Configure HFE Nodes in GCP

HFE Node Network Setup

The High-Availability Front End (HFE) node is a public-facing node that allows sub-second switchover between Active and Standby SBC instances of an HA pair, as it negates the need for any IP reassignment.

GCP requires each interface of a instance in a separate Virtual Private Network (VPC). Create a minimum of six VPCs for a full HFE set up (assuming all management interfaces for the SBC and the HFE node are in the same VPC).

HFE 2.1 Network Setup

HFE 2.1 has two HFE nodes, each responsible for a different type of traffic:

• Untrusted public traffic to the SBC (for PKT0). In this document, such a HFE node is referred to as "PKT0 HFE node".
• Trusted traffic from the SBC to other trusted networks (from PKT1). In this document, such a HFE node is referred to as the "PKT1 HFE node".

Both HFE nodes require three interfaces, as follows:

Supported OS for HFE

For HFE nodes, Ribbon supports the following operating systems:

• Ubuntu 18.04 LTS
• Debian 9/10
• CentOS 8
• Red Hat Enterprise Linux 8

Prerequisites for Creating the HFE Node

Ensure that the following are configured before creating the HFE node:

• Six VPC Networks. For more information, refer to Configure VPC Networks.
• GCE Service Account permission. For more information, refer to GCP Service Account Permissions.
• A GCP Bucket in Cloud Storage for the HFE script. For more information, refer to Create a Bucket in Cloud Storage for the HFE Script Upload.
• A Static External IP address reserved for the nic0. For more information, refer to the section "Reserve Static External IP Addresses" of the page Configure VPC Networks.
• When using HFE 2.1, create a cloud NAT gateway for the VPC used for nic0 of the PKT1 HFE node. This allows the HFE node to access the Google Servers to retrieve the script and query instance information. For more information, see Create a NAT Gateway.

Manual HFE Node Instance Creation

This section describes the manual creation of HFE nodes.

HFE 2.1 - Split HFE

1. Navigate to Compute Engine > VM instances.
2. Click CREATE INSTANCE.
3. Select Name, Region, and Zone.
4. Select Machine Type:
   • General-purpose
   • Series - N1

   • n1-standard-4

     Note

     The value for Machine Type are for illustration purposes only. For recommended configurations, refer to Instance Types Supported for SBC SWe in GCP.

     
     

     HFE 2.1 - Basic Configuration

     

5. To configure the OS, select Boot Disk - Change:
   1. Select a supported OS (for example, Ubuntu 19.04).

   2. Set the Size as 10 (GB).

      HFE 2.1 - Boot Disk

      

6. In Identity and API access - Service account, select the Service account created earlier.
7. In the Security tab, update the SSH Keys as to include:
   1. SSH Key for user: Insert ssh-rsa ... <user-name>.

   2. Check Block project-wide SSH keys.

      HFE 2.1 - Security

      

8. Update the Metadata to include the startup script for the HFE. For HFE 2.1, a copy of the startup script is available in GCE - HFE 2.1 Startup Script, with appropriate values for the following variables:
   • HFE_SCRIPT_LOCATION - The location of the HFE script stored in Google storage. For more information, refer to Create a Bucket in Google Cloud Storage for HFE Script Upload.
   • ACTIVE_SBC_NAME - Instance name of the Active SBC.
   • STANDBY_SBC_NAME - Instance name of the Standby SBC.
   • REMOTE_SSH_MACHINE_IP - The IP address of the remote machine to SSH from on the Management Interface. You can provide multiple IP addresses as a comma-separated list. For example, 10.0.0.1,10.0.0.2,10.0.0.3. 
     For more information, refer to Handling Multiple Remote SSH IPs to Connect to HFE Node.
   • ZONE - The Zone in which the SBCs are configured.
   • SBC_PKT_PORT_NAME - This tells the HFE if it is handling traffic for PKT0 or PKT1. The accepted values are PKT0 and PKT1. Use this variable only for HFE 2.1.
9. Update the Network Interfaces on the HFE by selecting the Networking tab.
10. Update the Network interfaces by configuring the three interfaces described in HFE 2.1 Network Set-up:
    1. For HFE PKT0 node, add Network interface for the Public Interface to receive traffic for SBC PKT0:
       1. Select the VPC created for the HFE 'Public' facing for PKT0 traffic.
       2. Select the Subnet created for the HFE 'Public' facing for PKT0 traffic.
       3. Set the Primary internal IP as Ephemeral (Automatic).
       4. Set the External IP as one of the static External IPs created earlier.

       5. Set IP forwarding to On.

          HFE 2.1 - HFE PKT0 Node - Public Interface - Network Interface - PKT0

          

    2. For HFE PKT1 node, add Network interface for Private interface to receive traffic for SBC PKT1:
       1. Select the VPC created for the HFE 'Private' facing for PKT1 traffic.
       2. Select the Subnet created for the HFE 'Private' facing for PKT1 traffic.
       3. Set the Primary internal IP as Ephemeral (Automatic).
       4. Set External IP as None.

       5. Set IP forwarding to On.

          HFE 2.1 - HFE PKT1 Node - Private Interface - Network Interface - PKT1

          

    3. Add Network interface for Management interface to HFE:
       1. Select the VPC created for SBC MGT0.
       2. Select Subnet which was created for SBC MGT0.
       3. Set the Primary internal IP as Ephemeral (Automatic).

       4. Set the External IP as Ephemeral (Automatic).

          HFE 2.1 - Management Interface - Network Interface - MGT0

          

    4. For PKT0 HFE node, add Network interface for Interface to communicate with SBC PKT0
       1. Select the VPC created for SBC PKT0.
       2. Select the Subnet created for SBC PKT0.
       3. Set the Primary internal IP as Ephemeral (Automatic).

       4. Set the External IP as None.

          HFE 2.1 - HFE PKT0 Node - Network Interface - PKT0

          

    5. For PKT1 HFE node, add Network interface for Interface to communicate with SBC PKT1
       1. Select the VPC created for SBC PKT1.
       2. Select the Subnet created for SBC PKT1.
       3. Set the Primary internal IP as Ephemeral (Automatic).

       4. Set the External IP as None.

          HFE 2.1 - HFE PKT1 Node - Network Interface - PKT1

          

11. Click CREATE.

Startup Script - Example

Refer to the following page for startup script examples:

• GCE - HFE 2.1 Startup Script

HFE Network Security Configuration

To use the HFE, add specific extra rules at the Google Network level. Ensure that routes and firewall rules are configured on the VPC networks containing the subnets in which PKT0 and PKT1 interfaces on the SBC are located. 

Google Network Routes

Create routes to send the traffic from PKT0 and PKT1 interfaces return through the HFE:

1. Go to VPC Networks.
2. Click on the VPC used for the PKT0 interface, and click Routes.
3. Click Add route:
   
   1. Provide a Name.
   2. Set Destination IP range to 0.0.0.0/0.
   3. Set Priority to anything under 1000 (The priority value for the default routes created). 

   4. Set an Instance tag. This is used to specify which instances use this rule.

      Note

      When creating an instance, set this tag as the Network Tag.

   5. Set the Next Hop using one of the following options:

      1. Specify an instance - Use an instance already created with an interface in this VPC. However, if the instance is deleted and another is created with the same name, traffic is routed to the new instance.

         Note

         To use the "Specify Instance" method, create the HFE instance before specifying it.

      2. Specify an IP address - Specify the private IP address of nic3 / nic4 (for the VPCs of PKT0 and PKT1 respectively)  interface on the HFE.

         Note

         You cannot edit VPC route rules on the Goggle Cloud Platform.

      Create a route

      

   6. Repeat for the VPC used for the SBC PKT1 interface.

Google Firewall Rules

By default, the Google network drops all packets unless firewall rules are configured. Ensure that rules are set for the VPCs to allow traffic from specific locations to reach the instances.

1. Go to VPC networks.
2. Click on the VPCs used for the PKT0 port on the SBCs, and click Firewall Rules.
3. There are two types of firewall required for using the HFE:
   
   1. An ingress and egress rule to allow all traffic (protocol and port) types from the source IP(s) of the traffic.
      1. Set Targets to "All instances in the network".
      2. The Source filter should be "IP ranges". 
      3. Set Source IP ranges as the source IPs for the traffic.

      4. Set Protocols and ports as "Allow all".

   2. An ingress and egress rule to allow all traffic from within the subnet to communicate (default):
      1. Set Targets to "All instances in the network".
      2. The Source filter should be "IP ranges". 
      3. Set Source IP ranges as the "subnet CIDR".
      4. Set Protocols and ports as "Allow all".

4. Repeat for the VPC used for the SBC PKT1 interface.

   VPC network details

   

Adding Custom Static Routes to HFE

For specialized deployments, users may need to add specific custom static routes to the HFE at the OS level. The HFE script supports this by using the HFE variable CUSTOM_ROUTES. It enables the HFE script to add these routes as part of its start-up process and verify these routes continue to be on the HFE throughout the uptime.

CUSTOM_ROUTES is a comma separated list of values in the form <DESTINATION_IP_CIDR>_<INTERFACE_NAME>. For example: 1.1.1.0/26_eth1, 2.2.2.0/28_eth2, 3.3.3.4/32_eth3.
To add the CUSTOM_ROUTES to the HFE startup-script, add the following line below /bin/echo "SBC_PKT_PORT_NAME=\"<SBC_PKT_PORT_NAME>\"" >> $NAT_VAR. For example:

/bin/echo "SBC_PKT_PORT_NAME=\"<SBC_PKT_PORT_NAME>\"" >> $NAT_VAR
/bin/echo "CUSTOM_ROUTES=\"<DESTINATION_IP_CIDR>_<INTERFACE_NAME>, <DESTINATION_IP_CIDR>_<INTERFACE_NAME>\"" >> $NAT_VAR
/bin/echo $(timestamp) "Copied natVars.input" >> $LOG_FILE

Creating a HFE Sysdump

The HFE_GCE.sh script (part of cloudTemplates.tar.gz) can create an archive of useful logs to help with debugging (similar to the SBC sysdump). Run the following command to collect the logs:

sudo /opt/HFE/HFE_GCE.sh sysdump

The following details are collected:

1. Output of:
   • Interfaces
   • Routes
   • IPtables
   • dmesg
   • conntrack count
   • conntrack extended list
   • The VM GCE metadata
   • journalctl errors
   • dhclient logs
   • System-networkd logs
2. The logs:
   • syslog
   • GCE startup-script
   • cloud-init logs
3. /opt/HFE/* (without previous sysdumps)
4. All user bash history

The sysdumps archives are stored in the .tar.gz format under /opt/HFE/sysdump/.

Enabling PKT DNS Support on HFE

The DNS queries on the SBC PKT port are sent using the primary IP. The HFE variable ENABLE_PKT_DNS_QUERY is used to enable the support for the HFE to forward these requests.

To enable it on a new HFE setup, add "ENABLE_PKT_DNS_QUERY=1" to the startup-script, below the SBC_PKT_PORT_NAME. For example: 

/bin/echo "SBC_PKT_PORT_NAME=\"<SBC_PKT_PORT_NAME>\"" >> $NAT_VAR
/bin/echo "ENABLE_PKT_DNS_QUERY=1" >> $NAT_VAR
/bin/echo $(timestamp) "Copied natVars.input" >> $LOG_FILE

HFE Node Logging

The HFE generates the following logs under /opt/HFE/log/:

• cloud-init-nat.log: Logs generated from the commands in the user-data script.
• HFE_conf.log: Logs generated from the set up of the HFE node. They contain information about:
  • SBC instance names
  • IPs for allowing SSH access of the HFE node
  • The configured zone
  • SBC IPs used to forward traffic
  • IP Tables rules
  • Routing rules
• HFE_conf.log.prev: A copy of the previous HFE_conf.log.
• HFE.log: 
  • Logs containing messages about any switchover action and connection errors. The logs generated are as follows:
    1. Connection error detected to Active SBC: <<IP>>. Attempting switchover.
       • Lost connection to the SBC. HFE node performs switchover action .
    2. Connection error ongoing - No connection to SBC PKT ports from HFE
       • This error means that the HFE node attempted a switchover, but no connection is established with new SBC.
       • The HFE node then continually switches between the SBCs until a connection is established.
       • This usually means there is a network issue or a configuration issue on the SBCs. 
    3. Switchover from old Active <<Old Active SBC IP>> to new Active <<New Active SBC IP>> complete. Connection established.
       • The switchover action is complete and connection is established to the Active SBC.
  • This log is rotated when it reaches 250 MB.
    
    • A maximum of four previous logs are saved.
    • The previous logs are compressed to save disk space.

Support for HFE node OS Upgrades and Security Patches

Ribbon tested the following upgrade scenarios on the HFE node, using ICMP packets to contact the PKT0/PKT1 ports on the SBC:

• Ubuntu 16.04 LTS > 18.04 LTS

• Ubuntu 19.04 > 19.10

  Note

  While updating Google-specific packages, the routes are removed. To restore them, reboot the instance after update.

  Ribbon recommends rebooting the HFE instance after installation or update of any package that affects networking.

• Debian 9 > Debian 10

• Installation of all updates available on CentOS 8.

  Note

  Ribbon does not support full OS upgrades on CentOS.

Create a NAT Gateway

Create a cloud NAT gateway for the VPC used by nic0 on the PKT1 HFE node. This allows the PKT1 HFE node to access the Google servers to retrieve the script and query instance information, and also prevents the instance from getting exposed to the outer world.

To create a cloud NAT gateway, perform the following steps:

1. In the GCP Console, navigate to Networking > Network services > Cloud NAT.
2. Click CREATE NAT GATEWAY.
3. Enter a Gateway name.
4. Select the VPC.
5. Select the Region
6. For Cloud Router, select Create New Router.
   1. Enter Name.
   2. Optionally, enter a Description.

   3. Click Create.

      Create a router

      

7. Retain the default NAT mapping.

8. Click Create.

   Create a NAT gateway