Page History

This section describes the extra steps (in addition to the Standalone SBC) necessary for creating a HFE/SBC on Azure. All commands used in this section is part of the Azure CLI.

HFE Node Network Setup

HFE nodes allow sub-second switchover between SBCs of an HA pair, as they negate the need for any IP reassignment. In the Microsoft Azure environment.

Azure requires that each interface of an instance is in a separate subnet, but within the same virtual network. Assuming all Mgmt interfaces are from the same subnet and HFE interfaces to the associated SBC PKT share a subnet, a minimum of six subnets are necessary for a full HFE setup.

Configure the HFE nodes in one of two ways:

• Use custom data on cloud-init enabled distributions.
• Use HFE_AZ_manual_setup.sh script. For more information, see HFE Node Initial Configuration.

Anchor
hfe_node_network_setup_hfe2-1 hfe_node_network_setup_hfe2-1

HFE 2.1

In HFE 2.1, there are two HFE nodes - one to handle untrusted public traffic to the SBC (for PKT0,) and the other to handle trusted traffic from the SBC to other trusted networks (from PKT1). In this section, the HFE node handling untrusted traffic is referred to as the "PKT0 HFE node", and the HFE node handling trusted traffic as the "PKT1 HFE node".

Both HFE nodes require 3 interfaces, as described below:

Steps to Create SBC HA with HFE Setup

To create the SBC HA with HFE, perform the following steps:

1. Install and login to the Azure CLI.
2. Create Resource Group and Network with six subnets.
3. Configure the Storage Account.
4. Create the User Assigned Managed Identity.
5. Create the HFE Node(s).
6. If using a non cloud-init enabled image, run the manual setup script. See HFE Node Initial Configuration.
7. Understand the extra steps necessary for the SBC creation in SBCs' Userdata.
8. Create two SBCs following the instructions in the sections Create SBC and SBCs' Userdata.
   
   

Resources for HFE Setup

To create HFE setup, use the HFE Azure Shell Script and the HFE Azure Manual Setup Shell Script, included in the cloudTemplates.tar.gz and called HFE_AZ.sh and HFE_AZ_manual_setup.sh.

HFE Azure User Data

Configure the Storage Account

The script HFE_AZ.sh is stored in a container within a storage account. This allow the HFE nodes to download and run the script during the VM startup.

To configure the storage account, perform the following steps:

1. Create a storage account by executing the following command:

   Code Block
   title Syntax
   az storage account create --name <NAME> --resource-group <RESOURCE_GROUP_NAME> --kind storageV2

   
   

   Code Block
   title Example
   az storage account create --name rbbnhfestorage --resource-group RBBN-SBC-RG --kind storageV2

   
   

2. Create a container by executing the following command:

   Code Block
   title Syntax
   az storage container create --name <NAME> --account-name <STORAGE ACCOUNT NAME> --public-access blob --auth-mode key

   
   

   Code Block
   title Example
   az storage container create --name hfescripts --account-name rbbnhfestorage --public-access blob --auth-mode key

   
   

3. Upload the script HFE_AZ.sh to the container by executing the following command:

   Code Block
   title Syntax
   az storage blob upload --name <NAME> --file <HFE_AZ.sh> --container-name <CONTAINER NAME> --account-name <STORAGE ACCOUNT NAME>

   
   

   Code Block
   title Example
   az storage blob upload --name HFE_AZ.sh --file /tmp/HFE_AZ.sh --container-name hfescripts --account-name rbbnhfestorage

   
   

4. Make the storage account accessible for the instances by allowing access to virtual machines in both subnets used for ETH0 on the HFE node (ensure that the subnet exists).

   Code Block
   title Syntax
   az storage account network-rule add --account-name <STORAGE ACCOUNT NAME> --subnet <SUBNET NAME of SUBNET USED FOR ETH0 of HFE NODE> --vnet-name <VIRTUAL NETWORK NAME>

   
   

   Code Block
   title Example
   az storage account network-rule add --account-name rbbnhfestorage --subnet hfepublic --vnet-name RibbonNet

   
   

HFE Node Initial Configuration

You can perform the initial configuration of the HFE node(s) in two ways:

• Using custom-data and cloud-init.
• Using the script HFE_AZ_manual_setup.sh.

The list of cloud-init enabled Linux VMs is available in Microsoft Azure Documentation.

HFE Variables

The HFE has variables that are required to be updated. When using cloud-init, update the the HFE variables in the custom data.

For manual setup, update the script HFE_AZ_manual_setup.sh (the portion of the script below the comment: UPDATE VARIABLES IN THIS SECTION).

The following table contains the values that you must update:

<HFE_SCRIPT_LOCATION>

<ACTIVE_SBC_NAME>

<STANDBY_SBC_NAME>

<REMOTE_SSH_MACHINE_IP>

<SBC_PKT_PORT_NAME>

Updating HFE Variables

Azure does not support updating Custom Data after a VM is created. To update an HFE variable, use the following procedure:

1. Log on to the HFE node as a Ribbon user.

2. Enter the updated variable to /opt/HFE/natVars.user. For example:

   Code Block
   echo "REMOTE_SSH_MACHINE_IP=\"10.27.0.54,10.36.9.6\"" | sudo tee -a /opt/HFE/natVars.user

   
   

3.  Reboot the HFE: 

   Code Block
   sudo reboot

   
   

   Info
   title Note
   Any variable added to /opt/HFE/natVars.user will overwrite the values set as the variables in custom data. To add in a new Remote SSH Machine IP, ensure to supply the full list of IPs you wish the routes to be created for.

   
   

Supported Images

The following images are generally supported for using as the HFE:

Cloud-init configuration

• Ubuntu 18.04

Manual configuration

• CentOS 7
• CentOS 8
• RHEL 7
• RHEL 8
• Debian 10

Custom Data Example

An example of the custom data for a HFE node is given below:

Manual Configuration

The script HFE_AZ_manual_setup.sh has two functions:

• It creates the systemd service "ribbon-hfe" and enables the service.
• Systemd runs it to download the script and write the variables out to /opt/HFE/natVars.input, similar to the role of custom-data does in the cloud-init. As the script is run as a service by systemd, it will automatically run if the instance reboots.

The steps required to initially configure the HFE node using the script HFE_AZ_manual_setup.sh are as follows:

1. Using SCP, upload the script HFE_AZ_manual_setup.sh onto the instance, in a file path that has executable permissions for the root.

2. Run the script with heightened permissions and the '-s' flag. For example:

   Code Block
   sudo /usr/sbin/HFE_AZ_manual_setup.sh -s

   
   

   Tip
   title Tip
   When you use the '-s' flag, systemd points at the location of the script. If you remove the file, run the script again with the '-s' flag.

   
   

3. Start the service by executing the following command:

   Code Block
   sudo systemctl start ribbon-hfe

   
   

Create HFE Nodes

To create HFE node(s), perform the steps described below.

Create Public IPs

Create at least one Public IP for ETH0 of the PKT0 HFE Node. Optionally, create up to two additional Public IPs to access the MGMT interfaces on both HFE nodes. For more information, refer to Create Public IPs (Standalone).

Create NICs

To create NICs, use the following command syntax:

az network nic create --name <NIC NAME>
                      --resource-group <RESOURCE-GROUP-NAME>
                      --vnet-name <VIRTUAL NETWORK NAME>
                      --subnet <SUBNET NAME>
                      --network-security-group <SECURITY GROUP NAME> 

HFE 2.1

For HFE 2.1, create a total of six NICs (three for each interface).

The following table contains the extra flags necessary for each interface:

Create the VM

To create the VM(s), use the following command syntax:

az vm create --name <INSTANCE NAME>
             --resource-group <RESOURCE_GROUP_NAME>
             --admin-username <UserName>
             --custom-data <USERDATA FILE>
             --image <IMAGE NAME>
             --location <LOCATION>
             --size <INSTANCE SIZE>
             --ssh-key-values <PUBLIC SSH KEY FILENAME>
             --nics <ETH0 NIC> <ETH1 NIC> <ETH2 NIC>
             --boot-diagnostics-storage <STORAGE ACCOUNT NAME>
             --assign-identity <USER ASSIGNED MANAGED IDENTITY ID>

The following table describes each flag:

HFE Routing 

The HFE setup requires routes in Azure to force all the traffic leaving PKT0 and PKT1 to route back through the HFE.

To create the routes, perform the following steps:

1. Create the route-table:
   

   Code Block
   title Syntax
   az network route-table create --name <NAME> --resource-group <RESOURCE_GROUP_NAME>

   
   

   Code Block
   title Example
   az network route-table create --name hfe-route-table --resource-group RBBN-SBC-RG

   
   

2. Create two rules for PKT0 and PKT1:
   

   Code Block
   title Syntax
   az network route-table route create --name <NAME> --resource-group <RESOURCE_GROUP_NAME> --address-prefix <CIDR OF ENDPOINT> --next-hop-type VirtualAppliance --route-table-name <ROUTE TABLE NAME> --next-hop-ip-address <IP FOR ETH3/ETH4 of HFE NODE>

   
   

   Code Block
   title Example
   az network route-table route create --name pkt0-route --resource-group RBBN-SBC-RG --adress-prefix 77.77.173.255/32 --next-hop-type VirtualAppliance --route-table-name hfe-route-table --next-hop-ip-address 10.2.6.5

   
   

3. Attach the route table to the PKT0/PKT1 subnets:
   

   Code Block
   title Syntax
   az network vnet subnet update --name <SUBNET NAME> --resource-group <RESOURCE_GROUP_NAME> --vnet-name <VIRTUAL NETWORK NAME> --route-table <ROUTE TABLE NAME>

   
   

   Code Block
   title Example
   az network vnet subnet update --name pkt0 --resource-group RBBN-SBC-RG --vnet-name RibbonNet --route-table pkt0-route

   
   

Additional Steps for SBC HFE Setup

To create the SBC HA with HFE setup, first perform all of the steps described in Create SBC (Standalone).

In addition to those steps, perform the steps described below.

Configure NICs

The SBC requires 4 NICs, each one attached to a individual subnet for MGMT, HA, PKT0 and PKT1.

To create a standard NIC, use the following syntax:

az network nic create --name <NIC NAME>
                      --resource-group <RESOURCE GROUP NAME>
                      --vnet-name <VIRTUAL NETWORK NAME>
                      --subnet <SUBNET NAME>
                      --network-security-group <SECURITY GROUP NAME>
                      --accelerated-networking <true/false>

See below for additional steps for when SBCs are in a HFE setup.

Secondary IPs

The HA SBCs require configuring Secondary IPs on both the PKT0 and PKT1 ports for both the Active and the Standby instances.

Create and attach Secondary IPs to a network interface by executing the following command:

az network nic ip-config create --name <NAME> --nic-name <PKT0/PKT1 NIC NAME> --resource-group <RESOURCE_GROUP_NAME>

az network nic ip-config create --name sbc1-pkt0-secIp --nic-name sbc1-pkt0 --resource-group RBBN-SBC-RG

Create NIC for PKT0 and PKT1

When creating the NICs for both SBC's PKT0 and PKT1 ports, include the flag --ip-forwarding for receiving the traffic sent to the HFE node. For example:

az network nic create --name sbc1-pkt0 --resource-group RBBN-SBC-RG --vnet-name RibbonNet --subnet pkt0 --network-security-group RbbnSbcSG --ip-forwarding

SBC Userdata

The SBCs in the HFE environment require the following user data: 

Create a JSON file using the following structure:

{
  "CEName" : "<SBC CE NAME>",
  "ReverseNatPkt0" : "True",
  "ReverseNatPkt1" : "True",
  "SystemName" : "<SYSTEM NAME>",
  "SbcPersonalityType": "isbc",
  "AdminSshKey" : "<ssh-rsa ...>",
  "ThirdPartyCpuAlloc" : "<0-4>",
  "ThirdPartyMemAlloc" : "<0-4096>",
  "CERole" : "<ACTIVE/STANDBY>",
  "PeerCEHa0IPv4Address" : "<PEER HA IP ADDRESS>",
  "ClusterIp" : "<PEER HA IP ADDRESS>",
  "PeerCEName" : "<PEER SBC CE NAME>",
  "SbcHaMode" : "1to1",
  "PeerInstanceName" : "<PEER INSTANCE NAME>",
  "Pkt0HfeInstanceName" : "<PKT0 HFE NODE INSTANCE NAME>",
  "Pkt1HfeInstanceName" : "<PKT1 HFE NODE INSTANCE NAME>"
}

Sample Meta Variable Table

Example Meta Variable table for a SBC HA is provided below:

admin@act-10.2.2.127> show table system metaVariable
CE NAME         NAME                  VALUE
-----------------------------------------------------
act-10.2.2.127  IF0.GWV4              10.2.0.1
act-10.2.2.127  IF0.IPV4              10.2.0.9
act-10.2.2.127  IF0.Port              Mgt0
act-10.2.2.127  IF0.RNat              True
act-10.2.2.127  IF1.GWV4              10.2.2.1
act-10.2.2.127  IF1.IPV4              10.2.2.127
act-10.2.2.127  IF1.Port              Ha0
act-10.2.2.127  IF1.RNat              True
act-10.2.2.127  IF2.GWV4              10.2.3.1
act-10.2.2.127  IF2.IPV4              10.2.3.10
act-10.2.2.127  IF2.Port              Pkt0
act-10.2.2.127  IF2.RNat              True
act-10.2.2.127  IF3.GWV4              10.2.4.1
act-10.2.2.127  IF3.IPV4              10.2.4.10
act-10.2.2.127  IF3.Port              Pkt1
act-10.2.2.127  IF3.RNat              True
act-10.2.2.127  IF0.FIPV4             137.117.73.22
act-10.2.2.127  IF0.PrefixV4          24
act-10.2.2.127  IF1.PrefixV4          24
act-10.2.2.127  IF2.PrefixV4          24
act-10.2.2.127  IF3.PrefixV4          24
act-10.2.2.127  HFE_IF2.FIPV4         52.168.34.216
act-10.2.2.127  HFE_IF3.FIPV4         10.2.2.7
act-10.2.2.127  HFE_IF2.IFName        IF_HFE_PKT0
act-10.2.2.127  HFE_IF3.IFName        IF_HFE_PKT1
act-10.2.2.127  secondaryIPList.Pkt0  ['10.2.3.10']
act-10.2.2.127  secondaryIPList.Pkt1  ['10.2.4.10']
sby-10.2.2.227  IF0.GWV4              10.2.0.1
sby-10.2.2.227  IF0.IPV4              10.2.0.14
sby-10.2.2.227  IF0.Port              Mgt0
sby-10.2.2.227  IF0.RNat              True
sby-10.2.2.227  IF1.GWV4              10.2.2.1
sby-10.2.2.227  IF1.IPV4              10.2.2.227
sby-10.2.2.227  IF1.Port              Ha0
sby-10.2.2.227  IF1.RNat              True
sby-10.2.2.227  IF2.GWV4              10.2.3.1
sby-10.2.2.227  IF2.IPV4              10.2.3.10
sby-10.2.2.227  IF2.Port              Pkt0
sby-10.2.2.227  IF2.RNat              True
sby-10.2.2.227  IF3.GWV4              10.2.4.1
sby-10.2.2.227  IF3.IPV4              10.2.4.10
sby-10.2.2.227  IF3.Port              Pkt1
sby-10.2.2.227  IF3.RNat              True
sby-10.2.2.227  IF0.FIPV4             40.76.8.39
sby-10.2.2.227  IF0.PrefixV4          24
sby-10.2.2.227  IF1.PrefixV4          24
sby-10.2.2.227  IF2.PrefixV4          24
sby-10.2.2.227  IF3.PrefixV4          24
sby-10.2.2.227  HFE_IF2.FIPV4         52.168.34.216
sby-10.2.2.227  HFE_IF3.FIPV4         10.2.2.7
sby-10.2.2.227  HFE_IF2.IFName        IF_HFE_PKT0
sby-10.2.2.227  HFE_IF3.IFName        IF_HFE_PKT1
sby-10.2.2.227  secondaryIPList.Pkt0  ['10.2.3.11']
sby-10.2.2.227  secondaryIPList.Pkt1  ['10.2.4.11']
[ok][2019-10-07 11:48:16]
admin@act-10.2.2.127>

Add New Endpoints to UAC

To add a new end point to the Public Endpoint side with HFE1 (for example, 52.52.52.52 is the new end point IP):

1. Add the end point IP to outbound security group.

   Caption
   0 Figure 1 Add outbound security rule

   
   

2. Add the end point IP to the PKT0 subnet custom route table. Name of the route table will be $instanceBaseName.

   Caption
   0 Figure 1 UAC - Route Table - Add Route

   
   Select Next hop type of Virtual Appliance, and the Next hop address as $hfePkt0OutIp.
   
   

3. Add the end point IP to the Inbound Security Rule of the Security group of nic1 of HFE1, and PKT0 of the SBC.

Add New Endpoints to UAS

Add the IP (for example, 10.2.3.9) to PKT1 subnet custom route table. The name of the route table is $instanceBaseName.

Select Next hop type of Virtual Appliance, and the Next hop address as $hfePkt1OutIp.

HFE Node Logging

The HFE generates the following logs under /opt/HFE/log/:

• cloud-init-nat.log: Logs generated from the initial configuration.
• HFE_conf.log: Logs generated from the setup of the HFE node. They contain information about:
  • SBC instance names
  • The IPs for allowing SSH into the HFE node
  • The configured zone
  • The SBC IPs being used to forward traffic to
  • Iptables rules
  • Routing rules
• HFE_conf.log.prev: A copy of the previous HFE_conf.log.
• HFE.log: 
  • Logs which contain messages about any switchover action, as well as connection errors. The logs generated are as follows:
    1. Connection error detected to Active SBC: <<IP>>. Attempting switchover.
       • We have lost connection to the SBC. HFE node now performing switchover action 
    2. Connection error ongoing - No connection to SBC PKT ports from HFE
       • This error means that a switchover has been attempted, but no connection could be established to the new SBC.
       • The HFE node then continually switches between the SBCs until a connection is established
       • This usually means there is a network issue or a configuration issue on the SBCs.
    3. Switchover from old Active <<Old Active SBC IP>> to new Active <<New Active SBC IP>> complete. Connection established.
       • The switchover action is complete and connection has been established to the 'Active' SBC
    4. Initial HFE startup configuration complete. Successfully connected to <<SBC Instance Name>>
       
       • The HFE node has successfully connected to the active SBC following a boot.
  • This log is rotated when it reaches 250MB:
    • Up to four previous logs are saved.
    • The previous logs are compressed to save disk space.

Adding Custom Static Routes to HFE

For specialized deployments, users may need to add specific custom static routes to the HFE at the OS level. The HFE script supports this by using the HFE variable CUSTOM_ROUTES. It enables the HFE script to add these routes as part of its start-up process and verify these routes continue to be on the HFE throughout the uptime.

CUSTOM_ROUTES is a comma separated list of values in the form <DESTINATION_IP_CIDR>_<INTERFACE_NAME>. For example: 1.1.1.0/26_eth1, 2.2.2.0/28_eth2, 3.3.3.4/32_eth3.
To add the CUSTOM_ROUTES to the HFE customData, add the following line below /bin/echo "SBC_PKT_PORT_NAME=\"<SBC_PKT_PORT_NAME>\"" >> $NAT_VAR. For example:

/bin/echo "SBC_PKT_PORT_NAME=\"<SBC_PKT_PORT_NAME>\"" >> $NAT_VAR
/bin/echo "CUSTOM_ROUTES=\"<DESTINATION_IP_CIDR>_<INTERFACE_NAME>, <DESTINATION_IP_CIDR>_<INTERFACE_NAME>\"" >> $NAT_VAR
/bin/echo $(timestamp) "Copied natVars.input" >> $LOG_FILE

If the HFE is already deployed, the variable is added to /opt/HFE/natVars.user. For example:

echo "CUSTOM_ROUTES=\"<DESTINATION_IP_CIDR>_<INTERFACE_NAME>, <DESTINATION_IP_CIDR>_<INTERFACE_NAME>\"" | sudo tee -a /opt/HFE/natVars.user

Creating a HFE Sysdump

The HFE_AZ.sh script can create an archive of useful logs to help with debugging (similar to the SBC sysdump). Run the following command to collect the logs:

sudo /opt/HFE/HFE_AZ.sh sysdump

The following details are collected:

1. Output of:
   • Interfaces
   • Routes
   • IPtables
   • dmesg
   • conntrack count
   • conntrack extended list
   • The VM Azure metadata
   • journalctl errors
   • dhclient logs
   • System-networkd logs
2. The logs:
   • syslog
   • waagent logs
   • cloud-init logs
3. /opt/HFE/* (without previous sysdumps)
4. All user bash history

The sysdumps archives are stored in the .tar.gz format under /opt/HFE/sysdump/.

Handling Multiple Remote SSH IPs to Connect to HFE Node

The following section contains the instructions to set multiple SSH IPs to access the HFE node as well as to update the instances to add in more SSH IPs.

Initial Orchestration

During orchestration, you can supply multiple IP addresses to the appropriate variable with a common separated list. For example, 10.0.0.1, 10.0.0.2, and 10.0.0.3. The following table represents the list of variables that need to be set for each orchestration type:

Updating Remote SSH IPs

The following steps describe the procedure to update the Remote SSH IPs on the Azure.

Azure

Azure does not support updating Custom Data after a VM is created. To update a HFE variable, use the following procedure:

1. Log on to the HFE node as a rbbn user.

2. Enter the updated variable to/opt/HFE/natVars.user. For example: 

   Code Block
   echo "REMOTE_SSH_MACHINE_IP=\"10.27.0.54,10.36.9.6\"" | sudo tee -a /opt/HFE/natVars.user

   
   

3. Reboot the HFE: 

   Code Block
   sudo reboot

   
   

   Info
   title Note
   Any variable added to /opt/HFE/natVars.user will overwrite the values set as the variables in custom data. Ensure you enter the complete list of IPs for which the routes need to be created.

   
   

Enabling PKT DNS Support on HFE

The DNS queries on the SBC PKT port are sent using the primary IP. The HFE variable ENABLE_PKT_DNS_QUERY is used to enable the support for the HFE to forward these requests correctly. 

To enable the PKT DNS Support on a new HFE setup, add "ENABLE_PKT_DNS_QUERY=1" to the customData, below SBC_PKT_PORT_NAME.

Example:

/bin/echo "SBC_PKT_PORT_NAME=\"<SBC_PKT_PORT_NAME>\"" >> $NAT_VAR
/bin/echo "ENABLE_PKT_DNS_QUERY=1" >> $NAT_VAR
/bin/echo $(timestamp) "Copied natVars.input" >> $LOG_FILE

To enable the PKT DNS Support option on an already configured HFE setup:

1. Log on to the HFE node as a RBBN user.

2. Add the natvar ENABLE_PKT_DNS_QUERY to /opt/HFE/natVars.user with the value 1.

   Code Block
   echo "ENABLE_PKT_DNS_QUERY=1" | sudo tee -a /opt/HFE/natVars.user

   
   

3. Reboot the HFE.

   Code Block
   sudo reboot