Upgrading SBC SWe Instances on OpenStack Using VNFM

You can upgrade the VM nodes in an SBC SWe VNF to a new version of software using VNFM if the nodes were instantiated using VNFM. Within VNFM, you specify the version of software to which you want to upgrade, select an upgrade method, and then upgrade the VM nodes beginning with the node that you determine currently has the standby role. To avoid service disruption you should only upgrade VM nodes when they are in standby mode.

The following procedure supports upgrading SBC N:1 HA deployments, where a maximum 4:1 HA deployment includes 5 VM instances (four active, one standby). However the process is similar regardless of the number of nodes in the cluster.

Prerequisites

Before initiating the upgrade ensure the following:

• If your deployment includes an Insight EMS system, upgrade it before upgrading OAM nodes (if present) and the SBC nodes. Refer to VNFM page, Upgrading a VNF, and EMS documentation for additional details.

• Download the required files for the new version of SBC software. Refer to the release notes for the new SBC software release. The required files include:
  • .qcow2 file that contains the SBC software
  • Script to generate a customized Cloud Service Archive (CSAR) package file
  • Virtual Network Function Descriptor (VNFD) template file to use with the script

• Run the script to generate a custom CSAR package file for your deployment. Refer to Creating a CSAR Package File. 
• Create an OpenStack Glance image using the SBC application software .qcow2 file. Refer to Creating a Glance Image.
• Onboard the CSAR package file you created for your deployment. Refer to Onboarding the SBC CSAR package file.
• Check the VNF Lifecycle window. The VNF you want to upgrade must be in Ready status, meaning that the status of each VM within it is Available.
• Ensure the pre-upgrade cluster configuration is saved to the EMS.  Note that for N:1 HA M-SBC upgrades, the Load Balancing Service (LBS) must be configured on the nodes prior to upgrading.
• Use the EMS to determine which node within the VNF currently has the standby role, or use the following CLI procedure: 
  1. Log into the CLI of one of the nodes. 
  2. Issue the command: show status system rgStatus 
  3. Check the output to determine which node shows its currentRole as standby.

Upgrading Deployments that Include OAM Nodes

If the deployment you are upgrading includes OAM nodes, VNFM instantiates the OAM nodes in their own cluster, separate from the redundancy groups containing the SBC nodes the OAM nodes manage. When upgrading the deployment, upgrade the OAM nodes in the VNF first, and then upgrade the SBC nodes. Typically, the OAM nodes are deployed in a 1:1 HA configuration including an active and a standby OAM node. Upgrade the standby OAM node first. After upgrade of the standby OAM completes, manually switchover the active OAM node (request system admin <system name> switchover) to switch roles. Then, upgrade the new standby (former active) OAM node. Once you complete upgrade of the OAM nodes you can continue with upgrading the SBC nodes.

Monitoring Node Status During Upgrade

Before and during the upgrade procedure, use the SBC CLI to check the status of the nodes in the redundancy group (RG) you are upgrading:

1. Log into the CLI of one of the nodes. 
2. Issue the command: show status system rgStatus and check the output to:
   • Determine which node shows its assignedRole as standby. This is the node to upgrade first.
   • Verify that the syncStatus of all the currently active SBC VM(s) is syncCompleted.
   • Verify that the syncStatus of the current standby SBC VM is unprotectedRunningStandby.

The following is an example of the type of output that would appear in a full 4:1 HA deployment. Smaller deployments would include fewer node entries. In a 1:1 HA configuration, the output would include entries for one active and one standby node, similar to the first two entries in the example.

> show status system rgStatus
rgStatus vsbc1-192.168.2.3 {
    actualCeName    vsbc1;
    assignedRole    active;
    currentRole     active;
    nodeId          1;
    serviceId       0;
    syncStatus      syncCompleted;
    usingMetavarsOf vsbc1-192.168.2.3;
    appVersion      V08.00.00;
}
rgStatus vsbc1-192.168.2.4 {
    actualCeName    vsbc1;
    assignedRole    standby;
    currentRole     standby;
    nodeId          2;
    serviceId       1;
    syncStatus      unprotectedRunningStandby;
    usingMetavarsOf vsbc1-192.168.2.4;
    appVersion      V08.00.00;
}
rgStatus vsbc1-192.168.2.5 {
    actualCeName    vsbc1;
    assignedRole    active;
    currentRole     active;
    nodeId          3;
    serviceId       2;
    syncStatus      syncCompleted;
    usingMetavarsOf vsbc1-192.168.2.5;
    appVersion      V08.00.00;
}
rgStatus vsbc1-192.168.2.6 {
    actualCeName    vsbc1;
    assignedRole    active;
    currentRole     active;
    nodeId          4;
    serviceId       3;
    syncStatus      syncCompleted;
    usingMetavarsOf vsbc1-192.168.2.6;
    appVersion      V08.00.00;
}
rgStatus vsbc1-192.168.2.7 {
    actualCeName    vsbc1;
    assignedRole    active;
    currentRole     active;
    nodeId          5;
    serviceId       4;
    syncStatus      syncCompleted;
    usingMetavarsOf vsbc1-192.168.2.7;
    appVersion      V08.00.00;
}

Throughout the upgrade process, maintain a CLI session in one of the VMs to monitor rgStatus. When it is time to upgrade the VM where you are monitoring rgStatus, start a CLI session in an upgraded VM to continue monitoring rgStatus. While upgrading M-SBC deployments it is also helpful to keep track of which node is the leader node for the cluster by issuing the command: 
show table system loadBalancingService leaderStatus

The command output lists all of the active nodes in the cluster and the leader node is the first node listed, for example:

> show table system loadBalancingService leaderStatus
LEADER  LEADER
ID      IPADDRESS  STATE
--------------------------
0 192.168.10.6 completed
1 192.168.10.10 completed
[ok][2018-11-15 21:52:51]

Upgrade Procedure

The upgrade procedure on VNFM occurs in two phases. First you upgrade the VNF, during which you specify the software version to which you want to upgrade. Then you upgrade the individual VM nodes (VNFCs) to the specified software version.

Upgrading a VNF

Use the following procedure to upgrade a VNF. 

1. Perform the procedure Performing Onboarding Through the UI. You must add a new version of the VNF to the VNF catalog.

2. Click VNF Lifecycle to display all VNFs and VNFCs.

3. Select the VNF you want to upgrade from the VNF Lifecycle panel.

4. Select Upgrade from the Select Action drop-down menu for the selected VNF.

   Note

   In the VNF Lifecycle panel, the status of the VNF you want to upgrade must be Ready. The Upgrade action is only available if the VNF is in the Ready state.

   
   

   VNF Ready to upgrade

   
   The Upgrade VNF window appears.

   Upgrade VNF

   

5. Provide a reason for the upgrade (optional). This information is included in logs and passes to the VNFs during the upgrade. 

6. Select the upgrade version from the Upgrade Version drop-down menu.

7. Click Save.
   The VNF status in the VNF Lifecycle panel updates to Upgrading.

8. Continue with the Upgrading a VNFC procedure to upgrade each VM node, one by one.
   After you upgrade each VNFC, the VNF status is Upgraded.

Upgrading a VNFC

Before you can upgrade a VNFC, you must complete the "Upgrading a VNF" procedure that specifies the software version. After completing that procedure, the status for the individual nodes (VNFCs) within the VNF appears as Upgrade Required. The Upgrade action only displays if the VNF status is Upgrading and the VNFC status is Upgrade Required.

Use the following procedure to upgrade a VNFC.

1. Log into the VNFM UI. Refer to Logging into the VNFM UI.
2. Click VNF Lifecycle to display all VNFs and all VNFCs.
3. Select the VNF you are upgrading in the VNF Lifecycle panel. The nodes within the selected VNF are listed below in the VNFC Lifecycle panel.
4. In the VNFC Lifecycle panel, select the node you previously determined to be the standby node. The standby node must be upgraded first.
5. Log into the CLI of the node you are upgrading and issue the command: sbxstop 
Note: Make sure the process to stop the SBC has completed before continuing.

6. Select Upgrade from the Select Action drop-down menu in the VNFC Lifecycle panel for the node you are upgrading.

   Upgrade VNFC

   
   The Upgrade VNFC window appears.

7. Select the upgrade option. Note that if the VM has an attached Cinder boot volume, the Rebuild option is not offered, you must use Recreate (reuse).

   • Rebuild - upgrades the VM using the upgrade image. The VM UUID will not change. Not available if the VM has an attached Cinder boot volume.
   • Recreate (reuse) - re-creates the VM instance using the upgrade image. The VM UUID will change.

8. Click Upgrade. VNFM initiates the upgrade.  

9. After the VM status changes to "Upgraded" and the node is back in service, determine the next node to upgrade. In a 1:1 HA deployment, this will be the currently active node. In an N:1 HA deployment, select one of the active nodes.  Before upgrading an instance, check the syncStatus field in the output of the rgStatus command to ensure that all the currently active instances are in the syncCompleted state. The standby instance displays syncStatus as unprotectedRunningStandby. 

10. Repeat steps 5 through 8 for the active node in a 1:1 HA deployment. In an N:1 HA deployment, repeat steps 5 through 8 for each active node, one by one, until all VMs are upgraded. 
    The VNF status in the VNF Lifecycle panel should be Upgrade in Progress until the VNFC upgrades are complete.  After each node is upgraded, its status changes to Upgraded. After the last node in the VNF is upgraded, the status of the active node(s) changes to Available. The status of the standby node changes to Maintenance for a short time before it comes into service and then it also becomes Available.
    The VNF status in the VNF Lifecycle panel then changes to Ready. 

Post-Upgrade Verification

To verify that the nodes instances are up and running with the upgraded software version:

1. Log into the CLI of one of the nodes.
2. Issue the command: show status system rgStatus 

3. In the output, check that the value for syncStatus for the currently active instance is syncCompleted. The value for the standby instance should appear as unprotectedRunningStandby.