Live Software Upgrade Using CLI

In this section:

Live Software Upgrade (LSWU) allows you to upgrade the SBC Core application in an HA environment without dropping active calls or interrupting service. This upgrade approach is recommended for High Availability SBC systems.

For information on average call rates and capacities during an LSWU, refer to LSWU Performance Metrics.

Prerequisites

Note: If you are upgrading from 5.x/6.0.x/7.0.x ...

Before upgrading to Release 7.1.0 release, you must have no sessions in the /var/log/sonus/evlog/ or /var/log/sonus/evlog/evlog/ directories.

(This Note does not apply if you are upgrading from 6.2.2, 7.1.0 or later versions.)

Check the version of firmware on both the servers. To determine the required firmware version, refer to SBC Release Information.
For more information, see Upgrading Firmware for guidance.

Verify the SBC platform is running in redundant mode using the following CLI command:

Example

admin@WFDSBC01a> show table system serverStatus
																		MGMT																				DAUGHTER
										PLATFORM		APPLICATION   	REDUNDANCY						APPLICATION UP		LAST RESTART					BOARD
NAME      HW TYPE SERIAL NUM PART NUM  VERSION			VERSION      	ROLE		UP TIME				TIME				REASON			SYNC STATUS		PRESENT
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
WFDSBC01a SBC 5200 2044090074 821-00430 V07.00.00R000 	V07.00.00R000 	active		6 Days	15:32:47	6 Days 15:31:04		systemRestart	syncCompleted	true
WFDSBC01b SBC 5200 2052090007 821-00430 V07.00.00R000 	V07.00.00R000 	standby		6 Days	15:32:47	6 Days 15:31:04		systemRestart	syncCompleted	true

Verify the sync status using the following CLI command.

Example

admin@WFDSBC01a> show table system syncStatus
SYNC MODULE                  STATUS
-------------------------------------------
Policy Data                 syncCompleted
Disk Mirroring              syncCompleted
Configuration Data          syncCompleted
Call/Registration Data      syncCompleted

Verify that at least one management port is in admnEnabledPortUp state on each server using the following CLI command:

Example

admin@WFDSBC01a> show table system ethernetPort mgmtPortStatus
CE   		PORT IF 											RX		TX		RX		TX		RX		TX
NAME 		NAME INDEX 	MAC ADDRESS 		LINK STATE			PACKETS	PACKETS	ERRORS	ERRORS	DROPPED	DROPPED
---------------------------------------------------------------------------------------------------------------------------
WFDSBC01a 	mgt0 1 		0:10:6b:2e:e6:9e 	admnEnabledPortUp	1728556	132270	0		0		0		0
WFDSBC01a 	mgt1 2 		0:10:6b:2e:e6:9f 	admnEnabledPortUp	6754292	6480	0		0		0		0
WFDSBC01b 	mgt0 3 		0:10:6b:2e:e5:ea 	admnEnabledPortUp	3457239	0		0		0		0		0
WFDSBC01b 	mgt1 4 		0:10:6b:2e:e5:eb 	admnEnabledPortUp	1038353	0		0		0		0		0

Verify the system call arrival rate (systemCongestionCallArrivalRate) using the following CLI command. The maximum call rate allowed for LSWU differs between various SBC platforms. For more information, refer to LSWU Performance Metrics page.

Example

admin@WFDSBC01a> show status system systemCongestionStatus
systemCongestionStatus entry {
	systemCongestionMCLevel				0;
	systemCongestionCPULevel			0;
	systemCongestionMemLevel			0;
	systemCongestionCallRateLevel		0;
	systemCongestionMCDuration			67397;
	systemCongestionCallArrivalRate		355;
	systemCongestionCallAcceptRate		100;
	systemCongestionCallAcceptCount		0;
	systemCongestionCallEqArrivalRate 	355;
	systemCongestionRegArrivalRate		0;
	systemCongestionIRTTLevel			0;
}
[ok]

Set the log filter levels (in configuration mode) to major using the following CLI command.

Example

admin@WFDSBC01a% set oam eventLog typeAdmin debug filterLevel major
admin@WFDSBC01a% set oam eventLog typeAdmin system filterLevel major
admin@WFDSBC01a% commit

Delete the LI configuration before performing the LSWU.

Note

If you are not using Lawful Intercept (LI), ignore this step.

Log on as Calea user with Calea credentials.

Check the current LI state using the command:

 > show table addressContext default intercept callDataChannel <callDataChannel_Name>
interceptStandard    packetcable;
vendorId             none;
priState             enabled;
secState             disabled;
priMode              active;
secMode              outOfService;
priIpAddress         0.0.0.0;
secIpAddress         0.0.0.0;
priPort              0;
kaTimer              5;
retries              3;
ipInterfaceGroupName IpIntGrp1;
[ok]

Delete the LI Configuration using the commands:

Prior to deleting an LI Configuration, set the following flags to:

priMode - outOfService

priState - disabled

secMode - outOfService

secState - disabled

% set addressContext default intercept callDataChannel <callDataChannel_Name> priMode outOfService
% commit

% set addressContext default intercept callDataChannel <callDataChannel_Name> secMode outOfService
% commit

% set addressContext default intercept callDataChannel <callDataChannel_Name> priState disabled
% commit

% set addressContext default intercept callDataChannel <callDataChannel_Name> secState disabled
% commit

% delete addressContext default intercept callDataChannel <callDataChannel_Name>
% commit

Verify if the system and the databases are in sync with each other.

For SBC 5100, SBC 5110, SBC 5200, and SBC 5210 only: Prior to performing a Live Software Upgrade (LSWU), verify if the system and the databases are in sync. The steps to accomplish this are included in the announcement Warning-14-00020748. Refer to the applicable release notes for instructions on viewing announcements.
Before upgrading the SBC application in SBC Software Edition (SBC SWe), verify the Misc.TimerMaxHardPeriod and Misc.TimerMinHardPeriod parameters of ESXi are set as per recommendations in the table of "ESXi Advanced Settings" found on the page For VMware.
Before LSWU, ensure that the number of rules across profiles in a system is limited to 2000, and the number of actions across profiles in a system is limited to 10000.

Transfer the Package to Target Servers

Download the following files included in SBC application bundle from the Ribbon Support Portal to your local PC or server. Refer to Downloading Software from the Ribbon Support Portal.
1. sbc-V07.xx.xxR000-connexip-os_06.xx.xx-R000_amd64.qcow2 (SBC application image file)
2. sbc-V07.xx.xx-R000.x86_64.tar.gz (SBC application installation image)
3. sbc-V07.xx.xx-R000.x86_64.md5 (md5 associated with the SBC application installation image)
4. sbc-V07.xx.xx-R000.x86_64.signature
  To determine the exact version, refer to SBC Release Information.

Validate the integrity of SBC tar file with the md5 file. See Validating MD5Sum with 'Checksums Calculator' for guidance.
Launch the SBC EMA.
Navigate to Administration > System Administration > File Upload to upload the SBC application package to the active and standby SBC servers.
1. Click File Upload tab.
2. Click Add Files to Queue.
3. Browse the SBC application package files in the File Upload screen and click Open.
  The files are listed in Upload Queue section.
4. Click Upload Files. The file upload starts.
5. Repeat step a through d for standby SBC server.

Perform Live Software Upgrade of Application

This procedure prepares the server for revert, upgrades OS (if required) and upgrades the SBC application. During this process, the server may reboot up to two times.

Log on to the CLI on the active server.

At the CLI prompt of active server, start the LSWU using the following CLI command:

Example

admin@WFDSBC01a> request system serverAdmin WFDSBC01b startSoftwareUpgrade package sbc-V07.01.00-R000.x86_64.tar.gz
This command will start live software upgrade. Do you want to proceed (yes/no): <yes>
Proceeding
result success
reason Success from: WFDSBC01b

Note

The package name must not contain the package path.

Once the upgrade starts, monitor the status using the following CLI command.
Example
```
admin@WFDSBC01a> show table system serverSoftwareUpgradeStatus
NAME             UPGRADE 
				 STATUS
----------------------------
WFDSBC01a        pendingUpgrade
WFDSBC01b        upgrading
```
At the beginning of the upgrade, the server is rebooted and disk partitioning is done automatically which takes approximately 15 to 20 minutes. Do not hit the power button or restart the server during this time. Open the JViewer console to watch the progress of the upgrade.
Verify the sync status is in syncCompleted state, using the following CLI command.
Example
```
admin@WFDSBC01a> show table system syncStatus
SYNC MODULE                  STATUS
-------------------------------------------
Policy Data                 syncCompleted
Disk Mirroring              syncCompleted
Configuration Data          syncCompleted
Call/Registration Data      syncCompleted
```
During an LSWU process, when the Standby SBC is in "Sync" state (which usually takes about 2 - 3 minutes), the new call data is queued on the Active SBC. Those calls are not marked as stable. Once the "Sync" is complete, the pending call data queue is moved from Active SBC to Standby SBC. Those calls are marked as stable. Therefore, check the callCountStatus when both Active and Standby SBCs complete the "Sync" state. IRTT congestion is expected while “Sync” is in progress and should get cleared when Active and Standby SBCs complete the “Sync” state.

Verify the software upgrade status is in Upgraded state, using the following CLI command.

Example

admin@WFDSBC01a> show table system serverSoftwareUpgradeStatus
NAME             UPGRADE 
				 STATUS
--------------------------------
WFDSBC01a        pendingUpgrade
WFDSBC01b        upgraded

This process takes approximately 45 minutes to complete on a single server.

Verify the system server status using the following CLI command.

Example

admin@WFDSBC01a> show table system serverStatus
																		MGMT																				
										PLATFORM		APPLICATION   	REDUNDANCY						APPLICATION UP		LAST RESTART					
NAME      HW TYPE SERIAL NUM PART NUM  VERSION			VERSION      	ROLE		UP TIME				TIME				REASON					SYNC STATUS		
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
WFDSBC01a SBC 5200 2044090074 821-00430 V07.00.00R000 	V07.00.00R001 	active		0 Days	02:44:22	0 Days 02:43:04		systemRestart			syncCompleted
WFDSBC01b SBC 5200 2052090007 821-00430 V07.01.00R000 	V07.01.00R000 	standby		0 Days	00:13:02	0 Days 00:07:23		softwareUpgradeOrRevert	syncCompleted

Once the WFDSBC01b state changes to upgraded, proceed with upgrading the next server.

Upgrade the next server using the following CLI command.

Example

admin@WFDSBC01a> request system serverAdmin WFDSBC01a startSoftwareUpgrade package sbc-V07.01.00-R000.x86_64.tar.gz
This command will start live software upgrade. Do you want to proceed (yes/no): <yes>
Proceeding
result success
reason Success from: WFDSBC01a
[ok]

This causes an automatic switch-over during which the active role is assigned to the original standby server (WFDSBC01b) which is already upgraded.
Once the upgrade starts, monitor the status using the following CLI command on the new active server.
Example
```
admin@WFDSBC01b> show table system serverSoftwareUpgradeStatus
NAME             UPGRADE 
				 STATUS
----------------------------
WFDSBC01a        upgrading
WFDSBC01b        upgraded
```
Note

At the beginning of the upgrade, the server is rebooted and disk partitioning is done automatically which takes approximately 15 to 20 minutes. Do not hit the power button or restart the server during this time. Open the JViewer console to watch the progress of upgrade.

Verify whether both the servers are upgraded using the following CLI command.

Example

admin@WFDSBC01b> show table system serverSoftwareUpgradeStatus
NAME             UPGRADE 
				 STATUS
----------------------------
WFDSBC01a        upgraded
WFDSBC01b        upgraded

After both the servers get upgraded and is in sync with its peer, an automatic switchover may happen.

Note

On all the SBC platforms other than the SBC 7000, an automatic switchover happens at the end of upgrade from pre-5.0 releases.

Verify the Live Software Upgrade status is in UpgradeDone state and both primary and secondary upgrade status reflect upgraded state. Use the following CLI command.

Example

admin@WFDSBC01a> show table system softwareUpgradeStatus
state					upgradeDone;
previousState			upgrading;
upgradeStartTime		"Wed Dec 06 05:37:04 2017";
revertStartTime			n/a;
package					/opt/sonus/external/sbc-V07.01.00R000-connexip-os_06.01.00-R000_amd64.qcow2;
rpmName					/opt/sonus/staging/sbc-V07.01.00-R000.x86_64.rpm;
upgradeScript			/opt/sonus/staging/sbxUpgrade.pl;
revertScript			/opt/sonus/sbxRevert.pl;
reason					successfulCompletion;
oldRelease				V07.00.00R000;
newRelease				V07.01.00R000;
primaryUpgradeStatus	upgraded;
secondaryUpgradeStatus	upgraded;

Verify the SBC server status using the following CLI command.

Example

admin@WFDSBC01a> show table system serverStatus
																		MGMT																				
										PLATFORM		APPLICATION   	REDUNDANCY						APPLICATION UP		LAST RESTART					
NAME      HW TYPE SERIAL NUM PART NUM  VERSION			VERSION      	ROLE		UP TIME				TIME				REASON					SYNC STATUS		
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
WFDSBC01a SBC 5200 2044090074 821-00430 V07.01.00R000 	V07.01.00R000 	active		0 Days	15:32:47	0 Days 15:31:04		softwareUpgradeOrRevert	syncCompleted
WFDSBC01b SBC 5200 2052090007 821-00430 V07.01.00R000 	V07.01.00R000 	standby		0 Days	15:32:47	0 Days 15:31:04		softwareUpgradeOrRevert	syncCompleted

Note

The LSWU procedure takes approximately 1.5 hours to complete on both the servers.

Note

Upgrade logs can be found at /var/log/sonus/upgrade/latest

Once the upgrade is successful, clear the browser cache.

Space shortcuts

Page tree

Prerequisites

Transfer the Package to Target Servers

Perform Live Software Upgrade of Application