Page History

Upgrade the Software

Start

1. Log onto the CPU as root that contains the stp_sp2k_21_x_x_nbxxxxxxx_upgrade.sh file file via the CPU terminal server connection.

2. Change the directory to /var/log.

   Code Block
   cd /var/log

   
   

   Info
   title Note
   After you upload the software package from your PC to the SP2000, the system automatically does the following: Confirms that the software upgrade package is found Checks that there is sufficient memory on the SP2000 for the new software Extracts the new software to the SP2000 that you are logged into (/var/log) Installs the scripts to all Management and Routing CPU cards

   
   
   

3. Enter the following command to run the Self-Extraction utility and press ENTER. 
   
   

   Code Block
   ./stp_sp2k_21_x_x_nbxxxxxxxx_upgrade.sh

   
   

   Code Block
   title Example
   [root@TMA15sp2kslot14 log]# ./stp_sp2k_21_x_x_nbxxxxxxxx_upgrade.sh DSC Self-Extraction Utility xx_x_x_xxxxxxxx ----------------------------------------------------- Verify archive integrity... Executing install script... Software Upgrade Preparation ---------------------------- root: UpgradeInstaller Executed upgrade installer xx_x_x_nbxxxxxxxx ---------------------------------------- Executed upgrade installer xx_x_x_nbxxxxxxxx Verify /upgrade ... Partition for /upgrade has 1.5 GB Verify execution point... Verify system state... Verify requisite files in the archive... Is it OK to proceed with preparation of the /upgrade directory? (y/n):

   
   

4. Enter y.

   Code Block
   title Example
   Preparing /upgrade directory... Expand OS image from archive... Place SW load file... Redirect pre_upgrade_check and upgrade... Place new pre_upgrade_check and upgrade... ---------------------------------------------------------------- To complete the process run the upgrade command. (May optionally run pre_upgrade_check independently beforehand). ---------------------------------------------------------------- complete. [root@TMA15sp2kslot14 log]#

   
   

5. Enter the following at the command line and press ENTER.

   Code Block
   upgrade

   The upgrade is automatically executed on the system. During this process, the CPU 14 and 24 initialize, discover, and install the software load and perform the upgrade/restore using the backup file. 
   

   Code Block
   title Example
   [root@TMA15sp2kslot14 log]# upgrade :::::::::::::::::::::::::::::::: Ribbon DSC :::::::::::::::::::::::::::::::: SP2000 CPU Upgrade _____________________________________________ Type "q" to quit from most prompts or CTRL-C. This process will prepare each CPU for upgrade before rebooting each CPU in order, isolating the reboots to avoid outages. Please run a pre-upgrade check. Type "yes" to execute this check (exiting otherwise):

   
   

6. Enter yes and press ENTER.
   

   Code Block
   title Example
   Calling pre-upgrade check... ======== Performing upgrade PRE-checks... ========= --------------------------------------------------- USB device check... --------------------------------------------------- Check USB device status on all the CPU or VM slots reported by cardlist/slotmon SLOT: 14 SLOT: 24 PASSED

    

7. Press ENTER.
   

   Code Block
   title Example
   --------------------------------------------------- Ping connectivity check... --------------------------------------------------- Ping all slots which are CPUS or IOs in /opt/cpu_ss7gw/current/data/slot.conf to check card availability Ping all the slots reported by cardlist/slotmon to check packet loss -Slot: 14 Packets dropped: 0% -Slot: 24 Packets dropped: 0% -Slot: 12 Packets dropped: 0% -Slot: 22 Packets dropped: 0% PASSED

    

8. Press ENTER.
   

   Code Block
   title Example
   --------------------------------------------------- ish connectivity check... --------------------------------------------------- Perform ish test between all CPUS in /opt/cpu_ss7gw/current/data/slot.conf to check slots interactive communication availability ish test from slot 14 to all slots in system... ish test from slot 14 to slot 14 passed. ish test from slot 14 to slot 24 passed. ish test from slot 24 to all slots in system... ish test from slot 24 to slot 14 passed. ish test from slot 24 to slot 24 passed. PASSED

    

9. Press ENTER.
   

   Code Block
   title Example
   --------------------------------------------------- RPM database check... --------------------------------------------------- Check RPM database status on all the CPU or VM slots reported by cardlist/slotmon SLOT: 14 RPM database files backed up to /var/lib/rpm/backup-20200115 RPM database is not corrupt. Verifying RPM database. Cleaning up RPM database is valid. SLOT: 24 RPM database files backed up to /var/lib/rpm/backup-20200115 RPM database is not corrupt. Verifying RPM database. Cleaning up RPM database is valid. PASSED

    

10. Press ENTER.
    

    Code Block
    title Example
    --------------------------------------------------- Free space check... --------------------------------------------------- -Available RAM space: 139 MB -Available HDD (/var/log) space: 54425 MB -Available Compact Flash (/opt directory) space: 1575 MB PASSED

     

11. Press ENTER.
    

    Code Block
    title Example
    Checking current Clock Settings... --------------------------------------------------- Current Clock Settings... --------------------------------------------------- TIMEZONE: CPU Slot 14: -05 (EST) CPU Slot 24: -05 (EST) IO Slot 12: -05 (EST) IO Slot 22: -05 (EST) HWCLOCK: CPU Slot 14: Wed 15 Jan 2020 11:51:06 AM EST -0.627910 seconds CPU Slot 24: Wed 15 Jan 2020 11:50:35 AM EST -0.375621 seconds IO Slot 12: IO Slot 22: WALL CLOCK: CPU Slot 14: Wed Jan 15 11:51:08 EST 2020 CPU Slot 24: Wed Jan 15 11:51:08 EST 2020 IO Slot 12: Wed Jan 15 11:51:09 EST 2020 IO Slot 22: Wed Jan 15 11:51:09 EST 2020

     

12. Press ENTER.
    

    Code Block
    title Example
    ---------------------------------------------------------------- Checking if IO netswitch is working... ---------------------------------------------------------------- PASSED

     

13. Press ENTER.
    

    Code Block
    title Example
    --------------------------------------------------- Check Kernel and Mount Points. --------------------------------------------------- No tests to perform.

     

14. Press ENTER.
    

    Code Block
    title Example
    --------------------------------------------------- Dumping slot status and watchdog processes... --------------------------------------------------- Dumping CPU status info... AVAILABLE SOFTWARE VERSIONS -------------------------------------------------------------------------------- No new load detected on system. Upload and run load .sh file on /var/log before attempting upgrade. CPU (current) stp_sp2k_21_x_x_nbxxxxxxxx OS: Ribbon7.4.6 (Servicepack) [none applied] CPU (previous) Not_Available OS: Ribbon7.4.18 (Servicepack) RUNNING SOFTWARE VERSIONS (*=update required, ?=update unknown) CARD BUILD SERVICE O/S -SLOT----TYPE----------RELEASE--VERSION----PACK---IMAGE-------CARD STATUS--- -------------------------------------------------------------------------------- Slot 14 MGMT/ROUTING 21.x.x (nbxxxxxxxx) SP: Ribbon7.4.6 24 procs running Slot 24 MGMT/ROUTING 21.x.x (nbxxxxxxxx) SP: Ribbon7.4.6 24 procs running (p=servicepack is partially applied) Dumping IO status info... AVAILABLE SOFTWARE VERSIONS -------------------------------------------------------------------------------- IO (current) AMC308 stp_sp2k_21_x_x_nbxxxxxxxxx OS: rel_12_00_p04.sp2k.img (Servicepack) [none applied] RUNNING SOFTWARE VERSIONS (*=update required, ?=update unknown) CARD BUILD SERVICE O/S -SLOT----TYPE----------RELEASE-VERSION------PACK-----IMAGE------- -------------------------------------------------------------------------------- Slot 12 IO AMC308 21.x.x (nbxxxxxxxx) SP: OS: rel_12_00_p04.sp2k.img Slot 22 IO AMC308 21.x.x (nbxxxxxxxx) SP: OS: rel_12_00_p04.sp2k.img (p=servicepack is partially applied) Dumping Watchdog process info for each CPU...

    
    

    Code Block
    title Example
    Watchdog Process Details on MGMT - slot14_0. WID PID STA RPT RHT HTV HTT COM 1 6491 S+ 0 1 10 10 slotmon --platform SP2K -s 14 -g 0 -p /op... 2 16651 S+ 0 0 10 10 pxbr -p /opt/cpu_ss7gw/current/data --pla... 3 16663 S+ 0 0 10 10 upp -p /opt/cpu_ss7gw/current/data/ --pla... 4 16677 S+ 0 0 10 10 sca -p /opt/cpu_ss7gw/current/data/ 5 16707 S+ 0 0 10 10 sysmon.default -p /opt/cpu_ss7gw/current/... 6 16855 S+ 0 0 10 10 sysmon.cpucheck -p /opt/cpu_ss7gw/current... 7 16966 S+ 0 0 10 10 sysmon.diskmon -p /opt/cpu_ss7gw/current/... 8 17136 S+ 0 0 10 10 sysmon.logcpu -p /opt/cpu_ss7gw/current/s... 9 17322 S+ 0 0 10 10 sysmon.logmem -p /opt/cpu_ss7gw/current/s... 10 17474 S+ 0 0 10 10 sysmon.usbcheck -p /opt/cpu_ss7gw/current... 11 17605 S+ 0 0 10 10 sysmon.tipcdump -p /opt/cpu_ss7gw/current... 12 17783 S+ 0 0 10 10 sysmon.slotinventory -p /opt/cpu_ss7gw/cu... 13 17971 S+ 0 0 10 10 sysmon.configuremtu -p /opt/cpu_ss7gw/cur... 14 18145 S+ 0 0 10 10 hwmon --platform SP2K -c 10 15 15308 S+ 0 0 10 10 licensed -r -f /opt/cpu_ss7gw/current/dat... 16 12019 S+ 0 1 10 10 snm --platform SP2K -p /opt/cpu_ss7gw/cur... 17 15331 S+ 0 0 10 10 imfProbe -p 7377 18 17440 S+ 0 5 10 10 gws --platform SP2K -p /opt/cpu_ss7gw/cur... 19 15722 S+ 0 5 10 10 dre --platform SP2K -p /opt/cpu_ss7gw/cur... 20 17439 S+ 0 2 10 10 sccp --platform SP2K --noGTT -p /opt/cpu_... 21 14281 S+ 0 3 10 10 gtt --platform SP2K -p /opt/cpu_ss7gw/cur... 22 17438 S+ 0 5 10 10 dinamo --platform SP2K -p /opt/cpu_ss7gw/... 23 13157 S+ 0 3 10 10 pce --platform SP2K --noGTT -p /opt/cpu_s... 24 15677 S+ 0 0 10 10 imf --platform SP2K -p /opt/cpu_ss7gw/cur...

    
    

    Code Block
    title Example
    Watchdog Process Details on MGMT - slot24_0. WID PID STA RPT RHT HTV HTT COM 1 15063 S+ 0 0 10 10 slotmon --platform SP2K -s 24 -g 0 -p /op... 2 18587 S+ 0 0 10 10 pxbr -p /opt/cpu_ss7gw/current/data --pla... 3 18830 S+ 0 0 10 10 upp -p /opt/cpu_ss7gw/current/data/ --pla... 4 19056 S+ 0 0 10 10 sca -p /opt/cpu_ss7gw/current/data/ 5 19293 S+ 0 0 10 10 sysmon.default -p /opt/cpu_ss7gw/current/... 6 19566 S+ 0 0 10 10 sysmon.cpucheck -p /opt/cpu_ss7gw/current... 7 19810 S+ 0 0 10 10 sysmon.diskmon -p /opt/cpu_ss7gw/current/... 8 20032 S+ 0 0 10 10 sysmon.logcpu -p /opt/cpu_ss7gw/current/s... 9 20289 S+ 0 0 10 10 sysmon.logmem -p /opt/cpu_ss7gw/current/s... 10 20529 S+ 0 0 10 10 sysmon.usbcheck -p /opt/cpu_ss7gw/current... 11 20775 S+ 0 0 10 10 sysmon.tipcdump -p /opt/cpu_ss7gw/current... 12 21038 S+ 0 0 10 10 sysmon.slotinventory -p /opt/cpu_ss7gw/cu... 13 21293 S+ 0 0 10 10 sysmon.configuremtu -p /opt/cpu_ss7gw/cur... 14 21542 S+ 0 0 10 10 hwmon --platform SP2K -c 20 15 20305 R+ 0 0 10 10 licensed -r -f /opt/cpu_ss7gw/current/dat... 16 20323 S+ 0 0 10 10 snm --platform SP2K -p /opt/cpu_ss7gw/cur... 17 20389 S+ 0 0 10 10 imfProbe -p 7377 18 20403 S+ 0 0 10 10 gws --platform SP2K -p /opt/cpu_ss7gw/cur... 19 20784 S+ 0 0 10 10 dre --platform SP2K -p /opt/cpu_ss7gw/cur... 20 20801 S+ 0 0 10 10 sccp --platform SP2K --noGTT -p /opt/cpu_... 21 20812 S+ 0 0 10 10 gtt --platform SP2K -p /opt/cpu_ss7gw/cur... 22 20829 S+ 0 0 10 10 dinamo --platform SP2K -p /opt/cpu_ss7gw/... 23 20841 R+ 0 0 10 10 pce --platform SP2K --noGTT -p /opt/cpu_s... 24 20856 S+ 0 0 10 10 imf --platform SP2K -p /opt/cpu_ss7gw/cur...

     

15. Press ENTER.
    

    Code Block
    title Example
    --------------------------------------------------- PCE Specific check for SCCP Parameter... --------------------------------------------------- PASSED

     

16. Press ENTER.
    

    Code Block
    title Example
    --------------------------------------------------- GWS Incoming Linkset check --------------------------------------------------- No GWS Incoming Linkset check needed for this SP2K version

     

17. Press Enter
    

    Code Block
    title Example
    ------------------------------------------------------------------------- Checking if DB Definition of GTT NA and its LPC registration match ------------------------------------------------------------------------- PASSED

     

18. Press ENTER.
    

    Code Block
    title Example
    --------------------------------------------------- Audit files check... --------------------------------------------------- 0 files were out of sync. Files: 110 Copied: 0 Deleted: 0 PASSED

     

19. Press ENTER.
    

    Code Block
    title Example
    --------------------------------------------------- Reminder to take a backup... --------------------------------------------------- -It is highly recommended to take a backup of the system prior to starting the upgrade. Label and keep the backup in a safe place.

     

20. Press ENTER.
    

    Code Block
    title Example
    ====== The results of the PRE upgrade check are stored in /var/log/cpu_ss7gw/upgrade_data/PRE_UPGRADE_DATA.stp_sp2k_xx_x_x_nbxxxxxxxx.Wed_Jan_15_11_49_28_EST_2020 ====== Upgrade PRE checks completed. All checks PASSED. You may now proceed with the upgrade. Collecting information from the system. Please wait... Getting software versions [ ==================== ] 100% Using: /shared/log/stp_sp2k_xx_x_x_nbxxxxxxxx_upgrade.sh Upgrading to: 21_x_x_nbxxxxxxxx Requires update: 24 14 Up to date: none I/O Cards: 12 22 Uncommissioned/Bad: Confirmation to proceed with upgrade operations? (yes/no):

     

21. Enter yes and press ENTER.
    

    Code Block
    title Example
    Copying upgrade package and preparing each CPU. Working on slot: 24 Execute /var/log/stp_sp2k_21_x_x_nbxxxxxxxx_upgrade.sh on 24 Issue backup on localhost Copy backup to peers [24, 14]. Should user input (a y/n prompt) be required after each CPU is upgraded? (yes/no) [n]:

     

22. Enter yes and press ENTER.
    

    Code Block
    title Example
    Upgrade execution order [24, 14]. To continue, press the Enter key:

     

23. Press ENTER.
    

    Code Block
    title Example
    Processing peer 24... Boot Update ----------- Available kernel version: 3.10.0-693.21.1.el7.sonus07.x86_64 Existing kernel version: 3.10.0-693.21.1.el7.sonus07.x86_64 Installed kernel version(s): 3.10.0-693.21.1.el7.sonus07.x86_64 Removing kernel file collection from defunct-upgrade location Removing defunct-upgrade grub config file Removing defunct-upgrade grub2 config file Moving current kernel file collection to rollback location Installing new kernel file collection: file installed: vmlinuz-3.10.0-693.21.1.el7.sonus07.x86_64 file installed: System.map-3.10.0-693.21.1.el7.sonus07.x86_64 file installed: config-3.10.0-693.21.1.el7.sonus07.x86_64 file installed: initramfs-3.10.0-693.21.1.el7.sonus07.x86_64.img file installed: symvers-3.10.0-693.21.1.el7.sonus07.x86_64.gz file installed: vmlinuz-0-rescue-29512d3199624fc5958c44c518e35267 file installed: initramfs-0-rescue-29512d3199624fc5958c44c518e35267.img Update mounts old backup is removed /upgrade/etc/.fstab_previous_0 generated backup /upgrade/etc/fstab_previous_0 set new load to FS2 Generate grub updates. old backup is removed /boot/grub/.grub.conf_previous_3.10.0-693.21.1.el7.sonus07.x86_64 generated backup /boot/grub/grub.conf_previous_3.10.0-693.21.1.el7.sonus07.x86_64 old backup is removed /boot/grub2/.grub.cfg_previous_3.10.0-693.21.1.el7.sonus07.x86_64 generated backup /boot/grub2/grub.cfg_previous_3.10.0-693.21.1.el7.sonus07.x86_64 Clean /upgrade over-mount directories. Preserve/set cloud-init disable Commit boot service Ready to reboot. Waiting for the new machine 24 to recover.

    
    

    Warning
    title Warning
    When the following "Unable to determine" error occurs, DO NOT interrupt the upgrade process on the current machine. This message is expected. DO NOT type "yes" until verified that this process is complete or a system failure will occur (this is continued in the next section on this page).

    
    

     

24. Open a second terminal window for your second CPU machine. 

25. Verify that the second machine is rebooted and running. 

    Tip
    title Tip
    If the prompt shows "dscbuild" for the second machine, the update is complete. When the machine is up and running, host name and slot number prompt will replace "dscbuild".

    
    

26. Type wd on the CPU that was just upgraded and ensure that processes are running without issues.

    Code Block
    title Example
    [root@SP2K00slot24 ~]# wd WID PID STA RPT RHT HTV HTT COM 1 15860 S+ 0 0 10 10 slotmon --platform SP2K -s 24 -g 0 -p /op... 2 19096 S+ 0 0 10 10 pxbr -p /opt/cpu_ss7gw/current/data --pla... 3 19114 S+ 0 0 10 10 upp -p /opt/cpu_ss7gw/current/data/ --pla... 4 19125 S+ 0 0 10 10 licensed -r -f /opt/cpu_ss7gw/current/dat... 5 19131 S+ 0 0 10 10 sca -p /opt/cpu_ss7gw/current/data/ 6 19193 S+ 0 0 10 10 snm --platform SP2K -p /opt/cpu_ss7gw/cur... 7 19212 S+ 0 0 10 10 sysmon.default -p /opt/cpu_ss7gw/current/... 8 19315 S+ 0 0 10 10 gws --platform SP2K -p /opt/cpu_ss7gw/cur... 9 19335 S+ 0 0 10 10 sysmon.cpucheck -p /opt/cpu_ss7gw/current... 10 19460 S+ 0 0 10 10 sysmon.diskmon -p /opt/cpu_ss7gw/current/... 11 19637 S+ 0 0 10 10 sysmon.logcpu -p /opt/cpu_ss7gw/current/s... 12 19812 S+ 0 0 10 10 sysmon.logmem -p /opt/cpu_ss7gw/current/s... 13 20028 S+ 0 0 10 10 sysmon.usbcheck -p /opt/cpu_ss7gw/current... 14 20216 S+ 0 0 10 10 dre --platform SP2K -p /opt/cpu_ss7gw/cur... 15 20243 S+ 0 0 10 10 sysmon.slotinventory -p /opt/cpu_ss7gw/cu... 16 20335 S+ 0 0 10 10 dinamo --platform SP2K -p /opt/cpu_ss7gw/... 17 20367 S+ 0 0 10 10 sysmon.configuremtu -p /opt/cpu_ss7gw/cur... 18 20528 S+ 0 0 10 10 sysmon.configaudit -p /opt/cpu_ss7gw/curr... 19 15509 S+ 1 0 10 10 hwmon --platform SP2K -c 20 [root@SP2K00slot24 ~]#

    
    

27. In the GUI, on the CPU slot that has been upgraded, compare your system's status to the state it was in before upgrading (refer to Verify and Record System Status). 
    1. Check Alarms to ensure no unexpected logs or alarms have been flagged.
    2. Verify MTP3 SS7 Link Status by going to the SP2000 GUI > Applications > MTP3 > All Links. All SS7 links should be In-Service.
    3. Verify SG M3UA ASP Path Status by going to the SP2000 Web GUI > Applications > SG > M3UA > M3UA Multi ASPs. All M3UA ASP Associations should be “UP”.
    4. Monitor C20 core alarm dashboard
28. Wait 10 minutes for applications and links to stabilize.

29. Proceed to stop the software on the other (non upgraded) CPU if you wish to perform some extra tests. When the upgrade process of the first CPU has been completed, tests can be performed to ensure stability and functionality of the new load.

...

1. In the GUI, stop the SS7 Routing Software Applications on the CPU slot that has not been upgraded.

2. Log into the WEB UI on the CPU that has not been upgraded and navigate to the System Start/Stop Commands Under System > Processes.

3. Choose the CPU Slot that has not been upgraded and select Stop Routing Software Applications. Click ‘Continue’.

4. Proceed with test calls.

5. In the terminal window from the previous section (Step 23), continue to upgrade the second CPU by typing yes.

   Code Block
   yes

   
   

   Note
   title Caution
   If the screen has timed out, and shows a black screen, do not attempt to "wake" the window- just type "yes".

   
   

6. Press ENTER. 

   Code Block
   title Example
   Processing local 14... Boot Update ----------- Available kernel version: 3.10.0-693.21.1.el7.sonus07.x86_64 Existing kernel version: 3.10.0-693.21.1.el7.sonus07.x86_64 Installed kernel version(s): 3.10.0-693.21.1.el7.sonus07.x86_64 Removing kernel file collection from defunct-upgrade location Removing defunct-upgrade grub config file Removing defunct-upgrade grub2 config file Moving current kernel file collection to rollback location Installing new kernel file collection: file installed: vmlinuz-3.10.0-693.21.1.el7.sonus07.x86_64 file installed: System.map-3.10.0-693.21.1.el7.sonus07.x86_64 file installed: config-3.10.0-693.21.1.el7.sonus07.x86_64 file installed: initramfs-3.10.0-693.21.1.el7.sonus07.x86_64.img file installed: symvers-3.10.0-693.21.1.el7.sonus07.x86_64.gz file installed: vmlinuz-0-rescue-29512d3199624fc5958c44c518e35267 file installed: initramfs-0-rescue-29512d3199624fc5958c44c518e35267.img Update mounts old backup is removed /upgrade/etc/.fstab_previous_0 generated backup /upgrade/etc/fstab_previous_0 set new load to FS1 Generate grub updates. old backup is removed /boot/grub/.grub.conf_previous_3.10.0-693.21.1.el7.sonus07.x86_64 generated backup /boot/grub/grub.conf_previous_3.10.0-693.21.1.el7.sonus07.x86_64 old backup is removed /boot/grub2/.grub.cfg_previous_3.10.0-693.21.1.el7.sonus07.x86_64 generated backup /boot/grub2/grub.cfg_previous_3.10.0-693.21.1.el7.sonus07.x86_64 Clean /upgrade over-mount directories. Preserve/set cloud-init disable Commit boot service Ready to reboot. You have new mail in /var/spool/mail/root [root@TMA15sp2kslot14 log]# Connection closing...Socket close.

    

7. If testing is unfavorable after the first CPU upgrade AND a rollback is required, apply the following steps BEFORE proceeding with the Rollback.  This will minimize the outage on the M3UA ASPs to the CORE.
   

...