The DSC Platform Disaster Recovery procedure performs a backup and restore on a VM if a system failure occurs.

Only after a new VM is installed, configured, and running data traffic as per operating requirements, an image of the VM is created and saved in a backup file that is used to rebuild the VM in the event of a catastrophic system failure. The VM will be restored to the same image and configuration status that proceeded the catastrophic failure event.

The Disaster Recovery procedure is closely coupled with the new VM Install procedure. The backup file contains an image of the new DSC Platform after it is installed, configured, and performing (running data traffic) as required. 

Note

It is highly recommended that a backup copy is made of the new DSC Platform installation and saved to another computer (for example, a dedicated backup server) for safe storage and later retrieval.

Workflow
StepAction
1Create a Backup of a VM.
2Restore a VM(s) on a KVM Host.
3Check the Status of VM(s).
4Re-authenticate pcsd Tokens and Verify VM Status.
5Create new SSH keys for the DSC Platform.

Create a Backup of a VM

Follow the instructions in this subsection for creating a backup of a VM.

Note

For a complete backup and restore of system configuration files on the DSC Platform, refer to the detailed procedures in the Platform Manager User Guide.

Start
  1. Login to the Web UI.

  2. In the System menu, click Backups.

  3. Select the Manual Configuration Backup.


  4. Click Complete configuration backup.

  5. Click Create Manual Backup.

    Tip

    The backup file (tar archive) is created and stored in the default directory: /var/log/backups. For safe storage and retrieval, save the backup file to the designated backup server. The location of the backup file destination can be changed by selecting the Backup Destination tab and clicking Create New Destination Definition. The Create Backup Destination screen appears where you can define the Destination Types (for example, local or FTP directories) and the pathname of the new destination.

  6. In the System Menu, click File Retrieval, and then click /var/log.


  7. Click the backup file, select Save File, and click OK.

  8. Log in to the VM as a root user using an SSH console port connection.

    login as: root
root@10.91.1.70's password:
*************************************
 Hostname --> DSCVMLab70slot1
 System IP -> 10.91.1.70
*************************************
[root@DSCVMLab70slot1 ~]#

  9. Verify the backup was created.

    [root@DSCVMLab70slot1 ~]# ls -lrt /var/log/backups/
-rw-r--r-- 1 root root 175004 Oct 22 15:11 DSCVMLab70slot1-bak-24-10-14-104357-Complete-DSCVM.14.4.tar.gz

  10. Copy the backup tar file to the backup server for later retrieval.

    #  scp /var/log/backups/<Backup_Filename> <Destination_Computer / Destination_Directory>

Restore a VM(s) on a KVM Host

Execute the following procedure to restore the VM(s) on the DSC Platform. After a successful VM restore, the Web UI is used to verify the slot status and to ensure all applications are running. 

Note

It is recommended that you perform the complete VM restore for the Routing and Management VM1 and VM2 before you restore the other Routing VMs.

Note

During the initial install, you are expected to save a copy of the install package, installation configuration and backup files. This procedure provides instructions about restoring a VM(s) on a KVM Host using the configuration file from the initial install and a complete backup file.

Prerequisites
  • You have copy of the install package and the installation configuration and backup files. 
Start
  1. Log onto a KVM Host or a Linux workstation on which you want to execute the restore.

  2. Enter the following command to start the restore process.

    ./dsc_swe_<release>_nb<date>_KVM.sh restore --config <filename>.conf --backup <backupFilename>


    Example
    # ./dsc_swe_17_0_0_nb20180604_KVM.sh restore --config DSC_TMAkvm.conf --backup TMAkvm-slot1-bak-06-06-18-110824-Complete-DSCVM.17.0.tar.gz
    Example 
    DSC SWe KVM Host Interaction
----------------------------
/tmp/kvminstall.py --self-exe ../dsc_swe_17_0_0_nb20180604_KVM.sh --offset 1374 --qcow2 dsc_swe_17_0_0_nb20180604.qcow2 --host-set dsc_swe_host_extra.tgz --print-command restore --config DSC_TMAkvm.conf --backup TMAkvm-slot1-bak-06-06-18-110824-Complete-DSCVM.17.0.tar.gz
Verify requisite files in the archive... 

Processing hosts
----------------
	Connected to caot-cloud1.sonusnet.com  libvirt 1005003 (lib 2000000)


VM Installation on KVM host
---------------------------
Install TMAkvm-slot1 on host caot-cloud1.sonusnet.com.
Install TMAkvm-slot2 on host caot-cloud1.sonusnet.com.

Continue with install of all listed VMs? (y/n):
extracting QCOW2 dsc_swe_17_0_0_nb20180604.qcow2 to /tmp ...

Installing VM TMAkvm-slot1 on host caot-cloud1.sonusnet.com
-----------------------------------------------------------
Checking for VM TMAkvm-slot1 on host caot-cloud1.sonusnet.com...
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'TMAkvm-slot1'
> issuing: ls /home/libvirt_images/TMAkvm-slot1.img
> issuing: ls /home/libvirt_images/TMAkvm-slot1_configdrive.iso


Process QCOW2 caot-cloud1.sonusnet.com TMAkvm-slot1
---------------------------------------------------
	Convert to raw image...
> issuing: qemu-img convert -O raw /tmp/dsc_swe_17_0_0_nb20180604.qcow2 /home/libvirt_images/TMAkvm-slot1.img
	Resize the raw image to 65G...
> issuing: qemu-img resize /home/libvirt_images/TMAkvm-slot1.img +53G

Generate VM Definition & Install caot-cloud1.sonusnet.com TMAkvm-slot1
----------------------------------------------------------------------
	Generate QEMU XML definition
	Define VM
	VM defined with UUID 7f5b9c06-7c66-498f-9ad5-7d3c0cd0c4fa
	Set autostart (start VM on host boot).
	using temporary space /tmp/dscLwO2jQ
> issuing: genisoimage -V config-2 -graft-points -J -rock -o /tmp/dscLwO2jQ/TMAkvm-slot1_configdrive.iso /tmp/dscLwO2jQ/iso
I: -input-charset not specified, using utf-8 (detected in locale settings)
Using VENDO000.JSO;1 for  /tmp/dscLwO2jQ/iso/openstack/latest/vendor_data2.json (vendor_data.json)
Total translation table size: 0
Total rockridge attributes bytes: 1369
Total directory bytes: 6144
Path table size(bytes): 52
Max brk space used 1b000 382 extents written (0 MB)
	removing temporary space /tmp/dscLwO2jQ

Installing VM TMAkvm-slot2 on host caot-cloud1.sonusnet.com
-----------------------------------------------------------

Checking for VM TMAkvm-slot2 on host caot-cloud1.sonusnet.com...
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'TMAkvm-slot2'
> issuing: ls /home/libvirt_images/TMAkvm-slot2.img
> issuing: ls /home/libvirt_images/TMAkvm-slot2_configdrive.iso

Process QCOW2 caot-cloud1.sonusnet.com TMAkvm-slot2
---------------------------------------------------
	Convert to raw image...
> issuing: qemu-img convert -O raw /tmp/dsc_swe_17_0_0_nb20180604.qcow2 /home/libvirt_images/TMAkvm-slot2.img
	Resize the raw image to 65G...
> issuing: qemu-img resize /home/libvirt_images/TMAkvm-slot2.img +53G

Generate VM Definition & Install caot-cloud1.sonusnet.com TMAkvm-slot2
----------------------------------------------------------------------
	Generate QEMU XML definition
	Define VM
	VM defined with UUID 373b5273-9953-4107-84f2-54f228671a9c
	Set autostart (start VM on host boot).
	using temporary space /tmp/dsciwDusf
> issuing: genisoimage -V config-2 -graft-points -J -rock -o /tmp/dsciwDusf/TMAkvm-slot2_configdrive.iso /tmp/dsciwDusf/iso
I: -input-charset not specified, using utf-8 (detected in locale settings)
Using VENDO000.JSO;1 for  /tmp/dsciwDusf/iso/openstack/latest/vendor_data2.json (vendor_data.json)
Total translation table size: 0
Total rockridge attributes bytes: 1369
Total directory bytes: 6144
Path table size(bytes): 52
Max brk space used 1b000
382 extents written (0 MB)
	removing temporary space /tmp/dsciwDusf
removing extracted qcow2
	Store config with VMs on hosts.
	warning: existing TMAkvm config adding time

Wrote configuration to file /tmp/DSC_TMAkvm.20180606140434.conf

Process VM startups
-------------------
System configuration file: /tmp/DSC_TMAkvm.20180606140434.conf

Store install log (/tmp/DSC_TMAkvm.20180606140434.install_log) with VMs on hosts.

Check the Status of VM(s)

Start
  1. Log onto a KVM Host or a Linux workstation for which you want to check VM(s) status.

  2. Enter the following command at the Linux prompt.{cod

    # ./dsc_swe_<release>_nb<date>_KVM.sh status --local --system <systemName>


    # ./dsc_swe_17_0_0_nb20180604_KVM.sh status --local --system samsvnf
    DSC Self-Extraction Utility 17_0_0_nb20180604
-----------------------------------------------------
Verify archive integrity... 

Executing install script... 

DSC SWe KVM Host Interaction
----------------------------
Verify requisite files in the archive... 

Processing hosts
----------------
	Connected to caot-cloud1.sonusnet.com  libvirt 1005003 (lib 2000000)

Installed DSC VMs
-----------------
		--+ Host caot-cloud1.sonusnet.com (6)
			+-- samsvnf-slot3
			+-- samsvnf-slot6
			+-- samsvnf-slot2
			+-- samsvnf-slot4
			+-- samsvnf-slot1
			+-- samsvnf-slot5
complete.

Re-authenticate pcsd Tokens and Verify VM Status

Use the following procedure to re-authenticate pcsd Tokens for the restored DSC Platform

Start
  1. Log in to the VM as a root user using an SSH Console port connection.

  2. Enter the following command to re-authenticate the pcsd tokens on the VM:

    # pcs cluster auth mgmt_cpu1_0 mgmt_cpu2_0 --force -u hacluster -p vmcluster

  3. Log onto the Web UI.

  4. From the Main Menu, click Slot Monitor.

  5. Make sure the Slot Status for the restored VM shows AVAILABLE.

Create new SSH keys for the DSC Platform

Use the following procedure to create new SSH Keys for the restored DSC Platform

Start
  1. Log in to the VM as a root user using an SSH Console port connection.

  2. Enter the following command to recreate the SSH Keys:

    # status CPU
  3. Log onto the Web UI.

  4. From the Main Menu, click Processes.

  5. Click Watchdog Processes.

  6. Select the corresponding restored VM, and enter Update.
  7. Make sure that all applications are up and running on the restored VM