Page History

Overview

The following

There are VM operating parameters you can set to improve system throughput for a single or multiple VMs installed on a KVM host. Some VM operating parameters are set on the KVM host and are modified any time when the VM instance is running, while others are set on the VM and are only configured when the VM instance is shut down.

The following sections contain VM performance tuning recommendations to improve system performance. These performance recommendations are general guidelines, and are not intended to be all-inclusive.

Refer to the exhaustive. Refer to the documentation provided by your Linux OS and KVM host vendors for complete details. For example, Redhat provides extensive documentation on using virt-manager and optimizing VM performance. Refer to the Redhat Virtualization Tuning and Optimization Guide for details.

The following general recommendations apply to any platform where SBC SWe is deployed:

Recommended BIOS Settings

Recommended BIOS Settings

Ribbon recommends applying the BIOS settings in the following table on all VMs for optimum performance:

For hardware virtualization

All server BIOS settings are different, but in general, the following guidelines apply:

• Set power profiles to maximum performance
• Set thermal configurations to optimal cooling
• Disable HW prefetcher

Procedure

Set CPU Frequency

on the Host

The CPU frequency setting

# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

determines the  operating clock speed of the processor and in turn the system performance. Red Hat offers a set of in-built tuning profiles and a tool called tuned-adm that helps in configuring the required tuning profile.

Ribbon recommends to apply the throughput-performance tuning profile, which allows the processor to operate at maximum frequency.

1. Determine the active tuning profile

1. :

   Code Block
   #

1. tuned-adm active Current active profile: powersave

   


2. Apply the throughput-performance tuning profile:
   

   Code Block
   # tuned-adm profile throughput-performance

   
   

This configuration is persistent across reboots, and takes effect immediately. There is no need to reboot the host after configuring this tuning profile.

Perform NUMA Pinning for the VM

Use the procedure below to accomplish NUMA pinning for the VM.

1. Determine the number of NUMA nodes on the host server.

   Code Block
   [root@srvr3320 ~]# lscpu | grep NUMA NUMA node(s):          2 NUMA node0 CPU(s):     0-7,16-23 NUMA node1 CPU(s):     8-15,24-31 [root@srvr3320 ~]#

   In this example, there are two NUMA nodes on the server.
   
   

2. Find out to which NUMA node the SR-IOV-enabled PF is connected, which will get allocated to the SBC VM for pkt interfaces.
   
   

   1. Obtain the bus-info of the PF interface using the command ethtool -I <PF interface name>.
      

      Code Block
      [root@srvr3320 ~]# ethtool -i ens4f0 driver: igb version: 5.6.0-k firmware-version: 1.52.0 expansion-rom-version: bus-info: 0000:81:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes [root@srvr3320 ~]#

      Anchor
      2.b 2.b

      
      

   2. Identify the NUMA node of the PCI device using cat /sys/bus/pci/devices/<PCI device>/numa_node.

      Code Block
      [root@srvr3320 ~]# cat /sys/bus/pci/devices/0000\:81\:00.0/numa_node 1

      
      

3. Repeat the previous step for other SR-IOV interfaces from which you plan to connect VFs.
   

   Info
   title Note
   Make sure that all PCI devices are connected to the same NUMA node.

   
   

4. Once the NUMA node is discovered, set the <numatune> of the SBC VM in the VM xml file.

   Code Block
   <numatune>     <memory mode='preferred' nodeset="1"/> </numatune>

   
   

Determine Host Processor and CPU Details

To determine the host system's processor and CPU details, enter the following command to determine how many vCPUs are assigned to host CPUs:

lscpu -p

[root@srvr3320 ~]# lscpu -p
# The following is the parsable format, which can be fed to other
# programs. Each different item in every column has an unique ID
# starting from zero.
# CPU,Core,Socket,Node,,L1d,L1i,L2,L3
0,0,0,0,,0,0,0,0
1,1,0,0,,1,1,1,0
2,2,0,0,,2,2,2,0
3,3,0,0,,3,3,3,0
4,4,0,0,,4,4,4,0
5,5,0,0,,5,5,5,0
6,6,0,0,,6,6,6,0
7,7,0,0,,7,7,7,0
8,8,1,1,,8,8,8,1
9,9,1,1,,9,9,9,1
10,10,1,1,,10,10,10,1
11,11,1,1,,11,11,11,1
12,12,1,1,,12,12,12,1
13,13,1,1,,13,13,13,1
14,14,1,1,,14,14,14,1
15,15,1,1,,15,15,15,1
16,0,0,0,,0,0,0,0
17,1,0,0,,1,1,1,0
18,2,0,0,,2,2,2,0
19,3,0,0,,3,3,3,0
20,4,0,0,,4,4,4,0
21,5,0,0,,5,5,5,0
22,6,0,0,,6,6,6,0
23,7,0,0,,7,7,7,0
24,8,1,1,,8,8,8,1
25,9,1,1,,9,9,9,1
26,10,1,1,,10,10,10,1
27,11,1,1,,11,11,11,1
28,12,1,1,,12,12,12,1
29,13,1,1,,13,13,13,1
30,14,1,1,,14,14,14,1
31,15,1,1,,15,15,15,1
[root@srvr3320 ~]#

The first column lists the logical CPU number of a CPU used by the Linux kernel. The second column lists the logical core number -- use this information for vCPU pinning.

Ensure Persistent CPU Pinning

CPU pinning ensures that a VM only gets CPU time from a specific CPU or set of CPUs. Pinning is performed on each logical CPU of the guest VM against each core ID in the host system. The CPU pinning information is lost every time the VM instance is shut down or restarted. To avoid entering the pinning information again, update the KVM configuration XML file on the host system.

 Use the following steps to update the pinning information in the KVM configuration guest VM XML file:

1. Shut down the VM instance.
   
   

2. Start virsh.

   Code Block
   language none
   virsh [root@kujo ~]# virsh Welcome to virsh, the virtualization interactive terminal. Type: 'help' for help with commands 'quit' to quit virsh #

   
   

3. Check the list of running instances:
   

   Code Block
   language none
   virsh # list --all Id Name State ---------------------------------------------------- 2 ISBC_SWE_VM running

   
   

4. Edit the VM instance:
   

   Code Block
   language none
   virsh # edit <KVM_instance_name>

   
   

5. Search for the vcpu placement attribute.

   Panel
   bgColor transparent
   <domain type='kvm'>   <name>ISBC_SWE_VM</name>   <uuid>c31953dc-726d-4725-a405-9f446696add5</uuid>   <memory unit='KiB'>33554432</memory>   <currentMemory unit='KiB'>33554432</currentMemory>   <vcpu placement='static'>4</vcpu>   <os>     <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>     <boot dev='hd'/>   </os>   <features>

   
   

6. Make sure the vCPUs are pinned to the correct NUMA node CPUs.

   Spacevars
   0 company

    recommends to reserve the 1-core siblings of each NUMA node for the host process (do not use for the VM). Since the PCI is connected to NUMA node1 (as determined in step step 2.b of NUMA Pinning procedure), you must pin the vCPUs of the VM from the CPU siblings in NUMA node1.
   
   

   1. Skip the first physical core siblings, 8 and 24, and pin the rest.

      Code Block
      <vcpu placement='static' cpuset='9,25,10,26'>4</vcpu> <cputune>            <vcpupin vcpu="0" cpuset="9"/>            <vcpupin vcpu="1" cpuset="25"/>            <vcpupin vcpu="2" cpuset="10"/>            <vcpupin vcpu="3" cpuset="26"/> </cputune>

      
      

      As the CPU Architecture Example shows , you must pin the cores to their siblings (i.e. the two Hyperthreads coming from the same physical core). The second column in the example shows the physical core number.

      Info
      title Note
      Note: As Sub-NUMA Clustering is disabled in the BIOS, each Socket will represent each numa node. So in this case socket 0 is NUMA node0 and Socket 1 is NUMA node1. Make sure that all the vCPUs are pinned to the same NUMA node and don’t cross the NUMA boundary.

      
      

      Tip
      title Tip
      Ensure that no two VM instances have the same physical core affinity. For example, if VM1 has an affinity of 9,25,10,26 assigned, then no other VM should be pinned to this core again. To Assign CPU pinning to other VMs, use the other available cores on the host, leaving the first 2 logical cores (as described in Perform Host Pinning) per NUMA node for the host.  Also, assign all other VM instances running on the same host with affinity; otherwise the VMs without affinity may impact the performance of VMs that have affinity.

      
      

7. Save and exit the XML file.

   Code Block
   :wq

   
   

8. Start the VM instance.
   

   Code Block
   language none
   virsh # start <KVM_instance_name>

   
   

   Info
   title Note
   If you require additional changes to the XML file (such as those described below), you can hold off on restarting until all changes area made.

   
   

Edit VM CPU Mode

Use the following steps to edit the VM CPU mode:

1. Shut down the VM instance.
   
   

2. Start virsh.

   Code Block
   language none
   virsh [root@kujo ~]# virsh Welcome to virsh, the virtualization interactive terminal. Type: 'help' for help with commands 'quit' to quit virsh #

   
   

3. Check the list of running instances:
   

   Code Block
   language none
   virsh # list --all Id Name State ---------------------------------------------------- 2 ISBC_SWE_VM running

   
   

4. Edit the VM instance:
   

   Code Block
   language none
   virsh # edit <KVM_instance_name>

   
   

5. Locate the <cpu mode='custom'> attribute in the default configuration.
   
   

   Panel
   bgColor transparent
   <cpu mode='custom' match='exact' check='partial'>   <model fallback='allow'>SandyBridge</model>   <vendor>Intel</vendor>   <feature policy='require' name='pbe'/>   <feature policy='require' name='tm2'/>   <feature policy='require' name='est'/>   <feature policy='require' name='vmx'/>   <feature policy='require' name='osxsave'/>   <feature policy='require' name='smx'/>   <feature policy='require' name='ss'/>   <feature policy='require' name='ds'/>   <feature policy='require' name='vme'/>   <feature policy='require' name='dtes64'/>   <feature policy='require' name='ht'/>   <feature policy='require' name='dca'/>   <feature policy='require' name='pcid'/>   <feature policy='require' name='tm'/>   <feature policy='require' name='pdcm'/>   <feature policy='require' name='pdpe1gb'/>   <feature policy='require' name='ds_cpl'/>   <feature policy='require' name='xtpr'/>   <feature policy='require' name='acpi'/>   <feature policy='require' name='monitor'/> </cpu>

   
   

6. Replace the entire CPU mode content shown above with the below content, containing the proper CPU topology of the VM. To identify the proper topology for your VM instance, use sockets=1 (as the VM has a single NUMA node), threads=2 (since the VM will support hyperthreading), cores=<number of vcCPUs for the VM/2>.


   Panel
   bgColor transparent
   <cpu mode='host-model'>   <topology sockets='1' cores='2' threads='2'/> </cpu>

   
   

   Info
   title Note
   Ensure to enter the topology details above to exactly match the topology details set while creating the VM instance (The number of cores equals the number of vCPUs allocated from the VM divided by 2).  For example, if the VM instance topology is set to 1 socket, 2 cores and 2 threads, enter the identical details in this XML file.

   
   

7. Save and exit the XML file.
   

   Code Block
   :wq

   
   

8. Start the VM instance.
   

   Code Block
   language none
   virsh # start <KVM_instance_name>

   
   

   Info
   title Note
   If you require additional changes to the XML file (such as those described below), you can hold off on restarting until all changes area made.

   
   

Increase the Transmit Queue Length for virt-io Interfaces

This section is applicable only for virt-io based interfaces. 

To increase the Transmit Queue Length to 4096:

1. Start virsh:

   Code Block
   language none
   virsh [root@kujo ~]# virsh Welcome to virsh, the virtualization interactive terminal. Type: 'help' for help with commands 'quit' to quit virsh #

   
   

2. Identify the available interfaces.

   Code Block
   language none
   domiflist <VM_instance_name>

   
   

   The list of active interfaces displays.

   Panel
   bgColor transparent
   virsh # domiflist ISBC_SWE_VM Interface  Type     Source    Model     MAC ------------------------------------------------------- macvtap4   direct   eno1      virtio    52:54:00:e5:e8:9f macvtap5   direct   eno1      virtio    52:54:00:2b:43:9b macvtap6   direct   ens3fl    virtio    52:54:00:aa:89:38 macvtap7   direct   ens3f0    virtio    52:54:00:b6:60:76 virsh #

   
   

3. Increase the Transmit Queue Lengths for the tap interfaces.

   Code Block
   language none
   ifconfig <interface_name> txqueuelen <length>

   
   The interface_name is the name of the interface you want to change, and length is the new queue length. For example, ifconfig macvtap4 txqueuelen 4096.
   
   

4. Verify the value of the interface length.

   Code Block
   language none
   ifconfig <interface_name>

   
   Example output:

   Panel
   bgColor transparent
   [root@kujo ~]# ifconfig macvtap4 txqueuelen 4096 [root@kujo ~]# ifconfig macvtap5 txqueuelen 4096 [root@kujo ~]# ifconfig macvtap6 txqueuelen 4096 [root@kujo ~]# ifconfig macvtap7 txqueuelen 4096 [root@kujo ~]# ifconfig macvtap4 macvtap4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500         inet6 fe80::5054:ff:fee5:e89f prefixlen 64 scopeid 0x20<linkk>         ether 52:54:00:e5:e8:9f txqueuelen 4096 (Ethernet)         RX packets 2547441 bytes 177005232 (168.8 MiB)         RX errors 260 dropped 260 overruns 0 frame 0         TX packets 50573 bytes 17987512 (17.1 MiB)         TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

   
   

5. To make this setting persistent across the reboot, do the following:

   1. Modify/Create the 60-tap.rules file and add the KERNEL command by adding the line: KERNEL=="tap*", RUN+="/sbin/ip link set %k txqueuelen 4096"

      Code Block
      # vim /etc/udev/rules.d/60-tap.rules KERNEL=="tap*", RUN+="/sbin/ip link set %k txqueuelen 4096" # udevadm control --reload-rules

      
      

   2. Apply the rules to already created interfaces.

      Code Block
      # udevadm trigger --attr-match=subsystem=net

      
      

   3. Reboot the host.
      

      Code Block
      reboot

      
      

      Info
      title Note
      If you require additional changes, you can hold off on rebooting until all changes area made.

      
      

Stop Kernel Same-page Metering (KSM)

Kernel same-page metering (KSM) is a technology which finds common memory pages inside a Linux system and merges the pages to save memory resources. In the event of one of the copies being updated, a new copy is created so the function is transparent to the processes on the system. For hypervisors, KSM is highly beneficial when multiple guests are running with the same level of the operating system. However, there is overhead due to the scanning process which may cause the applications to run slower, which is not desirable.

To turn off KSM in the host:

1. Deactivate KSM by stopping the ksmtuned and ksm services, as shown below. This does not persist across reboots.

   Code Block
   # systemctl stop ksm # systemctl stop ksmtuned

   
   

2. Disable KSM persistently as shown below:

   Code Block
   # systemctl disable ksm # systemctl disable ksmtuned

   
   

Perform Host Pinning

To avoid performance impact on VMs due to host-level Linux services, host pinning isolates physical cores where a guest VM is hosted from physical cores where the Linux host processes/services run. 

In this example, core 0 (Core 0 and core 16 are logical cores) and core 8 (Core 8 and core 24 are logical cores) each represent the first core in each CPU socket, and are reserved for Linux host processes.

1. Configure the CPUAffinity option in /etc/systemd/system.conf. To get the first core siblings of each socket, use lscpu as shown below, or any other equivalent command.
   As shown below, 0 and 16 are the first core siblings on NUMA node0 CPU(s), and 8 and 24 are the first core siblings on NUMA node1 CPU(s).
   

   Panel
   bgColor transparent
   [root@srvr3320 ~]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Thread(s) per core: 2 Core(s) per socket: 8 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 45 Model name: Intel(R) Xeon(R) CPU E5-2658 0 @ 2.10GHz Stepping: 7 CPU MHz: 1782.128 CPU max MHz: 2100.0000 CPU min MHz: 1200.0000 BogoMIPS: 4190.19 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 20480K NUMA node0 CPU(s): 0-7,16-23 NUMA node1 CPU(s): 8-15,24-31

   
   

2. To dedicate the physical CPUs 0 and 8 for host processing, specify CPUAffinity as 0 8 16 24 in the file /etc/systemd/system.conf.

   Code Block
   CPUAffinity=0 8 16 24

   
   

3. Reboot the system.
   

   Code Block
   reboot

   
   

Using <emulatorpin> Tag

The <emulatorpin> tag specifies to which host physical CPUs the emulator (a subset of a domain, not including vCPUs) is pinned. The <emulatorpin> tag provides a method of setting a precise affinity to emulator thread processes. As a result, vhost threads run on the same subset of physical CPUs and memory, thus benefit from cache locality. 

In the above example, since the VM is pinned to core siblings 9,25, 10,26 from NUMA node0, and 8 and 24 from NUMA node1 for host-level services, you can pin the emulator thread to any free cor siblings in the same NUMA node, such as 11 and 27, as shown below.

<cputune>
        <emulatorpin cpuset="11,27"/>
</cputune>

The <emulatorpin> tag is required in order to isolate the virtio network traffic to be pinned to a different core than the VM vCPUs. This greatly reduces the percentage steal seen inside the VMs.

Back Up VMs with 1G hugepages

The number of hugepages is decided based on the total memory available on the host. 

1. Configure the host server hugepage size for 1G hugepages by appending the following line to the kernel command line options in /etc/default/grub. In the example below, the host  has a total of 256G memory, out of which 200G is configured as hugepages. 

   Code Block
   GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console" GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200n8 crashkernel=auto intel_iommu=on iommu=pt default_hugepagesz=1GB hugepagesz=1G hugepages=200 rhgb quiet" GRUB_DISABLE_RECOVERY="true"

   
   

2. Regenerate the GRUB2 configuration as shown below: 

   1. If your system uses BIOS firmware, issue the command:

      Code Block
      # grub2-mkconfig -o /boot/grub2/grub.cfg

      
      

   2. If your system uses UEFI firmware, issue the command: 

      Code Block
      # grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

      
      

      Tip
      title Tip
      A simple method to find out if you are running UEFI or BIOS is to check for the presence of the folder /sys/firmware/efi.  Enter the ls command:   # ls -l /sys/firmware/efi If you get the error "ls: cannot access /sys/firmware/efi: No such file or directory",  you system is using BIOS firmware, If the folder is present, your system is using UEFI firmware.

      
      

3. Add the following lines in your instance XML file using virsh edit <instanceName> to allow the hypervisor to back the VM with hugepage memory.

   Info
   title Note
   Make sure that the PCI device (SR-IOV, vCPU and VM memory) comes from the same NUMA node. For virtual pkt interfaces, Also, ensure that the vCPU and memory comes from the same NUMA node.

   
   

   Code Block
   <memory unit='KiB'>33554432</memory> <currentMemory unit='KiB'>33554432</currentMemory> <memoryBacking> <hugepages> <page size='1048576' unit='KiB' nodeset='1'/> </hugepages> </memoryBacking>

   
   

   Tip
   This example pins the VM on NUMA node1. For hosting a second VM on other NUMA node use the proper NUMA node value in the nodeset = <NUMA Node>.

   
   

4. Restart the host.

   Code Block
   reboot

   
   

   Info
   title Note
   If you require additional changes, you can hold off on rebooting until all changes area made.

   
   

5. Obtain the PID of the VM:

   Code Block
   ps -eaf | grep qemu | grep -i <vm_name>

   
   

6. Verify VM memory is received from a single NUMA node:

   Code Block
   numastat -p  <vmpid>

   
   

Disable Flow Control

Perform the following steps to disable flow control.

To disable flow control:

1. Login to the host system as the root user.
   
   

2. Disable flow control for interfaces attached to the SWe VM.

   Tip
   title Tip
   Use the <interface name> from the actual configuration.

   
   

   Code Block
   ethtool -A <interface name> rx off tx off autoneg off

   
   

   Code Block
   title Example
   ethtool -A p4p3 rx off tx off autoneg off ethtool -A p4p4 rx off tx off autoneg off ethtool -A em3 rx off tx off autoneg off ethtool -A em4 rx off tx off autoneg off

   
   

To make the setting persistent:

The network service in CentOS/RedHat has the ability to make the setting persistent. The script /etc/sysconfig/network-scripts/ifup-post checks for the existence of /sbin/ifup-local. If it exists, the script runs it with the interface name as a parameter (e.g. /sbin/ifup-local eth0).

Perform the following steps:

1. Create this file using the  touch command:
   

   Code Block
   touch /sbin/ifup-local

   
   

2. Make the file executable using the chmod command:


   Code Block
   chmod +x /sbin/ifup-local

   
   

3. Set the file's SELinux context using the chcon command: 

   Code Block
   chcon --reference /sbin/ifup /sbin/ifup-local

   
   

   
   

   
   

4. Open the file in an editor.
   
   Here is an example of a simple script to apply the same settings to all interfaces (except lo):

   Code Block
   #!/bin/bash if [ -n "$1" ]; then     if [ "$1" != "lo" ];then         /sbin/ethtool -A $1 rx off tx off autoneg off     fi fi

   
   

Recap of Changes in the KVM Configuration XML File

Use the following example KVM configuraion XML file to verify all of the changed values (highlighted in red) you performed in the aforementioned performance tuning steps.  

Tune Interrupt Requests (IRQs)

This section applies only to virt-io-based packet interfaces. Virt-IO networking works by sending interrupts on the host core. SBC VM performance can be impacted if frequent processing interruptions occur on any core of the VM. To avoid this, the affinity of the IRQs for a virtio-based packet interface should be different from the cores assigned to the SBC VM.

The /proc/interrupts file lists the number of interrupts per CPU, per I/O device. IRQs have an associated "affinity" property, "smp_affinity," that defines which CPU cores are allowed to run the interrupt service routine (ISR) for that IRQ. Refer to the distribution guidelines of the host OS for the exact steps to locate and specify the IRQ affinity settings for a device.

External Reference: https://access.redhat.com/solutions/2144921

Validate Changes

Validate your configuration changes using the steps below.

1. BIOS settings: If changed, ensure BIOS settings are set properly (changes will become effective after system startup). 
   
   

2. CPU frequency on the Host: Determine the active tuning profile:

   Code Block
   # tuned-adm active Current active profile: throughput-performance

   
   

3. NUMA and CPU pinning: Validate pinning and other memory settings.
   
   

   1. Verify the NUMA node of the SR-IOV device, as described in the section "Perform NUMA Pinning for the VM".
      
      

   2. Check to ensure all vCPU pinnings match what was previousl assigned:

      Code Block
      virsh # vcpupin 2 VCPU: CPU Affinity -------------------------- 0: 9 1: 25 2: 10 3: 26

      
      

4. TXqueuelen value: Verify the TXqueuelen value matches the example output in step 4 of "Increase the Transmit Queue Length for virt-io Interfaces".
   
   

5. KSM settings: Validate that KSM is disabled:

   Code Block
   # systemctl list-unit-files | grep disabled | grep ksm ksm.service                                   disabled ksmtuned.service                              disabled

   
   

6. Host pinning: Check the CPUAffinity set on the host to ensure the CPU numbers match what you assigned earlier.

   Code Block
   # cat /etc/systemd/system.conf | grep CPUAffinity CPUAffinity=0 8 16 24

   
   

7. Flow control:

   1. Check the physical interfaces on the host to ensure that flow control is disabled.

      Code Block
      # ethtool -a <interface name> Pause parameters for <interface name>: Autonegotiate:  off RX:             off TX:             off

      
      

   2. If flow controls is not disabled, go back to Disable Flow Control to perform the steps.
      
      

8. Overall VM settings:  Verify all other VM instance settings remain intact aft the final reboot.
   

   Code Block
   # virsh # edit <instance name>

   
   

Processor and CPU Details

To determine the host system's processor and CPU details, perform the following steps:

1. Execute the following command to determine how many vCPUs are assigned to host CPUs:

   Code Block
   lscpu -p

   The command provides the following output:

   Caption
   0 Figure 1 CPU Architecture
   Image Removed

   The first column lists the logical CPU number of a CPU as used by the Linux kernel. The second column lists the logical core number, this information can be used for vCPU pinning.

Persistent CPU Pinning

CPU pinning ensures that a VM only gets CPU time from a specific CPU or set of CPUs. Pinning is performed on each logical CPU of the guest VM against each core ID in the host system. The CPU pinning information will be lost every time the VM instance is shutdown or restarted. To avoid entering the pinning information again, you must update the KVM configuration XML file on the host system.

 To update the pinning information in the KVM configuration XML file:

1. Shutdown the VM instance.

2. Enter the following command.

   Code Block
   language none
   virsh

   The command provides the following output:

   Caption
   0 Figure 1 virsh Prompt
   Image Removed

3. Enter the following command to edit the VM instance:

   Code Block
   language none
   virsh # edit <KVM_instance_name>

4. Search for the vcpu placement attribute.

   Caption
   0 Figure 1 vCPU Placement Attribute
   Image Removed

5. Enter CPU pinning information as shown below:

   Caption
   0 Figure 1 CPU Pinning Information
   Image Removed

   Tip
   title Tip
   Ensure that no two VM instances have the same physical core affinity. For example, if VM1 has affinity of 0,1,2,3 assigned, then no VM should be pinned to 0,1,2,3,8,9,10 or 11 as these CPUs belong to the physical core assigned to VM1. Also, all other VM instances running on the same host must be assigned with affinity, otherwise the VMs without affinity might impact the performance of VMs having affinity.

6. Enter the following command to save and exit the XML file.

   Code Block
   :wq

CPU Mode Configuration

Even if the Copy host CPU configuration was selected while creating a VM instance, the host configuration may not be copied on the VM instance. To resolve this issue, you must edit the CPU mode to host-passthrough using a virsh command in the host system.

To edit the VM CPU mode:

1. Shutdown the VM instance.

2. Enter the following command.

   Code Block
   virsh

   The command provides the following output:

   Caption
   0 Figure 1 virsh Prompt
   Image Removed

3. Enter the following command to edit the VM instance:

   Code Block
   language none
   edit <KVM_instance_name>

4. Search for the cpu mode attribute.

   Caption
   0 Figure 1 cpu mode
   Image Removed

5. Edit the cpu mode attribute with the following:

   Caption
   0 Figure 1 Editing CPU Mode
   Image Removed

   Tip
   title Tip
   The topology details entered must be same as the topology details that were set while creating the VM instance. For example, if the topology was set to 1 socket, 4 cores and 1 thread, the same must be entered in this XML file.

6. Enter the following command to save and exit the XML file.

   Code Block
   :wq

7. Enter the following command to start the VM instance.

   Code Block
   language none
   start <KVM_instance_name>

8. Enter the following command to verify the host CPU configuration on the VM instance:

   Code Block
   language none
   cat /proc/cpuinfo

   The command provides the following output.

   Caption
   0 Figure 1 Verifying CPU Configuration
   Image Removed

Increasing the Transmit Queue Length

By default, the transmit queue length is set to 500. To increase the transmit queue length to 4096:

1. Execute the following command to identify the available interfaces:

   Code Block
   language none
   virsh

   The virsh prompt is displayed.

2. Execute the following command.

   Code Block
   language none
   domiflist <VM_instance_name>

   The list of active interfaces is displayed.

   Caption
   0 Figure 1 Active Interfaces List
   Image Removed

3. Execute the following command to increase the transmit queue lengths for the tap interfaces.

   Code Block
   language none
   ifconfig <interface_name> txqueuelen <length>

   where interface_name is the name of the interface you want to change, and length is the new queue length. For example, ifconfig macvtap4 txqueuelen 4096.

4. Execute the following command to verify the value of the interface length.

   Code Block
   language none
   ifconfig <interface_name>

   The command provides the following output.

   Caption
   0 Figure 1 Interface Information
   Image Removed

Kernel Same-page Metering (KSM) Settings

Apply the following settings to all VMs installed on the host.

Kernel same-page metering (KSM) is a technology which finds common memory pages inside a Linux system and merges the pages to save memory resources. In the event of one of the copies being updated, a new copy is created so the function is transparent to the processes on the system. For hypervisors, KSM is highly beneficial when multiple guests are running with the same level of the operating system. However, there is overhead due to the scanning process which may cause the applications to run slower, which is not desirable. The SBC SWe requires that KSM is turned off.

The sample commands below are for Ubuntu 4.4; use the syntax that corresponds to your operating system.

# echo 0 >/sys/kernel/mm/ksm/run
# echo "KSM_ENABLED=0" > /etc/default/qemu-kvm

Once KSM is turned off, it is important to verify that there is still sufficient memory on the hypervisor. When the pages are not merged, it may increase memory usage and lead to swapping that negatively impacts performance.

Host Pinning

To avoid performance impact on VMs due to host-level Linux services, host pinning isolates physical cores where a guest VM is hosted from physical cores where the Linux host processes/services run. In this example, the core 0 (Core 0 and core 36 are logical cores) and core 1 (Core 1 and core 37 are logical cores) are reserved for Linux host processes.

The CPUAffinity option in /etc/systemd/system.conf sets affinity to systemd by default, as well as for everything it launches, unless their .service file overrides the CPUAffinity setting with its own value. Configure the CPUAffinity option in /etc/systemd/system.conf.

Execute the following command:

lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                72
On-line CPU(s) list:   0-71
Thread(s) per core:    2
Core(s) per socket:    18
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 79
Model name:            Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz
Stepping:              1
CPU MHz:               2699.984
BogoMIPS:              4604.99
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              46080K
NUMA node0 CPU(s):     0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70
NUMA node1 CPU(s):     1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71

To dedicate the physical CPUs 0 and 1 for host processing, in the file /etc/systemd/system.conf, specify CPUAffinity as 0 1 36 37, as shown below. Restart the system.

CPUAffinity=0 1 36 37

Back up VMs with hugepages

1. Mount the HugeTLB filesystem on the host.

   Code Block
   mkdir -p /hugepages

2. Add the following line in the /etc/fstab  file.

   Code Block
   hugetlbfs /hugepages hugetlbfs defaults 0 0

3. Configure the number of 2M hugepages equal to the vRAM requirement for hosting a VM:

   Code Block
   cat /etc/sysctl.conf# System default settings live in /usr/lib/sysctl.d/00-system.conf. # To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file # # For more information, see sysctl.conf(5) and sysctl.d(5). vm.nr_hugepages = 25000 (assuming a 24G VM) vm.hugetlb_shm_group = 36

4. Add lines in your instance XML file using virsh edit <instanceName>:

   Code Block
   <domain type='kvm' id='3'> <name>RENGALIVM01</name> <uuid>f1bae5a2-d26e-4fc0-b472-3638743def9a</uuid> <memory unit='KiB'>25165824</memory> <currentMemory unit='KiB'>25165824</currentMemory> <memoryBacking> <hugepages> <page size='2048' unit='KiB' nodeset='0'/> </hugepages> </memoryBacking>

   Tip
   title Tip
   The previous example pins the VM on NUMA node 0. For hosting a second VM on NUMA node 1, use nodeset = ‘1’.

5. Restart the host.

6. To verify, get the PID for the VM and execute the following command to check that VM memory is received from a single NUMA node:

   Code Block
   numastat -p <vmpid>

Disable Flow Control

Execute the following command to disable flow control for interfaces attached to the SWe VM.

ethtool -A <interface name> rx off tx off autoneg off  

Example:

ethtool -A p4p3 rx off tx off autoneg off
ethtool -A p4p4 rx off tx off autoneg off
ethtool -A em3 rx off tx off autoneg off
ethtool -A em4 rx off tx off autoneg off