VNF Performance Tuning

The sections below describe the best possible performance and scale for a given virtual machine resource profile:

Purpose

Real-time applications have stringent requirements with respect to jitter, latency, quality of service and packet loss. The migration of real-time applications to an all-software environment requires deterministic response to failures and performance in the scheduler of the hypervisor and the host Operating System (OS). Although OpenStack continually addresses carrier-grade performance, scalability, resiliency, manageability, modularity and interoperability; however, some fine-tuning is available to achieve maximum scale and reliable performance for the SBC SWe and ancillary applications. This document defines those areas that needs to be fine-tuned in OpenStack/KVM platforms.

The Sonus SBC SWe requires a reservation of CPU, memory and hard disk resources in virtual machines in addition to implementing certain performance tuning parameters for any production deployments that are over 100 concurrent sessions.

Recommended BIOS Settings

Sonus recommends applying the BIOS settings in the table below to all Nova compute hosts running the Sonus VMs for optimum performance:

• S-SBC
• M-SBC
• T-SBC
• SBC Configurator

CPU Pinning Overview

Apply below settings to all Nova compute hosts in the pinned host aggregate.

From the hypervisor's perspective, a virtual machine appears as a single process that should be scheduled on the available CPUs. By design, hypervisors can schedule clock cycle on a different processor. While this is certainly acceptable in environments where the hypervisor is allowed to over-commit, this contradicts the requirements for real-time applications. Hence, Sonus requires CPU pinning to prevent applications from sharing a core.

CPU with Unpinned Applications

CPU with Pinned Applications

 

By default, virtual CPUs are not assigned to a host CPU, but Sonus requires CPU pinning to maintain the requirements of real-time media traffic. The primary reason for pinning Sonus instances is to prevent other workloads (including those of the host OS) from causing significant jitter in media processing. It is also possible to introduce significant message queuing delays and buffer overflows at higher call rates. OpenStack states that no instance with pinned CPUs can use the CPUs of another pinned instance. This prevents resource contention and improves processor cache efficiency by reserving physical cores. Host Aggregate filters or Availability Zones can be used to select compute hosts for pinned and non-pinned instances. OpenStack clearly states that pinned instances must be separated from unpinned instances as latter will not respect the resourcing requirements of the former.

To enable CPU pinning, execute the following steps on every compute host where CPU pinning is to be enabled:

1. To retrieve the NUMA topology for the node, execute the below command:

   # lscpu  | grep NUMA
   NUMA node(s):          2
   NUMA node0 CPU(s):     0-11,24-35
   NUMA node1 CPU(s):     12-23,36-47

   In this case, there are two Intel Sockets with 12 cores each; configured for hyper-threading. CPUs are paired on physical cores in the pattern 0/24, 1/25, etc. (The pairs are also known as thread siblings).

2. When the following code is added at the end of /etc/default/grub, the system understands the cores that should be used by VMs (and not by host operating system):

   GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX hugepagesz=1G hugepages=256"

   For Red Hat RHEL based host OS, Red Hat recommended omitting the isolcpus reservation configuration.

   The number of hugepages depends on how many VM instances is created on this host and multiplied by the memory size of each instance. The hugepagesz should be the maximum hugespace value supported by the kernel being used.

3. A pin set limits KVM to placing guests on a subset of the physical cores and thread siblings. Omitting some cores from the pin set ensures that there are dedicated cores for the OpenStack processes and application. The pin set can also be a means of ensuring that KVM guests never use more than one thread/core while leaving the additional thread for shared KVM/OpenStack processes. This mechanism can boost performance of non-threaded guest applications by allowing the host OS to schedule closely related host OS processes on the same core with the guest OS (e.g. virtio processes). The following example built on the CPU and NUMA topology shown in Step 1 (above):
   
   

   • For Hyper Threading Host: Add the CPU pin set list to vcpu_pin_set in /etc/nova/nova.conf:

     vcpu_pin_set=2-11,14-23,26-35,38-47

     For compute nodes, servicing VMs which can be run on Hyper-Threaded host, the CPU PIN set includes all thread siblings except for the cores which are carved out and dedicated to host OS. The resulting CPU PIN in the example dedicates cores/threads 0/24,1/25 and 12/36,13/37 to the host OS. VMs uses cores/threads 2/26-11/35 on NUMA node 0, and cores/threads 14/38-23/47 on NUMA node 1.

4. Update the boot record and reboot the compute node.

5. Configure the Nova Scheduler to use NUMA Topology and Aggregate Instance Extra Specs on Nova Controller Hosts:

On each node where the OpenStack Compute Scheduler (openstack-nova-scheduler) runs, edit nova.conf file that is located at /etc/nova/nova.conf. Add the AggregateInstanceExtraSpecFilter and NUMATopologyFilter values to the list of scheduler_default_filters. These filters are used to segregate the compute nodes that can be used for CPU pinning from those that cannot and to apply NUMA aware scheduling rules when launching instances:

• • scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,

  • ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter,

  • PciPassthroughFilter,NUMATopologyFilter,AggregateInstanceExtraSpecsFilter
    
    In addition to support SR-IOV, enable the PciPassthroughFilter and restart the openstack-nova-scheduler service.

    systemctl restart openstack-nova-scheduler.service

    With CPU pinning now enabled, Nova must be configured to use it. See the section below for a method to use a combination of host-aggregate and nova flavor keys.

CPU Model Setting

Apply below settings to all Nova compute hosts where Sonus VMs are installed.

The CPU model defines the CPU flags and the CPU architecture that is exposed from the host processor to the guest.

Non-S/M/T-SBC Instances

Sonus supports either host-passthrough or host-model for non-S/M/T-SBC instances; this includes the SBC Configurator.

S/M/T-SBC Instances

Sonus recommends setting CPU Mode to host-passthrough for SBC instances so every detail of the host CPU can be known by SBC SWe. The host-model setting impacts how CPU L2 / L3 cache information is communicated to the guest OS since the libvert emulated CPU does not accurately represent the L2 and L3 CPU hardware caches to the guest OS.  In performance testing of the Sonus SBC, Sonus has seen significant performance degradation, which goes beyond just a simple reduction in capacity. This results in signaling latency and jitter that is outside of acceptable limits even at modest loads.

This setting is defined in /etc/nova/nova.conf:

This change is made in /etc/nova/nova-compute.conf:

Apply below settings to all Nova compute hosts where Sonus VMs are installed:

• EMS
• PSX-M
• PSX-Replica
• S-SBC
• M-SBC
• T-SBC
• SBC Configurator

The CPU model defines the CPU flags and the CPU architecture that is exposed from the host processor to the guest. Modify nova.conf file located at /etc/nova/nova.conf and set CPU-Mode to host-passthrough that helps SBC SWe to know every detail of the host CPU. The host-model setting impacts how CPU L2/L3 cache information is communicated to the guest OS since the libvert emulated CPU does not accurately represent the L2 and L3 CPU hardware caches to the guest OS. In performance testing of the Sonus SBC, Sonus has seen significant performance degradation that goes beyond simple reduction in capacity. This results in signaling latency and jitter that is outside of acceptable limits even at modest loads.

This setting is defined in /etc/nova/nova.conf:

[libvirt]
virt_type = kvm 
cpu_mode = host-passthrough

This change is made in /etc/nova/nova-compute.conf:

[libvirt]
virt_type = kvm

Removal of CPU and Memory Over Commit

Apply below settings to all Nova compute hosts where Sonus VMs are installed:

• S-SBC
• M-SBC
• T-SBC
• SBC Configurator

The default settings for CPU (1.16) and Memory (1:5). Modify nova.conf file located at /etc/nova/nova.conf, and change the default settings of cpu_allocation_ratio and ram_allocation_ratio to (1:1) for resource reservation.

cpu_allocation_ratio = 1.0
ram_allocation_ratio = 1.0

Adjusting the Tx Queue Length of Tap Device

Apply below settings to all Nova compute hosts where Sonus VMs are installed:

• S-SBC
• M-SBC
• T-SBC
• SBC Configurator

While using the centralized mode with virtual nics (virtio), OpenStack creates tap devices for each port on the guest VM. The Tx queue length of the tap devices is set to 500 by default that defines the queue between the OVS and the VM instance. The value 500 is too low on the queue that increases the possibility of packet drops at the tap device. Set Tx queue length to a higher value that increases performance and reliability. Use a value that matches your performance requirements. 

The sample commands below are for Ubuntu 4.4, please use the syntax that corresponds to your operating system.

Modify the 60-tap.rules file and add the KERNEL command
# vim /etc/udev/rules.d/60-tap.rules
KERNEL=="tap*", RUN+="/sbin/ip link set %k txqueuelen 1000" - Add this line
# udevadm control --reload-rules
Use the below command to apply the rules to already created interfaces:
# udevadm trigger --attr-match=subsystem=net

Kernel Same-page Metering (KSM) Settings

Apply below settings to all Nova compute hosts where Sonus VMs are installed:

• S-SBC
• M-SBC
• T-SBC
• SBC Configurator

Kernel same-page metering (KSM) is a technology which finds common memory pages inside a Linux system and merges the pages to save memory resources. In the event of one of the copies being updated, a new copy is created so the function is transparent to the processes on the system. For hypervisors, KSM is highly beneficial where multiple guests are running with the same level of operating system. However, there is an overhead due to the scanning process which may cause the applications to run slower which is not desirable. The SBC SWe requires KSM to be turned-off.

The sample commands below are for Ubuntu 4.4, please use the syntax that corresponds to your operating system

# echo 0 >/sys/kernel/mm/ksm/run
# echo "KSM_ENABLED=0" > /etc/default/qemu-kvm

Once the KSM is turned-off, it is important to verify that there is still sufficient memory on the hypervisor. When the pages are not merged, it may increase memory usage and lead to swapping that impacts performance negatively.

Hyper-Threading Support

Hyper-threading is designed to use "idle resources" on Intel processors. A physical core is split into 2 x logical cores for parallel threads. Each logical core has its own architectural state. This is shown in the diagram below:

Hyperthreading Support

The actual performance gains of using hyper-threading depends on the amount of idle resources on the physical CPU. Hyper-threading on a SBC SWe instance has yet to prove any quantifiable gain in performance for a given number of cores. Sonus is in the process of assessing this on various call flows. The performance should never drop below the values obtained without hyper-threading for the same number of cores and could increase, but we need additional engineering work to qualify there are no negative impacts.

Sonus VNF CPU Pinning and Hyper-threading Support

Hyper-threading can be enabled in the BIOS for all Sonus NFV elements.

Sonus VNF Tested Configurations

*Memory values rounded to the next power of 2 to prevent memory fragmentation in the nova compute scheduler.

Host Aggregate Method for SMP VM Placement

A few methods exist to influence VM placement in OpenStack. The method described in this section segregates Nova compute nodes into discrete host aggregates and use Nova flavor-key aggregate_instance_extra_specs so that specific flavors will use specific host aggregates. For this to work, all flavors must specify a host aggregate. This is accomplished by first assigning all existing flavors to a "normal" host aggregate, then assigning only the Nova compute hosts configured for non-Hyper-Threading to a "Pin-Isolate" host aggregate.

From the Openstack CLI, create the host aggregates and assign compute hosts:

% nova aggregate-create Active-Pin-Isolate
% nova aggregate-set-metadata Active-Pin-Isolate Active-Pin-Isolate=true
% nova aggregate-add-host Active-Pin-Isolate {first nova compute host in aggregate}
    {repeat for each compute host to be added to this aggregate}

% nova aggregate-create Active
% nova aggregate-set-metadata Active Active=true
% nova aggregate-add-host Active {first nova compute host in aggregate}
    {repeat for each compute host to be added to this aggregate}

From the Openstack CLI, assign all existing flavors to the non-pinned host aggregate:
% for FLAVOR in `nova flavor-list | cut -f 2 -d ' ' | grep -o [0-9]*`; \
    do nova flavor-key ${FLAVOR} set \
        "aggregate_instance_extra_specs:Active"="true"; \
    done

Tested Flavor Definitions

The flavor definitions mentioned below includes the following Extra Specs:

• hw:cpu_policy=dedicated: This setting enables CPU pinning.
• hw:cpu_thread_policy=isolate: The SBC VNFCs require non-threaded access to the pinned CPUs to guarantee real-time performance for signaling and media.
• hw:numa_nodes: This setting defines how the host processor cores are spread over the host NUMA nodes. Where this is set to 1, it ensures that the cores are not spread over more than 1 NUMA node ensuring the performance of having one; otherwise Nova would be free to split the cores up between available NUMA nodes.

• hw:cpu_max_sockets: This setting defines how KVM exposes the sockets and cores to the guest. Without this setting, KVM always exposes a socket for every core; each socket having one core. This requires a mapping in the host virtualization layer to convert the topology resulting in a measurable performance degradation. That performance overhead can be avoided by accurately matching the advertised cpu_sockets to the requested host numa_nodes. Using the *_max_* variable ensures that the value cannot be overridden in the image metadata supplied by tenant level users.
  
  

  From the Openstack CLI, Create flavors that will go on the Hyper-Threaded Nova compute hosts:
  EMS
  % nova flavor-create EMS-SK-E-01P auto 16384 40 8
  % nova flavor-key EMS-SK-E-01P set aggregate_instance_extra_specs:Active=true hw:cpu_policy=dedicated
  
  PSX Master
  % nova flavor-create PSX-SK-PM-01P auto 65536 180 20
  % nova flavor-key PSX-SK-PM-01P set aggregate_instance_extra_specs:Active=true hw:cpu_policy=dedicated
  
  SBC Configurator
  % nova flavor-create SBC-SK-C-01P auto 16384 80 4
  % nova flavor-key SBC-SK-C-01P set aggregate_instance_extra_specs:Active=true hw:cpu_policy=dedicated
  
  PSX Replica as SRv6 Proxy
  % nova flavor-create PSX-SK-SRV6-01P auto 32768 180 16
  % nova flavor-key PSX-SK-SRV6-01P set aggregate_instance_extra_specs:Active=true hw:cpu_policy=dedicated
  % nova flavor-key PSX-SK-SRV6-01P set hw:numa_nodes=1 hw:cpu_max_sockets=1
  
  PSX Replica as D+ for CSBC 
  % nova flavor-create PSX-SK-CD-01P auto 32768 180 16
  % nova flavor-key PSX-SK-CD-01P set aggregate_instance_extra_specs:Active=true hw:cpu_policy=dedicated
  % nova flavor-key PSX-SK-CD-01P set hw:numa_nodes=1 hw:cpu_max_sockets=1

  From the Openstack CLI, Create flavors that will go on the non-Hyper-Threaded Nova compute hosts:
  
  C-SBC Signaling
  % nova flavor-create SBC-SK-CS-01P auto 65536 80 20
  % nova flavor-key SBC-SK-CS-01P set aggregate_instance_extra_specs:Active-Pin-Isolate=true 
  % nova flavor-key SBC-SK-CS-01P set hw:cpu_policy=dedicated hw:cpu_thread_policy=isolate 
  % nova flavor-key SBC-SK-CS-01P set hw:mem_page_size=1048576
  % nova flavor-key SBC-SK-CS-01P set hw:numa_nodes=2 hw:cpu_max_sockets=2
   
  C-SBC Media
  % nova flavor-create SBC-SK-CM-01P auto 32768 80 10
  % nova flavor-key SBC-SK-CM-01P set aggregate_instance_extra_specs:Active-Pin-Isolate=true
  % nova flavor-key SBC-SK-CM-01P set hw:cpu_policy=dedicated hw:cpu_thread_policy=isolate
  % nova flavor-key SBC-SK-CM-01P set hw:numa_nodes=1 hw:cpu_max_sockets=1
  % nova flavor-key SBC-SK-CM-01P set hw:mem_page_size=1048576

References

• Host Aggregate Based Pinning Flavor Specification Reference: http://redhatstackblog.redhat.com/2015/05/05/cpu-pinning-and-numa-topology-awareness-in-openstack-compute/

• OpenStack Flavor Specification Reference: http://docs.openstack.org/admin-guide/compute-flavors.html

• OpenStack CPU Typologies Reference: http://docs.openstack.org/admin-guide/compute-cpu-topologies.html