Page History

...

1. To retrieve the NUMA topology for the node, execute the below command:

   Code Block
   # lscpu | grep NUMA NUMA node(s): 2 NUMA node0 CPU(s): 0-11,24-35 NUMA node1 CPU(s): 12-23,36-47

   Note
   In this case, there are two Intel Sockets with 12 cores each; configured for hyperHyper-threadingThreading. CPUs are paired on physical cores in the pattern 0/24, 1/25, etc. (The pairs are also known as thread siblings).

2. When the following code is added at the end of /etc/default/grub, the system understands the cores that should be used by VMs (and not by host operating system):

   Code Block
   GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX hugepagesz=1G hugepages=256"

   Note
   For Red Hat RHEL based host OS, Red Hat recommended omitting the isolcpus reservation configuration. The number of hugepages depends on how many VM instances is created on this host and multiplied by the memory size of each instance. The hugepagesz should be the maximum hugespace value supported by the kernel being used.

3. A pin set limits KVM to placing guests on a subset of the physical cores and thread siblings. Omitting some cores from the pin set ensures that there are dedicated cores for the OpenStack processes and application. The pin set can also be a means of ensuring that KVM guests never use more than one thread/core while leaving the additional thread for shared KVM/OpenStack processes. This mechanism can boost performance of non-threaded guest applications by allowing the host OS to schedule closely related host OS processes on the same core with the guest OS (e.g. virtio processes). The following example built on the CPU and NUMA topology shown in Step 1 (above):
   
   

   • For Hyper-Threading Host: Add the CPU pin set list to vcpu_pin_set in /etc/nova/nova.conf:

     Code Block
     vcpu_pin_set=2-11,14-23,26-35,38-47

     For compute nodes, servicing VMs which can be run on Hyper-Threaded host, the CPU PIN set includes all thread siblings except for the cores which are carved out and dedicated to host OS. The resulting CPU PIN in the example dedicates cores/threads 0/24,1/25 and 12/36,13/37 to the host OS. VMs uses cores/threads 2/26-11/35 on NUMA node 0, and cores/threads 14/38-23/47 on NUMA node 1.

4. Update the boot record and reboot the compute node.

5. Configure the Nova Scheduler to use NUMA Topology and Aggregate Instance Extra Specs on Nova Controller Hosts:

...

• • scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,

  • ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter,

  • PciPassthroughFilter,NUMATopologyFilter,AggregateInstanceExtraSpecsFilter
    
    In addition to support SR-IOV, enable the PciPassthroughFilter and restart the openstack-nova-scheduler service.

    Code Block
    systemctl restart openstack-nova-scheduler.service

    With CPU pinning now enabled, Nova must be configured to use it. See the section below for a method to use a combination of host-aggregate and nova flavor keys.

Pagebreak

Anchor
model model

CPU Model Setting

...

Hyper-Threading Support

Hyper-threading Threading is designed to use " idle resources " on Intel processors. A physical core is split into 2 x logical cores for parallel threads. Each logical core has its own architectural state. The actual performance gains of using Hyper-Threading depends on the amount of idle resources on the physical CPU.

This is shown in the diagram below:.

The actual performance gains of using hyper-threading depends on the amount of idle resources on the physical CPU. Hyper-threading on a SBC SWe instance has yet to prove any quantifiable gain in performance for a given number of cores. Sonus is in the process of assessing this on various call flows. The performance should never drop below the values obtained without hyper-threading for the same number of cores and could increase, but we need additional engineering work to qualify there are no negative impacts.

...

Sonus VNF CPU Pinning and Hyper-Threading Support

Hyper-Threading can be enabled in the BIOS for all Sonus NFV elements.

...

*Memory values rounded to the next power of 2 to prevent memory fragmentation in the nova compute scheduler.

...

• hw:cpu_policy=dedicated: This setting enables CPU pinning.
  
  
• hw:cpu_thread_policy=isolate: The SBC VNFCs require non-threaded access to the pinned CPUs to guarantee real-time performance for signaling and media.
  
  
• hw:numa_nodes: This setting defines how the host processor cores are spread over the host NUMA nodes. Where this is set to 1, it ensures that the cores are not spread over more than 1 NUMA node ensuring the performance of having one; otherwise Nova would be free to split the cores up between available NUMA nodes.
  
  

• hw:cpu_max_sockets: This setting defines how KVM exposes the sockets and cores to the guest. Without this setting, KVM always exposes a socket for every core; each socket having one core. This requires a mapping in the host virtualization layer to convert the topology resulting in a measurable performance degradation. That performance overhead can be avoided by accurately matching the advertised cpu_sockets to the requested host numa_nodes. Using the *_max_* variable ensures that the value cannot be overridden in the image metadata supplied by tenant level users.
  
  

EMS SWe and PSX SWe Flavor Examples

...

#EMS
% nova flavor-create EMS-SK-E-01P auto 16384 

...

60 8
% nova flavor-key EMS-SK-E-01P set aggregate_instance_extra_specs:Active=true hw:cpu_policy=dedicated

...

#PSX Master
% nova flavor-create PSX-SK-PM-01P auto 65536 180 20
% nova flavor-key PSX-SK-PM-01P set aggregate_instance_extra_specs:Active=true hw:cpu_policy=dedicated

...

#SBC Configurator
% nova flavor-create SBC-SK-C-01P auto 16384 80 4
% nova flavor-key SBC-SK-C-01P set aggregate_instance_extra_specs:Active=true hw:cpu_policy=dedicated

...

#PSX Replica as SRv6 Proxy
% nova flavor-create PSX-SK-SRV6-01P auto 32768 180 16
% nova flavor-key PSX-SK-SRV6-01P set aggregate_instance_extra_specs:Active=true hw:cpu_policy=dedicated
% nova flavor-key PSX-SK-SRV6-01P set hw:numa_nodes=1 hw:cpu_max_sockets=1

...

#PSX Replica as D+

...

 
% nova flavor-create PSX-SK-CD-01P auto 32768 180 16
% nova flavor-key PSX-SK-CD-01P set aggregate_instance_extra_specs:Active=true hw:cpu_policy=dedicated
% nova flavor-key PSX-SK-CD-01P set hw:numa_nodes=1 hw:cpu_max_sockets=1

...

SBC SWe Flavor Example

To create a M-SBC SWe flavor with 20 vCPUs, 32 GiB of RAM and 100 GB of Hard disk, enter the following Nova commands from the Openstack CLI.

% nova flavor-create Sonus-MSBC auto 32768 100 20
% nova flavor-

...

key 

...

Sonus-MSBC set hw:cpu_policy=dedicated hw:cpu_thread_policy=prefer
% nova flavor-key 

...

Sonus-MSBC set 

...

hw:cpu_

...

max_sockets=1
% nova flavor-key 

...

Sonus-MSBC set hw:

...

mem_page_size=

...

2048
% nova flavor-key 

...

Sonus-MSBC set hw:numa_nodes=

...

1

To create a S-SBC SWe flavor with 64 GiB RAM and 100 GB of Hard Disk based on 2 x NUMA nodes of 20 vCPUs each (For example, 40 vCPUs for S-SBC), enter the following Nova commands from the Openstack CLI.

% nova flavor-create 

...

Sonus-SSBC auto 

...

65536 

...

100 

...

40
% nova flavor-key 

...

Sonus-SSBC set hw:cpu_policy=dedicated hw:cpu_thread_policy=

...

prefer
% nova flavor-key 

...

Sonus-SSBC set hw:

...

cpu_max_sockets=

...

2
% nova flavor-key 

...

Sonus-SSBC set hw:mem_page_size=2048
% nova flavor-key Sonus-SSBC set hw:

...

numa_nodes=2

Regarding the default setting, numa_mempolicy=preferred, the NUMA memory allocation policy is set to "strict" which forces the kernel to allocate memory only from the local NUMA node where processes are scheduled. If memory on one of the NUMA node is exhausted for any reason, the kernel cannot allocate memory from another NUMA node even when memory is available on that node. With this in mind, using the default setting would have a negative impact on applications like the S-SBC. This setting is in reference to the link below:

https://specs.openstack.org/openstack/nova-specs/specs/juno/implemented/virt-driver-numa-placement.html

References

• Host Aggregate Based Pinning Flavor Specification Reference: http://redhatstackblog.redhat.com/2015/05/05/cpu-pinning-and-numa-topology-awareness-in-openstack-compute/

• OpenStack Flavor Specification Reference: http://docs.openstack.org/admin-guide/compute-flavors.html

• OpenStack CPU Typologies Reference: http://docs.openstack.org/admin-guide/compute-cpu-topologies.html

...