The following sections contain VMware performance tuning recommendations to improve system performance. These performance recommendations are general guidelines and are not all-inclusive.

BIOS Setting Recommendations

Ribbon recommends the following BIOS settings for optimum performance. The following table is intended as a reference. Exact values may vary based on vendor and HW.

ParameterRecommended Setting

CPU power management/ Power Regulator

Maximum performance, or Static High Performance

Intel Hyper-ThreadingEnabled
Intel Turbo BoostEnabled
Intel VT-x (Virtualization Technology)Enabled
Thermal Configuration

Optimal Cooling, or Maximum Cooling

Minimum Processor Idle Power Core C-stateNo C-states
Minimum Processor Idle Power Package C-stateNo Package state
Energy Performance BIASMax Performance

Sub-NUMA Clustering

Disabled
HW PrefetcherDisabled
SRIOVEnabled
Intel® VT-d

Enabled

The BIOS settings shown in the example below are recommended for HP DL380p Gen8 servers. For BIOS settings of other servers, refer to the respective vendor's website.

ParameterRibbon Recommended SettingsDefault Settings
HP Power Profile Maximum PerformanceBalanced Power and Performance
Thermal ConfigurationMaximum CoolingOptimal Cooling
HW PrefetchersDisabledEnabled
Adjacent Sector PrefetcherDisabledEnabled
Processor Power and Utilization MonitoringDisabledEnabled
Memory Pre-Failure NotificationDisabledEnabled
Memory Refresh Rate1x Refresh2x Refresh
Data Direct I/OEnabledDisabled
SR-IOVEnabledDisabled
Intel® VT-dEnabledDisabled

General Recommendations

  • Ensure the number of vCPUs in an instance is always an even number (4, 6, 8, and so on), as hyper threaded vCPUs are used.
  • For best performance, make sure a single instance is confined to a single NUMA. Performance degradation occurs if an instance spans across multiple NUMAs.
  • Ensure the physical NICs associated with an instance are connected to the same NUMA/socket where the instance is hosted. Doing so reduces the remote node memory access, which in turn helps improve the performance.
    • Log in to the ESX host

    • Check the NIC in use by using the esxcli network nic list command.

    • Find out the NUMA affinity of the NIC using the command vsish -e get /net/pNics/<vmnicx>/properties | grep "NUMA".

ESXi Host Configuration Parameters

Use the VMware vSphere client to configure the following ESXi host configuration parameters on the Advanced Settings page (see figure below) before installing the SBC SWe.

Path: Host > Manage > Advanced Settings


ESXi ParameterESXi 6.5 SettingsESXi 6.7 SettingsESXi 7.0 Settings

RecommendedDefaultRecommendedDefaultRecommendedDefault
Cpu.CoschedCrossCall010101
Cpu.CreditAgePeriod100030001000300010003000
DataMover.HardwareAcceleratedInit010101
DataMover.HardwareAcceleratedMove010101
Disk.SchedNumReqOutstandingn/an/an/an/an/an/a
Irq.BestVcpuRouting101010
Mem.BalancePeriodn/an/an/an/an/an/a
Mem.SamplePeriodn/an/an/an/an/an/a
Mem.ShareScanGHz040404
Mem.VMOverheadGrowthLimit042949672950429496729504294967295
Misc.TimerMaxHardPeriod200050000020005000002000500000
Misc.TimerMinHardPeriodn/an/an/an/an/an/a
Net.AllowPT111111
Net.MaxNetifRxQueueLenn/an/an/an/an/an/a
Net.MaxNetifTxQueueLen100020002000200020002000
Net.NetTxCompletionWorldletn/an/an/an/an/an/a
Net.NetVMTxType121212
Net.NetTxWordletn/an/an/an/an/an/a
Numa.LTermFairnessInterval050505
Numa.MonMigEnable010101
Numa.PageMigEnable010101
Numa.PreferHT101010
Numa.RebalancePeriod600002000600002000600002000
Numa.SwapInterval131313
Numa.SwapLoadEnable010101

VM Settings

CPU Settings

To edit CPU, go to Edit instance settings > Virtual Hardware > CPU.

Recommended CPU Settings

ParameterRecommended Settings
Cores Per Socket 
Reservation 

Value = (No. of vCPUs * CPU frequency)/2

or example,

No. of vCPUs associated with the SBC = 32 

CPU Model : Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz

Hence Value = 36800 MHz  ( i.e., 32 x 2300 / 2)

LimitUnlimited
SharesNormal

Memory Settings

To edit Memory, go to Edit instance settings > Virtual Hardware > Memory.

Recommended Memory Settings

ParameterRecommended Settings
RAMAs per requirement.
Reservation 

Same as RAM. 

Check "Reserve all guest memory (All locked)"

LimitUnlimited
SharesNormal

Latency Sensitivity Settings

To edit Latency sensitivity, go to Edit instance settings > VM Options > Advanced > Latency sensitivity.

Configure the VM Latency Sensitivity to High, if the ESX allows it.

Note
  • ESX 6.5 allows configuring latency sensitivity to High even with hyper-threaded CPU reservation.
  • ESX 6.7 and above does not allow configuring latency sensitivity to High without full CPU core reservation.

NUMA Settings

To edit NUMA settings, go to Edit instance settings > VM Options > AdvancedConfiguration Parameters > Edit Configuration.

Configure numa.nodeAffinity based on the NUMA node to which Pkt NICs are attached (as mentioned in General Recommendations). Ensure the VM memory fits in a single NUMA node, so that remote memory access does not happen.

Configure numa.vcpu.preferHT=TRUE. This is required for better cache optimizations. Refer to http://www.staroceans.org/ESXi_VMkernel_NUMA_Constructs.htm for further details.

Configure numa.autosize.once = FALSE.

Setting the Scheduling Affinity of a VMware Instance

By specifying a CPU affinity setting for each virtual machine, you can restrict the assignment of virtual machines to a subset of the available cores. By using this feature, you can assign each virtual machine to cores in the specified affinity set.

Each virtual machine on the Host instance (not just the SBC instance) must have a dedicated "CPU affinity" set, so that one VM will not interfere with another VM.

Procedure

  1. Browse to the virtual machine in the vSphere Client.
  2. Select VM, right-click on the virtual machine, and click Edit Settings.
  3. Under Virtual Hardware, expand CPU.
  4. Under Scheduling Affinity, specify the core numbers
    Use '-' for ranges and ',' to separate values.
    For example, "0, 2, 4-7" would indicate CPU cores 0, 2, 4, 5, 6 and 7.
  5. Select the cores where you want the virtual machine to run and click Save.

Both logical cores of a physical core need assignment in the "Scheduling affinity" set of the same virtual machine. Logical cores on the same physical core have consecutive CPU numbers, so that CPUs 0 and 1 are on the first core together, CPUs 2 and 3 are on the second core, and so on.

You can let the Host OS run some of the CPU cores.

Validation

Use the below ESXI CLI command to validate the "Scheduling affinity" of the VM:

sched-stats -t cpu| grep vcpu

Example output from the above command:

In the preceding example:

Column TitleDescription
NameDisplays the vCPU number: VM Name
CPUDisplays the logical CPU number where the vCPU thread is presently running
AffinityDisplays the "Scheduling Affinity" of the vCPU thread

VMware does not provide any method for the user to pin a vCPU thread to a particular logical core. You can only provide a "Scheduling affinity" set for the VM, and the vCPU thread will get scheduled within the given CPU affinity set.