Isolating interrupts (IRQs) from real time workloads like SC and SLB on different dedicated CPUs (Host reserved cores) can minimize or eliminate latency in real-time environments.
Approach 1
- On Openshift(OCP) K8 platform:
Set "globallyDisableIrqLoadBalancing
"
in the performance profile to "true
" to shield the isolated cores from IRQs.
Code Block |
---|
apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
name: manual
spec:
globallyDisableIrqLoadBalancing: true |
- On non-OCP K8s Platforms:
On the worker nodes /etc/sysconfig/irqbalance
file needs to be updated to have either the IRQBALANCE_BANNED_CPULIST
or IRQBALANCE_BANNED_CPUS
parameter with values of CPUs(based on the version of irqbalance service) that need to be banned from IRQ.
Info |
---|
- All the CPUs part of the isolcpus configuration in the grub-line has to be isolated from the IRQ servicing from the above configuration, as we don't know which set of CPUs the container workload would use in advance.
- After updating the above configuration, the irqbalance service has to be restarted using systemctl restart irqbalance.service
- Extra care is advised while allocating host reserved cores; a sufficient amount of CPUs should be allocated for the host processes(i.e., host reserved cores) in this scenario, as the IRQ will land only on the host reserved cores, leading to an increase in CPU utilization.
|
Approach 2
This approach applies to the OCP K8S environment. For certain workloads, the host reserved CPUs are not always sufficient for dealing with device interrupts, and for this reason, device interrupts are not globally disabled on the isolated CPUs.
Device interrupts are load-balanced between all isolated and reserved CPUs to avoid overloading CPUs, except for CPUs with a guaranteed pod running.
Guaranteed pod CPUs are prevented from processing device interrupts when the pod annotation, irq-load-balancing.crio.io, is defined with the value as disabled.
When configured, CRI-O disables device interrupts only when the pod is running.
The corresponding update will be visible (after the latency-sensitive workload pod has been scheduled) in the /etc/sysconfig/irqbalance file
- which would contain container CPUs in the IRQ banned list.
From the SBC CNF helm chart perspective, the following setting needs to be configured:
- disableIrqBalance: a boolean value.
To disable interrupt request processing on vCPUs allocated to the pod for enhanced performance, set this value to true.
true: Results in enhanced performance by banning the IRQ landing on the cores of lantency sensitive workload.
false: default - Allows interrupt request handling on vCPUs allocated to latency sensitive workload. performanceProfileName - a string value.
This parameter must be provided with the name of the OCP performance profile configured on worker nodes hosting latency sensitive pods when IRQ handling needs to be disabled.
Info |
---|
When the first approach is used, the above two parameters should be left with the default value (i.e., false and "") |
Additional worker node configuration for Approach 2
The irqbalance service gets restarted every time the /etc/sysconfig/irqbalance
file has to be updated with the container's CPU details as part of container scheduling.
Since the SC pods are dynamically scale-able entities based on traffic subjected to the SBC CNe cluster, the SC pods would frequently be created and destroyed - resulting in frequent restarts of irqbalance service on the worker node.
By default, the system allows five restarts (StartLimitBurst) in 10 seconds (StartLimitIntervalSec), which is not sufficient in certain scaling occasions, especially during the initial scale-out of SC deployment immediately after helm installation to minimum active SC pods.
Therefore, the irqbalance service configuration file /usr/lib/systemd/system/irqbalance.service
should be updated to have StartLimitBurst set to 60 to account for the maximum number of irqbalance service restarts upon the SC pod instantiation.
A sample configuration would be as follows:
Code Block |
---|
[Unit]Description=irqbalance daemon
ConditionVirtualization=!container
[Service]EnvironmentFile=/etc/sysconfig/irqbalance
ExecStart=/usr/sbin/irqbalance --foreground $IRQBALANCE_ARGS
StartLimitBurst=60 <----------------------------------------------------
New parameter.
[Install]
WantedBy=multi-user.target |
After modifying the irqbalance.service unit file, you need to reload systemd and then restart the service for the changes to take effect:
Reload systemd to pick up the changes to the unit files:
Code Block |
---|
systemctl daemon-reload |
Restart or reload the service:
Code Block |
---|
systemctl restart irqbalance.service |
or
Code Block |
---|
systemctl reload irqbalance.service |
For more information, refer to: https://docs.openshift.com/container-platform/4.15/scalability_and_performance/cnf-low-latency-tuning.html