The Ribbon CNF Solution which comprises the SBC CNe, PSX CNe and RAMP real-time communication products requires some specific settings/configurations on the K8s cluster worker Nodes for the proper functioning and to meet the expected KPIs. The user must complete or enable these settings/configurations on the cluster before onboarding the Ribbon CNF Solution components.
Ribbon provides a Cluster Readiness Checker Tool to run on the allocated cluster (Bare-metal or VM). The tool conducts a comprehensive assessment of the cluster's settings and configurations, comparing them against the expected criteria for onboarding the Ribbon CNF Solution and gives a "Readiness Report" of the current cluster settings/configurations to compare with the expected criteria to decide whether the cluster is ready for onboarding of the Ribbon CNF Solution.
The Cluster Readiness Checker Tool requires the following CPU and RAM configurations to operate.
The following checks are not included in this tool because they cannot be run from a container. You must check these items directly on the Worker Node.
- Check BIOS settings
- Check whether KSM is enabled
- Check whether CPU tuning profile is 'throughput-performance'.
To ensure the results meet the expected Key Performance Indicators (KPIs), you must apply specific settings and configurations to the K8s cluster worker nodes before onboarding the Ribbon CNF Solution components. Refer to the "Readme.md" file (located in the cluster-readiness-checker-12.1.0-latest.tgz file that is downloaded in the procedure below) for instructions on checking dependencies, running the tool, understanding its tasks, and reading the generated report.
To run the CNF Cluster Readiness Checker Tool:
- Download the
cluster-readiness-checker-12.1.2-latest.tgz from the artifactory and untar it into the Jump server.
- Refer to the "Readme.md" file for instructions on checking dependencies, running the tool, understanding its tasks, and reading the generated report.
- Bash:
- Grant 'execute' permission to the script:
chmod +x ReadinessChecker.py - Run the script with the desired pod name:
./ReadinessChecker.py <helm_deployement_name>
The script accomplishes the following actions:
- Obtains the cluster and nodes details.
- Launches the Cluster Readiness Checker pod
- Checks the PVC creation.
- Copies the
- Generates the ReadinessCheckReport.txt and ReadinessCheck.log on the Jump server where the
ReadinessCheck.py script is located.
The primary component of the Readiness Checker Tool is a Python script that performs the following tasks:
- Collects software and CPU version from the K8s cluster and Worker Nodes.
- Launches a Daemonset of the Readiness Checker POD in all Nodes.
- Checks the RWX, RWO and RWX PVC Quota Launch creation.
- Copies the
pod.log file from the pod to the local machine. - Generates the ReadinessCheckReport.txt and ReadinessCheck.log where the
ReadinessCheck.py is located.
The Readiness Checker Pod in each Worker Node accomplishes following tasks:
- The Readiness Checker POD in each Worker Node collects CPU, network, storage details etc.
- Each pod writes the data to the pod's
/var/log/ path.
Cluster Readiness Check Report
=================================
=================================
Date: 2024-03-11 14:24:20
Cluster level details
=======================
Number of nodes in the Cluster : 6
Kubectl Version: v1.26.12+9ed7eae
OpenShift Client Version : 4.9.0
OpenShift Server Version : 4.13.29
Helm Version : v3.7.1+g1d11fcb
FluxCD Version: 2.2.2
Argo Rollout Version: latest
RWX PVC Creation Check: Passed
RWO PVC Creation Check: Passed
RWX PVC Quota Launch Check : Passed
Multus CNI Plugin : Present
NODE Worker Node Level Details
==================================
Node : worker-1.blr-ocp3.lab.rbbn.com
-------------------------------------
Could able to launch Test Pod : Yes
Software Versions:
-----------------
Operating System = Linux CoreOS 413.92.202401100947-0
OS FLAVOUR = Red Hat Enterprise
Kernel Version = 5.14.0-284.48.1.el9_2.x86_64
SR-IOV enabled: Yes
hugepages-1Gi available : Yes
CPU:
----
Model name: Intel(R) Xeon(R) Gold 6230R CPU @ 2.10GHz
CPU op-mode(s): 32-bit, 64-bit
Architecture: x86_64
Byte Order: Little Endian
Core(s) per socket: 26
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU MHz: 2999.979
Testing ephemeral storage write performance (in sec)
-----------------------------------------------------
Results: 21.987 11.461 10.443
Min: 10.443 Max: 21.987 Avg: 14.63
Current IOPS on ephemeral storage : 14.63
Testing external volume write performance (in sec)
---------------------------------------------------
Results: 5.198 4.991 5.130
Min: 4.991 Max: 5.198 Avg: 5.11
Current IOPS on block storage : 5.11
Disk I/O testing using dd commands.
-----------------------------------
Disk checking for Ephemeral Storage : Writing to output file /var/log/myfile.img of block size 1M of input blocks to copy count=1024.
Output : 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.23932 s, 172 MB/s
Disk checking for EmptyDir : Writing to output file /emptydir/myfile.img of block size 1M of input blocks to copy count=1024.
Output : 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.18121 s, 492 MB/s
Disk checking for PVC : Writing to output file /mnt/diskcheck/myfile.img of block size 1M of input blocks to copy count=1024.
Output : 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.99925 s, 179 MB/s
Allotted cpu: 4
Provisioned cpu: 4
CPU Manager Policy : static
Topology_policy: single-numa-node
NIC_Make : Intel Corporation Ethernet
NIC_Version : E810-XXV-2 [8086:0003]
Kernel Driver : vfio-pci
NIC_Make : Intel Corporation Ethernet
NIC_Version : E810-XXV-2 [8086:0003]
Kernel Driver : vfio-pci
PVC Read and Write Check: Passed
Kernel Parameters
--------------------
kernel.sched_rt_runtime_us = -1
fs.file-max = 800000
net.ipv4.conf.default.disable_xfrm = 0
net.ipv4.conf.all.disable_xfrm = 0
net.ipv4.conf.default.disable_policy = 0
net.ipv4.conf.all.disable_policy = 0
vm.dirty_writeback_centisecs = 20
vm.dirty_expire_centisecs = 1000
vm.dirty_background_bytes = 2097152
vm.dirty_bytes = 4194304
net.core.xfrm_larval_drop = 1
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_max_tw_buckets = 262144
net.ipv4.conf.all.send_redirects = 1
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 1
net.ipv4.conf.default.accept_redirects = 1
net.ipv6.conf.default.accept_redirects = 1
net.ipv4.conf.default.secure_redirects = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.default.log_martians = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 0
net.ipv4.tcp_pacing_ss_ratio = 200
kernel.sysrq = 0
net.ipv4.ip_forward = 1
net.ipv4.ip_local_reserved_ports =
kernel.shmmax = 18446744073692774399
net.ipv4.ip_local_port_range = 32768 60999
net.ipv4.tcp_syncookies = 1
kernel.pid_max = 4194304
fs.inotify.max_user_instances = 8192
Role Verification
-------------------------------
For pods
=========================================
'pods' 'get' operation success
'pods' 'list' operation success
'pods' 'patch' operation success
'pods' 'delete' operation success
For endpoints
=========================================
'endpoints' 'get' operation success
'endpoints' 'list' operation success
'endpoints' 'patch' operation success
For deployments.apps
=========================================
'deployments.apps' 'get' operation success
'deployments.apps' 'list' operation success
'deployments.apps' 'patch' operation success
For services
=========================================
'services' 'get' operation success
'services' 'list' operation success
For statefulsets.apps
=========================================
'statefulsets.apps' 'get' operation success
'statefulsets.apps' 'list' operation success
For pods.metrics.k8s.io
=========================================
'pods.metrics.k8s.io' 'get' operation success
'pods.metrics.k8s.io' 'list' operation success
For roles.rbac.authorization.k8s.io
=========================================
'roles.rbac.authorization.k8s.io' 'get' operation success
'roles.rbac.authorization.k8s.io' 'list' operation success
For deployments.apps/scale
=========================================
'deployments.apps/scale' 'get' operation success
'deployments.apps/scale' 'patch' operation success
'deployments.apps/scale' 'update' operation success
For statefulsets.apps/scale
=========================================
'statefulsets.apps/scale' 'get' operation success
'statefulsets.apps/scale' 'patch' operation success
'statefulsets.apps/scale' 'update' operation success
For pods/log
=========================================
'pods/log' 'get' operation success
Index Calculation result for 6 Pods out of 6 Pods
---------------------------------------------------------
Average of Index Calculation
----------------------------
passthrough_index : 1.04
crypto_index : 0.94
transcode_index : 0.94
signalling_index : 0.99
Minimum of Index Calculation
----------------------------
passthrough_index : 1.03
crypto_index : 0.94
transcode_index : 0.93
signalling_index : 0.97
Maximum of Index Calculation
----------------------------
passthrough_index : 1.07
crypto_index : 0.95
transcode_index : 0.95
signalling_index : 1.0
Cluster Readiness Check Report
=================================
=================================
Date: 2024-03-11 14:31:50
Cluster level details
=======================
Number of nodes in the Cluster : 6
Kubectl Version: v1.26.12+9ed7eae
OpenShift Client Version : 4.9.0
OpenShift Server Version : 4.13.29
Helm Version : v3.7.1+g1d11fcb
FluxCD Version: 2.2.2
Argo Rollout Version: latest
RWX PVC Creation Check: Passed
RWO PVC Creation Check: Passed
RWX PVC Quota Launch Check : Passed
Multus CNI Plugin : Present
NODE Worker Node Level Details
==================================
Node : worker-1.blr-ocp3.lab.rbbn.com
-------------------------------------
Could able to launch Test Pod : No
Software Versions:
-----------------
Operating System = Linux CoreOS 413.92.202401100947-0
OS FLAVOUR = Red Hat Enterprise
Kernel Version = 5.14.0-284.48.1.el9_2.x86_64
SR-IOV enabled: Yes
hugepages-1Gi available : Yes
CPU:
----
Model name: NA
CPU op-mode(s): NA
Architecture: NA
Byte Order: NA
Core(s) per socket: NA
Socket(s): NA
NUMA node(s): NA
Vendor ID: NA
CPU MHz: NA
Testing ephemeral storage write performance (in sec)
-----------------------------------------------------
Results: NA
Min: NA Max: NA Avg: NA
Current IOPS on ephemeral storage : NA
Testing external volume write performance (in sec)
---------------------------------------------------
Results: NA
Min: NA Max: NA Avg: NA
Current IOPS on ephemeral storage : NA
Disk I/O testing using dd commands.
-----------------------------------
Disk checking for Ephemeral Storage : Writing to output file /var/log/myfile.img of block size 1M of input blocks to copy count=1024.
Output : NA
Disk checking for EmptyDir : Writing to output file /emptydir/myfile.img of block size 1M of input blocks to copy count=1024.
Output : NA
Disk checking for PVC : Writing to output file /mnt/diskcheck/myfile.img of block size 1M of input blocks to copy count=1024.
Output : NA
Allotted cpu: NA
Provisioned cpu: NA
CPU Manager Policy : NA
Topology_policy: NA
NIC_Make : NA
NIC_Version : NA
Kernel Driver : NA
NIC_Make : NA
NIC_Version : NA
Kernel Driver : NA
PVC Read and Write Check: NA
Kernel Parameters
----------------
kernel.sched_rt_runtime_us = NA
fs.file-max = NA
net.ipv4.conf.default.disable_xfrm = NA
net.ipv4.conf.all.disable_xfrm = NA
net.ipv4.conf.default.disable_policy = NA
net.ipv4.conf.all.disable_policy = NA
vm.dirty_writeback_centisecs = NA
vm.dirty_expire_centisecs = NA
vm.dirty_background_bytes = NA
vm.dirty_bytes = NA
net.core.xfrm_larval_drop = NA
net.ipv4.tcp_max_syn_backlog = NA
net.ipv4.tcp_max_tw_buckets = NA
net.ipv4.conf.all.send_redirects = NA
net.ipv4.conf.default.send_redirects = NA
net.ipv4.conf.all.accept_redirects = NA
net.ipv6.conf.all.accept_redirects = NA
net.ipv4.conf.default.accept_redirects = NA
net.ipv6.conf.default.accept_redirects = NA
net.ipv4.conf.default.secure_redirects = NA
net.ipv4.conf.all.accept_source_route = NA
net.ipv4.conf.default.accept_source_route = NA
net.ipv6.conf.all.accept_source_route = NA
net.ipv4.conf.all.log_martians = NA
net.ipv4.conf.default.log_martians = NA
net.ipv4.icmp_echo_ignore_broadcasts = NA
net.ipv4.icmp_ignore_bogus_error_responses = NA
net.ipv4.tcp_syncookies = NA
net.ipv4.conf.default.rp_filter = NA
net.ipv4.conf.all.rp_filter = NA
net.ipv4.tcp_pacing_ss_ratio = NA
kernel.sysrq = NA
net.ipv4.ip_forward = NA
net.ipv4.ip_local_reserved_ports = NA
kernel.shmmax = NA
net.ipv4.ip_local_port_range = NA
net.ipv4.tcp_syncookies = NA
kernel.pid_max = NA
fs.inotify.max_user_instances = NA
Role Verification
-------------------------------
POD start failed. Hence could not execute Role check.
Index Calculation result for 0 Pods out of 0 Pods
---------------------------------------------------------
Average of Index Calculation
----------------------------
passthrough_index : NA
crypto_index : NA
transcode_index : NA
signalling_index : NA
Minimum of Index Calculation
----------------------------
passthrough_index : NA
crypto_index : NA
transcode_index : NA
signalling_index : NA
Maximum of Index Calculation
----------------------------
passthrough_index : NA
crypto_index : NA
transcode_index : NA
signalling_index : NA