Page History
| Panel | ||||
|---|---|---|---|---|
Contents:
|
Overview
The Cloud Diagnostic tool is also known as cloud_diags.py script that facilitates debugging of Sonus cloud devices. The Cloud Diagnostic tool is a test agent that can be incorporated into all Sonus cloud instances and used in production cloud environment where the user has access only to the VM hosting the Sonus instance. It is currently supported on SBC, PSX and EMS.
- System information:
- CPU information
- Platform information
- Storage information
- Memory information
- Basic disk IOPS
- Network interfaces information
- Check if Metadata supported
- Check if Config-drive supported
- Check if DHCP used
- Check if SBC SR-IOV supported
- Network connectivity and performance testing:
- Packet load throughput measurements on specified interfaces (mgtx, pktx) for different packet sizes (TCP/UDP).
- Request/response round trip measurements (latency) with time stamps.
Prerequisites
Create an IP Access Control List (ACL) rules (only for SBC) that is used to allow access to incoming IP packets. A separate ACL rule is required for each protocol type. The ACL rule allows a packet on a specific IP address/port and can be configured to have an “fillRate” and “bucketSize” set to “Unlimited”. Refer Managing Access Control Lists.
An example of a user generated ACL rule on SBX CLI:
- set addressContext default ipAccessControlList rule <rule_name> protocol tcp destinationIpAddress <destination_ip> destinationAddressPrefixLength <dest_prefix_len> destinationPort any precedence <precedence_no>
- set addressContext default ipAccessControlList rule <rule_name> bucketSize unlimited fillRate unlimited
- set addressContext default ipAccessControlList rule <rule_name> state enabled
Limitations: Even with the ACL rule configured to Unlimited; the rule is limited by the Operator Aggregate Policer. The total amount of bandwidth allowed on the device is as follows:
- Hard coded to 20,000 packets/second
- Based on a calculation of 1500 bytes/packet. Expect to get up to 1500 bytes * 8 bytes/bit * 20000 pkts = 240Mbit/second
CLI Changes
| Note |
|---|
The outputs displayed in this section may differ based on the actual system configurations and various other requirements. You may consider the outputs as an example. |
The Cloud Diagnostic tool consists of following commands, each of these commands accomplishes one or more feature described in the feature overview section.
- To run on an SBC system, ssh to the SBC Linux shell as linuxadmin user. The script is in /home/linuxadmin/cloud_diags/cloud_diags.py directory.
- To run on a PSX or an EMS system, ssh to the system Linux shell as root user. The script is in /root/cloud_diags/cloud_diags.py directory.
| Note |
|---|
Within the tool, command names are case-sensitive and required to be entered as shown. Tab completion is supported for command names. |
help
This command provides information about the available commands or specific information about a command.
command syntax: help [command_name]
Optional parameter(s): command_name
Example:
To know the list of available commands:
Code Block (Enter command or ?) : ? Supported commands (type help <command_name>): ============================================== EOF showCpuInfo showStorageInfo checkDiskPerformance showFullSystemInfo startMessageSizeTest exit showMemoryInfo startNetworkServer help showNetworkInfo startNetworkTest quit showPlatformInfo
To know about a specific command: Command syntax: help command_name
Code Block (Enter command or ?) : help startNetworkTest Start the Network Test on the Client Command Syntax: startNetworkTest -p tcp --si <remote_server_ip> [--sp <remote_server_port_no>] --li <local_ip> [--lp <local_port_no>] [-r <msg_rate_per_sec>] [-m <message_size>] [-c <run_count>] startNetworkTest -p udp --si <remote_server_ip> [--sp <remote_server_port_no>] [-r <msg_rate_per_sec>] [-m <message_size>] [-c <run_count>] Mandatory Parameters: -p <protocol_type> --si <remote_server_ip> --li <local_client_ip> (only required for TCP) Optional parameters: --sp <remote_server_port_no> (default TCP=40000, UDP=5061) --lp <local_client_port_no> (default TCP=20000, UDP=5061) -r <msg_rate_per_sec> (default = 1000) -m <msg_size> (default = 500) -c <run_count> (default = 1) -f (output test result to file named <applicationType>_<hostname>_performanceResult_YYYYMMDD-HHMMSS.txt) Example: startNetworkTest -p tcp --si 10.1.10.1 --sp 40000 --li 10.20.2.20 --lp 20000 -r 1000 -m 500 -c 1 -f startNetworkTest -p udp --si 10.1.10.1 --sp 5061 -r 10000 -m 250 -c 5 -f Note: Please ensure destination network server is running before running this test.
showCpuInfo
Displays information on the CPU model, CPU virtualization, and number of cores available for the application.
Help information: Displays information on vCPU's, including clock speed, sockets, cores, and threads.
| Code Block |
|---|
(Enter command or ?) : showCpuInfo System CPU info : CPU Model = Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz (2596.958 MHz) Number of CPU's = 4 ( 4 Sockets * 1 Cores * 1 Threads) CPU Virtualization = VT-x |
showMemoryInfo
Displays information of total memory, used memory and available memory on the system.
Help information: Displays information on system memory including available and free memory.
| Code Block |
|---|
(Enter command or ?) : showMemoryInfo System memory info : Total Memory = 9.79 GB, Free = 5.21 GB, Buffers = 0.00 GB, Cached = 0.62 GB Actual Free = 5.84 GB (Free+Buffers+Cached) |
showStorageInfo
Displays information on the total storage, used storage and available capacity of the storage device on the system.
Help information: Displays information on system storage/disks including used and free space.
| Code Block |
|---|
(Enter command or ?) : showStorageInfo System storage info : Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/debian-root ext3 78G 8.9G 66G 12% / /dev/vda1 ext4 184M 34M 142M 20% /boot /dev/mapper/debian-root ext3 78G 8.9G 66G 12% /tmp |
showPlatformInfo
Displays Linux platform information.
Help information: Displays information on platform, includes node name, app type, Linux platform info, architecture, DHCP use, SRIOV, config-drive and metadata info.
| Code Block |
|---|
(Enter command or ?) : showPlatformInfo
System platform info :
Node name = vsbc1
Application type = SBC, Package name = sbc-V06.01.00-A013
Platform Info = Linux-3.2.84-x86_64-with-debian-8, Architecture = ('64bit', 'ELF')
DHCP = DHCP used
SR-IOV support = Not Supported
Config Drive = /dev/disk/by-label/config-2
Checking for Metadata .....
Metadata - Not available |
showNetworkInfo
Displays information on network interfaces on the system.
Help information: Displays information on network interfaces including configured IP addresses.
| Code Block |
|---|
(Enter command or ?) : showNetworkInfo
System network interfaces info :
pkt0 Link encap:Ethernet HWaddr fa:16:3e:bb:b8:be
inet6 addr: fe80::f816:3eff:febb:b8be/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
ha0 Link encap:Ethernet HWaddr fa:16:3e:9a:b6:82
inet addr:192.168.2.94 Bcast:192.168.2.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fe9a:b682/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
pkt1 Link encap:Ethernet HWaddr fa:16:3e:11:82:6c
inet6 addr: fe80::f816:3eff:fe11:826c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
mgt0 Link encap:Ethernet HWaddr fa:16:3e:a5:0c:22
inet addr:10.54.221.94 Bcast:10.54.221.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fea5:c22/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
|
checkDiskPerformance
Displays information collected from basic disk IO performance tests.
Help information:
- Performs basic check of the disk IO performance by reading and writing 10K data records 1,000 times from/to disk.
- Displays time and speed in MB/sec taken to read/write data.
| Code Block |
|---|
(Enter command or ?) : checkDiskPerformance
Basic disk IO performance (may take a few secs to complete) :
Performing write test - Writing 10K data records 1,000 times to disk :
1000+0 records in
1000+0 records out
10240000 bytes (10 MB) copied, 0.608691 s, 16.8 MB/s
Performing read test - Copying 10K data records 1,000 times from disk to null stream:
1000+0 records in
1000+0 records out
10240000 bytes (10 MB) copied, 0.0260588 s, 393 MB/s |
showFullSystemInfo
Displays full system information.
Command syntax: showFullSystemInfo [-f]
Optional Parameter(s): -f
If the optional -f flag is specified, output from the command is written to a file named ApplicationName_HostName_fullsysinfo_YYYYMMDD-HHMMSS.txt
Example:
- For SBC - SBC_vsbc_fullsysinfo_20170322-121008.txt
- For EMS - EMS_vems_fullsysinfo_20170322-121008.txt
- For PSX - PSX_vpsx_fullsysinfo_20170322-121008.txt
Where YYYYMMDD-HHMMSS is the date/time of the file created.
Note - In command syntax -f flag mentioned inside bracket to indicate its an optional parameter, actual command do not require use of brackets
Help information: Displays comprehensive system information including platform, CPU, memory, storage, network interfaces and checks disk IOPs.
Displays output of each of the following commands:
- showPlatformInfo
- showCpuInfo
- showMemoryInfo
- showStorageInfo
- showNetworkInfo
- checkDiskPerformance
| Code Block |
|---|
(Enter command or ?) : showFullSystemInfo
Full System Info, collected on - Thu Jun 1 03:07:33 2017
System platform info :
Node name = vsbc1
Application type = SBC, Package name = sbc-V06.01.00-A013
Platform Info = Linux-3.2.84-x86_64-with-debian-7.11, Architecture = ('64bit', 'ELF')
DHCP = DHCP used
SR-IOV support = Not Supported
Config Drive = /dev/disk/by-label/config-2
Checking for Metadata .....
Metadata - Not available
System CPU info :
CPU Model = Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz (2596.958 MHz)
Number of CPU's = 4 ( 4 Sockets * 1 Cores * 1 Threads)
CPU Virtualization = VT-x
System memory info :
Total Memory = 9.79 GB, Free = 5.22 GB, Buffers = 0.00 GB, Cached = 0.63 GB
Actual Free = 5.86 GB (Free+Buffers+Cached)
System storage info :
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/debian-root ext3 78G 8.9G 66G 12% /
/dev/vda1 ext4 184M 34M 142M 20% /boot
/dev/mapper/debian-root ext3 78G 8.9G 66G 12% /tmp
System network interfaces info :
pkt0 Link encap:Ethernet HWaddr fa:16:3e:bb:b8:be
inet6 addr: fe80::f816:3eff:febb:b8be/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
ha0 Link encap:Ethernet HWaddr fa:16:3e:9a:b6:82
inet addr:192.168.2.94 Bcast:192.168.2.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fe9a:b682/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
pkt1 Link encap:Ethernet HWaddr fa:16:3e:11:82:6c
inet6 addr: fe80::f816:3eff:fe11:826c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
mgt0 Link encap:Ethernet HWaddr fa:16:3e:a5:0c:22
inet addr:10.54.221.94 Bcast:10.54.221.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fea5:c22/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Basic disk IO performance (may take a few secs to complete) :
Performing write test - Writing 10K data records 1,000 times to disk :
1000+0 records in
1000+0 records out
10240000 bytes (10 MB) copied, 0.661109 s, 15.5 MB/s
Performing read test - Copying 10K data records 1,000 times from disk to null stream:
1000+0 records in
1000+0 records out
10240000 bytes (10 MB) copied, 0.106902 s, 95.8 MB/s |
| Code Block |
|---|
(Enter command or ?) : showFullSystemInfo -f
Full System Info, collected on - Thu Jun 1 03:08:33 2017
System platform info :
Node name = vsbc1
Application type = SBC, Package name = sbc-V06.01.00-A013
Platform Info = Linux-3.2.84-x86_64-with-debian-7.11, Architecture = ('64bit', 'ELF')
DHCP = DHCP used
SR-IOV support = Not Supported
Config Drive = /dev/disk/by-label/config-2
Checking for Metadata .....
Metadata - Not available
System CPU info :
CPU Model = Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz (2596.958 MHz)
Number of CPU's = 4 ( 4 Sockets * 1 Cores * 1 Threads)
CPU Virtualization = VT-x
System memory info :
Total Memory = 9.79 GB, Free = 5.21 GB, Buffers = 0.00 GB, Cached = 0.63 GB
Actual Free = 5.84 GB (Free+Buffers+Cached)
System storage info :
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/debian-root ext3 78G 8.9G 66G 12% /
/dev/vda1 ext4 184M 34M 142M 20% /boot
/dev/mapper/debian-root ext3 78G 8.9G 66G 12% /tmp
System network interfaces info :
pkt0 Link encap:Ethernet HWaddr fa:16:3e:bb:b8:be
inet6 addr: fe80::f816:3eff:febb:b8be/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
ha0 Link encap:Ethernet HWaddr fa:16:3e:9a:b6:82
inet addr:192.168.2.94 Bcast:192.168.2.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fe9a:b682/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
pkt1 Link encap:Ethernet HWaddr fa:16:3e:11:82:6c
inet6 addr: fe80::f816:3eff:fe11:826c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
mgt0 Link encap:Ethernet HWaddr fa:16:3e:a5:0c:22
inet addr:10.54.221.94 Bcast:10.54.221.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fea5:c22/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Basic disk IO performance (may take a few secs to complete) :
Performing write test - Writing 10K data records 1,000 times to disk :
1000+0 records in
1000+0 records out
10240000 bytes (10 MB) copied, 0.622806 s, 16.4 MB/s
Performing read test - Copying 10K data records 1,000 times from disk to null stream:
1000+0 records in
1000+0 records out
10240000 bytes (10 MB) copied, 0.0376839 s, 272 MB/s
Test output also written to file - SBC_vsbc1_fullsysinfo_20170601-030833.txt |
startNetworkServer
This command runs test server with parameters. The user is required to select the protocol type, IP address and optional port for the test. The server acts as a listener for that protocol type; receives and forwards messages from/to the Client.
To start the server from the diagnostics tool menu, execute below command:
| Code Block |
|---|
startNetworkServer -p <protocol_type> --li <servers_local_ip> [--lp <servers_local_port_no>] |
| Mandatory Parameters | ||
Parameter | Description / Usage | Example |
-p | Specify the Protocol Type of test server. Values:
| TCP/UDP |
--li | Specify the Local Server IP (IPv4 or IPv6) address. | IPV4 and Protocol - startNetwrokServer -p tcp --li 1.1.1.1 IPV6 and Protocol - startNetwrokServer -p udp --li 2001:0DB8:AC10:FE01:: |
| Optional Parameters | ||
| Parameter | Description / Usage | Example |
--lp | Specify the Local Server Port Number. Values:
If the user does not specify a Port Number, a default port number is assigned. | IP (IPV4 or IPV6), Protocol, and Port - startNetwrokServer -p tcp --li 1.1.1.1 --lp 5061 |
| Code Block |
|---|
This diagnostics tool is running as TCP Server to receive on IP = fd00:10:6b50:42c0::1c1 port = 40000
Waiting ...
Enter Ctrl-c to exit server mode and return to command prompt |
| Note |
|---|
When the diagnostics tool runs as a server, it listens to the port specified in --lp (or default port), and responds to any incoming packet from the client. The standard output connected to the server displays the message "Waiting ...", without displaying any further information. To exit the server mode and return to the command prompt, press the Ctrl+C key combination. |
startNetworkTest
This command runs client side performance test. The Client side is responsible for running the test and sending packets to the server. Server should be running when the client test is started.
To start the client test from the diagnostics tool menu, execute below command:
| Code Block |
|---|
startNetworkTest -p <protocol_type> --si <remote_server_ip> [--sp <remote_server_port_no>] --li <local_ip> [--lp <local_port_no>] [-r <msg_rate_per_sec>] [-m <message_size>] [-c <run_count>] |
Mandatory Parameters | |||
Parameter | Description / Usage | Length/Range | Example |
-p | Specify the Protocol Type of test server Values:
| N/A | TCP/UDP |
--si | Specify the Remote Server IP (IPv4 or IPv6) address. | N/A | IP4: 1.1.1.1 IPV6: 2001:0DB8:AC10:FE01:: |
--li | Specify the Local Server IP (IPv4 or IPv6) address. Mandatory for TCP and not applicable for UDP. | N/A | IP4: 1.1.1.1 IPV6: 2001:0DB8:AC10:FE01:: |
Optional Parameters | |||
Parameter | Description / Usage | Length/Range | Example/Default Value |
--sp | Specify the Remote Server Port Number. If the user does not specify a Port Number, a default port number is assigned. | TCP: 1024 - 65535 UDP: 5060 or 5061 | TCP - 40000 UDP - 5061 |
--lp | Specify the Local Client Port Number. This parameter is not applicable for UDP protocol. | TCP: 1024 - 65535 | TCP - 20000 |
-r | Specify Message Rate Per Second. | 1- 10000 | 1000 |
-m | Specify Message Size. | 100-10000 | 500 |
-c | Specify Run Count (the number of time the test is to be run). | 1-100 | 1 |
-f | Specify whether test result is written to a file. If, -f flag is specified, the result is displayed and exported to a text file. Naming convention of a text file:
| N/A | Application Name_Host Name_performanceResult_YYYYMMDD-HHMMSS.txt SBC_vsbc1_performanceResult_20170601-101112.txt EMS_vems1_performanceResult_20170601-101112.txt PSX_vpsx1_performanceResult_20170601-101112.txt |
StartNetwork command output with TCP protocol
| Code Block |
|---|
startNetworkTest -p tcp --si fd00:10:6b50:43a0::dd --li fd00:10:6b50:43a0::d8 -r 10000 -m 500
Performance Measurement for TCP
User Input(s) |
Server Address | fd00:10:6b50:43a0::dd 40000
Local Address | fd00:10:6b50:43a0::d8 20000
Protocol Type | TCP
Port Type | MGMT
Number of Messages| 10000
Message Size | 500
Performance Test Results
Fastest | 0.000619
Slowest | 0.017394
Bucket 1 (Range 0.000619 - 0.003974)| 4467
Bucket 2 (Range 0.003974 - 0.007329)| 3796
Bucket 3 (Range 0.007329 - 0.010684)| 1308
Bucket 4 (Range 0.010684 - 0.014039)| 228
Bucket 5 (Range 0.014039 - 0.017394)| 201
Total Time | 0.373096
Packets Lost | 0 |
Example Usage – TCP
| Note |
|---|
Scenarios below are for illustration purpose only. |
Scenario | Command |
User has All Parameters (without file store command) | startNetwrokTest -p tcp --si 1.1.1.1 --sp 40000 --li 2.2.2.2 --lp 20000 -r 2000 -m 1000 -c 2 |
User has All Parameters (with file store command) | startNetwrokTest -p tcp --si 1.1.1.1 --sp 40000 --li 2.2.2.2 --lp 20000 -r 2000 -m 1000 -c 2 -f |
User has only | |
IPv 4 and Protocol | startNetwrokTest -p tcp --si 1.1.1.1 --li 2.2.2.2 |
IPv 6 and Protocol | startNetwrokTest -p tcp --si 2001:0DB8:AC10:FE01:: --li 2001::0DB8:AC10 |
IP, Protocol and Server Port | startNetwrokTest -p tcp --si 1.1.1.1 --sp 40000 --li 2.2.2.2 |
IP, Protocol and local Port | startNetwrokTest -p tcp --si 1.1.1.1 --li 2.2.2.2 --lp 20000 |
IP, Protocol and Port | startNetwrokTest -p tcp --si 1.1.1.1 --sp 40000 --li 2.2.2.2 --lp 20000 |
StartNetwork command output with UDP protocol
| Code Block |
|---|
startNetworkTest -p udp --si fd00:10:6b50:43a0::dd -r 1000 -m 500
Performance Measurement for UDP
User Input(s) |
Server Address | fd00:10:6b50:43a0::dd 5061
Local Address | Not Applicable
Protocol Type | UDP
Port Type | MGMT
Number of Messages| 1000
Message Size | 500
Performance Test Results
Fastest | 0.000488
Slowest | 0.007309
Bucket 1 (Range 0.000488 - 0.001852)| 545
Bucket 2 (Range 0.001852 - 0.003216)| 146
Bucket 3 (Range 0.003216 - 0.004580)| 158
Bucket 4 (Range 0.004580 - 0.005944)| 124
Bucket 5 (Range 0.005944 - 0.007309)| 27
Total Time | 1.067999
Packets Lost | 0 |
Example Usage – UDP
Scenario | Command |
User has All Parameters (without file store command) | startNetwrokTest -p udp --si 1.1.1.1 --sp 5061 -r 2000 -m 1000 -c 2 |
User has All Parameters (with file store command) | startNetwrokTest -p udp --si 1.1.1.1 --sp 5061 -r 2000 -m 1000 -c 2 -f |
User has only | |
IPv 4 and Protocol | startNetwrokTest -p udp --si 1.1.1.1 |
IPv 6 and Protocol | startNetwrokTest -p udp --si 2001:0DB8:AC10:FE01:: |
IP, Protocol and Server Port | startNetwrokTest -p udp --si 1.1.1.1 --sp 5061 |
startMessageSizeTest
This command runs client side message size test. The Client side is responsible for running the test and sending packets to the server.
To start the message size test from the diagnostics tool menu, execute below command:
| Code Block |
|---|
startMessageSizeTest -p <protocol_type> --si <remote_server_ip> [--sp <remote_server_port_no>] |
Mandatory Parameters | |||
Parameter | Description / Usage | Length/Range | Example/Default Value |
-p | Specify the Protocol Type of test server, only UDP is allowed. | N/A | UDP |
--si | Specify the Remote Server IP (IPv4 or IPv6) address. Note: IP type must match in both server and message size test client to be successful. | N/A | IP4: 1.1.1.1 IPV6: 2001:0DB8:AC10:FE01:: |
Optional Parameters | |||
Parameter | Description / Usage | Length/Range | Example/Default Value |
--sp | Specify the Remote Server Port Number. If the user does not specify a Port Number, a default port number is assigned. | 5060 or 5061 | 5061 |
Example Usage – UDP
Scenario | Command |
User has All Parameters | startMessageSizeTest -p udp --si 1.1.1.1 --sp 5061 |
User has | |
IPv 4 and Protocol | startMessageSizeTest -p udp --si 1.1.1.1 |
IPv 6 and Protocol | startMessageSizeTest -p udp --si 2001:0DB8:AC10:FE01:: |
| Note |
|---|
Ensure server is active before running client message size test to be successful. |
quit/exit
Each of the command allows user to exit from the command mode of tool.
Command syntax: quit / exit
Example: Exit from tool command prompt and return to Linux terminal command prompt.
| Code Block |
|---|
(Enter command or ?) : ? Supported commands (type help <command_name>): ============================================== EOF showCpuInfo showStorageInfo checkDiskPerformance showFullSystemInfo startMessageSizeTest exit showMemoryInfo startNetworkServer help showNetworkInfo startNetworkTest quit showPlatformInfo |
Overview
Content Tools
Support diagnostics: