In this section:
Overview
Environment
- SBC SWe on VMware
- Require nine or more VLANs per PKT port
- X710 NIC
Purpose
Use the procedure below to enable the 'Trust' mode on the VFs of the X710 NIC in order to allow nine or more VLANs per PKT port. The Intel X710 does not support more than eight VLANs per VF when running in 'Untrusted' mode.
IMPORTANT
It is important to perform this NIC procedure to avoid potential network outages due to NIC misconfiguration.
Steps to Identify the Existence of the Issue
Prerequisites
On the VMware ESXi host server, ensure the X710 NIC i40en driver and firmware are installed as per recommendations on the page Attach SR-IOV Interface in VMware ESXi 7.0 and Above.
Use the following command to get the X710 NIC i40en driver and firmware versions.
esxcli network nic get -n <vmnic name>
(<vmnic name> is the X710 NIC port name)
Example:
[root@rengali:~] esxcli network nic get -n vmnic0
Advertised Auto Negotiation: true
Advertised Link Modes: Auto, 1000BaseSR/Full, 10000BaseSR/Full, 25000BaseSR/Full
Auto Negotiation: true
Cable Type: FIBRE
Current Message Level: 0
Driver Info:
Bus Info: 0000:04:00:1
Driver: i40en
Firmware Version: 7.20 0x80007a10 1.2585.0
Version: 1.10.9.0
Link Detected: true
Link Status: Up
Name: vmnic9
PHYAddress: 0
Pause Autonegotiate: false
Pause RX: false
Pause TX: false
Supported Ports: FIBRE
Supports Auto Negotiation: true
Supports Pause: true
Supports Wakeon: false
Transceiver:
Virtual Address: 00:50:56:5f:b9:7d
Wakeon: None
Note
In the above example, X710 NIC i40en driver version is '1.10.9', and the firmware version is '7.20'.
Identify the Issue
If the SBC SWe running on VMware, with untrusted VFs as PKT interface, is configured to have 9 or more VLANs tagged interfaces, then the following error logs are observed:
Check VLAN add failed error log message in the 'np.log' file on the SBC SWe.
Path: /home/log/sonus/np/np.logLog snippet:ERROR: pkt0: vlan 10 add failed with error -56.
Check VLAN add failed message in the 'vmkernel.log' file on the VMware ESXi host server.
Path: /var/log/vmkernel.logLog snippet:i40en: i40en_AddVlanMsg:3270: VF is not trusted, switch the VF to trusted mode to add more VLAN addresses i40en: i40en_SendMsgToVf:1689: vmnic0 VF 1 failed opcode 12, error: -5
From the above error message, make a note of the VF ID.
(In the above example error, the VF ID is "1" and the corresponding port name of the underlying host is "vmnic0")
These values are used to enable 'Trust' mode on the VFs associated with the SBC SWe from the host using the VMware's esxcli intnet commands (for use later in this document).
Solution
Enable the trust mode for the VFs associated with SBC SWe to resolve this issue.
Enable Trust Mode on X710 VFs for VMware
- Download and install the Intel ESXCLI plug-in for managing Intel® Ethernet Network Adapters on the VMware ESXi host server.
(For Intel ESXCLI plug-in and detailed documentation, go to https://www.intel.com/content/www/us/en/download/19380/682980/intel-esxcli-plug-in-for-managing-intel-ethernet-network-adapters.html) - Once the Intel ESXCLI plug-in 'Intel-intnetcli_1.6.5.0__esx7.0.zip' package file are downloaded, copy the package file to the VMWare ESXi host server and install it.
- Create a directory and copy the package file. We can create a directory under 'datastore1' directory. Path: /vmfs/volumes/datastore1.
Check the package file.
[root@rengali:/vmfs/volumes] ls -lrt lrwxr-xr-x 1 root root 35 Aug 16 06:42 datastore1 -> 607ebe6e-52de583e-3446-246e964e2464 [root@rengali:/vmfs/volumes] cd /vmfs/volumes/datastore1 [root@rengali:/vmfs/volumes/607ebe6e-52de583e-3446-246e964e2464] cd intnetcli_1.6.5.0 [root@rengali:/vmfs/volumes/607ebe6e-52de583e-3446-246e964e2464/intnetcli_1.6.5.0] ls -lrt total 64 -rwx------ 1 root root 40952 Aug 16 06:43 Intel-intnetcli_1.6.5.0__esx7.0.zip [root@rengali:/vmfs/volumes/607ebe6e-52de583e-3446-246e964e2464/intnetcli_1.6.5.0]
Unzip the package file.
[root@rengali:/vmfs/volumes/607ebe6e-52de583e-3446-246e964e2464/intnetcli_1.6.5.0] unzip Intel-intnetcli_1.6.5.0__esx7.0.zip Archive: Intel-intnetcli_1.6.5.0__esx7.0.zip inflating: Intel-intnetcli_intnetcli.1.6.5.0-700.15843807_18728558.zip inflating: doc/README.txt
Install the package.
[root@rengali:/vmfs/volumes/607ebe6e-52de583e-3446-246e964e2464/intnetcli_1.6.5.0] esxcli software component apply -d /vmfs/volumes/607ebe6e-52de583e-3446-246e964e2464/intnetcli_1.6.5.0/Intel-intnetcli_intnetcli.1.6.5.0-700.15843807_18728558.zip Installation Result Components Installed: Intel-intnetcli_intnetcli.1.6.5.0-700.15843807 Components Removed: Components Skipped: Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective. Reboot Required: true
- Reboot the VMware ESXi host server to apply the changes. Turn off all the VMs gracefully before rebooting the host.
- Get the VF trust mode status on the VMware ESXi host server. Use the following command from the ESXi shell on the SR-IOV vmnic.
esxcli intnet sriovnic vf get -v <vf number> -n <vmnic name>
Where,<vmnic name>is the SR-IOV port to which the SBC SWe VM is associated.<vf number>is the VF ID assigned to the SBC SWe VM instance.Example:[root@rengali:~] esxcli intnet sriovnic vf get -v 1 -n vmnic0 VF ID Trusted Spoof Check ----- ------- ----------- 1 false true
Locate the SR-IOV Port and VF ID Associated with the SBC SWe VM
Get the 'World ID' of the running SBC SWe VM using the command:
NoteThe below command gives the World ID for each VM in the running state. Ensure that the SBC VM is in a running state before issuing the command.
In this document, you will modify the trust mode of VFs associated with the SBC SWe VM named Test_VM1. The world ID for this VM is 2105037.
esxcli vm process listExample:[root@rengali:~] esxcli vm process list Test_VM1 World ID: 2105037 Process ID: 0 VMX Cartel ID: 2105036 UUID: 56 4d cf 76 f1 3f bf 1c-82 e0 9a 5f df 43 b5 bc Display Name: Test_VM1 Config File: /vmfs/volumes/607ebe6e-52de583e-3446-246e964e2464/Test_VM1/Test_VM1.vmx Test_VM2 World ID: 2107118 Process ID: 0 VMX Cartel ID: 2107117 UUID: 56 4d f8 55 80 52 b7 4f-25 71 9c 5f a1 ae 9a 1f Display Name: Test_VM2 Config File: /vmfs/volumes/607ebe6e-52de583e-3446-246e964e2464/Test_VM2/Test_VM2.vmx
Get the X710 NIC SR-IOV port name using the command:
esxcli network nic listExample:[root@rengali:~] esxcli network nic list Name PCI Device Driver Admin Status Link Status Speed Duplex MAC Address MTU Description ------- ------------ ---------- ------------ ----------- ----- ------ ----------------- ---- ----------- vmnic0 0000:04:00.0 i40en Up Up 10000 Full 3c:fd:fe:c8:dd:60 1500 Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 vmnic1 0000:04:00.1 i40en Up Up 10000 Full 3c:fd:fe:c8:dd:61 1500 Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 vmnic2 0000:01:00.2 igbn Up Up 1000 Full 24:6e:96:4e:24:66 1500 Intel Corporation Gigabit 4P I350-t rNDC vmnic3 0000:01:00.3 igbn Up Down 0 Half 24:6e:96:4e:24:67 1500 Intel Corporation Gigabit 4P I350-t rNDC
In this example, the X710 NIC SR-IOV ports names are
vmnic0andvmnic1. The port names may differ on another VMware ESXi setup.Get the X710 NIC SR-IOV port name and VF ID associated with the SBC SWe VM using the command:
esxcli network sriovnic vf list -n <vmnic name>Example:We can find the VF ID associated with the SBC SWe VM using the 'World ID'. As in the above case, 1: The 'World ID' for SBC SWe 'Test_VM1' is 2105037. 2: The X710 NIC SR-IOV ports names are vmnic0 and vmnic1. Check VF list of the both vmnic0 and vmnic1. [root@rengali:~] esxcli network sriovnic vf list -n vmnic0 VF ID Active PCI Address Owner World ID ----- ------ -------------- -------------- 0 true 00000:004:10.0 2107118 1 true 00000:004:10.1 2105037 2 true 00000:004:10.2 3 true 00000:004:10.3 4 false 00000:004:10.4 - 5 false 00000:004:10.5 - [root@rengali:~] esxcli network sriovnic vf list -n vmnic1 VF ID Active PCI Address Owner World ID ----- ------ -------------- -------------- 0 false 00000:004:02.0 - 1 false 00000:004:02.1 - 2 false 00000:004:02.2 - 3 false 00000:004:02.3 - 4 false 00000:004:02.4 - 5 false 00000:004:02.5 -NoteFrom the above output, you can infer that for the SBC SWe Test_VM1, the X710 NIC SR-IOV port name is 'vmnic0' and the VF ID is 1.
To enable VF trust mode, use the following command:
esxcli intnet sriovnic vf set -t true -v <vf number> -n <vmnic name>
Where,<vmnic name>is the SR-IOV port to which the SBC SWe VM is associated.<vf number>is the VF ID assigned to the SBC SWe VM instance.Example:[root@rengali:~] esxcli intnet sriovnic vf set -t true -v 1 -n vmnic0 Trusted mode is set to true and spoof check is set to true
Set VF trust mode persistent across reboots using the command.
esxcli system module parameters set -a -m i40en -p "trust_all_vfs=<array of int>"<array of int> , 0 = disable (default), 1 = enableFor example:[root@rengali:~] esxcli system module parameters set -a -m i40en -p "trust_all_vfs=1,1" [root@rengali:~] esxcli system module parameters list -m i40en Name Type Value Description ------------- ------------ ----------- ----------- DRSS array of int Enable/disable the DefQueue RSS(default = 0 ) EEE array of int Energy Efficient Ethernet feature (EEE): 0 = disable, 1 = enable, (default = 1) LLDP array of int Link Layer Discovery Protocol (LLDP) agent: 0 = disable, 1 = enable, (default = 1) MaxRdmaInts int Maximum RDMA Interrupts (default = 16) RDMA array of int Enable RDMA support 0 = disable, 1 = enable, (default = 0) RSS array of int Enable/disable the NetQueue RSS( default = 1 ) RxITR int Default RX interrupt interval (0..0xFFF), in microseconds (default = 50) TxITR int Default TX interrupt interval (0..0xFFF), in microseconds, (default = 100) VMDQ array of int Number of Virtual Machine Device Queues: 0/1 = disable, 2-16 enable (default =8) max_vfs array of int Maximum number of VFs to be enabled (0..128) trust_all_vfs array of int 1,1 Always set all VFs to trusted mode 0 = disable (default), other = enable
NoteIn the above example, "trust_all_vfs=1,1" are an array of values, which references each NIC in order using the i40en driver.
- Reboot the VMware ESXi host server.
Verification
To verify the VF is configured as trusted. use the following command.
esxcli intnet sriovnic vf get -v <vf number> -n <vmnic name>
Here,
<vmnic name> is the SR-IOV port to which the SBC SWe VM is associated.
<vf number> is the VF ID assigned to the SBC SWe VM instance.
Example:
[root@rengali:~] esxcli intnet sriovnic vf get -v 1 -n vmnic0 VF ID Trusted Spoof Check ----- ------- ----------- 1 true false
On the SBC SWe PKT port configure 9 or more VLANs and check VLANs added message in the np.log file.
Example:
[root@RAS ~]# grep -i vlan /var/log/sonus/np/np.log 2022-08-25 12:32:45 : | name | valid | strip_vlan | type | is_slave | sby_bcast_rx | kni_port | slave_kni_port | mac_addr | 2022-08-25 12:34:41 : WARNING: Ignoring vlan add for vlan 0 on pkt0 2022-08-25 12:34:41 : WARNING: Ignoring vlan add for vlan 0 on pkt1 2022-08-25 12:34:41 : pkt0: vlan 10 added. 2022-08-25 12:34:41 : pkt0: vlan 9 added. 2022-08-25 12:34:45 : pkt0: vlan 8 added. 2022-08-25 12:34:45 : pkt0: vlan 7 added. 2022-08-25 12:34:45 : pkt0: vlan 6 added. 2022-08-25 12:34:45 : pkt0: vlan 5 added. 2022-08-25 12:34:45 : pkt0: vlan 4 added. 2022-08-25 12:34:45 : pkt0: vlan 3 added. 2022-08-25 12:34:45 : pkt0: vlan 2 added. 2022-08-25 13:16:37 : pkt0: vlan 11 added. 2022-08-25 13:16:38 : pkt0: vlan 12 added. 2022-08-25 13:16:40 : pkt0: vlan 13 added. 2022-08-25 13:16:41 : pkt0: vlan 14 added. 2022-08-25 13:16:43 : pkt0: vlan 15 added. 2022-08-25 13:16:44 : pkt0: vlan 16 added.
In the above example logs, 15 VLANs are configured on the SBC SWe PKT port.
Overview
Content Tools