SBC Core 09.02.01R000 Release Notes

Portfix SBX-106989: Multiple Switchovers occurred because of a DSP core after an Upgrade to V09.02.00R000.

Scenario: A transcoded call with any Codec<=>G711 and RFC2833 DTMF relay enabled on both legs of a call, and the media probe is disabled.

Impact: If a peer device sends a RFC2833 without EOP packet (end-of-digit marker), then a DSP crash occurs.

Root Cause: The code tried to add an error condition (LOST_EOP flag) to media probe data structure without checking whether a media probe was enabled or not. This resulted in a NULL pointer access and a subsequent memory protection fault resulting in a dsp core dump.

Steps to Replicate: The customer case had a G711A<=>G711A transcoded call with RFC2833 enabled on both legs

Use the following setup:

SipP UAC=>SBX 5x10 => SipP UAS
G729AB(RFC2833) <=> G711(RFC2833)
show configuration system media mediaProbe state
state disabled;
From UAC send dtmf_2833_1_noEop.pcap with '1' digit w/o EOP packets

This resulted in dsp core dump before the fix and no core dump was observed after the fix and digit '1' was correctly sent to g711 side.

The code is modified so if the media probe is enabled and not write to media probe structure if media probe is disabled.

Workaround: Use RFC2833<=>Inband digits or RFC2833<=>SIP Info instead of RFC2833<=>RFC2833.

PortFix SBX-103881 to 9.2.x - MEO: Qseries CDR field not matching Qseries format

Impact: When rn parameter is received as part of SIP R-URI, this value is being used to populate field 10 of the QSBC format CDR record, rather than the received Called Number

Root Cause: In case of rn parameter received, source for field 10 of QSBC CDR is incorrect

Steps to Replicate: Send SIP INVITE with R-URI including rn parameter.

Change code such that if rn is received, field 10 of QSBC is populated with the "Called Number Before Translation" (field 24 of Ribbon format CDR). For other scenarios, field 10 is populated as before.

Workaround: None.

Portfix SBX-106206: An existing hairpin call gets silenced after a switchover.

Impact: When switchover occurs in the SWe SBC Active-Standby HA with the SBC internal loopback media calls flows, the media loopback is not working in existing calls. The looped back media is dropped internally in the SWe NP, because packet skb was not including the internal loopback flag set condition in the NP incoming path validation checks from the internally looped back media for the existing calls.

Root Cause: When a switchover occurs in the SWe SBC for existing medial calls, a loopback occurs in the SWe NP based on the matching SA. 

The DA IP address was configured in BRES by setting an internal loopback flag for packet work/skb, even though MAC DA is not updated in BRES flows to the new active instance interface MAC Address to match SA MAC.

However, this internal skb loopback flag settings was removed in the release 9.0 release NP refactoring code which caused the packet drops in incoming packet checks for MACs at NP from the looped back media of existing calls. The new calls will not have this issue post-switchover.

Steps to Replicate: With the SWe SBC Active-Standby HA setup, using AS/3GPP call flow scripts/setup, establish a call that creates the SBC internal loopback media flow. Verify the media flow and then Issue a switchover and verify the media again on this existing active call.

The code is modified to save NP the CPU cycles also while addressing the loopback media issue after a switchover issue for existing calls.

Workaround: None.

Portfix SBX-105269: The SBC crashed and a core dump created.

Impact: The SCM processes coredump due to NULL pointer access on a pointer that has been freed due to BYE and HOLD REINVITE in a transfer call scenario.

Root Cause: There was an illegal memory access, absence of NULL check, exposed due to REINVITE.

Steps to Replicate: A calls B, B REFERs to C and now A and C talk. After sometime, A sends BYE and C sends re-INVITE with a=inactive at the same time.

The code is modified to address the issue. 

Workaround: None.

Release 9.2.0 should have CAM version of "00610000".

New CDR fields were added in the 9.2.0R0 release, but the CAM version in the top of the ACT header file was not updated to "00610000". It was still set to ""00600000", which was used for release 9.1.0R0.

Impact: Incorrect CAM version in the top of the ACT header file.

Root Cause: The code to increment the CAM version was missed.

Steps to Verify Fix: Run a basic call and check the ACT log header to confirm the CAM version is set to "00610000"

The code is updated to correctly set the CAM version to "00610000".

The DSP Threshold setting is not generating a trap on SBC 5400.

Impact: The g711PacketThreshold, g729Threshold, and g726Threshold onset and abate traps are not sent.

Root Cause: The NRM did not receive CLI updates to the g711PacketThreshold, g729Threshold, and g726Threshold, and the trap generation code used wrong trap names.

Steps to Replicate: 

1. Provision the threshold levels:
   set oam traps dspAdmin dspAvailabilityTrap g729Threshold 40
   set oam traps dspAdmin dspAvailabilityTrap g726Threshold 60
   set oam traps dspAdmin dspAvailabilityTrap g711PacketThreshold 80
   commit 
   set oam traps admin sonusSbxDspAvailG729OnSetCrossThresholdNotification state enabled
   commit
   set oam traps admin sonusSbxDspAvailG729AbateCrossThresholdNotification state enabled
   commit
   set oam traps admin sonusSbxDspAvailG726AbateCrossThresholdNotification state enabled
   commit
   set oam traps admin sonusSbxDspAvailG726OnSetCrossThresholdNotification state enabled
   commit
   set oam traps admin sonusSbxDspAvailG711OnSetCrossThresholdNotification state enabled
   commit
   set oam traps admin sonusSbxDspAvailG711PacketAbateCrossThresholdNotification state enabled
   commit
2. Configure the trap target: set oam snmp trapTarget EMS160 ipAddress 10.xxx.xx.xxx port 162 trapType v2 state enabled commit.
3. Limit the compression resources to make the issue readily occur: set system mediaProfile tone 98 compression 2 commit.
4. Perform a bunch of transcoded (e.g. G711 to G729) calls that exceed threshold limits, and see onset traps are sent to the trap target.
5. Clear the transcoded calls, and see abate traps are sent to the trap target.

The code is modified to support g711PacketThreshold, g729Threshold, and g726Threshold values.

Deprecated the support for allThreshold, as it was never implemented in the SBC.

Workaround: None.

Calls are dropping when placed on hold through MS Teams.

Impact: When MS teams puts a call on hold and then attempts to retrieve it while the microphone is disabled, the action results in the call being cleared.

Root Cause: MS teams performs the call retrieve by sending an INVITE with a replaces message to the SBC and when the microphone is disabled this message includes a=recvonly.

The SBC send 200 Ok response to the INVITE with replaces with a=sendrecv while teams is expecting a=sendonly that causes the call to be cleared by MS teams.

Steps to Replicate: 

1. Establish a PSTN to teams call through the SBC.
2. Place call on hold from teams.
3. Disable the microphone in browser running teams and then retrieve (place off hold) the call.
4. The call should retrieve should be successful.

The code is modified to respond to an INVITE with replaces containing a=recvonly, with 200 OK containing a=sendonly.

Workaround: None.

The STI sends 2 Identity fields in Contact header, but SBC only passes one

Impact: In the case where multiple "Identity Headers" are present as embedded headers in a 3xx "Contact Header", the SBC sends out only 1 Identity Header for the re-directed INVITE.

Root Cause: The SBC did not support handling of the scenario where SIP headers are repeated in embedded contact headers in 3xx. The SBC would end up the sending only the last of the repeated headers in the re-directed Invite.

Steps to Replicate: 

1. The SBC is configured with ERE, set the config required for handling embedded headers in 3xx as shown below.
   "set profiles signaling ipSignalingProfile DEFAULT_SIP egressIpAttributes redirect flags forceRequeryForRedirection enable honorEmbeddedHeadersIn3xx enable"
2. Run a UAS script with 302 redirection and add multiple "Identity headers"
   as shown below.
   Ex- Contact: 9710622482<sip:9710622482@10.xx.xx.xx:xxxx?Identity=test1&Identity=test2&Identity=test3>
3. Run a 2nd UAS Script to handle the Redirected INVITE.
4. Run SIPP call, trigger UAC script.
   
   Expected Result: The redirect INVITE shall contain both the Identity headers received in 3xx as embedded contact header.

The code is modified to support repeated SIP headers.

Workaround: None.

A customer SBC upgrade had a spiltbrain SBC boot issue

Impact: One of the customer SBCs in N:1 mode is not coming up after upgrade.

Root Cause: The customer SBC node went for multiple reboots due to config profile change and split brains after upgrade exceeding max limit for reboot.

Steps to Replicate: Upgrade all customer SBC nodes in one shot so that they all come up together.

The code is modified to avoid split brain due to nodeId/serviceId collision by informing the peer nodes about the self node's nodeId/serviceid as soon as its allocated.

Workaround: Restart the customer SBC application manually on the node which has exceeded max reboot limit.

The BYE message was sent to the wrong port.

Impact: When endpoint is registered over TLS initiates a calls and after call is established, endpoint sends refresh register from modified port and also sends refresh INVITE with no SDP change, the SBC does not send BYE to modified port upon call disconnect.

Root Cause: The SBC does not update remote connection address when refresh INVITE is received from different port.

Steps to Replicate: 

1. Register endpoint over TLS.
2. Endpoint initiates a call, call gets connected.
3. Endpoint sends refresh register from modified port.
4. Endpoint sends refresh re-INVITE from modified port.
5. Called party disconnects the call.
6. Verify SBC sends BYE to modified port towards caller.

The code is modified to ensure the SBC update remote address when the refresh INVITE is received from modified port.

Workaround: None.

PreInstall Check improvements for file permissions.

Impact: PreUpgrade checks failure due to permission issues on external directory.

Root Cause: Pre-checks script failed to run commands on peer due to permission issues.

Steps to Replicate: Perform upgrade to the fix version and verify that upgrade is successful.

The code is modified to ensure permissions/ownership are set properly before running through pre-checks/upgrade.

Workaround: Set the right ownership for external directory as:
chgrp -R upload /opt/sonus/external/

The IAC:terraform apply output is displaying only mgt Active and Standby IP's. I

Impact: When orchestrating a HFE 2.0 or HFE 2.1 setup in Azure using Ribbon RAF modules, the Terraform output does not show any information about HFE node(s) created, even thought the HFE nodes are started correctly.

Root Cause: Terraform was missing the configuration to tell it to output the information about the HFE node(s).

Steps to Replicate: Create a HFE 2.0 or HFE 2.1 setup using Ribbon RAF modules.

The code is modified so that Terraform outputs the management IP and instance name for each HFE node when the orchestration is successful.

Workaround: None.

There are call failures due connection toggling between the SBC and SLB.

Impact: The SBC's can lose connectivity to the HFE nodes when there is high amounts of traffic on the network in Azure

Root Cause: The health check timeout from the HFE to SBC is too small, therefore was not receiving the replies back from the SBC, causing HFE to switchover

Steps to Replicate:

Increased the timeout from 20ms to 100ms

Workaround: Manually edit the following line in HFE_AZ.sh from:
$FPING -c 3 -t 20-p 200 ${ACTIVE_SBC_IP_ARR[0]} &> /dev/null
to:
$FPING -c 3 -t 100 -p 200 ${ACTIVE_SBC_IP_ARR[0]} &> /dev/null

The IaC is creating same NSG for all the network interfaces created. Recommended SGs should be created for each interface.

Impact: IaC is creating single NSG for all the 4 sbc interface.

Root Cause: Iac is creating one NSG and using same for all sbc interfaces created.

Steps to Replicate: Provided option to create different NSG for different interface created for the SBC.

Option available in terraform.vars Ex:
sbc_security_group_names = ["TF-SVT-SG-MGT0", "TF-SVT-SG-HA", "TF-SVT-SG-PKT0", "TF-SVT-SG-PKT1"]

The code is modified to create different NSG for different interface of the SBC.

Workaround: None

The Active SLB failed in the LCA to coming up "getInterfaceInfoFromMDS: Could not get the interface details in the metadata information!"

Impact: If the Azure fails on the request when verifying the authorization of the managed identity, then the cloud-init immediately fails.

Root Cause: The cloud-init is missing tolerance when testing the authorization.

Steps to Replicate: TBD

Add retries if the connection is reset or times out to address the issue.

Workaround: Reboot the instance after cloud-init has failed.

The LeakSanitizer detected memory leaks on various processes for a t140 Call.

Impact: Small memory leak while configuring SNMP trap targets.

Root Cause: While processing the configuration requests the SBC code was reading content from CDB into local memory blocks but failed to release the memory blocks at the end of the configuration action.

Steps to Replicate: Configure the SNMP trap targets.

The code is modified to correctly free the internal memory blocks used to hold the temporary CDB configuration data.

Workaround: None

The PES Process cored while testing Flex AD support with the ERE (SBX-72926).

Impact: PES process dumps core while executing AD service.

Root Cause: While executing the AD service, PES fetches the AD profile from cache. Since there is no AD profile was configured in this case, cache shall return NULL. The code was missing to check for NULL for the AD profile pointer.

Steps to Replicate: Configure a AD number translation criteria with ingress TG as the trigger criteria type. Make a call on the ingress TG. While executing the AD service, PES process will core dump.

The code is modified to check for a NULL for AD profile and AD attribute profile. If there is no profiles configured, the control returns and the service is marked as failed.

Workaround: Configure an AD Profile and AD Attribute profile so that the existing code can be executed further.

The SCM Process cored while testing SBX H323 Conformance and Interworking.

Impact: The SCM Process cored while running SIP-H323 interworking call to verify rfc2833 to inband DTMF.

Root Cause: Accessing of a NULL pointer caused the core dump.

Steps to Replicate: Run a SIP-H323 interworking call with SIP side RFC 2833 and H.323 side inband DTMF being enabled.

The code is modified to check for NULL pointer before accessing it.

Workaround: None.

Observed a PRS Process core followed by a "DsPr, Ssre, Cpx" core on both active and standby box, for the Openstack S-SBC HA pair and "Ccsp" process core on the TSBC active while running calls on the 2CPS with the ASAN build.

Impact: There was a memory leak detected by the ASAN in the active S-SBC while running the G.711 to G.729 transcoding load on the DSBC.

Root Cause: A code analysis of the ASAN reported function revealed that in some corner scenarios, the SBC may end up leaking the memory related to the place holder used to store the remote DSP resource.

Steps to Replicate: Re-run the same load in ASAN and ensure no memory leaks are detected

The code is modified to free this memory.

Workaround: None.

The SCM Process core dump was observed when REGISTER is received and useRandomUserInfoInContactHdr is enabled.

Impact: Invalid memory access issue when both embeddedRegInfoinUserPart and useRandomUserInfoInContactHdr flag were enabled

Root Cause: In the SBC, the appropriate memory was not allocated for contactUrl puchUsername in case both embeddedRegInfoinUserPart and useRandomUserInfoInContactHdr flag were enabled, due to this invalid memory access was happening.

Steps to Replicate:

1. On SLB - Enable remoteDeviceType to access
2. On SBCs enable the embeddedRegInfoinUserPart flag as:-
   set addressContext <ADDRESS_CONTEXT> zone <ZONE> sipTrunkGroup <TG> signaling embeddedRegInfoinUserPart enabled
3. On SBCs enabled useRandomUserinfoInContactHeader as :-
   set addressContext <ADDRESS_CONTEXT> zone <ZONE> sipTrunkGroup <TG> signaling useRandomUserInfoInContactHdr enabled

The code is modified to allocate appropriate memory for puchUsername based on embeddedRegInfoinUserPart or useRandomUserInfoInContactHdr enable case.

Workaround: As per the SBC design, any of the flags below should be enable not both.
embeddedRegInfoinUserPart or useRandomUserInfoInContactHdr flag

PortFix SBX-98177: The SBC 7000 TCP window size.

Impact: Add CLI support to set the kernel parameter net.ipv4.tcp_window_scaling.

Root Cause: In certain SBC deployment scenarios (a large number of devices running SIP-TLS), the tcp_window_scaling caused small TCP window size.
In such cases, the customer had to use Linux command to disable tcp_window_scaling on each SBC node.

Steps to Replicate: 

1. Show initial setting for tcp_window_scaling on both active (SBCSWE01a) and standby (SBCSWE01b)
   [root@SBCSWE01a ~]# cat /proc/sys/net/ipv4/tcp_window_scaling=1
   [root@SBCSWE01b ~]# cat /proc/sys/net/ipv4/tcp_window_scaling=1
   
   
2. Use CLI to change the kernel param value and display the changed value on both active and standby.
   admin@SBCSWE01a% set system admin SBCSWE01 kernelParams tcpWindowScaling disable
   admin@SBCSWE01a% commit
   Commit complete.
   
   [root@SBCSWE01a ~]# cat /proc/sys/net/ipv4/tcp_window_scaling=0
   
   [root@SBCSWE01b ~]# cat /proc/sys/net/ipv4/tcp_window_scaling=0

The code is modified for setting the kernel parameter net.ipv4.tcp_window_scaling. The Linux command is no longer needed.

Workaround: Use Linux command to set the kernel parameter on both active and standby.
[root@SBCSWE01a ~]# sysctl net.ipv4.tcp_window_scaling=1
[root@SBCSWE01a ~]# sysctl -w net.ipv4.tcp_window_scaling=0
[root@SBCSWE01b ~]# sysctl net.ipv4.tcp_window_scaling=1
[root@SBCSWE01b ~]# sysctl -w net.ipv4.tcp_window_scaling=0

To be persistent across SBC restart and system reboot, use Linux the following shell command on both active and standby.
[root@SBCSWE01a ~]# echo "net.ipv4.tcp_window_scaling=0" >> /etc/sysctl.conf
[root@SBCSWE01b ~]# echo "net.ipv4.tcp_window_scaling=0" >> /etc/sysctl.conf

Portfix SBX-105351: The 823R0 SCM Process cored.

Impact:The SCM cores after a long extensive load of the OOD relay.

Root Cause: The SBC accesses an invalid pointer of link list when the cleanup relay control block.

Steps to Replicate:System required HW_TYPE_SBS5000 and max support Relay per SCM is 4194304.

Run the OOD relay load (Notify). After a long period time (at least 4m relay per SCM).

The code is modified to properly initialize the link list.

Workaround: None.

Portfix SBX-105019: A crosstalk issue seen on the SBC 5400.

Impact: When there is a transcoded call with Opus codec and Opus termination does not send any packets to the SBC, then other termination of opus call, may hear audio from another unrelated opus transcoded call or it may hear white noise.

This occurs on SBC5x10/5400/7K but not on SWe platforms. This problem only happens if no opus packet is received. When the decoder receives first opus packet, subsequently problem does not happen for that call.

Root Cause: Initially, when no packet is received, Opus decoder is not called. As a result, the decoder's output buffer is uninitialized and it happened to be stale buffer of another opus call or a leftover buffer of a previous Opus call on that DSP. That buffer is used to encode data for other leg of the Opus call and that results in cross talk audio or distorted noise to other termination of Opus call.

Steps to Replicate: 

1. Make two sipp Opus<=>g711 calls, say call 1 and call 2 on SBC5x10/5400/7K
2. These two calls have to land on same DSP core. So issue a unhide debug command 'request sbx drm debug command "loadbalance disable" before making two calls. Without this command, both calls may not land on same DSP core and you may not see same problem.
3. Call 1 sends audio packet (.pcap) from Opus termination but call 2 does not send any packets to the SBC from it's Opus termination.
4. Call 2's g711 termination will hear audio from call 1's opus termination.

The fix is to clear the buffer (with silence) when the Opus decoder is not primed with any packet.

Workaround: 

1. Use a different codec other than Opus.
2. Use SWe platform instead of h/w SBC5x10/5400/7000.

Portfix SBX-102122: SAM Process core dumps with v06.02.01 F012.

Impact: The SAM and PRS process cored in the Standby node.

Root Cause: The DRBD split brain lead to DRBD GI reset. This leads to a DRBD full sync, increasing the i/o congestion and ultimately causing a healthcheck timeout.

Steps to Replicate: To test, run "drbdadm disconnect mirror". Once we get "standby split-brain" log in DBG logs, run "drbdadm disconnect mirror" again so that DRBD does not connect automatically and we get into the block which we need to test. Run "drbd-overview" and check if drbd does not went for full sync.

The code is modified to perform the DRBD checks after DRBD split-brain recovery.

Workaround: None.

PortFix SBX-102625: The SIP calls stop in receiving 183 (Precondition).

Impact: Call stopped working after 183 received without SDP.

Root Cause: As part of egress precondition interworking, to negotiate the SBC should send an UPDATE to the egress endpoint. When 183 received without SDP, SBC should not try to send UPDATE. SIPSG call state assumed UPDATE is sent. Therefore, when the next 183 is received it tried to queue the 183 assuming UPDATE is in progress.

Steps to Replicate: As mentioned in the JIRA

The code is modified to avoid this state transition when 183 received without SDP.

Workaround: Remove the 183 without a SDP message using SMM.

PortFix SBX-102837: The fra-tdg-sonus-01 SipSignalingPorts became OutOfService after a switchover.

Impact:The SIP sigPorts stuck in the OOS after a switchover.

Root Cause: The customer had the network/pkt port issue, on one of their HA node, which caused pkt port(s) to bounce randomly, i.e. pkt port went DOWN and came back UP within 2 to 3 seconds. So they have been keeping that node as standby node. They also have link detection enabled for pkt port(s) and have around 100 LIFs per pkt port, one per SIP SIGPORT. When the pkt port went down, NRS delays the port down event processing for 2 second to allow link failure detection to be ready and also to avoid the race condition between NRS and LVM. When a 2 second delay timer is up, the NRS starts to take down affected LIFs and notifies local SIPCM and SIPFE so they can take down affected SIP SIGPORTs.

In SIPCM, all the sockets on the affected sigPort are put in a delete pending table and starts a 1 tick timer. Then the socket(s) is being deleted after 1 tick timer is up.
When pkt port came back up, NRS processes the event with no delay and notifies SIPCM and SIPFE as well. Since there are around 100 LIFs, there were many messages exchanges between NRS/XRM/SIPCM/SIPFE. NRS LIF FSM has the mechanism in place to handle the timing issue and LIFs were all back in service. But SIPCM failed to activate some SIP SIGPORT(s) while binding the socket(s). These error messages indicated that SIPCM tried to activate the SIGPORT while it was still pending delete. Therefore SIGPORT got stuck in OOS(broken state) in both SIPCM and SIPFE on standby node. If there was a switchover happened later, user would then noticed one or more SIGPORTs were OOS. They have to manually bounce those SIGPORTs to bring them back in service.

Steps to Replicate: The nature of the problem was the timing caused race condition. There is no good way to re-create/verify the fixes.

Suggest to run regular SIP related regression tests.

The code is modified to:

1. Introduce a new 1 tick timer in SIPCM_DATA_STR, activateRetryTimerId.
2. The deletePendingSocketTable and if the sigPort is found in the table, then start the 1 tick timer. When the timer is up, SipCmActivateCallSigPort() is invoked again. Once the sigPort is activated successfully, SIPCM notifies SIPFE as usual.

Workaround: No workaround.

PortFix SBX-101161: A memory leak was observed in the SAM process.

Impact: A memory leak was observed in the SAM process.

Root Cause: There is race condition in the code that handles CALL_AUDITs and CALL_CLEANUPs that can cause a memory leak.

If the response to a CALL_AUDIT takes to long, the code that handles the "late" response has allocates memory for a structure that may never get freed.

Steps to Replicate: This issue is caused by a race condition that cannot be forced - therefore we cannot specify steps to reproduce this issue.

The code is modified to add a timer every time a fault structure is allocated. When the timer expires, if the the structure still exists it will be freed.

Workaround: There is no workaround.

PortFix SBX-104802: Enabling the Dialog Transparency causes calls to fail.

Impact: The call fails for the dialog transparency and forking.

Root Cause: When the ingress support PRACK and multiple 18x fork, the SBC fails to send 200OK to ingress.

Steps to Replicate: Configure dialog transparency and downstream forking on egress.
Ingress support PRACK, egress not support PRACK.

The Egress answers 180fork1, 180fork2, 180fork3, and 200OKfork2 simultaneously.

The code is modified so the SBC resets the PRACK pending status properly, causing 200OK stuck in the queue.

Workaround: Disable the dialog transparency.

PortFix SBX-104761: A SM Process coredump occurred on the server.

Impact: The SM Process crashed while executing the "show table system syncStatus" command.

Root Cause: The shell script used to get the oracle sync status - PolicyDBSyncStatus.sh - did not return within 10 seconds, causing a healthcheck timeout that caused the coredump.

Steps to Replicate: This problem is not reproducible.

The code is modified to disable heathchecks while fetching the syncStatus.

Workaround: There is no workaround.

PortFix SBX-103553: The SBC does not proceed multiple 183.

Impact: The GSX-GW-SBX call erroneously queues 183 message at SBX when X-Service-Type precondition is received and downstream forking enabled.

Root Cause: An interaction between downstream forking and GW-GW code.

Steps to Replicate: Make GSX -> SBX SIP-GW-SIP call.
Egress trunk group has X-Headers supported.
Egress trunk group has downstream forking enabled.

INVITE -->
<-- 183 with X-Service-Type: cf,precondition and SDP
<-- 183 with X Service-Type: cf,precondition and different SDP
<-- 180 with X Service-Type: cf and SDP
<-- 183 with X Service-Type: cf and no SDP
This final 183 is queued.

The code is modified to not perform the queueing in GW-GW scenarios.

Workaround: None.

PortFix SBX-104443: The SM Process and Cpx cores prevented fra-tdg-sonus-01-1 to come back up as standby after switchover

Impact: The SM Process and Cpx processes cored on slot 1 after LDG triggered switchover to slot 2.

The SM Process was deadlocked while retrieving peer NTP status.

The Cpx cores happened after SM Process coredump, when writing SMM profiles into shared memory.

Root Cause: The root cause was SM Process deadlock.

Steps to Replicate: This is time sensitive issue. The steps cannot be re-produced or verified.

Suggest to run full regression test with switchovers.

The code is modified to release smNtpLock_ after issuing a NTP restart command because we certainly do not need the lock while doing the restart.

Workaround: No workaround.

PortFix SBX-103254: The call setup delay with the Option Ping 40+ endpoints SBC SWe.

Impact: The configuration support to disable Path MTU Discovery by setting kernel parameter net.ipv4.ip_no_pmtu_disc (ipNoPmtuDisc). When ipNoPmtuDisc is to 2, the DF bit in IP packet header will be set 0.

Root Cause: The secure network does not support Path MTU Discovery per RFC-1191.

Steps to Replicate: Testing on a Standalone SBC SWe:

1. Start packet capture, and make SIP call (SIP-TLS). Verify Path MTU Discovery is enabled by checking outgoing IP packet's DF bit is 1.
- Test to show the issue
[root@SBCSWE03 ~]# date; tshark -t a -i pkt0 -f "ip host 10.x.xx.xx" -w t_tshark_pkt0_10.x.xx.xx_1127_pmtu_1027_01.pcap
Tue Oct 27 16:21:11 EDT 2020
Capturing on 'pkt0'
35 ^C
[root@SBCSWE03 ~]# tshark -V -r t_tshark_pkt0_10.7.20.29_1127_pmtu_1027_01.pcap | more

Internet Protocol Version 4, Src: 10.x.xx.xx (10.x.xx.xx), Dst: 10.xxx.xx.xxx (10.xxx.xx.xxx)
Version: 4
Header Length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport))
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00)
Total Length: 60
Identification: 0x0000 (0)
Flags: 0x02 (Don't Fragment)
0... .... = Reserved bit: Not set
.1.. .... = Don't fragment: Set
..0. .... = More fragments: Not set
Fragment offset: 0

[root@SBCSWE03 ~]#
--> The DF bit was set to 1.

2. Disable Path MTU Discovery.
admin@SBCSWE03% set system admin sbcswe03 kernelParams ipNoPmtuDisc ?
Description:
Configure /proc/sys/net/ipv4/ip_no_pmtu_disc (0 (default; disable); 1,2,3 (enabled modes))
1: when frag-required ICMP is received, PMTU to this destination is set to min_pmut 552.
2: implicitly setting IP_PMTUDISC_DONT on every created socket - outgoing IP packet DF is set to 0.
3: hardened pmtu discover mode. Please refer to Linux kernel document.

Possible completions:
<int, 0 .. 3>[0]
admin@SBCSWE03% set system admin sbcswe03 kernelParams ipNoPmtuDisc 2
admin@SBCSWE03% commit
Commit complete.
[ok][2020-10-27 16:36:27]

3. Restart the SBC. (If using HA, restart both nodes)
[root@SBCSWE03 ~]# date; sbxrestart
Tue Oct 27 16:36:56 EDT 2020

4. Start packet capture, and make SIP call (SIP-TLS). Verify Path MTU Discovery is disabled by checking outgoing IP packet's DF bit is 0.
[root@SBCSWE03 ~]# date; tshark -t a -i pkt0 -f "ip host 10.x.xx.xx" -w t_tshark_pkt0_10.x.xx.xx_1127_noPmtu_1027_02.pcap
Tue Oct 27 16:42:32 EDT 2020
Capturing on 'pkt0'
26 ^C
[root@SBCSWE03 ~]# tshark -V -r t_tshark_pkt0_10.x.xx.xx_1127_noPmtu_1027_02.pcap | more
...
Frame 2: 74 bytes on wire (592 bits), 74 bytes captured (592 bits) on interface 0
...
Internet Protocol Version 4, Src: 10.x.xx.xx (10.x.xx.xx), Dst: 10.xxx.xx.xxx (10.xxx.xx.xxx)
Version: 4
Header Length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport))
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00)
Total Length: 60
Identification: 0xe881 (59521)
Flags: 0x00
0... .... = Reserved bit: Not set
.0.. .... = Don't fragment: Not set
..0. .... = More fragments: Not set
Fragment offset: 0
...

[root@SBCSWE03 ~]#
--> The DF bit was set to 0, and Path MTU Discovery is disabled.

Disable the Path MTU Discovery by setting kernel parameter net.ipv4.ip_no_pmtu_disc to 2. When set to 2, the DF bit will be set to 0 for the transmitted packets on every created socket. The configuration parameter is "system admin <name of system> kernelParams ipNoPmtuDisc".

Workaround: The workaround is a hack. It can be used one time test only. The hack will not survive Linux reboots.

1. Make changes on the current Standby (assuming SBC1 is currently active, and SBC2 is currently standby)
   [root@SBC2 ~]# date; cat /proc/sys/net/ipv4/ip_no_pmtu_disc
   [root@SBC2 ~]# date; echo “2” > cat /proc/sys/net/ipv4/ip_no_pmtu_disc
   [root@SBC2 ~]# date; sbxrestart
2. Wait until SBC2 is up and synched. Ensure the p_no_pmtu_disc is set to 2.
   [root@SBC2 ~]# date; sbxstatus | tail -4
   [root@SBC2 ~]# cat /proc/sys/net/ipv4/ip_no_pmtu_disc 2
   [root@SBC2 ~]#
3. Make changes on the current Active.
   [root@SBC1 ~]# date; cat /proc/sys/net/ipv4/ip_no_pmtu_disc
   [root@SBC1 ~]# date; echo “2” > cat /proc/sys/net/ipv4/ip_no_pmtu_disc
   [root@SBC1 ~]# date; sbxrestart
4. Wait until the SBC1 is up and synched. Ensure the p_no_pmtu_disc is set to 2.
   [root@SBC1 ~]# date; sbxstatus | tail -4
   [root@SBC1 ~]# cat /proc/sys/net/ipv4/ip_no_pmtu_disc 2
   [root@SBC1 ~]#
5. May perform a switchover to make the SBC1 to be active.
6. Run your tests.

PortFix SBX-95982: The SBC was running version 6.2.2 and cannot capture media with MCT.

Impact: The MCT reload of configuration (sbx restart or switchover) will not succeed.

Root Cause: Missing small logic during the restart in order to properly reload the MCT configuration.

Steps to Replicate: Retest MCT reload and switchover scenarios.

The code is modified to properly reload the MCT configuration.

Workaround: User can delete the MCT config and reconfigure it after any restart/switchover.

PortFix SBX-103948: The SBC Application is crashing when CPaaS to PSTN call is made.

Impact: The SBC cores after a reboot/upgrade.

Root Cause: The issue was introduced when the SBC try to support 10000 SMM rules in 8.2.0.

Steps to Replicate: Configure the customer rules (delete SDP line). Major logs trigger but the rule still success.

After a reboot/upgrade, the core occurs.

The code is modified to address the issue. 

Workaround: Disable the delete SDP rule.

PortFix SBX-105156: The SBC was sending m=application 0 UDP/BFCP (null).

Impact: The SBC generates the syntax error m line UDP/BFCP in 183.

Root Cause: The SBC accesses the initialize data when format m line for UDP/BFCP

Steps to Replicate: Incoming call with m-line UDP/BFCP. ERE/PSX using script NO_ROUTE and trigger announcement. The internal 183 has m-line UDP/BFCP with an invalid syntax.

Make the m-line initialize properly to address the issue.

Workaround: Use the SMM to correct m-line syntax error.

PortFix SBX-96239: The display-name in the Diversion header or History-info header is deleted during the interworking.

Impact: The SBC does not send display name present in ingress INVITE's diversion header in history info header in egress INVITE.

The SBC does not send a display name present in ingress INVITE's history info header in egress INVITE's diversion header.

Root Cause: The SBC does not consider display name when interworking between diversion header and history info header.

Steps to Replicate: Ensure the display name is sent in history info header of egress INVITE when the SBC is converting diversion header to history info header.

Also, the SBC should send the display name in diversion header of egress INVITE when the SBC is converting history info to diversion header.

The code is modified to ensure the display name is sent when the SBC is interworking history info and diversion header.

Workaround: None.

PortFix SBX-104225: Both servers are registered but offline, and the SBCs cored after routing the changes from the customer.

Impact: The S-SBC Core dump with PEM enabled call-flow.

Root Cause: With PEM enabled, M-SBC performs NAPT learning for 1st RTP Packets. Upon receiving the Napt Indication for egress-leg from the M-SBC, the NRMA on the S-SBC performs flowChange/modification on both ingress and egress leg.

As part of this flowChange/modification in ingress-leg, the ingress is NULL, resulting in coredump.

The cktInfo on ingress should never be NULL, unless the call is being teared down.

In this case, the ingress-leg has received a CANCEL, resulting in the call being taken down, as part of this call tear-down, ingress-leg cktInfo is reset to NULL, and then the de-alloc indication for ingress-leg is sent to the M-SBC, while the S-SBC is waiting for response of this tear-down, it processes the NAPT indication on egress-leg from the M-SBC. This results in the NULL cktInfo pointer getting accesses for ingress-leg.

Steps to Replicate: 

1. Setup a D-SBC transcoded call with MRF.
2. Enable PEM on egress.
3. Ensure PRACK is enabled.
4. Ensure 180 without SDP, followed by 183 with SDP is received from UAS.
5. Ensure CANCEL is received from UAC, while 200 OK from MRF is received, such that the timing of RTP NAPT learning processing and CANCEL processing match.
   
   NOTE: This is a very tricky issue to recreate, as the timing may vary every time we attempt, it is a matter is few milliseconds.

The code is modified to address the issue.

Workaround: Disable NAPT or PEM. This workaround will ensure that we do not get to the race condition.

Portfix SBX-104561: The 8.1R5 customer SBC OAMs reaches 90% memory util and then restarts.

Impact: The memory leak was observed on the OAM node.

Root Cause: 

• The XRM and MRM are sending messages through the ICM to NRM.
• The NRM is not present/running on OAM.
• Since these are non-discardable messages, they will queue forever until we run out of memory.

Steps to Replicate: 

1. Deploy the fix.
2. Monitor memory usage of PRS process.
3. Validate that PRS memory is not increasing.

The code is modified so that these messages are not sent on OAM node.

Workaround: Before this fix, the workaround is to restart OAM app to recover the memory.

PortFix SBX-104484: The SBC is rejecting the second Update message from the Egress as 500 result into call failure.

Impact: The SBC rejects 2nd Update from egress due to DLRB feature.

Root Cause: The first Update SIPS response locally 200OK but not clear the server request message. As result the second Update, trigger the SIPS answer 500.

Steps to Replicate: Configure the DLRB, Egress response 183 without the SDP, first Update, and the second Update.

The code is modified so after a 200OK response, the SIPS clears the server request so it can handle the subsequent one.

Workaround: Disable the DLRB.

PortFix SBX-104146: The DNS Process observed a coredump when responding with a RCODE 1 error from the DNS server and deleting a DNS Group in the SBC.

Impact: The DNS Process dumps core after getting RCODE errors 1,2,4 for a EDNS query and the DNS Group is deleted immediately on the SBC in the monitoring Interval specified in the "ednsRetryAfter" configuration has values ranging from 60 to 180 seconds with 180 being default.

Root Cause: The EDNS failure timer callback is invoked, upon encountering an EDNS query failure based on the "ednsRetryAfter" configuration. The timer call back function is missing NULL check for the DNS Group

Steps to Replicate: 

1. The SBC configured with EDNS support.
2. The DNS server is configured to respond with the RCODE 1/2/4 error for EDNS query.
3. Make a SIP Call that triggers the EDNS query.
4. Disable the DNS Server and delete the DNS group within 180 seconds.

The code is modified to cancel the EDNS-failure timer, whenever the DNS Group is deleted and the NULL pointer check is added for the timer callback for robustness.

Workaround: None.

Portfix SBX-103207: The SBC SWe duplicates the audio from call B to call A.

Impact: If the CN is not negotiated for G711 codec and remote peer still sends CN packets that match default CN payload type (13) then user on other end may hear cross talk audio of completely unrelated channel or hear own reflect audio.

Root Cause: When a g711 side does not negotiate CN in signaling, the DSP does not initialize Comfort Noise Generation object. However, if remote peer still sends a CN packet that matches the g711 SID payload type configured in PSP (default 13), DSP accepts the packet. It processes that CN packet incorrectly and as a result uses stale voice buffer that happens to be of another channel. This continues until next voice packet for that channel arrives. As a result, we see cross talk audio from the other call during silence period. In some cases, the user may hear own audio also and that is different manifestation of the same problem. This issue is specific to SWe.

Steps to Replicate: Follow the step described in problem statement of JIRA.

The code is modified to initialize the comfort noise object even though the CN is not negotiated and process CN packets correctly.

Workaround: 

Work around 1:

Change the default payload type of comfort noise from 13 to something else (say 15) in PSP of peer that is sending CN packets. This will make DSP drop CN packets because payload type will not match.

Work around 2:

Enable silence suppression on PSP of peer that is sending CN packets and keep CN payload type that matches with CN payload type.

PortFix SBX-102704: Unable to Add or Edit route with # in Destination National, CLI or EMA.

Impact: User is not able to store few special characters for the destination number field in the route entity.

Root Cause: The pattern defined for the destination number filed did not allow special characters.

Steps to Replicate: Create a route with special character like #123 for the destination number filed in the route entity.

The code is modified to allow only few specific special characters.

So, reverted back the pattern to same as before 8.1 release, so that no customers face similar issue again who have configured special characters prior to 8.1.

Workaround: None.

PortFix SBX-102290: The DBG file was filling up with messages "SIPCM: *ThreadPool: messageSequence".

Impact: The DBG logs can be overrun with "SIPCM: *ThreadPool: messageSequence" messages.

Root Cause: The DBG logs can be overrun with "SIPCM: *ThreadPool: messageSequence" messages, when Double CRLF "pings" are received by the SBC over UDP transport.

Steps to Replicate: Send Double CRLF "pings" over UDP to the SBC.

The code is modified to properly dispose of Double CRLF "pings" received by the SBC over UDP.

Workaround: Inhibit the transmission (or reception) of Double CRLF "pings" over UDP.

PortFix SBX-104934: The SBC generates the same ICID for more than 1 call, porting the fix for SBX-101887 into 8.2.3F1 did not help

Impact: Duplicate the ICID generated for two different calls.

Root Cause: The instance specific value is set to 0 every other call causing the ICID generation to always use 0 in all instances. This results in the same ICID generated in more than one SCM instance when the calls land within the same microsecond

Steps to Replicate: Configure the SBC to generate ICID on the egress side. Run call load and check for duplicate ICIDs.

The code is modified to manipulate the last octet of the MAC address used in ICID generation.

Workaround: None.

Portfix SBX-105417: During an S-SBC and M-SBC cyclic switchover, the S-SBC app did not come up after few switchovers.

Impact: The service did not come up after a switchover due to inconsistent encryptedStore.

Root Cause: The store became inconsistent after the oam_config got applied, the time taken during deletePeerEntries should also be reduced to allow a ChmClearEncryptedStoreOfPeers to finish faster during startup.

Steps to Replicate: Perform multiple switchovers and ensure that:

1. ChmClearEncryptedStoreOfPeers does not take too long during startup.
2. The deletePeerEntries runs in background and does not take too long either.
3. The service comes up, role is assigned successfully and store is consistent.

The code is modified so:

1. deletePeerEntries run in background
2. decryptStore validates if decryption was successful to ensure the store is not inconsistent/corrupt.

Workaround: None.

PortFix SBX-103629: The SIP registrations are failing, and the SBC reporting REGISTER_PARSE_ERROR and replies with SIP 500 when it receives retransmitted initial REGISTER (CSEQ 1) followed by REGISTER with Authorization header (CSEQ 1).

Impact: When the SBC recv rexmit of registration (cseq=1) after sending 401/407, the SBC relay to AS. At the same time, the IAD sends new registration(cseq=2) for authentication. It triggers race condition to the SBC and response 500.

Root Cause: The SBC should relay rexmit (cseq=1) registration to application server.

Steps to Replicate: The IAD send registration(cseq1) to AS, AS response 401 to IAD, IAD rexmit the same sceq1, and send a new sceq2 with auth header.

The code is modified to response rexmit request (cseq=1). So that a subsequent one cseq=2 can handle properly relay to AS.

Workaround: None.

The SBC 9.1 MD5 check sum.

Impact: The SBC qcow2 image is shipped along with a sha256 checksum but Openstack glance supports only md5 checksum. Unable to validate the image in an automated way.

Root Cause: Checksum was updated from md5 to sha256 as md5 is weaker compared to sha256 algorithm and there are reported attacks on md5.

Steps to Replicate: With the fix build, ensure qcow2.md5 is one of the build artifacts.

The code is modified to facilitate openstack glance image verification.

Workaround: User can manually create md5 checksum by running below command:
md5sum sbc.qcow2 > sbc.qcow2.md5

The SNMPv3 traps from the MRFP are not displayed in EMS Fault Management.

Impact: The traps were not getting displayed in EMS.

Root Cause: The SBC generates Engine ID at the time of ISO installation. Due to this, all cloud instances share same engine id from the qcow2 generation time.

When same engine id is returned by two SBCs in different cluster, the EMS is unable to differentiate between them.

Steps to Replicate: Launch instances from qcow2 in different clusters. Traps generated should be shown correctly now.

Engine ID is created at the time cloud instance launch now to address the issue.

Workaround: None.

Configure the import issue.

Impact: The fields that supports \r like regex fields, changes it after a configuration Import(XML based).

Root Cause: The \r is not xml encoded when parsing intermediate xml file.

Steps to Replicate: 

1. Create SMM Profile with regex that has \r export configuration as xml-tar clear DB.
2. Import the configuration.
3. Verify SMM profile regex field.

The code is modified to handle \r as a special case.

Workaround: None.

Observed a SCM Process coredump in the gcored while running the 30 SWOs on 7000 Platform.

Impact: In case of switchover and switchback while running load, we expect some stale entries in SIPSG and CC control blocks. Such entries are cleared on the audit, but while clearing the SIPSG is sending accounting message to CC.

This SIPSG was reconstructed, but in CC the index corresponding to this GCID was allocated to some other GCID.

When the CC receives such message, the mismatch occurs and it logs a crash in sensitive mode.

Root Cause: During load run with multiple switchovers, various call modules get reconstructed when standby transitions to active. Due to this some stale entries are expected. If one of these stale entries sends some message (e.g accounting message) and if index for that call was allocated to another call, this error can happen. In case of such mismatch the SBC logs a crash in sensitive mode.

Steps to Replicate: Load run with multiple switchovers.

If the GCID sent from SIPSG module in accounting message is different from the GCID received in CC module, for switchover case do not do a sensitive crash. Instead add a major log to address the issue.

Workaround: Run load scenarios in normal mode instead of sensitive mode.

The emssftp fails in the 9.1R1.

Impact: The emssftp login was failing.

Root Cause: The sequence of asking for a password and keep credentials in sync caused the emssftp password failed.

Steps to Replicate: Checking login, after multiple reboots, rebuild with 1:1 and N: 1, along with 1-2 EMS registered.

The code is modified on the mode(HA vs standalone), included a new script to handle password request and updated for some HA cases.

Workaround: None.

Portfix SBX-104962: The HFE setInterfaceMap() can be incorrect.

Impact: The HFE_AZ.sh script can configure the HFE node incorrectly if the IP of an interface is a contained with in another.

Root Cause: The logic used to determine which interface associates while the NIC can find the incorrect interface.

Steps to Replicate: Create a HFE where the eth0 address contains either eth1 or eth2 IP address.

The code is modified to verify it the only the correct IP address is used.

Workaround: Request an updated HFE_AZ.sh script from Ribbon. Replace the HFE_AZ.sh in the storage account and reboot the HFE node(s).

Portfix SBX-104156: The RTT-TTY support verifies the call flow in the SBC.

Impact: Lower case characters from t140 packet were not getting converted correctly to Baudot (tty). Also, some other characters were not getting converted as per V.18 A2 table.

Root Cause: The translation table from T140 to Baudot(TTY) only accounted for the subset of characters that were present in the Baudot (TTY) character set.

Steps to Replicate:

1. Setup a T140=>Baudot (AMRNB=>G711) call.
2. Create t140 pcap file with all UTF8 characters (128) and validate the baudot generation.
3. Create a t140 pcap file with multi-byte sequence and validate the baudot.

The code is modified to:

• The t140=>Baudot translation table as per V.18 Annx2. 
• Validate multi-byte (2,3 and 4 bytes).
• Handle the Byte Order Mark (BOM).
• Ignore an invalid byte sequence and substitute the valid but unknown multi-byte seq with an apostrophe.

Workaround: None. Instead of lower case letters use upper case letters.

PortFix SBX-96783: Some counts in the callCurrentStatistics keep incrementing.

Impact: The "activeRegs" counter provided by the CLI zone callCurrentStatistics/ callIntervalStatistics command may continuously increase.

Root Cause: Badly formed SIP REGISTER messages received by the SBC
(i.e., a REGISTER message that the SIP parser determines is bad)
will increment the "activeRegs" counter and never decrement it.

Steps to Replicate: Send bad SIP REGISTER message(s) to the SBC, and see that the
"activeRegs" counter provided by the CLI zone callCurrentStatistics / callIntervalStatistics command increases (and does not decrease).

Decrement the "activeRegs" counter when the SIP REGISTER message fails the SIP parser to address the issue.

Workaround: None.

Portfix SBX-102364: The SBC behavior is inconsistent with the isfocus parameter handling.

Impact: 

Issue 1: The SBC is not transparently passing the complete Contact header in 200 OK of SUBSCRIBE when there is a isfocus parameter.

Issue 2: The SBC is not adding the Record-Route in any message when doing a full Contact header transparency

Root Cause: 

Issue 1: The SBC passes contact header transparently only when the isfocus is present in request as well as response and would not send the contact header transparently only when the contact header in response has isfocus parameter.

Issue 2: The code to add a record route for notify and its response is not present.

Steps to Replicate: 

1. Configure the SBC for A to B call.
2. Enable the flag contactTransparencyForIsFocusMediaTag on both TGs.
3. Make a SUBSCRIBE-NOTIFY call flow.

Issue 1: The code is modified to check if the service bit is not set (for the response) and check if the contact header has the isfocus parameter, if the parameter is present, set:

SIP_SERVICE_TYPE_CONTACT_TRANSPARENCY_
    FOR_ISFOCUS_PARAM

for a single contact scenario.

Issue 2: The code is modified to add the record route for notify and its response if isfocus parameter is present.

Workaround: None

Portfix SBX-103988: The ICE not working after upgrade to V08.02.03A950.

Impact: When a call with ICE changes from direct media to passthru, ICE learning does not get enabled on the call.

Root Cause: The code was not re-enabling the ICE FSM to start ICE learning when changing from direct media to passthru.

Steps to Replicate: 

Test 1
--------

1. Establish a call with ICE on ingress and xdmi on egress.
2. Send re-invite from egress without xdmi and complete the re-invite related signaling.
3. Send stuns to ingress media to complete ICE learning and verify call state changes to established and media can be exchanged between ingress and egress.

The code is modified to re-enable the ICE FSM when changing from direct media to passthru to allow the receipt and processing of stun messages to complete ICE learning.

Workaround: None.

The MGT0 cannot reach the GW.

Impact: The mgt0 interface appears to be inactive. The user cannot ping the gateway from the SBC 5400.

The ethtool -S mgt0 shows the tx_pause count rising rapidly.

Root Cause: This problem is specific to the SBX 5400 fix.

The SBC 5400 has a backpressure mechanism for management ports so that when the receive buffer exceeds a certain level, Design sends pause packets out to the interface to ask the switch/router to pause transmission momentarily.

The receive buffer count is incremented by a hardware module and decremented by software.
The notification was sent from Network Processor to application about sRTP Rollover Counter (ROC) reset is specifying the wrong interface so software decrements the wrong receive buffer count.

This causes the receive buffer count to wrap around, resulting in a high value. This triggers the back-pressure mechanism and Design continuously send out pause packets.

Steps to Replicate: 

1. Set up a pass-thru SRTP call.
2. The call should be a long duration call so that the Rollover Counter (ROC) in the sRTP packet becomes non-zero.
3. Modify the call so that the sRTP packet's the SSRC changes.

Update:

1. Do a long duration sRTP call (about 30 minutes.)
2. Hold the call.
3. Unhold the call.
4. As a result, this causes an SSRC change and triggers the problem.

Set the correct interface information in the notification to address the issue.

Workaround: Please contact Ribbon support for a script that will restore contact to the management interface.

Portfix SBX-103922: There was a PRS Process coredump on an active/A node in the middle of an upgrade.

Impact: The PRS Process cored during the LSWU while syncing ICE call data.

Root Cause: The ICE redundancy code is taking a long time to sync the call data with the standby during LSWU, causing a Healthcheck timeout and a core dump.

Steps to Replicate: The issue cannot be reproduced easily. 

The code is modified so the ICE syncs the call data to standby and addresses the issue. 

Workaround: None. 

PortFix SBX-104704: Duplicate the audio entries in the route PSP.

Impact: The duplicate codec entry is being sent in the policy response.

Root Cause: The function popPSPCodecEntryHlp that populates the codec entry was called twice, and as a result the codec entry was getting populated twice in the TLV.

Steps to Replicate: Add more then 4 codec entries in the ingress PSP and egress PSP. This will cause the issue to be reproduced.

The code is modified so that the duplicate codec entry is not passed on the d+ response.

Workaround: None.

PortFix SBX-101575: There was an offline upgrade failure from 8.2.1 to 8.2.2

Impact: An offline upgrade on a HA setup failed for the upgrade from 8.2.1 to 8.2.2.

Root Cause: An offline upgrade failed due to presence of a model update marker files that should not get created in case of offline upgrades.

Steps to Replicate: Perform an offline upgrade on a HA setup to the fix build and ensure upgrade is successful.

The code is modified to ensure nodeB is powered-off/shutdown (instead of the sbxstop) while upgrading a nodeA. This ensures that the model update related markers files do not get created.

Workaround: None.

PortFix SBX-104668: The SNMP PM collection is not working for newly added KPIs.

Impact: SNMP stats collection at EMS for G711_WITHOUT_XCODE_RES_PEAK_COUNT and G711_WITHOUT_XCODE_RES_AVG_COUNT of DspResDspUsageIntervalStats failed.

Root Cause: Some of the KPI stats fields are made obsolete as they are no longer supported. Due to this change EMS is not able to collect the stats through SNMP.

Steps to Replicate: Verify EMS SNMP walk should not fail when queried to the SBC.

1. Bring up SBC.
2. Register SBC to EMS.
3. Enable the sonusDspResDspUsageIntervalStatistics.

Expected Result:
The EMS fetches stats from the SBC.

The code is modified so the EMS is able to collect stats using SNMP.

Workaround: None.

PortFix SBX-103175: The large microflow profile was not getting enabled in the Custom Traffic Profile.

Impact: There are multiple issues:

1. For a default profile, max subs was always set to 256K.
2. For the custom and standard traffic profiles, the micro flow count in the NP resource was not proper.
3. Limited support of the 2M micro flows for standard profiles.

Root Cause:

1. The large micro flow support was not there for the custom traffic profiles.
2. For standard and default traffic profiles, there was restricted/incomplete support of large micro flow that is made generic as part of this issue.

Steps to Replicate:

1. Create an instance with 10GB memory and 16vpcu with the default traffic profile.Check microflow limits using "/opt/sonus/bin/cpsi -d summary" command.
2. Increase the memory to 50GB memory and check the microflow limits using "/opt/sonus/bin/cpsi -d summary" command.
3. Create a instance with 20GB memory and 16vpcu. Create and activate a custom traffic profile with access enabled in call mix. Check microflow limits using "/opt/sonus/bin/cpsi -d summary" command.
   
   

The code is modified to:

1. Introduce a new estimation parameter maxSubs based on the micro flow count is decided and the NP hugepages are reserved.
2. Enable the large micro flow support for all the standard/default profiles where mem > 48GB and vcpu_count >10.

Workaround: No workaround available. Need to use the build after the fix.

PortFix SBX-104347: The resource mem congestion level 3 is approaching threshold 90 sample 82.

Impact: The SIPCM is leaking a structure that is associated with the SIPMM functionality.

Root Cause: The code to free this memory under certain error/edge scenarios is missing.

Steps to Replicate: Since the exact call flow is not know that caused the customer to hit the error condition, no test steps can be used.

The code is modified to free the SIPMM related structure in error scenarios.

Workaround: Since the exact call flow is not know that caused the customer to hit the error condition, we cannot suggest a workaround.

PortFix SBX-102707: The network interfaces are not renamed as expected after an upgrade from 820R2 to 821R0.

Impact: Network interfaces are not properly mapped when the system is abruptly brought down after the first boot and rebooted.

Root Cause: Cloud-init service runs in parallel to sonusudev service and can cause issues with the network interface mapping as it tries to rename the interfaces.

Steps to Replicate: Reboot the system after sonusudev is run on first bootup.

The code is modified to start only after the sonusudev has run and mapped the interfaces properly.

Workaround: Reboot the instance again to recover from the issue.

PortFix SBX-99208: The SBC has the same issue as SBX-86420, this time for an outgoing REGISTER messages.

Impact: When the TLS port is not sent from the PSX, the SBC is not sending the DNS SRV query to the DNS server.

Root Cause: The root cause was the SBC just increments the port number by 1 sent by the PSX even if peer port was Zero.

Steps to Replicate: 

1. Configure the SBC and PSX to send a register and subscribe SIP request to relay.
2. Configure the serve FQDN and zero for the FQDN port.
3. Send a register/subscribe request.
4. The SBC should send SRV query to the DNS server.

If port number sent by PSX is zero then do not increment it by +1 by default for selecting TLS port to address the issue.

Workaround: None.

PortFix SBX-103561: The "tgrp" parameter are not passing transparently when the SIP in the core is enabled.

Impact: The SipCore calls is passing the wrong TGRP parameter value.

Root Cause: There was missing logic to support the SipCore.

Steps to Replicate: Configure the SipCore feature. Both the IPSP core and egress leg have "originating Trunk Group Options" set to "Include Tgrp with IP address"
Make a SipCore call.

The code is modified to support the SipCore feature.

Workaround: None.

PortFix SBX-103852: The SBC sent an unnecessary 481 for PRACK when the CANCEL is received.

Impact: The SBC sends unnecessary 481 for PRACK when received CANCEL.

Root Cause: The issue was introduced by intercept feature (SIP_EVENT_PRACK_INTERCEPT_RCVD).

When the SBC received PRACK, it save the message into the server list. Later the CANCEL arrives, UasCancelRequestCmd, the UAS found the message in server list, and thought PRACK is not completed therefore it try to send 481.

Steps to Replicate: A calls B, B response 180 and send 180 to A with PRACK required.
After 32 seconds A sends a CANCEL, the SBC sends an unnecessary 481 PRACK.

Once the 200OK for PRACK has send out, the SIPS deletes the PRACK message in the server list to address the issue.

Workaround: Enable the e2e PRACK if the egress also supports PRACK.

PortFix SBX-96788: The DTMF 2833 PT change (in offer) has incorrect OA SBX handling (wrong answer PT).

Impact: When the peer offer with new PT, the SBC is unable to process and respond to the previous one.

Root Cause: There was a logical error that checking the wrong field to detect PT change.

Steps to Replicate: A call B and connect with "0 100". An INVITE with an offer new PT "0 101". The SBC responds with "0 100".

The code is modified to check the correct field for PT change.

Workaround: None.

PortFix SBX-105428: Error when attempting to delete negative cache record upon a probe response.

Impact: The DNS query was not out sent from the SBC if a call is made immediately after the DNS server is recovered from a blacklist.

Root Cause: Due to record is in the negative cache entry from earlier DNS dummy query, the SBC was not sending a query.

Steps to Replicate: 

1. Configure the SBC with a basic call and also configure the DNS group.
2. Ensure that DNS server does not respond to the DNS query or stop DNS server.
3. Now make call and SBC query for DNS server for resolving FQDN, but DNS server is down and there no response from DNS server.
4. After retransmission's timeout value. DNS server will be blacklisted.
5. Now the SBC keep probing DNS server.
6. Bring the DNS server up.
7. The SBC receives probe response, DNS server is removed from Blacklist.
8. Immediately make SIP call.

Expected Result:

• The SBC selects a DNS query.

As soon as the DNS server is removed from blacklist entry, those records involved with the DNS Dummy query is removed from the negative cache and resolves the issue.

Workaround: None.

PortFix SBX-105010: Observed a congestion for the G711 to G711 transcode calls even before the CPU resource limit is exhausted.

Impact: The G711-G711 transcoded capacity test shows a lower capacity.
(Test is Pump make and break g711 to g711 transcode calls on one of the active instances with 50cps, 90 CHT and 4500 calls).

Root Cause: The root cause is an incorrectly built libfmtd.a that was built with a debug option and makes the FMTD module take more cycles. This bug was addressed originally in SBX-104122.

Steps to Replicate: Perform a test indicated in SBX-105010.

The code is modified to remove the -g flag and using -O3 flag for compilation (Port of SBX-104122).

Workaround: None.

PortFix SBX-105072: Password padding with random characters by the SBC causes the RADIUS server to reject the password.

Impact: The radius password sent to the server has no zero characters at the end following the password and a NULL.

Root Cause: The radius passwords are padded to 16 characters. The existing implementation did not set those padded characters to 0.

Steps to Replicate:

1. Configure a radius authentication server.
2. Use a password that is less than 15 characters long for the external radius user.
3. Set the externalAuthententication to true.
4. Run a tshark session.
5. Login to the CLI.
6. Stop the tshark.
7. View the radius password element in wireshark after configuring the shared secret in wireshark under protocol preferences.

The padded characters are now set to 0 to address the issue.

Workaround: Use 15 or 16 character passwords.

PortFix SBX-66831: Need to update/add proper warning message when deleting the ipInterface.

Impact: The warning message does not display in the cloud SBC when deleting the IP interface, but there is no problem with functionality.

Root Cause: This is working fine in the hardware SBC. We have the problem in the cloud SBC because when displaying the warning message, we checked few values if that values are matching. Then, show the warning popup but in case the cloud SBC values index are different and check is failing because of that, we are unable to show the warning popup when we delete IP Interface.

Steps to Replicate: 

1. Login to EMA.
2. Go to All -> Address Context ->IP Interface Group -> IP Interface.
3. Click on New IP Interface button.
4. Fill the correct info in text field, select state as enable, Mode as In Service and click on save button.
5. The creation should be successful.
6. Select the same entry from IP interface list section and click on delete button.
7. We should be able to see warning popup like "'addressContext PPP_AC ipInterfaceGroup PPP_IG ipInterface': Deleting ipInterface while in dryUp action can
   cause problems with calls on the ipInterface. Use force action to clear existing calls on the ipInterface before deleting. Do you wish to continue?"

The code is modified to handle the Cloud SBC scenario.

Workaround: None.

PortFix SBX-100286 to 9.2: The trace file containing SIP Rec PDU was not imported in Ribbon Protect properly.

Impact: When a level 4 call trace is active, if a SIP PDU is too large to fit on a single trace line, it is split over multiple lines. However, the Ribbon Protect only reads the first line - there is no way for it to know subsequent lines are continuation lines.

Root Cause: Design issue.

Steps to Replicate: 

1. Enable level 4 call trace.
2. Send a SIP PDU to be traced towards the SBC of total size 2000 bytes.

The code is modified to put the same header onto each level 4 trace line. Include in the header two new fields, the MSG ID and PART to allow Ribbon Protect to recombine multiple trace lines to recover the original message. The equivalent Jira VIGIL-17137 is required for Ribbon Protect compatibility.

Workaround: None.

PortFix SBX-100590: The wrong "Egress Zone Name" in ATTEMPT CDR for a GW-GW call.

Impact: For a call with multiple routes that include one or more local routes followed by routes on another gateway (for GW-GW) scenario, if the local routes fail and then other gateway route is selected, the resulting ingress gateway CDR for the call has incorrect egress zone name and ID.

Root Cause: When there are multiple routes, when a route fails the call cranks back to use the next route. When selecting the next route the code is not clearing the internal CDR information for the egress zone. So if the next route selected is on another gateway, the CDR information retains the egress zone information from the previous attempt rather than being blank.

Steps to Replicate:

1. With the SBC GW-GW setup create two routes for a call such that:
   - route 1 is routed through local egress trunk group on SBC1.
   - route 2 is routed through egress trunk group on SBC2.
   - Set crankback profile to enable attemptRecordGeneration and add reason 41 (503 is mapped to 41 in default SIP to CPC mapping profile).
2. Send a valid INVITE to the SBC1 ingress that will first select route1 and route out through the egress trunk group on the SBC1. From the egress endpoint reject that call attempt with a 503 Service Unavailable.
3. The call will crank back and re-route using route2 through the SBC2 egress trunk group. From the egress endpoint reject that call attempt with a 503 Service Unavailable.
4. Check there are two Attempt CDR records for the call on SBC1.
   1. ATTEMPT1 - should have the "Egress Zone Name" and "Egress Zone ID" fields populated correctly with zone information for zone of SBC1 egress.
   2. ATTEMPT2 - should have the "Egress Zone Name" empty and "Egress Zone ID" as 0 because this is an attempt for GW-GW route.

The code is modified to clear the internal CDR information for the egress zone when one route fails and the next route is selected.

Workaround: None.

PortFix SBX-74991: Saving changes to a SMM profile after doing an edit/update operation is taking long time[8 to 16 minutes].

Impact: The SIP Adaptor Profile with 250+ rules takes lot of time to create a profile including the update as well.

Root Cause: When the SIP Adapter Profile with more than one rule is created, each rule is committed individually. However, after each commit EMA internally retrieves all the SIP Adapter Profiles along with their rules, though this is not required it is causing slowness of both create and update operation.

Steps to Replicate: 

1. Login to EMA.
2. Navigate to Sip Adapter Profile Screen.
3. Create a profile with 200+ Rules.
4. Click on Save button.

The code is modified to prevent retrieval of sip adapter profiles after each commit

Workaround: The only workaround is to create the profile from the CLI.

PortFix SBX-101769: The SBC sends the wrong SDP in 200 OK for UPDATE received from the Ingress during tone play(LRBT).

Impact: The SBC responds with wrong codec in the answer to UPDATE received on ingress/calling side while local ring back tone is playing. As a result, the UPDATE is being used to change the codec on ingress. In this case, the SBC continues to play the RBT using the previously agreed codec.

Root Cause: There was a logical error in the code that was picking the currently active packet service profile to answer instead of honoring the modify.

Steps to Replicate: Configuration:

1. Configure SBC to make a Basic A-B call.
2. Configure PSX with G711 and G729 in codecEntry and this Leg/other Leg configuration.
3. Configure LRBT, and attach to Ingress TG.

Procedure:

1. Send an INVITE from UAC with G711.
2. Send an 180 from UAS, with no SDP.
3. Send an UPDATE from UAC, with G729.
4. Receive the 200 OK from INVITE from UAS.

The code is modified to address this issue.

Workaround: None.

PortFix SBX-103771: There are special characterss in DN hinders EMA display.

Impact: The EMA does not display route with destination national containing special characters.

Root Cause: The support for special characters was not available in EMA.

Steps to Replicate: 

1. In the SBC 7.2.4 R0, create a Route with Destination National containing special characters.
2. Upgrade to 8.2.4 R0.
3. After the upgrade, the EMA should display the route.

The code is modified to support special characters in the EMA.

Workaround: The CLI can be used to view route details.

PortFix SBX-104247: The resource memory congestion level 3 is approaching the threshold 90 sample 81.

Impact: There was a SAM Process is leaking memory when AKA is being used.

Root Cause: There was a SAM Process is leaking an AKA related structure when the code is handling an error case scenario.

Steps to Replicate: The issue cannot be reproduced.

The code is modified to correctly free the AKA structure in all scenario.

Workaround: There is no known workaround.

PortFix SBX-93898: The "request sbx xrm debug command sec -stat gcid <gcid>" was not showing ENC and DEC details on the SBC SWe.

Impact: This debug command in unhide section is not showing all the required fields populated in the SWe SBC.

Root Cause: NP response is not framed in expected order to application layer.

Steps to Replicate: Run the SRTP call and issue this debug command for internal use.

The code is modified so the NP Response is correctly framed in expected order to application layer.

Workaround: This debug command is to see SSN field value with the RoC, all other details can be seen from show call mediastatus.

PortFix SBX-103599; The SBC is sending multiple re-INVITEs to the ingress and egress leg during a 200OK of INVITE answered with Dialog-1.

Impact: The SBC is sending multiple re-INVITEs to the ingress and egress side if the first forked leg is answering with a 200 OK.

Root Cause: When the 200 OK is coming without SDP, we were taking last received SDP and that precondition attributes to further processing.

Steps to Replicate: Precondition Transparency is set on both the Ingress and Egress
Steps:

UAC sends INVITE with Supported: precondition and SDP with precondition attributes.

First forked UAS sends 183 with SDP with preconditions.

Second forked UAS sends 183 with SDP with preconditions.

First forked leg send 200 OK without SDP.

Expected Behaviour:

1. The SBC transparently passes precondition attributes in SDP.
2. The SBC gets first forked 183 with SDP with preconditions.
3. The SBC forwards the 183 to ingress with preconditions.
4. The SBC sends an UPDATE to ingress and egress to complete precondition state.
5. The SBC forward second the 183 to ingress with preconditions.
6. The SBC sends an UPDATE to ingress and egress to complete precondition state.
7. The SBC forwards 200 OK and get ACK.
8. The SBC completes reINVITE-200-ACK towards egress.

The code is modified to address the issue.

Workaround: None.

PortFix CHOR-6320: The UxPAD coredump observed on the media container while pumping 2X overload of G711U to G711U DSP (media transcoding disabled) call load.

Impact: The SWe_UXPAD crash was seen during an overload transcode test in a 2 vcpu SWe deployment in the public cloud.

Root Cause: Potential corruption in the packet buffers due to issues related to concurrency specifically in the 2 vcpu transcode overload scenarios.

Steps to Replicate: 

1. Configure a 2 vcpu instance.
2. Run 700 G711-G711 transcode sessions for overnight/long duration.
3. Check if there is any UXPAD/NP crash seen.

The code is modified to fix concurrency issues in transcode call scenarios in the 2 vcpu SWe deployments.

Workaround: No workaround apart from avoiding overload scenarios for transcoded calls.

PortFix SBX-105182: After the SBC receives RCODE = 2 from the DNS server, the SBC makes a strange query request to the DNS and subsequent new calls are rejected.

Impact: When RCODE error received for the DNS query and no other servers were available, the record was never being deleted from cache and this remained as a RECORD in "RESOLVING STATE" that impacted further queries with this domain and resulted in call failures.

Root Cause: This particular test scenario's was not covered during the testing feature.

Steps to Replicate: 

1. Setup with EDNS supported and DNS fall back disabled.
2. The single DNS server configured for the DNS Group.
3. The DNS Server configured to return RCODE Error 2 for SRV Query.
4. Setup a call that trigger NAPTR, SRV and A Query and the DNS Server responds with RCODE 2 for SVR.
5. Modify the DNS Server to return successful RCODE for SRV.
6. Make another call for the same domain. The call fails because the SRV record is in "RESOLVING STATE" in the cache.

The code is modified to delete cache entry (which is in "RESOLVING STATE") when the RCODE error is received.

Workaround: NA

Enhance the sbxPerf on the SBC for lesser resource consumption.

Impact: The SBC performance monitoring tools like top and top2 at times take 20% of a CPU core there by reducing total available CPU resources for management activities on the SBC.

Root Cause: The sbxPerf that contains list of tools such as top, mpstat, and iostat currently runs periodically without any linux value configured. This can cause these processes to get prioritized and scheduled ahead of other management processes such as ConfD and SSH.

Steps to Replicate: Install fix build and ensure processes such as top, top2, mpstat sbxPerf are running with corrects value of 15 using the top command.

The code is modified so that priority of these processes are less compared to other management processes on the SBC.

Workaround: None.

The ACL rule is missing on the SBC in LD triggered switch-overs and link recovered after.

Impact: When an IPACL rule is created that references an IP interface, but that IP interface has been down since the system started up, the rule is not successfully created.

Root Cause: When an IP interface is failed on system startup, the system does not add that interface to the kernel. When an IPACL rule is added that references that IP interface, it fails to be added to the NP.

Steps to Replicate: Start an SBC with an IP interface failed, that also has one or more IPACL rules that have been configured that reference that IP interface. Bring up the IP interface and logs will show the IPACL rules have been successfully added.

The code is modified to detect this situation and maintain a retry list. Once the IP interface comes up, it will retry the associated IPACL rules that were previously failed.

Workaround: This issue can be worked around by having IPACL rules that do not reference the IP interface.

There was a customer call forwarding MRF interaction failure on the receipt of an UPDATE.

Impact: The SBC sends INVITE towards MRF upon receiving UPDATE from Egress End Point

Root Cause: In some situations when there is delay in incoming PRACK request (In this situation PRACK from Ingress Endpoint), some internal logic of the SBC queues the SDP on ingress leg and let's SBC answer to egress, thereby unblocking the egress peer which can send a new offer using SIP UPDATE.

This causes the SBC to send INVITE towards MRF, even before sending an UPDATE out towards the Ingress.

Steps to Replicate: Configure the PSPs accordingly:

1. UAC sends Invite with AMR-WB, AMR, telephone-event.
2. UAS sends 183 with SDP with AMR-WB with 100rel.
3. UAC sends PRACK/200 OK.
4. UAS sends 180 without SDP without 100 rel.
5. PRACK for 180 is pending on Ingress.
6. UAS sends UPDATE with PCMU, telephone-event.

Expected Results:
==============

1. Upon receiving an UPDATE with PCMU, the SBC will not auto answer towards UAS.
2. The SBC will wait for PRACK to be received from UAC, then will relay UPDATE towards UAC.
3. Based on answer in 200 OK (Update) from UAC, the SBC will send INVITE towards MRF for reserving transcoding resources.

The code is modified to ensure that SBC does not auto answer to egress.

Workaround: None.

The No Way Video after A/V calls is resumed after hold.

Impact: No way video after audio video call is resumed using late media re-INVITE.

Root Cause: The SBC does not send "sendrecv" in 200 OK to late media re-INVITE, instead sends the same datapath mode that was negotiated last - that is - inactive.

Steps to Replicate: Setup an audio and video call. Put the call on hold by sending a=inactive in audio and video SDP. Then, send a late media re-INVITE to take the call off-hold. The idea is that the SBC will send SendRecv for audio and video in 200 OK and let the remote end choose if it wants to remain in hold or come out.

The code is modified to send sendrecv in other streams too when call is being "may be" resumed after hold.

Workaround: Do not use late media re-INVITE to come out of hold.

Observed a PRS process core dump "systemerror healthcheck timeout"

Impact: A PRS coredump was found due to healthcheck timeout upon executing a CLI debug command

Root Cause: The CLI debug command was taking a long time due to too many ACL entries resulting in a health check timeout.

Steps to Replicate: 

1. Add more than 1000 ACL entries
2. Execute below debug command:
   Run CLI debug command "request sbx xrm debug command acl\ -stat"
   
   NOTE: The debug commands are not available on customer sites.

The code is modified to process only 200 ACL entries to allow CLI to recover sooner.

Workaround: As we do not run debug commands on customer sites, this error should not  occur.

There was a problem with management of the maxptime and ptime in the Direct Media mode.

Impact: There was a problem with management of maxptime and ptime in the Direct Media mode.

Root Cause: Generally, the SBC ignores the ptime value if only maxptime attribute is present in incoming SDP, If "sendPtimeInSdp" IPSP flag is enabled and peer is sending both ptime and maxptime, SBC is preferring maxptime value while sending ptime,
The existing IPSP flag "sendPtimeInSdp" makes the SBC to send out a=ptime irrespective of what we receive from the peer.

Steps to Replicate: 

1. Do a basic call from A to B.
2. A sends ptime=20 and maxptime=40 towards the SBC.
3. An INVITE going out from the SBC should contain ptime=20 as we enabled both "sendPtimeInSdp" IPSP flag and "preferPtimeInSdp" TG flag.

A new flag "preferPtimeInSdp" is introduced under SBC CLI Trunk Group signaling.
If this new flag "preferPtimeInSdp" is enabled, the SBC prefers ptime value received in a=ptime over a=maxptime in incoming SDP.

This new flag "preferPtimeInSdp" needs be enabled in conjunction with existing flag "sendPtimeInSdp". If the SBC is configured to send a=ptime, then preferPtimeInSdp should be enabled on the incoming trunk and vice versa.

Workaround: None.

The RTP sourceAddressFiltering is not working (call leg dependency).

Impact: The source address validation is not done if call flow does not involve NAPT learning. For calls that want source address validation but does not have NAPT enabled, the SBC would not validate the source address and end up forwarding the RTP packet to the other endpoint.

Root Cause: There is no way for application to inform Network Processor to validate source if NAPT learning is not enabled.

Steps to Replicate: Involves a CISCO MOH server but probably something else can be used.

The code is modified to validate source even when NAPT learning is not enabled to address the issue.

Workaround: Enable NAPT learning for the call.

The SBC5210 upgrade failure during Starting DB_RESTORE stage.

Impact: An upgrade failure during Starting DB_RESTORE stage.

Root Cause: Creation of foreign key fails on table packet_service_profile column CODEC_ENTRY_FK9, as it had "0" in one of its rows and dbimpl.codec_entry table did not have this value.

Steps to Replicate: Upgrade a dump that has "0" in packet_service_profile.CODEC_ENTRY_FK9, it should be successful.

Before add the referential keys, the data is checked and ant value that is not there in parent table is nullified to address the issue.

Workaround: None.

Empty "Egress Local Signaling IP Addr" field in the CDR

Impact: When an incoming call is routed out of the SBC but is then rejected by the SBC with cause 132 Module failure before a backwards response message has been received, the resulting attempt CDR has an empty "Egress Local Signaling IP Addr" field.

Root Cause: The SBC currently populates the "Egress Local Signaling IP Addr" CDR field when it processes a backwards response message for the call, and as a result the field remains empty until a backwards response message is received and processed.

Steps to Replicate: 

1. Initiate a call by sending a valid INVITE to the SBC ingress TG to route towards the egress TG.
2. Once the INVITE is sent to egress by SBC, put the signaling port to egress out of service.
3. The SBC will clear the call by sending a 503 message towards egress. The resulting attemptCDR should have valid address in "Egress Local Signaling IP Addr" field.

The code is modified to populate the "Egress Local Signaling IP Addr" CDR field on sending out the egress INVITE message.

Workaround: None.

There was a memory leak on the glusterfsd.

Impact: The OAM application cores.

Root Cause: Memory leak is observed for the glusterfsd process, that is the brick process. The use of the "gluster volume heal info" function causes the process memory usage to increase and not go down afterward.

Steps to Replicate: Ensure both Active and Standby OAM are running.
Monitor glusterfsd memory usage on both nodes.

Use the "gluster volume heal statistics heal-count" command to determine the gluster bricks heal state to address the issue.

Workaround: Reboot the OAM node.

The SBC is ignoring Use Max Bitrate Only flag.

Impact: The SBC is ignoring Use Max Bitrate Only flag.

Root Cause: During Re-Invite answer processing, NrmaAdjustPeerT38MaxBitRate() routine is doing T38MaxBitRate adjustment based on packet size that is changing T38MaxBitRate value from 14400 to 4800.

Steps to Replicate: Call Flow:

1. Do a basic call between UAC and UAS.
2. The UAS sends fax re-invite to UAC with T38MaxBitrate set to 14400.
3. The SBC sends Invite out with T38MaxBitrate set to 14400.
4. The UAC answers with T38MaxBitrate set to 14400 in 200OK.
5. The SBC sends out 200ok with T38MaxBitrate set to 4800.

Actual Result:

------UAS(Re-Invite) ---------------------------> SBC --------------------------> UAC
T38MaxBitRate:14400 T38MaxBitRate:14400
-----UAC(200ok for Re-Invite) --------------> SBC ---------------------------> UAS
T38MaxBitRate:14400 T38MaxBitRate:4800

Expected Result:

------UAS(Re-Invite) ---------------------------> SBC --------------------------> UAC
T38MaxBitRate:14400 T38MaxBitRate:14400
-----UAC(200ok for Re-Invite) --------------> SBC ---------------------------> UAS
T38MaxBitRate:14400 T38MaxBitRate:14400

The code is modified to relay the T38MaxBitRate in SDP, which is received from peer for Direct Media calls.

Workaround: None.

The redirection NOA set wrong in the CDR.

Impact: The "Redirecting Orig Cd Num - NOA" subfield of "Redirection Feature Spec Data" is not set correctly in CDR record if the the Redirecting Original Called Number - NOA value has been modified by PSX.

Root Cause: The existing code was not updating the value for the CDR record based on route specific information returned from the PSX.

Steps to Replicate: Test Setup
========

1. Test requires call routing through the PSX.
   Routing on PSX, setup to route incoming call through routing label with two routes:
   - route 1, through local egress trunk group 1 on SBC to UE2.
   - route 2, through local egress trunk group 2 on SBC to UE3.
2. On egress trunk group 1 only, associate a DM/PM rule to change the "Number Type" to International for "redirecting Original Called Number".
3. Set the crankback profile to enable attemptRecordGeneration and add reason 41 (503 is mapped to 41 in default SIP to CPC mapping profile)
   set profiles callRouting crankbackProfile default attemptRecordGeneration enabled reason 41

Test Procedure
============

1. Send INVITE to the SBC ingress, INVITE should include a diversion header e.g.
   Diversion: <sip:+4969667740185@xxx.xxx.xxx.xxx>;privacy=off;screen=no;reason=unknown;counter=1
2. Call is routed using route 1, respond from first egress with 503 Service Unavailable
3. Call is re-routed using route 2, respond from second egress with 503 Service Unavailable
4. Check the CDR records for call on the SBC that should have two attempt records -
   1. ATTEMPT1 - should have "Redirection Feature Spec Data" that has subfield 22 "Redirecting Orig Cd Num - NOA" as 3 (Unique International Number), because the number was updated by PSX to international.
   2. ATTEMPT2 - should have "Redirection Feature Spec Data" that has subfield 22 "Redirecting Orig Cd Num - NOA" as 2 (Unique National Number), because the number was not updated by PSX
      
      

The code is modified to populate the CDR value from the route specific data returned from PSX.

Workaround: None.

There was an incorrect RTP time stamp in the SBC packet capture.

Impact: After an SBC upgrade to V09.01.00R001, RTP/RTCP time stamp in the SBC packet capture shows the year 2036.

Root Cause: Present logic was not considering the edge cases while checking the last modification time of the NTP log.

Steps to Replicate: 

1. Configure a remote syslog server:
   set oam eventLog platformRsyslog linuxLogs ntpLog enabled
   set oam eventLog platformRsyslog servers server1 port 10514 protocolType udp remoteHost <remote host ip>
   com
   set oam eventLog platformRsyslog syslogState enabled
   com
2. Create a second dummy NTP server (just to trigger some logs to be written to ntp.log):
   set system ntp serverAdmin 1.2.3.4
   com
3. Delete the dummy NTP server:
   del system ntp serverAdmin 1.2.3.4
   com
4. Wait at least several seconds.
5. Repeat steps 2 and 3.

The code is modified for the edge case of NTP log last modification time.

Workaround: None.

The MS Teams with FLRBT enabled hangs the call.

Impact: When the Force Local Ringback Tone is applied on a call that has been through SIP replace followed by SIP refer procedure, and the final trunk group has downstream forking enabled. The final 200 OK response may be queued indefinitely, meaning the call does not get completed.

Root Cause: Interactions between Force Local Ringback Tone, multi-party and downstream forking.

Steps to Replicate: 

1. Make a SIP to SIP call.
2. A calls B.
3. B' replaces B.
4. B' refers to C.
5. The trunk for A has Force Local Ringback Tone applied.
6. The trunk for C has downstream forking enabled.
7. C responds with 180 with no SDP, 183 with SDP, then 200 OK.

The code is modified to prevent the queuing.

Workaround: None.

The SIPCM (SAM) deadlock reported on SNMP request.

Impact: The SAM Process cores when getTlsStatus command is executed.

Root Cause: The issue is because when fetching the TLS Status, we were trying to access sessData that was no longer valid.

Steps to Replicate: Execute getTlsStatus command when running TLS load. SAM process cores randomly.

The code is modified to fetch the status from socketPtr instead of sessData.

Workaround: None.

The sbxAutoBackup.sh changes for AWS SWE.

Impact: The Cloud SBC uses default system name 'vsbcSystem' when creating system backups, making it difficult to distinguish between setups.

Root Cause: The Cloud SBC configuration file backup logic uses the system name to create the back up file instead of the specific name.

Steps to Replicate: Run /opt/sonus/sbx/scripts/sbxAutoBackup.sh on the SBC running in a cloud (e.g. AWS) that has SystemName configured too something specific in user-data.

Use the actual system name (configured in user-data) in the configuration backup filename to address the issue.

Workaround: None.

Unable to see the signaling messages on the PKT log file.

Impact: The Packet capture does not work properly for pkt ports.

Root Cause: The version of libpcap0.8 library in Debian9 had a defect that resulted in the capture not working.

Steps to Replicate: 

1. Login to the SBC and run a basic call configuration.
2. Create a signaling packet capture filter: set global callTrace signalingPacketCapture state enable devices pkt0 0.
3. Run the SIPP calls.

Expected Results:

1. The SBC should capture all the signaling messages on the pkt0 port in a PKT file under the gsxlog directory.

Update the libpcap0.8 to a newer version to address the issue.

Workaround: None.

The SBC REST API. pam phase authorization failed to login through a PAM: timeout

Impact: Too many requests on the REST API interface causes 401 error.

Root Cause: Issue with third party tool ConfD that is used for configuration management.

Steps to Replicate: 

1. Send REST API requests for long duration.
2. Open some CLI session for some time to cause max session limit.
3. REST APIs start failing with 401 error due to session limit, as expected.
4. Close the CLI sessions.
5. REST APIs keep failing, even after keeping CLI sessions under max limit.

The code is modified to fix the issue.

Workaround: None.

The SIPREC was not working. No request sent to recorder.

Impact: Without this fix, SBC is generating INVITE towards the SIPREC SRS with bad format on SDP header line 'label'. The 'label' header is missing the CR in the CRLF line break.

Root Cause: The code that formed the SDP header line "label" does not add CR character for CRLF line break.

Steps to Replicate: Make a call that would match the configured SIPRec criteria and thus triggering a SIPRec session for the corresponding Communication Session.

The code is modified to add CR character for CRLF line break, for SDP header line "label" .

Workaround: None.

The reenableOsAccount silently sets an account expiration to 30 days.

Impact: Using the reeanbleOsAccount will add account aging to any OS account.

Root Cause: The logic does not take into account the state of OS account aging or if OS account aging should be applied (e.g. root).

Steps to Replicate: Run with an OS account aging enabled:

1. Test root:
   1. ReeanbleOsAccount for user root.
   2. Check no aging has been applied at OS level: chage -l root.
2. Test the Confd user:
   1. Create user through CLI.
   2. ReeanbleOsAccount for this CLI user.
   3. Check no aging has been applied at OS level: chage -l <CLIUser>.
3. Test OS user with a password:
   1. Create OS user with a password.
   2. Disable and then Enable OS account aging state.
   3. ReeanbleOsAccount for this OS user.
   4. Check correct aging has been applied at OS level: chage -l <OSUser>.

The code is modified in the OS account aging is enabled and is applied to the account.

Workaround: None.

The active OAM node for the Signaling SBC generates header-only ACT files and pushing it to the billing server, where the OAM node should be responsible for the configuration only.

Impact: The active OAM node generates header-only ACT files and pushing it to billing server.

Root Cause: The SBC does not check whether it is OAM node or Managed VM while generating ACT files.

Steps to Replicate: Spawned a Nto1 OAM node, check whether .ACT files exist or not. Restart active Node, Check if any .ACT file gets created. Restart the current Active again (assigned role Standby) and check whether any .ACT file is present on OAM node or not.

Check whether VM is OAM before creating the ACT files to address the issue.

Workaround: None. 

The SBC fails to relay embedded header in Contact of 3xx with statusCode3xx relay enabled

Impact: The SBC fails to relay embedded header in Contact of 3xx with statusCode3xx relay enabled

Root Cause: The code required to send the embedded part in contact of 3xx was not present

Steps to Replicate: 

1. Configure the SBC for A to B call.
2. Enable the flag statusCode3xx on egress TG IPSP.
3. Send an INVITE from UAC.
4. Send 300 Multiple Choice from UAS with below Contact header that has embedded headers.

The code is modified so the contact header transparency is set in the relay 3xx scenario that is now being changed to full contact header transparency and set the SIP_HEADER_URI_HEADERS so that the embedded contact header is transparently passed in the 3xx. A new header type is introduced that is set only in the relay 3xx case to send the entire contact header transparently.

Workaround: None.

The SubsystemAdmin filter affects calltrace (TRC) logging showed useless logs.

Impact: After creating then deleting an entry in the oam eventLog subsystemAdmin, the original behaviour is not restored. INFO level events for that subsystem are no longer logged, even if the oam eventLog typeAdmin filterLevel is info.

Root Cause: Deleting an entry in oam eventLog subsystemAdmin does not restore settings back to default.

Steps to Replicate: 

1. Create an entry for a subsystem in the oam eventLog subsystemAdmin table.
2. Delete the entry.

The code is modified to process to deleted the oam eventLog subsystemAdmin entry so that settings are put back to default values.

Workaround: None.

Our SIPREC implementation is prone to failure to record a stream and to tear down the SRS call leg when SRS (SIP Recording Server) sends a re-INVITE to SBC

Impact: SIPREC handling had race conditions issues in the scenario where a RE-INVITE was received on the main call and also RE-INVITE was received from SRS server at the same time.

This resulted in recording failure and the SBC sent out BYE towards SRS.