SBC Core 09.02.00R002 Release Notes

Portfix SBX-106989: Multiple Switchovers occurred because of a DSP core after an Upgrade to V09.02.00R000.

Scenario: A transcoded call with any Codec<=>G711 and RFC2833 DTMF relay enabled on both legs of a call, and the media probe is disabled.

Impact: If a peer device sends a RFC2833 without EOP packet (end-of-digit marker), then a DSP crash occurs.

Root Cause: The code tried to add an error condition (LOST_EOP flag) to media probe data structure without checking whether a media probe was enabled or not. This resulted in a NULL pointer access and a subsequent memory protection fault resulting in a dsp core dump.

Steps to Replicate: The customer case had a G711A<=>G711A transcoded call with RFC2833 enabled on both legs

Use the following setup:

SipP UAC=>SBX 5x10 => SipP UAS
G729AB(RFC2833) <=> G711(RFC2833)
show configuration system media mediaProbe state
state disabled;
From UAC send dtmf_2833_1_noEop.pcap with '1' digit w/o EOP packets

This resulted in dsp core dump before the fix and no core dump was observed after the fix and digit '1' was correctly sent to g711 side.

The code is modified so if the media probe is enabled and not write to media probe structure if media probe is disabled.

Workaround: Use RFC2833<=>Inband digits or RFC2833<=>SIP Info instead of RFC2833<=>RFC2833.

Portfix SBX-106206: An existing hairpin call gets silenced after a switchover.

Impact: When switchover occurs in the SWe SBC Active-Standby HA with the SBC internal loopback media calls flows, the media loopback is not working in existing calls. The looped back media is dropped internally in the SWe NP, because packet skb was not including the internal loopback flag set condition in the NP incoming path validation checks from the internally looped back media for the existing calls.

Root Cause: When a switchover occurs in the SWe SBC for existing medial calls, a loopback occurs in the SWe NP based on the matching SA. 

The DA IP address was configured in BRES by setting an internal loopback flag for packet work/skb, even though MAC DA is not updated in BRES flows to the new active instance interface MAC Address to match SA MAC.

However, this internal skb loopback flag settings was removed in the release 9.0 release NP refactoring code which caused the packet drops in incoming packet checks for MACs at NP from the looped back media of existing calls. The new calls will not have this issue post-switchover.

Steps to Replicate: With the SWe SBC Active-Standby HA setup, using AS/3GPP call flow scripts/setup, establish a call that creates the SBC internal loopback media flow. Verify the media flow and then Issue a switchover and verify the media again on this existing active call.

The code is modified to save NP the CPU cycles also while addressing the loopback media issue after a switchover issue for existing calls.

Workaround: None.

Portfix SBX-105269: The SBC crashed and a core dump created.

Impact: The SCM processes coredump due to NULL pointer access on a pointer that has been freed due to BYE and HOLD REINVITE in a transfer call scenario.

Root Cause: There was an illegal memory access, absence of NULL check, exposed due to REINVITE.

Steps to Replicate: A calls B, B REFERs to C and now A and C talk. After sometime, A sends BYE and C sends re-INVITE with a=inactive at the same time.

The code is modified to address the issue. 

Workaround: None.

Release 9.2.0 should have CAM version of "00610000".

New CDR fields were added in the 9.2.0R0 release, but the CAM version in the top of the ACT header file was not updated to "00610000". It was still set to ""00600000", which was used for release 9.1.0R0.

Impact: Incorrect CAM version in the top of the ACT header file.

Root Cause: The code to increment the CAM version was missed.

Steps to Verify Fix: Run a basic call and check the ACT log header to confirm the CAM version is set to "00610000"

The code is updated to correctly set the CAM version to "00610000".

The DSP Threshold setting is not generating a trap on SBC 5400.

Impact: The g711PacketThreshold, g729Threshold, and g726Threshold onset and abate traps are not sent.

Root Cause: The NRM did not receive CLI updates to the g711PacketThreshold, g729Threshold, and g726Threshold, and the trap generation code used wrong trap names.

Steps to Replicate: 

1. Provision the threshold levels:
   set oam traps dspAdmin dspAvailabilityTrap g729Threshold 40
   set oam traps dspAdmin dspAvailabilityTrap g726Threshold 60
   set oam traps dspAdmin dspAvailabilityTrap g711PacketThreshold 80
   commit 
   set oam traps admin sonusSbxDspAvailG729OnSetCrossThresholdNotification state enabled
   commit
   set oam traps admin sonusSbxDspAvailG729AbateCrossThresholdNotification state enabled
   commit
   set oam traps admin sonusSbxDspAvailG726AbateCrossThresholdNotification state enabled
   commit
   set oam traps admin sonusSbxDspAvailG726OnSetCrossThresholdNotification state enabled
   commit
   set oam traps admin sonusSbxDspAvailG711OnSetCrossThresholdNotification state enabled
   commit
   set oam traps admin sonusSbxDspAvailG711PacketAbateCrossThresholdNotification state enabled
   commit
2. Configure the trap target: set oam snmp trapTarget EMS160 ipAddress 10.xxx.xx.xxx port 162 trapType v2 state enabled commit.
3. Limit the compression resources to make the issue readily occur: set system mediaProfile tone 98 compression 2 commit.
4. Perform a bunch of transcoded (e.g. G711 to G729) calls that exceed threshold limits, and see onset traps are sent to the trap target.
5. Clear the transcoded calls, and see abate traps are sent to the trap target.

The code is modified to support g711PacketThreshold, g729Threshold, and g726Threshold values.

Deprecated the support for allThreshold, as it was never implemented in the SBC.

Workaround: None.

Calls are dropping when placed on hold through MS Teams.

Impact: When MS teams puts a call on hold and then attempts to retrieve it while the microphone is disabled, the action results in the call being cleared.

Root Cause: MS teams performs the call retrieve by sending an INVITE with a replaces message to the SBC and when the microphone is disabled this message includes a=recvonly.

The SBC send 200 Ok response to the INVITE with replaces with a=sendrecv while teams is expecting a=sendonly that causes the call to be cleared by MS teams.

Steps to Replicate: 

1. Establish a PSTN to teams call through the SBC.
2. Place call on hold from teams.
3. Disable the microphone in browser running teams and then retrieve (place off hold) the call.
4. The call should retrieve should be successful.

The code is modified to respond to an INVITE with replaces containing a=recvonly, with 200 OK containing a=sendonly.

Workaround: None.

The STI sends 2 Identity fields in Contact header, but SBC only passes one

Impact: In the case where multiple "Identity Headers" are present as embedded headers in a 3xx "Contact Header", the SBC sends out only 1 Identity Header for the re-directed INVITE.

Root Cause: The SBC did not support handling of the scenario where SIP headers are repeated in embedded contact headers in 3xx. The SBC would end up the sending only the last of the repeated headers in the re-directed Invite.

Steps to Replicate: 

1. The SBC is configured with ERE, set the config required for handling embedded headers in 3xx as shown below.
   "set profiles signaling ipSignalingProfile DEFAULT_SIP egressIpAttributes redirect flags forceRequeryForRedirection enable honorEmbeddedHeadersIn3xx enable"
2. Run a UAS script with 302 redirection and add multiple "Identity headers"
   as shown below.
   Ex- Contact: 9710622482<sip:9710622482@10.xx.xx.xx:xxxx?Identity=test1&Identity=test2&Identity=test3>
3. Run a 2nd UAS Script to handle the Redirected INVITE.
4. Run SIPP call, trigger UAC script.
   
   Expected Result: The redirect INVITE shall contain both the Identity headers received in 3xx as embedded contact header.

The code is modified to support repeated SIP headers.

Workaround: None.

A customer SBC upgrade had a spiltbrain SBC boot issue

Impact: One of the customer SBCs in N:1 mode is not coming up after upgrade.

Root Cause: The customer SBC node went for multiple reboots due to config profile change and split brains after upgrade exceeding max limit for reboot.

Steps to Replicate: Upgrade all customer SBC nodes in one shot so that they all come up together.

The code is modified to avoid split brain due to nodeId/serviceId collision by informing the peer nodes about the self node's nodeId/serviceid as soon as its allocated.

Workaround: Restart the customer SBC application manually on the node which has exceeded max reboot limit.

The BYE message was sent to the wrong port.

Impact: When endpoint is registered over TLS initiates a calls and after call is established, endpoint sends refresh register from modified port and also sends refresh INVITE with no SDP change, the SBC does not send BYE to modified port upon call disconnect.

Root Cause: The SBC does not update remote connection address when refresh INVITE is received from different port.

Steps to Replicate: 

1. Register endpoint over TLS.
2. Endpoint initiates a call, call gets connected.
3. Endpoint sends refresh register from modified port.
4. Endpoint sends refresh re-INVITE from modified port.
5. Called party disconnects the call.
6. Verify SBC sends BYE to modified port towards caller.

The code is modified to ensure the SBC update remote address when the refresh INVITE is received from modified port.

Workaround: None.

PreInstall Check improvements for file permissions.

Impact: PreUpgrade checks failure due to permission issues on external directory.

Root Cause: Pre-checks script failed to run commands on peer due to permission issues.

Steps to Replicate: Perform upgrade to the fix version and verify that upgrade is successful.

The code is modified to ensure permissions/ownership are set properly before running through pre-checks/upgrade.

Workaround: Set the right ownership for external directory as:
chgrp -R upload /opt/sonus/external/

The IAC:terraform apply output is displaying only mgt Active and Standby IP's. I

Impact: When orchestrating a HFE 2.0 or HFE 2.1 setup in Azure using Ribbon RAF modules, the Terraform output does not show any information about HFE node(s) created, even thought the HFE nodes are started correctly.

Root Cause: Terraform was missing the configuration to tell it to output the information about the HFE node(s).

Steps to Replicate: Create a HFE 2.0 or HFE 2.1 setup using Ribbon RAF modules.

The code is modified so that Terraform outputs the management IP and instance name for each HFE node when the orchestration is successful.

Workaround: None.

There are call failures due connection toggling between the SBC and SLB.

Impact: The SBC's can lose connectivity to the HFE nodes when there is high amounts of traffic on the network in Azure

Root Cause: The health check timeout from the HFE to SBC is too small, therefore was not receiving the replies back from the SBC, causing HFE to switchover

Steps to Replicate:

Increased the timeout from 20ms to 100ms

Workaround: Manually edit the following line in HFE_AZ.sh from:
$FPING -c 3 -t 20-p 200 ${ACTIVE_SBC_IP_ARR[0]} &> /dev/null
to:
$FPING -c 3 -t 100 -p 200 ${ACTIVE_SBC_IP_ARR[0]} &> /dev/null

The IaC is creating same NSG for all the network interfaces created. Recommended SGs should be created for each interface.

Impact: IaC is creating single NSG for all the 4 sbc interface.

Root Cause: Iac is creating one NSG and using same for all sbc interfaces created.

Steps to Replicate: Provided option to create different NSG for different interface created for the SBC.

Option available in terraform.vars Ex:
sbc_security_group_names = ["TF-SVT-SG-MGT0", "TF-SVT-SG-HA", "TF-SVT-SG-PKT0", "TF-SVT-SG-PKT1"]

The code is modified to create different NSG for different interface of the SBC.

Workaround: None

The Active SLB failed in the LCA to coming up "getInterfaceInfoFromMDS: Could not get the interface details in the metadata information!"

Impact: If the Azure fails on the request when verifying the authorization of the managed identity, then the cloud-init immediately fails.

Root Cause: The cloud-init is missing tolerance when testing the authorization.

Steps to Replicate: TBD

Add retries if the connection is reset or times out to address the issue.

Workaround: Reboot the instance after cloud-init has failed.

The LeakSanitizer detected memory leaks on various processes for a t140 Call.

Impact: Small memory leak while configuring SNMP trap targets.

Root Cause: While processing the configuration requests the SBC code was reading content from CDB into local memory blocks but failed to release the memory blocks at the end of the configuration action.

Steps to Replicate: Configure the SNMP trap targets.

The code is modified to correctly free the internal memory blocks used to hold the temporary CDB configuration data.

Workaround: None

The PES Process cored while testing Flex AD support with the ERE (SBX-72926).

Impact: PES process dumps core while executing AD service.

Root Cause: While executing the AD service, PES fetches the AD profile from cache. Since there is no AD profile was configured in this case, cache shall return NULL. The code was missing to check for NULL for the AD profile pointer.

Steps to Replicate: Configure a AD number translation criteria with ingress TG as the trigger criteria type. Make a call on the ingress TG. While executing the AD service, PES process will core dump.

The code is modified to check for a NULL for AD profile and AD attribute profile. If there is no profiles configured, the control returns and the service is marked as failed.

Workaround: Configure an AD Profile and AD Attribute profile so that the existing code can be executed further.

The SCM Process cored while testing SBX H323 Conformance and Interworking.

Impact: The SCM Process cored while running SIP-H323 interworking call to verify rfc2833 to inband DTMF.

Root Cause: Accessing of a NULL pointer caused the core dump.

Steps to Replicate: Run a SIP-H323 interworking call with SIP side RFC 2833 and H.323 side inband DTMF being enabled.

The code is modified to check for NULL pointer before accessing it.

Workaround: None.

Observed a PRS Process core followed by a "DsPr, Ssre, Cpx" core on both active and standby box, for the Openstack S-SBC HA pair and "Ccsp" process core on the TSBC active while running calls on the 2CPS with the ASAN build.

Impact: There was a memory leak detected by the ASAN in the active S-SBC while running the G.711 to G.729 transcoding load on the DSBC.

Root Cause: A code analysis of the ASAN reported function revealed that in some corner scenarios, the SBC may end up leaking the memory related to the place holder used to store the remote DSP resource.

Steps to Replicate: Re-run the same load in ASAN and ensure no memory leaks are detected

The code is modified to free this memory.

Workaround: None.

The SCM Process core dump was observed when REGISTER is received and useRandomUserInfoInContactHdr is enabled.

Impact: Invalid memory access issue when both embeddedRegInfoinUserPart and useRandomUserInfoInContactHdr flag were enabled

Root Cause: In the SBC, the appropriate memory was not allocated for contactUrl puchUsername in case both embeddedRegInfoinUserPart and useRandomUserInfoInContactHdr flag were enabled, due to this invalid memory access was happening.

Steps to Replicate:

1. On SLB - Enable remoteDeviceType to access
2. On SBCs enable the embeddedRegInfoinUserPart flag as:-
   set addressContext <ADDRESS_CONTEXT> zone <ZONE> sipTrunkGroup <TG> signaling embeddedRegInfoinUserPart enabled
3. On SBCs enabled useRandomUserinfoInContactHeader as :-
   set addressContext <ADDRESS_CONTEXT> zone <ZONE> sipTrunkGroup <TG> signaling useRandomUserInfoInContactHdr enabled

The code is modified to allocate appropriate memory for puchUsername based on embeddedRegInfoinUserPart or useRandomUserInfoInContactHdr enable case.

Workaround: As per the SBC design, any of the flags below should be enable not both.
embeddedRegInfoinUserPart or useRandomUserInfoInContactHdr flag

PortFix SBX-98177: The SBC 7000 TCP window size.

Impact: Add CLI support to set the kernel parameter net.ipv4.tcp_window_scaling.

Root Cause: In certain SBC deployment scenarios (a large number of devices running SIP-TLS), the tcp_window_scaling caused small TCP window size.
In such cases, the customer had to use Linux command to disable tcp_window_scaling on each SBC node.

Steps to Replicate: 

1. Show initial setting for tcp_window_scaling on both active (SBCSWE01a) and standby (SBCSWE01b)
   [root@SBCSWE01a ~]# cat /proc/sys/net/ipv4/tcp_window_scaling=1
   [root@SBCSWE01b ~]# cat /proc/sys/net/ipv4/tcp_window_scaling=1
   
   
2. Use CLI to change the kernel param value and display the changed value on both active and standby.
   admin@SBCSWE01a% set system admin SBCSWE01 kernelParams tcpWindowScaling disable
   admin@SBCSWE01a% commit
   Commit complete.
   
   [root@SBCSWE01a ~]# cat /proc/sys/net/ipv4/tcp_window_scaling=0
   
   [root@SBCSWE01b ~]# cat /proc/sys/net/ipv4/tcp_window_scaling=0

The code is modified for setting the kernel parameter net.ipv4.tcp_window_scaling. The Linux command is no longer needed.

Workaround: Use Linux command to set the kernel parameter on both active and standby.
[root@SBCSWE01a ~]# sysctl net.ipv4.tcp_window_scaling=1
[root@SBCSWE01a ~]# sysctl -w net.ipv4.tcp_window_scaling=0
[root@SBCSWE01b ~]# sysctl net.ipv4.tcp_window_scaling=1
[root@SBCSWE01b ~]# sysctl -w net.ipv4.tcp_window_scaling=0

To be persistent across SBC restart and system reboot, use Linux the following shell command on both active and standby.
[root@SBCSWE01a ~]# echo "net.ipv4.tcp_window_scaling=0" >> /etc/sysctl.conf
[root@SBCSWE01b ~]# echo "net.ipv4.tcp_window_scaling=0" >> /etc/sysctl.conf

Portfix SBX-105351: The 823R0 SCM Process cored.

Impact:The SCM cores after a long extensive load of the OOD relay.

Root Cause: The SBC accesses an invalid pointer of link list when the cleanup relay control block.

Steps to Replicate:System required HW_TYPE_SBS5000 and max support Relay per SCM is 4194304.

Run the OOD relay load (Notify). After a long period time (at least 4m relay per SCM).

The code is modified to properly initialize the link list.

Workaround: None.

Portfix SBX-105019: A crosstalk issue seen on the SBC 5400.

Impact: When there is a transcoded call with Opus codec and Opus termination does not send any packets to the SBC, then other termination of opus call, may hear audio from another unrelated opus transcoded call or it may hear white noise.

This occurs on SBC5x10/5400/7K but not on SWe platforms. This problem only happens if no opus packet is received. When the decoder receives first opus packet, subsequently problem does not happen for that call.

Root Cause: Initially, when no packet is received, Opus decoder is not called. As a result, the decoder's output buffer is uninitialized and it happened to be stale buffer of another opus call or a leftover buffer of a previous Opus call on that DSP. That buffer is used to encode data for other leg of the Opus call and that results in cross talk audio or distorted noise to other termination of Opus call.

Steps to Replicate: 

1. Make two sipp Opus<=>g711 calls, say call 1 and call 2 on SBC5x10/5400/7K
2. These two calls have to land on same DSP core. So issue a unhide debug command 'request sbx drm debug command "loadbalance disable" before making two calls. Without this command, both calls may not land on same DSP core and you may not see same problem.
3. Call 1 sends audio packet (.pcap) from Opus termination but call 2 does not send any packets to the SBC from it's Opus termination.
4. Call 2's g711 termination will hear audio from call 1's opus termination.

The fix is to clear the buffer (with silence) when the Opus decoder is not primed with any packet.

Workaround: 

1. Use a different codec other than Opus.
2. Use SWe platform instead of h/w SBC5x10/5400/7000.

Portfix SBX-102122: SAM Process core dumps with v06.02.01 F012.

Impact: The SAM and PRS process cored in the Standby node.

Root Cause: The DRBD split brain lead to DRBD GI reset. This leads to a DRBD full sync, increasing the i/o congestion and ultimately causing a healthcheck timeout.

Steps to Replicate: To test, run "drbdadm disconnect mirror". Once we get "standby split-brain" log in DBG logs, run "drbdadm disconnect mirror" again so that DRBD does not connect automatically and we get into the block which we need to test. Run "drbd-overview" and check if drbd does not went for full sync.

The code is modified to perform the DRBD checks after DRBD split-brain recovery.

Workaround: None.

PortFix SBX-102625: The SIP calls stop in receiving 183 (Precondition).

Impact: Call stopped working after 183 received without SDP.

Root Cause: As part of egress precondition interworking, to negotiate the SBC should send an UPDATE to the egress endpoint. When 183 received without SDP, SBC should not try to send UPDATE. SIPSG call state assumed UPDATE is sent. Therefore, when the next 183 is received it tried to queue the 183 assuming UPDATE is in progress.

Steps to Replicate: As mentioned in the JIRA

The code is modified to avoid this state transition when 183 received without SDP.

Workaround: Remove the 183 without a SDP message using SMM.

PortFix SBX-102837: The fra-tdg-sonus-01 SipSignalingPorts became OutOfService after a switchover.

Impact:The SIP sigPorts stuck in the OOS after a switchover.

Root Cause: The customer had the network/pkt port issue, on one of their HA node, which caused pkt port(s) to bounce randomly, i.e. pkt port went DOWN and came back UP within 2 to 3 seconds. So they have been keeping that node as standby node. They also have link detection enabled for pkt port(s) and have around 100 LIFs per pkt port, one per SIP SIGPORT. When the pkt port went down, NRS delays the port down event processing for 2 second to allow link failure detection to be ready and also to avoid the race condition between NRS and LVM. When a 2 second delay timer is up, the NRS starts to take down affected LIFs and notifies local SIPCM and SIPFE so they can take down affected SIP SIGPORTs.

In SIPCM, all the sockets on the affected sigPort are put in a delete pending table and starts a 1 tick timer. Then the socket(s) is being deleted after 1 tick timer is up.
When pkt port came back up, NRS processes the event with no delay and notifies SIPCM and SIPFE as well. Since there are around 100 LIFs, there were many messages exchanges between NRS/XRM/SIPCM/SIPFE. NRS LIF FSM has the mechanism in place to handle the timing issue and LIFs were all back in service. But SIPCM failed to activate some SIP SIGPORT(s) while binding the socket(s). These error messages indicated that SIPCM tried to activate the SIGPORT while it was still pending delete. Therefore SIGPORT got stuck in OOS(broken state) in both SIPCM and SIPFE on standby node. If there was a switchover happened later, user would then noticed one or more SIGPORTs were OOS. They have to manually bounce those SIGPORTs to bring them back in service.

Steps to Replicate: The nature of the problem was the timing caused race condition. There is no good way to re-create/verify the fixes.

Suggest to run regular SIP related regression tests.

The code is modified to:

1. Introduce a new 1 tick timer in SIPCM_DATA_STR, activateRetryTimerId.
2. The deletePendingSocketTable and if the sigPort is found in the table, then start the 1 tick timer. When the timer is up, SipCmActivateCallSigPort() is invoked again. Once the sigPort is activated successfully, SIPCM notifies SIPFE as usual.

Workaround: No workaround.

PortFix SBX-101161: A memory leak was observed in the SAM process.

Impact: A memory leak was observed in the SAM process.

Root Cause: There is race condition in the code that handles CALL_AUDITs and CALL_CLEANUPs that can cause a memory leak.

If the response to a CALL_AUDIT takes to long, the code that handles the "late" response has allocates memory for a structure that may never get freed.

Steps to Replicate: This issue is caused by a race condition that cannot be forced - therefore we cannot specify steps to reproduce this issue.

The code is modified to add a timer every time a fault structure is allocated. When the timer expires, if the the structure still exists it will be freed.

Workaround: There is no workaround.

PortFix SBX-104802: Enabling the Dialog Transparency causes calls to fail.

Impact: The call fails for the dialog transparency and forking.

Root Cause: When the ingress support PRACK and multiple 18x fork, the SBC fails to send 200OK to ingress.

Steps to Replicate: Configure dialog transparency and downstream forking on egress.
Ingress support PRACK, egress not support PRACK.

The Egress answers 180fork1, 180fork2, 180fork3, and 200OKfork2 simultaneously.

The code is modified so the SBC resets the PRACK pending status properly, causing 200OK stuck in the queue.

Workaround: Disable the dialog transparency.

PortFix SBX-104761: A SM Process coredump occurred on the server.

Impact: The SM Process crashed while executing the "show table system syncStatus" command.

Root Cause: The shell script used to get the oracle sync status - PolicyDBSyncStatus.sh - did not return within 10 seconds, causing a healthcheck timeout that caused the coredump.

Steps to Replicate: This problem is not reproducible.

The code is modified to disable heathchecks while fetching the syncStatus.

Workaround: There is no workaround.

PortFix SBX-103553: The SBC does not proceed multiple 183.

Impact: The GSX-GW-SBX call erroneously queues 183 message at SBX when X-Service-Type precondition is received and downstream forking enabled.

Root Cause: An interaction between downstream forking and GW-GW code.

Steps to Replicate: Make GSX -> SBX SIP-GW-SIP call.
Egress trunk group has X-Headers supported.
Egress trunk group has downstream forking enabled.

INVITE -->
<-- 183 with X-Service-Type: cf,precondition and SDP
<-- 183 with X Service-Type: cf,precondition and different SDP
<-- 180 with X Service-Type: cf and SDP
<-- 183 with X Service-Type: cf and no SDP
This final 183 is queued.

The code is modified to not perform the queueing in GW-GW scenarios.

Workaround: None.

PortFix SBX-104443: The SM Process and Cpx cores prevented fra-tdg-sonus-01-1 to come back up as standby after switchover

Impact: The SM Process and Cpx processes cored on slot 1 after LDG triggered switchover to slot 2.

The SM Process was deadlocked while retrieving peer NTP status.

The Cpx cores happened after SM Process coredump, when writing SMM profiles into shared memory.

Root Cause: The root cause was SM Process deadlock.

Steps to Replicate: This is time sensitive issue. The steps cannot be re-produced or verified.

Suggest to run full regression test with switchovers.

The code is modified to release smNtpLock_ after issuing a NTP restart command because we certainly do not need the lock while doing the restart.

Workaround: No workaround.

PortFix SBX-103254: The call setup delay with the Option Ping 40+ endpoints SBC SWe.

Impact: The configuration support to disable Path MTU Discovery by setting kernel parameter net.ipv4.ip_no_pmtu_disc (ipNoPmtuDisc). When ipNoPmtuDisc is to 2, the DF bit in IP packet header will be set 0.

Root Cause: The secure network does not support Path MTU Discovery per RFC-1191.

Steps to Replicate: Testing on a Standalone SBC SWe:

1. Start packet capture, and make SIP call (SIP-TLS). Verify Path MTU Discovery is enabled by checking outgoing IP packet's DF bit is 1.
- Test to show the issue
[root@SBCSWE03 ~]# date; tshark -t a -i pkt0 -f "ip host 10.x.xx.xx" -w t_tshark_pkt0_10.x.xx.xx_1127_pmtu_1027_01.pcap
Tue Oct 27 16:21:11 EDT 2020
Capturing on 'pkt0'
35 ^C
[root@SBCSWE03 ~]# tshark -V -r t_tshark_pkt0_10.7.20.29_1127_pmtu_1027_01.pcap | more

Internet Protocol Version 4, Src: 10.x.xx.xx (10.x.xx.xx), Dst: 10.xxx.xx.xxx (10.xxx.xx.xxx)
Version: 4
Header Length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport))
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00)
Total Length: 60
Identification: 0x0000 (0)
Flags: 0x02 (Don't Fragment)
0... .... = Reserved bit: Not set
.1.. .... = Don't fragment: Set
..0. .... = More fragments: Not set
Fragment offset: 0

[root@SBCSWE03 ~]#
--> The DF bit was set to 1.

2. Disable Path MTU Discovery.
admin@SBCSWE03% set system admin sbcswe03 kernelParams ipNoPmtuDisc ?
Description:
Configure /proc/sys/net/ipv4/ip_no_pmtu_disc (0 (default; disable); 1,2,3 (enabled modes))
1: when frag-required ICMP is received, PMTU to this destination is set to min_pmut 552.
2: implicitly setting IP_PMTUDISC_DONT on every created socket - outgoing IP packet DF is set to 0.
3: hardened pmtu discover mode. Please refer to Linux kernel document.

Possible completions:
<int, 0 .. 3>[0]
admin@SBCSWE03% set system admin sbcswe03 kernelParams ipNoPmtuDisc 2
admin@SBCSWE03% commit
Commit complete.
[ok][2020-10-27 16:36:27]

3. Restart the SBC. (If using HA, restart both nodes)
[root@SBCSWE03 ~]# date; sbxrestart
Tue Oct 27 16:36:56 EDT 2020

4. Start packet capture, and make SIP call (SIP-TLS). Verify Path MTU Discovery is disabled by checking outgoing IP packet's DF bit is 0.
[root@SBCSWE03 ~]# date; tshark -t a -i pkt0 -f "ip host 10.x.xx.xx" -w t_tshark_pkt0_10.x.xx.xx_1127_noPmtu_1027_02.pcap
Tue Oct 27 16:42:32 EDT 2020
Capturing on 'pkt0'
26 ^C
[root@SBCSWE03 ~]# tshark -V -r t_tshark_pkt0_10.x.xx.xx_1127_noPmtu_1027_02.pcap | more
...
Frame 2: 74 bytes on wire (592 bits), 74 bytes captured (592 bits) on interface 0
...
Internet Protocol Version 4, Src: 10.x.xx.xx (10.x.xx.xx), Dst: 10.xxx.xx.xxx (10.xxx.xx.xxx)
Version: 4
Header Length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport))
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00)
Total Length: 60
Identification: 0xe881 (59521)
Flags: 0x00
0... .... = Reserved bit: Not set
.0.. .... = Don't fragment: Not set
..0. .... = More fragments: Not set
Fragment offset: 0
...

[root@SBCSWE03 ~]#
--> The DF bit was set to 0, and Path MTU Discovery is disabled.

Disable the Path MTU Discovery by setting kernel parameter net.ipv4.ip_no_pmtu_disc to 2. When set to 2, the DF bit will be set to 0 for the transmitted packets on every created socket. The configuration parameter is "system admin <name of system> kernelParams ipNoPmtuDisc".

Workaround: The workaround is a hack. It can be used one time test only. The hack will not survive Linux reboots.

1. Make changes on the current Standby (assuming SBC1 is currently active, and SBC2 is currently standby)
   [root@SBC2 ~]# date; cat /proc/sys/net/ipv4/ip_no_pmtu_disc
   [root@SBC2 ~]# date; echo “2” > cat /proc/sys/net/ipv4/ip_no_pmtu_disc
   [root@SBC2 ~]# date; sbxrestart
2. Wait until SBC2 is up and synched. Ensure the p_no_pmtu_disc is set to 2.
   [root@SBC2 ~]# date; sbxstatus | tail -4
   [root@SBC2 ~]# cat /proc/sys/net/ipv4/ip_no_pmtu_disc 2
   [root@SBC2 ~]#
3. Make changes on the current Active.
   [root@SBC1 ~]# date; cat /proc/sys/net/ipv4/ip_no_pmtu_disc
   [root@SBC1 ~]# date; echo “2” > cat /proc/sys/net/ipv4/ip_no_pmtu_disc
   [root@SBC1 ~]# date; sbxrestart
4. Wait until the SBC1 is up and synched. Ensure the p_no_pmtu_disc is set to 2.
   [root@SBC1 ~]# date; sbxstatus | tail -4
   [root@SBC1 ~]# cat /proc/sys/net/ipv4/ip_no_pmtu_disc 2
   [root@SBC1 ~]#
5. May perform a switchover to make the SBC1 to be active.
6. Run your tests.

PortFix SBX-95982: The SBC was running version 6.2.2 and cannot capture media with MCT.

Impact: The MCT reload of configuration (sbx restart or switchover) will not succeed.

Root Cause: Missing small logic during the restart in order to properly reload the MCT configuration.

Steps to Replicate: Retest MCT reload and switchover scenarios.

The code is modified to properly reload the MCT configuration.

Workaround: User can delete the MCT config and reconfigure it after any restart/switchover.

PortFix SBX-103948: The SBC Application is crashing when CPaaS to PSTN call is made.

Impact: The SBC cores after a reboot/upgrade.

Root Cause: The issue was introduced when the SBC try to support 10000 SMM rules in 8.2.0.

Steps to Replicate: Configure the customer rules (delete SDP line). Major logs trigger but the rule still success.

After a reboot/upgrade, the core occurs.

The code is modified to address the issue. 

Workaround: Disable the delete SDP rule.

PortFix SBX-105156: The SBC was sending m=application 0 UDP/BFCP (null).

Impact: The SBC generates the syntax error m line UDP/BFCP in 183.

Root Cause: The SBC accesses the initialize data when format m line for UDP/BFCP

Steps to Replicate: Incoming call with m-line UDP/BFCP. ERE/PSX using script NO_ROUTE and trigger announcement. The internal 183 has m-line UDP/BFCP with an invalid syntax.

Make the m-line initialize properly to address the issue.

Workaround: Use the SMM to correct m-line syntax error.

PortFix SBX-96239: The display-name in the Diversion header or History-info header is deleted during the interworking.

Impact: The SBC does not send display name present in ingress INVITE's diversion header in history info header in egress INVITE.

The SBC does not send a display name present in ingress INVITE's history info header in egress INVITE's diversion header.

Root Cause: The SBC does not consider display name when interworking between diversion header and history info header.

Steps to Replicate: Ensure the display name is sent in history info header of egress INVITE when the SBC is converting diversion header to history info header.

Also, the SBC should send the display name in diversion header of egress INVITE when the SBC is converting history info to diversion header.

The code is modified to ensure the display name is sent when the SBC is interworking history info and diversion header.

Workaround: None.

PortFix SBX-104225: Both servers are registered but offline, and the SBCs cored after routing the changes from the customer.

Impact: The S-SBC Core dump with PEM enabled call-flow.

Root Cause: With PEM enabled, M-SBC performs NAPT learning for 1st RTP Packets. Upon receiving the Napt Indication for egress-leg from the M-SBC, the NRMA on the S-SBC performs flowChange/modification on both ingress and egress leg.

As part of this flowChange/modification in ingress-leg, the ingress is NULL, resulting in coredump.

The cktInfo on ingress should never be NULL, unless the call is being teared down.

In this case, the ingress-leg has received a CANCEL, resulting in the call being taken down, as part of this call tear-down, ingress-leg cktInfo is reset to NULL, and then the de-alloc indication for ingress-leg is sent to the M-SBC, while the S-SBC is waiting for response of this tear-down, it processes the NAPT indication on egress-leg from the M-SBC. This results in the NULL cktInfo pointer getting accesses for ingress-leg.

Steps to Replicate: 

1. Setup a D-SBC transcoded call with MRF.
2. Enable PEM on egress.
3. Ensure PRACK is enabled.
4. Ensure 180 without SDP, followed by 183 with SDP is received from UAS.
5. Ensure CANCEL is received from UAC, while 200 OK from MRF is received, such that the timing of RTP NAPT learning processing and CANCEL processing match.
   
   NOTE: This is a very tricky issue to recreate, as the timing may vary every time we attempt, it is a matter is few milliseconds.

The code is modified to address the issue.

Workaround: Disable NAPT or PEM. This workaround will ensure that we do not get to the race condition.

Portfix SBX-104561: The 8.1R5 customer SBC OAMs reaches 90% memory util and then restarts.

Impact: The memory leak was observed on the OAM node.

Root Cause: 

• The XRM and MRM are sending messages through the ICM to NRM.
• The NRM is not present/running on OAM.
• Since these are non-discardable messages, they will queue forever until we run out of memory.

Steps to Replicate: 

1. Deploy the fix.
2. Monitor memory usage of PRS process.
3. Validate that PRS memory is not increasing.

The code is modified so that these messages are not sent on OAM node.

Workaround: Before this fix, the workaround is to restart OAM app to recover the memory.

PortFix SBX-104484: The SBC is rejecting the second Update message from the Egress as 500 result into call failure.

Impact: The SBC rejects 2nd Update from egress due to DLRB feature.

Root Cause: The first Update SIPS response locally 200OK but not clear the server request message. As result the second Update, trigger the SIPS answer 500.

Steps to Replicate: Configure the DLRB, Egress response 183 without the SDP, first Update, and the second Update.

The code is modified so after a 200OK response, the SIPS clears the server request so it can handle the subsequent one.

Workaround: Disable the DLRB.

PortFix SBX-104146: The DNS Process observed a coredump when responding with a RCODE 1 error from the DNS server and deleting a DNS Group in the SBC.

Impact: The DNS Process dumps core after getting RCODE errors 1,2,4 for a EDNS query and the DNS Group is deleted immediately on the SBC in the monitoring Interval specified in the "ednsRetryAfter" configuration has values ranging from 60 to 180 seconds with 180 being default.

Root Cause: The EDNS failure timer callback is invoked, upon encountering an EDNS query failure based on the "ednsRetryAfter" configuration. The timer call back function is missing NULL check for the DNS Group

Steps to Replicate: 

1. The SBC configured with EDNS support.
2. The DNS server is configured to respond with the RCODE 1/2/4 error for EDNS query.
3. Make a SIP Call that triggers the EDNS query.
4. Disable the DNS Server and delete the DNS group within 180 seconds.

The code is modified to cancel the EDNS-failure timer, whenever the DNS Group is deleted and the NULL pointer check is added for the timer callback for robustness.

Workaround: None.

Portfix SBX-103207: The SBC SWe duplicates the audio from call B to call A.

Impact: If the CN is not negotiated for G711 codec and remote peer still sends CN packets that match default CN payload type (13) then user on other end may hear cross talk audio of completely unrelated channel or hear own reflect audio.

Root Cause: When a g711 side does not negotiate CN in signaling, the DSP does not initialize Comfort Noise Generation object. However, if remote peer still sends a CN packet that matches the g711 SID payload type configured in PSP (default 13), DSP accepts the packet. It processes that CN packet incorrectly and as a result uses stale voice buffer that happens to be of another channel. This continues until next voice packet for that channel arrives. As a result, we see cross talk audio from the other call during silence period. In some cases, the user may hear own audio also and that is different manifestation of the same problem. This issue is specific to SWe.

Steps to Replicate: Follow the step described in problem statement of JIRA.

The code is modified to initialize the comfort noise object even though the CN is not negotiated and process CN packets correctly.

Workaround: 

Work around 1:

Change the default payload type of comfort noise from 13 to something else (say 15) in PSP of peer that is sending CN packets. This will make DSP drop CN packets because payload type will not match.

Work around 2:

Enable silence suppression on PSP of peer that is sending CN packets and keep CN payload type that matches with CN payload type.

PortFix SBX-102704: Unable to Add or Edit route with # in Destination National, CLI or EMA.

Impact: User is not able to store few special characters for the destination number field in the route entity.

Root Cause: The pattern defined for the destination number filed did not allow special characters.

Steps to Replicate: Create a route with special character like #123 for the destination number filed in the route entity.

The code is modified to allow only few specific special characters.

So, reverted back the pattern to same as before 8.1 release, so that no customers face similar issue again who have configured special characters prior to 8.1.

Workaround: None.

PortFix SBX-102290: The DBG file was filling up with messages "SIPCM: *ThreadPool: messageSequence".

Impact: The DBG logs can be overrun with "SIPCM: *ThreadPool: messageSequence" messages.

Root Cause: The DBG logs can be overrun with "SIPCM: *ThreadPool: messageSequence" messages, when Double CRLF "pings" are received by the SBC over UDP transport.

Steps to Replicate: Send Double CRLF "pings" over UDP to the SBC.

The code is modified to properly dispose of Double CRLF "pings" received by the SBC over UDP.

Workaround: Inhibit the transmission (or reception) of Double CRLF "pings" over UDP.

PortFix SBX-104934: The SBC generates the same ICID for more than 1 call, porting the fix for SBX-101887 into 8.2.3F1 did not help

Impact: Duplicate the ICID generated for two different calls.

Root Cause: The instance specific value is set to 0 every other call causing the ICID generation to always use 0 in all instances. This results in the same ICID generated in more than one SCM instance when the calls land within the same microsecond

Steps to Replicate: Configure the SBC to generate ICID on the egress side. Run call load and check for duplicate ICIDs.

The code is modified to manipulate the last octet of the MAC address used in ICID generation.

Workaround: None.

Portfix SBX-105417: During an S-SBC and M-SBC cyclic switchover, the S-SBC app did not come up after few switchovers.

Impact: The service did not come up after a switchover due to inconsistent encryptedStore.

Root Cause: The store became inconsistent after the oam_config got applied, the time taken during deletePeerEntries should also be reduced to allow a ChmClearEncryptedStoreOfPeers to finish faster during startup.

Steps to Replicate: Perform multiple switchovers and ensure that:

1. ChmClearEncryptedStoreOfPeers does not take too long during startup.
2. The deletePeerEntries runs in background and does not take too long either.
3. The service comes up, role is assigned successfully and store is consistent.

The code is modified so:

1. deletePeerEntries run in background
2. decryptStore validates if decryption was successful to ensure the store is not inconsistent/corrupt.

Workaround: None.

PortFix SBX-103629: The SIP registrations are failing, and the SBC reporting REGISTER_PARSE_ERROR and replies with SIP 500 when it receives retransmitted initial REGISTER (CSEQ 1) followed by REGISTER with Authorization header (CSEQ 1).

Impact: When the SBC recv rexmit of registration (cseq=1) after sending 401/407, the SBC relay to AS. At the same time, the IAD sends new registration(cseq=2) for authentication. It triggers race condition to the SBC and response 500.

Root Cause: The SBC should relay rexmit (cseq=1) registration to application server.

Steps to Replicate: The IAD send registration(cseq1) to AS, AS response 401 to IAD, IAD rexmit the same sceq1, and send a new sceq2 with auth header.

The code is modified to response rexmit request (cseq=1). So that a subsequent one cseq=2 can handle properly relay to AS.

Workaround: None.

The SBC 9.1 MD5 check sum.

Impact: The SBC qcow2 image is shipped along with a sha256 checksum but Openstack glance supports only md5 checksum. Unable to validate the image in an automated way.

Root Cause: Checksum was updated from md5 to sha256 as md5 is weaker compared to sha256 algorithm and there are reported attacks on md5.

Steps to Replicate: With the fix build, ensure qcow2.md5 is one of the build artifacts.

The code is modified to facilitate openstack glance image verification.

Workaround: User can manually create md5 checksum by running below command:
md5sum sbc.qcow2 > sbc.qcow2.md5

The SNMPv3 traps from the MRFP are not displayed in EMS Fault Management.

Impact: The traps were not getting displayed in EMS.

Root Cause: The SBC generates Engine ID at the time of ISO installation. Due to this, all cloud instances share same engine id from the qcow2 generation time.

When same engine id is returned by two SBCs in different cluster, the EMS is unable to differentiate between them.

Steps to Replicate: Launch instances from qcow2 in different clusters. Traps generated should be shown correctly now.

Engine ID is created at the time cloud instance launch now to address the issue.

Workaround: None.

Configure the import issue.

Impact: The fields that supports \r like regex fields, changes it after a configuration Import(XML based).

Root Cause: The \r is not xml encoded when parsing intermediate xml file.

Steps to Replicate: 

1. Create SMM Profile with regex that has \r export configuration as xml-tar clear DB.
2. Import the configuration.
3. Verify SMM profile regex field.

The code is modified to handle \r as a special case.

Workaround: None.

Observed a SCM Process coredump in the gcored while running the 30 SWOs on 7000 Platform.

Impact: In case of switchover and switchback while running load, we expect some stale entries in SIPSG and CC control blocks. Such entries are cleared on the audit, but while clearing the SIPSG is sending accounting message to CC.

This SIPSG was reconstructed, but in CC the index corresponding to this GCID was allocated to some other GCID.

When the CC receives such message, the mismatch occurs and it logs a crash in sensitive mode.

Root Cause: During load run with multiple switchovers, various call modules get reconstructed when standby transitions to active. Due to this some stale entries are expected. If one of these stale entries sends some message (e.g accounting message) and if index for that call was allocated to another call, this error can happen. In case of such mismatch the SBC logs a crash in sensitive mode.

Steps to Replicate: Load run with multiple switchovers.

If the GCID sent from SIPSG module in accounting message is different from the GCID received in CC module, for switchover case do not do a sensitive crash. Instead add a major log to address the issue.

Workaround: Run load scenarios in normal mode instead of sensitive mode.

The emssftp fails in the 9.1R1.

Impact: The emssftp login was failing.

Root Cause: The sequence of asking for a password and keep credentials in sync caused the emssftp password failed.

Steps to Replicate: Checking login, after multiple reboots, rebuild with 1:1 and N: 1, along with 1-2 EMS registered.

The code is modified on the mode(HA vs standalone), included a new script to handle password request and updated for some HA cases.

Workaround: None.

Portfix SBX-104962: The HFE setInterfaceMap() can be incorrect.

Impact: The HFE_AZ.sh script can configure the HFE node incorrectly if the IP of an interface is a contained with in another.

Root Cause: The logic used to determine which interface associates while the NIC can find the incorrect interface.

Steps to Replicate: Create a HFE where the eth0 address contains either eth1 or eth2 IP address.

The code is modified to verify it the only the correct IP address is used.

Workaround: Request an updated HFE_AZ.sh script from Ribbon. Replace the HFE_AZ.sh in the storage account and reboot the HFE node(s).

Portfix SBX-104156: The RTT-TTY support verifies the call flow in the SBC.

Impact: Lower case characters from t140 packet were not getting converted correctly to Baudot (tty). Also, some other characters were not getting converted as per V.18 A2 table.

Root Cause: The translation table from T140 to Baudot(TTY) only accounted for the subset of characters that were present in the Baudot (TTY) character set.

Steps to Replicate:

1. Setup a T140=>Baudot (AMRNB=>G711) call.
2. Create t140 pcap file with all UTF8 characters (128) and validate the baudot generation.
3. Create a t140 pcap file with multi-byte sequence and validate the baudot.

The code is modified to:

• The t140=>Baudot translation table as per V.18 Annx2. 
• Validate multi-byte (2,3 and 4 bytes).
• Handle the Byte Order Mark (BOM).
• Ignore an invalid byte sequence and substitute the valid but unknown multi-byte seq with an apostrophe.

Workaround: None. Instead of lower case letters use upper case letters.

PortFix SBX-96783: Some counts in the callCurrentStatistics keep incrementing.

Impact: The "activeRegs" counter provided by the CLI zone callCurrentStatistics/ callIntervalStatistics command may continuously increase.

Root Cause: Badly formed SIP REGISTER messages received by the SBC
(i.e., a REGISTER message that the SIP parser determines is bad)
will increment the "activeRegs" counter and never decrement it.

Steps to Replicate: Send bad SIP REGISTER message(s) to the SBC, and see that the
"activeRegs" counter provided by the CLI zone callCurrentStatistics / callIntervalStatistics command increases (and does not decrease).

Decrement the "activeRegs" counter when the SIP REGISTER message fails the SIP parser to address the issue.

Workaround: None.

Portfix SBX-102364: The SBC behavior is inconsistent with the isfocus parameter handling.

Impact: 

Issue 1: The SBC is not transparently passing the complete Contact header in 200 OK of SUBSCRIBE when there is a isfocus parameter.

Issue 2: The SBC is not adding the Record-Route in any message when doing a full Contact header transparency

Root Cause: 

Issue 1: The SBC passes contact header transparently only when the isfocus is present in request as well as response and would not send the contact header transparently only when the contact header in response has isfocus parameter.

Issue 2: The code to add a record route for notify and its response is not present.

Steps to Replicate: 

1. Configure the SBC for A to B call.
2. Enable the flag contactTransparencyForIsFocusMediaTag on both TGs.
3. Make a SUBSCRIBE-NOTIFY call flow.

Issue 1: The code is modified to check if the service bit is not set (for the response) and check if the contact header has the isfocus parameter, if the parameter is present, set:

SIP_SERVICE_TYPE_CONTACT_TRANSPARENCY_
    FOR_ISFOCUS_PARAM

for a single contact scenario.

Issue 2: The code is modified to add the record route for notify and its response if isfocus parameter is present.

Workaround: None

Portfix SBX-103988: The ICE not working after upgrade to V08.02.03A950.

Impact: When a call with ICE changes from direct media to passthru, ICE learning does not get enabled on the call.

Root Cause: The code was not re-enabling the ICE FSM to start ICE learning when changing from direct media to passthru.

Steps to Replicate: 

Test 1
--------

1. Establish a call with ICE on ingress and xdmi on egress.
2. Send re-invite from egress without xdmi and complete the re-invite related signaling.
3. Send stuns to ingress media to complete ICE learning and verify call state changes to established and media can be exchanged between ingress and egress.

The code is modified to re-enable the ICE FSM when changing from direct media to passthru to allow the receipt and processing of stun messages to complete ICE learning.

Workaround: None.

The MGT0 cannot reach the GW.

Impact: The mgt0 interface appears to be inactive. The user cannot ping the gateway from the SBC 5400.

The ethtool -S mgt0 shows the tx_pause count rising rapidly.

Root Cause: This problem is specific to the SBX 5400 fix.

The SBC 5400 has a backpressure mechanism for management ports so that when the receive buffer exceeds a certain level, Design sends pause packets out to the interface to ask the switch/router to pause transmission momentarily.

The receive buffer count is incremented by a hardware module and decremented by software.
The notification was sent from Network Processor to application about sRTP Rollover Counter (ROC) reset is specifying the wrong interface so software decrements the wrong receive buffer count.

This causes the receive buffer count to wrap around, resulting in a high value. This triggers the back-pressure mechanism and Design continuously send out pause packets.

Steps to Replicate: 

1. Set up a pass-thru SRTP call.
2. The call should be a long duration call so that the Rollover Counter (ROC) in the sRTP packet becomes non-zero.
3. Modify the call so that the sRTP packet's the SSRC changes.

Update:

1. Do a long duration sRTP call (about 30 minutes.)
2. Hold the call.
3. Unhold the call.
4. As a result, this causes an SSRC change and triggers the problem.

Set the correct interface information in the notification to address the issue.

Workaround: Please contact Ribbon support for a script that will restore contact to the management interface.

Portfix SBX-103922: There was a PRS Process coredump on an active/A node in the middle of an upgrade.

Impact: The PRS Process cored during the LSWU while syncing ICE call data.

Root Cause: The ICE redundancy code is taking a long time to sync the call data with the standby during LSWU, causing a Healthcheck timeout and a core dump.

Steps to Replicate: The issue cannot be reproduced easily. 

The code is modified so the ICE syncs the call data to standby and addresses the issue. 

Workaround: None. 

PortFix SBX-104704: Duplicate the audio entries in the route PSP.

Impact: The duplicate codec entry is being sent in the policy response.

Root Cause: The function popPSPCodecEntryHlp that populates the codec entry was called twice, and as a result the codec entry was getting populated twice in the TLV.

Steps to Replicate: Add more then 4 codec entries in the ingress PSP and egress PSP. This will cause the issue to be reproduced.

The code is modified so that the duplicate codec entry is not passed on the d+ response.

Workaround: None.

PortFix SBX-101575: There was an offline upgrade failure from 8.2.1 to 8.2.2

Impact: An offline upgrade on a HA setup failed for the upgrade from 8.2.1 to 8.2.2.

Root Cause: An offline upgrade failed due to presence of a model update marker files that should not get created in case of offline upgrades.

Steps to Replicate: Perform an offline upgrade on a HA setup to the fix build and ensure upgrade is successful.

The code is modified to ensure nodeB is powered-off/shutdown (instead of the sbxstop) while upgrading a nodeA. This ensures that the model update related markers files do not get created.

Workaround: None.

PortFix SBX-104668: The SNMP PM collection is not working for newly added KPIs.

Impact: SNMP stats collection at EMS for G711_WITHOUT_XCODE_RES_PEAK_COUNT and G711_WITHOUT_XCODE_RES_AVG_COUNT of DspResDspUsageIntervalStats failed.

Root Cause: Some of the KPI stats fields are made obsolete as they are no longer supported. Due to this change EMS is not able to collect the stats through SNMP.

Steps to Replicate: Verify EMS SNMP walk should not fail when queried to the SBC.

1. Bring up SBC.
2. Register SBC to EMS.
3. Enable the sonusDspResDspUsageIntervalStatistics.

Expected Result:
The EMS fetches stats from the SBC.

The code is modified so the EMS is able to collect stats using SNMP.

Workaround: None.

PortFix SBX-103175: The large microflow profile was not getting enabled in the Custom Traffic Profile.

Impact: There are multiple issues:

1. For a default profile, max subs was always set to 256K.
2. For the custom and standard traffic profiles, the micro flow count in the NP resource was not proper.
3. Limited support of the 2M micro flows for standard profiles.

Root Cause:

1. The large micro flow support was not there for the custom traffic profiles.
2. For standard and default traffic profiles, there was restricted/incomplete support of large micro flow that is made generic as part of this issue.

Steps to Replicate:

1. Create an instance with 10GB memory and 16vpcu with the default traffic profile.Check microflow limits using "/opt/sonus/bin/cpsi -d summary" command.
2. Increase the memory to 50GB memory and check the microflow limits using "/opt/sonus/bin/cpsi -d summary" command.
3. Create a instance with 20GB memory and 16vpcu. Create and activate a custom traffic profile with access enabled in call mix. Check microflow limits using "/opt/sonus/bin/cpsi -d summary" command.
   
   

The code is modified to:

1. Introduce a new estimation parameter maxSubs based on the micro flow count is decided and the NP hugepages are reserved.
2. Enable the large micro flow support for all the standard/default profiles where mem > 48GB and vcpu_count >10.

Workaround: No workaround available. Need to use the build after the fix.

PortFix SBX-104347: The resource mem congestion level 3 is approaching threshold 90 sample 82.

Impact: The SIPCM is leaking a structure that is associated with the SIPMM functionality.

Root Cause: The code to free this memory under certain error/edge scenarios is missing.

Steps to Replicate: Since the exact call flow is not know that caused the customer to hit the error condition, no test steps can be used.

The code is modified to free the SIPMM related structure in error scenarios.

Workaround: Since the exact call flow is not know that caused the customer to hit the error condition, we cannot suggest a workaround.

PortFix SBX-102707: The network interfaces are not renamed as expected after an upgrade from 820R2 to 821R0.

Impact: Network interfaces are not properly mapped when the system is abruptly brought down after the first boot and rebooted.

Root Cause: Cloud-init service runs in parallel to sonusudev service and can cause issues with the network interface mapping as it tries to rename the interfaces.

Steps to Replicate: Reboot the system after sonusudev is run on first bootup.

The code is modified to start only after the sonusudev has run and mapped the interfaces properly.

Workaround: Reboot the instance again to recover from the issue.

PortFix SBX-99208: The SBC has the same issue as SBX-86420, this time for an outgoing REGISTER messages.

Impact: When the TLS port is not sent from the PSX, the SBC is not sending the DNS SRV query to the DNS server.

Root Cause: The root cause was the SBC just increments the port number by 1 sent by the PSX even if peer port was Zero.

Steps to Replicate: 

1. Configure the SBC and PSX to send a register and subscribe SIP request to relay.
2. Configure the serve FQDN and zero for the FQDN port.
3. Send a register/subscribe request.
4. The SBC should send SRV query to the DNS server.

If port number sent by PSX is zero then do not increment it by +1 by default for selecting TLS port to address the issue.

Workaround: None.

PortFix SBX-103561: The "tgrp" parameter are not passing transparently when the SIP in the core is enabled.

Impact: The SipCore calls is passing the wrong TGRP parameter value.

Root Cause: There was missing logic to support the SipCore.

Steps to Replicate: Configure the SipCore feature. Both the IPSP core and egress leg have "originating Trunk Group Options" set to "Include Tgrp with IP address"
Make a SipCore call.

The code is modified to support the SipCore feature.

Workaround: None.

PortFix SBX-103852: The SBC sent an unnecessary 481 for PRACK when the CANCEL is received.

Impact: The SBC sends unnecessary 481 for PRACK when received CANCEL.

Root Cause: The issue was introduced by intercept feature (SIP_EVENT_PRACK_INTERCEPT_RCVD).

When the SBC received PRACK, it save the message into the server list. Later the CANCEL arrives, UasCancelRequestCmd, the UAS found the message in server list, and thought PRACK is not completed therefore it try to send 481.

Steps to Replicate: A calls B, B response 180 and send 180 to A with PRACK required.
After 32 seconds A sends a CANCEL, the SBC sends an unnecessary 481 PRACK.

Once the 200OK for PRACK has send out, the SIPS deletes the PRACK message in the server list to address the issue.

Workaround: Enable the e2e PRACK if the egress also supports PRACK.

PortFix SBX-96788: The DTMF 2833 PT change (in offer) has incorrect OA SBX handling (wrong answer PT).

Impact: When the peer offer with new PT, the SBC is unable to process and respond to the previous one.

Root Cause: There was a logical error that checking the wrong field to detect PT change.

Steps to Replicate: A call B and connect with "0 100". An INVITE with an offer new PT "0 101". The SBC responds with "0 100".

The code is modified to check the correct field for PT change.

Workaround: None.

PortFix SBX-105428: Error when attempting to delete negative cache record upon a probe response.

Impact: The DNS query was not out sent from the SBC if a call is made immediately after the DNS server is recovered from a blacklist.

Root Cause: Due to record is in the negative cache entry from earlier DNS dummy query, the SBC was not sending a query.

Steps to Replicate: 

1. Configure the SBC with a basic call and also configure the DNS group.
2. Ensure that DNS server does not respond to the DNS query or stop DNS server.
3. Now make call and SBC query for DNS server for resolving FQDN, but DNS server is down and there no response from DNS server.
4. After retransmission's timeout value. DNS server will be blacklisted.
5. Now the SBC keep probing DNS server.
6. Bring the DNS server up.
7. The SBC receives probe response, DNS server is removed from Blacklist.
8. Immediately make SIP call.

Expected Result:

• The SBC selects a DNS query.

As soon as the DNS server is removed from blacklist entry, those records involved with the DNS Dummy query is removed from the negative cache and resolves the issue.

Workaround: None.

PortFix SBX-105010: Observed a congestion for the G711 to G711 transcode calls even before the CPU resource limit is exhausted.

Impact: The G711-G711 transcoded capacity test shows a lower capacity.
(Test is Pump make and break g711 to g711 transcode calls on one of the active instances with 50cps, 90 CHT and 4500 calls).

Root Cause: The root cause is an incorrectly built libfmtd.a that was built with a debug option and makes the FMTD module take more cycles. This bug was addressed originally in SBX-104122.

Steps to Replicate: Perform a test indicated in SBX-105010.

The code is modified to remove the -g flag and using -O3 flag for compilation (Port of SBX-104122).

Workaround: None.

PortFix SBX-105072: Password padding with random characters by the SBC causes the RADIUS server to reject the password.

Impact: The radius password sent to the server has no zero characters at the end following the password and a NULL.

Root Cause: The radius passwords are padded to 16 characters. The existing implementation did not set those padded characters to 0.

Steps to Replicate:

1. Configure a radius authentication server.
2. Use a password that is less than 15 characters long for the external radius user.
3. Set the externalAuthententication to true.
4. Run a tshark session.
5. Login to the CLI.
6. Stop the tshark.
7. View the radius password element in wireshark after configuring the shared secret in wireshark under protocol preferences.

The padded characters are now set to 0 to address the issue.

Workaround: Use 15 or 16 character passwords.

PortFix SBX-66831: Need to update/add proper warning message when deleting the ipInterface.

Impact: The warning message does not display in the cloud SBC when deleting the IP interface, but there is no problem with functionality.

Root Cause: This is working fine in the hardware SBC. We have the problem in the cloud SBC because when displaying the warning message, we checked few values if that values are matching. Then, show the warning popup but in case the cloud SBC values index are different and check is failing because of that, we are unable to show the warning popup when we delete IP Interface.

Steps to Replicate: 

1. Login to EMA.
2. Go to All -> Address Context ->IP Interface Group -> IP Interface.
3. Click on New IP Interface button.
4. Fill the correct info in text field, select state as enable, Mode as In Service and click on save button.
5. The creation should be successful.
6. Select the same entry from IP interface list section and click on delete button.
7. We should be able to see warning popup like "'addressContext PPP_AC ipInterfaceGroup PPP_IG ipInterface': Deleting ipInterface while in dryUp action can
   cause problems with calls on the ipInterface. Use force action to clear existing calls on the ipInterface before deleting. Do you wish to continue?"

The code is modified to handle the Cloud SBC scenario.

Workaround: None.

PortFix SBX-100286 to 9.2: The trace file containing SIP Rec PDU was not imported in Ribbon Protect properly.

Impact: When a level 4 call trace is active, if a SIP PDU is too large to fit on a single trace line, it is split over multiple lines. However, the Ribbon Protect only reads the first line - there is no way for it to know subsequent lines are continuation lines.

Root Cause: Design issue.

Steps to Replicate: 

1. Enable level 4 call trace.
2. Send a SIP PDU to be traced towards the SBC of total size 2000 bytes.

The code is modified to put the same header onto each level 4 trace line. Include in the header two new fields, the MSG ID and PART to allow Ribbon Protect to recombine multiple trace lines to recover the original message. The equivalent Jira VIGIL-17137 is required for Ribbon Protect compatibility.

Workaround: None.

PortFix SBX-100590: The wrong "Egress Zone Name" in ATTEMPT CDR for a GW-GW call.

Impact: For a call with multiple routes that include one or more local routes followed by routes on another gateway (for GW-GW) scenario, if the local routes fail and then other gateway route is selected, the resulting ingress gateway CDR for the call has incorrect egress zone name and ID.

Root Cause: When there are multiple routes, when a route fails the call cranks back to use the next route. When selecting the next route the code is not clearing the internal CDR information for the egress zone. So if the next route selected is on another gateway, the CDR information retains the egress zone information from the previous attempt rather than being blank.

Steps to Replicate:

1. With the SBC GW-GW setup create two routes for a call such that:
   - route 1 is routed through local egress trunk group on SBC1.
   - route 2 is routed through egress trunk group on SBC2.
   - Set crankback profile to enable attemptRecordGeneration and add reason 41 (503 is mapped to 41 in default SIP to CPC mapping profile).
2. Send a valid INVITE to the SBC1 ingress that will first select route1 and route out through the egress trunk group on the SBC1. From the egress endpoint reject that call attempt with a 503 Service Unavailable.
3. The call will crank back and re-route using route2 through the SBC2 egress trunk group. From the egress endpoint reject that call attempt with a 503 Service Unavailable.
4. Check there are two Attempt CDR records for the call on SBC1.
   1. ATTEMPT1 - should have the "Egress Zone Name" and "Egress Zone ID" fields populated correctly with zone information for zone of SBC1 egress.
   2. ATTEMPT2 - should have the "Egress Zone Name" empty and "Egress Zone ID" as 0 because this is an attempt for GW-GW route.

The code is modified to clear the internal CDR information for the egress zone when one route fails and the next route is selected.

Workaround: None.

PortFix SBX-74991: Saving changes to a SMM profile after doing an edit/update operation is taking long time[8 to 16 minutes].

Impact: The SIP Adaptor Profile with 250+ rules takes lot of time to create a profile including the update as well.

Root Cause: When the SIP Adapter Profile with more than one rule is created, each rule is committed individually. However, after each commit EMA internally retrieves all the SIP Adapter Profiles along with their rules, though this is not required it is causing slowness of both create and update operation.

Steps to Replicate: 

1. Login to EMA.
2. Navigate to Sip Adapter Profile Screen.
3. Create a profile with 200+ Rules.
4. Click on Save button.

The code is modified to prevent retrieval of sip adapter profiles after each commit

Workaround: The only workaround is to create the profile from the CLI.

PortFix SBX-101769: The SBC sends the wrong SDP in 200 OK for UPDATE received from the Ingress during tone play(LRBT).

Impact: The SBC responds with wrong codec in the answer to UPDATE received on ingress/calling side while local ring back tone is playing. As a result, the UPDATE is being used to change the codec on ingress. In this case, the SBC continues to play the RBT using the previously agreed codec.

Root Cause: There was a logical error in the code that was picking the currently active packet service profile to answer instead of honoring the modify.

Steps to Replicate: Configuration:

1. Configure SBC to make a Basic A-B call.
2. Configure PSX with G711 and G729 in codecEntry and this Leg/other Leg configuration.
3. Configure LRBT, and attach to Ingress TG.

Procedure:

1. Send an INVITE from UAC with G711.
2. Send an 180 from UAS, with no SDP.
3. Send an UPDATE from UAC, with G729.
4. Receive the 200 OK from INVITE from UAS.

The code is modified to address this issue.

Workaround: None.

PortFix SBX-103771: There are special characterss in DN hinders EMA display.

Impact: The EMA does not display route with destination national containing special characters.

Root Cause: The support for special characters was not available in EMA.

Steps to Replicate: 

1. In the SBC 7.2.4 R0, create a Route with Destination National containing special characters.
2. Upgrade to 8.2.4 R0.
3. After the upgrade, the EMA should display the route.

The code is modified to support special characters in the EMA.

Workaround: The CLI can be used to view route details.

PortFix SBX-104247: The resource memory congestion level 3 is approaching the threshold 90 sample 81.

Impact: There was a SAM Process is leaking memory when AKA is being used.

Root Cause: There was a SAM Process is leaking an AKA related structure when the code is handling an error case scenario.

Steps to Replicate: The issue cannot be reproduced.

The code is modified to correctly free the AKA structure in all scenario.

Workaround: There is no known workaround.

PortFix SBX-93898: The "request sbx xrm debug command sec -stat gcid <gcid>" was not showing ENC and DEC details on the SBC SWe.

Impact: This debug command in unhide section is not showing all the required fields populated in the SWe SBC.

Root Cause: NP response is not framed in expected order to application layer.

Steps to Replicate: Run the SRTP call and issue this debug command for internal use.

The code is modified so the NP Response is correctly framed in expected order to application layer.

Workaround: This debug command is to see SSN field value with the RoC, all other details can be seen from show call mediastatus.

PortFix SBX-103599; The SBC is sending multiple re-INVITEs to the ingress and egress leg during a 200OK of INVITE answered with Dialog-1.

Impact: The SBC is sending multiple re-INVITEs to the ingress and egress side if the first forked leg is answering with a 200 OK.

Root Cause: When the 200 OK is coming without SDP, we were taking last received SDP and that precondition attributes to further processing.

Steps to Replicate: Precondition Transparency is set on both the Ingress and Egress
Steps:

UAC sends INVITE with Supported: precondition and SDP with precondition attributes.

First forked UAS sends 183 with SDP with preconditions.

Second forked UAS sends 183 with SDP with preconditions.

First forked leg send 200 OK without SDP.

Expected Behaviour:

1. The SBC transparently passes precondition attributes in SDP.
2. The SBC gets first forked 183 with SDP with preconditions.
3. The SBC forwards the 183 to ingress with preconditions.
4. The SBC sends an UPDATE to ingress and egress to complete precondition state.
5. The SBC forward second the 183 to ingress with preconditions.
6. The SBC sends an UPDATE to ingress and egress to complete precondition state.
7. The SBC forwards 200 OK and get ACK.
8. The SBC completes reINVITE-200-ACK towards egress.

The code is modified to address the issue.

Workaround: None.

PortFix CHOR-6320: The UxPAD coredump observed on the media container while pumping 2X overload of G711U to G711U DSP (media transcoding disabled) call load.

Impact: The SWe_UXPAD crash was seen during an overload transcode test in a 2 vcpu SWe deployment in the public cloud.

Root Cause: Potential corruption in the packet buffers due to issues related to concurrency specifically in the 2 vcpu transcode overload scenarios.

Steps to Replicate: 

1. Configure a 2 vcpu instance.
2. Run 700 G711-G711 transcode sessions for overnight/long duration.
3. Check if there is any UXPAD/NP crash seen.

The code is modified to fix concurrency issues in transcode call scenarios in the 2 vcpu SWe deployments.

Workaround: No workaround apart from avoiding overload scenarios for transcoded calls.

PortFix SBX-105182: After the SBC receives RCODE = 2 from the DNS server, the SBC makes a strange query request to the DNS and subsequent new calls are rejected.

Impact: When RCODE error received for the DNS query and no other servers were available, the record was never being deleted from cache and this remained as a RECORD in "RESOLVING STATE" that impacted further queries with this domain and resulted in call failures.

Root Cause: This particular test scenario's was not covered during the testing feature.

Steps to Replicate: 

1. Setup with EDNS supported and DNS fall back disabled.
2. The single DNS server configured for the DNS Group.
3. The DNS Server configured to return RCODE Error 2 for SRV Query.
4. Setup a call that trigger NAPTR, SRV and A Query and the DNS Server responds with RCODE 2 for SVR.
5. Modify the DNS Server to return successful RCODE for SRV.
6. Make another call for the same domain. The call fails because the SRV record is in "RESOLVING STATE" in the cache.

The code is modified to delete cache entry (which is in "RESOLVING STATE") when the RCODE error is received.

Workaround: NA

Enhance the sbxPerf on the SBC for lesser resource consumption.

Impact: The SBC performance monitoring tools like top and top2 at times take 20% of a CPU core there by reducing total available CPU resources for management activities on the SBC.

Root Cause: The sbxPerf that contains list of tools such as top, mpstat, and iostat currently runs periodically without any linux value configured. This can cause these processes to get prioritized and scheduled ahead of other management processes such as ConfD and SSH.

Steps to Replicate: Install fix build and ensure processes such as top, top2, mpstat sbxPerf are running with corrects value of 15 using the top command.

The code is modified so that priority of these processes are less compared to other management processes on the SBC.

Workaround: None.

The ACL rule is missing on the SBC in LD triggered switch-overs and link recovered after.

Impact: When an IPACL rule is created that references an IP interface, but that IP interface has been down since the system started up, the rule is not successfully created.

Root Cause: When an IP interface is failed on system startup, the system does not add that interface to the kernel. When an IPACL rule is added that references that IP interface, it fails to be added to the NP.

Steps to Replicate: Start an SBC with an IP interface failed, that also has one or more IPACL rules that have been configured that reference that IP interface. Bring up the IP interface and logs will show the IPACL rules have been successfully added.

The code is modified to detect this situation and maintain a retry list. Once the IP interface comes up, it will retry the associated IPACL rules that were previously failed.

Workaround: This issue can be worked around by having IPACL rules that do not reference the IP interface.

There was a customer call forwarding MRF interaction failure on the receipt of an UPDATE.

Impact: The SBC sends INVITE towards MRF upon receiving UPDATE from Egress End Point

Root Cause: In some situations when there is delay in incoming PRACK request (In this situation PRACK from Ingress Endpoint), some internal logic of the SBC queues the SDP on ingress leg and let's SBC answer to egress, thereby unblocking the egress peer which can send a new offer using SIP UPDATE.

This causes the SBC to send INVITE towards MRF, even before sending an UPDATE out towards the Ingress.

Steps to Replicate: Configure the PSPs accordingly:

1. UAC sends Invite with AMR-WB, AMR, telephone-event.
2. UAS sends 183 with SDP with AMR-WB with 100rel.
3. UAC sends PRACK/200 OK.
4. UAS sends 180 without SDP without 100 rel.
5. PRACK for 180 is pending on Ingress.
6. UAS sends UPDATE with PCMU, telephone-event.

Expected Results:
==============

1. Upon receiving an UPDATE with PCMU, the SBC will not auto answer towards UAS.
2. The SBC will wait for PRACK to be received from UAC, then will relay UPDATE towards UAC.
3. Based on answer in 200 OK (Update) from UAC, the SBC will send INVITE towards MRF for reserving transcoding resources.

The code is modified to ensure that SBC does not auto answer to egress.

Workaround: None.

The No Way Video after A/V calls is resumed after hold.

Impact: No way video after audio video call is resumed using late media re-INVITE.

Root Cause: The SBC does not send "sendrecv" in 200 OK to late media re-INVITE, instead sends the same datapath mode that was negotiated last - that is - inactive.

Steps to Replicate: Setup an audio and video call. Put the call on hold by sending a=inactive in audio and video SDP. Then, send a late media re-INVITE to take the call off-hold. The idea is that the SBC will send SendRecv for audio and video in 200 OK and let the remote end choose if it wants to remain in hold or come out.

The code is modified to send sendrecv in other streams too when call is being "may be" resumed after hold.

Workaround: Do not use late media re-INVITE to come out of hold.

Observed a PRS process core dump "systemerror healthcheck timeout"

Impact: A PRS coredump was found due to healthcheck timeout upon executing a CLI debug command

Root Cause: The CLI debug command was taking a long time due to too many ACL entries resulting in a health check timeout.

Steps to Replicate: 

1. Add more than 1000 ACL entries
2. Execute below debug command:
   Run CLI debug command "request sbx xrm debug command acl\ -stat"
   
   NOTE: The debug commands are not available on customer sites.

The code is modified to process only 200 ACL entries to allow CLI to recover sooner.

Workaround: As we do not run debug commands on customer sites, this error should not  occur.

There was a problem with management of the maxptime and ptime in the Direct Media mode.

Impact: There was a problem with management of maxptime and ptime in the Direct Media mode.

Root Cause: Generally, the SBC ignores the ptime value if only maxptime attribute is present in incoming SDP, If "sendPtimeInSdp" IPSP flag is enabled and peer is sending both ptime and maxptime, SBC is preferring maxptime value while sending ptime,
The existing IPSP flag "sendPtimeInSdp" makes the SBC to send out a=ptime irrespective of what we receive from the peer.

Steps to Replicate: 

1. Do a basic call from A to B.
2. A sends ptime=20 and maxptime=40 towards the SBC.
3. An INVITE going out from the SBC should contain ptime=20 as we enabled both "sendPtimeInSdp" IPSP flag and "preferPtimeInSdp" TG flag.

A new flag "preferPtimeInSdp" is introduced under SBC CLI Trunk Group signaling.
If this new flag "preferPtimeInSdp" is enabled, the SBC prefers ptime value received in a=ptime over a=maxptime in incoming SDP.

This new flag "preferPtimeInSdp" needs be enabled in conjunction with existing flag "sendPtimeInSdp". If the SBC is configured to send a=ptime, then preferPtimeInSdp should be enabled on the incoming trunk and vice versa.

Workaround: None.

The RTP sourceAddressFiltering is not working (call leg dependency).

Impact: The source address validation is not done if call flow does not involve NAPT learning. For calls that want source address validation but does not have NAPT enabled, the SBC would not validate the source address and end up forwarding the RTP packet to the other endpoint.

Root Cause: There is no way for application to inform Network Processor to validate source if NAPT learning is not enabled.

Steps to Replicate: Involves a CISCO MOH server but probably something else can be used.

The code is modified to validate source even when NAPT learning is not enabled to address the issue.

Workaround: Enable NAPT learning for the call.

The SBC5210 upgrade failure during Starting DB_RESTORE stage.

Impact: An upgrade failure during Starting DB_RESTORE stage.

Root Cause: Creation of foreign key fails on table packet_service_profile column CODEC_ENTRY_FK9, as it had "0" in one of its rows and dbimpl.codec_entry table did not have this value.

Steps to Replicate: Upgrade a dump that has "0" in packet_service_profile.CODEC_ENTRY_FK9, it should be successful.

Before add the referential keys, the data is checked and ant value that is not there in parent table is nullified to address the issue.

Workaround: None.

Empty "Egress Local Signaling IP Addr" field in the CDR

Impact: When an incoming call is routed out of the SBC but is then rejected by the SBC with cause 132 Module failure before a backwards response message has been received, the resulting attempt CDR has an empty "Egress Local Signaling IP Addr" field.

Root Cause: The SBC currently populates the "Egress Local Signaling IP Addr" CDR field when it processes a backwards response message for the call, and as a result the field remains empty until a backwards response message is received and processed.

Steps to Replicate: 

1. Initiate a call by sending a valid INVITE to the SBC ingress TG to route towards the egress TG.
2. Once the INVITE is sent to egress by SBC, put the signaling port to egress out of service.
3. The SBC will clear the call by sending a 503 message towards egress. The resulting attemptCDR should have valid address in "Egress Local Signaling IP Addr" field.

The code is modified to populate the "Egress Local Signaling IP Addr" CDR field on sending out the egress INVITE message.

Workaround: None.

There was a memory leak on the glusterfsd.

Impact: The OAM application cores.

Root Cause: Memory leak is observed for the glusterfsd process, that is the brick process. The use of the "gluster volume heal info" function causes the process memory usage to increase and not go down afterward.

Steps to Replicate: Ensure both Active and Standby OAM are running.
Monitor glusterfsd memory usage on both nodes.

Use the "gluster volume heal statistics heal-count" command to determine the gluster bricks heal state to address the issue.

Workaround: Reboot the OAM node.

The SBC is ignoring Use Max Bitrate Only flag.

Impact: The SBC is ignoring Use Max Bitrate Only flag.

Root Cause: During Re-Invite answer processing, NrmaAdjustPeerT38MaxBitRate() routine is doing T38MaxBitRate adjustment based on packet size that is changing T38MaxBitRate value from 14400 to 4800.

Steps to Replicate: Call Flow:

1. Do a basic call between UAC and UAS.
2. The UAS sends fax re-invite to UAC with T38MaxBitrate set to 14400.
3. The SBC sends Invite out with T38MaxBitrate set to 14400.
4. The UAC answers with T38MaxBitrate set to 14400 in 200OK.
5. The SBC sends out 200ok with T38MaxBitrate set to 4800.

Actual Result:

------UAS(Re-Invite) ---------------------------> SBC --------------------------> UAC
T38MaxBitRate:14400 T38MaxBitRate:14400
-----UAC(200ok for Re-Invite) --------------> SBC ---------------------------> UAS
T38MaxBitRate:14400 T38MaxBitRate:4800

Expected Result:

------UAS(Re-Invite) ---------------------------> SBC --------------------------> UAC
T38MaxBitRate:14400 T38MaxBitRate:14400
-----UAC(200ok for Re-Invite) --------------> SBC ---------------------------> UAS
T38MaxBitRate:14400 T38MaxBitRate:14400

The code is modified to relay the T38MaxBitRate in SDP, which is received from peer for Direct Media calls.

Workaround: None.

The redirection NOA set wrong in the CDR.

Impact: The "Redirecting Orig Cd Num - NOA" subfield of "Redirection Feature Spec Data" is not set correctly in CDR record if the the Redirecting Original Called Number - NOA value has been modified by PSX.

Root Cause: The existing code was not updating the value for the CDR record based on route specific information returned from the PSX.

Steps to Replicate: Test Setup
========

1. Test requires call routing through the PSX.
   Routing on PSX, setup to route incoming call through routing label with two routes:
   - route 1, through local egress trunk group 1 on SBC to UE2.
   - route 2, through local egress trunk group 2 on SBC to UE3.
2. On egress trunk group 1 only, associate a DM/PM rule to change the "Number Type" to International for "redirecting Original Called Number".
3. Set the crankback profile to enable attemptRecordGeneration and add reason 41 (503 is mapped to 41 in default SIP to CPC mapping profile)
   set profiles callRouting crankbackProfile default attemptRecordGeneration enabled reason 41

Test Procedure
============

1. Send INVITE to the SBC ingress, INVITE should include a diversion header e.g.
   Diversion: <sip:+4969667740185@xxx.xxx.xxx.xxx>;privacy=off;screen=no;reason=unknown;counter=1
2. Call is routed using route 1, respond from first egress with 503 Service Unavailable
3. Call is re-routed using route 2, respond from second egress with 503 Service Unavailable
4. Check the CDR records for call on the SBC that should have two attempt records -
   1. ATTEMPT1 - should have "Redirection Feature Spec Data" that has subfield 22 "Redirecting Orig Cd Num - NOA" as 3 (Unique International Number), because the number was updated by PSX to international.
   2. ATTEMPT2 - should have "Redirection Feature Spec Data" that has subfield 22 "Redirecting Orig Cd Num - NOA" as 2 (Unique National Number), because the number was not updated by PSX
      
      

The code is modified to populate the CDR value from the route specific data returned from PSX.

Workaround: None.

There was an incorrect RTP time stamp in the SBC packet capture.

Impact: After an SBC upgrade to V09.01.00R001, RTP/RTCP time stamp in the SBC packet capture shows the year 2036.

Root Cause: Present logic was not considering the edge cases while checking the last modification time of the NTP log.

Steps to Replicate: 

1. Configure a remote syslog server:
   set oam eventLog platformRsyslog linuxLogs ntpLog enabled
   set oam eventLog platformRsyslog servers server1 port 10514 protocolType udp remoteHost <remote host ip>
   com
   set oam eventLog platformRsyslog syslogState enabled
   com
2. Create a second dummy NTP server (just to trigger some logs to be written to ntp.log):
   set system ntp serverAdmin 1.2.3.4
   com
3. Delete the dummy NTP server:
   del system ntp serverAdmin 1.2.3.4
   com
4. Wait at least several seconds.
5. Repeat steps 2 and 3.

The code is modified for the edge case of NTP log last modification time.

Workaround: None.

The MS Teams with FLRBT enabled hangs the call.

Impact: When the Force Local Ringback Tone is applied on a call that has been through SIP replace followed by SIP refer procedure, and the final trunk group has downstream forking enabled. The final 200 OK response may be queued indefinitely, meaning the call does not get completed.

Root Cause: Interactions between Force Local Ringback Tone, multi-party and downstream forking.

Steps to Replicate: 

1. Make a SIP to SIP call.
2. A calls B.
3. B' replaces B.
4. B' refers to C.
5. The trunk for A has Force Local Ringback Tone applied.
6. The trunk for C has downstream forking enabled.
7. C responds with 180 with no SDP, 183 with SDP, then 200 OK.

The code is modified to prevent the queuing.

Workaround: None.

The SIPCM (SAM) deadlock reported on SNMP request.

Impact: The SAM Process cores when getTlsStatus command is executed.

Root Cause: The issue is because when fetching the TLS Status, we were trying to access sessData that was no longer valid.

Steps to Replicate: Execute getTlsStatus command when running TLS load. SAM process cores randomly.

The code is modified to fetch the status from socketPtr instead of sessData.

Workaround: None.

The sbxAutoBackup.sh changes for AWS SWE.

Impact: The Cloud SBC uses default system name 'vsbcSystem' when creating system backups, making it difficult to distinguish between setups.

Root Cause: The Cloud SBC configuration file backup logic uses the system name to create the back up file instead of the specific name.

Steps to Replicate: Run /opt/sonus/sbx/scripts/sbxAutoBackup.sh on the SBC running in a cloud (e.g. AWS) that has SystemName configured too something specific in user-data.

Use the actual system name (configured in user-data) in the configuration backup filename to address the issue.

Workaround: None.

Unable to see the signaling messages on the PKT log file.

Impact: The Packet capture does not work properly for pkt ports.

Root Cause: The version of libpcap0.8 library in Debian9 had a defect that resulted in the capture not working.

Steps to Replicate: 

1. Login to the SBC and run a basic call configuration.
2. Create a signaling packet capture filter: set global callTrace signalingPacketCapture state enable devices pkt0 0.
3. Run the SIPP calls.

Expected Results:

1. The SBC should capture all the signaling messages on the pkt0 port in a PKT file under the gsxlog directory.

Update the libpcap0.8 to a newer version to address the issue.

Workaround: None.

The SBC REST API. pam phase authorization failed to login through a PAM: timeout

Impact: Too many requests on the REST API interface causes 401 error.

Root Cause: Issue with third party tool ConfD that is used for configuration management.

Steps to Replicate: 

1. Send REST API requests for long duration.
2. Open some CLI session for some time to cause max session limit.
3. REST APIs start failing with 401 error due to session limit, as expected.
4. Close the CLI sessions.
5. REST APIs keep failing, even after keeping CLI sessions under max limit.

The code is modified to fix the issue.

Workaround: None.

The SIPREC was not working. No request sent to recorder.

Impact: Without this fix, SBC is generating INVITE towards the SIPREC SRS with bad format on SDP header line 'label'. The 'label' header is missing the CR in the CRLF line break.

Root Cause: The code that formed the SDP header line "label" does not add CR character for CRLF line break.

Steps to Replicate: Make a call that would match the configured SIPRec criteria and thus triggering a SIPRec session for the corresponding Communication Session.

The code is modified to add CR character for CRLF line break, for SDP header line "label" .

Workaround: None.

The reenableOsAccount silently sets an account expiration to 30 days.

Impact: Using the reeanbleOsAccount will add account aging to any OS account.

Root Cause: The logic does not take into account the state of OS account aging or if OS account aging should be applied (e.g. root).

Steps to Replicate: Run with an OS account aging enabled:

1. Test root:
   1. ReeanbleOsAccount for user root.
   2. Check no aging has been applied at OS level: chage -l root.
2. Test the Confd user:
   1. Create user through CLI.
   2. ReeanbleOsAccount for this CLI user.
   3. Check no aging has been applied at OS level: chage -l <CLIUser>.
3. Test OS user with a password:
   1. Create OS user with a password.
   2. Disable and then Enable OS account aging state.
   3. ReeanbleOsAccount for this OS user.
   4. Check correct aging has been applied at OS level: chage -l <OSUser>.

The code is modified in the OS account aging is enabled and is applied to the account.

Workaround: None.

The active OAM node for the Signaling SBC generates header-only ACT files and pushing it to the billing server, where the OAM node should be responsible for the configuration only.

Impact: The active OAM node generates header-only ACT files and pushing it to billing server.

Root Cause: The SBC does not check whether it is OAM node or Managed VM while generating ACT files.

Steps to Replicate: Spawned a Nto1 OAM node, check whether .ACT files exist or not. Restart active Node, Check if any .ACT file gets created. Restart the current Active again (assigned role Standby) and check whether any .ACT file is present on OAM node or not.

Check whether VM is OAM before creating the ACT files to address the issue.

Workaround: None. 

The SBC fails to relay embedded header in Contact of 3xx with statusCode3xx relay enabled

Impact: The SBC fails to relay embedded header in Contact of 3xx with statusCode3xx relay enabled

Root Cause: The code required to send the embedded part in contact of 3xx was not present

Steps to Replicate: 

1. Configure the SBC for A to B call.
2. Enable the flag statusCode3xx on egress TG IPSP.
3. Send an INVITE from UAC.
4. Send 300 Multiple Choice from UAS with below Contact header that has embedded headers.

The code is modified so the contact header transparency is set in the relay 3xx scenario that is now being changed to full contact header transparency and set the SIP_HEADER_URI_HEADERS so that the embedded contact header is transparently passed in the 3xx. A new header type is introduced that is set only in the relay 3xx case to send the entire contact header transparently.

Workaround: None.

The SubsystemAdmin filter affects calltrace (TRC) logging showed useless logs.

Impact: After creating then deleting an entry in the oam eventLog subsystemAdmin, the original behaviour is not restored. INFO level events for that subsystem are no longer logged, even if the oam eventLog typeAdmin filterLevel is info.

Root Cause: Deleting an entry in oam eventLog subsystemAdmin does not restore settings back to default.

Steps to Replicate: 

1. Create an entry for a subsystem in the oam eventLog subsystemAdmin table.
2. Delete the entry.

The code is modified to process to deleted the oam eventLog subsystemAdmin entry so that settings are put back to default values.

Workaround: None.

Our SIPREC implementation is prone to failure to record a stream and to tear down the SRS call leg when SRS (SIP Recording Server) sends a re-INVITE to SBC

Impact: SIPREC handling had race conditions issues in the scenario where a RE-INVITE was received on the main call and also RE-INVITE was received from SRS server at the same time.

This resulted in recording failure and the SBC sent out BYE towards SRS.

Root Cause: This race condition of RE-INIVITE from the main call end point and SRS server simultaneously was not handled on the SBC.

Steps to Replicate: 

1. Make a SIPREC call.
2. Send RE-INVITE with codec change/hold on main call (either from UAC or UAS) this triggers a RE-INVITE towards SRS.
3. SRS also sends RE-INVITE at the same time when SBC receives 200 OK for main call RE-INVITE.

The RE-INVITE triggered/sent towards SRS due to the main call RE-INVITE is queued in the situation where the SIPREC leg is in middle of processing the RE-INVITE that was received from the SRS. The queued RE-INVITE would be sent once the RE-INVITE transaction received from the SRS was completed. 

Workaround: The SRS can avoid sending immediate hold RE-INVITE if no media is observed on SIPRec leg.

The EMA codec entry screen was not usable.

Impact: The EMA codec entry screen was not usable.

Root Cause: The method call place mismatched.

Steps to Replicate: 

1. Log into EMA.
2. Navigate to Profiles > Media > Codec Entry
3. See that the screen is working correctly.

The code is modified so the EMA codec entry screen is placed at the appropriate position.

Workaround: None.

A SIPREC Codec issue.

Impact: Without this fix, the wrong codecs are negotiated with SRS when communication sessions is renegotiated.

Root Cause: Because of wrong conditions in code, the codec information that is to be sent towards SRS is fetched from peer packet service profile rather than active service profile from the communication call leg.

Steps to Replicate: 

1. The SBC receives INVITE with G711A, G711U
2. Egress leg responds with A law and RTP stream is A law.
3. The SBC sends an INVITE to SIPRec server with G711A law and SIPRec servers responds with A law.
4. Egress leg sends re-INVITE and changes the order of codecs, G711U and G711A respectively.
5. The SBC relays the re-INVITE to ingress peer with G711U, G711A respectively and ingress peer responds with G711A.
6. The SBC was sending re-INVITE with G711U codec to SIPRec Server.
With this fix, SBC now sends re-INVITE with G711A codec to SIPRec server.

The code is modified to fetch correct codec information from the communication session and the same is sent in recording session.

Workaround: None

The "numMatch notmatch" gets auto converted to "numMatch match" when SMM profile is modified/saved from GUI

Impact: The "numMatch notmatch" gets auto converted to "numMatch match" when SMM profile is modified/saved from GUI

Root Cause: The numMatch field was not there in the EMA.

Steps to Replicate: 

1. Navigate SipAdaptorProfile screen. (Profiles→Signalling→SipAdaptorProfile).
2. Create profile with the selection numMatch field.
3. Create profiles with all possible values.
4. Here the numMatch field was newly added.
5. Verify whether all field values are coming properly or not.

The code is modified to address the issue.

Workaround: None.

The SWe application will not start: The memory mismatch after a AWS instance restart.

Impact: The standby SBC shuts down if the memory product capability mismatches with the active SBC.

Root Cause: The SBC checks for a minimum required memory with the memory allocated to peer and it should be either equal or greater than the peer SBC.

Steps to Replicate: Launch the SBC HA pair on AWS to see if the memory allocated to the standby SBC is lower than an active SBC.

The code is modified to change the memory from 'Minimum required' to 'Informational' to not shut down the app in case of a mismatch and only raises a trap.

Workaround: Try rebooting in case memory allocated to standby the SBC is lower than the active SBC.

After a switchover, the MRFP does not send a Goodbye packet for the Text stream.

Impact: When a call is using T140/TTY interworking, after a switchover RTCP bye is not sent toward the T140 stream endpoint.

Root Cause: In a T140/TTY interworking, the resource for T140 stream is not mirrored properly to standby causing deactivating does not work after a switchover.

Steps to Replicate: 

1. Make call with T140/TTY interworking (for example, AMR-WB+T140 - PCMU).
2. Once call is established, switchover.
3. After a switchover was successful, end the call and see that RTCP bye is sent properly to T140 stream.

The code is modified to mirror resource correctly for T140 stream in T140/TTY interworking.

Workaround: None.

The SBC is not updating the DNS servers order in DnsServerStatistics on deleting and creating a fresh DNS Group.

Impact: The SBC was showing the incorrect DNS Servers Index in DnsServerStatistics command "show table addressContext default dnsGroup <dnsGroupName> dnsServerStatistics " when the DNS Servers were deleted and re-created in a different order with the same server IPs.

After creating 128 DNS Servers with different unique IPs are and then deleting some of them such that the total number of DNS Servers is less than 128, no new DNS servers created post the delete operations shall have the corresponding statistics information, even though the total number of servers is less than 128 on the system.

As an example, if on the SBC we create a DNS Server with IP1 and then delete it and then proceed on to repeat the create and delete operations with different IPs say IP2 to IP128, then from the IP128 onwards, no statistics shall be displayed, even though there is only one DNS Server on the system.

Root Cause: The DNS Server Statistic blocks are not deleted when the DNS Servers are deleted and continues to remain hashed with the IP with that it was created.

Steps to Replicate: 

1. Configure the DNSGROUP.
2. Add the 1st DNS server with IP1 to DNSGROUP.
3. Add the 2nd DNS server with IP2 to DNSGROUP.
4. Check the DnsServerStatistics.
   show table addressContext default dnsGroup <dnsGroupName> dnsServerStatistics
5. Delete the DNSGROUP.
6. Configure the DNSGROUP.
7. Add the 1st DNS server with IP2 to DNSGROUP.
8. Add the 2nd DNS server with IP1 to DNSGROUP with weight1.
9. Check the DnsServerStatistics.
10. The IP2 should have 3 as Index and IP1 should have 4 as index in dnsServerStatistics.

The code is modified to delete the DNS Server Statistics block and remove it from the IP hash when the DNS Server is deleted.

Workaround: None.

The  AddressSanitizer: heap-buffer-overflow on address 0x61d001429c18 at pc 0x556c876cf74c bp 0x7fb14f64fab0 sp 0x7fb14f64f260 READ of size 3488 at 0x61d001429c18 thread T21 in the SAM Process.

Impact: In a D-SBC environment while processing the remote media data command message, the SAM process could read off the end of the received message. Reading off the end of the allocated message block could, in the worst case, result in a coredump.

Root Cause: The SAM process was assuming that the size of the received message was larger than it actually was and this resulted in reading off the end of the received message buffer. In most cases, this does not cause a problem but in could potentially result in a coredump if the associated buffer is at the end of the addressable memory space.

Steps to Replicate: Run the RFC4117 test case for audio transcoding and video relay.

The code is modified to not read off the end of the received memory block.

Workaround: None.

The HA SBC Admin Connection failure issue within sometime for the ASAN build.

Impact: In an HA setup, the standby SCM process can read off the end of a memory block. Reading off the end of allocated memory blocks can cause unexpected behaviour and in the worst case, it could result in coredumps. This problem only impacts the standby server so will not be service impacting.

Root Cause: While making P-Asserted-Id header information redundant between active and standby, the standby was reading off the end of the allocated memory buffer because the active instance had passed over a bad data length value for P-Asserted-Id header.

Steps to Replicate: Run a call load where the INVITE messages contain P-Asserted-Id header in an HA setup.

The code is modified to correctly format the P-Asserted-Id header information with valid length when sending it to the standby to avoid the standby accessing invalid memory.

Workaround: None.

The CPX Process had a memory leak for single call on the OAM node.

Impact: The CPX process can leak small amounts of memory when creating/modifying the SNMP configuration.

Root Cause: While processing the SNMP configuration commands, the SBC was creating the temporary internal memory blocks to read and write information from/to the CDB. But it was not freeing up this memory at the end of the configuration action.

Steps to Replicate: Modify the SNMP configuration.

The code is modified to correctly free the temporary memory allocated during the configuration process.

Workaround: None.

Fix the Customer Telecom coverity issues.

Impact: While processing the SUBSCRIBE messages, the coverity tool has highlighted that the code could dereference a pointer that is potentially null. Although no bad behaviour has been observed during testing, there is a small chance that it could result in coredumps if the pointer really was null.

Root Cause: Based on other validation in the code, the coverity highlighted that some legs of code could result in accessing a pointer that might be null. Derefencing null pointers can cause unexpected behaviour and in the worst case coredumps.

Steps to Replicate: Run various SUBSCRIBE related test cases.

The code is modified to validate that the pointer is not null before using it to avoid any potential issues/coredumps.

Workaround: None.

The leaksanitizer had an CpxAaaGetNextEntry.

Impact: While processing the requests to display the user information/status, the Cpx process was leaking memory.

Root Cause: While processing the requests to display user information/status, the Cpx process had to allocate the internal memory blocks to collect information from CDB and was not freeing up this memory at the end of the request.

Steps to Replicate: From the CLI, run commands to request user information.

The code is modified to free up the memory at the end of the processing request.

Workaround: None.

De-reference after a NULL check and de-reference the NULL return value.

Impact: The coverity scanning tool identified a potential edge case scenario, where the lawful intercept code could potentially access a NULL pointer leading to unexpected behaviour and potentially a coredump.

Root Cause: While the code was performing X2 peer status updates, it retrieved the peer information from a hash table but did not verify that the pointers inside the record it retrieved were valid. It would be an extreme error case for this to result in reading of a null pointer and would likely need to be due to another problem. But coverity highlights it as an edge case issue to fix up.

Steps to Replicate: Run test cases for lawful intercept using packet cable V2.0 configuration.

The code is modified to validate the pointer is not null before using it and avoid any error scenarios.

Workaround: None.

The AddressSanitizer: detected heap-use-after-free in UasProcessMsgCmd on address 0x623000187198 at pc 0x5623cc7b3b42.

Impact: The SCM process is accessing memory after it has been freed during the timer expiry handling when there is no response to an OPTIONS message that was triggered due to debug optionsPing using an FQDN. Accessing memory after it has been freed can cause unexpected behavior and in the worst case, this potentially causes coredumps.

Ex:"request addressContext default cmds optionsPing peerFQDN bats3.gsxlab.com peerPort 14090 sigPort 2 transport udp"

Root Cause: The SIP dialog memory block was being removed in two places while handling the timeout for the debug optionsPing command, which could result in unexpected behaviour and potentially result in coredumps.

Steps to Replicate: 

1. Run debug optionsPing command
   "request addressContext default cmds optionsPing peerFQDN bats3.gsxlab.com peerPort 14090 sigPort 2 transport udp"
2. The OPTIONS message will be sent out.
3. Let the options timeout. (do not send any response).

The code is modified to address the issue.

Workaround: Do not use this debug command, or use it with an IP instead of an FQDN.

Observed that PRS process heap buffer overflow issue on 9.2 ASAN build 63

Impact: The BRM process was accessing memory after it had been freed. As the memory is being read immediately after being freed, then it is unlikely this would cause a problem.

Root Cause: If the BRM process received an unexpected message, then it was trying to print a debug message to indicate the message type. However, the message had already been freed up because it was unexpected. This was observed on the standby server and  generated at the point of switchover.

Steps to Replicate: Run a normal call load and perform switchovers.

The code is modified to collect the message type information before the message is freed so that the debug log content can be safely generated without accessing freed memory.

Workaround: None.

The SCM Process cores when the targets set for OPTIONS/MESSAGE.

Impact: Without this fix, the SBC will core dump when the Out Of Dialogue messages like OPTIONS/MESSAGE are intercepted.

Root Cause: Double free of a data structure is causing the code dump.

Steps to Replicate: 

1. Provision packet cable 2.0 Targets corresponding to the out of dialogue OPTIONS that will be sent to the SBC.
2. Send the OPTIONS to the SBC.

The code is modified to avoid double freeing of the corresponding data structure. Also, addressed couple of memory leaks concerned with the same data structure in error cases such as ARS blacklisting.

Workaround: None.

The LeakSanitizer: detected memory leaks in CpxAppProcess.

Impact: Small memory leak when making configuration changes to the GW signaling port.

Root Cause: The configuration logic was reading the interface group name from CDB into a temporary local variable in order to perform validation logic but never freed up this memory at the end of the validation.

Steps to Replicate: This issue is highlighted in the engineering lab when performing GW sig port configuration changes on an ASAN enabled build.

The code is modified to correctly free the temporary memory to avoid the leak.

Workaround: None.

The MRFP: CE_2N_Comp_CpxAppProc leak for port SWO (BFD).

Impact: There was a small memory leak during the configuration of packet port with BFD configuration associated.

Root Cause: The configuration code was reading information from the CDB and storing information in an internal memory block but not freeing it up.

Steps to Replicate: With a BFD configuration on the SBC, disable and enable the pkt port by setting mode OOS and state disabled and then enable the pkt port again.

The code is modified to correctly release the internal memory block at the end of the configuration action.

Workaround: None.

Fixing the NRMA coverity issue CID:11149.

Impact: The coverity scanning tool identified a potential code leg where a null pointer could be dereferenced and could potentially result in a coredump, although not observed in internal testing.

Root Cause: While trying to allocate resources for PCMU to PCMA call where the ingress leg is being tapped, there is a possibility the code could access and invalid pointer resulting in unexpected behaviour and in potential coredumps.

Steps to Replicate: This part of the code could be triggered for PCMU to PCMA call where the ingress leg is being tapped.

The code is modified to verify the pointer is not NULL before trying to use it to avoid potential coredumps.

Workaround: None.

The AddressSanitizer: detected heap-use-after-free on address 0x6080002177a0 at pc 0x55a942cce5b1 bp 0x7fb90b2df190 sp 0x7fb90b2df188.

Impact: There were two types of issues identified in this Jira:

1. There was a memory leak when putting a virtual media gateway in service.
2. The MRFP is accessing memory after it is freed while cleaning up a call following an internal resource allocation failure. Accessing memory after it has been freed can have unexpected behavior and in the worst case result in coredumps.

Root Cause: 

1. The MRFP was allocating internal memory while performing validation of the request to put a virtual media gateway inservice, but the memory was not being freed up at the end of the configuration action leading to a small memory leak.
2. In case of internal call failure, the call data was being deleted but the memory was being access later on in the clean up procedure.

Steps to Replicate: 

1. Configure a virtual media gateway in service.
2. Configure the MRFP without licenses and then try to make some calls.

1. The code is modified to free the memory at the end of the configuration action.
2. The code is modified to no longer access the call data after is free.

Workaround: None.

The LeakSanitizer detected memory leaks in the SCM Process.

Impact: While relaying REGISTER messages, the SBC may leak memory blocks allocated to hold the record-route header information.

Root Cause: The SBC allocates memory blocks to hold the contents of the record-route headers in the REGISTER message. But if the record-route header information is associated with the SBC IP address, then the SBC does not correctly free up the memory blocks causing a leak.

Steps to Replicate: Run test cases for stateless REGISTER relay call scenarios where the IP information in the record-route header is the SBC's IP.

The code is modified to correctly freed up memory allocated to store the record route header information.

Workaround: None.

The MRFP failed to come up after an sbxstart.

Impact: At startup of the SBC, there is a possible race condition where the SBC processes may go for an additional restart before they come up and restore the configuration. data.

Root Cause: The race condition in the code between threads.

Steps to Replicate: Installation and startup on the various platforms.

The race condition in setting some common variables is avoided by using static path of binaries to address the issue.

Workaround: None.

Due to the Adaptive Codec Change, the MRFP does not disable the text stream although there is conflict between the payload types that the non audio and text streams use.

Impact: In a audio and T140 call setup, with Adaptive Codec Change enabled and the MRFP does not disable the text stream when there is conflict between the payload types that the audio and text streams use.

Root Cause: In the Adaptive Codec Change handling, the code does not validate if the payload type used in the T140 stream are in conflict with payload types being used in the audio stream codecs. As a result, the T140 stream is created in the media plan that should be disabled in this scenario.

Steps to Replicate: 

1. Enable Adaptive Codec Change in the VMG configuration.
2. Setup a call audio and T140 media streams from both call terminations: A- side AMRWB and text 140 to B side, AMR, text T140, and Telephone event.
3. At the mid call, send a Re-Invite to change B's side to AMRWB, T140 and Telephone Event with its payload type is the same as the T140's payload type.
4. Verify that MRFP should disable the T140 stream by setting the t140 stream port number to zero in the reply to MGC(C3).

The code is modified to check the conflict for T140 payload types against the payload types being used in an audio stream, and reject T140 stream when a conflict is found.

Workaround: None.

The SBC fails to perform Alert-Info to PEM interworking when the relayPemState is disabled on the ingress.

Impact: The SBC fails to perform Alert-Info to PEM interworking when the relayPemState flag is disabled on the ingress TG.

Root Cause: The relayPemState flag implementation was not considered for Alert-Info to PEM Interworking scenarios.

Steps to Replicate: Configuration:

1. The relayPemState flag is disabled on the Ingress TG.
2. The iToPemInterworking flag is enabled on the Egress IPSP.
3. The acceptAlertInfo flag is enabled on the Egress IPSP.

1. The UAC sends an Invite with the SDP with PEM: supported to the SBC.
2. The UAS sends 180 without the SDP with an alert-info as rt to the SBC.
3. The SBC should interwork the alert-Info to PEM and insert PEM:inactive while sending 180 responses towards UAC.

The code is modified to consider the Alert-Info to PEM Interworking scenarios when the relayPemState flag is disabled.

Workaround: None.

The preferred payload number was not used for either dtmf or amrwb when the "different2833PayloadType" transcode option is enabled.

Impact: The preferred payload number was not used for either dtmf or amrwb when the "different2833PayloadType" transcode option is enabled.

Root Cause: Both the AMR-WB and dtmf do not use 101, as 101 was being neglected and thinking the other one was using it. So at the end, it was not being considered and was not being populated.

Steps to Replicate: Make a SIPP call using AMR-WB codec with dmtf
enable different2833palyloadtype flag.
enable honorSdpClockRate flag.
set preferredRtpPayloadType as 101
set preferredRtpPayloadTypeForDtmfRelay 101

Check for the Preferred payload number.

The code is modified to scan even the workingDynamicPtSet and verify that 101 is not being used and is considered.

Workaround: The workingDynamicPTSet represents a more accurate information w.r.t the PTs used. As a result, that condition is being added to scan workingDynamicPtSet along with peer PSP and Route PSP.

There was a CE_node error "glusterSetup.sh Abort: Mount is in bad state!!"

Impact: A glusterSetup.sh script error log showing in CE_Node log file.

Root Cause: A glusterSetup.sh script error text is printed to the terminal.

Steps to Replicate: 

1. Shut down the Standby OAM.
2. Ensure there is no glusterSetup.sh log appear in CE_Node log file on managed node.

The error text already captured in two separate log files, so remove the output to terminal to address the issue.

Workaround: None.

Implement SAN validation for cert authentication in VNFR REST Server

Impact: The VNFM request did not have SAN validation in addition to a cert chain verification.

Now, the SAN validation checks against IPv4 and IPv6 both if present in userData.

Root Cause: The SAN validation was not implemented.

The SAN validation used to work only in mode(either IPv4 or IPv6) depending upon the SBC instantiation mode.

Steps to Replicate: 

1. Bring VNFR(running HTTPS) from VNFM GR.
2. Perform a successful reassignment.
3. Do a sbxrestart.
4. Again perform a successful reassignment.
5. Bring up a dual stack VNFM, and spawn IPv4 and IPv6 SBC.
6. Register VNFR successfully in a simplex VNFM.

Implemented SAN validation using GnuTLS that is also used for cert chain verification.
Loading both IPv4 and IPv6 from userdata addresses the issue.

Workaround: No workaround.

The cinder volume detach/attach required a reboot from Openstack to bring up all the functioning related to the SBC.

Impact: In the case of Openstack, if cinder volume is detached from the running the SBC instance, it does not cause the SBC system to shutdown properly.

Root Cause: The shutdown triggered on the volume detach from a running instance was not going through successfully due to failure to run abnormal reboot command.

Steps to Replicate: Test cinder volume detach from a running instance and ensure the node goes for a reboot and if running on active, should switchover to standby successfully.

The code is modified to update the command to forcefully reboot the node on volume detach from a running instance.

Workaround: None.

A TEAMS user was unable to resume the call when transfer is failed, when the call is initiated in beginning by PSTN user.

Impact: During a Refer, if the transfer target sends early answer in 18x but then rejects the call, the SBC fails to resume the previously active call.

Root Cause: During call transfer, after tone play, while processing early answer in 183, the SBC wrongly freed the previous cut-thru context and instead retains the previously activated tone context (for A-B call).

As a result, after transfer target rejects the call, the SBC attempts to resumes the previously active call (A-B) which fails due to unavailability of correct context.

Steps to Replicate: 

1. PSTN1 to TEAMS call
TEAMS transfer call to PSTN2
PSTN2 rejects the call
TEAMS resume the call

2.PSTN1 To TEAMS call
TEAMS transfer call to PSTN2
PSTN2 does not answer the call
TEAMS resume the call

The code is modified to retain the previously activated cut-thru context and free the previously activated tone context if current tone context is more recent.

Workaround: No

Observed the "NrmUpdateLicensesForXferFeatureType cant get origGCID" MAJOR Logs on the 5400 Platform while running a call with Multiple INVITE Replace.

Impact: The Call Pickup logs were incorrectly printing a major error event.

Root Cause: The Call Pickup logs were incorrectly printing a major error event.

Steps to Replicate: None.

The code is modified to address the issue.

Workaround: None.

The outputAdaptor profile rule gets applied after the clearDB.

Impact: The SMM rule was not applied after the upgrade from 8.2 or 9.0 or 9.1 to 9.2, even the same issue is observed during restart or switchover.

Root Cause: The SMM rule was not applied due to wrong path in the switchover scenario.

Steps to Replicate: Configure the SMM profile in the SBC.

Perform any of the of the below steps.

1. Restart the SBC.
2. The SBC performs a switchover.
3. Upgrade from 8.2/9.0/9.1 to 9.2.

The code is modified to address the issue.

Workaround: ClearDB and load the configuration.

The SBC is not feeding the delayed RBT on monitoring a failure in a late media passthru scenario in the CLOUD ISBC and SWE ISBC.

Impact: The SBC is not feeding delayed RBT on monitoring the failure in a late media passthru call.

Root Cause: The fix for the SBC was not feeding delayed RBT on the monitoring failure in a late media passthru scenario was applicable only when bToneAsAnnc flag is enabled.

Steps to Replicate: Procedure:

1. An INVITE is received without SDP and no "PEM: supported" from the UAC.
2. The UAS sends the 183 with SDP and no PEM having PCMU, AMR.
3. The UAC send PRACK with SDP PCMU,G729 and UAS sends 200 OK for PRACK.
4. The UAS sends an authorized RTP.
5. The UAS sends 180 without SDP and no PEM.
6. The UAC send PRACK and UAS sends 200 OK for PRACK.

After a RTP Monitoring failure, the SBC should play the tone.

The code is modified to address the issue.

Workaround: None.

Unable to login to the CLI session.

Impact: The SWe Active profile configuration may cause a password change commit to hang.

Root Cause: The Internal Configuration change related to sweActiveProfile leaves the changes in the candidate database. This causes another commit from the SM Process deadlock, as sweActiveProfile change subscription needs the SM Subscribers acknowledged.

Steps to Replicate: This cannot be recreated through User Interfaces. This was an internal error condition.

The code is modified to revert the changes from the candidate database, if the sweActiveProfile commit fails.

Workaround: None.

The Audit Compliance issues are found in the MRFP Setup.

Impact: The Nessus Scan with the CIS Plugin shows compliance failures on the SBC.

Root Cause: The failures were due to a missing a user directory and bad file permissions.

Steps to Replicate: Run the Nessus scan with the same policy and verify that the failed compliances do not exist.

Below changes are made to meet the CIS requirements.

1. Set nologin shell for cwagent
2. Change remoteExecution.log file permission.

Workaround: None.

Dual NUMA support in the SWe.

Impact: The SWe does not come up in the VMs having multiple NUMA nodes except for the signaling traffic profile (SLB, S-SBC, SO-SBC).

Root Cause: Due to a deliberate software restriction to not allow multiple NUMAs, the issue was shown.

Steps to Replicate: 

1. Configure a KVM instance with the dual NUMA and install in Non-Gold/Non-GPU setup.
2. Let it come in default traffic profile.
3. Verify that the SBC would come up fine without any HostCheck errors.

The code is modified to allow multiple NUMA irrespective of the personality and profile.

Workaround: None.

The BFD status is not up for the MGMT port (the BFD packets are seen though the LDG state is disabled).

Impact: Even after disabling Link Monitor on the MGMT port, the BFD packets are still seen on the MGMT port.

Root Cause: The issue was that when state is disabled and the BFD session was deleted. A timer is started so the expiry initiates a BFD session.

Steps to Replicate: 

1. Create BFD session for MGT port.
2. Check the status of BFD session.
3. Disable the BFD session.
4. Observed the BFD packets even when the BFD session is disabled.

The code is modified to check for the Link Monitor state before initiating the BFD session.

Workaround: None.

The LeakSanitizer: detected memory leaks at the confd_malloc.

Impact: The small memory leak during the configuration of lawful intercept server.

Root Cause: The configuration code was reading information from the CDB and storing information in an internal memory block but not freeing it up.

Steps to Replicate: Create a lawful intercept server and make configuration changes.

The code is modified to correctly release the internal memory block at the end of the configuration action.

Workaround: None.

The Video NAT learning is failing on the D-SBC cloud.

Impact: The Video NAT learning failed in the D-SBC.

Root Cause: On the D-SBC, NAT learning for video stream was not processed successfully and as a result caused the video packets to be dropped by the SBC.

Steps to Replicate: 

1. Configure the SBC for an Audio and Video call.
2. Enable the Signaling NAT and Media NAT in the Ingress and Egress Trunk Groups.
3. Make an Audio and Video call between the NATed EndPoints EP1 and EP2.
4. After a call gets established, the EP1 and EP2 sends Audio and Video packets.
5. NAT learning happens for Audio and Video and then the packets are sent and received from EP1 and EP2.

The code is modified to process the NAT learning for the video stream in the D-SBC.

Workaround: None.

The AddressSanitizer: detected heap-use-after-free

/localstore/ws/jenkinsbuild/sbxmainasan/marlin/SIPSG/sipsgCallNotification.c:1123

in

SipSgSendCallNotificationApi(SIPSG_CCB_STR*, SIPSG_CALL_NOTIFY_TYPE_ENUM, sip_nameaddr_str*).

Impact: While freeing up a SIP call, the code is accessing the SIP call control memory block immediately after it has been freed up in the same processing loop.

Root Cause: This issue was detected using ASAN images, it has not been proven to cause bad behaviour using regular production images, but accessing memory after it has been freed can cause unexpected processing to happen which might potentially result in coredumps.

Steps to Replicate: Run regular call scenarios with ASAN images.

The code is modified to avoid using the memory block after it is free.

Workaround: None.

Anomalies were observed after decoding a ACT File by sbxCamDecoder.pl.

Impact: 

Issue 1: The camDecoder script "sbxCamDecoder.pl" does not decode a reboot, the switchover records.

Issue 2: It does not display the field names under the "Ingress Signaling Information" and "Egress Signaling Information" for an EVENT records as shown below.

26. Ingress Signaling Information :
26.1 : "sip:7587339665@10.xx.xxx.xxx
26.2 : 1-23946@10.xx.xx.xxx
26.3 : <sip:sipp@10.xx.xx.xxx:xxxx>;tag=23946SIPpTag011
26.4 : <sip:7587339665@10.xx.xxx.xxx>
26.5 :
26.6 : sip:sipp@10.xx.xx.xxx:xxxx
26.7 :
26.8 :
26.9 :
26.10 : 0
26.11 :
26.12 :
26.13 :
26.14 : 1
26.15 :
26.16 :
26.17 :
26.18 :
26.19 :
26.20 :
26.21 :
26.22 :
26.23 :
26.24 :
26.25 : "

Root Cause: The sbxCamDecoder script was not updated to support the reboot, on switchover records.

Also, the script was not updated to decode "Ingress Signaling Information" and "Egress Signaling Information" for EVENT records.

Steps to Replicate: 

Issue 1: The switchover record issue.
Setup: The SBC HA
Procedure:

1. Perform a switchover the SBC.
2. Decode the latest ACT log that contains "SWITCHOVER" record by using the sbcCamDecoder.pl.

Issue 2: REBOOT Record issue.

1. Reboot the SBC.
2. Decode the latest ACT log that contains the "REBOOT" record, by using sbcCamDecoder.pl.

Issue 3:

1. Send an OOD message (OPTIONS).
2. Decode the latest ACT log that has the event RECORD by using the sbxCamDecoder.pl.

sbxCamDecoder was updated to support decoding REBOOT,SWITCHOVER records and the "Ingress Signaling Information" and "Egress Signaling Information" in the EVENT Records.

Workaround: Decode the record manually.

The AddressSanitizer: detected heap-use-after-free on address 0x61b000018f84 at pc 0x556c7afd4213 bp 0x7fcbf0905680 sp 0x7fcbf0905678.

Impact: An invalid memory access of a termination object after it's been deleted. Accessing the memory after it has been freed can result in unexpected behaviour and in the worst case coredumps.

Root Cause: In the call teardown processing, after the last termination is deleted, the calltracing function was accessing the already deleted call termination object to retrieve the context ID.

Steps to Replicate: 

1. Setup a call in MRFP.
2. Teardown the call.
3. The previously mentioned invalid memory access occurs in handling the deletion of the last termination of the call.

In the same function, retrieve the context ID from the context object instead to address the issue.

Workaround: None.

The RECORDING CDR does not have correct value for the SRTP field during switchover scenario.

Impact: For the SIPREC sessions with SRTP, after a switchover the "RECORDING" CDR generated had values as 0 in the SRTP statistics fields as shown below:

24.7 ingress SRTP info1 : 0:0:0:0
24.12 egress SRTP info1 : 0:0:0:0

Root Cause: The standby processing code did not support copying the SRTP information into the SIPREC Standby data blocks and as a result was lost during a switch over.

Steps to Replicate: 

1. Execute a SIPREC call, with a SIPREC Session over the SRTP.
2. Perform a switchover.
3. Verify the RECORDING CDR for SRTP Statistics in the following fields:
   
   24.7 ingress SRTP info1 : 0:0:0:0
   24.12 egress SRTP info1 : 0:0:0:0

The code is modified to process the SIPREC SRTP information is added.

Workaround: None.

The SBC disables the SRTP for an audio when the EP disables and re-enables the audio.

Impact: The SBC disables the SRTP for audio when EP disables and re-enables the audio.

Root Cause: During the modify offer processing, the SRTP value is taken from previous Active security PSP without checking if the SRTP is valid or not.

Steps to Replicate: Test Procedure:

Setup Audio+Video RTP to SRTP pass-Thru call between UAC-UAS:

1. UAC sends invite with Audio and video.
2. UAS responds with Audio and Video cryptoline.
3. UAC sends re-invite to disable Audio stream port=0 and inactive and Video with valid media port and sendrecv.
4. UAS responds 200OK with port=0, a=inactive and no crypto line for audio and video with valid port and crypto line.
5. UAC sends re-invite to add Audio stream back with valid media port and sendrecv and video with valid media port and sendrecv.
6. UAS responds 200OK with valid Audio and Video crypto lines.

Expected Result:

1. RTP-SRTP A+V call gets established.
2. The SBC disables audio stream and sets up RTP-SRTP call with video stream.
3. The SBC re-establishes Audio+Video RTP-SRTP call.

Actual Result:

1. RTP-SRTP A+V call gets established.
2. The SBC disables audio stream and sets up RTP-SRTP call with video stream.
3. The SBC re-establishes re-Invite without SRTP for audio stream.

The code is modified to copy the Active security PSP only if the SRTP admin state is enabled.

Workaround: None.

There was a SM Process leak for the MRFP call on the MRFP active.

Impact: The redundancy group manager (RGM) that is used in conjunction with N:1 deployments has a memory leak.

Root Cause: The RGM handles the switchover and fault messages in N:1 deployments and it was not freeing up the ICM message after processing that results in a memory leak.

Steps to Replicate: This issue was observed in an MRFP configuration especially when restarted instances.

The code is modified to correctly free the ICM messages.

Workaround: None

The LeakSanitizer: detected memory leaks in the MrfRmProcessTrmAllocRpyMsg.

Impact: While testing call scenarios for RTCP for T.140 Pass-through functionality, it was observed that the SCM process could leak memory for calls associated with an MRF (external media transcoder).

Root Cause: The SBC was allocated memory while processing the SDP associated with this call but was not always freeing up the memory at the end of the call.

Steps to Replicate: Run various call scenarios with MRF where the SBC is using the SIP to allocated media resources on an external media transcoder (MRF) or T-SBC.

The code is modified to correctly free all the memory allocated for the call.

Workaround: None.

The AddressSanitizer: detected heap-buffer-overflow on address 0x61a0000cb9d9 observed in the SAM Process while running OCSP feature.

Impact: When the OCSP stapling feature is enabled on the SBC and the code was processing the response it writes to unallocated memory and in the worst case this could result in process coredumps.

Root Cause: While processing OCSP response the code was allocating a memory buffer large enough to hold the response, but then incorrectly writing one byte off the end of the memory buffer while attempt to try and null terminate the string.

Steps to Replicate: Setup - UAC > Dut<>Adapter -> UAS

1. Create an OCSP profile by configuring the defaultResponder and stapling enabled.
2. Attach the OCSP profile to the TLS profile configured.
3. From Endpoint A, Intiate the TLS call with Client Hello from the SIPP UAC having OCSP parameter in it.
4. The SBC should send server hello certifciate to the user client.

The code is modified to correctly terminate the OCSP response string without writing off the end of the memory buffer allocated to hold the response.

Workaround: None.

Observed the run time error in M-SBC "runtime error: load of value 42, which is not a valid value for type 'bool'"

Impact: The M-SBC could potentially configure the wrong data path mode for a call.

Root Cause: No issues were observed while running this functionality on a regular deployment image. But while testing with ASAN, it highlighted that some of the fields used in a call resource allocation message were not always initialized correctly, which could potentially lead to unexpected behaviour e.g. SYS_ERRs, wrong datapath mode setup.

Steps to Replicate: This issue was highlighted while while running test suite related to RFC-4117 MRF Mid-Call modification Enhancement

The code is modified to correctly initialize the resource allocation request fields to avoid issues.

Workaround: None.

The contact header is not transparently passed when the Ingress and Egress had different transport types.

Impact: Contact header is not passed transparently from Ingress, when egress side has different transport type, even when the IPSP flag 'passCompleteContactHeader' is enabled on both ingress/egress Trunk Groups.

Root Cause: In API SipSgCheckAndSetContactHeaderTrasparency(), irrespective of transparency control is enabled/disabled, if the egress Sig Zone is MS Teams,
then contact header was not transparently passed to egress.

Steps to Replicate: 

1. Configure, IPSP flag 'passCompleteContactHeader to enable.
2. Attach the IPSP to both ingress/egress TG's.
3. Set Ingress transport to TCP / Egress to TL.
4. Create pathCheck profile with hostName 'sip.pstnhub.microsoft.com' and attach to the Teams side.
5. Make a successful call.

The code is modified so regardless of MS Teams zone, if the transparency control flag is enabled then pass the contact header transparently to the egress.

Workaround: None.

The one medium vulnerability found after the Qualys Scan.

Impact: A medium vulnerability found after the Qualys Scan in Jquery 3.4.1 version.

Root Cause: The jquery 3.4.1 version has security issue.

Steps to Replicate: Run a Qualys Scan and vulnerability was not shown.

The code is modified to address the issue.

Workaround: No workaround.

The Confd updateAdmin user's configuration was failing.

Impact: In cloud deployments, when a new user is created, it throws the error:
"Cannot change accountAgingState to disabled while accountRemovalState is enabled"
even though accountAgingState is set from disabled and accountRemovalState to disabled.

Root Cause: The validation logic for user creation was done along with transformation callbacks. This makes the validation logic to be order dependent.

Steps to Replicate: 

1. Launch a cloud standalone instance.
2. Create a new user with accountAgingState to disabled and accountRemovalState to disabled:
   commit fails with error "Cannot change accountAgingState to disabled while accountRemovalState is enabled"

The code is modified to make an order independent.

Workaround: None.

The LeakSanitizer: detected memory leaks in DCmRestoreAllMetaVarsToStandbyContext.

Impact: A small memory leak was observed in the SAM process while performing a switchover from standby to active box.

Root Cause: The standby instance stores information about the metavars associated with signaling ports configured on the active instance. During the conversion from standby to active, the standby data is moved into active structures but the original standby memory blocks are not freed up correctly resulting in a memory leak.

Steps to Replicate: Perform a switchover from standby to active while running a basic call, no memory leak will be observed in the SAM process.

The code is modified to correctly free the memory associated with the standby data when it gets transitioned to active to avoid the memory leak.

Workaround: None.

The SWITCHOVER ACT Records are not generated in the SBC with HA mode set as Nto1.

Impact: On an N to 1 system, the SWITCHOVER record is not written to the accounting file on the new active node after a switchover.

Root Cause: The SWITCHOVER record is sent to the ENM process before it has opened the accounting file, so the record is not written.

Steps to Replicate: 

1. Perform a switchover a system.
2. Check that the switchover record is written to the accounting file.

The SWITCHOVER record is stored and then written out when the accounting file is opened to address the issue.

Workaround: None.

The SBC fails to relay to 3xx and picks the next route when Ethe nhancedLocalRedirection and StatusCode3xx flags are enabled.

Impact: The SBC fails to relay the received 3xx to the ingress side and picks the next route when the EnhancedLocalRedirection and StatusCode3xx flags are enabled with 2 routes.

Root Cause: When both the StatusCode3xx and EnhancedLocalRedirection are enabled for the 2 routes scenario, the performCrankback is set to true which leads to this issue.

Steps to Replicate: Procedure:

1. Configure the SBC for A to B call over ERE.
2. Configure the ERE to return two routes for Initial call R1, R2.
3. Enable the flag statusCode3xx, EnhancedLocalRedirection on TG IPSP.
4. Send the INVITE from UAC.
5. Send the 300 Multiple Choice from UAS with Contact header and embedded headers.

The code is modified to ensure that performCrankback is not set to true when both the EnhancedLocalRedirection and StatusCode3xx are enabled.

Workaround: None.

The DRBD tuning is not optimal.

Impact: On the non-cloud 1:1 SBCs, due to non-optimal tuning of the DRBD, the DRBD connection between active was getting disconnected leading to full sync and high i/o on the DRBD partition.

Root Cause: Due to the peer not responding within a certain time (ko-count * timeout), ko-count feature of DRBD was disconnecting the peer leading to full sync.

Steps to Replicate: The following are the steps to test the changes:

1. Decrease the timeout value in the DRBD conf file.
2. Restart the DRBD service.
3. Resume the DRBD sync if not already enabled.
4. Check syslog, the DRBD must not get disconnection logs.

The DRBD ko-count feature is disabled on the SBC to address the issue.

Workaround: None.

The SBC start is showing some errors related to a serf file and config-version file.

Impact: Missing the file error printed to terminal.

Root Cause: The gluster setup script does not redirect output of 'cat' command.

Steps to Replicate: Run the sbxstop; sbxstart

The 'cat' command error output is redirected to NULL to fix the issue. 

Workaround: None.

The segfault in pamValidator during PM login with incorrect credentials.

Impact: The segfault in pamValidator on a failed login for locked user.

Root Cause: The pamValidator defines a conversation function that is called by pam modules to exchange values. The pam module was freeing a struct member in the first call and accessing the same freed value in another call to the conversation function that was causing segmentation fault.

Steps to Replicate: Perform following steps on any of the SBC deployment and verify that segmentation fault does not happen:

## TEST 1: Ensure success for correct username and password:

1. Encode username and password to base 64 and set as environment variables USER and PSWD example: export USER="YWRtaW4=" ; export PSWD="U29udXNAMTIz"
2. Run pamValidator and verify it returns success.

##TEST 2: Authentication Failure for username and incorrect password:

1. Encode the correct username and incorrect password to base64 and export as env variables USER and PSWD.
2. Run pamValidator and verify it returns "Authentication Failure".
3. Run pamValidator again multiple times (atleast 3 more) and verify the authentication failure and no segmentation fault.
4. Wait for 30 seconds and try TEST#1 with the correct password and verify it succeeds.

The code is modified so the pam_response struct properly in between calls to the conversation function.

Workaround: None.

While expecting a 200 OK to ingress endpoint, the SBC was sending BYE to the Egress endpoint.

Impact: If the Media inactivity/activity monitoring is enabled on media leg, and if media is not received at all on that leg in initial 10 seconds, then the NP sends a media inactive notification to up layer.

But later even if media starts coming to the SBC on that leg, the NP is not sending media is active now in notification to Application layer.

As a result, the application layers were closing the call based on the action configured for media inactivity (peerAbsenceTrapAndDisconnect).

Root Cause: In the call, if the media is preceded by no media for few seconds.
The call can be terminated if the peerAbsenceAction is peerAbsenceTrapAndDisconnect.

Steps to Replicate: 

1. Configure the peerAbsenceAction as peerAbsenceTrapAndDisconnect in the PSP.
2. After the call is established, do not stream the media for initial few seconds (this duration should be less than inactivityTimeout), start it only after few seconds.
   > set system media mediaPeerInactivity inactivityTimeout 20
   > set profiles media packetServiceProfile DEFAULT peerAbsenceAction peerAbsenceTrapAndDisconnect
3. In this case, the pause should be less than 20 seconds.
4. With the fix, the calls will not be terminated.

The code is modified to fix the Inactive to Active detection functionality and address the issue.

Workaround: If media is expected to come late, they should not configure the peerAbsenceAction action in PSP to peerAbsenceTrapAndDisconnect. The action can be selected as the peerAbsenceTrap or none.

The ASAN leaksanitizer detected errors in the CPX and SCM.

Impact: The SCM and CPX processes have a small memory leak while changing the IPSEC and IP interface group configuration.

Root Cause: While processing configuration related commands, the SBC was reading information from CDB into temporary memory blocks but failed to release the memory at the end of the configuration action, resulting in a small memory leak.

Steps to Replicate: Make the configuration changes to IPSEC and IP interface group.

The code is modified to ensure the memory is correctly freed up to avoid the leak.

Workaround: None.

The AddressSanitizer: detected heap-buffer-overflow on address 0x61b000013128 at pc 0x55e3eea0419e bp 0x7ff7a52768a0 sp 0x7ff7a5276898 in ScmProcess_0.

Impact: There was invalid memory access when the SBC receives the 500 Internal Error to REGISTER.

Root Cause: The root cause of this issue is accessing invalid memory while accessing callData, trying to read off the data from the end of memory block. It can potentially cause the coredumps if the memory block is at the edge of the accessible memory region.

Steps to Replicate: Testcase:
Description:

1. Clean up SAs if unexpected response received

Procedure:

1. Monitor the exchange.
2. Send an initial reg message.
3. Receive the 401 challenge.
4. S-CSCF responses with 500.

Expected Results:

1. Verify IPSEC SA's deleted (ip xfrm state).

The code is modified to access the respective application data (It can be call data, registration data or subscription data) based on type of SIP message.

Workaround: None.

In a secure call, the SBC does not increment the port number in R-URI after processing Refer.

Impact: In a secure call with TLS configured, if the call is REFERed with a REFER-TO header containing a FQDN and port number, the SBC sends out a new INVITE to specified FQDN and port number. If that INVITE fails and the SBC then sends a subsequent INVITE on the next route, it does not correctly increment the RURI port number for the TLS.

Root Cause: The SBC code does not take into account that a re-route after a REFER-TO with FQDN and port number target needs to increment the port number for TLS, if the target after the re-route is different to the original target specified in the REFER-TO.

Steps to Replicate:

1. With the recommended SBC configuration for MS teams with TLS enabled between the SBC and MS teams, establish a call from the PSTN to MS Teams
2. Send a REFER from MS Teams that includes following header: 
   REFER-TO: <sip:sip2.pstnhub.microsoft.com:xxxx;transport=tls>
3. Based on the REFER, the SBC should route the call and send an INVITE to sip2.pstnhub.microsoft.com using port xxxx
4. Reject this INVITE from MS Teams with a 503.
5. The SBC should then send an INVITE out using the second route to sip3.pstnhub.microsoft.com using port xxxx.
6. Complete the call signaling and verify referred call is established correctly.

The code is modified to increment the RURI port number for TLS if performing a re-route to a target FQDN and port number that is different to the original target specified in the REFER-TO.

Workaround: None.

The CpxAppProc leak for the MRFP call.

Impact: During the SBC startup, the CPX process has a small memory leak.

Root Cause: During the SBC startup processing, the CPX process reads various CDB configuration and performs DB schema upgrade validation logic. As part of this processing, it was creating temporary internal memory blocks but not releasing them at the end of initialization.

Steps to Replicate: Restart the SBC after it has been configured.

The code is modified to correctly release the temporary memory blocks used for initialization processing.

Workaround: None.

The LeakSanitizer: detected memory leaks on the active OAM.

Impact: The CPX process was leaking small amounts of memory while processing BFD configuration changes.

Root Cause: The CPX process was allocating memory in order to interact with the CDB while process BFD configuration changes. But it was not freeing up the memory at the end of the configuration action, resulting in a memory leak.

Steps to Replicate: Make configuration changes to the BFD profile.

The code is modified to correctly free the memory required to process the BFD configuration changes and avoid the memory leak.

Workaround: None.

The SBC is sending an unexpected re-INVITE to the Egress side in the SRTP early media scenario.

Impact: The SBC is sending an unexpected re-INVITE to the Egress side in the SRTP early media scenario.

Root Cause: This issue is caused as a side-effect of SBX-103807.

Steps to Replicate: Run the following call flow:

1. The UAC sends an INVITE with 100 rel required.
2. The UAS sends 18x with SDP and SRTP( SHA-1-32).
3. Ingress sends PRACK with SDP and SRTP( SHA-1-32).

Copy an active security PSP only if audio stream is present to address the issue.

Workaround: None.

Observed SIPFE MAJOR logs on the n1-standard-4 SBC_HA_HFE_SPLIT instance.

Impact: The log was printed as major, when stale calls are present and whenever we start cleaning the stale calls, then the log is seen.

Root Cause: This log is expected when clearing any stale calls.

Steps to Replicate: Run call load and if any stale calls are seen, then this log issue is seen (only when the Minor logging is enabled).

The code is modified to address the issue.

Workaround: None.

Observed an error "runtime error: index -20 out of bounds for type 'uint8_t [16]' " on UXPAD when running T140 call with out-of-order of T.140 packets.

Impact: If the T140 packet contains any T140block that has more than 16 characters, then a debug buffer to display the ASCII characters of T140 packet may write beyond its size.

Root Cause: The debug buffer to display ASCII characters of T140 packet is 16 bytes in size. It saves the 16 bytes of last T140block. However, the T140 packet SBC accepts allows a maximum T140block size of 36 characters. The code did not properly limit the size of buffer to copy to 16 characters.

Steps to Replicate: 

1. Make a SIPP call (AMRNB=>G711) with t140=>baudot enabled.
2. Send PCAP with the T140 from AMRNB termination having T140block in packet exceeding 16 bytes.
3. There may not be any observable effect, but the debug buffer that displays ASCII characters writes beyond its bounds and corrupts some other fields in the structure.

The code is modified to limit the size of ASCII buffer to copy to 16 characters.

Workaround: None.

There was a CpxAppProc leak for MRFP calls.

Impact: The small memory leak occurs on the SBC/MRFP nodes when action/status/stats under the node branch is accessed through the OAM node CLI or EMS.

Root Cause: Resource references are cleared mistakenly after serving the request from the OAM node.

Steps to Replicate: Execute a node branch command repeatedly and monitor the CPX process size on the target node.

The resource references are cleared only when the system is shutting down. The resources are now getting reused by subsequent requests to address the issue.

Workaround: Avoid using the node branch commands in automated/periodic operations. Manual use should work as the leak is small.

The usage of a kill command output in the active and standby CE_node logs.

Impact: The kill command usage gets printed in the CE_Node log file of the managed nodes.

Root Cause: No glusterfs process is present when the kill command is executed by the glusterSetup.sh script called by sbxConfigUpdater.sh.

Steps to Replicate: 

1. Reboot the Standby OAM.
2. Ensure that kill usage does not appear in the CE_Node log file of the managed nodes.

Redirect the kill command error output to /dev/null to address the issue.

Workaround: None.

There are coverity issues in the OAMNODE.

Impact: When processing a show list command under the node branch from the OAM node, if the target node fails to read the command path in the request, the code will access memory immediately after freeing it. While in most cases this should not cause issues, accessing memory after it is freed is not good behaviour and could result in unexpected behaviour, potentially causing coredumps.

Root Cause: The error handling flow in the code is incorrect. The handling code hits some code for the success flow.

Steps to Replicate: Enter the CLI commands like "show table/status node SSBC-1 <path to some list> <partial key>" and hit "tab" for auto completion on a system with an unreliable HA network. Repeat until the target node showing this error in its DBG log:

CPX ConfdProxy::worker: could not deserialize parameter for PROXY_FIND_NEXT_REQ

The code is modified to address the issue.

Workaround: None.

The BFD packet forwarded by the router is not received by the MRFP.

Impact: Once the BFD session is established on a port (either primary or secondary),
an immediate port switch over is followed due to the BFD packets dropped at NP for a brief amount of time (~1 second). This eventually triggers a node switchover.

Root Cause: A race condition in ACL lookup is the cause of this issue.

Steps to Replicate: 

1. Ensure the BFD session is up on a port (either primary or secondary).
2. Initiate a port switchover.
3. Monitor if the BFD session comes back on the switched-over port and an immediate port switch over is not followed.

The code is modified to address the issue.

Workaround: A user ACL can be added as a workaround.

For an existing call trace, when the call trace feature is disabled on the MRFP, the MRFP should reject the trace request (in MODIFY with CALLTRACE/TRACEACTIVITYREQUEST=ON) from the C3 by sending a NOTIFY to the C3 with RES=FAILURE.

Impact: When the call trace feature is disabled in the MRFP (using CLI command: set global callTrace state disabled), and the C3 tries to enable call trace using the MODIFY command with CALLTRACE/TRACEACTIVITYREQUEST=ON, then the MRFP should reject the trace request from C3 by sending NOTIFY to C3 with {CALLTRACE/TRACACT{Stream=1,RES=FAILURE}}.

However, the MRFP (9.1 R0) is not sending any NOTIFY message for the Call Trace request received in the MODIFY command.

Root Cause: The issue was seen only when the MODIFY command does not have any SDP.

Steps to Replicate: 

1. Disable the call trace feature, using the CLI command: set global callTrace state disabled.
2. Establish a MRFP call for the SBC from C3 by sending two ADD termination commands.
3. Send MOD termination command from C3 with: CALLTRACE/TRACEACTIVITYREQUEST=ON.termId ip/1/intf/3523215361{ Media { Stream = 1 { LocalControl { Mode=SendReceive,CALLTRACE/TRACEACTIVITYREQUEST=ON } } } ,Events = 1 { NT/NETFAIL , ADID/IPSTOP { DT=50 } , HANGTERM/THB { TIMERX=1800 } } ,Signals { } }
4. Since call trace is disabled at the SBC, verify that the SBC sends a NOTIFY with res=Failure after sending a MODIFY reply.

The code is modified to send a NOTIFY for TRACEACTIVITYREQUEST when the MODIFY command does not result into any change in media parameters. The MRFP now sends a NOTIFY with RES=SUCCESS/FAILURE (based on call trace configuration) after sending reply for the MODIFY command.

Workaround: None.

A SCM core dump was observed when multiple gateway TGs are created in a GW-GW call on a HA setup.

Impact: On a HA setup, the Standby box SCM Process dumps core when the Standby starts to sync from Active post a switch over and as a result, the switchover occurs after a scenario where Gateway TG's are created and then a GW TG with a lower index (created earlier) is deleted.

Example:

1. Create GWTG1, GWTG2 and GWTG3.
2. Delete GWTG2
3. Switch over.

Root Cause: The coredump is caused due to difference in indexing the gateway TG's in active and standby boxes.

The GW TG indexes were out of sync between the Active and Standby. Active SBC had holes in the indices of the GW TGs after deletion. The Standby SBC does not have holes for the GW TGs that are present post deletion and occupy a different index when compared to active.

Steps to Replicate: 

1. The HA setup with GW-GW configurations.
2. Create 3 Gateway Trunkgroups: GWTG1, GWTG2 and GWTG3.
3. Delete the GWTG2.
4. Perform a switchover.

The code is modified to ensure that the GWTG indexes on the Active and Standby are in sync.

Workaround: None,

When multiple codecs are received in descriptor, the call is getting rejected if license of the first preferred codec is not present in the license bundle.

Impact: When the MRFP receives an ADD termination command with a list of codecs in the audio stream from the MGC, it rejects call if the license of first preferred codec is not present in the license bundle, even though MRFP could have succeeded the call with other codecs on the list.

Root Cause: The MRFP's codec filtering function chooses the first allowed codec from the list and send it to media plane without check the codec license, so that the media plane returns error in case of codec license validation failure. The call failed as a result.

Steps to Replicate: 

1. Ensure the MRFP is up and running
2. Generate a license xml with ONE unit of MRFP-RTU, MRFP-DSP-RTU MRFP-DSP-AMR and ZERO units MRFP-DSP-AMRWB. Install the license bundle b1 in MRFP.
3. Place a call from endpoint with AMRWB and AMRNB in SDP in same m line
4. Validate the call should be succeeded with AMRNB codec being used.

The code is modified so it always chooses a codec with a valid license to the media plane.

Workaround: None.

The route data was lost after offline PM upgrade from 625R0 to 823A13.

Impact: The special character data (e.g. ?,*,#,$) is not getting migrated from the Oracle version to postgres version.

Root Cause: Special characters were causing issues with Postgres data loading.

Steps to Replicate: 

1. Create 100+ routes that have special characters in the DN field (Domain Name) in the SBC 6.2.5.R0.
2. Upgrade to the SBC through the PM offline upgrade to 8.0 or above.
3. After upgrade all route data was lost while other data are unaltered.

The code is modified to address the issue.

Workaround: No workaround.

The media port range threshold alarm is not triggered after a switchover.

Impact: The sonusMrfpRealmMediaPortRangeThresholdExceededNotfication alarm does not get raised on a newly active box following a switchover, even if the conditions for the alarm are met.

Root Cause: The realm status is not getting properly mirrored to the standby box.

Steps to Replicate: 

1. Run calls such that 90% of the ports are used and alarm is triggered.
2. Trigger a switchover from the active MRFP.
3. Alarm should be triggered in new active MRFP.

1. The code is modified to check the UDP port usage, and to raise the alarm if appropriate.
2. The code is modified to mirror realm status so that, it can be used to raise alarm when transitioning to active or when new call on SBY arrives.

Workaround: None.

The createConfigDrive.py --file option throws an error when executing.

Impact: In the case of KVM/VMWare deployments with qcow2/OVA/vmdk, the createCOnfigDrive.py script that creates the config-drive is not working with the '--file' option.

Root Cause: There was an error in handling the '--file' option

Steps to Replicate: Generate a configuration drive with the '--file' option.

The code is modified to address the issue.

Workaround: Use the '--cli' option to generate a configuration drive.

The createConfigDrive.py --cli throws an IndentationError.

Impact: When deploying on the KVM/VMWare using qcow2/OVA/VMDK and generating config-drive using createConfigDrive.py with '--cli' option, the script returns with an error.

Root Cause: There was an indentation issue in the Python code.

Steps to Replicate: Generate a config-drive with the '--cli' option and verified the generated config-drive.

The code is modified to address the issue.

Workaround: None.

The LeakSanitizer: detected memory leaks at confd_malloc

Impact: The ASAN detected memory leaks while processing the E164 profile configuration changes.

Root Cause: The code was allocating memory while processing E164Profile configuration changes but not releasing the memory at the end of the configuration action, resulting in a small memory leak.

Steps to Replicate: Create and modify the E164Profile configuration.

The code is modified to release the internal memory block at the end of the configuration action.

Workaround: None.

The AddressSanitizer: detected heap-use-after-free on address 0x6180000ed180 at pc 0x5619a742719d bp 0x7f3b04960310 sp 0x7f3b04960308.

Impact: While the MRFP node is shutting down, it can access memory after it has been freed, this could result in unexpected behaviour and in the worst case a coredump. But would have limited impact as it only occurs when shutting down.

Root Cause: During the sbxstop/sbxrestart or switchover because of race-condition, when the SBC is in deactivation the oamNodeRegisterRetry can access already deallocated resource leading to a core dump.

Steps to Replicate: 

1. Setup a build with HA MRFP using OAM.
2. Do the sbxrestart/switchover of an active instance.

The code is modified to handle this race condition.

Workaround: None.

The trunkgroups are not displayed while assigning the SMM profile to TG on the EMA.

Impact: The trunkgroups are not displayed while assigning the SMM profile to TG on the EMA.

Root Cause: The dropdown height is limited, and as the result the last entry is not visible.

Steps to Replicate: 

1. Create more than one TG.
2. Create a SMM profile on EMA.
3. Click on Assign SIP Adaptor Profile.
4. Under 'Assign Message Manipulation Profile to TGs', check for Input Adaptor and Output Adaptor.

All the options should be available after performing the test steps.

The code is modified to show all the options to select.

Workaround: None.

The SBC is unable to find the TG for in-dialog NOTIFY message when received from different IPs and the dialogTransparency flag.

Impact: When the indialog NOTIFY comes from different source IP, then the SBC is dropping the NOTIFY.

Root Cause: The existing design for OOD does not support the indialog NOTIFYs when it comes from a different source.

Steps to Replicate: 

1. Initiate a SUBSCRIBE dialog.
2. Send an indialog NOTIFY from a different source.

The code is modified to accept the indialog requests when the source is different.

Workaround: None.