Call Trace and Packet Capture - Call Trace

In this section:

Use the Configure Trace and Media Packet Capture window to both configure Call Trace Filters and to initiate call traces to use in debugging policy and call routing issues.

On the SBC main screen, go to Troubleshooting → Call Trace/Logs/Monitors → Call Trace and Packet Capture → Call Trace. The Configure Trace and Media Packet Capture window opens.

Configure Trace and Media Packet Capture Window:

There are two panels in this window: the Call Trace Status and Setting panel and the Call Trace Filters panel. The Call Trace Filters panel lists any existing Call Trace Filters.

Call Trace Filters define conditions that determine which call events trigger logging PES call trace messages to the system trace data event log. When a call meets the criteria, the trace information is written to a tracing file (.TRC). The following default information is collected during a call trace:

Service group configuration
DSP T.38 event information, if applicable
DSP channel statistics
DEBUG/SYS log information

Warning

Do not configure a Call Trace Filter to trace all calls. Use of the Call Trace feature can have significant impact on the network performance of the SBC system. Ribbon strongly recommends configuring Call Filters which result in a small number of calls being traced to avoid heavy loads. Also, be sure to delete Call Trace Filters once you have obtained the necessary information.

Enter values for the Call Trace Status and Settings using the following table.

Parameter	Description
Call Trace Duration	Duration in minutes that call trace is enabled. (default = 180)
Call Trace Status	Displays the current status of the Call Trace. When there is no trace running, the status is Call trace stopped with a red dot. when a trace is running, the status is Call trace running with a green dot.
Sage Fingerprint	Enable this flag to generate an audio fingerprint for the incoming calls that can aid in robocalling campaign identification. The back-end systems (such as Identity Hub) use this feature to identify robocalling campaigns and classify individual calls that belong to a robocalling campaign. Audio fingerprinting is enabled by default. A global configuration option is provided to disable the fingerprinting on the SBC. Disable – Disable audio fingerprinting to free the memory allocated to store the associated captured media. Enable (default)
Sage Tracing	Use this flag to allow a sampling of the call flows in the network using the L4 call trace. When the Sage Tracing feature is enabled, the SBC captures a small set of level 4 SIP PDU trace logs automatically in order to help debug field issues. This logging utilizes a small amount of CPU, for example, 1%. This functionality is not utilized if Level 4 SIP PDU tracing is already enabled to capture allMessages for all IPs, or the trace files are disabled in the eventLog configuration. Disable Enable (default) Note: You might need to disable the Sage Tracing to reduce the TRC output and the speed of log rotation if you are trying to collect a specific call trace filter.

Call Trace Filters

Note:

Only Administrators, Field Service, Guests and Operators have the ability to access the Call Trace configuration screen.

You must create Call Trace Filters before you can start a call trace.

The maximum number of call trace filters allowed is 100.

The Call Trace Filter object allows you to create and configure Call Trace Filters that specify criteria that the SBC uses to determine which call events to trace.

To Create a Call Trace Filter

To create a new Call Trace Filter, click New Call Filter on the Call Trace Filters panel. The Create New Call Trace Filter window opens.

Create New Call Trace Filter Window

Enter values for the Call Trace Filter object using the following table and click Save.

Call Trace Filter parameters:

Parameter	Description
Capture calls that match these filters	Specifies the criteria to use determine which calls to capture in the logs in a Call Trace (these are optional). The options are: Called number - Specifies the called party number to trace. Field length is up to 30 characters. Calling number - Specifies the calling party number to trace. Field length is up to 30 characters. Contractor number - Specifies the contractor number to trace. Field length is up to 30 characters. Redirecting number - Specifies the redirecting number to trace. Field length is up to 30 characters. CDDN number - Specifies the Called Directory Number to trace. Field length is up to 30 characters. Note: For the preceding options, the following characters are accepted by the SBC when specifying Call Trace Filter criteria: Digits 0-9 Plus (+) symbol Wildcard matching is also supported. Use "X" or "x" wildcard character to match any single digit, and "%" symbol to match any digits from that point forward. For example, using "978xxx1212" returns all calls between 9780001212 and 9789991212, and using "978%" returns all calls with a 978 prefix. Transfer capability - Specifies the transfer capability of the call trace filter. Options are: Audio31Khz - 3.1 kHz Audio - ITC 3.1 kHz audio calls are traced. The default setting is disabled. Speech - ITC Speech based calls are traced. The default setting is disabled. Unrestricted - Calls with ITC Unrestricted are traced. The default setting is disabled. Unrestricted With Tones - Unrestricted Digital Information With Tones Announcements - Calls with ITC Unrestricted Digital Information With Tones Announcements are traced. The default setting is disabled. Video - Calls with ITC Video are traced. The default setting is disabled. Trunk Group - The name of a trunk group to filter the calls to trace. This string must match the trunk group name exactly. You cannot use wildcard characters for this argument. Field length is up to 23 characters. Peer IP address - Specifies the peer IPv4 or IPv6 address for call tracing.
Capture type	Specifies the type of logs to capture while running a Call Trace. The options are: Capture trace information (.trc logs) only Capture trace information (.trc logs) and media information (.pkt logs).
Detail level to log	Specifies the level of details that need to be capture in the log file when the trace is complete. The options are: Level 1 - Everything Level 2 - Everything but raw hex dump Level 3 - Only external message information and errors Level 4 - Only SIP PDU events, and GW-GW PDUs in Level 4 trace when enabled. Note: The Level 4 log is implemented to capture SIP Ladder diagrams for the interpretation of calls. Level 4 Call Trace Filter conditions/behavior: Level 4 traces all the SIP PDU events (without logging other trace messages) at line-rate without dropping any PDUs. These traces are used by SIP Ladder feature for further analysis. The Level 4 filters are based only on Peer IP address and this must be specified to trace calls. This call trace method does not trace/filter based on trunk group, called number, calling number, and so on. Configure a Level 4 filter with a wildcard Peer IP address of 255.255.255.255 to match ALL packets to/from any IP address. Level 4 filters identify GCID when possible. However, some messages do not contain GCIDs and may have 0xfffffffff as the GCID (registration, notify, options ping, and so on). Other filters may run simultaneously as trace level 4, but ensure configuring the parameter maxTriggerCount to a non-zero value by executing the following command: `set global callTrace maxTriggerCount <1-64>` where the allowed range for `maxTriggerCount` is `0-64`. For more information, refer to Call Trace - CLI. For more information on the effect of different levels with respect to the flag `SIPRec Legs Capture`, refer to Call Trace to Capture SIPRec Legs.
Name	Specifies the name of the Call Trace Filter of up to 23 characters.
SIPRec Legs Capture	Enable this flag to trace and capture packets for SIPREC legs. Disabled (default) Enabled For more information on this flag, refer to Call Trace to Capture SIPRec Legs.
State	Specifies the state of this filter. The options are: Disabled (default) - If disabled, it is not used for a Call Trace. Enabled - If enabled, this filter can be used for a Call Trace.
Stop Match	Specifies the condition to stop matching filters. The options are: When a match occurs in this filter, stop trying to match the other filters Continue to try to match up to two other filters after a match is found in this filter (default)

Note

As a general rule, set the Call Trace Filter for the ingress side except for the Called number.

To Edit a Call Trace Filter

Select an existing Call Trace Filter object by clicking the radio button adjacent to it in the Call Trace Filters window. The Edit Call Trace Filter window is displayed.
Make changes and click Save.

To Copy a Call Trace Filter

Select an existing Call Trace Filter object by clicking the radio button adjacent to it in the Call Trace Filters window.
Click Copy Call Filter. The Copy Call Trace Filter window is displayed.
Make changes and click Save.

To Delete a Call Trace Filter

Select an existing Call Trace Filter object by clicking the radio button adjacent to it in the Call Trace Filters window.
Click the X on the right end of the selected row.
Confirm the deletion when prompted.

Call Trace Status and Settings

Once you have one or more Call Trace Filters created, call traces can be started and stopped from the top panel of the Configure Trace and Media Packet Capture window. See the earlier figure of the Configure Trace and Media Packet Capture window. There you can start, save and stop traces using the enabled filters. The trace continues running until a specified duration elapses or the specified number of call filter matches is reached.

Note:

When you start a trace, by default, the first three enabled filters in the table are used for the trace.

To Run a Call Trace

In the Call Trace Status and Settings panel, choose the required conditions for the trace using the following table.

Call Trace Status and Settings parameters:

Parameter	Description
Call Trace Duration	Specifies the duration for the Call Trace to run. The options are: Run trace until stopped (by clicking the Stop Trace button) - If selected, the trace will keep running till it is stopped manually (by clicking the Stop Trace button). Stop trace after __ minutes - If selected, add the number of minutes for the Call Trace to run. The value ranges from 1 to 360 minutes and the default value is 180 minutes. If you configure a specific value for Call Trace Duration and the SBC experiences a failover or restart, the call trace starts again automatically and runs for the specified duration.
Call Trace Status	Displays the current status of the Call Trace. When there is no trace running, the status is Call trace stopped with a red dot. when a trace is running, the status is Call trace running with a green dot.
Number of Matches (optional)	Specifies the maximum number of calls to match before stopping the trace. The value ranges from 1 to 64 times.
Sage Fingerprint	Enable this flag to generate an audio fingerprint for the incoming calls that can aid in robocalling campaign identification. The back-end systems (such as Identity Hub) use this feature to identify robocalling campaigns and classify individual calls that belong to a robocalling campaign. Audio fingerprinting is enabled by default. A global configuration option is provided to disable the fingerprinting on the SBC. Disable – Disable audio fingerprinting to free the memory allocated to store the associated captured media. Enable (default)
Trace Calls with errors of the type	Check this object to enable specifying a system-wide call trace error filter trigger criteria. By default the option is unchecked which means there is no SIP PDU logging for this error filter. After checking the option the following options are available: Any - Log all SIP PDUs that meet the requirement of the Parse Error, Out of Dialog, and Early Attempt options below. Parse Error - Log received SIP PDUs that fail parsing. If an error is detected by SIP signaling, the accompanying PDUs will not be associated with any dialog, including an existing dialog. Hence all in dialog and out of dialog PDUs are assigned to this category. If an error is detected at the front end pre-parser, the PDU is logged by this software and not forwarded to SIP signaling. This prevents an offending PDU from being logged by both the pre-parser and SIP signaling. Out of Dialog - Log incoming syntactically correct SIP PDUs (except INVITE) that do not belong to an existing dialog. Early Attempt - Logs incoming SIP PDUs that result in early termination of the call (by either the pre-parser or SIP signaling). Note The system-wide error filter triggers a call trace based on the selected type of error. If you select to trace calls with errors as well as define call filters to trace, the error filter runs in parallel with the call filters. Both may trigger producing a double-PDU output, one for the conditions matched in the call filter, and one for the detected error. For further details on error logs, see the section Call Trace Log Messages.

Click Save & Start Trace to start the trace. The trace status is displayed in the same panel.

Call Trace Logs and Monitors - Call Trace Running

The trace stops automatically if a time limit is specified, otherwise you must stop the trace manually by clicking Stop Trace.

Figure 3: Call Trace Logs and Monitors - Call Trace Stop

Once the trace is complete, the logs are generated and stored at Log Management. Look for TRC and PKT log files (.trc and .pkt log files). For more details on log messages, see the section Call Trace Log Messages.

Call Trace Log Messages

The memory available for Call Trace Log messages are limited to 1800 bytes per log entry. If a log entry occupies memory more than the 1800 bytes allocated for it, the entry is divided into two entries. The next 1800 bytes are allocated for the new log entry with the same time-stamp and sequence number. This ensures that a human or a program can parse the entries, and if necessary, can reassemble the entries with the same time-stamp and sequence number into one single chunk.

A sample log entry with memory footprint of more than 1800 bytes is illustrated below (observe the comments within /* */):

A sample log entry with memory footprint of more than 1800 bytes

242 12162016 083615.736943:1.02.00.38938.Info .SIPSG:CALL:FILTER=Cisco_Test_Call GCID=0x00000DDB: 14443. TraceCallLogger invoked - From pstProperties: PeerIP/port: 10.100.100.10/5000 LocalIP/port:10.139.160.141/5000 ulMsgType-0x40000004

/* This is a comment for illustration and not part of the actual log entry produced by the SBC. */
/* The line just below this comment indicates the start of an entry in the log file. */

1787 12162016 083615.737177:1.01.00.38939.MAJOR .SIPSG:FILTER=Cisco_Test_Call GCID=0x00000DDB: EXTRA INFO: Ingress: received msg for CallId:804ee913ccfe61c2aa581a72a000 gcid:0xDDB from IP/port:10.100.100.10/5000, Local IP/port:10.139.160.141/5000
RAW PDU:
INVITE sip:+8005555555@abc.company.com:5000 SIP/2.0
From: "AU-User-A" <sip:+8005555556@fgh.company.com>;tag=2ea08b0a-13c4-5853a77f-e2852bca-687922
To: <sip:+8005555555@ijk.company.com>
Call-ID: 804ee913ccfe61c2aa581a72a000
CSeq: 1 INVITE
Via: SIP/2.0/TCP 10.100.100.10:5000;branch=z9hG4bK-1fe0f0-5853a77f-e2852bca-53b01fb3
P-AV-Message-Id: 1_1
History-Info: <sip:+8005555555@lmn.company.com>;index=1, "8005555555" <sip:+8005555555@lmn.company.com>;index=1.1
P-Asserted-Identity: "AU-User-A" <sip:+8005555556@fgh.company.com>
Max-Breadth: 60
P-Charging-Vector: icid-value="AAS:409-910e4e801e6cf3c1a58aac0a072"
Alert-Info: <cid:internal@opq.company.com>;avaya-cm-alert-type=internal
Accept-Language: en
Supported: 100rel,histinfo,join,replaces,sdp-anat,timer
User-Agent: Avaya CM/R016x.03.0.124.0 AVAYA-SM-6.3.18.0.631804
Max-Forwards: 66
Av-Global-Session-ID: 804e0e91-3ccf-4601-b7aa-581a72a00000
P-Location: SM;origlocname="ACM4 R3 AU";origsiglocname="ACM4 R3 AU";origmedialocname="ACM4 R3 AU";termlocname="T2A R3 AU";termsiglocname="T2A R3 AU";smaccounting="true"
Session-Expires: 1800;refresher=uac
Min-SE: 1800
Allow: INVITE,ACK,OPTIONS,BYE,CANCEL,SUBSCRIBE,NOTIFY,REFER,INFO,PRACK,PUBLISH,UPDATE
Contact: <sip:+8005555556@10.100.100.10:5000;transport=tcp>
Content-Type: application/sdp
Content-Length: 253
 
v=0
o=- 8420266 0 IN IP4 10.240.42.55
s=-
c=IN IP4 10.240.42.111
b=AS:64
t=0 0
a=avf:avc=n prio=n
a=csup:avf-v0
m=audio 49080 RTP/AVP 18 127
a=rtpmap:18 G729/8000
a=rtpmap

/* This is a comment for illustration and not part of the actual log entry produced by the SBC. */
/* The line just after this comment indicates the start of a new entry in the log file, but with the same sequence number and time-stamp. */
/* It indicates that the new entry is an extension of the previous entry, after the quota of 1800 bytes of memory was utilized by the previous entry. */

148 12162016 083615.737177:1.01.00.38939.MAJOR .SIPSG:FILTER=Cisco_Test_Call :127 telephone-event/8000
a=fmtp:18 annexb=no
a=ptime:20
a=sendrecv

Messages like INVITE, in combination with SIP Headers like P-Asserted-Identity, P-Location, etc. often tends to produce log entries with memory footprint of more than 1800 bytes per entry.

Additional information about a call is indicated by the EXTRA INFO delimiter (see the example above). Currently, it comprises of the following:

Ingress/Egress indication in text
Indication if the message is received or sent
CallId
Global Call ID (gcid)
from IP/Port, to indicate the source IP and port of the message
LOCAL IP/Port, to indicate the recipient IP and port of the message
RAW PDU, which is the raw PDU in ascii text but not encrypted. The RAW PDU section is the largest in a log entry, and often wraps between log entries. Sometimes, it even spans over multiple files.

Offline Call Trace Analysis Using LX Utility

The TRC files can be analyzed offline using the LX tool, which generates a SIP ladder diagram from the TRC or DBG files. The LX tool is a Windows PC application available free of charge to all customers and partners with a license agreement, and can be downloaded as part of your software download request. For details, refer to LXDOC.

LX Main User Interface:

Space shortcuts

Page tree