Page History
8a00a02355cd1c2f0155cd26cb8305e9
-include
Observability
PageWithExcerpt SBC Cloud-Native edition - SBC CNe
System and Application Performance Metrics
such as
and
for example, the
Trunkgroup
and
.
for example, the
we can have
such as time series format.
| Panel | ||||
|---|---|---|---|---|
In this section:
|
| Multiexcerpt |
|---|
|
|
Logging
The
logs from all microservices/containers are streamed to the Observability backend. Some examples of Observability backends for centralized logging are EFK, Kafka, and so on. The logs are streamed to the Observability backend in a uniform format. In CNF deployments, a new format is introduced for some of the logs, as explained below.Spacevars 0 product4
Old Format
| Code Block |
|---|
Format: Size, filter flag, month, day, year, hour, minute, second, tenths of seconds, shelf, slot, instance, sequence number, level, subsystem, trace type, trace name, event text.
For example: "206 05022023 131215.414913:1.01.00.00004.MAJOR .SM: *HwModuleServer::procNodeCeProductCapability: serverName: vsbc1, capName: SPS100, capValue: not present, checkType: Minimum Required ActualCeName vsbc1" |
New Format
| Code Block |
|---|
Format: year, month, day, hour, minute, second, tenths of seconds, time zone, level, Size, shelf, slot, instance, sequence number, trace type, subsystem, trace name, event text.
For example: "2023-05-02 11:52:12,367076 UTC MAJOR 132 1.01.00.00001 .SM: *ConfigManager::configUpdater: ALARM CLEARED - Able to process config updates" |
For backward compatibility of the logging format, a CLI configuration is added for the debug, system and security logs. You can switch the files of these log formats at runtime using the following CLI commands.
OverviewAll CNF applications produce logs and metrics that are essential to understanding their state and health. Because pods are typically ephemeral, it is essential to get this data to a centralized observability backend. The Ribbon CNFs support EFK and Kafka as observability backends for centralized logging, and Prometheus as the metrics logging backend. The interaction with the backends is typically through an integrated telemetry agent. |
| Multiexcerpt | |||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||||||||||
LoggingThe CNF framework logs of all the microservices/containers are directly streamed to an Elastic backend when Elastic is configured and streamed using RAMP when KAFKA is configured. When a pod restarts, crashes, or is evicted, any logs that are locally stored in an ephemeral file system are lost. The Observability framework streams the logs to the backend to ensure they remain available and are easily searchable. The new CNF logging format adheres to the Elastic Common Schema that allows you to interoperate with storage backends that handle structured data. Below is an example of the new format. Additional details are provided upon request. Supported logging mechanisms include:
Old Format
New Format
Configuring Logs in HelmThe following configuration example in the Helm configures the Observability backend server for logging.
|
| Multiexcerpt | ||||
|---|---|---|---|---|
| ||||
SBC CNe Logging Backwards CompatibilityFor | ||||
| Code Block | ||||
%
|
Some points to note:
| ||||
|
| Multiexcerpt | ||
|---|---|---|
| ||
Ribbon CNF MetricsAs part of the |
| Spacevars | ||
|---|---|---|
|
Ribbon CNF solution, the system metrics ( |
e.g., CPU, Memory, |
Disk read/write, etc.) and application performance metrics ( |
e.g., number of calls per |
trunk group, active calls, |
attempted calls, etc.) are collected and monitored using Prometheus |
The metrics are sent in the Prometheus format. In the Observability backend ( |
e.g., Grafana dashboard), |
multiple queries are available to display these metrics in a graphical format |
CNe Prometheus Metrics - Call Traffic (Example)
. Once the metrics are stored in a time-series database such as Prometheus, you can display these metrics. Ribbon provides the Grafana dashboard templates with the solution to display compelling metrics. Metrics-based AlertsYou may optionally install the provided metrics-based alerting rules using the Prometheus query language (promql) directly through the Helm chart if using the prometheus-operator. |
SBC CNe System and Application Performance Metrics
Interval Statistics
The interval statistics files are generated by I-SBC/SLB/CS the isbc/slb/cs pods at the configured time interval. The files are shared with the OAM pod Pod via PVC.
- The OAM Pod aggregates the statistics files to provide a consolidated statistics pm files to the RAMP
- The OAM pod Pod populates the DB to get the performance statistics per pod and aggregated statistics. With this, we can see the pod wise interval statistics and aggregated statistics via CLI.
- The individual pod level interval statistics are also streamed to observability backend.
In a VNF environmentVNF environment, managed pods used to connect to an EMS and stream intervals statistics individually to the EMS, where as whereas in a CNF environment, only OAM is connected to the RAMP and OAM streams aggregated/consolidated interval statistics from all pods to the RAMP. All the existing performance statistics are supported with some of the statistics being replaced with the new statistics.
New CNe equivalent CNF-equivalent commands are introduced if the SBC CNe cannot aggregate statistics data from multiple pods can't be aggregated for the following reasons. A new key cnfPodName has been added in the cnf CNF-equivalent commands.
- If one or more non integer type fields like Ip Address, Time, Average data are present.)
- If aggregation is possible but aggregated data will not add value as the statistics command is meant to reflect pod specific data (like memory and /CPU utililization utility information).
For above cases mentioned,
The list of newly introduced stats in CNF deployment is captured in the table below:
Existing versus New CNF Stats:
Existing StatsName | CNF StatsName |
|---|---|
IpAclOverallStats | |
Exisiting Stats Name | CNF Stats Name |
IpAclOverallStats | CnfIpAclOverallStats |
IpAclRuleStats | CnfIpAclRuleStats |
IpGeneralGroupStats | CnfIpGeneralGroupStats |
IpPolicingAclOffendersListIntStats | CnfIpPolicingAclOffendersListIntStats |
IpPolicingAggregateOffendersIntStats | CnfIpPolicingAggregateOffendersIntStats |
IpPolicingArpOffendersListIntStats | CnfIpPolicingArpOffendersListIntStats |
IpPolicingBadEtherIpHdrOffendersIntStats | CnfIpPolicingBadEtherIpHdrOffendersIntStats |
IpPolicingDiscardRuleOffendersIntStats | CnfIpPolicingDiscardRuleOffendersIntStats |
IpPolicingIpSecDecryptOffendersIntStats | CnfIpPolicingIpSecDecryptOffendersIntStats |
IpPolicingMediaOffendersIntStats | CnfIpPolicingMediaOffendersIntStats |
IpPolicingRogueMediaIntStats | CnfIpPolicingRogueMediaIntStats |
IpPolicingSrtpDecryptOffendersIntStats | CnfIpPolicingSrtpDecryptOffendersIntStats |
IpPolicingSystemIntStats | CnfIpPolicingSystemIntStats |
IpPolicinguFlowOffendersListIntStats | CnfIpPolicinguFlowOffendersListIntStats |
LinkDetectionGroupStats | CnfLinkDetectionGroupStats |
SysCpuUtilIntStatsSts | CnfSysCpuUtilIntStatsSts |
SysMemoryUtilIntStatsSts | CnfSysMemoryUtilIntStatsSts |
SystemCongestionIntervalStats | CnfSystemCongestionIntervalStats |
TcpGeneralGroupStats | CnfTcpGeneralGroupStats |
DiamNodeRfIntervalStatistics | CnfDiamNodeRfIntervalStatistics |
EthernetPortMgmtStats | CnfEthernetPortMgmtStats |
EthernetPortPacketStats | CnfEthernetPortPacketStats |
IcmpGeneralGroupStats | CnfIcmpGeneralGroupStats |
sipOcsCallIntervalStatistics | cnfSipOcsCallIntervalStatistics |
The CLI view of one of the CNe equivalent commands, cnfSipOcsCallIntervalStatistics, where pod level data is displayed.Example CNF-equivalent CLI commands to display pod-level data using , cnfSipOcsCallIntervalStatistics:
| Code Block |
|---|
admin@vsbc1> show status service SC podName ALL addressContext default zone PR_ZONE_INGRESS cnfSipOcsCallIntervalStatistics 24
Possible completions:
displaylevel - Depth to show
prupgrade-sc-8695fcdc64-jd8xp - This object indicates the PodName.
prupgrade-sc-8695fcdc64-mh42j - This object indicates the PodName.
prupgrade-sc-8695fcdc64-mshm7 - This object indicates the PodName.
Possible match completions:
attemptedCalls - Current Attempted ocs Call statistics.
establishedCalls - Current Established ocs Call statistics.
failedCalls - Current Failed ocs Call statistics.
intervalValid - The member indicating the validity of the interval.
pendingCalls - Current Pending ocs Call statistics.
rejectedCalls - Current SBX Rejected ocs Call statistics.
relayedCalls - Current Realyed ocs Invite to Engress side statistics.
successfulCalls - Current Successful ocs Call statistics.
time - The system up time when the interval statisitic is collected.
admin@vsbc1> show status service SC podName ALL addressContext default zone PR_ZONE_INGRESS cnfSipOcsCallIntervalStatistics 24 prupgrade-sc-8695fcdc64-jd8xp
cnfSipOcsCallIntervalStatistics 24 prupgrade-sc-8695fcdc64-jd8xp PR_INGRESS_TG {
intervalValid true;
time 1683091200;
attemptedCalls 0;
relayedCalls 0;
establishedCalls 0;
successfulCalls 0;
failedCalls 0;
pendingCalls 0;
rejectedCalls 0;
}
cnfSipOcsCallIntervalStatistics 24 prupgrade-sc-8695fcdc64-jd8xp PR_INGRESS_TG1 {
intervalValid true;
time 1683091200;
attemptedCalls 0;
relayedCalls 0;
establishedCalls 0;
successfulCalls 0;
failedCalls 0;
pendingCalls 0;
rejectedCalls 0;
} |
CLI example that shows of callCountIntervalStatistics aggregated info of callCountIntervalStatistics:
| Code Block |
|---|
admin@vsbc1> show table service SC podName ALL global callCountIntervalStatistics ENHANCED AMRNB AMRWB EVRC NICE MRF SIP EV INTERVAL CALL ENCRYPT SRTP VIDEO LEG LEG LEG REC SESSIONS REC TRANSCODE PDCS LE NUMBER NAME VALID TIME COUNT COUNT COUNT COUNT COUNT COUNT COUNT COUNT COUNT COUNT COUNT COUNT CO --------------------------------------------------------------------------------------------------------------------------------------- 55 entry true 1683100500 0 0 0 0 0 0 0 0 0 0 0 0 0 56 entry true 1683100800 0 0 0 0 0 0 0 0 0 0 0 0 0 57 entry true 1683101100 0 0 0 0 0 0 0 0 0 0 0 0 0 58 entry true 1683101400 0 0 0 0 0 0 0 0 0 0 0 0 0 |
The data view from RAMP for some of the interval statistics is depicted below.
Example 1: CnfSysCpuUtilIntStatsSts streamed by OAM, where pod level view is retained as data aggregation is not possible.
Example 2: CallCountIntervalStats streamed by OAM where data from all pods are aggregated.
Current Statistics
0 0 0 |
Current Statistics
In a CNF environment, the service level statistics from all SC Pods are aggregated at the OAM and presented In CNe environment, under service level the statistics from all the SC pods are aggregated at OAM and are presented to the CLI.
Under A service-level, clusterwise CurrentStatistics can be seen like belowcluster-wide Current Statistics example is shown below :
| Code Block |
|---|
admin@vsbc1> show table service SC podName ALL global callCountCurrentStatistics ENHANCED AMRNB AMRWB EVRC NICE MRF SIP EVS SILK SLB CALL ENCRYPT SRTP VIDEO LEG LEG LEG REC SESSIONS REC TRANSCODE PDCS LEG LEG LICENSE SESSIONS NAME COUNT COUNT COUNT COUNT COUNT COUNT COUNT COUNT COUNT COUNT COUNT COUNT COUNT COUNT MODE COUNT --------------------------------------------------------------------------------------------------------------------------------------- entry 36129 0 0 0 0 0 0 0 0 0 0 0 0 0 domain 36129 [ok][2023-05-03 16:35:07] admin@vsbc1> |
Pod-level view of current Statisticsstatistics:
| Code Block |
|---|
admin@vsbc1> show table service SC podName npbasedtones-sc-86647c7d94-djg2s global callCountCurrentStatistics ENHANCED AMRNB AMRWB EVRC NICE MRF SIP EVS SILK SLB CALL ENCRYPT SRTP VIDEO LEG LEG LEG REC SESSIONS REC TRANSCODE PDCS LEG LEG LICENSE SESSIONS NAME COUNT COUNT COUNT COUNT COUNT COUNT COUNT COUNT COUNT COUNT COUNT COUNT COUNT COUNT MODE COUNT --------------------------------------------------------------------------------------------------------------------------------------- entry 12173 0 0 0 0 0 0 0 0 0 0 0 0 0 domain 12173 [ok][2023-05-03 16:36:18] admin@vsbc1> |
Status Commands
Under service level, new a key has been is added to all status commands to prevent data aggregation as since CNF status commands details are meant to reflect pod-specific data. All the root level commands remains same.
| Code Block |
|---|
admin@vsbc1> show table service SC podName ALL addressContext default zone MR_ZONE_INGRESS trunkGroupQoeStatus
INBOUND OUTBOUND
RFACTOR INBOUND RFACTOR OUTBOUND
INBOUND NUM RFACTOR OUTBOUND NUM RFACTOR
RFACTOR CRITICAL NUM MAJOR RFACTOR CRITICAL NUM MAJOR
INBOUND FROM THRESHOLD THRESHOLD OUTBOUND FROM THRESHOLD THRESHOLD CURR
POD NAME NAME RFACTOR SBXBOOT BREACHED BREACHED RFACTOR SBXBOOT BREACHED BREACHED ASR
---------------------------------------------------------------------------------------------------------------------------------------
npbasedtones-sc-86647c7d94-djg2s MR_TG_INGRESS 94 94 0 0 94 94 0 0 90
npbasedtones-sc-86647c7d94-plxp6 MR_TG_INGRESS 94 94 0 0 94 94 0 0 90
npbasedtones-sc-86647c7d94-xf625 MR_TG_INGRESS 94-xf625 MR_TG_INGRESS 94 94 0 0 94 94 0 0 94 0 0 90
[ok][2023-05-03 17:02:06]
admin@vsbc1> |
Action commands
Under service level, the podName object is added to the action command response to differentiate responses from multiple pods.
Action commands at the local level (For example, oam, system, addressContext, etc.) are unchanged.
| Code Block |
|---|
admin@vsbc1> request service SC podName ALL oam eventLog typeAdmin debug rolloverLogNow response { podname npbasedtones-sc-86647c7d94-plxp6 94result success reason } response { 94 podname npbasedtones-sc-86647c7d94-xf625 result 0success reason } response { 0podname npbasedtones-sc-86647c7d94-djg2s result success 90reason } [ok][2023-05-03 17:0237:0652] admin@vsbc1> |
Action commands
Under Service level, podname has been added to action command response to differentiate responses from multiple pods.
Action commands at local level (as in the OAM, system, addressContext, etc.) remains same.
| Code Block |
|---|
admin@vsbc1> request service SC podName ALL oam eventLog typeAdmin debug rolloverLogNow
response {
podname npbasedtones-sc-86647c7d94-plxp6
result success
reason
}
response {
podname npbasedtones-sc-86647c7d94-xf625
result success
reason
}
response {
podname npbasedtones-sc-86647c7d94-djg2s
result success
reason
}
[ok][2023-05-03 17:37:52] |
| Note |
|---|
All CLI commands for unsupported features are hidden in a CNF deployment. |
| Multiexcerpt | |||||
|---|---|---|---|---|---|
| |||||
CNF Traps/AlarmsThe CNF traps from all pods are consolidated on the OAM Pod and streamed to RAMP's Observability backend. The traps include Kubernetes-specific details with the addition of the varbinds Use the following CLI command to view the alarms in the OAM Pod. In this example, the pod name (
| |||||
| Note | |||||
| All CLI commands for unsupported features mentioned in SBC CNe Observability section has been hidden in CNF deployment. |
Overview
Content Tools