Secure Remote Syslog for TCP Support

Overview

Security analytics is dependent on centralized Syslog aggregation, with the expectation that the Syslog communication is secured. The primary Syslog security threats to address are:

• Masquerade
• Modification
• Disclosure

To eliminate these threats, RFC 5425 defines a TLS Transport Mapping for Syslog. This feature supports the RFC 5425-compliant transport option in addition to the existing UDP, TCP, and RELP Syslog remote protocols.

The SBC is enhanced to:

• broadcast Syslog traffic to multiple Syslog servers
• support three Syslog servers per event log type
• offer support for the Linux logs

The SBC supports the Rsyslog method of sending event messages to the Syslog server with the following capabilities:

• Using Transport Layer Security (TLS) as a secure transport medium supported over TCP. The SBC adds the TLS  to the existing event logs and the Linux logs.
• Spooling to reduce message loss: When the connection to the Syslog server is down, the SBC spools the log entries locally to reduce message loss for all log types. The supported protocols include TCP, RELP, and TLS over TCP.

• Broadcasting to the three Syslog servers
• Sending additional Linux logs to Syslog servers: This allows for the new configuration to configure the specific /var/log/ files to transfer over Syslog. It captures the Linux session console logs and transfers it through the Syslog.
  The new Linux log option sends the Linux session console logs to the user's console real-time to any Syslog server configured. The SBC writes the new console logs real-time to the newly created files in a new directory on the SBC: /var/log/session/session.* files and pushes it to the Syslog Server.
  
  The Linux logs include the following:
  • platformAuditLog - Platform Linux Audit log messages  
  • consoleLog - Console activity messages
  • sftpLog - internal-sftp messages
  • kernLog - kernel messages
  • userLog - user-level messages
  • daemonLog - system daemon messages
  • authLog - auth and authpriv - security/authorization messages
  • syslogLog - internally generated by syslog messages
  • ntpLog - NTP subsystem messages
  • cronLog - clock daemon messages
  • fipsLog - fips messages
• Validating IPv4/IPv6 to the Rsyslog remote host fields to prevent adding the invalid format IP address in the Remote Host Server fields.

Supporting TLS for Rsyslog

The TLS is a new transport medium that uses Rsyslog service and ensures secure communication between the SBC (acting as a client to the Syslog server) and the Syslog server.

The SBC sends the system, debug, trace, accounting, security, audit, and the Linux logs, (including the existing Platform Audit logs) to the remote Syslog server securely using TLS over TCP.

To support this service, the SBC provisions a new TLS protocol type and the required certificate details, which is summarized as follows:

• Provisioning of Syslog server CA Certificate details and Client Certificates. The user can create and store the certificates using the PKI interface
• Provisioning of Rsyslog TLS profile
• Provisioning of the remote host TLS protocol type

The SBC acts as a TLS Client and establishes a TLS connection towards the Syslog server. The SBC and Syslog server authenticate each other using certificates.

Certificate Provisioning

The SBC requires certificates for TLS communication using Rsyslog. The SBC adds these generated certificates to the rsyslog.conf file to allow communication through TLS over TCP or RELP.

The SBC can either create Certificates internally or import the externally created certificates.

The configuration of Rsyslog service to use TLS on each server requires three certificates: the CA certificate, the server certificate and the server key set in rsyslog.conf

The SBC creates the certificates by generating the RSA key pairs and Certificate Signing Request (CSR) locally on the SBC, export the CSR and import the signed certificate generated by the CA.

Alternatively, the SBC generates the RSA key pair and CSR externally, then import the private key and the signed certificate to the SBC.

SBC-provisioned Certificates

The SBC generates certificates using the PKI interface, and after copying it, creates new commands to generate specific certificates to support TLS using Rsyslog.

To generate the PKI certificate, refer to Generating PKI Certificates

This interface simplifies the certificates and keys managing process and provides more security since the private key never leaves the SBC system.

Generating Certificates for TLS

To enable Rsyslog communication over TLS:

• Create the client certificate and server CA certificates in a certification store using PKI interface provided in the system/security.
• After creating the certificates, create Rsyslog TLS profile.
• After creating Rsyslog TLS profile,  configure syslogRemoteProtocol as tls-tcp.

To generate the certificate using Rsyslog for TLS:

1. The SBC application generates and install RSA key pairs and generate a Certificate Signing Request (CSR) on the SBC.

1. Create a configuration object to store the locally generated RSA Key Pair:

   set system security pki certificate <certName> type local-internal

2. Generate Key pair and CSR for submission to a Certificate Authority (CA):

   request system security pki certificate <certName> generateCSR csrSub <csrSub> keySize <keySize>

2. Contact the Certificate Authority (CA) to request certificate using the generated CSR. This is external to the SBC.
   
   1. Copy the generated CSR into a file named csrToBeSigned.csr
   2. Generate the local Certificate as localCert.pem and the remote Certificate as rootCA.der. Using openssl commands on the CA run the following:

   1. Generate the rootCA.key

      openssl genrsa -out rootCA.key 2048 

   2. Generate RSA private key for rootCA.key

      openssl genrsa -des3 -out rootCA.key 2048

   3. Generate rootCA.pem file:

      openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.pem

   4. Generate the localCert.pem file:

      openssl x509 -req -in csrToBeSigned.csr -CA rootCA.pem -CAkey rootCA.key
       -CAcreateserial -out localCert.pem -days <no_days> -sha256

   5. Generate the rootCA.der file from the localCert.pem file

      openssl x509 -outform der -in localCert.pem -out rootCA.der

Once Certificate Authority issues the certificates, place the certificate in the SBC directory: /opt/sonus/external and install the certificates on the SBC using the following commands:

set system security pki certificate <loal_cert_name> fileName <local_pem_filename> state enabled
set system security pki certificate <remote_cert_name> fileName <remote_der_filename> type remote state enabled

The following three Certificates are added to the rsyslog.conf file to support TLS communication, these are generated from the existing local and remote Certificates already copied to the SBC:

• <machine-cert>.pem
• <machine-key>.key
• <ca-cert>.pem

Externally-provisioned Certificates

Externally-provisioned certificates follows the same steps as mentioned in the SBC-provisioned Certificates. The only difference being that Step one is configured on an external Server to the SBC

Support to Broadcast to Three Syslog Servers

The ACL rules  support the number of acknowledged messages received to the SBC. The ACK message count is predominantly dependent on the SBC platform as a result of the CPS that SBCs can support. The ACL rule has a credit Rate set to 1200 packets per second, which limits the number of messages that the SBC can receive.

The following example describes the worst case Credit Rate required, on an SBC7000 which supports 1350 cps

1350 cps * 2 call detail records (start/stop) * 3 Servers = 8100 pkt/sec + additional logs.

This results in an approximate credit rate of up to 10,000 pkts/sec in the worst case scenario.

The credit rate is calculated as: Calculated SBCs CPS * 2 (start/stop record) * 3 (remote servers) + approx. 25% for all other logs types.

Support for Spooling

The SBC supports spooling for all log types, including event logs and the Linux Logs only. It provides a buffer for all log types on a local SBC client.

 The SBC spooling support is summarized as follows:

• When the SBC loses the connection to the remote Server, it uses the allocated buffer.
• Each log type is allocated a 50 MB buffer per server configured.
• The rsyslog.conf file is updated appropriately to support spooling for all log types.
• A new secure spooling directory is created, with root only access, to buffer the log messages.
  • The spool files are plain text and not encrypted, but only readable by root.
  • The spool files use the SBC's common partition and write to the path /home/log/spool. Spooling does not use the DRBD partition, and there is no need to replicate the data using DRBD as Syslog service is always running to send out logs even when the SBC is standby.
• When the connection is re-established, the SBC clears the buffer, and the message logging continues as usual.

Spooling reduces message loss but does not guarantee the server receives all the messages. However, when re-establishing the connection to the server, the SBC sends some duplicate messages to the server when the spooling is configured.

For spooling, the SBC detects the connection status using Connection Orientated protocol types TCP and RELP.

The SBC supports spooling for the following protocol types:

• TCP
• RELP
• TLS over TCP
  
  

Spooling is not supported for UDP because the Rsyslog service cannot detect, if the connection is down.

To keep the amount of message loss to a minimum, use the Spooling feature to configure the latest version of Rsyslog on the SBC. The SBC supports the following new and updated packages/libraries which indicate the minimum package/library versions required. 

New/Updated packages are:

• rsyslog_8.40.0-1~bpo9+1_amd64.deb
• rsyslog-gnutls_8.40.0-1~bpo9+1_amd64.deb
• rsyslog-relp_8.40.0-1~bpo9+1_amd64.deb

Data sent to Management Port

The Rsyslog service sends its messages to the Syslog servers on the management interface. With the introduction of three servers for the Rsyslog service, the amount of data the SBC sends on the Management Port increases. However, the Management Ports for each SBC platform have physical limitations.

The worst case scenario message size that can be sent on an SBC 52x0:

450 cps * 2 call detail records (start/stop) * approx. 2k per record * 3 Servers = 5.4 MB/sec + additional logs

This does not include the additional ACKs received for TCP and RELP.

The following table indicates for each SBC platform the potential data usage on the Management port for the Rsyslog service configured with the maximum three servers:

Figure 1: Data sent to Management Port