7.4. Logs

Observability Framework stores the log messages generated by Connext applications in third-party backends (for example, Grafana Loki).

When a Connext application starts, it may generate log messages before Monitoring Library 2.0 is loaded and enabled. Any such log messages are not stored by Observability Framework.

Each log message is divided into six parts to facilitate analysis, as illustrated in Table 7.22.

Table 7.22 Log message components
Component	Description	Example	Always present
Timestamp	The time the log message was generated.	`[2023-02-02 21:32:38.049836]`	Yes
Syslog Facility	The facility (MIDDLEWARE, SECURITY_EVENT, SERVICE, or USER) and the sequence number of the log message for the facility.	`MIDDLEWARE(sn: 123)`	Yes
Syslog Level	The severity (EMERGENCY, ALERT, CRITICAL, ERROR, WARNING, NOTICE, INFORMATIONAL, or DEBUG).	`WARNING`	Yes
Activity Context	The DDS context in which the log was generated. See Activity Context.	`[010105A0.81551B17.4AA10C9B.80000007{Entity=DR,MessageKind=DATA}\|RECEIVE FROM 0101C41F.40A68B3A.C9442BC5.8000A502]`	No
Message	The message contents.	`PRESCstReaderCollator_isNewerSample:[Topic: 'Temperature', Type: 'Temperature'] Dropped sample from DataWriter (0101C41F.40A68B3A.C9442BC5.8000A502). The source timestamp (2023-02-02 21:32:40.049765) is greater than the received timestamp (2023-02-02 21:32:38.049820) by more than the source_timestamp_tolerance. The system clocks for the DataWriter and DataReader may not be synchronized.`	Yes

7.4.1. Syslog Levels and Facilities

All the log messages generated by the Connext applications are members of a Syslog facility. Syslog facilities are numerical codes that represent the source of the log message, allowing the system or network administrator to categorize and filter log messages based on their origin. These facilities help organize log data, making it easier to manage and analyze.

This release supports the following Syslog facilities:

MIDDLEWARE (23): Messages generated by the Connext middleware.
SECURITY_EVENT (10): Security-related messages generated by the Security Plugins Logging Plugin.
SERVICE (22): Messages generated by infrastructure services, such as Routing Service.
USER (1): Messages generated by the Connext logger APIs that log user messages.

The Syslog facility is always present in the log message, and it is followed by a sequence number that uniquely identifies the log message within the facility. The sequence number is useful for tracking the order of log messages within the same facility and for identifying missing log messages.

The available Syslog levels are: EMERGENCY, CRITICAL, ALERT, ERROR, WARNING, NOTICE, INFORMATIONAL, or DEBUG.

For SECURITY_EVENT and USER facilities, you can get messages with any Syslog level. However, for the other facilities (MIDDLEWARE and SERVICE), the Syslog level of the message is determined by translating the Connext builtin logging level associated with the message to the Syslog level.

The mapping between Connext’s builtin logging levels (NDDS_Config_LogLevel) and Syslog Levels (NDDS_Config_SysLogLevel) is as follows:

Table 7.23 Log Level Mapping
NDDS_Config_LogLevel	Syslog Level	Minimum Syslog Verbosity that lets the message pass through
NDDS_CONFIG_LOG_LEVEL_FATAL_ERROR	EMERGENCY (1)	ERROR
NDDS_CONFIG_LOG_LEVEL_ERROR	ERROR (15)	ERROR
NDDS_CONFIG_LOG_LEVEL_WARNING	WARNING (31)	WARNING
NDDS_CONFIG_LOG_LEVEL_STATUS_LOCAL	INFORMATIONAL (127)	INFORMATIONAL
NDDS_CONFIG_LOG_LEVEL_STATUS_REMOTE	INFORMATIONAL (127)	INFORMATIONAL
NDDS_CONFIG_LOG_LEVEL_DEBUG	DEBUG (255)	DEBUG

For additional information on Syslog levels and facilities, see Configuring Connext Logging in the RTI Connext Core Libraries User’s Manual.

7.4.2. Activity Context

The Activity Context provides context for the log message associated with it. The information provided by the Activity Context includes a sequence of activities and resources to which the activities apply. Comparing the Activity Context to traces and spans in OpenTelemetry, you can think of the Activity Context as a trace and the individual activities within the Activity Context as spans within the trace. For additional information on the Activity Context, see Format of Logged Messages in the RTI Connext Core Libraries User’s Manual.

The Activity Context is available by default in all log messages generated by a Connext application. However, you can disable this information by using the APIs in the C language binding: see NDDS_Config_Logger_set_print_format and NDDS_Config_Logger_set_print_format_by_log_level. The same APIs are available in other language bindings.

7.4.3. Log Labels

As with metrics, logs also have an associated set of labels. In Grafana Loki, labels are key/value pairs that act as metadata to describe a log stream. The combination of every label key and value defines a log stream. If just one label value changes, this creates a new stream. Refer to the official Grafana Loki documentation for further details about labels.

Table 7.24 describes the log labels generated by Connext applications.

Table 7.24 Log Labels
Loki Label Name	Description
`job`	The source of the log message. This label is useful when multiple system resources share the same Loki instance. For Connext applications, the `job` label is always `connext_logger`.
`resource_guid`	A GUID that identifies the specific Connext application that generated the log message.
`category`	The logging category of the message. For the MIDDLEWARE and SERVICE facilities, the category logically classifies messages with the same meaning across facilities. Valid values are `Discovery`, `Security`, or a combination of both separated by a comma (for example, `Discovery,Security`). For the SECURITY_EVENT facility, the category identifies the security plugin class that generated the message. Valid values are `Auth`, `Access`, `Crypto`, or `Logging`. The USER facility does not support categories. If the logging category is not available for a log message of any facility, its value is `N/A`. For example, this could happen for MIDDLEWARE log messages that are not related to discovery or security.
`plugin_class`	Only for messages with SECURITY_EVENT facility. The standard plugin class name that originated the message, as defined in the OMG ‘DDS Security’ specification, version 1.2. Here, the `category` label is just a more friendly name for these standard plugin class names. Table 7.25 shows the relationship between the plugin class name and the category.

Table 7.25 Relationship between plugin_class and category
Standard plugin class names	Associated category
`DDS:Auth:PKI-DH` `DDS:Auth:PSK` `RTI:Auth`	`Auth`
`DDS:Access:Permissions` `DDS:Access:PSK` `RTI:Access`	`Access`
`DDS:Crypto:AES-GCM-GMAC` `DDS:Crypto:PSK` `RTI:Crypto`	`Crypto`
`DDS:Logging:DDS_LogTopic` `RTI:Logging`	`Logging`
`RTI:Common`	`N/A`

Following are a few examples of how you could use log labels in Grafana Loki:

Use the resource_guid label to query all the log messages generated by a specific Connext application.
Use the category label to query all the log messages related to discovery.
Use the plugin_class label to query all the log messages related to authentication logged by the Security Plugins (RTI:Auth plugin class).

7.4.4. Collection and Forwarding Verbosity

Monitoring Library 2.0 has two verbosity settings:

Collection verbosity controls the level of log messages an application generates.
Forwarding verbosity controls the level of log messages an application forwards to the Observability Collector Service (making the messages visible in the dashboard).

By default, Monitoring Library 2.0 only forwards error and warning log messages, even if the applications generate more verbose logging. Forwarding messages at a higher verbosity for all applications may saturate the network and the different Observability Framework components, such as Observability Collector Service and the logging aggregation backend (for example, Grafana Loki).

Both the collection and forwarding verbosity can be set locally by changing the configuration of a Connext application or remotely by sending a command.

7.4.4.1. Changing Verbosity Levels Locally

The collection level can be changed locally using the NDDS_Config_Logger_set_verbosity_by_category and NDDS_Config_Logger_set_verbosity APIs for C or equivalent in other languages. You can also use the logging XML tag under participant_factory_qos. The collection level can be changed at any time.

The forwarding level can be changed per facility using the participant_factory_qos.monitoring.telemetry_data.logs.<facility>_forwarding_level field in the MONITORING QosPolicy (DDS Extension). This QoS policy can be configured programmatically or via XML. When set programmatically using QoS, the forwarding level must be changed before the Monitoring Library 2.0 is enabled.

7.4.4.2. Changing Verbosity Levels Remotely

The collection and forwarding levels can both be changed remotely. There are two methods available to send remote commands:

Using Observability Dashboards, as described in Change the Application Logging Verbosity
Using the Collector Service REST API as described in the Collector Service REST API Reference.

The Observability Dashboards only allow setting the following Syslog levels: ERROR, WARNING, INFORMATIONAL, and DEBUG. The Collector Service REST API allows you to set all the Syslog levels.