7. Observability Library

Connext Observability Library is one component of the Connext Observability Framework. It allows collecting and distributing telemetry data (metrics and logs) associated with the resources created by a DDS application. These observable resources are DomainParticipants, Publishers, Subscribers, DataWriters, DataReaders, Topics, and applications (refer to Resources.) The library also accepts remote commands (via dashboards) to change the set of distributed telemetry data at runtime.

The data distributed by Observability Library is sent to a Collector Service instance. Collector Service forwards the data to other Collector Service instances, or stores it to a third-party observability backend such as Prometheus or Grafana Loki.

Observability Library is a separate library (rtimonitoring2) and applications can use it in three different modes:

  • Dynamically loaded: This is the default mode, which requires that the rtimonitoring2 shared library is in the library search path.

  • Dynamic Linking: The application is linked with the rtimonitoring2 shared library.

  • Static Linking: The application is linked with the rtimonitoring2 static library.

The last two modes require calling the API RTI_Monitoring_initialize in your application before any other Connext APIs. This API is defined in the header file ndds/monitoring/monitoring_monitoringClass.h.

Dynamic and static linking are only supported in C and C++ applications.

To enable usage of Observability Library and to configure its behavior, you have to use the MONITORING QosPolicy (DDS Extension) on the DomainParticipantFactory and set participant_factory_qos.monitoring.enable to true. This QoS policy can be configured programmatically or via XML.

Observability Library uses three different built-in Topics to distribute telemetry data to Observability Collector Service:

  • Periodic: A best-effort Topic for distributing periodic metric data (for example, dds_datawriter_protocol_pushed_samples_total). The data is sent periodically, with a configurable period.

  • Event: A reliable Topic for distributing event metric data (for example, dds_datawriter_liveliness_lost_total). The data is sent when it changes.

  • Logging: A reliable Topic for distributing log data. The data is sent when a log event occurs.

The library creates one DomainParticipant and three DataWriters, one for each topic type (periodic, event, and logging). Each DataWriter is created within its own Publisher.

The Observability domain ID can be configured by setting participant_factory_qos.monitoring.distribution_settings.dedicated_participant.domain_id.

7.1. Observability QoS Profile

By default, the DDS entities created by the Observability Library use the built-in profile BuiltinQosLib::Generic.Observability (located under <install dir>/resource/resource/xml/BuiltinProfiles.documentationONLY.xml) to configure their QoS. You can provide a different profile name for each entity by changing the Monitoring Qos Policy. It is recommended that if you provide a different profile name, you inherit from the BuiltinQosLib::Generic.Observability profile. For example:

 <qos_library name="MyQosLibrary">
     <qos_profile name="MyObservabilityProfile" base_name="BuiltinQosLib::Generic.Observability">
         <domain_participant_qos>
             <discovery>
                 <!-- Set the initial peers for the Observability
                      DomainParticipant
                 -->
                 <initial_peers>
                     <element>192.168.1.2</element>
                 </initial_peers>
             </discovery>
         </domain_participant_qos>
     </qos_profile>

     <qos_profile name="MyApplicationProfile">
         <participant_factory_qos>
             <monitoring>
                 <enable>true</enable>
                 <distribution_settings>
                     <dedicated_participant>
                         <participant_qos_profile_name>
                             MyQosLibrary::MyObservabilityProfile
                         </participant_qos_profile_name>
                     </dedicated_participant>
                 </distribution_settings>
             </monitoring>
         </participant_factory_qos>
     </qos_profile>
 </qos_library>

There are a few things to note in the BuiltinQosLib::Generic.Observability profile:

  • The BuiltinQosLib::Generic.Observability disables the use of multicast discovery by setting <multicast_receive_addresses/> for the Observability Library DomainParticipant in order to avoid issues with multicast discovery in networks where multicast is not available. In addition, a single application should connect only to one Collector Service instance. Using multicast may lead to multiple Collector Service instances receiving the same data.

    To enable multicast discovery, create a profile that overwrites the BuiltinQosLib::Generic.Observability profile and set the <multicast_receive_addresses> element to the multicast address.

 <qos_profile name="MyObservabilityProfile" base_name="BuiltinQosLib::Generic.Observability">
     <domain_participant_qos>
         <discovery>
             <multicast_receive_addresses>239.255.0.1</multicast_receive_addresses>
         </discovery>
     </domain_participant_qos>
 </qos_profile>

  • In most cases, the initial peer used to connect to Collector Service will be different from the initial peers used by the application. To change the initial peers for the Observability Library DomainParticipant, create a profile that overwrites the BuiltinQosLib::Generic.Observability profile and set the <initial_peers> element to the initial peer used to connect to Collector Service.

     <qos_profile name="MyObservabilityProfile" base_name="BuiltinQosLib::Generic.Observability">
     <domain_participant_qos>
         <discovery>
             <initial_peers>
                 <element>192.168.1.2</element>
             </initial_peers>
         </discovery>
     </domain_participant_qos>
 </qos_profile>

  • The BuiltinQosLib::Generic.Observability sets the DDS domain for the Observability Library DomainParticipant to 2 using the <domain_id/> tag.

    To modify the domain for the Observability Library DomainParticipant data, create a profile that overwrites the BuiltinQosLib::Generic.Observability profile and set the <domain_id> element to the required DDS domain (7 in this example).

 <qos_library name="MyQosLibrary">
     <qos_profile name="MyObservabilityProfile" base_name="BuiltinQosLib::Generic.Observability">
         <domain_participant_qos>
             <discovery>
                 <!-- Set the initial peers for the Observability
                      DomainParticipant
                 -->
                 <initial_peers>
                     <element>192.168.1.2</element>
                 </initial_peers>
             </discovery>
         </domain_participant_qos>
     </qos_profile>

     <qos_profile name="MyApplicationProfile">
         <participant_factory_qos>
             <monitoring>
                 <enable>true</enable>
                 <distribution_settings>
                     <dedicated_participant>
                         <domain_id>7</domain_id>
                         <participant_qos_profile_name>
                             MyQosLibrary::MyObservabilityProfile
                         </participant_qos_profile_name>
                     </dedicated_participant>
                 </distribution_settings>
             </monitoring>
         </participant_factory_qos>
     </qos_profile>
 </qos_library>