8. Monitoring Library 2.0

RTI Monitoring Library 2.0 is one component of the Connext Observability Framework. It allows collecting and distributing telemetry data (metrics and logs) associated with the resources created by a DDS application. These observable resources are DomainParticipants, Publishers, Subscribers, DataWriters, DataReaders, Topics, and applications (refer to Resources). The library also accepts remote commands (via dashboards) to change the set of generated and forwarded telemetry data at runtime.

The data forwarded by Monitoring Library 2.0 is sent to a Collector Service instance. Collector Service forwards the data to other Collector Service instances, or stores it to a third-party observability backend such as Prometheus or Grafana Loki.

Monitoring Library 2.0 is a separate library (rtimonitoring2) and applications can use it in three different modes:

  • Dynamically loaded: This is the default mode, which requires that the rtimonitoring2 shared library is in the library search path.

  • Dynamic Linking: The application is linked with the rtimonitoring2 shared library.

  • Static Linking: The application is linked with the rtimonitoring2 static library.

The last two modes require calling the API RTI_Monitoring_initialize in your application before any other Connext APIs. This API is defined in the header file ndds/monitoring/monitoring_monitoringClass.h.

Dynamic and static linking are only supported in C and C++ applications.

Monitoring Library 2.0 creates a dedicated Participant and uses three different built-in Topics to forward telemetry data to Observability Collector Service:

  • Periodic: A best-effort Topic for distributing periodic metric data (for example, dds_datawriter_protocol_pushed_samples_total). The data is sent periodically, with a configurable period.

  • Event: A reliable Topic for distributing event metric data (for example, dds_datawriter_liveliness_lost_total). The data is sent when it changes.

  • Logging: A reliable Topic for distributing log data. The data is sent when a log event occurs.

The library creates one DomainParticipant and three DataWriters, one for each topic type (periodic, event, and logging). Each DataWriter is created within its own Publisher.

8.1. Enabling Monitoring Library 2.0

To enable usage of Monitoring Library 2.0 and to configure its behavior, you have to use the MONITORING QosPolicy (DDS Extension) on the DomainParticipantFactory and set participant_factory_qos.monitoring.enable to true. This QoS policy can be configured programmatically or via XML. Next, there is an example that shows how to enable Monitoring Library 2.0 in your XML configuration file:

<qos_library name="MyQosLibrary">
    <qos_profile name="MyApplicationProfile">
        <participant_factory_qos>
            <monitoring>
                <enable>true</enable>
            </monitoring>
        </participant_factory_qos>
    </qos_profile>
</qos_library>

Alternatively, you can also use the snippet BuiltinQosSnippetLib::Feature.Monitoring2.Enable to enable the Monitoring Library 2.0 in your XML configuration file:

<qos_library name="MyQosLibrary">
    <qos_profile name="MyApplicationProfile">
       <base_name>
         <element>BuiltinQosSnippetLib::Feature.Monitoring2.Enable</element>
       </base_name>
    </qos_profile>
</qos_library>

In a typical application in addition to enabling the Monitoring Library 2.0, you will also want to configure the application name, the initial peer of the Collector Service to which the telemetry data will be forwarded, and the Observability Domain ID. The next XML configuration shows how to configure these parameters:

 <qos_library name="MyQosLibrary">
     <qos_profile name="MyApplicationProfile">
         <participant_factory_qos>
             <monitoring>
                 <enable>true</enable>
                 <!-- Change the application name -->
                 <application_name>MyApplication</application_name>
                 <distribution_settings>
                     <dedicated_participant>
                         <!-- Change the Observability Domain ID -->
                         <domain_id>7</domain_id>
                         <!-- Change the initial peers of the
                              Observability DomainParticipant -->
                         <collector_initial_peers>
                             <element>192.168.1.2</element>
                         </collector_initial_peers>
                     </dedicated_participant>
                 </distribution_settings>
             </monitoring>
         </participant_factory_qos>
     </qos_profile>
 </qos_library>

The following sections describe in detail the most common configuration options for Monitoring Library 2.0. For a complete list of configuration options, refer to the MONITORING QosPolicy (DDS Extension).

8.2. Setting the Application Name

To modify the application name used by Monitoring Library 2.0, use the participant_factory_qos.monitoring.application_name field. For example:

 <qos_library name="MyQosLibrary">
     <qos_profile name="MyApplicationProfile">
         <participant_factory_qos>
             <monitoring>
                 <enable>true</enable>
                 <application_name>MyApplication</application_name>
             </monitoring>
         </participant_factory_qos>
     </qos_profile>
 </qos_library>

Assigning an application name is important because it help identifying the resource that represents your Connext application. The resource identifier representing the application will be:

/applications/<application_name>

This is the resource identifier that will be used to send commands to this application from the RTI Observability Dashboards.

The application_name should be unique across theConnext system; however, the Monitoring Library 2.0 does not currently enforce uniqueness.

When application_name is not set, the RTI Observability Library will automatically assign a resource identifier with this format:

/applications/<host_name:process_id:uuid>

8.3. Changing the Default Observability Domain ID

To modify the domain for the Monitoring Library 2.0 DomainParticipant data, use the participant_factory_qos.monitoring.distribution_settings.dedicated_participant.domain_id field. The default value is 2.

 <qos_library name="MyQosLibrary">
     <qos_profile name="MyApplicationProfile">
         <participant_factory_qos>
             <monitoring>
                 <enable>true</enable>
                 <distribution_settings>
                     <dedicated_participant>
                         <domain_id>7</domain_id>
                     </dedicated_participant>
                 </distribution_settings>
             </monitoring>
         </participant_factory_qos>
     </qos_profile>
 </qos_library>

8.4. Configuring QoS for Monitoring Library 2.0 Entities

By default, the DDS entities created by Monitoring Library 2.0 use the built-in profile BuiltinQosLib::Generic.Monitoring2 (located under <install dir>/resource/resource/xml/BuiltinProfiles.documentationONLY.xml) to configure their QoS. You can provide a different profile name (MyObservabilityProfile in the example below) for each entity by changing the Monitoring Qos Policy. It is recommended that if you provide a different profile name, you create this profile to inherit from the BuiltinQosLib::Generic.Monitoring2 profile. For example:

<qos_library name="MyQosLibrary">
    <qos_profile name="MyObservabilityProfile" base_name="BuiltinQosLib::Generic.Monitoring2">
        <domain_participant_qos>
            <participant_name>
                <!-- Change the name of the Observability
                     DomainParticipant
                -->
                <name>Monitoring Participant</name>
            </participant_name>
        </domain_participant_qos>
        <domain_participant_qos>
            <participant_name>
                <!-- Change the name of the Observability
                     DomainParticipant
                -->
                <name>Monitoring Participant</name>
            </participant_name>
        </domain_participant_qos>

        <datawriter_qos topic_filter="DCPSEventStatusMonitoring">
            <publication_name>
                <!-- Change the name of the Observability
                     Event DataWriter
                -->
                <name>Monitoring Event DataWriter</name>
            </publication_name>
        </datawriter>

        <datawriter_qos topic_filter="DCPSPeriodicStatusMonitoring">
            <publication_name>
                <!-- Change the name of the Observability
                     Periodic DataWriter
                -->
                <name>Monitoring Periodic DataWriter</name>
            </publication_name>
        </datawriter>

        <datawriter_qos topic_filter="DCPSLoggingStatusMonitoring">
            <publication_name>
                <!-- Change the name of the Observability
                     Logging DataWriter
                -->
                <name>Monitoring Logging DataWriter</name>
            </publication_name>
        </datawriter>
    </qos_profile>

    <qos_profile name="MyApplicationProfile">
        <participant_factory_qos>
            <monitoring>
                <enable>true</enable>
                <distribution_settings>
                    <dedicated_participant>
                        <!-- Change the configuration of the
                             Observability DomainParticipant -->
                        <participant_qos_profile_name>
                            MyQosLibrary::MyObservabilityProfile
                        </participant_qos_profile_name>
                    </dedicated_participant>
                    <!-- Change the configuration of the
                         Observability Publishers -->
                    <publisher_qos_profile_name>
                        BuiltinQosLib::::Generic.Monitoring2
                    </publisher_qos_profile_name>
                    <event_settings>
                        <!-- Change the configuration of the
                             Observability Event DataWriter -->
                        <datawriter_qos_profile_name>
                            BuiltinQosLib::::Generic.Monitoring2
                        </datawriter_qos_profile_name>
                    </event_settings>
                    <periodic_settings>
                        <!-- Change the configuration of the
                             Observability Periodic DataWriter -->
                        <datawriter_qos_profile_name>
                            BuiltinQosLib::::Generic.Monitoring2
                        </datawriter_qos_profile_name>
                    </periodic_settings>
                    <logging_settings>
                        <!-- Change the configuration of the
                             Observability Logging DataWriter -->
                        <datawriter_qos_profile_name>
                            BuiltinQosLib::::Generic.Monitoring2
                        </datawriter_qos_profile_name>
                    </logging_settings>
                </distribution_settings>
            </monitoring>
        </participant_factory_qos>
    </qos_profile>
</qos_library>

The BuiltinQosLib::Generic.Monitoring2 profile disables the use of multicast discovery by setting the <multicast_receive_addresses/> element for the Monitoring Library 2.0 DomainParticipant. This setting prevents issues with multicast discovery in networks where multicast is not available. In addition, a single application should connect only to one Collector Service instance. Using multicast may lead to multiple Collector Service instances receiving the same data.

To enable multicast discovery, create a profile that overwrites the BuiltinQosLib::Generic.Monitoring2 profile and set the <multicast_receive_addresses> element to the multicast address.

 <qos_library name="MyQosLibrary">
     <qos_profile name="MyObservabilityProfile" base_name="BuiltinQosLib::Generic.Monitoring2">
         <domain_participant_qos>
             <discovery>
                 <multicast_receive_addresses>239.255.0.1</multicast_receive_addresses>
             </discovery>
         </domain_participant_qos>
     </qos_profile>
 </qos_library>

8.5. Setting Collector Service Initial Peers

In most cases, the initial peer used to connect to Collector Service will be different from the initial peers used by the application. To change the initial peers for the Monitoring Library 2.0 DomainParticipant, use the participant_factory_qos.monitoring.distribution_settings.dedicated_participant.collector_initial_peers field:

 <qos_library name="MyQosLibrary">
     <qos_profile name="MyApplicationProfile">
         <participant_factory_qos>
             <monitoring>
                 <enable>true</enable>
                 <distribution_settings>
                     <dedicated_participant>
                         <collector_initial_peers>
                             <element>192.168.1.2</element>
                         </collector_initial_peers>
                     </dedicated_participant>
                 </distribution_settings>
             </monitoring>
         </participant_factory_qos>
     </qos_profile>
 </qos_library>

If no collector_initial_peers are specified, or if it is explicitly set to an empty list, Monitoring Library 2.0 will use the initial_peers list of the QoS profile specified by participant_factory_qos.monitoring.distribution_settings.dedicated_participant.participant_qos_profile_name as the initial peers of the Monitoring Library 2.0 DomainParticipant.

If both are present, the value in collector_initial_peers takes precedence over initial_peers.