8. Monitoring Library 2.0
RTI Monitoring Library 2.0 is one component of Connext Observability Framework. It allows collecting and distributing telemetry data, which includes observables (metrics and non-metrics) and logs associated with the resources created by a DDS application. These observable resources are DomainParticipants, Publishers, Subscribers, DataWriters, DataReaders, Topics, types, and applications (refer to Resources). The library also accepts remote commands to change the set of collected and forwarded metrics and logs at runtime.
The observables and logs collected by Monitoring Library 2.0 are distributed to a Collector Service instance. Collector Service forwards the data to other Collector Service instances, or stores it to a third-party observability backend such as Prometheus or Grafana Loki. The collected non-metric observables are used by RTI Admin Console to enable its remote debugging feature.
The non-metric observables required by the remote debugging feature are only collected and forwarded if an Admin Console session requests them.
Monitoring Library 2.0 is a separate library (rtimonitoring2); applications can use it in three different modes:
Dynamically loaded: This is the default mode, which does not require linking with your application. The only requirement is that the rtimonitoring2 shared library must be in the library search path. The library is loaded when the monitoring library is enabled. See Enabling Monitoring Library 2.0.
Dynamic Linking: The application is linked with the rtimonitoring2 shared library. When the application runs, the rtimonitoring2 shared library must be in the library search path.
Static Linking: The application is linked with the rtimonitoring2 static library.
The last two modes (dynamic and static linking) are only supported in C and C++
and require calling the API RTI_Monitoring_initialize in your application
before any other Connext APIs. This API is defined in the header file
ndds/monitoring/monitoring_monitoringClass.h.
Regardless of the mode, to start monitoring your application, enable monitoring as described in Enabling Monitoring Library 2.0.
Monitoring Library 2.0 creates a dedicated Participant and uses three different built-in Topics to forward telemetry data to Collector Service:
Periodic: A best-effort Topic for distributing periodic metric data (for example,
dds_data_writer_protocol_pushed_samples_total). The data is sent periodically, with a configurable period.Event: A reliable Topic for distributing event metric data (for example,
dds_data_writer_liveliness_lost_total). The data is sent when it changes.Logging: A reliable Topic for distributing log data. The data is sent when a log event occurs.
The library creates one DomainParticipant and three DataWriters, one for each Topic type (periodic, event, and logging). Each DataWriter is created within its own Publisher.
When Monitoring Library 2.0 is enabled for an application
(participant_factory_qos.monitoring.enable is TRUE), every DDS Entity created
by the application will be “registered” with the library as an observable
resource. Monitoring Library 2.0 is able to monitor all DDS Entities across
multiple DomainParticipants. You can select the metrics and logs that you want collected and
forwarded for an observable
resource via an initial configuration, and/or change that data at runtime
using remote commands. To set the initial configuration for the collection of
metrics
in Monitoring Library 2.0, see Setting Initial Metrics and Log Configuration.
To change metric collection configuration dynamically at runtime, use the REST
API as
described in Collector Service REST API Reference. For an example
of how to dynamically change the metric collection configuration using
the Observability Dashboards, see Change the Metric Configuration.
Monitoring Library 2.0 receives remote commands on the built-in ServiceRequest Topic. The Monitoring Library 2.0 DomainParticipant creates a DataReader for this Topic.
Note
You are not expected to use the built-in Topics directly in your applications. The builtin Topics are internal channels between Monitoring Library 2.0 and Collector Service.
To send remote commands, use the REST API (see Collector Service REST API Reference). This API sends configuration commands to Collector Service, which forwards the commands to the appropriate Monitoring Library 2.0 instance.
To access the telemetry data, connect to the third-party backends where the data is stored by Collector Service. You can visualize the telemetry data through the reference Grafana dashboards (see Observability Dashboards).
8.1. Enabling Monitoring Library 2.0
To enable usage of Monitoring Library 2.0 and configure
its behavior, you have to use the
MONITORING QosPolicy (DDS Extension)
on the DomainParticipantFactory and set
participant_factory_qos.monitoring.enable to true. This QoS policy can be
configured programmatically or via XML. Following is an example that shows
how to enable Monitoring Library 2.0 in your XML configuration file:
<qos_library name="MyQosLibrary">
<qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
<participant_factory_qos>
<!-- Enable monitoring -->
<monitoring>
<enable>true</enable>
</monitoring>
</participant_factory_qos>
</qos_profile>
</qos_library>
The default configuration of Monitoring Library 2.0 uses the SHMEM and UDPv4 transports, along with the default Connext initial peers (builtin.udpv4://127.0.0.1, builtin.shmem://, builtin.udpv4://239.255.0.1) to connect to Collector Service.
Note
Although multicast is included in the default initial peers, it is not permitted by Collector Service. Therefore, the Monitoring Library 2.0 DomainParticipant will not use multicast when connecting to Collector Service.
For details on how to overwrite the initial peers, see Setting Collector Service Initial Peers.
If you want to connect to a Collector Service instance over the WAN, see Connecting to Collector Service Over WAN.
8.2. Setting Initial Metrics and Log Configuration
By default all metric collection is disabled, and all log forwarding is set
to level WARNING. To configure the initial behavior of telemetry data
in Monitoring Library 2.0, you have to use the MONITORING QosPolicy (DDS Extension)
on the DomainParticipantFactory and configure the
participant_factory_qos.monitoring.telemetry_data structure. This QoS
policy can be configured programmatically or via XML. It
is also changeable at runtime, allowing you to enable or disable
Monitoring Library 2.0 at runtime.
For details on how to set the resource_selection fields,
see Resource Pattern Definitions.
For details on how to set the enabled_metrics_selection and
disabled_metrics_selection fields, see Metric Pattern Definitions.
The following sections describe in detail the most common configuration options for Monitoring Library 2.0. For a complete list of configuration options, refer to the MONITORING QosPolicy (DDS Extension).
8.2.1. Enable All Metrics Manually
The following XML example shows how to configure Monitoring Library 2.0 parameters to collect all metrics.
<qos_library name="MyQosLibrary">
<qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
<participant_factory_qos>
<monitoring>
<!-- Enable monitoring -->
<enable>true</enable>
<!-- Enable all metrics -->
<telemetry_data>
<metrics>
<element>
<resource_selection>//*</resource_selection>
<enabled_metrics_selection>
<element>*</element>
</enabled_metrics_selection>
</element>
</metrics>
</telemetry_data>
<!-- Change the application name -->
<application_name>MyApplication</application_name>
<distribution_settings>
<dedicated_participant>
<!-- Change the Observability Domain ID -->
<domain_id>7</domain_id>
<!-- Change the initial peers of the
Observability DomainParticipant -->
<collector_initial_peers>
<element>192.168.1.2</element>
</collector_initial_peers>
</dedicated_participant>
</distribution_settings>
</monitoring>
</participant_factory_qos>
</qos_profile>
</qos_library>
8.2.2. Enable All Metrics Using a Snippet
Alternatively, you can use the snippet BuiltinQosSnippetLib::Feature.Monitoring2.Enable
in your XML configuration file. The following snippet enables
Monitoring Library 2.0 and all metrics
for collection and forwarding:
<qos_library name="MyQosLibrary">
<qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
<base_name>
<element>BuiltinQosSnippetLib::Feature.Monitoring2.Enable</element>
</base_name>
<participant_factory_qos>
<monitoring>
<application_name>MyApplication</application_name>
<distribution_settings>
<dedicated_participant>
<!-- Change the Observability Domain ID -->
<domain_id>7</domain_id>
<!-- Change the initial peers of the
Observability DomainParticipant -->
<collector_initial_peers>
<element>192.168.1.2</element>
</collector_initial_peers>
</dedicated_participant>
</distribution_settings>
</monitoring>
</participant_factory_qos>
</qos_profile>
</qos_library>
8.2.3. Enable a Custom Set of Metrics
The following XML example shows how to configure Monitoring Library 2.0 parameters to collect a custom set of metrics. For a list of all available metrics, see Telemetry Data.
<qos_library name="MyQosLibrary">
<qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
<participant_factory_qos>
<monitoring>
<enable>true</enable>
<telemetry_data>
<metrics>
<element>
<!-- enable all application metrics -->
<resource_selection>/applications/*</resource_selection>
<enabled_metrics_selection>
<element>*</element>
</enabled_metrics_selection>
</element>
<element>
<!-- enable all domain_participant metrics -->
<resource_selection>//domain_participants/*</resource_selection>
<enabled_metrics_selection>
<element>*</element>
</enabled_metrics_selection>
</element>
<element>
<!-- enable all topic metrics -->
<resource_selection>//topics/*</resource_selection>
<enabled_metrics_selection>
<element>*</element>
</enabled_metrics_selection>
</element>
<element>
<!-- enable all data_writer metrics except those that end in "_bytes" -->
<resource_selection>//data_writers/*</resource_selection>
<enabled_metrics_selection>
<element>*</element>
</enabled_metrics_selection>
<disabled_metrics_selection>
<element>dds_data_writer_*_bytes</element>
</disabled_metrics_selection>
</element>
<element>
<!-- enable all data_reader metrics except those related to "protocol" -->
<resource_selection>//data_readers/*</resource_selection>
<enabled_metrics_selection>
<element>*</element>
</enabled_metrics_selection>
<disabled_metrics_selection>
<element>dds_data_reader_protocol_*</element>
</disabled_metrics_selection>
</element>
</metrics>
<logs>
<!-- set initial MIDDLEWARE forwarding level to ERROR -->
<middleware_forwarding_level>ERROR</middleware_forwarding_level>
<!-- set initial SECURITY_EVENT forwarding level to ERROR -->
<security_event_forwarding_level>ERROR</security_event_forwarding_level>
<!-- set initial SERVICE forwarding level to ERROR -->
<service_forwarding_level>ERROR</service_forwarding_level>
<!-- set initial USER forwarding level to ERROR -->
<user_forwarding_level>ERROR</user_forwarding_level>
</logs>
</telemetry_data>
</monitoring>
</participant_factory_qos>
</qos_profile>
</qos_library>
8.3. Configuring Distribution Settings
In a typical application, after enabling Monitoring Library 2.0, you can configure the following parameters in your XML configuration file:
A name for the application being monitored
The DDS domain ID to use for observability
The locator (address), as an initial_peer, of the Collector Service instance to which the telemetry data will be forwarded
8.3.1. Setting Application Name
To modify the application name used by Monitoring Library 2.0,
use the participant_factory_qos.monitoring.application_name
field. For example:
<qos_library name="MyQosLibrary">
<qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
<participant_factory_qos>
<monitoring>
<enable>true</enable>
<application_name>MyApplication</application_name>
</monitoring>
</participant_factory_qos>
</qos_profile>
</qos_library>
Assigning an application name is important because it helps identify the resource that represents your Connext application. The resource identifier representing the application will be:
/applications/<application_name>
This is the resource identifier that will be used to send commands to this application from the Observability Dashboards.
The application_name should be unique across the Connext system; however, Monitoring Library 2.0 does not currently enforce uniqueness.
When application_name is not set, Monitoring Library 2.0 will automatically assign a resource identifier with this format:
/applications/<host_name:process_id:uuid>
8.3.2. Changing the Default Observability Domain ID
To modify the domain used by Monitoring Library 2.0’s DomainParticipant to connect to
Collector Service, use the participant_factory_qos.monitoring.distribution_settings.dedicated_participant.domain_id
field. The default value is 2.
<qos_library name="MyQosLibrary">
<qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
<participant_factory_qos>
<monitoring>
<enable>true</enable>
<distribution_settings>
<dedicated_participant>
<domain_id>7</domain_id>
</dedicated_participant>
</distribution_settings>
</monitoring>
</participant_factory_qos>
</qos_profile>
</qos_library>
8.3.3. Setting Collector Service Initial Peers
To connect Monitoring Library 2.0 to Collector Service, configure the library
with the locator/address of the Collector Service via the
Monitoring Library 2.0’s DomainParticipant initial peers list. Set this list
(usually just a single locator) using the
participant_factory_qos.monitoring.distribution_settings.dedicated_participant.collector_initial_peers
field in the Monitoring Library 2.0 XML QoS configuration. The locator/address of
the collector service uses the same format as the
DISCOVERY QosPolicy (DDS Extension)
initial_peers field.
<qos_library name="MyQosLibrary">
<qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
<participant_factory_qos>
<monitoring>
<enable>true</enable>
<distribution_settings>
<dedicated_participant>
<collector_initial_peers>
<element>192.168.1.2</element>
</collector_initial_peers>
</dedicated_participant>
</distribution_settings>
</monitoring>
</participant_factory_qos>
</qos_profile>
</qos_library>
If collector_initial_peers is not specified, or if it is explicitly set to
an empty list, Monitoring Library 2.0 will use the value set in the
domain_participant_qos.discovery.initial_peers of the QoS profile specified
by participant_factory_qos.monitoring.distribution_settings.dedicated_participant.participant_qos_profile_name
as the initial peers for the Monitoring Library 2.0’s DomainParticipant.
If both values are present, the value in collector_initial_peers in the Monitoring
QosPolicy will be used instead of the value of initial_peers in the Discovery
QosPolicy for the Monitoring Library 2.0’s DomainParticipant.
8.4. Configuring QoS for Entities
You may want to change the QoS of the Entities responsible
for distributing the monitoring data. By default, the DDS Entities created by
Monitoring Library 2.0 use the built-in profile BuiltinQosLib::Generic.Monitoring2
(as documented in <install dir>/resource/resource/xml/BuiltinProfiles.documentationONLY.xml)
to configure their QoS. You can provide a different profile name
(MyObservabilityProfile in the example below) for each Entity
by changing the Monitoring QoS Policy. It is recommended that if you provide a
different profile name, you create this profile to inherit from the
BuiltinQosLib::Generic.Monitoring2 or the
BuiltinQosLib::Generic.Monitoring2.WAN profiles, depending on whether you
want to connect to a Collector Service instance over a LAN or WAN.
The BuiltinQosLib::Generic.Monitoring2.WAN profile replaces the
UDPv4 and SHMEM transports with the RTI Real-Time WAN Transport in the
Monitoring Library 2.0 DomainParticipant.
The following example demonstrates the mechanism for changing the QoS for monitoring Entities by changing the name of the Entities.
<qos_library name="MyQosLibrary">
<qos_profile name="MyObservabilityProfile" base_name="BuiltinQosLib::Generic.Monitoring2">
<domain_participant_qos>
<participant_name>
<!-- Change the name of the Observability
DomainParticipant
-->
<name>Monitoring Participant</name>
</participant_name>
</domain_participant_qos>
<datawriter_qos topic_filter="DCPSEventStatusMonitoring">
<publication_name>
<!-- Change the name of the Observability
Event DataWriter
-->
<name>Monitoring Event DataWriter</name>
</publication_name>
</datawriter>
<datawriter_qos topic_filter="DCPSPeriodicStatusMonitoring">
<publication_name>
<!-- Change the name of the Observability
Periodic DataWriter
-->
<name>Monitoring Periodic DataWriter</name>
</publication_name>
</datawriter>
<datawriter_qos topic_filter="DCPSLoggingStatusMonitoring">
<publication_name>
<!-- Change the name of the Observability
Logging DataWriter
-->
<name>Monitoring Logging DataWriter</name>
</publication_name>
</datawriter>
</qos_profile>
<qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
<participant_factory_qos>
<monitoring>
<enable>true</enable>
<distribution_settings>
<dedicated_participant>
<!-- Change the configuration of the
Observability DomainParticipant -->
<participant_qos_profile_name>
MyQosLibrary::MyObservabilityProfile
</participant_qos_profile_name>
</dedicated_participant>
<!-- Change the configuration of the
Observability Publishers -->
<publisher_qos_profile_name>
MyQosLibrary::MyObservabilityProfile
</publisher_qos_profile_name>
<event_settings>
<!-- Change the configuration of the
Observability Event DataWriter -->
<datawriter_qos_profile_name>
MyQosLibrary::MyObservabilityProfile
</datawriter_qos_profile_name>
</event_settings>
<periodic_settings>
<!-- Change the configuration of the
Observability Periodic DataWriter -->
<datawriter_qos_profile_name>
MyQosLibrary::MyObservabilityProfile
</datawriter_qos_profile_name>
</periodic_settings>
<logging_settings>
<!-- Change the configuration of the
Observability Logging DataWriter -->
<datawriter_qos_profile_name>
MyQosLibrary::MyObservabilityProfile
</datawriter_qos_profile_name>
</logging_settings>
</distribution_settings>
</monitoring>
</participant_factory_qos>
</qos_profile>
</qos_library>
8.5. Connecting to Collector Service Over WAN
To connect to a Collector Service instance over a WAN, set
participant_factory_qos.monitoring.distribution_settings.dedicated_participant.participant_qos_profile_name
to the built-in profile
BuiltinQosLib::Generic.Monitoring2.WAN. This profile is configured to use the
the Real-Time WAN Transport, and it inherits from
BuiltinQosLib::Generic.Monitoring2.
In addition, you must set
participant_factory_qos.monitoring.distribution_settings.dedicated_participant.collector_initial_peers
to the locator/address of the Collector Service instance you want to connect to.
<qos_library name="MyQosLibrary">
<qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
<participant_factory_qos>
<monitoring>
<enable>true</enable>
<distribution_settings>
<dedicated_participant>
<collector_initial_peers>
<element>udpv4_wan://38.21.45.34:4500</element>
</collector_initial_peers>
<participant_qos_profile_name>
BuiltinQosLib::Generic.Monitoring2.WAN
</participant_qos_profile_name>
</dedicated_participant>
</distribution_settings>
</monitoring>
</participant_factory_qos>
</qos_profile>
</qos_library>