.. _section-monitoring-library-2:

|MONITORINGLIBRARY2_HEADING|
****************************
|RTI| |MONITORINGLIBRARY2| is one component of |CONNEXT| 
|OBSERVABILITY|. It is supported on all |Connext| platforms and 
included in the |RTI| |PRO| target packages. The library enables
the collection and distribution of telemetry data (metrics and logs) associated with the resources 
created by a DDS application. These observable resources are 
|DPs|, |PUBs|, |SUBs|, |DWs|, 
|DRs|, |TOPICs|, and *applications* 
(refer to :ref:`section-observable-resources`). The library also accepts 
remote commands to change the set of collected and forwarded telemetry data
at runtime.

The data collected by |MONITORINGLIBRARY2| is distributed to
an |OCS| instance. |OCS| forwards the data to other 
|OCS| instances, or stores it to a third-party
observability backend such as Prometheus or Grafana Loki.

|MONITORINGLIBRARY2| is a separate library (rtimonitoring2);
applications can use it in three different modes:

- **Dynamically loaded**: This is the default mode, which 
  does not require linking with your application. The only requirement is that the
  rtimonitoring2 shared library must be in the library search path. The library is
  loaded when the monitoring library is enabled. See :ref:`section-enabling_monitoring2-library`.

- **Dynamic Linking**: The application is linked with the rtimonitoring2 shared
  library. When the application runs, the rtimonitoring2 shared library must be
  in the library search path.
  
- **Static Linking**: The application is linked with the 
  rtimonitoring2 static library.
               
The last two modes (dynamic and static linking) are only supported in C and C++
and require calling the API ``RTI_Monitoring_initialize`` in your application
before any other |Connext| APIs. This API is defined in the header file
``ndds/monitoring/monitoring_monitoringClass.h``.

Regardless of the mode, to start monitoring your application,
enable monitoring as described in :ref:`section-enabling_monitoring2-library`.

|MONITORINGLIBRARY2| creates a dedicated Participant and uses three 
different built-in |TOPICs| to forward telemetry data to |OCS|:

- **Periodic**: A best-effort |TOPIC| for distributing periodic metric data 
  (for example, ``dds_data_writer_protocol_pushed_samples_total``). The data 
  is sent periodically, with a configurable period.

- **Event**: A reliable |TOPIC| for distributing event metric data
  (for example, ``dds_data_writer_liveliness_lost_total``). The data is sent  
  when it changes.

- **Logging**: A reliable |TOPIC| for distributing log data. The data is sent 
  when a log event occurs.

The library creates one |DP| and three |DWs|, 
one for each |TOPIC| type (periodic, event, and logging). Each |DW| is
created within its own |PUB|. 

When |MONITORINGLIBRARY2| is enabled for an application
(``participant_factory_qos.monitoring.enable`` is TRUE), every DDS Entity created
by the application will be “registered” with the library as an observable
resource. |MONITORINGLIBRARY2| is able to monitor all DDS Entities across
multiple |DPs|. You can select the telemetry data that you want collected and
forwarded for an observable
resource via an initial configuration, and/or change that data at runtime
using remote commands. To set the initial configuration for the collection of
metrics
in |MONITORINGLIBRARY2|, see :ref:`section-setting-initial-metrics`.
To change metric collection configuration dynamically at runtime, use the REST
API as
described in :ref:`section-collector-service-rest-api-reference`. For an example
of how to dynamically change the metric collection configuration using
the |DASHBOARDS|, see :ref:`section-change-metric-configuration`.

|MONITORINGLIBRARY2| receives remote commands on the built-in ServiceRequest
|TOPIC|. The |MONITORINGLIBRARY2| |DP| creates a |DR| for this |TOPIC|.

.. note:: 
    
    You are not expected to use the built-in |TOPICs| directly in your
    applications. The builtin |TOPICs| are internal channels between
    |MONITORINGLIBRARY2| and |OCS|.
    
    To send remote commands, use the REST API (see :ref:`section-collector-service-rest-api-reference`).
    This API sends configuration commands to |OCS|, which forwards the commands
    to the appropriate |MONITORINGLIBRARY2| instance.

    To access the telemetry data, connect to the third-party backends
    where the data is stored by |OCS|. You can visualize the telemetry data
    through the reference Grafana dashboards (see :ref:`section-observability-dashboards`).

.. _section-enabling_monitoring2-library:

Enabling |MONITORINGLIBRARY2_HEADING|
-------------------------------------

To enable usage of |MONITORINGLIBRARY2| and to configure
its behavior, you have to use the 
:link_connext_dds_pro_um:`MONITORING QosPolicy (DDS Extension) <#users_manual/MONITORING_QosPolicy.htm>`
on the DomainParticipantFactory and set 
``participant_factory_qos.monitoring.enable`` to true. This QoS policy can be 
configured programmatically or via XML. Next, there is an example that shows
how to enable |MONITORINGLIBRARY2| in your XML configuration file:

.. code-block:: xml
    :emphasize-lines: 5-7

    <qos_library name="MyQosLibrary">
        <qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
            <participant_factory_qos>
                <!-- Enable monitoring -->
                <monitoring>
                    <enable>true</enable>
                </monitoring>
            </participant_factory_qos>
        </qos_profile>
    </qos_library>

In a typical application, after enabling |MONITORINGLIBRARY2|, you can also
configure which metrics to collect from which resources; the DDS domain ID to
use for observability; a name for the application being monitored; and the
locator (address), as an initial_peer, of the |OCS| instance to which the
telemetry data will be forwarded. The following XML example 
shows how to configure these parameters:

.. code-block:: xml
   :emphasize-lines: 8-17, 19, 23, 26-28

    <qos_library name="MyQosLibrary">
        <qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
            <participant_factory_qos>
                <monitoring>
                    <!-- Enable monitoring -->
                    <enable>true</enable>
                    <!-- Enable all metrics -->
                    <telemetry_data>
                        <metrics>
                            <element>
                                <resource_selection>//*</resource_selection>
                                <enabled_metrics_selection>
                                    <element>*</element>
                                </enabled_metrics_selection>
                            </element>
                        </metrics>
                    </telemetry_data>
                    <!-- Change the application name -->
                    <application_name>MyApplication</application_name>
                    <distribution_settings>
                        <dedicated_participant>
                            <!-- Change the Observability Domain ID -->
                            <domain_id>7</domain_id>
                            <!-- Change the initial peers of the
                                 Observability DomainParticipant -->
                            <collector_initial_peers>
                                <element>192.168.1.2</element>
                            </collector_initial_peers>
                        </dedicated_participant>
                    </distribution_settings>
                </monitoring>
            </participant_factory_qos>
        </qos_profile>
    </qos_library>

Alternatively, you can use the snippet ``BuiltinQosSnippetLib::Feature.Monitoring2.Enable`` 
in your XML configuration file. This snippet enables |MONITORINGLIBRARY2| and
all metrics for collection and forwarding:

.. code-block:: xml
    :emphasize-lines: 3-5

    <qos_library name="MyQosLibrary">
        <qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
            <base_name>
                <element>BuiltinQosSnippetLib::Feature.Monitoring2.Enable</element>
            </base_name>
            <participant_factory_qos>
                <monitoring>
                    <application_name>MyApplication</application_name>
                    <distribution_settings>
                        <dedicated_participant>
                            <!-- Change the Observability Domain ID -->
                            <domain_id>7</domain_id>
                            <!-- Change the initial peers of the
                                 Observability DomainParticipant -->
                            <collector_initial_peers>
                                <element>192.168.1.2</element>
                            </collector_initial_peers>
                        </dedicated_participant>
                    </distribution_settings>
                </monitoring>
            </participant_factory_qos>
        </qos_profile>
    </qos_library>

The :link_connext_dds_pro_um:`MONITORING QosPolicy (DDS Extension) <#users_manual/MONITORING_QosPolicy.htm>` 
is changeable at runtime. This means that you can enable or disable 
|MONITORINGLIBRARY2| at runtime.

The following sections describe in detail the most common configuration options
for |MONITORINGLIBRARY2|. For a complete list of configuration options,
refer to the :link_connext_dds_pro_um:`MONITORING QosPolicy (DDS Extension) <#users_manual/MONITORING_QosPolicy.htm>`.

.. _section-setting-initial-metrics:

Setting the Initial Metrics and Log Configuration
-------------------------------------------------

By default all metric collection is disabled, and all log forwarding is set
to level WARNING. To configure the initial behavior of telemetry data 
in |MONITORINGLIBRARY2|, you have to use the :link_connext_dds_pro_um:`MONITORING QosPolicy (DDS Extension) <#users_manual/MONITORING_QosPolicy.htm>`
on the DomainParticipantFactory and configure the 
``participant_factory_qos.monitoring.telemetry_data`` structure. This QoS 
policy can be configured programmatically or via XML. For details
on how to set the ``resource_selection`` fields, see :ref:`section-observable-resource-patterns`. 
For details on how to set the ``enabled_metrics_selection`` and 
``disabled_metrics_selection`` fields, see :ref:`section-observable-metric-patterns`. 
The following example shows how to configure the initial metric and log 
collection and forwarding for |MONITORINGLIBRARY2| in your XML configuration file:

.. code-block:: xml
    :emphasize-lines: 10-13, 17-20, 24-27, 31-37, 41-47, 52, 54, 56, 58

    <qos_library name="MyQosLibrary">
        <qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
            <participant_factory_qos>
                <monitoring>
                    <enable>true</enable>
                    <telemetry_data>
                        <metrics>
                            <element>
                                <!-- enable all application metrics -->
                                <resource_selection>/applications/*</resource_selection>
                                <enabled_metrics_selection>
                                    <element>*</element>
                                </enabled_metrics_selection>
                            </element>
                            <element>
                                <!-- enable all domain_participant metrics -->
                                <resource_selection>//domain_participants/*</resource_selection>
                                <enabled_metrics_selection>
                                    <element>*</element>
                                </enabled_metrics_selection>
                            </element>
                            <element>
                                <!-- enable all topic metrics -->
                                <resource_selection>//topics/*</resource_selection>
                                <enabled_metrics_selection>
                                    <element>*</element>
                                </enabled_metrics_selection>
                            </element>
                            <element>
                                <!-- enable all data_writer metrics except those that end in "_bytes" -->
                                <resource_selection>//data_writers/*</resource_selection>
                                <enabled_metrics_selection>
                                    <element>*</element>
                                </enabled_metrics_selection>
                                <disabled_metrics_selection>
                                    <element>dds_data_writer_*_bytes</element>
                                </disabled_metrics_selection>
                            </element>
                            <element>
                                <!-- enable all data_reader metrics except those related to "protocol" -->
                                <resource_selection>//data_readers/*</resource_selection>
                                <enabled_metrics_selection>
                                    <element>*</element>
                                </enabled_metrics_selection>
                                <disabled_metrics_selection>
                                    <element>dds_data_reader_protocol_*</element>
                                </disabled_metrics_selection>
                            </element>
                        </metrics>
                        <logs>
                            <!-- set initial MIDDLEWARE forwarding level to ERROR -->
                            <middleware_forwarding_level>ERROR</middleware_forwarding_level>
                            <!-- set initial SECURITY_EVENT forwarding level to ERROR -->
                            <security_event_forwarding_level>ERROR</security_event_forwarding_level>
                            <!-- set initial SERVICE forwarding level to ERROR -->
                            <service_forwarding_level>ERROR</service_forwarding_level>
                            <!-- set initial USER forwarding level to ERROR -->
                            <user_forwarding_level>ERROR</user_forwarding_level>
                        </logs>
                    </telemetry_data>
                </monitoring>
            </participant_factory_qos>
        </qos_profile>
    </qos_library>

Setting the Application Name
----------------------------

To modify the application name used by |MONITORINGLIBRARY2|,
use the ``participant_factory_qos.monitoring.application_name``
field. For example:

.. code-block:: xml
   :emphasize-lines: 6

    <qos_library name="MyQosLibrary">
        <qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
            <participant_factory_qos>
                <monitoring>
                    <enable>true</enable>
                    <application_name>MyApplication</application_name>
                </monitoring>
            </participant_factory_qos>
        </qos_profile>
    </qos_library>

Assigning an application name is important because it helps identify 
the resource that represents your |Connext| application. The resource identifier 
representing the application will be:

``/applications/<application_name>``

This is the resource identifier that will be used to send commands to this 
application from the |DASHBOARDS|.

The application_name should be unique across the |Connext| system; however, 
|MONITORINGLIBRARY2| does not currently enforce uniqueness.

When application_name is not set, |MONITORINGLIBRARY2| will 
automatically assign a resource identifier with this format:

``/applications/<host_name:process_id:uuid>``

Changing the Default Observability Domain ID
--------------------------------------------

To modify the domain used by |MONITORINGLIBRARY2|'s |DP| to connect to
|OCS|, use the ``participant_factory_qos.monitoring.distribution_settings.dedicated_participant.domain_id``
field. The default value is 2.

.. code-block:: xml
   :emphasize-lines: 8

    <qos_library name="MyQosLibrary">
        <qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
            <participant_factory_qos>
                <monitoring>
                    <enable>true</enable>
                    <distribution_settings>
                        <dedicated_participant>
                            <domain_id>7</domain_id>
                        </dedicated_participant>
                    </distribution_settings>
                </monitoring>
            </participant_factory_qos>
        </qos_profile>
    </qos_library>

Configuring QoS for |MONITORINGLIBRARY2_HEADING| Entities
---------------------------------------------------------

By default, the DDS entities created by |MONITORINGLIBRARY2| use the 
built-in profile ``BuiltinQosLib::Generic.Monitoring2`` (as documented in 
``<install dir>/resource/resource/xml/BuiltinProfiles.documentationONLY.xml``) 
to configure their QoS. You can provide a different profile name 
(``MyObservabilityProfile`` in the example below) for each entity 
by changing the Monitoring QoS Policy. It is recommended that if you provide a 
different profile name, you create this profile to inherit from the 
``BuiltinQosLib::Generic.Monitoring2`` profile. For example:

.. code-block:: xml
    :emphasize-lines: 48-50, 54-56, 60-62, 67-69, 74-76

    <qos_library name="MyQosLibrary">
        <qos_profile name="MyObservabilityProfile" base_name="BuiltinQosLib::Generic.Monitoring2">
            <domain_participant_qos>
                <participant_name>
                    <!-- Change the name of the Observability
                         DomainParticipant
                    -->
                    <name>Monitoring Participant</name>
                </participant_name>
            </domain_participant_qos>

            <datawriter_qos topic_filter="DCPSEventStatusMonitoring">
                <publication_name>
                    <!-- Change the name of the Observability
                         Event DataWriter
                    -->
                    <name>Monitoring Event DataWriter</name>
                </publication_name>
            </datawriter>

            <datawriter_qos topic_filter="DCPSPeriodicStatusMonitoring">
                <publication_name>
                    <!-- Change the name of the Observability
                         Periodic DataWriter
                    -->
                    <name>Monitoring Periodic DataWriter</name>
                </publication_name>
            </datawriter>

            <datawriter_qos topic_filter="DCPSLoggingStatusMonitoring">
                <publication_name>
                    <!-- Change the name of the Observability
                         Logging DataWriter
                    -->
                    <name>Monitoring Logging DataWriter</name>
                </publication_name>
            </datawriter>
        </qos_profile>

        <qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
            <participant_factory_qos>
                <monitoring>
                    <enable>true</enable>
                    <distribution_settings>
                        <dedicated_participant>
                            <!-- Change the configuration of the
                                 Observability DomainParticipant -->
                            <participant_qos_profile_name>
                                MyQosLibrary::MyObservabilityProfile
                            </participant_qos_profile_name>
                        </dedicated_participant>
                        <!-- Change the configuration of the
                             Observability Publishers -->
                        <publisher_qos_profile_name>
                            MyQosLibrary::MyObservabilityProfile
                        </publisher_qos_profile_name>
                        <event_settings>
                            <!-- Change the configuration of the
                                 Observability Event DataWriter -->
                            <datawriter_qos_profile_name>
                                MyQosLibrary::MyObservabilityProfile
                            </datawriter_qos_profile_name>
                        </event_settings>
                        <periodic_settings>
                            <!-- Change the configuration of the
                                 Observability Periodic DataWriter -->
                            <datawriter_qos_profile_name>
                                MyQosLibrary::MyObservabilityProfile
                            </datawriter_qos_profile_name>
                        </periodic_settings>
                        <logging_settings>
                            <!-- Change the configuration of the
                                 Observability Logging DataWriter -->
                            <datawriter_qos_profile_name>
                                MyQosLibrary::MyObservabilityProfile
                            </datawriter_qos_profile_name>
                        </logging_settings>
                    </distribution_settings>
                </monitoring>
            </participant_factory_qos>
        </qos_profile>
    </qos_library>

.. note::

    The ``BuiltinQosLib::Generic.Monitoring2`` profile disables the use of
    multicast discovery by setting the ``<multicast_receive_addresses/>`` element
    for the |MONITORINGLIBRARY2|'s |DP|. Using multicast may lead to multiple
    |OCS| instances receiving the same data. Your
    applications (that is, each instance of |MONITORINGLIBRARY2|), should
    configure the address (initial_peer) of the |OCS| that they connect to
    explicitly as described in :ref:`section-setting-initial-peers`.

.. _section-setting-initial-peers:

Setting Collector Service Initial Peers
---------------------------------------

To connect |MONITORINGLIBRARY2| to |OCS|, configure the library
with the locator/address of the |OCS| via the
|MONITORINGLIBRARY2|'s |DP| initial peers list. Set this list
(usually just a single locator) using the
``participant_factory_qos.monitoring.distribution_settings.dedicated_participant.collector_initial_peers``
field in the |MONITORINGLIBRARY2| XML QoS configuration. The locator/address of
the collector service uses the same format as the 
:link_connext_dds_pro_um:`DISCOVERY QosPolicy (DDS Extension) <#users_manual/DISCOVERY_Qos.htm>`
``initial_peers`` field.

.. code-block:: xml
   :emphasize-lines: 8-10

    <qos_library name="MyQosLibrary">
        <qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
            <participant_factory_qos>
                <monitoring>
                    <enable>true</enable>
                    <distribution_settings>
                        <dedicated_participant>
                            <collector_initial_peers>
                                <element>192.168.1.2</element>
                            </collector_initial_peers>
                        </dedicated_participant>
                    </distribution_settings>
                </monitoring>
            </participant_factory_qos>
        </qos_profile>
    </qos_library>

If ``collector_initial_peers`` is not specified, or if it is explicitly set to
an empty list, |MONITORINGLIBRARY2| will use the value set in the 
``domain_participant_qos.discovery.initial_peers`` of the QoS profile specified
by ``participant_factory_qos.monitoring.distribution_settings.dedicated_participant.participant_qos_profile_name``
as the initial peers for the |MONITORINGLIBRARY2|'s |DP|.

If both values are present, the value in ``collector_initial_peers`` in the Monitoring
QosPolicy will be used instead of the value of ``initial_peers`` in the Discovery
QosPolicy for the |MONITORINGLIBRARY2|'s |DP|.