.. _section-monitoring-library-2:

|MONITORINGLIBRARY2_HEADING|
****************************
|RTI| |MONITORINGLIBRARY2| is one component of |CONNEXT| 
|OBSERVABILITY|. It allows collecting and distributing 
telemetry data, which includes observables (metrics and non-metrics) 
and logs associated with the resources created by a DDS application. 
These observable resources are 
|DPs|, |PUBs|, |SUBs|, |DWs|, |DRs|, |TOPICs|, types, and applications 
(refer to :ref:`section-observable-resources`). The library also accepts
remote commands to change the set of collected and forwarded 
metrics and logs at runtime.

The observables and logs collected by |MONITORINGLIBRARY2| are distributed to
a |OCS| instance. |OCS| forwards the data to other 
|OCS| instances, or stores it to a third-party
observability backend such as Prometheus or Grafana Loki.
The collected non-metric observables are used by 
|RTI_ADMINCONSOLE| to enable its :link_tools_admin_console:`remote debugging feature <p2_administration/features_reference/ref_remote_debugging.html>`.

The non-metric observables required by the remote debugging feature
are only collected and forwarded if an |ADMINCONSOLE| session
requests them.

|MONITORINGLIBRARY2| is a separate library (rtimonitoring2);
applications can use it in three different modes:

- **Dynamically loaded**: This is the default mode, which 
  does not require linking with your application. The only requirement is that the
  rtimonitoring2 shared library must be in the library search path. The library is
  loaded when the monitoring library is enabled. See :ref:`section-enabling_monitoring2-library`.

- **Dynamic Linking**: The application is linked with the rtimonitoring2 shared
  library. When the application runs, the rtimonitoring2 shared library must be
  in the library search path.
  
- **Static Linking**: The application is linked with the 
  rtimonitoring2 static library.
               
The last two modes (dynamic and static linking) are only supported in C and C++
and require calling the API ``RTI_Monitoring_initialize`` in your application
before any other |Connext| APIs. This API is defined in the header file
``ndds/monitoring/monitoring_monitoringClass.h``.

Regardless of the mode, to start monitoring your application,
enable monitoring as described in :ref:`section-enabling_monitoring2-library`.

|MONITORINGLIBRARY2| creates a dedicated Participant and uses three 
different built-in |TOPICs| to forward telemetry data to |OCS|:

- **Periodic**: A best-effort |TOPIC| for distributing periodic metric data 
  (for example, ``dds_data_writer_protocol_pushed_samples_total``). The data 
  is sent periodically, with a configurable period.

- **Event**: A reliable |TOPIC| for distributing event metric data
  (for example, ``dds_data_writer_liveliness_lost_total``). The data is sent  
  when it changes.

- **Logging**: A reliable |TOPIC| for distributing log data. The data is sent 
  when a log event occurs.

The library creates one |DP| and three |DWs|, 
one for each |TOPIC| type (periodic, event, and logging). Each |DW| is
created within its own |PUB|. 

When |MONITORINGLIBRARY2| is enabled for an application
(``participant_factory_qos.monitoring.enable`` is TRUE), every DDS *Entity* created
by the application will be “registered” with the library as an observable
resource. |MONITORINGLIBRARY2| is able to monitor all DDS *Entities* across
multiple |DPs|. You can select the metrics and logs that you want collected and
forwarded for an observable
resource via an initial configuration, and/or change that data at runtime
using remote commands. To set the initial configuration for the collection of
metrics
in |MONITORINGLIBRARY2|, see :ref:`section-setting-initial-metrics`.
To change metric collection configuration dynamically at runtime, use the REST
API as
described in :ref:`section-collector-service-rest-api-reference`. For an example
of how to dynamically change the metric collection configuration using
the |DASHBOARDS|, see :ref:`section-change-metric-configuration`.

|MONITORINGLIBRARY2| receives remote commands on the built-in ServiceRequest
|TOPIC|. The |MONITORINGLIBRARY2| |DP| creates a |DR| for this |TOPIC|.

.. note:: 
    
    You are not expected to use the built-in |TOPICs| directly in your
    applications. The builtin |TOPICs| are internal channels between
    |MONITORINGLIBRARY2| and |OCS|.
    
    To send remote commands, use the REST API (see :ref:`section-collector-service-rest-api-reference`).
    This API sends configuration commands to |OCS|, which forwards the commands
    to the appropriate |MONITORINGLIBRARY2| instance.

    To access the telemetry data, connect to the third-party backends
    where the data is stored by |OCS|. You can visualize the telemetry data
    through the reference Grafana dashboards (see :ref:`section-observability-dashboards`).

.. _section-enabling_monitoring2-library:

Enabling |MONITORINGLIBRARY2_HEADING|
=====================================

To enable usage of |MONITORINGLIBRARY2| and configure
its behavior, you have to use the 
:link_connext_dds_pro_um:`MONITORING QosPolicy (DDS Extension) <#users_manual/MONITORING_QosPolicy.htm>`
on the DomainParticipantFactory and set 
``participant_factory_qos.monitoring.enable`` to true. This QoS policy can be 
configured programmatically or via XML. Following is an example that shows
how to enable |MONITORINGLIBRARY2| in your XML configuration file:

.. code-block:: xml
    :emphasize-lines: 5-7

    <qos_library name="MyQosLibrary">
        <qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
            <participant_factory_qos>
                <!-- Enable monitoring -->
                <monitoring>
                    <enable>true</enable>
                </monitoring>
            </participant_factory_qos>
        </qos_profile>
    </qos_library>

The default configuration of |MONITORINGLIBRARY2| uses the SHMEM and 
UDPv4 transports, along with the default |CONNEXT| initial peers (builtin.udpv4://127.0.0.1,
builtin.shmem://, builtin.udpv4://239.255.0.1) to connect to |OCS|.

.. note::

  Although multicast is included in the default initial peers, it is not 
  permitted by |OCS|. Therefore, the |MONITORINGLIBRARY2| |DP| will not use 
  multicast when connecting to |OCS|.

For details on how to overwrite the initial peers, see :ref:`section-setting-initial-peers`.

If you want to connect to a |OCS| instance over the WAN, see
:ref:`section-connecting-to-a-collector-service-over-a-wan`.

.. _section-setting-initial-metrics:

Setting Initial Metrics and Log Configuration
=============================================

By default all metric collection is disabled, and all log forwarding is set
to level WARNING. To configure the initial behavior of telemetry data 
in |MONITORINGLIBRARY2|, you have to use the :link_connext_dds_pro_um:`MONITORING QosPolicy (DDS Extension) <#users_manual/MONITORING_QosPolicy.htm>`
on the DomainParticipantFactory and configure the 
``participant_factory_qos.monitoring.telemetry_data`` structure. This QoS 
policy can be configured programmatically or via XML. It 
is also changeable at runtime, allowing you to enable or disable 
|MONITORINGLIBRARY2| at runtime.

For details on how to set the ``resource_selection`` fields, 
see :ref:`section-observable-resource-patterns`. 
For details on how to set the ``enabled_metrics_selection`` and 
``disabled_metrics_selection`` fields, see :ref:`section-observable-metric-patterns`. 

The following sections describe in detail the most common configuration options
for |MONITORINGLIBRARY2|. For a complete list of configuration options,
refer to the :link_connext_dds_pro_um:`MONITORING QosPolicy (DDS Extension) <#users_manual/MONITORING_QosPolicy.htm>`.

.. _section-enable-metrics-manually:

Enable All Metrics Manually
---------------------------
The following XML example shows how to configure |MONITORINGLIBRARY2|
parameters to collect all metrics. 

.. code-block:: xml
   :emphasize-lines: 8-17

    <qos_library name="MyQosLibrary">
        <qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
            <participant_factory_qos>
                <monitoring>
                    <!-- Enable monitoring -->
                    <enable>true</enable>
                    <!-- Enable all metrics -->
                    <telemetry_data>
                        <metrics>
                            <element>
                                <resource_selection>//*</resource_selection>
                                <enabled_metrics_selection>
                                    <element>*</element>
                                </enabled_metrics_selection>
                            </element>
                        </metrics>
                    </telemetry_data>
                    <!-- Change the application name -->
                    <application_name>MyApplication</application_name>
                    <distribution_settings>
                        <dedicated_participant>
                            <!-- Change the Observability Domain ID -->
                            <domain_id>7</domain_id>
                            <!-- Change the initial peers of the
                                 Observability DomainParticipant -->
                            <collector_initial_peers>
                                <element>192.168.1.2</element>
                            </collector_initial_peers>
                        </dedicated_participant>
                    </distribution_settings>
                </monitoring>
            </participant_factory_qos>
        </qos_profile>
    </qos_library>

.. _section-enable-metrics-snippet:

Enable All Metrics Using a Snippet
----------------------------------

Alternatively, you can use the snippet ``BuiltinQosSnippetLib::Feature.Monitoring2.Enable`` 
in your XML configuration file. The following snippet enables 
|MONITORINGLIBRARY2| and all metrics
for collection and forwarding:

.. code-block:: xml
    :emphasize-lines: 3-5

    <qos_library name="MyQosLibrary">
        <qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
            <base_name>
                <element>BuiltinQosSnippetLib::Feature.Monitoring2.Enable</element>
            </base_name>
            <participant_factory_qos>
                <monitoring>
                    <application_name>MyApplication</application_name>
                    <distribution_settings>
                        <dedicated_participant>
                            <!-- Change the Observability Domain ID -->
                            <domain_id>7</domain_id>
                            <!-- Change the initial peers of the
                                 Observability DomainParticipant -->
                            <collector_initial_peers>
                                <element>192.168.1.2</element>
                            </collector_initial_peers>
                        </dedicated_participant>
                    </distribution_settings>
                </monitoring>
            </participant_factory_qos>
        </qos_profile>
    </qos_library>

.. _section-enable-metrics-custom:

Enable a Custom Set of Metrics
------------------------------

The following XML example shows how to configure |MONITORINGLIBRARY2|
parameters to collect a custom set of metrics. For a list of all available 
metrics, see :ref:`section-telemetry-index`.


.. code-block:: xml
    :emphasize-lines: 10-13, 17-20, 24-27, 31-37, 41-47, 52, 54, 56, 58

    <qos_library name="MyQosLibrary">
        <qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
            <participant_factory_qos>
                <monitoring>
                    <enable>true</enable>
                    <telemetry_data>
                        <metrics>
                            <element>
                                <!-- enable all application metrics -->
                                <resource_selection>/applications/*</resource_selection>
                                <enabled_metrics_selection>
                                    <element>*</element>
                                </enabled_metrics_selection>
                            </element>
                            <element>
                                <!-- enable all domain_participant metrics -->
                                <resource_selection>//domain_participants/*</resource_selection>
                                <enabled_metrics_selection>
                                    <element>*</element>
                                </enabled_metrics_selection>
                            </element>
                            <element>
                                <!-- enable all topic metrics -->
                                <resource_selection>//topics/*</resource_selection>
                                <enabled_metrics_selection>
                                    <element>*</element>
                                </enabled_metrics_selection>
                            </element>
                            <element>
                                <!-- enable all data_writer metrics except those that end in "_bytes" -->
                                <resource_selection>//data_writers/*</resource_selection>
                                <enabled_metrics_selection>
                                    <element>*</element>
                                </enabled_metrics_selection>
                                <disabled_metrics_selection>
                                    <element>dds_data_writer_*_bytes</element>
                                </disabled_metrics_selection>
                            </element>
                            <element>
                                <!-- enable all data_reader metrics except those related to "protocol" -->
                                <resource_selection>//data_readers/*</resource_selection>
                                <enabled_metrics_selection>
                                    <element>*</element>
                                </enabled_metrics_selection>
                                <disabled_metrics_selection>
                                    <element>dds_data_reader_protocol_*</element>
                                </disabled_metrics_selection>
                            </element>
                        </metrics>
                        <logs>
                            <!-- set initial MIDDLEWARE forwarding level to ERROR -->
                            <middleware_forwarding_level>ERROR</middleware_forwarding_level>
                            <!-- set initial SECURITY_EVENT forwarding level to ERROR -->
                            <security_event_forwarding_level>ERROR</security_event_forwarding_level>
                            <!-- set initial SERVICE forwarding level to ERROR -->
                            <service_forwarding_level>ERROR</service_forwarding_level>
                            <!-- set initial USER forwarding level to ERROR -->
                            <user_forwarding_level>ERROR</user_forwarding_level>
                        </logs>
                    </telemetry_data>
                </monitoring>
            </participant_factory_qos>
        </qos_profile>
    </qos_library>

.. _section-configure-distribution-settings:

Configuring Distribution Settings
=================================

In a typical application, after enabling |MONITORINGLIBRARY2|, you can 
configure the following parameters in your XML configuration file: 

- A name for the application being monitored 
- The DDS domain ID to use for observability 
- The locator (address), as an initial_peer, of the |OCS| instance to which the
  telemetry data will be forwarded 

Setting Application Name
------------------------

To modify the application name used by |MONITORINGLIBRARY2|,
use the ``participant_factory_qos.monitoring.application_name``
field. For example:

.. code-block:: xml
   :emphasize-lines: 6

    <qos_library name="MyQosLibrary">
        <qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
            <participant_factory_qos>
                <monitoring>
                    <enable>true</enable>
                    <application_name>MyApplication</application_name>
                </monitoring>
            </participant_factory_qos>
        </qos_profile>
    </qos_library>

Assigning an application name is important because it helps identify 
the resource that represents your |Connext| application. The resource identifier 
representing the application will be:

``/applications/<application_name>``

This is the resource identifier that will be used to send commands to this 
application from the |DASHBOARDS|.

The application_name should be unique across the |Connext| system; however, 
|MONITORINGLIBRARY2| does not currently enforce uniqueness.

When application_name is not set, |MONITORINGLIBRARY2| will 
automatically assign a resource identifier with this format:

``/applications/<host_name:process_id:uuid>``

Changing the Default Observability Domain ID
--------------------------------------------

To modify the domain used by |MONITORINGLIBRARY2|'s |DP| to connect to
|OCS|, use the ``participant_factory_qos.monitoring.distribution_settings.dedicated_participant.domain_id``
field. The default value is 2.

.. code-block:: xml
   :emphasize-lines: 8

    <qos_library name="MyQosLibrary">
        <qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
            <participant_factory_qos>
                <monitoring>
                    <enable>true</enable>
                    <distribution_settings>
                        <dedicated_participant>
                            <domain_id>7</domain_id>
                        </dedicated_participant>
                    </distribution_settings>
                </monitoring>
            </participant_factory_qos>
        </qos_profile>
    </qos_library>

.. _section-setting-initial-peers:

Setting Collector Service Initial Peers
---------------------------------------

To connect |MONITORINGLIBRARY2| to |OCS|, configure the library
with the locator/address of the |OCS| via the
|MONITORINGLIBRARY2|'s |DP| initial peers list. Set this list
(usually just a single locator) using the
``participant_factory_qos.monitoring.distribution_settings.dedicated_participant.collector_initial_peers``
field in the |MONITORINGLIBRARY2| XML QoS configuration. The locator/address of
the collector service uses the same format as the 
:link_connext_dds_pro_um:`DISCOVERY QosPolicy (DDS Extension) <#users_manual/DISCOVERY_Qos.htm>`
``initial_peers`` field.

.. code-block:: xml
   :emphasize-lines: 8-10

    <qos_library name="MyQosLibrary">
        <qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
            <participant_factory_qos>
                <monitoring>
                    <enable>true</enable>
                    <distribution_settings>
                        <dedicated_participant>
                            <collector_initial_peers>
                                <element>192.168.1.2</element>
                            </collector_initial_peers>
                        </dedicated_participant>
                    </distribution_settings>
                </monitoring>
            </participant_factory_qos>
        </qos_profile>
    </qos_library>

If ``collector_initial_peers`` is not specified, or if it is explicitly set to
an empty list, |MONITORINGLIBRARY2| will use the value set in the 
``domain_participant_qos.discovery.initial_peers`` of the QoS profile specified
by ``participant_factory_qos.monitoring.distribution_settings.dedicated_participant.participant_qos_profile_name``
as the initial peers for the |MONITORINGLIBRARY2|'s |DP|.

If both values are present, the value in ``collector_initial_peers`` in the Monitoring
QosPolicy will be used instead of the value of ``initial_peers`` in the Discovery
QosPolicy for the |MONITORINGLIBRARY2|'s |DP|.

.. _section-configuring-qos-entities:

Configuring QoS for Entities
============================

You may want to change the QoS of the *Entities* responsible
for distributing the monitoring data. By default, the DDS *Entities* created by
|MONITORINGLIBRARY2| use the built-in profile ``BuiltinQosLib::Generic.Monitoring2``
(as documented in ``<install dir>/resource/resource/xml/BuiltinProfiles.documentationONLY.xml``) 
to configure their QoS. You can provide a different profile name 
(``MyObservabilityProfile`` in the example below) for each *Entity* 
by changing the Monitoring QoS Policy. It is recommended that if you provide a 
different profile name, you create this profile to inherit from the 
``BuiltinQosLib::Generic.Monitoring2`` or the 
``BuiltinQosLib::Generic.Monitoring2.WAN`` profiles, depending on whether you 
want to connect to a |OCS| instance over a LAN or WAN.

The ``BuiltinQosLib::Generic.Monitoring2.WAN`` profile replaces the
UDPv4 and SHMEM transports with the *RTI* |RWT| in the 
|MONITORINGLIBRARY2| |DP|.

The following example demonstrates the
mechanism for changing the QoS for monitoring *Entities* by changing the 
name of the *Entities*.

.. code-block:: xml
    :emphasize-lines: 48-50, 54-56, 60-62, 67-69, 74-76

    <qos_library name="MyQosLibrary">
        <qos_profile name="MyObservabilityProfile" base_name="BuiltinQosLib::Generic.Monitoring2">
            <domain_participant_qos>
                <participant_name>
                    <!-- Change the name of the Observability
                         DomainParticipant
                    -->
                    <name>Monitoring Participant</name>
                </participant_name>
            </domain_participant_qos>

            <datawriter_qos topic_filter="DCPSEventStatusMonitoring">
                <publication_name>
                    <!-- Change the name of the Observability
                         Event DataWriter
                    -->
                    <name>Monitoring Event DataWriter</name>
                </publication_name>
            </datawriter>

            <datawriter_qos topic_filter="DCPSPeriodicStatusMonitoring">
                <publication_name>
                    <!-- Change the name of the Observability
                         Periodic DataWriter
                    -->
                    <name>Monitoring Periodic DataWriter</name>
                </publication_name>
            </datawriter>

            <datawriter_qos topic_filter="DCPSLoggingStatusMonitoring">
                <publication_name>
                    <!-- Change the name of the Observability
                         Logging DataWriter
                    -->
                    <name>Monitoring Logging DataWriter</name>
                </publication_name>
            </datawriter>
        </qos_profile>

        <qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
            <participant_factory_qos>
                <monitoring>
                    <enable>true</enable>
                    <distribution_settings>
                        <dedicated_participant>
                            <!-- Change the configuration of the
                                 Observability DomainParticipant -->
                            <participant_qos_profile_name>
                                MyQosLibrary::MyObservabilityProfile
                            </participant_qos_profile_name>
                        </dedicated_participant>
                        <!-- Change the configuration of the
                             Observability Publishers -->
                        <publisher_qos_profile_name>
                            MyQosLibrary::MyObservabilityProfile
                        </publisher_qos_profile_name>
                        <event_settings>
                            <!-- Change the configuration of the
                                 Observability Event DataWriter -->
                            <datawriter_qos_profile_name>
                                MyQosLibrary::MyObservabilityProfile
                            </datawriter_qos_profile_name>
                        </event_settings>
                        <periodic_settings>
                            <!-- Change the configuration of the
                                 Observability Periodic DataWriter -->
                            <datawriter_qos_profile_name>
                                MyQosLibrary::MyObservabilityProfile
                            </datawriter_qos_profile_name>
                        </periodic_settings>
                        <logging_settings>
                            <!-- Change the configuration of the
                                 Observability Logging DataWriter -->
                            <datawriter_qos_profile_name>
                                MyQosLibrary::MyObservabilityProfile
                            </datawriter_qos_profile_name>
                        </logging_settings>
                    </distribution_settings>
                </monitoring>
            </participant_factory_qos>
        </qos_profile>
    </qos_library>

.. _section-connecting-to-a-collector-service-over-a-wan:

Connecting to Collector Service Over WAN
========================================

To connect to a |OCS| instance over a WAN, set
``participant_factory_qos.monitoring.distribution_settings.dedicated_participant.participant_qos_profile_name``
to the built-in profile 
``BuiltinQosLib::Generic.Monitoring2.WAN``. This profile is configured to use the
the |RWT|, and it inherits from 
``BuiltinQosLib::Generic.Monitoring2``.

In addition, you must set
``participant_factory_qos.monitoring.distribution_settings.dedicated_participant.collector_initial_peers``
to the locator/address of the |OCS| instance you want to connect to.

.. code-block:: xml
   :emphasize-lines: 8-13

    <qos_library name="MyQosLibrary">
        <qos_profile name="MyApplicationProfile" is_default_participant_factory_profile="true">
            <participant_factory_qos>
                <monitoring>
                    <enable>true</enable>
                    <distribution_settings>
                        <dedicated_participant>
                            <collector_initial_peers>
                                <element>udpv4_wan://38.21.45.34:4500</element>
                            </collector_initial_peers>
                            <participant_qos_profile_name>
                                BuiltinQosLib::Generic.Monitoring2.WAN
                            </participant_qos_profile_name>
                        </dedicated_participant>
                    </distribution_settings>
                </monitoring>
            </participant_factory_qos>
        </qos_profile>
    </qos_library>