.. _section-installation: Installing and Running |OBSERVABILITY_HEADING| ********************************************** |RTI_CONNEXT| |OBSERVABILITY| is not installed as part of |RTI| |PRO| with the exception of |MONITORINGLIBRARY2| which is included in the |RTI| |PRO| target package. |MONITORINGLIBRARY2| is supported in all |Connext| platforms. |OBSERVABILITY| must be downloaded and installed separately. For information on how to obtain the |OBSERVABILITY| package, check the `RTI Customer portal `_, contact support@rti.com, or contact your account team. There is one |OBSERVABILITY| package, as outlined in :numref:`TablePackages`. .. list-table:: Observability Framework Packages :name: TablePackages :widths: 50 50 50 40 :header-rows: 1 * - Package Name - Package Contents - Use Case - Supported Platform * - rti_observability-|CONNEXT_CURRENT_VERSION|-host-x64Linux.rtipkg - The host package contains the files required to run the |OBSERVABILITY| collection, storage, and visualization components using Docker and Docker Compose. This package also includes |OBSERVABILITY| documentation. - Install this package if you need to run the collection, storage, and visualization components. - These components are only supported in Linux. The host package can be installed on a Virtual Machine (VM); for more information, see :ref:`section-docker-compose-env`. In the rest of this chapter, ```` refers to the installation directory for |Connext|. .. important:: |OBSERVABILITY| is an experimental product that includes example configuration files for use with several third-party components (Prometheus, Grafana Loki, and Grafana). This release is an evaluation distribution; use it to explore the new observability features that support |Connext| applications. Do not deploy any |OBSERVABILITY| components in production. Installing the Host Package =========================== There are two ways to install the documentation and files supporting the Docker containers used by |OBSERVABILITY|: using *RTI Launcher* or the ``rtipkginstall`` command-line utility. Prerequisites ------------- The following applications must be installed before installing the experimental |OBSERVABILITY| product. - |CONNEXT| |CONNEXT_CURRENT_VERSION|. For installation instructions, see the :link_connext_dds_pro_ig:`RTI Connext Installation Guide<#installation_guide/Installing.htm>`. - Docker Engine v20.10.x or higher. For installation instructions, see `Docker's Engine installation overview `__. - Docker Compose Plugin v2.x or higher. For installation instructions, see `Docker's installation instructions `__. .. note:: The |OBSERVABILITY| host package has been tested on the platforms noted in :ref:`section-docker-compose-env`. Install from RTI Launcher ------------------------- To install the |OBSERVABILITY| host package from *RTI Launcher*: #. Start *Launcher* from the Start menu, or from the command line using: ``/bin/rtilauncher``. #. From the **Configuration** tab, click **Install RTI Packages**. #. Use the plus (+) sign to add the ``rti_observability--host-x64Linux.rtipkg`` file. #. Click **Install**. Install from the Command Line ----------------------------- To install the |OBSERVABILITY| host package from the command line, run: .. code-block:: text $ /bin/rtipkginstall //rti_observability--host-x64Linux.rtipkg .. _section-install-components: Configuring, Running, and Removing |OBSERVABILITY_HEADING| Components Using Docker Compose ========================================================================================== The telemetry data forwarded by |MONITORINGLIBRARY2| is processed, stored, and visualized using the following components: - `RTI Observability Collector Service `_ - `Prometheus `_ - `Grafana Loki `_ - `Grafana `_ - `OpenTelemetry Collector `_ [Optional]: |OBSERVABILITY| can be configured to launch an instance of OpenTelemetry Collector that will store the telemetry data in Prometheus and Loki instead of this being done by the |RTI| |OCS|. In this configuration mode, |OCS| sends the data to OpenTelemetry Collector. - `NGINX `_ [Optional]: when using security the |OBSERVABILITY| runs and instance of NGINX to secure communications with Grafana Loki and the OpenTelemetry Collector. The files required to run these components are installed by the |OBSERVABILITY| host package. In this release, the collection, storage, and visualization components only run in a single Linux host using Docker and Docker Compose. Future releases will offer the ability to install the components independently without using Docker. |OBSERVABILITY| can be deployed with or without using the OpenTelemetry Collector. Both deployment options can be configured to be secure or non-secure and to work on a LAN or WAN. For additional information on the deployment options, see :ref:`section-docker-compose-prepackaged`. .. warning:: |OBSERVABILITY| uses third-party software that is subject to each product's license terms and conditions. IT IS YOUR RESPONSIBILITY TO ENSURE THAT YOUR USE OF THIRD-PARTY SOFTWARE COMPLIES WITH THE CORRESPONDING THIRD-PARTY LICENSE TERMS AND CONDITIONS. .. _section-configure-docker: Configuring the Docker Workspace for |OBSERVABILITY_HEADING| ------------------------------------------------------------ Before creating and running the Docker containers for |OBSERVABILITY|, the associated configuration files that comprise the Docker workspace must be created and copied to the ``rti_workspace//user_config/observability`` directory. This is done using the ``/bin/rtiobservability`` script. There are several optional, user-defined variables you can use to configure |OBSERVABILITY|. These variables are specified in a JSON file. .. note:: To reconfigure an existing Docker workspace you must first remove the existing workspace as described in section :ref:`section-remove-observability-workspace`. .. _section-configure-json-file: Configure the JSON File ^^^^^^^^^^^^^^^^^^^^^^^ Before creating your workspace, you will need to provide your configuration using a JSON file. This file can contain all the specific ports, names, and certificates to be used by the different services. The following default JSON file is included in the installation folder at ``/resource/app/app_support/observability/default.json``. You can copy this file to another location, then modify it as needed to create the |OBSERVABILITY| configuration for your environment. Alternately, you can create your own JSON file. .. code-block:: JSON { "hostname": "localhost", "observabilityDomain": 2 } :numref:`TableJSONConfigurationfile` :ref:`TableJSONConfigurationfile` describes all of the JSON configuration fields and default values. .. note:: All of the JSON configuration fields are optional except ``hostname``. If configuration for ``securityConfig`` is required, then all its fields must be provided. .. list-table:: JSON Configuration file :name: TableJSONConfigurationfile :widths: 50 50 10 10 :header-rows: 1 * - Field Name - Description - Type - Default Value * - hostname - Hostname to be used to configure all of the services. This field is required. - String - N/A * - observabilityDomain - DDS Domain to be used to exchange Observability data. - int - 2 * - **lgpStackConfig** - - - * - lgpStackConfig.grafanaPort - The Grafana server port. This is the port that the Grafana service listens to. - int - 3000 * - lgpStackConfig.prometheusPort - The Prometheus server port. This is the port that the Prometheus service listens to. - int - 9090 * - lgpStackConfig.lokiPort - The Loki server port. This is the port that the Loki service listens to. - int - 3100 * - **collectorConfig** - - - * - collectorConfig.prometheusExporterPort - The |OCS| Prometheus endpoint port for exporting telemetry data to Prometheus. This is the port that the Prometheus endpoint service listens to and uses to provide telemetry data via scrapes from the Prometheus server. - int - 19090 * - collectorConfig.controlPort - The |OCS| server port for control commands. This is the port that the service listens to. - int - 19098 * - collectorConfig.controlPublicHostname - The |OCS| public server hostname for control commands. - String - `hostname` * - collectorConfig.controlPublicPort - The |OCS| public server port for control commands. This is the port exposed to the public network. - int - `collectorConfig.controlPort` * - collectorConfig.rtwPublicAddress - The WAN public address used by |RWT|. - String - `hostname` * - collectorConfig.rtwPort - The WAN port used by |RWT|. This is both the private port where |RWT| receives data, and the public port exposed to the public network. - int - 30000 * - **otelConfig** - - - * - otelConfig.otelHttpReceiverPort - The OpenTelemetry Collector server port. This is the port the OpenTelemetry Collector listens to for telemetry data. - int - N/A * - **securityConfig** - - - * - securityConfig.basicAuthUsername - Username used for HTTP Basic authentication. - String - N/A * - securityConfig.basicAuthPassword - Password used for HTTP Basic authentication. - String - N/A * - **securityConfig.httpsSecurity** - - - * - securityConfig.httpsSecurity.caCertificate - Certificate authority (CA) used to verify the SSL certificates signed by this CA. - String - N/A * - securityConfig.httpsSecurity.serverCertificate - Server Certificate used with HTTPs. - String - N/A * - securityConfig.httpsSecurity.serverKey - Server Key for the Server Certificate. - String - N/A * - **securityConfig.ddsSecurity** - - - * - securityConfig.ddsSecurity.identityCaCertificate - File to be passed as ``dds.sec.auth.identity_ca`` - String - N/A * - securityConfig.ddsSecurity.permissionsCaCertificate - File to be passed as ``dds.sec.access.permissions_ca`` - String - N/A * - securityConfig.ddsSecurity.identityCertificate - File to be passed as ``dds.sec.auth.identity_certificate`` - String - N/A * - securityConfig.ddsSecurity.identityKey - File to be passed as ``dds.sec.auth.private_key`` - String - N/A * - securityConfig.ddsSecurity.signedPermissionsFile - File to be passed as ``dds.sec.access.permissions`` - String - N/A * - securityConfig.ddsSecurity.signedGovernanceFile - File to be passed as ``dds.sec.access.governance`` - String - N/A Complete examples of both secure and non-secure configurations of the |OBSERVABILITY| may be found in the section :ref:`section-configure-operation-mode` of the Getting started Guide. An example of a fully-defined JSON file, with security and OpenTelemetry configured, follows. You can follow this example to create your own custom configuration. .. code-block:: JSON { "hostname": "localhost", "observabilityDomain": 2, "lgpStackConfig": { "grafanaPort": 3000, "prometheusPort": 9090, "lokiPort": 3100 }, "collectorConfig": { "prometheusExporterPort": 19090, "controlPort": 19098 }, "otelConfig": { "otelHttpReceiverPort": 4318 }, "securityConfig": { "basicAuthUsername": "yourusername", "basicAuthPassword": "yourpassword", "httpsSecurity": { "caCertificate": "path/to/ca_cert.pem", "serverCertificate": "path/to/server_cert.pem", "serverKey": "path/to/server_key.pem" } "ddsSecurity": { "identityCaCertificate": "path/to/identityCaCert.pem", "permissionsCaCertificate": "path/to/permissionsCaCert.pem", "identityCertificate": "path/to/identityCert.pem", "identityKey": "path/to/identityKey.pem", "signedPermissionsFile": "path/to/signedPermissions.p7s", "signedGovernanceFile": "path/to/signedGovernance.p7s" } } } .. note:: The tilde (~) Linux shortcut for a user home directory is not supported in the JSON configuration file. Run the Observability script to create the Observability workspace ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To configure the Docker workspace for |OBSERVABILITY|, run the ``/bin/rtiobservability`` script with the ``-c `` option. .. warning:: The ``/bin/rtiobservability`` script requires Python3. If any Python package dependencies are missing, the script detects them and provides the command to install them. The required packages are detailed in the ``/resource/app/app_support/observability/requirements.txt`` file. The following image shows the types of errors returned when running the script with a missing dependency. .. code-block:: console $ rtiobservability -c NonSecureLAN.json ******************************************************************************** * * The Observability Docker Containers created by this script may include images * from third-parties, including: * * Prometheus * (https://hub.docker.com/r/prom/prometheus) * Grafana Loki * (https://hub.docker.com/r/grafana/loki) * Grafana * (https://hub.docker.com/r/grafana/grafana-enterprise) * NGINX * (https://hub.docker.com/_/nginx) * OpenTelemetry Collector * (https://hub.docker.com/r/otel/opentelemetry-collector-contrib) * * Such third-party software is subject to third-party license terms and * conditions. IT IS YOUR RESPONSIBILITY TO ENSURE THAT YOUR USE OF THIRD-PARTY * SOFTWARE COMPLIES WITH THE CORRESPONDING THIRD-PARTY LICENSE TERMS AND * CONDITIONS. * ******************************************************************************** Do you wish to continue setting up the Connext Observability Framework[Y/n]? Y Generating configuration for the Connext Observability Framework 2023-08-03 01:36:46,017 - root - ERROR - Some requirements are missing: No module named 'jinja2'. 2023-08-03 01:36:46,017 - root - ERROR - Please install them with the following command: pip3 install -r /home/test/rti_connext_dds-7.2.0/resource/app/app_support/observability/requirements.txt 1. Run ``/bin/rtiobservability -c `` to configure the Docker workspace. .. code-block:: console $ rtiobservability -c NonSecureLAN.json ******************************************************************************** * * The Observability Docker Containers created by this script may include images * from third-parties, including: * * Prometheus * (https://hub.docker.com/r/prom/prometheus) * Grafana Loki * (https://hub.docker.com/r/grafana/loki) * Grafana * (https://hub.docker.com/r/grafana/grafana-enterprise) * NGINX * (https://hub.docker.com/_/nginx) * OpenTelemetry Collector * (https://hub.docker.com/r/otel/opentelemetry-collector-contrib) * * Such third-party software is subject to third-party license terms and * conditions. IT IS YOUR RESPONSIBILITY TO ENSURE THAT YOUR USE OF THIRD-PARTY * SOFTWARE COMPLIES WITH THE CORRESPONDING THIRD-PARTY LICENSE TERMS AND * CONDITIONS. * ******************************************************************************** Do you wish to continue setting up the Connext Observability Framework[Y/n]? 2. Select ``Y/y`` (or simply enter) to acknowledge the license statement. .. code-block:: console Do you wish to continue setting up the Connext Observability Framework[Y/n]? y Generating configuration for the Connext Observability Framework 2023-07-19 19:28:10,277 - exporter - INFO - Config: { "hostname": "localhost", "observabilityDomain": 2, "otelConfig": null, "lgpStackConfig": { "grafanaPort": 3000, "prometheusPort": 9090, "lokiPort": 3100 }, "collectorConfig": { "prometheusExporterPort": 19090, "controlPort": 19098, "rtwPort": null }, "securityConfig": null } If you attempt to configure an existing Docker workspace for |OBSERVABILITY|, you will see the following warning. .. code-block:: console $ rtiobservability -c NonSecureOTelLAN.json ******************************************************************************** * * The Connext Observability Framework already exist in: * /home/rtrentini/rti_workspace/7.2.0/user_config/observability * * Remove or rename the directory manually if you want to start over ... * ******************************************************************************** .. _section-initializing-and-running-docker: Initialize and Run Docker Containers ------------------------------------------ .. important:: An RTI license is always required to run |OCS| in a Docker container. The following table indicates the RTI licenses required based on your answers to the questions in the first two columns. .. list-table:: License Requirements Table :name: License Requirements Table :widths: 25 25 25 :header-rows: 1 * - Do you need to secure telemetry data exchanged between applications and |OCS| using |RTI_SP_PRODUCT|? - Do you need to send telemetry data to |OCS| over the WAN using |RWT|? - Required License * - NO - NO - |PRO| * - YES - NO - |PRO| and |RTI_SP_PRODUCT| * - YES - YES - |PRO| & |RTI_SP_PRODUCT| & |CDS| & |RWT| * - NO - YES - |PRO| & |CDS| & |RWT| For instructions on how to install a license file, see :link_connext_dds_pro_ig:`Installing the License File<#installation_guide/License_Management.htm>` in the *RTI Connext Installation Guide*. After the Docker workspace is configured and created, run ``/bin/rtiobservability -i`` to initialize and run the Docker containers for |OBSERVABILITY|. The ``-i`` option calls ``docker compose up -d`` to create the required storage volumes and containers, then starts the containers. .. code-block:: console $ rtiobservability -i ... using Docker version 24.0.4, build 3713ee1. ... using Docker Compose version v2.19.1. Initializing and running the Connext Observability Framework [+] Running 6/6 ✔ Volume "observability_grafana_data" Created 0.0s ✔ Volume "observability_prometheus_data" Created 0.0s ✔ Container collector_service_observability Started 0.2s ✔ Container prometheus_observability Started 0.2s ✔ Container grafana_observability Started 0.2s ✔ Container loki_observability Started 0.2s Three things happen upon running ``/bin/rtiobservability`` with the ``-i`` option. #. The Docker images for Grafana Loki, Prometheus, Grafana, and |OCS| are pulled from Docker Hub to your local Docker image store. Note that this will only happen if there are no local images found. #. The Docker data volumes are created for the Prometheus and Grafana data storage. #. Docker containers for |OBSERVABILITY| are started for the four components (Loki, Prometheus, Grafana, and |OCS|). At this point, the Docker containers used by |OBSERVABILITY| are started and all components should be running. Verify Docker Containers are Running ------------------------------------------ To verify that all Docker containers used by |OBSERVABILITY| are running, run the command ``docker ps -a``. Examine the STATUS column and verify that all containers report a status of ``Up``, as shown below. .. code-block:: console CONTAINER ID IMAGE COMMAND CREATED STATUS NAMES 6651d7ed9810 prom/prometheus:v2.37.5 "/bin/prometheus --c…" 5 minutes ago Up 5 minutes prometheus_observability 25050d16b1b5 grafana/grafana-enterprise:9.2.1-ubuntu "/run.sh" 5 minutes ago Up 5 minutes grafana_observability 08611ea9b255 rticom/collector-service: "/rti_connext_dds-7.…" 5 minutes ago Up 5 minutes collector_service_observability 55568de5120f grafana/loki:2.7.0 "/usr/bin/loki --con…" 5 minutes ago Up 5 minutes loki_observability When a container does not start, the STATUS column displays ``Restarting`` to indicate the ``prometheus-observability`` container failed to start and repeatedly tried to restart. .. code-block:: console CONTAINER ID IMAGE COMMAND CREATED STATUS NAMES 08f75e0fadb2 prom/prometheus:v2.37.5 "/bin/prometheus --c…" 5 minutes ago Restarting (1) 27 seconds ago prometheus_observability 9a3964b561ec grafana/loki:2.7.0 "/usr/bin/loki --con…" 5 minutes ago Up 5 minutes loki_observability b6a6ffa201f3 rticom/collector-service: "/rti_connext_dds-7.…" 5 minutes ago Up 5 minutes collector_service_observability 26658f76cfdc grafana/grafana-enterprise:9.2.1-ubuntu "/run.sh" 5 minutes ago Up 5 minutes grafana_observability If a container fails to start, refer to section :ref:`section-docker-containers-fail-to-start` for troubleshooting suggestions. .. _section-configure-grafana: Configure Grafana ------------------- Initial Login ^^^^^^^^^^^^^ To access |DASHBOARDS|, open a new browser window and go to **http://:** to access Grafana (3000 is the default grafanaPort). Log in using the credentials **admin : admin**, then change the password when prompted. If you are using a secure configuration, the url to access Grafana will be **https://:** and the Grafana credentials will be the values configured in the **basicAuthUsername** and **basicAuthPassword** fields in the JSON configuration. .. figure:: static/grafana_login.png :figwidth: 90 % :alt: Grafana Login Screen :name: GrafanaLogin :align: center Once you are logged in you will see the **RTI Alert Home** dashboard. .. figure:: static/dashboard_home.png :figwidth: 90 % :alt: Grafana Homne Dashboard :name: GrafanaHome :align: center .. _section-options: Configuration Options ^^^^^^^^^^^^^^^^^^^^^ You can configure the Grafana dashboard to meet your specific needs. For more information, refer to the Grafana article `Use dashboards `__. Create Accounts (Optional) """""""""""""""""""""""""" You can create additional users as needed. Refer to the Grafana article `Manage Grafana Users `__ for information about user roles and permissions. Change the Default Time Range (Optional) """""""""""""""""""""""""""""""""""""""" The default visualization time range can be modified. The default relative time range is one hour. You may want to update the range as follows: #. Go to the **Alert Home** dashboard, #. From the toolbar, select the **time picker**. #. Select the desired time range from the dropdown list. The dashboard refreshes to display the selected time range. .. figure:: static/grafana_time_range.png :figwidth: 90 % :alt: Grafana Dashboard Time Range :name: GrafanaTimeRange :align: center #. From the toolbar, select **Save dashboard**. .. figure:: static/grafana_save_dashboard_icon.png :figwidth: 90 % :alt: Grafana Save Icon :name: GrafanaSaveIcon :align: center #. In the **Save dashboard** dialog, select **Save current time range as dashboard default** and then click **Save**. .. figure:: static/grafana_save_dashboard.png :figwidth: 90 % :alt: Grafana Save Dashboard Dialog :name: GrafanaSaveDialog :align: center #. To confirm the new time range, navigate to another dashboard and then click the Home icon at the top left to go back to the Alert Home dashboard. Stop Docker Containers -------------------------- Once |OBSERVABILITY| Docker containers are running, you can stop them by running ``/bin/rtiobservability -t``. The ``-t`` option terminates the running Docker containers for |OBSERVABILITY| by calling ``docker compose stop``. .. code-block:: console $ rtiobservability -t ... using Docker version 24.0.4, build 3713ee1. ... using Docker Compose version v2.19.1. Terminating the running Connext Observability Framework [+] Stopping 4/4 ✔ Container collector_service_observability Stopped 10.1s ✔ Container prometheus_observability Stopped 0.1s ✔ Container grafana_observability Stopped 0.2s ✔ Container loki_observability Stopped 2.1s This command stops the existing Docker containers for |OBSERVABILITY|, but leaves associated storage volumes and configuration for a future run. Start Existing Docker Containers --------------------------------------- To restart existing Docker containers used by |OBSERVABILITY|, run ``/bin/rtiobservability -s``. The ``-s`` option starts existing Docker containers for |OBSERVABILITY| by calling ``docker compose start``. .. code-block:: console $ rtiobservability -s ... using Docker version 24.0.4, build 3713ee1. ... using Docker Compose version v2.19.1. Starting the existing Connext Observability Framework [+] Running 4/4 ✔ Container prometheus_observability Started 0.1s ✔ Container collector_service_observability Started 0.1s ✔ Container grafana_observability Started 0.1s ✔ Container loki_observability Started 0.2s This command starts any existing Docker containers created by |OBSERVABILITY|. .. _section-stop-and-remove-docker-containers: Stop and Remove Docker Containers ------------------------------------------------- To clean up, or stop and remove, all Docker containers and storage volumes used by |OBSERVABILITY|, run ``/bin/rtiobservability -d``. The ``-d`` option stops and removes Docker containers for |OBSERVABILITY| by calling ``docker compose down``, and subsequently removes storage volumes. .. warning:: Running ``/bin/rtiobservability -d`` removes all Docker containers and storage volumes used by |OBSERVABILITY|. This command removes all changes to your current |OBSERVABILITY| Docker environment including: - metric data in Prometheus - log data in Loki - all Grafana user and dashboard configurations .. code-block:: console $ rtiobservability -d ... using Docker version 24.0.4, build 3713ee1. ... using Docker Compose version v2.19.1. ******************************************************************************** * * You have requested to clean up and remove the existing Connext Observability * Framework. If you continue you will lose all changes to your current * environment including: * - metric data in Prometheus * - log data in Loki * - all Grafana user and dashboard configuration * ******************************************************************************** Do you wish to continue cleaning and removing the existing Connext Observability Framework[y/N]? When prompted to confirm that you want to remove all Docker containers and storage volumes for |OBSERVABILITY|: * Select ``N/n`` (or simply enter) to cancel the cleanup. .. code-block:: console Do you wish to continue cleaning and removing the existing Connext Observability Framework[y/N]? n Cleaning up and removing the existing Connext Observability Framework canceled. * Select ``Y/y`` to proceed with the cleanup and remove all Docker containers and storage volumes used by |OBSERVABILITY|. .. code-block:: console Do you wish to continue cleaning and removing the existing Connext Observability Framework[y/N]? y Cleaning up and removing the existing Connext Observability Framework [+] Running 4/5 ✔ Container prometheus_observability Removed 0.1s ✔ Container grafana_observability Removed 0.1s ✔ Container loki_observability Removed 1.5s ✔ Container collector_service_observability Removed 10.1s observability_grafana_data observability_prometheus_data .. _section-remove-observability-workspace: Removing the Docker Workspace for |OBSERVABILITY_HEADING| --------------------------------------------------------- There may be a time that you need to remove your existing Docker Workspace for |OBSERVABILITY|. This could be because you want to change the existing configuration in some way. Things that you would want to change could include hostname, port configurations, and enabling or disabling security. The ``rtiobservability`` script will not overwrite an existing workspace. This prevents inadvertently corrupting or deleting an existing configuration. The following steps should be followed to remove an existing workspace to allow re-configuration. #. You must first stop and remove any existing containers created with the current workspace configuration as detailed in section :ref:`section-stop-and-remove-docker-containers`. |br| |br| #. Once the docker containers have been stopped and removed you must manually delete the ``rti_workspace//user_config/observability`` directory. .. tabs:: .. group-tab:: Linux .. code-block:: console $ rm -rf //user_config/observability