Note
RTI® Connext® Observability Framework is now considered production-ready.
Observability Framework includes example configuration files for use with several third-party components (Prometheus®, Grafana Loki™, Grafana®, NGINX®, and OpenTelemetry™ Collector). This release supports Connext applications with new observability features. For support, you may contact support@rti.com.
Feel confident deploying Observability Framework components in production environments.
5. Installing and Running Observability Framework
RTI Connext Observability Framework is not installed as part of RTI Connext Professional with the exception of Monitoring Library 2.0 which is included in the RTI Connext Professional target package. Monitoring Library 2.0 is supported in all Connext platforms. Observability Framework must be downloaded and installed separately. For information on how to obtain the Observability Framework package, check the RTI Customer portal, contact support@rti.com, or contact your account team.
There is one Observability Framework package, as outlined in Table 5.1.
Package Name |
Package Contents |
Use Case |
Supported Platform |
|---|---|---|---|
rti_observability-7.3.0.2-host-x64Linux.rtipkg |
The host package contains the files required to run the Observability Framework collection, storage, and visualization components using Docker and Docker Compose. This package also includes Observability Framework documentation. |
Install this package if you need to run the collection, storage, and visualization components. |
These components are only supported in Linux. The host package can be installed on a Virtual Machine (VM); for more information, see Supported Docker Compose Environments. |
In the rest of this chapter, <installdir> refers to the installation
directory for Connext.
5.1. Installing the Host Package
There are two ways to install the documentation and files supporting the
Docker containers used by Observability Framework: using RTI Launcher
or the rtipkginstall command-line utility.
5.1.1. Prerequisites
The following applications must be installed before installing Observability Framework.
Connext 7.3.0.2. For installation instructions, see the RTI Connext Installation Guide.
Docker Engine v20.10.x or higher. For installation instructions, see Docker’s Engine installation overview.
Docker Compose Plugin v2.x or higher. For installation instructions, see Docker’s installation instructions.
Note
The Observability Framework host package has been tested on the platforms noted in Supported Docker Compose Environments.
5.1.2. Install from RTI Launcher
To install the Observability Framework host package from RTI Launcher:
Start Launcher from the Start menu, or from the command line using:
<installdir>/bin/rtilauncher.From the Configuration tab, click Install RTI Packages.
Use the plus (+) sign to add the
rti_observability-<version>-host-x64Linux.rtipkgfile.Click Install.
5.1.3. Install from the Command Line
To install the Observability Framework host package from the command line, run:
$ <installdir>/bin/rtipkginstall /<path-to-observability-framework-file>/rti_observability-<version>-host-x64Linux.rtipkg
5.2. Configuring, Running, and Removing Observability Framework Components Using Docker Compose
The telemetry data forwarded by Monitoring Library 2.0 is processed, stored, and visualized using the following components:
OpenTelemetry Collector [Optional]: Observability Framework can be configured to launch an instance of OpenTelemetry Collector that will store the telemetry data in Prometheus and Loki instead of this being done by the RTI Observability Collector Service. In this configuration mode, Observability Collector Service sends the data to OpenTelemetry Collector.
NGINX [Optional]: when using security the Observability Framework runs and instance of NGINX to secure communications with Grafana Loki and the OpenTelemetry Collector.
The files required to run these components are installed by the Observability Framework host package. In this release, the collection, storage, and visualization components only run in a single Linux host using Docker and Docker Compose. Future releases will offer the ability to install the components independently without using Docker.
Observability Framework can be deployed with or without using the OpenTelemetry Collector. Both deployment options can be configured to be secure or non-secure and to work on a LAN or WAN. For additional information on the deployment options, see Docker Compose (Prepackaged).
Warning
Observability Framework uses third-party software that is subject to each product’s license terms and conditions. IT IS YOUR RESPONSIBILITY TO ENSURE THAT YOUR USE OF THIRD-PARTY SOFTWARE COMPLIES WITH THE CORRESPONDING THIRD-PARTY LICENSE TERMS AND CONDITIONS.
5.2.1. Configuring the Docker Workspace for Observability Framework
Before creating and running the Docker containers for Observability Framework,
the associated configuration files that comprise the Docker
workspace must be created and copied to the
rti_workspace/<version>/user_config/observability directory. This is done using
the <installdir>/bin/rtiobservability script.
There are several optional, user-defined variables you can use to configure Observability Framework. These variables are specified in a JSON file.
Note
To reconfigure an existing Docker workspace you must first remove the existing workspace as described in section Removing the Docker Workspace for Observability Framework.
5.2.1.1. Configure the JSON File
Before creating your workspace, you will need to provide your configuration using a JSON file. This file can contain all the specific ports, names, and certificates to be used by the different services.
The following default JSON file is included in the installation folder at
<rti_installation>/resource/app/app_support/observability/default.json. You
can copy this file to another location, then modify it as needed to create
the Observability Framework configuration for your environment. Alternately,
you can create your own JSON file.
{
"hostname": "localhost",
"observabilityDomain": 2
}
Table 5.2 JSON Configuration file describes all of the JSON configuration fields and default values.
Note
All of the JSON configuration fields are optional except hostname. If configuration for securityConfig is
required, then all its fields must be provided.
Field Name |
Description |
Type |
Default Value |
|---|---|---|---|
hostname |
Hostname to be used to configure all of the services. This field is required. |
String |
N/A |
observabilityDomain |
DDS Domain to be used to exchange Observability data. |
int |
2 |
lgpStackConfig |
|||
lgpStackConfig.grafanaPort |
The Grafana server port. This is the port that the Grafana service listens to. |
int |
3000 |
lgpStackConfig.prometheusPort |
The Prometheus server port. This is the port that the Prometheus service listens to. |
int |
9090 |
lgpStackConfig.lokiPort |
The Loki server port. This is the port that the Loki service listens to. |
int |
3100 |
collectorConfig |
|||
collectorConfig.prometheusExporterPort |
The Observability Collector Service Prometheus endpoint port for exporting telemetry data to Prometheus. This is the port that the Prometheus endpoint service listens to and uses to provide telemetry data via scrapes from the Prometheus server. |
int |
19090 |
collectorConfig.controlPort |
The Observability Collector Service server port for control commands. This is the port that the service listens to. |
int |
19098 |
collectorConfig.controlPublicHostname |
The Observability Collector Service public server hostname for control commands. |
String |
hostname |
collectorConfig.controlPublicPort |
The Observability Collector Service public server port for control commands. This is the port exposed to the public network. |
int |
collectorConfig.controlPort |
collectorConfig.rtwPublicAddress |
The WAN public address used by Real-Time WAN Transport. |
String |
hostname |
collectorConfig.rtwPort |
The WAN port used by Real-Time WAN Transport. This is both the private port where Real-Time WAN Transport receives data, and the public port exposed to the public network. |
int |
30000 |
otelConfig |
|||
otelConfig.otelHttpReceiverPort |
The OpenTelemetry Collector server port. This is the port the OpenTelemetry Collector listens to for telemetry data. |
int |
N/A |
securityConfig |
|||
securityConfig.basicAuthUsername |
Username used for HTTP Basic authentication. |
String |
N/A |
securityConfig.basicAuthPassword |
Password used for HTTP Basic authentication. |
String |
N/A |
securityConfig.httpsSecurity |
|||
securityConfig.httpsSecurity.caCertificate |
Certificate authority (CA) used to verify the SSL certificates signed by this CA. |
String |
N/A |
securityConfig.httpsSecurity.serverCertificate |
Server Certificate used with HTTPs. |
String |
N/A |
securityConfig.httpsSecurity.serverKey |
Server Key for the Server Certificate. |
String |
N/A |
securityConfig.ddsSecurity |
|||
securityConfig.ddsSecurity.identityCaCertificate |
File to be passed as |
String |
N/A |
securityConfig.ddsSecurity.permissionsCaCertificate |
File to be passed as |
String |
N/A |
securityConfig.ddsSecurity.identityCertificate |
File to be passed as |
String |
N/A |
securityConfig.ddsSecurity.identityKey |
File to be passed as |
String |
N/A |
securityConfig.ddsSecurity.signedPermissionsFile |
File to be passed as |
String |
N/A |
securityConfig.ddsSecurity.signedGovernanceFile |
File to be passed as |
String |
N/A |
Complete examples of both secure and non-secure configurations of the Observability Framework may be found in the section Configure Observability Framework for the Appropriate Operation Mode of the Getting started Guide.
An example of a fully-defined JSON file, with security and OpenTelemetry configured, follows. You can follow this example to create your own custom configuration.
{
"hostname": "localhost",
"observabilityDomain": 2,
"lgpStackConfig": {
"grafanaPort": 3000,
"prometheusPort": 9090,
"lokiPort": 3100
},
"collectorConfig": {
"prometheusExporterPort": 19090,
"controlPort": 19098
},
"otelConfig": {
"otelHttpReceiverPort": 4318
},
"securityConfig": {
"basicAuthUsername": "yourusername",
"basicAuthPassword": "yourpassword",
"httpsSecurity": {
"caCertificate": "path/to/ca_cert.pem",
"serverCertificate": "path/to/server_cert.pem",
"serverKey": "path/to/server_key.pem"
}
"ddsSecurity": {
"identityCaCertificate": "path/to/identityCaCert.pem",
"permissionsCaCertificate": "path/to/permissionsCaCert.pem",
"identityCertificate": "path/to/identityCert.pem",
"identityKey": "path/to/identityKey.pem",
"signedPermissionsFile": "path/to/signedPermissions.p7s",
"signedGovernanceFile": "path/to/signedGovernance.p7s"
}
}
}
Note
The tilde (~) Linux shortcut for a user home directory is not supported in the JSON configuration file.
5.2.1.2. Run the Observability script to create the Observability workspace
To configure the Docker workspace for Observability Framework, run the
<installdir>/bin/rtiobservability script with the -c <json_file> option.
Warning
The <installdir>/bin/rtiobservability script requires Python3. If any
Python package dependencies are missing, the script detects them and provides
the command to install them. The required packages are detailed in the
<installdir>/resource/app/app_support/observability/requirements.txt file.
The following image shows the types of errors returned when running the
script with a missing dependency.
$ rtiobservability -c NonSecureLAN.json
********************************************************************************
*
* The Observability Docker Containers created by this script may include images
* from third-parties, including:
*
* Prometheus
* (https://hub.docker.com/r/prom/prometheus)
* Grafana Loki
* (https://hub.docker.com/r/grafana/loki)
* Grafana
* (https://hub.docker.com/r/grafana/grafana-enterprise)
* NGINX
* (https://hub.docker.com/_/nginx)
* OpenTelemetry Collector
* (https://hub.docker.com/r/otel/opentelemetry-collector-contrib)
*
* Such third-party software is subject to third-party license terms and
* conditions. IT IS YOUR RESPONSIBILITY TO ENSURE THAT YOUR USE OF THIRD-PARTY
* SOFTWARE COMPLIES WITH THE CORRESPONDING THIRD-PARTY LICENSE TERMS AND
* CONDITIONS.
*
********************************************************************************
Do you wish to continue setting up the Connext Observability Framework[Y/n]? Y
Generating configuration for the Connext Observability Framework
2023-08-03 01:36:46,017 - root - ERROR - Some requirements are missing: No module named 'jinja2'.
2023-08-03 01:36:46,017 - root - ERROR - Please install them with the following command:
pip3 install -r /home/test/rti_connext_dds-7.2.0/resource/app/app_support/observability/requirements.txt
Run
<installdir>/bin/rtiobservability -c <json_file>to configure the Docker workspace.$ rtiobservability -c NonSecureLAN.json ******************************************************************************** * * The Observability Docker Containers created by this script may include images * from third-parties, including: * * Prometheus * (https://hub.docker.com/r/prom/prometheus) * Grafana Loki * (https://hub.docker.com/r/grafana/loki) * Grafana * (https://hub.docker.com/r/grafana/grafana-enterprise) * NGINX * (https://hub.docker.com/_/nginx) * OpenTelemetry Collector * (https://hub.docker.com/r/otel/opentelemetry-collector-contrib) * * Such third-party software is subject to third-party license terms and * conditions. IT IS YOUR RESPONSIBILITY TO ENSURE THAT YOUR USE OF THIRD-PARTY * SOFTWARE COMPLIES WITH THE CORRESPONDING THIRD-PARTY LICENSE TERMS AND * CONDITIONS. * ******************************************************************************** Do you wish to continue setting up the Connext Observability Framework[Y/n]?
Select
Y/y(or simply enter) to acknowledge the license statement.Do you wish to continue setting up the Connext Observability Framework[Y/n]? y Generating configuration for the Connext Observability Framework 2023-07-19 19:28:10,277 - exporter - INFO - Config: { "hostname": "localhost", "observabilityDomain": 2, "otelConfig": null, "lgpStackConfig": { "grafanaPort": 3000, "prometheusPort": 9090, "lokiPort": 3100 }, "collectorConfig": { "prometheusExporterPort": 19090, "controlPort": 19098, "rtwPort": null }, "securityConfig": null }
If you attempt to configure an existing Docker workspace for Observability Framework, you will see the following warning.
$ rtiobservability -c NonSecureOTelLAN.json ******************************************************************************** * * The Connext Observability Framework already exist in: * /home/rtrentini/rti_workspace/7.2.0/user_config/observability * * Remove or rename the directory manually if you want to start over ... * ********************************************************************************
5.2.2. Initialize and Run Docker Containers
Important
An RTI license is always required to run Observability Collector Service in a Docker container. The following table indicates the RTI licenses required based on your answers to the questions in the first two columns.
Do you need to secure telemetry data exchanged between applications and Observability Collector Service using Security Plugins? |
Do you need to send telemetry data to Observability Collector Service over the WAN using Real-Time WAN Transport? |
Required License |
|---|---|---|
NO |
NO |
Connext Professional |
YES |
NO |
Connext Professional and Security Plugins |
YES |
YES |
Connext Professional & Security Plugins & Cloud Discovery Service & Real-Time WAN Transport |
NO |
YES |
Connext Professional & Cloud Discovery Service & Real-Time WAN Transport |
For instructions on how to install a license file, see Installing the License File in the RTI Connext Installation Guide.
After the Docker workspace is configured and created, run
<installdir>/bin/rtiobservability -i to initialize and run the Docker
containers for Observability Framework. The -i option calls
docker compose up -d to create the required
storage volumes and containers, then starts the containers.
$ rtiobservability -i ... using Docker version 24.0.4, build 3713ee1. ... using Docker Compose version v2.19.1. Initializing and running the Connext Observability Framework [+] Running 6/6 ✔ Volume "observability_grafana_data" Created 0.0s ✔ Volume "observability_prometheus_data" Created 0.0s ✔ Container collector_service_observability Started 0.2s ✔ Container prometheus_observability Started 0.2s ✔ Container grafana_observability Started 0.2s ✔ Container loki_observability Started 0.2s
Three things happen upon running <installdir>/bin/rtiobservability
with the -i option.
The Docker images for Grafana Loki, Prometheus, Grafana, and Observability Collector Service are pulled from Docker Hub to your local Docker image store. Note that this will only happen if there are no local images found.
The Docker data volumes are created for the Prometheus and Grafana data storage.
Docker containers for Observability Framework are started for the four components (Loki, Prometheus, Grafana, and Observability Collector Service).
At this point, the Docker containers used by Observability Framework are started and all components should be running.
5.2.3. Verify Docker Containers are Running
To verify that all Docker containers used by Observability Framework are
running, run the command docker ps -a. Examine the STATUS column and
verify that all containers report a status of Up, as shown below.
CONTAINER ID IMAGE COMMAND CREATED STATUS NAMES
6651d7ed9810 prom/prometheus:v2.37.5 "/bin/prometheus --c…" 5 minutes ago Up 5 minutes prometheus_observability
25050d16b1b5 grafana/grafana-enterprise:9.2.1-ubuntu "/run.sh" 5 minutes ago Up 5 minutes grafana_observability
08611ea9b255 rticom/collector-service:<version> "/rti_connext_dds-7.…" 5 minutes ago Up 5 minutes collector_service_observability
55568de5120f grafana/loki:2.7.0 "/usr/bin/loki --con…" 5 minutes ago Up 5 minutes loki_observability
When a container does not start,
the STATUS column displays Restarting to indicate the
prometheus-observability container failed to start and repeatedly tried
to restart.
CONTAINER ID IMAGE COMMAND CREATED STATUS NAMES
08f75e0fadb2 prom/prometheus:v2.37.5 "/bin/prometheus --c…" 5 minutes ago Restarting (1) 27 seconds ago prometheus_observability
9a3964b561ec grafana/loki:2.7.0 "/usr/bin/loki --con…" 5 minutes ago Up 5 minutes loki_observability
b6a6ffa201f3 rticom/collector-service:<version> "/rti_connext_dds-7.…" 5 minutes ago Up 5 minutes collector_service_observability
26658f76cfdc grafana/grafana-enterprise:9.2.1-ubuntu "/run.sh" 5 minutes ago Up 5 minutes grafana_observability
If a container fails to start, refer to section Docker Container[s] Failed to Start for troubleshooting suggestions.
5.2.4. Configure Grafana
5.2.4.1. Initial Login
To access Observability Dashboards, open a new browser window and go to http://<hostname>:<grafanaPort> to access Grafana (3000 is the default grafanaPort). Log in using the credentials admin : admin, then change the password when prompted.
If you are using a secure configuration, the url to access Grafana will be https://<hostname>:<grafanaPort> and the Grafana credentials will be the values configured in the basicAuthUsername and basicAuthPassword fields in the JSON configuration.
Once you are logged in you will see the RTI Alert Home dashboard.
5.2.4.2. Configuration Options
You can configure the Grafana dashboard to meet your specific needs. For more information, refer to the Grafana article Use dashboards.
5.2.4.2.1. Create Accounts (Optional)
You can create additional users as needed. Refer to the Grafana article Manage Grafana Users for information about user roles and permissions.
5.2.4.2.2. Change the Default Time Range (Optional)
The default visualization time range can be modified. The default relative time range is one hour. You may want to update the range as follows:
Go to the Alert Home dashboard,
From the toolbar, select the time picker.
Select the desired time range from the dropdown list. The dashboard refreshes to display the selected time range.
From the toolbar, select Save dashboard.
In the Save dashboard dialog, select Save current time range as dashboard default and then click Save.
To confirm the new time range, navigate to another dashboard and then click the Home icon at the top left to go back to the Alert Home dashboard.
5.2.5. Stop Docker Containers
Once Observability Framework Docker containers are running, you can
stop them by running <installdir>/bin/rtiobservability -t. The -t
option terminates the running Docker containers for Observability Framework
by calling docker compose stop.
$ rtiobservability -t ... using Docker version 24.0.4, build 3713ee1. ... using Docker Compose version v2.19.1. Terminating the running Connext Observability Framework [+] Stopping 4/4 ✔ Container collector_service_observability Stopped 10.1s ✔ Container prometheus_observability Stopped 0.1s ✔ Container grafana_observability Stopped 0.2s ✔ Container loki_observability Stopped 2.1s
This command stops the existing Docker containers for Observability Framework, but leaves associated storage volumes and configuration for a future run.
5.2.6. Start Existing Docker Containers
To restart existing Docker containers used by Observability Framework,
run <installdir>/bin/rtiobservability -s. The -s option starts existing
Docker containers for Observability Framework by calling docker compose start.
$ rtiobservability -s ... using Docker version 24.0.4, build 3713ee1. ... using Docker Compose version v2.19.1. Starting the existing Connext Observability Framework [+] Running 4/4 ✔ Container prometheus_observability Started 0.1s ✔ Container collector_service_observability Started 0.1s ✔ Container grafana_observability Started 0.1s ✔ Container loki_observability Started 0.2s
This command starts any existing Docker containers created by Observability Framework.
5.2.7. Stop and Remove Docker Containers
To clean up, or stop and remove, all Docker containers and storage volumes used
by Observability Framework, run <installdir>/bin/rtiobservability -d. The
-d option stops and removes Docker containers for Observability Framework
by calling docker compose down, and subsequently removes storage volumes.
Warning
Running <installdir>/bin/rtiobservability -d removes all Docker
containers and storage volumes
used by Observability Framework. This command removes all changes to
your current Observability Framework Docker environment including:
metric data in Prometheus
log data in Loki
all Grafana user and dashboard configurations
$ rtiobservability -d
... using Docker version 24.0.4, build 3713ee1.
... using Docker Compose version v2.19.1.
********************************************************************************
*
* You have requested to clean up and remove the existing Connext Observability
* Framework. If you continue you will lose all changes to your current
* environment including:
* - metric data in Prometheus
* - log data in Loki
* - all Grafana user and dashboard configuration
*
********************************************************************************
Do you wish to continue cleaning and removing the existing Connext Observability Framework[y/N]?
When prompted to confirm that you want to remove all Docker containers and storage volumes for Observability Framework:
Select
N/n(or simply enter) to cancel the cleanup.
Do you wish to continue cleaning and removing the existing Connext Observability Framework[y/N]? n
Cleaning up and removing the existing Connext Observability Framework canceled.
Select
Y/yto proceed with the cleanup and remove all Docker containers and storage volumes used by Observability Framework.
Do you wish to continue cleaning and removing the existing Connext Observability Framework[y/N]? y
Cleaning up and removing the existing Connext Observability Framework
[+] Running 4/5
✔ Container prometheus_observability Removed 0.1s
✔ Container grafana_observability Removed 0.1s
✔ Container loki_observability Removed 1.5s
✔ Container collector_service_observability Removed 10.1s
observability_grafana_data
observability_prometheus_data
5.2.8. Removing the Docker Workspace for Observability Framework
There may be a time that you need to remove your existing Docker Workspace for
Observability Framework. This could be because you want to change the existing
configuration in some way. Things that you would want to change could include
hostname, port configurations, and enabling or disabling security. The rtiobservability
script will not overwrite an existing workspace. This prevents inadvertently
corrupting or deleting an existing configuration. The following steps should be
followed to remove an existing workspace to allow re-configuration.
You must first stop and remove any existing containers created with the current workspace configuration as detailed in section Stop and Remove Docker Containers.
Once the docker containers have been stopped and removed you must manually delete the
rti_workspace/<version>/user_config/observabilitydirectory.$ rm -rf <path_to_workspace>/<version>/user_config/observability