RTI Connext Observability Framework
7.3.1
Copyrights and Notices
1. What is Connext Observability Framework?
1.1. Telemetry Data
1.2. Distribution of Telemetry Data
1.3. Flexible Storage
1.4. Visualization of Telemetry Data
1.5. Control and Selection of Telemetry Data
1.6. Security
2. Components
2.1. Monitoring Library 2.0
2.2. Observability Collector Service
2.2.1. Storage Components
2.3. Observability Dashboards
3. Observability Framework Deployments
3.1. Current Release
3.1.1. Docker Compose (Prepackaged)
3.1.1.1. Collection, Storage, and Visualization Components
3.1.2. Docker (Separate Deployment)
3.1.2.1. Collector Service
3.2. Future releases
3.2.1. Collector Service
3.2.1.1. Executable
3.2.1.2. Collector Service Deployments
4. Security
4.1. Secure Communication between Connext Applications and Collector Service
4.1.1. Secure Communication between Connext Applications and Collector Service (Pre-Packaged Deployment)
4.1.2. Secure Communication between Connext Applications and Collector Service (Separate Deployment)
4.2. Secure Communication with Collector Service HTTP Servers
4.2.1. Secure Collector Service HTTP Servers (Pre-Packaged Deployment)
4.2.2. Secure Collector Service HTTP Servers (Separate Deployment)
4.3. Secure Communication with Third-Party Component HTTP Servers
4.3.1. Secure Third-Party Component HTTP Servers (Pre-Packaged Deployment)
4.3.2. Secure Third-Party Component HTTP Servers (Separate Deployment)
4.4. Generating the Observability Framework Security Artifacts
4.4.1. Generating DDS Security Artifacts
4.4.2. Generating HTTPS Security Artifacts
4.4.2.1. Preliminary Steps
4.4.2.2. Generating a New Root CA
4.4.2.3. Generating Server Certificates
4.4.2.4. BASIC-Auth Password File
5. Installing and Running Observability Framework
5.1. Configuring the Docker Workspace for Observability Framework
5.1.1. Configure the JSON File
5.1.2. Run the Observability script to create the Observability workspace
5.2. Initialize and Run Docker Containers
5.3. Verify Docker Containers are Running
5.4. Configure Grafana
5.4.1. Initial Login
5.4.2. Configuration Options
5.4.2.1. Create Accounts (Optional)
5.4.2.2. Change the Default Time Range (Optional)
5.5. Stop Docker Containers
5.6. Start Existing Docker Containers
5.7. Stop and Remove Docker Containers
5.8. Removing the Docker Workspace for Observability Framework
6. Getting Started Guide
6.1. About the Observability Example
6.1.1. Applications
6.1.2. Data Model
6.1.3. DDS Entity Mapping
6.1.4. Command-Line Parameters
6.1.4.1. Publishing Application
6.1.4.2. Subscribing Application
6.2. Before Running the Example
6.2.1. Set Up Environment Variables
6.2.2. Compile the Example
6.2.2.1. Non-Windows Systems
6.2.2.2. Windows Systems
6.2.3. Install Observability Framework
6.2.3.1. Configure Observability Framework for the Appropriate Operation Mode
6.2.4. Start the Collection, Storage, and Visualization Docker Containers
6.3. Running the Example
6.3.1. Start the Applications
6.3.2. Changing the Time Range in Dashboards
6.3.3. Simulate Sensor Failure
6.3.4. Simulate Slow Sensor Data Consumption
6.3.5. Simulate Time Synchronization Failures
6.3.6. Change the Application Logging Verbosity
6.3.7. Change the Metric Configuration
6.3.7.1. Resources used in this example
6.3.7.2. Changing metrics collected for a single DataWriter
6.3.7.3. Changing metrics collected for all DataWriters of an application
6.3.8. Close the Applications
7. Telemetry Data
7.1. Introduction
7.2. Resources
7.2.1. Resource Pattern Definitions
7.3. Metrics
7.3.1. Metric Pattern Definitions
7.3.2. Application Metrics
7.3.3. Participant Metrics
7.3.4. Topic Metrics
7.3.5. DataWriter Metrics
7.3.6. DataReader Metrics
7.3.7. Derived Metrics Generated by Prometheus Recording Rules
7.3.7.1. DDS Entity Proxy Metrics
7.3.7.2. Raw Error Metrics
7.3.7.3. Aggregated Error Metrics
7.3.7.4. Enable a Raw Error Metric
7.3.7.5. Custom Error Metrics
7.4. Logs
7.4.1. Syslog Levels and Facilities
7.4.2. Activity Context
7.4.3. Log Labels
7.4.4. Collection and Forwarding Verbosity
7.4.4.1. Changing Verbosity Levels Locally
7.4.4.2. Changing Verbosity Levels Remotely
8. Monitoring Library 2.0
8.1. Enabling Monitoring Library 2.0
8.2. Setting the Initial Metrics and Log Configuration
8.3. Setting the Application Name
8.4. Changing the Default Observability Domain ID
8.5. Configuring QoS for Monitoring Library 2.0 Entities
8.6. Setting Collector Service Initial Peers
9. Collector Service REST API Reference
9.1. Definitions
9.2. Root endpoint (base URL)
9.3. API Overview
9.4. API Reference
10. Observability Dashboards
10.1. System Status Dashboards
10.1.1. System Status Dashboard Common Elements
10.1.2. Alert Home Dashboard
10.1.3. Alert Category Dashboards
10.2. Entity List Dashboards
10.3. Entity Status List Dashboards
10.4. Entity Status Dashboards
10.5. Log Dashboards
10.5.1. Log Dashboard
10.5.2. Entity Log Dashboards
10.6. Control Dashboards
10.6.1. Log Control Dashboard
10.6.2. Metric Control Dashboards
10.6.2.1. Single Entity Metric Control Dashboards
10.6.2.2. Multiple Entity Metric Control Dashboards
11. Troubleshooting Observability Framework
11.1. Docker Container[s] Failed to Start
11.1.1. Check for Port Conflicts
11.1.2. Check that You Have the Correct File Permissions
11.2. No Data in Dashboards
11.2.1. Check that Collector Service has Discovered Your Applications
11.2.2. Check that Prometheus can Access Collector Service
11.2.3. Check that Grafana can Access Prometheus
11.2.4. Check that Grafana can Access Loki
12. Glossary
13. Release Notes
13.1. Supported Platforms
13.2. Compatibility
13.2.1. Monitoring Library 2.0 compatibility with Collector Service
13.2.2. Collector Service compatibility with Monitoring Library 2.0
13.3. Supported Docker Compose Environments
13.4. Supported Docker Environments for Collector Service
13.5. What’s New in 7.3.1
13.5.1. Disallow unrealistic values of polling_period, to avoid errors collecting data for Observability Framework
13.5.2. Third-Party Software Changes
13.6. What’s Fixed in 7.3.1
13.6.1. Debuggability
13.6.1.1.
[Major]
Query condition errors on discovering more than four applications simultaneously
13.6.1.2.
[Major]
Potential timeout on commands addressed to more than one application
13.6.2. Crashes
13.6.2.1.
[Critical]
Potential crash when removing unready resources
13.6.2.2.
[Critical]
Collector Service crashed when processing malformed HTTP commands
13.6.3. Hangs
13.6.3.1.
[Critical]
Collector Service hung when several applications discovered simultaneously
13.6.4. Memory Leaks/Growth
13.6.4.1.
[Critical]
Possible memory leak due to error in processing child resources
13.6.4.2.
[Critical]
Unbounded memory growth when using remote administration commands in Observability Framework
13.6.5. Vulnerabilities
13.6.5.1.
[Critical]
Potential stack overflow in Collector Service when using XML configuration file referencing environment variables
13.6.5.2.
[Critical]
Potential crash in Collector Service when processing a malicious Set-Cookie header in an HTTP response
13.6.6. Other
13.6.6.1.
[Minor]
Collector Service did not export all DDS Entities when periodic metrics not enabled
13.6.6.2.
[Minor]
Prometheus listener port was not correctly configured
13.7. Previous Releases
13.7.1. What’s New in 7.3.0
13.7.1.1. Enhanced control of entities distributed across various Collector Service instances
13.7.1.2. New REST API in Collector Service to control telemetry data collection and distribution
13.7.1.3. Support for more flexible Observability Framework deployments
13.7.1.4. Control which metrics are collected
13.7.1.5. New Syslog facilities provide expanded log management
13.7.1.6. New logging category and plugin class labels enable more precise third-party backend queries
13.7.1.7. Updated dashboards support enhanced logging and dynamic metric control
13.7.1.8. Name change for some observability metrics
13.7.1.9. Third-party software upgrades
13.7.2. What’s Fixed in 7.3.0
13.7.2.1. Crashes
13.7.2.2. Vulnerabilities
13.7.3. What’s New in 7.2.0
13.7.3.1. Observability Collector Service compatible with Monitoring Library 2.0
13.7.3.2. Support for most observability backends with OpenTelemetry integration
13.7.3.3. Support for Observability Collector Service security
13.7.3.4. Name change from “RTI Observability Library” to “RTI Monitoring Library 2.0”
13.7.3.5. Name change for some Observability metrics
13.7.3.6. Secured communications between Monitoring Library 2.0 and Observability Collector Service
13.7.3.7. Ability to set initial forwarding verbosity in MONITORING QoS policy
13.7.3.8. Ability to set collector initial peers in MONITORING QoS policy
13.7.3.9. Third-Party software changes
13.7.4. What’s Fixed in 7.2.0
13.7.4.1. Collector Service might have crashed on startup
13.7.4.2. Controllability issues on applications with same name
13.7.4.3. Unhandled exceptions may have caused segmentation fault
13.7.4.4. Race condition when processing remote commands led to failures and memory leaks when shutting down Collector Service
13.7.4.5. Collector Service could discard samples when monitoring large DDS applications
13.7.5. What’s New in 7.1.0
13.7.5.1. Third-Party Software
RTI Connext Observability Framework
7.
Telemetry Data
View page source
7.
Telemetry Data
7.1. Introduction
7.2. Resources
7.2.1. Resource Pattern Definitions
7.3. Metrics
7.3.1. Metric Pattern Definitions
7.3.2. Application Metrics
7.3.3. Participant Metrics
7.3.4. Topic Metrics
7.3.5. DataWriter Metrics
7.3.6. DataReader Metrics
7.3.7. Derived Metrics Generated by Prometheus Recording Rules
7.3.7.1. DDS Entity Proxy Metrics
7.3.7.2. Raw Error Metrics
7.3.7.3. Aggregated Error Metrics
7.3.7.4. Enable a Raw Error Metric
7.3.7.5. Custom Error Metrics
7.4. Logs
7.4.1. Syslog Levels and Facilities
7.4.2. Activity Context
7.4.3. Log Labels
7.4.4. Collection and Forwarding Verbosity
7.4.4.1. Changing Verbosity Levels Locally
7.4.4.2. Changing Verbosity Levels Remotely