Communicate between two Docker containers using RTI Connext DDS and shared memory
Introduction
Docker enables containers to communicate with one another and with the host machine using interprocess communication (IPC). As explained in the Docker’s documentation:
IPC (POSIX/SysV IPC) namespace provides separation of named shared memory segments, semaphores and message queues.
Shared memory segments are used to accelerate inter-process communication at memory speed, rather than through pipes or through the network stack. Shared memory is commonly used by databases and custom-built (typically C/OpenMPI, C++/using boost libraries) high performance applications for scientific computing and financial services industries. If these types of applications are broken into multiple containers, you might need to share the IPC mechanisms of the containers.
By default, the IPC namespace of a Docker container is isolated from the host machine and other containers. Fortunately, Docker provides a method to share the IPC of a Docker container with other containers or with the host machine.
Note that this functionality may not be available in containers run by an orchestrator: some of them may be running in different machines and will therefore be unable to communicate through shared memory.
Enabling IPC Namespace Sharing in Docker
To enable Docker containers to communicate through Shared Memory they must share the same IPC namespace. For that you have two options:
- Use the host's IPC namespace to communicate with other containers and the host. This can be done by adding the --ipc=host option to the run command:
docker run --ipc=host -ti <Docker image> <Program>
- Share the same IPC namespace with the container you want to communicate with. This can be done by adding the --ipc="container:<name>" option to the run command:
docker run --ipc="container:<name>" -ti <Docker image> <Program>
Running DDS Applications on Docker Containers
Building an Example DDS Application
In this example we use a simple rtiddsgen-generated DDS application. You may use any existing IDL file, such as ShapeType.idl:
$ rtiddsgen -ppDisable -language C++ -create typefiles \
-create examplefiles -create makefiles -platform x64Linux3gcc5.4.0 \
~/rti_workspace/6.1.1/user_config/rtiddsgen/shape_type/ShapeType.idl
rtiddsgen generates a makefile that you may use to build your application as follows:
$ make -f makefile_ShapeType_x64Linux3gcc5.4.0
Once you have built the Publisher and Subscriber applications, you will need to copy them to your Docker containers (or mount the directory where your applications are as a volume).
Configuration
The configuration of the Docker container might affect the ability of RTI Connext to communicate over shared memory. This is due to the impact of different parameters (e.g., the Process ID of the DDS application and the IP address of the instance) in the generation of the GUID that identifies every Entity in RTI Connext. For more information, see Controlling How the GUID is Set (rtps_auto_id_kind), in the RTI Connext DDS User’s Manual.
Configuring Connext at/after Release 6.0.0
Starting from Connext 6.0.0 the Shared Memory compatibility detection is based on checking if it is possible to attach to a shared memory segment compatible with Connext (see Improved shared memory transport compatibility detection, in What’s New for 6.0.0). Connext will check if there is a live process that has created the shared memory segment on the same port.
a) [RECOMMENDED] Use the same PID namespace for your containers:
You can configure your Docker containers to share the same process namespace by using the --pid=container:<name> option to the run command:
$ docker run --pid="container:<name>" (...) <Docker image> <Program>This way Connext will detect other application(s) using Shared Memory from a different container without having to modify any QoS configuration.
b) Set the participant_id QoS
To avoid participant_id collisions you can manually set a different value for each RTI Connext application. For example:
<qos_profile name="Application_A"> <participant_qos> <wire_protocol> <participant_id>1</participant_id> </wire_protocol> </participant_qos> </qos_profile> <qos_profile name="Application_B"> <participant_qos> <wire_protocol> <participant_id>2</participant_id> </wire_protocol> </participant_qos> </qos_profile>
This will make DomainParticipants have different Participant IDs regardless of the Docker’s configuration. But you must ensure that each Participant has a unique participant_id.
Keep in mind that if you set a Participant ID bigger than 4, you may also need to adjust your max_participant_index in your peer list (max_participant_index is 4 by default) to ensure discovery. You can find more information on how to set the max_participant_index in this article from our Knowledge Base.
Configuring Connext at/before Release 5.3.1
- The generation of the GUID is based on the IP address of the running instance (i.e., if the containers have different IP addresses, they will have different Host IDs and there will not be communication).
- The Application ID is based on the Process ID (which could be the same in both containers).
- Set the same Host ID for all your RTI Connext applications.
- If not sharing the same IPC namespace, set a different RTPS App ID for each RTI Connext application to uniquely identify them.
- Set different Participant IDs for all your DomainParticipants.
1. Setting the Host ID
The configuration of the Host ID depends on the version of RTI Connext that you are running.
For RTI Connext DDS 5.2.5 through 5.3.1
Set the property dds.transport.shmem.builtin.host_id with the same value in each RTI Connext application you want to enable to communicate.
<participant_qos> <property> <value> <element> <name>dds.transport.shmem.builtin.host_id</name> <value>1</value> </element> </value> </property> </participant_qos>
For RTI Connext DDS 5.2.0 to 5.2.4
Set the rtps_host_id QoS parameter with the same value in each RTI Connext application. Note that this solution also applies to versions 5.2.5 and above, but the change will affect all the enabled transports.
<participant_qos> <wire_protocol> <rtps_host_id>1</rtps_host_id> </wire_protocol> </participant_qos>
2. Setting the Application ID
Note: This is not needed if both containers are sharing the same PID namespace (--pid="container:<name>")
You can configure the Application ID by changing the rtps_app_id QoS parameter.
<participant_qos> <wire_protocol> <rtps_app_id>1</rtps_app_id> </wire_protocol> </participant_qos>
If you go with this approach you must ensure the rtps_app_id is different between applications.
3. Setting the Participant ID
By default, Connext will choose the Participant ID based on the unicast ports available on the transports enabled (see “9.5.9.1 Choosing Participant IDs in the RTI Connext DDS Core Libraries User's Manual” for more information).
If the Docker containers are running on different networks, it can happen that the same port number is available for both containers. Hence, Connext will assign the same Participant ID for both DomainParticipants. This will make the DomainPartipants believe they are on different hosts, and shared memory communication won’t take place.
If you are running into this scenario, set the participant_id QoS parameter with a different value for each RTI Connext application. See “Set the participant_id QoS” in the section “Configuring Connext at/after Release 6.0.0” in this article.