Communicate two Docker containers using RTI Connext DDS and shared memory

Note: Connext DDS does not currently support configurations with Docker + SHMEM only. In order for Connext to work properly, you must use Docker + (UDPv4 | SHMEM). This article applies to Connext DDS versions 5.2.0 through 5.3.1

Introduction

Docker enables containers to communicate with one another and with the host machine using Interprocess communication (IPC). As explained in the Docker’s documentation:

IPC (POSIX/SysV IPC) namespace provides separation of named shared memory segments, semaphores and message queues.
Shared memory segments are used to accelerate inter-process communication at memory speed, rather than through pipes or through the network stack. Shared memory is commonly used by databases and custom-built (typically C/OpenMPI, C++/using boost libraries) high performance applications for scientific computing and financial services industries. If these types of applications are broken into multiple containers, you might need to share the IPC mechanisms of the containers.

By default, the IPC namespace of a Docker container is isolated from the host machine and other containers. Fortunately, Docker provides a method to share the IPC of a Docker container with other containers or with the host machine. 

Note that this functionality may not be available in containers run by an orchestrator: some of them may be running in different machines and will therefore be unable to communicate through shared memory.

Enabling IPC Namespace Sharing in Docker

To enable Docker containers to use the host's IPC namespace to communicate with other containers and the host, add the --ipc=host option to the run command:

$ docker run --ipc=host -ti <Docker image> <Program>

Running DDS Applications on Docker Containers

Building an Example DDS Application

In this example we use a simple rtiddsgen-generated DDS application. You may use any existing IDL file, such as ShapeType.idl:

$ rtiddsgen -ppDisable -language C++ -create typefiles \
    -create examplefiles -create makefiles -platform x64Linux3gcc5.4.0 \
    ~/rti_workspace/5.3.0/user_config/rtiddsgen/shape_type/ShapeType.idl  

rtiddsgen generates a makefile that you may use to build your application as follows:

$ make -f makefile_ShapeType_x64Linux3gcc5.4.0 

Once you have built the Publisher and Subscriber applications, you will need to copy them to your Docker containers (or mount the directory where your applications are as a volume).

Configuring DDS Applications

The configuration of the Docker container affects the ability of RTI Connext DDS to communicate over shared memory. This is due to the impact of different parameters (e.g., the IP address of the instance and the Process ID of the DDS application) in the generation of the GUID that identifies every Entity in RTI Connext DDS. By default, the GUID is set as follows (for more information, see section “8.5.9.4 Controlling How the GUID is Set” of the RTI Connext DDS User’s Manual): 

  • 32 bits for the Host ID ( rtps_host_id). The Host ID is currently based upon the IP address (although this might change in future versions). This will allow you to determine the node having the GUID problem.

  • 16 low bits for the Application ID ( rtps_app_id).

    • If the originating node is a Windows system, the relevant Process ID will be the value of GetCurrentProcessId().

    • On a VxWorks system, it will be the Task ID of a specific task.

    • On a Linux system, it will be the Process ID that you can see as the output of the command ps -ef.

  • 8 bits for an Internal Counter. This counter allows an application to create multiple DomainParticipants in the same domain.

  • 8 bits containing a Constant Value (0x01).

One of the challenges of enabling communication across Docker containers is that, in order to communicate over shared memory, they must have the same Host ID and a different Application ID. This is due to the fact that, as we explained before:

  1. The generation of the GUID is based on the IP address of the running instance (i.e., if the containers have different IP addresses, they will have different Host IDs and there will not be communication).
  2. The Application ID is based on the Process ID (which could be the same in both containers).

To solve this situation, you need to manually:

  • Set the same Host ID for all your RTI Connext DDS applications.

  • Set a different RTPS App ID for each RTI Connext DDS application to uniquely identify them.

Setting the Host ID

The configuration of the Host ID depends on the version of RTI Connext DDS that you are running.

For RTI Connext DDS 5.2.5 through 5.3.1

Set the property dds.transport.shmem.builtin.host_id with the same value in each RTI Connext DDS application you want to enable to communicate.

<participant_qos>  
    <property>
       <value>
           <element>
               <name>dds.transport.shmem.builtin.host_id</name>
               <value>1</value>
           </element>
       </value>
   </property>
</participant_qos>
For RTI Connext DDS  5.2.0 to 5.2.4

Set the rtps_host_id QoS parameter with the same value in each RTI Connext DDS application. Note that this solution also applies to versions 5.2.5 and above, but the change will affect all the enabled transports.

<participant_qos>    
    <wire_protocol>
        <rtps_host_id>1</rtps_host_id>
    <wire_protocol>
</participant_qos>

 

Setting the Application ID

You can configure the Application ID by changing the rtps_app_id QoS parameter.

<participant_qos>
    <wire_protocol>     
        <rtps_app_id>1</rtps_app_id>   
    </wire_protocol>
</participant_qos>
Platform: