Communicate between two Docker containers using RTI Connext DDS and shared memory

Submitted by israel on Mon, 10/02/2017 - 09:52

Introduction

Containers can communicate using Connext over Shared Memory with one another and with the host machine using interprocess communication (IPC). This is available in Docker as explained in the Docker’s documentation:

IPC (POSIX/SysV IPC) namespace provides separation of named shared memory segments, semaphores and message queues.
Shared memory segments are used to accelerate inter-process communication at memory speed, rather than through pipes or through the network stack. Shared memory is commonly used by databases and custom-built (typically C/OpenMPI, C++/using boost libraries) high performance applications for scientific computing and financial services industries. If these types of applications are broken into multiple containers, you might need to share the IPC mechanisms of the containers.

By default, the IPC and PID namespaces of a Docker container is isolated from the host machine and other containers. Fortunately, Docker provides a method to share the IPC/PID of a Docker container with other containers or with the host machine. 

Note that this functionality may not be available in containers run by an orchestrator: some of them may be running in different machines and will therefore be unable to communicate through shared memory.

Enabling IPC Namespace Sharing in Docker

To enable Docker containers to communicate through Shared Memory they must share the same IPC namespace. For that you have two options:

  • Use the host's IPC namespace to communicate with other containers and the host. This can be done by adding the --ipc=host option to the run command:
docker run --ipc=host -ti <Docker image> <Program>
  • Share the same IPC namespace with the container you want to communicate with. This can be done by adding the --ipc="container:<name>" option to the run command:
docker run --ipc="container:<name>" -ti <Docker image> <Program>

Enabling PID Namespace Sharing in Docker (Recommended)

 

Connext relies on visibility into the actual Process ID (PID) of the creating application to verify if the process is still alive and to safely manage, initialize, or reuse shared memory resources. This among other parameters (e.g., the Process ID of the DDS application and the IP address of the instance) also affect in the generation of the GUID that identifies every Entity in RTI Connext. For more information, see Controlling How the GUID is Set (rtps_auto_id_kind), in the RTI Connext DDS User’s Manual. 

Because of this it's necessary to configure your Docker containers to share the same process namespace to perform Shared Memory communication out of the box. This can be achieved in docker e.g., by using the --pid=container:<name> option to the run command:

$ docker run --pid="container:<name>" --ipc="container:<name>" (...) <Docker image> <Program>
Or in Kubernetes by setting shareProcessNamespace: true (see kubernetes doc)
 
This way Connext will detect other application(s) using Shared Memory from a different container without having to modify any QoS configuration.

If PID namespace cannot be shared between containers

If sharing the PID namespace is not an option for your (e.g., because of process isolation constraints) there is an option to manually set a unique participant_id for each DomainParticipant to successfully establish communication. 

This is to prevent participant_id collisions due to not having visibility on the PID namespace. You can manually set a different value for each RTI Connext application. For example:

<qos_profile name="Application_A">
    <participant_qos>
        <wire_protocol>
            <participant_id>1</participant_id>
        </wire_protocol>
    </participant_qos>
</qos_profile>

<qos_profile name="Application_B">  
    <participant_qos>
        <wire_protocol>
            <participant_id>2</participant_id>
        </wire_protocol>
    </participant_qos>
</qos_profile>

This will make DomainParticipants have different Participant IDs regardless of the Docker’s configuration. But you must ensure that each Participant has a unique participant_id.

Keep in mind that if you set a Participant ID bigger than 4, you may also need to adjust your max_participant_index in your peer list (max_participant_index is 4 by default) to ensure discovery. You can find more information on how to set the max_participant_index in this article from our Knowledge Base.

Additional configuration for Connext at/before Release 5.3.1

In releases prior to Connext 6, the Shared Memory compatibility detection was based on both DomainParticipants having the same shared memory locator. Hence, to communicate over shared memory, the DomainParticipants must have the same Host ID and a different Application ID. This is because, as explained before:
  1. The generation of the GUID is based on the IP address of the running instance (i.e., if the containers have different IP addresses, they will have different Host IDs and there will not be communication).
  2. The Application ID is based on the Process ID (which could be the same in both containers).
To solve this situation, you need to manually:
  • Set the same Host ID for all your RTI Connext applications.
  • If not sharing the same IPC namespace, set a different RTPS App ID for each RTI Connext application to uniquely identify them.
  • Set different Participant IDs for all your DomainParticipants (see section "If PID namespace cannot be shared between containers").

Setting the Host ID

The configuration of the Host ID depends on the version of RTI Connext that you are running.

For RTI Connext DDS 5.2.5 through 5.3.1

Set the property dds.transport.shmem.builtin.host_id with the same value in each RTI Connext application you want to enable to communicate.

<participant_qos>  
    <property>
       <value>
           <element>
               <name>dds.transport.shmem.builtin.host_id</name>
               <value>1</value>
           </element>
       </value>
   </property>
</participant_qos>
For RTI Connext DDS  5.2.0 to 5.2.4

Set the rtps_host_id QoS parameter with the same value in each RTI Connext application. Note that this solution also applies to versions 5.2.5 and above, but the change will affect all the enabled transports.

<participant_qos>    
    <wire_protocol>
        <rtps_host_id>1</rtps_host_id>
    </wire_protocol>
</participant_qos>

Setting the Application ID

Note: This is not needed if both containers are sharing the same PID namespace (--pid="container:<name>")

You can configure the Application ID by changing the rtps_app_id QoS parameter.

<participant_qos>
    <wire_protocol>     
        <rtps_app_id>1</rtps_app_id>   
    </wire_protocol>
</participant_qos>

If you go with this approach you must ensure the rtps_app_id is different between applications.

Testing a Contenarized DDS Application over Shared Memory

To test if communication over shared memory works on your environment you can use our official RTI DDS Ping docker image available in our Docker Hub repository.

Run the rtiddsping publisher container only enabling Shared Memory transport f("-transport 2"):

$ docker run -it --rm --ipc=host --pid=host --network host --name=dds_ping_publisher rticom/dds-ping:latest -domainId 0 -transport 2

And similarly the rtiddsping subscriber container

$ docker run -it --rm --ipc=host --pid=host --network host --name=dds_ping_subscriber rticom/dds-ping:latest -sub -domainId 0 -transport 2

Your subscriber should start receiving samples.

Product:
Platform:
Keywords: