3. Using Protocol Buffers Extension

Protocol Buffers Extension will become available when the Connext installation is loaded in the shell environment, for example by using one of the provided scripts:

source <installdir>/resource/scripts/rtisetenv_<architecture>.bash

The script will customize the PATH, and LD_LIBRARY_PATH variables to include the Connext installation’s subdirectories.

3.1. Hello DDS Protocol Buffers

The following examples show how Protocol Buffers Extension enables the easy integration of Protocol Buffers components with the Connext Databus.

Note

You can find the complete source code for this example in directory <installdir>/resource/template/rti_workspace/examples/connext_dds/c++11/protobuf-sdk.

3.1.1. Defining Protocol Buffers Types

Protocol Buffers Extension allows applications to use their existing Protocol Buffers messages with Connext.

For this example, we will reference the AddressBook example included in the Protocol Buffers documentation, which describes a simple data model composed of a single .proto file:

// addressbook.proto

syntax = "proto3";
package tutorial;

import "google/protobuf/timestamp.proto";

message Person {
  string name = 1;
  int32 id = 2;  // Unique ID number for this person.
  string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    string number = 1;
    PhoneType type = 2;
  }

  repeated PhoneNumber phones = 4;

  google.protobuf.Timestamp last_updated = 5;
}

message AddressBook {
  repeated Person people = 1;
}

Most Protocol Buffers data models can be used without modifications, thanks to the detailed mapping between Protocol Buffers and IDL4 defined by Protocol Buffers Extension.

3.1.2. Exchanging Protocol Buffers Data

The Protocol Buffers types must be converted into types for the programming language used by application code, using protoc, and rtiddsgen. For example, to generate C++ code:

# First use the Protocol Buffers compiler with the additional plugins
protoc --idl4_out=build \
        --cpp_out=build \
        --connext-cpp_out=build \
        -I . \
        addressbook.proto \
        google/protobuf/timestamp.proto

# Then use the Connext compiler to generate additional code from the generated IDL
rtiddsgen -language C++11 \
          -standard PROTOBUF_CPP \
          -I build \
          build/addressbook.idl \
          build/google/protobuf/timestamp.idl

The generated “type support” code can be used by applications to exchange Protocol Buffers data directly over DDS topics:

Publish data to a DDS Topic using the Protocol Buffers types.

// addressbook_publisher.cxx

// Include the standard DDS C++ API.
#include <dds/dds.hpp>

// Include the generated header files for the Protocol Buffers types.
// addressbook.hpp is generated from addressbook.idl by rtiddsgen.
// It includes addressbook.pb.h, which is generated by protoc
// from addressbook.proto.
#include "addressbook.hpp"

// Instantiate DDS entities to publish an AddressBook topic on domain 0.
dds::domain::DomainParticipant participant(0);
dds::topic::Topic<tutorial::AddressBook> topic(participant, "Example tutorial_AddressBook");
dds::pub::Publisher publisher(participant);
dds::pub::DataWriter<tutorial::AddressBook> writer(publisher, topic);

// Create an instance of the AddressBook message and populate it.
tutorial::AddressBook data;

auto person = data.add_people();
person->set_name("John Doe");
person->set_id(1);
person->set_email("johndoe@example.org");

auto phone = person->add_phones();
phone->set_number("867-5309");
phone->set_type(tutorial::Person_PhoneType_HOME);

// Write the AddressBook message using the typed DataWriter.
writer.write(data);

Subscribe to data on a DDS Topic using the Protocol Buffers types.

// addressbook_subscriber.cxx

// Include the standard DDS C++ API.
#include <dds/dds.hpp>

// Include the generated header files for the Protocol Buffers types.
#include "addressbook.hpp"

// Instantiate DDS entities to subscribe to an AddressBook topic on domain 0.
dds::domain::DomainParticipant participant(0);
dds::topic::Topic<tutorial::AddressBook> topic(participant, "Example tutorial_AddressBook");
dds::sub::Subscriber subscriber(participant);
dds::sub::DataReader<tutorial::AddressBook> reader(subscriber, topic);

// Take samples from the typed DataReader.
dds::sub::LoanedSamples<tutorial::AddressBook> samples = reader.take();

for (const auto &sample : samples) {
    if (sample.info().valid()) {
        // Samples are instances of the Protocol Buffers AddressBook type.
        tutorial::AddressBook &data = sample.data();
        std::cout << data.DebugString() << std::endl;
    }
}

3.1.3. Communicating with other DDS applications

All .proto files used with Protocol Buffers Extension will be automatically converted into .idl files, that describe an equivalent data model for the type system defined by the Extensible and Dynamic Topic Types for DDS specification (XTypes).

For example, the AddressBook data model will be converted to the following equivalent IDL4/XTypes data model:

// addressbook.idl

// -----------------------------------------------------------------------------
// WARNING ---------------------------------------------------------------------
// This file was automatically generated by RTI's IDL4 plugin for protoc,
// from Protobuf source file: addressbook.proto
// Do not edit this file manually. ALL CHANGES WILL BE LOST!
// -----------------------------------------------------------------------------

#ifndef tutorial_addressbook_proto_IDL4_
#define tutorial_addressbook_proto_IDL4_
#include "google/protobuf/timestamp.idl"

module tutorial {
  @containing_type("Person")
  enum Person_PhoneType {
    @value(0) @default_literal Person_PhoneType_MOBILE,
    @value(1) Person_PhoneType_HOME,
    @value(2) Person_PhoneType_WORK
  }; // enum Person_PhoneType

  struct Person_PhoneNumber;

  struct Person;

  struct AddressBook;

  @nested
  @containing_type("Person")
  @mutable
  struct Person_PhoneNumber {
    @id(1) @field_presence(implicit) string number;
    @id(2) @field_presence(implicit) ::tutorial::Person_PhoneType type;
  }; // struct Person_PhoneNumber

  @mutable
  struct Person {
    @id(1) @field_presence(implicit) string name;
    @id(2) @field_presence(implicit) int32 id;
    @id(3) @field_presence(implicit) string email;
    @id(4) sequence<::tutorial::Person_PhoneNumber> phones;
    @id(5) @optional ::google::protobuf::Timestamp last_updated;
  }; // struct Person

  @mutable
  struct AddressBook {
    @id(1) sequence<::tutorial::Person> people;
  }; // struct AddressBook

}; // tutorial
#endif // tutorial_addressbook_proto_IDL4_

The generated IDL4 types can be used in other applications, using any of the Programming Language supported by Connext. For example, to use the “modern C++” API with the latest language binding defined by the IDL4 specification:

mkdir build-dds

rtiddsgen -language C++11 \
          -standard IDL4_CPP \
          -I build \
          -d build-dds \
          build/addressbook.idl

mkdir -p build-dds/google/protobuf

rtiddsgen -language C++11 \
          -standard IDL4_CPP \
          -I build \
          -d build-dds/google/protobuf \
          build/google/protobuf/timestamp.idl

3.1.4. Building & Running the Example

Load Connext and Protocol Buffers Extension in the shell environment:

source <installdir>/resource/scripts/rtisetenv_<architecture>.bash

Build the example applications:

// Create a build directory and enter it
mkdir build
cd build

// Generate the CMake build files
cmake <installdir>/resource/template/rti_workspace/examples/connext_dds/c++11/protobuf-sdk

// Build the example applications
cmake --build .

Publish data samples to a DDS topic using the Protocol Buffers types with one of the supported language bindings:
1. Start a publisher using the Protocol Buffers C++ language binding:
```
./addressbook_publisher
```
2. Start a publisher using the Connext C++ language binding:
```
./addressbook_publisher_dds
```
Subscribe to the DDS topic using the Protocol Buffers types:
1. Start a subscriber using the Protocol Buffers C++ language binding:
```
./addressbook_subscriber
```
2. Start a subscriber using the Connext C++ language binding:
```
./addressbook_subscriber_dds
```
3. Start rtiddsspy to subscribe to all data in the DDS domain:
```
rtiddsspy -printSample
```

3.2. Generating Code from Protocol Buffers Messages

Protocol Buffers Extension relies on a two-phased generative process to produce source code from .proto files, which will allow Connext applications to use the Protocol Buffers types with DDS topics.

The protoc compiler must be invoked first to:

Convert each .proto file into an equivalent .idl file.
Generate source code for each Protocol Buffers type.

All input .proto files must be processed by three protoc plugins:

The idl4 converter plugin, provided by Protocol Buffers Extension, which generates an .idl file for each input .proto.
The built-in cpp plugin, which is part of the Protocol Buffers distribution, and it is responsible for generating the C++ classes associated with each Protocol Buffers type.
The connext-cpp plugin, provided by Protocol Buffers Extension, which decorates the C++ code produced by cpp, so that it may be efficiently integrated with Connext.

The two protoc plugins provided by Protocol Buffers Extension are located under <installdir>/bin. The directory should be added to environment variable PATH to make them usable by protoc.

Alternatively, it is possible to specify the explicit path of each plugin through the protoc command line, without modifying PATH:

protoc --plugin=<installdir>/bin/protoc-gen-idl4 \
       --plugin=<installdir>/bin/protoc-gen-connext-cpp \
       ...

The second phase relies on rtiddsgen, Connext’s code generator, to process the .idl files produced by protoc, in order to generate the remaining source code required to use the Protocol Buffers types as DDS topics in Connext applications.

Table 3.1 summarizes the steps, with an example of the associate command-line tool invocation, and a list of the files it would produce.

Table 3.1 Code Generation Steps
Code Generation Step	Input	Output
protoc --idl4_out=<outputdir> \ --cpp_out=<outputdir> \ --connext-cpp_out=<outputdir> \ <input>	`message.proto`	`message.idl` `message.pb.h` `message.pb.cc`
rtiddsgen -language C++11 \ -standard PROTOBUF_CPP \ <input>	`message.idl`	`message.hpp` `message.cxx` `messagePlugin.hpp` `messagePlugin.cxx`

Note

The example protoc invocation runs all plugins at once, but this is not a strict requirement. Each protoc plugin can be invoked independently, with the only caveat that connext-cpp must be run with (or after) the built-in cpp plugin.

Nevertheless, since other options (e.g. include paths) should also be kept consistent across different invocations of protoc, it is recommended to perform generation with all plugins in a single command line.

If an input .proto file imports omg/dds/descriptor.proto, the directory containing the file must be specified using the -I option in the protoc command line, e.g.:

protoc --idl4_out=<outputdir> \
       --cpp_out=<outputdir> \
       --connext-cpp_out=<outputdir> \
       -I <installdir>/resource/proto \
       <input>

3.2.1. Code Generation with Multiple Files

Most projects will use multiple .proto files, which include import statements to reference the types they depend on.

The corresponding .idl files generated by the idl4 plugin will contain #include statements that expect the .idl files to have been generated in a directory structure similar to the one used by the input files.

For example, consider a project containing the following file structure:

.src/
├── a.proto
├── foo
│   └── b.proto
└── bar
    └── c.proto

We could imagine a.proto referencing the other file via import statements, e.g.:

import "foo/b.proto";
import "bar/c.proto";

After processing a.proto with the idl4 plugin, the generated a.idl would then have corresponding #include statements:

#include "foo/b.idl"
#include "bar/c.idl"

To ensure that the generated #include statements are valid, all of the imported files must have been converted to .idl files before a.idl can be processed by rtiddsgen. The .idl files must be placed in a similar directory structure:

.build/       # generated by:
├── a.idl     # └── protoc --idl4_out=...
├── foo
│   └── b.idl
└── bar
    └── c.idl

A similar ordering requirement may also exist between b.proto and c.proto, forcing, for example, b.idl to be generated before c.idl can be processed. The two files may also be independent, and thus allow for faster processing in parallel.

In general, the complexity of managing these dependencies precisely increases rapidly with the number of input files, typically requiring additional “bookkeeping”, and making it particularly difficult to implement a “fully parallelized” solution.

It is recommended to forego a bit of parallelization efficiency in favor of simpler build rules by breaking up the code generation in two separate steps:

First, process all .proto files with protoc, e.g.:

protoc --idl4_out=build \
       --cpp_out=build \
       --connext-cpp_out=build \
       -I src \
       src/a.proto \
       src/foo/b.proto \
       src/bar/c.proto

Then, process all .idl files with rtiddsgen, e.g.:

rtiddsgen -language C++11 \
          -standard PROTOBUF_CPP \
          -I build \
          build/a.idl \
          build/foo/b.idl \
          build/bar/c.idl

Warning

While both protoc and rtiddsgen can process multiple input files from a single command line, it is recommended to use a single input file for each invocation of these tools when automating code generation.

This will ensure that the length of the command line invocation never exceeds the maximum allowed by the operating system, which may otherwise happen as the number of .proto files used by a project grows.

At the end of the code generation process, the following C++ files will be available for compilation in the output directory:

.build/              # generated by:
├── a.pb.cc          # ├── protoc --cpp_out=... --connext-cpp_out=...
├── a.pb.h           # ├── protoc --cpp_out=... --connext-cpp_out=...
├── a.cxx            # ├── rtiddsgen -language C++11 -standard PROTOBUF_CPP
├── a.hpp            # ├── rtiddsgen -language C++11 -standard PROTOBUF_CPP
├── aPlugin.cxx      # ├── rtiddsgen -language C++11 -standard PROTOBUF_CPP
├── aPlugin.hpp      # └── rtiddsgen -language C++11 -standard PROTOBUF_CPP
├── foo
│   ├── b.pb.cc
│   ├── b.pb.h
│   ├── b.cxx
│   ├── b.hpp
│   ├── bPlugin.cxx
│   └── bPlugin.hpp
└── bar
    ├── c.pb.cc
    ├── c.pb.h
    ├── c.cxx
    ├── c.hpp
    ├── cPlugin.cxx
    └── cPlugin.hpp