.. _section-using: Using |PROTOBUF_EXT| ******************** |PROTOBUF_EXT| will become available when the |CONNEXT| installation is loaded in the shell environment, for example by using one of the provided scripts: .. code-block:: shell source /resource/scripts/rtisetenv_.bash The script will customize the ``PATH``, and ``LD_LIBRARY_PATH`` variables to include the |CONNEXT| installation's subdirectories. Hello DDS |PROTOBUF| ==================== The following examples show how |PROTOBUF_EXT| enables the easy integration of |PROTOBUF| components with the |CONNEXT| Databus. .. note:: You can find the complete source code for this example in directory ``/resource/template/rti_workspace/examples/connext_dds/c++11/protobuf-sdk``. .. _section-hello-types: Defining |PROTOBUF| Types ------------------------- |PROTOBUF_EXT| allows applications to use their existing |PROTOBUF| messages with |CONNEXT|. For this example, we will reference the `AddressBook example `__ included in the |PROTOBUF| documentation, which describes a simple data model composed of a single ``.proto`` file: .. code-block:: protobuf // addressbook.proto syntax = "proto3"; package tutorial; import "google/protobuf/timestamp.proto"; message Person { string name = 1; int32 id = 2; // Unique ID number for this person. string email = 3; enum PhoneType { MOBILE = 0; HOME = 1; WORK = 2; } message PhoneNumber { string number = 1; PhoneType type = 2; } repeated PhoneNumber phones = 4; google.protobuf.Timestamp last_updated = 5; } message AddressBook { repeated Person people = 1; } Most |PROTOBUF| data models can be used without modifications, thanks to the :ref:`detailed mapping ` between |PROTOBUF| and |IDL| defined by |PROTOBUF_EXT|. Exchanging |PROTOBUF| Data -------------------------- The |PROTOBUF| types must be converted into types for the programming language used by application code, using ``protoc``, and ``rtiddsgen``. For example, to generate C++ code: .. code-block:: shell # First use the Protocol Buffers compiler with the additional plugins protoc --idl4_out=build \ --cpp_out=build \ --connext-cpp_out=build \ -I . \ addressbook.proto \ google/protobuf/timestamp.proto # Then use the Connext compiler to generate additional code from the generated IDL rtiddsgen -language C++11 \ -standard PROTOBUF_CPP \ -I build \ build/addressbook.idl \ build/google/protobuf/timestamp.idl The generated "type support" code can be used by applications to exchange |PROTOBUF| data directly over DDS topics: 1. Publish data to a DDS Topic using the |PROTOBUF| types. .. code-block:: c++ // addressbook_publisher.cxx // Include the standard DDS C++ API. #include // Include the generated header files for the Protocol Buffers types. // addressbook.hpp is generated from addressbook.idl by rtiddsgen. // It includes addressbook.pb.h, which is generated by protoc // from addressbook.proto. #include "addressbook.hpp" // Instantiate DDS entities to publish an AddressBook topic on domain 0. dds::domain::DomainParticipant participant(0); dds::topic::Topic topic(participant, "Example tutorial_AddressBook"); dds::pub::Publisher publisher(participant); dds::pub::DataWriter writer(publisher, topic); // Create an instance of the AddressBook message and populate it. tutorial::AddressBook data; auto person = data.add_people(); person->set_name("John Doe"); person->set_id(1); person->set_email("johndoe@example.org"); auto phone = person->add_phones(); phone->set_number("867-5309"); phone->set_type(tutorial::Person_PhoneType_HOME); // Write the AddressBook message using the typed DataWriter. writer.write(data); 2. Subscribe to data on a DDS Topic using the |PROTOBUF| types. .. code-block:: c++ // addressbook_subscriber.cxx // Include the standard DDS C++ API. #include // Include the generated header files for the Protocol Buffers types. #include "addressbook.hpp" // Instantiate DDS entities to subscribe to an AddressBook topic on domain 0. dds::domain::DomainParticipant participant(0); dds::topic::Topic topic(participant, "Example tutorial_AddressBook"); dds::sub::Subscriber subscriber(participant); dds::sub::DataReader reader(subscriber, topic); // Take samples from the typed DataReader. dds::sub::LoanedSamples samples = reader.take(); for (const auto &sample : samples) { if (sample.info().valid()) { // Samples are instances of the Protocol Buffers AddressBook type. tutorial::AddressBook &data = sample.data(); std::cout << data.DebugString() << std::endl; } } Communicating with other DDS applications ----------------------------------------- All ``.proto`` files used with |PROTOBUF_EXT| will be automatically converted into ``.idl`` files, that describe an equivalent data model for the type system defined by the `Extensible and Dynamic Topic Types for DDS `__ specification (*XTypes*). For example, the :ref:`AddressBook data model ` will be converted to the following equivalent |IDL|/*XTypes* data model: .. code-block:: omg-idl // addressbook.idl // ----------------------------------------------------------------------------- // WARNING --------------------------------------------------------------------- // This file was automatically generated by RTI's IDL4 plugin for protoc, // from Protobuf source file: addressbook.proto // Do not edit this file manually. ALL CHANGES WILL BE LOST! // ----------------------------------------------------------------------------- #ifndef tutorial_addressbook_proto_IDL4_ #define tutorial_addressbook_proto_IDL4_ #include "google/protobuf/timestamp.idl" module tutorial { @containing_type("Person") enum Person_PhoneType { @value(0) @default_literal Person_PhoneType_MOBILE, @value(1) Person_PhoneType_HOME, @value(2) Person_PhoneType_WORK }; // enum Person_PhoneType struct Person_PhoneNumber; struct Person; struct AddressBook; @nested @containing_type("Person") @mutable struct Person_PhoneNumber { @id(1) @field_presence(implicit) string number; @id(2) @field_presence(implicit) ::tutorial::Person_PhoneType type; }; // struct Person_PhoneNumber @mutable struct Person { @id(1) @field_presence(implicit) string name; @id(2) @field_presence(implicit) int32 id; @id(3) @field_presence(implicit) string email; @id(4) sequence<::tutorial::Person_PhoneNumber> phones; @id(5) @optional ::google::protobuf::Timestamp last_updated; }; // struct Person @mutable struct AddressBook { @id(1) sequence<::tutorial::Person> people; }; // struct AddressBook }; // tutorial #endif // tutorial_addressbook_proto_IDL4_ The generated |IDL| types can be used in other applications, using any of the Programming Language supported by |CONNEXT|. For example, to use the "modern C++" API with the latest language binding defined by the |IDL| specification: .. code-block:: shell mkdir build-dds rtiddsgen -language C++11 \ -standard IDL4_CPP \ -I build \ -d build-dds \ build/addressbook.idl mkdir -p build-dds/google/protobuf rtiddsgen -language C++11 \ -standard IDL4_CPP \ -I build \ -d build-dds/google/protobuf \ build/google/protobuf/timestamp.idl Building & Running the Example ------------------------------ 1. Load |CONNEXT| and |PROTOBUF_EXT| in the shell environment: .. code-block:: shell source /resource/scripts/rtisetenv_.bash 2. Build the example applications: .. code-block:: shell // Create a build directory and enter it mkdir build cd build // Generate the CMake build files cmake /resource/template/rti_workspace/examples/connext_dds/c++11/protobuf-sdk // Build the example applications cmake --build . 3. Publish data samples to a DDS topic using the |PROTOBUF| types with one of the supported language bindings: a. Start a publisher using the |PROTOBUF| C++ language binding: .. code-block:: shell ./addressbook_publisher b. Start a publisher using the |CONNEXT| C++ language binding: .. code-block:: shell ./addressbook_publisher_dds 4. Subscribe to the DDS topic using the |PROTOBUF| types: a. Start a subscriber using the |PROTOBUF| C++ language binding: .. code-block:: shell ./addressbook_subscriber b. Start a subscriber using the |CONNEXT| C++ language binding: .. code-block:: shell ./addressbook_subscriber_dds c. Start ``rtiddsspy`` to subscribe to all data in the DDS domain: .. code-block:: shell rtiddsspy -printSample Generating Code from |PROTOBUF| Messages ======================================== |PROTOBUF_EXT| relies on a two-phased generative process to produce source code from ``.proto`` files, which will allow |CONNEXT| applications to use the |PROTOBUF| types with DDS topics. .. figure:: static/Protobuf_Extension_Dev_Workflow.png :alt: Code Generation Workflow :name: Figure - How Code Generation Workflow :align: center :figWidth: 100% The ``protoc`` compiler must be invoked first to: - Convert each ``.proto`` file into an equivalent ``.idl`` file. - Generate source code for each |PROTOBUF| type. All input ``.proto`` files must be processed by three ``protoc`` plugins: - The ``idl4`` converter plugin, provided by |PROTOBUF_EXT|, which generates an ``.idl`` file for each input ``.proto``. - The built-in ``cpp`` plugin, which is part of the |PROTOBUF| distribution, and it is responsible for generating the C++ classes associated with each |PROTOBUF| type. - The ``connext-cpp`` plugin, provided by |PROTOBUF_EXT|, which decorates the C++ code produced by ``cpp``, so that it may be efficiently integrated with |CONNEXT|. The two ``protoc`` plugins provided by |PROTOBUF_EXT| are located under ``/bin``. The directory should be added to environment variable ``PATH`` to make them usable by ``protoc``. Alternatively, it is possible to specify the explicit path of each plugin through the ``protoc`` command line, without modifying ``PATH``: .. code-block:: shell protoc --plugin=/bin/protoc-gen-idl4 \ --plugin=/bin/protoc-gen-connext-cpp \ ... The second phase relies on ``rtiddsgen``, |CONNEXT|'s code generator, to process the ``.idl`` files produced by ``protoc``, in order to generate the remaining source code required to use the |PROTOBUF| types as DDS topics in |CONNEXT| applications. :numref:`CodeGenerationSteps` summarizes the steps, with an example of the associate command-line tool invocation, and a list of the files it would produce. .. list-table:: Code Generation Steps :name: CodeGenerationSteps :widths: 50 50 50 :header-rows: 1 * - Code Generation Step - Input - Output * - .. code-block:: shell protoc --idl4_out= \ --cpp_out= \ --connext-cpp_out= \ - ``message.proto`` - * ``message.idl`` * ``message.pb.h`` * ``message.pb.cc`` * - .. code-block:: shell rtiddsgen -language C++11 \ -standard PROTOBUF_CPP \ - ``message.idl`` - * ``message.hpp`` * ``message.cxx`` * ``messagePlugin.hpp`` * ``messagePlugin.cxx`` .. note:: The example ``protoc`` invocation runs all plugins at once, but this is not a strict requirement. Each ``protoc`` plugin can be invoked independently, with the only caveat that ``connext-cpp`` must be run with (or after) the built-in ``cpp`` plugin. Nevertheless, since other options (e.g. include paths) should also be kept consistent across different invocations of ``protoc``, it is recommended to perform generation with all plugins in a single command line. If an input ``.proto`` file imports :ref:`section-omg-dds-descriptor`, the directory containing the file must be specified using the ``-I`` option in the ``protoc`` command line, e.g.: .. code-block:: shell protoc --idl4_out= \ --cpp_out= \ --connext-cpp_out= \ -I /resource/proto \ Code Generation with Multiple Files ----------------------------------- Most projects will use multiple ``.proto`` files, which include ``import`` statements to reference the types they depend on. The corresponding ``.idl`` files generated by the ``idl4`` plugin will contain ``#include`` statements that expect the ``.idl`` files to have been generated in a directory structure similar to the one used by the input files. For example, consider a project containing the following file structure: .. code-block:: shell .src/ ├── a.proto ├── foo │ └── b.proto └── bar └── c.proto We could imagine ``a.proto`` referencing the other file via ``import`` statements, e.g.: .. code-block:: protobuf import "foo/b.proto"; import "bar/c.proto"; After processing ``a.proto`` with the ``idl4`` plugin, the generated ``a.idl`` would then have corresponding ``#include`` statements: .. code-block:: omg-idl #include "foo/b.idl" #include "bar/c.idl" To ensure that the generated ``#include`` statements are valid, all of the imported files must have been converted to ``.idl`` files before ``a.idl`` can be processed by ``rtiddsgen``. The ``.idl`` files must be placed in a similar directory structure: .. code-block:: shell .build/ # generated by: ├── a.idl # └── protoc --idl4_out=... ├── foo │ └── b.idl └── bar └── c.idl A similar ordering requirement may also exist between ``b.proto`` and ``c.proto``, forcing, for example, ``b.idl`` to be generated before ``c.idl`` can be processed. The two files may also be independent, and thus allow for faster processing in parallel. In general, the complexity of managing these dependencies precisely increases rapidly with the number of input files, typically requiring additional "bookkeeping", and making it particularly difficult to implement a "fully parallelized" solution. It is recommended to forego a bit of parallelization efficiency in favor of simpler build rules by breaking up the code generation in two separate steps: 1. First, process all ``.proto`` files with ``protoc``, e.g.: .. code-block:: shell protoc --idl4_out=build \ --cpp_out=build \ --connext-cpp_out=build \ -I src \ src/a.proto \ src/foo/b.proto \ src/bar/c.proto 2. Then, process all ``.idl`` files with ``rtiddsgen``, e.g.: .. code-block:: shell rtiddsgen -language C++11 \ -standard PROTOBUF_CPP \ -I build \ build/a.idl \ build/foo/b.idl \ build/bar/c.idl .. warning:: While both ``protoc`` and ``rtiddsgen`` can process multiple input files from a single command line, it is recommended to use a single input file for each invocation of these tools when automating code generation. This will ensure that the length of the command line invocation never exceeds the maximum allowed by the operating system, which may otherwise happen as the number of ``.proto`` files used by a project grows. At the end of the code generation process, the following C++ files will be available for compilation in the output directory: .. code-block:: shell .build/ # generated by: ├── a.pb.cc # ├── protoc --cpp_out=... --connext-cpp_out=... ├── a.pb.h # ├── protoc --cpp_out=... --connext-cpp_out=... ├── a.cxx # ├── rtiddsgen -language C++11 -standard PROTOBUF_CPP ├── a.hpp # ├── rtiddsgen -language C++11 -standard PROTOBUF_CPP ├── aPlugin.cxx # ├── rtiddsgen -language C++11 -standard PROTOBUF_CPP ├── aPlugin.hpp # └── rtiddsgen -language C++11 -standard PROTOBUF_CPP ├── foo │ ├── b.pb.cc │ ├── b.pb.h │ ├── b.cxx │ ├── b.hpp │ ├── bPlugin.cxx │ └── bPlugin.hpp └── bar ├── c.pb.cc ├── c.pb.h ├── c.cxx ├── c.hpp ├── cPlugin.cxx └── cPlugin.hpp