.. _section-using: Integrating |PROTOBUF| with |CONNEXT_HEADING| ********************************************** |PROTOBUF_EXT| allows your applications to use their existing |PROTOBUF| messages with |CONNEXT|. It uses a generative process to produce C++ source code from ``.proto`` files. The generated code enables |CONNEXT| applications to use the |PROTOBUF| types with DDS |TOPICS|. .. figure:: static/Protobuf_Extension_Dev_Workflow.png :alt: Code Generation Workflow :name: Figure - How Code Generation Workflow :align: center :figWidth: 100% There are two methods to generate the necessary source code from an input ``.proto`` file using |PROTOBUF_EXT|: * :ref:`Two-phase generation `. Execute ``protoc`` using the RTI plugins and all necessary options, then run ``rtiddsgen`` on the resulting IDL file. We recommend this workflow because it simplifies the protoc configuration process. * :ref:`Single-phase generation `. Invoke ``rtiddsgen`` using the ``.proto`` file as the direct input to generate all necessary files. This streamlined method is ideal for simple ``.proto`` files that do not contain includes or require specific ``protoc`` configuration options. .. _section-two-phase: Two-Phase Code Generation ========================= In phase one, the protocol buffers compiler (``protoc``) is invoked to: - Convert each ``.proto`` file into an equivalent ``.idl`` file. - Generate C++ source code for each |PROTOBUF| type. In phase two, all input ``.proto`` files are processed by three ``protoc`` plugins: - The |PROTOBUF_EXT| :ref:`section-idl4-plugin` (``idl4``) generates an ``.idl`` file for each input ``.proto`` file. The generated ``.idl`` files use the type system defined by the `Extensible and Dynamic Topic Types for DDS `__ specification (DDS-XTypes) from the Object Management Group (OMG). - The built-in C++ code generator (``cpp``), part of the |PROTOBUF| distribution, generates the C++ classes associated with each |PROTOBUF| type. - The |PROTOBUF_EXT| :ref:`section-connext-cpp-plugin` (``connext-cpp``) decorates the C++ code produced by ``cpp`` so that it can be efficiently integrated with |CONNEXT|. Phase two relies on |RTI_CODEGEN| (``rtiddsgen``) to process the ``.idl`` files generated by ``protoc``. ``rtiddsgen`` produces the additional C++ source code necessary to use |PROTOBUF| types as DDS |TOPICS| within |CONNEXT| applications. .. _section-single-phase: Single-Phase Code Generation ============================ You can generate code from a ``.proto`` file in a single step using ``rtiddsgen``. When a ``.proto`` file is used as input, ``rtiddsgen`` automatically invokes ``protoc`` internally using the :ref:`RTI plugins `. Use the ``-protocPath`` option to specify the ``protoc`` executable location if it is not in your system path. Use the ``-protocOption`` flag to pass specific parameters directly to ``protoc``. Code Generation Commands ======================== :numref:`CodeGenerationCommands` shows the two commands used to generate the necessary source code from an input ``.proto`` file using the :ref:`two-phase code generation method `. It lists the input files, the commands used to generate the output, and the resulting files. .. list-table:: Protocol Buffers Extensions Code Generation Commands :name: CodeGenerationCommands :widths: 50 50 50 :header-rows: 1 * - Command - Input - Output * - .. code-block:: shell protoc --idl4_out= \ --cpp_out= \ --connext-cpp_out= \ - ``message.proto`` - * ``message.idl`` * ``message.pb.h`` * ``message.pb.cc`` * - .. code-block:: shell rtiddsgen -language C++11 \ -standard PROTOBUF_CPP \ - ``message.idl`` - * ``message.hpp`` * ``message.cxx`` * ``messagePlugin.hpp`` * ``messagePlugin.cxx`` In the example ``protoc`` command above, all plugins are run at once. This is not a strict requirement; each ``protoc`` plugin can be invoked independently. However, it is recommended to run ``protoc`` with all plugins in a single command because other options (for example, ``#include`` paths) should be kept consistent across different ``protoc`` calls. .. important:: If the plugins are called independently, RTI's ``connext-cpp`` plugin must be run with (or after) the built-in ``cpp`` plugin. If an input ``.proto`` file imports the RTI :ref:`omg/dds/descriptor.proto ` file, the directory containing the file must be specified using the ``-I`` option in the ``protoc`` command line. For example: .. code-block:: shell :emphasize-lines: 4 protoc --idl4_out= \ --cpp_out= \ --connext-cpp_out= \ -I /resource/proto \ When using the :ref:`single-phase code generation method `, a similar ``rtiddsgen`` command is used, but the input file is the original ``.proto`` file instead of the generated IDL file. The following example generates source code directly from ``example.proto``: .. code-block:: shell rtiddsgen -language C++11 \ -standard PROTOBUF_CPP example.proto \ -protocPath \ -protocOption=--idl4_out= Generating Code for Multiple Files ================================== Most projects use multiple ``.proto`` files with dependencies between them. These dependencies are typically expressed using ``import`` statements in the ``.proto`` files. For these projects, you need to make sure the code generation process respects the dependency order. .. note:: If your project has multiple ``.proto`` files with dependencies, it is recommended to use the :ref:`two-phase code generation method `. This method simplifies the management of dependencies between files. Integrating .proto files with dependencies ------------------------------------------ As an example, consider the following file structure: .. code-block:: shell .src/ ├── a.proto ├── foo │ └── b.proto └── bar └── c.proto Here, ``a.proto`` imports ``b.proto`` and ``c.proto`` using ``import`` statements: .. code-block:: protobuf import "common/b.proto"; import "utils/c.proto"; After processing ``a.proto`` with the IDL4 Converter Plugin, the generated ``a.idl`` would then have corresponding ``#include`` statements: .. code-block:: omg-idl #include "foo/b.idl" #include "bar/c.idl" To ensure the generated ``#include`` statements are valid: - all imported ``.proto`` files must be converted to ``.idl`` files before ``a.idl`` can be processed by ``rtiddsgen``. - the generated ``.idl`` files must be placed in a similar directory structure. For example: .. code-block:: shell .build/ # generated by: ├── a.idl # └── protoc --idl4_out=... ├── foo │ └── b.idl └── bar └── c.idl If a similar ordering requirement existed between ``b.proto`` and ``c.proto``, ``b.idl`` would have to be generated before ``c.idl`` can be processed. If the two files were independent, they could be processed simultaneously for faster processing in parallel. In this example, code generation can be executed as follows using the :ref:`two-phase code generation method `: 1. First, process all ``.proto`` files with ``protoc``. .. code-block:: shell protoc --idl4_out=build \ --cpp_out=build \ --connext-cpp_out=build \ -I src \ src/a.proto \ src/foo/b.proto \ src/bar/c.proto This command compiles multiple related ``.proto`` files, generating both standard C++ code and |CONNEXT|-specific integration code, with all output going to a single build directory. 2. Then, process all ``.idl`` files with ``rtiddsgen``. .. code-block:: shell rtiddsgen -language C++11 \ -standard PROTOBUF_CPP \ -I build \ build/a.idl \ build/foo/b.idl \ build/bar/c.idl This command generates C++ code from the related IDL files that were converted from |PROTOBUF| types in step 1. It processes IDL files organized in different directories and resolves cross-file dependencies using the ``#include`` path. When code generation is complete, the following C++ files are generated and made available for compilation in the output directory: .. code-block:: shell .build/ # generated by: ├── a.pb.cc # ├── protoc --cpp_out=... --connext-cpp_out=... ├── a.pb.h # ├── protoc --cpp_out=... --connext-cpp_out=... ├── a.cxx # ├── rtiddsgen -language C++11 -standard PROTOBUF_CPP ├── a.hpp # ├── rtiddsgen -language C++11 -standard PROTOBUF_CPP ├── aPlugin.cxx # ├── rtiddsgen -language C++11 -standard PROTOBUF_CPP ├── aPlugin.hpp # └── rtiddsgen -language C++11 -standard PROTOBUF_CPP ├── foo │ ├── b.pb.cc │ ├── b.pb.h │ ├── b.cxx │ ├── b.hpp │ ├── bPlugin.cxx │ └── bPlugin.hpp └── bar ├── c.pb.cc ├── c.pb.h ├── c.cxx ├── c.hpp ├── cPlugin.cxx └── cPlugin.hpp .. warning:: Both ``protoc`` and ``rtiddsgen`` can process multiple input files from a single command line. However, when automating code generation, it is recommended to use a single input file for each invocation of these plugins. This approach ensures that the length of the command-line invocation does not exceed system command-line length limits, which is a risk as the number of ``.proto`` files used by a project grows. Managing dependencies --------------------- Managing these precise dependencies becomes increasingly complex as the number of input files grows. This complexity typically introduces additional maintenance overhead and makes a fully parallelized solution difficult to implement. It is recommended to forego a bit of parallelization efficiency in favor of simpler build rules. That's why we recommend using the :ref:`two-phase code generation ` method when working with multiple ``.proto`` files with dependencies.