.. _section-proto2idl: |PROTOBUF| to |DDS-XTYPES| Mapping ******************************************* This chapter details how Protocol Buffers types are mapped to |DDS-XTYPES| via the |PROTOBUF_EXT| IDL4 Converter Plugin. For every ``.proto`` file processed, the plugin generates an ``.idl`` representation. The resulting output contains |DDS-XTYPES| definitions that are functionally equivalent to the input |PROTOBUF| types. The generated |IDL| files contain type definitions for all |PROTOBUF| messages, enumerations, and other constructs, which are mapped to |IDL| constructs. A C preprocessor guard protects the definitions in each generated file, allowing them to be included by multiple files without causing redefinition errors. .. list-table:: |MAPPING| for .proto files :name: Proto2IdlMapping :widths: 50 50 :header-rows: 1 * - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - ``message.proto`` - ``message.idl`` * - .. code-block:: protobuf message MyMessage { } - .. code-block:: omg-idl #ifndef message_proto_IDL4_ #define message_proto_IDL4_ @mutable struct MyMessage { }; #endif // message_proto_IDL4_ * - ``myapp/message.proto`` - ``myapp/message.idl`` * - .. code-block:: protobuf package myapp; message MyMessage { } - .. code-block:: omg-idl #ifndef myapp_message_proto_IDL4_ #define myapp_message_proto_IDL4_ module myapp { @mutable struct MyMessage { }; }; // module myapp #endif // message_proto_IDL4_ User Types ========== |PROTOBUF| supports two user-defined types: - :ref:`section-proto2idl-messages` define the schema for structured data fields. - :ref:`section-proto2idl-enumerations` define the discrete set of values, or literals, allowed for a field. .. _section-proto2idl-messages: |PROTOBUF| messages ------------------- |PROTOBUF| messages are mapped to |IDL| structs. Mutability ^^^^^^^^^^ Each mapped |PROTOBUF| message uses the ``@mutable`` extensibility. This setting enables type modifications without breaking communication and supports declaration of ``@optional`` members. As a result, the XCDR serialization of these types behaves similarly to |PROTOBUF| serialization behavior. To change the default extensibility setting for the mapped |IDL| types, use the ``.omg.dds.type.extensibility`` DDS option. .. list-table:: |MAPPING| for Messages :name: MessagesMapping :widths: 50 50 :header-rows: 1 * - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - .. code-block:: protobuf message MyMessage { } - .. code-block:: omg-idl @mutable struct MyMessage { }; .. _section-nested-messages: Nested messages ^^^^^^^^^^^^^^^ |IDL| does not support nested type declarations. Therefore, |PROTOBUF| messages declared inside other messages are mapped to top-level |IDL| types using a specific naming convention and dedicated annotations. The name of the |IDL| type associated with a nested message encodes the names of all containing types, concatenated by underscores. This naming convention matches that followed by |PROTOBUF| when deriving the names of types associated with nested messages in several programming languages. Nested types are annotated with ``@nested`` to prevent them from being used as top-level types in DDS |TOPICS|. This convention better matches the visibility of nested types in |PROTOBUF|, which is limited to the ``.proto`` file where they are declared. Nested types are also annotated with ``@containing_type`` to further encode their relationship to the containing type. .. list-table:: |MAPPING| for Nested Messages :name: NestedMessagesMapping :widths: 50 50 :header-rows: 1 * - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - .. code-block:: protobuf message MyMessage { message NestedMessage { } } - .. code-block:: omg-idl @nested @containing_type("MyMessage") @mutable struct MyMessage_NestedMessage { }; .. _section-proto2idl-enumerations: |PROTOBUF| enumerations ----------------------- |PROTOBUF| enumerations are mapped to |IDL| ``enum`` types. Enumeration literals ^^^^^^^^^^^^^^^^^^^^ The value of each literal is propagated to |IDL| using the ``@value`` annotation. The default literal, which |PROTOBUF| typically defines as the first value, is always marked in |IDL| with ``@default_literal`` for greater robustness. The literals are exposed as top-level symbols without any prefix, unless the enumeration is nested inside a message. .. list-table:: |MAPPING| for Enumerations :name: EnumerationsMapping :widths: 50 50 :header-rows: 1 * - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - .. code-block:: protobuf enum MyEnum { HELLO = 0; WORLD = 1; } - .. code-block:: omg-idl enum MyEnum { @value(0) @default_literal HELLO, @value(1) WORLD }; Nested Enumerations ^^^^^^^^^^^^^^^^^^^ Nested enumerations follow the same naming conventions as :ref:`nested messages `. The only difference is that enumeration literals are also prefixed with the nested type's name. Consistently with message handling, this naming convention mirrors how |PROTOBUF| transforms nested enumeration names when generating code for languages such as C++. .. list-table:: |MAPPING| for Nested Enumerations :name: NestedEnumerationsMapping :widths: 50 50 :header-rows: 1 * - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - .. code-block:: protobuf message MyMessage { enum NestedEnum { HELLO = 0; WORLD = 1; } } - .. code-block:: omg-idl @containing_type("MyMessage") enum MyMessage_NestedEnum { @value(0) @default_literal MyMessage_NestedEnum_HELLO, @value(1) MyMessage_NestedEnum_WORLD }; Primitive Types =============== Each |PROTOBUF| primitive (or scalar) type is mapped to the appropriate |IDL| counterpart. This type mapping results in very similar representations in both |PROTOBUF| and the other language bindings supported by |CONNEXT|. The |PROTOBUF| ``bytes`` type is the only notable primitive mapping exception. ``bytes`` is mapped to a sequence of octets in |IDL| (``sequence``) rather than a primitive type. |PROTOBUF| typically represents `bytes` as a string; this structural difference may cause some inconsistencies when accessing these values from different language bindings. In C++ the ``bytes`` type is mapped to: - ``std::string`` when using the |PROTOBUF| C++ API. - ``std::vector`` when using the standard DDS C++ API. .. list-table:: |MAPPING| for Primitive Types :name: PrimitiveTypesMapping :widths: 50 50 :header-rows: 1 * - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - ``double`` - ``double`` * - ``float`` - ``float`` * - ``int32`` - ``int32`` * - ``int64`` - ``int64`` * - ``uint32`` - ``uint32`` * - ``uint64`` - ``uint64`` * - ``sint32`` - ``int32`` * - ``sint64`` - ``int64`` * - ``fixed32`` - ``uint32`` * - ``fixed64`` - ``uint64`` * - ``sfixed32`` - ``int32`` * - ``sfixed64`` - ``int64`` * - ``bool`` - ``boolean`` * - ``string`` - ``string`` * - ``bytes`` - ``sequence`` Collections =========== |PROTOBUF| offers two ways to represent collections of data: - :ref:`Repeated fields ` represent a sequence of values - :ref:`Map fields ` represent key-value pairs .. _section-proto2idl-repeated-fields: Repeated fields --------------- |PROTOBUF| repeated fields are mapped to unbounded |IDL| sequences. Use the ``.omg.dds.member.optional`` DDS option to mark the corresponding |IDL| member as @optional. When a message includes a repeated ``bytes`` field, the generated |IDL| defines an alias for ``sequence`` scoped within the message type. This alias is required for the definition of the associated ``struct`` member because |IDL| does not support sequences of anonymous sequences. For example, syntax like ``sequence>`` is not valid |IDL|. .. list-table:: |MAPPING| for Repeated Fields :name: RepeatedFieldsMapping :widths: 40 40 :header-rows: 1 * - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - .. code-block:: protobuf repeated TYPE my_field ... ; - .. code-block:: omg-idl sequence< TYPE > my_field; * - .. code-block:: protobuf message MyMessage { repeated bytes my_field = 1; } - .. code-block:: omg-idl typedef sequence MyMessage_OctetSeq; struct MyMessage { @id(1) sequence my_field; }; .. _section-proto2idl-map-fields: Map fields ---------- |PROTOBUF| map fields are mapped to |IDL| sequences of ``key`` and ``value`` pairs. These pairs are represented using an automatically generated MapPair ``struct`` type. A MapPair ``struct`` is defined for each pair of key and value types used by map fields in a |PROTOBUF| message using the ``_MapPair__`` naming convention. The same type is reused by all map fields within a message that share the same key-value definitions. MapPair ``struct`` types are marked with ``@map_pair``, to indicate their use, and with ``@containing_type``, to trace their relationship to the containing message type. The ``@nested`` annotation is also used to prevent their use as top-level types for DDS |TOPICS|. Members associated with a map field are annotated with ``@map`` to make sure they are represented as map constructs in the language bindings that support them. .. list-table:: |MAPPING| for Map Fields :name: MapFieldsMapping :widths: 40 40 :header-rows: 1 * - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - .. code-block:: protobuf message MyMessage { map my_field = 1; } - .. code-block:: omg-idl @nested @final @map_pair @containing_type("MyMessage") struct MyMessage_MapPair_K_V { K key; V value; }; struct MyMessage { @id(1) @map sequence< MyMessage_MapPair_K_V > my_field; }; .. warning:: The ``@map`` and ``@map_pair`` annotations are a temporary solution to allow the |PROTOBUF| C++ language binding to support map fields. The annotations are ignored by language bindings where maps must be accessed as linear collections rather than as associative containers. Map field representations will be updated to use the native |IDL| ``map`` type in a future |CONNEXT| release. Packages ======== Each |PROTOBUF| package is mapped to an |IDL| ``module``. The name of the |PROTOBUF| package is split into segments by the dot (``.``) character, and each segment is mapped to a nested |IDL| module. To convert qualified |PROTOBUF| ``package`` names to |IDL| ``module`` names, replace dots (``.``) with double colons (``::``). The name of the |PROTOBUF| package is also included as part of the C preprocessor guard used to wrap all definitions in the generated |IDL| file. .. list-table:: |MAPPING| for Packages :name: PackagesMapping :widths: 50 50 :header-rows: 1 * - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - .. code-block:: protobuf package my.messages.package; // Type definitions... - .. code-block:: omg-idl #ifndef my_messages_package_proto_IDL4_ #define my_messages_package_proto_IDL4_ module my { module messages { module package { // Type definitions... }; // module package }; // module messages }; // module my #endif // my_messages_package_proto_IDL4_ Imported Files ============== Each ``import`` statement in a |PROTOBUF| file is converted to an ``#include`` statement in the generated |IDL| file. The ``#include`` uses the same path, but with the ``.idl`` extension instead of ``.proto``. All imported ``.proto`` files must be independently converted to ``.idl`` for the generated ``#include`` statements to be valid. Any builtin |PROTOBUF| file imported by a ``.proto`` file (for example, ``google/protobuf/timestamp/proto``) also needs to be explicitly converted to |IDL|. Typically, |PROTOBUF| applications do not need to generate artifacts from these files, because the associated code is already part of the |PROTOBUF| binary distribution. |PROTOBUF_EXT| does not include a pre-generated |IDL| version of the built-in |PROTOBUF| files. This approach allows developers to precisely match the interfaces included in their preferred version of |PROTOBUF|. .. list-table:: |MAPPING| for Imported Files :name: ImportedFilesMapping :widths: 50 50 :header-rows: 1 * - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - .. code-block:: protobuf import "another.proto"; - .. code-block:: omg-idl #include "another.idl" .. _section-proto2idl-field-presence: Field Presence ============== Based on whether a field has ``implicit presence`` or ``explicit presence``, ``protoc`` generates a different API for the message type: - Fields with ``explicit presence`` have an associated ``has`` method, which can be used to check whether the field has been set. - Fields with ``implicit presence`` do not have an associated ``has`` method, and their value can only be checked against the default value for their type (or the default value assigned via options). |PROTOBUF_EXT| relies on this classification to determine whether to map a field to an ``@optional`` member in |IDL|. Every field with a ``has`` method (that is, with ``explicit presence``) not marked as ``required`` is mapped to an ``@optional`` member in |IDL|. The behavior of fields with ``explicit presence`` depends on the |PROTOBUF| language revision used in the ``.proto`` file. For details about field presence, see the `Protocol Buffers documentation `__. To override the default mapping of the ``@optional`` annotation, use the ``.omg.dds.member.optional`` DDS option. Required fields --------------- The ``proto2`` syntax allows the declaration of ``required`` fields. This functionality is also available when using the ``edition = "2023"`` syntax via the ``features.field_presence = LEGACY_REQUIRED`` option. Because a required field must be present in every serialized instance of a message, |PROTOBUF_EXT| maps required fields to |IDL| ``struct`` members without the ``@optional`` annotation. .. list-table:: |MAPPING| for Required Fields :name: RequiredFieldsMapping :widths: 30 50 50 :header-rows: 1 * - File Syntax - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - ``proto2`` - .. code-block:: protobuf required TYPE my_field ... ; - .. code-block:: omg-idl TYPE my_field; * - ``proto3`` - N/A - N/A * - ``edition = "2023"`` - .. code-block:: protobuf TYPE my_field ... [ features.field_presence = LEGACY_REQUIRED ]; - .. code-block:: omg-idl TYPE my_field; Optional fields --------------- All |PROTOBUF| syntaxes allow the declaration of ``optional`` fields, which represent singular values that may be omitted from a serialized message. ``proto2`` and ``proto3`` provide the ``optional`` keyword to denote such fields, while ``edition = "2023"`` automatically marks every singular field as ``optional``. With ``edition = "2023"``, fields can be further controlled using the ``features.field_presence = EXPLICIT`` option. Fields with ``optional`` semantics are mapped to |IDL| members with the ``@optional`` annotation. .. list-table:: |MAPPING| for Optional Fields :name: OptionalFieldsMapping :widths: 30 50 50 :header-rows: 1 * - File Syntax - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - ``proto2`` - .. code-block:: protobuf optional TYPE my_field ... ; - .. code-block:: omg-idl @optional TYPE my_field; * - ``proto3`` - .. code-block:: protobuf optional TYPE my_field ... ; MESSAGE_TYPE my_msg_field; - .. code-block:: omg-idl @optional TYPE my_field; @optional MESSAGE_TYPE my_msg_field; * - ``edition = "2023"`` - .. code-block:: protobuf TYPE my_field ... ; TYPE my_field ... [ features.field_presence = EXPLICIT ]; - .. code-block:: omg-idl @optional TYPE my_field; @optional TYPE my_field; Implicit fields --------------- The ``proto3`` revision introduced the concept of ``implicit presence``. This revision does not provide ``Hazzer`` methods for singular fields of non-message types that are not explicitly marked as ``optional``. In ``edition = "2023"``, ``implicit presence`` is enabled by setting the ``features.field_presence = IMPLICIT`` option. In addition to omitting the ``@optional`` annotation, |IDL| members derived from fields with ``implicit presence`` are annotated with ``@field_presence(implicit)``. This annotation encodes the API behavior in the |IDL| data model. .. warning:: |PROTOBUF| discourage using *implicit presence*. .. list-table:: |MAPPING| for Implicit Fields :name: ImplicitFieldsMapping :widths: 30 50 50 :header-rows: 1 * - File Syntax - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - ``proto2`` - N/A - N/A * - ``proto3`` - .. code-block:: protobuf NON_MESSAGE_TYPE my_field ... ; - .. code-block:: omg-idl @field_presence(implicit) NON_MESSAGE_TYPE my_field; * - ``edition = "2023"`` - .. code-block:: protobuf NON_MESSAGE_TYPE my_field ... [ features.field_presence = IMPLICIT ]; - .. code-block:: omg-idl @field_presence(implicit) NON_MESSAGE_TYPE my_field; .. _section-proto2idl-fieldpresence-options: Field presence examples ----------------------- .. list-table:: Example use of the ``.omg.dds.member.optional`` DDS option :name: OptionalMemberOption :widths: 30 50 50 :header-rows: 1 * - Example Element - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - Explicitly mark any field as ``@optional`` - .. code-block:: protobuf TYPE my_field ... [ .omg.dds.optional = true ]; - .. code-block:: omg-idl @optional TYPE my_field; * - Prevent default mapping to ``@optional`` - .. code-block:: protobuf optional TYPE my_field ... [ .omg.dds.optional = false ]; - .. code-block:: omg-idl TYPE my_field; * - Mark a ``repeated`` field as ``@optional`` - .. code-block:: protobuf repeated TYPE my_field ... [ .omg.dds.optional = true ]; - .. code-block:: omg-idl @optional sequence< TYPE > my_field; * - Mark a ``map`` field as ``@optional`` - .. code-block:: protobuf map my_field ... [ .omg.dds.optional = true ]; - .. code-block:: omg-idl @map @optional sequence< MyMessage_MapPair_K_V > my_field; OneOf Fields ============ |PROTOBUF| uses the ``oneof`` construct to group multiple optional fields, allowing only one field in the group to be set at any given time. Every field included in a ``oneof`` is treated as an ``optional`` message field. The associated |IDL| members are annotated with ``@optional`` and ``@oneof`` to encode the name of the ``oneof`` group. The ``@oneof`` annotation is most useful when the types are used with the |PROTOBUF| C++ language binding. Applications using other language bindings must enforce the ``oneof`` semantics manually by setting/resetting fields in the same group. .. list-table:: |MAPPING| for OneOf Fields :name: OneOfFieldsMapping :widths: 40 40 :header-rows: 1 * - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - .. code-block:: protobuf message MyMessage { // other fields... oneof oneof_field { A oneof_a ...; B oneof_b ...; C oneof_c ...; } } - .. code-block:: omg-idl @mutable struct MyMessage { // other fields... @optional @oneof("oneof_field") A oneof_a; @optional @oneof("oneof_field") B oneof_b; @optional @oneof("oneof_field") C oneof_c; }; Field Groups ============ Field groups are a |PROTOBUF| feature that allows grouping fields within a message without requiring a separate message declaration. In |IDL|, each field group is mapped to a nested ``struct`` associated with the declaring message type. .. warning:: |PROTOBUF| has deprecated field groups; they are only available when using the older ``proto2`` syntax. .. list-table:: |MAPPING| for Field Groups :name: FieldGroupsMapping :widths: 30 50 50 :header-rows: 1 * - File Syntax - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - ``proto2`` - .. code-block:: protobuf message MyMessage { required group my_req_group = 1 { } optional group my_opt_group = 2 { } repeated group my_rep_group = 3 { } } - .. code-block:: omg-idl @nested @containing_type("MyMessage") @mutable struct MyMessage_MyReqGroup { }; @nested @containing_type("MyMessage") @mutable struct MyMessage_MyOptGroup { }; @nested @containing_type("MyMessage") @mutable struct MyMessage_MyRepGroup { }; struct MyMessage { @id(1) MyMessage_MyReqGroup my_req_group; @id(2) @optional MyMessage_MyOptGroup my_opt_group; @id(3) sequence my_rep_group; }; * - ``proto3`` - N/A - N/A * - ``edition = "2023"`` - N/A - N/A Unsupported Features ==================== The following |PROTOBUF| features are not supported by |PROTOBUF_EXT|: - Message extensions - Self-referencing messages