.. _section-proto2idl: |PROTOBUF| to |DDS-XTYPES| Mapping ********************************** Every ``.proto`` file passed to the ``idl4`` plugin will be converted into an ``.idl`` file containing |DDS-XTYPES| data types equivalent to the |PROTOBUF| ones passed as input. The generated |IDL| files will contain type definitions for all |PROTOBUF| messages, enumerations, and other constructs which can be mapped to |IDL| constructs. The definitions will be guarded by a C preprocessor guard, to ensure that the generated file can be included by multiple files without redefinition errors. .. list-table:: |MAPPING| for |PROTOBUF| files to |IDL| files :name: Proto2IdlMapping :widths: 50 50 :header-rows: 1 * - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - ``message.proto`` - ``message.idl`` * - .. code-block:: protobuf message MyMessage { } - .. code-block:: omg-idl #ifndef message_proto_IDL4_ #define message_proto_IDL4_ @mutable struct MyMessage { }; #endif // message_proto_IDL4_ * - ``myapp/message.proto`` - ``myapp/message.idl`` * - .. code-block:: protobuf package myapp; message MyMessage { } - .. code-block:: omg-idl #ifndef myapp_message_proto_IDL4_ #define myapp_message_proto_IDL4_ module myapp { @mutable struct MyMessage { }; }; // module myapp #endif // message_proto_IDL4_ User Types ========== |PROTOBUF| supports two types of user-defined types: - :ref:`section-proto2idl-messages` - :ref:`section-proto2idl-enumerations` .. _section-proto2idl-messages: Messages -------- |PROTOBUF| messages are mapped to |IDL| ``struct``'s, with ``@mutable`` extensibility. The use of the ``@mutable`` extensibility allows modifications to the types without breaking communication, as well as declaration of ``@optional`` members. This makes the *XCDR* serialization of these types behave similarly to their |PROTOBUF| serialization. The extensibility of the mapped |IDL| types can be changed with option ``.omg.dds.type.extensibility``. |IDL| does not support nested type declarations, therefore |PROTOBUF| messages declared inside other messages, are mapped to top-level |IDL| types using a special naming convention and dedicated annotations. The name of the |IDL| type associated with a nested message will encode the names of all containing types, concatenated by underscores. This naming convention matches the one followed by |PROTOBUF| in several programming languages (e.g. C++) when deriving the names of types associated with nested messages. Nested types are marked as ``@nested``, to prevent them from being used as top-level types in DDS topics. This choice is made to better match the visibility of nested types in |PROTOBUF|, which is limited to the ``.proto`` file where they are declared. Nested types are also marked with ``@containing_type``, to further encode their relationship to the containing type. .. list-table:: |MAPPING| for Messages :name: MessagesMapping :widths: 50 50 :header-rows: 1 * - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - .. code-block:: protobuf message MyMessage { } - .. code-block:: omg-idl @mutable struct MyMessage { }; * - .. code-block:: protobuf message MyMessage { message NestedMessage { } } - .. code-block:: omg-idl @nested @containing_type("MyMessage") @mutable struct MyMessage_NestedMessage { }; .. _section-proto2idl-enumerations: Enumerations ------------ |PROTOBUF| enumerations are mapped to |IDL| ``enum`` types. The value of each enumeration literal is propagated to |IDL| using the ``@value`` annotation. The default literal, which |PROTOBUF| typically defines as the first value, is always marked in |IDL| with ``@default_literal``, for greater robustness. The literals are exposed as top-level symbols without any prefix, unless the enumeration is nested inside a message. Nested enumerations follow the same naming conventions as nested messages, with the only difference being that enumeration literals are also prefixed with the nested type's name. Similar to the handling of messages, this naming choice matches how |PROTOBUF| "mangles" the names of nested enumerations when translating them to programming languages such as C++. .. list-table:: |MAPPING| for Enumerations :name: EnumerationsMapping :widths: 50 50 :header-rows: 1 * - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - .. code-block:: protobuf enum MyEnum { HELLO = 0; WORLD = 1; } - .. code-block:: omg-idl enum MyEnum { @value(0) @default_literal HELLO, @value(1) WORLD }; * - .. code-block:: protobuf message MyMessage { enum NestedEnum { HELLO = 0; WORLD = 1; } } - .. code-block:: omg-idl @containing_type("MyMessage") enum MyMessage_NestedEnum { @value(0) @default_literal MyMessage_NestedEnum_HELLO, @value(1) MyMessage_NestedEnum_WORLD }; Primitive Types =============== All |PROTOBUF| primitive (or "scalar") types are mapped to their |IDL| counterparts, which results in very similar representations in both |PROTOBUF|, and the many other "language bindings" supported by |CONNEXT|. The only considerable deviation is with the |PROTOBUF| type ``bytes``, which is mapped to a sequence of octests in |IDL| (i.e. ``sequence``), clearly not a primitive. `bytes` is typically represented as a string by |PROTOBUF|, which may cause some confusion when accessing these values from different language bindings. For example, in C++, the ``bytes`` type will be mapped to: - ``std::string``, when using the |PROTOBUF|/C++ API. - ``std::vector``, when using the standard DDS C++ API. .. list-table:: |MAPPING| for Primitive Types :name: PrimitiveTypesMapping :widths: 50 50 :header-rows: 1 * - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - ``double`` - ``double`` * - ``float`` - ``float`` * - ``int32`` - ``int32`` * - ``int64`` - ``int64`` * - ``uint32`` - ``uint32`` * - ``uint64`` - ``uint64`` * - ``sint32`` - ``int32`` * - ``sint64`` - ``int64`` * - ``fixed32`` - ``uint32`` * - ``fixed64`` - ``uint64`` * - ``sfixed32`` - ``int32`` * - ``sfixed64`` - ``int64`` * - ``bool`` - ``boolean`` * - ``string`` - ``string`` * - ``bytes`` - ``sequence`` Collections =========== |PROTOBUF| offers two ways to represent collections of data: - :ref:`section-proto2idl-repeated-fields` - :ref:`section-proto2idl-map-fields` .. _section-proto2idl-repeated-fields: Repeated Fields --------------- |PROTOBUF| repeated fields are mapped to unbounded |IDL| sequences. The corresponding |IDL| member may be marked as ``@optional`` by using option ``.omg.dds.member.optional``. When a message includes a repeated field of type ``bytes``, the generated |IDL| will include an alias for ``sequence`` scoped within the message type. The alias will be used in the definition of the associated ``struct`` member, because |IDL| does not allow defining sequences of "anonymous" sequences (i.e. ``sequence>`` is not valid |IDL|). .. list-table:: |MAPPING| for Repeated Fields :name: RepeatedFieldsMapping :widths: 40 40 :header-rows: 1 * - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - .. code-block:: protobuf repeated TYPE my_field ... ; - .. code-block:: omg-idl sequence< TYPE > my_field; * - .. code-block:: protobuf message MyMessage { repeated bytes my_field = 1; } - .. code-block:: omg-idl typedef sequence MyMessage_OctetSeq; struct MyMessage { @id(1) sequence my_field; }; .. _section-proto2idl-map-fields: Map Fields ---------- |PROTOBUF| map fields are mapped to |IDL| sequences of ``key`` and ``value`` pairs represented using an automatically-generated "MapPair" ``struct`` type. A "MapPair" type will be defined for each pair of key and value types used by map fields in a |PROTOBUF| message, following the naming scheme ``_MapPair__``. The same type will be reused by all map fields in a message which share the same key and value types, The "MapPair" ``struct`` types are marked with ``@map_pair`` to indicate their use, and with ``@containing_type`` to trace their relationship to the containing message type. The ``@nested`` annotation is also used to prevent their use as top-level types for DDS topics. The members associated with a map field will be annotated with ``@map`` to indicate that they should be mapped to map constructs in those language bindings that support them. .. list-table:: |MAPPING| for Map Fields :name: MapFieldsMapping :widths: 40 40 :header-rows: 1 * - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - .. code-block:: protobuf message MyMessage { map my_field = 1; } - .. code-block:: omg-idl @nested @final @map_pair @containing_type("MyMessage") struct MyMessage_MapPair_K_V { K key; V value; }; struct MyMessage { @id(1) @map sequence< MyMessage_MapPair_K_V > my_field; }; .. warning:: The mapping for map fields will be updated to the native |IDL| ``map`` type once support for this construct has been introduced by a future release of |CONNEXT|. The ``@map`` and ``@map_pair`` annotations are a temporary solutition to allow the |PROTOBUF|/C++ langauge binding to support map fields. The annotations will be ignored by all other language bindings, where maps must be accessed as linear collections, rather than as associative containers. Packages ======== |PROTOBUF| packages are mapped to |IDL| ``module``'s. The name of the |PROTOBUF| package is split into segments by the dot (``.``) character, and each segment is mapped to a nested |IDL| module. Qualified |PROTOBUF| ``package`` names can be converted to |IDL| ``module`` names by replacing dots (``.``) with double colons (``::``). The name of the |PROTOBUF| package will also be used as part of the C preprocesor guard used to wrap all definitions in the generated |IDL| file. .. list-table:: |MAPPING| for Packages :name: PackagesMapping :widths: 50 50 :header-rows: 1 * - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - .. code-block:: protobuf package my.messages.package; // Type definitions... - .. code-block:: omg-idl #ifndef my_messages_package_proto_IDL4_ #define my_messages_package_proto_IDL4_ module my { module messages { module package { // Type definitions... }; // module package }; // module messages }; // module my #endif // my_messages_package_proto_IDL4_ Imported Files ============== Each ``import`` statement in a |PROTOBUF| file will be converted to an ``#include`` statement in the generated |IDL| file, using the same path, but with the ``.idl`` extension instead of ``.proto``. All imported ``.proto`` files must be independently converted to ``.idl`` for the generated ``#include`` statements to become valid. Any "built-in" |PROTOBUF| file imported by a ``.proto`` file (e.g. ``google/protobuf/timestamp/proto``) will also need to be explicitly converted to |IDL|. Typically, |PROTOBUF| applications do not need to generate artifacts from these files, because the associated code is already part of the |PROTOBUF| binary distribution. |PROTOBUF_EXT| does not include a pre-generated |IDL| version of the "built-in" |PROTOBUF| files in order to allow users to precisely match the interfaces included in their preferred version of |PROTOBUF|. .. list-table:: |MAPPING| for Imported Files :name: ImportedFilesMapping :widths: 50 50 :header-rows: 1 * - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - .. code-block:: protobuf import "another.proto"; - .. code-block:: omg-idl #include "another.idl" Field Presence ============== To quote the |PROTOBUF| `documentation about Field Presence `__: .. epigraph:: Field presence is the notion of whether a protobuf field has a value. There are two different manifestations of presence for protobufs: - *implicit presence*, where the generated message API stores field values (only). - *explicit presence*, where the API also stores whether or not a field has been set. Based on whether a field has "implicit" or "explicit" presence, the |PROTOBUF| compiler will generate a different API for the message type: - Fields with *explicit presence* have an associated "has" method, which can be used to check whether the field has been set. - Fields with *implicit presence* do not have an associated "has" method, and their value can only be checked against the default value for their type (or the default value assigned via options). |PROTOBUF_EXT| relies on this classification to determine whether the field should be mapped to an ``@optional`` member in |IDL|, or not. Every field with a "has" method (i.e. which *explicit presence*), which is not marked as "required", will be mapped to an ``@optional`` member in |IDL|. Refer to the |PROTOBUF| `documentation about Field Presence `__ for a detailed summary of which fields have *explicit presence*, which changes depending on the syntax/edition used. It is possible to override the default mapping of the ``@optional`` annotation using option ``.omg.dds.member.optional``. Required Fields --------------- The ``proto2`` syntax allows the declaration of ``required`` fields. This is also possible when using the ``edition = "2023"`` syntax, by using the ``features.field_presence = LEGACY_REQUIRED`` option. A field marked as "required" must be present in every serialization of the message. As such, |PROTOBUF_EXT| maps "required" fields to members of |IDL| ``struct``'s without the ``@optional`` annotation. .. list-table:: |MAPPING| for Required Fields :name: RequiredFieldsMapping :widths: 30 50 50 :header-rows: 1 * - File Syntax - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - ``proto2`` - .. code-block:: protobuf required TYPE my_field ... ; - .. code-block:: omg-idl TYPE my_field; * - ``proto3`` - N/A - N/A * - ``edition = "2023"`` - .. code-block:: protobuf TYPE my_field ... [ features.field_presence = LEGACY_REQUIRED ]; - .. code-block:: omg-idl TYPE my_field; Optional Fields --------------- All |PROTOBUF| syntaxes allow the declaration of "optional" fields, which are singular values that may or may not appear in the network serialization of a message. ``proto2`` and ``proto3`` provide the ``optional`` keyword to denote such fields, while ``edition = "2023"`` automatically marks every singular field as "optional". With ``edition = "2023"``, fields can be futher controlled using the ``features.field_presence = EXPLICIT`` option. Any field with "optional" semantics will be mapped to an |IDL| member with the ``@optional`` annotation. .. list-table:: |MAPPING| for Optional Fields :name: OptionalFieldsMapping :widths: 30 50 50 :header-rows: 1 * - File Syntax - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - ``proto2`` - .. code-block:: protobuf optional TYPE my_field ... ; - .. code-block:: omg-idl @optional TYPE my_field; * - ``proto3`` - .. code-block:: protobuf optional TYPE my_field ... ; MESSAGE_TYPE my_msg_field; - .. code-block:: omg-idl @optional TYPE my_field; @optional MESSAGE_TYPE my_msg_field; * - ``edition = "2023"`` - .. code-block:: protobuf TYPE my_field ... ; TYPE my_field ... [ features.field_presence = EXPLICIT ]; - .. code-block:: omg-idl @optional TYPE my_field; @optional TYPE my_field; Implicit Fields --------------- ``proto3`` introduced the concept of *implicit presence*, by not providing "hazzer" methods for any singular field of a non-message type, which is not explicitly marked as ``optional``. The same *implicit presence* can be achieved in ``edition = "2023"`` by using the ``features.field_presence = IMPLICIT`` option. Beside not being marked as ``@optional``, |IDL| members derived from fields with *implicit presence* will be annotated with ``@field_presence(implicit)`` to encode this peculiar API choice in the |IDL| data model. .. warning:: The use of *implicit presence* is discouraged by |PROTOBUF|. .. list-table:: |MAPPING| for Implicit Fields :name: ImplicitFieldsMapping :widths: 30 50 50 :header-rows: 1 * - File Syntax - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - ``proto2`` - N/A - N/A * - ``proto3`` - .. code-block:: protobuf NON_MESSAGE_TYPE my_field ... ; - .. code-block:: omg-idl @field_presence(implicit) NON_MESSAGE_TYPE my_field; * - ``edition = "2023"`` - .. code-block:: protobuf NON_MESSAGE_TYPE my_field ... [ features.field_presence = IMPLICIT ]; - .. code-block:: omg-idl @field_presence(implicit) NON_MESSAGE_TYPE my_field; .. _section-proto2idl-fieldpresence-options: Field Presence Examples ----------------------- .. list-table:: Example use of option ``.omg.dds.member.optional`` :name: OptionalMemberOption :widths: 30 50 50 :header-rows: 1 * - Example Element - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - Explicitly mark any field as ``@optional`` - .. code-block:: protobuf TYPE my_field ... [ .omg.dds.optional = true ]; - .. code-block:: omg-idl @optional TYPE my_field; * - Prevent default mapping to ``@optional`` - .. code-block:: protobuf optional TYPE my_field ... [ .omg.dds.optional = false ]; - .. code-block:: omg-idl TYPE my_field; * - Mark a ``repeated`` field as ``@optional`` - .. code-block:: protobuf repeated TYPE my_field ... [ .omg.dds.optional = true ]; - .. code-block:: omg-idl @optional sequence< TYPE > my_field; * - Mark a ``map`` field as ``@optional`` - .. code-block:: protobuf map my_field ... [ .omg.dds.optional = true ]; - .. code-block:: omg-idl @map @optional sequence< MyMessage_MapPair_K_V > my_field; OneOf Fields ============ |PROTOBUF| provides the ``oneof`` construct to group multiple optional fields together, allowing only one of them to be set at a time. Every field that is part of a ``oneof`` will be treated as an "optional" field of the message. The associated |IDL| members will be annotated with ``@optional``, and with ``@oneof`` to encode the name of the ``oneof`` group. The ``@oneof`` annotation is most useful when the types are used with |PROTOBUF|/C++ language binding. Applications using the types with other language bindings must enforce the "oneof" semantics by manually setting/resetting fields in the same group. .. list-table:: |MAPPING| for OneOf Fields :name: OneOfFieldsMapping :widths: 40 40 :header-rows: 1 * - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - .. code-block:: protobuf message MyMessage { // other fields... oneof oneof_field { A oneof_a ...; B oneof_b ...; C oneof_c ...; } } - .. code-block:: omg-idl @mutable struct MyMessage { // other fields... @optional @oneof("oneof_field") A oneof_a; @optional @oneof("oneof_field") B oneof_b; @optional @oneof("oneof_field") C oneof_c; }; Field Groups ============ Field groups are a |PROTOBUF| feature that allows grouping of fields within a message without the need to declare another dedicated message. In |IDL|, each field group is mapped to a nested ``struct`` associated with the declaring message type. .. warning:: Field groups are **deprecated** by |PROTOBUF|, and they are only available when using the older ``proto2`` syntax. .. list-table:: |MAPPING| for Field Groups :name: FieldGroupsMapping :widths: 30 50 50 :header-rows: 1 * - File Syntax - |PROTOBUF| - |DDS-XTYPES|/|IDL| * - ``proto2`` - .. code-block:: protobuf message MyMessage { required group my_req_group = 1 { } optional group my_opt_group = 2 { } repeated group my_rep_group = 3 { } } - .. code-block:: omg-idl @nested @containing_type("MyMessage") @mutable struct MyMessage_MyReqGroup { }; @nested @containing_type("MyMessage") @mutable struct MyMessage_MyOptGroup { }; @nested @containing_type("MyMessage") @mutable struct MyMessage_MyRepGroup { }; struct MyMessage { @id(1) MyMessage_MyReqGroup my_req_group; @id(2) @optional MyMessage_MyOptGroup my_opt_group; @id(3) sequence my_rep_group; }; * - ``proto3`` - N/A - N/A * - ``edition = "2023"`` - N/A - N/A Unsupported Features ==================== - Message extensions. - Self-referencing messages.