6. Protocol Buffers to DDS-XTYPES Mapping

This chapter details how Protocol Buffers types are mapped to DDS-XTYPES via the Protocol Buffers Extension IDL4 Converter Plugin.

For every .proto file processed, the plugin generates an .idl representation. The resulting output contains DDS-XTYPES definitions that are functionally equivalent to the input Protocol Buffers types.

The generated IDL4 files contain type definitions for all Protocol Buffers messages, enumerations, and other constructs, which are mapped to IDL4 constructs.

A C preprocessor guard protects the definitions in each generated file, allowing them to be included by multiple files without causing redefinition errors.

Table 6.1 Protocol Buffers to IDL4 Mapping for .proto files
Protocol Buffers	DDS-XTYPES/IDL4
`message.proto`	`message.idl`
message MyMessage { }	#ifndef message_proto_IDL4_ #define message_proto_IDL4_ @mutable struct MyMessage { }; #endif // message_proto_IDL4_
`myapp/message.proto`	`myapp/message.idl`
package myapp; message MyMessage { }	#ifndef myapp_message_proto_IDL4_ #define myapp_message_proto_IDL4_ module myapp { @mutable struct MyMessage { }; }; // module myapp #endif // message_proto_IDL4_

6.1. User Types

Protocol Buffers supports two user-defined types:

Protocol Buffers messages define the schema for structured data fields.
Protocol Buffers enumerations define the discrete set of values, or literals, allowed for a field.

6.1.1. Protocol Buffers messages

Protocol Buffers messages are mapped to IDL4 structs.

6.1.1.1. Mutability

Each mapped Protocol Buffers message uses the @mutable extensibility. This setting enables type modifications without breaking communication and supports declaration of @optional members. As a result, the XCDR serialization of these types behaves similarly to Protocol Buffers serialization behavior.

To change the default extensibility setting for the mapped IDL4 types, use the .omg.dds.type.extensibility DDS option.

Table 6.2 Protocol Buffers to IDL4 Mapping for Messages
Protocol Buffers	DDS-XTYPES/IDL4
message MyMessage { }	@mutable struct MyMessage { };

6.1.1.2. Nested messages

IDL4 does not support nested type declarations. Therefore, Protocol Buffers messages declared inside other messages are mapped to top-level IDL4 types using a specific naming convention and dedicated annotations.

The name of the IDL4 type associated with a nested message encodes the names of all containing types, concatenated by underscores. This naming convention matches that followed by Protocol Buffers when deriving the names of types associated with nested messages in several programming languages.

Nested types are annotated with @nested to prevent them from being used as top-level types in DDS Topics. This convention better matches the visibility of nested types in Protocol Buffers, which is limited to the .proto file where they are declared.

Nested types are also annotated with @containing_type to further encode their relationship to the containing type.

Table 6.3 Protocol Buffers to IDL4 Mapping for Nested Messages
Protocol Buffers	DDS-XTYPES/IDL4
message MyMessage { message NestedMessage { } }	@nested @containing_type("MyMessage") @mutable struct MyMessage_NestedMessage { };

6.1.2. Protocol Buffers enumerations

Protocol Buffers enumerations are mapped to IDL4 enum types.

6.1.2.1. Enumeration literals

The value of each literal is propagated to IDL4 using the @value annotation. The default literal, which Protocol Buffers typically defines as the first value, is always marked in IDL4 with @default_literal for greater robustness. The literals are exposed as top-level symbols without any prefix, unless the enumeration is nested inside a message.

Table 6.4 Protocol Buffers to IDL4 Mapping for Enumerations
Protocol Buffers	DDS-XTYPES/IDL4
enum MyEnum { HELLO = 0; WORLD = 1; }	enum MyEnum { @value(0) @default_literal HELLO, @value(1) WORLD };

6.1.2.2. Nested Enumerations

Nested enumerations follow the same naming conventions as nested messages. The only difference is that enumeration literals are also prefixed with the nested type’s name.

Consistently with message handling, this naming convention mirrors how Protocol Buffers transforms nested enumeration names when generating code for languages such as C++.

Table 6.5 Protocol Buffers to IDL4 Mapping for Nested Enumerations

Protocol Buffers

DDS-XTYPES/IDL4

message MyMessage {
    enum NestedEnum {
        HELLO = 0;
        WORLD = 1;
    }
}

@containing_type("MyMessage")
enum MyMessage_NestedEnum {
    @value(0)
    @default_literal
    MyMessage_NestedEnum_HELLO,
    @value(1)
    MyMessage_NestedEnum_WORLD
};

6.2. Primitive Types

Each Protocol Buffers primitive (or scalar) type is mapped to the appropriate IDL4 counterpart. This type mapping results in very similar representations in both Protocol Buffers and the other language bindings supported by Connext.

The Protocol Buffers bytes type is the only notable primitive mapping exception. bytes is mapped to a sequence of octets in IDL4 (sequence<octet>) rather than a primitive type. Protocol Buffers typically represents bytes as a string; this structural difference may cause some inconsistencies when accessing these values from different language bindings. In C++ the bytes type is mapped to:

std::string when using the Protocol Buffers C++ API.
std::vector<uint8_t> when using the standard DDS C++ API.

Table 6.6 Protocol Buffers to IDL4 Mapping for Primitive Types
Protocol Buffers	DDS-XTYPES/IDL4
`double`	`double`
`float`	`float`
`int32`	`int32`
`int64`	`int64`
`uint32`	`uint32`
`uint64`	`uint64`
`sint32`	`int32`
`sint64`	`int64`
`fixed32`	`uint32`
`fixed64`	`uint64`
`sfixed32`	`int32`
`sfixed64`	`int64`
`bool`	`boolean`
`string`	`string`
`bytes`	`sequence<octet>`

6.3. Collections

Protocol Buffers offers two ways to represent collections of data:

Repeated fields represent a sequence of values
Map fields represent key-value pairs

6.3.1. Repeated fields

Protocol Buffers repeated fields are mapped to unbounded IDL4 sequences. Use the .omg.dds.member.optional DDS option to mark the corresponding IDL4 member as @optional.

When a message includes a repeated bytes field, the generated IDL4 defines an alias for sequence<octet> scoped within the message type. This alias is required for the definition of the associated struct member because IDL4 does not support sequences of anonymous sequences. For example, syntax like sequence<sequence<T>> is not valid IDL4.

Table 6.7 Protocol Buffers to IDL4 Mapping for Repeated Fields

Protocol Buffers

DDS-XTYPES/IDL4

repeated TYPE my_field ... ;

sequence< TYPE > my_field;

message MyMessage {
    repeated bytes my_field = 1;
}

typedef sequence<octet> MyMessage_OctetSeq;

struct MyMessage {
    @id(1)
    sequence<MyMessage_OctetSeq> my_field;
};

6.3.2. Map fields

Protocol Buffers map fields are mapped to IDL4 sequences of key and value pairs. These pairs are represented using an automatically generated MapPair struct type.

A MapPair struct is defined for each pair of key and value types used by map fields in a Protocol Buffers message using the <message>_MapPair_<key-type>_<value-type> naming convention. The same type is reused by all map fields within a message that share the same key-value definitions.

MapPair struct types are marked with @map_pair, to indicate their use, and with @containing_type, to trace their relationship to the containing message type. The @nested annotation is also used to prevent their use as top-level types for DDS Topics.

Members associated with a map field are annotated with @map to make sure they are represented as map constructs in the language bindings that support them.

Table 6.8 Protocol Buffers to IDL4 Mapping for Map Fields

Protocol Buffers

DDS-XTYPES/IDL4

message MyMessage {
    map<K, V> my_field = 1;
}

@nested
@final
@map_pair
@containing_type("MyMessage")
struct MyMessage_MapPair_K_V {
    K key;
    V value;
};


struct MyMessage {
  @id(1)
  @map
  sequence< MyMessage_MapPair_K_V >
  my_field;
};

Warning

The @map and @map_pair annotations are a temporary solution to allow the Protocol Buffers C++ language binding to support map fields. The annotations are ignored by language bindings where maps must be accessed as linear collections rather than as associative containers.

Map field representations will be updated to use the native IDL4 map type in a future Connext release.

6.4. Packages

Each Protocol Buffers package is mapped to an IDL4 module.

The name of the Protocol Buffers package is split into segments by the dot (.) character, and each segment is mapped to a nested IDL4 module. To convert qualified Protocol Buffers package names to IDL4 module names, replace dots (.) with double colons (::).

The name of the Protocol Buffers package is also included as part of the C preprocessor guard used to wrap all definitions in the generated IDL4 file.

Table 6.9 Protocol Buffers to IDL4 Mapping for Packages

Protocol Buffers

DDS-XTYPES/IDL4

package my.messages.package;

// Type definitions...

#ifndef my_messages_package_proto_IDL4_
#define my_messages_package_proto_IDL4_

module my {
module messages {
module package {

    // Type definitions...

}; // module package
}; // module messages
}; // module my

#endif // my_messages_package_proto_IDL4_

6.5. Imported Files

Each import statement in a Protocol Buffers file is converted to an #include statement in the generated IDL4 file. The #include uses the same path, but with the .idl extension instead of .proto.

All imported .proto files must be independently converted to .idl for the generated #include statements to be valid. Any builtin Protocol Buffers file imported by a .proto file (for example, google/protobuf/timestamp/proto) also needs to be explicitly converted to IDL4.

Typically, Protocol Buffers applications do not need to generate artifacts from these files, because the associated code is already part of the Protocol Buffers binary distribution.

Protocol Buffers Extension does not include a pre-generated IDL4 version of the built-in Protocol Buffers files. This approach allows developers to precisely match the interfaces included in their preferred version of Protocol Buffers.

Table 6.10 Protocol Buffers to IDL4 Mapping for Imported Files
Protocol Buffers	DDS-XTYPES/IDL4
import "another.proto";	#include "another.idl"

6.6. Field Presence

Based on whether a field has implicit presence or explicit presence, protoc generates a different API for the message type:

Fields with explicit presence have an associated has method, which can be used to check whether the field has been set.
Fields with implicit presence do not have an associated has method, and their value can only be checked against the default value for their type (or the default value assigned via options).

Protocol Buffers Extension relies on this classification to determine whether to map a field to an @optional member in IDL4.

Every field with a has method (that is, with explicit presence) not marked as required is mapped to an @optional member in IDL4.

The behavior of fields with explicit presence depends on the Protocol Buffers language revision used in the .proto file. For details about field presence, see the Protocol Buffers documentation.

To override the default mapping of the @optional annotation, use the .omg.dds.member.optional DDS option.

6.6.1. Required fields

The proto2 syntax allows the declaration of required fields. This functionality is also available when using the edition = "2023" syntax via the features.field_presence = LEGACY_REQUIRED option.

Because a required field must be present in every serialized instance of a message, Protocol Buffers Extension maps required fields to IDL4 struct members without the @optional annotation.

Table 6.11 Protocol Buffers to IDL4 Mapping for Required Fields
File Syntax	Protocol Buffers	DDS-XTYPES/IDL4
`proto2`	required TYPE my_field ... ;	TYPE my_field;
`proto3`	N/A	N/A
`edition = "2023"`	TYPE my_field ... [ features.field_presence = LEGACY_REQUIRED ];	TYPE my_field;

6.6.2. Optional fields

All Protocol Buffers syntaxes allow the declaration of optional fields, which represent singular values that may be omitted from a serialized message.

proto2 and proto3 provide the optional keyword to denote such fields, while edition = "2023" automatically marks every singular field as optional. With edition = "2023", fields can be further controlled using the features.field_presence = EXPLICIT option.

Fields with optional semantics are mapped to IDL4 members with the @optional annotation.

File Syntax

Protocol Buffers

DDS-XTYPES/IDL4

proto2

optional TYPE my_field ... ;

@optional
TYPE my_field;

proto3

optional TYPE my_field ... ;

MESSAGE_TYPE my_msg_field;

@optional
TYPE my_field;

@optional
MESSAGE_TYPE my_msg_field;

edition = "2023"

TYPE my_field ... ;

TYPE my_field ... [
  features.field_presence = EXPLICIT
];

@optional
TYPE my_field;

@optional
TYPE my_field;

6.6.3. Implicit fields

The proto3 revision introduced the concept of implicit presence. This revision does not provide Hazzer methods for singular fields of non-message types that are not explicitly marked as optional.

In edition = "2023", implicit presence is enabled by setting the features.field_presence = IMPLICIT option.

In addition to omitting the @optional annotation, IDL4 members derived from fields with implicit presence are annotated with @field_presence(implicit). This annotation encodes the API behavior in the IDL4 data model.

Warning

Protocol Buffers discourage using implicit presence.

File Syntax

Protocol Buffers

DDS-XTYPES/IDL4

proto2

N/A

proto3

NON_MESSAGE_TYPE my_field ... ;

@field_presence(implicit)
NON_MESSAGE_TYPE my_field;

edition = "2023"

NON_MESSAGE_TYPE my_field ... [
  features.field_presence = IMPLICIT
];

@field_presence(implicit)
NON_MESSAGE_TYPE my_field;

6.6.4. Field presence examples

Table 6.14 Example use of the `.omg.dds.member.optional` DDS option
Example Element	Protocol Buffers	DDS-XTYPES/IDL4
Explicitly mark any field as `@optional`	TYPE my_field ... [ .omg.dds.optional = true ];	@optional TYPE my_field;
Prevent default mapping to `@optional`	optional TYPE my_field ... [ .omg.dds.optional = false ];	TYPE my_field;
Mark a `repeated` field as `@optional`	repeated TYPE my_field ... [ .omg.dds.optional = true ];	@optional sequence< TYPE > my_field;
Mark a `map` field as `@optional`	map<K, V> my_field ... [ .omg.dds.optional = true ];	@map @optional sequence< MyMessage_MapPair_K_V > my_field;

6.7. OneOf Fields

Protocol Buffers uses the oneof construct to group multiple optional fields, allowing only one field in the group to be set at any given time.

Every field included in a oneof is treated as an optional message field. The associated IDL4 members are annotated with @optional and @oneof to encode the name of the oneof group.

The @oneof annotation is most useful when the types are used with the Protocol Buffers C++ language binding. Applications using other language bindings must enforce the oneof semantics manually by setting/resetting fields in the same group.

Table 6.15 Protocol Buffers to IDL4 Mapping for OneOf Fields

Protocol Buffers

DDS-XTYPES/IDL4

message MyMessage {

  // other fields...

  oneof oneof_field {
    A oneof_a ...;
    B oneof_b ...;
    C oneof_c ...;
  }

}

@mutable
struct MyMessage {

  // other fields...

  @optional
  @oneof("oneof_field")
  A oneof_a;

  @optional
  @oneof("oneof_field")
  B oneof_b;

  @optional
  @oneof("oneof_field")
  C oneof_c;
};

6.8. Field Groups

Field groups are a Protocol Buffers feature that allows grouping fields within a message without requiring a separate message declaration.

In IDL4, each field group is mapped to a nested struct associated with the declaring message type.

Warning

Protocol Buffers has deprecated field groups; they are only available when using the older proto2 syntax.

File Syntax

Protocol Buffers

DDS-XTYPES/IDL4

proto2

message MyMessage {
    required group my_req_group = 1 {  }

    optional group my_opt_group = 2 {  }

    repeated group my_rep_group = 3 {  }
}

@nested
@containing_type("MyMessage")
@mutable
struct MyMessage_MyReqGroup {  };

@nested
@containing_type("MyMessage")
@mutable
struct MyMessage_MyOptGroup {  };

@nested
@containing_type("MyMessage")
@mutable
struct MyMessage_MyRepGroup {  };

struct MyMessage {
    @id(1)
    MyMessage_MyReqGroup my_req_group;

    @id(2)
    @optional
    MyMessage_MyOptGroup my_opt_group;

    @id(3)
    sequence<MyMessage_MyRepGroup> my_rep_group;
};

proto3

N/A

edition = "2023"

N/A

6.9. Unsupported Features

The following Protocol Buffers features are not supported by Protocol Buffers Extension:

Message extensions
Self-referencing messages