Open topic with navigation

23.4.2 Using FlatData Language Binding

For examples of FlatData language binding and Zero Copy transfer over shared memory, including example code, see https://community.rti.com/kb/flatdata-and-zerocopy-examples.

23.4.2.1 Selecting FlatData Language Binding

To select FlatData as the language binding of a type, annotate it with @language_binding(FLAT_DATA). (See 3.3.9.9 The @language_binding Annotation.)

For example, consider a surveillance application in which high-definition (HD) video signal is published and subscribed to. The application publishes a Topic of the type CameraImage. This is the IDL: 

enum Format {
   RGB,
   HSV,
   YUV
};
 
@final
@language_binding(FLAT_DATA)
struct Resolution {
 long height;
 long width;
};
 
@final
@language_binding(FLAT_DATA)
struct Pixel {
   octet red;
   octet green;
   octet blue;
};
 
const long MAX_IMAGE_SIZE = 8294400;
 
@mutable
@language_binding(FLAT_DATA)
struct CameraImage {
 string<128> source;
 Format format;
 Resolution resolution;
 sequence<Pixel, MAX_IMAGE_SIZE> pixels;
};

The language binding annotation supports two values: FLAT_DATA and PLAIN (default). PLAIN refers to the regular in-memory representation, where an IDL struct maps to a C++ class or C struct.

There are some restrictions regarding the kinds of structures, value types, and unions to which the FlatData language binding can be applied.

For final types, the FlatData language binding can be applied only to fixed-size types. A fixed-size type is a type whose wire representation always has the same size. This includes primitive types, arrays of fixed-size types, and structs containing only members of fix-size types. Unions are not fixed-size types.1 These restrictions on final types only apply to the FlatData language binding. Final types with the plain language binding can be variable-size.

The FlatData language binding can be applied to any mutable type. This enables support for variable-size types containing bounded sequences, bounded strings, or optional members (unbounded sequences or strings are not supported with FlatData). It also allows using unions.

FlatData cannot be applied to extensible types.

Final types provide the best performance, while mutable types are the most flexible. Typically, the best compromise between flexibility and performance comes from a mutable type whose largest member is either a final type or a sequence of final elements. In the CameraImage example, the top-level type is mutable, which allows for type evolution, optional members, and variable-size members (such as the source string member). On the other hand, its member pixels, which contains the bulk of the data, is defined as a sequence of the final type Pixel, which allows for an efficient manipulation.

23.4.2.2 Programming with FlatData Language Binding

When a type is marked with the FlatData language binding, the in-memory representation for samples of this type is equal to the wire representation (according to XCDR version 22 See Data Representation in the RTI Connext DDS Core Libraries Getting Started Guide Addendum for Extensible Types for more information on XCDR2.). That is, the data sample is in its serialized format at all times. To facilitate accessing and setting the sample content, RTI Code Generator generates helper types that provide the operations to create and access these data samples. These helper types are Samples, Offsets, and Builders.

A FlatData Sample is a buffer holding the wire representation of the data. In the code generated for the previous IDL, a sample of the type CameraImage contains this buffer. This is the top-level object that can be written or read:

typedef rti::flat::Sample<CameraImageOffset> CameraImage;

(Note: These examples show code for the Modern C++ API. See 23.4.2.3 Languages Supported by FlatData Language Binding.)

To access this sample, applications use Offset types. An Offset represents the type of a member and its location in the buffer. An Offset can be described as an “iterator,” a light-weight object that points to the data, but doesn’t own it. Copying an Offset copies the “iterator,” not the data it points to.

class NDDSUSERDllExport CameraImageConstOffset : public rti::flat::MutableOffset {
 public:
   const rti::flat::StringOffset source() const;
   Format format() const;
   Resolution::ConstOffset resolution() const;
   rti::flat::SequenceOffset<Pixel::ConstOffset> pixels() const;
};
 
class NDDSUSERDllExport CameraImageOffset : public rti::flat::MutableOffset {
 public:
   typedef CameraImageConstOffset ConstOffset;
 
   // Const accessors
   const rti::flat::StringOffset source() const;
   Format format() const;
   Resolution::ConstOffset resolution() const;
   rti::flat::SequenceOffset<Pixel::ConstOffset> pixels() const;
 
   // Modifiers
   rti::flat::StringOffset source();
   bool format(Format value);
   Resolution::Offset resolution();
   rti::flat::SequenceOffset<Pixel::Offset> pixels();
};

There are two kinds of Offset types:

For details on all the Offset types and their interface, see the API Reference HTML documentation, under RTI Connext DDS API Reference > Topic Module > FlatData Topic-Types.

The function CameraImage::root() provides the Offset to the top-level type (CameraImageOffset). If the sample is const (for example, in a LoanedSamples container), root() returns a read-only offset (CameraImageConstOffset).

To create variable-size (mutable) data-samples, applications use Builders. A Builder type provides the interface to create a mutable sample member by member. Once all the desired members for a sample have been added, the Builder is “finished,” returning the built sample, which can be published.

class NDDSUSERDllExport CameraImageBuilder : public rti::flat::AggregationBuilder {
 public:
   typedef CameraImageOffset Offset;
 
   Offset finish();
   CameraImage * finish_sample();
 
   rti::flat::StringBuilder build_source();
   bool add_format(Format value);
   Resolution::Offset add_resolution();
   rti::flat::FinalSequenceBuilder<Pixel::Offset> build_pixels();
};

Builders provide three kinds of functions:

Similarly to Offsets, Builders can correspond to user-defined struct and union types, or other IDL types such as sequences, arrays, and strings. For details on all the Builder types see the API Reference HTML documentation.

The following sections summarize how to use FlatData language binding:

Creating a FlatData sample

The following sections assume you have created a DataWriter for the type Pixel or CameraImage, following the usual process.

To write FlatData, first create a FlatData sample. The way to create a sample varies depending on whether the type is final or mutable. In both cases, this section shows how to create DataWriter-managed samples. See also Working with unmanaged FlatData samples.

Creating a FlatData sample for a final type

In this section we will create a sample for the final type Pixel. To create a sample for the mutable type CameraImage, see Creating a FlatData sample for a mutable type after this.

Samples for final FlatData types are created directly with a single call to the DataWriter function get_loan. The DataWriter manages this sample and will return it to a pool at some point after the sample is written.

Pixel *pixel_sample = writer.extensions().get_loan();

pixel_sample contains the buffer that can be written. To set its values, first locate the position of the top-level type:

PixelOffset pixel = pixel_sample->root();

The root() function returns PixelOffset, which points to the position where the data begins. To set the values, use the following setters:

pixel.red(10);
pixel.green(20);
pixel.blue(30);

Creating a FlatData sample for a mutable type

Samples for mutable types are created using Builders. To obtain a CameraImageBuilder to build a CameraImage sample, use the function build_data:

CameraImageBuilder image_builder = rti::flat::build_data(writer);

This function loans the memory necessary to create a CameraImage sample from the DataWriter and provides a CameraImageBuilder to populate it. Use the Builder functions to set the sample’s members (in any order). Non-key members can be omitted, even when they are not optional.3 See "Optional Members" in the RTI Connext DDS Core Libraries Getting Started Guide Addendum for Extensible Types. These Builder functions work on a pre-allocated buffer; they do not allocate any additional memory.

First, we add the member format. As a primitive member, the function add_format directly adds the member and sets its value:

image_builder.add_format(Format::RGB);

Next, we add the member resolution. Its type being final, the function add_resolution adds the member and provides the Offset that allows setting its values:

ResolutionOffset resolution = image_builder.add_resolution();
resolution.height(100);
resolution.width(200);

To build the string member source, the function build_source returns a StringBuilder. We use this builder (in this case it’s as simple as calling set_string), and then call finish. The function finish (not to be confused with finish_sample) completes the construction of the member and renders source_builder invalid.

auto source_builder = image_builder.build_source();
source_builder.set_string(“CAM-1”);
source_builder.finish();

Since this builder is so simple, it is possible to simplify the above code:

image_builder.build_source().set_string("CAM-1");

(The Builder destructor takes care of calling finish.)

To create the pixels member, we build a sequence of Pixels:

auto pixels_builder = image_builder.build_pixels();

There are two ways to populate this member.

Method 1: add and initialize each element:

for (int i = 0; i < 20000; i++) {
   PixelOffset pixel = pixels_builder.add_next();
   pixel.red(i % 256);
   pixel.green((i + 1) % 256);
   pixel.blue((i + 2) % 256);
}
pixels_builder.finish();

Builders for sequences with elements of a final type provide the function add_next to add the elements. When the element type is mutable, the sequence (and array) Builder provides the function build_next, which provides a Builder for each element. See more details in the API Reference HTML documentation.

Method 2: cast the elements in the sequence to the equivalent C++ plain type. This method only works for types that meet the conditions required by rti::flat::plain_cast, as described in the API Reference HTML documentation. Basically, the in-memory representation must match the XCDR2 serialized representation. Pixel meets these conditions.

Method 2 is more efficient. First, we use the Builder function add_n to add 20000 elements at once, leaving them uninitialized. Then, after finishing the Builder, we obtain the Offset to the member, cast it, and manipulate the data as a plain C++ type:

pixels_builder.add_n(20000);
auto pixels_offset = pixels_builder.finish();
 
auto plain_pixels = rti::flat::plain_cast(pixels_offset);
for (int i = 0; i < 20000; i++) {
   plain_pixels[i].red(i % 256);
   plain_pixels[i].green((i + 1) % 256);
   plain_pixels[i].blue((i + 2) % 256);
}

The function rti::flat::plain_cast casts the position in memory that pixels_offset points to into a C-style array of PixelPlainHelper, a type with the same IDL definition as Pixel, but with @language_binding(PLAIN). plain_cast can receive an offset to a final struct, or an offset to an array or sequence of final structs or primitive types. See the API Reference HTML documentation for more information.

Finally, call finish_sample to obtain the complete sample. After this, the Builder instance is invalid and cannot be further used.

CameraImage *image_sample = image_builder.finish_sample();

Once the sample has been created, it is still possible to modify its values, as long as these modifications don’t change the size. For example, it is possible to change the value of an existing pixel, but it’s not possible to add a new one:

auto pixels_offset = image_sample->root().pixels();
pixels_offset.get_element(100).blue(0);

The next section shows how to write the sample.

Writing a FlatData sample

When you write a sample using a regular DataWriter (for a type with a plain language binding), the DataWriter copies the sample in its internal queue, so when write() ends, the application still owns the sample. A DataWriter for a FlatData type, however, doesn’t copy the sample; it keeps a reference. You yield ownership of the data sample from the moment you call write().

writer.write(*image_sample);

The DataWriter will decide when to return samples created with get_loan or build_data to a pool, where the sample will be reused.

To write a new sample, don’t use image_sample again, but obtain a new one with get_loan or build a new one with build_data.

If the sample cannot be written, to return it to the DataWriter pool call:

writer.extensions().discard_loan(*image_sample);

Or, if the sample has not been completely built yet, discard the Builder:

rti::flat::discard_builder(writer, image_builder);

Reading a FlatData sample

The method for reading data for a FlatData type is the same regardless of whether the type is final or mutable.

Create a DataReader as you normally would; see 8.3.1 Creating DataReaders.

Read the data samples:

dds::sub::LoanedSamples<CameraImage> samples = camera_reader.take();

Let’s work with the first sample (assuming samples.length() > 0 and samples[0].info().valid()):

const CameraImage& image_sample = samples[0].data();

Using the root Offset and the Offset to the members, the following code prints the sample values. Note that in this example, image_sample is const, so camera_image is a CameraImageConstOffset, which only allows reading the buffer, not modifying it.

auto camera_image = image_sample->root();
 
std::cout << "Source: " << camera_image.source().get_string() << std::endl;
std::cout << "Timestamp: " << camera_image.timestamp() << std::endl;
std::cout << "Format: " << camera_image.format() << std::endl;
 
auto resolution = camera_image.resolution();
std::cout << "Resolution (height: " << resolution.height()
          << ", width: " << resolution.width() << ")\n";

To access the sequence of pixels, the same two methods that allowed building it (element by element or plain cast) are available:

Method 1 (access each element offset):

for (auto pixel : camera_image.pixels()) {
   std::cout << "Pixel (" << pixel.red() << ", " << pixel.green()
             << ", " << pixel.blue() << ")\n";
}

Method 2 (plain_cast):

auto pixel_count = camera_image.pixels().element_count();
auto plain_pixels = rti::flat::plain_cast(camera_image.pixels());
for (int i = 0; i < pixel_count; i++) {
    const auto& pixel = plain_pixels[i];
    std::cout << "Pixel (" << pixel.red() << ", " << pixel.green()
              << ", " << pixel.blue() << ")\n";
}

Method 2 is more efficient, provided that the type meets the requirements of plain_cast. Also, the endianness of the publishing application must be the same as the local endianness.

Note that you can directly print the sample:

std::cout << *image_sample << std::endl;

Working with unmanaged FlatData samples

The previous sections describe how to create and write DataWriter-managed samples (via get_loan or build_data). While this is the recommended and easiest way, sometimes applications may need to use unmanaged samples. For example, they may need to reuse the same sample after it is written or to obtain the memory from some other source.

Note that a given DataWriter cannot write both unmanaged and managed samples. The functions get_loan or build_data will fail if an unmanaged sample has been written. Conversely, the DataWriter will fail to write an unmanaged sample if get_loan or build_data have been called.

To create a CameraImage using memory from an arbitrary buffer, my_buffer, with a capacity of my_buffer_size bytes, use the following constructor:

unsigned char *my_buffer = ...;
unsigned int my_buffer_size = ...;
CameraImageBuilder image_builder(my_buffer, my_buffer_size);
// use image_builder...
CameraImage *image_sample = image_builder.finish_sample();

image_builder will fail if it runs out of space. The maximum size of a CameraImage can be obtained from its dynamic type:

unsigned int max_size =
rti::topic::dynamic_type<CameraImage>::get().cdr_serialized_sample_max_size();

After writing image_sample, the DataWriter takes ownership of it. In order to reuse the sample, the application needs to monitor the on_sample_removed callback in the DataWriter listener, and correlate the cookie it receives with the sample. The following is a simple DataWriterListener implementation that does that:

class FlatDataWriterListener
   : public dds::pub::NoOpDataWriterListener<CameraImage> {
public:
   void on_sample_removed(
           dds::pub::DataWriter<CameraImage>& writer,
           const rti::core::Cookie& cookie) override
   {
        // The cookie identifies the sample being removed
        last_removed_sample = cookie.to_pointer<CameraImage>();
}
 
CameraImage *last_removed_sample = NULL;
};

The application will need to wait until last_removed_sample is equal to image_sample. This indicates that the DataWriter no longer needs to hold ownership of image_sample.

Another way to create an unmanaged sample is CameraImage::create_data() or Pixel::create_data() (the result of CameraImage::create_data() must be passed to the CameraImageBuilder constructor mentioned before). Samples can be copied with the clone() function. These samples need to be released with the respective delete_data() functions. See the API Reference HTML documentation for more information.

Multi-threading notes

Notes on Extensible Types

There are a few differences in how a plain and a FlatData DataReader behave when they receive samples of types that are different but compatible.

Before a DataReader and DataWriter can communicate, their types are inspected to determine if they are compatible. The same is true when using FlatData; however, even after two types have been deemed compatible, there may be specific data samples that are not.

DataReaders for plain types verify sample compatibility during data deserialization, but DataReaders for FlatData types don’t deserialize the data, passing FlatData samples directly to the application. For that reason, there may be situations where a plain DataReader would drop a data-sample, while a DataReader for a FlatData type with the same definition will pass the same sample to the application. Therefore, if you are using FlatData you may need to explicitly check if all the received samples are consistent with your application logic. For more information on the rules that determine the assignability of a sample, see the RTI Connext DDS Core Libraries Getting Started Guide Addendum for Extensible Types (see the section “Verifying Sample Consistency: Sample Assignability”) or the "Extensible and Dynamic Topic Types for DDS" (DDS-XTypes) specification.

For example, a FlatData DataReader won’t drop a sample when a sequence (or a string) member exceeds the bounds in the reader’s type definition, and the application will be able to read this sequence or string. (This can only happen if ignore_sequence_bounds or ignore_string_bounds in TypeConsistencyEnforcement has been set to true; otherwise the DataWriter’s type wouldn’t have matched the DataReader’s.) The @min and @max annotations are another example. FlatData DataReaders will not enforce the @min/@max range set for a member, and applications will be able to access such samples.

Another difference in behavior involves the reception of samples that don’t include some data members. When a regular DataReader for a mutable (plain) type receives a data sample that doesn’t include one of its non-optional members, it automatically assigns a default value during the data deserialization. A FlatData DataReader for a mutable (FlatData) type will not do that. Instead, if the application tries to access that member, the corresponding member getter will return a null Offset. Only if the member is primitive will it return a default value. This means that, for a FlatData DataReader in this case, all non-primitive members will be treated as if they were optional.

23.4.2.3 Languages Supported by FlatData Language Binding

The FlatData language binding is supported in the Modern and Traditional C++ APIs:

The FlatData language binding is basically the same in both APIs, as described in the previous sections, with a few differences:

© 2020 RTI

Open topic with navigation