Dynamic Data: Binary Blob field

5 posts / 0 new
Last post
Offline
Last seen: 3 years 3 weeks ago
Joined: 05/05/2021
Posts: 2
Dynamic Data: Binary Blob field

As part of an effort to convert a legacy system to DDS I'm working on a Routing Service Adapter that commmunicates over a socket with the legacy system. For messages that only go one place we just pull them off the wire and pull out some key fields to filter on but the message is basicly a variable length binary blob.

The .idl looks like this. 

struct LegacyEvents
{
int32 message_type;
sequence<char,LEGACY_BODY_MAX_LENGTH> body;
};

I'm basing this off the adapter example so using the dynamic data API. (I only started with DDS 6 months ago). The question is:

What is the best way to deal with the body field?  It would be nice for it to be variable length for efficiency.  You can access the elements of a sequence normally but is there some way to get a pointer to do a block copy (memcopy?). Would be nice to be able to recieve into the field also, but not sure if that is possible.

Can't imagine this is the first time this has come up. Thank you.

asorbini's picture
Offline
Last seen: 1 year 2 months ago
Joined: 02/06/2013
Posts: 11

Hi jnugent,

The `body` member is already of variable size, since it's defined as a sequence, which is a dynamically-sized container: samples shared over DDS will only contain elements up to the sequence field's actual length, not its maximum (as would have instead been the case, had the member been declared as an array). My only suggestion is to consider changing the element type from `char` to `octet` in order to guarantee that the included bytes will never be interpreted as anything but a raw chunk of bytes by any of the various DDS language mappings (unless the contents are actually to be considered (utf-8) characters).

You can access the "contiguous buffer" associated with a sequence using FooSeq_get_contiguous_buffer (in C, but the same operation is also exposed in other language mappings). You should be able to receive from the socket directly into the sequence buffer with the following pattern:

  1. Allocate memory in the sequence by calling FooSeq_set_maximum()+FooSeq_set_length() (or the convenience wrapper FooSeq_ensure_length, which allows to modify both max and length in a single call). Set maximum and length to the maximum size of a UDP packet you expect your application to receive.
  2. Retrieve a pointer to the sequence's contiguous buffer (with FooSeq_get_contiguous_buffer) and pass it to your UDP receive call.
  3. Update the sequence's length to the actual length of the received packet (by calling FooSeq_set_length with the actual size of the received packet returned by the UDP receive call).

Finally, you might be interested in taking a look the RTI Connext Gateway repository, which contains a collection of more advanced plugins for Routing Service, including some custom Adapters, that may prove useful as a reference to go beyond "trivial" examples.

Offline
Last seen: 3 years 3 weeks ago
Joined: 05/05/2021
Posts: 2

Thanks for the info. I'll give it a try. 

I didn't know about the gateway repository. That looks usefull. Thank you.

 

asorbini's picture
Offline
Last seen: 1 year 2 months ago
Joined: 02/06/2013
Posts: 11

I have thought of one possible caveat that I neglected to mention in my previous post: while it is true that you should be able to receive from the UDP socket into the buffer of a sequence (e.g. `DDS_OctetSeq`, which is the type of `LegacyEvents::data` in C if element is changed to `octet`) you will still need to perform one copy when setting this value into the DynamicData sample that will be propagated through RoutingService.

This is because DDS_DynamicData_set_octet_seq() (and other setters for sequence members) implicitly perform a copy of the sequence passed as input. The same is true for the corresponding getters, which also copy the sequence buffer into a user supplied sequence. There is no way currently to access a raw pointer to the buffer backing the sequence member of a DynamicData sample.

Because of this copy, you might want to consider using a sequence to simply wrap the receive buffer allocated by your application, instead of using the sequence API to do memory management of this buffer as suggested in my previous post (via the `set_maximum()` and `set_length()` operations), which would be mostly useful if you were to draw memory from the actual sample (which is possible for statically typed data, but as I said, it's not possible with DynamicData samples).

Instead, you could keep using your own buffer, "loan it" to a sequence via the FooSeq_loan_contiguous() operation, and then pass the value to `DynamicData_seq_XXX_seq()` to actually store it in a sample.

Hope this helps! Let me know if you have any questions.

pschmitt's picture
Offline
Last seen: 3 years 5 months ago
Joined: 07/28/2021
Posts: 1

Hi James,

I will post our off-line email yesterday:

Rephase your requirements:
1) You want to use a sequence in the idl to take advantage of the variable length?
2) You would like to directly copy the body into the sequence using something like a points?, but..
3) You want to do this with dynamic data api?
 
Would this work..... 
1) Define your idl as a normal struct
2) use the set_octet_array api to dynamically copy data into the body - this makes it a sequence on the wire and efficient for DDS
 
typedef octet Body[LEGACY_BODY_MAX_LENGTH];
 

struct LegacyEvents
{
int32 message_type;
Body body;
};

Below is code I used for a project that copied a deviceID into an octet array - I modified it (semi-pseudocode) to give you a flavor of what I did relative to your structure.

class WriterTopic : public Topic {
public:
WriterTopic(DDSDomainParticipant * participant, enum TOPICS_E topicEnum, bool prefillDevId=true);
virtual ~WriterTopic(void) = 0; // Abstract base class

DDSDynamicDataWriter * myWriter;
DDS_DynamicData * myData;
};

class fooWriterTopic : public WriterTopic {
public:
fooWriterTopic(DDSDomainParticipant * participant, enum TOPICS_E topicEnum, bool prefillDevId=true);
~fooWriterTopic();
DDSDynamicDataWriter * getMyWriter(); // needed for Requests to get the response writer
DDS_DynamicData * getMyDataInstance();
 
private:
WriterEventsThreadInfo * myWriterEventsThreadInfo;

};

 

fooWriterTopic * legacy_event_topic;
 
retcode = legacy_event_topic->getMyDataInstance()->set_octet_array
("legacy_event.body", DDS_DYNAMIC_DATA_MEMBER_ID_UNSPECIFIED, actual_length_to_copy, (const DDS_Octet *)&my_body);