I am looking at creating a plugin for the dds recorder that can serialize dds data to XCDR2 format and send over a network stream.
I am using rti connext 6.0.1 and the modern c++ API. I can use the function to_cdr_buffer to serialize a DynamicData object but am unsure how to deserialize the byte stream?
The documentation for from_cdr_buffer says you should be able to use DynamicData as TopicType but that fails to compile (DynamicData constructor private).
I can use from_cdr_buffer_no_alloc with an existing DynamicData, but that requires me to know the exact type.
So two questions really, first does the serialized DynamicData retain the type information in the XCDR2 format or is it just the binary data?
If not, is it possible to also serialize/deserialize the type object (DynamicType)? If so how?
I don't think DynamicData and XCDR2 are what you think they are.
1) XCDR2 is a standardized mapping of a typed data structure to a binary representation that can be sent/received. A serilalized XCDR2 object only contains the contents of the data structure, no description, not even the name of the data structure nor the typename of the data structure, let alone type information.
2) A DynamicData object is an object that can hold data of any datatype. But a DynamicData object has a data type. You have to tell the DynamicData object what its type is for it to be able to deserialize a buffer into itself.
You can't create a DynamicData object...it's fundamentally an abstract base class. You can instance one of the derived classes, create it from the type support code generated from an IDL description of the type, create it from an XML description of the type, and some others...see:
https://community.rti.com/static/documentation/connext-dds/6.0.1/doc/api/connext_dds/api_cpp2/classdds_1_1core_1_1xtypes_1_1DynamicType.html
But fundmentally, a DynamicData object needs to know what type it is to deserialize (or serialize) itself from (to) XCDR2 representation.
However, I am curious, you say that you're trying to write a plugin for RTI Recording Service to subscribe to DDS topics, then serialize the received values into XCDR2 format and then send over a network. What is the use case? DDS just sent the data over a network in XCDR2 format to the Recording Service....you could have just subscribe to the data using DDS over the network directly. What's on the other end of the network that can't use DDS?
Thank you for the answer.
Yes it is maybe an unconventional use case. The dds recording service is running on a machine that does not have storage for the logging, but there is another machine that can log network data to disk that I want to send the serialized stream to. The deserializing would then be done offline later.
The problem is that the logging will be done for software that is evolving over time and the dds data types will sometimes change, so it would be great if I didn't have to keep track of the exact types used but instead was able to store the the type information in the logged data as well.
I would like to know how the recording service manages to discover the type information dynamically, since in the recording plugin you get DynamicData objects with the correct type for each topic. Since it is possible there I thought there was a way for me to do it as well, but maybe there is no public API?
Can't you use DDS between the host that you would be running RTI Recording Service and the host that you want to use to log the data to disk?
Are those two hosts are running on the same network as the one that's carring the data that needs to be recorded?
What's the reason you can't run RTI Recording Service on the host that does have disk storage for logging?
If the situation is that you have 2 networks, one that is carrying the data that you want to log and the host (aka logging host) that you can use to log data isn't directly connected to that network, but you do have a host (aka relay host ) that can connect to both networks (assume that's the one that you're currently contemplating running RTI Recording Service with a custom plugin), then instead of doing the complicated thing that you're considering...assuming you are able to run RTI Recording Service on the logging host, then you can run RTI Routing Service on the relay host and configure the Routing Service to subscribe to data from one network and then send it via DDS to the Recording Service on the other network.
But yes, it is possible for you to write your own application to discover published topics, and then dynamically register data types that were discovered and subscribe to the topics (create DataReaders), and then use the DynamicData API to manipulate the received data...all without having to hardcode/compile any knowledge of data types apriori. This is how all of RTI's Services, e.g., Recording Service, Routing Service, and tools, e.g., Admin Console is able to deal with users' topics and datatypes.
Please see this documentation.
There are code examples here
https://community.rti.com/examples/builtin-topics
Lets just assume that this is how I have to do it so we can leave that out. I have very little control over the setup, the other machine (with the disk) is not a generic PC and it is not possible to install the RTI recording service there.
The documentation for builtin topics was helpful for me to understand how this works, but I don't want to create my own recording service. I am not sure if the example code is updated for 6.0, am a bit confused about type codes vs type objects.
I saw that it is possible to get the idl from the DynamicType objects, so maybe that is good enough, if I include that then I can run rtiddsgen and build a decoder. It would be nice if it was possible to do it dynamically though. You can create a DynamicType from xml but not from idl.
If you can't run Recording Service, can you run an application that uses DDS on the other machine (with disk)? If so, you can just use RTI Routing Service to relay the data to the other machine and write an application that uses the builtin discovery listeners to detect and subscribe to the topics published by RTI Routing Service.
Type code is the term used in the Dynamic Data API. On the wire, the DDS specification calls the binary encoding of the data type definition, type object...which is passed around on discovery. Confusingly, RTI had offered a non-standard format, now deprecated, which we called type code. It's actually still there for backwards compatibility, but it needs to be turned on via QOS.
Are you trying to write code for the Recording Service plugin?
Actually, I would suggest that you look into RTI Routing Service...which you can write an adapter to redirect the data to a network connection, just like Recording Service...but since Recording Service is actually built on top of Routing Service...and what Routing Service was designed to do is actually what you're trying to do, you should switch your efforts to Routing Service.
Is the "decoder" that you're referring to the code that you think needs to be in the Routing Service?
In the Routing Service adapter (and the Recording Service plugin), you'll be provided the data to be relayed in a Dynamic Data object that already has the typecode. It should be all ready for your code to send over a network (after serialization).
So, I assume you're talking about the code on the logging machine which has to receive the serialized data and store it on disk.
In which case, are you assuming that you'll be using APIs from the RTI Connext DDS libraries to help you write the logging application? All of the Dynamic Type code, etc.? If so, then there will be DDS on that logging machine...which then begs the question, what is preventing you from running a full DDS application that uses DDS to receive the data on the logging machine?
Thank you, I will look into the routing service.
The logging machine is a 3rd party product that can log network traffic, I have no control over what is installed there. I have access to the disk when the logging is done and can copy the logged data, the decoder in this case would do postprocessing on another machine (which I have control over). It could be possible to replay the logged network traffic though, so using the routing service is still an interresting possibility.
OK, I see, thanks for details. Wow that's certainly painful. It seems like it would be cheaper solution just to add a workstation-level pc with as much storage as needed to your network and run Recording Service (which can do playback already), than to spend time writing a Routing Service plugin, and then a decoder app and a playback app. About $3000 to $4000 vs a month of your time...
In any case, what does the network traffic logger actually store and then allow you to access?
Full UDP packets? A TCP stream? Or is it at the Ethernet packet level, ala wireshark?
If it's not basically what come in a TCP stream, then you'll have to deal with how to reassemble data fragments into what constitutes a whole piece of data.
If storing at the Ethernet level, you have to deal with reassembly of 1500 byte IP fragments for any data > 1450 bytes or so.
If able to store full UDP packets, then you have to deal with reassembly of any data that is > 64000 bytes or so....DDS has fragmentation/reassembly algorithms built in, but if your adapter is using UDP to send data to the logger, your adapter now needs to fragment the data into what fits into a UDP packet and your decoder/replayer would have to ressemble the UDP packets into the full data that was originally sent.
In addition, I assume that you want to store the data for multiple topics that being published simultaneously. If so, you'll have to consider how to MUX all of the data into a single network stream that's recorded by the logger...and then how to DEMUX the recorded data back into the streams for different topics. At least, you'll have to put some sort of topic identifier with each piece of data that you send to the logger.
When referring to "replay", are you saying that the 3rd party network traffic logger can replay the traffic that it logged onto the network? If so, does it do that with the same timing that it received the data? If this is the case, if your goal is for the logged data to be replayed as DDS data, then you'll have to create another adapter for the Routing Service that receives the data stream from the network traffic logger, DEMUX the data into separate DDS topic streams, and then repackage the data for Routing Service to publish as DDS data.
Good luck!