Recording Large User Data Types

3 posts / 0 new
Last post
Offline
Last seen: 9 years 11 months ago
Joined: 07/29/2013
Posts: 7
Recording Large User Data Types

Hello,

I'm trying to figure out a solution for recording and doing post-analysis on large user types. I have an IDL file with many, many sequences and according to the RTI Recording documentation recording should be done with RTI_DESERIALIZEMODE_NEVER. Which is fine.. it doesn't record anything unless I use this method. However, I need to export such data into a readable format for post-analysis. The rtirecconv tool seems to fail because it could not get the type information.

My question is: I have another IDL file with a smaller set of sequences which records in serialized mode just fine. Conversion afterwards is also fine. Why does the smaller IDL work but not the larger one? What can I do to reconstruct my original IDL into something readable?

Thanks!

Offline
Last seen: 10 years 5 months ago
Joined: 06/24/2013
Posts: 8

Hi:

There a a number of issues that can affect the recording, replay, and conversion, of complex data types using Recording Service. (Note that the issue at stake here is the number of elements in the data type, not the physical size of the data - so I'm not calling it "large data" types. Its the data-type that is large, not the data itself).

It sounds like your larger IDL has more data elements that your smaller IDL, and is more complex.

The first limit that might be reached is the default limit on the size of the type code that can be sent by DDS. More complex data types have larger type codes. There is a knowledge base article on "How to send big TypeCodes and TypeObjects" that describes how to work with large type codes:

http://community.rti.com/kb/how-send-big-typecodes-and-typeobjects

This publisher's limit on the size of the type code that can be sent may be what is preventing recorder from knowing the type of the data, preventing recorder from being able to deserialize it, and preventing rtirecconv from being able to convert it. Neither recorder nor rtirecconv has access to the type code, because it wasn't sent.This limit applies to all DDS communications, not just recorder.

Note that the modified QoS to allow sending larger type codes must be applied to the original publisher. That is, you won't be able to change that just by modifying recorder's configuration.

Another limit that is unique to recorder is that when recording data in de-serialized mode, recorder will "flatten" then data representation - i.e., store each member of a sequence in its own column (or even more if the sequence is a sequence of complex types). Because there is a limit to the number of columns that recorder can store in a user data table (I think the limit is 999 or 1000), data types that have more primitive members than the limit, can not be stored in de-serialized form. As far as I know, this limit cannot be changed. In any case, I don't think that you need to store data de-serialized to accomplish what you want, but it is useful to know why less complex data types can be recorded de-serialized while more complex types can't.

One last reason why rtirecconv might be unable to obtain the type code is that if recorder is using multiple file segments (storing the data in mutiple files of a maximum file size), by default the type code will be contained in the first file, but not in subsequent files of the file set. This may lead rtirecconv to not be able to convert subsequent files of a file set.

However, even if a type code is unavailable because of size, or not being present in one file of a file set, there is still a way to convert your data.

rtirecconv also accepts an external type code definition using the -typeConfig option with a XML description of the type code. This option is described in more detail in the RTI Recording Service User's Manual. The XML representation of the type code, needed by rtirecconv, can be generated using the -convertToXml option of the rtiddsgen tool, decribed in more detail in the RTI Core Libraries and Utilities User's Manual. By generating an XML representation of your type code, and using that as an input to rtirecconv, you should be able to convert the recorded data from your large IDL type.

I hope one of these solutions - either increasing the size of the type code that can be sent, or using an XML description of the type code for rtirecconv - works for you to enable you to access your recorded data in readable format.

Offline
Last seen: 3 years 1 week ago
Joined: 01/15/2013
Posts: 94

Hi,

Just a little side note about Recorder's column limitation. The 999 column limit is going to be increased in the next general access release.