Hi,
I was wondering if there is any existing package that allows reading the data recorded by RTI Recording Service (database files) into Matlab and making queries based on the topic names so that all samples obtained under that topic can be decoded into Matlab struct arrays.
Thanks and regards
Goksan
Read data recorded by RTI Recorder into Matlab
Tue, 08/23/2022 - 22:30
#1
Read data recorded by RTI Recorder into Matlab
Hi Goksan,
Well, RTI Recording Service uses Sqlite as the format of the data that's recorded. So, you can use any package that takes tables in Sqlite files and ingests them into Matlab, for example:
https://www.mathworks.com/help/database/ug/sqlite.html
https://www.mathworks.com/help/database/ug/sqlite.fetch.html
Note: RTI Recording Service can be configured to store data in 2 ways. 1) Serialized data in binary form, most efficient and performant or 2) Deserialized to a JSON formated string, takes more CPU, less performant. By default, data is stored in serialized form.
https://community.rti.com/static/documentation/connext-dds/6.1.1/doc/manuals/connext_dds_professional/services/recording_service/recorder/record_configuration.html#sqlite
If you load a Topic table recorded in serialized form into Matlab...that won't be useful unless you are able deserialize the data in Matlab...theoretically possible, but not practical. So, if you want to go this route, you should configure RTI Recording Service to record data in JSON format...and make sure that the Recording Service performance meets your system requirements.
However, if you load JSON data from the Sqlite file with Matlab, with Matlab, you'll have to do an additional conversion of the JSON-formated data to however you want to use it using MatLab's internal routines to deal with JSON formated data...I'm not sure how easy this is to do...you may need to try and experiment.
Another way is to use "rticonverter" to convert the stored data (serialized form is OK) in the recording file into CSV...which Matlab can load directly. Please see this documentation on "rticonverter"
https://community.rti.com/static/documentation/connext-dds/6.1.1/doc/manuals/connext_dds_professional/services/recording_service/converter/converter_index.html
A final way that you can try to do this is to use the Replay Service of RTI Recording Service and "playback" the data of recorded topics and then use the DDS Blockset in Matlab to receive and store the data in Matab. Details on the DDS Blockset can be found here:
https://www.mathworks.com/products/dds.html
Hi Howard,
Thanks a lot for your thorough explanation.
I'm logging the data in the serialized XCDR format indeed, and I think converting the data to CSV format via rticonverter seems like the easiest solution. However, when I just try to run the converter with the default configuration XML, I'm getting the following error. Note that just running the rticonverter without specifying the cfgFile works fine.
/opt/rti/rti_connext_dds-6.0.1/bin/rticonverter -cfgFile /opt/rti/rti_connext_dds-6.0.1/resource/xml/RTI_CONVERTER.xml
RTIXMLUTILSVisitor_visit: error visiting element with name defaultToJson
[/recording_services/defaultToJson|CREATE] ROUTERCfgFileParser_initializeExtensionsExt:!parse at line=2 for tag=dds
[/recording_services/defaultToJson|CREATE] ROUTERCfgFileParser_loadString:!init parser extensions
[/recording_services/defaultToJson|CREATE] ROUTERService_load:!load string
[/recording_services/defaultToJson|CREATE] ROUTERService_initialize:!load service
[/recording_services/defaultToJson|CREATE] ROUTERService_newWithDescription:!init service
[/recording_services/defaultToJson|CREATE] RTI_RoutingService_new_from_description:!init routing service
Stopping RTI Recording Service
RTI Recording Service stopped
main:!Service:!create native router
Hi a few things:
1) the configuration file
/opt/rti/rti_connext_dds-6.0.1/resource/xml/RTI_CONVERTER.xml
is already loaded by rticonverter. If you specify it to be loaded it will be loaded twice and produce an error.
There is a user configuration file that you are supposed to use to create your own rticonverter configurations located in
/home/<user>/rti_workspace/6.0.1/user_config/recording_service/USER_CONVERTER.xml
You would modify this file to configure rticoverter, and load it via "-cfgFile". And you should also specify a "-cfgName" parameter.
2) The CSV plugin for rticonverter is only available in RTI Connext 6.1.x. It's not a part of the 6.0.1 release.