File metadata and Recording Service

4 posts / 0 new
Last post
Offline
Last seen: 1 year 5 months ago
Joined: 06/18/2021
Posts: 2
File metadata and Recording Service

Hi.

We've been prototyping with the RTI Recording Service for a bit with two different plugins and it seems to work well overall but it does seem like we could need some help.

We are prototyping a system with several RTI Recording Service instances. We need the ability to run several RTI Recording Service instances at once to provision for eventual network/disk capacity limitations. They have to be started and stopped from a central control client. We need to add a unique identifier to each recorded file from all recorders done during each continuous recording event. This is to associate each file with it's unique continuous recording event, essentially so we can see which files belong together and together make up the whole data set for that recording event.

Do you have any suggestions on how such id-related metadata could be included in each recorded file using the RTI "Remote Administration Platform"?

If anyone has information related to this that would be very helpful, thank you!

Howard's picture
Offline
Last seen: 6 hours 53 min ago
Joined: 11/29/2012
Posts: 447

You can create a special topic that has the identifier and then in your remote control application, have it send the unique id, which the recording services will store in their files.

There's also an tagging feature, where you can add a symbolic tag to a timestamp.

https://community.rti.com/static/documentation/connext-dds/6.1.0/doc/manuals/connext_dds_professional/services/recording_service/recorder/record_administration.html#record-storage-tag-timestamp

That might work for you.

Offline
Last seen: 1 year 5 months ago
Joined: 06/18/2021
Posts: 2

Hi Howard, thank you for your reply! Much appreciated!

Special topic

As to the first suggestion (using a special topic) this is something we've discussed internally prior to posing the question and although it might work the team brought up a couple of concerns:

  • It's an additional special topic which isn't a part of the RTI Recording Service interface so it has to be dealt with accordingly.
  • The id is on a separate topic and therefore not directly associated with the commands given to the recorder instances, can one then rely on a topic update by the control client being propagated before the control command? Essentially: if the control client updates the special topic with a new session id and then sends a start command can one be sure that the recorder plugin gets the new session id prior to the start of the recording? Just want to be sure there aren't any synchronization/race issues with this approach.

tag_timestamp

The second suggestion looked interesting but it seems to have an issue: the documentation clearly states that tag_timestamp is "Only valid for builtin SQLite plugin." which seems problematic for us since we've already developed another plugin and we'd like this to work for both plugins and other ones too.

Is it possible and appropriate to extend custom plugins such as ours to support tag_timestamp? Even so it might not be the ideal solution for us since it would probably mean that all plugins would have to be adapted to support this rather than work out of the box.

Brainstorming/speculating about other approaches

Or maybe there are some existing unique id(s) at some level in the RTI Recording Service interface that can be reused for this purpose? Or maybe it's possible to for instance use TOPIC_DATA / GROUP_DATA / USER_DATA on existing topics/entities to propagate this metadata? Or maybe there's another way entirely? Or perhaps the special topic approach is still the way to go.

Howard's picture
Offline
Last seen: 6 hours 53 min ago
Joined: 11/29/2012
Posts: 447

The id is on a separate topic and therefore not directly associated with the commands given to the recorder instances, can one then rely on a topic update by the control client being propagated before the control command? Essentially: if the control client updates the special topic with a new session id and then sends a start command can one be sure that the recorder plugin gets the new session id prior to the start of the recording? Just want to be sure there aren't any synchronization/race issues with this approach.

By configuring the SessionID topic with QoS Durability.kind = TRANSIENT_LOCAL (both in your recorder admin client as well as the Recording Service) (as well as Reliability.kind = RELIABLE), then, there is no synchronization/race conditon.  Your client can send a value for the SessionID, and even if the Recording Service has not fully initialized the DataReader for the SessionID topic yet.  That's what the Durability QoS is designed to do.

The second suggestion looked interesting but it seems to have an issue: the documentation clearly states that tag_timestamp is "Only valid for builtin SQLite plugin." which seems problematic for us since we've already developed another plugin and we'd like this to work for both plugins and other ones too.

Sorry, I don't know anything about this...you should ask support@rti.com if you have a M&S contract with RTI.  Otherwise, you can look into the plugin APIs that we offer and see if a tag is being passed in anywhere.  I don't know why the SQLite plugin would be special.  I'm pretty sure it's just a plugin using the Recording Service plugin interface...no different than what you're using for your own plugins.

Or maybe there are some existing unique id(s) at some level in the RTI Recording Service interface that can be reused for this purpose? Or maybe it's possible to for instance use TOPIC_DATA / GROUP_DATA / USER_DATA on existing topics/entities to propagate this metadata? Or maybe there's another way entirely? Or perhaps the special topic approach is still the way to go.

Yes, that could work.. the Discovery information is being recorded by default.  The TOPIC_DATA/GROUP_DATA/USER_DATA that you can set in applications are definitely propagated by Discovery.