Issue with replaying data from multiple domains

8 posts / 0 new
Last post
Offline
Last seen: 2 years 8 months ago
Joined: 06/21/2021
Posts: 4
Issue with replaying data from multiple domains

Hello all! First time posting, tried to search for answer but couldn't find anything, sorry if this is a duplicate.

I have logs recorded by recording service in SQLite format, they contain data coming from multiple domains. I use RTI Administration Console to view the results.

  1. When I try to replay thse logs, I see that topic writers are created in both domain 0 and 4, although only a single table (with correct domain ID specified after @ sign) exists in the SQLite file. The writer in domain that does not exist in SQLite file will never transmit any data. Is there a way to configure the replay in such way that writers for tables which do not exist will not be created?
  2. When I try to enable  dds.transport.UDPv4.builtin.parent.allow_interfaces to participants to restrict the interfaces on which the data is being sent (see .xml file), domain 4 disappears completely in RTI Administration Console. Is there a way to fix this?
  3. Are  begin_time and  end_time options (see .xml file) specifying absolute or relative time in the logs? If the former, is there a way to make them work as relative?

Thanks for help!

AttachmentSize
File dummy.xml2.78 KB
Howard's picture
Offline
Last seen: 18 hours 23 min ago
Joined: 11/29/2012
Posts: 565

When I try to replay thse logs, I see that topic writers are created in both domain 0 and 4, although only a single table (with correct domain ID specified after @ sign) exists in the SQLite file. The writer in domain that does not exist in SQLite file will never transmit any data. Is there a way to configure the replay in such way that writers for tables which do not exist will not be created?

I'm not sure what you mean.  Did you have DataWriters sending data in both domain 0 and 4?  Did you have a DataWriter that didn't send data?

And, how does it affect your system that Replay Service creates a DataWriter for a Topic for which no data was recorded?  If you know that no data was recorded in a Domain, you don't have to configure Replay Service to start a Participant in that Domain for replay...

When I try to enable  dds.transport.UDPv4.builtin.parent.allow_interfaces to participants to restrict the interfaces on which the data is being sent (see .xml file), domain 4 disappears completely in RTI Administration Console. Is there a way to fix this?

So, your dummy.xml restricted ReplayService only to use the local loopback adapter.  Are you running AdminConsole on the same host as the ReplayService? 

Are  begin_time and  end_time options (see .xml file) specifying absolute or relative time in the logs? If the former, is there a way to make them work as relative?

Sorry, not at this time.

 

Offline
Last seen: 2 years 8 months ago
Joined: 06/21/2021
Posts: 4

Hi Howard, thanks for answer! Here are some clarifications:

I'm not sure what you mean.  Did you have DataWriters sending data in both domain 0 and 4?

In the system on which the data is recorded I have WriterX in Domain 0 and WriterY in Domain 4, both sending data. When I replay this data, in RTI Admin Console WriterX exists in Domains 0 and 4, but only the one in Domain 0 sends any data. Same for WriterY, but with Domain 4. When manually inspecting recorded data (SQLite database), it doesn't look like it contains duplicated records (the only existing tables are WriterX@0 and WriterY@4). So at first glance it doesn't look like there is a problem with Recording service and I see no (obvious) reason for Replay service to create Writers in Domains in which no data has ever been recorded from them. 

And, how does it affect your system that Replay Service creates a DataWriter for a Topic for which no data was recorded? 

  • Performance-wise. When replaying such data I see a lot of "table does not exist" errors in the terminal and it takes long time before replay actually starts. This seems to scale up with every new Domain added. Also, if these writers really exist they are unnecessarily hogging resources which they should not.
  • Readibility-wise. In RTI Admin Console each Domain now contains complete set of all the writers in the system and the user must manually find these which are sending data.

Are you running AdminConsole on the same host as the ReplayService?

Yes, that is the case. Any idea how it can be connected to disappearance of one Domain?

Sorry, not at this time.

Then my suggestion is that documentation should be updated to state this explicitly because although the description of time_range contains info that a timestamp is expected, it has no information about inner tags sec and nanosec, and the example configuration file for Replay service has 200 as sec which doesn' look like a proper timestamp. I guess maybe it works if the data is recorded on system with time set to 1st of Jan 1970 :) 

 

Howard's picture
Offline
Last seen: 18 hours 23 min ago
Joined: 11/29/2012
Posts: 565

So, I was able to recreate your issue.  I check our bugs database and found something similar that was marked as resolved...but think that it wasn't actually addressed.  I'm inquiring with the dev team...but at this time, it looks like a bug.

With respect to the time_range...there is a request already field to make the time range relative to the start of the recording (note the start of the recording may not mean that there's any data at that time...just like a tape recorder can be started before any sound is actually recorded).

It's likely that you'll have to wait until a future release of the Recording Service to see the bug and feature improvement addressed.

I'll also add a documentation bug issue to clarify what the time base for the time range. The most current documentation for Connext 6.1.0 does not have the example of using 200 seconds for the time range anymore...which I also agree that 200 seconds is not a useful example in the current situation.

Thanks for reporting these problems...by the way if you have paid support, you should actually submit these questions through Support.  When support enters issues into our bug database, they attach customer ids...and bugs with customer ids get higher visibility than ones that don't...

Finally, the Admin Console on the same host....yes, if you have fundamentally disabled multicast addressing by enabling only 127.0.0.1, then likely you're running into a "max participants" issue.

Please see this writeup:

https://community.rti.com/kb/why-cant-more-5-domainparticipants-communicate-within-same-machine

Offline
Last seen: 2 years 8 months ago
Joined: 06/21/2021
Posts: 4

Thanks! I will use the Support in future!

Got a question about the write-up you mentioned. I tried the solution like this:

<domain_participant name="Participant0">
  <domain_id>0</domain_id>
  <participant_qos>
    <discovery>
      <initial_peers>
        <element>14@builtin.udpv4://127.0.0.1</element>
        <element>14@builtin.shmem://</element>
       </initial_peers>
    </discovery>
    <property>
      <value>
        <element>
          <name>dds.transport.UDPv4.builtin.parent.allow_interfaces</name>
          <value>127.0.0.1</value>
        </element>
      </value>
    </property>
  </participant_qos>
</domain_participant>

but it didn't work, RTI Admin Console doesn't show Domain 4. Any idea?

As a bonus, I realized that if I copy the code snippet directly from the write-up page, some non-whitespace characters are present between tags and I was getting error like this:

Element 'initial_peers': Character content other than whitespace is not allowed because the content type is 'element-only'.
Element 'initial_peers': Character content other than whitespace is not allowed because the content type is 'element-only'.
Element 'initial_peers': Character content other than whitespace is not allowed because the content type is 'element-only'.
Element 'discovery': Character content other than whitespace is not allowed because the content type is 'element-only'.

Another question, are there any plans for next release of replay service?

 

Howard's picture
Offline
Last seen: 18 hours 23 min ago
Joined: 11/29/2012
Posts: 565

Did you do the same for Domain 4?

Our next release is scheduled sometime later this year or early next year.

Offline
Last seen: 2 years 8 months ago
Joined: 06/21/2021
Posts: 4

Sorry for late reply. Yes, exactly the same for Domain 4. I did an experiment and removed Domain 0 participant completely from the config file, then RTI Admin Console was not showing any traffic at all. As soon as I removed dds.transport.UDPv4.builtin.parent.allow_interfaces parameter from Domain 4 participant, I was able to see traffic on Domain 4 in the Admin Console.

Howard's picture
Offline
Last seen: 18 hours 23 min ago
Joined: 11/29/2012
Posts: 565

Hmm, I assume you're restricting the interface to 127.0.0.1?

I don't know off hand why that would stop it from talking to Admin Console on the local host.  But you can try to restrict Admin Console to also use 127.0.0.1

If you have wireshark, you can use wireshark to sniff the packets on 127.0.0.1 and see if there are any packets back and forth from your app to Admin Console.  If you can't interpret the trace yourself, then I would suggest going to RTI's Support team through your customer portal...assuming that your project has an active maintenance and support agreement.

Finally, why are you trying to restrict your participants to use 127.0.0.1?  If you don't want participants to communicate off-board, then I suggest turning off UDP altogether and only use the shared memory transport.  The Knowledge Base on community.rti.com have posting on how to do that.