Problem with recording of large data packets (~8MB) from DDS Micro

Hi Niclas,

I've let Maxx Becker, the RTI FAE responsible for helping Zukunft Mobility at this time to know about your question.  I'm sure he'll get in touch soon.

In the meantime, we'll probably need to clarify what you're actually doing.

For your 0.3 Hz rate, are you sending from an appllication using Connext Micro to another application using Connext Micro?  Or are you using Connext Micro to send data to RTI Recording Service?

How have you configure your RTI Recording Service to record the data (if you are using the Recording Service)?  If using a Connext Micro application to subscribe, how have you configured the QOS for the DataReader?

Will your real system be running the sending and receiving applications on the same host and thus can take advantage of using Shared Memory?  Or is that only available for this testing and you will be using an Ethernet or other type of network for the actual scenario?

If can use shared memory, what is the configuration of your shared memory and how did you set it up (can copy/paste the code), including how you set up the Transports to use shared memory.

Finally, please use whatever means that you can to localize the place in your code, the specific Connext DDS Micro API that your code called, that produces the warning message that you reported:

From the self written pure node in the console: ModuleID=7 Errcode=500 which is according to the docu "Could not allocate a resource of the specified kind"

This issue may be related to the poor performance that you are experiencing.

Hi Niclas,

Thank you for the information. A few more questions...

For the data type for the image information, how large is the array that holds the pixel information? Is it unbounded? How much data do you have in each sample when you are issuing the write() call? Are you attempting 8MB data samples at 25Hz? If you can provide the IDL, or any code snippets, that would help.

From what I see, it seems that the Data Writer may be exceeding its resouce limits as defined, blocks, and eventually times out (throwing the DDS_RETCODE_TIMEOUT). This could be because the Data Reader (the Recording Service) is not acknowledging samples, and therefore the Data Writer queues are filling up. It looks like the shared memory resources are configured as defaults; the receive_buffer_size ends up being 1MB. We may have to look at how data is being handled in the receive queues. 

Feel free to follow up with me directly via email through the ongoing channel I have with the rest of the team. 

Thanks,

Maxx

Hi Niclas,

So, I'm sure that Maxx will be able to help you to get this working...but let me give you guys some pointers since I have a bit more experience than Maxx.

1) When using RELIABLE reliability, you can't just set RELIABLE and KEEP_ALL history.  Unfortunately, the parameters for the reliable protocol used by DDS, which is turned on when you set RELIABLE, are not well configured for any specific use case. 

So, for best practice, we suggest that for any DataWriter or DataReader that is supposed to send or receive data reliably, you should use a QOS Profile derived from the Builtin QOS Profiles, and specifically for strict, TCP, lossless, reliability, you should use the "Generic.StrictReliable.LargeData" (since the data is 8 MB) profile from the "BuiltinQosLib" qos library.  This for applications using the full Connext DDS, including the Recording Service.

So in your Recording Service XML configuration file, you should use this (which will set the QOS to be RELIABLE and KEEP_ALL and thus don't need to set those anymore)

<datareader_qos base_name="BuiltinQosLib::Generic.StrictReliable.LargeData"></datareader_qos>

Also, you should configure the DomainParticipant created by Recording Service to better handle large data by using the following QOS setting in the Recording Service XML file,

        <domain_participant name="Participant0">

         <domain_id>0</domain_id>
         <participant_qos base_name="BuiltinQosLib::Generic.Participant.LargeData"></participant_qos>

     </domain_participant>

Unfortunately, Connext Micro doesn't support the concept of QOS profiles and thus you will have to make modifications to code.

For DataWriters, I would recommend:

      int NUM_SAMPLES = 2; dw_qos.resource_limits.max_samples = NUM_SAMPLES;

    dw_qos.resource_limits.max_samples_per_instance = NUM_SAMPLES;

    dw_qos.resource_limits.max_instances = 1; // 10 ms

    dw_qos.protocol.rtps_reliable_writer.heartbeat_period.sec = 0;

    dw_qos.protocol.rtps_reliable_writer.heartbeat_period.nanosec = 10000000;

    dw_qos.protocol.rtps_reliable_writer.heartbeats_per_max_samples = NUM_SAMPLES;

where NUM_SAMPLES should be set to the number of data samples that you want DDS to be able to hold while DDS is using the reliable protocol to send the data...if you're sending very fast, then NUM_SAMPLES should be larger (typically for small data sent continuously, we recommend setting to 40).

However, for large data, if you set NUM_SAMPLES to 1, then after you sent a data sample, the next call to DataWriter::write() will block (assuming RELIABLE and KEEP_ALL) until DDS has successfully sent and confirmed the receipt of the data.  Setting NUM_SAMPLES to 2 will allow you to write another sample while DDS is still sending the first, i.e., if you are sending the data twice in a row without any delay in between.

NOTE, the larger the value of NUM_SAMPLES, the more memory will be allocated by DDS to hold those samples.

Additionally, if only sending through shared memory (and not over UDP on a network), there are several optimizations that you can make such as changing the MTU (message_size_max) of the shared memory transport to hold the largest data that you can send without having to fragment the data.  Right now, your configuration sets the message_size_max to 64K Bytes, and only allocates 1 MB of total shared memory space.  This basically forces DDS to break up your 8 MB data into 64KB fragments to send over shared memory which can only hold 1 MB (actually each 64 KB packet only contains up to 65507 bytes of user data, the other bytes will be taken up by a header).  Fundamentally, if the 1 MB shared memory buffer isn't serviced fast enough by the receiving application, the buffer will become full, and shared memory packets will be dropped...but will be resent because of the reliability protocol.  But any dropped packets means delays and inefficient use of the CPU.

So to fix, you should set your shared memory segment to have a much larger message_size_max (over 8 MB if your largest data is 8 MB) and larger buffer (some N x 8 MB, so that multiple data can be buffered without loss in shared memory).

This needs to be configured in Recording Service if Recording Service is receiving the data (think of the Shared Memory as a mailbox owned by the receiving application).  And if Micro is receiving the data, then it should be configured similarly.

For Recording service it would be like

<domain_participant name="Participant0">
            <domain_id>0</domain_id>
        <participant_qos base_name="BuiltinQosLib::Generic.Participant.LargeData">

                <transport_builtin>
                    <mask> SHMEM </mask>
                    <shmem>
                        <message_size_max>  10485760 </message_size_max>
                        <receive_buffer_size> 20971520 </receive_buffer_size>
                    </shmem>
                </transport_builtin>

</participant_qos>

For Micro it would be:

// Set sizing for SHMEM transport

shmem_property.message_size_max = 10*1024*1024;

shmem_property.receive_buffer_size = 2*10*1024*1024; // register SHMEM transport

if (!registry->register_component(NETIO_DEFAULT_SHMEM_NAME, NETIO_SHMEMInterfaceFactory_get_interface(),

                                  &shmem_property._parent._parent, nullptr))

The above won't help at all if you are also sending the data over a network, which has a MTU of 64 KB.  When sending the data to multiple participants, DDS will use the smallest MTU to fragment the data, so if UDP is 64 KB and shared memory is 10 MB, DDS will fragment at 64 KB.

Finally, this is advanced stuff and usually we recommend getting some training first before diving into this, if you are only sending data through shared memory, you can consider using the ZeroCopy Shared Memory option..., see this documentation: https://community.rti.com/static/documentation/connext-micro/3.0.3/doc/html/usersmanual/zerocopy.html

Hi Niclas,

Did you set both

<message_size_max>  10485760 </message_size_max>

in the Recording Service XML configuration, as well as 

shmem_property.message_size_max = 10*1024*1024;

In the Micro source configuration? The error you are seeing points to a shared memory configuration mismatch. So they should both have the same value, 10485760. 

Maxx

Hi Niclas,

Yes, the shared memory configuration should match in the XML (for the Recording Service) and in the code (of the Publisher, using Micro). I am not sure what you mean by the receiver in your last reply, if it is not the recording service... is this the target that subscribes to the replay of the recorded data?

In any case, glad to know it is working! 

Maxx