Blocking calls.

Hi Rafael,

I think that you're on a project that has a support contract with RTI.  Your question may be best asked to and answered by RTI's professional support team.

I believe that Johan Schutzer and Kristhoffer Hultenius are supported developers and can submit your issue to RTI support for resolution.  However, in your submission to RTI's support team, it would be great if you could answer the following questions to clarify your situation:

1) Why did you want to change your application from using synchronous DataWriters to asynchronous DataWriters with flow controller?  What behavior are you trying to avoid to trying to achieve?

2) How many DataWriters are involved?  How big is the data for each DataWriter?  What is the period at which data is sent?  And are you setting all of the DataWriters to be ASYNCHRONOUS Publish Mode with the same flow controller?

3) What time are you measuring with respect to your statement "measuring on the caller thread, using async with the flow controller is actually slower; it's blocking the writer thread for more time"? 

I guess your application uses a single thread to call DataWriter::write() on multiple topics and then at the end of the loop

Where is your writer thread being blocked?

trigger_flow() should not block...it's only giving a semaphore under-the-hood.

4) "From my understanding something configured asynchronously shouldn't block the caller."  No, whether or not the DataWriter::write() call blocks or not is not related to whether or not the Publish Mode is synchronous or asynchronous.

If the publish mode is synchronous, then the write() call wil pass the data to the network layer before it returns.  Asynchronous write() calls only serializes and copies the data to the DataWriter queue.  So theoretically, a synchronous write() will take more time to return (executes more code) than an asynchrnous write().

HOWEVER, whether a write() call will block is independent of the publish mode.  A DataWrite::write() call will block if there is no space left in the DataWriter queue to store a copy of the data.  This will only happen for RELIABLE Reliability DataWriters that have one or more RELIABLE DataReaders, i.e., there is a reliable connection between the DataWriter and one or more DataReaders. 

Sent data is stored in the Writer's history queue (aka writer cache) until it's been ACK'ed by all reliable DataReaders.  If one or more DataReaders are not acknowledging send data fast enough, then the queue can eventually be filled.  How fast depends on

a) how fast data is being sent

b) how big the Send Windows is set for the Writer queue

c) configuration of the parameters that control the reliability protocol

which begs the question, are you using the Generic.StrictReliable builtin QOS profile to create the QOS used for your DataWriter/DataReaders that need to be reliable?  If not, I would certainly start there.  The out-of-the-box configuration of the reliability protocol is not designed for any sort of high performance situation.

Again, you should work with RTI's support team for them to understand what you're doing and offer suggestions on how to achieve your desired behavior.

Are the application sending all of the data from a single thread?  Does your CPU/OS support multiple cores?

If not, then unfortunately, the only thing that you can really try is to optimize your code to be more CPU efficient.  If all of the data needs to be published at 40 Hz, then there is a certain level of processing, aka, CPU usage, needed by DDS to do that work...whether done as the application calls write() synchronously or at the bottom of the send loop all at once with the asynchronous writer thread...I don't there is there significantly less work one way versus the other way.

In this case, I would use a software profiling tool to see where is most of the CPU being spent.  And after knowing that, try to work on optimizing those paths...

On the other hand, if the application is running on a multi-core system, then you can try to divide the work across different threads.  Of course, at some point there is a bottleneck, i.e., the network stack.  But you can certainly try to send different data using different threads.

It's possible that when sending "large" data...not sure what you consider to be large, that DDS will have to do extra work to fragment that data to send it.  In that case, I would certainly try to send that data in a different thread.

Now having said that, you will have to create your DataWriters differently if you want the calls to different DataWriters() to be multi-threaded.  By design, the DataWriters created by the same DDSPublisher object all shared a mutex.  So write() calls to the DataWriters of the same DDSPublisher will be executed in a single thread manner.

To have the write() calls to different DataWriters be multi-threaded, i.e., simultaneously executable by threads on different cores, you will have to create the DataWriters with different instances of the DDSPublisher.   All of the DataWriters that will be sent by the same thread can use the same DDSPublisher.

3) Measuring on a the (by instrumenting) that would need to do those calls at 40Hz.

I'm not sure what you mean by this sentence above.  My original question was how were you measuring "time"?  How do you know which API calls are taking up more time than others?  How are you determining that the write() call is blocking?

What you described is a syncrhonous writer, as it is always blocking; it just happens that on async mode with free queue space it returns sooner than on other configuration modes. That was surprising. I'd expect an asynchronous writer to never block, specially not for that long, or if it can do so to have a per-call timeout parameter. Given the API (no send message events/callbacks/waitsets) I assumed that writers would throw on full internal queues.

While your definition of what is "synchronous" is different, ours comes from the DDS Standard.  A write() call only has the potential for blocking when the connection is set to be RELIABLE via the QOS.

In the same Reliability QosPolicy, you can set the max_blocking_time parameter to 0...so that the write() will return if the internal queues are full.  DDSDataWriter::write() does have a return value in C and traditional C++ for which you can detect this condition, DDS_RETCODE_TIMEOUT, or throws an exception for the same situation with the Modern C++ API.

Generally, you don't want write() to return an error just because the queue is full...because it's not unusual for this case and is used as a mechanism to throttle the writer when the writer is sending too fast.  This is the default behavior of TCP socket for example.  When sending a burst of data, a full queue will prevent a sender to flood the network causing packet losses which necessitates repairing lost data at the expense of additional CPU and network bandwidth.  So, slowing down a writer, usually for under a millisecond, will allow the system to optimally use the network.

Probably enabling FlatData would help.

Using FlatData will save a copy of the data in the write() path, however, the application code that access the data structure will have to change.  You can't just access elements of a structure directly...instead, you'll have to use assessor methods.  And the time savings for small data is minimal.  Even for "large" data, it's probably not significant until the data is in the MB range.

I suggest doing some performance tests before refactoring your application to use FlatData just to understand how much savings you can expect.

Good to know about max_blocking_time. This seems to give us exactly the desired behavior. Every cycle/iteration we know how much time we have left, is it possible to change this per-call?

Sorry, the value of max_blocking_time and the rest of the Reliabilty QOS Policy is fixed once the entity (DataWriter/DataReader) is created.

Which are the parameters to set the bytesize of the library internal DDS async write/read queues? I assume that we have to tweak those both on Readers and Writers after going asynchronous.

There is no internal queue/buffer being used by the internal thread used to send data for DataWriters that are configured for ASYNCHRONOUS publish mode.  The thread takes the data that is stored with each DataWriter.

Each DataWriter has a send queue (aka writer cache) that is configured physically through the DataWriter's ResourceLimits QOS Policy, although for reliable connections, the configuration of the History QOS Policy as well as the configuration of the send_window for the reliability protocol will affect how the allocated queue is used.

In addition, if you configure the DataWriter to use a FlowController, the internal publish thread will use the FlowController to determine how much data it will send for each DataWriter every time it wakes up. 

I assume that we have to tweak those both on Readers and Writers after going asynchronous.

I'm not sure what you mean by this.  The sizes of the queues for the Writers/Readers (aka send queue/receive queue aka writer/reader caches) are certainly something that you may want to configure...but the fact that a Writer is configured to use the ASYNCHRONOUS publish mode is usually not a factor in queue sizing.

Yes, the sizes of queues are specified in the ResourceLimits QOS Policy in terms of data samples, not data bytes.

Do you prefer this to go through support? Things seem to be generic enough that if kept public they might be solve doubts for other people for those rare specimens using the search function...

Generally, if your project has a support contract with RTI's support team, it would be to your advantage to use that resource to answer all of your technical questions about RTI products.  The team is dedicated and global...they are able to respond to your questions in your own timezone.  Also, your questions are recorded in your account's record so that you have a record of all issues as well as their resolution that would be available to you and your colleagues for review.  Your project is paying for the service, you should get something useful from it.

The community forum is completely self-help/self-serve.  There is no guarantee of a timely response to a posting or even if a question will be answered.  While RTI engineers do monitor and often provide some answers/advice, we're usually doing this in our "spare" time.  There's no-one at RTI who has the responsibility to make sure that all posting to the forum are answered.  We're hoping that other members of the RTI community are able to help each other via the forum as many of our users don't have a support contract (researchers/university program members/etc).