Tuning Queue Sizes and Other Resource Limits
Set the HISTORY QosPolicy appropriately to accommodate however many DDS samples should be saved in the DataWriter’s send queue or the DataReader’s receive queue. The defaults may suit your needs; if so, you do not have to modify this QosPolicy.
Set the DDS_RtpsReliableWriterProtocol_t in the DATA_WRITER_PROTOCOL QosPolicy (DDS Extension) appropriately to accommodate the number of unacknowledged DDS samples that can be in-flight at a time from a DataWriter.
For more information, see the following sections:
- Understanding the Send Queue and Setting its Size
- Understanding the Receive Queue and Setting Its Size
Note: The HistoryQosPolicy’s depth must be less than or equal to the ResourceLimitsQosPolicy’s max_samples_per_instance; max_samples_per_instance must be less than or equal to the ResourceLimitsQosPolicy’s max_samples (see RESOURCE_LIMITS QosPolicy), and max_samples_per_remote_writer (see DATA_READER_RESOURCE_LIMITS QosPolicy (DDS Extension)) must be less than or equal to max_samples.
- depth <= max_samples_per_instance <= max_samples
- max_samples_per_remote_writer <= max_samples
Examples:
DataWriter
writer_qos.resource_limits.initial_instances = 10; writer_qos.resource_limits.initial_samples = 200; writer_qos.resource_limits.max_instances = 100; writer_qos.resource_limits.max_samples = 2000; writer_qos.resource_limits.max_samples_per_instance = 20; writer_qos.history.depth = 20;
DataReader
reader_qos.resource_limits.initial_instances = 10; reader_qos.resource_limits.initial_samples = 200; reader_qos.resource_limits.max_instances = 100; reader_qos.resource_limits.max_samples = 2000; reader_qos.resource_limits.max_samples_per_instance = 20; reader_qos.history.depth = 20; reader_qos.reader_resource_limits.max_samples_per_remote_writer = 20;
Understanding the Send Queue and Setting its Size
A DataWriter’s send queue is used to store each DDS sample it writes. A DDS sample will be removed from the send queue after it has been acknowledged (through an ACKNACK) by all the reliable DataReaders. A DataReader can request that the DataWriter resend a missing DDS sample (through an ACKNACK). If that DDS sample is still available in the send queue, it will be resent. To elicit timely ACKNACKs, the DataWriter will regularly send heartbeats to its reliable DataReaders.
A DataWriter’s send queue size is determined by its RESOURCE_LIMITS QosPolicy, specifically the max_samples field. The appropriate value depends on application parameters such as how fast the publication calls write().
A DataWriter has a "send window" that is the maximum number of unacknowledged DDS samples allowed in the send queue at a time. The send window enables configuration of the number of DDS samples queued for reliability to be done independently from the number of DDS samples queued for history. This is of great benefit when the size of the history queue is much different than the size of the reliability queue. For example, you may want to resend a large history to late-joining DataReaders, so the send queue size is large. However, you do not want performance to suffer due to a large send queue; this can happen when the send rate is greater than the read rate, and the DataWriter has to resend many DDS samples from its large historical send queue. If the send queue size was both the historical and reliability queue size, then both these goals could not be met. Now, with the send window, having a large history with good live reliability performance is possible.
The send window is determined by the DataWriterProtocolQosPolicy, specifically the fields min_send_window_size and max_send_window_size within the rtps_reliable_writer field of type DDS_RtpsReliableWriterProtocol_t. Other fields control a dynamic send window, where the send window size changes in response to network congestion to maximize the effective send rate. Like for max_samples, the appropriate values depend on application parameters.
Strict reliability:If a DataWriter does not receive ACKNACKs from one or more reliable DataReaders, it is possible for the reliability send queue—either its finite send window, or max_samples if its send window is infinite—to fill up. If you want to achieve strict reliability, the kind field in the HISTORY QosPolicy for both the DataReader and DataWriter must be set to KEEP_ALL, positive acknowledgments must be enabled for both the DataReader and DataWriter, and your publishing application should wait until space is available in the reliability queue before writing any more DDS samples. Connext DDS provides two mechanisms to do this:
- Allow the write() operation to block until there is space in the reliability queue again to store the DDS sample. The maximum time this call blocks is determined by the max_blocking_time field in the RELIABILITY QosPolicy (also discussed in Blocking until the Send Queue Has Space Available).
- Use the DataWriter’s Listener to be notified when the reliability queue fills up or empties again.
When the HISTORY QosPolicy on the DataWriter is set to KEEP_LAST, strict reliability is not guaranteed. When there are depth number of DDS samples in the queue (set in the HISTORY QosPolicy, see Controlling Queue Depth with the History QosPolicy) the oldest DDS sample will be dropped from the queue when a new DDS sample is written. Note that in such a reliable mode, when the send window is larger than max_samples, the DataWriter will never block, but strict reliability is no longer guaranteed. If there is a request for the purged DDS sample from any DataReaders, the DataWriter will send a heartbeat that no longer contains the sequence number of the dropped DDS sample (it will not be able to send the DDS sample).
Alternatively, a DataWriter with KEEP_LAST may block on write() when its send window is smaller than its send queue. The DataWriter will block when its send window is full. Only after the blocking time has elapsed, the DataWriter will purge a DDS sample, and then strict reliability is no longer guaranteed.
The send queue size is set in the max_samples field of the RESOURCE_LIMITS QosPolicy. The appropriate size for the send queue depends on application parameters (such as the send rate), channel parameters (such as end-to-end delay and probability of packet loss), and quality of service requirements (such as maximum acceptable probability of DDS sample loss).
The DataReader’s receive queue size should generally be larger than the DataWriter’s send queue size. Receive queue size is discussed in Understanding the Receive Queue and Setting Its Size.
A good rule of thumb, based on a simple model that assumes individual packet drops are not correlated and time-independent, is that the size of the reliability send queue, N, is as shown in Figure: Calculating Minimum Send Queue Size for a Desired Level of Reliability .
Figure: Calculating Minimum Send Queue Size for a Desired Level of Reliability
N = 2RT(log(1-Q))/log(p))
Simple formula for determining the minimum size of the send queue required for strict reliability
In the above equation, R is the rate of sending DDS samples, T is the round-trip transmission time, p is the probability of a packet loss in a round trip, and Q is the required probability that a DDS sample is eventually successfully delivered. Of course, network-transport dropouts must also be taken into account and may influence or dominate this calculation.
Required Size of the Send Queue for Different Network Parameters gives the required size of the send queue for several common scenarios.
|
p2"p" is the probability that any single packet gets lost in the network. |
R4"R" is the rate at which the publisher is sending updates. |
|||
|
99% |
1% |
0.0016The typical round-trip delay for a dedicated 100 Mbit/second ethernet is about 0.001 seconds. sec |
100 Hz |
1 |
|
99% |
1% |
0.001 sec |
2000 Hz |
2 |
|
99% |
5% |
0.001 sec |
100 Hz |
1 |
|
99% |
5% |
0.001 sec |
2000 Hz |
4 |
|
99.99% |
1% |
0.001 sec |
100 Hz |
1 |
|
99.99% |
1% |
0.001 sec |
2000 Hz |
6 |
|
99.99% |
5% |
0.001 sec |
100 Hz |
1 |
|
99.99% |
5% |
0.001 sec |
2000 Hz |
8 |
Note: Packet loss on a network frequently happens in bursts, and the packet loss events are correlated. This means that the probability of a packet being lost is much higher if the previous packet was lost because it indicates a congested network or busy receiver. For this situation, it may be better to use a queue size that can accommodate the longest period of network congestion, as illustrated in Figure: Calculating Minimum Send Queue Size for Networks with Dropouts.
Figure: Calculating Minimum Send Queue Size for Networks with Dropouts
N = RD (Q)
Send queue size as a function of send rate "R" and maximum dropout time D
In the above equation R is the rate of sending DDS samples, D(Q) is a time such that Q percent of the dropouts are of equal or lesser length, and Q is the required probability that a DDS sample is eventually successfully delivered. The problem with the above formula is that it is hard to determine the value of D(Q) for different values of Q.
For example, if we want to ensure that 99.9% of the DDS samples are eventually delivered successfully, and we know that the 99.9% of the network dropouts are shorter than 0.1 seconds, then we would use N = 0.1*R. So for a rate of 100Hz, we would use a send queue of N = 10; for a rate of 2000Hz, we would use N = 200.
Understanding the Receive Queue and Setting Its Size
DDS samples are stored in the DataReader’s receive queue, which is accessible to the user’s application.
A DDS sample is removed from the receive queue after it has been accessed by take(), as described in Accessing DDS Data Samples with Read or Take. Note that read() does not remove DDS samples from the queue.
A DataReader's receive queue size is limited by its RESOURCE_LIMITS QosPolicy, specifically the max_samples field. The storage of out-of-order DDS samples for each DataWriter is also allocated from the DataReader’s receive queue; this DDS sample resource is shared among all reliable DataWriters. That is, max_samples includes both ordered and out-of-order DDS samples.
A DataReader can maintain reliable communications with multiple DataWriters (e.g., in the case of the OWNERSHIP_STRENGTH QosPolicy setting of SHARED). The maximum number of out-of-order DDS samples from any one DataWriter that can occupy in the receive queue is set in the max_samples_per_remote_writer field of the DATA_READER_RESOURCE_LIMITS QosPolicy (DDS Extension); this value can be used to prevent a single DataWriter from using all the space in the receive queue. max_samples_per_remote_writer must be set to be <= max_samples.
The DataReader will cache DDS samples that arrive out of order while waiting for missing DDS samples to be resent. (Up to 256 DDS samples can be resent; this limitation is imposed by the wire protocol.) If there is no room, the DataReader has to reject out-of-order DDS samples and request them again later after the missing DDS samples have arrived.
The appropriate size of the receive queue depends on application parameters, such as the DataWriter’s sending rate and the probability of a dropped DDS sample. However, the receive queue size should generally be larger than the send queue size. Send queue size is discussed in Understanding the Send Queue and Setting its Size.
Figure: Effect of Receive-Queue Size on Performance: Large Queue Size and Figure: Effect of Receive Queue Size on Performance: Small Queue Size compare two hypothetical DataReaders, both interacting with the same DataWriter. The queue on the left represents an ordering cache, allocated from receive queue—DDS samples are held here if they arrive out of order. The DataReader in Figure: Effect of Receive-Queue Size on Performance: Large Queue Size has a sufficiently large receive queue (max_samples) for the given send rate of the DataWriter and other operational parameters. In both cases, we assume that all DDS samples are taken from the DataReader in the Listener callback. (See Accessing DDS Data Samples with Read or Take for information on take() and related operations.)
In Figure: Effect of Receive Queue Size on Performance: Small Queue Size , max_samples is too small to cache out-of-order DDS samples for the same operational parameters. In both cases, the DataReaders eventually receive all the DDS samples in order. However, the DataReader with the larger max_samples will get the DDS samples earlier and with fewer transactions. In particular, DDS sample “4” is never resent for the DataReader with the larger queue size.
Figure: Effect of Receive-Queue Size on Performance: Large Queue Size
Figure: Effect of Receive Queue Size on Performance: Small Queue Size
© 2018 RTI