Hi,
Currently using 6.1.1 LTS.
My setup is that I have multiple domain participants with a few large topic structures, due to nested sequences. All of them are bounded.
So I'm facing this issue when I attempt to run the setup on a virtual machine(VM) running Windows 10, there reaches a point where I am unable to initialise all the participants.
The error messages are "no space on heap for buffer" and "Failed to create writer buffer pool", which are quite self-explanatory.
The interesting thing to me is that this setup runs totally fine on my laptop with way less RAM than the VM, but that seems more of a VM setup issue than an RTI DDS issue.
Of course, everything works fine if I modify the IDL and change all the sequences to unbounded, but that comes with other issues which are out of the scope of my question.
To help investigate this, I'm hoping to find a way to monitor the heap usage of all my domain participants, both at startup, and when running for extended periods of time. Are there any logging tools already available in the RTI suite that would help with this? Or is there any helper function in the RTI libraries that can take memory usage snapshots of a participant?
Thank you.
Monitoring heap usage
Tue, 07/12/2022 - 20:12
#1
Monitoring heap usage
For DataWriters/DataReaders, Connext will prealloc all of the data buffers needed for the send and receive queues of each entity respectively.
Of course the size of the buffers depend on the size of the datatype. For unbounded strings/sequences, the size of an element can be quite big.
So, typically, we suggest to use dynamic memory allocation for DataWriters/DataReaders of large data types...or ones with variable sized data that has a large maximum size.
For variable sized data, typically, most of the time, the data isn't the maximum size possible. The average size of the data can be much less than the maximum size.
In which case, you can set a QoS for the DataWriter and/or DataReader to preallocate buffers of a certain size (small) and dynamically allocate/free buffers for data above the specified size.
For the DataWriterQos
<property>
<value>
<element>
<name>
dds.data_writer.history.memory_manager.fast_pool.pool_buffer_max_size
</name>
<value>4096</value>
</element>
</value>
</property>
for DataReaderQos
<property>
<value>
<element>
<name>
dds.data_reader.history.memory_manager.fast_pool.pool_buffer_max_size
</name>
<value>4096</value>
</element>
</value>
</property>
In the above examples, DDS will preallocate 4K buffers, but buffers for an data that is above 4K in size will be allocated dynamically (and freed when not needed).
For direct heap monitoring, if you are on a project that has an RTI support contract, I suggest that your team contacts RTI support who may be able to help you figure out how much and where memory is being allocated by DDS in your application.
If you are using one of the latest versions of Connext there is a way you can monitor heap usage and take snapshots
See: https://community.rti.com/static/documentation/connext-dds/6.1.1/doc/manuals/connext_dds_professional/users_manual/index.htm#users_manual/MonitoringNativeHeapMemory.htm
As Howard mentions, you may want to work with our support team and/or local field application engineer who can help you analyze the information this API provides.
I understand Howard, I will see if I can better manage memory usage with that suggested QOS. Thank you.
Also, thanks r, for the links to relevant documentation, I will give those a look.