As shmem transport properties are shown in https://community.rti.com/static/documentation/connext-dds/6.0.1/doc/api/connext_dds/api_cpp/structNDDS__Transport__Shmem__Property__t.html, e.g.,
dds.transport.shmem.builtin.receive_buffer_size
dds.transport.shmem.builtin.received_message_count_max
How to understand the relation between shmem properties & process memory usage (e.g. Linux/Ubuntu 'top': RES)?
For examples, we have 2 topics: TopicX & TopicY, and 3 processes on Linux/Ubuntu: TaskA ->TopicX -> TaskB ->TopicY -> TaskC, where all of them are using SHMEM, and '->' means a publisher or a subscriber connecting on a topic.
If we increase dds.transport.shmem.builtin.receive_buffer_size/received_message_count_max in TaskB, we might expect its memory usage increasing (shown in Linux command 'top': RES column).
What should be expected on TaskA & TaskC memory usage? Memory usage increasing or not?
If, at the same time, we have 2 more topics: TopicZ & TopicW, TaskA <- TopicZ <- TaskB <- TopicW <- TaskC.
What should be expected on TaskA & TaskC memory usage? Memory usage increasing or not?
In other words, if the properties increase memory usage, will it affect the current process ONLY, or all processes in a system/SoC, or depending on topic connections?
Thanks in advance!
The configuration of the shared memory transport, including the size of the allocated shared memory segment, only affects the participant for which the configuration changes are applied.
So, in your example, if you change the shared memory receive_buffer_size for participant in Task B, it will only affect the memory usage for Task B. No other Tasks will be affected unless they somehow are also using the same QoS configuration file and thus change their shared memory configuration as well.
Note, however, the memory used by DDS is also affected by the number of DataWriters and DataReaders created by the participant. This is independent of shared memory. And how much memory that is used by a DataWriter or DataReader depends on the configuration of the ResourceLimits QoS policy set for those entities as well as the maximum size of the datatype used by the Topics of those entities.
e.g., if the ResourceLimits.max_samples = 40, and the max size of the data type is 1 MB, then creating a DW or DR with that setting and datatype will allocate at least 40 MB just for the DW or DR data caches (i.e., send/receive queues).
NOTE: The reporting of memory usage of a process in Linux is pretty complicated and there are multiple ways to try to get the information. You have to understand what's being reported. I suggest looking on the web for blog/articles like:
https://www.linuxfoundation.org/blog/blog/classic-sysadmin-linux-101-5-commands-for-checking-memory-usage-in-linux
https://stackoverflow.com/questions/131303/how-can-i-measure-the-actual-memory-usage-of-an-application-or-process
Thanks for your quickly response.
As my understanding, dds.transport.shmem.builtin.receive_buffer_size/received_message_count_max in TaskB is for all topics of a participant. If so, since TaskA & TaskC are communicating with TaskB with SHMEM, it means the shared memory segments (where queue sizes are depending on 'received_message_count_max') in TaskB might be accesed by TaskA / TaskC partially. And then, I guess, the "RES" number in 'top' command of TaskA & TaskC might be increasing. Not sure whether it is correct.
BTW, how is 'received_message_count_max' distributed to DataReaders of a participant? For examples, received_message_count_max = 100, a DataReader/TopicX & a DataReader/TopicY. Will a DataWriter directly access DataReader's SHMEM receiving buffers?
Yes, it seems the big issue is how to get an accurate "memory usage" of a process in Linux, since the 'top' command "RES" column is not so accurate. Thanks for your shared blog/articles.
The shared memory transport is an abstraction of a transport that internally uses shared memory for communications. It's no different than a UDP transport or a TCP transport.
Each shared memory transport allocates a shared memory segment (think of it as a mailbox) to which other participants (maybe in other tasks) will send their RTPS formated messages...which may contain data for Topics or are meta messages like discovery, heartbeats, acknacks, etc.
Each shared memory transport is configured to have at the most "receive_message_count_max" in the shared memory buffer (that's used like a FIFO) queue...this is the same as the UDP socket for a UDP transport. So, yes, all data for all topics of a participant may pass through the shared memory transport before they are processed...but the same transport is used for non-Topic messages as well.
DataWriters and DataReaders do NOT access shared memory directly (at least not by default). A copy of the data must be passed into shared memory for a DataWriter or a data must be copied out of shared memory for a DataReader.
If you are looking for direct access, you need to configure and use the Connext Zero Copy feature....which starts with defining your datatype as zero-copy enabled. See docs here: https://community.rti.com/static/documentation/connext-dds/6.1.2/doc/manuals/connext_dds_professional/users_manual/index.htm#users_manual/SendingLDZeroCopy.htm%3FTocPath%3DPart%25203%253A%2520Advanced%2520Concepts%7C23.%2520Sending%2520Large%2520Data%7C23.1%2520Reducing%2520Latency%7C23.1.5%2520Zero%2520Copy%2520Transfer%2520Over%2520Shared%2520Memory%7C_____0