Hi,
Imagine that you need transmit a lot of different data through you system. To do that, you have to define all data types required and the topics associated to these data types.
For each data type defined can be one or more Topics associated, so imagine that the number of Topics in the system is around 1000. It means that the number of DataWriters and DataReaders in the system will be higher because the Topics are shared among several participants.
I've done several tests about the percentatge of CPU used and memory used depending of the number of entities created.
In the first test I created 1 participant, 1 publisher, 1 DataWriter 1 Topic X Instance, being X= 10, 100, 1000.
An the second test I created 1 participant, 1 publisher, X DataWriter 1 Topic 1 Instance, being X= 10, 100, 1000.
And so on.
When I ran a test when X=1000 the time to create all entities (1000 publishers or 1000 datawriters, etc...) was extremly long, the magintude is in minutes.
So, my question is: Where is the middle-point where grouping several data types in a single one is better than use several data types?
Thanks a lot.
This is not an easy question! The location of the "sweepspot" is going to depend on how efficient the discovery mechanism is, whether the network supports multicast, and whether it is a flat network or you have "Routing Service" components that provide some levels of isolation federation.
I will not try to provide concrete numbers because that really depends on the deployment scenario and the best woud be do do some kind of experimental analysis or benchmark.
In general having a DataWriter is much more expensive than having anTopic Instance. A DataWriter is an entity that needs to be discovered by every participant in the system. It also creates a separate "RTPS" session which means state needs to be maintained on the matched DataReaders, the sequence numbers that each matched DataReader has acknowledged, it requires some level of network heartbeats, etc. This means additional discovery traffic to discover every DataWriter as well as additional resources to maintain the state.
So if discovery time and memory resources were the only consideration then fewer DataWriters would be better. But another consideration is the efficiency of actually distributing the Data. Here having more DataWriters can be better because that way a DataWriter is only matched with the DataReaders that are interested on the data it publishes and with more Topics there will be fewer matches per Topic. If you have many Topic instances then you have to rely on content filters topis to prevent the instances that are not of interest from being sent to the DataReaders. RTI Connext DDS is very efficient in doing writer-side filtering. But creating this content filters can be complicated for large numbers of instances, and also it becomes a lot harder to do multicast.
So which is more important depends on how much resources you have, how often you will write. How commonality there is aong subscribers so multicast can be leveraged, etc. As I mentioned earlier answering this in seems like it would require some scenario-driven research.
I have observed in practical scenarios is that to keep discovery time reasonable it is good not to exceed a few hundred Topics on a flat network. Hundreds of thousands of Instances on the other hand can be managed well. So in this case a rule of thumb would be to keep NumTopics < 1000 x NumInstances. But this is relly handwavy.
Gerardo
Good explanation Gerardo.
I just want to add one thing: one of the most heavy information that is exchanged in the discovery traffic is the topic type code and object information (that is the topic data type).
When we moved from an 1 Gigabit lan configuration to a 380Kb network (using radio link), the initial discovery took more than 10 minutes to complete. By disabling type code/object discovery traffic, we reduced to a 50 seconds the time required to complete the initial discovery. This was achieved by setting to zero the corresponding participant properties in our QOS configuration file.
<resource_limits><type_object_max_serialized_length>0</type_object_max_serialized_length>
<type_code_max_serialized_length>0</type_code_max_serialized_length>
</resource_limits>
Of course disabling this kind of traffic requires that all participants share the topic data type. It also prevents valuable RTI tools (such as the AdminConsole) to work properly.