Hi there,
I am a current PhD student working on "Experimental and Model-based Evaluation of DDS Performance and Security". As part of my research I am looking for a benchmarking tool that would allow me to performance test multiple DDS products such as Vortex OpenSplice and OpenDDS. I have experience performance testing using RTI's PerfTest application and would like to compare the results with other DDS products.
So far, I have found various other benchmarking tools that seem outdated or unsupported such as DDS Benchmark Environment, Open Architecture Benchmark and DDSBench. If there are no available benchmarking tools then I may have to create one. Does anyone have any advice on searching for or creating a benchmarking tool able to evaluate the performance of multiple DDS products?
Thank you for taking the time to read through this post and wish you a great day!
Thanks,
Kaleem Peeroo
Hi Kaleem,
So, first I will say that it's very difficult to create a performance test that actually measure what you're looking to measure correctly and accurately. There are a variety of variables that depend on the use case not limited to size of data, data structure complexity or simplicity, CPU and network speeds, rate at which data is being sent, number of subscribers for Topic, number of topics, number of threads sending/number of threads processing received data, sending on the same host, same network, across subnets, across NATs and firewalls.
Fundamentally, in our experience we have not found a huge difference in perfomance for the most basic tests, e.g, 1 publisher to 1 subscriber with a single Topic and simple data structure. It's not like one DDS is 10x or even 2x better in latency/throughput. You're talking 10% to 30% maybe.
Practically speaking, select a DDS based on performances like buying a car based on its top speed. People don't do that. Any DDS that is selected must meet the performance requirements of the project. Given that one DDS is able to meet the performance requirements, it's more than likely that several different DDS will. Performance is a necessary condition for selection...it's often not a sufficient condition.
Just like most people decide on the car that they want to purchase based on non-performance criteria, styling, manufacturer, accessories, technology...sure for some people, there's a minimum performance criteria that they use to consider a car in the first place.
So, having said that...as I first wrote, performance tests are hard to create correctly. AND, even though DDS is a standard, how to optimize a particular implementation for performance will require different configuration settings...some of which may be specific to the DDS implementation.
Thus, a performance test offer by a DDS vendor will certainly be optimized for the performance of the DDS that they offer, but likely not optimized for any other DDS implementation that it claims to test.
A performance test offered by a "neutral" third party is likely to not to have been optimized to get the best performance of any of the DDS implementations that it works with...
So having stated all of this, RTI does offer a performance test in source code that can be (and have been) used to test different DDS implementations. It's certainly critical to use the same test infrastructure to test multiple DDSs if you want to eliminate the influence of the test framework on the results.
You can get the code and build for RTI Connext DDS out-of-the-box from
https://github.com/rticommunity/rtiperftest
You can (others have) create plugins for other DDS implementations...even for non-DDS communication protocols and run the test framework against those other implementations.
However, it's easier said than done since even though the source is designed to execute other middleware through a middleware plugin, we haven't provided much documentation on how to do it.
If you're a good coder, you should be able to see how we've implemented the RTI Connext DDS plugin and added it to the test framework code as well as the build infrastructure...and then basically copy, duplicate, modify for other implementations. But again, we haven't documented how to do it, but lots of people have been able to figure it out and done tests on different DDSes.
Good luck!
Hi Howard,
Thank you for the amazingly fast, insightful and detailed response!
Do you have any advice when looking to create plugins for various other DDS implementations? Do you know anyone who has done what I aim to do and would it be possible to come into contact with them? Have you got any snippets of code that show exactly what needs to be done?
I appreciate you taking the time to answer my questions and hope you have a fantastic day/weekend!
Thanks,
Kaleem
Hi Kaleem,
The way that RTI Perftest is structured is to use a plugin to access middleware...this includes RTI Connext DDS.
So the RTIDDSImpl is actually a plugin that invokes RTI Connext DDS and you can use it as an example to build something to create plugins that invokes other middleware.
I used the plugin infrastructure myself to test vSOME/IP. The trick is that you can't just create the plugin, you also have to modify the framework to know about your plugin and to allow it to be tested via command line arguments.
I'll attach a zip file with all of the additional plugin files as well as existing files that I had to change to use the plugin. You can use those files as an example to creat your own plugins. I would NOT copy these files directly into your own rtiperftest source tree, but only as an example of how and where to modify.
You can do diffs to see the difference between what was modified and what you get out of github to figure out what I needed to change to add in a plugin for vSOME/IP.
Good luck!
Hi Howard,
Thank you again for the amazing speed of your response!
Just to clarify, Perftest is the benchmarking tool and it uses a plugin to work with Connext. Therefore, in order to achieve what I intend in terms of a benchmarking tool that evaluates multiple DDS products I need to create plugins for the other products and then modify Perftest to use these plugins?
Thank you very much,
Kaleem
Hi Kaleem,
Yes, Perftest is a benchmarking tool that RTI freely provides. It has a plugin architecture so that it can be modified to test middlewares other than Connext DDS. Out-of-the box, it can test Connext DDS, Connext DDS Micro and raw UDP.
You would have to create plugins to test other DDS implementations or other types of middleware.
--Howard