Hi,
is it feasible to transfer large chunks of data through dds? I need to send images, pdf, ... from a client to a server and vice versa. Or should I use something other (e.g. websockets?).
What QoS settings are needed for such a task?
Thx,
Christian
Hi Christian,
To send large data through DDS reliably you will need to use an Asynchronous Publisher, which sends the user data in a external middleware thread.
There is an example on GitHub that shows how to configure Asynchronous Publication, but probably the easiest option is to use one of the built-in QoS profiles designed for that purpose. You could use:
Generic.StrictReliable.LargeData, which enables asynchronous publication with the default flow controller (i.e,. no flow control).Generic.StrictReliable.LargeData.FastFlow, which enables strict reliable communication for large data with a 100 MB/sec (838 Mb/sec) flow controller.Generic.StrictReliable.LargeData.MediumFlow,which enables strictly reliable communication for large data with a 25 MB/sec (209 Mb/sec) flow controller.Generic.StrictReliable.LargeData.SlowFlow, which enables strictly reliable communication for large data with a 6.25 MB/sec (52 Mb/sec) flow controller.Let me know if these solutions work for you. Otherwise, we can go over your example for a more specific configuration.
Thanks,
Fernando.
Hi,
thx for the information. We've the problem that rti crashes / throws an exception when we try to transport large data. Looks like there is a bug deep inside the serialization because it only crashes under certain conditions. I've attached a testcase which shows the problem (tested with i86Win32VS2015 and x64Linux3gcc4.8.2). The qos must be an async qos like BuiltinQosLibExp::Generic.StrictReliable.LargeData
The idl must be exact the given idl. Removing one of the parameters does not trigger the problem. Also the size of the content must not be 1024*1024. Therefore it looks like some kind of alignment problem to me.
Here the error:
RTICdrStream_serializeParameterLength:!precondition: me == ((void *)0) || lengthPosition == ((void *)0) || state == ((void *)0)PRESWriterHistoryDriver_initializeSample:!serialize
WriterHistoryMemoryPlugin_addEntryToSessions:!initialize sample
WriterHistoryMemoryPlugin_getEntry:!add virtual sample to sessions
WriterHistoryMemoryPlugin_addSample:!get entry
PRESWriterHistoryDriver_addWrite:!add_sample
PRESPsWriter_writeInternal:!collator addWrite
I'll file a bug report for this.
/edit: forgot to add that we're using 5.2.3-6
Hey Christian,
In case you are referring to a crash when receiving the sample and trying to deserialize it:
This may be a long shot but I recall certain problems occurring due to windows fragmentation not working properly (that is, dds tries to send a message that is bigger than what the network interface is willing to send, causing a fragmentation at the OS level to occur, unfortunately, with some problematic bits).
This may be a bit difficult to spot but if you have the exact dds sample that causes the crash, you may want to use wireshark to capture the network while the sample is being sent.
If you are experiencing the crash before the sample is sent this seems irrelevant.
On a general note I would recommend using DDS based on the following video (which mostly addresses IOT but is also relevant for general RTI DDS usage):
https://www.youtube.com/watch?v=umtYEp67pHc
To sum it up:
RTI DDS does well with:
reliability (and more generally, a lot of configurability)
performance (latencies of milli or micro seconds / scalability)
integration (given a good design of the data, you can achieve very quick growth while not being limited to one operating system / language [and DDS also allows for easy integration with other DDS vend ors])
It's true that other solutions may also do the above with varying success, but this is what RTI DDS (at least according to the CEO of RTI) claims to be good at, and from my experience (of 4.5 years), it's pretty much true.
On a personal note I would still recommend examining your use-case more accurately (that is: do you REALLY need millisecond latencies? or does it just sound like a good thing to have, do you REALLY need configurable reliability? or will you be okay with built in reliabilities offered by 'simpler' solutions).
DDS is great but I would not recommend any product as a 'solve all problems' kinda thing (other than whiskey).
Have fun,
Roy.
To transfer large files, I use DDS for the C2, and either sftp or similar to send the actual file (i.e, Destination sends a "what you got?", Source sends "all of these", Destination sends "want that one, here's my in-box", Source sftp or scp the requested file to the Destination's in-box, Destination sends "file tx complete".
agreeing with Roy (also on the Whisky part)... use the tool best suited to the purpose.
rip