Useful tools to debug DDS issues

Sometimes DDS users face issues while developing their applications, like for example communication issues. In these situations, usually we get this question: how can I figure out the issue and solve it?

There are several tools and features that can help you to debug your DDS issues:

1 - Look for log warnings or error messages.

You can enable log messages in your DDS applications. There are different verbosity levels depending on the kind of messages that you would like to see. Usually we recommend customers to use “Warning” verbosity level to see most common problems. If you need further details in a certain situation, you can increase the verbosity level to “All”. However, be aware that setting the highest verbosity will output lots of messages.

You can set logging via XML or code. To do it via XML, for example with warning verbosity and sending the output to a file, you need to add the following settings to your XML QoS profile:

  <participant_factory_qos>
     <logging>
        <output_file>log.txt</output_file>
        <verbosity>WARNING</verbosity>
        <print_format>TIMESTAMPED</print_format>
     </logging>
  </participant_factory_qos>

To enable logging via code, add the following lines to your application main function (C++ example):

  NDDSConfigLogger::get_instance()->
  set_verbosity_by_category(NDDS_CONFIG_LOG_CATEGORY_ALL,
  NDDS_CONFIG_LOG_VERBOSITY_WARNING);

  FILE *myLogFile = fopen("my_logfile.txt", "a+");
  NDDSConfigLogger::get_instance()->set_output_file(myLogFile);

Note that this code needs to be added before you call any other operations on the RTI Connext DDS API to ensure that any output goes to the file instead of the standard output.

If you need to limit the size of the log output file, follow the steps in this article.

RTI Log Parser

Once you have the log messages, you can parse them using our RTI Log Parser. RTI Log Parser is a command-line tool that processes and enhances Connext DDS log messages making it easier to debug applications.

Let’s take as an example the following log messages:  

  COMMENDSrReaderService_onSubmessage:[1484571004,3507893055] reader oid 0x80000004 received DATA of sn(0000000000,00001346), vSn(0000000000,00001346) from writer 0xffffffff.49e6.1.80000003
  COMMENDSrReaderService_onSubmessage: accepted sn(0000000000,00001346), dataRcvd.lead(0,1346), nextRelSn(0,1347), reservedCount(1)
  COMMENDSrReaderService_onSubmessage:[1484571004,3577888137] reader oid 0x80000004 received DATA of sn(0000000000,00001348), vSn(0000000000,00001348) from writer 0xffffffff.49e6.1.80000003
  COMMENDSrReaderService_onSubmessage: accepted sn(0000000000,00001348), dataRcvd.lead(0,1347), nextRelSn(0,1347), reservedCount(2)
  MIGInterpreter_parse:HEARTBEAT from 0xffffffff,0X49E6
  COMMENDSrReaderService_onSubmessage:[1484571004,3578261799] reader oid 0x80000004 received HB for sn (0000000000,00001345)-(0000000000,00001348), epoch(347) from writer 0xffffffff.49e6.1.80000003
  COMMENDSrReaderService_sendAckNacks:[1484571004,3578261799] reader oid 0x80000004 sent NACK of bitmap lead(0000000000,00001347), bitcount(2), epoch(353) to writer 0xffffffff.49e6.1.80000003
  COMMENDSrReaderService_onSubmessage:[1484571004,3578751425] reader oid 0x80000004 received DATA of sn(0000000000,00001347), vSn(0000000000,00001347) from writer 0xffffffff.49e6.1.80000003
  COMMENDSrReaderService_onSubmessage: accepted sn(0000000000,00001347), dataRcvd.lead(0,1347), nextRelSn(0,1349), reservedCount(-1)
  MIGInterpreter_parse:HEARTBEAT from 0xffffffff,0X49E6
  COMMENDSrReaderService_onSubmessage:[1484571004,3579107907] reader oid 0x80000004 received HB for sn (0000000000,00001347)-(0000000000,00001348), epoch(348) from writer 0xffffffff.49e6.1.80000003
  COMMENDSrReaderService_onSubmessage:[1484571004,3579107907] reader oid 0x80000004 sent ACK of bitmap lead(0000000000,00001349), bitcount(0), epoch(354) to writer 0xffffffff.49e6.1.80000003

This is how the output will look like after being parsed by RTI Log Parser:  

  2017-01-16T12:50:03.919122 | ---> | H2.A2.P1 | R-K_800000 | Received DATA [1346] from writer W-K_800000 (reliable)
  2017-01-16T12:50:03.919122 | | | | *Warning: Missing packet for H2.A2.P1.W-K_800000 to R-K_800000*
  2017-01-16T12:50:03.919122 | ---> | H2.A2.P1 | R-K_800000 | Received DATA [1348] from writer W-K_800000 (reliable)
  2017-01-16T12:50:03.919122 | ---> | H2.A2.P1 | R-K_800000 | Received HB [347] from writer W-K_800000 for samples in [1345, 1348]
  2017-01-16T12:50:03.919122 | <--- | H2.A2.P1 | R-K_800000 | Sent NACK [353] to writer W-K_800000 for 1347 count 2
  2017-01-16T12:50:03.919122 | ---> | H2.A2.P1 | R-K_800000 | Received DATA [1347] from writer W-K_800000 (reliable)
  2017-01-16T12:50:03.919122 | ---> | H2.A2.P1 | R-K_800000 | Received HB [348] from writer W-K_800000 for samples in [1347, 1348]
  2017-01-16T12:50:03.919122 | <--- | H2.A2.P1 | R-K_800000 | Sent ACK [354] to writer W-K_800000 for 1349 count 0

For more information about how to configure DDS logging, please refer to the User’s Manual here for the current release or here for release 5.2.3 (which is the version this article is based on).

2 - Check status of DDS entities

  1. Use RTI Admin Console

    Admin Console is a GUI tool that includes many features to easily debug your DDS applications. For example, with Admin Console you can:

  1. Enable Monitoring Libraries

    RTI Monitoring Library enables RTI Connext DDS applications to provide monitoring data. The monitoring data can be visualized with RTI Monitor, a separate GUI application that can run on the same host as Monitoring Library or on a different host. Monitoring Library periodically queries the status of all Connext DDS entities and brings up all the entity creation/deletions and QoS changes, among other statistics.

    You can enable the Monitoring Libraries by setting values in the DomainParticipant’s PropertyQosPolicy (via code or through an XML QoS profile). The specific instructions to enable Monitoring Library are here for the current release or here for release 5.2.3. If you want to configure the behavior and statistics obtained by Monitoring Library, you can refer to the User’s Manual, here for the current release or here for release 5.2.3.

  1. Check entity status

    DDS entities have different statuses associated to them. The statuses give information about important changes in the entity state. For example, there are statuses that are triggered when a sample is lost or rejected, and also a status to detect the liveliness loss of an entity. An entity status can be retrieved by:

    • Directly checking the status with the appropriate get API.
    • Using listeners.
    • Using StatusConditions and WaitSets.

    This article explains the difference between these methods.

    You can find the different statuses associated with an entity in the User’s Manual and in the online API documentation. For example see Statuses for DataWriters and DataReaders.

3 - Check what’s happening on the wire

Looking at the traffic generated by your applications can give you a lot of information about what is going on. To do this you can use Wireshark .

  • You can find documentation about using Wireshark with RTPS messages in Wireshark for RTI Connext.

  • Wireshark cannot capture traffic from the loopback interface on Windows. You can use RawCap instead to capture this traffic on Windows. Once traffic is captured in a pcap file, you can open it with Wireshark to analyze. These are the steps to use RawCap:

    1. Run your applications with shared memory disabled. You can disable shared memory via XML by adding into your <participant_qos> this setting:

       <transport_builtin>
          <mask>UDPv4</mask>
       </transport_builtin>      
    2. In a terminal run RawCap.exe and select the interface in which you wish to capture traffic.

  • You can configure Wireshark to show RTPS packets with specific colors, as explained in this article.

4 - Finally, you can also find useful tips for system-level basic debugging in this article.