Common NDDS Errors Seen on VxWorks Systems and Their Meanings

Submitted by juan on Thu, 01/17/2013 - 08:11

Note: Applies to RTI NDDS 2.x and earlier, and NDDS 3.x.

This Solution describes errors you may see when running an NDDS application on VxWorks 5.4.x or 5.5.x. These error messages most likely mean that the amount and/or sizing of your stack buffers are incorrect.

NDDS is unable to send

"NDDS Alarm: Unable to serialize fired message"

This message indicates NDDS cannot use the stack to send a local UDP message, typically because the stack is out of "data pool" buffers.

NDDS is unable to create sockets

"RtiDGramInitialize(): socket() failed because OS errno = 0x37"

This message usually occurs during domain creation and indicates that NDDS couldn't create a new datagram socket. The errno value suggests that the stack is out of "system pool" buffers.

Packet loss

When the TCP/IP stack runs out of buffers, it will silently drop packets. You will notice that a best-effort subscription will start losing issues.

Poor performance using reliable protocols

When packet loss occurs, the RTPS reliable protocols will resend packets; this will cause increased latency and decreased throughput.

The stack stops working completely

This is the most drastic symptom: all communication going through the TCP/IP stack stops and the target seems dead. To check whether the problem is related to the stack, try to connect to the target over a communication line that does not use the TCP/IP stack (such as a serial connection to a target shell). In this error situation, the target will behave normally (there are usually no fatal error messages from the OS side) but the stack is locked up (does not send nor receive).

To determine if your problem is related to buffer availability, try sending less data by either:

  • sending fewer packets (e.g., decrease the send-rate of the publications or increase the minimum separation of the best-effort subscriptions)
  • sending packets with less data (decrease the user-data size)

If the symptoms disappear when you send less data, it is most likely that your problem is buffer related. The VxWorks TCP/IP stack uses a fixed number of buffers. Although NDDS doesn't use TCP (only UDP) these buffers are used for sending and receiving messages through the UDP side of the stack as well. The number of buffers and their sizes can only be changed by altering the kernel configuration parameters. This is different from non real-time OS's (such as Windows/UNIX systems) where the buffers can be allocated as needed from main memory.

When using NDDS on VxWorks, it is sometimes necessary to modify the buffer allocation parameters to achieve optimal performance or to avoid running out of buffers. In certain cases, running out of buffers is a fatal event, which can stop the stack from sending or receiving further data until the target is rebooted. See Solution on "How do I resolve a problem with VxWorks buffers?" for more details on modifying buffers in VxWorks.

Errors of type 'setsockopt'

"NDDSDGram::NDDSDGram: setsockopt(SO_SNDBUF) failed"
"NDDSDGram::NDDSDGram: setsockopt(SO_RECVBUF) failed"

These errors mean that NDDS is not able to increase the send socket buffer size and receive socket buffer size. They could occur during the call to NddsInit() or when you try to directly set the sizes for the send/receive buffers. They should not cause the application to crash or lose functionality, but will affect performance. If you are getting the performance you want, then this may not be a problem, since those errors do not cause loss of functionality and you can safely ignore them.

Also, there might be an upper limit to the send and receive socket buffer sizes that can be changed. You might want to take a look at your VxWorks kernel settings to see what that upper limit is and change it.

Product:
Platform:
Keywords: