DDSTheParticipantFactory->delete_participant hangs

Hello gandriotakis, 

One possible culprit for this:

Our receive threads are blocked on sockets, and they are woken up when the delete_participant() call sends a message over loopback to each socket our receive threads are blocking on.  This allows the thread to wake up, realize that it's shutting down, and clean itself up.  

In some cases, this message doesn't get received – especially when a firewall is running – and one or more recieve thread fails to wake up to be shut down.

So, can you tell me a bit more about your system?  Do you have a firewall running?

Thank you!

Rose

Hello gandriotakis, 

I checked, and we have a known issue on certain versions of Linux where the delete_participant call may hang forever.  I am checking whether there is a workaround.  

Edit: I just checked the details of the bug report that I mentioned, and it looks like it is on systems where that last shutdown packet doesn't get through to the DataReader (such as being blocked by a firewall).  I doubt that this problem is due to your code, but I am not sure why it is happening in 4.5f but not in the previous version.  What was the previous version you were using? What OS are you running on?

Thank you!

Rose

Hello gandriotakis,

We have only seen this problem on Linux.  But two more questions for you:  Do you have multiple DomainParticipants in your application?  Is multicast enabled in your application?

We have reproduced this problem on Linux, but only if there are two DomainParticipants in an application, and if you delete them in a different order than they were created. 

One other thing worth exploring:  It would be helpful to see which threads are still running and preventing the call from completing.  Do you have Process Explorer?  If you do, it will show you more details about which threads are running.

Thank you!

Rose

Hello gandriotakis, 

It would be very strange if the other threads shut down correctly, but the call is still hanging.  Can you increase the verbosity of DDS?  That will slow down your execution time, but it might give me a hint where it's hanging.

NDDSConfigLogger::get_instance()->set_output_file( File *yourfile );
NDDSConfigLogger::get_instance()->set_verbosity( NDDS_CONFIG_LOG_VERBOSITY_STATUS_ALL ); 

Also, can you attach the QoS configuration you are using?  One other place that can have problems is if you set the DomainParticipant's shutdown_cleanup_period to 0.  This usually only causes problems on RTOSes, though, so it is a long shot. 

Thank you!

Rose

Hello gandriotakis, 

As a test, can you comment out this property?

           <element>
                   <name>dds.transport.UDPv4.builtin.ignore_loopback_interface</name> 
                   <value>0</value> 
           </element> 

Thank you!

Rose

There is nothing else in the QoS that looks like it could be responsible.

(There is a mismatch in some of the profiles between the UDP sizes that you  might want to change: 65530 vs. 65535.  That is unlikely to be related to this, though.)

Can you build a debug version, open the application in Visual Studio, and double-check which threads are running when it hangs?  I believe you might be able to get our thread stack traces that way.  (I am still reading through the log file, but in retrospect the stack trace may be faster).

Thank you!

Rose

The delay isn't a problem – glad to know this isn't holding you up too badly.

Can you reproduce the same problem when building against our debug libraries?  I'm hoping that gives a more detailed stack trace.

Thank you!

Rose

MODIFIED: I had not noticed you had already attached the verbosity output to a previous posting. Please ignore the request below...

Hi,

We are still shooting in the dark. It seems like for some reason whe the operating system dumps the stack it not able to find the symbols for the nddscppd.dll

Maybe you can call the operation:

 NDDSConfigLogger::set_verbosity(NDDS_CONFIG_LOG_VERBOSITY_STATUS_ALL); 

to enable verbose output prior to calling participant->delete_contained_entities() operation or before calling DDSTheParticipantFactory->delete_participant(). That may shed some light into what the RTIDDS core is doing when it hangs.

Gerardo

Thank you for the additional details!

That stack trace says that we are definitely waiting for a thread to shut down.  It's still not 100% certain that it's a receive thread, but in combination with the error messages in the log file you sent, I am willing to make a strong guess that that is the reason why the main thread never shuts down.

Here is what I see:  At about the time the application is calling delete_contained_entities(), there are error messages in your log file indicate that the WSASendTo call gets error 10093, WSANOTINITIALISED.

[D0101|DELETE_CONTAINED]NDDS_Transport_UDPv4_send:OS WSASendTo() failure, error 0X276D

[D0101|DELETE_CONTAINED]NDDS_Transport_UDPv4_send:U0000136c sent 88 bytes to 0X100007F:7660 

This error is being printed as DDS is sending the final messages at shutdown.  We send these final messages for two reasons:

1. To notify other applications about reader/writer deletions.  
2. To wake up our own receive threads for a clean shutdown

Given the error messages, there is a very good chance that wake up data is not being received, causing the main thread to wait forever for the receive threads to unblock and shutdown cleanly.  

Looking closely at this WSANOTINITIALISED error, it looks like this error could happen in two cases:

1. if we didn't call WSAStartup (in which case we would see this error at the beginning of the file, too), or
2. if somebody called WSACleanup() before these final messages are sent.  

I just took a look through our code, and if you are using the default UDP transport we should not be calling WSACleanup() at all in 4.5f.  Is it possible that your application is calling WSACleanup() somewhere, or using another library that calls WSACleanup()?  (Also, if there is anything else you can think of that might cause a network disconnection during the shutdown process, let me know.)

Thank you!

Rose

After all DomainParticipants are deleted in your application, we don't need to use the network anymore.  So really, just watiting for delete_participant() call (or calls) to complete should be enough.  You don't have to finalize the DomainParticipantFactory before making a WSACleanup() call.

If for some reason you call WSACleanup() and then need to create/enable a new DomainParticipant, it calls the WSAStartup() API again, which should be safe.

Thank you!

Rose

The only indication that we have succesfully shut down is when all delete_participant() calls complete and return a return code saying the calls were successful.  It should be possible to build logic around this to signal the other code that DDS has finished cleaning up, but there is no callback mechanism for that.

Thank you!

Rose

That's great that it solved the problem!  

Thank you,

Rose