Hallo,
I need some help with a data race occuring in libnddscore.
The data race seems to occur between two threads DDS threads that belongs to a DoaminParticipant.
When the data race occurs a lost of samples can be monitored.
Data race
================== WARNING: ThreadSanitizer: data race (pid=15202) Read of size 8 at
0x7f8c44580110 by thread T27 (mutexes: write M2899): #0 memcpy /build/gcc-multilib/src/gcc/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:596 (libtsan.so.0+0x00000002fda2) #1 REDACursor_copyReadWriteArea
(libnddscore.so+0x00000049c152) Previous write of size 8 at 0x7f8c44580110 by thread T57 (mutexes: write M270914287804219624): #0 memcpy /build/gcc-multilib/src/gcc/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:596 (libtsan.so.0+0x00000002fda2) #1 PRESPsService_readerNotifyOfReaderQueueChanges (libnddscore.so+0x0000001f2c66)
The location seem to be allocated within a DataReader implementation Location is heap block of size 221728 at 0x7f8c44557000 allocated by thread T320:
#0 calloc /build/gcc-multilib/src/gcc/libsanitizer/tsan/tsan_interceptors.cc:606 (libtsan.so.0+0x000000027a27) #1 RTIOsapiHeap_reallocateMemoryInternal (libnddscore.so+0x0000004a670f) #2 rti::sub::DataReaderImpl::DataReaderImpl(dds::sub::TSubscriber const&, dds::topic::TopicDescription<Commands::Session::Create::Reply, rti::topic::TopicDescriptionImpl> const&, dds::core::TEntityQos const&, dds::sub::DataReaderListener*, dds::core::status::StatusMask const&) /usr/include/ndds/hpp/rti/sub/DataReaderImpl.hpp:254 (libdds_test_lib.so+0x0000036a6d06) #3 dds::sub::DataReader<Commands::Session::Create::Reply, rti::sub::DataReaderImpl>::DataReader(dds::sub::TSubscriber const&, dds::topic::Topic<Commands::Session::Create::Reply, rti::topic::TopicImpl> const&, dds::core::TEntityQos const&, dds::sub::DataReaderListener*, dds::core::status::StatusMask const&) /usr/include/ndds/hpp/dds/sub/TDataReader.hpp:590 (libdds_test_lib.so+0x0000036a6e43)
Thread creation:
DDSFactory.cpp line 62: dds::domain::DomainParticipant domainParticipant(domainId, m_qosPovider->participant_qos(kQosDefaultLibaryName.str() + "::participant"));
Thread T27 (tid=15508, running) created by thread T29 at: #0 pthread_create /build/gcc-multilib/src/gcc/libsanitizer/tsan/tsan_interceptors.cc:900 (libtsan.so.0+0x000000028874) #1 RTIOsapiThread_initializeWithStack (libnddscore.so+0x0000004b1eae) #2 DDSFactory::getDomainParticipant(int) DDSFactory.cpp:62
Thread T57 (tid=15521, running) created by thread T29 at: #0 pthread_create /build/gcc-multilib/src/gcc/libsanitizer/tsan/tsan_interceptors.cc:900 (libtsan.so.0+0x000000028874) #1 RTIOsapiThread_initializeWithStack (libnddscore.so+0x0000004b1eae) #2 DDSFactory::getDomainParticipant(int) DDSFactory.cpp:62
Mutex creation:
Mutex M2899 (0x7b1c000619e8) created at: #0 pthread_mutex_init /build/gcc-multilib/src/gcc/libsanitizer/tsan/tsan_interceptors.cc:1117 (libtsan.so.0+0x0000000291af) #1 RTIOsapiSemaphore_new (libnddscore.so+0x0000004aa540) DDSFactory::getDomainParticipant(int)
Note: the data writer is not created from the same thread than the domain participant. We implemented a factory that ensures that we only create one domain participant per domain in our application. The domain paricipant access as well as the topic access and the creation of requester / replier / writer / reader is mutex protected.
I would be thankful for any hints.
Sorry for not providing source code, but the current implementation would be to complex and I have not found time to implement a simple example that reproduces the data race.
Today I found some time to build the rti lib as debug build. Here is the thread sanitizer output:
================== WARNING: ThreadSanitizer:data race (pid=23398) Read of size 8 at 0x7fc198ccc8c0 by thread T42(mutexes: write M2876 ) :#0 memcpy /build/gcc-multilib/src/gcc/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:596 (libtsan.so.0+0x00000002fda2)#1 REDACursor_copyReadWriteArea packages_rti_ndds/src/rti-ndds/reda.1.0/srcC/table/Cursor.c:739 (libnddscore.so+0x000000472072)
Previous write of size 8 at 0x7fc198ccc8c0 by thread T66 (mutexes: write M271758712723658504):#0 memcpy /build/gcc-multilib/src/gcc/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:596 (libtsan.so.0+0x00000002fda2)
#1 PRESPsService_readerNotifyOfReaderQueueChanges packages_rti_ndds/src/rti-ndds/pres.1.0/srcC/psService/PsServiceImpl.c:2187 (libnddscore.so+0x0000001db026)
Location is heap block of size 221728 at 0x7fc198cc7000 allocated by thread T284:#0 calloc /build/gcc-multilib/src/gcc/libsanitizer/tsan/tsan_interceptors.cc:606 (libtsan.so.0+0x000000027a27)
#1 RTIOsapiHeap_reallocateMemoryInternal packages_rti_ndds/src/rti-ndds/osapi.1.0/srcC/memory/heap.c:748 (libnddscore.so+0x00000047bfb5)
#2 rti::sub::DataReaderImpl::DataReaderImpl(dds::sub::TSubscriber const&, dds::topic::TopicDescription<Commands::Session::Create::Reply, rti::topic::TopicDescriptionImpl> const&, dds::core::TEntityQos const&, dds::sub::DataReaderListener*, dds::core::status::StatusMask const&) /usr/include/ndds/hpp/rti/sub/DataReaderImpl.hpp:254 (libdds_test_lib.so+0x0000036a6d06)
#3 dds::sub::DataReader<Commands::Session::Create::Reply, rti::sub::DataReaderImpl>::DataReader(dds::sub::TSubscriber const&, dds::topic::Topic<Commands::Session::Create::Reply, rti::topic::TopicImpl> const&, dds::core::TEntityQos const&, dds::sub::DataReaderListener*, dds::core::status::StatusMask const&) /usr/include/ndds/hpp/dds/sub/TDataReader.hpp:590 (libdds_test_lib.so+0x0000036a6e43)
Mutex M2876 (0x7b1c00017bc8) created at:#0 pthread_mutex_init /build/gcc-multilib/src/gcc/libsanitizer/tsan/tsan_interceptors.cc:1117 (libtsan.so.0+0x0000000291af)
#1 RTIOsapiSemaphore_new packages_rti_ndds/src/rti-ndds/osapi.1.0/srcC/semaphore/Semaphore.c:1250 (libnddscore.so+0x00000047fd70)
#2 DDSFactory::getDomainParticipant(int) DDSFactory.cpp:64 (libdds_test_l_dds.so+0x00000001b4c3)
Mutex M271758712723658504 is already destroyed.Thread T42 (tid=23655, running) created by thread T58 at:#0 pthread_create /build/gcc-multilib/src/gcc/libsanitizer/tsan/tsan_interceptors.cc:900 (libtsan.so.0+0x000000028874)
#1 RTIOsapiThread_initializeWithStack packages_rti_ndds/src/rti-ndds/osapi.1.0/srcC/thread/Thread.c:1905 (libnddscore.so+0x000000487056) #2 DDSFactory::getDomainParticipant(int) DDSFactory.cpp:64 (libdds_test_l_dds.so+0x00000001b4c3)
Thread T66 (tid=23665, running) created by thread T58 at:#0 pthread_create /build/gcc-multilib/src/gcc/libsanitizer/tsan/tsan_interceptors.cc:900 (libtsan.so.0+0x000000028874)#1 RTIOsapiThread_initializeWithStack packages_rti_ndds/src/rti-ndds/osapi.1.0/srcC/thread/Thread.c:1905 (libnddscore.so+0x000000487056)
#2 DDSFactory::getDomainParticipant(int) DDSFactory.cpp:64 (libdds_test_l_dds.so+0x00000001b4c3)
#3 DDSData::participant() const DDSData.h:55 (libdds_test_lib.so+0x000002b524eb)
Thread T284 (tid=24853, running) created by thread T785 at:#0 pthread_create /build/gcc-multilib/src/gcc/libsanitizer/tsan/tsan_interceptors.cc:900 (libtsan.so.0+0x000000028874)
#1 __gthread_create /build/gcc-multilib/src/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:662 (libstdc++.so.6+0x0000000b9d65)
#2 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) /build/gcc-multilib/src/gcc/libstdc++-v3/src/c++11/thread.cc:163 (libstdc++.so.6+0x0000000b9d65)
SUMMARY:ThreadSanitizer: data race packages_rti_ndds/src/rti-ndds/reda.1.0/srcC/table/Cursor.c:739 in REDACursor_copyReadWriteArea ==================