Chapter 73 Identifying Threads Used by Connext

Connext uses multiple internal threads for sending and receiving data, maintaining internal state, and calling user code when events occur. Further details regarding Connext’s threading model can be found in Part 11: Connext Threading Model. This section explains how these threads can be identified in your system.

73.1 Checking Thread Names at the OS Level

On some systems, it is possible to check the internal name of RTI threads directly at the operating system level. Threads created by Connext will have RTI-specific thread names, unless otherwise stated in the RTI Connext Core Libraries Platform Notes, which lists architectures that do not support setting thread names.

In general, thread names follow this pattern:

r<Module>[<Participant identifier>][<Thread index>][Transport name>]<Task type>

Where:

• The maximum length for a thread name is 16, including the '\0'.
• r indicates this is a thread from RTI.
• The second and third characters identify the <Module>:

• <Participant identifier> is represented with five characters, as follows:
• If participant_name is set: the participant identifier will be the first three characters and the last two characters of the participant_name.
• If participant_name is not set: the identifier is computed as domain_id (three characters), participant_id (two characters).
• If participant_name is not set and the participant_id is set to -1 (default value): the participant identifier is computed as the last five digits of the rtps_instance_id in the participant GUID.

• <Thread index> - index used to distinguish among threads with the same name.

For example, there are several instantiations of the receive thread; the thread index is used to differentiate them:

rCo32265##00Rcv

rCo32265##01Rcv

rCo32265##02Rcv

rCo32265##03Rcv

rCo32265##04Rcv

• <Transport name> is represented with four characters:

• <taskType> - the type of thread is represented with three characters:

The details on checking the thread names depend on the operating system. The following is an example output from a publisher application running on VxWorks 6.9.4:

-> taskSpawn "test", 255, <floating_point_option>, 150000, publisher_main, 1, 100

value = 83748528 = 0x4fde6b0

-> i

NAME         ENTRY       TID    PRI   STATUS      PC       SP     ERRNO  CPU #

----------  ------------ -------- --- ---------- -------- -------- ------- -----

[...]

rCoHelnt##> RTIOsapiThr>  444a010  71 PEND         37b218  53f7c00       0     -

rCoHelnt##> RTIOsapiThr>  540b2b0  71 PEND         37b218  542ac00       0     -

rCoHelnt##> RTIOsapiThr>  543e080  71 PEND         37b218  545dc90       0     -

rCoHelnt##> RTIOsapiThr>  543ea78  71 PEND         37b218  5490c00       0     -

rCoHelnt##> RTIOsapiThr>  5471860  71 PEND         37b218  54c5c90       0     -

Test        REDATester_>  456c010 100 STOP         2cf594  501cc94       0     -

rCoHelnt##> RTIOsapiThr>  44d4358 110 PEND+T       37b218  735fda0  3d0010     -

rCoHelnt##> RTIOsapiThr>  44dea68 120 PEND+T       37b218  730fe04       0     -

rTrHelntUD> RTIOsapiThr>  53bbcb0 120 PEND+T       37b218  53c4e2c       0     -

tZynq7Task  2c8e9c        43f0228 240 DELAY        384288  4df5fa4       0     -

miiBusMoni> 2c9974        464bb60 252 DELAY        384288  4654fb8       0     -

test        publisher_m>  4fde6b0 255 DELAY        384288  7231ee8  3d0010     -

tIdleTask0  idleTaskEnt>  43d5418 287 READY        37a918  43d53ec       0     -

tIdleTask1  idleTaskEnt>  43d9670 287 READY        37a918  43d9644       0     -

value = 0 = 0x0

-> ti 0x444a010

NAME     	ENTRY   	TID	PRI   STATUS  	PC   	SP 	ERRNO  DELAY

----------  ------------ -------- --- ---------- -------- -------- ------- -----

rCoHelnt##> RTIOsapiThr>  444a010  71 PEND     	37b218  53f7c00   	0 	0

full task name : rCoHelnt##00Rcv

task entry     : RTIOsapiThreadChild_onSpawned

task affinity  : 0x00000000

[...]

Where <floating_point_option> is a numeric value that varies depending on the hardware. See Enabling Floating Point Coprocessor in Kernel Tasks, in the RTI Connext Core Libraries Platform Notes.

In this example, the i command in VxWorks retrieves information about the running threads. The > at the end of the name (rCoHelnt##>) indicates that the full thread name could not be displayed, because it exceeds 10 characters. You can use the ti command in VxWorks (shown above), followed by the thread ID (TID), to retrieve information about a specific thread, including its full name (in this case, rCoHelnt##00Rcv).

The following is an example from running a subscriber on a Linux machine:

$ ./objs/x64Linux3gcc5.4.0/HelloWorld_subscriber

HelloWorld subscriber sleeping for 4 sec...

HelloWorld subscriber sleeping for 4 sec...

HelloWorld subscriber sleeping for 4 sec...

[...]

$ ps -eT | grep rC

22966 22967 pts/19   00:00:00 rCo32265####Dtb

22966 22968 pts/19   00:00:00 rCo32265####Evt

22966 22970 pts/19   00:00:00 rCo32265##00Rcv

22966 22971 pts/19   00:00:00 rCo32265##01Rcv

22966 22972 pts/19   00:00:00 rCo32265##02Rcv

22966 22973 pts/19   00:00:00 rCo32265##03Rcv

22966 22974 pts/19   00:00:00 rCo32265##04Rcv

Table 73.4 Example Thread Names shows names for the majority of threads created by Connext:

Table 73.4 Example Thread Names

Thread Information	Name	Fields	Example: Domain: 111 Participant Id : 22 ThreadIndex: 33 Topic: HelloWorld DataBase: Test Application Name: TestPersistence
Receive thread	rCo%5s##%02dRcv	Participant identifier, thread index	rCo11122##33Rcv
Asynchronous waitset thread	rCo%5s##%02dAWs	Participant identifier, thread index	rCo11122##33AWs
Database thread	rCo%5s####Dtb	Participant identifier	rCo11122####Dtb
Dispatcher (i.e., asynchronous publishing) thread	rCo%5s##%02dDsp	Participant identifier, thread index	rCo11122##33Dsp
Asynchronous batch flushing thread	rCo%5s####ABF	Participant identifier	rCo11122####ABF
Topic query publication thread	rCo%5s####TQP	Participant identifier	rCo11122####TQP
Event thread	rCo%5s####Evt	Participant identifier	rCo11122####Evt
DNS tracker thread	rCo%5s####DNS	Participant identifier	rCo11122####DNS
Distributed logger writer thread	rDl#########Wri		rDl#########Wri
Secure distributed logger thread	rSe%5s####Log	Participant identifier	rSe11122####Log
TCP control thread	rTr%5s%04sCtr	Participant identification, transportName (TCP4)	rTr11122TCP4Ctr
TCP event thread	rTr%5s%04sEvt	Participant identification, transportName (TCP4)	rTr11122TCP4Evt
TLS receive thread	rTr%5s%04sRcv	Participant identification, transportName (TLS)	rTr11122#TLSRcv
WAN receive thread	rTr%5s%04sRcv	Participant identification, transportName (WAN)	rTr11122#WANRcv
WAN server thread	rTr%5s%04sSvr	Participant identification, transportName (WAN)	rTr11122#WANCtr
Interface tracking thread	rTr%5s%04sITr	Participant identification, transportName (UDP4, UDP6, TCP4)	rTr11122UDP4ITr
Persistence Service receive administration command request thread	rPs%03d######RAC	domainId	rPs111######RAC
Persistence Service discovery thread	rPs%09sDis	Application name	rPsTestPersiDis
Persistence Service reception thread (topic)	rPs%07s%02dRcv	topic name, thread index	rPsHello##33Rea
Persistence Service publication thread	rPs%07s%02dPub	topic name, thread index	rPsHello##33Pub
Persistence Service reception thread (TopicSet)	rPsTopic##%02dRcv	thread index	rPsHello##33Rea
Persistence Service event thread	rPs#########Evt		rPsHello##33Rea
Recording Service timer thread	rRe#########Tim		rRe#########Tim
Monitor event thread	rMo%5s####Evt	Participant identifier	rREHelloWorlPub
Routing Service polling timer thread	rRs#########Tim		rRs#########Tim
Routing Service filter tracker event thread	rRsFilterTr#Evt		rRsFilterTr#Evt
Routing Service monitor statistics event thread	rRsMoSta####Evt		rRsMoSta####Evt
Routing Service monitor publication event thread	rRsMoPub####Evt		rRsMoPub####Evt
Routing Service discovery event thread	rRsDisc#####Evt		rRsDisc#####Evt
Routing Service aysnchronous admin thread	rRsAdmin##%02dAWs	thread index	rRsAdmin##33AWs
Routing Service aysnchronous discovery thread	rRsDisc###%02dAWs	thread index	rRsDisc###33dAWs
Database Integrated Service discovery thread	rDs#########Dis		rDs#########Dis
Database Integrated Service connection thread	rDs%.9sCon	Database name	rDsTestsCon
Database Integrated Service refresh thread	rDs%.9sRef	Database name	rDsTestsRef
Database Integrated Service finalization Library thread	rDsFinalizeLib#		rDsFinalizeLib#
Database Integrated Service event manager thread	rDsManager##Evt		rDsManager##Evt
Web Integrated Service access control list dataBase thread	rWsACL######Dtb		rWsACL######Dtb

73.2 Checking Thread Names from the Call Stack

Thread names are only available in a subset of architectures. See the RTI Connext Core Libraries Platform Notes for which architectures support checking thread names at the OS level. This section lists the correspondence between Connext threads and the functions they run. You can use this information to identify Connext threads from the call stack, independently of your architecture. If you are using VxWorks or Integrity, see 73.1 Checking Thread Names at the OS Level.

This is the correspondence between threads and the functions they run:

• Database Thread: RTIEventActiveDatabaseThread_loop()
• (Main) Event Thread: RTIEventActiveGeneratorThread_loop(). Note that this function is generic to all the event threads. That is, all of the event threads run RTIEventActiveGeneratorThread_loop(), which detects and handles events. For this reason, it can be difficult to distinguish the Main Event Thread from other event threads (such as the Topic Query Publication Event Thread); however, to better make this distinction, you can check whether some (sub)functions are called (for example, the subfunctions related to the Asynchronous Batch Flushing Event Thread and Topic Query Publication Event Thread below).
• Receive Thread: COMMENDActiveFacadeReceiver_loop(), which calls to a different function depending on what transport is being used to get the (meta)data:
• Shared Memory: NDDS_Transport_Shmem_receive_rEA()
• UDP: NDDS_Transport_UDP_receive_rEA()
• TCP: NDDS_Transport_TCP_receive_rEA()

• Interface Tracking Thread: RTIOsapiInterfaceTracker_()
• Transport-Specific Threads:
• TCP Control Thread: NDDS_Transport_TCPv4_Plugin_threadLoop()
• TCP Event Thread: RTIEventActiveGeneratorThread_loop() and NDDS_Transport_TCPv4_Plugin_clientOn<event_name>()

• Asynchronous Publishing Thread: RTIEventJobDispatcherThread_spawnedFnc()
• Asynchronous Batch Flushing Event Thread: RTIEventActiveGeneratorThread_loop() and PRESPsWriter_onFlushBatch()
• Topic Query Publication Event Thread: RTIEventActiveGeneratorThread_loop() and PRESPsService_onWriterServiceDispatchActiveTopicQueriesEvent()

For example, if you are on GNU/Linux, you can run the following command on gdb to get the call stack:

(gdb) thread apply all backtrace

The same information can be seen with Visual Studio. To see this information in Visual Studio, select Debug > Windows > Threads, then do Ctrl+D, T. You will need to add a breakpoint and start the application in debug mode.

73.3 Checking Thread Names Using the Worker’s Name

Connext uses the concept of a worker as an abstraction for threads. Workers are RTI-specific entities used internally to manage critical sections and to provide access to thread-specific storage. Most of the threads created by Connext have an associated worker. In addition, user threads calling certain APIs from Connext will have a worker associated with them. Workers are given a name when they are created. If you have the proper debug symbols, you can use the worker’s name to identify the thread (on a debugger, for instance).

To check the workers’ names, first locate these workers in the threads. You can do that by selecting a thread and printing its full backtrace. Another option is moving up and down through the frames on the thread’s stack. The worker will be either a local variable or the last argument to one of the RTI functions. Here is an example using gdb on GNU/Linux to identify a thread with the method just described:

(gdb) info thread

  Id   Target Id         Frame

* 1    Thread 0x7ffff7fce700 (LWP 6801) "HelloWorld_publ" __clock_nanosleep (clock_id=<optimized out>, flags=0, req=0x7fffffffcb20, rem=0x7fffffffcb30) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:48

  2    Thread 0x7ffff6ec1700 (LWP 6805) "HelloWorld_publ" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225

  3    Thread 0x7ffff66c0700 (LWP 6806) "HelloWorld_publ" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225

[...]

(gdb) thread 2

[Switching to thread 2 (Thread 0x7ffff6ec1700 (LWP 6805))]

(gdb) backtrace full

[...]

#3  0x0000000000c6095b in RTIEventActiveDatabaseThread_loop (param=0x13fc8c0) at ActiveDatabase.c:156

        timeStr = 0x7ffff6ec0dc0 "{0000003d,00000000}"

        t = 0x13fc8c0

        canBeDeleted = 0

        timeBuf = "{0000003d,00000000}"

        workerName = 0x1351cb0 "rDtb2081a9101"

        METHOD_NAME = 0x100a0d0 "RTIEventActiveDatabaseThread_loop"

[...]

As you can see in the example, workers follow the same naming convention as threads (in some cases, a shortened version of it). Workers associated with user threads use the following convention: U<threadId>, where:

• U - indicator that this is a User Thread
• <threadId> - ID given to the thread by the OS