15.8.8 Exclusive Areas (EAs)

Listener callbacks are invoked by internal Connext threads. To prevent undesirable, multi-threaded interaction, the internal threads may take and hold semaphores (mutexes) used for mutual exclusion. In your listener callbacks, you may want to invoke functions provided by the Connext API. Internally, those Connext functions also may take mutexes to prevent errors due to multi-threaded access to critical data or operations.

Once there are multiple mutexes to protect different critical regions, the possibility for deadlock exists. Consider Figure 15.5: Multiple Mutexes Leading to a Deadlock Condition’s scenario, in which there are two threads and two mutexes.

Figure 15.5: Multiple Mutexes Leading to a Deadlock Condition

Thread1 takes MutexA while simultaneously Thread2 takes MutexB. Then, Thread1 takes MutexB and simultaneously Thread2 takes MutexA. Now both threads are blocked since they hold a mutex that the other thread is trying to take. This is a deadlock condition.

While the probability of entering the deadlock situation in Figure 15.5: Multiple Mutexes Leading to a Deadlock Condition depends on execution timing, when there are multiple threads and multiple mutexes, care must be taken in writing code to prevent those situations from existing in the first place. Connext has been carefully created and analyzed so that we know our threads internally are safe from deadlock interactions.

However, when Connext threads that are holding mutexes call user code in listeners, it is possible for user code to inadvertently cause the threads to deadlock if Connext APIs that try to take other mutexes are invoked. To help you avoid this situation, RTI has defined a concept known as Exclusive Areas, some restrictions regarding the use of Connext APIs within user callback code, and a QoS policy that allows you to configure Exclusive Areas.

Connext uses Exclusive Areas (EAs) to encapsulate mutexes and critical regions. Only one thread at a time can be executing code within an EA. The formal definition of EAs and their implementation ensures safety from deadlock and efficient entering and exiting of EAs. While every Entity created by Connext has an associated EA, EAs may be shared among several Entities. A thread is automatically in the entity's EA when it is calling the entity’s listener.

Connext allows you to configure all the Entities within an application in a single DDS domain to share a single Exclusive Area. This would greatly restrict the concurrency of thread execution within Connext’s multi-threaded core. However, doing so would release all restrictions on using Connext APIs within your callback code.

You may also have the best of both worlds by configuring a set of Entities to share a global EA and others to have their own. For the Entities that have their own EAs, the types of Connext operations that you can call from the Entity’s callback are restricted.

To understand why the general EA framework limits the operations that can be called in an EA, consider a modification to the example previously presented in Figure 15.5: Multiple Mutexes Leading to a Deadlock Condition. Suppose we create a rule that is followed when we write our code. “For all situations in which a thread has to take multiple mutexes, we write our code so that the mutexes are always taken in the same order.” Following the rule will ensure us that the code we write cannot enter a deadlock situation due to the taking of the mutexes, see Figure 15.6: Taking Multiple Mutexes in a Specific Order to Eliminate Deadlock.

Figure 15.6: Taking Multiple Mutexes in a Specific Order to Eliminate Deadlock

By creating an order in which multiple mutexes are taken, you can guarantee that no deadlock situation will arise. In this case, if a thread must take both MutexA and MutexB, we write our code so that in those cases MutexA is always taken before MutexB.

Connext defines an ordering of the mutexes it creates. Generally speaking, there are three ordered levels of Exclusive Areas:

  • ParticipantEA

    • There is only one ParticipantEA per participant. The creation and deletion of all Entities (create_xxx(), delete_xxx()) take the ParticipantEA. In addition, the enable() method for an Entity and the setting of the Entity’s QoS, set_qos(), also take the ParticipantEA. There are other functions that take the ParticipantEA: get_discovered_participants(), get_publishers(), get_subscribers(), get_discovered_topics(), ignore_participant(), ignore_topic(), ignore_publication(), ignore_subscription(), remove_peer(), and register_type().

  • SubscriberEA

    • This EA is created on a per-Subscriber basis by default. You can assume that the methods of a Subscriber will take the SubscriberEA. In addition, the DataReaders created by a Subscriber share the EA of its parent. This means that the methods of a DataReader (including take() and read()) will take the EA of its Subscriber. Therefore, operations on DataReaders of the same Subscriber, will be serialized, even when invoked from multiple concurrent application threads. As mentioned, the enable() and set_qos() methods of both Subscribers and DataReaders will take the ParticipantEA. The same is true for the create_datareader() and delete_datareader() methods of the Subscriber.

  • PublisherEA

    • This EA is created on a per-Publisher basis by default. You can assume that the methods of a Publisher will take the PublisherEA. In addition, the DataWriters created by a Publisher share the EA of its parent. This means that the methods of a DataWriter including write() will take the EA of its Publisher. Therefore, operations on DataWriters of the same Publisher will be serialized, even when invoked from multiple concurrent application threads. As mentioned, the enable() and set_qos() methods of both Publishers and DataWriters will take the ParticipantEA, as well as the create_datawriter() and delete_datawriter() methods of the Publisher.

In addition, you should also be aware that:

  • The three EA levels are ordered in the following manner:
    ParticipantEA < SubscriberEA < PublisherEA
  • When executing user code in a listener callback of an Entity, the internal Connext thread is already in the EA of that Entity or used by that Entity.
  • If a thread is in an EA, it can call methods associated with either a higher EA level or that share the same EA. It cannot call methods associated with a lower EA level nor ones that use a different EA at the same level.