Introduction
Depending on your resource limit configuration, restarting Connext Micro applications can sometimes lead to applications never being able to re-discover each other, or you may see delays in re-discovery. This article explains the cause of these behaviors and provides guidance on optimizing your configuration to minimize them.
The best solution for this issue is to upgrade to Micro 2.4.14.3 or Micro 4.3.0. Those versions include features that allow two applications to instantly re-discover each other when one is restarted. These features can be accessed by enabling the following settings listed below:
Enabling participant_discovery_by_name ensures that previously discovered remote participants can be instantly re-discovered after they are restarted. Instead of relying on the participant’s GUID (which changes on every restart), the system now matches participants by their name. When a remote participant with a new GUID is detected whose name already exists in the host’s participant database, the system will immediately release the resources held by the old instance and register the newly discovered participant.
Enabling endpoint_discovery_queue ensures that even if endpoint resources are temporarily insufficient when discovering a restarted remote participant, the system will queue the endpoint discovery samples. These samples will be held and later committed to the host participant’s internal database once sufficient resources become available, allowing discovery to complete successfully.
If upgrading to Micro 2.4.14.3 or Micro 4.3.0 is not possible, please continue reading the article
Resource Limits and Their Effect on Re-Discovery
The remote_participant_allocation is the main parameter that directly affects re-discovery. Connext relies on liveliness to detect when a remote participant is lost and to free database space. Since Participant A is only configured to allocate space for a single remote Participant, restarting Participant B will result in the following behavior: Participant A cannot discover the new Participant B until it confirms the loss of the old Participant B. Confirmation of loss and initiation of re-discovery is directly affected by two QoS parameters:
If the remote_participant_allocation is greater than the exact number of participants discoverable in the domain, you may encounter a scenario in which re-discovery will never occur again. This is because an extra allocated participant also requires extra allocations in the endpoint resource limits. If there is sufficient space for the restarted participant but not enough for its endpoints, you may encounter a situation where the remote participant is successfully discovered, but its endpoints are dropped due to endpoint resource limits. This is why it is vital that the remote_participant_allocation is set to the exact number of participants discoverable in the domain.
Setting Resource Limits
In Connext Micro, resource optimization is key. If your domain only involves two DomainParticipants (A and B), you should configure each to allocate space to discover only one remote participant. This also applies for DataReaders and DataWriters: if each participant only manages a single reader and writer, allocate the absolute minimum amount of resources at both the DomainParticipant and endpoint (DataWriter and DataReader) levels. Refer to the QoS parameters below to manage these limits effectively.
Domain Participant QoS:
Endpoint QoS:
In What Time Frame Should I Expect Re-Discovery to Complete?
Re-discovery timing is determined by the lease duration and assertion period. Assuming a 100s lease duration and 30s assertion period, here is an example sequence of events when Participant B restarts (with Participant A limited to one remote participant):
T=0s: New Participant B sends its first Data(p) assertion. Participant A rejects this because its slot is still occupied by Old Participant B.
T=100s: Participant A's lease duration for Old Participant B expires, freeing the slot.
T=120s: New Participant B sends its next scheduled Data(p) assertion. Participant A accepts this, discovering New Participant B.
The worst-case wait time is the sum of the lease duration and the assertion period.
How Do I Decrease the Time it Takes for Re-Discovery To Occur?
To tune the re-discovery time, you should decrease the liveliness_lease_duration and liveliness_assert_period accordingly. For example, using default values, the worst case scenario is 130 seconds. If you set the lease duration to 10 seconds, and the assert period to 3 seconds, then the worst case scenario is 13 seconds.
