86.10 Synchronizing of Persistence Service Instances
By default, different Persistence Service instances do not synchronize with each other. For example, in a scenario with two Persistence Service instances, the first persistence service could receive a DDS sample ‘S1’ from the original DataWriter that is not received by the second persistence service. If the disk where the first persistence service stores its DDS samples fails, ‘S1’ will be lost.
86.10.1 Synchronization Configuration
To enable synchronization between Persistence Service instances, use the tag <synchronization> under <persistence_service>. When it comes to synchronization, there are two different kinds of information that can be synchronized independently:
- Information about Durable Subscriptions and their states (see 86.9 Configuring Durable Subscriptions in Persistence Service)
- DDS data samples
|
Tags within |
Description |
Number of Tags Allowed |
|
<data_synchronization> |
Configure sample synchronization protocol for redundant Persistence Service instances The data synchronization protocol ensures that DDS data samples are synchronized across redundant Persistence Service instances. When the <enable> tag in <data_synchronization> is set to true, the Persistence Service instances communicate with each other to synchronize their samples. This protocol guarantees eventual consistency at the DDS instance level if the Persistence Service instances are configured with the following QoS policies:
When original DataWriters publishing samples stop writing for a DDS instance, all Persistence Service instances will eventually converge to the same last value for that instance. The synchronization protocol is designed to minimize DDS sample exchanges. Typically, each Persistence Service instance receives a DDS sample only once. Refer to Table 86.9 Data Synchronization Tags for additional configuration details. |
0 or 1 |
|
<synchronize_data> (DEPRECATED) |
Enables the old sample synchronization protocol and should not be used. Using this protocol leads to high bandwidth usage. This is because each Persistence Service instance transmits received data samples to all other Persistence Service instances, regardless of whether those instances have already received the samples. Default: false |
0 or 1 |
|
<synchronize_durable_subscription> |
Enables synchronization of Durable Subscriptions in redundant Persistence Service instances. When set to true, the different Persistence Service instances will synchronize their Durable Subscription information. This information includes the set of Durable Subscriptions as well as information about the Durable Subscription’s state, such as the DDS samples that have already been received by the Durable Subscriptions. Default: false |
0 or 1 |
|
<durable_subscription_synchronization_period> |
The period (in milliseconds) at which the information about Durable Subscriptions is synchronized. Default: 5000 milliseconds |
0 or 1 |
|
Description |
Number of Tags Allowed |
|
|
<enable> |
When set to true, enables DDS data synchronization between Persistence Service instances. Default: false |
0 or 1 |
|
<remote_provider_selection_period> |
Controls how often (expressed in milliseconds) a Persistence Service instance evaluates the remote DataWriter (remote provider) from which it will be getting DDS samples published by the original DataWriter. If a Persistence Service instance is connected directly to the original DataWriter, it will pick the original DataWriter as its remote provider. Therefore, a remote provider can be another Persistence Service DataWriter or the original DataWriter. Default: 5000 ms |
0 or 1 |
|
<stable_provider_period_count> |
Specifies the number of consecutive remote_provider_selection_periods during which the selected set of remote providers must remain unchanged before the selection is considered stable. When a new set of remote providers is selected for a (DataReader, DataWriter) pair, the Persistence Service waits for this number of periods without any change in the selection. Once stability is confirmed, the ContentFilteredTopic associated with the DataReader is updated to begin receiving samples from the selected remote providers. If set to 1, the Persistence Service considers the selection stable immediately after it is made, and switches to the new providers without waiting. Default: 1 |
0 or 1 |
|
<max_remote_provider_change_period_count> |
Sets an upper limit on how long the Persistence Service may tolerate instability in remote provider selection for a given (DataReader, DataWriter) pair. Specifically, this property defines the maximum number of consecutive remote_provider_selection_periods during which the set of selected remote providers is allowed to change. If this threshold is reached without the selection stabilizing, the Persistence Service will stop waiting, adopt the most recently selected set of remote providers, and consider the selection stable. This property must be greater than or equal to stable_provider_period_count. Its purpose is to prevent indefinite oscillation in provider selection, which may occur due to fluctuating network conditions, intermittent provider availability, or other transient issues. Such instability can lead to unreliable system behavior, increased latency in data reception, and excessive processing overhead. By enforcing this limit, the service ensures progress and system responsiveness even when perfect stability is not achievable. Default: 1 |
0 or 1 |
86.10.2 Data Synchronization Protocol Overview
The data synchronization protocol ensures that DDS data samples are synchronized across redundant Persistence Service instances. When the <data_synchronization>/<enable> tag is set to true, the Persistence Service instances communicate with each other to synchronize their samples.
This protocol guarantees eventual consistency at the DDS instance level if the Persistence Service instances are configured with the following QoS policies:
- History QoS policy: Persist the last value of an instance (KEEP_LAST = 1).
- DestinationOrder QoS policy: Enforced by source timestamp on both DataWriter and DataReader.
When original DataWriters publishing samples stop writing for a DDS instance, all Persistence Service instances will eventually converge to the same last value for that instance.
The data synchronization protocol is designed to minimize DDS sample exchanges. Typically, each Persistence Service instance receives a DDS sample only once. This sample may come directly from the original DataWriter or from a remote Persistence Service DataWriter. A remote DataWriter that provides samples to the Persistence Service DataReader associated with a Persistence Service DataWriter is referred to as a remote provider.
To ensure that a DDS sample is received only once by a Persistence Service DataReader, the synchronization protocol utilizes ContentFilteredTopics (CFTs). To optimize performance and reduce bandwidth usage, the protocol uses a built-in content filter rather than the built-in SQL filter. The filter expression, accessible through DDS_SubscriptionBuiltinTopicData.content_filter_property.filter_expression, is not intended to be interpreted by the application.
If a Persistence Service instance decides to change any of its selected remote providers (for example, one of the existing remote providers disconnects) for one of its DataReaders, it will delete the DataReader and recreate a new one with a new CFT expression.
Because recreating a DataReader consumes bandwidth, changes to CFTs should be minimized. The process of changing the remote provider set for a Persistence Service DataReader involves two steps:
- Remote provider selection
- Remote provider change
86.10.2.1 Remote Provider Selection
This step runs periodically, based on the value specified in the <remote_provider_selection_period> tag. During this step, the Persistence Service evaluates whether the set of remote providers for any of its DataReaders should change.
A change in a remote provider is considered if:
- The current remote provider is lagging in sequence numbers, or
- A candidate provider offers a shorter replication path than the current provider.
At the end of this step, the Persistence Service may decide that a different set of remote providers would be preferable. However, any actual change in remote providers occurs only in the second step: remote provider change.
86.10.2.2 Remote Provider Change
A change in the selected remote providers for a Persistence Service DataReader is applied only when the new selection remains stable for a number of selection periods, as configured using the <stable_provider_period_count> tag. Alternatively, if a maximum number of selection periods—configured with <max_remote_provider_change_period_count>—passes without reaching stability, the Persistence Service will stop waiting. In that case, it adopts the most recently selected set of remote providers and considers the selection stable.
When a change in remote providers is applied, the Persistence Service deletes the affected DataReader and re-creates it with a new ContentFilteredTopic (CFT) based on the updated selection.
Important: Because the Persistence Service uses CFTs to minimize network traffic, the CFT expression will grow in size with the number of original DataWriters sending samples to the Persistence Service instances. This may result in long filter expressions. To prevent issues when handling these lengthy filter expressions, it's recommended to increase the default value of participant_qos.resource_limits.contentfilter_property_max_length for local participants containing DataWriters that send samples to the Persistence Service. There is no need to do this for the Persistence Service participants, since this value is automatically set to 65535 (the maximum value) when synchronization is enabled.