86.10 Synchronizing of Persistence Service Instances

By default, different Persistence Service instances do not synchronize with each other. For example, in a scenario with two Persistence Service instances, the first persistence service could receive a DDS sample ‘S1’ from the original DataWriter that is not received by the second persistence service. If the disk where the first persistence service stores its DDS samples fails, ‘S1’ will be lost.

86.10.1 Synchronization Configuration

To enable synchronization between Persistence Service instances, use the tag <synchronization> under <persistence_service>. When it comes to synchronization, there are two different kinds of information that can be synchronized independently:

• Information about Durable Subscriptions and their states (see 86.9 Configuring Durable Subscriptions in Persistence Service)
• DDS data samples

Table 86.8 Synchronization Tags

Tags within <synchronization>	Description	Number of Tags Allowed
<data_synchronization>	Configure sample synchronization protocol for redundant Persistence Service instances The data synchronization protocol ensures that DDS data samples are synchronized across redundant Persistence Service instances. When the <enable> tag in <data_synchronization> is set to true, the Persistence Service instances communicate with each other to synchronize their samples. This protocol guarantees eventual consistency at the DDS instance level if the Persistence Service instances are configured with the following QoS policies: History QoS policy: Persist the last value of an instance (KEEP_LAST = 1). DestinationOrder QoS policy: Enforced by source timestamp on both DataWriter and DataReader. When original DataWriters publishing samples stop writing for a DDS instance, all Persistence Service instances will eventually converge to the same last value for that instance. The synchronization protocol is designed to minimize DDS sample exchanges. Typically, each Persistence Service instance receives a DDS sample only once. Refer to Table 86.9 Data Synchronization Tags for additional configuration details.	0 or 1
<synchronize_data> (DEPRECATED)	Enables the old sample synchronization protocol and should not be used. Using this protocol leads to high bandwidth usage. This is because each Persistence Service instance transmits received data samples to all other Persistence Service instances, regardless of whether those instances have already received the samples. Default: false	0 or 1
<synchronize_durable_subscription>	Enables synchronization of Durable Subscriptions in redundant Persistence Service instances. When set to true, the different Persistence Service instances will synchronize their Durable Subscription information. This information includes the set of Durable Subscriptions as well as information about the Durable Subscription’s state, such as the DDS samples that have already been received by the Durable Subscriptions. Default: false	0 or 1
<durable_subscription_synchronization_period>	The period (in milliseconds) at which the information about Durable Subscriptions is synchronized. Default: 5000 milliseconds	0 or 1

Table 86.9 Data Synchronization Tags

Tags within <data_synchronization>	Description	Number of Tags Allowed
<enable>	When set to true, enables DDS data synchronization between Persistence Service instances. See also 86.10.2.3 Affected QoS Settings. Default: false	0 or 1
<remote_provider_selection_period>	Controls how often (expressed in milliseconds) a Persistence Service instance evaluates the remote DataWriter (remote provider) from which it will be getting DDS samples published by the original DataWriter. If a Persistence Service instance is connected directly to the original DataWriter, it will pick the original DataWriter as its remote provider. Therefore, a remote provider can be another Persistence Service DataWriter or the original DataWriter. Default: 5000 ms	0 or 1
<stable_provider_period_count>	Specifies the number of consecutive remote_provider_selection_periods during which the selected set of remote providers must remain unchanged before the selection is considered stable. When a new set of remote providers is selected for a (DataReader, DataWriter) pair, the Persistence Service waits for this number of periods without any change in the selection. Once stability is confirmed, the ContentFilteredTopic associated with the DataReader is updated to begin receiving samples from the selected remote providers. If set to 1, the Persistence Service considers the selection stable immediately after it is made, and switches to the new providers without waiting. Default: 1	0 or 1
<max_remote_provider_change_period_count>	Sets an upper limit on how long the Persistence Service may tolerate instability in remote provider selection for a given (DataReader, DataWriter) pair. Specifically, this property defines the maximum number of consecutive remote_provider_selection_periods during which the set of selected remote providers is allowed to change. If this threshold is reached without the selection stabilizing, the Persistence Service will stop waiting, adopt the most recently selected set of remote providers, and consider the selection stable. This property must be greater than or equal to stable_provider_period_count. Its purpose is to prevent indefinite oscillation in provider selection, which may occur due to fluctuating network conditions, intermittent provider availability, or other transient issues. Such instability can lead to unreliable system behavior, increased latency in data reception, and excessive processing overhead. By enforcing this limit, the service ensures progress and system responsiveness even when perfect stability is not achievable. Default: 1	0 or 1
<enable_sync_builtin_filter>	Enables or disables use of the synchronization built-in content filter. See 86.10.2 Data Synchronization Protocol Overview for more information. When set to false, this tag disables the use of the synchronization built-in content filter, and the Persistence Service will synchronize data using the built-in SQL-based filter, which is less efficient. Default: true	0 or 1

 

86.10.2 Data Synchronization Protocol Overview

The data synchronization protocol ensures that DDS data samples are synchronized across redundant Persistence Service instances. When the <data_synchronization>/<enable> tag is set to true, the Persistence Service instances communicate with each other to synchronize their samples.

The data synchronization protocol does not work when there are Routing Service instances between the Persistence Service instances synchronizing their data.

This protocol guarantees eventual consistency at the DDS instance level if the Persistence Service instances are configured with the following QoS policies:

• History QoS policy: Persist the last value of an instance (KEEP_LAST = 1).
• DestinationOrder QoS policy: Enforced by source timestamp on both DataWriter and DataReader.

When original DataWriters publishing samples stop writing for a DDS instance, all Persistence Service instances will eventually converge to the same last value for that instance.

The data synchronization protocol is designed to minimize DDS sample exchanges. Typically, each Persistence Service instance receives a DDS sample only once. This sample may come directly from the original DataWriter or from a remote Persistence Service DataWriter. A remote DataWriter that provides samples to the Persistence Service DataReader associated with a Persistence Service DataWriter is referred to as a remote provider.

To ensure that a DDS sample is received only once by a Persistence Service DataReader, the synchronization protocol utilizes ContentFilteredTopics (CFTs). To optimize performance and reduce bandwidth usage, the protocol uses a built-in content filter rather than the built-in SQL filter. The filter expression, accessible through DDS_SubscriptionBuiltinTopicData.content_filter_property.filter_expression, is not intended to be interpreted by the application.

If a Persistence Service instance decides to change any of its selected remote providers (for example, one of the existing remote providers disconnects) for one of its DataReaders, it will delete the DataReader and recreate a new one with a new CFT expression.

Because recreating a DataReader consumes bandwidth, changes to CFTs should be minimized. The process of changing the remote provider set for a Persistence Service DataReader involves two steps:

• Remote provider selection
• Remote provider change

86.10.2.1 Remote Provider Selection

This step runs periodically, based on the value specified in the <remote_provider_selection_period> tag. During this step, the Persistence Service evaluates whether the set of remote providers for any of its DataReaders should change.

A change in a remote provider is considered if:

• The current remote provider is lagging in sequence numbers, or
• A candidate provider offers a shorter replication path than the current provider.

At the end of this step, the Persistence Service may decide that a different set of remote providers would be preferable. However, any actual change in remote providers occurs only in the second step: remote provider change.

86.10.2.2 Remote Provider Change

A change in the selected remote providers for a Persistence Service DataReader is applied only when the new selection remains stable for a number of selection periods, as configured using the <stable_provider_period_count> tag. Alternatively, if a maximum number of selection periods—configured with <max_remote_provider_change_period_count>—passes without reaching stability, the Persistence Service will stop waiting. In that case, it adopts the most recently selected set of remote providers and considers the selection stable.

When a change in remote providers is applied, the Persistence Service deletes the affected DataReader and re-creates it with a new ContentFilteredTopic (CFT) based on the updated selection.

Important: Because the Persistence Service uses CFTs to minimize network traffic, the CFT expression will grow in size with the number of original DataWriters sending samples to the Persistence Service instances. This may result in long filter expressions. To prevent issues when handling these lengthy filter expressions, it's recommended to increase the default value of participant_qos.resource_limits.contentfilter_property_max_length for local participants containing DataWriters that send samples to the Persistence Service. There is no need to do this for the Persistence Service participants, since this value is automatically set to 65535 (the maximum value) when synchronization is enabled.

86.10.2.3 Affected QoS Settings

When data synchronization is enabled (by setting <data_synchronization>/<enable> to true), Persistence Service overwrites the following QoS settings:

• <propagate_related_source_guid> (see 86.8 Creating Persistence Groups) - ignored and treated as false because the synchronization protocol relies on its own mechanisms to track and synchronize samples between Persistence Service instances; propagating the related source GUID would interfere with that
• propagate_dispose_of_unregistered_instances (see 60.1 DATA_READER_PROTOCOL QosPolicy (DDS Extension)) - set to true, otherwise some Persistence Service instances or DataReaders might never learn that a sample instance was disposed, leading to inconsistencies; for example, a late-joining DataReader might see an instance as alive when it was disposed earlier

86.10.2.4 Detecting Samples Replicated by Persistence Service

This section explains how to verify that a sample published by an application has been replicated by one or more RTI Persistence Service instances.

86.10.2.4.1 SyncAnnouncement Topic

The synchronization protocol introduces a special Topic called SyncAnnouncement. Each Persistence Service uses SyncAnnouncement to broadcast its synchronization state. Applications can subscribe to this Topic to determine whether a given sample has been received by a Persistence Service.

Replication detection relies on comparing:

• the identity of an application sample (writer_guid and sequence_number)
• the announcements periodically published by Persistence Service instances

86.10.2.4.2 Replication Verification Workflow

Follow this standard workflow to use replication detection in your application:

1. Assign a unique ID to each Persistence Service participant
2. Subscribe to SyncAnnouncement
3. Get the identity of each written sample
4. Confirm replication by matching the sample identity with SyncAnnouncement

The following section provides a step-by-step walkthrough of this process, including functional code examples.

86.10.2.4.3 Steps to Detect Replication

Step 1: Assign a unique identity to each Persistence Service participant

Every Persistence Service instance must have a unique identity. To achieve this, we recommend configuring a unique identifier via the Property QoS policy:

<persistence_service name="ExamplePS">

	<synchronization>

		<data_synchronization>

			<enable>true</enable>

		</data_synchronization>

	</synchronization>

	<participant name="PSParticipant">

		<domain_participant_qos>

			<property>

				<value>

					<element>

						<name>participant.replication_name</name>

						<value>PersistenceService1</value>

						<propagate>true</propagate>

					</element>

				</value>

			</property>

		</domain_participant_qos>

	</participant>

</persistence>

Other mechanisms, such as the UserData QoS policy, can also be used.

Note: If you use the EntityName QoS policy, Persistence Service automatically prefixes names with RTI Persistence Service: <application name> <participant name>.

Step 2: Subscribe to SyncAnnouncement

Use the following IDL definition for the SyncAnnouncement Topic:

module RTI {

	module PersistenceService {

		module Sync {

			@nested

			struct GUID {

				octet value[16];

			};

			@nested

			struct SequenceNumber {

				int32 high;

				uint32 low;

			};

			const string ANNOUNCEMENT_TOPIC_NAME = "SyncAnnouncement";

			struct Announcement {

				@key GUID source_virtual_guid;

				@key GUID provider_virtual_guid;

				int32 distance;

				SequenceNumber last_published_sn;

				sequence<GUID> path_provider_guids;

			};

		};

	};

};

Important: When generating typecode for SyncAnnouncement, use the -unboundedSupport argument in rtiddsgen. See the RTI Code Generator User's Manual for more information.

Use the following DataReader QoS to subscribe to the Topic:

auto reader_qos = dds::core::QosProvider::Default().datareader_qos(

		"BuiltinQosLib::Generic.KeepLastReliable.TransientLocal");

reader_qos << dds::core::policy::History::KeepLast(1);

Step3: Obtain the identity of each written sample

When writing data, use the DataWriter::write_w_params operation to capture each sample’s identity.

The field identity (params.identity()) contains the sample’s identity after the write call finishes. The identity consists of:

• writer_guid: the 16-byte unique DataWriter identifier
• sequence_number: the unique identifier for each sample published by a specific DataWriter

Step 4: Match sample identity with SyncAnnouncement

To confirm replication:

1. Listen for SyncAnnouncement samples
2. Validate that:
   • the writer_guid publishing the target sample matches the source_virtual_guid in the announcement sample from the Persistence Service instance replicating the sample (announcement.source_virtual_guid == writer_guid)
   • the Persistence Service instance replicating the sample published all samples up to the sequence number of the target sample (announcement.last_published_sn >= sequence_number)

You can identify which Persistence Service published the announcement using the SampleInfo::publication_handle instance handle and the DDS_DataReader_get_matched_publication_participant_data Builtin Topic API.

The following Modern C++ function illustrates how to check if a sample has been replicated:

bool has_sample_been_replicated(

		const std::string& persistence_service_participant_name,

		const rti::core::SampleIdentity& sample_identity,

		dds::sub::DataReader<::RTI::PersistenceService::Sync::Announcement>& reader,

		const dds::sub::SampleInfo& announcement_info,

		const ::RTI::PersistenceService::Sync::Announcement& announcement)

{

	// Get participant data for the matched publication (throws on error)

	dds::topic::ParticipantBuiltinTopicData participant_data;

	try {

		participant_data = rti::sub::matched_publication_participant_data(

			reader,

			announcement_info.publication_handle());

	} catch (const dds::core::Exception& e) {

		std::cerr << "Error getting participant data for announcement: "

			<< e.what() << std::endl;

		return false;

	}

	// Check if the participant name matches the expected one

	auto replication_name = participant_data->property().try_get(

			"participant.replication_name");

	if (!replication_name.has_value()) {

		// Property not found: ignore the announcement

		return false;

	}

	if (*replication_name != persistence_service_participant_name) {

		// Name mismatch: ignore the announcement

		return false;

	}

	// Check if the source virtual GUID matches the writer GUID

	const rti::core::Guid& writer_guid = sample_identity.writer_guid();

	for (uint32_t i = 0; i < rti::core::Guid::LENGTH; ++i) {

		if (announcement.source_virtual_guid.value[i] != writer_guid[i]) {

			return false;

		}

	}

	// Check if the last published sequence number covers the sample's sequence number

	rti::core::SequenceNumber last_published_sn(

		announcement.last_published_sn.high,

		announcement.last_published_sn.low);

	if (last_published_sn < sample_identity.sequence_number()) {

		return false;

	}

	// All checks passed: the sample has been replicated

	return true;

}

Note: The above example implementation is provided for illustrative purposes. To optimize performance, cache the publication_handle rather than re-fetching it each time.

86.10.2.4.4 Performance Considerations

Persistence Service does not announce every received sample. Instead, it publishes SyncAnnouncement periodically, with the frequency governed by the <remote_provider_selection_period> synchronization parameter located in the <data_synchronization> tag. Note that:

• units are in milliseconds
• lower values prioritize faster replication detection but result in higher system overhead
• higher values prioritize bandwidth efficiency but result in slower replication discovery

<persistence_service name="SyncTesterMultiNew">

	<synchronization>

		<data_synchronization>

			<enable>true</enable>

			<remote_provider_selection_period>2000</remote_provider_selection_period>

		</data_synchronization>

	</synchronization>

</persistence_service>