How to read "historical" persistent data

Hi Dallas,

You can find a good code example on how to use content filters here. Also, the RTI Connext Core Libraries and Utilities User's Manual includes an explaination on how to set up content filters--including the use of parameters that can be changed at run-time--in its Chapter 12 (Section 5.4.3).

Basically, to be able to modify a set parameters on a filter expression at run-time, you have to create your content filtered topic indicating the sequence of parameters that can be modified.

    DDS_StringSeq parameters(2);
    const char* param_list[] = {"1", "4"};
    parameters.from_array(param_list, 2);

    DDSContentFilteredTopic *cft = NULL;
    if (sel_cft) {
        cft = participant->create_contentfilteredtopic(
            "ContentFilteredTopic", topic, "(x >= %0 and x <= %1)", parameters);
        if (cft == NULL) {
            printf("create_contentfilteredtopic error\n");
            subscriber_shutdown(participant);
            return -1;
        }
    }

Then, whenever you want to change the value of those parameters, you have to (1) change their value in the parameters string squence, and (2) call set_expression_parameters() to update the filter.

    parameters[0] = DDS_String_dup("5");
    parameters[1] = DDS_String_dup("9");
    retcode = cft->set_expression_parameters(parameters);

Regarding your first question, KEEP_LAST_HISTORY_QOS indicates the number of samples per instance that the middleware stores locally for data writers and data readers. If the data writer of your publisher application (or the persitence service data writer) keeps the last 20 samples of each of your 10 instances, it won't be able to send data readers more samples than the number of samples stored for each instance. On the data reader side, only the samples that passing the filter will be stored.

Therefore, if you want to increase the number of samples you receive for each instance you have to increase the KEEP_LAST_HISTORY_QOS depth on your data writers (including persistence service for its potential use), and accordingly its depth on your data readers.

Fernando.

Hi,

Regarding the depth of the history received by the DataReader, I think it is enough if you increase the history depth on the pesistence service DataWriter.  RTI Connext DDS will automaticaly combine what is in the persistence service DataWriter with the history in the original DataWriter (if it still around) so that the DataReader that specified a DURABILITY of TRANSIENT or PERSISTENT will get the larger history.

Gerardo

Hello,

Your observation below is exactly right:

    "dynamic changes to the content filtered topic only affect the reception of newly published samples"

This is how it works:  

• If a DataReader is subscribed to a ContentFilteredTopic and the filter parameters are changed, the change causes DDS to send an update of the new DataReader filter to the DataWriters via the discovery writer (BuiltinSubscriptionDataWriter).   Once the DataWriter sees the change, it will apply the new filter parameters to the samples it writes after it receives the change in the filter.  
• At the same time, as soon as the filter parameters are changed, the DataReader starts using the new filter parameters. So if the DataWriter is still sending samples filtered with the old filter parameters, the DataReader will see that the filter is different and apply the new filter paramaters effectively refiltering the samples.
• So the change of parameters will cause the stream to be first filtered with the old filter, then with both filters, and then with the new filter.

In this process the DataWriter never re-evaluates the history of samples that were already filtered for that DataReader. The reason is that it would be quite complicated to define the proper behavior. For instance:

1. Is the DataReader suppossed to see duplicate samples (if it passed both the old and new filters) or only the ones that pass the new filter, but not the old one?
2. Is the DataReader allowed to see "old" samples prior to the ones that it has already received. Doing so would make the time "go backwards" and leave it with the impression that a particular instance has a value when it really has somethign different.

These may or may not be problems in partuclar situations. But it is very hard to get the behavior to be correct and intuitive in general so for this reason the DataWriter does not go back and replay its history to the DataReader...

So if you need this old history you have 2 options I think:

1. You can create a new DataReader with the new filter.  For this to work as you expect you also need the DataReader and DataWriter configured with DURABILITY QoS kind set to something other than VOLATILE (for example you configure both to TRANSIENT_LOCAL; or else you configure the DataReader to TRANSIENT and have a PersistenceService configured to maintain the durable data). And in addition you also need the DataWriter HISTORY QoS to have the RefilterQosPolicyKind set to DDS_ALL_REFILTER_QOS.
2. You can create the DataReader without a filter so that it receives all the samples, and then use a QueryCondition to locally read the samples that match the filter of interest. As the samples are all in the DataReader cache you can change the filter parameters in the QueryCondition and read other samples in the cache.

The first option works because a DataWriter that is configured with DURABILITY kind TRANSIENT_LOCAL, TRANSIENT, or PERSISTENT will keep samples in its cache for future, to-be-discovered DataReaders. When the DataWriter discovers a new DataReader that also specifies non-VOLATILE DURABILITY the DataWriter will send the data in its history cache. If the DataWriter HISTORY QoS has been configured with the RefilterQosPolicyKind set to DDS_ALL_REFILTER_QOS then the DataWriter will filter the historical samples and only sent the ones that pass the filter. If the RefilterQosPolicyKind is set to DDS_NONE_REFILTER_QOS then it will just send all the samples and let the DataReader filter them, which is probably not what you are looking for.

Let us know if this helps,

Gerardo