ContentFilteredTopics seem very appropriate for message / event type data, such as the "news story feed" example provided in the documentation.
However, when used on a topic representing a set of objects, there seems to be no way to affirm that an object with previously received instance samples has since left the filter criteria, and that the previously receieved samples are now inconsistent with the current state of the object.
Let's take an extension of the shapes demo. Say the shape state data was much larger, perhaps containing a large X/Y dataset representing a complex polygonal outline. Because of its size, I want to send it send it only when it actually changes shape or color, so I use a reliable delivery + transient durability QOS. I split off the x/y position into a separate topic that is sent with a best effort + deadline QOS, and my client app reads both topics and ties the two together using matching numeric keys. All is well.
Now I start content filtering by position, and also by color.
When the object leaves my position filter bounding box, I just stop getting updates. There’s no callback or method I could see, to get an affirmative notification that it left the bounds, but, given deadline, I can eventually get an on_deadline_missed() callback that I can use to infer that there should be updates coming, but that they have been filtered, and therefore the position must currently be outside my bounding box. Along the way, I suppose I need to also check liveliness on the publisher, if I need to differentiate bounds departure and sender failure, which adds a whole lot of complexity, but OK. I can’t get an affirmative notification that the position is outside the region of interest, but I can sort of work around it to infer it.
What about the shape state? Say I have an instance of a shape is blue, that I’ve gotten samples of before, and after a period of time, the publisher changes it to red, and it now doesn’t pass the filter. Because it doesn’t pass the filter, my client doesn’t get the sample, but still has an outdated sample showing that it’s blue, so client keeps obliviously drawing a blue shape that it shouldn’t be drawing at all.
If I use a deadline QOS so I can get on_deadline_missed(), I now need to send the state data periodically, wasting bandwidth. So I’m motivated to use a large timeout. 10 seconds. OK, but now, my client app shows incorrect state for up to 10 seconds, because it’s still sitting there with its last recieved (fliter-passing) sample from when the shape was blue. So now I need to shorten the deadline to mitigate this inconsistent presentation to the user, and my bandwidth use goes even higher.
The only other method I can think of is to use a regular Topic instead of a ContentFilteredTopic, and when I process data from DDS, that I first use a QueryCondition to read_w_condition() all the samples that do match the filter, then do a take() to get and flush out all the DDS_NOT_READ_SAMPLE_STATE samples, which given that they didn't get returned from read_w_condition(), must be samples that don’t match the filter. Also, if I'm using a history QOS, I might need to iterate the taken samples, and do a take_instance() to clear out any stale matching samples from the DDS buffer? At least with this case I don’t have to heartbeat my data periodically, but I still end up with many samples I’m not interested in going over the wire.
What I need is a method to identify the following case:
- An instance X is a data-object that is known to a DataReader() bound to a ContentFilteredTopic.
- A new sample of Instance X was just written.
- The last sample of X passed the DataReader's filter, but the new one does not.
That method doesn’t need to (shoulnd't!) deliver the filtered-out sample. It doesn’t get triggered if later samples come that are also out of filter. Only if the last sample matched and the current sample doesn’t.
In other words, it signifies the transition of a member variable outside of the filter bounds. It doesn't appear that any API to reveal such a transition exists.
Surely I’m not the only one who’s needed this use case before? Is there another solution I’m missing?
Unfortunately what you describe is a limitation of both DDS and RTPS wire protocol specifications. It is open issue (Issue # 11035) that was filed against the specificartion and is pending resolution by the OMG RTPS Revision Task Force.
Issue 11035 has been open for a while and the last revision of RTPS (version 2.2) did not address it due to its complexity. This is two fold: First resolving the issue requires enhancements to both the wire protocol (RTPS) and the DDS API. Second it is very hard to implement this efficiently.
The problem is that for efficiency it is very desirable (even essential) that the filters are applied by the DataWriter so that the "filtered out" data is not sent on the wire. In this scenario for a DataWriter to notify the DataReader that a particular instance has transitioned from "not filtered" to "filtered" it must remember the "filtered" status of each instance relative to each DataReader. This requires memory resources on the order of
NumInstances x NumDataReaderswhich in the general case can be quite large. Worse, nothing else on the DataWriter side requires resources that scale withNumInstances x NumDataReaders. All other resources scale either withNumInstancesorNumDataReaders. So this would have a huge impact on the overall reasources consumed by the DataWriter.The next revision of RTPS (version 2.3) is scheduled to come out on June 2015. I think there is a significant chance that this issue will be resolved given that he Revision Task Force is actively working on it and there are a couple of proposals being discussed.
In the meantime, as far as I can see there are three posible approaches, two of which you already described in your posting.
The third approach is a bit more complex but can be more efficient as long the number of instances in the system is not too large. The idea is to modify the filter to also "whitelist" a set of Instances and to chnage the "whitelist" dynamically so that the transition from "not-filtered" to "filtered" is captured. I will explain with an example.
Lets use the shapes demo, assume the application subscribes to "Square"
Topicand wants a filter"x < 100".The DataReader starts creting a content filtered topic with the filter
"x < 100"then as soon as it receives an object/instance, it modifies the filter to "whitelist" that instance. Say for example it receives a "Green" square. When this happensthe DataReader modifies the filter so that the new filter is"color = 'Green' OR x < 100". As it receives other objects it keeps doing the same, e.g. if it receives a "Red" square the filter is modified to"color = 'Green' OR color = 'Red' OR x < 100"Now say the "Green" Square goes outside the filter (e.g. x = 110). Because the color 'Green' was whitelisted the DataReader will still receive that Square. Upon noticing that the value does not pass the
"x < 100"filter, it modifies the filter to remove the whitelisting for "Green" so that future updates to the Green square are filtered by the DataWriter until they pass the"x<100"filter again.Note that DDS does not allow the ContentFilter expression to be modified, but the paramaters can. However there is no limitation on what is the "expression" and what are the "parameters" so you can create an expression that contains a paramater for the whitelist as in
"%1 OR x < 100". Then you can set%1to be initially the boolean value FALSE, and then set the paramater to be the whitelist"color = 'Green', etc.To implement this logic DataReader needs to remember which objects/instances it has already seen. It would be nice if it could rely on the DDS DataReader cache and the state flags for this. But unfortunately I do not think it is possible, so the applicationn will have to keep that list (or seach for that color in the current filter).
Also to detect that a particular instance/object no longer matches the filter you could use the trick or reading with a QueryCondition that is the complement of the filter (in this case
"x >= 100"). Anything you read here indicates that the object no longer passes the filter and you can remove it from the whitelist.I realize this is pretty cumbersome. Hopefully we will get this resoved in the spec soon...
Gerardo