Use content filter on Recording service

4 posts / 0 new
Last post
Offline
Last seen: 2 months 3 weeks ago
Joined: 04/13/2023
Posts: 4
Use content filter on Recording service

I'm using recording service with RTI DDS Connext 6.1.2 to capture data of type KeyedString on the same topic name ('MOVEMENT_LINE') 

The key is a normal string, and the value is an xml content. 

Bellow example of data where some messages are used as responses and others as requests: 

 

[

  {

    "key": "2b44272e-694c-4666-a3f3-0ad125812042",

    "value": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\n<movementLineResponse creationTime=\"2024-04-26T11:46:31.315+02:00\" messageId=\"2b44272e-694c-4666-a3f3-0ad125812042\"  xmlns=\"http://www.dummy.com/ABC/mvtResponse/1\">...</movementLineResponse>..."

  },

  {

    "key": "8b30d34f-818c-4e3f-bc15-0621ea51fcd5",

    "value": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?><movementLine version=\"6\" creationTime=\"2024-04-26T11:47:08.528+02:00\" messageId=\"8b30d34f-818c-4e3f-bc15-0621ea51fcd5\" xmlns=\"http://www.dummy.com/ABC/mvt/1\">...</movementLine>..."

  },

 {

    "key": "RESPONSE",

    "value": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\n<movementLineResponse creationTime=\"2024-04-26T11:46:31.315+02:00\" messageId=\"2b44272e-694c-4666-a3f3-0ad125812042\"  xmlns=\"http://www.dummy.com/ABC/mvtResponse/1\">...</movementLineResponse>..."

  },

  {

    "key": "REQUEST",

    "value": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?><movementLine version=\"6\" creationTime=\"2024-04-26T11:47:08.528+02:00\" messageId=\"8b30d34f-818c-4e3f-bc15-0621ea51fcd5\" xmlns=\"http://www.dummy.com/ABC/mvt/1\">...</movementLine>..."

  }

]

 

The meaningful difference between the data is only available in the value so the only way to capture the request is by filtering on the data example the xmlns of the request is http://www.dummy.com/ABC/mvt/1 or the request does not contain movementLineResponse tag." 

 My issue is I never succeeded to filter on request data. Here after the config filter, I'm using: 

 I wanted to test it on filtering the key that  match 'REQUEST', I tested it with  builtin.sql and stringMatch kinds  and with LIKE keyword  or  sign equal (=) and never succeeded:

 

<dds xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="file:/opt/rti_connext_dds/resource/schema/rti_recording_service.xsd">

<types>
<include file="./builtinTypes.xml" />
</types>

...

<session name="DefaultSession">
<topic_group name="Recorder_part1" participant_ref="part1">
<allow_topic_name_filter>MOVEMENT_LINE</allow_topic_name_filter>
<deny_topic_name_filter>rti/*</deny_topic_name_filter>
<datareader_qos base_name="QosLibrary::StrictReliable"/>
<content_filter name="MVT_LINE_filter" kind="builtin.stringMatch">
<expression>key = 'REQUEST'</expression>
</content_filter>
</topic_group>

</session>
</recording_service>

</dds>

 

Also, I tested on value field  and it never worked for me. 

The type definition of keyedString is : 

 

<?xml version="1.0" encoding="UTF-8"?>
<types xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="file:/opt/rti_connext_dds/resource/app/app_support/rtiddsgen/schema/rti_dds_topic_types.xsd">
<module name="DDS">
<struct name="KeyedString">
<member name="key" type="string" key="true" stringMaxLength="1024" />
<member name="value" type="string" stringMaxLength="2147483647" />
</struct>
</module>

 

</types>

For filtering I used the following syntax: 

 

 <content_filter name="MVT_LINE_filter" kind="builtin.stringMatch">

 

<expression>key = %0</expression>
<expression_parameters>
<element>REQUEST</element>
</expression_parameters>
</content_filter>

 

 I don't know if my config is correct or not because on all circumstances the application starts without issue and the problem is that I can't see in rtiadminconsole the reader of the recording service so I can investigate further the content filter. 

Can you provide any tips or help on this topic?

  

 

 

 

 

 

 

 

 

 

 
 
 
 
 
Howard's picture
Offline
Last seen: 11 hours 47 min ago
Joined: 11/29/2012
Posts: 584

So, you're running to a number of issues including what I think may be a bug in RTI Recording Service.

1) there may be (to be confirmed) a bug preventing the use of content-filters with <topic_group>.  Instead, you will need to use <topic> and apply content-filters on a per-topic basis

So, this doesn't work (even though it's supposed to)

             <topic_groupname="RecordAll"participant_ref="Participant0">
                <allow_topic_name_filter>*</allow_topic_name_filter>
                <deny_topic_name_filter>rti/*</deny_topic_name_filter>
                <content_filtername="aaa"kind="builtin.stringMatch">
                    <expression>color MATCH 'RED'</expression>
                </content_filter>
            </topic_group> 
 

but this DOES work

       <topic name="Square" participant_ref="Participant0">
                <registered_type_name> ShapeType </registered_type_name>
                <content_filter name="aaa" kind="builtin.stringMatch">
                    <expression>color MATCH 'RED'</expression>
                </content_filter>
       </topic>
 
2) NOTE if you are using the "builtin.stringMatch" filter, you should use the "MATCH" operator.  Not the "=" operator.
 
3) if you do use the "builtin.sql" filter, and want to use ">", "<" operators, in XML you have to use the escaped versions of those operators as mentioned in the documentation.
 
4) When specifying string literals, you need to include the single quote, ', as a part of the literal.  e.g., 'RED", or in your case 'REQUEST', not just REQUEST
 
 
So change to use <topic> and things should work.  A bug will be filed about supporting content-filters in <topic_group>.
 
 
And finally, in RTI Admin Console, if you want the Admin Console to display all of the "internal" DDS topics created by RTI Services, you can goto the menu "View" and uncheck the option "Hide all RTI internal Topics".  This will enable Admin Console to show the DataReaders created by RTI Recording Service, so you can select them and inspect the QoS with which they were created (including content-filter if any).
 
 
Offline
Last seen: 2 months 3 weeks ago
Joined: 04/13/2023
Posts: 4

Hello  Howard, 

Your recomendation helped me and I can make it work and  filter my data.

The problem I have right know is when the field  contains the special character '/', the filtering never work. 

Example of data: 

key: "/t REQUEST /n" or " REQUEST /"

The filter expression I used is:  <expression>key MATCH '*REQUEST*' </expression>. 

I tested it with stringMatch and sql filter kind and it gives the same results. 

So when the field contains this special caracter, the filter always evalued to false and the data was rejected. 

 

 

 

Offline
Last seen: 2 months 1 week ago
Joined: 02/20/2020
Posts: 6

Hi Saber,

Although we have created a dedicated support case for this with you, I track this here too for completness.
The behavior with the '/' is due to the FNM_PATHNAME flag we use for fnmatch. With it, the '/' won't match the '*' wildcard expression.

That means that for your filter expression you need to know the number of '/' that are before and after the substring you want to match. For example for your keys: "/t REQUEST /n" or " REQUEST /" there is not a single filter expression that match both. You will require two: key MATCH '*/*REQUEST*/*' or key MATCH '*REQUEST*/*'

I have created a request for improvement for not using the FNM_PATHNAME flag in Content Filters. Keep the ID for future references: CORE-14725.