6. ContentFilteredTopics

Prerequisites

Time to complete

45 minutes

Concepts covered in this module

Content Filtering

In the Publish/Subscribe communication pattern, DataWriters send data to DataReaders with the same Topic. But sometimes a DataReader may be interested in only a subset of the data that is being sent. Some examples are:

  • A DataReader only cares about the temperature when it goes outside of a certain bound.

  • A DataReader only cares about log messages with levels WARNING or ERROR.

  • A DataReader only cares about the ChocolateLotState if the DataReader belongs to the next station that processes the chocolate lot.

Note that in all of these examples, there might be some DataReaders that want to receive a subset of data, and other DataReaders that might want to receive all data. For example, in our chocolate factory, an individual station only cares about a chocolate lot if it is the next station to process the lot, but the monitoring application wants to receive every update about every chocolate lot.

Connext offers a mechanism to allow DataReaders to filter data so that they only receive the data they are interested in. Applications can do this by creating ContentFilteredTopics instead of normal Topics. Since different DataReaders may want to receive a different subset of the data, ContentFilteredTopics are specified on the DataReaders. No changes are required on DataWriters to use content-filters: they continue to use a normal Topic, and they can communicate both with DataReaders that use a normal Topic and DataReaders that use a ContentFilteredTopic.

Although ContentFilteredTopics are specified on DataReaders, in most cases the DataWriter does the filtering, in which case no network bandwidth is used for those filtered samples. (The DataWriter finds out during discovery that a DataReader has a ContentFilteredTopic.)

Definition

A ContentFilteredTopic is a Topic with filtering properties. It makes it possible to subscribe to Topics and at the same time specify that you are only interested in a subset of the Topic’s data.

A ContentFilteredTopic consists of a Topic, a filter expression, and optional parameters. The filter expression and parameters can be changed at runtime. You can think of the filter expression as being like the “where” clause in SQL.

Content Filter

Figure 6.1 A ContentFilteredTopic that only allows samples with degrees > 32.

6.1. The Complete Chocolate Factory

ContentFilteredTopics are the final piece we need to put together the last part of the Chocolate Factory: the ingredient stations.

If you recall, the process the chocolate lot takes through the chocolate factory looks like this:

Ingredient Stations Using Content Filtering

Figure 6.2 DataReaders in our system will use content filtering to filter out any ChocolateLotState that doesn’t have “next_station” equal to that DataReader’s station.

The Monitoring/Control Application starts each lot by writing a sample to the ChocolateLotState Topic, with next_station set to one of the ingredient stations. Each ingredient station processes the lot by adding an ingredient, and then updates the next_station. The last station is the tempering machine, which processes the lot by tempering it, and then calls dispose() on the lot instance to say that the lot has completed.

Next Station

Figure 6.3 A chocolate lot is waiting, processing, or completed by an ingredient station. When a lot is completed, the next_station field indicates the next station to process the chocolate lot.

Last Station

Figure 6.4 A chocolate lot is waiting, processing, or disposed by the tempering application.

The Monitoring/Control Application wants to monitor every state of the chocolate lots as they are processed by the stations. However, an individual station only wants to know about a lot if the next_station field indicates it is the station that will process the lot next.

In the previous exercises, we had code in the ProcessLot function in TemperingApplication.cs to check whether the tempering station was the next station:

private void ProcessLot(
    DataReader<ChocolateLotState> lotStateReader,
    DataWriter<ChocolateLotState> lotStateWriter)
{
   using var samples = lotStateReader.Take();
   foreach (var sample in samples.ValidData())
   {
         if (sample.next_station == StationKind.TEMPERING_CONTROLLER)
         {
             ...
         }
   }
}

Instead of doing that, we can use a ContentFilteredTopic to filter out all the lot updates where the tempering controller is not the next_station. This cleans up the logic when the tempering application receives data, because it no longer needs to check whether a chocolate lot update is intended for it. This can also save network bandwidth because DataWriters perform the filtering. When the application uses a ContentFilteredTopic, Connext can filter out the data more efficiently than the application, and may not even send updates to an application that is not interested.

6.2. Hands-On 1: Update the ChocolateLotState DataReader with a ContentFilteredTopic

We are going to update the DataReader in the Tempering Application to use a ContentFilteredTopic instead of checking whether the data is important to this application every time a DataWriter updates the ChocolateLotState. Remember that the Tempering Application only cares about a lot if the next_station field indicates that the tempering machine is the next station.

  1. Find the examples as described in Clone Repository.

  2. Generate the C# code for chocolate_factory.idl as explained in Build the Applications.

  3. Create a ContentFilteredTopic that filters out ChocolateLotState data unless the next_station field in the data refers to this application.

    Open TemperingApplication.cs and look for the comment:

    // Exercise #1.1: Create a Content-Filtered Topic that filters out
    // chocolate lot state unless the next_station = TEMPERING_CONTROLLER
    

    Add the following code after the comment, so it looks like the following:

    // Exercise #1.1: Create a Content-Filtered Topic that filters out
    // chocolate lot state unless the next_station = TEMPERING_CONTROLLER
    ContentFilteredTopic<ChocolateLotState> filteredLotStateTopic =
          participant.CreateContentFilteredTopic(
             name: "FilteredLot",
             relatedTopic: lotStateTopic,
             filter: new Filter(
                expression: "next_station = %0",
                parameters: new string[] { "'TEMPERING_CONTROLLER'" }));
    

    This code creates a ContentFilteredTopic, using the ChocolateLotState Topic, that will filter out all chocolate lot state data that the Tempering Application does not care about.

  4. Use the ContentFilteredTopic instead of a plain Topic.

    In TemperingApplication.cs, look for the comment:

    // Exercise #1.2: Change the DataReader's Topic to use a
    // Content-Filtered Topic
    

    Replace these lines:

    DataReader<ChocolateLotState> lotStateReader = subscriber.CreateDataReader(
          lotStateTopic,
          ...);
    

    So the code looks like this (the highlighted line has changed):

    // Exercise #1.2: Change the DataReader's Topic to use a
    // Content-Filtered Topic
    DataReader<ChocolateLotState> lotStateReader = subscriber.CreateDataReader(
          filteredLotStateTopic,
          ...);
    

    Now the DataReader is using the ContentFilteredTopic. You don’t need to make any changes on the DataWriter side.

  5. Remove the code from the ProcessLot that checks that next_station is the Tempering Application.

    In TemperingApplication.cs, look for the comment:

    // Exercise #1.3: Remove the check that the Tempering Application is
    // the next_station. This will now be filtered automatically.
    

    Delete the following if condition:

    if (sample.next_station == StationKind.TEMPERING_CONTROLLER)
    

    So the code looks like this:

    foreach (var sample in samples.ValidData())
    {
       // Exercise #1.3: Remove the check that the Tempering Application is
       // the next_station. This will now be filtered automatically.
       Console.WriteLine("Processing lot " + sample.lot_id);
       ...
       Console.WriteLine("Lot completed");
    }
    

    Now Connext automatically filters out ChocolateLotState data before your application processes it.

6.3. Hands-On 2: Review the Temperature DataReader’s ContentFilteredTopic

We have already made a similar change as you made in Hands-On 1 to the Monitoring/Control Application’s Temperature DataReader: we added content filtering to it. That application now filters out data unless the temperature is outside of the expected range. (And it prints an error when the temperature is above or below that range.)

  1. In MonitoringCtrlApplication.cs, review the following code:

    // A Topic has a name and a datatype. Create a Topic with type
    // ChocolateLotState.  Topic name is a constant defined in the IDL file.
    Topic<ChocolateLotState> lotStateTopic =
          participant.CreateTopic<ChocolateLotState>("ChocolateLotState");
    // Add a Topic for Temperature to this application
    Topic<Temperature> temperatureTopic =
          participant.CreateTopic<Temperature>("ChocolateTemperature");
    ContentFilteredTopic<Temperature> filteredTemperatureTopic =
          participant.CreateContentFilteredTopic(
             name: "FilteredTemperature",
             relatedTopic: temperatureTopic,
             filter: new Filter(
                expression: "degrees > %0 or degrees < %1",
                parameters: new string[] { "32", "30" }));
    

    Notice that we have added a ContentFilteredTopic that will filter out temperature data unless it’s outside of the expected range. This uses a slightly more complex filter expression than the one you added to the Tempering Application in Hands-On 1: Update the ChocolateLotState DataReader with a ContentFilteredTopic. More information about the possible filter expressions can be found in SQL Filter Expression Notation, in the RTI Connext Core Libraries User’s Manual.

  2. You can see later in the code that the Temperature DataReader has been updated to use the ContentFilteredTopic:

    DataReader<Temperature> temperatureReader =
        subscriber.CreateDataReader(filteredTemperatureTopic, readerQos);
    
  3. The logic in the MonitorTemperature function no longer checks whether the temperature is out of range, because temperature data is only received when it’s out of range. Now the function looks like this:

    foreach (var data in samples.ValidData())
    {
          // Receive updates from tempering station about chocolate temperature.
          // Only an error if below 30 or over 32 degrees Fahrenheit.
          Console.WriteLine("Temperature high: " + data);
    }
    
  4. Build and run the applications.

    $ dotnet build
    

    Use the script to start all the applications at once:

    $ ./start_all.sh
    

    The script runs four copies of the Ingredient Application, each one specifying a different type of ingredient it is providing. The script also runs the Tempering Application and the Monitoring/Control Application. You can watch each lot get processed by the ingredient applications, and then the tempering machine.

    Applications Automatically Launched

    If you do not want to run the script, you can start up each of the applications by opening a new terminal for each application, and starting each application as follows:

    $ dotnet run -p IngredientApplication -- --station-kind COCOA_BUTTER_CONTROLLER
    
    $ dotnet run -p IngredientApplication -- --station-kind SUGAR_CONTROLLER
    
    $ dotnet run -p IngredientApplication -- --station-kind MILK_CONTROLLER
    
    $ dotnet run -p IngredientApplication -- --station-kind VANILLA_CONTROLLER
    
    $ dotnet run -p TemperingApplication
    
    $ dotnet run -p MonitoringCtrlApplication
    

6.4. Hands-On 3: Review the Larger System in Admin Console

We now have six applications that are publishing and subscribing to data, which is not a very large distributed system. This can start to look overwhelming, so we’re going to open Admin Console to get an overview of this system. This hands-on exercise does not relate directly to content filtering, but shows how multiple applications start to look in Admin Console.

  1. Make sure all of your applications are still running.

  2. Open Admin Console.

    • Open Admin Console from RTI Launcher.

    • Choose the Administration view:

      Administration View

      (The Administration view might be selected for you already.)

  3. In the DDS Logical View, click on the Domain in your system:

    Adminstrative View Domain
  4. Explore the chocolate factory system. Even though it looks complex with multiple applications running, you can navigate by clicking on the Topic name or the application. This makes it much easier to visualize the larger system.

    Applications in Admin Console

    Figure 6.5 Each Ingredient Application shows one DataReader and one DataWriter communicating on the ChocolateLotState Topic