Use Keyed Topics

Submitted by rose on Mon, 06/17/2013 - 14:50

For each data type, indicate to RTI Connext DDS the fields that uniquely identify the data object.

RTI Connext DDS allows you to represent individual real-world objects inside your Topics by specifying a set of key fields that form the unique identifiers of your data objects. The data objects you are representing are called instances.

Examples using keys:

  • If you are representing the positions of multiple vehicles, a vehicle ID or VIN will be the key for your data type.
  • If you are representing flight plans, the combination of the airline name and flight number will be the key for your data type.
  • If you are representing application state, an application unique ID will be the key for your data type.
  • If you are representing data coming from multiple sensors, the sensor ID will be the key for your data type.

While there are some valid scenarios in which a Topic should not have a key, these are the exception. Any time your data is describing more than one unique real-world object, you should specify keys. You can compare this with the primary key in a database that uniquely identifies an object – specifying a primary key allows you to represent a unique object within a table row.

The primary reason not to define a key is when the Topic represents a pure message and there is no natural way to view it as an update to a data object. This situation is rare – often when you think that you have data that does not represent an update to object state, you will see that there are objects being represented when you look more closely. For example, imagine an instant-messaging system where users or agents exchange messages (or a typical phone SMS system). In such a case, the key that defines the instance might be the identity of the message sender. Even though the messages do not strictly represent updates to data objects, you can treat the sender as a data object and receive the benefits described below.

One of the primary benefits of representing your data as unique instances is that RTI Connext Quality of Service policies and built-in services provide more value when your data is keyed:

  • Notifications of delayed object updates. RTI Connext DDS contains a built-in mechanism that notifies the application if an instance is not updated regularly. The notifications are configured using the DEADLINE QoS policy. If a data object is not updated within the DEADLINE period, the application is immediately notified with the object’s ID. If a key is not specified, this information is only available at the granularity of the Topic.  This means that without keys, as long as there has been some update of the Topic within the DEADLINE period, you will not receive a notification – regardless of whether you have delayed object updates.

  • History cache. RTI Connext DDS can keep a cache of the last set of updates (e.g., the last 10 changes) for each instance. This is configured on DataWriters and DataReaders, but is applied separately for each instance. This prevents a rapidly changing instance from “replacing” the last value of another instance that changes less frequently. This per-instance cache can also be maintained by the source application and the persistence service that initializes late-joining readers. This allows late-joining readers to be initialized with the current value of each object (or the last few changes that happened to each instance). Without keyed data, RTI Connext DDS cannot perform smart per-instance caching. This forces late joiners to process a larger history before getting to the current value of an instance.  This also allows the data bus itself to become the source of object data.  When you use keys and specify a history, the middleware now knows how much cache you need for each data-object, and it can differentiate between your data-objects.  With this information, it can become the source for your data-objects, even if your application fails or is shut down.

  • Ownership arbitration and failover management. RTI Connext DDS applications can use the OWNERSHIP QoS policy to specify that instances can only be updated exclusively by one writer, the owner of the instance. The owner of each instance is automatically managed by DDS based on the presence of DataWriters, their OWNERSHIP_STRENGTH value, and their ability to meet the QoS contract with the DataReader. DDS will automatically fail over to the highest-strength active DataWriter.   This allows redundancy and failover at the granularity of instances – if a DataWriter that is writing instances A and B fails, it can fail over gracefully without affecting the DataWriter that is writing instances C and D.

  • High-performance content-based filtering. RTI Connext DDS has the capability to filter data based on the content of the data update.  If the Topic has defined a set of key fields and the DataReaders are selecting the data of interest based only the value of the key fields, then the filtering can be done with high efficiency.  Additionally, this high-performance filtering based on key fields can be used to filter data from one data stream to multiple multicast channels using the MULTICHANNEL QoS.  This allows the network hardware to provide even higher filtering performance when DataReaders need a subset of the instance updates.

  • Integration with relational databases. Data distributed over RTI Connext DDS may originate or terminate in relational database tables. Individual records in a table are uniquely identified by the values of the fields marked as the “primary key” for that table. If those same fields are also marked as the key fields for the corresponding Topic, then the integration is seamless and the RTI Connext DDS data cache can work hand-in-hand with the database table storage.

  • Visualization in Excel and other table-oriented displays. Understanding and monitoring your system often requires visualization of your data in the DDS Global Data Space. It is typical to visualize this information using tabular displays, where each row represents the value of a specific data object. These displays can be updated live as applications publish updates to these objects, record object creations and deletions, provide historical views of the evolution of each data object, etc.  RTI Connext DDS’s ability to represent unique data-objects makes this possible with little or no programming. For example, with RTI’s Spreadsheet Add-in for Microsoft Excel, you can view live updates of data on the DDS bus.

  • Achieving consistency among observers of the Global Data Space. There are many scenarios where RTI Connext DDS is used to communicate and synchronize portions of a system’s state. An example of this state could be the current arrival and departure times of aircraft, the current locations of vehicles in a fleet of delivery trucks, or the current rotation rate of a wind turbine rotor. Many of these scenarios can have different sources contributing to different portions of the global state. Building a consistent state picture in these scenarios can be a very challenging problem. In the past, it required very complex application code. DDS can handle this problem via QoS policies such as RELIABILITY, DESTINATION_ORDER and OWNERSHIP. Using DDS mechanisms, even if the state observed by different consumers is temporarily different due to timing or short-term message loss, DDS ensures it eventually converges.  The middleware provides this by keeping track of which instance data has been delivered and in what order it should be given to the application.  However, this will only work correctly if the application has specified keys for the Topic – otherwise there are situations where portions of the state could be lost.

 

There are Very Few Reasons not to Use Keys

The main reasons why people do not define keys are because (1) they do not understand the implications, (2) they are concerned that using them would “lock” them into using DDS, as keys are not supported by other publish-subscribe middleware like JMS, or (3) they are afraid it will impact performance or resource usage.  

If you avoid using keys in order to get greater portability, you typically have to reimplement the same capabilities in your applications.   This incurs costs to develop, test, and maintain this logic. The application also cannot do this job as efficiently as the middleware, because it is insulated from the middeware’s protocol state.

Performance when using keys is not usually a significant concern. Extensive benchmarking on RTI Connext DDS shows that in almost all scenarios, the performance impact introduced by using keys is negligible. Latency differences are typically less than 5%. In most situations, the extra resources consumed by keys are very small and typically less than what the application would otherwise use itself to implement the equivalent functionality. However, if you expect to have millions of instances it may be advisable to benchmark the memory usage and performance in your system.

 
Tags: