.. include:: vars.rst .. _section-dataIntegration: ************************************************** Data Integration: Combining Different Data Domains ************************************************** In chapters :ref:`section-routingData` and :ref:`section-controllingData` we showed how |RS| is a powerful solution to scale and aggregate DDS systems. You can define data flows between publication and subscription |TOPICs|, and also perform stream processing using a custom |PROCESSOR|. Up to this point we have shown these capabilities only in the presence of DDS data sources and destinations. However, |RS| can provide the same capabilities for any other data technology and protocol through the concept of an |ADAPTER|, which makes |RS| a suitable framework for data integration. .. figure:: static/RouterDataIntegrationAdapter.svg :figwidth: 90 % :name: FigureRouterDataIntegrationAdapter :align: center Data Integration in |RS| An |ADAPTER| is a *pluggable* component that allows you to access any data domain pertaining to any technology. |ADAPTERs| provide a connection point to data domains so the information can flow back and forth to |RS|. The main |ADAPTER| interfaces are: - *Plugin*: Entry point to the custom implementation. It consists of a creation method that |RS| can call to instantiate the |ADAPTER| implementation. (see :ref:`section-Common-PluginManagement`). - |CONNECTION|: Entity responsible for *accessing* a concrete data domain. (see :ref:`section-CoreConcepts-Rm-Connection`). For example, a socket connection, database connection, or |DP|. The |CONNECTION| is the factory of |SR| and |SW|. - |SR|: Entity responsible for *reading* data streams from a concrete data domain and with a single |INPUT|. - |SW|: Entity responsible for *writing* data streams to a concrete data domain and associated with a single |OUTPUT|. :numref:`FigureRouterDataIntegrationAdapterConcept` illustrates the concept of the |ADAPTER| and how it fits within the |RS| entity model. .. figure:: static/RouterDataIntegrationAdapterConcept.svg :figwidth: 100 % :name: FigureRouterDataIntegrationAdapterConcept :align: center |ADAPTER| concept |RS| relies on concrete |ADAPTER| implementations to *read and write* data streams as part of the configured data flows. Similar to the |TR| object presented in :ref:`section-routingData-SingleTopic`, a |ROUTE| represents a generalization of a |TR| whose |INPUTs| and |OUTPUTs| can interact with any data domain. Each |INPUT| and |OUTPUT| are attached to a |CONNECTION|, which through the underlying |ADAPTER| connection entity creates appropriate |SRs| and |SWs|, respectively. These |SRs| and |SWs| provide read and write access to data streams, respectively. .. note:: All the following sections require you to be familiar with the routing concepts explained in :ref:`section-routingData`. We also recommended becoming familiar with :ref:`section-controllingData`. This section requires software programming knowledge in C/C++, understanding of CMake, and shared library management. Unified Data Representation =========================== |RS| architecture allows all the data-related components such as |ADAPTER|, |PROCESSOR|, and |TRANSF| to interoperate and coexist without knowing details of each other. |RS| achieves this by defining a unified data representation that all components are required to use. The unified data representation model is provided by |DD|, a concept presented in :ref:`section-controllingData-DynData`. |RS| imposes |DD| as the data interface for all the components that have to deal with data streams. This contract for the unified data representation is the key element that enables data integration in |RS|. Therefore, the main responsibility of an |ADAPTER| implementation is to provide a translation between the domain-specific data representation to |DD| and vice versa. .. figure:: static/RouterDataIntegrationDynDataStreams.svg :figwidth: 100 % :name: FigureRouterDataIntegrationDynDataStreams :align: center Unified Data Representation Model In :numref:`FigureRouterDataIntegrationDynDataStreams`, you can see all the data-related components interacting with each other independently of the domain-specific format of the data. All the data streams that flow across different components are presented as streams of |DD| objects. The following sections will guide you through an example that implements an |ADAPTER| that manipulates data from a file system. We will cover each step necessary to implement a custom |ADAPTER| and explain the purpose of each entity. .. _section-dataIntegration-FileBaseDomain: Integrating a File-Based Domain =============================== This section will guide you through an example of how to implement a custom |ADAPTER| to integrate with a non-DDS technology. The example shows how to feed data stored in a set of ``CSV`` files back and forth between a DDS domain. The file integration example is shown in :numref:`FigureRouterDataIntegrationFileAdapter`. .. figure:: static/RouterDataIntegrationFileAdapter.svg :figwidth: 100 % :name: FigureRouterDataIntegrationFileAdapter :align: center Example of data integration with a simple CSV file adapter The example requires the implementation of a custom *File* |ADAPTER|, which provides the ability to read and write from a set files and convert their content into a stream of |DD| samples. Let's review all the tasks you need to do to create a custom |ADAPTER| using the :ref:`section-Tutorials-ExampleFileAdapter`. You can run it first to see it in action, but you can also run one step at a time. We explain each method. Develop a Custom |ADAPTER_HEADING| ---------------------------------- As mentioned earlier, there are three main |ADAPTER| interfaces that must be implemented in order to provide access to, read, and write in a data domain. The most important step in designing a custom |ADAPTER| is to properly define the **mapping** between the adapter interfaces and specific entities or agents involved in the adapted data domain. For this example, the mapping is very simple and consists of the following: - ``FileConnection`` A simple factory class for `FileStreamReader` and `FileStreamWriter`. - ``FileStreamReader`` Reads data from a single file and converts it to |DD|. - ``FileStreamWriter`` Writes data to a single file after being converted. Both the ``FileStreamReader`` and ``FileStreamWriter`` process files in a custom and consistent ``CSV`` format. For simplicity, they also expect and understand the ``ShapeType`` only. To better understand how these implementations work, we will split the focus into two separate concepts: reading and writing. Implement a |SR_HEADING| for Reading Data ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Reading from a data domain is the responsibility of the |SR|. If you need to provide read access from your integrated data domain, you will need to implement this part of the |ADAPTER|, although it's optional. .. figure:: static/RouterDataIntegrationFileAdapterRead.svg :figwidth: 80 % :name: FigureRouterDataIntegrationFileAdapterRead :align: center Routing from the file adapter to DDS |SR_HEADING| Creation ''''''''''''''''''''' Creating |SRs| is the responsibility of the |CONNECTION|. Hence the |ADAPTER| connection interface has an abstract method to implement the creation of a |SR|. In this method you will find, among others, two important parameters: - Information about the *Stream* for which the |SR| is created. This parameter has type ``rti::routing::StreamInfo`` and contains: - *Stream* name: This is the name provided as part of the |INPUT| configuration in the ```` tag. - Type information: The registered name and ``TypeCode`` of the type of the input data stream. This information is encapsulated in a ``TypeInfo`` structure that contains: - ``type_name``` is the registered type name, as specified in the |INPUT| configuration in the ```` tag. - ``type_representation`` is the type definition as ``TypeCode``, obtained either from XML or from *Stream discovery*. You can learn more details about type registration in :ref:`section-Config-StreamPort-Types`. - A ``StreamReaderListener`` object to provide asynchronous notifications about data available to read. This is an object provided by the |RS| engine and the implementation can use it to signal the availability of input stream data and generate an event that's notified to the owner |ROUTE|. Read Operation '''''''''''''' ``FileStreamReader`` inherits from the ``rti::routing::adapter::DynamicDataStreamReader`` interface, which has different abstract method overload to read data. Which read operation version is called depends on the behavior of the ``Processor`` set in the parent |ROUTE|. The default forwarding |PROCESSOR| only calls the basic ``take()`` and is the one our example implements. When implementing a |SR|, there are two main tasks that require special attention: - **Providing an input stream of loaned** |DD| **samples**: All of the abstract read operations have two output parameters that shall hold the returned samples: list of user-data objects, and a list for info-data objects. The ``FileStreamReader::take()`` implementation reads one ``CSV`` text line at a time, parses each member, and converts it to a ``DynamicData`` object. In this case, the take operation can only read one sample at a time, and a heap-allocated ``DynamicData`` is provided as part of the output sample list. Note that ``FileStreamReader::return_loan()`` frees this heap-allocated object. The ``return_loan()`` operation is called automatically by the processor implementation when the sample loan from the take operation is no longer needed. Note that the take operation may also return a list of info-objects. These objects are meant to provide metadata associated with the user-data objects, such as reception timestamps or sequence numbers (which metadata is available depends on the data domain being adapted). Our example does not provide any metadata and hence the list is returned empty. - **Notifying** |RS| **about available data**: This is an important yet subtle step involved in the data processing pipeline. If you look at the ``Connection::create_stream_reader`` operation you will notice that one of the input parameters is an object of ``rti::routing::adapter::StreamReaderListener``. This object is provided by the |RS| engine and you can use it to indicate to |RS| about the existence of data available from the |SR|. When ``StreamReaderListener::on_data_available`` is called, it will trigger the generation of a ``DATA_ON_INPUTS`` event that will be dispatched to the |PROCESSOR| installed in the parent |ROUTE|. In our example implementation, the ``FileStreamReader`` *spawns a thread* that reads a text line from the file and notifies the ``StreamReaderListener`` right after, repeating this sequence in a loop until the whole file is read. Note that if we didn't notify the ``StreamReaderListener``, then the only way for |RS| to read data would be through a *periodic event* (see :ref:`section-ControllingData-Periodic`). Read vs. Take ''''''''''''' In the |SR| you will find that there are always two parallel operations with the same signature but different names: one called ``read()`` and one called ``take()``. Their behavior should be the same except for one main difference: ``take()`` will return samples from a |SR| only once, while ``read()`` allows the same samples to be returned more than once. In the DDS world, this is similar to the read and take operations of a |DR|. While the behavior is the same in both of them, the take operation will remove the samples from the |DR|'s cache (freeing space and preventing them from being read again), while the read will leave the cache intact, simply marking the samples with ``READ`` status. Implement a |SW_HEADING| for Writing Data ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |SW_HEADING| Creation ''''''''''''''''''''' Creating |SWs| is responsibility of of the |CONNECTION|. Hence the |ADAPTER| connection interface has an abstract method to implement the creation of a |SW|. In this method you will find, among others, an important parameter that identifies the *Stream* for which the |SR| is created. This parameter has type ``rti::routing::StreamInfo`` and its content and purpose are the same as explained in the reading section above. Write Implementation '''''''''''''''''''' Writing to a data domain is the responsibility of the |SW|. In our example, ``FileStreamWriter`` inherits from the ``rti::routing::adapter::DynamicDataStreamWriter`` interface, which has abstract methods to write data. Similar to the reading part, the write operation is called by the installed |PROCESSOR| of the parent |ROUTE|. The default |PROCESSOR| calls the write operation, passing the same samples read from the |INPUTs| belonging to the same parent |ROUTE|. .. figure:: static/RouterDataIntegrationFileAdapterWrite.svg :figwidth: 90 % :name: FigureRouterDataIntegrationFileAdapterWrite :align: center Routing from DDS to the file adapter The abstract write operation receives two input parameters: a list of user-data |DD| objects, and a list of info-data objects of type ``SampleInfo``. The info-data list may be empty if no such information is available, though if it's not, then it has the same size as the user-data objects (a 1:1 mapping between user-data and info-data objects). Our ``FileStreamWrite::write()`` implementation is as simple as iterating over the list of user-data objects and storing each of them in a file as a separate ``CSV`` text line. However, our example does not use the info-data list, though it could have used it to store, for example, the timestamps of the samples. .. note:: Implementing either the |SR| and |SW| is optional. You can implement only the side that you need, that is, reading or writing. .. seealso:: :link_router_sdk_api_cpp_s:`Adapter C++ API reference ` :link_router_sdk_api_cpp_s:`Processor Events ` Overview for the Processor API. :ref:`section-CoreConcepts-BuiltinPlugins-ForwardingProcessor` Details on the default forwarding |PROCESSOR| of the |TRs|. Create a Shared Library ----------------------- Once the |ADAPTER| implementation is finished, we need to create a *shared library* that |RS| can load. In this example we use ``CMake`` as the build system to create the shared library. We specify the generation of a library named ``fileadapter``: .. code-block:: C ... add_library( fileadapter "${CMAKE_CURRENT_SOURCE_DIR}/FileAdapter.cxx" "${CMAKE_CURRENT_SOURCE_DIR}/FileConnection.cxx" "${CMAKE_CURRENT_SOURCE_DIR}/FileInputDiscoveryStreamReader.cxx" "${CMAKE_CURRENT_SOURCE_DIR}/FileStreamReader.cxx" "${CMAKE_CURRENT_SOURCE_DIR}/FileStreamWriter.cxx" ) ... The generated library contains the compiled code of our implementation, contained in multiple ``.cxx`` files. A key aspect of the generated library is that it must export an external function that instantiates the ``FileAdapter``, and it is the function that |RS| will call to instantiate the |ADAPTER|. This external symbol is denoted *entry point* and you can declare it as follows: .. code-block:: C RTI_ADAPTER_PLUGIN_CREATE_FUNCTION_DECL(FileAdapter); The macro declares an external exported function with the following signature: .. code-block:: C struct RTI_RoutingServicAdapterPlugin* FileAdapter_create_adapter_plugin( const struct RTI_RoutingServiceProperties *, RTI_RoutingServiceEnvironment *); which is the signature |RS| requires and will assume for the entry point to create a custom |ADAPTER|. Note that the implementation of this function requires using the macro ``RTI_ADAPTER_PLUGIN_CREATE_FUNCTION_DEF`` in the source file. Define a Configuration that Integrates DDS with the File Adapter ---------------------------------------------------------------- This is similar to the process explained in :ref:`section-routingData-SingleTopic`, except that we will use a |CONNECTION| from the file adapter and only one |DP|. The example configuration file contains three different configurations that perform the integration in multiple combinations: **file to DDS, DDS to file, and file to file**. Note that all combinations could fit in a single |RS| configuration, but we chose this model to better explain the adapter capabilities. Below are the steps you need to follow. Configure a Plugin Library ^^^^^^^^^^^^^^^^^^^^^^^^^^ Within the root element of the XML configuration, you can define a *plugin library* element that contains the description of all the plugins that can be loaded by |RS|. In our case, we define a plugin library with a single entry for our *File Adapter* plugin: .. code-block:: xml fileadapter FileAdapter_create_adapter_plugin The values specified for the ``name`` attributes can be set at your convenience and they shall uniquely identify a plugin instance. We will use these names later within the |CONNECTION| to refer to this plugin. For the definition of our ADAPTER plugin, we have specified two elements: - ``dll`` is the name of the shared library we specified in the build step. We just provide the library name so |RS| will try to load it from the working directory, or assume that the library path is set accordingly. - ```` is the name of the entry point (external function) that creates the plugin object, exactly as we defined in code with the ``RTI_ADAPTER_PLUGIN_CREATE_FUNCTION_DECL`` macro. Once we have the plugin defined in the library, we can move to the next step and define a |CONNECTION| to the data domain of this plugin and the |ROUTE| for the data flows for reading and writing. .. warning:: When a name is specified in the element, |RS| will automatically append a **d** suffix when running the debug version of |RS|. .. seealso:: :ref:`section-Config-Plugins` Documentation about the ```` element. :ref:`section-Common-PluginManagement` For in-depth understanding of plugins. Define a |CONNECTION_HEADING| Linked to the Adapter ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ As mentioned before, the |CONNECTION| is the entity that enables access to a specific domain. To do so, the connection configuration shall refer to the |ADAPTER| plugin from which the underlying domain connection shall be created. In this example, the connection configuration is defined as follows: .. code-block:: xml There are three key elements that shall be set in a |CONNECTION| configuration: - ``name`` is the attribute that represents the name of the |CONNECTION| entity. You can choose any name you like that helps you identify the data domain. This name will be used later by the |INPUT| and |OUTPUT| configurations to indicate from which |CONNECTION| their underlying |SR| and |SW|, respectively, are created. In our case, we named it ``FileConnection``. - ``plugin_name`` is the attribute that must refer to the |ADAPTER| plugin from which the adapter connection is created. The value of this attribute must be the fully qualified name of the adapter plugin within the plugin library. The fully qualified name of the plugin is built using the values from the ``name`` attributes of the plugin library and plugin element. In our case, the fully qualified name of the file adapter plugin is given by ``AdapterLib::FileAdapter``. - ``register_type`` is an element tag that refers to a type definition (``TypeCode``) described in XML. This element has two attributes: ``name`` to uniquely identify and register a type, and ``type_ref`` to point to an existing type in XML providing its fully qualified name. This element can optionally appear as many times as needed. You will need to use this element if your adapter does not support discovery and |RS| cannot provide it through means of others adapters. Our file adapter example is quite basic. It only works with the ``ShapeType`` and it requires the definition to be available in XML (you can find it under the ```` section). Define the Data Flows that Read and Write from Your Adapter ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Once a |CONNECTION| to the adapted data domain is available, we need to define the |ROUTEs| (or |ARs|) that will indicate how data *streams* flow from inputs to outputs. |INPUTs| and |OUTPUTs| are ultimately the entities that hold |SRs| and |SWs| that perform the reading and writing. The file adapter example maps a separate ``CSV`` file for each *stream*. This allows us to nicely perform a 1:1 mapping between a DDS |TOPIC| and a file stream. In general, the expectation is that data that is read from an input's |SR| shall originate from a single input stream. Likewise, the data written to an output's |SW| shall be sent to a single stream. As mentioned at the beginning, this example provides three different |RS| configurations, each with a single |ROUTE| that defines the data flow for a specific combination. We will review each separately. .. _section-DataIntegration-FileAdapter-Reading: Routing from a File Stream to a DDS |TOPIC_HEADING| ''''''''''''''''''''''''''''''''''''''''''''''''''' For this case we define a |ROUTE| with: - An input attached to the file adapter (````). This requires setting the following elements: - ``connection`` is the attribute that specifies from which |CONNECTION| the underlying |SR| is created. This attribute shall refer to the name of the |CONNECTION| configuration exactly as it was set in its ``name`` attribute. - ```` is the *stream* name associated with this input. The impact of this value is specific to each |ADAPTER| implementation. - ```` indicates the associated type to the input stream. This ultimately translates into finding a ``TypeCode`` that matches this name and providing it on the |SR| creation as part of the ``StreamInfo``. In our case, this name matches the value in the ``name`` attribute of the ```` element in the connection configuration, so the type is the one defined in XML. - ```` is the adapter-specific configuration in the form of name-value pairs. This content is passed directly as a set of name-value string pairs on the creation of the |SR|. Our file |SR| receives the name of the ``CSV`` file from where data is read and a period at which the file is read. - An output attached to the built-in DDS adapter (````). This requires setting the following elements: - ``participant`` is the attribute that specifies from which |DP| the underlying |SW| is created. This attribute shall refer to the name of the |DP| configuration exactly as it was set in its ``name`` attribute. - ```` is the name of the |TOPIC| the underlying |DW| writes to. - ```` indicates the type associated with the |TOPIC|. This has the same behavior as for the input. The XML is shown below. .. code-block:: xml ON_DOMAIN_MATCH $(SHAPE_TOPIC) ShapeType example.adapter.input_file Input_$(SHAPE_TOPIC).csv example.adapter.sample_period_sec 1 ON_ROUTE_MATCH ShapeType $(SHAPE_TOPIC) .. _section-DataIntegration-FileAdapter-Writing: Routing from a DDS |TOPIC_HEADING| to a File Stream ''''''''''''''''''''''''''''''''''''''''''''''''''' For this case we define a |ROUTE| with: - An input attached to the built-in DDS adapter (````). This requires setting the following elements: - ``participant`` is the attribute that specifies from which |DP| the underlying |SR| is created. This attribute shall refer to the name of the |DP| configuration exactly as it was set in its ``name`` attribute. - ```` is the name of the |TOPIC| the underlying |DR| reads data from. - ```` indicates the type associated with the |TOPIC|. This has the same behavior as for the input. - An output attached to the file adapter (````). This requires setting the following elements: - ``connection`` is the attribute that specifies from which |CONNECTION| the underlying |SW| is created. This attribute shall refer to the name of the |CONNECTION| configuration exactly as it was set in its ``name`` attribute. - ```` is the *stream* name associated with this output. The impact of this value is specific to each |ADAPTER| implementation. - ````: indicates the associated type to the output stream. This ultimately translates into finding a ``TypeCode`` that matches this name and providing it on the |SW| creation as part of the ``StreamInfo``. In our case, this name matches the value in the ``name`` attribute of the ```` element in the connection configuration, so the type is the one defined in XML. - ```` is the adapter-specific configuration in the form of name-value pairs. This content is passed directly as a set of name-value string pairs on the creation of the |SW|. Our file |SW| receives the name of the ``CSV`` file where data is written. The XML is shown below. .. code-block:: xml ON_ROUTE_MATCH ShapeType $(SHAPE_TOPIC) ON_ROUTE_MATCH ShapeType $(SHAPE_TOPIC) example.adapter.output_file Output_$(SHAPE_TOPIC).csv .. _section-DataIntegration-FileAdapter-Both: Routing from a File Stream to Another File Stream ''''''''''''''''''''''''''''''''''''''''''''''''' This scenario represents a case where both the input and output are attached to the file |ADAPTER|. Hence, the routing path of this configuration generates a flow from file to file. This scenario demonstrates the flexibility and abstraction of |RS| working agnostically with data domains. For this case, the |ROUTE| configuration is defined with the same input configuration from :ref:`section-DataIntegration-FileAdapter-Reading` and the same output configuration from :ref:`section-DataIntegration-FileAdapter-Writing`. The XML is shown below. .. code-block:: xml ON_DOMAIN_MATCH $(SHAPE_TOPIC) ShapeType example.adapter.input_file Input_$(SHAPE_TOPIC).csv example.adapter.sample_period_sec 1 ON_ROUTE_MATCH ShapeType $(SHAPE_TOPIC) example.adapter.output_file Output_$(SHAPE_TOPIC).csv .. note:: In all configurations, the Stream and |TOPIC| names are set using the XML variable ``SHAPES_TOPIC``. Its purpose is to allow reusing the same configuration providing the actual desired name at run time. Another alternative is to use an |AR| instead (see :ref:`section-routingData-TopicGroup`). .. _section-DataIntegration-Discovery: Discovery Capabilities ====================== Besides allowing integration with application or user data, the |ADAPTER| interface also provides *data stream discovery* capabilities. Data communication frameworks may offer the ability to detect at run time which streams of user-data information are available and which endpoints (producers and consumers) are involved in the communication. Such is the case with DDS for example, which has a builtin discovery protocol to detect and notify applications of the presence of |TOPICs|. Discovery is very useful because it eliminates deployment configuration complexity and allows dynamic systems where endpoints come and go to function autonomously. |RS| can interoperate with discovery streams from any data domain through the |ADAPTER| by defining the concept of *Stream discovery*. *Stream* discovery refers to the ability to detect the presence of streams of information that carry user data. User-data streams are categorized as: - *publication or input streams*: These are data streams that originate from producer endpoints and from which |RS| receives data using |SRs|. An input stream is read-only. - *subscription or output streams*: These are data streams that originate from the consumer endpoints and to which |RS| sends data using |SWs|. An output stream is write-only. .. figure:: static/RouterDataIntegrationDiscovery.svg :figwidth: 100 % :name: FigureRouterDataIntegrationDiscovery :align: center Integration with discovery capabilities of data domains |RS| uses stream discovery for two main activities: - Detecting the **generation or disposal of streams** that trigger the filter matching with |ARs| (see :ref:`section-routingData-TopicGroup`) and the creation of |SRs| and |SWs| based on the input and output *creation modes* (see :ref:`section-Config-StreamPort-CreationMode`). - Receiving **information about the type of the samples** carried on the user-data streams. |RS| needs to obtain the *Stream* type information (``TypeCode``) beforehand in order to create the |SRs| and |SWs|. *Stream* discovery provides a channel for the reception of types. The discovery information in |RS| is represented in a unified way by defining a common type to describe *Stream* information: *StreamInfo*. Key information that a *StreamInfo* object provides: - *Stream* name: A unique identifier of a *stream* within a particular data domain connection (e.g., a |TOPIC| name in a DDS domain). - Alive or dispose: Whether or not the stream has any alive endpoints associated with it. - *TypeInfo*: Contains the unique identifier for the registration name of the type, as well as the type description as a ``TypeCode``. |RS| receives *StreamInfo* objects through the *Discovery* |SR| interfaces from the |ADAPTER|. Namely, there are two discovery |SRs| to read *StreamInfo* samples, one for each input and output stream. Implementation of discovery in the |ADAPTER| is optional. The |CONNECTION| is responsible for the provision of the *Discovery* |SRs| and its interface has two abstract methods to retrieve them. |RS| calls these operations upon enabling the parent |DR| (typically at startup) and will use the returned |SRs| (if any) to obtain the *StreamInfo* objects from them. |RS| has a dedicated *discovery thread* to call the read and return loan operations from all discovery |SRs|. The next section shows an example of how to provide discovery information using the file |ADAPTER|. .. _section-DataIntegration-Discovery-FileAdapter: Discovery in a File-Based Domain -------------------------------- When working with files on a file system, there are many ways in which discovery information can be useful. One of them is to provide notification about the creation and removal of files. Our file adapter example shows a basic way to recreate this. The file adapter example implements only the input stream *Discovery* |SR|. It provides information about which files are available to read and when the user |SRs| are done reading them. The class ``FileInputDiscoveryStreamReader`` inherits from the abstract class ``rti::routing::adapter::DiscoveryStreamReader`` and represents the implementation of the |SR| that provides discovery information about input streams. The implementation of this class is similar to the user-data |SR|. You will find that the abstract take operation is implemented by returning a list of ``rti::routing::StreamInfo`` objects. The implementation also uses an ``rti::routing::adapter::StreamReaderListener`` object to notify |RS| about discovery information that is available to read. The file ``FileInputDiscoveryStreamReader`` has two ways to generate ``StreamInfo`` objects: - On class instantiation, which in this case occurs when |RS| calls the ``FileConnection::get_input_stream_discovery_reader``. The constructors checks for the existence of ``CSV`` files containing the user data in hard-coded locations. - When user |SRs| obtain an end-of-file token, they call ``FileInputDiscoveryStreamReader::dispose``, which will generate a ``StreamInfo`` object marked as disposed for each finished file. The file adapter has basic code to illustrate how to implement the discovery functionality. More useful behavior could include providing continuous notifications about new files (hence new streams) to be read. It could also implement the *output Discovery* |SR| by also detecting when a file is placed in a directory as a signal to write data obtained from a peer input stream. Key Terms ========= .. glossary:: Data Integration The process of combining data from multiple and different sources for analysis, processing, or system integration purposes. Adapter Pluggable component that allows access to a custom data domain. Info Object A structure that contains metadata associated with the user-data object. In DDS, this is defined as ``SampleInfo``. Sample A structure composed of a data object and its associated info object. Loaned samples A list of samples returned by a |SR| for which a return loan operation is perform. Stream Discovery A mechanism through which |RS| detects the presence of user-data streams. StreamInfo A common data structure to represent discovery information across all data domains.