.. include:: ../../../router.1.0/srcDoc/vars.rst .. _section-converter_configuration: Converter Configuration ======================= When you start *Converter*, you can specify a configuration file in XML format. In this file, you can specify the properties that control the behavior of the service. This section describes how to write a configuration file. How to Load the XML Configuration --------------------------------- *Converter* loads its XML configuration from multiple locations. Here are the various sources of configuration files, listed in load order: 1. ``[working directory]/USER_CONVERTER.xml`` This file is loaded automatically if it exists. 2. ``[NDDSHOME]/resource/xml/RTI_CONVERTER.xml`` This file is loaded automatically if it exists. 3. One or more files (semicolon-separated) specified using the command-line parameter -cfgFile. .. note:: ``[working directory]`` indicates the path to the current working directory from which you run *Converter*. ``[NDDSHOME]`` indicates the path to your *Connext DDS* installation. See :numref:`section-doc-paths`. You may use a combination of the above sources and load multiple configuration files. Here is an example configuration file.  You will learn the meaning of each line as you read the rest of this section. .. code-block:: xml XCDR_AUTO cdr_recording JSON_SQLITE converted rti_recorder_default.converted.dat 0 * rti/* XML Syntax and Validation ------------------------- The XML representation of DDS-related resources must follow these syntax rules: - It shall be a well-formed XML document according to the criteria defined in clause 2.1 of :link_xml_spec:`the Extensible Markup Language standard <>`. - It shall use UTF-8 character encoding for XML elements and values. - It shall use ```` as the root tag of every document. To validate the loaded configuration, *Converter* relies on an XSD file that describes the format of the XML content. We recommend including a reference to this document in the XML file that contains the service’s configuration; this document provides helpful features in code editors such as Visual Studio®, Eclipse®, and NetBeans®, including validation and auto-completion while you are editing the XML file. The XSD definitions of the XML elements are in ``$NDDSHOME/resources/schema/rti_converter.xsd``. To include a reference to the XSD document in your XML file, use the attribute ``xsi:noNamespaceSchemaLocation`` in the ```` tag. For example: .. code-block:: xml Builtin Configuration of Converter ---------------------------------- *Converter* comes pre-configured with a configuration that converts a file called rti_recorder_default.dat from XCDR format to a file called rti_recorder_default_converted.dat in JSON_SQLITE format. See :numref:`section-builtin-config-converter` for details. .. _section-convert-config-XML-tags: XML Tags for Configuring Converter ---------------------------------- This section describes the XML tags you can use in a *Converter* configuration file. :numref:`FigureConverterTopLevelTags` and :numref:`TableConverterTopLevelTags` describe the top-level tags allowed within the root ```` tag. .. figure:: ../static/ConverterTopLevelTags.png :alt: Top-Level Tags in Converter's Configuration File :name: FigureConverterTopLevelTags :align: center :figwidth: 50 % Top-level Tags in the *Converter* Configuration File .. list-table:: Top-Level Tags in Converter's Configuration file :name: TableConverterTopLevelTags :widths: 20 70 10 :header-rows: 1 * - Tags within :litrep:`` - Description - Multiplicity * - :litrep:`` - Defines types that can be used by *Converter*. This tag is needed if data types are not available in the discovery table. The type description is done using the *Connext DDS* XML format for type definitions. See :link_userman_types_in_xml:`Creating User Data Types with Extensible Markup Language (XML), in the RTI Connext DDS Core Libraries User's Manual <>`. - 0..* * - :litrep:`` - Contains a list of storage libraries that you may have implemented to support custom databases. For more information, see :numref:`section-Record-Config-Plugins` and :numref:`section-Replay-Config-Plugins`. - 0..* * - :litrep:`` - **Required**. |br| Specifies a *Converter* configuration. Attributes - ``name``: uniquely identifies a *Converter* configuration. **Required**. Example .. code-block:: xml See :numref:`section-convert-config-converter-tag`. - 1..* .. _section-convert-config-converter-tag: Converter Tag ------------- A configuration file must have at least one ```` tag. This tag is used to configure an execution of *Converter*. A configuration file may contain multiple ```` tags. When you start *Converter*, you can specify which ```` tag to use to configure the service using the ``-cfgName`` command-line parameter. This means one file can be used to configure multiple *Converter* executions. :numref:`FigureConverterTags` and :numref:`TableConverterTags` describe the tags allowed within a ```` tag. .. figure:: ../static/ConverterTags.png :alt: Converter Tags in Converter's Configuration File :name: FigureConverterTags :align: center :figwidth: 50 % Converter Tags in the *Converter* Configuration File .. list-table:: Converter Tags in Converter's Configuration File :name: TableConverterTags :widths: 20 50 10 :header-rows: 1 :class: longtable * - Tags within :litrep:`` - Description - Multiplicity * - :litrep:`` - Describes the storage that *Converter* will read as input. *Converter* expects the storage to be in CDR (binary) format. If storage is not specified, data will be stored in a SQLite file using a default name. See :numref:`section-convert-config-input-storage-tag`. - 0..1 * - :litrep:`` - Describes the storage that *Converter* will write as output. This output will be in JSON format. If this storage is not specified, data will be stored in a SQLite file using a default name. See :numref:`section-convert-config-output-storage-tag`. - 0..1 * - :litrep:`` - Enables selection of a subset of data to convert. Supports selecting data for conversion by time (or tagged time). See :numref:`section-converter_config_dataselection_tag`. - 0..1 * - :litrep:`` - Allows you to create one or more sessions in which to convert data, which gives you the option of specifying multiple threads for conversion. See :numref:`section-converter_config_session_tag`. Attributes: - ``name``: Uniquely defines a session. **Required**. - ``default_participant_ref``: Refers to a ``domain_participant`` tag, that specifies the domain ID to use from the database when converting this Topic. Children can override this by specifying their own participant. - 0..* * - :litrep:`` - **Required.** |br| Describes the domains *Converter* will select for conversion. *Converter* is not a DDS application so this definition will not create a DDS Domain Participant, it just works as a domain ID selection utility. |br| *Note*: the domain ID has to correspond to one of the domain IDs defined in the Recorder configuration. |br| See :numref:`section-converter_config_domainparticipant_tag`. Attributes: - ``name``: Uniquely defines a DomainParticipant. **Required**. - 1..* Example:  Specify a Configuration in XML ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: xml Starting *Converter* with the following command will use the ```` tag with the name "ConvertToJSON". .. code-block:: bash $NDDSHOME/bin/rticonverter -cfgFile file.xml -cfgName ConvertToJSON .. _section-convert-config-input-storage-tag: Input Storage ------------- The ```` tag allows you to configure the storage from which data will be read. You can choose between using the builtin SQLite storage or implementing your own storage plugin. .. list-table:: Input Storage Tags in Converter's Configuration File :name: TableInputStorageTag :widths: 20 70 10 :header-rows: 1 :class: longtable * - Tags within :litrep:`` - Description - Multiplicity * - :litrep:`` - Enables converting data from an SQLite database file. See :numref:`section-convert-config-sqlite-tag`. - 0..1 * - :litrep:`` - Enables converting data from a legacy version of *Recording Service* to a database usable by this version. See :numref:`section-converter_config_legacy_tag`. - 0..1 * - :litrep:`` - Enables converting data from storage using an external library that you specify. See :numref:`section-converter_config_plugin_tag`. - 0..1 * - :litrep:`` - Specifies the maximum number of samples to be returned by the read() operation. When the recorded tables contain huge amounts of samples, this setting can help improve the responsiveness of *Converter*, because the tool won't - for each table - attempt to get all samples at once. If not specified, the default value is 1024 samples. A negative value means an unlimited number of samples. - 0..1 .. _section-convert-config-output-storage-tag: Output Storage -------------- The ```` tag allows you to configure the storage into which converted data will be written. You can choose between using the builtin SQLite storage or implementing your own storage plugin. .. list-table:: Output Storage Tags in Converter's Configuration File :name: TableOutputStorageTag :widths: 20 70 10 :header-rows: 1 * - Tags within :litrep:`` - Description - Multiplicity * - :litrep:`` - Enables writing converted data into an SQLite database file. See :numref:`section-convert-config-sqlite-tag`. - 0..1 * - :litrep:`` - Enables writing converted data into storage of your choosing, using a plugin library that you specify. See :numref:`section-converter_config_plugin_tag`. - 0..1 .. _section-convert-config-sqlite-tag: SQLite ------ The ```` tag allows you to specify the name and file extension of a SQLite file to read data from, or write data to. Currently, the only storage format supported for input is CDR, and the only format supported for output is JSON. .. list-table:: SQLite Tags in Converter's Configuration File :name: TableConverterSqliteTag :widths: 30 60 10 :header-rows: 1 :class: longtable * - Tags within :litrep:`` - Description - Multiplicity * - :litrep:`` (output only) - Set of files to write to, and parameters for creating files and directories in that set. Used when creating more than one output file. See :numref:`section-convert-config-fileset-tag`. - 0..1 * - :litrep:`` (output only) - Name of file to write converted database file(s) to. Used when creating only a single output file. Default: rti_recorder_default_converted - 0..1 * - :litrep:`` (output only) - Suffix of the output file. Used when creating only a single output file. Default: dat - 0..1 * - :litrep:`` (input only) - Directory containing file(s) to convert. Default: Current working directory. - 0..1 * - :litrep:`` - Specifies what format the data is stored in.  The options are: - [Input Default] XCDR_AUTO: This is the binary format used by *Connext DDS* when sending data over the network. This has the highest performance for recording, but can only be viewed by using *Converter* to convert the data to a readable format, or by using *Replay* to replay the data. This will internally store data in XCDR or XCDR2 depending on the format received. - JSON_SQLITE: This format can be queried, but recording in this format has lower performance because data must be deserialized before it can be stored. - XCDR: The format to use when communicating with *Connext DDS* before 6.0.0. - XCDR2: More efficient than XCDR, used by *Connext DDS* 6.0.0 and later. - 0..1 * - :litrep:`` - Defines what to do when database files are already present in the current Converter execution directory. There are two options: - OVERWRITE: delete old files and replace them with newly created ones. - DO_NOT_TOUCH: do not delete any old files and just exit. Default: OVERWRITE - 0..1 * - :litrep:`` - Configures a SQL expression to use when establishing SQLite connections using this plugin. You can use this to add an index to a table to speed up replay, but this must be done for each table you want to index. Example: .. code-block:: xml DROP INDEX IF EXISTS pingtopic_0_by_src_timestamp; CREATE INDEX pingtopic_0_by_src_timestamp ON 'PingTopic@0' (SampleInfo_source_timestamp); Default: PRAGMA SYNCHRONOUS = OFF; PRAGMA JOURNAL_MODE = MEMORY; - 0..1 .. _section-convert-config-fileset-tag: Fileset ------- The ```` tag allows you to specify a set of files for *Converter* to write to. The behavior allowed by *Converter* is more limited than the behavior allowed by *Recording Service*. .. list-table:: Fileset Tags in Converter's Configuration File :name: TableConverterFilesetTag :widths: 20 70 10 :header-rows: 1 :class: longtable * - Tags within :litrep:`` - Description - Multiplicity * - :litrep:`` - **Required**. |br| Base directory where *Converter* should output files. Must be different from the input ``database_dir`` directory. Default: The current working directory - 1 * - :litrep:`` - **Required**. |br| The file name *Converter* will use for the generated user data file(s). This expression in combination with the rollover configuration will determine how many (and the names of) files *Converter* will output. This setting accepts text and any combination of the following parameters in it: - Autonumeric. **Format: %auto:M%.** This parameter describes an integer that auto-increments every time *Converter* has a rollover event due to time or file size. Example: .. code-block:: xml test_files_%auto:0%.dat This will create a series of files starting with test_files_0.dat, and incrementing each time *Converter* rolls over (due to the file size or time limits being reached). - Timestamp. **Format: %ts%.** This parameter will take the current timestamp in the system (the time represented as number of seconds since Epoch). - Time. **Format: %T%.** Current time expressed in ISO 8601 time format (HH:MM:SS). Example: 14:55:02. This parameter uses the strftime() parameter %T. - Short date. **Format: %F%.** Short date in YYYY-MM-DD format. Example: 2001-08-23. This parameter uses the strftime() parameter %F. **Note:** this parameter will not vary in 24 hours, so use with caution in combination with the rollover time limit feature (time limit should be greater than 1 day; otherwise, you may overwrite the same file continously). - Date and time. **Format: %c%.** Date and time representation, locale-dependent. This parameter uses the strftime() parameter %c. Default: rti_recorder_default_%auto:0%.dat (auto-numeric starting at zero, unlimited) - 1 * - :litrep:`` - Configuration for rolling over the file after a size limit is reached. See :numref:`section-convert-config-rollover-tag`. - 0..1 .. _section-convert-config-rollover-tag: Rollover -------- Rollover enables *Converter* to write to the next file in a set when a limit is reached. *Converter's* rollover functionality is decoupled from the files that are input, so many files may be merged into one, or one file may be split into many depending on *Converter's* rollover configuration. Currently *Converter* supports rolling over by size. .. list-table:: Rollover in Converter's Configuration File :name: TableConverterRolloverTag :widths: 20 50 10 :header-rows: 1 * - Tags within :litrep:`` - Description - Multiplicity * - :litrep:`` - The maximum allowed size for a file in a set. Units can be specified as an attribute. Attributes: - :litrep:`unit`: (Optional) The unit in which the size is expressed. The following values are allowed (Default: ``KILOBYTES``): - ``BYTES`` - ``KILOBYTES`` - ``MEGABYTES`` - ``GIGABYTES`` Example: .. code-block:: xml 500 - 0..1 .. _section-converter_config_legacy_tag: Legacy ------ This tag configures how legacy databases are converted to the newer format. (Note: You can continue to use the older format with *Replay*, by specifying "legacy" storage in *Replay's* configuration.) .. list-table:: Legacy in the Configuration File :name: TableConverterLegacyTag :widths: 20 50 10 :header-rows: 1 * - Tags within :litrep:`` - Description - Multiplicity * - :litrep:`` - The path to a legacy file to be converted. File set and version properties will be obtained automatically from the file itself. - 0..1 * - :litrep:`` - This tag allows you to link legacy domain names with domain IDs. See :numref:`section-converter_config_domainmapping_tag`. - 0..1 .. _section-converter_config_domainmapping_tag: Domain Mapping -------------- When converting a legacy database, the domain may not have been recorded. This tag provides a way to map a table with a domain ID. Later versions of the old Recorder database allowed you to use field filters. By default, the domain ID field was not recorded (it was filtered out by default). Thus, there is no information available in these legacy databases to relate the domain name used to record the data with a domain ID. This tag allows you to link legacy domain names with domain IDs. .. list-table:: Domain Mapping in the Configuration File :name: TableConverterDomainMappingTag :widths: 20 50 10 :header-rows: 1 * - Tags within :litrep:`` - Description - Multiplicity * - :litrep:`` - **Required**. |br| A link between a recorded legacy domain name and a domain ID. Attributes: - ``legacy_domain_name``: The recorded domain name specified in the old Recording Service domain tag, for example: . - ``domain_id``: The domain ID you want to associate with the legacy domain name. - 1..* .. _section-converter_config_plugin_tag: Plugin ------ This tag enables you to convert data using an external library that you specify. .. list-table:: Storage plugin Tag in the Configuration File :name: ConverterTablePluginTag :widths: 20 70 10 :header-rows: 1 * - Tags within :litrep:`` - Description - Multiplicity * - :litrep:`` - Name/value pairs of properties to pass to a storage plugin. - 0 or 1 .. _section-converter_config_dataselection_tag: Data Selection -------------- This tag selects what data to convert from the database, based on a time range. .. list-table:: Data Selection Tags in Converter's Configuration File :name: TableConverterDataSelectionTag :widths: 20 50 10 :header-rows: 1 * - Tags within :litrep:`` - Description - Multiplicity * - :litrep:`` - Select data to convert from the database, based on start and end times. See :numref:`section-converter_config_timerange_tag`. - 0..1 .. _section-converter_config_timerange_tag: Time Range ---------- The ```` tag allows you to specify the begin and end times of the data you want to convert. This can be specified either as timestamps or as symbolic timestamps called "timestamp tags." Tags may have been added to the recording through remote administration, and can be viewed using the script ``rtirecordingservice_list_tags`` (see :numref:`section-record-list-tags`). .. list-table:: Time Range Tags in Converter's Configuration File :name: TableConverterTimeRangeTag :widths: 20 50 10 :header-rows: 1 :class: longtable * - Tags within :litrep:`` - Description - Multiplicity * - :litrep:`` - Select data to convert from the database, with a timestamp beginning at this time. Specify the time in seconds and nanoseconds. |br| Default: 0, start at beginning of the file. - 0..1 * - :litrep:`` - Can be used instead of begin_time. Select data to start converting from the database by specifying the string name of a timestamp tag. Timestamp tags associate a timestamp with a name. Then you can refer to a timestamp by that name. The timestamp must have been tagged with that name during recording. See :numref:`section-record-tags`. |br| Default: Start at beginning of the file. - 0..1 * - :litrep:`` - Select data to convert from the database, with a timestamp ending at this time. Specify the time in seconds and nanoseconds. |br| Default: End at end of file. - 0..1 * - :litrep:`` - Can be used instead of end_time. Select when to stop converting data by specifying the string name of a timestamp tag. Timestamp tags associate a timestamp with a name. Then you can refer to a timestamp by that name. The timestamp must have been tagged with that name during recording. See :numref:`section-record-tags`. |br| Default: Stop at end of file. - 0..1 .. _section-converter_config_domainparticipant_tag: DomainParticipant ----------------- The ```` tag allows you to specify the domain IDs you want to convert from the database. Specify a name for the DomainParticipant, and the domain_id you want to convert. Associate the DomainParticipant name with a session, topic or topic_group that you want to convert, to convert data within a domain. This does not create a DDS DomainParticipant. .. list-table:: DomainParticipant Tags in Converter's Configuration File :name: TableDomainParticipantConverterTag :widths: 20 50 10 :header-rows: 1 * - Tags within :litrep:`` - Description - Multiplicity * - :litrep:`` - The domain ID of tables loaded from the database. This is used to determine which domain IDs to convert. - 1 * - :litrep:`` - Registers a type name and associates it with a type representation. When you define a type in the configuration file, you have to register the type in order to use it in a ````. Attributes: - ``name``: Name that the data type is registered with if no :litrep:`` is specified. The same data type may be registered with different names. **Required**. - ``type_ref``: Definition of this data type. It must refer to one of the defined types in the ```` section by specifying the fully qualified name. Tags within this tag: - :litrep:``: Name the data type is registered with. The same data type may be registered with different names. Not required. - 0..* .. _section-converter_config_session_tag: Session ------- The ```` tag configures the threads that will be used to convert data.  You also specify the Topics and groups of Topics to convert inside the tag. .. list-table:: Session Tags in Converter's Configuration File :name: TableConverterSessionTag :widths: 20 50 10 :header-rows: 1 :class: longtable * - Tags within :litrep:`` - Description - Multiplicity * - :litrep:`` - Specifies the number of threads used by this session, and the settings used when creating them. - 0..1 * - :litrep:`` - Specifies an individual Topic to convert. See :numref:`section-converter_config_topic_tag`. Attributes: - ``name``: The name of the Topic to convert. - ``participant_ref``: Refers to a ``domain_participant`` tag, that specifies the domain ID to use from the database when converting this Topic. If the parent ```` specifies a ``default_participant_ref``, this attribute is optional. - 0..* * - :litrep:`` - Specifies a group of Topics to convert. See :numref:`section-converter_config_topicgroup_tag`. Attributes: - ``name``: The name of the topic group. - ``participant_ref``: Refers to a ``domain_participant`` tag, that specifies the domain ID to use from the database when converting this Topic. If the parent ```` specifies a ``default_participant_ref``, this attribute is optional. - 0..* .. _section-converter_config_topicgroup_tag: Topic Group ----------- The ```` tag allows you to convert a group of Topics, using regular expressions to describe which Topics to convert. .. list-table:: Topic Group Tags in Converter's Configuration File :name: ConverterTableTopicGroupTag :widths: 20 70 10 :header-rows: 1 * - Tags within :litrep:`` - Description - Multiplicity * - :litrep:`` - A regular expression (fnmatch) describing which Topics should be converted. You may use a comma-separated list to specify more than one filter. |br| Example: .. code-block:: xml CONTROL_*,DATA_* - 0..1 * - :litrep:`` - A regular expression (FNMATCH) describing which Topics should not be converted. This is applied after the ``allow_topic_name_filter``. You may use a comma-separated list to specify more than one filter. - 0..1 .. _section-converter_config_topic_tag: Topic ----- The ```` tag specifies an individual Topic to convert. .. list-table:: Topic Tags in Converter's Configuration File :name: ConverterTableTopicTag :widths: 20 70 10 :header-rows: 1 :class: longtable * - Tags within :litrep:`` - Description - Multiplicity * - :litrep:`` - The name of the DDS topic to be converted. If this tag is not present, the name attribute of the `` will be used. |br| *Note*: we recommend using this tag to define the topic name. There may be characters that cause the XML validation to fail if they are part of the topic name attribute. Also, the '/' character and '::' separator may cause Converter to fail when found in the topic name attribute. - 0..1 * - :litrep:`` - The name of the data type that will be converted for this topic. **Required**. - 1 * - :litrep:`` - The transformation library to be applied to this Topic's data when converting. This is a user library that can modify the data after it is received from input storage and before it is sent to output storage. Transformations implement APIs identical to *Routing Service's* transformations. For more on using transformations, see these sections in the *RTI Routing Service User's Manual*: - :link_router_manual_config_transformation:`Data Transformation <>` - :link_router_manual_tutorials:`Tutorials <>` Attributes: - ``plugin_name``: The name of the plugin to load, qualified by the plugin library name. Example: .. code-block:: xml ModifyTestID_create modify_test_id_library - 0..1 .. _section-builtin-config-converter: Converter's Builtin Configuration Details ----------------------------------------- *Converter* comes with a builtin configuration, which selects the name of an input file to convert from and an output file to convert to. The 'defaultToJson' configuration specifies JSON SQLite format by default as the output format, with 'rti_recorder_default.dat' as the input file and 'rti_recorder_default_converted.dat' as the output file. It expects to find the file(s) to convert in a directory called 'cdr_recording'. This is the default directory when recording in XCDR format in the default Recorder configuration. .. code-block:: xml XCDR_AUTO cdr_recording JSON_SQLITE converted rti_recorder_default.converted.dat 0