5.2. Converter Configuration

When you start Converter, you can specify a configuration file in XML format. In this file, you can specify the properties that control the behavior of the service. This section describes how to write a configuration file.

5.2.1. How to Load the XML Configuration

Converter loads its XML configuration from multiple locations. Here are the various sources of configuration files, listed in load order:

  1. [working directory]/USER_CONVERTER.xml This file is loaded automatically if it exists.
  2. [NDDSHOME]/resource/xml/RTI_CONVERTER.xml This file is loaded automatically if it exists.
  3. File specified using the command-line parameter -cfgFile.

Note

[working directory] indicates the path to the current working directory from which you run Converter.

[NDDSHOME] indicates the path to your Connext DDS installation. See Section 1.3.

You may use a combination of the above sources and load multiple configuration files.

Here is an example configuration file.  You will learn the meaning of each line as you read the rest of this section.

<?xml version="1.0" encoding="UTF-8"?>

<dds xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="https://community.rti.com/schema/6.0.0/rti_converter.xsd">

    <!-- Available types -->
    <types />

    <!--  A definition of a Converter instance to run -->
    <converter name="defaultToJson">
        <!--  Input storage settings -->
        <input_storage>
            <sqlite>
                <storage_format>XCDR_AUTO</storage_format>
                <database_dir>cdr_recording</database_dir>
            </sqlite>
        </input_storage>

        <!--  Output storage settings -->
        <output_storage>
            <sqlite>
                <storage_format>JSON_SQLITE</storage_format>
                <fileset>
                    <workspace_dir>converted</workspace_dir>
                    <filename_expression>rti_recorder_default.converted.dat</filename_expression>
                </fileset>
            </sqlite>
        </output_storage>

        <!--  Domain selection: assume 0 by default.
              Converter is not a DDS application, so this won't create a
              DDS Domain Participant, but allows you to select which domains
              to convert -->
        <domain_participant name="Domain0">
            <domain_id>0</domain_id>
        </domain_participant>

        <session name="DefaultSession"
                 default_participant_ref="Domain0">
            <!-- Topics to convert in this session -->
            <topic_group name="DefaultTopicGroup">
                <!-- Rules describing the topics to convert -->
                <allow_topic_name_filter>*</allow_topic_name_filter>
                <deny_topic_name_filter>rti/*</deny_topic_name_filter>
            </topic_group>
        </session>

    </converter>
</dds>

5.2.2. XML Syntax and Validation

The XML representation of DDS-related resources must follow these syntax rules:

  • It shall be a well-formed XML document according to the criteria defined in clause 2.1 of the Extensible Markup Language standard.
  • It shall use UTF-8 character encoding for XML elements and values.
  • It shall use <dds> as the root tag of every document.

To validate the loaded configuration, Converter relies on an XSD file that describes the format of the XML content. We recommend including a reference to this document in the XML file that contains the service’s configuration; this document provides helpful features in code editors such as Visual Studio®, Eclipse®, and NetBeans®, including validation and auto-completion while you are editing the XML file.

The XSD definitions of the XML elements are in $NDDSHOME/resources/schema/rti_converter.xsd.

To include a reference to the XSD document in your XML file, use the attribute xsi:noNamespaceSchemaLocation in the <dds> tag. For example:

<dds xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../schema/rti_converter.xsd">
    <!-- ... -->
</dds>

5.2.3. Builtin Configuration of Converter

Converter comes pre-configured with a configuration that converts a file called rti_recorder_default.dat from XCDR format to a file called rti_recorder_default_converted.dat in JSON_SQLITE format. See Section 5.2.20 for details.

5.2.4. XML Tags for Configuring Converter

This section describes the XML tags you can use in a Converter configuration file. Figure 5.1 and Table 5.2 describe the top-level tags allowed within the root <dds> tag.

Top-Level Tags in Converter's Configuration File

Figure 5.1 Top-level Tags in the Converter Configuration File

Table 5.2 Top-Level Tags in Converter’s Configuration file
Tags within <dds> Description Multiplicity
<types> Defines types that can be used by Converter. This tag is needed if data types are not available in the discovery table. The type description is done using the Connext DDS XML format for type definitions. See Creating User Data Types with Extensible Markup Language (XML), in the RTI Connext DDS Core Libraries User’s Manual. 0..*
<plugin_library> Contains a list of storage libraries that you may have implemented to support custom databases. For more information, see Section 3.5.4. 0..*
<converter>

Required.
Specifies a Converter configuration.

Attributes
  • name: uniquely identifies a Converter configuration. Required.
Example
<converter name="ConvertAll">
</converter>

See Section 5.2.5.

 1..*

5.2.5. Converter Tag

A configuration file must have at least one <converter> tag. This tag is used to configure an execution of Converter.

A configuration file may contain multiple <converter> tags. When you start Converter, you can specify which <converter> tag to use to configure the service using the -cfgName command-line parameter. This means one file can be used to configure multiple Converter executions.

Figure 5.2 and Table 5.3 describe the tags allowed within a <converter> tag.

Converter Tags in Converter's Configuration File

Figure 5.2 Converter Tags in the Converter Configuration File

Table 5.3 Converter Tags in Converter’s Configuration File
Tags within <converter> Description Multiplicity
<input_storage>

Describes the storage that Converter will read as input. Converter expects the storage to be in CDR (binary) format.

If storage is not specified, data will be stored in a SQLite file using a default name.

See Section 5.2.6.

 0..1
<output_storage>

Describes the storage that Converter will write as output. This output will be in JSON format.

If this storage is not specified, data will be stored in a SQLite file using a default name.

See Section 5.2.7.

 0..1
<data_selection> Enables selection of a subset of data to convert. Supports selecting data for conversion by time (or tagged time). See Section 5.2.14. 0..1
<session>

Allows you to create one or more sessions in which to convert data, which gives you the option of specifying multiple threads for conversion. See Section 5.2.17.

Attributes:
  • name: Uniquely defines a session. Required.
  • default_participant_ref: Refers to a domain_participant tag, that specifies the domain ID to use from the database when converting this Topic. Children can override this by specifying their own participant.
 0..*
<domain_participant>

Required.
Describes the domains Converter will select for conversion. Converter is not a DDS application so this definition will not create a DDS Domain Participant, it just works as a domain ID selection utility.
Note: the domain ID has to correspond to one of the domain IDs defined in the Recorder configuration.
See Section 5.2.16.

Attributes:
  • name: Uniquely defines a DomainParticipant. Required.
 1..*

5.2.5.1. Example:  Specify a Configuration in XML

<dds>
    <converter name="ConvertToJSON">
        <!-- ... Required entities -->
    </converter>
</dds>

Starting Converter with the following command will use the <converter> tag with the name “ConvertToJSON”.

$NDDSHOME/bin/rticonverter -cfgFile file.xml -cfgName ConvertToJSON

5.2.6. Input Storage

The <input_storage> tag allows you to configure the storage from which data will be read. You can choose between using the builtin SQLite storage or implementing your own storage plugin.

Table 5.4 Input Storage Tags in Converter’s Configuration File
Tags within <input_storage> Description Multiplicity
<sqlite> Enables converting data from an SQLite database file. See Section 5.2.8. 0..1
<legacy> Enables converting data from a legacy version of Recording Service to a database usable by this version. See Section 5.2.11. 0..1
<plugin> Enables converting data from storage using an external library that you specify. See Section 5.2.13. 0..1
<max_samples_per_read>

Specifies the maximum number of samples to be returned by the read() operation.

When the recorded tables contain huge amounts of samples, this setting can help improve the responsiveness of Converter, because the tool won’t - for each table - attempt to get all samples at once.

If not specified, the default value is 1024 samples.

A negative value means an unlimited number of samples.

 0..1

5.2.7. Output Storage

The <output_storage> tag allows you to configure the storage into which converted data will be written. You can choose between using the builtin SQLite storage or implementing your own storage plugin.

Table 5.5 Output Storage Tags in Converter’s Configuration File
Tags within <output_storage> Description Multiplicity
<sqlite> Enables writing converted data into an SQLite database file. See Section 5.2.8. 0..1
<plugin> Enables writing converted data into storage of your choosing, using a plugin library that you specify. See Section 5.2.13. 0..1

5.2.8. SQLite

The <sqlite> tag allows you to specify the name and file extension of a SQLite file to read data from, or write data to. Currently, the only storage format supported for input is CDR, and the only format supported for output is JSON.

Table 5.6 SQLite Tags in Converter’s Configuration File
Tags within <sqlite> Description Multiplicity
<fileset> (output only) Set of files to write to, and parameters for creating files and directories in that set. Used when creating more than one output file. See Section 5.2.9. 0..1
<file> (output only)

Name of file to write converted database file(s) to. Used when creating only a single output file.

Default: rti_recorder_default_converted

 0..1
<file_suffix> (output only)

Suffix of the output file. Used when creating only a single output file.

Default: dat

 0..1
<database_dir> (input only)

Directory containing file(s) to convert.

Default: Current working directory.

 0..1
<storage_format>

Specifies what format the data is stored in.  The options are:

  • [Input Default] XCDR_AUTO: This is the binary format used by Connext DDS when sending data over the network. This has the highest performance for recording, but can only be viewed by using Converter to convert the data to a readable format, or by using Replay to replay the data. This will internally store data in XCDR or XCDR2 depending on the format received.
  • JSON_SQLITE: This format can be queried, but recording in this format has lower performance because data must be deserialized before it can be stored.
  • XCDR: The format to use when communicating with Connext DDS before 6.0.0.
  • XCDR2: More efficient than XCDR, used by Connext DDS 6.0.0 and later.
 0..1
<overwrite_policy_kind>

Defines what to do when database files are already present in the current Converter execution directory.

There are two options:
  • OVERWRITE: delete old files and replace them with newly created ones.
  • DO_NOT_TOUCH: do not delete any old files and just exit.

Default: OVERWRITE

 0..1
<sql_initialization_string>

Configures a SQL expression to use when establishing SQLite connections using this plugin. You can use this to add an index to a table to speed up replay, but this must be done for each table you want to index.

Example:
<sql_initialization_string>
    DROP INDEX IF EXISTS pingtopic_0_by_src_timestamp;
    CREATE INDEX pingtopic_0_by_src_timestamp ON 'PingTopic@0' (SampleInfo_source_timestamp);
</sql_initialization_string>

Default: PRAGMA SYNCHRONOUS = OFF; PRAGMA JOURNAL_MODE = MEMORY;

 0..1

5.2.9. Fileset

The <fileset> tag allows you to specify a set of files for Converter to write to.

The behavior allowed by Converter is more limited than the behavior allowed by Recording Service.

Table 5.7 Fileset Tags in Converter’s Configuration File
Tags within <fileset> Description Multiplicity
<workspace_dir>

Required.
Base directory where Converter should output files. Must be different from the input database_dir directory.

Default: The current working directory

 1
<filename_expression>

Required.
The file name Converter will use for the generated user data file(s). This expression in combination with the rollover configuration will determine how many (and the names of) files Converter will output. This setting accepts text and any combination of the following parameters in it:

  • Autonumeric. Format: %auto:M%. This parameter describes an integer that auto-increments every time Converter has a rollover event due to time or file size.

    Example:
    <filename_expression>
  test_files_%auto:0%.dat
</filename_expression>

    This will create a series of files starting with test_files_0.dat, and incrementing each time Converter rolls over (due to the file size or time limits being reached).

  • Timestamp. Format: %ts%. This parameter will take the current timestamp in the system (the time represented as number of seconds since Epoch).

  • Time. Format: %T%. Current time expressed in ISO 8601 time format (HH:MM:SS). Example: 14:55:02. This parameter uses the strftime() parameter %T.

  • Short date. Format: %F%. Short date in YYYY-MM-DD format. Example: 2001-08-23. This parameter uses the strftime() parameter %F. Note: this parameter will not vary in 24 hours, so use with caution in combination with the rollover time limit feature (time limit should be greater than 1 day; otherwise, you may overwrite the same file continously).

  • Date and time. Format: %c%. Date and time representation, locale-dependent. This parameter uses the strftime() parameter %c.

Default: rti_recorder_default_%auto:0%.dat (auto-numeric starting at zero, unlimited)

 1
<rollover> Configuration for rolling over the file after a size limit is reached. See Section 5.2.10. 0..1

5.2.10. Rollover

Rollover enables Converter to write to the next file in a set when a limit is reached. Converter’s rollover functionality is decoupled from the files that are input, so many files may be merged into one, or one file may be split into many depending on Converter’s rollover configuration. Currently Converter supports rolling over by size.

Table 5.8 Rollover in Converter’s Configuration File
Tags within <rollover> Description Multiplicity
<file_size_limit>

The maximum allowed size for a file in a set. Units can be specified as an attribute.

Example:
<file_size_limit unit="KILOBYTES">500
</file_size_limit>
0..1

5.2.11. Legacy

This tag configures how legacy databases are converted to the newer format. (Note: You can continue to use the older format with Replay, by specifying “legacy” storage in Replay’s configuration.)

Table 5.9 Legacy in the Configuration File
Tags within <legacy> Description Multiplicity
<file_path> The path to a legacy file to be converted. File set and version properties will be obtained automatically from the file itself. 0..1
<domain_mapping> This tag allows you to link legacy domain names with domain IDs. See Section 5.2.12. 0..1

5.2.12. Domain Mapping

When converting a legacy database, the domain may not have been recorded. This tag provides a way to map a table with a domain ID.

Later versions of the old Recorder database allowed you to use field filters. By default, the domain ID field was not recorded (it was filtered out by default). Thus, there is no information available in these legacy databases to relate the domain name used to record the data with a domain ID. This tag allows you to link legacy domain names with domain IDs.

Table 5.10 Domain Mapping in the Configuration File
Tags within <domain_mapping> Description Multiplicity
<domain_map>

Required.
A link between a recorded legacy domain name and a domain ID.

Attributes:
  • legacy_domain_name: The recorded domain name specified in the old Recording Service domain tag, for example: <domain name=”domain0”>.
  • domain_id: The domain ID you want to associate with the legacy domain name.
 1..*

5.2.13. Plugin

This tag enables you to convert data using an external library that you specify.

Table 5.11 Plugin Tags in the Configuration File
Tags within <plugin> Description Multiplicity
<property> Name/value pairs of properties to pass to a storage plugin. 0 or 1

5.2.14. Data Selection

This tag selects what data to convert from the database, based on a time range.

Table 5.12 Data Selection Tags in Converter’s Configuration File
Tags within <data_selection> Description Multiplicity
<time_range> Select data to convert from the database, based on start and end times. See Section 5.2.15. 0..1

5.2.15. Time Range

The <time_range> tag allows you to specify the begin and end times of the data you want to convert. This can be specified either as timestamps or as symbolic timestamps called “timestamp tags.” Tags may have been added to the recording through remote administration, and can be viewed using the script rtirecordingservice_list_tags (see Section 3.5.7).

Table 5.13 Time Range Tags in Converter’s Configuration File
Tags within <time_range> Description Multiplicity
<begin_time> Select data to convert from the database, with a timestamp beginning at this time. Specify the time in seconds and nanoseconds.
Default: 0, start at beginning of the file.		 0..1
<begin_tag> Can be used instead of begin_time. Select data to start converting from the database by specifying the string name of a timestamp tag. Timestamp tags associate a timestamp with a name. Then you can refer to a timestamp by that name. The timestamp must have been tagged with that name during recording. See Section 3.5.6.
Default: Start at beginning of the file.		 0..1
<end_time> Select data to convert from the database, with a timestamp ending at this time. Specify the time in seconds and nanoseconds.
Default: End at end of file.		 0..1
<end_tag> Can be used instead of end_time. Select when to stop converting data by specifying the string name of a timestamp tag. Timestamp tags associate a timestamp with a name. Then you can refer to a timestamp by that name. The timestamp must have been tagged with that name during recording. See Section 3.5.6.
Default: Stop at end of file.		 0..1

5.2.16. DomainParticipant

The <domain_participant> tag allows you to specify the domain IDs you want to convert from the database. Specify a name for the DomainParticipant, and the domain_id you want to convert. Associate the DomainParticipant name with a session, topic or topic_group that you want to convert, to convert data within a domain.

This does not create a DDS DomainParticipant.

Table 5.14 DomainParticipant Tags in Converter’s Configuration File
Tags within <domain_participant> Description Multiplicity
<domain_id> The domain ID of tables loaded from the database. This is used to determine which domain IDs to convert. 1
<register_type> Registers a type name and associates it with a type representation. When you define a type in the configuration file, you have to register the type in order to use it in a <topic>. 0..*

5.2.17. Session

The <session> tag configures the threads that will be used to convert data.  You also specify the Topics and groups of Topics to convert inside the <session> tag.

Table 5.15 Session Tags in Converter’s Configuration File
Tags within <session> Description Multiplicity
<thread_pool> Specifies the number of threads used by this session, and the settings used when creating them. 0..1
<topic>

Specifies an individual Topic to convert. See Section 5.2.19.

Attributes:
  • name: The name of the Topic to convert.
  • participant_ref: Refers to a domain_participant tag, that specifies the domain ID to use from the database when converting this Topic. If the parent <session> specifies a default_participant_ref, this attribute is optional.
 0..*
<topic_group>

Specifies a group of Topics to convert. See Section 5.2.18.

Attributes:
  • name: The name of the topic group.
  • participant_ref: Refers to a domain_participant tag, that specifies the domain ID to use from the database when converting this Topic. If the parent <session> specifies a default_participant_ref, this attribute is optional.
 0..*

5.2.18. Topic Group

The <topic_group> tag allows you to convert a group of Topics, using regular expressions to describe which Topics to convert.

Table 5.16 Topic Group Tags in Converter’s Configuration File
Tags within <topic_group> Description Multiplicity
<allow_topic_name_filter>

A regular expression describing which Topics should be converted.

Example:
<topic_group name="ConvertAll">
    <allow_topic_name_filter>*</allow_topic_name_filter>
</topic_group>
0..1
<deny_topic_name_filter> A regular expression describing which Topics should not be converted. This is applied after the allow_topic_name_filter. 0..1

5.2.19. Topic

The <topic> tag specifies an individual Topic to convert.

Table 5.17 Topic Tags in Converter’s Configuration File
Tags within <topic> Description Multiplicity
<registered_type_name> The name of the data type that will be converted for this topic. Required. 1
<transformation>

The transformation library to be applied to this Topic’s data when converting. This is a user library that can modify the data after it is received from input storage and before it is sent to output storage.

Transformations implement APIs identical to Routing Service’s transformations. For more on using transformations, see these sections in the RTI Routing Service User’s Manual:

Attributes:
  • plugin_name: The name of the plugin to load, qualified by the plugin library name.
Example:
<dds>
    <plugin_library name="ConverterTransformations">
        <transformation_plugin name="ModifyTestID">
            <create_function>ModifyTestID_create</create_function>
            <dll>modify_test_id_library</dll>
        </transformation_plugin>
    </plugin_library>
    <!-- ... -->
    <converter>
        <!-- ... -->
        <topic name="TestTopic">
            <transformation plugin_name="ConverterTransformations::ModifyTestID" />
        </topic>
    </converter>
</dds>
0..1

5.2.20. Converter’s Builtin Configuration Details

Converter comes with a builtin configuration, which selects the name of an input file to convert from and an output file to convert to. The ‘defaultToJson’ configuration specifies JSON SQLite format by default as the output format, with ‘rti_recorder_default.dat’ as the input file and ‘rti_recorder_default_converted.dat’ as the output file. It expects to find the file(s) to convert in a directory called ‘cdr_recording’. This is the default directory when recording in XCDR format in the default Recorder configuration.

<?xml version="1.0" encoding="UTF-8"?>

<dds xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="http://community.rti.com/schema/6.0.0/rti_converter.xsd">

    <!-- Available types -->
    <types />

    <!--  A definition of a Converter instance to run -->
    <converter name="defaultToJson">
        <!--  Input storage settings -->
        <input_storage>
            <sqlite>
                <storage_format>XCDR_AUTO</storage_format>
                <database_dir>cdr_recording</database_dir>
            </sqlite>
        </input_storage>

        <!--  Output storage settings -->
        <output_storage>
            <sqlite>
                <storage_format>JSON_SQLITE</storage_format>
                <fileset>
                    <workspace_dir>converted</workspace_dir>
                    <filename_expression>rti_recorder_default.converted.dat</filename_expression>
                </fileset>
            </sqlite>
        </output_storage>

        <!--  Domain selection: assume 0 by default -->
        <domain_participant name="Domain0">
            <domain_id>0</domain_id>
        </domain_participant>

    </converter>
</dds>