.. include:: ../../../router.1.0/srcDoc/vars.rst
.. _section-converter_configuration:
Converter Configuration
=======================
When you start *Converter*, you can specify a configuration file
in XML format. In this file, you can specify the properties that
control the behavior of the service. This section describes how to
write a configuration file.
How to Load the XML Configuration
---------------------------------
*Converter* loads its XML configuration from multiple
locations. Here are the various sources of configuration files, listed
in load order:
1. ``[working directory]/USER_CONVERTER.xml`` This file
is loaded automatically if it exists.
2. ``[NDDSHOME]/resource/xml/RTI_CONVERTER.xml`` This
file is loaded automatically if it exists.
3. One or more files (semicolon-separated) specified using the command-line
parameter -cfgFile.
.. note::
``[working directory]`` indicates the path to the current working
directory from which you run *Converter*.
``[NDDSHOME]`` indicates the path to your *Connext DDS* installation.
See :numref:`section-doc-paths`.
You may use a combination of the above sources and load multiple configuration
files.
Here is an example configuration file. You will learn the meaning of
each line as you read the rest of this section.
.. code-block:: xml
XCDR_AUTO
cdr_recording
JSON_SQLITE
converted
rti_recorder_default.converted.dat
0
*
rti/*
XML Syntax and Validation
-------------------------
The XML representation of DDS-related resources must follow these syntax
rules:
- It shall be a well-formed XML document according to the criteria
defined in clause 2.1 of :link_xml_spec:`the Extensible Markup Language standard <>`.
- It shall use UTF-8 character encoding for XML elements and values.
- It shall use ```` as the root tag of every document.
To validate the loaded configuration, *Converter* relies on an XSD
file that describes the format of the XML content. We recommend
including a reference to this document in the XML file that contains the
service’s configuration; this document provides helpful features in code
editors such as Visual Studio®, Eclipse®, and NetBeans®, including
validation and auto-completion while you are editing the XML file.
The XSD definitions of the XML elements are in ``$NDDSHOME/resources/schema/rti_converter.xsd``.
To include a reference to the XSD document in your XML file, use the
attribute ``xsi:noNamespaceSchemaLocation`` in the ```` tag. For example:
.. code-block:: xml
Builtin Configuration of Converter
----------------------------------
*Converter* comes pre-configured with a configuration that converts
a file called rti_recorder_default.dat from XCDR format to a file called
rti_recorder_default_converted.dat in JSON_SQLITE format.
See :numref:`section-builtin-config-converter` for details.
.. _section-convert-config-XML-tags:
XML Tags for Configuring Converter
----------------------------------
This section describes the XML tags you can use in a *Converter*
configuration file. :numref:`FigureConverterTopLevelTags`
and :numref:`TableConverterTopLevelTags` describe
the top-level tags allowed within the root ```` tag.
.. figure:: ../static/ConverterTopLevelTags.png
:alt: Top-Level Tags in Converter's Configuration File
:name: FigureConverterTopLevelTags
:align: center
:figwidth: 50 %
Top-level Tags in the *Converter* Configuration File
.. list-table:: Top-Level Tags in Converter's Configuration file
:name: TableConverterTopLevelTags
:widths: 20 70 10
:header-rows: 1
* - Tags within :litrep:``
- Description
- Multiplicity
* - :litrep:``
- Defines types that can be used by *Converter*. This tag is needed
if data types are not available in the discovery table.
The type description is done using the
*Connext DDS* XML format for type definitions.
See :link_userman_types_in_xml:`Creating User Data Types with
Extensible Markup Language (XML), in the RTI Connext DDS
Core Libraries User's Manual <>`.
- 0..*
* - :litrep:``
- Contains a list of storage libraries that you may have implemented
to support custom databases. For more information, see
:numref:`section-Record-Config-Plugins` and
:numref:`section-Replay-Config-Plugins`.
- 0..*
* - :litrep:``
- **Required**. |br|
Specifies a *Converter* configuration.
Attributes
- ``name``: uniquely identifies a *Converter* configuration.
**Required**.
Example
.. code-block:: xml
See :numref:`section-convert-config-converter-tag`.
- 1..*
.. _section-convert-config-converter-tag:
Converter Tag
-------------
A configuration file must have at least one ```` tag.
This tag is used to configure an execution of *Converter*.
A configuration file may contain multiple ```` tags.
When you start *Converter*, you can specify which
```` tag to use to configure the service using the
``-cfgName`` command-line parameter. This means one file can
be used to configure multiple *Converter* executions.
:numref:`FigureConverterTags` and :numref:`TableConverterTags`
describe the tags allowed within a ```` tag.
.. figure:: ../static/ConverterTags.png
:alt: Converter Tags in Converter's Configuration File
:name: FigureConverterTags
:align: center
:figwidth: 50 %
Converter Tags in the *Converter* Configuration File
.. list-table:: Converter Tags in Converter's Configuration File
:name: TableConverterTags
:widths: 20 50 10
:header-rows: 1
:class: longtable
* - Tags within :litrep:``
- Description
- Multiplicity
* - :litrep:``
- Describes the storage that *Converter* will read as input. *Converter*
expects the storage to be in CDR (binary) format.
If storage is not specified, data will be stored in a SQLite file using
a default name.
See :numref:`section-convert-config-input-storage-tag`.
- 0..1
* - :litrep:``
- Describes the storage that *Converter* will write as output. This output
will be in JSON format.
If this storage is not specified, data will be stored in a
SQLite file using a default name.
See :numref:`section-convert-config-output-storage-tag`.
- 0..1
* - :litrep:``
- Enables selection of a subset of data to convert. Supports selecting
data for conversion by time (or tagged time).
See :numref:`section-converter_config_dataselection_tag`.
- 0..1
* - :litrep:``
- Allows you to create one or more sessions in which to convert data,
which gives you the option of specifying multiple threads for
conversion.
See :numref:`section-converter_config_session_tag`.
Attributes:
- ``name``: Uniquely defines a session. **Required**.
- ``default_participant_ref``: Refers to a ``domain_participant``
tag, that specifies the domain ID to use from the database
when converting this Topic. Children can override this by
specifying their own participant.
- 0..*
* - :litrep:``
- **Required.** |br|
Describes the domains *Converter* will select for conversion. *Converter*
is not a DDS application so this definition will not create a DDS Domain
Participant, it just works as a domain ID selection utility. |br|
*Note*: the domain ID has to correspond to one of the domain IDs defined
in the Recorder configuration. |br|
See :numref:`section-converter_config_domainparticipant_tag`.
Attributes:
- ``name``: Uniquely defines a DomainParticipant. **Required**.
- 1..*
Example: Specify a Configuration in XML
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: xml
Starting *Converter* with the following command will use the
```` tag with the name "ConvertToJSON".
.. code-block:: bash
$NDDSHOME/bin/rticonverter -cfgFile file.xml -cfgName ConvertToJSON
.. _section-convert-config-input-storage-tag:
Input Storage
-------------
The ```` tag allows you to configure the storage from which
data will be read. You can choose between using the builtin SQLite storage or
implementing your own storage plugin.
.. list-table:: Input Storage Tags in Converter's Configuration File
:name: TableInputStorageTag
:widths: 20 70 10
:header-rows: 1
:class: longtable
* - Tags within :litrep:``
- Description
- Multiplicity
* - :litrep:``
- Enables converting data from an SQLite database file.
See :numref:`section-convert-config-sqlite-tag`.
- 0..1
* - :litrep:``
- Enables converting data from a legacy version of *Recording Service*
to a database usable by this version.
See :numref:`section-converter_config_legacy_tag`.
- 0..1
* - :litrep:``
- Enables converting data from storage using an external library that you
specify.
See :numref:`section-converter_config_plugin_tag`.
- 0..1
* - :litrep:``
- Specifies the maximum number of
samples to be returned by the read() operation.
When the recorded tables contain huge amounts of samples,
this setting can help improve the responsiveness of
*Converter*, because the tool won't - for each table -
attempt to get all samples at once.
If not specified, the default value is 1024 samples.
A negative value means an unlimited number of samples.
- 0..1
.. _section-convert-config-output-storage-tag:
Output Storage
--------------
The ```` tag allows you to configure the storage into which converted
data will be written. You can choose between using the builtin SQLite storage or
implementing your own storage plugin.
.. list-table:: Output Storage Tags in Converter's Configuration File
:name: TableOutputStorageTag
:widths: 20 70 10
:header-rows: 1
* - Tags within :litrep:``
- Description
- Multiplicity
* - :litrep:``
- Enables writing converted data into an SQLite database file.
See :numref:`section-convert-config-sqlite-tag`.
- 0..1
* - :litrep:``
- Enables writing converted data into storage of your choosing, using a
plugin library that you specify. See
:numref:`section-converter_config_plugin_tag`.
- 0..1
.. _section-convert-config-sqlite-tag:
SQLite
------
The ```` tag allows you to specify the name and file extension of a
SQLite file to read data from, or write data to. Currently, the only storage
format supported for input is CDR, and the only format supported for output is
JSON.
.. list-table:: SQLite Tags in Converter's Configuration File
:name: TableConverterSqliteTag
:widths: 30 60 10
:header-rows: 1
:class: longtable
* - Tags within :litrep:``
- Description
- Multiplicity
* - :litrep:``
(output only)
- Set of files to write to, and parameters for creating files and
directories in that set. Used when creating more than one output file.
See :numref:`section-convert-config-fileset-tag`.
- 0..1
* - :litrep:``
(output only)
- Name of file to write converted database file(s) to. Used when
creating only a single output file.
Default: rti_recorder_default_converted
- 0..1
* - :litrep:``
(output only)
- Suffix of the output file. Used when creating only a single output
file.
Default: dat
- 0..1
* - :litrep:``
(input only)
- Directory containing file(s) to convert.
Default: Current working directory.
- 0..1
* - :litrep:``
- Specifies what format the data is stored in. The options are:
- [Input Default] XCDR_AUTO: This is the binary format used
by *Connext DDS* when sending data over the network. This has
the highest performance for recording, but can only be viewed by
using *Converter* to convert the data to a readable format, or
by using *Replay* to replay the data. This will internally
store data in XCDR or XCDR2 depending on the format received.
- JSON_SQLITE: This format can be queried, but recording in this format
has lower performance because data must be deserialized before
it can be stored.
- XCDR: The format to use when communicating with
*Connext DDS* before 6.0.0.
- XCDR2: More efficient than XCDR, used by *Connext DDS* 6.0.0 and
later.
- 0..1
* - :litrep:``
- Defines what to do when database files are already
present in the current Converter execution directory.
There are two options:
- OVERWRITE: delete old files and replace them with
newly created ones.
- DO_NOT_TOUCH: do not delete any old files and just
exit.
Default: OVERWRITE
- 0..1
* - :litrep:``
- Configures a SQL expression to use when establishing SQLite
connections using this plugin. You can use this to add an index to a
table to speed up replay, but this must be
done for each table you want to index.
Example:
.. code-block:: xml
DROP INDEX IF EXISTS pingtopic_0_by_src_timestamp;
CREATE INDEX pingtopic_0_by_src_timestamp ON 'PingTopic@0' (SampleInfo_source_timestamp);
Default:
PRAGMA SYNCHRONOUS = OFF; PRAGMA JOURNAL_MODE = MEMORY;
- 0..1
.. _section-convert-config-fileset-tag:
Fileset
-------
The ```` tag allows you to specify a set of files for *Converter* to
write to.
The behavior allowed by *Converter* is more limited than the
behavior allowed by *Recording Service*.
.. list-table:: Fileset Tags in Converter's Configuration File
:name: TableConverterFilesetTag
:widths: 20 70 10
:header-rows: 1
:class: longtable
* - Tags within :litrep:``
- Description
- Multiplicity
* - :litrep:``
- **Required**. |br|
Base directory where *Converter* should output files. Must be
different from the input ``database_dir`` directory.
Default: The current working directory
- 1
* - :litrep:``
- **Required**. |br|
The file name *Converter* will use for the generated user
data file(s). This expression in combination with the rollover
configuration will determine how many (and the names of) files
*Converter* will output.
This setting accepts text and any combination of the
following parameters in it:
- Autonumeric. **Format: %auto:M%.** This parameter
describes an integer that auto-increments every
time *Converter* has a rollover event due to time or file size.
Example:
.. code-block:: xml
test_files_%auto:0%.dat
This will create a series of files starting with test_files_0.dat,
and incrementing each time *Converter* rolls over (due to the file
size or time limits being reached).
- Timestamp. **Format: %ts%.** This parameter will take
the current timestamp in the system (the time
represented as number of seconds since Epoch).
- Time. **Format: %T%.** Current time expressed in ISO
8601 time format (HH:MM:SS). Example: 14:55:02.
This parameter uses the strftime() parameter %T.
- Short date. **Format: %F%.** Short date in YYYY-MM-DD
format. Example: 2001-08-23.
This parameter uses the strftime() parameter %F.
**Note:** this parameter will not vary in 24 hours, so
use with caution in combination with the rollover
time limit feature (time limit should be greater
than 1 day; otherwise, you may overwrite the same
file continously).
- Date and time. **Format: %c%.** Date and time
representation, locale-dependent.
This parameter uses the strftime() parameter %c.
Default: rti_recorder_default_%auto:0%.dat (auto-numeric
starting at zero, unlimited)
- 1
* - :litrep:``
- Configuration for rolling over the file after a size limit
is reached. See :numref:`section-convert-config-rollover-tag`.
- 0..1
.. _section-convert-config-rollover-tag:
Rollover
--------
Rollover enables *Converter* to write to the next file in a
set when a limit is reached. *Converter's* rollover functionality is decoupled
from the files that are input, so many files may be merged into one, or one
file may be split into many depending on *Converter's* rollover configuration.
Currently *Converter* supports rolling over by size.
.. list-table:: Rollover in Converter's Configuration File
:name: TableConverterRolloverTag
:widths: 20 50 10
:header-rows: 1
* - Tags within :litrep:``
- Description
- Multiplicity
* - :litrep:``
- The maximum allowed size for a file in a set. Units can be specified
as an attribute.
Attributes:
- :litrep:`unit`: (Optional) The unit in which the size is expressed.
The following values are allowed (Default: ``KILOBYTES``):
- ``BYTES``
- ``KILOBYTES``
- ``MEGABYTES``
- ``GIGABYTES``
Example:
.. code-block:: xml
500
- 0..1
.. _section-converter_config_legacy_tag:
Legacy
------
This tag configures how legacy databases are converted to the newer format.
(Note: You can continue to use the older format with *Replay*, by
specifying "legacy" storage in *Replay's* configuration.)
.. list-table:: Legacy in the Configuration File
:name: TableConverterLegacyTag
:widths: 20 50 10
:header-rows: 1
* - Tags within :litrep:``
- Description
- Multiplicity
* - :litrep:``
- The path to a legacy file to be converted. File set and
version properties will be obtained automatically from
the file itself.
- 0..1
* - :litrep:``
- This tag allows you to link legacy domain names with domain IDs.
See :numref:`section-converter_config_domainmapping_tag`.
- 0..1
.. _section-converter_config_domainmapping_tag:
Domain Mapping
--------------
When converting a legacy database, the domain may not have been recorded.
This tag provides a way to map a table with a domain ID.
Later versions of the old Recorder database allowed you to use field filters.
By default, the domain ID field was not recorded (it was filtered out by
default). Thus, there is no information available in these legacy databases
to relate the domain name used to record the data with a domain ID. This
tag allows you to link legacy domain names with domain IDs.
.. list-table:: Domain Mapping in the Configuration File
:name: TableConverterDomainMappingTag
:widths: 20 50 10
:header-rows: 1
* - Tags within :litrep:``
- Description
- Multiplicity
* - :litrep:``
- **Required**. |br|
A link between a recorded legacy domain name and a domain ID.
Attributes:
- ``legacy_domain_name``: The recorded domain name
specified in the old Recording Service domain tag, for example:
.
- ``domain_id``: The domain ID you want to associate with
the legacy domain name.
- 1..*
.. _section-converter_config_plugin_tag:
Plugin
------
This tag enables you to convert data using an external library that you specify.
.. list-table:: Storage plugin Tag in the Configuration File
:name: ConverterTablePluginTag
:widths: 20 70 10
:header-rows: 1
* - Tags within :litrep:``
- Description
- Multiplicity
* - :litrep:``
- Name/value pairs of properties to pass to a storage plugin.
- 0 or 1
.. _section-converter_config_dataselection_tag:
Data Selection
--------------
This tag selects what data to convert from the database, based on a time range.
.. list-table:: Data Selection Tags in Converter's Configuration File
:name: TableConverterDataSelectionTag
:widths: 20 50 10
:header-rows: 1
* - Tags within :litrep:``
- Description
- Multiplicity
* - :litrep:``
- Select data to convert from the database, based on start and end times.
See :numref:`section-converter_config_timerange_tag`.
- 0..1
.. _section-converter_config_timerange_tag:
Time Range
----------
The ```` tag allows you to specify the begin and end times
of the data you want to convert. This can be specified either as timestamps
or as symbolic timestamps called "timestamp tags." Tags may have been
added to the recording through remote administration, and can be viewed
using the script ``rtirecordingservice_list_tags``
(see :numref:`section-record-list-tags`).
.. list-table:: Time Range Tags in Converter's Configuration File
:name: TableConverterTimeRangeTag
:widths: 20 50 10
:header-rows: 1
:class: longtable
* - Tags within :litrep:``
- Description
- Multiplicity
* - :litrep:``
- Select data to convert from the database, with a timestamp beginning at
this time. Specify the time in seconds and nanoseconds. |br|
Default: 0, start at beginning of the file.
- 0..1
* - :litrep:``
- Can be used instead of begin_time. Select data to start converting
from the database by specifying the string name of a timestamp tag.
Timestamp tags associate a timestamp with a name. Then you
can refer to a timestamp by that name.
The timestamp must have been tagged with that name during recording.
See :numref:`section-record-tags`. |br|
Default: Start at beginning of the file.
- 0..1
* - :litrep:``
- Select data to convert from the database, with a timestamp ending at
this time. Specify the time in seconds and nanoseconds. |br|
Default: End at end of file.
- 0..1
* - :litrep:``
- Can be used instead of end_time. Select when to stop converting data
by specifying the string name of a timestamp tag.
Timestamp tags associate a timestamp with a name. Then you
can refer to a timestamp by that name.
The timestamp must have been tagged with that name during recording.
See :numref:`section-record-tags`. |br|
Default: Stop at end of file.
- 0..1
.. _section-converter_config_domainparticipant_tag:
DomainParticipant
-----------------
The ```` tag allows you to specify the
domain IDs you want to convert from the database. Specify a name for the
DomainParticipant, and the domain_id you want to convert. Associate the
DomainParticipant name with a session, topic or topic_group that you want to
convert, to convert data within a domain.
This does not create a DDS DomainParticipant.
.. list-table:: DomainParticipant Tags in Converter's Configuration File
:name: TableDomainParticipantConverterTag
:widths: 20 50 10
:header-rows: 1
* - Tags within :litrep:``
- Description
- Multiplicity
* - :litrep:``
- The domain ID of tables loaded from the database. This is used
to determine which domain IDs to convert.
- 1
* - :litrep:``
- Registers a type name and associates it with a type representation.
When you define a type in the configuration file, you have to
register the type in order to use it in a ````.
Attributes:
- ``name``: Name that the data type is registered with if no
:litrep:`` is specified. The same data type may be
registered with different names. **Required**.
- ``type_ref``: Definition of this data type. It must refer to one of
the defined types in the ```` section by specifying the fully
qualified name.
Tags within this tag:
- :litrep:``: Name the data type is registered with.
The same data type may be registered with different names. Not required.
- 0..*
.. _section-converter_config_session_tag:
Session
-------
The ```` tag configures the threads that will be used to convert
data. You also specify the Topics and groups of Topics to convert inside the
tag.
.. list-table:: Session Tags in Converter's Configuration File
:name: TableConverterSessionTag
:widths: 20 50 10
:header-rows: 1
:class: longtable
* - Tags within :litrep:``
- Description
- Multiplicity
* - :litrep:``
- Specifies the number of threads used by this session, and the settings
used when creating them.
- 0..1
* - :litrep:``
- Specifies an individual Topic to convert.
See :numref:`section-converter_config_topic_tag`.
Attributes:
- ``name``: The name of the Topic to convert.
- ``participant_ref``: Refers to a ``domain_participant`` tag, that
specifies the domain ID to use from the database when converting this
Topic. If the parent ````
specifies a ``default_participant_ref``, this attribute is optional.
- 0..*
* - :litrep:``
- Specifies a group of Topics to convert.
See :numref:`section-converter_config_topicgroup_tag`.
Attributes:
- ``name``: The name of the topic group.
- ``participant_ref``: Refers to a ``domain_participant`` tag, that
specifies the domain ID to use from the database when converting this
Topic. If the parent ````
specifies a ``default_participant_ref``, this attribute is optional.
- 0..*
.. _section-converter_config_topicgroup_tag:
Topic Group
-----------
The ```` tag allows you to convert a group of Topics,
using regular expressions to describe which Topics to convert.
.. list-table:: Topic Group Tags in Converter's Configuration File
:name: ConverterTableTopicGroupTag
:widths: 20 70 10
:header-rows: 1
* - Tags within :litrep:``
- Description
- Multiplicity
* - :litrep:``
- A regular expression (fnmatch) describing which Topics should be
converted.
You may use a comma-separated list to specify more than one filter. |br|
Example:
.. code-block:: xml
CONTROL_*,DATA_*
- 0..1
* - :litrep:``
- A regular expression (FNMATCH) describing which Topics should not be
converted. This is applied after the ``allow_topic_name_filter``.
You may use a comma-separated list to specify more than one filter.
- 0..1
.. _section-converter_config_topic_tag:
Topic
-----
The ```` tag specifies an individual Topic to convert.
.. list-table:: Topic Tags in Converter's Configuration File
:name: ConverterTableTopicTag
:widths: 20 70 10
:header-rows: 1
:class: longtable
* - Tags within :litrep:``
- Description
- Multiplicity
* - :litrep:``
- The name of the DDS topic to be converted. If this tag is not present,
the name attribute of the `` will be used. |br|
*Note*: we recommend using this tag to define the topic name. There may
be characters that cause the XML validation to fail if they are part of
the topic name attribute. Also, the '/' character and '::' separator may
cause Converter to fail when found in the topic name attribute.
- 0..1
* - :litrep:``
- The name of the data type that will be converted for this topic.
**Required**.
- 1
* - :litrep:``
- The transformation library to be applied to this Topic's data when
converting. This is a user library that can modify the data after
it is received from input storage and before it is sent to output
storage.
Transformations implement APIs identical to *Routing Service's*
transformations. For more on using transformations,
see these sections in the *RTI Routing Service User's Manual*:
- :link_router_manual_config_transformation:`Data Transformation <>`
- :link_router_manual_tutorials:`Tutorials <>`
Attributes:
- ``plugin_name``: The name of the plugin to load, qualified by the
plugin library name.
Example:
.. code-block:: xml
ModifyTestID_create
modify_test_id_library
- 0..1
.. _section-builtin-config-converter:
Converter's Builtin Configuration Details
-----------------------------------------
*Converter* comes with a builtin configuration, which selects the name of
an input file to convert from and an output file to convert to. The
'defaultToJson' configuration specifies JSON SQLite format by default as the
output format, with 'rti_recorder_default.dat' as the input file and
'rti_recorder_default_converted.dat' as the output file. It expects to find the
file(s) to convert in a directory called 'cdr_recording'. This is the default
directory when recording in XCDR format in the default Recorder configuration.
.. code-block:: xml
XCDR_AUTO
cdr_recording
JSON_SQLITE
converted
rti_recorder_default.converted.dat
0