.. include:: ../../../recorder.4.0/srcDoc/vars.rst

.. _section-storageutils-plugins:

Storage Utility Plugins
=======================

.. _section-storageutils-plugins-csv:

CSV
----
This is a type of output storage plugin that stores the data provided to it 
in Comma-Separated Value (CSV) format. It is meant to be used with *Converter* 
to perform conversion of a recorded database into CSV output files. Typically 
it helps solve use cases like offline analysis or incident investigation. For 
more information about the configuration tags, see 
:numref:`section-convert-config-csv-tag`.

By default, the plug-in generates a separate ``csv`` file for each recorded *Topic*. 
The content and format of each file is as follows:

.. list-table:: CSV file format
    :name: TableCsvFileFormat
    :widths: 15 20 10 10 10 10

    * - Row 1: Topic Entry
      - ``Topic name: <Name>``
      -
      -
      -
      -
    * - Row 2: Type Header
      - ``timestamp``
      - ``member1``
      - ``member2``
      - ...
      - ``memberN``
    * - Row 3: Data values
      - Reception timestamp of the sample (in nanoseconds)
      - Value for member1
      - Value for member2
      - ...
      - Value for memberN

The filename matches the topic name, with a ``.csv`` extension. All 
the files are placed in a directory that can be specified in the plug-in 
configuration.

.. note::

    If the topic name contains characters that cannot appear in the file name
    because they are reserved by the underlying operating system, these characters
    will be replaced by the token ``#``.

Mapping a data sample into columns
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

General case
""""""""""""

As shown in the table above, a data sample is represented in a single row
comprised of multiple columns. Each cell holds only a value for a *final* or
leaf member in a complex data type. That is, for a member whose type is
a *Simple* type (integer, char, short), enumeration, or *String*.

Each data value in a cell corresponds to a member whose name is in the
type header. Given a complex data type, the name for a member is constructed
as follows:

::

    .<parent_member1>.<parent_member2>...<parent_memberN>.<final_member>

where ``<parent_member>`` is the name of the parent complex member that
contains the subsequent member.

If either ``<parent_member>`` or ``<final_member>`` is a *Collection* type
(array or sequence), the member name is suffixed with ``[<index>]``, and a column
for each possible element in the *Collection* is created.

For example, consider the following type described in IDL:

.. code-block:: IDL

    struct NestedStruct {
        long m_long;
    };

    struct TopLevelStruct {
        String m_string;
        long m_array[2];
        NestedStruct m_complex;
        NestedStruct m_complex_array[2];
    };

The resulting type header row will look like this:

.. list-table::
    :name: TableExampleTypeHeaderRow

    * - .m_string
      - .m_array[0]
      - .m_array[1]
      - .m_complex.a_long
      - .m_complex_array[0].a_long
      - .m_complex_array[1].a_long


.. note::

    Considerations about collection types: Due to the column consistency
    required by the CSV format, mapping a collection type **requires us 
    to generate as many columns as the number of elements that can be present 
    in the collection**, even if for a given data sample only a few of them 
    are present (such as for a ``Sequence`` type). If the recorded type has 
    collections with large sizes, the generated file may hit the column limit 
    of some CSV processors.

Sequences
"""""""""

Mapping a ``Sequence`` type is based on the mapping of a *Collection*
type explained above, plus an additional column to indicate the length 
of the sequence. That is:

::

    .<seq_member.length>.<seq_member[0]>...<seq_member[N-1]>

where the column ``<seq_member.length>`` indicates how many elements are set
in the sequence (the number of columns with non-null values) and ``N`` is the
maximum length of the sequence.

For example, consider the following type described in IDL:

.. code-block:: IDL

    struct StructType {
        sequence<long, 4> m_seq;
    };

The resulting type header row will look like this:

.. list-table::
    :name: TableSeqTypeHeaderRow

    * - .m_seq.length
      - .m_seq[0]
      - .m_seq[1]
      - .m_seq[2]
      - .m_seq[3]


Unions
""""""

The mapping of a ``Union`` type is similar to a ``Struct`` type except that
a discriminator column with name ``disc`` is placed before all the members.

For example, consider the following type described in IDL:

.. code-block:: IDL

    union UnionType switch (long) {
        case 0:
        long case1;

        case 1:
        StructType case2;

        default:
        long case_default;
    };

    struct StructType {
        UnionType m_union;
    };

The resulting type header row will look like this:

.. list-table::
    :name: TableUnionTypeHeaderRow

    * - .m_union.disc
      - .m_union.case1
      - .m_union.case2
      - .m_union.default


Data Values
"""""""""""
For a given data sample, the value for each member is placed under the 
corresponding column represented as a ``String``, which applies to all primitive 
types. For primitive types, the output text will correspond to the standard 
conventions for the type (e.g., decimal point for floating point numbers, 
single quotes around a character, etc.). By default, enumerations are printed 
with their corresponding text label. This behavior for enums is configurable.

There may not be values for every column in a sample. This may occur for
the following situations:

* A Sequence member that does not contain all the possible elements.
* A Union member, which can only set a member at a time.
* An optional member, which may or may not be set.

By default, the value of an empty member is represented as ``nil``.

For example, consider the following type described in IDL:

.. code-block:: IDL

    struct StructType {
        sequence<long, 2> m_sequence;
        @optional String m_optional;
    };

And two samples with the following values (represented in JSON):

.. code:: JSON

    {
        "m_sequence": [1, 2],
        "m_optional": "hello"
    }

    {
        "m_sequence": [1],
        "m_optional":
    }

The resulting type header row and the two data value rows will look like this:

.. list-table::
    :name: TableEmptyMembersExample

    * - .m_sequence.length
      - .m_sequence[0]
      - .m_sequence[1]
      - .m_optional
    * - 2
      - 1
      - 2
      - hello
    * - 1
      - 1
      - nil
      - nil