RTI Converter segfaulting when outputting to SQL

6 posts / 0 new
Last post
Offline
Last seen: 5 years 4 months ago
Joined: 07/29/2013
Posts: 7
RTI Converter segfaulting when outputting to SQL

Hello,

When using the recorder conversion tool (rtirecconv) to convert a serialized database to something human readable, it causes a segmentation fault. I did a backtrace with a core dump and the offending function seems to be sqlite3Select(). I really don't have any information besides that because I don't have debugging symbols.


Our current method uses the converter to convert the data into XML first and then we parse it into SQLite, but this method takes quite a while for substantially large files. Can I get any insight into why the segfault is happening? I have tried 5.0.0 and 5.0.0.4 and they both do the same thing. My system is running CentOS 6.4 with sqlite3 3.6.20.

Offline
Last seen: 1 month 1 week ago
Joined: 01/15/2013
Posts: 88

Hi,

Do you have any log message or any hint on the types you're trying to convert?

Converter has a built-in utility to convert serialized format conserving the SQLite format. If you specify the "-format SQL" flag when calling Converter it will create a new database file with the same tables but the samples will be deserialized.

Thanks,

Juanlu

Offline
Last seen: 5 years 4 months ago
Joined: 07/29/2013
Posts: 7

Unfortunately I can't share our types but I have narrowed it down to it being a sequence problem. I recreated the problem using another IDL.

module com {

module test {

struct Message {

  long messageId;

  long messageLen;

  string message;

};

struct Envelope {

  long seqNum;

  sequence<Message, 512> messages;

};

}; // test

}; // com

If I were to guess, it seems that the converter will try to create more columns than sqlite will allow. Our original IDL files are much more complex and contain much higher sequences (e.g. upto 10k). I ran a recording test with the above IDL and wrote 10 envelopes with 5 to 25 messages inside them and could recreate the segfault when I tried to convert it to SQL format.

Let me know if you need anything else.

Offline
Last seen: 1 month 1 week ago
Joined: 01/15/2013
Posts: 88

Hi,

Did you try different conversion methods? Conversion to XML, CSV or HTML shouldn't be limited to the number of allows SQLite columns. Does it happen in these cases too?

Offline
Last seen: 5 years 4 months ago
Joined: 07/29/2013
Posts: 7

Conversion to XML works fine, which is what we use right now. The problem is that it has now become a two-step process in that we convert to XML and then back to SQLite.

I would like to migrate away from from XML because our databases are several gigabytes in size and it currently takes several hours to do the conversion.

I tried CSV in the past and it didn't work well either. Either it crashed outputting to CSV, or the resulting CSV output was not readable in a spreadsheet (probably because it was too large). Never tried HTML.

Offline
Last seen: 1 month 1 week ago
Joined: 01/15/2013
Posts: 88

Hi,

One thing that may interest you is that "conversion to SQL" (deserialization) speed has been greatly improved in Converter 5.1. Also, the SQLite column limits have been improved (now it can hold up to 5000 columns, better for types with sequences and arrays), which could be one of the reasons sequences are not working properly for you. One thing you could do is to try out Recording Service 5.1. However, you won't be able to use Converter 5.1 with already existing (pre 5.1) databases, as compatibility has been broken in order to improve the tool.

You mention that your databases are several GB in size. The disk space usage has been greatly improved in Recorder 5.1. You could check out that too because with more efficient storage you would also get more efficient conversion.

If you decide to try, beware that Recording Service 5.1 uses Connext 5.1, which has some default QoS incompatibilities against Connext 5.0.x. You would need to use the new built-in QoS profile for 5.0 compatibility ("Baseline.5.0.0" profile; see http://community.rti.com/examples/built-qos-profiles) with your Recording Service 5.1 entities.

Thanks,

Juanlu