Hello,
When using the recorder conversion tool (rtirecconv) to convert a serialized database to something human readable, it causes a segmentation fault. I did a backtrace with a core dump and the offending function seems to be sqlite3Select(). I really don't have any information besides that because I don't have debugging symbols.
Our current method uses the converter to convert the data into XML first and then we parse it into SQLite, but this method takes quite a while for substantially large files. Can I get any insight into why the segfault is happening? I have tried 5.0.0 and 5.0.0.4 and they both do the same thing. My system is running CentOS 6.4 with sqlite3 3.6.20.
Hi,
Do you have any log message or any hint on the types you're trying to convert?
Converter has a built-in utility to convert serialized format conserving the SQLite format. If you specify the "-format SQL" flag when calling Converter it will create a new database file with the same tables but the samples will be deserialized.
Thanks,
Juanlu
Unfortunately I can't share our types but I have narrowed it down to it being a sequence problem. I recreated the problem using another IDL.
module com {module test {struct Message {long messageId;long messageLen;string message;};struct Envelope {long seqNum;sequence<Message, 512> messages;};}; // test}; // comIf I were to guess, it seems that the converter will try to create more columns than sqlite will allow. Our original IDL files are much more complex and contain much higher sequences (e.g. upto 10k). I ran a recording test with the above IDL and wrote 10 envelopes with 5 to 25 messages inside them and could recreate the segfault when I tried to convert it to SQL format.
Let me know if you need anything else.
Hi,
Did you try different conversion methods? Conversion to XML, CSV or HTML shouldn't be limited to the number of allows SQLite columns. Does it happen in these cases too?
Conversion to XML works fine, which is what we use right now. The problem is that it has now become a two-step process in that we convert to XML and then back to SQLite.
I would like to migrate away from from XML because our databases are several gigabytes in size and it currently takes several hours to do the conversion.
I tried CSV in the past and it didn't work well either. Either it crashed outputting to CSV, or the resulting CSV output was not readable in a spreadsheet (probably because it was too large). Never tried HTML.
Hi,
One thing that may interest you is that "conversion to SQL" (deserialization) speed has been greatly improved in Converter 5.1. Also, the SQLite column limits have been improved (now it can hold up to 5000 columns, better for types with sequences and arrays), which could be one of the reasons sequences are not working properly for you. One thing you could do is to try out Recording Service 5.1. However, you won't be able to use Converter 5.1 with already existing (pre 5.1) databases, as compatibility has been broken in order to improve the tool.
You mention that your databases are several GB in size. The disk space usage has been greatly improved in Recorder 5.1. You could check out that too because with more efficient storage you would also get more efficient conversion.
If you decide to try, beware that Recording Service 5.1 uses Connext 5.1, which has some default QoS incompatibilities against Connext 5.0.x. You would need to use the new built-in QoS profile for 5.0 compatibility ("Baseline.5.0.0" profile; see http://community.rti.com/examples/built-qos-profiles) with your Recording Service 5.1 entities.
Thanks,
Juanlu