[SOLVED] rticonnextdds_connector + node.js + durability problem

Hello Alex, 

Thanks for trying out the connector!
Let me try to answer to number 2 first. 

You are correctly using the builtin qos profile BuiltinQosLibExp::Generic.StrictReliable. After that in your qos file you are changing the data reader qos setting history to KEEP_LAST with depth 1:

<datareader_qos>
<durability>
<kind>TRANSIENT_LOCAL_DURABILITY_QOS</kind>
</durability>
<history>
<kind>KEEP_LAST_HISTORY_QOS</kind>
<depth>1</depth>
</history>
<reliability>
<kind>RELIABLE_RELIABILITY_QOS</kind>
</reliability>
</datareader_qos>
</qos_profile>
</qos_library>

 To be really strict reliable the history kind has to be <strong>KEEP_ALL</strong> otherwise you may loose samples due to other resource limits. See this chapter of the user manual:

https://community.rti.com/static/documentation/connext-dds/5.2.3/doc/manuals/connext_dds/html_files/RTI_ConnextDDS_CoreLibraries_UsersManual/index.htm#UsersManual/ControllingQueueDepthwithHistory.htm?Highlight=strict reliable

By changing the reader qos to this:

<datareader_qos>
<durability>
<kind>TRANSIENT_LOCAL_DURABILITY_QOS</kind>
</durability>
<history>
<kind>KEEP_ALL_HISTORY_QOS</kind>
<!-- <depth>1</depth> -->
</history>
<reliability>
<kind>RELIABLE_RELIABILITY_QOS</kind>
</reliability>
</datareader_qos>

 it should work fine and you should not loose any samples.

 Also, you want to have two different participant in you XML: one containing only the Reader and one containing only the reader. 

In your example there was one participant containing both. That means that the sender will create a reader as well and, since is reliable the writer will have to be sure that one received the samples too. 

 I attached the xml with my modifications. 

 I also did some small modification to the receiver and the sender:

• at the beginning i put a sleep of 4 seconds after creating the input and output: that will allow for the entities to discover. 

 A suggestions. In your code you do:

 console.log(JSON.stringify(input.samples.getJSON(i)));

 to print the whole sample: that can be slow. The way it works is that core create a json string and then we covert it to a js object. Creating the string can be pretty slow if the sample is big. Also, to print it on the screen you are transforming back the js obj to a string!  

I would suggest you to use the direct API to access a specific field:

console.log(input.samples.getString(i,'color'));

I attached my modified sender and receiver as well. 

As for the first question:

1. What are you plans and roadmap on adding more support for node.js - official release and support, fulll API support etc.?

Connector is an experimental feature that has been developed as a research project. The plan is to gather customer feedback and understand what is the interest before adding official support. 

Regarding the full API support: connector is a simplified API. We do not plan to support the full DDS Api at the moment. That being said, we are open to suggestion on how to improve the current API. 

I hope I was helpful,

Best, 
Gianpiero

Hello Alex,

First of all I want to clarify that the JS or PY Connector are using the same DDS libraries that C# uses. So all the QoS Settings have the same meaning and they should work the same way. That said let me try to answer to your questions the best i can. 

Question #2 

Problem A 

Even if you are sending different instances, each one of those instances is a sample. In your specific example each sample belongs to a different instance; but they still have to be stored in the reader queue and then processed by the application. 

I should look more in deep to both application but, if I have to guess, I think the reason why you notice a difference in the C# compared to the JS code is because the processing of each sample takes way longer. As I explained in the post above, in Javascript you are using dynamic data and in your code your processing for each sample is the following:

console.log(JSON.stringify(input.samples.getJSON(i))); 

That is slow! If you try to remove that line, and you just increase the counter, you will not lose any sample. Of course that is not realistic, and that’s why i think using keep_all is the right solution. 

Problem A (late joiner)

I think you identified a bug in the javascript layer. I spent some time investigating. So if you don’t use the connector.on but instead you do polling (just put a for loop instead):

Using connector.on:

// Subscribe

connector.on('on_data_available',

    function () {

        input.take();

        var len = input.samples.getLength()

        // console.log(len)

        for (i = 1; i <= len; i++) {

            if (input.infos.isValid(i)) {

                console.log(JSON.stringify(input.samples.getJSON(i)));

//                console.log(input.samples.getString(i,'color'));

                received_count++;

            } else {

                console.log(">>> INVALID SAMPLES");

            }

        }

    });

Using a for loop:

// Subscribe

for(;;) {

        input.take();

        var len = input.samples.getLength()

        // console.log(len)

        for (i = 1; i <= len; i++) {

            if (input.infos.isValid(i)) {

                console.log(JSON.stringify(input.samples.getJSON(i)));

//                console.log(input.samples.getString(i,'color'));

                received_count++;

            } else {

                console.log(">>> INVALID SAMPLES");

            }

        }

}

 all the sample is received! That means that there is something wrong with the mechanism that implements the connector.on! Internally we have a wait set that gets triggered when there are new data. At that point the function you specified into the connector.on gets called and the take will reset the waitset. Unfortunately

 I think there is a bug in how i implemented this: I first notify the js listeners and then go back to wait in c. I think what is happening is that, by the time i finish to notify all the js listeners and then call the take, all the sample are received by the middleware so the take resets the wait set and connector.on is never called again. 

That’s why if you poll (the for loop) you will get the samples! What is not happening is the triggering of the function that does the read in the right way. 

 I changed the js layer of the connector to call the c wait set before calling the javascript listener and that seems to fix the problem on my tests(see attached zip file): started the sender, waited for it to send all the samples, started the receiver, received all the samples.

 Could you try and confirm that for me? If it works for you, i will review the code with some coworker and then update the github repo. 

To try, use the attached file called rticonnextdds-connector.js and replace the one you have in node_modules/rticonnextdds-connector 

Question #1

I agree with you: nodejs is very trendy! We are planning to add more features to the connector js. Filters and the capability of adding new reader/writers dynamically as well as discovery topics and some status changes. The goal for the connector is to give enough flexibility but a smaller set of API. I also see that there is space for a full dds api in node separated from the connector. 

My suggestion is to contact your distributor and maybe set up a meeting with product management here in RTI and we can figure out together what the need and how to get there. What do you think?

Best,
  Gianpiero

Alex,

Thanks for taking the time to test. I will do a code review internally and update the connector on github and npm. I will let you know when is up. 

Best,
  Gianpiero

Alex,

The fix should be now on github and on npm. (v 0.2.1) 

Best,
  Gianpiero