changing max size of sequence

13 posts / 0 new
Last post
Offline
Last seen: 6 years 11 months ago
Joined: 10/29/2012
Posts: 1
changing max size of sequence

Hi,

I'm trying to send a variable length sequence so I've written the idl to have a char sequence and generated the code using rtiddsgen. But I need to increase the maximum size of the sequence depending on the data and it can go over the default of 100 which is set by rtiddsgen.

I've changed the sequenceSize when running rtiddsgen and this works but I'd rather change it in my code when I know the exact size rather than setting an arbitary limit. Is this possible?

I managed to change the size of the instance of the sequence but when sending it I got an error saying RTICdrStream_serializePrimitiveSequence:sequence length (1000) exceeds maximum (100), and I noticed that the 100 limit is in the rtiddsgen generated code in the serialize function.

Do I need to just write my own serialize functions or am I missing something?

Thanks

Caleb

Organization:
Gerardo Pardo's picture
Offline
Last seen: 3 weeks 14 hours ago
Joined: 06/02/2010
Posts: 589

Hello Caleb,

There are two ways to address the situation you describe. Unfortunately they both require some edits on the generated code:

  1. Modify the code generated by rtiddsgen to "trick" it into delaying the setting of the maximum length of the sequence
  2. Modify the type-plugin and configure the DataWriter and DataReader to use dynamic memory allocation for the sequence.

The most appropriate one depends on your actual use-case.

If all you need is to configure the maximum length at run-time from your program, and your program will know this length (both on the publisher and the subscriber side) before it ever uses the data-type, then the first approach (modifying the code generated by rtiddsgen) may be the simplest and most efficient.

If you have a more complex scenario then you may need to use the second approach and relay on dynamic memory allocation.

I will describe both so you can decide which one fits you best. What I describe here will work in C/C++ for Java the approach is a little bit easier. If that is the programming language you woudl want to use ley us know and we will try to come up with a Java example as well.

Approach 1: Modify the code generated by rtiddsgen to "trick" it into delaying the setting of the maximum length of the sequence

It is easiest to describe this with an example. I have attached the files I used so you can see whet I did. The example can be found attached to the end of this posting.

Say you define a data-type that contains a sequence as follows (see  SequenceWithRuntimeMaxLen.idl)

const long myAdjustableSeq_MAX_LEN=4;

struct SequenceWithRuntimeMaxLen {
    long key; //@Key
    sequence<long, myAdjustableSeq_MAX_LEN> myAdjustableSeq;
};

Then when you run rtiddsgen for a language like C or C++ it will create several files, the  relevant ones here are the ones that define the data-type and type-code namely:  SequenceWithRuntimeMaxLen.h and SequenceWithRuntimeMaxLen.cxx

annapurna: gerardo$  rtiddsgen -example x64Darwin10gcc4.2.1 -language C++ -namespace -replace SequenceWithRuntimeMaxLen.idl
Running rtiddsgen version 5.0.0, please wait ...
Done

The reason why you cannot change the maximum length at run-time can be seen by examining the  SequenceWithRuntimeMaxLen.h  On line 29 you can see there the declaration:

static const DDS_Long myAdjustableSeq_MAX_LEN = 4;

This is the variable that is used to size the maximum size of the sequence. Since this variable is declared static const in a header file it gets "compiled in" and cannot be changed at run-time.

To delay de setting of this variable all we need to do it replace that declaration in the header file with an external declaration as in:

// SequenceWithRuntimeMaxLen.h  line 29:
extern DDS_Long myAdjustableSeq_MAX_LEN;

Now the storage for the variable must be declared somewhere else. So we should add it to the SequenceWithRuntimeMaxLen.cxx

I edited SequenceWithRuntimeMaxLen.cxx  and added the declaration on line 40, right after the inclusion of the SequenceWithRuntimeMaxLen.h

// SequenceWithRuntimeMaxLen.c  lines 39-40:
#include "SequenceWithRuntimeMaxLen.h"
DDS_Long myAdjustableSeq_MAX_LEN = 4;

Now it is possible to set the myAdjustableSeq_MAX_LEN at run-time. You must do it before the call to SequenceWithRuntimeMaxLenTypeSupport::register_type() and you must to it consistently both in the publishing and subscribing sides.

I modified the generated examples SequenceWithRuntimeMaxLen_publication.cxx and SequenceWithRuntimeMaxLen_subscription.cxx  and set myAdjustableSeq_MAX_LEN to a larger value prior to calling SequenceWithRuntimeMaxLenTypeSupport::register_type()

// SequenceWithRuntimeMaxLen_publication.cxx lines 136-140

    /* Register type before creating topic */
    myAdjustableSeq_MAX_LEN = 10;

    type_name = SequenceWithRuntimeMaxLenTypeSupport::get_type_name();
    retcode = SequenceWithRuntimeMaxLenTypeSupport::register_type(
        participant, type_name);
// SequenceWithRuntimeMaxLen_publication.cxx lines 196-200

    /* Register the type before creating the topic */
    myAdjustableSeq_MAX_LEN = 10;

    type_name = SequenceWithRuntimeMaxLenTypeSupport::get_type_name();
    retcode = SequenceWithRuntimeMaxLenTypeSupport::register_type(
          participant, type_name);

To show it all works I modified the length of the sequence on the SequenceWithRuntimeMaxLen_publication.cxx to something greater than the original 4. 

// SequenceWithRuntimeMaxLen_publication.cxx lines 194-195

        /* Modify the data to be sent here */
        instance->myAdjustableSeq.length(8);

Now when I run the example applications I can see I get a sequence with a length of 8.

Output from the publisher:

annapurna: gerardo$ ./objs/x64Darwin10gcc4.2.1/SequenceWithRuntimeMaxLen_publisher 
Writing SequenceWithRuntimeMaxLen, count 0
Writing SequenceWithRuntimeMaxLen, count 1

Output from the subscriber:

annapurna: gerardo$ ./objs/x64Darwin10gcc4.2.1/SequenceWithRuntimeMaxLen_subscriber 

SequenceWithRuntimeMaxLen subscriber sleeping for 4 sec...
   key: 0
   myAdjustableSeq:
      myAdjustableSeq[0]: 0
      myAdjustableSeq[1]: 0
      myAdjustableSeq[2]: 0
      myAdjustableSeq[3]: 0
      myAdjustableSeq[4]: 0
      myAdjustableSeq[5]: 0
      myAdjustableSeq[6]: 0
      myAdjustableSeq[7]: 0
SequenceWithRuntimeMaxLen subscriber sleeping for 4 sec...
   key: 0
   myAdjustableSeq:
      myAdjustableSeq[0]: 0
      myAdjustableSeq[1]: 0
      myAdjustableSeq[2]: 0
      myAdjustableSeq[3]: 0
      myAdjustableSeq[4]: 0
      myAdjustableSeq[5]: 0
      myAdjustableSeq[6]: 0
      myAdjustableSeq[7]: 0

Approach 2: Modify the type-plugin and configure the DataWriter and DataReader to use dynamic memory allocation for the sequence.

This approach also modifies the type plugin code to do dynamic memory allocation for the sequence. It also takes advantage of a new feature introduced by RTI Connext DDS 5.0.0 that allows configuring the DataWriter cache to use dynamic memory for buffers that exceed a given size.

With this approach the maximum-length of the sequence becomes a property of each specific data-object and not a global property on the data-type.  The example can be found attached to the end of this posting.

To achieve this we need the following 5 steps:

STEP 2.1) Set the maximum length to some very large value, larger than it will be needed, but not so large that it overflows integers when computing the size on bytes for the arrays.

This does not waste space because with the changes we are making we never allocate anything to the maximum declared size. However there are checks in the serialize/deserialize functions that check that the length does not exceed this maximum so we need to set it to something large.

I used the value 1000000 so my test IDL looks like:

const long myAdjustableSeq_MAX_LEN=10000000;

struct SequenceWithRuntimeMaxLen {
  long key; //@Key
  sequence<long, myadjustableseq_max_len="" >="" myadjustableseq;="" };="">

STEP 2.2) Modify the type plugin to initialize the sequence with a zero maximum size, rather than the maximum:

This is done modifying the SequenceWithRuntimeMaxLen_initialize_ex() function found in SequenceWithRuntimeMaxLen.c

Replace:

        if (!DDS_LongSeq_set_maximum(&sample->myAdjustableSeq,
                ((myAdjustableSeq_MAX_LEN)))) {

            return RTI_FALSE;
        }

With:

        if (!DDS_LongSeq_set_maximum(&sample->myAdjustableSeq, 0)) {
            return RTI_FALSE;
        }

STEP 2.3) Modify the type plugin deserialization to set the size of the sequence to fit what it needs to accomodate the sample received.

This is done modifying the SequenceWithRuntimeMaxLenPlugin_deserialize_sample() function found in SequenceWithRuntimeMaxLenPlugin.cxx

Code added to SequenceWithRuntimeMaxLenPlugin.cxx line 454 after the declaration of

RTICdrUnsignedLong sequence_length;

// Peek into the stream to get the length of the sequence
RTICdrStream_deserializLong(stream,  &sequence_length);

// Rewind the stream 4 bytes to leave it as it was before we peeked
RTICdrStream_incrementCurrentPosition(stream, -4);

// Set the maximum length of the sequence to the actual length
DDS_LongSeq_set_maximum(&sample->myAdjustableSeq, sequence_length);

STEP 2.4) Modify the type plugin to set the maximum to zero again when is no longer needed. This releases the memory for the sequence elements

This is done modifying the SequenceWithRuntimeMaxLenPlugin_return_sample() which is a macro defined in SequenceWithRuntimeMaxLenPlugin.h  

I commented out that definition SequenceWithRuntimeMaxLenPlugin_return_sample in SequenceWithRuntimeMaxLenPlugin.h  line 53:

// #define SequenceWithRuntimeMaxLenPlugin_return_sample PRESTypePluginDefaultEndpointData_returnSample 

 

And added the declaration of this function in SequenceWithRuntimeMaxLenPlugin.c to first set the maximum length of the sequence to zero:

void SequenceWithRuntimeMaxLenPlugin_return_sample(
    PRESTypePluginEndpointData endpointData,
    void *sample, void *handle);
{
    SequenceWithRuntimeMaxLen * sampleData = sample;
    DDS_LongSeq_set_maximum(&sample->myAdjustableSeq, 0);
    PRESTypePluginDefaultEndpointData_returnSample(endpointData, sample, handle);
}

STEP 2.5) Configure the data-writer to use dynamic memory

By default a DataWriter allocates buffers to hold serialized samples of the maximum size. This can be wasteful for data-types where the average sample is significantly smaller that the largest possible sample.  Specifically if a data-type contains a sequences the DataWriter by default allocates buffers sized to hold samples where the sequence has the maximum declared size.

RTI Connext DDS version 5.0.0 introduced a new feature that can configure a DataWriter to use dynamic memory allocation for large samples.  With this feature the DataWriter allocates buffers to only hold a configured 'pool_buffer_max_size' and dynamically allocates the buffer if it needs to serialize a sample that does not fit on that size.  More details on this feature see Chapter 20 of The RTI Connext DDS Users's Guide http://community.rti.com/rti-doc/500/ndds.5.0.0/doc/pdf/RTI_CoreLibrariesAndUtilities_UsersManual.pdf

The configuration of the DataWriter to use dynamic allocation for large samples is done using the PropertyQoSPolicy in the DataWriterQoS. The property that configures it is called "dds.data_writer.history.memory_manager.fast_pool.pool_buffer_max_size"

I configured this by editing the USER_QOS_PROFILE.xml located in the working directory of the application. The  USER_QOS_PROFILE.xml  was generated by rtiddsgen and already contains a default configuration for the DataWriter QoS. I modified the configuration and added:

	        <property>
		    <value>
		        <element>
			  <name> dds.data_writer.history.memory_manager.fast_pool.pool_buffer_max_size</name>
			  <value>1000</value>
			</element>
		    </value>
		</property>

Directly inside the <datawriter_qos>  element.

VERIFICATION STEP) 

To show it all works I modified both the maximum-length and the length of the sequence on the SequenceWithRuntimeMaxLen_publication.cxx each time I publish a sample. Note that this does not really prove that the memory is not being allocated to the maximum size, but with the changes we made it should not happen.

// SequenceWithRuntimeMaxLen_publication.cxx lines 194-195

        /* Modify the data to be sent here */
	instance->myAdjustableSeq.maximum(1 + 3*count);
        instance->myAdjustableSeq.length(1 + 3*count);

Now when I run the example applications I can see I get a sequence with a varying length

Output from the publisher:

annapurna: gerardo$ ./objs/x64Darwin10gcc4.2.1/SequenceWithRuntimeMaxLen_publisher 
Writing SequenceWithRuntimeMaxLen, count 0
Writing SequenceWithRuntimeMaxLen, count 1

Output from the subscriber:

annapurna: gerardo$ ./objs/x64Darwin10gcc4.2.1/SequenceWithRuntimeMaxLen_subscriber 
SequenceWithRuntimeMaxLen subscriber sleeping for 4 sec...
SequenceWithRuntimeMaxLen subscriber sleeping for 4 sec...
   key: 0
   myAdjustableSeq:
      myAdjustableSeq[0]: 0
      myAdjustableSeq[1]: 0
      myAdjustableSeq[2]: 0
      myAdjustableSeq[3]: 0
SequenceWithRuntimeMaxLen subscriber sleeping for 4 sec...

   key: 0
   myAdjustableSeq:
      myAdjustableSeq[0]: 0
      myAdjustableSeq[1]: 0
      myAdjustableSeq[2]: 0
      myAdjustableSeq[3]: 0
      myAdjustableSeq[4]: 0
      myAdjustableSeq[5]: 0
      myAdjustableSeq[6]: 0
SequenceWithRuntimeMaxLen subscriber sleeping for 4 sec...
   key: 0
   myAdjustableSeq:
      myAdjustableSeq[0]: 0
      myAdjustableSeq[1]: 0
      myAdjustableSeq[2]: 0
      myAdjustableSeq[3]: 0
      myAdjustableSeq[4]: 0
      myAdjustableSeq[5]: 0
      myAdjustableSeq[6]: 0
      myAdjustableSeq[7]: 0
      myAdjustableSeq[8]: 0
      myAdjustableSeq[9]: 0
File Attachments: 
frgo's picture
Offline
Last seen: 1 year 7 months ago
Joined: 12/28/2011
Posts: 20

Hi Gerardo,

approach 2 is the solution I need for all my (quite large) sequences. It is really cumbersome to have to introduce all this manually - especially if you have like more than 50 sequences defined, as is the case here.

It would be so much more productive to have this as an option to rtiddsgen. As it seems it all can be done in the code generation templates used by rtiddsgen. So, no big deal - just let the user decide which directory to use for reading the code templates...

Where do I have to file an enhancement request ? ;-)

Thanks!

Regards

  Frank

rip
rip's picture
Offline
Last seen: 1 day 2 hours ago
Joined: 04/06/2012
Posts: 316

Hallo Frank,

It's a known issue, and is already a request.  :D Ich habe es ja eben SELBST in der Verbesserungsdatenbank eingefuehrt, und zwar vor einem halben Jahr schon...

We are looking at a solution for a future release.  Depending on whether the enhancement request is completed in time (Which I'm agitating for*), the out of the box solution will be sufficient, or may only be a partial solution.  In the event of a partial solution, I plan to supply a process workaround (+ java application) as experimental code. 

MfG,

Rip
--
Richard Williamson
Field Application Engineer
RTI
 
* Man kann auch sagen, dass ich da oben Deutsch benutzt habe, nuer als reine Agitproptheater...
frgo's picture
Offline
Last seen: 1 year 7 months ago
Joined: 12/28/2011
Posts: 20

Hi Rip,

thanks for jumping in. And "Danke - Woher können Sie so gut Deutsch?" ;-) 

Is there a way to "up-vote" this feature request? I am still in the "idea phase" of the application I'm building and therefore have very frequent changes to the IDL files and the data defined therein. I am already thinking of building my own "frgoddsgen" to support loading a different set of templates from a user specified directory ...

Any time horizon for a release of the programmin tools supporting this?

Thx and kind regards

  Frank

 

rip
rip's picture
Offline
Last seen: 1 day 2 hours ago
Joined: 04/06/2012
Posts: 316

We have a log for tracking these things, so it's already counted as an up-vote.   We haven't been given a release date, sorry.

... Ach, Sie wissen wie es geht... auf Partys und so ... bekam ich immer die gleiche Frage:  Sprechen Sie deutsch? ... 'Eh. Laut meiner (ehemaligen) Frau, kein einziges Wort..."  

rip

frgo's picture
Offline
Last seen: 1 year 7 months ago
Joined: 12/28/2011
Posts: 20

@rip


Hi rip,

any news on this ? I still need this feature. Rather urgently now. Manipulating generated code is a major nuissance...

Thanks and HNY !


Kind regards

  Frank

 

rip
rip's picture
Offline
Last seen: 1 day 2 hours ago
Joined: 04/06/2012
Posts: 316

Hi Frank,

In order to easily implement the code template changes, (for this, among other changes that we'd like to make, and the key word here is "easily"), we're moving to a new code generation engine. 

rtiddsgen "pre 1.0" was based on a perl script (I believe...before my time at RTI).

rtiddsgen "1.0" (the current one) was based on xslt transforms.  Changing the templates was very, very painful.

rtiddsgen2 is based off the Velocity Template Engine.  It uses "plain text" templates that are marked up in such a way that the engine can compile them using information in a tree generated from the IDL.

Trying to change the xslt to support your use case would have been a non-starter, due to the complexity of the changes, the underlying format (xslt) and the fact that the xslt method was Going Away.

With the next release (Real Soon Now) we'll be able to implement the above ("Approach 2") in the templates, and it should be straightforward.  Are you on Windows?  I could look at the pre-release templates and investigate if this as "straightforward" as it looks.

Regards,

Rip

rip
rip's picture
Offline
Last seen: 1 day 2 hours ago
Joined: 04/06/2012
Posts: 316

Hi Frank,

NOTE:  The following is NOT SUPPORTED.  No warranty is expressed or implied.  Make applicable backups before hacking the template files!  This is extremely brittle and may not work in all cases.  If you bump into an edge case, create an idl that reproduces it and post it on this thread.

Make the following changes to %NDDSHOME%\ndds.5.1.0\resource\rtiddsgen\templates\c:

File typeMacros.vm line 1197, Replace:

 

--*##if($member.dimensionList)[i]#end, ($member.seqSize) )) {

with:

--*##if($member.dimensionList)[i]#end, /* rip ($member.seqSize) */ (0) /* /rip */)) {

 

File typePluginBody.vm Line 1734, after "...finalize_optional_members..." line, Add:

/* rip */
#*--*##foreach($member in $node.memberFieldMapList)
#*----*##if($member.seqSize)
${member.nativeTypeSeqFQName}_set_maximum(${member.sampleRef}, 0);
#*----*##end
#*--*##end
/* /rip */

 

File: typePluginMacros.vm Line 1588, after the RTICdrUnsignedLong sequence_length line, Add:

/* rip */
    // Peek into the stream to get the length of the sequence
RTICdrStream_deserializeLong(stream,  &sequence_length);

    // Rewind the stream 4 bytes to leave it as it was before we peeked
RTICdrStream_incrementCurrentPosition(stream, -4);

    // Set the maximum length of the sequence to the actual length
DDS_LongSeq_set_maximum(${member.sampleRef}#if($member.dimensionList)[i]#end, sequence_length);
/* /rip */

 

In directory c:\RTI\ndds.5.1.0\resource\rtiddsgen\templates\example,

File qosProfile.vm Line 142, Insert after </publication_name> and before </datawriter_qos>:

                <!-- rip -->
                <property>
                    <value>
                        <element>
                            <name> dds.data_writer.history.memory_manager.fast_pool.pool_buffer_max_size </name>
                            <value>1000</value>
                        </element>
                    </value>
                </property>
                <!-- /rip -->

 

Result:

c:\Users\rip\Documents\tmp>.\objs\i86Win32VS2010\FooType_subscriber.exe
FooType subscriber sleeping for 4 sec...
FooType subscriber sleeping for 4 sec...
FooType subscriber sleeping for 4 sec...
FooType subscriber sleeping for 4 sec...
FooType subscriber sleeping for 4 sec...

   key: 0
   myAdjustableSeq:
      myAdjustableSeq[0]: 0
      myAdjustableSeq[1]: 0
      myAdjustableSeq[2]: 0
      myAdjustableSeq[3]: 0
FooType subscriber sleeping for 4 sec...

   key: 0
   myAdjustableSeq:
      myAdjustableSeq[0]: 0
      myAdjustableSeq[1]: 0
      myAdjustableSeq[2]: 0
      myAdjustableSeq[3]: 0
      myAdjustableSeq[4]: 0
      myAdjustableSeq[5]: 0
      myAdjustableSeq[6]: 0
FooType subscriber sleeping for 4 sec...

   key: 0
   myAdjustableSeq:
      myAdjustableSeq[0]: 0
      myAdjustableSeq[1]: 0
      myAdjustableSeq[2]: 0
      myAdjustableSeq[3]: 0
      myAdjustableSeq[4]: 0
      myAdjustableSeq[5]: 0
      myAdjustableSeq[6]: 0
      myAdjustableSeq[7]: 0
      myAdjustableSeq[8]: 0
      myAdjustableSeq[9]: 0
FooType subscriber sleeping for 4 sec...

   key: 0
   myAdjustableSeq:
      myAdjustableSeq[ 0]: 0
      myAdjustableSeq[ 1]: 0
      myAdjustableSeq[ 2]: 0
      myAdjustableSeq[ 3]: 0
      myAdjustableSeq[ 4]: 0
      myAdjustableSeq[ 5]: 0
      myAdjustableSeq[ 6]: 0
      myAdjustableSeq[ 7]: 0
      myAdjustableSeq[ 8]: 0
      myAdjustableSeq[ 9]: 0
      myAdjustableSeq[10]: 0
      myAdjustableSeq[11]: 0
      myAdjustableSeq[12]: 0
FooType subscriber sleeping for 4 sec...
^C
c:\Users\rip\Documents\tmp>

 

Offline
Last seen: 2 months 3 weeks ago
Joined: 03/25/2015
Posts: 33

@RIP / Gerardo

Is the enhancement gone into any release? I am also currently looking into dynamic arrays. I don't understand why it is not supported in first place - probably it is inherently limited by the DDS design and performance.. Not sure. So, even taking DDS5.2 into consideration do we only have the above 2 approaches? Anyway approach #1 is compile-time approach. Effectively there is only one approach (#2) for run-time handling of array size ?

@Frank

Did you follow the same approach (#2) or is there any alternative? It is not a pain, but with approach #2 we will have to maintain the generated code too as part of versioning system.

Thanks.

Uday

Offline
Last seen: 1 year 11 months ago
Joined: 09/17/2015
Posts: 53

Yes, I described it in one of my posts. It is pretty easy now, but you need the right QOS settings.

https://community.rti.com/forum-topic/unbounded-sequences

 

Offline
Last seen: 2 months 3 weeks ago
Joined: 03/25/2015
Posts: 33

Hmmm.. now after seeing the post and some related information in manual, I feel like sticking to bounded sequences but with some inheritance-kind of approach where I can define the 2 bounds appropriately. The reason for leaning towards this is that I dont want dynamic allocation during serialization and deserialization as in my case the publisher keeps sending the data at good rate and subscriber needs to keep up with it.

Thanks.

Uday

Offline
Last seen: 1 year 11 months ago
Joined: 09/17/2015
Posts: 53

Note that dynamic memory allocation only occurs if data is larger than a certain size. This limit can you configure yourself and tweek performance accordingly.