Inherited dds topic structs

21 posts / 0 new
Last post
Offline
Last seen: 5 years 4 months ago
Joined: 03/25/2015
Posts: 33
Inherited dds topic structs

Hi,

I am looking for inheritance kind of model with DDS topics. For example, I want the listener/receiver to listen to "one" topic (say, Shape) but the sender to send any of the inherited topics (say, square, rectangle etc..). Is it possible to achieve this in DDS first of all?

I have specific usecases for this. So trust me, when I say I need it.

Thanks in advance.

Uday

Organization:
Offline
Last seen: 11 months 2 days ago
Joined: 02/11/2016
Posts: 144

Hello udayk,

I actually went over the posts in the forums recently and saved some interesting posts, including this one.

To sum up how, to my understanding, this post is relevant to your question:

While you can set a reader to support type A and have various writers writing different types that extend A, and while data written by those writers will reach the reader, the reader will only be able to extract data that exists in type A.

That is, you will be able to receive the general shape properties sent by the "square" writer, by the "rectangle" writer, etc, but you will not be able to get the extra rectangle data.

In such situations I recommend one of the following alternatives:

1. Make the shape properties part of a shared struct that will be a field within triangle/rectangle/square

2. Use unions (similar to the first option)

3. If possible, use an enum to denote shape kind per sample and allow all "shape" readers and writers to send all shape types (this will, of course, introduce difficulties in creating shape type specific logic)

 

There may be other alternatives I haven't thought of but hopefully these will do for your use case.

 

Good luck!

Fernando Garcia's picture
Offline
Last seen: 4 months 4 weeks ago
Joined: 05/18/2011
Posts: 200

Hi udayk,

You can alternatively use Extensible types for your scenario. For instance, if you defined a ShapeType and a ShapeTypeExtended:

enum ShapeFillKind {
    SOLID_FILL,
    TRANSPARENT_FILL,
    HORIZONTAL_HATCH_FILL,
    VERTICAL_HATCH_FILL
};

struct ShapeType {
    string<128> color; //@key
    long x;
    long y;
    long shapesize;
};//@Extensibility EXTENSIBLE_EXTENSIBILITY

struct ShapeTypeExtended : ShapeType {
    ShapeFillKind fillKind;
    float angle;
};//@Extensibility EXTENSIBLE_EXTENSIBILITY

Your subscriber application could create a Square topic using ShapeType, which would allow it to receive both other Squares defined as ShapeType and Squares that contain the extra fields in ShapeTypeExtended.

For more information on how to use Extensible Types check out the following section in the User's Manual. Note that this example is taken from RTI Shapes Demo, which enables you to publish Shapes using ShapeTypeExtended or ShapeType.

Best,
Fernando.

Offline
Last seen: 11 months 2 days ago
Joined: 02/11/2016
Posts: 144

Hey Fernando,

I don't know so for sure but given what udayk wrote in his original post I have a hard time believing your solution works for him:

1. The subscriber, subscribing on ShapeType, will not get the added information that is sent by a publisher that publishes ShapeTypeExtended

2. Even with a solution that reverses the case (say, the subscriber subscribes to ShapeTypeExtended and is therefore able to receive all the data published by ShapeType publishers AND ShapeTypeExtended publishers) he will still not be able to get the polymorphic behavior he seems to crave (as is suggested by "I want the listener/receiver to listen to "one" topic (say, Shape) but the sender to send any of the inherited topics")

 

Perhaps I'm assuming too much regarding udayks' needs, but I believe that isn't the case.

 

Roy.

Offline
Last seen: 5 years 4 months ago
Joined: 03/25/2015
Posts: 33

Roy.. You understood my intent well. And after going through the Fernando's post, I too felt the same.. that it wouldn't solve the problem I am trying to solve.. I couldn't get time to experiment over the options that are listed in both of your posts, and moreover I am an vacation this week :-).. 

I have gone through the user manual on extensible types. It sounds like there is no way to solve the problem in exactly the same way I intend to solve (C++ kind of polymorphism). The best option, as of now, I could think of is to define a single topic struct with simulating the fields in extended structs as "optional"..

Unions is also an option.. Infact, in my current implementation I had to go for that instead of optional fields as it was in DDS5.0.. Now that I am moving to DDS5.2 I can experiment with 'optional'..

Let me know if you guys think of any better ideas..

Thanks,

Uday 

Offline
Last seen: 11 months 2 days ago
Joined: 02/11/2016
Posts: 144

I feared as much.

If you gave an explanation about your "problem" and current architecture perhaps we could suggest alternative architectures that don't require c++ like polymorphism.

Roy.

Offline
Last seen: 5 years 4 months ago
Joined: 03/25/2015
Posts: 33

Ok.. Let me put it in the best possible way I can..

I have to publish my "data" for further processing. In some cases, the data is needed by external subsystems and in other cases it is needed by the same subsystem. Now, the data is of different kinds. There are basic set of fields that are needed to be populated in each of the kind. On top of that some additional fields are populated in those different kinds. The data I am publishing is basically image data. Since there are different kind of images there are some additional information that is needed depending on the type of image. Now, this is at the publishing side.

The subscriber basically listens to these images and constructs a single entity. The single entity holds information appropriately depending on the type of image - basically some 'null' fields in those that are not applicable. There is nothing that can be done from the perspective of single entity construction. The current architecture has the communication done in raw bytes - like that of socket communication, unlike DDS which is data-based. The structure that constitutes the raw format basically holds information of all kinds of images . The subscriber, depending, on the kind of image accesses the appropriate fields. This is basically in the same lines as the one I mentioned in my previous post - taking approach of "optional" fields.

I can define different topic structs for each kind of image, and let the subcriber listen to each of the topics. But, I didn't like that approach due to scalability reasons. I want the subscriber listen to one kind of topic - that is my end target.

Hope you got my thinking this time.

Uday

Offline
Last seen: 11 months 2 days ago
Joined: 02/11/2016
Posts: 144

Hey udayk,

 

I'll tell you why I think the situation you are describing is not as problematic as you first portrayed it but let me ask some questions:

1. Does the subscriber NEED the extra data that some formats can supply? If so, how does it handle images in formats where this data does not exist.

2. Does the subscriber WANT the extra data that some formats can supply? If so, why not have the subscriber subscribe to an additional topic, where "additional data relevant to a certain format" can be sent if an image is of that format?

3. Does the (supposedly polymorphic) subscriber not care about the additional fields? If so, you can simply have it listening on type Image and have the publishers publishing various extensions of Image (so that only fields of Image are received by this subscriber and additional fields relevant to each extension can be received by other subscribers interested in that specific type)

 

Of course, if the subscriber wants the additional fields you could also use a large type with optional fields but that architecture tends to be quite ugly.

.

I hope this helped a bit.

 

Gerardo Pardo's picture
Offline
Last seen: 1 month 1 week ago
Joined: 06/02/2010
Posts: 602

Hello Uday,

If I understood correctly, you can accomplish what you want with XTYPES as long as the subscriber is aware of the whole set of types that can be published. If that is not the case then you would need to define an extensibility mechanism inside the type itself. I will cover that later.

KickR's description of the XTYPES behavior on his first post is exactly right:

While you can set a reader to support type A and have various writers writing different types that extend A, and while data written by those writers will reach the reader, the reader will only be able to extract data that exists in type A.

The reason is this: Unlike the situation with C++/Java/C# polymorphism, the DataReader only has compiled-in knowledge for Type A. So even if it receives data of other types A1, A2 compatible (or derived) from A, it can only treat that type as an "A" type as it is the only code that it ever compiled. However, it is true that would be possible to get the derived types but doing that would require using a dynamic data API to read the data (as there is no compile-time code of anything other than A) and also would require using dynamic deserialization that looks at the type serialized by the sender. This is possible, but when we created XTYPES we considered that this was going to end up being quite complex and heavy and, therefore, settled for rules that allowed data to be de-serialized looking only at the (single) type the reader knows (i.e. not  looking into each message to dispatch based on the type the writer sent).

Now, assuming the DataReader does know what all the possible types are, then you can use the following approach to effectively getting the "polymorphic" behavior:

Define your type hierarchy using MUTABLE extensibility, and define the added members as OPTIONAL members. For example:

struct BaseImage {
   long              image_type;   //@ID 0
   string            format;       //@ID 1
   string            description;  //@ID 2
   sequence          image_data;   //@ID 3
 
}; //@Extensibility MUTABLE_EXTENSIBILITY

// Derived types are careful to define their members names and ID values 
// to not collide with each other
struct ImageKind1 : BaseImage {
   // Extra stuff for ImageKind1
   long    image1_extra1;  //@ID 101
                           //@Optional
   string  image1_extra2;  //@ID 102
                           //@Optional
   //  ...
}; //@Extensibility MUTABLE_EXTENSIBILITY
 
struct ImageKind2 : BaseImage {   
   // Extra stuff for ImageKind2 
   long  image2_extra1;  //@ID 201 
                         //@Optional
   long  image2_extra2;  //@ID 202  
                         //@Optional
   //  ...
}; //@Extensibility MUTABLE_EXTENSIBILITY


// We need a receiver "consolidated" or type that combines all member IDs
// This can be done using inheritance or copying the members 
// as lonf as member ID and names were chosen to be different on each derived type 
// it is possible to build a ConsolidatedImage  
struct ConsolidatedImage : BaseImage  {   
  // Extra stuff for ImageKind1
   long    image1_extra1;  //@ID 101
                           //@Optional
   string  image1_extra2;  //@ID 102
                           //@Optional
   
   // Extra stuff for ImageKind2 
   long  image2_extra1;  //@ID 201 
                         //@Optional
   long  image2_extra2;  //@ID 202  
                         //@Optional
}; //@Extensibility MUTABLE_EXTENSIBILITY
 

Note that since structs only support single inheritance each time you need to  "inherit" from more than one type you need to "copy" the elements inside instead. This is what I had to do for the ConsolidatedImage which logically would want to also "inherit" from ImageKInd1 and ImageKind2.

Now the receiver has all the code needed to recognize any kind of image. It can recognize the actual image either using the image_type member, or alternatively by checking the values of the added members (image1_extra1, etc.). Since these were marked //@Optional the receiver will set those members to "NULL" pointers if they were not part of the type sent.

The above approach works as long as you can compile the reader with the knowledge of all the types. If that is not the case you would have to use a different approach, such as defining something inside the type that you can use for effectively extending it, without actually changing the type itself. For example:

For example:

  
struct NameValuePair {
    string name;
    string value;
};
typedef sequence NameValuePairSequence;

struct NameBinaryValuePair {
    string name;
    sequence value;
}; 
typedef sequence NameBinaryValuePairSequence;

struct ExtensibleImage {
   long              image_type;   
   string            format;       
   string            description;  
   sequence   image_data;   
   
   NameValuePairSequence       string_nvps;
   NameBinaryValuePairSequence binary_nvps;
}; 
 

This approach allows you to put extra info into the ExtensibleImage using the string_nvps and binary_nvps fields. But if you want to put structured data then have to do the encoding/decoding yourself...

-Gerardo

Offline
Last seen: 11 months 2 days ago
Joined: 02/11/2016
Posts: 144

Hey Gerardo,

Thank you for verifying what I said and also for a detailed explanation of reasonable solutions.

Obviously both of them have their own disadvantages but then this is a tough case for DDS to handle.

Your 2nd solution actually relates to a thought I had about using JSON over DDS to gain much more flexibility (obviously at the price of performance, among other things) without losing the discovery and multicast abilities of DDS.

Food for thought, I suppose.

 

Roy.

Gerardo Pardo's picture
Offline
Last seen: 1 month 1 week ago
Joined: 06/02/2010
Posts: 602

Hi,

Yes adding a JSON string inside a string to get extensibility will give you similar capabilities. But if you used JSON as the actual way to encode all your data (i.e. you told DDS it was a string but fill-in a JSON-formatted string) then, performance aside, there would be other side-effects:

You would lose type-compatibility checking. 

You would not be able to leverage content-filtered Topics, unless you define custom filters that are able to understand the JSON format

The tools (DataVisualizationAdminConsole, recording/replay, etc.) would not be able to understand the content beyond it being a string so you would loose some capabilities.

So  I think a mixed approach where you define some types fully in the IDL and perhaps have extra 'string' fields to fill with data encoded in other formats jile JSON or name-value pairs is a good compromise...

Gerardo

Offline
Last seen: 11 months 2 days ago
Joined: 02/11/2016
Posts: 144

Hey,

Firstly, I wrote "among other things" :)

Of course I realize there are other prices to pay when using DDS to send JSON.

By "Food for thought" I meant that I need to consider those prices against the benefits my system could gain from such an approach (mainly type-flexibility)

 

Roy.

Offline
Last seen: 5 years 4 months ago
Joined: 03/25/2015
Posts: 33

Back from vacation, so late in replying.

Thanks to all for your valuable suggestions.

From what I understood till now from Gerardo's and Roy's posts is that the subscriber should be aware of all the types at compilation time. I think that is fine assumption to me, as in my case I dont intend the subscriber to handle something that it can't recognize as it would have image-type-dependent logic in it's subsystem. The only ambitious thinking I had is that depending on the 'image-type' I can cast to an appropriate struct (subscriber side) - similar to how we do dynamic casting. It is clear to me, now, that this is not possible. I can have the subscriber subscribe to multiple topics where each of them are designed in an inheritance approach - similar to what Gerardo has posted.

Name value pair approach doesn't work for me as it is not just additional images but it could be additional fields which can be of boolean/int/short/double/string etc..

@Gerardo - I have one question from your post. What is the point of having 'Optional' fields in an inherited topic? I thought the approach should be either 'optional fields' or 'inherited topic'.

 As mentioned, my original design (before transitioning to DDS5.2) was based on unions.. It looks like this..

struct TOPIC_STURCT {

// all the common stuff

EXTRA something1;

EXTRA something2;

}

union EXTRA switch (EXTRA_TYPE) {

case EXTRA_TYPE1: // type1 fields

case EXTRA_TYPE2: // type2 fields

case EXTRA_NONE: // empty byte

}

With DDS5.2, it can be like this:

struct TOPIC_STURCT {

// all the common stuff

type1 fields; // @Optional

type2 fields; // @Optional

}

OR

struct TOPIC_STURCT {

// all the common stuff

}

struct EXTRA1_TOPIC_STRUCT : TOPIC_STRUCT {

// type1 fields

}

struct EXTRA2_TOPIC_STRUCT : TOPIC_STRUCT {

// type2 fields

}

Did I summarize your suggestions correctly?

Thanks.

Uday

Offline
Last seen: 11 months 2 days ago
Joined: 02/11/2016
Posts: 144

Hey udayk,

You said "Name value pair approach doesn't work for me as it is not just additional images but it could be additional fields which can be of boolean/int/short/double/string etc.." but you should note that Gerardo did say that if you want to use other kinds of data (apart from Strings and Bytes) you will have to also implement application level encoding / decoding (as is common for people using tcp/udp sockets for communication over binary streams or people using websockets/http for communication over strings).

About your questions regarding optional and inherited:

Optional (for mutuable extensibility) means that a field may be sent empty (and thus, they can also be expected to sometimes arrive as "empty").

That allows the reader (which can get just the basic fields which are not optional, can get the basic fields + the optional fields of of image kind 1, or can ge the basic fields + the optional fields of image kind2) to have a built in "empty" value (instead of having a "default" value which may cause some problems in detecting that a field was not sent by the writer).

Inheritance doesn't contradict with the above, but allows for a better understood structure model.

Regarding your last question:

No.

That is, if you defined your structs like the 2nd option you listed:

You are able to send the common stuff from EXTRA1 and from EXTRA2 to the TOPIC_STRUCT but you cannot create a reader which can receive all of the data from both extended types.

To complete this option you need to definte a consolidating type which inherits topic_struct and also defines type1 fields and type2 fields (or a type which inherits type1 and also defines type2 fields).

In all of the above options you must make sure that the ordinals are a. consistent between types (field x in type extra1 should have the same ordinal as field x in a consolidated type) and do not contradict (field x of type 1 should not have the same ordinal as field y of type 2).

 

Good luck,

Roy.

Offline
Last seen: 5 years 4 months ago
Joined: 03/25/2015
Posts: 33

Hi Roy,

I understand that having optional fields in the inherited topic is necessary only when I am not sure if all the fields are populated in it. The reason for mentioning "either of the approach" is good is that I didn't have a case in mind where I would want to send only few of the fields in the inherited topic.

And the reason for not considering consolidated image struct at the receiver is that I am leaning towards having the subscriber listen to "both EXTRA1_TOPIC_STRUCT and EXTRA2_TOPIC_STRUCT". If I want it to listen to just one topic, then yes - defining consolidated image is necessary..

Thanks.

Uday

Offline
Last seen: 5 years 4 months ago
Joined: 03/25/2015
Posts: 33

I have one basic doubt. In the approach which Gerardo mentioned, and I am leaning to finally, how does the discovery happen? On the publisher side I have either ImageKind1, ImageKind2 or BaseImage (from Gerardo's illustration). But on the other end (subscriber side) I have one single struct - ConsolidatedImage. I am trying to understand what goes on behind the scenes.

Thanks.

Uday

Offline
Last seen: 11 months 2 days ago
Joined: 02/11/2016
Posts: 144

This: https://community.rti.com/rti-doc/500/ndds.5.0.0/doc/pdf/RTI_CoreLibrariesAndUtilities_GettingStarted_ExtensibleTypesAddendum.pdf (and specifically section 2.3.1) should help you.

Note that it also directs you at the XTypes documentation which may come in handy.

 

Roy.

Offline
Last seen: 5 years 4 months ago
Joined: 03/25/2015
Posts: 33

Roy, Fernanado, Gerardo,

I am doing size analysis of topic size with the options at my hand.. My current design (based on 5.0) is based on "Unions". The max serialized data of the type turns out to be 92750096.. By just replacing the union based fields with optional fields the size came down to 46466032.. I am yet to find the size when I design the same with inheritance approach..

My question is does this size (serialized_sample_max_size) play any role in performance? If yes, how to address such a case?

Thanks.

Uday

Offline
Last seen: 5 years 4 months ago
Joined: 03/25/2015
Posts: 33

Hi,

I tried implementing, in my scratchbook, pub-sub of inherited types. Not sure what I am missing, but the discovery seems to be not happening between the inherited types. I have attached the files, in zip format, for better understanding of the situation.

The zip file holds 4 files:

- ImagePayload.idl, ImagePayload_publisher.cxx, ImagePayload_subscriber.cxx, qos.xml

The typed I defined fall in the same line Gerardo mentioned in this post..

const unsigned long MAX_IMAGE1_WIDTH=512;
const unsigned long MAX_IMAGE1_HEIGHT=512;

const unsigned long MAX_IMAGE2_WIDTH=828;
const unsigned long MAX_IMAGE2_HEIGHT=3500;

struct ImageDim{
long width;
long height;
}; //@Extensibility MUTABLE_EXTENSIBILITY

struct Image1 : ImageDim {
sequence<short,MAX_IMAGE1_WIDTH*MAX_IMAGE1_HEIGHT> image1_data; // @ID 101
// @Optional
}; //@Extensibility MUTABLE_EXTENSIBILITY

struct Image2 : ImageDim {
sequence<short,MAX_IMAGE2_WIDTH*MAX_IMAGE2_HEIGHT> image2_data; // @ID 201
// @Optional
}; //@Extensibility MUTABLE_EXTENSIBILITY

struct ConsolidatedImage : ImageDim {
sequence<short,MAX_IMAGE1_WIDTH*MAX_IMAGE1_HEIGHT> image1_data; // @ID 101
// Optional
sequence<short,MAX_IMAGE2_WIDTH*MAX_IMAGE2_HEIGHT> image2_data; // @ID 201
// @Optional
}; //@Extensibility MUTABLE_EXTENSIBILITY

On the publisher side I created a topic using 'Image1' type. And on the subscriber side I created a topic using 'ConsolidatedImage' type. I have 'discovery' check in the publisher - waiting for a reader. That check is not passing.

Can anybody help me on what could be missing? I checked the qos.xml file and I even see that 'DDS_ALLOW_TYPE_COERCION' is set.

Thanks.

Uday

 

File Attachments: 
Offline
Last seen: 11 months 2 days ago
Joined: 02/11/2016
Posts: 144

Hello udayk,

By setting the type consistency to be DDS_ALLOW_TYPE_COERCION (assuming you've verified that the qos that you believe that you are using really is being used) you've allowed for writers and readers using these different types to communicate.

However, as specified in the link I gave you in my last post, there are more requirements to be met for that to work.

Primarily, as I understand, the reader and writer need to use TypeObjects during discovery for a match to be made.

I am not certain if mutable extensibility is fully supported in rti dds 5.0.0 but it would seem unlikely that even a simple case like this wouldn't work.

I would recommend trying ImageDim and Image1 communication (in both ways) to see how big your problem may be.

 

Good luck,

Roy.

 

Offline
Last seen: 5 years 4 months ago
Joined: 03/25/2015
Posts: 33

It was a dumb mistake.. I thought of sharing it immediately..

I was using 'different' topic name (along with different type) both on subscriber side as well as publisher side.. I realized it only after seeing the visualized picture from rtiadminconsole. I was able to subscribe ConsolidatedImage while publishing Image1, after using a common topic name 'ImageTopic'.. This happens with 'create_topic_with_profile(...)' call..

Thanks.

Uday