Dynamic Discovery and Type Codes

Hi Arthur,

I will try to answer your questions:

1) Regarding how to publish data for a type/topic yoru discover dynamically:

If you know how to subscribe to data dynamically then the process to publish data dynamically is almost the same. You would follow the same steps to discover the Type and Topic, register the type with the local DomainParticipant and create the Topic. For details on this you can refer back to the MonitorData.java example and the forum thread titled DynamicData API usage for receiving arbitrary data on a topic.

After the Type has been registered and the Topic created instead of creating the DataReader you can create a DataWriter for that Topic, or you could create both. This will get you a DynamicDataWriter which takes a DynamicData as the data-type. You can use the create_data (or create_data_exp) operation on your DynamicDataWriter to create DynamicData objects with the right type, fill them, and write them using the DynamicDataWriter.

2) If I understood correctly your question is how to externalize or serialize the TypeCode so that you can use some out-of-band (i.e. non-DDS) mechanism to send it to some other process where it can be used to publish or subscribe to Topics of this type. 

The approach to do this differs depending on the programming language. I assume you are using Java as you mentioned that in other postings. If so the following code illustrates both the serialization and deserialization process:

import java.util.ArrayList;
import java.util.Arrays;

import com.rti.dds.cdr.CdrInputStream;
import com.rti.dds.cdr.CdrOutputStream;
import com.rti.dds.typecode.TypeCode;
import com.rti.dds.typecode.TypeCodeFactory;
import com.rti.dds.util.NativeInterface;

/**
 * This class contains utility functions that illustrate how to serialize a TypeCode into
 * a byte representation and how to reconstruct a TypeCode by deserializing its byte representation
 *
 */
public class TypeCodeSerialization {
    
    /** This function illustrates how to serialize a TypeCode in Java such that it can be saved in 
     * a file or sent out-of band.
     *
     * @param typeCode a TypeCode such as the one received via discovery in 
     * the BuitinPublicationTopicData or BuitinSubscriptionTopicData
     * 
     * @return A byte array containing a serialized raw-byte representation of the TypeCode
     */
    public static byte[] serializeTypeCode(TypeCode typeCode) {
        int bufferSize = typeCode.get_serialized_size(0);
        CdrOutputStream stream = new CdrOutputStream(bufferSize);
        typeCode.serialize(stream);
        return stream.getBuffer().getBuffer();
    }
    
    /**
     *  This function illustrates how to create a TypeCode in Java from the serialized representation
     *  obtained by the call to serializeTypeCode(). 
     *  
     * @param serializedTypeCode an array of bytes containing the serialized representation of
     *        a TypeCode such as the one returned by serializeTypeCode()
     *        
     * @return A TypeCode that corresponds to that serialized representation
     */
    public static TypeCode deserializeTypeCode(byte[] serializedTypeCode) {
        boolean needByteSwap =
                NativeInterface.getInstance().isNativeByteOrderLittleEndian();

        CdrInputStream stream = new CdrInputStream(serializedTypeCode, needByteSwap);           
        TypeCode typeCode = 
                TypeCodeFactory.get_instance().create_tc_from_stream(stream);
        
        return typeCode;
    }
} 

The above code can be found in the File Exchange in the file TypeCodeSerialization.java it is runnable code but it depends on an updated version of MonitorDiscoveredTypes.java 

Gerardo

Great example. How would this be done in C++? I've looked through the various API for the other languages and have noticed differences in the available functions. I've tried the following code, but it does not seem to work:

code 

      DDS_ExceptionCode_t ec;
      int buffSz = dataSeq[i].get_type()->get_serialized_size(ec);
      std::vector< char > tempBuff( buffSz );
      std::fill( tempBuff.begin(), tempBuff.end(), 0 );
     
      RTICdrStream stream;
      RTICdrStream_init( &stream );
      RTICdrStream_set( &stream, &tempBuff[0], tempBuff.size() ) ;
      int result = RTICdrTypeCode_serialize( NULL, &(dataSeq[i].get_type()->_data), &stream, true, RTI_CDR_ENCAPSULATION_ID_CDR_NATIVE, true, NULL );

    
      if( result )
      {
        std::cout << "Serialize Passed." << std::endl;
      }
      else
      {
        std::cout << "Serialize Failed." << std::endl;
      }

      myfile.write( &tempBuff[0], tempBuff.size() );
      myfile.close();

code 

Hello Michael,

Apologies for the delay. Yes this can also be done in C++. Your code is almost correct. The reason why it fails is because you cannot pass NULL as the last parameter to the RTICdrTypeCode_serialize(). The last parameter must be a pointer to an unsigned long containing a value that will be used as the limit for the maximum allowed serialization length.  I realize this is not intuitive. RTICdrTypeCode_serialize() was not meant to be a public API...

In yoru situation you do not need any limits (beyond the ones implicit on the configured size for the RTICdrStream you are passing in) so the following code should work:

    // We do not want any limits. Assigning (-1) to an unsigned int
    //   results in the maximum insigned int
    unsigned int serializedTypecodeMaxSize = -1;

    int result = RTICdrTypeCode_serialize(
            NULL, &typeCode->_data, &stream,
            false, /* No need to serialize an encapsulation */
            RTI_CDR_ENCAPSULATION_ID_CDR_NATIVE, /* ignored because no encapsulation */
            true,  &serializedTypecodeMaxSize /* limits the typecode serialized size */ );

I have uploaded a working C++ example to the File Exchange titled serialize_typecode.zip.

Gerardo

Hello Arthur,

Sorry for the lack of documentation. Your use appears correct.

Basically to construct DynamicData objects in Java there are three possibilities:

(1) Use the class constructor:

        DynamicDataProperty_t ddProperty = new DynamicDataProperty_t();       
        DynamicData sample = new DynamicData(typeCode, ddProperty);

This approach requires you to have the TypeCode handy. The use of the DynamicDataProperty_t is documented in the on-line docs for the DynamicData constructor.

(2) Use the DynamicDataTypeSupport as you are using:

      DynamicData sample = (DynamicData) dynamicTypeSupport.create_data();

This is equivalent to using the constructor, except that the DynamicDataProperty_t is automatically created from the DynamicDataTypeProperty_t that was used to create the TypeSupport. If you notice,  the DynamicDataTypeProperty_t contains a DynamicDataProperty_t within, so this is the one used to create the DynamicData.

(3) Use the DynamicDataWriter.create_data_exp() operation. 

        DynamicDataProperty_t ddProperty = new DynamicDataProperty_t();       
        DynamicData sample = new DynamicData(typeCode, ddProperty);

This is a convenience function that accesses the TypeCode that is associated with the DataWriter so you are sure to get a DynamicData bound to the correct data-type. This operation just takes a DynamicDataProperty_t and the interpretation is the same as described for the DynamicData constructor. 

The documentation is missing because it is still considered "experimental", hence the "_exp" suffix. We intend to make it availabe in one of the next releases at which point we will drop the "_exp" suffix.  Our plan is to make this operation is intended to be available in all DataWriter and DataReader objects.

 Once you have the DynamicData object you can set the member values as you show in the code you wrote and use it to write to the DataWriter.

Gerardo

 Thanks Gerardo for the reply and the example code. This is exactly what I needed for the type code serialization. Looks like I was close with my initial code :).

I understand that these functions aren't documented because they weren't meant to be public, so thanks for still providing an example on how to use them.

Since we were're still on the topic (no pun intended) of serialization, how do the autogenerated serialize/deserialize functions of user-defined types differ from the "to_stream" and "from_stream" functions for DynamicData types The reason I ask is because I'm encountering issues with writing out a deserialized dynamic sample.

I took a look at the  "compare_data" example that you previously uploaded to the File Exchange and attempted to create a projec that did something similar, except using file i/o. The example below attempts to serialized a known type, which in this case is ShapeType. I then serialize it using the autogenerated functions, and then write it out to a file. I then read the file back and attempt to construct a DynamicData instance and write it back out, but the write always fails. I'm not sure where i'm messing up.The following errors are outputted upon the call to the write function.

Errors:

DDS_DynamicData_to_key_stream:serialization error: reserialize member
DDS_DynamicData_to_key_stream:serialization error: reserialize member
PRESWriterHistoryDriver_initializeInstance:!serializeKey
WriterHistoryMemoryPlugin_addInstance:!initialize instance
WriterHistoryMemoryPlugin_addSample:!add instance
PRESWriterHistoryDriver_addWrite:!add_sample
PRESPsWriter_writeInternal:!collator addWrite

Test Code:

////////////////////////////////////////////////////////////////////////////////
///
////////////////////////////////////////////////////////////////////////////////
void TestSampleSerializeDeserialize( DDSDomainParticipant* participant )
{
  int rc = 0;

  // Create a shapetype and give it some values.
  ShapeType shape;
  ShapeType_initialize( &shape );
  std::string myColor( "BLUE" );
  strcpy( shape.color, myColor.c_str() );
  shape.shapesize = 30;
  shape.x         = 0x79;
  shape.y         = 0x93;

  // Create binary file so that we can serilize the sample and save it.
  std::ofstream fout( "shapetypesample.bin", std::ios::binary );
  const unsigned int shapeDataSize = 
    ShapeTypePlugin_get_serialized_sample_max_size( NULL, RTI_FALSE, 0, 0 ) + RTI_CDR_STREAM_ALIGNMENT;

  // Create a stream so that we can serialize the sample using the generated 
  std::vector< char > shapeDataBuff( shapeDataSize, 0x00 );
  RTICdrStream shapeDataStream;
  RTICdrStream_init( &shapeDataStream );
  RTICdrStream_set( &shapeDataStream, &shapeDataBuff[0], shapeDataSize );

  // Serialize the sample and write it out to the file
  ShapeTypePlugin_serialize( NULL, &shape, &shapeDataStream, RTI_FALSE, 0, RTI_TRUE, NULL );
  fout.write( &shapeDataBuff[0], shapeDataSize );
  fout.close();

  // Now open the file and read back the sample
  std::ifstream fin( "shapetypesample.bin", std::ios::binary );
  std::vector< char > serializedSample( shapeDataSize, 0 );
  fin.read( (char *)&serializedSample[0], shapeDataSize );
  fin.close();

  // Create a stream and set it to the input buffer containing serialized sample
  RTICdrStream shapeInStream;
  RTICdrStream_init( &shapeInStream );
  RTICdrStream_set( &shapeInStream, &serializedSample[0], shapeDataSize ) ;

  // Create a dynamic data type support instance and register the type
  DDSDynamicDataTypeSupport* ddts = new DDSDynamicDataTypeSupport( 
    ShapeType_get_typecode(), DDS_DYNAMIC_DATA_TYPE_PROPERTY_DEFAULT ) ;
  rc = ddts->register_type( participant, ShapeTypeTYPENAME );

  // Now create a topic and a data writer so we can publish it back out
  DDSTopic* topic = 
    participant->create_topic( "Square", 
                               ShapeTypeTYPENAME, 
                               DDS_TOPIC_QOS_DEFAULT,
                               NULL,
                               DDS_STATUS_MASK_NONE );

  DDSDynamicDataWriter* dynamicWriter = 
  DDSDynamicDataWriter::narrow( 
    participant->create_datawriter( topic,
                                    DDS_DATAWRITER_QOS_DEFAULT,
                                    NULL,
                                    DDS_STATUS_MASK_NONE ) );

  // Create a dynamic data sample and attempt to read it from the stream
  DDS_DynamicData* dd = ddts->create_data();
  rc = dd->from_stream( shapeInStream );

  // Now pubish it (this doesn't work)
  rc = dynamicWriter->write( *dd, DDS_HANDLE_NIL );

  rc = ddts->delete_data( dd );

  delete ddts;
}

////////////////////////////////////////////////////////////////////////////////
///
////////////////////////////////////////////////////////////////////////////////
int main(void)
{
  DDSDomainParticipantFactory* factory = DDSDomainParticipantFactory::get_instance();

  DDS_DomainParticipantQos pQos;
  factory->get_default_participant_qos( pQos );

  DDSDomainParticipant* participant = factory->create_participant( 
    0, pQos, NULL, DDS_STATUS_MASK_NONE );

  NDDSConfigLogger::get_instance()->set_verbosity_by_category(
                      NDDS_CONFIG_LOG_CATEGORY_API,
                      NDDS_CONFIG_LOG_VERBOSITY_STATUS_ALL);

  TestSampleSerializeDeserialize( participant );

  participant->delete_contained_entities();

  factory->delete_participant( participant );
  
  return( 0 );
}

Hi Arthur,

Have you found a way to serialize /deserialize dynamic data? I've encountered the same problem and haven't found a solution yet

Hi Arthur, Michael,

We don't have anything to natively translate a DynamicData object to some other encoding scheme (e.g. XML).  That being said, I'm attaching a simple example project that I've used with others that have asked a similar question.

The logic translates from a DynamicData sample to an XML string, in Java.  Technically, we went from a Java DynamicData sample to a Java XML object (javax.xml.parsers.DocumentBuilder and org.w3c.dom.Document).  In theory, it should work with any arbitrary type where typeCode is available (typeCode is needed to create the DynamicData subscription).

The logic to translate from DDS sample to XML is found in the file (DdsToXmlProcessor.java).  We did not investigate going from XML->DDS, though I imagine it would be relatively similar.  Our DDS->XML logic parses a DynamicData sample to create an XML document; the reverse would parse an XML dom and create a DynamicData sample.

There are four directories inside this bundle:

1. app: a simple DDS app that publishes a data type defined in IDL.  The app is a slightly modified version of what gets generated by rtiddsgen when run with the -example option.
2. src: the source for the Dds to Xml logic. 
3. bin: (compiled) java classes of the above
4. cMulticastApp: a simple C-language app that listens on a multicast address.  Note this app does not use any DDS API whatsoever.

The Java source is sort of like rtiddsspy in that it creates subscriptions for discovered topics, except we print the data as XML.  Also, the app can send the XML string as a raw multicast packet rather than to stdout.  To run the Java app:

java -classpath $NDDSHOME/class/nddsjava.jar:. DdsToXmlLauncher

The arguments (which can be seen if you add the -usage argument):

 java -classpath $CLASSPATH DdsToXmlLauncher [args]:
     -topics <topic list> : comma separated list of the initial dds topics to listen for
     -addUnknown : if true, subscribe to discovered topics. Default is false
     -ddsDomain <id> : the dds domain id. Default is 0
     -period <value> : the period at which to read dds data. Default is 1000ms
     -address : the multicast address to use for publishing. Default is 239.255.0.11
     -port : the multicast port for publishing. Default is 4446
     -stdout : send xml output to stdout rather than to multicast.
     -usage : print this message

As I mentioned, this was some example code that was meant to motivate more complex use cases.  So caveat utilitor.  Hopefully this is in some way useful.

Feel free to ask any further questions. 

Hi Sumeet,

Thanks for the example, it's very helpful. Am I right in assuming that the sample in the dynamic data object is stored in some sort of serialized state? So basically, if I were to use DdsToXml and then serialize the xml object (if I, for example,  want to store it on disk and maybe deserialize later), the whole process would include several deserializations and serializations.

Is there a way to retreive the sample in its initial serialized state, as found in dynamic data object, to avoid unneccesary deserializations and serializations?

Thanks for the help,

Michael

I've also enountered a problem in DynamicDataUtilities.java

In isPrimitiveKind, TCKind.TK_ENUM is considered primitive, but there is no "get_enum" function in initialize(), which results in errors when trying to convert samples containing enums.

What can be done about it?

Thanks,

Michael

Hi Michael,

Let me answer your latter question about enums first.  You can add the following two lines to the DynamicDataUtilities initialize() function:

_kindToClass.put(TCKind.TK_ENUM, Integer.class);

_kindToMethod.put(TCKind.TK_ENUM, dynClass.getMethod("get_int", args));

In Java, enums are accessed using the get_int() function.  The DdsToXml processor will encode the ordinal value of the enumerator, not the literal name.  It's possible to get the name by looking up the ordinal value in the TypeCode object, but it was beyond the scope of this prototype code. 

As for your question about accessing the internal format.... it's possible to get to the raw serialized data in C/C++, but not in Java.  If you look at the DynamicData API, you'll see a function called set_buffer() that allows you to provide the memory for the underlying buffer of a single DynamicData object.  So you'd allocate your own buffer, configure the DynamicData sample to use this buffer, and then call take() on the DataReader.  The serialized data will be written to your buffer. With some additional API calls, you can get how much of the buffer is actually being used for serialized data. 

However, I'm not aware of any API that would allow you to later reload and deserialize that raw data.  This is why I provided my example of converting into some other format (XML).  Note that Java doesn't have these functions because the DynamicData buffer is not kept in Java memory, but rather in the native memory.

-sumeet

Sumeet

Your last comment here that the DynamicData buffer is not kept in Java memory, but in the native memory, suggests that it may be possible to serialise/deserialize directly through the JNI.  Do you have any thoughts on this?

My requirement is to save DynamicData structure and data content to file so that it can be reloaded, edited and re-saved. I've created a JTableTree editor to display the DynamicData content and used the DDSToXML code to write XML (though I'm currently struggling to persuade nested sequences to format properly).  I need to write the XMLToDDS stuff yet and if I can avoid this it would be a help.  There is no specific requirement to output XML to file so a serialised DDS binary would be fine.

John