HOWTO Use RTI Connext DDS With a Leap Motion Controller
This document is aimed at software engineers who are new to the DDS standard. It assumes no background knowledge on DDS or the Leap SDK, in order to make this document usable by both Leap Developers new to DDS, and DDS Developers new to the Leap Motion.
The data path is to get data generated by a Leap Motion Controller into the cloud. The path starts with the Leap development SDK and its API. With the SDK, the data is presented as "frames" via either a callback to, or via polling by the application that is making use of the Leap's data. This HowTo explains one way to create a bridge application that reads the frames as native Leap data, mediates it (reformats it) as DDS instances, and publishes it as HandType data on Leap::Hand. The patterns described below would also be suitable for PointableType data on Leap::Pointable, for GestureType data on Leap::Gesture, etc. There will be less obvious Types and Topics, either general or specific to your application.
In order to keep the tutorial to a manageable size, the Leap::Hand data does not include embedded Pointable information that is attached to the Hand in the frame data. If Pointable or Gesture information is also needed, the model below is suitable as a starting point. [not yet] The source code described below and shown in extract is available in the File Exchange Folder and is linked to below.
What you need/Download links
RTI Connext DDS (Standard 30day Eval, requires registration) (Live bootable Ubuntu CD .iso, suitable for a VM) RTI Shapes Demo (if necessary, included in the 30day eval and the live CD by default) Community Github code repository
Install the development environment or boot the CD as appropriate.
(Pro tip: If installing on Windows, install into C:\RTI to avoid problems with those pesky spaces in Windows directory paths.)
If you are using the Standard 30day Eval for experimentation, you will need to follow the Getting Started guide for your host system. Make sure to set your NDDSHOME environment variable correctly, it should point to <install directory>/ndds.<version> (for example: C:\RTI\ndds.5.1.0)
How DDS works
DDS is the Object Management Group "Data Distribution Standard for Real-Time Systems" suite of standards. There are three primary legs to the DDS standard: The API used by an application to control the middleware; the "Real-Time Publish-Subscribe" (RTPS) protocol for how the data is serialized, formatted and wrapped when it is sent to a recipient; and a set of well-defined Quality of Service profiles that define contracts between readers and writers (governing reliability, durability, fault-tolerance, filtering, etc).
DDS is server-less and uses automatic discovery (well-known multicast addresses and ports) and peer-to-peer distribution of data between interested parties. To exchange data, one or more writers and one or more readers (collectively: endpoints) must be on the same Domain, on the same Topic, and have compatible Quality of Service configurations. One or more writers sending durable, reliable data on the topic “Leap::Hand” on domain id 112 will send data to one or more durable, reliable readers of that topic on that domain.
A DDS-enabled application creates a domain participant for the domain it wants to be on. A domain is identified by an integer, for the below discussion I am using the domain 112, but you are free to substitute any domain 0..120. The participant is then used as a factory to create Topic, Publisher and Subscriber entities; Publishers are factories for DataWriters (linked to a given Topic), and Subscribers are factories for DataReaders (also linked to a given Topic). An application that is limited to only one domain needs only one participant (ie, you don’t need a participant per endpoint).
Topics, DataReaders and DataWriters are strongly typed. This means that they know their Type. Types are defined in IDL (Interface Description Language), and then passed through a code generator (rtiddsgen) which creates the language-specific structure (C), class (C++) or POJO (Java) objects, as well as the strongly typed DataReader and DataWriter entities that the application will use. A publishing application will create a new instance of data, populate its fields and then call a typed writer to write the instance. A subscribing application will have a typed reader that will use a listener (callback), a waitset (analogous to a socket "select()") or simple polling to identify when new data is available to it. What the reader receives from the middleware is another of the typed object instances, with its fields already filled in.
This discussion does not touch on Quality of Service contracts. Examine a USER_QOS_PROFILES.xml (generated by using the -example <arch> flag with rtiddsgen) to see what named profiles are available to start. For simplicity, all Leap Topics in this tutorial will use the default QoS settings (reliable, volatile writer and best-effort, volatile readers). The data reader is best-effort and so neither durable (no historical data available), nor reliable (the reliability protocol is internal to DDS, so when enabled it works even when the underlying transport is best-effort, e.g. UDP).
The Leap to DDS Bridge
The Leap to DDS bridge is a good place to start looking at how the DDS API can be used to publish high-rate data. In the fileset attached to this post, find the file LeapDDS_Hands.cxx. This file is based heavily on the <Type>_publisher.cxx file that is generated using the -example <arch> (in this case, i86Win32VS2010) flag used with rtiddsgen.
Starting from the top of that file, there is a LeapListener::onFrame implementation (see LeapFrameHandler.h). The handler forwards the controller argument to the “handleHands(...)” method in LeapFrameHandler.h.
Moving to the end of the LeapDDS_Hands file, find the main(argc, argv) function. Main processes the command line arguments, sets up the entities and initializes them. From the DDS standpoint, the interesting ones are participant_main, and writer_main.
Participant_main creates a domain participant for either domain 112 or as by -domainId flag. In this instance I’m also using it to register the type (“HandType”) and the topic (“Leap::Hand”) for use by my readers and writers, as Types and Topics should only be registered with a given Participant once.
Writer_main creates the writer entity and the instance of HandType we will be populating with data from the frames. For this application I’m using the “default” Publisher entity rather than creating one for use.
When a new Frame arrives via onFrame, handleHands calls the LeapListener’s handFrameToInstance method for each Hand in the frame. HandFrameToInstance populates our single HandType instance with the metadata about the hand (isValid, etc), the palm velocity and position and so on, and also for each Pointable it populates a PointableType in the HandType.pointables sequence.
In DDS, a “sequence” is a bounded array of like elements, that may be 0..<max> long. An “array” is also possible to define, but that is a fixed length. The <max> is defined in the IDL, in HandType.pointables’ case, it is defined as “sequence<PointableType, 12> pointables;”, so a sequence of Pointables up to 12 instances long. This is important for network bandwidth utilization; when serialized for sending, an empty sequence only includes a field to give its length (0), while each value in an "empty" array is fully serialized, all to zeros.
On return from handFrameToInstance, handleHands calls the HandTypeDataWriter’s write method. HandType data is now available on the wire, if there are valid subscribers. If there are no subscribers, the HandType data does not leave the application.
Writing your first DDS Application
The demo applications we'll be writing target the Shapes Demo as the output medium. DDS is strongly typed; we'll be using the ShapeTypeExtended Type. The Shapes installation includes an IDL file with the complete type descriptions, this is an extract with additional commentary:
// Note: The comments (other than //@key, which is an annotation) are
// related to how the Shapes Demo understands the data. They are not limits
// on the type, only on what the Shapes Demo is prepared to correctly
// display.
// struct ShapeType {
string<128> color; //@key
// RED, ORANGE, YELLOW, GREEN, BLUE, CYAN, MAGENTA, PURPLE, BLACK (any other string will show as Black)
long x; // 0 <= x <= 243
long y; // 0 <= y <= 254
long shapesize; // shapesize >= 0
};
// ShapeTypeExtended InheritsFrom ShapeType
struct ShapeTypeExtended : ShapeType {
ShapeFillKind fillKind; // see the supplied IDL
float angle;
};
rtiddsgen
In order to make use of the IDL-defined type, you have to create the instance type object (a C struct, a C++ class, a Java POJO, etc), which is done by passing the idl through the idl compiler, rtiddsgen. Create a working directory for your project, and copy the ShapeType.idl file there.
rtiddsgen -inputIdl ShapeType.idl -language <C|C++|Java|C++/CLI|C#> -ppDisable -replace -example <arch (see below)>
To determine what <arch> is/are available to you, list the directory $NDDSHOME/lib (%NDDSHOME%\lib on Windows). The listing will show items like "x64Win64VS2010" or "x64Darwin12clang4.1". If you are writing your code in Java, use one of the architectures which ends in "jdk". If using Visual Studio Express, use one of the i86Win32VS* architectures (Visual Studio Express won't do 64bit compiles, ordinarily). If you are confused as to which architecture you should be using, ask in the comments and we'll help you out.
For Java, I will be using
shallweplayagame:communityPost rip$ rtiddsgen -inputIdl ShapeType.idl -language Java -ppDisable -replace -example x64Darwin12clang4.1jdk Running rtiddsgen version 5.1.0, please wait ... Done shallweplayagame:communityPost rip$ ls ShapeFillKind.java ShapeTypeExtendedPublisher.java ShapeFillKindSeq.java ShapeTypeExtendedSeq.java ShapeFillKindTypeCode.java ShapeTypeExtendedSubscriber.java ShapeFillKindTypeSupport.java ShapeTypeExtendedTypeCode.java ShapeType.idl ShapeTypeExtendedTypeSupport.java ShapeType.java ShapeTypeSeq.java ShapeTypeDataReader.java ShapeTypeTypeCode.java ShapeTypeDataWriter.java ShapeTypeTypeSupport.java ShapeTypeExtended.java USER_QOS_PROFILES.xml ShapeTypeExtendedDataReader.java makefile_ShapeType_x64Darwin12clang4.1jdk ShapeTypeExtendedDataWriter.java shallweplayagame:communityPost rip$
(Pro tip: Quickly rename the ShapeTypeExtendedPublisher.java and ShapeTypeExtendedSubscriber.java files to something like STE_ExamplePub.java/STE_ExampleSub.java. Because reasons, related to the use of the "-replace" flag to rtiddsgen. I will use the STE_ forms in the discussion, below).
There are fewer files in the C++ or C builds. With the Java build, this directory is suitable for import into Eclipse, into a project. Create a new Java project, context-click the project name and select "import...". Use the General > File System... entry and simply point it at your working directory. Add $NDDSHOME/class/nddsjava.jar to the build path, and you can edit/debug in the GUI. If the language is C or C++, we supply a .sln solution file for Visual Studio, and a makefile for Linux.
Edit the STE_ExampleSub.java file, to change the Topic name (from "Example ShapeTypeExtended") to "Triangle". Now, run the class file either from Eclipse, or from the command line:
java -cp ".;${NDDSHOME}/class/nddsjava.jar" STE_ExampleSub
Launch the Shapes Demo on a machine in the subnet, which should start in ShapeTypeExtended mode and enabled. Create a Triangle publisher (from the GUI or from the appropriate menu). Don't worry about the QoS settings for now, simply accept the default. Do however add a rotation rate (lower left corner of the dialog). You should see a moving Triangle appear in the display, bouncing off the walls. More importantly, you should see a bunch of text (each sample's information) dumping to the console (eclipse or command line). If you launched both applications on the same machine, they are communicating over shared memory. If on different machines, they are communicating over UDP, using anonymous (simple) discovery. If you don't see data appearing in the STE Subscriber, read the comments below, or post a new comment if the previous comments didn't resolve the problem.
Now, edit the STE_ExamplePub.java file. Again, start by changing the Topic name to "Triangle". Further down, find where the instance is created, and add some lines:
ShapeTypeExtended instance = new ShapeTypeExtender(); // pre-existing
instance.color = new String("Yellow");
instance.x = 150;
instance.y = 150;
instance.shapesize = 25;
instance.fillKind = ShapeFillKind.SOLID_FILL;
instance.angle = 0.0f;For a more interesting demo, move the instance.x, .y and .angle down into the run loop and modify them on each pass. I like to have my triangles loop in a circle.
In the Shapes Demo, use the "Delete All" label to delete the existing Publisher, if it is still running. STE_ExampleSub should stop showing new data. Create a Triangle Subscriber, again accepting the defaults.
Now, run the STE_ExamplePub application, using the same style of command line for the ExampleSub. You should see a Yellow Triangle appear in the GUI (as well as seeing the publication information in the STE_ExampleSub window). Because Discovery is automatic, the Shapes GUI and the two command line applications will all find each other and react correctly.
Congratulations, you've executed a DDS-based publisher, and a DDS-based subscriber, application! And published to, and subscribed from, a 3d party application, simply by knowing what Type (ShapeTypeExtended) and Topic (Triangle) to use.
The LeapDDS Bridge
In the github repo for this post, grab the LeapTypes.idl file and open it. You can perform the same rtiddsgen step to it, to generate the types for HandType, PointableType and GestureType which you will need for your DDS-based Leap-enabled application. Also in the git repo is a rudimentary bridge (no more than PoC at this time). If you want to edit the IDL, feel free, and then regenerate for C++, and recompile the bridge to get access to your specific Types. There are applicable readmes. If you find issues with the Bridge code, readme.txt files or anything else, please leave a comment below.
The LeapDDSBridge PoC can be started using
./obs/<arch>/LeapDDSBridge <domainId:0> <bridgeId:0>
The <arch> string is identical as used in rtiddsgen. The domainId and bridgeId default to 0 (bridgeId can be used by a subscriber application, to filter input to just a single system, which in a multiuser game would be necessary for the local player's own bridge. The "game" application itself would not filter, and so could see all the data from everyone's systems).
With the bridge running, the databus now includes the potential for HandType, PointableType and GestureType, although since there are no subscribers, the actual data isn't being publishes. If you want to see that the data is there, use the command-line tool 'rtiddsspy -domainId 0 -topicRegex "*::Hand" -printSample" and you'll see the data as you wave your hand over your connected Leap device.
At this point you have all the necessary tools: The Leap Motion, the LeapDDS Bridge which reads the Leap data and writes the DDS data, and the Types that are being written (The topic Leap::Hand is HandType, Leap::Pointable is PointableType, etc), and you have a display medium in the Shapes GUI.
The only thing you need is to mediate between the Leap data, and the Shapes Types. Things you can target:
Leap::Hand palm orientation to Triangle. Rotate your palm and the Triangle rotates accordingly, move your hand in the Leap's field and the Triangle moves accordingly. You're using the Shape to reflect the current state of your hand. Or, palm orientation to Square angle, and the hand's pointables as Triangles (remember to subscribe to both Squares and Triangles in the GUI). For this, you may want to set the QoS "History" to Keep_Last with a depth of 1. The color is a key value (like RDBMS key columns), so the "RED" square is a different instance from the "BLUE" square, so each will have a History depth of 1. If they were unkeyed, then "Square" would only keep the last instance it received, a later "BLUE" square would overwrite the preceding "RED" square).
Leap::Pointable tip data to Circle (and also Triangle). You can use Circle to indicate location, and Triangle to indicate direction of movement and angle. You're using the Shapes to reflect the current state of your finger tips.
Leap::Gesture to change the direction of a moving/moveable Shape (ie, say you have a Circle at 150, 150, and you swipe left -- move the Circle -20, 0 from its current location). In this case, you are using Gestures to affect the state of a known object.
What else?