Output buffer problem using the RTI Connector for Python

8 posts / 0 new
Log in or register to post comments
Last post
Mon, 10/19/2020 - 07:34
#1
garzob
Offline
Last seen: 4 years 1 month ago
Joined: 10/17/2020
Posts: 10
Output buffer problem using the RTI Connector for Python

Hey!

Im writing a DDS application, that sends long strings that are numpy arrays turned into strings. Im using Python for this app, and using the python connector (RTI Connector for Python). When I'm trying to read the sent string back I get the following error.

DDS_DynamicData2_get_string: Output buffer too small for member (name = "frame", id = 1). Provided size (128), requires size (630).

My XML file:

<?xml version="1.0" ?>
<dds version="6.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://community.rti.com/schema/6.0.0/rti_dds_profiles.xsd">
  <qos_library name="QosLibrary">
    <qos_profile base_name="BuiltinQosLib::Generic.StrictReliable.LargeData" is_default_qos="true" name="DefaultProfile">
      <datareader_qos>
      </datareader_qos>
      <participant_qos>
        <participant_name>
          <name>Connector Example</name>
        </participant_name>
      </participant_qos>
    </qos_profile>
  </qos_library>

  <types>
    <struct name="ImageType">
      <member name="frame" stringMaxLength="1000000" type="string"/>
    </struct>
  </types>
  <domain_library name="MyDomainLibrary">
    <domain domain_id="0" name="MyDomain">
      <register_type name="ImageType" type_ref="ImageType"/>
      <topic name="Image" register_type_ref="ImageType"/>
    </domain>
  </domain_library>
  <domain_participant_library name="MyParticipantLibrary">
    <domain_participant domain_ref="MyDomainLibrary::MyDomain" name="MyPubParticipant">
      <publisher name="MyPublisher">
        <data_writer name="MyImageWriter" topic_ref="Image"/>
      </publisher>
    </domain_participant>
    <domain_participant domain_ref="MyDomainLibrary::MyDomain" name="MySubParticipant">
      <subscriber name="MySubscriber">
        <data_reader name="MyImageReader" topic_ref="Image"/>
      </subscriber>
    </domain_participant>
  </domain_participant_library>
</dds>


 
If anyone more involved in the python connector could please help, that would be great!

 

 

Top
Mon, 10/19/2020 - 07:45
samr
Offline
Last seen: 2 months 1 week ago
Joined: 10/22/2018
Posts: 91

Hi,

Please can you provide a code snippets of:
- How you are setting + sending the sample (on the output)

- How you are accessing the data within the sample (on the input)

Although the content of this article isn't related to your use-case, there are some code snippets there that show the use of numpy with Connector: https://community.rti.com/kb/improve-performance-accessing-types-containing-sequences-and-arrays-rti-connector

Thanks,
Sam

Top
Mon, 10/19/2020 - 07:56
garzob
Offline
Last seen: 4 years 1 month ago
Joined: 10/17/2020
Posts: 10

Hy, I'm going to include my project zipped together, but also I'm going to paste it here. My Project only consists of a ImageReader.py and a ImageWriter.py file also an image and an xml.

ImageReader:

with rti.open_connector(
        config_name="MyParticipantLibrary::MySubParticipant",
        url="./Image.xml") as connector:

    input = connector.get_input("MySubscriber::MyImageReader")

    print("Waiting for publications...")
    input.wait_for_publications() # wait for at least one matching publication

    print("Waiting for data...")
    for i in range(1, 500):
        input.wait() # wait for data on this input
        input.take()
        for sample in input.samples.valid_data_iter:
            # You can get all the fields in a get_dictionary()
           imageString = sample.get_string("frame")
           image = np.fromstring(imageString,dtype=int)
           cv2.imshow("image",image)
 
This is without the imports.
 
ImageWriter:
 
file_path = os_path.dirname(os_path.realpath(__file__))
sys_path.append(file_path + "/../../../")
import rticonnextdds_connector as rti
with rti.open_connector(
        config_name="MyParticipantLibrary::MyPubParticipant",
        url="./Image.xml" ) as connector:
  frameOg = cv2.imread('asd.png')
  frame = cv2.cvtColor(frameOg, cv2.COLOR_BGR2RGB)
  frameString = np.array_str(frame)
  print(type(frameString))
  print(len(frameString))
  output = connector.get_output("MyPublisher::MyImageWriter")

  print("Waiting for subscriptions...")
  output.wait_for_subscriptions()
  output.instance.set_string("frame",frameString)
  output.write()
  output.wait()
 
Also without the imports.
 
File Attachments: 
Package icon imageproject.zip
Top
Wed, 10/21/2020 - 04:00
samr
Offline
Last seen: 2 months 1 week ago
Joined: 10/22/2018
Posts: 91

Hey,

I took a quick look at your application and managed to reproduce your issue.

The error you are seeing "DDS_DynamicData2_get_string: Output buffer too small for member (name = "frame", id = 1). Provided size (128), requires size (630)." appears to be unrelated to your issue. I am investigating internally what is happening there, but printing out the data on the reader I confirmed that in fact the entire string is recieved.

It looks to me like your issue is with the use of the numpy / cv2 APIs.

On the input side, you should probably do something like this:


           imageString = sample.get_string("frame")
           jpg_original = base64.b64decode(imageString)
           jpg_as_np = np.frombuffer(jpg_original, dtype=np.uint8)
           img = cv2.imdecode(jpg_as_np, flags=1)
           cv2.imshow("image", img)

When I do that, the imshow API fails due to the fact that there is something wrong with the received string, so probably something going on with the output side of things and how you are setting the string. I'm not super familiar with these libraries but 630 seems like a very small string size to represent an entire image.

 

Sam

Top
Wed, 10/21/2020 - 07:07
samr
Offline
Last seen: 2 months 1 week ago
Joined: 10/22/2018
Posts: 91

Hi again,

I have confirmed that you are running into a known bug, with internal issue # CON-157.

The error message being printed out can safely be ignored, it doesn't have any impact on the application (other than the error message itself being printed).

Sam

Top
Thu, 10/22/2020 - 11:30
garzob
Offline
Last seen: 4 years 1 month ago
Joined: 10/17/2020
Posts: 10

Hi Sam!

I managed to get your "Improve performance accessing types containing sequences and arrays in RTI Connector" example code working. As you have mentioned the devil was in the image encoding and decoding and also me switching to using WSL 2.

However now I'm trying to make a working videostream with DDS, but I can't seem to get all my frames the same byte size, so I get the following message:

DDS_DynamicData2_checkBounds: Invalid length for member (name = "data", id = 1). Provided length (291072), max length (291308).
DDS_DynamicData2_set_octet_array:invalid length
Writing...
DDS_DynamicData2_checkBounds: Invalid length for member (name = "data", id = 1). Provided length (291680), max length (291308).
DDS_DynamicData2_set_octet_array:invalid length
Writing...

With this error no data is being sent out sadly. I don't know why it's a must for the max length and provided length to be the exact same, but my frame sending only happens when they are the same.

Do you maybe know a way to get past this error? Maybe have a unbounded byte array somehow or just set a big length and get past that issue that I mentioned.

Thank you for your help so far! I hope you can also help me on this one! I'll definetly quote you on my project! =)

 

Ben

Top
Mon, 10/26/2020 - 05:13
samr
Offline
Last seen: 2 months 1 week ago
Joined: 10/22/2018
Posts: 91

In this code (taken from the article I linked above)

def setSequenceNatively(output, field_name, python_arr):
    # We need access to the native DynamicData sample which we can supply
    # to the DDS C APIs
    native_dynamic_data = output.instance.native
 
    # We then need to build the sequence such that it is compatible with C.
    # This is accomplished using ctypes
    # Unsigned long for the length of the array
    array_length = ctypes.c_ulong(len(python_arr))
    # Create a new type
    in_array_type = ctypes.c_char * len(python_arr)
    # Set the member ID to 0 (translates to unknown in DDS C DynamicData API)
    member_id = 0
    # Create an instance of our new type using the python array
    in_array = in_array_type(*python_arr)
    # Here we set the values into the member
    DDS_DynamicData_set_octet_array(
            ctypes.cast(native_dynamic_data, ctypes.c_void_p),
            field_name.encode("utf8"),
            member_id,
            array_length,
            ctypes.byref(in_array))
 

Can you check that in the equivalent in your application you are updating the array_length your pare passing to the native function?

Top
Mon, 10/26/2020 - 08:00
garzob
Offline
Last seen: 4 years 1 month ago
Joined: 10/17/2020
Posts: 10

Hi!

Yes I'm using the code you wrote in that article. My problem is that in the XML file you have to specifie an arrayDimensions variable and that can not be updated while the program is running. While the setSequenceNatively can write the exact length of the byte array I wanna pass through, my XML file can't be changed, and if I set a upwards estimate for my arrayDimensions variable, the program wont transfer the data. Do you know how to bypass that? With set arrayDimensions variable I simply cannot do it. With sequenceMaxLength variable set to -1, which indicates unbound sequence, I get an intresting crash, probably because your code is desinged for arrays not sequences.

Top