I am trying to connect a basic DDS publisher/writer to a Kafka topic using the Routing Service application. I have been trying for two days to get this to work, and am starting to lose hope.
Versions:
OS: Rocky 9 Linux
OpenSSL: 3.2.2
RTI Connext :7.5.0
Kafka: 2.13-4.0.0
Environment: Everything (publisher app, kafka broker) is running on a single VM running Rocky 9 linux.
Steps:
1. I have configured a Routing Service between a basic publisher and Kafka according to the documentation found here (configuration file attached):
https://community.rti.com/static/documentation/connext-dds/current/doc/manuals/connext_dds_professional/services/routing_service/configuration.html
https://community.rti.com/static/documentation/gateway/current/adapters/kafka/configuration.html#kafka-configuration-properties
2. I have built the rticonnextdds-gateway library and pointed `LD_LIBRARY_PATH` to /rticonnextdds-gateway/install/lib, where librtikafkaadapter.so lives.
3. I have also installed the required library librdkafka from confluent, according to their instructions here: https://github.com/confluentinc/librdkafka?tab=readme-ov-file#librdkafka---the-apache-kafka-cc-client-library
4. I successfully build and run my C++ publisher app
5. Configure and start the Kafka server according to the official quick-start instructions: https://kafka.apache.org/quickstart
6. Create a Kafka topic with
7. Start the routing service with the attached configuration, which results in:
RROR [/routing_services/DDS2KafkaService|START|/domain_routes/DDS2KafkaRoute|CREATE] RTIOsapiLibrary_openEx:FAILED TO OPEN FILE | library=librtikafkaadapter.so: /lib64/libldap.so.2: undefined symbol: EVP_md2, version OPENSSL_3.0.0
ERROR [/routing_services/DDS2KafkaService|START|/domain_routes/DDS2KafkaRoute|CREATE] ROUTERPlugin_load:!load plugin shared library:
ERROR [/routing_services/DDS2KafkaService|START|/domain_routes/DDS2KafkaRoute|CREATE] ROUTERPlugin_load:rtikafkaadapter
ERROR [/routing_services/DDS2KafkaService|START|/domain_routes/DDS2KafkaRoute|CREATE] ROUTERPlugin_getInstance:!load shared library
ERROR [/routing_services/DDS2KafkaService|START|/domain_routes/DDS2KafkaRoute|CREATE] ROUTERDomainRoute_initialize:!get adapter plugin instance
ERROR [/routing_services/DDS2KafkaService|START] ROUTERDomainRoute_new:!init ROUTERDomainRoute object
ERROR [/routing_services/DDS2KafkaService|START] ROUTERService_createDomainRoute:!create domain route
ERROR [/routing_services/DDS2KafkaService|START] ROUTERService_start:!create domain route
ERROR RTI_RoutingService_start:!start routing service
ERROR ROUTERServiceApp_main:!start service
Here are the steps that I've used to troubleshoot:
I believe md2 is a legacy algorithm, so I updated openssl.conf to enable legacy algorithms. This was verified to have worked with `openssl list -providers`:
I have verified the dependencies of the kafka adapter:
I have also verified the dependencies of libldap.so:
I am truly at my wits' end here. Any help is appreciated.
| Attachment | Size |
|---|---|
 routing service configuration | 7.46 KB |
Unfortunately, this is not a RTI Routing Service nor gateway Kafka adapter plugin issue...this is an issue with your openssl installation.
If you google for "/lib64/libldap.so.2: undefined symbol: EVP_md2, version OPENSSL_3.0.0"
you'll find that this problem will occur if your openssl libs aren't compiled with "
enable-md2"...Hi Howard, I am on Rocky 9, and as you can see here OpenSSL is compiled with enable-md2: https://git.rockylinux.org/staging/rpms/openssl/-/blob/r9/SPECS/openssl.spec?ref_type=heads#L311
I have not touched the OpenSSL system libraries, and I have also verified that I can run the md2 algorithm from the command line.
I suggest that
1) you check your system openssl libs to make sure that they have the EVP_md2 defined
nm -D /lib64/libcrypto.so.3 (I think that's where EVP_md2 is defined)
2) you create a script in which you set LD_LIBRARY_PATH to *only* the paths from where you want the OS to find all of the required libraries, specifically the Openssl ones used by libldap.so.
You may have multiple installations of Openssl installed. Certainly, if you installed RTI Connext DDS's Security plugins and the openssl distribution that we provide...]
so make sure that /lib64 is first on the path.
3) execute the rtiroutingserviceapp binary directly, and not via the rti_connext_dds-7.5.0/bin/rtiroutingservice script (the script may modify the LD_LIBRARY_PATH). You should be able to find the executable in the resource/app/bin/<arch> directory.
Trust that the error message is accurate. When the OS tried to load the librtikafkaadapter.so plugin, it needed to also load libldap.so, and that need to load an Openssl library (libcryto.so?) that defines the mssing symbol.
In which case, the openssl library that it found and loaded didn't have that symbol defined...
Hi Howard, I have already verified that the symbol is defined, and I will try the rest of your suggestions when I go into work tomorrow. Trusting the error makes sense. I'll let you know what I find.
Hi Howard, I believe I have found the issue.
The script used to run the routing service adds this path to LD_LIBRARY_PATH: rti_connext_dds-7.5.0/resource/app/lib/x64Linux4gcc7.3.0/ which contains libcrypto.so.3
Upon inspecting the dependency chain BEFORE running the script we see that everything is fine. /lib64/libldap.so.2 points to /lib64/libcrypto.so.3 which provides the EVP_md2 symbol:
However, if I add a debug statement to check the output of ldd /lib64/libldap.so.2 | grep libcrypto to the rti provided script, we see that the dependency chain has changed:
So it does indeed appear that the routing service script recommended in the tutorials/examples points LD_LIBRARY_PATH to a version of libcrypto that is incompatable with the rti kafka adapter. You suggested running the executable directly instead of using the provided script, but after looking at the script I see it does a lot of setup that is probably necessary.
I think I will rename/move the libcrypto.so.3 provided by RTI and see if that fixes the issue.
Great, I'm glad you found the issue. Sorry the libcrypto that we've distributed doesn't work with the Kakfa shared library...
With respect to the rtiroutingservice script... I don't think that there's alot of the "setup" in the script is necessary. If you set the LD_LIBRARY_PATH directly as I suggested and then run the executable directly from resource/app/bin/<etc>, it should work cleaning and just fine.
I have done this myself when I need to run a particular version of the executable and not use the one that the script has chosen for my platform...
In any case, I'll file an internal bug about the build of our libcrypto.so.
Thank you Howard. I learned a lot from troubleshooting this issue, and your tips helped me a lot!
So, the support of md2 is disabled by default in Openssl.
It was disabled by default back in this commit (openssl 1.0.0): https://github.com/openssl/openssl/commit/c155d83f5bf3285daddf1a92e836dc238560afca
Likely due to the security vulnerability of md2: https://en.wikipedia.org/wiki/MD2_(hash_function)#Security
So, we're not likely going to enable it in what we build/distribute.
I'm not sure why your libldap.so needs to have EVP_md2.
For example, on an ubuntu 20.04 installation
nm -D /usr/lib/x86_64-linux-gnu/libldap-2.4.so.2 | grep md2
returns nothing..also checked on a ubuntu 18.04 installation.
The question is why does Rocky9 build openssl with enable-md2 (and subsequently libldap.so that uses openssl) when the distribution from openssl doesn't...
Hi Howard, RHEL 9 builds with md2 enabled for the purposes of supporting legacy software, and Rocky 9 aims for direct binary compatability with RHEL 9.
I'm not sure why the LDAP lib uses that deprecated functionality though. I've been unable to identify whether the version I have on my VM is what came stock with Rocky 9 or was added later.