3.2.1.2. OpenSSL 1.1 vs OpenSSL 3.0 vs WolfSSL 5.5

This section compares the performance of the cryptographic libraries that RTI Connext supports. We compare the performance of the OpenSSL 3, OpenSSL 1.1.1, and wolfSSL 5.5 cryptographic libraries. Changes in the patch versions of those libraries (last digit in the OpenSSL 3 or wolfSSL version numbers, letter in the OpenSSL 1 version number) should not have a great effect on the performance results.

Notice that these tests use the cryptographic libraries along to the RTI Connext code, however, in order to prove that these differences are not due to the RTI Connext specific code we also gathered some metrics isolating the Cryptographic operations, see Cryptographic Libraries section.

  • Not using security libraries

  • No protection

  • RTPS Sign

  • RTPS Encrypt

  • RTPS Sign with Original Auth, Data Encrypt

  • RTPS Sign, Submessage Encrypt with Original Auth, Data Encrypt

  • RTPS Sign, Submessage Encrypt

Perftest Scripts

To produce these tests, we executed RTI Perftest for C++98. The scripts used to execute the tests can be found here:

  1#!/bin/bash
  2filename=$0
  3script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)
  4
  5export datasizes="32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000"
  6export datasizes_extended="${datasizes} 100000 500000 1048576 1548576 4194304 10485760"
  7
  8export domain="2"
  9export exec_time=20
 10export num_reps=1
 11export instance_number=100000
 12export core=0
 13
 14# We will use some colors to improve visibility of errors and info messages.
 15RED='\033[0;31m'
 16GREEN='\033[0;32m'
 17YELLOW='\033[0;33m'
 18BLUE='\033[0;34m'
 19LIGHTBLUE='\033[0;36m'
 20NC='\033[0m'
 21INFO_TAG="${GREEN}[INFO]:${NC}"
 22WARNING_TAG="${YELLOW}[WARNING]:${NC}"
 23ERROR_TAG="${RED}[ERROR]:${NC}"
 24
 25export ip_machine_1="10.2.78.20"
 26export ip_machine_2="10.2.78.21"
 27export if10Gbps="enp1s0f0"
 28export if1Gbps="eno1"
 29
 30################################################################################
 31
 32function disable_colors() {
 33    export RED=""
 34    export GREEN=""
 35    export YELLOW=""
 36    export NC=""
 37    export BLUE=""
 38    export LIGHTBLUE=""
 39    export INFO_TAG="${GREEN}[INFO]:${NC}"
 40    export WARNING_TAG="${YELLOW}[WARNING]:${NC}"
 41    export ERROR_TAG="${RED}[ERROR]:${NC}"
 42}
 43
 44function change_domain() {
 45    if [[ "$domain" == "1" ]]; then
 46        export domain="2"
 47    else
 48        export domain="1"
 49    fi
 50}
 51
 52# Usage: execute_test <keyed/unkeyed> <rel/be> <datasizes> <batchSize>
 53function execute_test() {
 54
 55    local keyed_unkeyed=$1
 56    local rel_be=$2
 57    local datasizes_test=$3
 58    local other_args=$4
 59    local name_suffix=$5
 60
 61    local commands_string_test=$commands_string
 62    local tag=""
 63
 64    if [[ "${keyed_unkeyed}" == "keyed" ]]; then
 65        commands_string_test="${commands_string_test} -keyed -instances $instance_number"
 66        tag="[${YELLOW}${transport}${NC}|${BLUE}K${NC}|"
 67    else
 68        tag="[${YELLOW}${transport}${NC}|${LIGHTBLUE}UK${NC}|"
 69    fi
 70
 71    if [[ "${rel_be}" == "be" ]]; then
 72        commands_string_test="${commands_string_test} -bestEffort"
 73        tag="${tag}${YELLOW}BE${NC}]"
 74    else
 75        tag="${tag}${RED}REL${NC}]"
 76    fi
 77
 78    tag="${tag}[${LIGHTBLUE}${lat_thr}${NC}]"
 79
 80    local output_file=$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}.csv
 81
 82    if [[ "$role" == "pub" ]]; then
 83        echo -e "${YELLOW}[TEST]: $keyed_unkeyed, $rel_be. ${NC}"
 84    fi
 85
 86    if [[ "$LANGUAGE" != "java" && "$LANGUAGE" != "cs" ]]; then
 87        export pre_command_string="taskset -c $core"
 88    fi
 89
 90    if [[ "$LANGUAGE" == "python" ]]; then
 91        export pre_command_string="python3 "
 92    fi
 93
 94    # Get the aprox time this will take:
 95    total_tests=$((`wc -w <<< "$datasizes_test"` * num_reps))
 96    total_time=$((total_tests * exec_time))
 97
 98    touch $output_file
 99    local no_headers=""
100    local current_test=0
101    for index in $(seq 1 ${num_reps}); do
102        for DATALEN in ${datasizes_test}; do
103            current_test=$((current_test + 1))
104            export command="$pre_command_string $executable -domain $domain -dataLen $DATALEN $commands_string_test $other_args $no_headers"
105            if [[ "$role" == "pub" ]]; then
106                echo -e "Test ${tag} (${current_test}/${total_tests}) -- Total time = ${total_time}s"
107                echo -e ${BLUE}$command${NC}
108            else
109                echo -e ${LIGHTBLUE}$command${NC}
110            fi
111            if [[ "$LANGUAGE" == "cs" && "$role" == "pub" ]]; then
112                sleep 3
113            fi
114            if [[ "$raw" == "1" && "$role" == "sub" ]]; then
115                sleep 5
116            fi
117            if [[ "${get_netstat_info}" == "1" ]]; then
118                echo -e "${INFO_TAG} Getting netstat info before"
119                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt
120            fi
121            eval $command >> $output_file;
122            if [[ "${get_netstat_info}" == "1" ]]; then
123                echo -e "${INFO_TAG} Getting netstat info after"
124                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt
125                touch "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
126                python3 $script_location/../../../tools/diff_netstat_output.py \
127                    -n $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt \
128                    -o $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt \
129                    -d $DATALEN $no_header_netstat \
130                    -csv >> "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
131                rm -rf $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_*.txt
132                no_header_netstat=" -nh"
133            fi
134            no_headers=" -noOutputHeaders"
135
136            change_domain
137        done
138    done
139}
140
141################################################################################
142# PARSE COMMAND LINE OPTIONS:
143
144while [ "$1" != "" ]; do
145    case $1 in
146        --executable)
147            executable=$2
148            shift
149            ;;
150        --output-folder)
151            output_folder=$2
152            shift
153            ;;
154        --sub-folder)
155            sub_folder=$2
156            shift
157            ;;
158        --role)
159            export role=$2
160            shift
161            ;;
162        --core)
163            export core=$2
164            shift
165            ;;
166        --test-kind)
167            export lat_thr=$2
168            shift
169            ;;
170        --interface1)
171            export interface=$2
172            shift
173            ;;
174        --interface2)
175            export interface2=$2
176            shift
177            ;;
178        --ip1)
179            export ip1=$2
180            shift
181            ;;
182        --ip2)
183            export ip2=$2
184            shift
185            ;;
186        --repetitions)
187            export num_reps=$2
188            shift
189            ;;
190        --domain)
191            export domain=$2
192            shift
193            ;;
194        --execution-time)
195            export exec_time=$2
196            shift
197            ;;
198        --transport)
199            export transport=$2
200            shift
201            ;;
202        --datalen)
203            export datalen_input=$2
204            shift
205            ;;
206        --file-suffix)
207            export file_suffix=$2
208            shift
209            ;;
210        --executable-suffix)
211            export executable_suffix=$2
212            shift
213            ;;
214        --extra-arguments)
215            export extra_arguments=$2
216            shift
217            ;;
218        --extra-arguments-pub)
219            export extra_arguments_pub=$2
220            shift
221            ;;
222        --extra-arguments-sub)
223            export extra_arguments_sub=$2
224            shift
225            ;;
226        --skip-no-batching)
227            export skip_no_batching="1"
228            ;;
229        --skip-be)
230            export skip_be_tests="1"
231            ;;
232        --skip-rel)
233            export skip_rel_tests="1"
234            ;;
235        --skip-keyed)
236            export skip_keyed_data="1"
237            ;;
238        --skip-large-data)
239            export skip_large_data="1"
240            ;;
241        --large-data)
242            export large_data="1"
243            ;;
244        --keyed)
245            export skip_unkeyed="1"
246            ;;
247        --unkeyed)
248            export skip_keyed_data="1"
249            ;;
250        --no-batching)
251            export no_batching_only="1"
252            ;;
253        --reliable)
254            export skip_be_tests="1"
255            ;;
256        --best-effort)
257            export skip_rel_tests="1"
258            ;;
259        --security)
260            export security_only="$2"
261            shift
262            ;;
263        --micro)
264            export micro="1"
265            ;;
266        --raw | --raw-transport)
267            export raw="1"
268            ;;
269        --tss)
270            export tss="1"
271            ;;
272        --no-colors)
273            export NO_COLORS="1"
274            ;;
275        --language)
276            export LANGUAGE=$2
277            shift
278            ;;
279        --loss-rate)
280            export loss_rate=$2
281            shift
282            ;;
283        --get-netstat-info)
284            export get_netstat_info="1"
285            ;;
286        *)
287            echo -e "unknown parameter \"$1\""
288            exit 255
289            ;;
290    esac
291    shift
292done
293
294if [[ "$NO_COLORS" == "1" ]]; then
295    disable_colors
296fi
297
298export folder_base="$(dirname "${executable}")"/../../..
299
300if [[ $LANGUAGE == "java"  || "$LANGUAGE" == "cs" ]]; then
301    export folder_base="$(dirname "${executable}")"/../..
302fi
303if [[ $tss == "1" ]]; then
304    export folder_base="$(dirname "${executable}")"/../../../../..
305fi
306
307if [[ "${executable_suffix}" != "" ]]; then
308    export executable="${executable}${executable_suffix}"
309fi
310
311if [[ "${sub_folder}" != "" ]]; then
312    export output_folder="${output_folder}/${sub_folder}"
313fi
314
315echo -e "${INFO_TAG} Perftest executable is: $executable"
316echo -e "${INFO_TAG} Output folder is: $output_folder"
317
318################################################################################
319
320if [[ "$LANGUAGE" == "python" ]]; then
321    export skip_keyed_data="1"
322    export skip_large_data="1"
323    export skip_be_tests="1"
324    export skip_no_batching="1"
325fi
326
327if [[ "${skip_large_data}" == "1" ]]; then
328    export datasizes_extended=${datasizes}
329elif [[ "${large_data}" == "1" ]]; then
330    export datasizes=${datasizes_extended}
331fi
332
333if [[ "${datalen_input}" != "" ]]; then
334    echo -e "${YELLOW}[TEST] Testing only for ${datalen_input}${NC}"
335    export datasizes=${datalen_input}
336    export datasizes_extended=${datalen_input}
337    if [[ "${no_batching_only}" != "1" ]]; then
338        export skip_large_data="1"
339    fi
340fi
341
342if [[ "$role" != "pub" && "$role" != "sub" ]]; then
343    echo -e "${ERROR_TAG} It must be either publisher or subscriber"
344    exit 255
345fi
346
347if [[ "$lat_thr" != "thr" && "$lat_thr" != "lat" ]]; then
348    echo -e "${ERROR_TAG} It must be either lat or thr"
349    exit 255
350fi
351
352if [[ "${interface}" == "" ]]; then
353    echo "Using default nics"
354    export nic_publisher=${ip_machine_1}
355    export nic_subscriber=${ip_machine_2}
356else
357    export nic_publisher=$interface
358    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"
359
360    if [[ "${interface2}" == "" ]]; then
361        export nic_subscriber=$interface
362    else
363        export nic_subscriber=$interface2
364    fi
365    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"
366
367    if [[ "${ip1}" != "" ]]; then
368        export ip_publisher=$ip1
369        echo "Using ip_publisher: ${ip_publisher}"
370    fi
371
372    if [[ "${ip2}" != "" ]]; then
373        export ip_subscriber=$ip2
374        echo "Using ip_subscriber: ${ip_subscriber}"
375    fi
376
377fi
378
379export transport_string="-transport $transport"
380
381if [[ "$transport" == "UDPv4" ]]; then
382
383    export transport_string_pub="$transport_string -nic $nic_publisher"
384    export transport_string_sub="$transport_string -nic $nic_subscriber"
385
386    if [[ "$micro" == "1" || "$raw" == "1" ]]; then
387        export transport_string_pub="$transport_string_pub -peer ${ip_subscriber}"
388        export transport_string_sub="$transport_string_sub -peer ${ip_publisher}"
389    fi
390
391elif [[ "$transport" == "TCP" ]]; then
392    export transport_string_pub="$transport_string \
393        -nic $nic_publisher \
394        -peer 0@tcpv4_lan://${ip_subscriber}:7400"
395    export transport_string_sub="$transport_string \
396        -nic $nic_subscriber \
397        -peer 0@tcpv4_lan://${ip_publisher}:7400"
398elif [[ "$transport" == "TLS" ]]; then
399    export transport_string_pub="$transport_string \
400        -nic $nic_publisher \
401        -peer tlsv4_lan://${ip_subscriber}:7400"
402    export transport_string_sub="$transport_string \
403        -nic $nic_subscriber \
404        -peer tlsv4_lan://${ip_publisher}:7400"
405elif [[ "$transport" == "UDPv4_WAN" ]]; then
406    export transport_string_pub="$transport_string \
407        -nic $nic_publisher \
408        -transportPublicAddress $ip_publisher:7400"
409    export transport_string_sub="$transport_string \
410        -nic $nic_subscriber \
411        -peer 0@udpv4_wan://${ip_publisher}:7400"
412else
413    export transport_string_pub="$transport_string"
414    export transport_string_sub="$transport_string"
415fi
416
417################################################################################
418
419export pub_string="-pub \
420        ${transport_string_pub} \
421        -noPrintIntervals \
422        -executionTime $exec_time"
423
424if [[ ${lat_thr} == "lat" ]]; then
425    export pub_string="$pub_string \
426        -latencyTest"
427fi
428
429export sub_string="-sub \
430        ${transport_string_sub} \
431        -noPrintIntervals"
432
433if [[ "$role" == "pub" ]]; then
434    echo -e "$INFO_TAG Publisher side running"
435    export commands_string=${pub_string}
436    export extra_arguments="${extra_arguments} ${extra_arguments_pub}"
437else
438    echo -e "$INFO_TAG Subscriber side running"
439    export commands_string=${sub_string}
440    export extra_arguments="${extra_arguments} ${extra_arguments_sub}"
441fi
442
443###############################################################################
444
445echo -e "${INFO_TAG} Executing: /set_${lat_thr}_mode.sh"
446sudo /set_${lat_thr}_mode.sh
447sleep 5
448
449echo -e "${INFO_TAG} Disabling any loss rate"
450sudo tc qdisc add dev $nic_publisher root netem loss 0%
451sudo tc qdisc del dev $nic_publisher root netem loss 0%
452
453if [[ "$role" == "pub" && "${loss_rate}" != "" ]]; then
454    echo -e "${INFO_TAG} Setting loss rate to ${loss_rate}%"
455    sudo tc qdisc add dev $nic_publisher root netem loss $loss_rate%
456fi
457
458cd $folder_base
459echo -e "${INFO_TAG} Folder Base is: $PWD"
460mkdir -p $output_folder
461
462# Tests that may use batching (when doing throughput tests)
463if [[ ${no_batching_only} != "1" ]]; then
464
465    # UNKEYED
466    if [[ "${skip_unkeyed}" == "" ]]; then
467
468        # RELIABLE
469        if [[ "${skip_rel_tests}" == "" ]]; then
470            execute_test "unkeyed" "rel" "${datasizes_extended}" "${extra_arguments}" "$file_suffix"
471        fi
472
473        # BEST EFFORT
474        if [[ "${skip_be_tests}" == "" ]]; then
475            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
476        fi
477    fi
478
479    # KEYED
480    if [[ "${skip_keyed_data}" == "" ]]; then
481
482        # RELIABLE
483        if [[ "${skip_rel_tests}" == "" ]]; then
484            execute_test "keyed" "rel" "${datasizes}" "${extra_arguments}" "$file_suffix"
485        fi
486
487        # BEST EFFORT
488        if [[ "${skip_be_tests}" == "" ]]; then
489            execute_test "keyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
490        fi
491    fi
492
493fi
494
495if [[ "${skip_no_batching}" == "" || "${no_batching_only}" == "1" ]]; then
496    no_batching_tests="1"
497fi
498
499# Tests that will not use batching
500if [[ "${lat_thr}" == "thr" && "${no_batching_tests}" == "1" ]]; then
501
502    if [[ "$role" == "pub" ]]; then
503        export commands_string="${commands_string} -batchSize 0"
504    fi
505
506    # UNKEYED
507    if [[ "${skip_unkeyed}" == "" ]]; then
508
509        # RELIABLE
510        if [[ "${skip_rel_tests}" == "" ]]; then
511            execute_test "unkeyed" "rel" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
512        fi
513
514        # BEST EFFORT
515        if [[ "${skip_be_tests}" == "" ]]; then
516            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
517        fi
518    fi
519fi
520
521if [[ "$role" == "pub" && "${loss_rate}" != "" ]]; then
522    echo -e "${INFO_TAG} Disabling loss rate"
523    sudo tc qdisc del dev $nic_publisher root netem loss $loss_rate%
524fi

 1#!/bin/bash
 2filename=$0
 3script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)
 4
 5export input_params=$@
 6
 7while [ "$1" != "" ]; do
 8    case $1 in
 9        --executable)
10            executable=$2
11            shift
12            ;;
13        --security)
14            export security_only=$2
15            shift
16            ;;
17        *)
18            ;;
19    esac
20    shift
21done
22
23echo $security_only
24
25export folder_base="$(dirname "${executable}")"/../../..
26export PATH_TO_GOVERNANCE_FILES_FOLDER=$folder_base/resource/secure
27
28if [[ "${security_only}" == "none" || "${security_only}" == "" ]]; then
29    echo -e "[Calling base_script/script.sh] -- No Security"
30    "${script_location}/../base_script/script.sh" $input_params --transport UDPv4 \
31        --skip-no-batching --skip-be --skip-keyed --skip-large-data \
32        --file-suffix "_security_none"
33    sleep 5;
34fi
35
36if [[ "${security_only}" == "no_protection" || "${security_only}" == "" ]]; then
37    echo -e "[Calling base_script/script.sh] -- No Protection"
38    "${script_location}/../base_script/script.sh" $input_params --transport UDPv4 \
39        --skip-no-batching --skip-be --skip-keyed --skip-large-data \
40        --extra-arguments "-secureGovernanceFile $PATH_TO_GOVERNANCE_FILES_FOLDER/signed_PerftestGovernance_.xml " \
41        --file-suffix "_security_no_protection"
42fi
43
44if [[ "${security_only}" == "rtps_sign" || "${security_only}" == "" ]]; then
45    echo -e "[Calling base_script/script.sh] -- RTPS Sign"
46    "${script_location}/../base_script/script.sh" $input_params --transport UDPv4 \
47        --skip-no-batching --skip-be --skip-keyed --skip-large-data \
48        --extra-arguments "-secureGovernanceFile $PATH_TO_GOVERNANCE_FILES_FOLDER/signed_PerftestGovernance_RTPSSign.xml " \
49        --file-suffix "_security_rtps_sign"
50fi
51
52if [[ "${security_only}" == "rtps_encrypt" || "${security_only}" == "" ]]; then
53    echo -e "[Calling base_script/script.sh] -- RTPS Encrypt"
54    "${script_location}/../base_script/script.sh" $input_params --transport UDPv4 \
55        --skip-no-batching --skip-be --skip-keyed --skip-large-data \
56        --extra-arguments "-secureGovernanceFile $PATH_TO_GOVERNANCE_FILES_FOLDER/signed_PerftestGovernance_RTPSEncrypt.xml " \
57        --file-suffix "_security_rtps_encrypt"
58fi
59
60if [[ "${security_only}" == "rtps_sign_submessage_encrypt" || "${security_only}" == "" ]]; then
61    echo -e "[Calling base_script/script.sh] -- RTPS Sign, Submessage Encrypt"
62    "${script_location}/../base_script/script.sh" $input_params --transport UDPv4 \
63        --skip-no-batching --skip-be --skip-keyed --skip-large-data \
64        --extra-arguments "-secureGovernanceFile $PATH_TO_GOVERNANCE_FILES_FOLDER/signed_PerftestGovernance_SignEncryptSubmessage.xml " \
65        --file-suffix "_security_rtps_sign_submessage_encrypt"
66fi
67
68if [[ "${security_only}" == "rtps_sign_submessage_encrypt_orig_data_encrypt" || "${security_only}" == "" ]]; then
69    echo -e "[Calling base_script/script.sh] -- RTPS Sign, Submessage Encrypt with original auth, Data Encrypt"
70    "${script_location}/../base_script/script.sh" $input_params --transport UDPv4 \
71        --skip-no-batching --skip-be --skip-keyed --skip-large-data \
72        --extra-arguments "-secureGovernanceFile $PATH_TO_GOVERNANCE_FILES_FOLDER/signed_PerftestGovernance_RTPSSignEncryptSubmessageWithOrigAuthEncryptData.xml " \
73        --file-suffix "_security_rtps_sign_submessage_encrypt_orig_data_encrypt"
74fi
75
76if [[ "${security_only}" == "rtps_sign_orig_data_encrypt" || "${security_only}" == "" ]]; then
77    echo -e "[Calling base_script/script.sh] -- RTPS Sign with Original auth, Data Encrypt"
78    "${script_location}/../base_script/script.sh" $input_params --transport UDPv4 \
79        --skip-no-batching --skip-be --skip-keyed --skip-large-data \
80        --extra-arguments "-secureGovernanceFile $PATH_TO_GOVERNANCE_FILES_FOLDER/signed_PerftestGovernance_RTPSSignWithOrigAuthEncryptData.xml " \
81        --file-suffix "_security_rtps_sign_orig_data_encrypt"
82    sleep 5;
83fi

Security Profiles

Test Hardware

The following hardware was used to perform these tests:

Linux Nodes

Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 20.04 -- gcc 9.3.0

Switch

Dell 2048 -- 10Gbps switch (10Gbps and 1Gbps interfaces)