1.2.2. Modern C++ API
The following tests have been performed by executing an RTI Perftest C++11 Publisher and Subscriber between two nodes, connected to a switch via Ethernet. The communication has been restricted to a single interface and the transport has been set to UDPv4.
Find information about the hardware, network, and command-line parameters after each of the tests.
1.2.2.1. UDPv4, C++11
The graph below shows the one-way latency without load between a Publisher and a Subscriber running in two Linux nodes in a 10Gbps network. The numbers are for best-effort as well as strict reliable reliability scenarios.
Note
We use the median (50th percentile) instead of the average in order to get a more stable measurement that does not account for spurious outliers. We also calculate the average value and other percentile values, which can be seen in the Detailed Statistics section below.
Detailed Statistics
The following tables contain the raw numbers presented by RTI Perftest. These numbers are the exact output with no further processing.
Best Effort
Sample Size (Bytes)  | 
Ave (μs)  | 
Std (μs)  | 
Min (μs)  | 
Max (μs)  | 
50% (μs)  | 
90% (μs)  | 
99% (μs)  | 
99.99% (μs)  | 
99.9999% (μs)  | 
|---|---|---|---|---|---|---|---|---|---|
32  | 
15  | 
1.0  | 
15  | 
65  | 
15  | 
16  | 
17  | 
45  | 
65  | 
64  | 
16  | 
1.0  | 
15  | 
57  | 
15  | 
16  | 
17  | 
45  | 
57  | 
128  | 
16  | 
1.0  | 
15  | 
59  | 
16  | 
16  | 
17  | 
45  | 
59  | 
256  | 
16  | 
0.9  | 
15  | 
55  | 
16  | 
16  | 
17  | 
46  | 
55  | 
512  | 
19  | 
1.6  | 
16  | 
70  | 
19  | 
21  | 
22  | 
49  | 
70  | 
1024  | 
25  | 
4.7  | 
17  | 
71  | 
26  | 
29  | 
33  | 
56  | 
71  | 
2048  | 
29  | 
31.9  | 
20  | 
5025  | 
30  | 
33  | 
42  | 
68  | 
5025  | 
4096  | 
31  | 
17.6  | 
25  | 
4196  | 
30  | 
35  | 
48  | 
69  | 
4196  | 
8192  | 
44  | 
82.4  | 
35  | 
4982  | 
37  | 
56  | 
68  | 
4436  | 
4982  | 
16384  | 
84  | 
24.9  | 
45  | 
167  | 
83  | 
121  | 
133  | 
154  | 
167  | 
32768  | 
158  | 
19.1  | 
75  | 
1904  | 
165  | 
175  | 
178  | 
388  | 
1904  | 
63000  | 
184  | 
2.5  | 
101  | 
223  | 
184  | 
186  | 
189  | 
221  | 
223  | 
Reliable
Sample Size (Bytes)  | 
Ave (μs)  | 
Std (μs)  | 
Min (μs)  | 
Max (μs)  | 
50% (μs)  | 
90% (μs)  | 
99% (μs)  | 
99.99% (μs)  | 
99.9999% (μs)  | 
|---|---|---|---|---|---|---|---|---|---|
32  | 
17  | 
1.1  | 
16  | 
68  | 
17  | 
17  | 
21  | 
46  | 
68  | 
64  | 
17  | 
1.1  | 
16  | 
58  | 
17  | 
17  | 
22  | 
47  | 
58  | 
128  | 
17  | 
1.1  | 
16  | 
60  | 
17  | 
17  | 
23  | 
47  | 
60  | 
256  | 
18  | 
1.9  | 
17  | 
62  | 
17  | 
19  | 
26  | 
48  | 
62  | 
512  | 
20  | 
2.7  | 
17  | 
74  | 
19  | 
22  | 
30  | 
49  | 
74  | 
1024  | 
25  | 
4.9  | 
18  | 
71  | 
26  | 
28  | 
42  | 
58  | 
71  | 
2048  | 
29  | 
31.9  | 
20  | 
5025  | 
30  | 
33  | 
42  | 
68  | 
5025  | 
4096  | 
31  | 
17.6  | 
25  | 
4196  | 
30  | 
35  | 
48  | 
69  | 
4196  | 
8192  | 
44  | 
82.4  | 
35  | 
4982  | 
37  | 
56  | 
68  | 
4436  | 
4982  | 
16384  | 
87  | 
25.9  | 
46  | 
168  | 
86  | 
125  | 
138  | 
156  | 
168  | 
32768  | 
160  | 
19.1  | 
75  | 
1904  | 
165  | 
175  | 
178  | 
388  | 
1904  | 
63000  | 
186  | 
2.6  | 
108  | 
227  | 
186  | 
187  | 
191  | 
216  | 
227  | 
100000  | 
208  | 
7.3  | 
159  | 
295  | 
210  | 
212  | 
217  | 
247  | 
295  | 
500000  | 
612  | 
31.6  | 
512  | 
933  | 
614  | 
656  | 
666  | 
736  | 
933  | 
1048576  | 
1250  | 
58.4  | 
1074  | 
1841  | 
1286  | 
1307  | 
1320  | 
1841  | 
1841  | 
1548576  | 
1690  | 
14.4  | 
1525  | 
2643  | 
1689  | 
1699  | 
1714  | 
2643  | 
2643  | 
4194304  | 
4070  | 
69.5  | 
4027  | 
6958  | 
4066  | 
4078  | 
4103  | 
6958  | 
6958  | 
10485760  | 
10228  | 
241.6  | 
10195  | 
17463  | 
10218  | 
10230  | 
10257  | 
17463  | 
17463  | 
Perftest Scripts
To produce these tests, we executed RTI Perftest for C++11. The exact script used can be found here:
  1#!/bin/bash
  2filename=$0
  3script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)
  4
  5export datasizes="32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000"
  6export datasizes_extended="${datasizes} 100000 500000 1048576 1548576 4194304 10485760"
  7
  8export domain="2"
  9export exec_time=20
 10export num_reps=1
 11export instance_number=100000
 12export core=0
 13export run_batching_tests="1"
 14export run_no_batching_tests="1"
 15
 16# We will use some colors to improve visibility of errors and info messages.
 17RED='\033[0;31m'
 18GREEN='\033[0;32m'
 19YELLOW='\033[0;33m'
 20BLUE='\033[0;34m'
 21LIGHTBLUE='\033[0;36m'
 22NC='\033[0m'
 23INFO_TAG="${GREEN}[INFO][$role]:${NC}"
 24WARNING_TAG="${YELLOW}[WARNING][$role]:${NC}"
 25ERROR_TAG="${RED}[ERROR][$role]:${NC}"
 26
 27################################################################################
 28
 29function disable_colors() {
 30    export RED=""
 31    export GREEN=""
 32    export YELLOW=""
 33    export NC=""
 34    export BLUE=""
 35    export LIGHTBLUE=""
 36    export INFO_TAG="${GREEN}[INFO][$role]:${NC}"
 37    export WARNING_TAG="${YELLOW}[WARNING][$role]:${NC}"
 38    export ERROR_TAG="${RED}[ERROR][$role]:${NC}"
 39}
 40
 41function change_domain() {
 42    if [[ "$domain" == "1" ]]; then
 43        export domain="2"
 44    else
 45        export domain="1"
 46    fi
 47}
 48
 49# Function to process and append lines to the output file
 50function append_to_output_file() {
 51    local line=${1?line required}; readonly line
 52    local append=$(echo "${2?append required}" | tr -s ' '); readonly append
 53    local file=${3?file required}; readonly file
 54    if [[ "${line: -1}" == $'\n' ]]; then
 55        line="${line::-1}"
 56    fi
 57    echo "${line}${append}" >> "$file"
 58}
 59
 60# Usage: execute_test <keyed/unkeyed> <rel/be> <datasizes> <batchSize>
 61function execute_test() {
 62
 63    local keyed_unkeyed=$1
 64    local rel_be=$2
 65    local datasizes_test=$3
 66    local other_args=$4
 67    local name_suffix=$5
 68
 69    local commands_string_test=$commands_string
 70    local tag=""
 71
 72    if [[ "${keyed_unkeyed}" == "keyed" ]]; then
 73        commands_string_test="${commands_string_test} -keyed -instances $instance_number"
 74        tag="[${YELLOW}${transport}${NC}|${BLUE}K${NC}|"
 75    else
 76        tag="[${YELLOW}${transport}${NC}|${LIGHTBLUE}UK${NC}|"
 77    fi
 78
 79    if [[ "${rel_be}" == "be" ]]; then
 80        commands_string_test="${commands_string_test} -bestEffort"
 81        tag="${tag}${YELLOW}BE${NC}]"
 82    else
 83        tag="${tag}${RED}REL${NC}]"
 84    fi
 85
 86    # If batch_size is set, we will add it to the command line of the publisher side.
 87    if [[ "$batch_size" != "" && "$role" == "pub" ]]; then
 88        commands_string_test="${commands_string_test} -batchSize $batch_size"
 89    fi
 90
 91    tag="${tag}[${LIGHTBLUE}${lat_thr}${NC}]"
 92
 93    local output_file=$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}.csv
 94
 95    if [[ "$role" == "pub" ]]; then
 96        echo -e "${YELLOW}[TEST]: $keyed_unkeyed, $rel_be, Is a no-batching test = $no_batching_tests. ${NC}"
 97    fi
 98
 99    if [[ "$thread_cpu_affinity" != "" ]]; then
100        commands_string_test="${commands_string_test} -threadCPUAffinity $thread_cpu_affinity"
101    fi
102
103    local pre_command_string=""
104
105    if [[ "$thread_priorities" != "" ]]; then
106        echo -e "${WARNING_TAG} Thread priorities enabled, this requires using sudo"
107        # When using sudo, the LD_LIBRARY_PATH is emptied, hence we need to set it again in the command itself
108        export LD_LIBRARY_PATH_COPIED="$LD_LIBRARY_PATH"
109        pre_command_string="sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH_COPIED "
110        commands_string_test="${commands_string_test} -threadPriorities $thread_priorities"
111    fi
112
113    if [[ "$pin_memory" != "" ]]; then
114        commands_string_test="${commands_string_test} -pinMemory"
115    fi
116
117    if [[ "$no_taskset" == "" && "$thread_cpu_affinity" == "" && "$LANGUAGE" != "java" && "$LANGUAGE" != "cs" ]]; then
118        pre_command_string="$pre_command_string taskset -c $core"
119    fi
120
121    if [[ "$LANGUAGE" == "python" ]]; then
122        pre_command_string="$pre_command_string python3 "
123    fi
124
125    if [[ "$DOCKER" == "1" ]]; then
126        pre_command_string="$pre_command_string docker run --net=host -v /home/perfuser/rti_license_connextpro.dat:/opt/rti.com/rti_connext_dds-7.3.0/rti_license.dat rticom/perftest:7.3.0-EAR "
127        executable=""
128    fi
129
130    # Get the aprox time this will take:
131    total_tests=$((`wc -w <<< "$datasizes_test"` * num_reps))
132    total_time=$((total_tests * exec_time))
133
134    touch $output_file
135    local no_headers=""
136    local current_test=0
137    for index in $(seq 1 ${num_reps}); do
138        for DATALEN in ${datasizes_test}; do
139            current_test=$((current_test + 1))
140
141            if [[ ! -s $output_file ]]; then
142                echo -e "${INFO_TAG} Output file is empty, filling the header."
143                no_headers=""
144            else
145                echo -e "${INFO_TAG} Output file is not empty."
146                no_headers=" -noOutputHeaders"
147            fi
148
149            export command="$pre_command_string $executable -domain $domain -dataLen $DATALEN $commands_string_test $other_args $no_headers"
150            if [[ "$role" == "pub" ]]; then
151                echo -e "Test ${tag} (${current_test}/${total_tests}) -- Total time = ${total_time}s"
152                echo -e ${BLUE}$command${NC}
153            else
154                echo -e ${LIGHTBLUE}$command${NC}
155            fi
156
157            # In certain cases we need to wait a bit before running the test, this is
158            # because the previous test might not be finished on the other side yet, or because the
159            # discovery mechanism does not work like in connext DDS.
160            if [[ "$LANGUAGE" == "cs" && "$role" == "pub" ]]; then
161                sleep 3
162            fi
163            if [[ "$raw" == "1" && "$role" == "sub" ]]; then
164                sleep 5
165            fi
166
167            # Gather netstat info before running the test
168            if [[ "${get_netstat_info}" == "1" ]]; then
169                echo -e "${INFO_TAG} Getting netstat info before"
170                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt
171            fi
172
173            # Execute the command and capture the output
174            if [[ $LANGUAGE == "c++98" || $LANGUAGE == "" ]]; then
175                echo -e "${INFO_TAG} Using C++98, using -outputFile option"
176                touch ${output_file}_tmp_line
177                chmod 777 ${output_file}_tmp_line
178                eval $command -outputFile ${output_file}_tmp_line
179            else
180                eval $command > ${output_file}_tmp_line
181            fi
182
183            number_of_lines=$(wc -l < ${output_file}_tmp_line)
184            echo -e "${INFO_TAG} Size of the output text: $number_of_lines lines"
185
186            # Check if this is the first test for this output file.
187            local is_first_test=0
188            if [[ ! -s $output_file ]]; then
189                is_first_test=1
190            fi
191            local expected_lines=$((is_first_test+1))
192
193            if [[ $number_of_lines -ne ${expected_lines} ]]; then
194                echo -e "${WARNING_TAG} The output text should have ${expected_lines} lines, but it has ${number_of_lines}."
195                echo -e "${RED} The content is: \n\"\"\""
196                cat ${output_file}_tmp_line
197                echo -e "\"\"\""
198                echo -e "Not adding it to the output file. ${NC}"
199            else
200                if [[ $number_of_lines -gt 1 ]]; then
201                    header_line=$(head -n 1 "${output_file}_tmp_line")
202                    echo -e "${INFO_TAG} Header line is: $header_line"
203                fi
204                result_line=$(tail -n 1 "${output_file}_tmp_line")
205                echo -e "${INFO_TAG} Result line is: $result_line"
206
207                # If this is the first test, we need to add the header line to the output file
208                if [[ $is_first_test -eq 1 ]]; then
209                    append_to_output_file "$header_line" ", command-line" $output_file
210                fi
211
212                command=$(echo $command | sed 's/,/:comma:/g')
213                # Remove the `"` characters from the command line
214                command=$(echo $command | sed 's/\"//g')
215
216                # Append the result line to the output file
217                append_to_output_file "$result_line" ", $command" $output_file
218            fi
219
220            # Always remove the temporary file
221            rm -rf ${output_file}_tmp_line
222
223            # Gather netstat info after running the test
224            if [[ "${get_netstat_info}" == "1" ]]; then
225                echo -e "${INFO_TAG} Getting netstat info after"
226                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt
227                touch "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
228                python3 $script_location/../../../tools/diff_netstat_output.py \
229                    -n $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt \
230                    -o $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt \
231                    -d $DATALEN $no_header_netstat \
232                    -csv >> "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
233                rm -rf $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_*.txt
234                no_header_netstat=" -nh"
235            fi
236
237            change_domain
238        done
239    done
240}
241
242################################################################################
243# PARSE COMMAND LINE OPTIONS:
244
245while [ "$1" != "" ]; do
246    case $1 in
247        --executable)
248            executable=$2
249            shift
250            ;;
251        --docker)
252            DOCKER="1"
253            ;;
254        --output-folder)
255            output_folder=$2
256            shift
257            ;;
258        --role)
259            export role=$2
260            shift
261            ;;
262        --core)
263            export core=$2
264            shift
265            ;;
266        --assign-core-list | --thread-cpu-affinity)
267            export thread_cpu_affinity=$2
268            shift
269            ;;
270        --assign-thread-priorities | --thread-priorities)
271            export thread_priorities=$2
272            shift
273            ;;
274        --pin-memory)
275            export pin_memory="1"
276            ;;
277        --test-kind)
278            export lat_thr=$2
279            shift
280            ;;
281        --interface1)
282            export interface=$2
283            shift
284            ;;
285        --interface2)
286            export interface2=$2
287            shift
288            ;;
289        --ip1)
290            export ip1=$2
291            shift
292            ;;
293        --ip2)
294            export ip2=$2
295            shift
296            ;;
297        --repetitions)
298            export num_reps=$2
299            shift
300            ;;
301        --domain)
302            export domain=$2
303            shift
304            ;;
305        --execution-time)
306            export exec_time=$2
307            shift
308            ;;
309        --transport)
310            export transport=$2
311            shift
312            ;;
313        --datalen)
314            export datalen_input=$2
315            shift
316            ;;
317        --file-suffix)
318            export file_suffix=$2
319            shift
320            ;;
321        --folder-suffix)
322            export folder_suffix=$2
323            shift
324            ;;
325        --sub-folder)
326            sub_folder=$2
327            shift
328            ;;
329        --executable-suffix)
330            export executable_suffix=$2
331            shift
332            ;;
333        --extra-arguments)
334            export extra_arguments=$2
335            shift
336            ;;
337        --extra-arguments-pub)
338            export extra_arguments_pub=$2
339            shift
340            ;;
341        --extra-arguments-sub)
342            export extra_arguments_sub=$2
343            shift
344            ;;
345        --skip-be)
346            export skip_be_tests="1"
347            ;;
348        --skip-rel)
349            export skip_rel_tests="1"
350            ;;
351        --skip-keyed)
352            export skip_keyed_data="1"
353            ;;
354        --skip-large-data)
355            export skip_large_data="1"
356            ;;
357        --large-data)
358            export large_data="1"
359            ;;
360        --keyed)
361            export skip_unkeyed="1"
362            ;;
363        --unkeyed)
364            export skip_keyed_data="1"
365            ;;
366        --no-batching | --skip-batching)
367            export run_batching_tests="0"
368            export run_no_batching_tests="1"
369            ;;
370        --skip-no-batching)
371            export run_batching_tests="1"
372            export run_no_batching_tests="0"
373            ;;
374        --batch-size)
375            export batch_size=$2
376            shift
377            ;;
378        --reliable)
379            export skip_be_tests="1"
380            ;;
381        --best-effort)
382            export skip_rel_tests="1"
383            ;;
384        --security-gov)
385            export security_only="$2"
386            shift
387            ;;
388        --micro)
389            export micro="1"
390            ;;
391        --cert)
392            export cert="1"
393            ;;
394        --raw | --raw-transport)
395            export raw="1"
396            ;;
397        --tss)
398            export tss="1"
399            ;;
400        --no-colors)
401            export NO_COLORS="1"
402            ;;
403        --language)
404            export LANGUAGE=$2
405            shift
406            ;;
407        --loss-rate)
408            export loss_rate=$2
409            shift
410            ;;
411        --get-netstat-info | --netstat)
412            export get_netstat_info="1"
413            ;;
414        --reduced-data-sizes-set)
415            export REDUCED_DATA_SIZES_SET="1"
416            ;;
417        --dont-change-tuned-settings)
418            export dont_change_tuned_settings="1"
419            ;;
420        --no-taskset)
421            export no_taskset=1
422            ;;
423        *)
424            echo -e "unknown parameter \"$1\""
425            exit 255
426            ;;
427    esac
428    shift
429done
430
431if [[ "$NO_COLORS" == "1" ]]; then
432    disable_colors
433fi
434
435export folder_base="$(dirname "${executable}")"/../../..
436
437if [[ $LANGUAGE == "java"  || "$LANGUAGE" == "cs" ]]; then
438    export folder_base="$(dirname "${executable}")"/../..
439fi
440if [[ $tss == "1" ]]; then
441    export folder_base="$(dirname "${executable}")"/../../../../..
442fi
443
444if [[ "${executable_suffix}" != "" ]]; then
445    export executable="${executable}${executable_suffix}"
446fi
447
448if [[ -n "${folder_suffix}" ]]; then
449    export output_folder="${output_folder}${folder_suffix}"
450fi
451
452if [[ -n "${sub_folder}" ]]; then
453    export output_folder="${output_folder}/${sub_folder}"
454fi
455
456echo -e "${INFO_TAG} Perftest executable is: $executable"
457echo -e "${INFO_TAG} Output folder is: $output_folder"
458
459################################################################################
460
461if [[ "$LANGUAGE" == "python" ]]; then
462    export skip_keyed_data="1"
463    export skip_large_data="1"
464    export skip_be_tests="1"
465    export run_no_batching_tests="0"
466fi
467
468if [[ "${skip_large_data}" == "1" ]]; then
469    export datasizes_extended=${datasizes}
470elif [[ "${large_data}" == "1" ]]; then
471    export datasizes=${datasizes_extended}
472fi
473
474if [[ "${datalen_input}" != "" ]]; then
475    echo -e "${YELLOW}[TEST] Testing only for ${datalen_input}${NC}"
476    export datasizes=${datalen_input}
477    export datasizes_extended=${datalen_input}
478    if [[ "${run_no_batching_tests}" == "1" && "${run_batching_tests}" == "0" ]]; then
479        export skip_large_data="1"
480    fi
481else 
482    if [[ "${REDUCED_DATA_SIZES_SET}" != "" ]]; then
483        echo -e "${YELLOW}[TEST] Testing Reduced set of datasizes ${NC}"
484        export datasizes="32 128 512 2048 8192 32768 63000"
485        export datasizes_extended="${datasizes} 102400 1048576 10485760"
486    fi
487fi
488
489if [[ "$role" != "pub" && "$role" != "sub" ]]; then
490    echo -e "${ERROR_TAG} It must be either publisher or subscriber"
491    exit 255
492fi
493
494if [[ "$lat_thr" != "thr" && "$lat_thr" != "lat" ]]; then
495    echo -e "${ERROR_TAG} It must be either lat or thr"
496    exit 255
497fi
498
499if [[ "${interface}" == "" ]]; then
500    echo "Using default nics"
501    export nic_publisher=${ip_machine_1}
502    export nic_subscriber=${ip_machine_2}
503elif [[ "${interface}" == "both" ]]; then
504    export nic_publisher="enp1s0f0,eno1"
505    export nic_subscriber="enp1s0f0,eno1"
506    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"
507    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"
508else
509    export nic_publisher=$interface
510    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"
511
512    if [[ "${interface2}" == "" ]]; then
513        export nic_subscriber=$interface
514    else
515        export nic_subscriber=$interface2
516    fi
517    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"
518
519    if [[ "${ip1}" != "" ]]; then
520        export ip_publisher=$ip1
521        echo "Using ip_publisher: ${ip_publisher}"
522    fi
523
524    if [[ "${ip2}" != "" ]]; then
525        export ip_subscriber=$ip2
526        echo "Using ip_subscriber: ${ip_subscriber}"
527    fi
528
529fi
530
531export transport_string="-transport $transport"
532
533if [[ "$transport" == "UDPv4" ]]; then
534
535    export transport_string_pub="$transport_string -nic $nic_publisher"
536    export transport_string_sub="$transport_string -nic $nic_subscriber"
537
538    if [[ "$raw" == "1" ]]; then
539        export transport_string_pub="$transport_string_pub -peer ${ip_subscriber}"
540        export transport_string_sub="$transport_string_sub -peer ${ip_publisher}"
541    fi
542
543    if [[ "$micro" == "1" || "$cert" == "1" ]]; then
544        export transport_string_pub="$transport_string_pub -peer _udp://${ip_subscriber}"
545        export transport_string_sub="$transport_string_sub -peer _udp://${ip_publisher}"
546    fi
547
548elif [[ "$transport" == "TCP" ]]; then
549    export transport_string_pub="$transport_string \
550        -nic $nic_publisher \
551        -peer 0@tcpv4_lan://${ip_subscriber}:7400"
552    export transport_string_sub="$transport_string \
553        -nic $nic_subscriber \
554        -peer 0@tcpv4_lan://${ip_publisher}:7400"
555elif [[ "$transport" == "TLS" ]]; then
556    export transport_string_pub="$transport_string \
557        -nic $nic_publisher \
558        -peer tlsv4_lan://${ip_subscriber}:7400"
559    export transport_string_sub="$transport_string \
560        -nic $nic_subscriber \
561        -peer tlsv4_lan://${ip_publisher}:7400"
562elif [[ "$transport" == "UDPv4_WAN" ]]; then
563    export transport_string_pub="$transport_string \
564        -nic $nic_publisher \
565        -transportPublicAddress $ip_publisher:7400"
566    export transport_string_sub="$transport_string \
567        -nic $nic_subscriber \
568        -peer 0@udpv4_wan://${ip_publisher}:7400"
569else
570    export transport_string_pub="$transport_string"
571    export transport_string_sub="$transport_string"
572fi
573
574################################################################################
575
576export pub_string="-pub \
577        ${transport_string_pub} \
578        -noPrintIntervals \
579        -executionTime $exec_time"
580
581if [[ ${lat_thr} == "lat" ]]; then
582    export pub_string="$pub_string \
583        -latencyTest"
584fi
585
586export sub_string="-sub \
587        ${transport_string_sub} \
588        -noPrintIntervals"
589
590if [[ "$role" == "pub" ]]; then
591    echo -e "$INFO_TAG Publisher side running"
592    export commands_string=${pub_string}
593    export extra_arguments="${extra_arguments} ${extra_arguments_pub}"
594else
595    echo -e "$INFO_TAG Subscriber side running"
596    export commands_string=${sub_string}
597    export extra_arguments="${extra_arguments} ${extra_arguments_sub}"
598fi
599
600###############################################################################
601
602if [[ "$dont_change_tuned_settings" != "1" ]]; then
603    echo -e "${INFO_TAG} Executing: /set_${lat_thr}_mode.sh"
604    sudo /set_${lat_thr}_mode.sh
605    sleep 5
606fi
607
608echo -e "${INFO_TAG} Disabling any loss rate"
609sudo tc qdisc add dev $nic_publisher root netem loss 0%
610sudo tc qdisc del dev $nic_publisher root netem loss 0%
611
612if [[ "$role" == "pub" && "${loss_rate}" != "" ]]; then
613    echo -e "${INFO_TAG} Setting loss rate to ${loss_rate}%"
614    sudo tc qdisc add dev $nic_publisher root netem loss $loss_rate%
615fi
616
617cd $folder_base
618echo -e "${INFO_TAG} Folder Base is: $PWD"
619mkdir -p $output_folder
620
621if [[ "${batch_size}" != "" ]]; then
622    if [[ "${lat_thr}" == "thr" ]]; then
623        if [[ "${batch_size}" -eq 0 ]]; then
624            echo -e "${INFO_TAG} Batch size is set to 0"
625            export run_batching_tests="0"
626            export run_no_batching_tests="1"
627        else
628            echo -e "${INFO_TAG} Batch size is set to ${batch_size}"
629            export run_batching_tests="1"
630            export run_no_batching_tests="0"
631        fi
632    else
633        echo -e "${INFO_TAG} Batch size is set to ${batch_size}. This value will be ignored for latency tests."
634        unset batch_size
635    fi
636fi
637
638# Tests that may use batching (when doing throughput tests). Also Latency tests.
639if [[ ${run_batching_tests} == "1" ]]; then
640
641    # UNKEYED
642    if [[ "${skip_unkeyed}" == "" ]]; then
643
644        # RELIABLE
645        if [[ "${skip_rel_tests}" == "" ]]; then
646            execute_test "unkeyed" "rel" "${datasizes_extended}" "${extra_arguments}" "$file_suffix"
647        fi
648
649        # BEST EFFORT
650        if [[ "${skip_be_tests}" == "" ]]; then
651            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
652        fi
653    fi
654
655    # KEYED
656    if [[ "${skip_keyed_data}" == "" ]]; then
657
658        # RELIABLE
659        if [[ "${skip_rel_tests}" == "" ]]; then
660            execute_test "keyed" "rel" "${datasizes}" "${extra_arguments}" "$file_suffix"
661        fi
662
663        # BEST EFFORT
664        if [[ "${skip_be_tests}" == "" ]]; then
665            execute_test "keyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
666        fi
667    fi
668
669fi
670
671# Tests that will not use batching
672if [[ "${lat_thr}" == "thr" && "${run_no_batching_tests}" == "1" ]]; then
673
674    if [[ "$role" == "pub" ]]; then
675        export commands_string="${commands_string} -batchSize 0"
676    fi
677
678    # UNKEYED
679    if [[ "${skip_unkeyed}" == "" ]]; then
680
681        # RELIABLE
682        if [[ "${skip_rel_tests}" == "" ]]; then
683            execute_test "unkeyed" "rel" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
684        fi
685
686        # BEST EFFORT
687        if [[ "${skip_be_tests}" == "" ]]; then
688            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
689        fi
690    fi
691
692    # KEYED
693    if [[ "${skip_keyed_data}" == "" ]]; then
694
695        # RELIABLE
696        if [[ "${skip_rel_tests}" == "" ]]; then
697            execute_test "keyed" "rel" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
698        fi
699
700        # BEST EFFORT
701        if [[ "${skip_be_tests}" == "" ]]; then
702            execute_test "keyed" "be" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
703        fi
704    fi
705
706fi
707
708if [[ "$role" == "pub" && "${loss_rate}" != "" ]]; then
709    echo -e "${INFO_TAG} Disabling loss rate"
710    sudo tc qdisc del dev $nic_publisher root netem loss $loss_rate%
711fi
1#!/bin/bash
2filename=$0
3script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)
4
5echo -e "[Calling base_script/script.sh]"
6"${script_location}/../base_script/script.sh" $@ --transport UDPv4
Test Hardware
The following hardware was used to perform these tests:
Linux Nodes
Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 20.04 -- gcc 9.3.0
Switch
Dell 2048 -- 10Gbps switch (10Gbps and 1Gbps interfaces)
The graph below shows the expected throughput behavior when performing a 1-1 communication between two Linux nodes in a 10Gbps network. The numbers are for best-effort as well as strict reliable reliability scenarios.
Note
By default, RTI Perftest enables batching when performing a Maximum Throughput test. The batching feature allows sending more than one data sample per RTPS packet, improving network performance for small data sizes. See the RTI Connext Core Libraries User’s Manual for more information on batching.
The batch maximum size is set by RTI Perftest to be 8192 bytes; after 8192 bytes, batching is not enabled.
Detailed Statistics
This table contains the raw numbers presented by RTI Perftest. These numbers are the exact output with no further processing.
Best Effort
Sample Size (Bytes)  | 
Total Samples  | 
Avg Samples/s  | 
Avg Mbps  | 
Lost Samples  | 
Lost Samples (%)  | 
|---|---|---|---|---|---|
32  | 
96201984  | 
4807775  | 
1230.8  | 
256  | 
0.00  | 
64  | 
87351460  | 
4362868  | 
2233.8  | 
0  | 
0.00  | 
128  | 
77301771  | 
3862449  | 
3955.1  | 
0  | 
0.00  | 
256  | 
62113607  | 
3103058  | 
6355.1  | 
0  | 
0.00  | 
512  | 
42699824  | 
2132513  | 
8734.8  | 
0  | 
0.00  | 
1024  | 
23815888  | 
1190135  | 
9749.6  | 
0  | 
0.00  | 
2048  | 
11952736  | 
597354  | 
9787.1  | 
0  | 
0.00  | 
4096  | 
5987620  | 
299253  | 
9805.9  | 
0  | 
0.00  | 
8192  | 
3006329  | 
150276  | 
9848.5  | 
0  | 
0.00  | 
16384  | 
1509068  | 
75446  | 
9888.9  | 
38  | 
0.00  | 
32768  | 
756063  | 
37800  | 
9909.3  | 
5  | 
0.00  | 
63000  | 
393530  | 
19675  | 
9916.3  | 
3  | 
0.00  | 
Reliable
Sample Size (Bytes)  | 
Total Samples  | 
Avg Samples/s  | 
Avg Mbps  | 
Lost Samples  | 
Lost Samples (%)  | 
|---|---|---|---|---|---|
32  | 
91819198  | 
4589513  | 
1174.9  | 
0  | 
0.00  | 
64  | 
82504832  | 
4121824  | 
2110.4  | 
0  | 
0.00  | 
128  | 
67678840  | 
3381720  | 
3462.9  | 
0  | 
0.00  | 
256  | 
52099028  | 
2601721  | 
5328.3  | 
0  | 
0.00  | 
512  | 
34868016  | 
1742216  | 
7136.1  | 
0  | 
0.00  | 
1024  | 
20971817  | 
1047873  | 
8584.2  | 
0  | 
0.00  | 
2048  | 
11694996  | 
583948  | 
9567.4  | 
0  | 
0.00  | 
4096  | 
5985007  | 
299041  | 
9799.0  | 
0  | 
0.00  | 
8192  | 
3005301  | 
150212  | 
9844.3  | 
0  | 
0.00  | 
16384  | 
1508835  | 
75431  | 
9887.0  | 
0  | 
0.00  | 
32768  | 
756045  | 
37797  | 
9908.3  | 
0  | 
0.00  | 
63000  | 
393524  | 
19674  | 
9915.8  | 
0  | 
0.00  | 
100000  | 
245123  | 
12252  | 
9802.2  | 
0  | 
0.00  | 
500000  | 
46396  | 
2318  | 
9275.5  | 
0  | 
0.00  | 
1048576  | 
14115  | 
704  | 
5913.6  | 
0  | 
0.00  | 
1548576  | 
14448  | 
720  | 
8922.0  | 
0  | 
0.00  | 
4194304  | 
4936  | 
246  | 
8278.2  | 
0  | 
0.00  | 
10485760  | 
2048  | 
101  | 
8540.5  | 
0  | 
0.00  | 
Perftest Scripts
To produce these tests, we executed RTI Perftest for C++11. The exact script used can be found here:
  1#!/bin/bash
  2filename=$0
  3script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)
  4
  5export datasizes="32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000"
  6export datasizes_extended="${datasizes} 100000 500000 1048576 1548576 4194304 10485760"
  7
  8export domain="2"
  9export exec_time=20
 10export num_reps=1
 11export instance_number=100000
 12export core=0
 13export run_batching_tests="1"
 14export run_no_batching_tests="1"
 15
 16# We will use some colors to improve visibility of errors and info messages.
 17RED='\033[0;31m'
 18GREEN='\033[0;32m'
 19YELLOW='\033[0;33m'
 20BLUE='\033[0;34m'
 21LIGHTBLUE='\033[0;36m'
 22NC='\033[0m'
 23INFO_TAG="${GREEN}[INFO][$role]:${NC}"
 24WARNING_TAG="${YELLOW}[WARNING][$role]:${NC}"
 25ERROR_TAG="${RED}[ERROR][$role]:${NC}"
 26
 27################################################################################
 28
 29function disable_colors() {
 30    export RED=""
 31    export GREEN=""
 32    export YELLOW=""
 33    export NC=""
 34    export BLUE=""
 35    export LIGHTBLUE=""
 36    export INFO_TAG="${GREEN}[INFO][$role]:${NC}"
 37    export WARNING_TAG="${YELLOW}[WARNING][$role]:${NC}"
 38    export ERROR_TAG="${RED}[ERROR][$role]:${NC}"
 39}
 40
 41function change_domain() {
 42    if [[ "$domain" == "1" ]]; then
 43        export domain="2"
 44    else
 45        export domain="1"
 46    fi
 47}
 48
 49# Function to process and append lines to the output file
 50function append_to_output_file() {
 51    local line=${1?line required}; readonly line
 52    local append=$(echo "${2?append required}" | tr -s ' '); readonly append
 53    local file=${3?file required}; readonly file
 54    if [[ "${line: -1}" == $'\n' ]]; then
 55        line="${line::-1}"
 56    fi
 57    echo "${line}${append}" >> "$file"
 58}
 59
 60# Usage: execute_test <keyed/unkeyed> <rel/be> <datasizes> <batchSize>
 61function execute_test() {
 62
 63    local keyed_unkeyed=$1
 64    local rel_be=$2
 65    local datasizes_test=$3
 66    local other_args=$4
 67    local name_suffix=$5
 68
 69    local commands_string_test=$commands_string
 70    local tag=""
 71
 72    if [[ "${keyed_unkeyed}" == "keyed" ]]; then
 73        commands_string_test="${commands_string_test} -keyed -instances $instance_number"
 74        tag="[${YELLOW}${transport}${NC}|${BLUE}K${NC}|"
 75    else
 76        tag="[${YELLOW}${transport}${NC}|${LIGHTBLUE}UK${NC}|"
 77    fi
 78
 79    if [[ "${rel_be}" == "be" ]]; then
 80        commands_string_test="${commands_string_test} -bestEffort"
 81        tag="${tag}${YELLOW}BE${NC}]"
 82    else
 83        tag="${tag}${RED}REL${NC}]"
 84    fi
 85
 86    # If batch_size is set, we will add it to the command line of the publisher side.
 87    if [[ "$batch_size" != "" && "$role" == "pub" ]]; then
 88        commands_string_test="${commands_string_test} -batchSize $batch_size"
 89    fi
 90
 91    tag="${tag}[${LIGHTBLUE}${lat_thr}${NC}]"
 92
 93    local output_file=$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}.csv
 94
 95    if [[ "$role" == "pub" ]]; then
 96        echo -e "${YELLOW}[TEST]: $keyed_unkeyed, $rel_be, Is a no-batching test = $no_batching_tests. ${NC}"
 97    fi
 98
 99    if [[ "$thread_cpu_affinity" != "" ]]; then
100        commands_string_test="${commands_string_test} -threadCPUAffinity $thread_cpu_affinity"
101    fi
102
103    local pre_command_string=""
104
105    if [[ "$thread_priorities" != "" ]]; then
106        echo -e "${WARNING_TAG} Thread priorities enabled, this requires using sudo"
107        # When using sudo, the LD_LIBRARY_PATH is emptied, hence we need to set it again in the command itself
108        export LD_LIBRARY_PATH_COPIED="$LD_LIBRARY_PATH"
109        pre_command_string="sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH_COPIED "
110        commands_string_test="${commands_string_test} -threadPriorities $thread_priorities"
111    fi
112
113    if [[ "$pin_memory" != "" ]]; then
114        commands_string_test="${commands_string_test} -pinMemory"
115    fi
116
117    if [[ "$no_taskset" == "" && "$thread_cpu_affinity" == "" && "$LANGUAGE" != "java" && "$LANGUAGE" != "cs" ]]; then
118        pre_command_string="$pre_command_string taskset -c $core"
119    fi
120
121    if [[ "$LANGUAGE" == "python" ]]; then
122        pre_command_string="$pre_command_string python3 "
123    fi
124
125    if [[ "$DOCKER" == "1" ]]; then
126        pre_command_string="$pre_command_string docker run --net=host -v /home/perfuser/rti_license_connextpro.dat:/opt/rti.com/rti_connext_dds-7.3.0/rti_license.dat rticom/perftest:7.3.0-EAR "
127        executable=""
128    fi
129
130    # Get the aprox time this will take:
131    total_tests=$((`wc -w <<< "$datasizes_test"` * num_reps))
132    total_time=$((total_tests * exec_time))
133
134    touch $output_file
135    local no_headers=""
136    local current_test=0
137    for index in $(seq 1 ${num_reps}); do
138        for DATALEN in ${datasizes_test}; do
139            current_test=$((current_test + 1))
140
141            if [[ ! -s $output_file ]]; then
142                echo -e "${INFO_TAG} Output file is empty, filling the header."
143                no_headers=""
144            else
145                echo -e "${INFO_TAG} Output file is not empty."
146                no_headers=" -noOutputHeaders"
147            fi
148
149            export command="$pre_command_string $executable -domain $domain -dataLen $DATALEN $commands_string_test $other_args $no_headers"
150            if [[ "$role" == "pub" ]]; then
151                echo -e "Test ${tag} (${current_test}/${total_tests}) -- Total time = ${total_time}s"
152                echo -e ${BLUE}$command${NC}
153            else
154                echo -e ${LIGHTBLUE}$command${NC}
155            fi
156
157            # In certain cases we need to wait a bit before running the test, this is
158            # because the previous test might not be finished on the other side yet, or because the
159            # discovery mechanism does not work like in connext DDS.
160            if [[ "$LANGUAGE" == "cs" && "$role" == "pub" ]]; then
161                sleep 3
162            fi
163            if [[ "$raw" == "1" && "$role" == "sub" ]]; then
164                sleep 5
165            fi
166
167            # Gather netstat info before running the test
168            if [[ "${get_netstat_info}" == "1" ]]; then
169                echo -e "${INFO_TAG} Getting netstat info before"
170                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt
171            fi
172
173            # Execute the command and capture the output
174            if [[ $LANGUAGE == "c++98" || $LANGUAGE == "" ]]; then
175                echo -e "${INFO_TAG} Using C++98, using -outputFile option"
176                touch ${output_file}_tmp_line
177                chmod 777 ${output_file}_tmp_line
178                eval $command -outputFile ${output_file}_tmp_line
179            else
180                eval $command > ${output_file}_tmp_line
181            fi
182
183            number_of_lines=$(wc -l < ${output_file}_tmp_line)
184            echo -e "${INFO_TAG} Size of the output text: $number_of_lines lines"
185
186            # Check if this is the first test for this output file.
187            local is_first_test=0
188            if [[ ! -s $output_file ]]; then
189                is_first_test=1
190            fi
191            local expected_lines=$((is_first_test+1))
192
193            if [[ $number_of_lines -ne ${expected_lines} ]]; then
194                echo -e "${WARNING_TAG} The output text should have ${expected_lines} lines, but it has ${number_of_lines}."
195                echo -e "${RED} The content is: \n\"\"\""
196                cat ${output_file}_tmp_line
197                echo -e "\"\"\""
198                echo -e "Not adding it to the output file. ${NC}"
199            else
200                if [[ $number_of_lines -gt 1 ]]; then
201                    header_line=$(head -n 1 "${output_file}_tmp_line")
202                    echo -e "${INFO_TAG} Header line is: $header_line"
203                fi
204                result_line=$(tail -n 1 "${output_file}_tmp_line")
205                echo -e "${INFO_TAG} Result line is: $result_line"
206
207                # If this is the first test, we need to add the header line to the output file
208                if [[ $is_first_test -eq 1 ]]; then
209                    append_to_output_file "$header_line" ", command-line" $output_file
210                fi
211
212                command=$(echo $command | sed 's/,/:comma:/g')
213                # Remove the `"` characters from the command line
214                command=$(echo $command | sed 's/\"//g')
215
216                # Append the result line to the output file
217                append_to_output_file "$result_line" ", $command" $output_file
218            fi
219
220            # Always remove the temporary file
221            rm -rf ${output_file}_tmp_line
222
223            # Gather netstat info after running the test
224            if [[ "${get_netstat_info}" == "1" ]]; then
225                echo -e "${INFO_TAG} Getting netstat info after"
226                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt
227                touch "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
228                python3 $script_location/../../../tools/diff_netstat_output.py \
229                    -n $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt \
230                    -o $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt \
231                    -d $DATALEN $no_header_netstat \
232                    -csv >> "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
233                rm -rf $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_*.txt
234                no_header_netstat=" -nh"
235            fi
236
237            change_domain
238        done
239    done
240}
241
242################################################################################
243# PARSE COMMAND LINE OPTIONS:
244
245while [ "$1" != "" ]; do
246    case $1 in
247        --executable)
248            executable=$2
249            shift
250            ;;
251        --docker)
252            DOCKER="1"
253            ;;
254        --output-folder)
255            output_folder=$2
256            shift
257            ;;
258        --role)
259            export role=$2
260            shift
261            ;;
262        --core)
263            export core=$2
264            shift
265            ;;
266        --assign-core-list | --thread-cpu-affinity)
267            export thread_cpu_affinity=$2
268            shift
269            ;;
270        --assign-thread-priorities | --thread-priorities)
271            export thread_priorities=$2
272            shift
273            ;;
274        --pin-memory)
275            export pin_memory="1"
276            ;;
277        --test-kind)
278            export lat_thr=$2
279            shift
280            ;;
281        --interface1)
282            export interface=$2
283            shift
284            ;;
285        --interface2)
286            export interface2=$2
287            shift
288            ;;
289        --ip1)
290            export ip1=$2
291            shift
292            ;;
293        --ip2)
294            export ip2=$2
295            shift
296            ;;
297        --repetitions)
298            export num_reps=$2
299            shift
300            ;;
301        --domain)
302            export domain=$2
303            shift
304            ;;
305        --execution-time)
306            export exec_time=$2
307            shift
308            ;;
309        --transport)
310            export transport=$2
311            shift
312            ;;
313        --datalen)
314            export datalen_input=$2
315            shift
316            ;;
317        --file-suffix)
318            export file_suffix=$2
319            shift
320            ;;
321        --folder-suffix)
322            export folder_suffix=$2
323            shift
324            ;;
325        --sub-folder)
326            sub_folder=$2
327            shift
328            ;;
329        --executable-suffix)
330            export executable_suffix=$2
331            shift
332            ;;
333        --extra-arguments)
334            export extra_arguments=$2
335            shift
336            ;;
337        --extra-arguments-pub)
338            export extra_arguments_pub=$2
339            shift
340            ;;
341        --extra-arguments-sub)
342            export extra_arguments_sub=$2
343            shift
344            ;;
345        --skip-be)
346            export skip_be_tests="1"
347            ;;
348        --skip-rel)
349            export skip_rel_tests="1"
350            ;;
351        --skip-keyed)
352            export skip_keyed_data="1"
353            ;;
354        --skip-large-data)
355            export skip_large_data="1"
356            ;;
357        --large-data)
358            export large_data="1"
359            ;;
360        --keyed)
361            export skip_unkeyed="1"
362            ;;
363        --unkeyed)
364            export skip_keyed_data="1"
365            ;;
366        --no-batching | --skip-batching)
367            export run_batching_tests="0"
368            export run_no_batching_tests="1"
369            ;;
370        --skip-no-batching)
371            export run_batching_tests="1"
372            export run_no_batching_tests="0"
373            ;;
374        --batch-size)
375            export batch_size=$2
376            shift
377            ;;
378        --reliable)
379            export skip_be_tests="1"
380            ;;
381        --best-effort)
382            export skip_rel_tests="1"
383            ;;
384        --security-gov)
385            export security_only="$2"
386            shift
387            ;;
388        --micro)
389            export micro="1"
390            ;;
391        --cert)
392            export cert="1"
393            ;;
394        --raw | --raw-transport)
395            export raw="1"
396            ;;
397        --tss)
398            export tss="1"
399            ;;
400        --no-colors)
401            export NO_COLORS="1"
402            ;;
403        --language)
404            export LANGUAGE=$2
405            shift
406            ;;
407        --loss-rate)
408            export loss_rate=$2
409            shift
410            ;;
411        --get-netstat-info | --netstat)
412            export get_netstat_info="1"
413            ;;
414        --reduced-data-sizes-set)
415            export REDUCED_DATA_SIZES_SET="1"
416            ;;
417        --dont-change-tuned-settings)
418            export dont_change_tuned_settings="1"
419            ;;
420        --no-taskset)
421            export no_taskset=1
422            ;;
423        *)
424            echo -e "unknown parameter \"$1\""
425            exit 255
426            ;;
427    esac
428    shift
429done
430
431if [[ "$NO_COLORS" == "1" ]]; then
432    disable_colors
433fi
434
435export folder_base="$(dirname "${executable}")"/../../..
436
437if [[ $LANGUAGE == "java"  || "$LANGUAGE" == "cs" ]]; then
438    export folder_base="$(dirname "${executable}")"/../..
439fi
440if [[ $tss == "1" ]]; then
441    export folder_base="$(dirname "${executable}")"/../../../../..
442fi
443
444if [[ "${executable_suffix}" != "" ]]; then
445    export executable="${executable}${executable_suffix}"
446fi
447
448if [[ -n "${folder_suffix}" ]]; then
449    export output_folder="${output_folder}${folder_suffix}"
450fi
451
452if [[ -n "${sub_folder}" ]]; then
453    export output_folder="${output_folder}/${sub_folder}"
454fi
455
456echo -e "${INFO_TAG} Perftest executable is: $executable"
457echo -e "${INFO_TAG} Output folder is: $output_folder"
458
459################################################################################
460
461if [[ "$LANGUAGE" == "python" ]]; then
462    export skip_keyed_data="1"
463    export skip_large_data="1"
464    export skip_be_tests="1"
465    export run_no_batching_tests="0"
466fi
467
468if [[ "${skip_large_data}" == "1" ]]; then
469    export datasizes_extended=${datasizes}
470elif [[ "${large_data}" == "1" ]]; then
471    export datasizes=${datasizes_extended}
472fi
473
474if [[ "${datalen_input}" != "" ]]; then
475    echo -e "${YELLOW}[TEST] Testing only for ${datalen_input}${NC}"
476    export datasizes=${datalen_input}
477    export datasizes_extended=${datalen_input}
478    if [[ "${run_no_batching_tests}" == "1" && "${run_batching_tests}" == "0" ]]; then
479        export skip_large_data="1"
480    fi
481else 
482    if [[ "${REDUCED_DATA_SIZES_SET}" != "" ]]; then
483        echo -e "${YELLOW}[TEST] Testing Reduced set of datasizes ${NC}"
484        export datasizes="32 128 512 2048 8192 32768 63000"
485        export datasizes_extended="${datasizes} 102400 1048576 10485760"
486    fi
487fi
488
489if [[ "$role" != "pub" && "$role" != "sub" ]]; then
490    echo -e "${ERROR_TAG} It must be either publisher or subscriber"
491    exit 255
492fi
493
494if [[ "$lat_thr" != "thr" && "$lat_thr" != "lat" ]]; then
495    echo -e "${ERROR_TAG} It must be either lat or thr"
496    exit 255
497fi
498
499if [[ "${interface}" == "" ]]; then
500    echo "Using default nics"
501    export nic_publisher=${ip_machine_1}
502    export nic_subscriber=${ip_machine_2}
503elif [[ "${interface}" == "both" ]]; then
504    export nic_publisher="enp1s0f0,eno1"
505    export nic_subscriber="enp1s0f0,eno1"
506    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"
507    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"
508else
509    export nic_publisher=$interface
510    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"
511
512    if [[ "${interface2}" == "" ]]; then
513        export nic_subscriber=$interface
514    else
515        export nic_subscriber=$interface2
516    fi
517    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"
518
519    if [[ "${ip1}" != "" ]]; then
520        export ip_publisher=$ip1
521        echo "Using ip_publisher: ${ip_publisher}"
522    fi
523
524    if [[ "${ip2}" != "" ]]; then
525        export ip_subscriber=$ip2
526        echo "Using ip_subscriber: ${ip_subscriber}"
527    fi
528
529fi
530
531export transport_string="-transport $transport"
532
533if [[ "$transport" == "UDPv4" ]]; then
534
535    export transport_string_pub="$transport_string -nic $nic_publisher"
536    export transport_string_sub="$transport_string -nic $nic_subscriber"
537
538    if [[ "$raw" == "1" ]]; then
539        export transport_string_pub="$transport_string_pub -peer ${ip_subscriber}"
540        export transport_string_sub="$transport_string_sub -peer ${ip_publisher}"
541    fi
542
543    if [[ "$micro" == "1" || "$cert" == "1" ]]; then
544        export transport_string_pub="$transport_string_pub -peer _udp://${ip_subscriber}"
545        export transport_string_sub="$transport_string_sub -peer _udp://${ip_publisher}"
546    fi
547
548elif [[ "$transport" == "TCP" ]]; then
549    export transport_string_pub="$transport_string \
550        -nic $nic_publisher \
551        -peer 0@tcpv4_lan://${ip_subscriber}:7400"
552    export transport_string_sub="$transport_string \
553        -nic $nic_subscriber \
554        -peer 0@tcpv4_lan://${ip_publisher}:7400"
555elif [[ "$transport" == "TLS" ]]; then
556    export transport_string_pub="$transport_string \
557        -nic $nic_publisher \
558        -peer tlsv4_lan://${ip_subscriber}:7400"
559    export transport_string_sub="$transport_string \
560        -nic $nic_subscriber \
561        -peer tlsv4_lan://${ip_publisher}:7400"
562elif [[ "$transport" == "UDPv4_WAN" ]]; then
563    export transport_string_pub="$transport_string \
564        -nic $nic_publisher \
565        -transportPublicAddress $ip_publisher:7400"
566    export transport_string_sub="$transport_string \
567        -nic $nic_subscriber \
568        -peer 0@udpv4_wan://${ip_publisher}:7400"
569else
570    export transport_string_pub="$transport_string"
571    export transport_string_sub="$transport_string"
572fi
573
574################################################################################
575
576export pub_string="-pub \
577        ${transport_string_pub} \
578        -noPrintIntervals \
579        -executionTime $exec_time"
580
581if [[ ${lat_thr} == "lat" ]]; then
582    export pub_string="$pub_string \
583        -latencyTest"
584fi
585
586export sub_string="-sub \
587        ${transport_string_sub} \
588        -noPrintIntervals"
589
590if [[ "$role" == "pub" ]]; then
591    echo -e "$INFO_TAG Publisher side running"
592    export commands_string=${pub_string}
593    export extra_arguments="${extra_arguments} ${extra_arguments_pub}"
594else
595    echo -e "$INFO_TAG Subscriber side running"
596    export commands_string=${sub_string}
597    export extra_arguments="${extra_arguments} ${extra_arguments_sub}"
598fi
599
600###############################################################################
601
602if [[ "$dont_change_tuned_settings" != "1" ]]; then
603    echo -e "${INFO_TAG} Executing: /set_${lat_thr}_mode.sh"
604    sudo /set_${lat_thr}_mode.sh
605    sleep 5
606fi
607
608echo -e "${INFO_TAG} Disabling any loss rate"
609sudo tc qdisc add dev $nic_publisher root netem loss 0%
610sudo tc qdisc del dev $nic_publisher root netem loss 0%
611
612if [[ "$role" == "pub" && "${loss_rate}" != "" ]]; then
613    echo -e "${INFO_TAG} Setting loss rate to ${loss_rate}%"
614    sudo tc qdisc add dev $nic_publisher root netem loss $loss_rate%
615fi
616
617cd $folder_base
618echo -e "${INFO_TAG} Folder Base is: $PWD"
619mkdir -p $output_folder
620
621if [[ "${batch_size}" != "" ]]; then
622    if [[ "${lat_thr}" == "thr" ]]; then
623        if [[ "${batch_size}" -eq 0 ]]; then
624            echo -e "${INFO_TAG} Batch size is set to 0"
625            export run_batching_tests="0"
626            export run_no_batching_tests="1"
627        else
628            echo -e "${INFO_TAG} Batch size is set to ${batch_size}"
629            export run_batching_tests="1"
630            export run_no_batching_tests="0"
631        fi
632    else
633        echo -e "${INFO_TAG} Batch size is set to ${batch_size}. This value will be ignored for latency tests."
634        unset batch_size
635    fi
636fi
637
638# Tests that may use batching (when doing throughput tests). Also Latency tests.
639if [[ ${run_batching_tests} == "1" ]]; then
640
641    # UNKEYED
642    if [[ "${skip_unkeyed}" == "" ]]; then
643
644        # RELIABLE
645        if [[ "${skip_rel_tests}" == "" ]]; then
646            execute_test "unkeyed" "rel" "${datasizes_extended}" "${extra_arguments}" "$file_suffix"
647        fi
648
649        # BEST EFFORT
650        if [[ "${skip_be_tests}" == "" ]]; then
651            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
652        fi
653    fi
654
655    # KEYED
656    if [[ "${skip_keyed_data}" == "" ]]; then
657
658        # RELIABLE
659        if [[ "${skip_rel_tests}" == "" ]]; then
660            execute_test "keyed" "rel" "${datasizes}" "${extra_arguments}" "$file_suffix"
661        fi
662
663        # BEST EFFORT
664        if [[ "${skip_be_tests}" == "" ]]; then
665            execute_test "keyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
666        fi
667    fi
668
669fi
670
671# Tests that will not use batching
672if [[ "${lat_thr}" == "thr" && "${run_no_batching_tests}" == "1" ]]; then
673
674    if [[ "$role" == "pub" ]]; then
675        export commands_string="${commands_string} -batchSize 0"
676    fi
677
678    # UNKEYED
679    if [[ "${skip_unkeyed}" == "" ]]; then
680
681        # RELIABLE
682        if [[ "${skip_rel_tests}" == "" ]]; then
683            execute_test "unkeyed" "rel" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
684        fi
685
686        # BEST EFFORT
687        if [[ "${skip_be_tests}" == "" ]]; then
688            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
689        fi
690    fi
691
692    # KEYED
693    if [[ "${skip_keyed_data}" == "" ]]; then
694
695        # RELIABLE
696        if [[ "${skip_rel_tests}" == "" ]]; then
697            execute_test "keyed" "rel" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
698        fi
699
700        # BEST EFFORT
701        if [[ "${skip_be_tests}" == "" ]]; then
702            execute_test "keyed" "be" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
703        fi
704    fi
705
706fi
707
708if [[ "$role" == "pub" && "${loss_rate}" != "" ]]; then
709    echo -e "${INFO_TAG} Disabling loss rate"
710    sudo tc qdisc del dev $nic_publisher root netem loss $loss_rate%
711fi
1#!/bin/bash
2filename=$0
3script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)
4
5echo -e "[Calling base_script/script.sh]"
6"${script_location}/../base_script/script.sh" $@ --transport UDPv4
Test Hardware
The following hardware was used to perform these tests:
Linux Nodes
Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 20.04 -- gcc 9.3.0
Switch
Dell 2048 -- 10Gbps switch (10Gbps and 1Gbps interfaces)
The graph below shows the one-way latency without load between a Publisher and a Subscriber running in two Linux nodes in a 10Gbps network. The numbers are for best-effort as well as strict reliable reliability scenarios.
Note
We use the median (50th percentile) instead of the average in order to get a more stable measurement that does not account for spurious outliers. We also calculate the average value and other percentile values, which can be seen in the Detailed Statistics section below.
Detailed Statistics
The following tables contain the raw numbers presented by RTI Perftest. These numbers are the exact output with no further processing.
Best Effort
Sample Size (Bytes)  | 
Ave (μs)  | 
Std (μs)  | 
Min (μs)  | 
Max (μs)  | 
50% (μs)  | 
90% (μs)  | 
99% (μs)  | 
99.99% (μs)  | 
99.9999% (μs)  | 
|---|---|---|---|---|---|---|---|---|---|
32  | 
19  | 
145.1  | 
17  | 
100084  | 
18  | 
19  | 
21  | 
49  | 
100084  | 
64  | 
19  | 
1.2  | 
17  | 
161  | 
18  | 
19  | 
21  | 
49  | 
161  | 
128  | 
18  | 
1.0  | 
17  | 
166  | 
18  | 
19  | 
21  | 
48  | 
166  | 
256  | 
19  | 
1.1  | 
18  | 
165  | 
19  | 
20  | 
22  | 
50  | 
165  | 
512  | 
20  | 
1.2  | 
18  | 
162  | 
20  | 
20  | 
22  | 
50  | 
162  | 
1024  | 
25  | 
2.8  | 
20  | 
169  | 
26  | 
28  | 
29  | 
55  | 
169  | 
2048  | 
40  | 
169.1  | 
22  | 
103155  | 
32  | 
62  | 
63  | 
78  | 
103155  | 
4096  | 
37  | 
150.7  | 
26  | 
100094  | 
32  | 
50  | 
56  | 
79  | 
100094  | 
8192  | 
51  | 
195.4  | 
35  | 
100103  | 
49  | 
64  | 
71  | 
90  | 
100103  | 
16384  | 
89  | 
27.2  | 
47  | 
255  | 
89  | 
129  | 
143  | 
160  | 
255  | 
32768  | 
177  | 
2.7  | 
68  | 
317  | 
177  | 
178  | 
181  | 
213  | 
317  | 
63000  | 
188  | 
3.4  | 
105  | 
330  | 
188  | 
190  | 
193  | 
222  | 
330  | 
Reliable
Sample Size (Bytes)  | 
Ave (μs)  | 
Std (μs)  | 
Min (μs)  | 
Max (μs)  | 
50% (μs)  | 
90% (μs)  | 
99% (μs)  | 
99.99% (μs)  | 
99.9999% (μs)  | 
|---|---|---|---|---|---|---|---|---|---|
32  | 
20  | 
1.2  | 
19  | 
162  | 
20  | 
21  | 
24  | 
51  | 
162  | 
64  | 
20  | 
1.3  | 
19  | 
160  | 
20  | 
21  | 
25  | 
51  | 
160  | 
128  | 
20  | 
1.2  | 
19  | 
163  | 
20  | 
21  | 
24  | 
51  | 
163  | 
256  | 
20  | 
1.7  | 
20  | 
163  | 
20  | 
22  | 
29  | 
52  | 
163  | 
512  | 
22  | 
13.8  | 
20  | 
5056  | 
21  | 
24  | 
35  | 
53  | 
5056  | 
1024  | 
26  | 
4.2  | 
21  | 
166  | 
26  | 
28  | 
49  | 
59  | 
166  | 
2048  | 
29  | 
10.8  | 
23  | 
5066  | 
30  | 
32  | 
46  | 
67  | 
5066  | 
4096  | 
31  | 
11.5  | 
28  | 
4459  | 
30  | 
33  | 
52  | 
67  | 
4459  | 
8192  | 
45  | 
7.9  | 
38  | 
200  | 
41  | 
56  | 
75  | 
102  | 
200  | 
16384  | 
92  | 
27.4  | 
49  | 
228  | 
91  | 
132  | 
148  | 
163  | 
228  | 
32768  | 
176  | 
7.6  | 
78  | 
252  | 
178  | 
179  | 
183  | 
207  | 
252  | 
63000  | 
190  | 
3.0  | 
109  | 
262  | 
190  | 
191  | 
196  | 
221  | 
262  | 
Perftest Scripts
To produce these tests, we executed RTI Perftest for C++11. The exact script used can be found here:
  1#!/bin/bash
  2filename=$0
  3script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)
  4
  5export datasizes="32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000"
  6export datasizes_extended="${datasizes} 100000 500000 1048576 1548576 4194304 10485760"
  7
  8export domain="2"
  9export exec_time=20
 10export num_reps=1
 11export instance_number=100000
 12export core=0
 13export run_batching_tests="1"
 14export run_no_batching_tests="1"
 15
 16# We will use some colors to improve visibility of errors and info messages.
 17RED='\033[0;31m'
 18GREEN='\033[0;32m'
 19YELLOW='\033[0;33m'
 20BLUE='\033[0;34m'
 21LIGHTBLUE='\033[0;36m'
 22NC='\033[0m'
 23INFO_TAG="${GREEN}[INFO][$role]:${NC}"
 24WARNING_TAG="${YELLOW}[WARNING][$role]:${NC}"
 25ERROR_TAG="${RED}[ERROR][$role]:${NC}"
 26
 27################################################################################
 28
 29function disable_colors() {
 30    export RED=""
 31    export GREEN=""
 32    export YELLOW=""
 33    export NC=""
 34    export BLUE=""
 35    export LIGHTBLUE=""
 36    export INFO_TAG="${GREEN}[INFO][$role]:${NC}"
 37    export WARNING_TAG="${YELLOW}[WARNING][$role]:${NC}"
 38    export ERROR_TAG="${RED}[ERROR][$role]:${NC}"
 39}
 40
 41function change_domain() {
 42    if [[ "$domain" == "1" ]]; then
 43        export domain="2"
 44    else
 45        export domain="1"
 46    fi
 47}
 48
 49# Function to process and append lines to the output file
 50function append_to_output_file() {
 51    local line=${1?line required}; readonly line
 52    local append=$(echo "${2?append required}" | tr -s ' '); readonly append
 53    local file=${3?file required}; readonly file
 54    if [[ "${line: -1}" == $'\n' ]]; then
 55        line="${line::-1}"
 56    fi
 57    echo "${line}${append}" >> "$file"
 58}
 59
 60# Usage: execute_test <keyed/unkeyed> <rel/be> <datasizes> <batchSize>
 61function execute_test() {
 62
 63    local keyed_unkeyed=$1
 64    local rel_be=$2
 65    local datasizes_test=$3
 66    local other_args=$4
 67    local name_suffix=$5
 68
 69    local commands_string_test=$commands_string
 70    local tag=""
 71
 72    if [[ "${keyed_unkeyed}" == "keyed" ]]; then
 73        commands_string_test="${commands_string_test} -keyed -instances $instance_number"
 74        tag="[${YELLOW}${transport}${NC}|${BLUE}K${NC}|"
 75    else
 76        tag="[${YELLOW}${transport}${NC}|${LIGHTBLUE}UK${NC}|"
 77    fi
 78
 79    if [[ "${rel_be}" == "be" ]]; then
 80        commands_string_test="${commands_string_test} -bestEffort"
 81        tag="${tag}${YELLOW}BE${NC}]"
 82    else
 83        tag="${tag}${RED}REL${NC}]"
 84    fi
 85
 86    # If batch_size is set, we will add it to the command line of the publisher side.
 87    if [[ "$batch_size" != "" && "$role" == "pub" ]]; then
 88        commands_string_test="${commands_string_test} -batchSize $batch_size"
 89    fi
 90
 91    tag="${tag}[${LIGHTBLUE}${lat_thr}${NC}]"
 92
 93    local output_file=$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}.csv
 94
 95    if [[ "$role" == "pub" ]]; then
 96        echo -e "${YELLOW}[TEST]: $keyed_unkeyed, $rel_be, Is a no-batching test = $no_batching_tests. ${NC}"
 97    fi
 98
 99    if [[ "$thread_cpu_affinity" != "" ]]; then
100        commands_string_test="${commands_string_test} -threadCPUAffinity $thread_cpu_affinity"
101    fi
102
103    local pre_command_string=""
104
105    if [[ "$thread_priorities" != "" ]]; then
106        echo -e "${WARNING_TAG} Thread priorities enabled, this requires using sudo"
107        # When using sudo, the LD_LIBRARY_PATH is emptied, hence we need to set it again in the command itself
108        export LD_LIBRARY_PATH_COPIED="$LD_LIBRARY_PATH"
109        pre_command_string="sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH_COPIED "
110        commands_string_test="${commands_string_test} -threadPriorities $thread_priorities"
111    fi
112
113    if [[ "$pin_memory" != "" ]]; then
114        commands_string_test="${commands_string_test} -pinMemory"
115    fi
116
117    if [[ "$no_taskset" == "" && "$thread_cpu_affinity" == "" && "$LANGUAGE" != "java" && "$LANGUAGE" != "cs" ]]; then
118        pre_command_string="$pre_command_string taskset -c $core"
119    fi
120
121    if [[ "$LANGUAGE" == "python" ]]; then
122        pre_command_string="$pre_command_string python3 "
123    fi
124
125    if [[ "$DOCKER" == "1" ]]; then
126        pre_command_string="$pre_command_string docker run --net=host -v /home/perfuser/rti_license_connextpro.dat:/opt/rti.com/rti_connext_dds-7.3.0/rti_license.dat rticom/perftest:7.3.0-EAR "
127        executable=""
128    fi
129
130    # Get the aprox time this will take:
131    total_tests=$((`wc -w <<< "$datasizes_test"` * num_reps))
132    total_time=$((total_tests * exec_time))
133
134    touch $output_file
135    local no_headers=""
136    local current_test=0
137    for index in $(seq 1 ${num_reps}); do
138        for DATALEN in ${datasizes_test}; do
139            current_test=$((current_test + 1))
140
141            if [[ ! -s $output_file ]]; then
142                echo -e "${INFO_TAG} Output file is empty, filling the header."
143                no_headers=""
144            else
145                echo -e "${INFO_TAG} Output file is not empty."
146                no_headers=" -noOutputHeaders"
147            fi
148
149            export command="$pre_command_string $executable -domain $domain -dataLen $DATALEN $commands_string_test $other_args $no_headers"
150            if [[ "$role" == "pub" ]]; then
151                echo -e "Test ${tag} (${current_test}/${total_tests}) -- Total time = ${total_time}s"
152                echo -e ${BLUE}$command${NC}
153            else
154                echo -e ${LIGHTBLUE}$command${NC}
155            fi
156
157            # In certain cases we need to wait a bit before running the test, this is
158            # because the previous test might not be finished on the other side yet, or because the
159            # discovery mechanism does not work like in connext DDS.
160            if [[ "$LANGUAGE" == "cs" && "$role" == "pub" ]]; then
161                sleep 3
162            fi
163            if [[ "$raw" == "1" && "$role" == "sub" ]]; then
164                sleep 5
165            fi
166
167            # Gather netstat info before running the test
168            if [[ "${get_netstat_info}" == "1" ]]; then
169                echo -e "${INFO_TAG} Getting netstat info before"
170                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt
171            fi
172
173            # Execute the command and capture the output
174            if [[ $LANGUAGE == "c++98" || $LANGUAGE == "" ]]; then
175                echo -e "${INFO_TAG} Using C++98, using -outputFile option"
176                touch ${output_file}_tmp_line
177                chmod 777 ${output_file}_tmp_line
178                eval $command -outputFile ${output_file}_tmp_line
179            else
180                eval $command > ${output_file}_tmp_line
181            fi
182
183            number_of_lines=$(wc -l < ${output_file}_tmp_line)
184            echo -e "${INFO_TAG} Size of the output text: $number_of_lines lines"
185
186            # Check if this is the first test for this output file.
187            local is_first_test=0
188            if [[ ! -s $output_file ]]; then
189                is_first_test=1
190            fi
191            local expected_lines=$((is_first_test+1))
192
193            if [[ $number_of_lines -ne ${expected_lines} ]]; then
194                echo -e "${WARNING_TAG} The output text should have ${expected_lines} lines, but it has ${number_of_lines}."
195                echo -e "${RED} The content is: \n\"\"\""
196                cat ${output_file}_tmp_line
197                echo -e "\"\"\""
198                echo -e "Not adding it to the output file. ${NC}"
199            else
200                if [[ $number_of_lines -gt 1 ]]; then
201                    header_line=$(head -n 1 "${output_file}_tmp_line")
202                    echo -e "${INFO_TAG} Header line is: $header_line"
203                fi
204                result_line=$(tail -n 1 "${output_file}_tmp_line")
205                echo -e "${INFO_TAG} Result line is: $result_line"
206
207                # If this is the first test, we need to add the header line to the output file
208                if [[ $is_first_test -eq 1 ]]; then
209                    append_to_output_file "$header_line" ", command-line" $output_file
210                fi
211
212                command=$(echo $command | sed 's/,/:comma:/g')
213                # Remove the `"` characters from the command line
214                command=$(echo $command | sed 's/\"//g')
215
216                # Append the result line to the output file
217                append_to_output_file "$result_line" ", $command" $output_file
218            fi
219
220            # Always remove the temporary file
221            rm -rf ${output_file}_tmp_line
222
223            # Gather netstat info after running the test
224            if [[ "${get_netstat_info}" == "1" ]]; then
225                echo -e "${INFO_TAG} Getting netstat info after"
226                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt
227                touch "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
228                python3 $script_location/../../../tools/diff_netstat_output.py \
229                    -n $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt \
230                    -o $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt \
231                    -d $DATALEN $no_header_netstat \
232                    -csv >> "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
233                rm -rf $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_*.txt
234                no_header_netstat=" -nh"
235            fi
236
237            change_domain
238        done
239    done
240}
241
242################################################################################
243# PARSE COMMAND LINE OPTIONS:
244
245while [ "$1" != "" ]; do
246    case $1 in
247        --executable)
248            executable=$2
249            shift
250            ;;
251        --docker)
252            DOCKER="1"
253            ;;
254        --output-folder)
255            output_folder=$2
256            shift
257            ;;
258        --role)
259            export role=$2
260            shift
261            ;;
262        --core)
263            export core=$2
264            shift
265            ;;
266        --assign-core-list | --thread-cpu-affinity)
267            export thread_cpu_affinity=$2
268            shift
269            ;;
270        --assign-thread-priorities | --thread-priorities)
271            export thread_priorities=$2
272            shift
273            ;;
274        --pin-memory)
275            export pin_memory="1"
276            ;;
277        --test-kind)
278            export lat_thr=$2
279            shift
280            ;;
281        --interface1)
282            export interface=$2
283            shift
284            ;;
285        --interface2)
286            export interface2=$2
287            shift
288            ;;
289        --ip1)
290            export ip1=$2
291            shift
292            ;;
293        --ip2)
294            export ip2=$2
295            shift
296            ;;
297        --repetitions)
298            export num_reps=$2
299            shift
300            ;;
301        --domain)
302            export domain=$2
303            shift
304            ;;
305        --execution-time)
306            export exec_time=$2
307            shift
308            ;;
309        --transport)
310            export transport=$2
311            shift
312            ;;
313        --datalen)
314            export datalen_input=$2
315            shift
316            ;;
317        --file-suffix)
318            export file_suffix=$2
319            shift
320            ;;
321        --folder-suffix)
322            export folder_suffix=$2
323            shift
324            ;;
325        --sub-folder)
326            sub_folder=$2
327            shift
328            ;;
329        --executable-suffix)
330            export executable_suffix=$2
331            shift
332            ;;
333        --extra-arguments)
334            export extra_arguments=$2
335            shift
336            ;;
337        --extra-arguments-pub)
338            export extra_arguments_pub=$2
339            shift
340            ;;
341        --extra-arguments-sub)
342            export extra_arguments_sub=$2
343            shift
344            ;;
345        --skip-be)
346            export skip_be_tests="1"
347            ;;
348        --skip-rel)
349            export skip_rel_tests="1"
350            ;;
351        --skip-keyed)
352            export skip_keyed_data="1"
353            ;;
354        --skip-large-data)
355            export skip_large_data="1"
356            ;;
357        --large-data)
358            export large_data="1"
359            ;;
360        --keyed)
361            export skip_unkeyed="1"
362            ;;
363        --unkeyed)
364            export skip_keyed_data="1"
365            ;;
366        --no-batching | --skip-batching)
367            export run_batching_tests="0"
368            export run_no_batching_tests="1"
369            ;;
370        --skip-no-batching)
371            export run_batching_tests="1"
372            export run_no_batching_tests="0"
373            ;;
374        --batch-size)
375            export batch_size=$2
376            shift
377            ;;
378        --reliable)
379            export skip_be_tests="1"
380            ;;
381        --best-effort)
382            export skip_rel_tests="1"
383            ;;
384        --security-gov)
385            export security_only="$2"
386            shift
387            ;;
388        --micro)
389            export micro="1"
390            ;;
391        --cert)
392            export cert="1"
393            ;;
394        --raw | --raw-transport)
395            export raw="1"
396            ;;
397        --tss)
398            export tss="1"
399            ;;
400        --no-colors)
401            export NO_COLORS="1"
402            ;;
403        --language)
404            export LANGUAGE=$2
405            shift
406            ;;
407        --loss-rate)
408            export loss_rate=$2
409            shift
410            ;;
411        --get-netstat-info | --netstat)
412            export get_netstat_info="1"
413            ;;
414        --reduced-data-sizes-set)
415            export REDUCED_DATA_SIZES_SET="1"
416            ;;
417        --dont-change-tuned-settings)
418            export dont_change_tuned_settings="1"
419            ;;
420        --no-taskset)
421            export no_taskset=1
422            ;;
423        *)
424            echo -e "unknown parameter \"$1\""
425            exit 255
426            ;;
427    esac
428    shift
429done
430
431if [[ "$NO_COLORS" == "1" ]]; then
432    disable_colors
433fi
434
435export folder_base="$(dirname "${executable}")"/../../..
436
437if [[ $LANGUAGE == "java"  || "$LANGUAGE" == "cs" ]]; then
438    export folder_base="$(dirname "${executable}")"/../..
439fi
440if [[ $tss == "1" ]]; then
441    export folder_base="$(dirname "${executable}")"/../../../../..
442fi
443
444if [[ "${executable_suffix}" != "" ]]; then
445    export executable="${executable}${executable_suffix}"
446fi
447
448if [[ -n "${folder_suffix}" ]]; then
449    export output_folder="${output_folder}${folder_suffix}"
450fi
451
452if [[ -n "${sub_folder}" ]]; then
453    export output_folder="${output_folder}/${sub_folder}"
454fi
455
456echo -e "${INFO_TAG} Perftest executable is: $executable"
457echo -e "${INFO_TAG} Output folder is: $output_folder"
458
459################################################################################
460
461if [[ "$LANGUAGE" == "python" ]]; then
462    export skip_keyed_data="1"
463    export skip_large_data="1"
464    export skip_be_tests="1"
465    export run_no_batching_tests="0"
466fi
467
468if [[ "${skip_large_data}" == "1" ]]; then
469    export datasizes_extended=${datasizes}
470elif [[ "${large_data}" == "1" ]]; then
471    export datasizes=${datasizes_extended}
472fi
473
474if [[ "${datalen_input}" != "" ]]; then
475    echo -e "${YELLOW}[TEST] Testing only for ${datalen_input}${NC}"
476    export datasizes=${datalen_input}
477    export datasizes_extended=${datalen_input}
478    if [[ "${run_no_batching_tests}" == "1" && "${run_batching_tests}" == "0" ]]; then
479        export skip_large_data="1"
480    fi
481else 
482    if [[ "${REDUCED_DATA_SIZES_SET}" != "" ]]; then
483        echo -e "${YELLOW}[TEST] Testing Reduced set of datasizes ${NC}"
484        export datasizes="32 128 512 2048 8192 32768 63000"
485        export datasizes_extended="${datasizes} 102400 1048576 10485760"
486    fi
487fi
488
489if [[ "$role" != "pub" && "$role" != "sub" ]]; then
490    echo -e "${ERROR_TAG} It must be either publisher or subscriber"
491    exit 255
492fi
493
494if [[ "$lat_thr" != "thr" && "$lat_thr" != "lat" ]]; then
495    echo -e "${ERROR_TAG} It must be either lat or thr"
496    exit 255
497fi
498
499if [[ "${interface}" == "" ]]; then
500    echo "Using default nics"
501    export nic_publisher=${ip_machine_1}
502    export nic_subscriber=${ip_machine_2}
503elif [[ "${interface}" == "both" ]]; then
504    export nic_publisher="enp1s0f0,eno1"
505    export nic_subscriber="enp1s0f0,eno1"
506    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"
507    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"
508else
509    export nic_publisher=$interface
510    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"
511
512    if [[ "${interface2}" == "" ]]; then
513        export nic_subscriber=$interface
514    else
515        export nic_subscriber=$interface2
516    fi
517    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"
518
519    if [[ "${ip1}" != "" ]]; then
520        export ip_publisher=$ip1
521        echo "Using ip_publisher: ${ip_publisher}"
522    fi
523
524    if [[ "${ip2}" != "" ]]; then
525        export ip_subscriber=$ip2
526        echo "Using ip_subscriber: ${ip_subscriber}"
527    fi
528
529fi
530
531export transport_string="-transport $transport"
532
533if [[ "$transport" == "UDPv4" ]]; then
534
535    export transport_string_pub="$transport_string -nic $nic_publisher"
536    export transport_string_sub="$transport_string -nic $nic_subscriber"
537
538    if [[ "$raw" == "1" ]]; then
539        export transport_string_pub="$transport_string_pub -peer ${ip_subscriber}"
540        export transport_string_sub="$transport_string_sub -peer ${ip_publisher}"
541    fi
542
543    if [[ "$micro" == "1" || "$cert" == "1" ]]; then
544        export transport_string_pub="$transport_string_pub -peer _udp://${ip_subscriber}"
545        export transport_string_sub="$transport_string_sub -peer _udp://${ip_publisher}"
546    fi
547
548elif [[ "$transport" == "TCP" ]]; then
549    export transport_string_pub="$transport_string \
550        -nic $nic_publisher \
551        -peer 0@tcpv4_lan://${ip_subscriber}:7400"
552    export transport_string_sub="$transport_string \
553        -nic $nic_subscriber \
554        -peer 0@tcpv4_lan://${ip_publisher}:7400"
555elif [[ "$transport" == "TLS" ]]; then
556    export transport_string_pub="$transport_string \
557        -nic $nic_publisher \
558        -peer tlsv4_lan://${ip_subscriber}:7400"
559    export transport_string_sub="$transport_string \
560        -nic $nic_subscriber \
561        -peer tlsv4_lan://${ip_publisher}:7400"
562elif [[ "$transport" == "UDPv4_WAN" ]]; then
563    export transport_string_pub="$transport_string \
564        -nic $nic_publisher \
565        -transportPublicAddress $ip_publisher:7400"
566    export transport_string_sub="$transport_string \
567        -nic $nic_subscriber \
568        -peer 0@udpv4_wan://${ip_publisher}:7400"
569else
570    export transport_string_pub="$transport_string"
571    export transport_string_sub="$transport_string"
572fi
573
574################################################################################
575
576export pub_string="-pub \
577        ${transport_string_pub} \
578        -noPrintIntervals \
579        -executionTime $exec_time"
580
581if [[ ${lat_thr} == "lat" ]]; then
582    export pub_string="$pub_string \
583        -latencyTest"
584fi
585
586export sub_string="-sub \
587        ${transport_string_sub} \
588        -noPrintIntervals"
589
590if [[ "$role" == "pub" ]]; then
591    echo -e "$INFO_TAG Publisher side running"
592    export commands_string=${pub_string}
593    export extra_arguments="${extra_arguments} ${extra_arguments_pub}"
594else
595    echo -e "$INFO_TAG Subscriber side running"
596    export commands_string=${sub_string}
597    export extra_arguments="${extra_arguments} ${extra_arguments_sub}"
598fi
599
600###############################################################################
601
602if [[ "$dont_change_tuned_settings" != "1" ]]; then
603    echo -e "${INFO_TAG} Executing: /set_${lat_thr}_mode.sh"
604    sudo /set_${lat_thr}_mode.sh
605    sleep 5
606fi
607
608echo -e "${INFO_TAG} Disabling any loss rate"
609sudo tc qdisc add dev $nic_publisher root netem loss 0%
610sudo tc qdisc del dev $nic_publisher root netem loss 0%
611
612if [[ "$role" == "pub" && "${loss_rate}" != "" ]]; then
613    echo -e "${INFO_TAG} Setting loss rate to ${loss_rate}%"
614    sudo tc qdisc add dev $nic_publisher root netem loss $loss_rate%
615fi
616
617cd $folder_base
618echo -e "${INFO_TAG} Folder Base is: $PWD"
619mkdir -p $output_folder
620
621if [[ "${batch_size}" != "" ]]; then
622    if [[ "${lat_thr}" == "thr" ]]; then
623        if [[ "${batch_size}" -eq 0 ]]; then
624            echo -e "${INFO_TAG} Batch size is set to 0"
625            export run_batching_tests="0"
626            export run_no_batching_tests="1"
627        else
628            echo -e "${INFO_TAG} Batch size is set to ${batch_size}"
629            export run_batching_tests="1"
630            export run_no_batching_tests="0"
631        fi
632    else
633        echo -e "${INFO_TAG} Batch size is set to ${batch_size}. This value will be ignored for latency tests."
634        unset batch_size
635    fi
636fi
637
638# Tests that may use batching (when doing throughput tests). Also Latency tests.
639if [[ ${run_batching_tests} == "1" ]]; then
640
641    # UNKEYED
642    if [[ "${skip_unkeyed}" == "" ]]; then
643
644        # RELIABLE
645        if [[ "${skip_rel_tests}" == "" ]]; then
646            execute_test "unkeyed" "rel" "${datasizes_extended}" "${extra_arguments}" "$file_suffix"
647        fi
648
649        # BEST EFFORT
650        if [[ "${skip_be_tests}" == "" ]]; then
651            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
652        fi
653    fi
654
655    # KEYED
656    if [[ "${skip_keyed_data}" == "" ]]; then
657
658        # RELIABLE
659        if [[ "${skip_rel_tests}" == "" ]]; then
660            execute_test "keyed" "rel" "${datasizes}" "${extra_arguments}" "$file_suffix"
661        fi
662
663        # BEST EFFORT
664        if [[ "${skip_be_tests}" == "" ]]; then
665            execute_test "keyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
666        fi
667    fi
668
669fi
670
671# Tests that will not use batching
672if [[ "${lat_thr}" == "thr" && "${run_no_batching_tests}" == "1" ]]; then
673
674    if [[ "$role" == "pub" ]]; then
675        export commands_string="${commands_string} -batchSize 0"
676    fi
677
678    # UNKEYED
679    if [[ "${skip_unkeyed}" == "" ]]; then
680
681        # RELIABLE
682        if [[ "${skip_rel_tests}" == "" ]]; then
683            execute_test "unkeyed" "rel" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
684        fi
685
686        # BEST EFFORT
687        if [[ "${skip_be_tests}" == "" ]]; then
688            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
689        fi
690    fi
691
692    # KEYED
693    if [[ "${skip_keyed_data}" == "" ]]; then
694
695        # RELIABLE
696        if [[ "${skip_rel_tests}" == "" ]]; then
697            execute_test "keyed" "rel" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
698        fi
699
700        # BEST EFFORT
701        if [[ "${skip_be_tests}" == "" ]]; then
702            execute_test "keyed" "be" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
703        fi
704    fi
705
706fi
707
708if [[ "$role" == "pub" && "${loss_rate}" != "" ]]; then
709    echo -e "${INFO_TAG} Disabling loss rate"
710    sudo tc qdisc del dev $nic_publisher root netem loss $loss_rate%
711fi
1#!/bin/bash
2filename=$0
3script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)
4
5echo -e "[Calling base_script/script.sh]"
6"${script_location}/../base_script/script.sh" $@ --transport UDPv4
Test Hardware
The following hardware was used to perform these tests:
Linux Nodes
Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 20.04 -- gcc 9.3.0
Switch
Dell 2048 -- 10Gbps switch (10Gbps and 1Gbps interfaces)
The graph below shows the expected throughput behavior when performing a 1-1 communication between two Linux nodes in a 1Gbps network. The numbers are for best-effort as well as strict reliable reliability scenarios.
Note
By default, RTI Perftest enables batching when performing a Maximum Throughput test. The batching feature allows sending more than one data sample per RTPS packet, improving network performance for small data sizes. See the RTI Connext Core Libraries User’s Manual for more information on batching.
The batch maximum size is set by RTI Perftest to be 8192 bytes; after 8192 bytes, batching is not enabled.
Detailed Statistics
This table contains the raw numbers presented by RTI Perftest. These numbers are the exact output with no further processing.
Best Effort
Sample Size (Bytes)  | 
Total Samples  | 
Avg Samples/s  | 
Avg Mbps  | 
Lost Samples  | 
Lost Samples (%)  | 
|---|---|---|---|---|---|
32  | 
23539456  | 
1175914  | 
301.0  | 
7168  | 
0.03  | 
64  | 
23404288  | 
1168912  | 
598.5  | 
13440  | 
0.06  | 
128  | 
22312320  | 
1113968  | 
1140.7  | 
16064  | 
0.07  | 
256  | 
20279264  | 
1012938  | 
2074.5  | 
17408  | 
0.09  | 
512  | 
17852416  | 
891271  | 
3650.6  | 
16256  | 
0.09  | 
1024  | 
13748016  | 
686826  | 
5626.5  | 
10896  | 
0.08  | 
2048  | 
9618608  | 
480411  | 
7871.1  | 
1904  | 
0.02  | 
4096  | 
5938548  | 
296547  | 
9717.3  | 
0  | 
0.00  | 
8192  | 
2998234  | 
149843  | 
9820.2  | 
0  | 
0.00  | 
16384  | 
1506933  | 
75337  | 
9874.7  | 
27  | 
0.00  | 
32768  | 
755517  | 
37773  | 
9902.1  | 
10  | 
0.00  | 
63000  | 
393377  | 
19667  | 
9912.6  | 
4  | 
0.00  | 
Reliable
Sample Size (Bytes)  | 
Total Samples  | 
Avg Samples/s  | 
Avg Mbps  | 
Lost Samples  | 
Lost Samples (%)  | 
|---|---|---|---|---|---|
32  | 
21613056  | 
1079070  | 
276.2  | 
0  | 
0.00  | 
64  | 
21518464  | 
1074347  | 
550.1  | 
0  | 
0.00  | 
128  | 
21417408  | 
1069769  | 
1095.4  | 
0  | 
0.00  | 
256  | 
18183683  | 
908275  | 
1860.1  | 
0  | 
0.00  | 
512  | 
15399196  | 
768920  | 
3149.5  | 
0  | 
0.00  | 
1024  | 
11533848  | 
575914  | 
4717.9  | 
0  | 
0.00  | 
2048  | 
7671125  | 
383035  | 
6275.7  | 
0  | 
0.00  | 
4096  | 
4727580  | 
236054  | 
7735.0  | 
0  | 
0.00  | 
8192  | 
2956248  | 
147709  | 
9680.3  | 
0  | 
0.00  | 
16384  | 
1506771  | 
75322  | 
9872.7  | 
0  | 
0.00  | 
32768  | 
755507  | 
37769  | 
9901.2  | 
0  | 
0.00  | 
63000  | 
393393  | 
19666  | 
9912.0  | 
0  | 
0.00  | 
Perftest Scripts
To produce these tests, we executed RTI Perftest for C++11. The exact script used can be found here:
  1#!/bin/bash
  2filename=$0
  3script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)
  4
  5export datasizes="32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000"
  6export datasizes_extended="${datasizes} 100000 500000 1048576 1548576 4194304 10485760"
  7
  8export domain="2"
  9export exec_time=20
 10export num_reps=1
 11export instance_number=100000
 12export core=0
 13export run_batching_tests="1"
 14export run_no_batching_tests="1"
 15
 16# We will use some colors to improve visibility of errors and info messages.
 17RED='\033[0;31m'
 18GREEN='\033[0;32m'
 19YELLOW='\033[0;33m'
 20BLUE='\033[0;34m'
 21LIGHTBLUE='\033[0;36m'
 22NC='\033[0m'
 23INFO_TAG="${GREEN}[INFO][$role]:${NC}"
 24WARNING_TAG="${YELLOW}[WARNING][$role]:${NC}"
 25ERROR_TAG="${RED}[ERROR][$role]:${NC}"
 26
 27################################################################################
 28
 29function disable_colors() {
 30    export RED=""
 31    export GREEN=""
 32    export YELLOW=""
 33    export NC=""
 34    export BLUE=""
 35    export LIGHTBLUE=""
 36    export INFO_TAG="${GREEN}[INFO][$role]:${NC}"
 37    export WARNING_TAG="${YELLOW}[WARNING][$role]:${NC}"
 38    export ERROR_TAG="${RED}[ERROR][$role]:${NC}"
 39}
 40
 41function change_domain() {
 42    if [[ "$domain" == "1" ]]; then
 43        export domain="2"
 44    else
 45        export domain="1"
 46    fi
 47}
 48
 49# Function to process and append lines to the output file
 50function append_to_output_file() {
 51    local line=${1?line required}; readonly line
 52    local append=$(echo "${2?append required}" | tr -s ' '); readonly append
 53    local file=${3?file required}; readonly file
 54    if [[ "${line: -1}" == $'\n' ]]; then
 55        line="${line::-1}"
 56    fi
 57    echo "${line}${append}" >> "$file"
 58}
 59
 60# Usage: execute_test <keyed/unkeyed> <rel/be> <datasizes> <batchSize>
 61function execute_test() {
 62
 63    local keyed_unkeyed=$1
 64    local rel_be=$2
 65    local datasizes_test=$3
 66    local other_args=$4
 67    local name_suffix=$5
 68
 69    local commands_string_test=$commands_string
 70    local tag=""
 71
 72    if [[ "${keyed_unkeyed}" == "keyed" ]]; then
 73        commands_string_test="${commands_string_test} -keyed -instances $instance_number"
 74        tag="[${YELLOW}${transport}${NC}|${BLUE}K${NC}|"
 75    else
 76        tag="[${YELLOW}${transport}${NC}|${LIGHTBLUE}UK${NC}|"
 77    fi
 78
 79    if [[ "${rel_be}" == "be" ]]; then
 80        commands_string_test="${commands_string_test} -bestEffort"
 81        tag="${tag}${YELLOW}BE${NC}]"
 82    else
 83        tag="${tag}${RED}REL${NC}]"
 84    fi
 85
 86    # If batch_size is set, we will add it to the command line of the publisher side.
 87    if [[ "$batch_size" != "" && "$role" == "pub" ]]; then
 88        commands_string_test="${commands_string_test} -batchSize $batch_size"
 89    fi
 90
 91    tag="${tag}[${LIGHTBLUE}${lat_thr}${NC}]"
 92
 93    local output_file=$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}.csv
 94
 95    if [[ "$role" == "pub" ]]; then
 96        echo -e "${YELLOW}[TEST]: $keyed_unkeyed, $rel_be, Is a no-batching test = $no_batching_tests. ${NC}"
 97    fi
 98
 99    if [[ "$thread_cpu_affinity" != "" ]]; then
100        commands_string_test="${commands_string_test} -threadCPUAffinity $thread_cpu_affinity"
101    fi
102
103    local pre_command_string=""
104
105    if [[ "$thread_priorities" != "" ]]; then
106        echo -e "${WARNING_TAG} Thread priorities enabled, this requires using sudo"
107        # When using sudo, the LD_LIBRARY_PATH is emptied, hence we need to set it again in the command itself
108        export LD_LIBRARY_PATH_COPIED="$LD_LIBRARY_PATH"
109        pre_command_string="sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH_COPIED "
110        commands_string_test="${commands_string_test} -threadPriorities $thread_priorities"
111    fi
112
113    if [[ "$pin_memory" != "" ]]; then
114        commands_string_test="${commands_string_test} -pinMemory"
115    fi
116
117    if [[ "$no_taskset" == "" && "$thread_cpu_affinity" == "" && "$LANGUAGE" != "java" && "$LANGUAGE" != "cs" ]]; then
118        pre_command_string="$pre_command_string taskset -c $core"
119    fi
120
121    if [[ "$LANGUAGE" == "python" ]]; then
122        pre_command_string="$pre_command_string python3 "
123    fi
124
125    if [[ "$DOCKER" == "1" ]]; then
126        pre_command_string="$pre_command_string docker run --net=host -v /home/perfuser/rti_license_connextpro.dat:/opt/rti.com/rti_connext_dds-7.3.0/rti_license.dat rticom/perftest:7.3.0-EAR "
127        executable=""
128    fi
129
130    # Get the aprox time this will take:
131    total_tests=$((`wc -w <<< "$datasizes_test"` * num_reps))
132    total_time=$((total_tests * exec_time))
133
134    touch $output_file
135    local no_headers=""
136    local current_test=0
137    for index in $(seq 1 ${num_reps}); do
138        for DATALEN in ${datasizes_test}; do
139            current_test=$((current_test + 1))
140
141            if [[ ! -s $output_file ]]; then
142                echo -e "${INFO_TAG} Output file is empty, filling the header."
143                no_headers=""
144            else
145                echo -e "${INFO_TAG} Output file is not empty."
146                no_headers=" -noOutputHeaders"
147            fi
148
149            export command="$pre_command_string $executable -domain $domain -dataLen $DATALEN $commands_string_test $other_args $no_headers"
150            if [[ "$role" == "pub" ]]; then
151                echo -e "Test ${tag} (${current_test}/${total_tests}) -- Total time = ${total_time}s"
152                echo -e ${BLUE}$command${NC}
153            else
154                echo -e ${LIGHTBLUE}$command${NC}
155            fi
156
157            # In certain cases we need to wait a bit before running the test, this is
158            # because the previous test might not be finished on the other side yet, or because the
159            # discovery mechanism does not work like in connext DDS.
160            if [[ "$LANGUAGE" == "cs" && "$role" == "pub" ]]; then
161                sleep 3
162            fi
163            if [[ "$raw" == "1" && "$role" == "sub" ]]; then
164                sleep 5
165            fi
166
167            # Gather netstat info before running the test
168            if [[ "${get_netstat_info}" == "1" ]]; then
169                echo -e "${INFO_TAG} Getting netstat info before"
170                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt
171            fi
172
173            # Execute the command and capture the output
174            if [[ $LANGUAGE == "c++98" || $LANGUAGE == "" ]]; then
175                echo -e "${INFO_TAG} Using C++98, using -outputFile option"
176                touch ${output_file}_tmp_line
177                chmod 777 ${output_file}_tmp_line
178                eval $command -outputFile ${output_file}_tmp_line
179            else
180                eval $command > ${output_file}_tmp_line
181            fi
182
183            number_of_lines=$(wc -l < ${output_file}_tmp_line)
184            echo -e "${INFO_TAG} Size of the output text: $number_of_lines lines"
185
186            # Check if this is the first test for this output file.
187            local is_first_test=0
188            if [[ ! -s $output_file ]]; then
189                is_first_test=1
190            fi
191            local expected_lines=$((is_first_test+1))
192
193            if [[ $number_of_lines -ne ${expected_lines} ]]; then
194                echo -e "${WARNING_TAG} The output text should have ${expected_lines} lines, but it has ${number_of_lines}."
195                echo -e "${RED} The content is: \n\"\"\""
196                cat ${output_file}_tmp_line
197                echo -e "\"\"\""
198                echo -e "Not adding it to the output file. ${NC}"
199            else
200                if [[ $number_of_lines -gt 1 ]]; then
201                    header_line=$(head -n 1 "${output_file}_tmp_line")
202                    echo -e "${INFO_TAG} Header line is: $header_line"
203                fi
204                result_line=$(tail -n 1 "${output_file}_tmp_line")
205                echo -e "${INFO_TAG} Result line is: $result_line"
206
207                # If this is the first test, we need to add the header line to the output file
208                if [[ $is_first_test -eq 1 ]]; then
209                    append_to_output_file "$header_line" ", command-line" $output_file
210                fi
211
212                command=$(echo $command | sed 's/,/:comma:/g')
213                # Remove the `"` characters from the command line
214                command=$(echo $command | sed 's/\"//g')
215
216                # Append the result line to the output file
217                append_to_output_file "$result_line" ", $command" $output_file
218            fi
219
220            # Always remove the temporary file
221            rm -rf ${output_file}_tmp_line
222
223            # Gather netstat info after running the test
224            if [[ "${get_netstat_info}" == "1" ]]; then
225                echo -e "${INFO_TAG} Getting netstat info after"
226                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt
227                touch "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
228                python3 $script_location/../../../tools/diff_netstat_output.py \
229                    -n $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt \
230                    -o $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt \
231                    -d $DATALEN $no_header_netstat \
232                    -csv >> "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
233                rm -rf $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_*.txt
234                no_header_netstat=" -nh"
235            fi
236
237            change_domain
238        done
239    done
240}
241
242################################################################################
243# PARSE COMMAND LINE OPTIONS:
244
245while [ "$1" != "" ]; do
246    case $1 in
247        --executable)
248            executable=$2
249            shift
250            ;;
251        --docker)
252            DOCKER="1"
253            ;;
254        --output-folder)
255            output_folder=$2
256            shift
257            ;;
258        --role)
259            export role=$2
260            shift
261            ;;
262        --core)
263            export core=$2
264            shift
265            ;;
266        --assign-core-list | --thread-cpu-affinity)
267            export thread_cpu_affinity=$2
268            shift
269            ;;
270        --assign-thread-priorities | --thread-priorities)
271            export thread_priorities=$2
272            shift
273            ;;
274        --pin-memory)
275            export pin_memory="1"
276            ;;
277        --test-kind)
278            export lat_thr=$2
279            shift
280            ;;
281        --interface1)
282            export interface=$2
283            shift
284            ;;
285        --interface2)
286            export interface2=$2
287            shift
288            ;;
289        --ip1)
290            export ip1=$2
291            shift
292            ;;
293        --ip2)
294            export ip2=$2
295            shift
296            ;;
297        --repetitions)
298            export num_reps=$2
299            shift
300            ;;
301        --domain)
302            export domain=$2
303            shift
304            ;;
305        --execution-time)
306            export exec_time=$2
307            shift
308            ;;
309        --transport)
310            export transport=$2
311            shift
312            ;;
313        --datalen)
314            export datalen_input=$2
315            shift
316            ;;
317        --file-suffix)
318            export file_suffix=$2
319            shift
320            ;;
321        --folder-suffix)
322            export folder_suffix=$2
323            shift
324            ;;
325        --sub-folder)
326            sub_folder=$2
327            shift
328            ;;
329        --executable-suffix)
330            export executable_suffix=$2
331            shift
332            ;;
333        --extra-arguments)
334            export extra_arguments=$2
335            shift
336            ;;
337        --extra-arguments-pub)
338            export extra_arguments_pub=$2
339            shift
340            ;;
341        --extra-arguments-sub)
342            export extra_arguments_sub=$2
343            shift
344            ;;
345        --skip-be)
346            export skip_be_tests="1"
347            ;;
348        --skip-rel)
349            export skip_rel_tests="1"
350            ;;
351        --skip-keyed)
352            export skip_keyed_data="1"
353            ;;
354        --skip-large-data)
355            export skip_large_data="1"
356            ;;
357        --large-data)
358            export large_data="1"
359            ;;
360        --keyed)
361            export skip_unkeyed="1"
362            ;;
363        --unkeyed)
364            export skip_keyed_data="1"
365            ;;
366        --no-batching | --skip-batching)
367            export run_batching_tests="0"
368            export run_no_batching_tests="1"
369            ;;
370        --skip-no-batching)
371            export run_batching_tests="1"
372            export run_no_batching_tests="0"
373            ;;
374        --batch-size)
375            export batch_size=$2
376            shift
377            ;;
378        --reliable)
379            export skip_be_tests="1"
380            ;;
381        --best-effort)
382            export skip_rel_tests="1"
383            ;;
384        --security-gov)
385            export security_only="$2"
386            shift
387            ;;
388        --micro)
389            export micro="1"
390            ;;
391        --cert)
392            export cert="1"
393            ;;
394        --raw | --raw-transport)
395            export raw="1"
396            ;;
397        --tss)
398            export tss="1"
399            ;;
400        --no-colors)
401            export NO_COLORS="1"
402            ;;
403        --language)
404            export LANGUAGE=$2
405            shift
406            ;;
407        --loss-rate)
408            export loss_rate=$2
409            shift
410            ;;
411        --get-netstat-info | --netstat)
412            export get_netstat_info="1"
413            ;;
414        --reduced-data-sizes-set)
415            export REDUCED_DATA_SIZES_SET="1"
416            ;;
417        --dont-change-tuned-settings)
418            export dont_change_tuned_settings="1"
419            ;;
420        --no-taskset)
421            export no_taskset=1
422            ;;
423        *)
424            echo -e "unknown parameter \"$1\""
425            exit 255
426            ;;
427    esac
428    shift
429done
430
431if [[ "$NO_COLORS" == "1" ]]; then
432    disable_colors
433fi
434
435export folder_base="$(dirname "${executable}")"/../../..
436
437if [[ $LANGUAGE == "java"  || "$LANGUAGE" == "cs" ]]; then
438    export folder_base="$(dirname "${executable}")"/../..
439fi
440if [[ $tss == "1" ]]; then
441    export folder_base="$(dirname "${executable}")"/../../../../..
442fi
443
444if [[ "${executable_suffix}" != "" ]]; then
445    export executable="${executable}${executable_suffix}"
446fi
447
448if [[ -n "${folder_suffix}" ]]; then
449    export output_folder="${output_folder}${folder_suffix}"
450fi
451
452if [[ -n "${sub_folder}" ]]; then
453    export output_folder="${output_folder}/${sub_folder}"
454fi
455
456echo -e "${INFO_TAG} Perftest executable is: $executable"
457echo -e "${INFO_TAG} Output folder is: $output_folder"
458
459################################################################################
460
461if [[ "$LANGUAGE" == "python" ]]; then
462    export skip_keyed_data="1"
463    export skip_large_data="1"
464    export skip_be_tests="1"
465    export run_no_batching_tests="0"
466fi
467
468if [[ "${skip_large_data}" == "1" ]]; then
469    export datasizes_extended=${datasizes}
470elif [[ "${large_data}" == "1" ]]; then
471    export datasizes=${datasizes_extended}
472fi
473
474if [[ "${datalen_input}" != "" ]]; then
475    echo -e "${YELLOW}[TEST] Testing only for ${datalen_input}${NC}"
476    export datasizes=${datalen_input}
477    export datasizes_extended=${datalen_input}
478    if [[ "${run_no_batching_tests}" == "1" && "${run_batching_tests}" == "0" ]]; then
479        export skip_large_data="1"
480    fi
481else 
482    if [[ "${REDUCED_DATA_SIZES_SET}" != "" ]]; then
483        echo -e "${YELLOW}[TEST] Testing Reduced set of datasizes ${NC}"
484        export datasizes="32 128 512 2048 8192 32768 63000"
485        export datasizes_extended="${datasizes} 102400 1048576 10485760"
486    fi
487fi
488
489if [[ "$role" != "pub" && "$role" != "sub" ]]; then
490    echo -e "${ERROR_TAG} It must be either publisher or subscriber"
491    exit 255
492fi
493
494if [[ "$lat_thr" != "thr" && "$lat_thr" != "lat" ]]; then
495    echo -e "${ERROR_TAG} It must be either lat or thr"
496    exit 255
497fi
498
499if [[ "${interface}" == "" ]]; then
500    echo "Using default nics"
501    export nic_publisher=${ip_machine_1}
502    export nic_subscriber=${ip_machine_2}
503elif [[ "${interface}" == "both" ]]; then
504    export nic_publisher="enp1s0f0,eno1"
505    export nic_subscriber="enp1s0f0,eno1"
506    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"
507    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"
508else
509    export nic_publisher=$interface
510    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"
511
512    if [[ "${interface2}" == "" ]]; then
513        export nic_subscriber=$interface
514    else
515        export nic_subscriber=$interface2
516    fi
517    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"
518
519    if [[ "${ip1}" != "" ]]; then
520        export ip_publisher=$ip1
521        echo "Using ip_publisher: ${ip_publisher}"
522    fi
523
524    if [[ "${ip2}" != "" ]]; then
525        export ip_subscriber=$ip2
526        echo "Using ip_subscriber: ${ip_subscriber}"
527    fi
528
529fi
530
531export transport_string="-transport $transport"
532
533if [[ "$transport" == "UDPv4" ]]; then
534
535    export transport_string_pub="$transport_string -nic $nic_publisher"
536    export transport_string_sub="$transport_string -nic $nic_subscriber"
537
538    if [[ "$raw" == "1" ]]; then
539        export transport_string_pub="$transport_string_pub -peer ${ip_subscriber}"
540        export transport_string_sub="$transport_string_sub -peer ${ip_publisher}"
541    fi
542
543    if [[ "$micro" == "1" || "$cert" == "1" ]]; then
544        export transport_string_pub="$transport_string_pub -peer _udp://${ip_subscriber}"
545        export transport_string_sub="$transport_string_sub -peer _udp://${ip_publisher}"
546    fi
547
548elif [[ "$transport" == "TCP" ]]; then
549    export transport_string_pub="$transport_string \
550        -nic $nic_publisher \
551        -peer 0@tcpv4_lan://${ip_subscriber}:7400"
552    export transport_string_sub="$transport_string \
553        -nic $nic_subscriber \
554        -peer 0@tcpv4_lan://${ip_publisher}:7400"
555elif [[ "$transport" == "TLS" ]]; then
556    export transport_string_pub="$transport_string \
557        -nic $nic_publisher \
558        -peer tlsv4_lan://${ip_subscriber}:7400"
559    export transport_string_sub="$transport_string \
560        -nic $nic_subscriber \
561        -peer tlsv4_lan://${ip_publisher}:7400"
562elif [[ "$transport" == "UDPv4_WAN" ]]; then
563    export transport_string_pub="$transport_string \
564        -nic $nic_publisher \
565        -transportPublicAddress $ip_publisher:7400"
566    export transport_string_sub="$transport_string \
567        -nic $nic_subscriber \
568        -peer 0@udpv4_wan://${ip_publisher}:7400"
569else
570    export transport_string_pub="$transport_string"
571    export transport_string_sub="$transport_string"
572fi
573
574################################################################################
575
576export pub_string="-pub \
577        ${transport_string_pub} \
578        -noPrintIntervals \
579        -executionTime $exec_time"
580
581if [[ ${lat_thr} == "lat" ]]; then
582    export pub_string="$pub_string \
583        -latencyTest"
584fi
585
586export sub_string="-sub \
587        ${transport_string_sub} \
588        -noPrintIntervals"
589
590if [[ "$role" == "pub" ]]; then
591    echo -e "$INFO_TAG Publisher side running"
592    export commands_string=${pub_string}
593    export extra_arguments="${extra_arguments} ${extra_arguments_pub}"
594else
595    echo -e "$INFO_TAG Subscriber side running"
596    export commands_string=${sub_string}
597    export extra_arguments="${extra_arguments} ${extra_arguments_sub}"
598fi
599
600###############################################################################
601
602if [[ "$dont_change_tuned_settings" != "1" ]]; then
603    echo -e "${INFO_TAG} Executing: /set_${lat_thr}_mode.sh"
604    sudo /set_${lat_thr}_mode.sh
605    sleep 5
606fi
607
608echo -e "${INFO_TAG} Disabling any loss rate"
609sudo tc qdisc add dev $nic_publisher root netem loss 0%
610sudo tc qdisc del dev $nic_publisher root netem loss 0%
611
612if [[ "$role" == "pub" && "${loss_rate}" != "" ]]; then
613    echo -e "${INFO_TAG} Setting loss rate to ${loss_rate}%"
614    sudo tc qdisc add dev $nic_publisher root netem loss $loss_rate%
615fi
616
617cd $folder_base
618echo -e "${INFO_TAG} Folder Base is: $PWD"
619mkdir -p $output_folder
620
621if [[ "${batch_size}" != "" ]]; then
622    if [[ "${lat_thr}" == "thr" ]]; then
623        if [[ "${batch_size}" -eq 0 ]]; then
624            echo -e "${INFO_TAG} Batch size is set to 0"
625            export run_batching_tests="0"
626            export run_no_batching_tests="1"
627        else
628            echo -e "${INFO_TAG} Batch size is set to ${batch_size}"
629            export run_batching_tests="1"
630            export run_no_batching_tests="0"
631        fi
632    else
633        echo -e "${INFO_TAG} Batch size is set to ${batch_size}. This value will be ignored for latency tests."
634        unset batch_size
635    fi
636fi
637
638# Tests that may use batching (when doing throughput tests). Also Latency tests.
639if [[ ${run_batching_tests} == "1" ]]; then
640
641    # UNKEYED
642    if [[ "${skip_unkeyed}" == "" ]]; then
643
644        # RELIABLE
645        if [[ "${skip_rel_tests}" == "" ]]; then
646            execute_test "unkeyed" "rel" "${datasizes_extended}" "${extra_arguments}" "$file_suffix"
647        fi
648
649        # BEST EFFORT
650        if [[ "${skip_be_tests}" == "" ]]; then
651            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
652        fi
653    fi
654
655    # KEYED
656    if [[ "${skip_keyed_data}" == "" ]]; then
657
658        # RELIABLE
659        if [[ "${skip_rel_tests}" == "" ]]; then
660            execute_test "keyed" "rel" "${datasizes}" "${extra_arguments}" "$file_suffix"
661        fi
662
663        # BEST EFFORT
664        if [[ "${skip_be_tests}" == "" ]]; then
665            execute_test "keyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
666        fi
667    fi
668
669fi
670
671# Tests that will not use batching
672if [[ "${lat_thr}" == "thr" && "${run_no_batching_tests}" == "1" ]]; then
673
674    if [[ "$role" == "pub" ]]; then
675        export commands_string="${commands_string} -batchSize 0"
676    fi
677
678    # UNKEYED
679    if [[ "${skip_unkeyed}" == "" ]]; then
680
681        # RELIABLE
682        if [[ "${skip_rel_tests}" == "" ]]; then
683            execute_test "unkeyed" "rel" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
684        fi
685
686        # BEST EFFORT
687        if [[ "${skip_be_tests}" == "" ]]; then
688            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
689        fi
690    fi
691
692    # KEYED
693    if [[ "${skip_keyed_data}" == "" ]]; then
694
695        # RELIABLE
696        if [[ "${skip_rel_tests}" == "" ]]; then
697            execute_test "keyed" "rel" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
698        fi
699
700        # BEST EFFORT
701        if [[ "${skip_be_tests}" == "" ]]; then
702            execute_test "keyed" "be" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
703        fi
704    fi
705
706fi
707
708if [[ "$role" == "pub" && "${loss_rate}" != "" ]]; then
709    echo -e "${INFO_TAG} Disabling loss rate"
710    sudo tc qdisc del dev $nic_publisher root netem loss $loss_rate%
711fi
1#!/bin/bash
2filename=$0
3script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)
4
5echo -e "[Calling base_script/script.sh]"
6"${script_location}/../base_script/script.sh" $@ --transport UDPv4
Test Hardware
The following hardware was used to perform these tests:
Linux Nodes
Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 20.04 -- gcc 9.3.0
Switch
Dell 2048 -- 10Gbps switch (10Gbps and 1Gbps interfaces)