Java API

The following tests have been performed by executing an RTI Perftest Java Publisher and Subscriber between two nodes, connected to a switch via Ethernet. The communication has been restricted to a single interface and the transport has been set to UDPv4.

Find information about the hardware, network, and command-line parameters after each of the tests.

Unkeyed, UDPv4 10Gbps Network, Java

The graph below shows the one-way latency without load between a Publisher and a Subscriber running in two Linux nodes in a 10Gbps network. The numbers are for best-effort as well as strict reliable reliability scenarios.

Note

We use the median (50th percentile) instead of the average in order to get a more stable measurement that does not account for spurious outliers. We also calculate the average value and other percentile values, which can be seen in the Detailed Statistics section below.

Detailed Statistics

The following tables contain the raw numbers presented by RTI Perftest. These numbers are the exact output with no further processing.

  • Best Effort

Sample Size (Bytes)

Ave (us)

Std (us)

Min (us)

Max (us)

50% (μs)

90% (us)

99% (us)

99.99% (us)

99.9999% (us)

32

19

5.7

18

2432

19

19

22

18

18

64

19

5.7

18

2420

19

20

23

18

18

128

19

5.6

18

2371

19

20

23

18

18

256

20

5.7

19

2383

20

20

24

19

19

512

20

5.7

19

2262

20

21

23

19

19

1024

26

7.0

20

2499

27

28

29

20

20

2048

49

284.8

23

100085

48

62

70

23

23

4096

32

131.7

27

100100

30

35

63

27

27

8192

46

11.2

38

2368

42

59

62

38

38

16384

91

24.7

49

2362

90

127

147

49

49

32768

179

13.9

101

2448

179

181

187

101

101

63000

193

14.7

137

2462

192

195

200

137

137

  • Reliable

Sample Size (Bytes)

Ave (us)

Std (us)

Min (us)

Max (us)

50% (μs)

90% (us)

99% (us)

99.99% (us)

99.9999% (us)

32

20

5.7

19

2244

20

21

24

19

19

64

21

5.8

19

2324

21

21

24

19

19

128

21

5.8

20

2170

21

22

25

20

20

256

22

9.9

20

5050

21

22

28

20

20

512

22

9.8

20

5052

22

24

27

20

20

1024

26

14.1

21

5055

26

29

46

21

21

2048

30

38.7

24

4999

30

33

52

24

24

4096

33

40.3

28

4986

31

34

63

28

28

8192

47

14.8

40

3889

43

60

85

40

40

16384

95

29.4

50

2407

93

132

147

50

50

32768

178

16.4

78

2586

181

183

189

78

78

63000

195

13.5

142

2286

195

197

203

142

142

Perftest Scripts

To produce these tests, we executed RTI Perftest for Java. The exact script used can be found here:

  1#!/bin/bash
  2filename=$0
  3script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)
  4
  5export datasizes="32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000"
  6export datasizes_extended="${datasizes} 100000 500000 1048576 1548576 4194304 10485760"
  7
  8export domain="2"
  9export exec_time=20
 10export num_reps=1
 11export instance_number=100000
 12export core=0
 13export run_batching_tests="1"
 14export run_no_batching_tests="1"
 15export nic_loss_rate=0
 16export nic_delay_ms=0
 17
 18# We will use some colors to improve visibility of errors and info messages.
 19RED='\033[0;31m'
 20GREEN='\033[0;32m'
 21YELLOW='\033[0;33m'
 22BLUE='\033[0;34m'
 23LIGHTBLUE='\033[0;36m'
 24NC='\033[0m'
 25INFO_TAG="${GREEN}[INFO][$role]:${NC}"
 26WARNING_TAG="${YELLOW}[WARNING][$role]:${NC}"
 27ERROR_TAG="${RED}[ERROR][$role]:${NC}"
 28
 29################################################################################
 30
 31function disable_colors() {
 32    export RED=""
 33    export GREEN=""
 34    export YELLOW=""
 35    export NC=""
 36    export BLUE=""
 37    export LIGHTBLUE=""
 38    export INFO_TAG="${GREEN}[INFO][$role]:${NC}"
 39    export WARNING_TAG="${YELLOW}[WARNING][$role]:${NC}"
 40    export ERROR_TAG="${RED}[ERROR][$role]:${NC}"
 41}
 42
 43function change_domain() {
 44    if [[ "$domain" == "1" ]]; then
 45        export domain="2"
 46    else
 47        export domain="1"
 48    fi
 49}
 50
 51# Function to process and append lines to the output file
 52function append_to_output_file() {
 53    local line=${1?line required}; readonly line
 54    local append=$(echo "${2?append required}" | tr -s ' '); readonly append
 55    local file=${3?file required}; readonly file
 56    if [[ "${line: -1}" == $'\n' ]]; then
 57        line="${line::-1}"
 58    fi
 59    echo "${line}${append}" >> "$file"
 60}
 61
 62# Usage: execute_test <keyed/unkeyed> <rel/be> <datasizes> <batchSize>
 63function execute_test() {
 64
 65    local keyed_unkeyed=$1
 66    local rel_be=$2
 67    local datasizes_test=$3
 68    local other_args=$4
 69    local name_suffix=$5
 70
 71    local commands_string_test=$commands_string
 72    local tag=""
 73
 74    if [[ "${keyed_unkeyed}" == "keyed" ]]; then
 75        commands_string_test="${commands_string_test} -keyed -instances $instance_number"
 76        tag="[${YELLOW}${transport}${NC}|${BLUE}K${NC}|"
 77    else
 78        tag="[${YELLOW}${transport}${NC}|${LIGHTBLUE}UK${NC}|"
 79    fi
 80
 81    if [[ "${rel_be}" == "be" ]]; then
 82        commands_string_test="${commands_string_test} -bestEffort"
 83        tag="${tag}${YELLOW}BE${NC}]"
 84    else
 85        tag="${tag}${RED}REL${NC}]"
 86    fi
 87
 88    # If batch_size is set to a non-zero value, we will add it to the command line of the publisher side.
 89    # When batch_size is 0, it is already handled by the no-batching block which appends -batchSize 0 to commands_string.
 90    if [[ "$batch_size" != "" && "$batch_size" != "0" && "$role" == "pub" ]]; then
 91        commands_string_test="${commands_string_test} -batchSize $batch_size"
 92    fi
 93
 94    tag="${tag}[${LIGHTBLUE}${lat_thr}${NC}]"
 95
 96    local output_file=$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}.csv
 97
 98    if [[ "$role" == "pub" ]]; then
 99        echo -e "${YELLOW}[TEST]: $keyed_unkeyed, $rel_be, Is a no-batching test = $no_batching_tests. ${NC}"
100    fi
101
102    if [[ "$thread_cpu_affinity" != "" && "$LANGUAGE" != "java" && "$LANGUAGE" != "cs" ]]; then
103        commands_string_test="${commands_string_test} -threadCPUAffinity $thread_cpu_affinity"
104    fi
105
106    local pre_command_string=""
107
108    if [[ "$thread_priorities" != "" && "$LANGUAGE" != "java" && "$LANGUAGE" != "cs" ]]; then
109        echo -e "${WARNING_TAG} Thread priorities enabled, this requires using sudo"
110        # When using sudo, the LD_LIBRARY_PATH is emptied, hence we need to set it again in the command itself
111        export LD_LIBRARY_PATH_COPIED="$LD_LIBRARY_PATH"
112        pre_command_string="sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH_COPIED "
113        commands_string_test="${commands_string_test} -threadPriorities $thread_priorities"
114    fi
115
116    if [[ "$pin_memory" != "" && "$LANGUAGE" != "java" && "$LANGUAGE" != "cs" ]]; then
117        commands_string_test="${commands_string_test} -pinMemory"
118    fi
119
120    if [[ "$no_taskset" == "" && "$thread_cpu_affinity" == "" && "$LANGUAGE" != "java" && "$LANGUAGE" != "cs" ]]; then
121        pre_command_string="$pre_command_string taskset -c $core"
122    fi
123
124    if [[ "$LANGUAGE" == "python" ]]; then
125        pre_command_string="$pre_command_string python3 "
126    fi
127
128    if [[ "$DOCKER" == "1" ]]; then
129        pre_command_string="$pre_command_string docker run --net=host -v /home/perfuser/rti_license_connextpro.dat:/opt/rti.com/rti_connext_dds-7.3.0/rti_license.dat rticom/perftest:7.3.0-EAR "
130        executable=""
131    fi
132
133    # Get the aprox time this will take:
134    total_tests=$((`wc -w <<< "$datasizes_test"` * num_reps))
135    total_time=$((total_tests * exec_time))
136
137    touch $output_file
138    local no_headers=""
139    local current_test=0
140    for index in $(seq 1 ${num_reps}); do
141        for DATALEN in ${datasizes_test}; do
142            current_test=$((current_test + 1))
143
144            if [[ ! -s $output_file ]]; then
145                echo -e "${INFO_TAG} Output file is empty, filling the header."
146                no_headers=""
147            else
148                echo -e "${INFO_TAG} Output file is not empty."
149                no_headers=" -noOutputHeaders"
150            fi
151
152            export command="$pre_command_string $executable -domain $domain -dataLen $DATALEN $commands_string_test $other_args $no_headers"
153            if [[ "$role" == "pub" ]]; then
154                echo -e "Test ${tag} (${current_test}/${total_tests}) -- Total time = ${total_time}s"
155                echo -e ${BLUE}$command${NC}
156            else
157                echo -e ${LIGHTBLUE}$command${NC}
158            fi
159
160            # In certain cases we need to wait a bit before running the test, this is
161            # because the previous test might not be finished on the other side yet, or because the
162            # discovery mechanism does not work like in connext DDS.
163            if [[ "$LANGUAGE" == "cs" && "$role" == "pub" ]]; then
164                sleep 3
165            fi
166            if [[ "$raw" == "1" && "$role" == "sub" ]]; then
167                sleep 5
168            fi
169
170            # Gather netstat info before running the test
171            if [[ "${get_netstat_info}" == "1" ]]; then
172                echo -e "${INFO_TAG} Getting netstat info before"
173                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt
174            fi
175
176            # Execute the command and capture the output
177            if [[ $LANGUAGE == "c++98" || $LANGUAGE == "" ]]; then
178                echo -e "${INFO_TAG} Using C++98, using -outputFile option"
179                touch ${output_file}_tmp_line
180                chmod 777 ${output_file}_tmp_line
181                eval $command -outputFile ${output_file}_tmp_line
182            else
183                eval $command > ${output_file}_tmp_line
184            fi
185
186            number_of_lines=$(wc -l < ${output_file}_tmp_line)
187            echo -e "${INFO_TAG} Size of the output text: $number_of_lines lines"
188
189            # Check if this is the first test for this output file.
190            local is_first_test=0
191            if [[ ! -s $output_file ]]; then
192                is_first_test=1
193            fi
194            local expected_lines=$((is_first_test+1))
195
196            if [[ $number_of_lines -ne ${expected_lines} ]]; then
197                echo -e "${WARNING_TAG} The output text should have ${expected_lines} lines, but it has ${number_of_lines}."
198                echo -e "${RED} The content is: \n\"\"\""
199                cat ${output_file}_tmp_line
200                echo -e "\"\"\""
201                echo -e "Not adding it to the output file. ${NC}"
202            else
203                if [[ $number_of_lines -gt 1 ]]; then
204                    header_line=$(head -n 1 "${output_file}_tmp_line")
205                    echo -e "${INFO_TAG} Header line is: $header_line"
206                fi
207                result_line=$(tail -n 1 "${output_file}_tmp_line")
208                echo -e "${INFO_TAG} Result line is: $result_line"
209
210                # If this is the first test, we need to add the header line to the output file
211                if [[ $is_first_test -eq 1 ]]; then
212                    append_to_output_file "$header_line" ", command-line" $output_file
213                fi
214
215                command=$(echo $command | sed 's/,/:comma:/g')
216                # Remove the `"` characters from the command line
217                command=$(echo $command | sed 's/\"//g')
218
219                # Append the result line to the output file
220                append_to_output_file "$result_line" ", $command" $output_file
221            fi
222
223            # Always remove the temporary file
224            rm -rf ${output_file}_tmp_line
225
226            # Gather netstat info after running the test
227            if [[ "${get_netstat_info}" == "1" ]]; then
228                echo -e "${INFO_TAG} Getting netstat info after"
229                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt
230                touch "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
231                python3 $script_location/../../../tools/diff_netstat_output.py \
232                    -n $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt \
233                    -o $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt \
234                    -d $DATALEN $no_header_netstat \
235                    -csv >> "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
236                rm -rf $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_*.txt
237                no_header_netstat=" -nh"
238            fi
239
240            change_domain
241        done
242    done
243}
244
245################################################################################
246# PARSE COMMAND LINE OPTIONS:
247
248while [ "$1" != "" ]; do
249    case $1 in
250        --executable)
251            executable=$2
252            shift
253            ;;
254        --docker)
255            DOCKER="1"
256            ;;
257        --output-folder)
258            output_folder=$2
259            shift
260            ;;
261        --role)
262            export role=$2
263            shift
264            ;;
265        --core)
266            export core=$2
267            shift
268            ;;
269        --assign-core-list | --thread-cpu-affinity)
270            export thread_cpu_affinity=$2
271            shift
272            ;;
273        --assign-thread-priorities | --thread-priorities)
274            export thread_priorities=$2
275            shift
276            ;;
277        --pin-memory)
278            export pin_memory="1"
279            ;;
280        --test-kind)
281            export lat_thr=$2
282            shift
283            ;;
284        --interface1)
285            export interface=$2
286            shift
287            ;;
288        --interface2)
289            export interface2=$2
290            shift
291            ;;
292        --ip1)
293            export ip1=$2
294            shift
295            ;;
296        --ip2)
297            export ip2=$2
298            shift
299            ;;
300        --repetitions)
301            export num_reps=$2
302            shift
303            ;;
304        --domain)
305            export domain=$2
306            shift
307            ;;
308        --execution-time)
309            export exec_time=$2
310            shift
311            ;;
312        --transport)
313            export transport=$2
314            shift
315            ;;
316        --datalen)
317            export datalen_input=$2
318            shift
319            ;;
320        --file-suffix)
321            export file_suffix=$2
322            shift
323            ;;
324        --folder-suffix)
325            export folder_suffix=$2
326            shift
327            ;;
328        --sub-folder)
329            sub_folder=$2
330            shift
331            ;;
332        --executable-suffix)
333            export executable_suffix=$2
334            shift
335            ;;
336        --extra-arguments)
337            export extra_arguments=$2
338            shift
339            ;;
340        --extra-arguments-pub)
341            export extra_arguments_pub=$2
342            shift
343            ;;
344        --extra-arguments-sub)
345            export extra_arguments_sub=$2
346            shift
347            ;;
348        --skip-be)
349            export skip_be_tests="1"
350            ;;
351        --skip-rel)
352            export skip_rel_tests="1"
353            ;;
354        --skip-keyed)
355            export skip_keyed_data="1"
356            ;;
357        --skip-large-data)
358            export skip_large_data="1"
359            ;;
360        --large-data)
361            export large_data="1"
362            ;;
363        --keyed)
364            export skip_unkeyed="1"
365            ;;
366        --unkeyed)
367            export skip_keyed_data="1"
368            ;;
369        --no-batching | --skip-batching)
370            export run_batching_tests="0"
371            export run_no_batching_tests="1"
372            ;;
373        --skip-no-batching)
374            export run_batching_tests="1"
375            export run_no_batching_tests="0"
376            ;;
377        --batch-size)
378            export batch_size=$2
379            shift
380            ;;
381        --reliable)
382            export skip_be_tests="1"
383            ;;
384        --best-effort)
385            export skip_rel_tests="1"
386            ;;
387        --security-gov)
388            export security_only="$2"
389            shift
390            ;;
391        --micro)
392            export micro="1"
393            ;;
394        --cert)
395            export cert="1"
396            ;;
397        --raw | --raw-transport)
398            export raw="1"
399            ;;
400        --tss)
401            export tss="1"
402            ;;
403        --no-colors)
404            export NO_COLORS="1"
405            ;;
406        --language)
407            export LANGUAGE=$2
408            shift
409            ;;
410        --loss-rate | --nic-loss-rate | --nic-loss-rate-percent)
411            export nic_loss_rate=$2
412            export configure_network="1"
413            shift
414            ;;
415        --delay | --nic-delay-ms | --nic-delay)
416            export nic_delay_ms=$2
417            export configure_network="1"
418            shift
419            ;;
420        --get-netstat-info | --netstat)
421            export get_netstat_info="1"
422            ;;
423        --reduced-data-sizes-set)
424            export REDUCED_DATA_SIZES_SET="1"
425            ;;
426        --dont-change-tuned-settings)
427            export dont_change_tuned_settings="1"
428            ;;
429        --no-taskset)
430            export no_taskset=1
431            ;;
432        --asynchronous | --async)
433            export ASYNC="1"
434            ;;
435        *)
436            echo -e "unknown parameter \"$1\""
437            exit 255
438            ;;
439    esac
440    shift
441done
442
443if [[ "$NO_COLORS" == "1" ]]; then
444    disable_colors
445fi
446
447export folder_base="$(dirname "${executable}")"/../../..
448
449if [[ $LANGUAGE == "java"  || "$LANGUAGE" == "cs" ]]; then
450    export folder_base="$(dirname "${executable}")"/../..
451fi
452if [[ $tss == "1" ]]; then
453    export folder_base="$(dirname "${executable}")"/../../../../..
454fi
455
456if [[ "${executable_suffix}" != "" ]]; then
457    export executable="${executable}${executable_suffix}"
458fi
459
460if [[ -n "${folder_suffix}" ]]; then
461    export output_folder="${output_folder}${folder_suffix}"
462fi
463
464if [[ -n "${sub_folder}" ]]; then
465    export output_folder="${output_folder}/${sub_folder}"
466fi
467
468if [[ "${ASYNC}" == "1" ]]; then
469    export output_folder="${output_folder}/async"
470fi
471
472echo -e "${INFO_TAG} Perftest executable is: $executable"
473echo -e "${INFO_TAG} Output folder is: $output_folder"
474
475################################################################################
476
477if [[ "$LANGUAGE" == "python" ]]; then
478    export skip_keyed_data="1"
479    export skip_large_data="1"
480    export skip_be_tests="1"
481    export run_no_batching_tests="0"
482fi
483
484if [[ "$LANGUAGE" == "java" || "$LANGUAGE" == "cs" ]]; then
485    # Java and C# perftest do not support these execution options.
486    unset thread_cpu_affinity
487    unset thread_priorities
488    unset pin_memory
489    export no_taskset=1
490fi
491
492if [[ "${skip_large_data}" == "1" ]]; then
493    export datasizes_extended=${datasizes}
494elif [[ "${large_data}" == "1" ]]; then
495    export datasizes=${datasizes_extended}
496fi
497
498if [[ "${datalen_input}" != "" ]]; then
499    echo -e "${YELLOW}[TEST] Testing only for ${datalen_input}${NC}"
500    export datasizes=${datalen_input}
501    export datasizes_extended=${datalen_input}
502    if [[ "${run_no_batching_tests}" == "1" && "${run_batching_tests}" == "0" ]]; then
503        export skip_large_data="1"
504    fi
505else 
506    if [[ "${REDUCED_DATA_SIZES_SET}" != "" ]]; then
507        echo -e "${YELLOW}[TEST] Testing Reduced set of datasizes ${NC}"
508        export datasizes="32 128 512 2048 8192 32768 63000"
509        export datasizes_extended="${datasizes} 102400 1048576 10485760"
510    fi
511fi
512
513if [[ "$role" != "pub" && "$role" != "sub" ]]; then
514    echo -e "${ERROR_TAG} It must be either publisher or subscriber"
515    exit 255
516fi
517
518if [[ "$lat_thr" != "thr" && "$lat_thr" != "lat" ]]; then
519    echo -e "${ERROR_TAG} It must be either lat or thr"
520    exit 255
521fi
522
523if [[ "${interface}" == "" ]]; then
524    echo "Using default nics"
525    export nic_publisher=${ip_machine_1}
526    export nic_subscriber=${ip_machine_2}
527elif [[ "${interface}" == "both" ]]; then
528    export nic_publisher="enp1s0f0,eno1"
529    export nic_subscriber="enp1s0f0,eno1"
530    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"
531    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"
532else
533    export nic_publisher=$interface
534    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"
535
536    if [[ "${interface2}" == "" ]]; then
537        export nic_subscriber=$interface
538    else
539        export nic_subscriber=$interface2
540    fi
541    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"
542
543    if [[ "${ip1}" != "" ]]; then
544        export ip_publisher=$ip1
545        echo "Using ip_publisher: ${ip_publisher}"
546    fi
547
548    if [[ "${ip2}" != "" ]]; then
549        export ip_subscriber=$ip2
550        echo "Using ip_subscriber: ${ip_subscriber}"
551    fi
552
553fi
554
555if [[ "$transport" != "" ]]; then
556    export transport_string="-transport $transport"
557
558    if [[ "$transport" == "UDPv4" ]]; then
559
560        export transport_string_pub="$transport_string -nic $nic_publisher"
561        export transport_string_sub="$transport_string -nic $nic_subscriber"
562
563        if [[ "$raw" == "1" ]]; then
564            export transport_string_pub="$transport_string_pub -peer ${ip_subscriber}"
565            export transport_string_sub="$transport_string_sub -peer ${ip_publisher}"
566        fi
567
568        if [[ "$micro" == "1" || "$cert" == "1" ]]; then
569            export transport_string_pub="$transport_string_pub -peer _udp://${ip_subscriber}"
570            export transport_string_sub="$transport_string_sub -peer _udp://${ip_publisher}"
571        fi
572
573    elif [[ "$transport" == "TCP" ]]; then
574        export transport_string_pub="$transport_string \
575            -nic $nic_publisher \
576            -peer 0@tcpv4_lan://${ip_subscriber}:7400"
577        export transport_string_sub="$transport_string \
578            -nic $nic_subscriber \
579            -peer 0@tcpv4_lan://${ip_publisher}:7400"
580    elif [[ "$transport" == "TLS" ]]; then
581        export transport_string_pub="$transport_string \
582            -nic $nic_publisher \
583            -peer tlsv4_lan://${ip_subscriber}:7400"
584        export transport_string_sub="$transport_string \
585            -nic $nic_subscriber \
586            -peer tlsv4_lan://${ip_publisher}:7400"
587    elif [[ "$transport" == "UDPv4_WAN" ]]; then
588        export transport_string_pub="$transport_string \
589            -nic $nic_publisher \
590            -transportPublicAddress $ip_publisher:7400"
591        export transport_string_sub="$transport_string \
592            -nic $nic_subscriber \
593            -peer 0@udpv4_wan://${ip_publisher}:7400"
594    else
595        export transport_string_pub="$transport_string"
596        export transport_string_sub="$transport_string"
597    fi
598fi
599
600################################################################################
601
602export pub_string="-pub \
603        ${transport_string_pub} \
604        -noPrintIntervals \
605        -executionTime $exec_time"
606
607if [[ ${lat_thr} == "lat" ]]; then
608    export pub_string="$pub_string \
609        -latencyTest"
610fi
611
612if [[ "$role" == "pub" ]]; then
613
614    if [[ "$configure_network" == "1" ]]; then
615        echo -e "${INFO_TAG} Adding -initialBurst 1 to publisher side"
616        export pub_string="$pub_string -initialBurst 1"
617    fi
618
619    if [[ "$ASYNC" == "1" ]]; then
620        export pub_string="${pub_string} -asynchronous"
621    fi
622fi
623
624export sub_string="-sub \
625        ${transport_string_sub} \
626        -noPrintIntervals"
627
628if [[ "$role" == "pub" ]]; then
629    echo -e "$INFO_TAG Publisher side running"
630    export commands_string=${pub_string}
631    export extra_arguments="${extra_arguments} ${extra_arguments_pub}"
632else
633    echo -e "$INFO_TAG Subscriber side running"
634    export commands_string=${sub_string}
635    export extra_arguments="${extra_arguments} ${extra_arguments_sub}"
636fi
637
638###############################################################################
639
640if [[ "$dont_change_tuned_settings" != "1" ]]; then
641    echo -e "${INFO_TAG} Executing: /set_${lat_thr}_mode.sh"
642    sudo /set_${lat_thr}_mode.sh
643    sleep 5
644fi
645
646if [[ "$role" == "pub" ]]; then
647    export interface_to_configure=$nic_publisher
648else
649    export interface_to_configure=$nic_subscriber
650fi
651
652echo -e "${INFO_TAG} Resetting network interface to default state"
653# Try to delete any existing netem rules (ignore errors if none exist)
654sudo tc qdisc del dev $interface_to_configure root netem 2>/dev/null || true
655# Reset to default (pfifo_fast)
656sudo tc qdisc del dev $interface_to_configure root 2>/dev/null || true
657
658if [[ "${configure_network}" == "1" ]]; then
659    echo -e "${INFO_TAG} Setting loss rate to ${nic_loss_rate}% and delay to ${nic_delay_ms}ms"
660    sudo tc qdisc add dev $interface_to_configure root netem loss ${nic_loss_rate}% delay ${nic_delay_ms}ms limit 10000000
661fi
662
663cd $folder_base
664echo -e "${INFO_TAG} Folder Base is: $PWD"
665mkdir -p $output_folder
666
667if [[ "${batch_size}" != "" ]]; then
668    if [[ "${lat_thr}" == "thr" ]]; then
669        if [[ "${batch_size}" -eq 0 ]]; then
670            echo -e "${INFO_TAG} Batch size is set to 0"
671            export run_batching_tests="0"
672            export run_no_batching_tests="1"
673        else
674            echo -e "${INFO_TAG} Batch size is set to ${batch_size}"
675            export run_batching_tests="1"
676            export run_no_batching_tests="0"
677        fi
678    else
679        echo -e "${INFO_TAG} Batch size is set to ${batch_size}. This value will be ignored for latency tests."
680        unset batch_size
681    fi
682fi
683
684# Tests that may use batching (when doing throughput tests). Also Latency tests.
685if [[ ${run_batching_tests} == "1" ]]; then
686
687    # UNKEYED
688    if [[ "${skip_unkeyed}" == "" ]]; then
689
690        # RELIABLE
691        if [[ "${skip_rel_tests}" == "" ]]; then
692            # For the time being, we will skip the throughput reliable tests for CERT.
693            # More info in PERF-1175
694            if ! [[ "${cert}" == "1" && "${lat_thr}" == "thr" && "${transport}" == "UDPv4" ]]; then
695                execute_test "unkeyed" "rel" "${datasizes_extended}" "${extra_arguments}" "$file_suffix"
696            else
697                echo -e "${INFO_TAG} Skipping reliable throughput tests for CERT. See PERF-1175 for more information."
698            fi
699        fi
700
701        # BEST EFFORT
702        if [[ "${skip_be_tests}" == "" ]]; then
703            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
704        fi
705    fi
706
707    # KEYED
708    if [[ "${skip_keyed_data}" == "" ]]; then
709
710        # RELIABLE
711        if [[ "${skip_rel_tests}" == "" ]]; then
712            execute_test "keyed" "rel" "${datasizes}" "${extra_arguments}" "$file_suffix"
713        fi
714
715        # BEST EFFORT
716        if [[ "${skip_be_tests}" == "" ]]; then
717            execute_test "keyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
718        fi
719    fi
720
721fi
722
723# Tests that will not use batching
724if [[ "${lat_thr}" == "thr" && "${run_no_batching_tests}" == "1" ]]; then
725
726    if [[ "$role" == "pub" ]]; then
727        export commands_string="${commands_string} -batchSize 0"
728    fi
729
730    # UNKEYED
731    if [[ "${skip_unkeyed}" == "" ]]; then
732
733        # RELIABLE
734        if [[ "${skip_rel_tests}" == "" ]]; then
735            execute_test "unkeyed" "rel" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
736        fi
737
738        # BEST EFFORT
739        if [[ "${skip_be_tests}" == "" ]]; then
740            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
741        fi
742    fi
743
744    # KEYED
745    if [[ "${skip_keyed_data}" == "" ]]; then
746
747        # RELIABLE
748        if [[ "${skip_rel_tests}" == "" ]]; then
749            execute_test "keyed" "rel" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
750        fi
751
752        # BEST EFFORT
753        if [[ "${skip_be_tests}" == "" ]]; then
754            execute_test "keyed" "be" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
755        fi
756    fi
757
758fi
759
760# Replace the cleanup section at the end:
761if [[ "${configure_network}" == "1" ]]; then
762    echo -e "${INFO_TAG} Resetting network interface to default state"
763    sudo tc qdisc del dev $interface_to_configure root netem 2>/dev/null || true
764    sudo tc qdisc del dev $interface_to_configure root 2>/dev/null || true
765fi

1#!/bin/bash
2filename=$0
3script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)
4
5echo -e "[Calling base_script/script.sh]"
6"${script_location}/../base_script/script.sh" $@ --transport UDPv4

Test Hardware

The following hardware was used to perform these tests:

Linux Nodes

Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 20.04 -- gcc 9.3.0

Switch

Dell 2048 -- 10Gbps switch (10Gbps and 1Gbps interfaces)

Keyed, UDPv4 10Gbps Network, Java

The graph below shows the one-way latency without load between a Publisher and a Subscriber running in two Linux nodes in a 10Gbps network. The numbers are for best-effort as well as strict reliable reliability scenarios.

Note

We use the median (50th percentile) instead of the average in order to get a more stable measurement that does not account for spurious outliers. We also calculate the average value and other percentile values, which can be seen in the Detailed Statistics section below.

Detailed Statistics

The following tables contain the raw numbers presented by RTI Perftest. These numbers are the exact output with no further processing.

  • Best Effort

Sample Size (Bytes)

Ave (us)

Std (us)

Min (us)

Max (us)

50% (μs)

90% (us)

99% (us)

99.99% (us)

99.9999% (us)

32

22

139.7

20

76073

21

23

25

20

20

64

22

139.5

20

77896

22

23

25

20

20

128

23

137.9

21

73111

22

24

26

21

21

256

23

148.4

21

79488

22

24

25

21

21

512

24

146.8

22

75814

23

25

27

22

22

1024

27

154.7

23

76554

27

28

29

23

23

2048

49

336.3

25

100142

47

65

74

25

25

4096

52

335.1

29

100143

55

63

76

29

29

8192

49

181.0

41

80380

46

59

62

41

41

16384

97

22.4

52

228

96

133

158

52

52

32768

185

2.8

84

260

185

187

189

84

84

63000

196

2.7

116

270

196

198

202

116

116

  • Reliable

Sample Size (Bytes)

Ave (us)

Std (us)

Min (us)

Max (us)

50% (μs)

90% (us)

99% (us)

99.99% (us)

99.9999% (us)

32

24

143.5

22

75003

23

24

25

22

22

64

24

137.7

22

69765

23

24

26

22

22

128

24

153.7

22

82448

24

25

26

22

22

256

25

152.3

23

78269

24

25

27

23

23

512

25

150.2

23

78692

25

26

28

23

23

1024

27

157.4

25

77513

27

29

34

25

25

2048

32

161.6

27

71640

30

33

55

27

27

4096

37

191.3

31

78810

34

39

66

31

31

8192

50

225.9

43

84549

46

60

90

43

43

16384

98

29.2

54

251

98

141

159

54

54

32768

185

3.2

87

260

185

187

190

87

87

63000

197

6.0

113

272

198

199

205

113

113

Perftest Scripts

To produce these tests, we executed RTI Perftest for Java. The exact script used can be found here:

  1#!/bin/bash
  2filename=$0
  3script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)
  4
  5export datasizes="32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000"
  6export datasizes_extended="${datasizes} 100000 500000 1048576 1548576 4194304 10485760"
  7
  8export domain="2"
  9export exec_time=20
 10export num_reps=1
 11export instance_number=100000
 12export core=0
 13export run_batching_tests="1"
 14export run_no_batching_tests="1"
 15export nic_loss_rate=0
 16export nic_delay_ms=0
 17
 18# We will use some colors to improve visibility of errors and info messages.
 19RED='\033[0;31m'
 20GREEN='\033[0;32m'
 21YELLOW='\033[0;33m'
 22BLUE='\033[0;34m'
 23LIGHTBLUE='\033[0;36m'
 24NC='\033[0m'
 25INFO_TAG="${GREEN}[INFO][$role]:${NC}"
 26WARNING_TAG="${YELLOW}[WARNING][$role]:${NC}"
 27ERROR_TAG="${RED}[ERROR][$role]:${NC}"
 28
 29################################################################################
 30
 31function disable_colors() {
 32    export RED=""
 33    export GREEN=""
 34    export YELLOW=""
 35    export NC=""
 36    export BLUE=""
 37    export LIGHTBLUE=""
 38    export INFO_TAG="${GREEN}[INFO][$role]:${NC}"
 39    export WARNING_TAG="${YELLOW}[WARNING][$role]:${NC}"
 40    export ERROR_TAG="${RED}[ERROR][$role]:${NC}"
 41}
 42
 43function change_domain() {
 44    if [[ "$domain" == "1" ]]; then
 45        export domain="2"
 46    else
 47        export domain="1"
 48    fi
 49}
 50
 51# Function to process and append lines to the output file
 52function append_to_output_file() {
 53    local line=${1?line required}; readonly line
 54    local append=$(echo "${2?append required}" | tr -s ' '); readonly append
 55    local file=${3?file required}; readonly file
 56    if [[ "${line: -1}" == $'\n' ]]; then
 57        line="${line::-1}"
 58    fi
 59    echo "${line}${append}" >> "$file"
 60}
 61
 62# Usage: execute_test <keyed/unkeyed> <rel/be> <datasizes> <batchSize>
 63function execute_test() {
 64
 65    local keyed_unkeyed=$1
 66    local rel_be=$2
 67    local datasizes_test=$3
 68    local other_args=$4
 69    local name_suffix=$5
 70
 71    local commands_string_test=$commands_string
 72    local tag=""
 73
 74    if [[ "${keyed_unkeyed}" == "keyed" ]]; then
 75        commands_string_test="${commands_string_test} -keyed -instances $instance_number"
 76        tag="[${YELLOW}${transport}${NC}|${BLUE}K${NC}|"
 77    else
 78        tag="[${YELLOW}${transport}${NC}|${LIGHTBLUE}UK${NC}|"
 79    fi
 80
 81    if [[ "${rel_be}" == "be" ]]; then
 82        commands_string_test="${commands_string_test} -bestEffort"
 83        tag="${tag}${YELLOW}BE${NC}]"
 84    else
 85        tag="${tag}${RED}REL${NC}]"
 86    fi
 87
 88    # If batch_size is set to a non-zero value, we will add it to the command line of the publisher side.
 89    # When batch_size is 0, it is already handled by the no-batching block which appends -batchSize 0 to commands_string.
 90    if [[ "$batch_size" != "" && "$batch_size" != "0" && "$role" == "pub" ]]; then
 91        commands_string_test="${commands_string_test} -batchSize $batch_size"
 92    fi
 93
 94    tag="${tag}[${LIGHTBLUE}${lat_thr}${NC}]"
 95
 96    local output_file=$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}.csv
 97
 98    if [[ "$role" == "pub" ]]; then
 99        echo -e "${YELLOW}[TEST]: $keyed_unkeyed, $rel_be, Is a no-batching test = $no_batching_tests. ${NC}"
100    fi
101
102    if [[ "$thread_cpu_affinity" != "" && "$LANGUAGE" != "java" && "$LANGUAGE" != "cs" ]]; then
103        commands_string_test="${commands_string_test} -threadCPUAffinity $thread_cpu_affinity"
104    fi
105
106    local pre_command_string=""
107
108    if [[ "$thread_priorities" != "" && "$LANGUAGE" != "java" && "$LANGUAGE" != "cs" ]]; then
109        echo -e "${WARNING_TAG} Thread priorities enabled, this requires using sudo"
110        # When using sudo, the LD_LIBRARY_PATH is emptied, hence we need to set it again in the command itself
111        export LD_LIBRARY_PATH_COPIED="$LD_LIBRARY_PATH"
112        pre_command_string="sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH_COPIED "
113        commands_string_test="${commands_string_test} -threadPriorities $thread_priorities"
114    fi
115
116    if [[ "$pin_memory" != "" && "$LANGUAGE" != "java" && "$LANGUAGE" != "cs" ]]; then
117        commands_string_test="${commands_string_test} -pinMemory"
118    fi
119
120    if [[ "$no_taskset" == "" && "$thread_cpu_affinity" == "" && "$LANGUAGE" != "java" && "$LANGUAGE" != "cs" ]]; then
121        pre_command_string="$pre_command_string taskset -c $core"
122    fi
123
124    if [[ "$LANGUAGE" == "python" ]]; then
125        pre_command_string="$pre_command_string python3 "
126    fi
127
128    if [[ "$DOCKER" == "1" ]]; then
129        pre_command_string="$pre_command_string docker run --net=host -v /home/perfuser/rti_license_connextpro.dat:/opt/rti.com/rti_connext_dds-7.3.0/rti_license.dat rticom/perftest:7.3.0-EAR "
130        executable=""
131    fi
132
133    # Get the aprox time this will take:
134    total_tests=$((`wc -w <<< "$datasizes_test"` * num_reps))
135    total_time=$((total_tests * exec_time))
136
137    touch $output_file
138    local no_headers=""
139    local current_test=0
140    for index in $(seq 1 ${num_reps}); do
141        for DATALEN in ${datasizes_test}; do
142            current_test=$((current_test + 1))
143
144            if [[ ! -s $output_file ]]; then
145                echo -e "${INFO_TAG} Output file is empty, filling the header."
146                no_headers=""
147            else
148                echo -e "${INFO_TAG} Output file is not empty."
149                no_headers=" -noOutputHeaders"
150            fi
151
152            export command="$pre_command_string $executable -domain $domain -dataLen $DATALEN $commands_string_test $other_args $no_headers"
153            if [[ "$role" == "pub" ]]; then
154                echo -e "Test ${tag} (${current_test}/${total_tests}) -- Total time = ${total_time}s"
155                echo -e ${BLUE}$command${NC}
156            else
157                echo -e ${LIGHTBLUE}$command${NC}
158            fi
159
160            # In certain cases we need to wait a bit before running the test, this is
161            # because the previous test might not be finished on the other side yet, or because the
162            # discovery mechanism does not work like in connext DDS.
163            if [[ "$LANGUAGE" == "cs" && "$role" == "pub" ]]; then
164                sleep 3
165            fi
166            if [[ "$raw" == "1" && "$role" == "sub" ]]; then
167                sleep 5
168            fi
169
170            # Gather netstat info before running the test
171            if [[ "${get_netstat_info}" == "1" ]]; then
172                echo -e "${INFO_TAG} Getting netstat info before"
173                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt
174            fi
175
176            # Execute the command and capture the output
177            if [[ $LANGUAGE == "c++98" || $LANGUAGE == "" ]]; then
178                echo -e "${INFO_TAG} Using C++98, using -outputFile option"
179                touch ${output_file}_tmp_line
180                chmod 777 ${output_file}_tmp_line
181                eval $command -outputFile ${output_file}_tmp_line
182            else
183                eval $command > ${output_file}_tmp_line
184            fi
185
186            number_of_lines=$(wc -l < ${output_file}_tmp_line)
187            echo -e "${INFO_TAG} Size of the output text: $number_of_lines lines"
188
189            # Check if this is the first test for this output file.
190            local is_first_test=0
191            if [[ ! -s $output_file ]]; then
192                is_first_test=1
193            fi
194            local expected_lines=$((is_first_test+1))
195
196            if [[ $number_of_lines -ne ${expected_lines} ]]; then
197                echo -e "${WARNING_TAG} The output text should have ${expected_lines} lines, but it has ${number_of_lines}."
198                echo -e "${RED} The content is: \n\"\"\""
199                cat ${output_file}_tmp_line
200                echo -e "\"\"\""
201                echo -e "Not adding it to the output file. ${NC}"
202            else
203                if [[ $number_of_lines -gt 1 ]]; then
204                    header_line=$(head -n 1 "${output_file}_tmp_line")
205                    echo -e "${INFO_TAG} Header line is: $header_line"
206                fi
207                result_line=$(tail -n 1 "${output_file}_tmp_line")
208                echo -e "${INFO_TAG} Result line is: $result_line"
209
210                # If this is the first test, we need to add the header line to the output file
211                if [[ $is_first_test -eq 1 ]]; then
212                    append_to_output_file "$header_line" ", command-line" $output_file
213                fi
214
215                command=$(echo $command | sed 's/,/:comma:/g')
216                # Remove the `"` characters from the command line
217                command=$(echo $command | sed 's/\"//g')
218
219                # Append the result line to the output file
220                append_to_output_file "$result_line" ", $command" $output_file
221            fi
222
223            # Always remove the temporary file
224            rm -rf ${output_file}_tmp_line
225
226            # Gather netstat info after running the test
227            if [[ "${get_netstat_info}" == "1" ]]; then
228                echo -e "${INFO_TAG} Getting netstat info after"
229                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt
230                touch "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
231                python3 $script_location/../../../tools/diff_netstat_output.py \
232                    -n $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt \
233                    -o $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt \
234                    -d $DATALEN $no_header_netstat \
235                    -csv >> "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
236                rm -rf $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_*.txt
237                no_header_netstat=" -nh"
238            fi
239
240            change_domain
241        done
242    done
243}
244
245################################################################################
246# PARSE COMMAND LINE OPTIONS:
247
248while [ "$1" != "" ]; do
249    case $1 in
250        --executable)
251            executable=$2
252            shift
253            ;;
254        --docker)
255            DOCKER="1"
256            ;;
257        --output-folder)
258            output_folder=$2
259            shift
260            ;;
261        --role)
262            export role=$2
263            shift
264            ;;
265        --core)
266            export core=$2
267            shift
268            ;;
269        --assign-core-list | --thread-cpu-affinity)
270            export thread_cpu_affinity=$2
271            shift
272            ;;
273        --assign-thread-priorities | --thread-priorities)
274            export thread_priorities=$2
275            shift
276            ;;
277        --pin-memory)
278            export pin_memory="1"
279            ;;
280        --test-kind)
281            export lat_thr=$2
282            shift
283            ;;
284        --interface1)
285            export interface=$2
286            shift
287            ;;
288        --interface2)
289            export interface2=$2
290            shift
291            ;;
292        --ip1)
293            export ip1=$2
294            shift
295            ;;
296        --ip2)
297            export ip2=$2
298            shift
299            ;;
300        --repetitions)
301            export num_reps=$2
302            shift
303            ;;
304        --domain)
305            export domain=$2
306            shift
307            ;;
308        --execution-time)
309            export exec_time=$2
310            shift
311            ;;
312        --transport)
313            export transport=$2
314            shift
315            ;;
316        --datalen)
317            export datalen_input=$2
318            shift
319            ;;
320        --file-suffix)
321            export file_suffix=$2
322            shift
323            ;;
324        --folder-suffix)
325            export folder_suffix=$2
326            shift
327            ;;
328        --sub-folder)
329            sub_folder=$2
330            shift
331            ;;
332        --executable-suffix)
333            export executable_suffix=$2
334            shift
335            ;;
336        --extra-arguments)
337            export extra_arguments=$2
338            shift
339            ;;
340        --extra-arguments-pub)
341            export extra_arguments_pub=$2
342            shift
343            ;;
344        --extra-arguments-sub)
345            export extra_arguments_sub=$2
346            shift
347            ;;
348        --skip-be)
349            export skip_be_tests="1"
350            ;;
351        --skip-rel)
352            export skip_rel_tests="1"
353            ;;
354        --skip-keyed)
355            export skip_keyed_data="1"
356            ;;
357        --skip-large-data)
358            export skip_large_data="1"
359            ;;
360        --large-data)
361            export large_data="1"
362            ;;
363        --keyed)
364            export skip_unkeyed="1"
365            ;;
366        --unkeyed)
367            export skip_keyed_data="1"
368            ;;
369        --no-batching | --skip-batching)
370            export run_batching_tests="0"
371            export run_no_batching_tests="1"
372            ;;
373        --skip-no-batching)
374            export run_batching_tests="1"
375            export run_no_batching_tests="0"
376            ;;
377        --batch-size)
378            export batch_size=$2
379            shift
380            ;;
381        --reliable)
382            export skip_be_tests="1"
383            ;;
384        --best-effort)
385            export skip_rel_tests="1"
386            ;;
387        --security-gov)
388            export security_only="$2"
389            shift
390            ;;
391        --micro)
392            export micro="1"
393            ;;
394        --cert)
395            export cert="1"
396            ;;
397        --raw | --raw-transport)
398            export raw="1"
399            ;;
400        --tss)
401            export tss="1"
402            ;;
403        --no-colors)
404            export NO_COLORS="1"
405            ;;
406        --language)
407            export LANGUAGE=$2
408            shift
409            ;;
410        --loss-rate | --nic-loss-rate | --nic-loss-rate-percent)
411            export nic_loss_rate=$2
412            export configure_network="1"
413            shift
414            ;;
415        --delay | --nic-delay-ms | --nic-delay)
416            export nic_delay_ms=$2
417            export configure_network="1"
418            shift
419            ;;
420        --get-netstat-info | --netstat)
421            export get_netstat_info="1"
422            ;;
423        --reduced-data-sizes-set)
424            export REDUCED_DATA_SIZES_SET="1"
425            ;;
426        --dont-change-tuned-settings)
427            export dont_change_tuned_settings="1"
428            ;;
429        --no-taskset)
430            export no_taskset=1
431            ;;
432        --asynchronous | --async)
433            export ASYNC="1"
434            ;;
435        *)
436            echo -e "unknown parameter \"$1\""
437            exit 255
438            ;;
439    esac
440    shift
441done
442
443if [[ "$NO_COLORS" == "1" ]]; then
444    disable_colors
445fi
446
447export folder_base="$(dirname "${executable}")"/../../..
448
449if [[ $LANGUAGE == "java"  || "$LANGUAGE" == "cs" ]]; then
450    export folder_base="$(dirname "${executable}")"/../..
451fi
452if [[ $tss == "1" ]]; then
453    export folder_base="$(dirname "${executable}")"/../../../../..
454fi
455
456if [[ "${executable_suffix}" != "" ]]; then
457    export executable="${executable}${executable_suffix}"
458fi
459
460if [[ -n "${folder_suffix}" ]]; then
461    export output_folder="${output_folder}${folder_suffix}"
462fi
463
464if [[ -n "${sub_folder}" ]]; then
465    export output_folder="${output_folder}/${sub_folder}"
466fi
467
468if [[ "${ASYNC}" == "1" ]]; then
469    export output_folder="${output_folder}/async"
470fi
471
472echo -e "${INFO_TAG} Perftest executable is: $executable"
473echo -e "${INFO_TAG} Output folder is: $output_folder"
474
475################################################################################
476
477if [[ "$LANGUAGE" == "python" ]]; then
478    export skip_keyed_data="1"
479    export skip_large_data="1"
480    export skip_be_tests="1"
481    export run_no_batching_tests="0"
482fi
483
484if [[ "$LANGUAGE" == "java" || "$LANGUAGE" == "cs" ]]; then
485    # Java and C# perftest do not support these execution options.
486    unset thread_cpu_affinity
487    unset thread_priorities
488    unset pin_memory
489    export no_taskset=1
490fi
491
492if [[ "${skip_large_data}" == "1" ]]; then
493    export datasizes_extended=${datasizes}
494elif [[ "${large_data}" == "1" ]]; then
495    export datasizes=${datasizes_extended}
496fi
497
498if [[ "${datalen_input}" != "" ]]; then
499    echo -e "${YELLOW}[TEST] Testing only for ${datalen_input}${NC}"
500    export datasizes=${datalen_input}
501    export datasizes_extended=${datalen_input}
502    if [[ "${run_no_batching_tests}" == "1" && "${run_batching_tests}" == "0" ]]; then
503        export skip_large_data="1"
504    fi
505else 
506    if [[ "${REDUCED_DATA_SIZES_SET}" != "" ]]; then
507        echo -e "${YELLOW}[TEST] Testing Reduced set of datasizes ${NC}"
508        export datasizes="32 128 512 2048 8192 32768 63000"
509        export datasizes_extended="${datasizes} 102400 1048576 10485760"
510    fi
511fi
512
513if [[ "$role" != "pub" && "$role" != "sub" ]]; then
514    echo -e "${ERROR_TAG} It must be either publisher or subscriber"
515    exit 255
516fi
517
518if [[ "$lat_thr" != "thr" && "$lat_thr" != "lat" ]]; then
519    echo -e "${ERROR_TAG} It must be either lat or thr"
520    exit 255
521fi
522
523if [[ "${interface}" == "" ]]; then
524    echo "Using default nics"
525    export nic_publisher=${ip_machine_1}
526    export nic_subscriber=${ip_machine_2}
527elif [[ "${interface}" == "both" ]]; then
528    export nic_publisher="enp1s0f0,eno1"
529    export nic_subscriber="enp1s0f0,eno1"
530    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"
531    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"
532else
533    export nic_publisher=$interface
534    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"
535
536    if [[ "${interface2}" == "" ]]; then
537        export nic_subscriber=$interface
538    else
539        export nic_subscriber=$interface2
540    fi
541    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"
542
543    if [[ "${ip1}" != "" ]]; then
544        export ip_publisher=$ip1
545        echo "Using ip_publisher: ${ip_publisher}"
546    fi
547
548    if [[ "${ip2}" != "" ]]; then
549        export ip_subscriber=$ip2
550        echo "Using ip_subscriber: ${ip_subscriber}"
551    fi
552
553fi
554
555if [[ "$transport" != "" ]]; then
556    export transport_string="-transport $transport"
557
558    if [[ "$transport" == "UDPv4" ]]; then
559
560        export transport_string_pub="$transport_string -nic $nic_publisher"
561        export transport_string_sub="$transport_string -nic $nic_subscriber"
562
563        if [[ "$raw" == "1" ]]; then
564            export transport_string_pub="$transport_string_pub -peer ${ip_subscriber}"
565            export transport_string_sub="$transport_string_sub -peer ${ip_publisher}"
566        fi
567
568        if [[ "$micro" == "1" || "$cert" == "1" ]]; then
569            export transport_string_pub="$transport_string_pub -peer _udp://${ip_subscriber}"
570            export transport_string_sub="$transport_string_sub -peer _udp://${ip_publisher}"
571        fi
572
573    elif [[ "$transport" == "TCP" ]]; then
574        export transport_string_pub="$transport_string \
575            -nic $nic_publisher \
576            -peer 0@tcpv4_lan://${ip_subscriber}:7400"
577        export transport_string_sub="$transport_string \
578            -nic $nic_subscriber \
579            -peer 0@tcpv4_lan://${ip_publisher}:7400"
580    elif [[ "$transport" == "TLS" ]]; then
581        export transport_string_pub="$transport_string \
582            -nic $nic_publisher \
583            -peer tlsv4_lan://${ip_subscriber}:7400"
584        export transport_string_sub="$transport_string \
585            -nic $nic_subscriber \
586            -peer tlsv4_lan://${ip_publisher}:7400"
587    elif [[ "$transport" == "UDPv4_WAN" ]]; then
588        export transport_string_pub="$transport_string \
589            -nic $nic_publisher \
590            -transportPublicAddress $ip_publisher:7400"
591        export transport_string_sub="$transport_string \
592            -nic $nic_subscriber \
593            -peer 0@udpv4_wan://${ip_publisher}:7400"
594    else
595        export transport_string_pub="$transport_string"
596        export transport_string_sub="$transport_string"
597    fi
598fi
599
600################################################################################
601
602export pub_string="-pub \
603        ${transport_string_pub} \
604        -noPrintIntervals \
605        -executionTime $exec_time"
606
607if [[ ${lat_thr} == "lat" ]]; then
608    export pub_string="$pub_string \
609        -latencyTest"
610fi
611
612if [[ "$role" == "pub" ]]; then
613
614    if [[ "$configure_network" == "1" ]]; then
615        echo -e "${INFO_TAG} Adding -initialBurst 1 to publisher side"
616        export pub_string="$pub_string -initialBurst 1"
617    fi
618
619    if [[ "$ASYNC" == "1" ]]; then
620        export pub_string="${pub_string} -asynchronous"
621    fi
622fi
623
624export sub_string="-sub \
625        ${transport_string_sub} \
626        -noPrintIntervals"
627
628if [[ "$role" == "pub" ]]; then
629    echo -e "$INFO_TAG Publisher side running"
630    export commands_string=${pub_string}
631    export extra_arguments="${extra_arguments} ${extra_arguments_pub}"
632else
633    echo -e "$INFO_TAG Subscriber side running"
634    export commands_string=${sub_string}
635    export extra_arguments="${extra_arguments} ${extra_arguments_sub}"
636fi
637
638###############################################################################
639
640if [[ "$dont_change_tuned_settings" != "1" ]]; then
641    echo -e "${INFO_TAG} Executing: /set_${lat_thr}_mode.sh"
642    sudo /set_${lat_thr}_mode.sh
643    sleep 5
644fi
645
646if [[ "$role" == "pub" ]]; then
647    export interface_to_configure=$nic_publisher
648else
649    export interface_to_configure=$nic_subscriber
650fi
651
652echo -e "${INFO_TAG} Resetting network interface to default state"
653# Try to delete any existing netem rules (ignore errors if none exist)
654sudo tc qdisc del dev $interface_to_configure root netem 2>/dev/null || true
655# Reset to default (pfifo_fast)
656sudo tc qdisc del dev $interface_to_configure root 2>/dev/null || true
657
658if [[ "${configure_network}" == "1" ]]; then
659    echo -e "${INFO_TAG} Setting loss rate to ${nic_loss_rate}% and delay to ${nic_delay_ms}ms"
660    sudo tc qdisc add dev $interface_to_configure root netem loss ${nic_loss_rate}% delay ${nic_delay_ms}ms limit 10000000
661fi
662
663cd $folder_base
664echo -e "${INFO_TAG} Folder Base is: $PWD"
665mkdir -p $output_folder
666
667if [[ "${batch_size}" != "" ]]; then
668    if [[ "${lat_thr}" == "thr" ]]; then
669        if [[ "${batch_size}" -eq 0 ]]; then
670            echo -e "${INFO_TAG} Batch size is set to 0"
671            export run_batching_tests="0"
672            export run_no_batching_tests="1"
673        else
674            echo -e "${INFO_TAG} Batch size is set to ${batch_size}"
675            export run_batching_tests="1"
676            export run_no_batching_tests="0"
677        fi
678    else
679        echo -e "${INFO_TAG} Batch size is set to ${batch_size}. This value will be ignored for latency tests."
680        unset batch_size
681    fi
682fi
683
684# Tests that may use batching (when doing throughput tests). Also Latency tests.
685if [[ ${run_batching_tests} == "1" ]]; then
686
687    # UNKEYED
688    if [[ "${skip_unkeyed}" == "" ]]; then
689
690        # RELIABLE
691        if [[ "${skip_rel_tests}" == "" ]]; then
692            # For the time being, we will skip the throughput reliable tests for CERT.
693            # More info in PERF-1175
694            if ! [[ "${cert}" == "1" && "${lat_thr}" == "thr" && "${transport}" == "UDPv4" ]]; then
695                execute_test "unkeyed" "rel" "${datasizes_extended}" "${extra_arguments}" "$file_suffix"
696            else
697                echo -e "${INFO_TAG} Skipping reliable throughput tests for CERT. See PERF-1175 for more information."
698            fi
699        fi
700
701        # BEST EFFORT
702        if [[ "${skip_be_tests}" == "" ]]; then
703            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
704        fi
705    fi
706
707    # KEYED
708    if [[ "${skip_keyed_data}" == "" ]]; then
709
710        # RELIABLE
711        if [[ "${skip_rel_tests}" == "" ]]; then
712            execute_test "keyed" "rel" "${datasizes}" "${extra_arguments}" "$file_suffix"
713        fi
714
715        # BEST EFFORT
716        if [[ "${skip_be_tests}" == "" ]]; then
717            execute_test "keyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
718        fi
719    fi
720
721fi
722
723# Tests that will not use batching
724if [[ "${lat_thr}" == "thr" && "${run_no_batching_tests}" == "1" ]]; then
725
726    if [[ "$role" == "pub" ]]; then
727        export commands_string="${commands_string} -batchSize 0"
728    fi
729
730    # UNKEYED
731    if [[ "${skip_unkeyed}" == "" ]]; then
732
733        # RELIABLE
734        if [[ "${skip_rel_tests}" == "" ]]; then
735            execute_test "unkeyed" "rel" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
736        fi
737
738        # BEST EFFORT
739        if [[ "${skip_be_tests}" == "" ]]; then
740            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
741        fi
742    fi
743
744    # KEYED
745    if [[ "${skip_keyed_data}" == "" ]]; then
746
747        # RELIABLE
748        if [[ "${skip_rel_tests}" == "" ]]; then
749            execute_test "keyed" "rel" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
750        fi
751
752        # BEST EFFORT
753        if [[ "${skip_be_tests}" == "" ]]; then
754            execute_test "keyed" "be" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
755        fi
756    fi
757
758fi
759
760# Replace the cleanup section at the end:
761if [[ "${configure_network}" == "1" ]]; then
762    echo -e "${INFO_TAG} Resetting network interface to default state"
763    sudo tc qdisc del dev $interface_to_configure root netem 2>/dev/null || true
764    sudo tc qdisc del dev $interface_to_configure root 2>/dev/null || true
765fi

1#!/bin/bash
2filename=$0
3script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)
4
5echo -e "[Calling base_script/script.sh]"
6"${script_location}/../base_script/script.sh" $@ --transport UDPv4

Test Hardware

The following hardware was used to perform these tests:

Linux Nodes

Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 20.04 -- gcc 9.3.0

Switch

Dell 2048 -- 10Gbps switch (10Gbps and 1Gbps interfaces)