UDPv4

The following tests have been performed by executing an RTI Perftest C++98 Publisher and Subscriber between two nodes, connected to a switch via Ethernet. The communication has been restricted to a single interface and the transport has been set to UDPv4.

Find information about the hardware, network, and command-line parameters after each of the tests.

Unkeyed, UDPv4 1Gbps Network, C++98

Linux x64Linux ARM v8

Minimum LatencyMaximum Throughput

The graph below shows the one-way latency without load between a Publisher and a Subscriber running in two Linux® nodes in a 1Gbps network. The numbers are for best-effort as well as strict reliable reliability scenarios.

Note

We use the median (50th percentile) instead of the average in order to get a more stable measurement that does not account for spurious outliers. We also calculate the average value and other percentile values, which can be seen in the Detailed Statistics section below.

Detailed Statistics

The following tables contain the raw numbers presented by RTI Perftest. These numbers are the exact output with no further processing.

• Best Effort

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)
32	52	1.6	48	158	52	55	56	82	158
64	55	1.8	50	118	55	57	58	84	118
128	56	1.6	51	119	56	58	60	85	119
256	61	1.5	56	158	61	63	64	90	158
512	67	1.4	62	125	67	68	70	96	125
1024	80	1.4	75	136	80	81	83	108	136
2048	108	1.2	96	158	108	109	110	136	158
4096	160	1.3	154	212	160	161	163	188	212
8192	265	1.5	258	318	265	267	268	295	318
16384	341	1.5	336	397	341	343	344	371	397
32768	474	1.4	467	536	474	475	476	503	536
63000	735	1.6	728	802	735	737	738	766	802

• Reliable

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)
32	55	2.2	47	118	56	57	61	84	118
64	57	1.6	48	105	57	59	62	86	105
128	58	1.8	50	121	58	60	63	88	121
256	64	1.4	57	123	63	65	67	92	123
512	69	1.5	64	129	68	70	73	98	129
1024	82	1.5	76	138	82	84	86	111	138
2048	109	1.4	104	161	109	111	114	138	161
4096	162	1.3	156	211	162	163	166	191	211
8192	267	1.5	261	320	267	268	271	295	320
16384	343	1.5	337	398	343	345	346	374	398
32768	475	1.5	468	535	475	477	478	505	535
63000	737	1.7	730	804	737	738	741	772	804
100000	1044	3.9	1037	1140	1043	1050	1055	1140	1140
500000	4314	10.6	4306	4616	4312	4316	4360	4616	4616
1048576	8798	33.6	8774	9348	8795	8806	8912	9348	9348
1548576	12887	30.7	12865	13655	12885	12887	12906	13655	13655
4194304	34644	117.7	34616	36420	34633	34641	34672	36420	36420
10485760	86671	417.8	86603	90632	86616	86627	88726	90632	90632

Perftest Scripts

To produce these tests, we executed RTI Perftest for C++98. The exact script used can be found here:

#!/bin/bash
filename=$0
script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)

export datasizes="32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000"
export datasizes_extended="${datasizes} 100000 500000 1048576 1548576 4194304 10485760"

export domain="2"
export exec_time=20
export num_reps=1
export instance_number=100000
export core=0
export run_batching_tests="1"
export run_no_batching_tests="1"

# We will use some colors to improve visibility of errors and info messages.
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
BLUE='\033[0;34m'
LIGHTBLUE='\033[0;36m'
NC='\033[0m'
INFO_TAG="${GREEN}[INFO][$role]:${NC}"
WARNING_TAG="${YELLOW}[WARNING][$role]:${NC}"
ERROR_TAG="${RED}[ERROR][$role]:${NC}"

################################################################################

function disable_colors() {
    export RED=""
    export GREEN=""
    export YELLOW=""
    export NC=""
    export BLUE=""
    export LIGHTBLUE=""
    export INFO_TAG="${GREEN}[INFO][$role]:${NC}"
    export WARNING_TAG="${YELLOW}[WARNING][$role]:${NC}"
    export ERROR_TAG="${RED}[ERROR][$role]:${NC}"
}

function change_domain() {
    if [[ "$domain" == "1" ]]; then
        export domain="2"
    else
        export domain="1"
    fi
}

# Function to process and append lines to the output file
function append_to_output_file() {
    local line=${1?line required}; readonly line
    local append=$(echo "${2?append required}" | tr -s ' '); readonly append
    local file=${3?file required}; readonly file
    if [[ "${line: -1}" == $'\n' ]]; then
        line="${line::-1}"
    fi
    echo "${line}${append}" >> "$file"
}

# Usage: execute_test <keyed/unkeyed> <rel/be> <datasizes> <batchSize>
function execute_test() {

    local keyed_unkeyed=$1
    local rel_be=$2
    local datasizes_test=$3
    local other_args=$4
    local name_suffix=$5

    local commands_string_test=$commands_string
    local tag=""

    if [[ "${keyed_unkeyed}" == "keyed" ]]; then
        commands_string_test="${commands_string_test} -keyed -instances $instance_number"
        tag="[${YELLOW}${transport}${NC}|${BLUE}K${NC}|"
    else
        tag="[${YELLOW}${transport}${NC}|${LIGHTBLUE}UK${NC}|"
    fi

    if [[ "${rel_be}" == "be" ]]; then
        commands_string_test="${commands_string_test} -bestEffort"
        tag="${tag}${YELLOW}BE${NC}]"
    else
        tag="${tag}${RED}REL${NC}]"
    fi

    # If batch_size is set, we will add it to the command line of the publisher side.
    if [[ "$batch_size" != "" && "$role" == "pub" ]]; then
        commands_string_test="${commands_string_test} -batchSize $batch_size"
    fi

    tag="${tag}[${LIGHTBLUE}${lat_thr}${NC}]"

    local output_file=$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}.csv

    if [[ "$role" == "pub" ]]; then
        echo -e "${YELLOW}[TEST]: $keyed_unkeyed, $rel_be, Is a no-batching test = $no_batching_tests. ${NC}"
    fi

    if [[ "$thread_cpu_affinity" != "" ]]; then
        commands_string_test="${commands_string_test} -threadCPUAffinity $thread_cpu_affinity"
    fi

    local pre_command_string=""

    if [[ "$thread_priorities" != "" ]]; then
        echo -e "${WARNING_TAG} Thread priorities enabled, this requires using sudo"
        # When using sudo, the LD_LIBRARY_PATH is emptied, hence we need to set it again in the command itself
        export LD_LIBRARY_PATH_COPIED="$LD_LIBRARY_PATH"
        pre_command_string="sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH_COPIED "
        commands_string_test="${commands_string_test} -threadPriorities $thread_priorities"
    fi

    if [[ "$pin_memory" != "" ]]; then
        commands_string_test="${commands_string_test} -pinMemory"
    fi

    if [[ "$no_taskset" == "" && "$thread_cpu_affinity" == "" && "$LANGUAGE" != "java" && "$LANGUAGE" != "cs" ]]; then
        pre_command_string="$pre_command_string taskset -c $core"
    fi

    if [[ "$LANGUAGE" == "python" ]]; then
        pre_command_string="$pre_command_string python3 "
    fi

    if [[ "$DOCKER" == "1" ]]; then
        pre_command_string="$pre_command_string docker run --net=host -v /home/perfuser/rti_license_connextpro.dat:/opt/rti.com/rti_connext_dds-7.3.0/rti_license.dat rticom/perftest:7.3.0-EAR "
        executable=""
    fi

    # Get the aprox time this will take:
    total_tests=$((`wc -w <<< "$datasizes_test"` * num_reps))
    total_time=$((total_tests * exec_time))

    touch $output_file
    local no_headers=""
    local current_test=0
    for index in $(seq 1 ${num_reps}); do
        for DATALEN in ${datasizes_test}; do
            current_test=$((current_test + 1))

            if [[ ! -s $output_file ]]; then
                echo -e "${INFO_TAG} Output file is empty, filling the header."
                no_headers=""
            else
                echo -e "${INFO_TAG} Output file is not empty."
                no_headers=" -noOutputHeaders"
            fi

            export command="$pre_command_string $executable -domain $domain -dataLen $DATALEN $commands_string_test $other_args $no_headers"
            if [[ "$role" == "pub" ]]; then
                echo -e "Test ${tag} (${current_test}/${total_tests}) -- Total time = ${total_time}s"
                echo -e ${BLUE}$command${NC}
            else
                echo -e ${LIGHTBLUE}$command${NC}
            fi

            # In certain cases we need to wait a bit before running the test, this is
            # because the previous test might not be finished on the other side yet, or because the
            # discovery mechanism does not work like in connext DDS.
            if [[ "$LANGUAGE" == "cs" && "$role" == "pub" ]]; then
                sleep 3
            fi
            if [[ "$raw" == "1" && "$role" == "sub" ]]; then
                sleep 5
            fi

            # Gather netstat info before running the test
            if [[ "${get_netstat_info}" == "1" ]]; then
                echo -e "${INFO_TAG} Getting netstat info before"
                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt
            fi

            # Execute the command and capture the output
            if [[ $LANGUAGE == "c++98" || $LANGUAGE == "" ]]; then
                echo -e "${INFO_TAG} Using C++98, using -outputFile option"
                touch ${output_file}_tmp_line
                chmod 777 ${output_file}_tmp_line
                eval $command -outputFile ${output_file}_tmp_line
            else
                eval $command > ${output_file}_tmp_line
            fi

            number_of_lines=$(wc -l < ${output_file}_tmp_line)
            echo -e "${INFO_TAG} Size of the output text: $number_of_lines lines"

            # Check if this is the first test for this output file.
            local is_first_test=0
            if [[ ! -s $output_file ]]; then
                is_first_test=1
            fi
            local expected_lines=$((is_first_test+1))

            if [[ $number_of_lines -ne ${expected_lines} ]]; then
                echo -e "${WARNING_TAG} The output text should have ${expected_lines} lines, but it has ${number_of_lines}."
                echo -e "${RED} The content is: \n\"\"\""
                cat ${output_file}_tmp_line
                echo -e "\"\"\""
                echo -e "Not adding it to the output file. ${NC}"
            else
                if [[ $number_of_lines -gt 1 ]]; then
                    header_line=$(head -n 1 "${output_file}_tmp_line")
                    echo -e "${INFO_TAG} Header line is: $header_line"
                fi
                result_line=$(tail -n 1 "${output_file}_tmp_line")
                echo -e "${INFO_TAG} Result line is: $result_line"

                # If this is the first test, we need to add the header line to the output file
                if [[ $is_first_test -eq 1 ]]; then
                    append_to_output_file "$header_line" ", command-line" $output_file
                fi

                command=$(echo $command | sed 's/,/:comma:/g')
                # Remove the `"` characters from the command line
                command=$(echo $command | sed 's/\"//g')

                # Append the result line to the output file
                append_to_output_file "$result_line" ", $command" $output_file
            fi

            # Always remove the temporary file
            rm -rf ${output_file}_tmp_line

            # Gather netstat info after running the test
            if [[ "${get_netstat_info}" == "1" ]]; then
                echo -e "${INFO_TAG} Getting netstat info after"
                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt
                touch "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
                python3 $script_location/../../../tools/diff_netstat_output.py \
                    -n $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt \
                    -o $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt \
                    -d $DATALEN $no_header_netstat \
                    -csv >> "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
                rm -rf $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_*.txt
                no_header_netstat=" -nh"
            fi

            change_domain
        done
    done
}

################################################################################
# PARSE COMMAND LINE OPTIONS:

while [ "$1" != "" ]; do
    case $1 in
        --executable)
            executable=$2
            shift
            ;;
        --docker)
            DOCKER="1"
            ;;
        --output-folder)
            output_folder=$2
            shift
            ;;
        --role)
            export role=$2
            shift
            ;;
        --core)
            export core=$2
            shift
            ;;
        --assign-core-list | --thread-cpu-affinity)
            export thread_cpu_affinity=$2
            shift
            ;;
        --assign-thread-priorities | --thread-priorities)
            export thread_priorities=$2
            shift
            ;;
        --pin-memory)
            export pin_memory="1"
            ;;
        --test-kind)
            export lat_thr=$2
            shift
            ;;
        --interface1)
            export interface=$2
            shift
            ;;
        --interface2)
            export interface2=$2
            shift
            ;;
        --ip1)
            export ip1=$2
            shift
            ;;
        --ip2)
            export ip2=$2
            shift
            ;;
        --repetitions)
            export num_reps=$2
            shift
            ;;
        --domain)
            export domain=$2
            shift
            ;;
        --execution-time)
            export exec_time=$2
            shift
            ;;
        --transport)
            export transport=$2
            shift
            ;;
        --datalen)
            export datalen_input=$2
            shift
            ;;
        --file-suffix)
            export file_suffix=$2
            shift
            ;;
        --folder-suffix)
            export folder_suffix=$2
            shift
            ;;
        --sub-folder)
            sub_folder=$2
            shift
            ;;
        --executable-suffix)
            export executable_suffix=$2
            shift
            ;;
        --extra-arguments)
            export extra_arguments=$2
            shift
            ;;
        --extra-arguments-pub)
            export extra_arguments_pub=$2
            shift
            ;;
        --extra-arguments-sub)
            export extra_arguments_sub=$2
            shift
            ;;
        --skip-be)
            export skip_be_tests="1"
            ;;
        --skip-rel)
            export skip_rel_tests="1"
            ;;
        --skip-keyed)
            export skip_keyed_data="1"
            ;;
        --skip-large-data)
            export skip_large_data="1"
            ;;
        --large-data)
            export large_data="1"
            ;;
        --keyed)
            export skip_unkeyed="1"
            ;;
        --unkeyed)
            export skip_keyed_data="1"
            ;;
        --no-batching | --skip-batching)
            export run_batching_tests="0"
            export run_no_batching_tests="1"
            ;;
        --skip-no-batching)
            export run_batching_tests="1"
            export run_no_batching_tests="0"
            ;;
        --batch-size)
            export batch_size=$2
            shift
            ;;
        --reliable)
            export skip_be_tests="1"
            ;;
        --best-effort)
            export skip_rel_tests="1"
            ;;
        --security-gov)
            export security_only="$2"
            shift
            ;;
        --micro)
            export micro="1"
            ;;
        --cert)
            export cert="1"
            ;;
        --raw | --raw-transport)
            export raw="1"
            ;;
        --tss)
            export tss="1"
            ;;
        --no-colors)
            export NO_COLORS="1"
            ;;
        --language)
            export LANGUAGE=$2
            shift
            ;;
        --loss-rate)
            export loss_rate=$2
            shift
            ;;
        --get-netstat-info | --netstat)
            export get_netstat_info="1"
            ;;
        --reduced-data-sizes-set)
            export REDUCED_DATA_SIZES_SET="1"
            ;;
        --dont-change-tuned-settings)
            export dont_change_tuned_settings="1"
            ;;
        --no-taskset)
            export no_taskset=1
            ;;
        *)
            echo -e "unknown parameter \"$1\""
            exit 255
            ;;
    esac
    shift
done

if [[ "$NO_COLORS" == "1" ]]; then
    disable_colors
fi

export folder_base="$(dirname "${executable}")"/../../..

if [[ $LANGUAGE == "java"  || "$LANGUAGE" == "cs" ]]; then
    export folder_base="$(dirname "${executable}")"/../..
fi
if [[ $tss == "1" ]]; then
    export folder_base="$(dirname "${executable}")"/../../../../..
fi

if [[ "${executable_suffix}" != "" ]]; then
    export executable="${executable}${executable_suffix}"
fi

if [[ -n "${folder_suffix}" ]]; then
    export output_folder="${output_folder}${folder_suffix}"
fi

if [[ -n "${sub_folder}" ]]; then
    export output_folder="${output_folder}/${sub_folder}"
fi

echo -e "${INFO_TAG} Perftest executable is: $executable"
echo -e "${INFO_TAG} Output folder is: $output_folder"

################################################################################

if [[ "$LANGUAGE" == "python" ]]; then
    export skip_keyed_data="1"
    export skip_large_data="1"
    export skip_be_tests="1"
    export run_no_batching_tests="0"
fi

if [[ "${skip_large_data}" == "1" ]]; then
    export datasizes_extended=${datasizes}
elif [[ "${large_data}" == "1" ]]; then
    export datasizes=${datasizes_extended}
fi

if [[ "${datalen_input}" != "" ]]; then
    echo -e "${YELLOW}[TEST] Testing only for ${datalen_input}${NC}"
    export datasizes=${datalen_input}
    export datasizes_extended=${datalen_input}
    if [[ "${run_no_batching_tests}" == "1" && "${run_batching_tests}" == "0" ]]; then
        export skip_large_data="1"
    fi
else 
    if [[ "${REDUCED_DATA_SIZES_SET}" != "" ]]; then
        echo -e "${YELLOW}[TEST] Testing Reduced set of datasizes ${NC}"
        export datasizes="32 128 512 2048 8192 32768 63000"
        export datasizes_extended="${datasizes} 102400 1048576 10485760"
    fi
fi

if [[ "$role" != "pub" && "$role" != "sub" ]]; then
    echo -e "${ERROR_TAG} It must be either publisher or subscriber"
    exit 255
fi

if [[ "$lat_thr" != "thr" && "$lat_thr" != "lat" ]]; then
    echo -e "${ERROR_TAG} It must be either lat or thr"
    exit 255
fi

if [[ "${interface}" == "" ]]; then
    echo "Using default nics"
    export nic_publisher=${ip_machine_1}
    export nic_subscriber=${ip_machine_2}
elif [[ "${interface}" == "both" ]]; then
    export nic_publisher="enp1s0f0,eno1"
    export nic_subscriber="enp1s0f0,eno1"
    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"
    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"
else
    export nic_publisher=$interface
    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"

    if [[ "${interface2}" == "" ]]; then
        export nic_subscriber=$interface
    else
        export nic_subscriber=$interface2
    fi
    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"

    if [[ "${ip1}" != "" ]]; then
        export ip_publisher=$ip1
        echo "Using ip_publisher: ${ip_publisher}"
    fi

    if [[ "${ip2}" != "" ]]; then
        export ip_subscriber=$ip2
        echo "Using ip_subscriber: ${ip_subscriber}"
    fi

fi

export transport_string="-transport $transport"

if [[ "$transport" == "UDPv4" ]]; then

    export transport_string_pub="$transport_string -nic $nic_publisher"
    export transport_string_sub="$transport_string -nic $nic_subscriber"

    if [[ "$raw" == "1" ]]; then
        export transport_string_pub="$transport_string_pub -peer ${ip_subscriber}"
        export transport_string_sub="$transport_string_sub -peer ${ip_publisher}"
    fi

    if [[ "$micro" == "1" || "$cert" == "1" ]]; then
        export transport_string_pub="$transport_string_pub -peer _udp://${ip_subscriber}"
        export transport_string_sub="$transport_string_sub -peer _udp://${ip_publisher}"
    fi

elif [[ "$transport" == "TCP" ]]; then
    export transport_string_pub="$transport_string \
        -nic $nic_publisher \
        -peer 0@tcpv4_lan://${ip_subscriber}:7400"
    export transport_string_sub="$transport_string \
        -nic $nic_subscriber \
        -peer 0@tcpv4_lan://${ip_publisher}:7400"
elif [[ "$transport" == "TLS" ]]; then
    export transport_string_pub="$transport_string \
        -nic $nic_publisher \
        -peer tlsv4_lan://${ip_subscriber}:7400"
    export transport_string_sub="$transport_string \
        -nic $nic_subscriber \
        -peer tlsv4_lan://${ip_publisher}:7400"
elif [[ "$transport" == "UDPv4_WAN" ]]; then
    export transport_string_pub="$transport_string \
        -nic $nic_publisher \
        -transportPublicAddress $ip_publisher:7400"
    export transport_string_sub="$transport_string \
        -nic $nic_subscriber \
        -peer 0@udpv4_wan://${ip_publisher}:7400"
else
    export transport_string_pub="$transport_string"
    export transport_string_sub="$transport_string"
fi

################################################################################

export pub_string="-pub \
        ${transport_string_pub} \
        -noPrintIntervals \
        -executionTime $exec_time"

if [[ ${lat_thr} == "lat" ]]; then
    export pub_string="$pub_string \
        -latencyTest"
fi

export sub_string="-sub \
        ${transport_string_sub} \
        -noPrintIntervals"

if [[ "$role" == "pub" ]]; then
    echo -e "$INFO_TAG Publisher side running"
    export commands_string=${pub_string}
    export extra_arguments="${extra_arguments} ${extra_arguments_pub}"
else
    echo -e "$INFO_TAG Subscriber side running"
    export commands_string=${sub_string}
    export extra_arguments="${extra_arguments} ${extra_arguments_sub}"
fi

###############################################################################

if [[ "$dont_change_tuned_settings" != "1" ]]; then
    echo -e "${INFO_TAG} Executing: /set_${lat_thr}_mode.sh"
    sudo /set_${lat_thr}_mode.sh
    sleep 5
fi

echo -e "${INFO_TAG} Disabling any loss rate"
sudo tc qdisc add dev $nic_publisher root netem loss 0%
sudo tc qdisc del dev $nic_publisher root netem loss 0%

if [[ "$role" == "pub" && "${loss_rate}" != "" ]]; then
    echo -e "${INFO_TAG} Setting loss rate to ${loss_rate}%"
    sudo tc qdisc add dev $nic_publisher root netem loss $loss_rate%
fi

cd $folder_base
echo -e "${INFO_TAG} Folder Base is: $PWD"
mkdir -p $output_folder

if [[ "${batch_size}" != "" ]]; then
    if [[ "${lat_thr}" == "thr" ]]; then
        if [[ "${batch_size}" -eq 0 ]]; then
            echo -e "${INFO_TAG} Batch size is set to 0"
            export run_batching_tests="0"
            export run_no_batching_tests="1"
        else
            echo -e "${INFO_TAG} Batch size is set to ${batch_size}"
            export run_batching_tests="1"
            export run_no_batching_tests="0"
        fi
    else
        echo -e "${INFO_TAG} Batch size is set to ${batch_size}. This value will be ignored for latency tests."
        unset batch_size
    fi
fi

# Tests that may use batching (when doing throughput tests). Also Latency tests.
if [[ ${run_batching_tests} == "1" ]]; then

    # UNKEYED
    if [[ "${skip_unkeyed}" == "" ]]; then

        # RELIABLE
        if [[ "${skip_rel_tests}" == "" ]]; then
            execute_test "unkeyed" "rel" "${datasizes_extended}" "${extra_arguments}" "$file_suffix"
        fi

        # BEST EFFORT
        if [[ "${skip_be_tests}" == "" ]]; then
            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
        fi
    fi

    # KEYED
    if [[ "${skip_keyed_data}" == "" ]]; then

        # RELIABLE
        if [[ "${skip_rel_tests}" == "" ]]; then
            execute_test "keyed" "rel" "${datasizes}" "${extra_arguments}" "$file_suffix"
        fi

        # BEST EFFORT
        if [[ "${skip_be_tests}" == "" ]]; then
            execute_test "keyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
        fi
    fi

fi

# Tests that will not use batching
if [[ "${lat_thr}" == "thr" && "${run_no_batching_tests}" == "1" ]]; then

    if [[ "$role" == "pub" ]]; then
        export commands_string="${commands_string} -batchSize 0"
    fi

    # UNKEYED
    if [[ "${skip_unkeyed}" == "" ]]; then

        # RELIABLE
        if [[ "${skip_rel_tests}" == "" ]]; then
            execute_test "unkeyed" "rel" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
        fi

        # BEST EFFORT
        if [[ "${skip_be_tests}" == "" ]]; then
            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
        fi
    fi

    # KEYED
    if [[ "${skip_keyed_data}" == "" ]]; then

        # RELIABLE
        if [[ "${skip_rel_tests}" == "" ]]; then
            execute_test "keyed" "rel" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
        fi

        # BEST EFFORT
        if [[ "${skip_be_tests}" == "" ]]; then
            execute_test "keyed" "be" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
        fi
    fi

fi

if [[ "$role" == "pub" && "${loss_rate}" != "" ]]; then
    echo -e "${INFO_TAG} Disabling loss rate"
    sudo tc qdisc del dev $nic_publisher root netem loss $loss_rate%
fi

#!/bin/bash
filename=$0
script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)

echo -e "[Calling base_script/script.sh]"
"${script_location}/../base_script/script.sh" $@ --transport UDPv4

Test Software

The following software was used to perform these tests:

RTI Connext DDS 6.1.1 Host and Target Libraries for x64 Linux (x64Linux4gcc7.3.0)

Test Hardware

The following hardware was used to perform these tests:

Linux Nodes

Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 20.04 -- gcc 9.3.0

Switch

Dell 2048 -- 10Gbps switch (10Gbps and 1Gbps interfaces)

Unkeyed, UDPv4 10Gbps Network, C++98

Linux x64

Minimum LatencyMaximum Throughput

The graph below shows the one-way latency without load between a Publisher and a Subscriber running in two Linux nodes in a 10Gbps network. The numbers are for best-effort as well as strict reliable reliability scenarios.

Note

We use the median (50th percentile) instead of the average in order to get a more stable measurement that does not account for spurious outliers. We also calculate the average value and other percentile values, which can be seen in the Detailed Statistics section below.

Detailed Statistics

The following tables contain the raw numbers presented by RTI Perftest. These numbers are the exact output with no further processing.

• Best Effort

Sample Size (Bytes)	Ave (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)
32	15	132.0	14	100050	15	15	16	44	100050
64	15	0.9	14	64	15	16	17	44	64
128	15	1.0	15	65	15	16	17	45	65
256	15	1.0	14	73	15	15	16	44	73
512	19	1.9	15	69	19	21	22	47	69
1024	25	3.3	17	64	26	28	29	53	64
2048	32	194.2	18	100076	30	55	69	80	100076
4096	32	131.3	23	100058	30	34	54	69	100058
8192	40	8.7	33	89	34	55	62	77	89
16384	83	22.5	44	173	82	118	131	143	173
32768	172	2.2	65	216	172	173	174	201	216
63000	183	2.7	99	213	183	185	186	212	213

• Reliable

Sample Size (Bytes)	Ave (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)
32	16	1.0	15	96	16	17	20	46	96
64	17	1.1	16	68	16	17	21	46	68
128	17	1.3	16	67	16	17	23	46	67
256	17	2.1	16	64	17	19	28	46	64
512	20	3.0	17	70	19	21	32	49	70
1024	25	4.8	18	85	26	29	39	57	85
2048	29	31.9	20	4917	30	33	44	68	4917
4096	31	18.1	24	4235	30	36	48	68	4235
8192	41	26.6	34	5016	36	57	65	82	5016
16384	86	25.3	45	161	85	123	136	146	161
32768	171	8.0	74	203	173	174	176	202	203
63000	169	15.0	104	223	167	186	188	215	223
100000	205	9.2	154	254	208	210	216	241	254
500000	619	32.6	504	2920	619	651	697	1011	2920
1048576	1180	43.9	1069	1683	1204	1207	1224	1683	1683
1548576	1704	20.5	1541	2331	1701	1730	1752	2331	2331
4194304	4052	47.9	4010	6083	4047	4073	4099	6083	6083
10485760	10185	166.8	10152	15009	10177	10188	10211	15009	15009

Perftest Scripts

To produce these tests, we executed RTI Perftest for C++98. The exact script used can be found here:

#!/bin/bash
filename=$0
script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)

export datasizes="32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000"
export datasizes_extended="${datasizes} 100000 500000 1048576 1548576 4194304 10485760"

export domain="2"
export exec_time=20
export num_reps=1
export instance_number=100000
export core=0
export run_batching_tests="1"
export run_no_batching_tests="1"

# We will use some colors to improve visibility of errors and info messages.
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
BLUE='\033[0;34m'
LIGHTBLUE='\033[0;36m'
NC='\033[0m'
INFO_TAG="${GREEN}[INFO][$role]:${NC}"
WARNING_TAG="${YELLOW}[WARNING][$role]:${NC}"
ERROR_TAG="${RED}[ERROR][$role]:${NC}"

################################################################################

function disable_colors() {
    export RED=""
    export GREEN=""
    export YELLOW=""
    export NC=""
    export BLUE=""
    export LIGHTBLUE=""
    export INFO_TAG="${GREEN}[INFO][$role]:${NC}"
    export WARNING_TAG="${YELLOW}[WARNING][$role]:${NC}"
    export ERROR_TAG="${RED}[ERROR][$role]:${NC}"
}

function change_domain() {
    if [[ "$domain" == "1" ]]; then
        export domain="2"
    else
        export domain="1"
    fi
}

# Function to process and append lines to the output file
function append_to_output_file() {
    local line=${1?line required}; readonly line
    local append=$(echo "${2?append required}" | tr -s ' '); readonly append
    local file=${3?file required}; readonly file
    if [[ "${line: -1}" == $'\n' ]]; then
        line="${line::-1}"
    fi
    echo "${line}${append}" >> "$file"
}

# Usage: execute_test <keyed/unkeyed> <rel/be> <datasizes> <batchSize>
function execute_test() {

    local keyed_unkeyed=$1
    local rel_be=$2
    local datasizes_test=$3
    local other_args=$4
    local name_suffix=$5

    local commands_string_test=$commands_string
    local tag=""

    if [[ "${keyed_unkeyed}" == "keyed" ]]; then
        commands_string_test="${commands_string_test} -keyed -instances $instance_number"
        tag="[${YELLOW}${transport}${NC}|${BLUE}K${NC}|"
    else
        tag="[${YELLOW}${transport}${NC}|${LIGHTBLUE}UK${NC}|"
    fi

    if [[ "${rel_be}" == "be" ]]; then
        commands_string_test="${commands_string_test} -bestEffort"
        tag="${tag}${YELLOW}BE${NC}]"
    else
        tag="${tag}${RED}REL${NC}]"
    fi

    # If batch_size is set, we will add it to the command line of the publisher side.
    if [[ "$batch_size" != "" && "$role" == "pub" ]]; then
        commands_string_test="${commands_string_test} -batchSize $batch_size"
    fi

    tag="${tag}[${LIGHTBLUE}${lat_thr}${NC}]"

    local output_file=$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}.csv

    if [[ "$role" == "pub" ]]; then
        echo -e "${YELLOW}[TEST]: $keyed_unkeyed, $rel_be, Is a no-batching test = $no_batching_tests. ${NC}"
    fi

    if [[ "$thread_cpu_affinity" != "" ]]; then
        commands_string_test="${commands_string_test} -threadCPUAffinity $thread_cpu_affinity"
    fi

    local pre_command_string=""

    if [[ "$thread_priorities" != "" ]]; then
        echo -e "${WARNING_TAG} Thread priorities enabled, this requires using sudo"
        # When using sudo, the LD_LIBRARY_PATH is emptied, hence we need to set it again in the command itself
        export LD_LIBRARY_PATH_COPIED="$LD_LIBRARY_PATH"
        pre_command_string="sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH_COPIED "
        commands_string_test="${commands_string_test} -threadPriorities $thread_priorities"
    fi

    if [[ "$pin_memory" != "" ]]; then
        commands_string_test="${commands_string_test} -pinMemory"
    fi

    if [[ "$no_taskset" == "" && "$thread_cpu_affinity" == "" && "$LANGUAGE" != "java" && "$LANGUAGE" != "cs" ]]; then
        pre_command_string="$pre_command_string taskset -c $core"
    fi

    if [[ "$LANGUAGE" == "python" ]]; then
        pre_command_string="$pre_command_string python3 "
    fi

    if [[ "$DOCKER" == "1" ]]; then
        pre_command_string="$pre_command_string docker run --net=host -v /home/perfuser/rti_license_connextpro.dat:/opt/rti.com/rti_connext_dds-7.3.0/rti_license.dat rticom/perftest:7.3.0-EAR "
        executable=""
    fi

    # Get the aprox time this will take:
    total_tests=$((`wc -w <<< "$datasizes_test"` * num_reps))
    total_time=$((total_tests * exec_time))

    touch $output_file
    local no_headers=""
    local current_test=0
    for index in $(seq 1 ${num_reps}); do
        for DATALEN in ${datasizes_test}; do
            current_test=$((current_test + 1))

            if [[ ! -s $output_file ]]; then
                echo -e "${INFO_TAG} Output file is empty, filling the header."
                no_headers=""
            else
                echo -e "${INFO_TAG} Output file is not empty."
                no_headers=" -noOutputHeaders"
            fi

            export command="$pre_command_string $executable -domain $domain -dataLen $DATALEN $commands_string_test $other_args $no_headers"
            if [[ "$role" == "pub" ]]; then
                echo -e "Test ${tag} (${current_test}/${total_tests}) -- Total time = ${total_time}s"
                echo -e ${BLUE}$command${NC}
            else
                echo -e ${LIGHTBLUE}$command${NC}
            fi

            # In certain cases we need to wait a bit before running the test, this is
            # because the previous test might not be finished on the other side yet, or because the
            # discovery mechanism does not work like in connext DDS.
            if [[ "$LANGUAGE" == "cs" && "$role" == "pub" ]]; then
                sleep 3
            fi
            if [[ "$raw" == "1" && "$role" == "sub" ]]; then
                sleep 5
            fi

            # Gather netstat info before running the test
            if [[ "${get_netstat_info}" == "1" ]]; then
                echo -e "${INFO_TAG} Getting netstat info before"
                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt
            fi

            # Execute the command and capture the output
            if [[ $LANGUAGE == "c++98" || $LANGUAGE == "" ]]; then
                echo -e "${INFO_TAG} Using C++98, using -outputFile option"
                touch ${output_file}_tmp_line
                chmod 777 ${output_file}_tmp_line
                eval $command -outputFile ${output_file}_tmp_line
            else
                eval $command > ${output_file}_tmp_line
            fi

            number_of_lines=$(wc -l < ${output_file}_tmp_line)
            echo -e "${INFO_TAG} Size of the output text: $number_of_lines lines"

            # Check if this is the first test for this output file.
            local is_first_test=0
            if [[ ! -s $output_file ]]; then
                is_first_test=1
            fi
            local expected_lines=$((is_first_test+1))

            if [[ $number_of_lines -ne ${expected_lines} ]]; then
                echo -e "${WARNING_TAG} The output text should have ${expected_lines} lines, but it has ${number_of_lines}."
                echo -e "${RED} The content is: \n\"\"\""
                cat ${output_file}_tmp_line
                echo -e "\"\"\""
                echo -e "Not adding it to the output file. ${NC}"
            else
                if [[ $number_of_lines -gt 1 ]]; then
                    header_line=$(head -n 1 "${output_file}_tmp_line")
                    echo -e "${INFO_TAG} Header line is: $header_line"
                fi
                result_line=$(tail -n 1 "${output_file}_tmp_line")
                echo -e "${INFO_TAG} Result line is: $result_line"

                # If this is the first test, we need to add the header line to the output file
                if [[ $is_first_test -eq 1 ]]; then
                    append_to_output_file "$header_line" ", command-line" $output_file
                fi

                command=$(echo $command | sed 's/,/:comma:/g')
                # Remove the `"` characters from the command line
                command=$(echo $command | sed 's/\"//g')

                # Append the result line to the output file
                append_to_output_file "$result_line" ", $command" $output_file
            fi

            # Always remove the temporary file
            rm -rf ${output_file}_tmp_line

            # Gather netstat info after running the test
            if [[ "${get_netstat_info}" == "1" ]]; then
                echo -e "${INFO_TAG} Getting netstat info after"
                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt
                touch "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
                python3 $script_location/../../../tools/diff_netstat_output.py \
                    -n $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt \
                    -o $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt \
                    -d $DATALEN $no_header_netstat \
                    -csv >> "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
                rm -rf $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_*.txt
                no_header_netstat=" -nh"
            fi

            change_domain
        done
    done
}

################################################################################
# PARSE COMMAND LINE OPTIONS:

while [ "$1" != "" ]; do
    case $1 in
        --executable)
            executable=$2
            shift
            ;;
        --docker)
            DOCKER="1"
            ;;
        --output-folder)
            output_folder=$2
            shift
            ;;
        --role)
            export role=$2
            shift
            ;;
        --core)
            export core=$2
            shift
            ;;
        --assign-core-list | --thread-cpu-affinity)
            export thread_cpu_affinity=$2
            shift
            ;;
        --assign-thread-priorities | --thread-priorities)
            export thread_priorities=$2
            shift
            ;;
        --pin-memory)
            export pin_memory="1"
            ;;
        --test-kind)
            export lat_thr=$2
            shift
            ;;
        --interface1)
            export interface=$2
            shift
            ;;
        --interface2)
            export interface2=$2
            shift
            ;;
        --ip1)
            export ip1=$2
            shift
            ;;
        --ip2)
            export ip2=$2
            shift
            ;;
        --repetitions)
            export num_reps=$2
            shift
            ;;
        --domain)
            export domain=$2
            shift
            ;;
        --execution-time)
            export exec_time=$2
            shift
            ;;
        --transport)
            export transport=$2
            shift
            ;;
        --datalen)
            export datalen_input=$2
            shift
            ;;
        --file-suffix)
            export file_suffix=$2
            shift
            ;;
        --folder-suffix)
            export folder_suffix=$2
            shift
            ;;
        --sub-folder)
            sub_folder=$2
            shift
            ;;
        --executable-suffix)
            export executable_suffix=$2
            shift
            ;;
        --extra-arguments)
            export extra_arguments=$2
            shift
            ;;
        --extra-arguments-pub)
            export extra_arguments_pub=$2
            shift
            ;;
        --extra-arguments-sub)
            export extra_arguments_sub=$2
            shift
            ;;
        --skip-be)
            export skip_be_tests="1"
            ;;
        --skip-rel)
            export skip_rel_tests="1"
            ;;
        --skip-keyed)
            export skip_keyed_data="1"
            ;;
        --skip-large-data)
            export skip_large_data="1"
            ;;
        --large-data)
            export large_data="1"
            ;;
        --keyed)
            export skip_unkeyed="1"
            ;;
        --unkeyed)
            export skip_keyed_data="1"
            ;;
        --no-batching | --skip-batching)
            export run_batching_tests="0"
            export run_no_batching_tests="1"
            ;;
        --skip-no-batching)
            export run_batching_tests="1"
            export run_no_batching_tests="0"
            ;;
        --batch-size)
            export batch_size=$2
            shift
            ;;
        --reliable)
            export skip_be_tests="1"
            ;;
        --best-effort)
            export skip_rel_tests="1"
            ;;
        --security-gov)
            export security_only="$2"
            shift
            ;;
        --micro)
            export micro="1"
            ;;
        --cert)
            export cert="1"
            ;;
        --raw | --raw-transport)
            export raw="1"
            ;;
        --tss)
            export tss="1"
            ;;
        --no-colors)
            export NO_COLORS="1"
            ;;
        --language)
            export LANGUAGE=$2
            shift
            ;;
        --loss-rate)
            export loss_rate=$2
            shift
            ;;
        --get-netstat-info | --netstat)
            export get_netstat_info="1"
            ;;
        --reduced-data-sizes-set)
            export REDUCED_DATA_SIZES_SET="1"
            ;;
        --dont-change-tuned-settings)
            export dont_change_tuned_settings="1"
            ;;
        --no-taskset)
            export no_taskset=1
            ;;
        *)
            echo -e "unknown parameter \"$1\""
            exit 255
            ;;
    esac
    shift
done

if [[ "$NO_COLORS" == "1" ]]; then
    disable_colors
fi

export folder_base="$(dirname "${executable}")"/../../..

if [[ $LANGUAGE == "java"  || "$LANGUAGE" == "cs" ]]; then
    export folder_base="$(dirname "${executable}")"/../..
fi
if [[ $tss == "1" ]]; then
    export folder_base="$(dirname "${executable}")"/../../../../..
fi

if [[ "${executable_suffix}" != "" ]]; then
    export executable="${executable}${executable_suffix}"
fi

if [[ -n "${folder_suffix}" ]]; then
    export output_folder="${output_folder}${folder_suffix}"
fi

if [[ -n "${sub_folder}" ]]; then
    export output_folder="${output_folder}/${sub_folder}"
fi

echo -e "${INFO_TAG} Perftest executable is: $executable"
echo -e "${INFO_TAG} Output folder is: $output_folder"

################################################################################

if [[ "$LANGUAGE" == "python" ]]; then
    export skip_keyed_data="1"
    export skip_large_data="1"
    export skip_be_tests="1"
    export run_no_batching_tests="0"
fi

if [[ "${skip_large_data}" == "1" ]]; then
    export datasizes_extended=${datasizes}
elif [[ "${large_data}" == "1" ]]; then
    export datasizes=${datasizes_extended}
fi

if [[ "${datalen_input}" != "" ]]; then
    echo -e "${YELLOW}[TEST] Testing only for ${datalen_input}${NC}"
    export datasizes=${datalen_input}
    export datasizes_extended=${datalen_input}
    if [[ "${run_no_batching_tests}" == "1" && "${run_batching_tests}" == "0" ]]; then
        export skip_large_data="1"
    fi
else 
    if [[ "${REDUCED_DATA_SIZES_SET}" != "" ]]; then
        echo -e "${YELLOW}[TEST] Testing Reduced set of datasizes ${NC}"
        export datasizes="32 128 512 2048 8192 32768 63000"
        export datasizes_extended="${datasizes} 102400 1048576 10485760"
    fi
fi

if [[ "$role" != "pub" && "$role" != "sub" ]]; then
    echo -e "${ERROR_TAG} It must be either publisher or subscriber"
    exit 255
fi

if [[ "$lat_thr" != "thr" && "$lat_thr" != "lat" ]]; then
    echo -e "${ERROR_TAG} It must be either lat or thr"
    exit 255
fi

if [[ "${interface}" == "" ]]; then
    echo "Using default nics"
    export nic_publisher=${ip_machine_1}
    export nic_subscriber=${ip_machine_2}
elif [[ "${interface}" == "both" ]]; then
    export nic_publisher="enp1s0f0,eno1"
    export nic_subscriber="enp1s0f0,eno1"
    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"
    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"
else
    export nic_publisher=$interface
    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"

    if [[ "${interface2}" == "" ]]; then
        export nic_subscriber=$interface
    else
        export nic_subscriber=$interface2
    fi
    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"

    if [[ "${ip1}" != "" ]]; then
        export ip_publisher=$ip1
        echo "Using ip_publisher: ${ip_publisher}"
    fi

    if [[ "${ip2}" != "" ]]; then
        export ip_subscriber=$ip2
        echo "Using ip_subscriber: ${ip_subscriber}"
    fi

fi

export transport_string="-transport $transport"

if [[ "$transport" == "UDPv4" ]]; then

    export transport_string_pub="$transport_string -nic $nic_publisher"
    export transport_string_sub="$transport_string -nic $nic_subscriber"

    if [[ "$raw" == "1" ]]; then
        export transport_string_pub="$transport_string_pub -peer ${ip_subscriber}"
        export transport_string_sub="$transport_string_sub -peer ${ip_publisher}"
    fi

    if [[ "$micro" == "1" || "$cert" == "1" ]]; then
        export transport_string_pub="$transport_string_pub -peer _udp://${ip_subscriber}"
        export transport_string_sub="$transport_string_sub -peer _udp://${ip_publisher}"
    fi

elif [[ "$transport" == "TCP" ]]; then
    export transport_string_pub="$transport_string \
        -nic $nic_publisher \
        -peer 0@tcpv4_lan://${ip_subscriber}:7400"
    export transport_string_sub="$transport_string \
        -nic $nic_subscriber \
        -peer 0@tcpv4_lan://${ip_publisher}:7400"
elif [[ "$transport" == "TLS" ]]; then
    export transport_string_pub="$transport_string \
        -nic $nic_publisher \
        -peer tlsv4_lan://${ip_subscriber}:7400"
    export transport_string_sub="$transport_string \
        -nic $nic_subscriber \
        -peer tlsv4_lan://${ip_publisher}:7400"
elif [[ "$transport" == "UDPv4_WAN" ]]; then
    export transport_string_pub="$transport_string \
        -nic $nic_publisher \
        -transportPublicAddress $ip_publisher:7400"
    export transport_string_sub="$transport_string \
        -nic $nic_subscriber \
        -peer 0@udpv4_wan://${ip_publisher}:7400"
else
    export transport_string_pub="$transport_string"
    export transport_string_sub="$transport_string"
fi

################################################################################

export pub_string="-pub \
        ${transport_string_pub} \
        -noPrintIntervals \
        -executionTime $exec_time"

if [[ ${lat_thr} == "lat" ]]; then
    export pub_string="$pub_string \
        -latencyTest"
fi

export sub_string="-sub \
        ${transport_string_sub} \
        -noPrintIntervals"

if [[ "$role" == "pub" ]]; then
    echo -e "$INFO_TAG Publisher side running"
    export commands_string=${pub_string}
    export extra_arguments="${extra_arguments} ${extra_arguments_pub}"
else
    echo -e "$INFO_TAG Subscriber side running"
    export commands_string=${sub_string}
    export extra_arguments="${extra_arguments} ${extra_arguments_sub}"
fi

###############################################################################

if [[ "$dont_change_tuned_settings" != "1" ]]; then
    echo -e "${INFO_TAG} Executing: /set_${lat_thr}_mode.sh"
    sudo /set_${lat_thr}_mode.sh
    sleep 5
fi

echo -e "${INFO_TAG} Disabling any loss rate"
sudo tc qdisc add dev $nic_publisher root netem loss 0%
sudo tc qdisc del dev $nic_publisher root netem loss 0%

if [[ "$role" == "pub" && "${loss_rate}" != "" ]]; then
    echo -e "${INFO_TAG} Setting loss rate to ${loss_rate}%"
    sudo tc qdisc add dev $nic_publisher root netem loss $loss_rate%
fi

cd $folder_base
echo -e "${INFO_TAG} Folder Base is: $PWD"
mkdir -p $output_folder

if [[ "${batch_size}" != "" ]]; then
    if [[ "${lat_thr}" == "thr" ]]; then
        if [[ "${batch_size}" -eq 0 ]]; then
            echo -e "${INFO_TAG} Batch size is set to 0"
            export run_batching_tests="0"
            export run_no_batching_tests="1"
        else
            echo -e "${INFO_TAG} Batch size is set to ${batch_size}"
            export run_batching_tests="1"
            export run_no_batching_tests="0"
        fi
    else
        echo -e "${INFO_TAG} Batch size is set to ${batch_size}. This value will be ignored for latency tests."
        unset batch_size
    fi
fi

# Tests that may use batching (when doing throughput tests). Also Latency tests.
if [[ ${run_batching_tests} == "1" ]]; then

    # UNKEYED
    if [[ "${skip_unkeyed}" == "" ]]; then

        # RELIABLE
        if [[ "${skip_rel_tests}" == "" ]]; then
            execute_test "unkeyed" "rel" "${datasizes_extended}" "${extra_arguments}" "$file_suffix"
        fi

        # BEST EFFORT
        if [[ "${skip_be_tests}" == "" ]]; then
            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
        fi
    fi

    # KEYED
    if [[ "${skip_keyed_data}" == "" ]]; then

        # RELIABLE
        if [[ "${skip_rel_tests}" == "" ]]; then
            execute_test "keyed" "rel" "${datasizes}" "${extra_arguments}" "$file_suffix"
        fi

        # BEST EFFORT
        if [[ "${skip_be_tests}" == "" ]]; then
            execute_test "keyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
        fi
    fi

fi

# Tests that will not use batching
if [[ "${lat_thr}" == "thr" && "${run_no_batching_tests}" == "1" ]]; then

    if [[ "$role" == "pub" ]]; then
        export commands_string="${commands_string} -batchSize 0"
    fi

    # UNKEYED
    if [[ "${skip_unkeyed}" == "" ]]; then

        # RELIABLE
        if [[ "${skip_rel_tests}" == "" ]]; then
            execute_test "unkeyed" "rel" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
        fi

        # BEST EFFORT
        if [[ "${skip_be_tests}" == "" ]]; then
            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
        fi
    fi

    # KEYED
    if [[ "${skip_keyed_data}" == "" ]]; then

        # RELIABLE
        if [[ "${skip_rel_tests}" == "" ]]; then
            execute_test "keyed" "rel" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
        fi

        # BEST EFFORT
        if [[ "${skip_be_tests}" == "" ]]; then
            execute_test "keyed" "be" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
        fi
    fi

fi

if [[ "$role" == "pub" && "${loss_rate}" != "" ]]; then
    echo -e "${INFO_TAG} Disabling loss rate"
    sudo tc qdisc del dev $nic_publisher root netem loss $loss_rate%
fi

#!/bin/bash
filename=$0
script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)

echo -e "[Calling base_script/script.sh]"
"${script_location}/../base_script/script.sh" $@ --transport UDPv4

Test Software

The following software was used to perform these tests:

RTI Connext DDS 6.1.2 Host and Target Libraries for x64 Linux (x64Linux4gcc7.3.0)

Test Hardware

The following hardware was used to perform these tests:

Linux Nodes

Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 20.04 -- gcc 9.3.0

Switch

Dell 2048 -- 10Gbps switch (10Gbps and 1Gbps interfaces)

Keyed, UDPv4 1Gbps Network, C++98

Linux x64Linux ARM v8

Minimum LatencyMaximum Throughput

The graph below shows the one-way latency without load between a Publisher and a Subscriber running in two Linux nodes in a 1Gbps network. The numbers are for best-effort as well as strict reliable reliability scenarios.

Note

We use the median (50th percentile) instead of the average in order to get a more stable measurement that does not account for spurious outliers. We also calculate the average value and other percentile values, which can be seen in the Detailed Statistics section below.

Detailed Statistics

The following tables contain the raw numbers presented by RTI Perftest. These numbers are the exact output with no further processing.

• Best Effort

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)
32	58	1.7	53	207	58	60	62	88	207
64	59	1.6	54	209	59	61	62	89	209
128	61	1.7	55	208	61	62	64	90	208
256	65	1.3	60	213	65	66	68	94	213
512	71	1.5	64	216	70	72	73	99	216
1024	84	1.4	78	236	84	85	87	114	236
2048	111	1.4	105	255	111	112	114	142	255
4096	164	1.5	158	311	163	165	167	194	311
8192	269	1.7	262	414	269	270	273	299	414
16384	345	1.8	338	490	345	347	348	382	490
32768	478	1.9	471	624	478	479	481	519	624
63000	739	2.1	731	883	739	740	742	810	883

• Reliable

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)
32	61	1.7	54	206	60	62	66	89	206
64	61	1.8	55	208	61	63	66	91	208
128	63	1.6	57	212	62	64	68	92	212
256	67	1.5	61	217	67	68	71	96	217
512	73	1.6	67	221	72	74	78	102	221
1024	86	1.7	80	234	85	87	91	115	234
2048	113	1.5	107	188	113	115	118	142	188
4096	166	1.5	158	236	165	167	170	195	236
8192	271	1.6	265	326	271	272	275	303	326
16384	347	1.6	341	405	347	348	350	376	405
32768	479	1.5	472	541	479	481	482	507	541
63000	740	1.9	734	813	740	742	744	772	813

Perftest Scripts

To produce these tests, we executed RTI Perftest for C++98. The exact script used can be found here:

#!/bin/bash
filename=$0
script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)

export datasizes="32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000"
export datasizes_extended="${datasizes} 100000 500000 1048576 1548576 4194304 10485760"

export domain="2"
export exec_time=20
export num_reps=1
export instance_number=100000
export core=0
export run_batching_tests="1"
export run_no_batching_tests="1"

# We will use some colors to improve visibility of errors and info messages.
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
BLUE='\033[0;34m'
LIGHTBLUE='\033[0;36m'
NC='\033[0m'
INFO_TAG="${GREEN}[INFO][$role]:${NC}"
WARNING_TAG="${YELLOW}[WARNING][$role]:${NC}"
ERROR_TAG="${RED}[ERROR][$role]:${NC}"

################################################################################

function disable_colors() {
    export RED=""
    export GREEN=""
    export YELLOW=""
    export NC=""
    export BLUE=""
    export LIGHTBLUE=""
    export INFO_TAG="${GREEN}[INFO][$role]:${NC}"
    export WARNING_TAG="${YELLOW}[WARNING][$role]:${NC}"
    export ERROR_TAG="${RED}[ERROR][$role]:${NC}"
}

function change_domain() {
    if [[ "$domain" == "1" ]]; then
        export domain="2"
    else
        export domain="1"
    fi
}

# Function to process and append lines to the output file
function append_to_output_file() {
    local line=${1?line required}; readonly line
    local append=$(echo "${2?append required}" | tr -s ' '); readonly append
    local file=${3?file required}; readonly file
    if [[ "${line: -1}" == $'\n' ]]; then
        line="${line::-1}"
    fi
    echo "${line}${append}" >> "$file"
}

# Usage: execute_test <keyed/unkeyed> <rel/be> <datasizes> <batchSize>
function execute_test() {

    local keyed_unkeyed=$1
    local rel_be=$2
    local datasizes_test=$3
    local other_args=$4
    local name_suffix=$5

    local commands_string_test=$commands_string
    local tag=""

    if [[ "${keyed_unkeyed}" == "keyed" ]]; then
        commands_string_test="${commands_string_test} -keyed -instances $instance_number"
        tag="[${YELLOW}${transport}${NC}|${BLUE}K${NC}|"
    else
        tag="[${YELLOW}${transport}${NC}|${LIGHTBLUE}UK${NC}|"
    fi

    if [[ "${rel_be}" == "be" ]]; then
        commands_string_test="${commands_string_test} -bestEffort"
        tag="${tag}${YELLOW}BE${NC}]"
    else
        tag="${tag}${RED}REL${NC}]"
    fi

    # If batch_size is set, we will add it to the command line of the publisher side.
    if [[ "$batch_size" != "" && "$role" == "pub" ]]; then
        commands_string_test="${commands_string_test} -batchSize $batch_size"
    fi

    tag="${tag}[${LIGHTBLUE}${lat_thr}${NC}]"

    local output_file=$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}.csv

    if [[ "$role" == "pub" ]]; then
        echo -e "${YELLOW}[TEST]: $keyed_unkeyed, $rel_be, Is a no-batching test = $no_batching_tests. ${NC}"
    fi

    if [[ "$thread_cpu_affinity" != "" ]]; then
        commands_string_test="${commands_string_test} -threadCPUAffinity $thread_cpu_affinity"
    fi

    local pre_command_string=""

    if [[ "$thread_priorities" != "" ]]; then
        echo -e "${WARNING_TAG} Thread priorities enabled, this requires using sudo"
        # When using sudo, the LD_LIBRARY_PATH is emptied, hence we need to set it again in the command itself
        export LD_LIBRARY_PATH_COPIED="$LD_LIBRARY_PATH"
        pre_command_string="sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH_COPIED "
        commands_string_test="${commands_string_test} -threadPriorities $thread_priorities"
    fi

    if [[ "$pin_memory" != "" ]]; then
        commands_string_test="${commands_string_test} -pinMemory"
    fi

    if [[ "$no_taskset" == "" && "$thread_cpu_affinity" == "" && "$LANGUAGE" != "java" && "$LANGUAGE" != "cs" ]]; then
        pre_command_string="$pre_command_string taskset -c $core"
    fi

    if [[ "$LANGUAGE" == "python" ]]; then
        pre_command_string="$pre_command_string python3 "
    fi

    if [[ "$DOCKER" == "1" ]]; then
        pre_command_string="$pre_command_string docker run --net=host -v /home/perfuser/rti_license_connextpro.dat:/opt/rti.com/rti_connext_dds-7.3.0/rti_license.dat rticom/perftest:7.3.0-EAR "
        executable=""
    fi

    # Get the aprox time this will take:
    total_tests=$((`wc -w <<< "$datasizes_test"` * num_reps))
    total_time=$((total_tests * exec_time))

    touch $output_file
    local no_headers=""
    local current_test=0
    for index in $(seq 1 ${num_reps}); do
        for DATALEN in ${datasizes_test}; do
            current_test=$((current_test + 1))

            if [[ ! -s $output_file ]]; then
                echo -e "${INFO_TAG} Output file is empty, filling the header."
                no_headers=""
            else
                echo -e "${INFO_TAG} Output file is not empty."
                no_headers=" -noOutputHeaders"
            fi

            export command="$pre_command_string $executable -domain $domain -dataLen $DATALEN $commands_string_test $other_args $no_headers"
            if [[ "$role" == "pub" ]]; then
                echo -e "Test ${tag} (${current_test}/${total_tests}) -- Total time = ${total_time}s"
                echo -e ${BLUE}$command${NC}
            else
                echo -e ${LIGHTBLUE}$command${NC}
            fi

            # In certain cases we need to wait a bit before running the test, this is
            # because the previous test might not be finished on the other side yet, or because the
            # discovery mechanism does not work like in connext DDS.
            if [[ "$LANGUAGE" == "cs" && "$role" == "pub" ]]; then
                sleep 3
            fi
            if [[ "$raw" == "1" && "$role" == "sub" ]]; then
                sleep 5
            fi

            # Gather netstat info before running the test
            if [[ "${get_netstat_info}" == "1" ]]; then
                echo -e "${INFO_TAG} Getting netstat info before"
                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt
            fi

            # Execute the command and capture the output
            if [[ $LANGUAGE == "c++98" || $LANGUAGE == "" ]]; then
                echo -e "${INFO_TAG} Using C++98, using -outputFile option"
                touch ${output_file}_tmp_line
                chmod 777 ${output_file}_tmp_line
                eval $command -outputFile ${output_file}_tmp_line
            else
                eval $command > ${output_file}_tmp_line
            fi

            number_of_lines=$(wc -l < ${output_file}_tmp_line)
            echo -e "${INFO_TAG} Size of the output text: $number_of_lines lines"

            # Check if this is the first test for this output file.
            local is_first_test=0
            if [[ ! -s $output_file ]]; then
                is_first_test=1
            fi
            local expected_lines=$((is_first_test+1))

            if [[ $number_of_lines -ne ${expected_lines} ]]; then
                echo -e "${WARNING_TAG} The output text should have ${expected_lines} lines, but it has ${number_of_lines}."
                echo -e "${RED} The content is: \n\"\"\""
                cat ${output_file}_tmp_line
                echo -e "\"\"\""
                echo -e "Not adding it to the output file. ${NC}"
            else
                if [[ $number_of_lines -gt 1 ]]; then
                    header_line=$(head -n 1 "${output_file}_tmp_line")
                    echo -e "${INFO_TAG} Header line is: $header_line"
                fi
                result_line=$(tail -n 1 "${output_file}_tmp_line")
                echo -e "${INFO_TAG} Result line is: $result_line"

                # If this is the first test, we need to add the header line to the output file
                if [[ $is_first_test -eq 1 ]]; then
                    append_to_output_file "$header_line" ", command-line" $output_file
                fi

                command=$(echo $command | sed 's/,/:comma:/g')
                # Remove the `"` characters from the command line
                command=$(echo $command | sed 's/\"//g')

                # Append the result line to the output file
                append_to_output_file "$result_line" ", $command" $output_file
            fi

            # Always remove the temporary file
            rm -rf ${output_file}_tmp_line

            # Gather netstat info after running the test
            if [[ "${get_netstat_info}" == "1" ]]; then
                echo -e "${INFO_TAG} Getting netstat info after"
                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt
                touch "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
                python3 $script_location/../../../tools/diff_netstat_output.py \
                    -n $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt \
                    -o $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt \
                    -d $DATALEN $no_header_netstat \
                    -csv >> "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
                rm -rf $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_*.txt
                no_header_netstat=" -nh"
            fi

            change_domain
        done
    done
}

################################################################################
# PARSE COMMAND LINE OPTIONS:

while [ "$1" != "" ]; do
    case $1 in
        --executable)
            executable=$2
            shift
            ;;
        --docker)
            DOCKER="1"
            ;;
        --output-folder)
            output_folder=$2
            shift
            ;;
        --role)
            export role=$2
            shift
            ;;
        --core)
            export core=$2
            shift
            ;;
        --assign-core-list | --thread-cpu-affinity)
            export thread_cpu_affinity=$2
            shift
            ;;
        --assign-thread-priorities | --thread-priorities)
            export thread_priorities=$2
            shift
            ;;
        --pin-memory)
            export pin_memory="1"
            ;;
        --test-kind)
            export lat_thr=$2
            shift
            ;;
        --interface1)
            export interface=$2
            shift
            ;;
        --interface2)
            export interface2=$2
            shift
            ;;
        --ip1)
            export ip1=$2
            shift
            ;;
        --ip2)
            export ip2=$2
            shift
            ;;
        --repetitions)
            export num_reps=$2
            shift
            ;;
        --domain)
            export domain=$2
            shift
            ;;
        --execution-time)
            export exec_time=$2
            shift
            ;;
        --transport)
            export transport=$2
            shift
            ;;
        --datalen)
            export datalen_input=$2
            shift
            ;;
        --file-suffix)
            export file_suffix=$2
            shift
            ;;
        --folder-suffix)
            export folder_suffix=$2
            shift
            ;;
        --sub-folder)
            sub_folder=$2
            shift
            ;;
        --executable-suffix)
            export executable_suffix=$2
            shift
            ;;
        --extra-arguments)
            export extra_arguments=$2
            shift
            ;;
        --extra-arguments-pub)
            export extra_arguments_pub=$2
            shift
            ;;
        --extra-arguments-sub)
            export extra_arguments_sub=$2
            shift
            ;;
        --skip-be)
            export skip_be_tests="1"
            ;;
        --skip-rel)
            export skip_rel_tests="1"
            ;;
        --skip-keyed)
            export skip_keyed_data="1"
            ;;
        --skip-large-data)
            export skip_large_data="1"
            ;;
        --large-data)
            export large_data="1"
            ;;
        --keyed)
            export skip_unkeyed="1"
            ;;
        --unkeyed)
            export skip_keyed_data="1"
            ;;
        --no-batching | --skip-batching)
            export run_batching_tests="0"
            export run_no_batching_tests="1"
            ;;
        --skip-no-batching)
            export run_batching_tests="1"
            export run_no_batching_tests="0"
            ;;
        --batch-size)
            export batch_size=$2
            shift
            ;;
        --reliable)
            export skip_be_tests="1"
            ;;
        --best-effort)
            export skip_rel_tests="1"
            ;;
        --security-gov)
            export security_only="$2"
            shift
            ;;
        --micro)
            export micro="1"
            ;;
        --cert)
            export cert="1"
            ;;
        --raw | --raw-transport)
            export raw="1"
            ;;
        --tss)
            export tss="1"
            ;;
        --no-colors)
            export NO_COLORS="1"
            ;;
        --language)
            export LANGUAGE=$2
            shift
            ;;
        --loss-rate)
            export loss_rate=$2
            shift
            ;;
        --get-netstat-info | --netstat)
            export get_netstat_info="1"
            ;;
        --reduced-data-sizes-set)
            export REDUCED_DATA_SIZES_SET="1"
            ;;
        --dont-change-tuned-settings)
            export dont_change_tuned_settings="1"
            ;;
        --no-taskset)
            export no_taskset=1
            ;;
        *)
            echo -e "unknown parameter \"$1\""
            exit 255
            ;;
    esac
    shift
done

if [[ "$NO_COLORS" == "1" ]]; then
    disable_colors
fi

export folder_base="$(dirname "${executable}")"/../../..

if [[ $LANGUAGE == "java"  || "$LANGUAGE" == "cs" ]]; then
    export folder_base="$(dirname "${executable}")"/../..
fi
if [[ $tss == "1" ]]; then
    export folder_base="$(dirname "${executable}")"/../../../../..
fi

if [[ "${executable_suffix}" != "" ]]; then
    export executable="${executable}${executable_suffix}"
fi

if [[ -n "${folder_suffix}" ]]; then
    export output_folder="${output_folder}${folder_suffix}"
fi

if [[ -n "${sub_folder}" ]]; then
    export output_folder="${output_folder}/${sub_folder}"
fi

echo -e "${INFO_TAG} Perftest executable is: $executable"
echo -e "${INFO_TAG} Output folder is: $output_folder"

################################################################################

if [[ "$LANGUAGE" == "python" ]]; then
    export skip_keyed_data="1"
    export skip_large_data="1"
    export skip_be_tests="1"
    export run_no_batching_tests="0"
fi

if [[ "${skip_large_data}" == "1" ]]; then
    export datasizes_extended=${datasizes}
elif [[ "${large_data}" == "1" ]]; then
    export datasizes=${datasizes_extended}
fi

if [[ "${datalen_input}" != "" ]]; then
    echo -e "${YELLOW}[TEST] Testing only for ${datalen_input}${NC}"
    export datasizes=${datalen_input}
    export datasizes_extended=${datalen_input}
    if [[ "${run_no_batching_tests}" == "1" && "${run_batching_tests}" == "0" ]]; then
        export skip_large_data="1"
    fi
else 
    if [[ "${REDUCED_DATA_SIZES_SET}" != "" ]]; then
        echo -e "${YELLOW}[TEST] Testing Reduced set of datasizes ${NC}"
        export datasizes="32 128 512 2048 8192 32768 63000"
        export datasizes_extended="${datasizes} 102400 1048576 10485760"
    fi
fi

if [[ "$role" != "pub" && "$role" != "sub" ]]; then
    echo -e "${ERROR_TAG} It must be either publisher or subscriber"
    exit 255
fi

if [[ "$lat_thr" != "thr" && "$lat_thr" != "lat" ]]; then
    echo -e "${ERROR_TAG} It must be either lat or thr"
    exit 255
fi

if [[ "${interface}" == "" ]]; then
    echo "Using default nics"
    export nic_publisher=${ip_machine_1}
    export nic_subscriber=${ip_machine_2}
elif [[ "${interface}" == "both" ]]; then
    export nic_publisher="enp1s0f0,eno1"
    export nic_subscriber="enp1s0f0,eno1"
    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"
    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"
else
    export nic_publisher=$interface
    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"

    if [[ "${interface2}" == "" ]]; then
        export nic_subscriber=$interface
    else
        export nic_subscriber=$interface2
    fi
    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"

    if [[ "${ip1}" != "" ]]; then
        export ip_publisher=$ip1
        echo "Using ip_publisher: ${ip_publisher}"
    fi

    if [[ "${ip2}" != "" ]]; then
        export ip_subscriber=$ip2
        echo "Using ip_subscriber: ${ip_subscriber}"
    fi

fi

export transport_string="-transport $transport"

if [[ "$transport" == "UDPv4" ]]; then

    export transport_string_pub="$transport_string -nic $nic_publisher"
    export transport_string_sub="$transport_string -nic $nic_subscriber"

    if [[ "$raw" == "1" ]]; then
        export transport_string_pub="$transport_string_pub -peer ${ip_subscriber}"
        export transport_string_sub="$transport_string_sub -peer ${ip_publisher}"
    fi

    if [[ "$micro" == "1" || "$cert" == "1" ]]; then
        export transport_string_pub="$transport_string_pub -peer _udp://${ip_subscriber}"
        export transport_string_sub="$transport_string_sub -peer _udp://${ip_publisher}"
    fi

elif [[ "$transport" == "TCP" ]]; then
    export transport_string_pub="$transport_string \
        -nic $nic_publisher \
        -peer 0@tcpv4_lan://${ip_subscriber}:7400"
    export transport_string_sub="$transport_string \
        -nic $nic_subscriber \
        -peer 0@tcpv4_lan://${ip_publisher}:7400"
elif [[ "$transport" == "TLS" ]]; then
    export transport_string_pub="$transport_string \
        -nic $nic_publisher \
        -peer tlsv4_lan://${ip_subscriber}:7400"
    export transport_string_sub="$transport_string \
        -nic $nic_subscriber \
        -peer tlsv4_lan://${ip_publisher}:7400"
elif [[ "$transport" == "UDPv4_WAN" ]]; then
    export transport_string_pub="$transport_string \
        -nic $nic_publisher \
        -transportPublicAddress $ip_publisher:7400"
    export transport_string_sub="$transport_string \
        -nic $nic_subscriber \
        -peer 0@udpv4_wan://${ip_publisher}:7400"
else
    export transport_string_pub="$transport_string"
    export transport_string_sub="$transport_string"
fi

################################################################################

export pub_string="-pub \
        ${transport_string_pub} \
        -noPrintIntervals \
        -executionTime $exec_time"

if [[ ${lat_thr} == "lat" ]]; then
    export pub_string="$pub_string \
        -latencyTest"
fi

export sub_string="-sub \
        ${transport_string_sub} \
        -noPrintIntervals"

if [[ "$role" == "pub" ]]; then
    echo -e "$INFO_TAG Publisher side running"
    export commands_string=${pub_string}
    export extra_arguments="${extra_arguments} ${extra_arguments_pub}"
else
    echo -e "$INFO_TAG Subscriber side running"
    export commands_string=${sub_string}
    export extra_arguments="${extra_arguments} ${extra_arguments_sub}"
fi

###############################################################################

if [[ "$dont_change_tuned_settings" != "1" ]]; then
    echo -e "${INFO_TAG} Executing: /set_${lat_thr}_mode.sh"
    sudo /set_${lat_thr}_mode.sh
    sleep 5
fi

echo -e "${INFO_TAG} Disabling any loss rate"
sudo tc qdisc add dev $nic_publisher root netem loss 0%
sudo tc qdisc del dev $nic_publisher root netem loss 0%

if [[ "$role" == "pub" && "${loss_rate}" != "" ]]; then
    echo -e "${INFO_TAG} Setting loss rate to ${loss_rate}%"
    sudo tc qdisc add dev $nic_publisher root netem loss $loss_rate%
fi

cd $folder_base
echo -e "${INFO_TAG} Folder Base is: $PWD"
mkdir -p $output_folder

if [[ "${batch_size}" != "" ]]; then
    if [[ "${lat_thr}" == "thr" ]]; then
        if [[ "${batch_size}" -eq 0 ]]; then
            echo -e "${INFO_TAG} Batch size is set to 0"
            export run_batching_tests="0"
            export run_no_batching_tests="1"
        else
            echo -e "${INFO_TAG} Batch size is set to ${batch_size}"
            export run_batching_tests="1"
            export run_no_batching_tests="0"
        fi
    else
        echo -e "${INFO_TAG} Batch size is set to ${batch_size}. This value will be ignored for latency tests."
        unset batch_size
    fi
fi

# Tests that may use batching (when doing throughput tests). Also Latency tests.
if [[ ${run_batching_tests} == "1" ]]; then

    # UNKEYED
    if [[ "${skip_unkeyed}" == "" ]]; then

        # RELIABLE
        if [[ "${skip_rel_tests}" == "" ]]; then
            execute_test "unkeyed" "rel" "${datasizes_extended}" "${extra_arguments}" "$file_suffix"
        fi

        # BEST EFFORT
        if [[ "${skip_be_tests}" == "" ]]; then
            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
        fi
    fi

    # KEYED
    if [[ "${skip_keyed_data}" == "" ]]; then

        # RELIABLE
        if [[ "${skip_rel_tests}" == "" ]]; then
            execute_test "keyed" "rel" "${datasizes}" "${extra_arguments}" "$file_suffix"
        fi

        # BEST EFFORT
        if [[ "${skip_be_tests}" == "" ]]; then
            execute_test "keyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
        fi
    fi

fi

# Tests that will not use batching
if [[ "${lat_thr}" == "thr" && "${run_no_batching_tests}" == "1" ]]; then

    if [[ "$role" == "pub" ]]; then
        export commands_string="${commands_string} -batchSize 0"
    fi

    # UNKEYED
    if [[ "${skip_unkeyed}" == "" ]]; then

        # RELIABLE
        if [[ "${skip_rel_tests}" == "" ]]; then
            execute_test "unkeyed" "rel" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
        fi

        # BEST EFFORT
        if [[ "${skip_be_tests}" == "" ]]; then
            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
        fi
    fi

    # KEYED
    if [[ "${skip_keyed_data}" == "" ]]; then

        # RELIABLE
        if [[ "${skip_rel_tests}" == "" ]]; then
            execute_test "keyed" "rel" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
        fi

        # BEST EFFORT
        if [[ "${skip_be_tests}" == "" ]]; then
            execute_test "keyed" "be" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
        fi
    fi

fi

if [[ "$role" == "pub" && "${loss_rate}" != "" ]]; then
    echo -e "${INFO_TAG} Disabling loss rate"
    sudo tc qdisc del dev $nic_publisher root netem loss $loss_rate%
fi

#!/bin/bash
filename=$0
script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)

echo -e "[Calling base_script/script.sh]"
"${script_location}/../base_script/script.sh" $@ --transport UDPv4

Test Software

The following software was used to perform these tests:

RTI Connext DDS 6.1.1 Host and Target Libraries for x64 Linux (x64Linux4gcc7.3.0)

Test Hardware

The following hardware was used to perform these tests:

Linux Nodes

Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 20.04 -- gcc 9.3.0

Switch

Dell 2048 -- 10Gbps switch (10Gbps and 1Gbps interfaces)

Keyed, UDPv4 10Gbps Network, C++98

Linux x64

Minimum LatencyMaximum Throughput

The graph below shows the one-way latency without load between a Publisher and a Subscriber running in two Linux nodes in a 10Gbps network. The numbers are for best-effort as well as strict reliable reliability scenarios.

Note

We use the median (50th percentile) instead of the average in order to get a more stable measurement that does not account for spurious outliers. We also calculate the average value and other percentile values, which can be seen in the Detailed Statistics section below.

Detailed Statistics

The following tables contain the raw numbers presented by RTI Perftest. These numbers are the exact output with no further processing.

• Best Effort

Sample Size (Bytes)	Ave (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)
32	17	1.0	17	167	17	18	20	47	167
64	17	1.2	17	170	17	18	21	46	170
128	18	1.1	17	167	18	19	21	48	167
256	18	1.1	17	165	18	19	21	48	165
512	19	1.2	17	175	19	20	22	49	175
1024	25	3.5	19	175	26	28	29	56	175
2048	32	171.7	21	100070	30	61	64	79	100070
4096	33	158.2	26	100062	30	44	58	70	100062
8192	46	214.6	35	100070	41	60	64	85	100070
16384	93	25.2	48	276	82	126	131	151	276
32768	176	2.5	70	321	176	177	179	206	321
63000	188	2.8	105	331	188	189	191	217	331

• Reliable

Sample Size (Bytes)	Ave (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)
32	19	1.2	18	166	19	20	23	49	166
64	20	1.3	18	166	19	20	24	50	166
128	19	1.3	18	167	19	20	25	49	167
256	20	5.0	18	3169	19	21	27	50	3169
512	21	2.3	19	171	20	22	29	51	171
1024	26	8.5	20	3437	26	28	46	58	3437
2048	29	14.5	22	4215	30	32	43	66	4215
4096	31	18.9	27	5027	30	33	51	66	5027
8192	44	48.3	37	4854	39	55	73	3352	4854
16384	93	24.9	48	270	86	126	132	152	270
32768	162	15.8	78	244	167	177	180	205	244
63000	189	2.7	108	260	188	190	194	219	260

Perftest Scripts

To produce these tests, we executed RTI Perftest for C++98. The exact script used can be found here:

#!/bin/bash
filename=$0
script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)

export datasizes="32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000"
export datasizes_extended="${datasizes} 100000 500000 1048576 1548576 4194304 10485760"

export domain="2"
export exec_time=20
export num_reps=1
export instance_number=100000
export core=0
export run_batching_tests="1"
export run_no_batching_tests="1"

# We will use some colors to improve visibility of errors and info messages.
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
BLUE='\033[0;34m'
LIGHTBLUE='\033[0;36m'
NC='\033[0m'
INFO_TAG="${GREEN}[INFO][$role]:${NC}"
WARNING_TAG="${YELLOW}[WARNING][$role]:${NC}"
ERROR_TAG="${RED}[ERROR][$role]:${NC}"

################################################################################

function disable_colors() {
    export RED=""
    export GREEN=""
    export YELLOW=""
    export NC=""
    export BLUE=""
    export LIGHTBLUE=""
    export INFO_TAG="${GREEN}[INFO][$role]:${NC}"
    export WARNING_TAG="${YELLOW}[WARNING][$role]:${NC}"
    export ERROR_TAG="${RED}[ERROR][$role]:${NC}"
}

function change_domain() {
    if [[ "$domain" == "1" ]]; then
        export domain="2"
    else
        export domain="1"
    fi
}

# Function to process and append lines to the output file
function append_to_output_file() {
    local line=${1?line required}; readonly line
    local append=$(echo "${2?append required}" | tr -s ' '); readonly append
    local file=${3?file required}; readonly file
    if [[ "${line: -1}" == $'\n' ]]; then
        line="${line::-1}"
    fi
    echo "${line}${append}" >> "$file"
}

# Usage: execute_test <keyed/unkeyed> <rel/be> <datasizes> <batchSize>
function execute_test() {

    local keyed_unkeyed=$1
    local rel_be=$2
    local datasizes_test=$3
    local other_args=$4
    local name_suffix=$5

    local commands_string_test=$commands_string
    local tag=""

    if [[ "${keyed_unkeyed}" == "keyed" ]]; then
        commands_string_test="${commands_string_test} -keyed -instances $instance_number"
        tag="[${YELLOW}${transport}${NC}|${BLUE}K${NC}|"
    else
        tag="[${YELLOW}${transport}${NC}|${LIGHTBLUE}UK${NC}|"
    fi

    if [[ "${rel_be}" == "be" ]]; then
        commands_string_test="${commands_string_test} -bestEffort"
        tag="${tag}${YELLOW}BE${NC}]"
    else
        tag="${tag}${RED}REL${NC}]"
    fi

    # If batch_size is set, we will add it to the command line of the publisher side.
    if [[ "$batch_size" != "" && "$role" == "pub" ]]; then
        commands_string_test="${commands_string_test} -batchSize $batch_size"
    fi

    tag="${tag}[${LIGHTBLUE}${lat_thr}${NC}]"

    local output_file=$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}.csv

    if [[ "$role" == "pub" ]]; then
        echo -e "${YELLOW}[TEST]: $keyed_unkeyed, $rel_be, Is a no-batching test = $no_batching_tests. ${NC}"
    fi

    if [[ "$thread_cpu_affinity" != "" ]]; then
        commands_string_test="${commands_string_test} -threadCPUAffinity $thread_cpu_affinity"
    fi

    local pre_command_string=""

    if [[ "$thread_priorities" != "" ]]; then
        echo -e "${WARNING_TAG} Thread priorities enabled, this requires using sudo"
        # When using sudo, the LD_LIBRARY_PATH is emptied, hence we need to set it again in the command itself
        export LD_LIBRARY_PATH_COPIED="$LD_LIBRARY_PATH"
        pre_command_string="sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH_COPIED "
        commands_string_test="${commands_string_test} -threadPriorities $thread_priorities"
    fi

    if [[ "$pin_memory" != "" ]]; then
        commands_string_test="${commands_string_test} -pinMemory"
    fi

    if [[ "$no_taskset" == "" && "$thread_cpu_affinity" == "" && "$LANGUAGE" != "java" && "$LANGUAGE" != "cs" ]]; then
        pre_command_string="$pre_command_string taskset -c $core"
    fi

    if [[ "$LANGUAGE" == "python" ]]; then
        pre_command_string="$pre_command_string python3 "
    fi

    if [[ "$DOCKER" == "1" ]]; then
        pre_command_string="$pre_command_string docker run --net=host -v /home/perfuser/rti_license_connextpro.dat:/opt/rti.com/rti_connext_dds-7.3.0/rti_license.dat rticom/perftest:7.3.0-EAR "
        executable=""
    fi

    # Get the aprox time this will take:
    total_tests=$((`wc -w <<< "$datasizes_test"` * num_reps))
    total_time=$((total_tests * exec_time))

    touch $output_file
    local no_headers=""
    local current_test=0
    for index in $(seq 1 ${num_reps}); do
        for DATALEN in ${datasizes_test}; do
            current_test=$((current_test + 1))

            if [[ ! -s $output_file ]]; then
                echo -e "${INFO_TAG} Output file is empty, filling the header."
                no_headers=""
            else
                echo -e "${INFO_TAG} Output file is not empty."
                no_headers=" -noOutputHeaders"
            fi

            export command="$pre_command_string $executable -domain $domain -dataLen $DATALEN $commands_string_test $other_args $no_headers"
            if [[ "$role" == "pub" ]]; then
                echo -e "Test ${tag} (${current_test}/${total_tests}) -- Total time = ${total_time}s"
                echo -e ${BLUE}$command${NC}
            else
                echo -e ${LIGHTBLUE}$command${NC}
            fi

            # In certain cases we need to wait a bit before running the test, this is
            # because the previous test might not be finished on the other side yet, or because the
            # discovery mechanism does not work like in connext DDS.
            if [[ "$LANGUAGE" == "cs" && "$role" == "pub" ]]; then
                sleep 3
            fi
            if [[ "$raw" == "1" && "$role" == "sub" ]]; then
                sleep 5
            fi

            # Gather netstat info before running the test
            if [[ "${get_netstat_info}" == "1" ]]; then
                echo -e "${INFO_TAG} Getting netstat info before"
                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt
            fi

            # Execute the command and capture the output
            if [[ $LANGUAGE == "c++98" || $LANGUAGE == "" ]]; then
                echo -e "${INFO_TAG} Using C++98, using -outputFile option"
                touch ${output_file}_tmp_line
                chmod 777 ${output_file}_tmp_line
                eval $command -outputFile ${output_file}_tmp_line
            else
                eval $command > ${output_file}_tmp_line
            fi

            number_of_lines=$(wc -l < ${output_file}_tmp_line)
            echo -e "${INFO_TAG} Size of the output text: $number_of_lines lines"

            # Check if this is the first test for this output file.
            local is_first_test=0
            if [[ ! -s $output_file ]]; then
                is_first_test=1
            fi
            local expected_lines=$((is_first_test+1))

            if [[ $number_of_lines -ne ${expected_lines} ]]; then
                echo -e "${WARNING_TAG} The output text should have ${expected_lines} lines, but it has ${number_of_lines}."
                echo -e "${RED} The content is: \n\"\"\""
                cat ${output_file}_tmp_line
                echo -e "\"\"\""
                echo -e "Not adding it to the output file. ${NC}"
            else
                if [[ $number_of_lines -gt 1 ]]; then
                    header_line=$(head -n 1 "${output_file}_tmp_line")
                    echo -e "${INFO_TAG} Header line is: $header_line"
                fi
                result_line=$(tail -n 1 "${output_file}_tmp_line")
                echo -e "${INFO_TAG} Result line is: $result_line"

                # If this is the first test, we need to add the header line to the output file
                if [[ $is_first_test -eq 1 ]]; then
                    append_to_output_file "$header_line" ", command-line" $output_file
                fi

                command=$(echo $command | sed 's/,/:comma:/g')
                # Remove the `"` characters from the command line
                command=$(echo $command | sed 's/\"//g')

                # Append the result line to the output file
                append_to_output_file "$result_line" ", $command" $output_file
            fi

            # Always remove the temporary file
            rm -rf ${output_file}_tmp_line

            # Gather netstat info after running the test
            if [[ "${get_netstat_info}" == "1" ]]; then
                echo -e "${INFO_TAG} Getting netstat info after"
                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt
                touch "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
                python3 $script_location/../../../tools/diff_netstat_output.py \
                    -n $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt \
                    -o $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt \
                    -d $DATALEN $no_header_netstat \
                    -csv >> "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
                rm -rf $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_*.txt
                no_header_netstat=" -nh"
            fi

            change_domain
        done
    done
}

################################################################################
# PARSE COMMAND LINE OPTIONS:

while [ "$1" != "" ]; do
    case $1 in
        --executable)
            executable=$2
            shift
            ;;
        --docker)
            DOCKER="1"
            ;;
        --output-folder)
            output_folder=$2
            shift
            ;;
        --role)
            export role=$2
            shift
            ;;
        --core)
            export core=$2
            shift
            ;;
        --assign-core-list | --thread-cpu-affinity)
            export thread_cpu_affinity=$2
            shift
            ;;
        --assign-thread-priorities | --thread-priorities)
            export thread_priorities=$2
            shift
            ;;
        --pin-memory)
            export pin_memory="1"
            ;;
        --test-kind)
            export lat_thr=$2
            shift
            ;;
        --interface1)
            export interface=$2
            shift
            ;;
        --interface2)
            export interface2=$2
            shift
            ;;
        --ip1)
            export ip1=$2
            shift
            ;;
        --ip2)
            export ip2=$2
            shift
            ;;
        --repetitions)
            export num_reps=$2
            shift
            ;;
        --domain)
            export domain=$2
            shift
            ;;
        --execution-time)
            export exec_time=$2
            shift
            ;;
        --transport)
            export transport=$2
            shift
            ;;
        --datalen)
            export datalen_input=$2
            shift
            ;;
        --file-suffix)
            export file_suffix=$2
            shift
            ;;
        --folder-suffix)
            export folder_suffix=$2
            shift
            ;;
        --sub-folder)
            sub_folder=$2
            shift
            ;;
        --executable-suffix)
            export executable_suffix=$2
            shift
            ;;
        --extra-arguments)
            export extra_arguments=$2
            shift
            ;;
        --extra-arguments-pub)
            export extra_arguments_pub=$2
            shift
            ;;
        --extra-arguments-sub)
            export extra_arguments_sub=$2
            shift
            ;;
        --skip-be)
            export skip_be_tests="1"
            ;;
        --skip-rel)
            export skip_rel_tests="1"
            ;;
        --skip-keyed)
            export skip_keyed_data="1"
            ;;
        --skip-large-data)
            export skip_large_data="1"
            ;;
        --large-data)
            export large_data="1"
            ;;
        --keyed)
            export skip_unkeyed="1"
            ;;
        --unkeyed)
            export skip_keyed_data="1"
            ;;
        --no-batching | --skip-batching)
            export run_batching_tests="0"
            export run_no_batching_tests="1"
            ;;
        --skip-no-batching)
            export run_batching_tests="1"
            export run_no_batching_tests="0"
            ;;
        --batch-size)
            export batch_size=$2
            shift
            ;;
        --reliable)
            export skip_be_tests="1"
            ;;
        --best-effort)
            export skip_rel_tests="1"
            ;;
        --security-gov)
            export security_only="$2"
            shift
            ;;
        --micro)
            export micro="1"
            ;;
        --cert)
            export cert="1"
            ;;
        --raw | --raw-transport)
            export raw="1"
            ;;
        --tss)
            export tss="1"
            ;;
        --no-colors)
            export NO_COLORS="1"
            ;;
        --language)
            export LANGUAGE=$2
            shift
            ;;
        --loss-rate)
            export loss_rate=$2
            shift
            ;;
        --get-netstat-info | --netstat)
            export get_netstat_info="1"
            ;;
        --reduced-data-sizes-set)
            export REDUCED_DATA_SIZES_SET="1"
            ;;
        --dont-change-tuned-settings)
            export dont_change_tuned_settings="1"
            ;;
        --no-taskset)
            export no_taskset=1
            ;;
        *)
            echo -e "unknown parameter \"$1\""
            exit 255
            ;;
    esac
    shift
done

if [[ "$NO_COLORS" == "1" ]]; then
    disable_colors
fi

export folder_base="$(dirname "${executable}")"/../../..

if [[ $LANGUAGE == "java"  || "$LANGUAGE" == "cs" ]]; then
    export folder_base="$(dirname "${executable}")"/../..
fi
if [[ $tss == "1" ]]; then
    export folder_base="$(dirname "${executable}")"/../../../../..
fi

if [[ "${executable_suffix}" != "" ]]; then
    export executable="${executable}${executable_suffix}"
fi

if [[ -n "${folder_suffix}" ]]; then
    export output_folder="${output_folder}${folder_suffix}"
fi

if [[ -n "${sub_folder}" ]]; then
    export output_folder="${output_folder}/${sub_folder}"
fi

echo -e "${INFO_TAG} Perftest executable is: $executable"
echo -e "${INFO_TAG} Output folder is: $output_folder"

################################################################################

if [[ "$LANGUAGE" == "python" ]]; then
    export skip_keyed_data="1"
    export skip_large_data="1"
    export skip_be_tests="1"
    export run_no_batching_tests="0"
fi

if [[ "${skip_large_data}" == "1" ]]; then
    export datasizes_extended=${datasizes}
elif [[ "${large_data}" == "1" ]]; then
    export datasizes=${datasizes_extended}
fi

if [[ "${datalen_input}" != "" ]]; then
    echo -e "${YELLOW}[TEST] Testing only for ${datalen_input}${NC}"
    export datasizes=${datalen_input}
    export datasizes_extended=${datalen_input}
    if [[ "${run_no_batching_tests}" == "1" && "${run_batching_tests}" == "0" ]]; then
        export skip_large_data="1"
    fi
else 
    if [[ "${REDUCED_DATA_SIZES_SET}" != "" ]]; then
        echo -e "${YELLOW}[TEST] Testing Reduced set of datasizes ${NC}"
        export datasizes="32 128 512 2048 8192 32768 63000"
        export datasizes_extended="${datasizes} 102400 1048576 10485760"
    fi
fi

if [[ "$role" != "pub" && "$role" != "sub" ]]; then
    echo -e "${ERROR_TAG} It must be either publisher or subscriber"
    exit 255
fi

if [[ "$lat_thr" != "thr" && "$lat_thr" != "lat" ]]; then
    echo -e "${ERROR_TAG} It must be either lat or thr"
    exit 255
fi

if [[ "${interface}" == "" ]]; then
    echo "Using default nics"
    export nic_publisher=${ip_machine_1}
    export nic_subscriber=${ip_machine_2}
elif [[ "${interface}" == "both" ]]; then
    export nic_publisher="enp1s0f0,eno1"
    export nic_subscriber="enp1s0f0,eno1"
    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"
    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"
else
    export nic_publisher=$interface
    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"

    if [[ "${interface2}" == "" ]]; then
        export nic_subscriber=$interface
    else
        export nic_subscriber=$interface2
    fi
    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"

    if [[ "${ip1}" != "" ]]; then
        export ip_publisher=$ip1
        echo "Using ip_publisher: ${ip_publisher}"
    fi

    if [[ "${ip2}" != "" ]]; then
        export ip_subscriber=$ip2
        echo "Using ip_subscriber: ${ip_subscriber}"
    fi

fi

export transport_string="-transport $transport"

if [[ "$transport" == "UDPv4" ]]; then

    export transport_string_pub="$transport_string -nic $nic_publisher"
    export transport_string_sub="$transport_string -nic $nic_subscriber"

    if [[ "$raw" == "1" ]]; then
        export transport_string_pub="$transport_string_pub -peer ${ip_subscriber}"
        export transport_string_sub="$transport_string_sub -peer ${ip_publisher}"
    fi

    if [[ "$micro" == "1" || "$cert" == "1" ]]; then
        export transport_string_pub="$transport_string_pub -peer _udp://${ip_subscriber}"
        export transport_string_sub="$transport_string_sub -peer _udp://${ip_publisher}"
    fi

elif [[ "$transport" == "TCP" ]]; then
    export transport_string_pub="$transport_string \
        -nic $nic_publisher \
        -peer 0@tcpv4_lan://${ip_subscriber}:7400"
    export transport_string_sub="$transport_string \
        -nic $nic_subscriber \
        -peer 0@tcpv4_lan://${ip_publisher}:7400"
elif [[ "$transport" == "TLS" ]]; then
    export transport_string_pub="$transport_string \
        -nic $nic_publisher \
        -peer tlsv4_lan://${ip_subscriber}:7400"
    export transport_string_sub="$transport_string \
        -nic $nic_subscriber \
        -peer tlsv4_lan://${ip_publisher}:7400"
elif [[ "$transport" == "UDPv4_WAN" ]]; then
    export transport_string_pub="$transport_string \
        -nic $nic_publisher \
        -transportPublicAddress $ip_publisher:7400"
    export transport_string_sub="$transport_string \
        -nic $nic_subscriber \
        -peer 0@udpv4_wan://${ip_publisher}:7400"
else
    export transport_string_pub="$transport_string"
    export transport_string_sub="$transport_string"
fi

################################################################################

export pub_string="-pub \
        ${transport_string_pub} \
        -noPrintIntervals \
        -executionTime $exec_time"

if [[ ${lat_thr} == "lat" ]]; then
    export pub_string="$pub_string \
        -latencyTest"
fi

export sub_string="-sub \
        ${transport_string_sub} \
        -noPrintIntervals"

if [[ "$role" == "pub" ]]; then
    echo -e "$INFO_TAG Publisher side running"
    export commands_string=${pub_string}
    export extra_arguments="${extra_arguments} ${extra_arguments_pub}"
else
    echo -e "$INFO_TAG Subscriber side running"
    export commands_string=${sub_string}
    export extra_arguments="${extra_arguments} ${extra_arguments_sub}"
fi

###############################################################################

if [[ "$dont_change_tuned_settings" != "1" ]]; then
    echo -e "${INFO_TAG} Executing: /set_${lat_thr}_mode.sh"
    sudo /set_${lat_thr}_mode.sh
    sleep 5
fi

echo -e "${INFO_TAG} Disabling any loss rate"
sudo tc qdisc add dev $nic_publisher root netem loss 0%
sudo tc qdisc del dev $nic_publisher root netem loss 0%

if [[ "$role" == "pub" && "${loss_rate}" != "" ]]; then
    echo -e "${INFO_TAG} Setting loss rate to ${loss_rate}%"
    sudo tc qdisc add dev $nic_publisher root netem loss $loss_rate%
fi

cd $folder_base
echo -e "${INFO_TAG} Folder Base is: $PWD"
mkdir -p $output_folder

if [[ "${batch_size}" != "" ]]; then
    if [[ "${lat_thr}" == "thr" ]]; then
        if [[ "${batch_size}" -eq 0 ]]; then
            echo -e "${INFO_TAG} Batch size is set to 0"
            export run_batching_tests="0"
            export run_no_batching_tests="1"
        else
            echo -e "${INFO_TAG} Batch size is set to ${batch_size}"
            export run_batching_tests="1"
            export run_no_batching_tests="0"
        fi
    else
        echo -e "${INFO_TAG} Batch size is set to ${batch_size}. This value will be ignored for latency tests."
        unset batch_size
    fi
fi

# Tests that may use batching (when doing throughput tests). Also Latency tests.
if [[ ${run_batching_tests} == "1" ]]; then

    # UNKEYED
    if [[ "${skip_unkeyed}" == "" ]]; then

        # RELIABLE
        if [[ "${skip_rel_tests}" == "" ]]; then
            execute_test "unkeyed" "rel" "${datasizes_extended}" "${extra_arguments}" "$file_suffix"
        fi

        # BEST EFFORT
        if [[ "${skip_be_tests}" == "" ]]; then
            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
        fi
    fi

    # KEYED
    if [[ "${skip_keyed_data}" == "" ]]; then

        # RELIABLE
        if [[ "${skip_rel_tests}" == "" ]]; then
            execute_test "keyed" "rel" "${datasizes}" "${extra_arguments}" "$file_suffix"
        fi

        # BEST EFFORT
        if [[ "${skip_be_tests}" == "" ]]; then
            execute_test "keyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
        fi
    fi

fi

# Tests that will not use batching
if [[ "${lat_thr}" == "thr" && "${run_no_batching_tests}" == "1" ]]; then

    if [[ "$role" == "pub" ]]; then
        export commands_string="${commands_string} -batchSize 0"
    fi

    # UNKEYED
    if [[ "${skip_unkeyed}" == "" ]]; then

        # RELIABLE
        if [[ "${skip_rel_tests}" == "" ]]; then
            execute_test "unkeyed" "rel" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
        fi

        # BEST EFFORT
        if [[ "${skip_be_tests}" == "" ]]; then
            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
        fi
    fi

    # KEYED
    if [[ "${skip_keyed_data}" == "" ]]; then

        # RELIABLE
        if [[ "${skip_rel_tests}" == "" ]]; then
            execute_test "keyed" "rel" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
        fi

        # BEST EFFORT
        if [[ "${skip_be_tests}" == "" ]]; then
            execute_test "keyed" "be" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
        fi
    fi

fi

if [[ "$role" == "pub" && "${loss_rate}" != "" ]]; then
    echo -e "${INFO_TAG} Disabling loss rate"
    sudo tc qdisc del dev $nic_publisher root netem loss $loss_rate%
fi

#!/bin/bash
filename=$0
script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)

echo -e "[Calling base_script/script.sh]"
"${script_location}/../base_script/script.sh" $@ --transport UDPv4

Test Software

The following software was used to perform these tests:

RTI Connext DDS 6.1.2 Host and Target Libraries for x64 Linux (x64Linux4gcc7.3.0)

Test Hardware

The following hardware was used to perform these tests:

Linux Nodes

Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 20.04 -- gcc 9.3.0

Switch

Dell 2048 -- 10Gbps switch (10Gbps and 1Gbps interfaces)