1.2.5. Comparison

This page contains the results of the performance comparison between the different API implementations.

The following tests have been performed by executing 2 RTI Perftest applications, Publisher and Subscriber, between two nodes, connected to a switch via Ethernet. The communication has been restricted to a single interface and the transport has been set to UDPv4.

Find information about the hardware, network, and command-line parameters after each of the tests.

1.2.5.1. UDPv4

Linux x64 (10Gbps Network)

Unkeyed

Minimum LatencyMaximum Throughput

The graph below shows the one-way latency without load between a Publisher and a Subscriber running in two Linux nodes in a 10Gbps network. The numbers are for best-effort as well as strict reliable reliability scenarios.

Note

We use the median (50th percentile) instead of the average in order to get a more stable measurement that does not account for spurious outliers. We also calculate the average value and other percentile values, which can be seen in the Detailed Statistics section below.

Perftest Scripts

To produce these tests, we executed RTI Perftest for C++11. The exact script used can be found here:

#!/bin/bash
filename=$0
script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)

export datasizes="32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000"
export datasizes_extended="${datasizes} 100000 500000 1048576 1548576 4194304 10485760"

export domain="2"
export exec_time=20
export num_reps=1
export instance_number=100000
export core=0

# We will use some colors to improve visibility of errors and info messages.
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
BLUE='\033[0;34m'
LIGHTBLUE='\033[0;36m'
NC='\033[0m'
INFO_TAG="${GREEN}[INFO]:${NC}"
WARNING_TAG="${YELLOW}[WARNING]:${NC}"
ERROR_TAG="${RED}[ERROR]:${NC}"

export ip_machine_1="10.2.78.20"
export ip_machine_2="10.2.78.21"
export if10Gbps="enp1s0f0"
export if1Gbps="eno1"

################################################################################

function disable_colors() {
    export RED=""
    export GREEN=""
    export YELLOW=""
    export NC=""
    export BLUE=""
    export LIGHTBLUE=""
    export INFO_TAG="${GREEN}[INFO]:${NC}"
    export WARNING_TAG="${YELLOW}[WARNING]:${NC}"
    export ERROR_TAG="${RED}[ERROR]:${NC}"
}

function change_domain() {
    if [[ "$domain" == "1" ]]; then
        export domain="2"
    else
        export domain="1"
    fi
}

# Usage: execute_test <keyed/unkeyed> <rel/be> <datasizes> <batchSize>
function execute_test() {

    local keyed_unkeyed=$1
    local rel_be=$2
    local datasizes_test=$3
    local other_args=$4
    local name_suffix=$5

    local commands_string_test=$commands_string
    local tag=""

    if [[ "${keyed_unkeyed}" == "keyed" ]]; then
        commands_string_test="${commands_string_test} -keyed -instances $instance_number"
        tag="[${YELLOW}${transport}${NC}|${BLUE}K${NC}|"
    else
        tag="[${YELLOW}${transport}${NC}|${LIGHTBLUE}UK${NC}|"
    fi

    if [[ "${rel_be}" == "be" ]]; then
        commands_string_test="${commands_string_test} -bestEffort"
        tag="${tag}${YELLOW}BE${NC}]"
    else
        tag="${tag}${RED}REL${NC}]"
    fi

    tag="${tag}[${LIGHTBLUE}${lat_thr}${NC}]"

    local output_file=$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}.csv

    if [[ "$role" == "pub" ]]; then
        echo -e "${YELLOW}[TEST]: $keyed_unkeyed, $rel_be. ${NC}"
    fi

    if [[ "$NO_TASKSET" == "" ]]; then
        if [[ "$LANGUAGE" != "java" && "$LANGUAGE" != "cs" ]]; then
            export pre_command_string="taskset -c $core"
        fi
    fi

    if [[ "$LANGUAGE" == "python" ]]; then
        export pre_command_string="python3 "
    fi

    if [[ "$DOCKER" == "1" ]]; then
        export pre_command_string="taskset -c $core docker run --net=host -v /home/perfuser/rti_license_connextpro.dat:/opt/rti.com/rti_connext_dds-7.3.0/rti_license.dat rticom/perftest:7.3.0-EAR "
        executable=""
    fi

    # Get the aprox time this will take:
    total_tests=$((`wc -w <<< "$datasizes_test"` * num_reps))
    total_time=$((total_tests * exec_time))

    touch $output_file
    local no_headers=""
    local current_test=0
    for index in $(seq 1 ${num_reps}); do
        for DATALEN in ${datasizes_test}; do
            current_test=$((current_test + 1))
            export command="$pre_command_string $executable -domain $domain -dataLen $DATALEN $commands_string_test $other_args $no_headers"
            if [[ "$role" == "pub" ]]; then
                echo -e "Test ${tag} (${current_test}/${total_tests}) -- Total time = ${total_time}s"
                echo -e ${BLUE}$command${NC}
            else
                echo -e ${LIGHTBLUE}$command${NC}
            fi
            if [[ "$LANGUAGE" == "cs" && "$role" == "pub" ]]; then
                sleep 3
            fi
            if [[ "$raw" == "1" && "$role" == "sub" ]]; then
                sleep 5
            fi
            if [[ "${get_netstat_info}" == "1" ]]; then
                echo -e "${INFO_TAG} Getting netstat info before"
                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt
            fi
            eval $command >> $output_file;
            if [[ "${get_netstat_info}" == "1" ]]; then
                echo -e "${INFO_TAG} Getting netstat info after"
                netstat -s -u | grep -e "error" -e "packet" > $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt
                touch "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
                python3 $script_location/../../../tools/diff_netstat_output.py \
                    -n $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_after.txt \
                    -o $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_before.txt \
                    -d $DATALEN $no_header_netstat \
                    -csv >> "$output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat.csv"
                rm -rf $output_folder/${lat_thr}_${role}_${keyed_unkeyed}_${rel_be}${name_suffix}_netstat_*.txt
                no_header_netstat=" -nh"
            fi
            no_headers=" -noOutputHeaders"

            change_domain
        done
    done
}

################################################################################
# PARSE COMMAND LINE OPTIONS:

while [ "$1" != "" ]; do
    case $1 in
        --executable)
            executable=$2
            shift
            ;;
        --docker)
            DOCKER="1"
            ;;
        --output-folder)
            output_folder=$2
            shift
            ;;
        --sub-folder)
            sub_folder=$2
            shift
            ;;
        --role)
            export role=$2
            shift
            ;;
        --core)
            export core=$2
            shift
            ;;
        --test-kind)
            export lat_thr=$2
            shift
            ;;
        --interface1)
            export interface=$2
            shift
            ;;
        --interface2)
            export interface2=$2
            shift
            ;;
        --ip1)
            export ip1=$2
            shift
            ;;
        --ip2)
            export ip2=$2
            shift
            ;;
        --repetitions)
            export num_reps=$2
            shift
            ;;
        --domain)
            export domain=$2
            shift
            ;;
        --execution-time)
            export exec_time=$2
            shift
            ;;
        --transport)
            export transport=$2
            shift
            ;;
        --datalen)
            export datalen_input=$2
            shift
            ;;
        --file-suffix)
            export file_suffix=$2
            shift
            ;;
        --executable-suffix)
            export executable_suffix=$2
            shift
            ;;
        --extra-arguments)
            export extra_arguments=$2
            shift
            ;;
        --extra-arguments-pub)
            export extra_arguments_pub=$2
            shift
            ;;
        --extra-arguments-sub)
            export extra_arguments_sub=$2
            shift
            ;;
        --skip-no-batching)
            export skip_no_batching="1"
            ;;
        --skip-be)
            export skip_be_tests="1"
            ;;
        --skip-rel)
            export skip_rel_tests="1"
            ;;
        --skip-keyed)
            export skip_keyed_data="1"
            ;;
        --skip-large-data)
            export skip_large_data="1"
            ;;
        --large-data)
            export large_data="1"
            ;;
        --keyed)
            export skip_unkeyed="1"
            ;;
        --unkeyed)
            export skip_keyed_data="1"
            ;;
        --no-batching | --skip-batching)
            export no_batching_only="1"
            ;;
        --reliable)
            export skip_be_tests="1"
            ;;
        --best-effort)
            export skip_rel_tests="1"
            ;;
        --security)
            export security_only="$2"
            shift
            ;;
        --micro)
            export micro="1"
            ;;
        --raw | --raw-transport)
            export raw="1"
            ;;
        --tss)
            export tss="1"
            ;;
        --no-colors)
            export NO_COLORS="1"
            ;;
        --language)
            export LANGUAGE=$2
            shift
            ;;
        --loss-rate)
            export loss_rate=$2
            shift
            ;;
        --get-netstat-info | --netstat)
            export get_netstat_info="1"
            ;;
        --no-taskset)
            export NO_TASKSET="1"
            ;;
        *)
            echo -e "unknown parameter \"$1\""
            exit 255
            ;;
    esac
    shift
done

if [[ "$NO_COLORS" == "1" ]]; then
    disable_colors
fi

export folder_base="$(dirname "${executable}")"/../../..

if [[ $LANGUAGE == "java"  || "$LANGUAGE" == "cs" ]]; then
    export folder_base="$(dirname "${executable}")"/../..
fi
if [[ $tss == "1" ]]; then
    export folder_base="$(dirname "${executable}")"/../../../../..
fi

if [[ "${executable_suffix}" != "" ]]; then
    export executable="${executable}${executable_suffix}"
fi

if [[ "${sub_folder}" != "" ]]; then
    export output_folder="${output_folder}/${sub_folder}"
fi

echo -e "${INFO_TAG} Perftest executable is: $executable"
echo -e "${INFO_TAG} Output folder is: $output_folder"

################################################################################

if [[ "$LANGUAGE" == "python" ]]; then
    export skip_keyed_data="1"
    export skip_large_data="1"
    export skip_be_tests="1"
    export skip_no_batching="1"
fi

if [[ "${skip_large_data}" == "1" ]]; then
    export datasizes_extended=${datasizes}
elif [[ "${large_data}" == "1" ]]; then
    export datasizes=${datasizes_extended}
fi

if [[ "${datalen_input}" != "" ]]; then
    echo -e "${YELLOW}[TEST] Testing only for ${datalen_input}${NC}"
    export datasizes=${datalen_input}
    export datasizes_extended=${datalen_input}
    if [[ "${no_batching_only}" != "1" ]]; then
        export skip_large_data="1"
    fi
fi

if [[ "$role" != "pub" && "$role" != "sub" ]]; then
    echo -e "${ERROR_TAG} It must be either publisher or subscriber"
    exit 255
fi

if [[ "$lat_thr" != "thr" && "$lat_thr" != "lat" ]]; then
    echo -e "${ERROR_TAG} It must be either lat or thr"
    exit 255
fi

if [[ "${interface}" == "" ]]; then
    echo "Using default nics"
    export nic_publisher=${ip_machine_1}
    export nic_subscriber=${ip_machine_2}
elif [[ "${interface}" == "both" ]]; then
    export nic_publisher="enp1s0f0,eno1"
    export nic_subscriber="enp1s0f0,eno1"
    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"
    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"
else
    export nic_publisher=$interface
    echo -e "${INFO_TAG} Using nic_publisher: ${nic_publisher}"

    if [[ "${interface2}" == "" ]]; then
        export nic_subscriber=$interface
    else
        export nic_subscriber=$interface2
    fi
    echo -e "${INFO_TAG} Using nic_subscriber: ${nic_subscriber}"

    if [[ "${ip1}" != "" ]]; then
        export ip_publisher=$ip1
        echo "Using ip_publisher: ${ip_publisher}"
    fi

    if [[ "${ip2}" != "" ]]; then
        export ip_subscriber=$ip2
        echo "Using ip_subscriber: ${ip_subscriber}"
    fi

fi

export transport_string="-transport $transport"

if [[ "$transport" == "UDPv4" ]]; then

    export transport_string_pub="$transport_string -nic $nic_publisher"
    export transport_string_sub="$transport_string -nic $nic_subscriber"

    if [[ "$micro" == "1" || "$raw" == "1" ]]; then
        export transport_string_pub="$transport_string_pub -peer ${ip_subscriber}"
        export transport_string_sub="$transport_string_sub -peer ${ip_publisher}"
    fi

elif [[ "$transport" == "TCP" ]]; then
    export transport_string_pub="$transport_string \
        -nic $nic_publisher \
        -peer 0@tcpv4_lan://${ip_subscriber}:7400"
    export transport_string_sub="$transport_string \
        -nic $nic_subscriber \
        -peer 0@tcpv4_lan://${ip_publisher}:7400"
elif [[ "$transport" == "TLS" ]]; then
    export transport_string_pub="$transport_string \
        -nic $nic_publisher \
        -peer tlsv4_lan://${ip_subscriber}:7400"
    export transport_string_sub="$transport_string \
        -nic $nic_subscriber \
        -peer tlsv4_lan://${ip_publisher}:7400"
elif [[ "$transport" == "UDPv4_WAN" ]]; then
    export transport_string_pub="$transport_string \
        -nic $nic_publisher \
        -transportPublicAddress $ip_publisher:7400"
    export transport_string_sub="$transport_string \
        -nic $nic_subscriber \
        -peer 0@udpv4_wan://${ip_publisher}:7400"
else
    export transport_string_pub="$transport_string"
    export transport_string_sub="$transport_string"
fi

################################################################################

export pub_string="-pub \
        ${transport_string_pub} \
        -noPrintIntervals \
        -executionTime $exec_time"

if [[ ${lat_thr} == "lat" ]]; then
    export pub_string="$pub_string \
        -latencyTest"
fi

export sub_string="-sub \
        ${transport_string_sub} \
        -noPrintIntervals"

if [[ "$role" == "pub" ]]; then
    echo -e "$INFO_TAG Publisher side running"
    export commands_string=${pub_string}
    export extra_arguments="${extra_arguments} ${extra_arguments_pub}"
else
    echo -e "$INFO_TAG Subscriber side running"
    export commands_string=${sub_string}
    export extra_arguments="${extra_arguments} ${extra_arguments_sub}"
fi

###############################################################################

echo -e "${INFO_TAG} Executing: /set_${lat_thr}_mode.sh"
sudo /set_${lat_thr}_mode.sh
sleep 5

echo -e "${INFO_TAG} Disabling any loss rate"
sudo tc qdisc add dev $nic_publisher root netem loss 0%
sudo tc qdisc del dev $nic_publisher root netem loss 0%

if [[ "$role" == "pub" && "${loss_rate}" != "" ]]; then
    echo -e "${INFO_TAG} Setting loss rate to ${loss_rate}%"
    sudo tc qdisc add dev $nic_publisher root netem loss $loss_rate%
fi

cd $folder_base
echo -e "${INFO_TAG} Folder Base is: $PWD"
mkdir -p $output_folder

# Tests that may use batching (when doing throughput tests)
if [[ ${no_batching_only} != "1" ]]; then

    # UNKEYED
    if [[ "${skip_unkeyed}" == "" ]]; then

        # RELIABLE
        if [[ "${skip_rel_tests}" == "" ]]; then
            execute_test "unkeyed" "rel" "${datasizes_extended}" "${extra_arguments}" "$file_suffix"
        fi

        # BEST EFFORT
        if [[ "${skip_be_tests}" == "" ]]; then
            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
        fi
    fi

    # KEYED
    if [[ "${skip_keyed_data}" == "" ]]; then

        # RELIABLE
        if [[ "${skip_rel_tests}" == "" ]]; then
            execute_test "keyed" "rel" "${datasizes}" "${extra_arguments}" "$file_suffix"
        fi

        # BEST EFFORT
        if [[ "${skip_be_tests}" == "" ]]; then
            execute_test "keyed" "be" "${datasizes}" "${extra_arguments}" "$file_suffix"
        fi
    fi

fi

if [[ "${skip_no_batching}" == "" || "${no_batching_only}" == "1" ]]; then
    no_batching_tests="1"
fi

# Tests that will not use batching
if [[ "${lat_thr}" == "thr" && "${no_batching_tests}" == "1" ]]; then

    if [[ "$role" == "pub" ]]; then
        export commands_string="${commands_string} -batchSize 0"
    fi

    # UNKEYED
    if [[ "${skip_unkeyed}" == "" ]]; then

        # RELIABLE
        if [[ "${skip_rel_tests}" == "" ]]; then
            execute_test "unkeyed" "rel" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
        fi

        # BEST EFFORT
        if [[ "${skip_be_tests}" == "" ]]; then
            execute_test "unkeyed" "be" "${datasizes}" "${extra_arguments}" "_noBatch${file_suffix}"
        fi
    fi
fi

if [[ "$role" == "pub" && "${loss_rate}" != "" ]]; then
    echo -e "${INFO_TAG} Disabling loss rate"
    sudo tc qdisc del dev $nic_publisher root netem loss $loss_rate%
fi

#!/bin/bash
filename=$0
script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)

echo -e "[Calling base_script/script.sh]"
"${script_location}/../base_script/script.sh" $@ --transport UDPv4

Test Hardware

The following hardware was used to perform these tests:

Linux Nodes

Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 20.04 -- gcc 9.3.0

Switch

Dell 2048 -- 10Gbps switch (10Gbps and 1Gbps interfaces)