2.2.2. Modern C++ API

The following tests have been performed by executing an RTI Perftest C++11 Publisher and Subscriber between two nodes, connected to a switch via Ethernet. The communication has been restricted to a single interface and the transport has been set to UDPv4.

Find information about the hardware, network, and command-line parameters after each of the tests.

2.2.2.1. Unkeyed, UDPv4 10Gbps Network, C++11

Linux x64

Minimum LatencyMaximum Throughput

The graph below shows the one-way latency without load between a Publisher and a Subscriber running in two Linux nodes in a 10Gbps network. The numbers are for best-effort as well as strict reliable reliability scenarios.

Note

We use the median (50th percentile) instead of the average in order to get a more stable measurement that does not account for spurious outliers. We also calculate the average value and other percentile values, which can be seen in the Detailed Statistics section below.

Detailed Statistics

The following tables contain the raw numbers presented by RTI Perftest. These numbers are the exact output with no further processing.

• Best Effort

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)
32	15	0.9	14	63	15	16	17	44	63
64	15	0.9	14	75	15	16	17	45	75
128	16	1.0	15	67	15	16	17	45	67
256	16	0.9	15	68	16	16	17	45	68
512	19	1.8	16	69	19	21	22	47	69
1024	25	4.7	17	93	26	29	33	55	93
2048	46	312.2	19	100083	46	55	64	82	100083
4096	44	154.0	23	100087	40	59	73	85	100087
8192	42	204.3	33	100096	35	57	64	81	100096
16384	85	25.1	46	247	84	123	135	151	247
32768	173	2.5	67	226	173	174	176	203	226
63000	185	2.7	102	215	185	187	189	214	215

• Reliable

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)
32	17	1.1	16	69	16	17	21	46	69
64	17	1.1	16	66	17	17	21	46	66
128	17	6.3	16	4535	17	17	21	46	4535
256	18	1.8	16	97	17	19	29	47	97
512	20	2.7	17	83	19	21	29	49	83
1024	25	5.0	18	77	26	28	43	57	77
2048	29	33.5	20	5004	30	33	44	71	5004
4096	31	15.5	25	4722	30	35	47	67	4722
8192	42	48.3	35	5034	37	57	66	3203	5034
16384	88	25.5	47	177	87	126	140	158	177
32768	174	6.8	75	224	175	176	179	204	224
63000	170	15.3	107	225	168	188	190	216	225
100000	198	12.4	159	273	204	211	215	242	273
500000	655	24.5	569	929	670	673	692	778	929
1048576	1143	49.6	1051	1768	1176	1186	1195	1768	1768
1548576	1689	27.4	1520	2556	1683	1749	1758	2556	2556
4194304	4142	58.9	4088	6786	4141	4161	4181	6786	6786
10485760	10526	225.6	10484	17166	10513	10543	10565	17166	17166

Perftest Scripts

To produce these tests, we executed RTI Perftest for C++11. The exact script used can be found here:

echo EXECUTABLE IS $1
export executable=$1

export folder_base="$(dirname "${executable}")"/../../..

echo OUTPUT PATH IS $2
export output_folder=$2
export pub_sub="pub"
export lat_thr="lat"
export num_reps="1 2 3 4"
export dataLens="32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000"
export domain="2"

if [[ -z "$3" ]]; then
    echo "You need a third argument with publisher or subscriber"
    exit -1
else
    if [[ "$3" == "publisher" ]]; then
        echo "Publisher"
        export pub_sub="pub"
        
    elif [[ "$3" == "subscriber" ]]; then
        echo "Subscriber"
        export pub_sub="sub"
    else
        echo "It must be either publisher or subscriber"
        exit -1
    fi
fi

if [[ -z "$4" ]]; then
    echo "You need a forth argument with lat or thr"
    exit -1
else
    if [[ "$4" == "thr" ]]; then
        echo "Throughput test"
        export ${lat_thr}_thr="thr"
        
    elif [[ "$4" == "lat" ]]; then
        echo "Latency test"
        export ${lat_thr}_thr="lat"
    else
        echo "It must be either lat or thr"
        exit -1
    fi
fi

if [[ -z "$5" ]]; then
    echo "Using default nics"
    export nic1=172.16.0.1
    export nic2=172.16.0.2
else
    echo "Using custom nic: $5"
    export nic1=$5
    export nic2=$5
fi

sudo /set_${lat_thr}_mode.sh
sleep 10

export exec_time=20

export pub_string="-pub \
        -transport UDPv4 \
        -nic $nic1 \
        -noPrint \
        -exec $exec_time"

if [[ ${lat_thr} == "lat" ]]; then
    export pub_string="$pub_string \
        -latencyTest"
fi

export sub_string="-sub \
        -transport UDPv4 \
        -nic $nic2 \
        -noPrint"

if [[ "$pub_sub" == "pub" ]]; then
    echo "Publisher side"
    export commands_string=${pub_string}
else
    echo "Subscriber side"
    export commands_string=${sub_string}
fi


cd $folder_base
mkdir -p $output_folder

export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_rel.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in 32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000 100000 500000 1048576 1548576 4194304 10485760; do
        export command="taskset -c 0 \
        $executable -domain $domain -datalen $DATALEN $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_be.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in ${dataLens}; do
        export command="taskset -c 0 \
        $executable -domain $domain -datalen $DATALEN -best $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

export my_file=$output_folder/${lat_thr}_${pub_sub}_keyed_rel.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in ${dataLens}; do
        export command="taskset -c 0 \
        $executable -domain $domain -datalen $DATALEN -keyed -instances 100000 $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

export my_file=$output_folder/${lat_thr}_${pub_sub}_keyed_be.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in ${dataLens}; do
        export command="taskset -c 0 \
        $executable -domain $domain -datalen $DATALEN -keyed -instances 100000 -best $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

if [[ ${lat_thr} == "thr" ]]; then

    if [[ "$pub_sub" == "pub" ]]; then
        echo "Publisher side"
        export commands_string="${commands_string} -batchSize 0"
    fi

    export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_be_noBatch.csv
    touch $my_file
    export extra_args=""
    for index in ${num_reps}; do
        for DATALEN in ${dataLens}; do
            export command="taskset -c 0 \
            $executable -domain $domain -datalen $DATALEN -best $commands_string $extra_args"
            echo $command ---- $index
            $command >> $my_file;
            if [[ "$domain" == "1" ]]; then
                export domain="2"
            else
                export domain="1"
            fi
            export extra_args=" -noOutputHeaders "
        done
    done

    export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_rel_noBatch.csv
    touch $my_file
    export extra_args=""
    for index in ${num_reps}; do
        for DATALEN in ${dataLens}; do
            export command="taskset -c 0 \
            $executable -domain $domain -datalen $DATALEN $commands_string $extra_args"
            echo $command ---- $index
            $command >> $my_file;
            if [[ "$domain" == "1" ]]; then
                export domain="2"
            else
                export domain="1"
            fi
            export extra_args=" -noOutputHeaders "
        done
    done

fi

Test Hardware

The following hardware was used to perform these tests:

Linux Nodes

Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 20.04 -- gcc 9.3.0

Switch

Dell 2048 -- 10Gbps switch

2.2.2.2. Keyed, UDPv4 10Gbps Network, C++11

Linux x64

Minimum LatencyMaximum Throughput

The graph below shows the one-way latency without load between a Publisher and a Subscriber running in two Linux nodes in a 10Gbps network. The numbers are for best-effort as well as strict reliable reliability scenarios.

Note

We use the median (50th percentile) instead of the average in order to get a more stable measurement that does not account for spurious outliers. We also calculate the average value and other percentile values, which can be seen in the Detailed Statistics section below.

Detailed Statistics

The following tables contain the raw numbers presented by RTI Perftest. These numbers are the exact output with no further processing.

• Best Effort

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)
32	18	1.1	17	166	18	19	21	48	166
64	18	1.1	17	163	18	19	21	48	163
128	18	1.1	17	162	18	19	21	48	162
256	19	1.1	18	162	19	20	22	49	162
512	20	1.2	18	166	19	20	22	49	166
1024	25	2.8	19	172	26	28	29	56	172
2048	46	243.8	21	100096	49	61	63	80	100096
4096	30	1.2	26	175	30	31	32	58	175
8192	54	170.1	36	100104	53	64	71	91	100104
16384	94	24.2	48	220	83	126	132	155	220
32768	177	2.2	71	319	177	179	180	208	319
63000	189	3.2	106	333	189	191	193	220	333

• Reliable

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)
32	20	1.2	18	161	19	20	24	49	161
64	20	1.3	18	161	20	21	24	50	161
128	20	1.3	19	164	20	21	26	50	164
256	20	1.9	19	162	20	21	31	50	162
512	21	8.0	19	5057	21	23	29	50	5057
1024	26	9.4	20	4987	26	28	48	58	4987
2048	29	16.4	23	5028	30	32	47	67	5028
4096	31	10.1	27	4618	30	33	53	66	4618
8192	44	12.7	38	4477	40	56	76	101	4477
16384	93	26.7	50	272	92	132	149	167	272
32768	164	16.8	79	218	170	179	182	208	218
63000	190	2.8	112	346	190	192	196	221	346

Perftest Scripts

To produce these tests, we executed RTI Perftest for C++11. The exact script used can be found here:

echo EXECUTABLE IS $1
export executable=$1

export folder_base="$(dirname "${executable}")"/../../..

echo OUTPUT PATH IS $2
export output_folder=$2
export pub_sub="pub"
export lat_thr="lat"
export num_reps="1 2 3 4"
export dataLens="32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000"
export domain="2"

if [[ -z "$3" ]]; then
    echo "You need a third argument with publisher or subscriber"
    exit -1
else
    if [[ "$3" == "publisher" ]]; then
        echo "Publisher"
        export pub_sub="pub"
        
    elif [[ "$3" == "subscriber" ]]; then
        echo "Subscriber"
        export pub_sub="sub"
    else
        echo "It must be either publisher or subscriber"
        exit -1
    fi
fi

if [[ -z "$4" ]]; then
    echo "You need a forth argument with lat or thr"
    exit -1
else
    if [[ "$4" == "thr" ]]; then
        echo "Throughput test"
        export ${lat_thr}_thr="thr"
        
    elif [[ "$4" == "lat" ]]; then
        echo "Latency test"
        export ${lat_thr}_thr="lat"
    else
        echo "It must be either lat or thr"
        exit -1
    fi
fi

if [[ -z "$5" ]]; then
    echo "Using default nics"
    export nic1=172.16.0.1
    export nic2=172.16.0.2
else
    echo "Using custom nic: $5"
    export nic1=$5
    export nic2=$5
fi

sudo /set_${lat_thr}_mode.sh
sleep 10

export exec_time=20

export pub_string="-pub \
        -transport UDPv4 \
        -nic $nic1 \
        -noPrint \
        -exec $exec_time"

if [[ ${lat_thr} == "lat" ]]; then
    export pub_string="$pub_string \
        -latencyTest"
fi

export sub_string="-sub \
        -transport UDPv4 \
        -nic $nic2 \
        -noPrint"

if [[ "$pub_sub" == "pub" ]]; then
    echo "Publisher side"
    export commands_string=${pub_string}
else
    echo "Subscriber side"
    export commands_string=${sub_string}
fi


cd $folder_base
mkdir -p $output_folder

export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_rel.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in 32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000 100000 500000 1048576 1548576 4194304 10485760; do
        export command="taskset -c 0 \
        $executable -domain $domain -datalen $DATALEN $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_be.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in ${dataLens}; do
        export command="taskset -c 0 \
        $executable -domain $domain -datalen $DATALEN -best $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

export my_file=$output_folder/${lat_thr}_${pub_sub}_keyed_rel.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in ${dataLens}; do
        export command="taskset -c 0 \
        $executable -domain $domain -datalen $DATALEN -keyed -instances 100000 $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

export my_file=$output_folder/${lat_thr}_${pub_sub}_keyed_be.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in ${dataLens}; do
        export command="taskset -c 0 \
        $executable -domain $domain -datalen $DATALEN -keyed -instances 100000 -best $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

if [[ ${lat_thr} == "thr" ]]; then

    if [[ "$pub_sub" == "pub" ]]; then
        echo "Publisher side"
        export commands_string="${commands_string} -batchSize 0"
    fi

    export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_be_noBatch.csv
    touch $my_file
    export extra_args=""
    for index in ${num_reps}; do
        for DATALEN in ${dataLens}; do
            export command="taskset -c 0 \
            $executable -domain $domain -datalen $DATALEN -best $commands_string $extra_args"
            echo $command ---- $index
            $command >> $my_file;
            if [[ "$domain" == "1" ]]; then
                export domain="2"
            else
                export domain="1"
            fi
            export extra_args=" -noOutputHeaders "
        done
    done

    export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_rel_noBatch.csv
    touch $my_file
    export extra_args=""
    for index in ${num_reps}; do
        for DATALEN in ${dataLens}; do
            export command="taskset -c 0 \
            $executable -domain $domain -datalen $DATALEN $commands_string $extra_args"
            echo $command ---- $index
            $command >> $my_file;
            if [[ "$domain" == "1" ]]; then
                export domain="2"
            else
                export domain="1"
            fi
            export extra_args=" -noOutputHeaders "
        done
    done

fi

Test Hardware

The following hardware was used to perform these tests:

Linux Nodes

Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 20.04 -- gcc 9.3.0

Switch

Dell 2048 -- 10Gbps switch