Real-Time WAN Transport (UDPv4_WAN)

The following tests have been performed by executing an RTI Perftest C++98 Publisher and Subscriber between two nodes, connected to a switch via Ethernet. The communication has been restricted to a single interface and the transport has been set to the Real-Time WAN Transport (UDPv4_WAN).

Note

Even though we are using the Real-Time WAN Transport, this testing has been performed on a LAN to be able to compare the results with the built-in UDPv4 transport. The differences are negligible as shown in the graphs below. Find more information and tests in the Real-Time WAN vs UDPv4 in LAN section.

Find information about the hardware, network, and command-line parameters after each of the tests.

Unkeyed, UDPv4_WAN 10Gbps Network, C++98

Linux x64

Minimum LatencyMaximum Throughput

The graph below shows the one-way latency without load between a Publisher and a Subscriber running in two Linux nodes in a 10Gbps network. The numbers are for best-effort as well as strict reliable reliability scenarios.

Note

We use the median (50th percentile) instead of the average in order to get a more stable measurement that does not account for spurious outliers. We also calculate the average value and other percentile values, which can be seen in the Detailed Statistics section below.

Detailed Statistics

The following tables contain the raw numbers presented by RTI Perftest. These numbers are the exact output with no further processing.

• Best Effort

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)
32	16	0.9	15	61	16	17	18	45	61
64	16	0.9	15	57	16	17	18	45	57
128	17	1.0	16	56	16	17	18	46	56
256	17	0.9	16	65	17	17	19	46	65
512	19	1.1	17	58	19	20	21	47	58
1024	25	1.0	18	61	25	26	27	53	61
2048	39	980.1	20	100057	30	31	31	63	100057
4096	30	256.4	25	100061	29	30	31	58	100061
8192	41	7.7	35	88	36	54	61	74	88
16384	85	27.5	45	180	84	124	135	146	180
32768	173	2.7	66	212	173	174	175	203	212
63000	185	2.8	103	295	185	187	189	215	295

• Reliable

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)
32	18	1.2	17	129	17	18	22	47	129
64	18	1.2	17	58	18	18	22	47	58
128	18	1.2	16	59	18	18	24	47	59
256	19	1.6	17	63	18	20	26	47	63
512	20	2.5	18	62	19	25	29	48	62
1024	25	3.6	19	72	26	27	40	55	72
2048	29	27.7	21	4906	30	31	37	62	4906
4096	31	15.5	26	3748	30	31	43	63	3748
8192	41	7.2	36	146	37	54	63	92	146
16384	87	25.9	46	162	86	125	139	153	162
32768	173	6.5	75	214	174	176	178	205	214
63000	186	2.7	113	225	186	188	191	217	225
100000	208	11.8	167	281	210	223	227	271	281
500000	613	66.6	518	8035	615	654	680	816	8035
1048576	1143	51.4	1013	1579	1157	1170	1281	1579	1579
1548576	1687	205.9	1470	9498	1643	1815	1825	9498	9498
4194304	4055	246.7	3942	11032	3996	4169	4875	11032	11032
10485760	10585	291.3	10452	14520	10520	10561	11562	14520	14520

Perftest Scripts

To produce these tests, we executed RTI Perftest for C++98. The exact script used can be found here:

echo EXECUTABLE IS $1
export executable=$1

export folder_base="$(dirname "${executable}")"/../../..

echo OUTPUT PATH IS $2
export output_folder=$2
export pub_sub="pub"
export lat_thr="lat"
export num_reps="1 2"
export dataLens="32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000"
export domain="2"

if [[ -z "$3" ]]; then
    echo "You need a third argument with publisher or subscriber"
    exit -1
else
    if [[ "$3" == "publisher" ]]; then
        echo "Publisher"
        export pub_sub="pub"
        
    elif [[ "$3" == "subscriber" ]]; then
        echo "Subscriber"
        export pub_sub="sub"
    else
        echo "It must be either publisher or subscriber"
        exit -1
    fi
fi

if [[ -z "$4" ]]; then
    echo "You need a forth argument with lat or thr"
    exit -1
else
    if [[ "$4" == "thr" ]]; then
        echo "Throughput test"
        export ${lat_thr}_thr="thr"
        
    elif [[ "$4" == "lat" ]]; then
        echo "Latency test"
        export ${lat_thr}_thr="lat"
    else
        echo "It must be either lat or thr"
        exit -1
    fi
fi

if [[ -z "$5" ]]; then
    echo "Using default nics"
    export nic1=10.2.78.20
    export nic2=10.2.78.21
else
    echo "Using custom nic NOT IMPLEMENTED YET."
    export nic1=10.2.78.20
    export nic2=10.2.78.21
fi


sudo /set_${lat_thr}_mode.sh
sleep 10

export exec_time=20

export pub_string="-pub \
        -transport UDPv4_WAN \
        -transportPublicAddress $nic1:7400 \
        -nic $nic1 \
        -noPrint \
        -exec $exec_time"

if [[ ${lat_thr} == "lat" ]]; then
    export pub_string="$pub_string \
        -latencyTest"
fi

export sub_string="-sub \
        -transport UDPv4_WAN \
        -peer 0@udpv4_wan://$nic1:7400 \
        -nic $nic2 \
        -noPrint"

if [[ "$pub_sub" == "pub" ]]; then
    echo "Publisher side"
    export commands_string=${pub_string}
else
    echo "Subscriber side"
    export commands_string=${sub_string}
fi


cd $folder_base
mkdir -p $output_folder

export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_rel.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in 32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000 100000 500000 1048576 1548576 4194304 10485760; do
        export command="taskset -c 0 \
        $executable -domain $domain -datalen $DATALEN $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_be.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in ${dataLens}; do
        export command="taskset -c 0 \
        $executable -domain $domain -datalen $DATALEN -best $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

export my_file=$output_folder/${lat_thr}_${pub_sub}_keyed_rel.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in ${dataLens}; do
        export command="taskset -c 0 \
        $executable -domain $domain -datalen $DATALEN -keyed -instances 100000 $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

export my_file=$output_folder/${lat_thr}_${pub_sub}_keyed_be.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in ${dataLens}; do
        export command="taskset -c 0 \
        $executable -domain $domain -datalen $DATALEN -keyed -instances 100000 -best $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

# if [[ ${lat_thr} == "thr" ]]; then

#     if [[ "$pub_sub" == "pub" ]]; then
#         echo "Publisher side"
#         export commands_string="${commands_string} -batchSize 0"
#     fi

#     export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_be_noBatch.csv
#     touch $my_file
#     export extra_args=""
#     for index in ${num_reps}; do
#         for DATALEN in ${dataLens}; do
#             export command="taskset -c 0 \
#             $executable -domain $domain -datalen $DATALEN -best $commands_string $extra_args"
#             echo $command ---- $index
#             $command >> $my_file;
#             if [[ "$domain" == "1" ]]; then
#                 export domain="2"
#             else
#                 export domain="1"
#             fi
#             export extra_args=" -noOutputHeaders "
#         done
#     done

#     export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_rel_noBatch.csv
#     touch $my_file
#     export extra_args=""
#     for index in ${num_reps}; do
#         for DATALEN in ${dataLens}; do
#             export command="taskset -c 0 \
#             $executable -domain $domain -datalen $DATALEN $commands_string $extra_args"
#             echo $command ---- $index
#             $command >> $my_file;
#             if [[ "$domain" == "1" ]]; then
#                 export domain="2"
#             else
#                 export domain="1"
#             fi
#             export extra_args=" -noOutputHeaders "
#         done
#     done

# fi

Test Hardware

The following hardware was used to perform these tests:

Linux Nodes

Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 20.04 -- gcc 9.3.0

Switch

Dell 2048 -- 10Gbps switch

Keyed, UDPv4_WAN 10Gbps Network, C++98

Linux x64

Minimum LatencyMaximum Throughput

The graph below shows the one-way latency without load between a Publisher and a Subscriber running in two Linux nodes in a 10Gbps network. The numbers are for best-effort as well as strict reliable reliability scenarios.

Note

We use the median (50th percentile) instead of the average in order to get a more stable measurement that does not account for spurious outliers. We also calculate the average value and other percentile values, which can be seen in the Detailed Statistics section below.

Detailed Statistics

The following tables contain the raw numbers presented by RTI Perftest. These numbers are the exact output with no further processing.

• Best Effort

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)
32	19	1.1	18	166	19	20	22	49	166
64	19	1.1	18	166	19	20	22	49	166
128	19	1.1	18	166	19	20	22	49	166
256	20	1.2	18	167	20	20	22	49	167
512	20	1.1	19	169	20	21	23	50	169
1024	25	1.2	20	171	25	26	27	54	171
2048	30	315.4	23	100062	29	30	32	58	100062
4096	30	180.7	27	100064	29	31	32	62	100064
8192	42	6.8	37	190	39	55	61	78	190
16384	90	24.2	48	280	89	127	144	158	280
32768	177	2.9	69	321	177	178	180	210	321
63000	189	3.0	107	333	189	190	193	226	333

• Reliable

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)
32	21	1.3	19	170	20	21	25	50	170
64	21	1.3	19	172	21	21	25	50	172
128	21	1.3	20	178	21	22	27	50	178
256	21	1.6	20	178	21	22	30	51	178
512	22	2.0	20	170	22	25	29	52	170
1024	26	7.2	21	3095	26	28	36	56	3095
2048	29	3.1	24	173	30	31	38	60	173
4096	31	5.8	28	2688	30	31	46	66	2688
8192	43	5.9	39	187	40	53	61	82	187
16384	92	26.9	49	212	91	131	147	163	212
32768	172	10.1	78	251	177	179	183	208	251
63000	190	2.8	109	263	190	191	195	220	263

Perftest Scripts

To produce these tests, we executed RTI Perftest for C++98. The exact script used can be found here:

echo EXECUTABLE IS $1
export executable=$1

export folder_base="$(dirname "${executable}")"/../../..

echo OUTPUT PATH IS $2
export output_folder=$2
export pub_sub="pub"
export lat_thr="lat"
export num_reps="1 2"
export dataLens="32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000"
export domain="2"

if [[ -z "$3" ]]; then
    echo "You need a third argument with publisher or subscriber"
    exit -1
else
    if [[ "$3" == "publisher" ]]; then
        echo "Publisher"
        export pub_sub="pub"
        
    elif [[ "$3" == "subscriber" ]]; then
        echo "Subscriber"
        export pub_sub="sub"
    else
        echo "It must be either publisher or subscriber"
        exit -1
    fi
fi

if [[ -z "$4" ]]; then
    echo "You need a forth argument with lat or thr"
    exit -1
else
    if [[ "$4" == "thr" ]]; then
        echo "Throughput test"
        export ${lat_thr}_thr="thr"
        
    elif [[ "$4" == "lat" ]]; then
        echo "Latency test"
        export ${lat_thr}_thr="lat"
    else
        echo "It must be either lat or thr"
        exit -1
    fi
fi

if [[ -z "$5" ]]; then
    echo "Using default nics"
    export nic1=10.2.78.20
    export nic2=10.2.78.21
else
    echo "Using custom nic NOT IMPLEMENTED YET."
    export nic1=10.2.78.20
    export nic2=10.2.78.21
fi


sudo /set_${lat_thr}_mode.sh
sleep 10

export exec_time=20

export pub_string="-pub \
        -transport UDPv4_WAN \
        -transportPublicAddress $nic1:7400 \
        -nic $nic1 \
        -noPrint \
        -exec $exec_time"

if [[ ${lat_thr} == "lat" ]]; then
    export pub_string="$pub_string \
        -latencyTest"
fi

export sub_string="-sub \
        -transport UDPv4_WAN \
        -peer 0@udpv4_wan://$nic1:7400 \
        -nic $nic2 \
        -noPrint"

if [[ "$pub_sub" == "pub" ]]; then
    echo "Publisher side"
    export commands_string=${pub_string}
else
    echo "Subscriber side"
    export commands_string=${sub_string}
fi


cd $folder_base
mkdir -p $output_folder

export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_rel.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in 32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000 100000 500000 1048576 1548576 4194304 10485760; do
        export command="taskset -c 0 \
        $executable -domain $domain -datalen $DATALEN $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_be.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in ${dataLens}; do
        export command="taskset -c 0 \
        $executable -domain $domain -datalen $DATALEN -best $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

export my_file=$output_folder/${lat_thr}_${pub_sub}_keyed_rel.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in ${dataLens}; do
        export command="taskset -c 0 \
        $executable -domain $domain -datalen $DATALEN -keyed -instances 100000 $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

export my_file=$output_folder/${lat_thr}_${pub_sub}_keyed_be.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in ${dataLens}; do
        export command="taskset -c 0 \
        $executable -domain $domain -datalen $DATALEN -keyed -instances 100000 -best $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

# if [[ ${lat_thr} == "thr" ]]; then

#     if [[ "$pub_sub" == "pub" ]]; then
#         echo "Publisher side"
#         export commands_string="${commands_string} -batchSize 0"
#     fi

#     export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_be_noBatch.csv
#     touch $my_file
#     export extra_args=""
#     for index in ${num_reps}; do
#         for DATALEN in ${dataLens}; do
#             export command="taskset -c 0 \
#             $executable -domain $domain -datalen $DATALEN -best $commands_string $extra_args"
#             echo $command ---- $index
#             $command >> $my_file;
#             if [[ "$domain" == "1" ]]; then
#                 export domain="2"
#             else
#                 export domain="1"
#             fi
#             export extra_args=" -noOutputHeaders "
#         done
#     done

#     export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_rel_noBatch.csv
#     touch $my_file
#     export extra_args=""
#     for index in ${num_reps}; do
#         for DATALEN in ${dataLens}; do
#             export command="taskset -c 0 \
#             $executable -domain $domain -datalen $DATALEN $commands_string $extra_args"
#             echo $command ---- $index
#             $command >> $my_file;
#             if [[ "$domain" == "1" ]]; then
#                 export domain="2"
#             else
#                 export domain="1"
#             fi
#             export extra_args=" -noOutputHeaders "
#         done
#     done

# fi

Test Hardware

The following hardware was used to perform these tests:

Linux Nodes

Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 20.04 -- gcc 9.3.0

Switch

Dell 2048 -- 10Gbps switch