4.1.1.1. Real-Time WAN vs UDPv4 in LAN

In this initial test we aim to show that the performance obtained with the Real-Time WAN Transport is similar to the one obtained by using the UDPv4 transport. In other words, Real-Time WAN Transport barely adds overheads when used in LAN.

Unkeyed, UDPv4_WAN 10Gbps Network, C++98

The graph below shows the one-way latency without load between a Publisher and a Subscriber running in two Linux nodes in a 10Gbps network. The numbers are for best-effort as well as strict reliable reliability scenarios.

Note

We use the median (50th percentile) instead of the average in order to get a more stable measurement that does not account for spurious outliers. We also calculate the average value and other percentile values, which can be seen in the Detailed Statistics section below.

Test Hardware

The following hardware was used to perform these tests:

Linux Nodes

Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 20.04 -- gcc 9.3.0

Switch

Dell 2048 -- 10Gbps switch

The graph below shows the expected throughput behavior when performing a 1-1 communication between two Linux nodes in a 10Gbps network. The numbers are for best-effort as well as strict reliable reliability scenarios.

Note

By default, RTI Perftest enables batching when performing a Maximum Throughput test. The batching feature allows sending more than one data sample per RTPS packet, improving network performance for small data sizes. See the RTI Connext DDS Core Libraries User’s Manual for more information on batching.

The batch maximum size is set by RTI Perftest to be 8192 bytes; after 8192 bytes, batching is not enabled.

Test Hardware

The following hardware was used to perform these tests:

Linux Nodes

Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 20.04 -- gcc 9.3.0

Switch

Dell 2048 -- 10Gbps switch

4.1.1.2. TCP/TLS vs Real-Time WAN and Security

The following tests compare the differences in performance between the use of TCP/TLS and Real-Time WAN Transport (UDPv4 over WAN) with and without Security in a more realistic scenario: the communications between a machine in our lab (HQ) and an Amazon Web Services (AWS) virtual machine.

Real-Time WAN Transport is a transport that enables secure, scalable, and high-performance communication over wide area networks (WANs), including public networks. It can be used to extend Connext capabilities to WAN environments. Real-Time WAN Transport uses UDPv4 as the underlying IP transport-layer protocol to better anticipate and adapt to the challenges of diverse network conditions, device mobility, and the dynamic nature of WAN system architectures. For more information, see RTI Real-Time WAN Transport, in the RTI Connext Core Libraries User’s Manual.

The following tests compare TCP vs Real-Time WAN Transport (RWT). They also compare TLS, which implements encryption over TCP, with an equivalent level of security using Security Plugins over the Real-Time WAN Transport.

Data Tables

TCP

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)	CPU (%)
32	56513	14895.8	33342	67423	66165	66662	66886	67423	67423	1.46
64	58089	14076.4	33344	67196	66373	66632	66696	67196	67196	1.53
128	52247	15623.7	33317	68177	64415	66504	67374	68177	68177	1.43
256	45870	12688.9	33333	66482	37122	63221	64828	66482	66482	1.52
512	44657	8328.1	33396	58792	43262	55690	58237	58792	58792	1.62
1024	45735	9063.0	33375	58479	42885	57472	58323	58479	58479	1.52
2048	45714	6598.6	33318	55801	47170	53022	54134	55801	55801	1.66
4096	45500	12585.6	33315	63342	37913	62437	63234	63342	63342	1.57
8192	45531	7616.5	33659	56629	45414	54773	56410	56629	56629	1.56
16384	45135	8924.8	33634	58757	41972	57825	58459	58757	58757	1.62
32768	45282	8389.4	33732	100317	44533	54253	56350	100317	100317	1.76
63000	45459	10948.8	34140	167265	46374	52047	53323	167265	167265	1.89
100000	47832	16429.1	34480	200825	38074	63125	63873	200825	200825	2.26
200000	70078	16839.4	68067	267995	68263	68583	100636	267995	267995	2.26
300000	129572	27437.3	101993	334909	135258	136129	334909	334909	334909	1.61
500000	182037	36188.5	169316	429641	170131	202979	429641	429641	429641	1.57
1048576	441878	36148.1	406666	604516	438370	441713	604516	604516	604516	1.34

TLS

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)	CPU (%)
32	56513	14895.8	33342	67423	66165	66662	66886	67423	67423	1.46
64	58089	14076.4	33344	67196	66373	66632	66696	67196	67196	1.53
128	52247	15623.7	33317	68177	64415	66504	67374	68177	68177	1.43
256	45870	12688.9	33333	66482	37122	63221	64828	66482	66482	1.52
512	44657	8328.1	33396	58792	43262	55690	58237	58792	58792	1.62
1024	45735	9063.0	33375	58479	42885	57472	58323	58479	58479	1.52
2048	45714	6598.6	33318	55801	47170	53022	54134	55801	55801	1.66
4096	45500	12585.6	33315	63342	37913	62437	63234	63342	63342	1.57
8192	45531	7616.5	33659	56629	45414	54773	56410	56629	56629	1.56
16384	45135	8924.8	33634	58757	41972	57825	58459	58757	58757	1.62
32768	45282	8389.4	33732	100317	44533	54253	56350	100317	100317	1.76
63000	45459	10948.8	34140	167265	46374	52047	53323	167265	167265	1.89
100000	47832	16429.1	34480	200825	38074	63125	63873	200825	200825	2.26
200000	70078	16839.4	68067	267995	68263	68583	100636	267995	267995	2.26
300000	129572	27437.3	101993	334909	135258	136129	334909	334909	334909	1.61
500000	182037	36188.5	169316	429641	170131	202979	429641	429641	429641	1.57
1048576	441878	36148.1	406666	604516	438370	441713	604516	604516	604516	1.34

RWT (No Security)

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)	CPU (%)
32	33120	151.6	33041	34835	33088	33141	33775	34835	34835	1.46
64	33100	78.4	33038	33755	33083	33126	33463	33755	33755	1.46
128	33098	49.9	33029	33441	33088	33119	33377	33441	33441	1.46
256	33133	102.4	33039	33971	33105	33186	33738	33971	33971	1.54
512	33146	96.5	33064	33979	33125	33171	33730	33979	33979	1.51
1024	33203	128.2	33130	35021	33174	33271	33619	35021	35021	1.55
2048	33370	2202.3	33165	71419	33224	33283	33881	71419	71419	1.46
4096	33369	176.3	33216	35475	33312	33520	34076	35475	35475	1.42
8192	33542	245.1	33371	35081	33455	33728	34710	35081	35081	1.55
16384	33840	133.8	33664	35177	33822	33933	34520	35177	35177	2.01
32768	34561	641.9	34291	42429	34477	34672	38765	42429	42429	2.32
63000	43309	14773.6	34792	74219	35463	70384	73205	74219	74219	2.64
100000	38143	233.0	37885	39789	38096	38407	38932	39789	39789	2.19
200000	41480	334.6	41147	43421	41359	41847	43006	43421	43421	2.69
300000	44838	410.9	44466	47410	44675	45296	46821	47410	47410	2.91
500000	52639	2783.1	51122	86690	52120	54400	58473	86690	86690	3.55
1048576	153956	12409.0	104219	177577	153248	171977	177577	177577	177577	3.16

RWT (RTPS Sign + Encrypt)

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)	CPU (%)
32	33123	132.8	33055	34731	33097	33158	33654	34731	34731	1.78
64	33154	205.7	33018	35093	33104	33208	34105	35093	35093	1.81
128	33140	150.9	33051	34837	33103	33186	33819	34837	34837	1.74
256	33153	87.2	33049	33742	33118	33266	33541	33742	33742	1.78
512	33163	107.7	33060	34415	33137	33218	33586	34415	34415	1.83
1024	33227	216.0	33106	35591	33175	33264	34272	35591	35591	1.73
2048	33327	2139.2	33128	70276	33180	33239	34013	70276	70276	1.74
4096	33283	143.4	33187	34889	33249	33357	33868	34889	34889	1.83
8192	33448	155.0	33309	34622	33398	33586	34382	34622	34622	1.82
16384	34116	145.9	33950	35284	34083	34252	34922	35284	35284	1.92
32768	35096	128.7	34928	36186	35066	35206	35796	36186	36186	2.05
63000	36838	231.5	36342	39799	36810	36982	37188	39799	39799	6.01
100000	38076	249.2	37827	40478	38007	38282	39070	40478	40478	3.33
200000	41518	275.8	41191	42922	41425	41865	42670	42922	42922	4.05
300000	44895	472.6	44590	48221	44769	45230	47457	48221	48221	4.96
500000	52658	1262.9	51200	57150	52351	54487	56292	57150	57150	6.30
1048576	151806	12836.5	104333	176735	152939	167952	176735	176735	176735	6.05

Data Tables

TCP

Sample Size (Bytes)	Total Samples	Avg Samples/s	Avg Mbps	CPU (%)
32	945664	47103	12.1	0.23
64	526720	26216	13.4	0.19
128	278656	13922	14.3	0.17
256	144000	7186	14.7	0.17
512	72992	3632	14.9	0.16
1024	36664	1825	15.0	0.17
2048	18424	920	15.1	0.17
4096	9274	461	15.1	0.18
8192	4637	231	15.2	0.17
16384	2323	115	15.2	0.17
32768	1157	58	15.2	0.16
63000	599	30	15.2	0.17
100000	423	19	15.3	0.24
200000	191	9	15.3	0.18
300000	150	6	14.7	0.16
500000	5		0.8	0.00
1048576	3		0.6	0.01

TLS

Sample Size (Bytes)	Total Samples	Avg Samples/s	Avg Mbps	CPU (%)
32	938496	46746	12.0	0.24
64	524544	26181	13.4	0.20
128	277952	13826	14.2	0.19
256	142880	7121	14.6	0.17
512	72336	3606	14.8	0.18
1024	36392	1809	14.8	0.17
2048	18208	908	14.9	0.17
4096	9164	456	15.0	0.19
8192	4589	228	15.0	0.16
16384	2297	114	15.0	0.16
32768	1154	57	15.2	0.16
63000	595	29	15.1	0.16
100000	419	19	15.2	0.17
200000	199	9	15.1	0.16
300000	140	6	14.6	0.16
500000	72	2	8.2	0.15
1048576	52		5.9	0.12

RWT (No Security)

Sample Size (Bytes)	Total Samples	Avg Samples/s	Avg Mbps	CPU (%)
32	3711233	185519	47.5	1.76
64	1829121	91425	46.8	1.64
128	925185	46249	47.4	1.63
256	462625	23123	47.4	1.60
512	230833	11539	47.3	1.57
1024	115201	5759	47.2	1.57
2048	57709	2885	47.3	1.55
4096	28945	1447	47.4	1.58
8192	14353	717	47.0	1.55
16384	11989	599	78.6	1.12
32768	9431	471	123.6	0.97
63000	5433	270	136.3	0.56
100000	3224	160	128.0	0.53
200000	1747	85	136.9	0.49
300000	1069	51	124.5	0.41
400000	862	42	135.9	0.47
500000	553	26	106.6	0.49
1048576	301	14	125.3	0.43

RWT (RTPS Sign + Encrypt)

Sample Size (Bytes)	Total Samples	Avg Samples/s	Avg Mbps	CPU (%)
32	3642113	182062	46.6	1.91
64	1835137	91655	46.9	1.85
128	923904	46184	47.3	1.83
256	461601	23076	47.3	1.81
512	230849	11540	47.3	1.79
1024	115057	5751	47.1	1.78
2048	57868	2893	47.4	1.77
4096	28815	1440	47.2	1.76
8192	14377	718	47.1	1.75
16384	12052	602	79.0	1.27
32768	9161	458	120.1	0.98
63000	5561	278	140.1	0.74
100000	2709	133	106.7	0.57
200000	1605	79	127.2	0.63
300000	803	39	95.8	0.44
400000	771	38	122.8	0.62
500000	668	33	132.4	0.54
1048576	281	13	117.1	0.52

Test Hardware

The following hardware was used to perform these tests:

Local Node

Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 20.04 -- gcc 9.3.0
Location: United States of America, California

AWS Machine

Kind: Ubuntu Server 20.04 LTS (HVM), SSD Volume Type
Processor: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz (1 core)
RAM: 1GB
Location: United States of America, East Coast

Data Tables

TCP

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)	CPU (%)
32	64448	7665.2	33269	67448	66595	66786	67020	67448	67448	0.02
64	66843	3117.6	33448	79962	66557	69506	70607	79962	79962	0.02
128	65453	5848.4	33301	68486	66542	66735	68411	68486	68486	0.02
256	63945	9399.1	33299	88690	66563	66988	71025	88690	88690	0.02
512	44617	12167.9	33373	63713	36631	62510	63526	63713	63713	0.02
1024	44643	7676.6	33310	56535	44417	54158	55528	56535	56535	0.02
2048	44718	11271.2	33304	63071	38575	60785	62485	63071	63071	0.03
4096	44276	10361.0	33345	61413	39054	59267	60720	61413	61413	0.03
8192	44696	7115.5	33608	53354	46443	52217	53085	53354	53354	0.03
16384	51365	15648.9	33547	68028	63872	66548	66841	68028	68028	0.03
32768	45050	7730.7	33824	102497	48539	50714	53184	102497	102497	0.06
63000	44949	12019.9	34116	170815	43199	54762	58257	170815	170815	0.09
100000	67620	11644.8	39267	203470	66665	66901	93761	203470	203470	0.07
200000	70004	14607.8	68239	240789	68445	68751	101881	240789	240789	0.10
300000	137864	16370.6	134474	276582	135821	137226	276582	276582	276582	0.09
500000	174815	31399.7	169215	409727	169790	172659	409727	409727	409727	0.11
1048576	415427	43991.3	402717	626209	403679	435947	626209	626209	626209	0.10

TLS

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)	CPU (%)
32	60281	12473.1	33376	66777	66226	66354	66622	66777	66777	0.03
64	57168	14596.1	33475	66798	66247	66675	66790	66798	66798	0.02
128	52150	11615.4	33813	70781	48195	69599	70663	70781	70781	0.02
256	44157	12079.9	33241	63556	36360	62131	63459	63556	63556	0.03
512	44674	8229.5	33315	57619	43416	55078	57562	57619	57619	0.03
1024	44492	9342.8	33298	58389	41162	57193	58181	58389	58389	0.03
2048	44911	6960.1	33358	53630	46972	52877	53468	53630	53630	0.03
4096	44472	12728.1	33402	64275	36351	62622	63939	64275	64275	0.03
8192	43924	8014.1	33481	55878	43950	54285	55624	55878	55878	0.03
16384	44963	9119.5	33639	67167	41891	58083	58493	67167	67167	0.04
32768	45451	8418.3	33849	100466	44791	54590	55901	100466	100466	0.07
63000	44669	9356.4	34087	133867	45875	52048	53352	133867	133867	0.09
100000	49863	12433.4	36788	135595	41394	61997	63419	135595	135595	0.09
200000	70439	14033.9	68760	234097	69002	69338	100551	234097	234097	0.11
300000	127088	24672.3	101738	301154	134572	135164	301154	301154	301154	0.10
500000	181016	32465.4	169561	402557	170388	202892	402557	402557	402557	0.10
1048576	428978	46682.1	403150	639097	435818	437529	639097	639097	639097	0.11

RWT (No Security)

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)	CPU (%)
32	33120	151.6	33041	34835	33088	33141	33775	34835	34835	1.46
64	33100	78.4	33038	33755	33083	33126	33463	33755	33755	1.46
128	33098	49.9	33029	33441	33088	33119	33377	33441	33441	1.46
256	33133	102.4	33039	33971	33105	33186	33738	33971	33971	1.54
512	33146	96.5	33064	33979	33125	33171	33730	33979	33979	1.51
1024	33203	128.2	33130	35021	33174	33271	33619	35021	35021	1.55
2048	33370	2202.3	33165	71419	33224	33283	33881	71419	71419	1.46
4096	33369	176.3	33216	35475	33312	33520	34076	35475	35475	1.42
8192	33542	245.1	33371	35081	33455	33728	34710	35081	35081	1.55
16384	33953	45.5	33876	34169	33947	34012	34094	34169	34169	0.18
32768	34465	111.5	34343	35651	34443	34533	35023	35651	35651	0.18
63000	35442	133.0	35274	36829	35416	35557	35758	36829	36829	0.18
100000	35863	134.9	35610	36687	35836	36026	36241	36687	36687	0.20
200000	37850	4758.5	36811	78056	37153	37576	75152	78056	78056	0.22
300000	38936	4654.8	37778	77752	38215	38967	76090	77752	77752	0.25
500000	41465	3783.6	39829	82164	40746	43122	45101	82164	82164	0.29
1048576	87688	8442.9	81632	128161	85634	89649	123899	128161	128161	0.23

RWT (RTPS Sign + Encrypt)

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)	CPU (%)
32	33241	178.4	33174	35438	33213	33249	33767	35438	35438	0.22
64	33217	29.6	33178	33549	33212	33245	33315	33549	33549	0.22
128	33256	209.6	33181	35460	33220	33262	34058	35460	35460	0.23
256	33234	44.7	33185	33644	33227	33251	33507	33644	33644	0.22
512	33263	55.5	33214	33793	33254	33284	33554	33793	33793	0.23
1024	33299	24.8	33258	33392	33294	33333	33382	33392	33392	0.22
2048	33430	51.0	33372	33875	33420	33470	33722	33875	33875	0.23
4096	33496	62.4	33433	34410	33488	33533	33617	34410	34410	0.22
8192	33636	45.3	33560	33917	33629	33681	33828	33917	33917	0.23
16384	33944	52.0	33837	34235	33933	34007	34166	34235	34235	0.23
32768	34471	93.2	34347	35393	34458	34533	34807	35393	35393	0.23
63000	35458	101.8	35285	35844	35431	35606	35827	35844	35844	0.23
100000	35817	157.2	35630	36588	35762	36060	36391	36588	36588	0.26
200000	37562	3369.7	36877	76535	37190	37572	39519	76535	76535	0.29
300000	38532	3435.1	37718	77416	38115	38825	40647	77416	77416	0.31
500000	40778	886.6	39838	44315	40492	42005	43834	44315	44315	0.37
1048576	86786	6416.3	82230	129337	85741	88412	124847	129337	129337	0.31

Data Tables

TCP

Sample Size (Bytes)	Total Samples	Avg Samples/s	Avg Mbps	CPU (%)
32	1606400	80206	20.5	5.95
64	871297	43550	22.3	5.10
128	432449	21610	22.1	3.97
256	214592	10696	21.9	3.07
512	109873	5481	22.5	3.33
1024	56160	2802	23.0	3.17
2048	27924	1394	22.9	2.86
4096	14075	702	23.0	3.02
8192	6971	348	22.8	2.79
16384	3029	151	19.9	2.59
32768	2195	109	28.8	3.06
63000	1176	58	29.7	1.52
100000	821	39	31.4	1.64
200000	379	18	28.9	3.01
300000	199	7	18.8	2.04
500000	46	1	5.4	0.44
1048576	51	1	8.4	0.44

TLS

Sample Size (Bytes)	Total Samples	Avg Samples/s	Avg Mbps	CPU (%)
32	1347840	67381	17.2	7.27
64	849792	42385	21.7	5.57
128	405952	20266	20.8	5.57
256	216896	10813	22.1	4.06
512	105856	5283	21.6	4.02
1024	54857	2742	22.5	4.07
2048	27429	1369	22.4	3.73
4096	13626	680	22.3	3.63
8192	7015	350	23.0	3.51
16384	4088	203	26.7	2.48
32768	2205	110	29.0	2.13
63000	1149	57	29.0	2.12
100000	808	38	31.2	2.34
200000	432	19	31.0	2.25
300000	294	12	30.4	2.07
500000	101	4	18.8	1.67
1048576	38	1	9.9	0.72

RWT (No Security)

Sample Size (Bytes)	Total Samples	Avg Samples/s	Avg Mbps	CPU (%)
32	1798913	89370	22.9	9.37
64	1673473	83559	42.8	9.62
128	888961	44442	45.5	7.64
256	456897	22803	46.7	6.32
512	230513	11500	47.1	6.40
1024	116025	5791	47.4	5.96
2048	58189	2909	47.7	5.64
4096	28423	1417	46.4	5.56
8192	13941	696	45.7	5.15
16384	7924	394	51.7	3.68
32768	2591	128	33.7	2.69
63000	1398	67	33.9	2.45
100000	684	31	25.4	2.42
200000	581	27	44.2	2.57
300000	470	21	50.6	2.70
400000	305	14	45.2	2.59
500000	324	13	55.5	2.80
1048576	170	5	48.4	2.64

RWT (RTPS Sign + Encrypt)

Sample Size (Bytes)	Total Samples	Avg Samples/s	Avg Mbps	CPU (%)
32	1660673	82608	21.1	10.59
64	1665537	83083	42.5	11.82
128	880257	43917	45.0	9.77
256	450753	22529	46.1	8.63
512	230353	11509	47.1	8.53
1024	116137	5798	47.5	7.88
2048	56509	2824	46.3	7.88
4096	28437	1421	46.6	7.52
8192	14071	701	46.0	7.18
16384	8061	401	52.6	5.44
32768	2578	124	32.7	3.97
63000	1450	70	35.5	3.69
100000	660	29	23.6	3.81
200000	588	28	46.0	3.97
300000	480	21	52.6	4.28
400000	374	16	53.4	4.32
500000	327	14	56.9	4.41
1048576	172	5	48.9	4.10

Test Hardware

The following hardware was used to perform these tests:

Local Node

Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 20.04 -- gcc 9.3.0
Location: United States of America, California

AWS Machine

Kind: Ubuntu Server 20.04 LTS (HVM), SSD Volume Type
Processor: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz (1 core)
RAM: 1GB
Location: United States of America, East Coast