TCP

The following tests have been performed by executing RTI Perftest C++98 Publisher and a Subscriber between two nodes, connected to a switch via Ethernet. The communication has been restricted to a single interface and the transport has been set to TCP.

Find the information about the hardware, network and command-line parameters after each of the tests.

Unkeyed, TCP 10Gbps, C++98

The graph below shows the one-way latency without load between a Publisher and a Subscriber running in two Linux nodes in a 10Gbps network. The numbers are for best-effort as well as strict reliable reliability scenarios.

Note

We use the median (50th percentile) instead of the average in order to get a more stable measurement that does not account for spurious outliers. We also calculate the average value and other percentile values, which can be seen in the Detailed Statistics section below.

Detailed Statistics

The following tables contain the raw numbers presented by RTI Perftest. These numbers are the exact output with no further processing.

  • Best Effort

Sample Size (Bytes)

Avg (μs)

Std (μs)

Min (μs)

Max (μs)

50% (μs)

90% (μs)

99% (μs)

99.99% (μs)

99.9999% (μs)

32

25

0.8

24

93

25

26

27

35

93

64

26

0.7

24

89

26

26

27

35

89

128

26

0.7

25

90

26

27

28

35

90

256

26

0.7

25

91

26

27

28

35

91

512

28

1.0

26

95

28

29

30

39

95

1024

29

1.2

27

90

29

31

32

39

90

2048

32

0.9

30

135

32

33

34

47

135

4096

38

1.0

36

104

38

39

40

53

104

8192

49

1.3

47

116

49

51

52

75

116

16384

56

1.0

53

120

56

57

59

83

120

32768

70

1.0

68

120

70

71

72

98

120

63000

103

0.9

101

159

102

103

105

127

159

  • Reliable

Sample Size (Bytes)

Avg (μs)

Std (μs)

Min (μs)

Max (μs)

50% (μs)

90% (μs)

99% (μs)

99.99% (μs)

99.9999% (μs)

32

27

0.8

25

92

26

27

30

38

92

64

27

0.8

26

94

27

27

30

40

94

128

27

0.9

26

94

27

28

30

39

94

256

28

0.8

27

94

28

29

31

39

94

512

30

1.3

27

86

29

31

33

42

86

1024

31

1.2

28

96

30

32

34

45

96

2048

34

1.3

31

99

34

35

37

47

99

4096

39

1.2

37

137

39

41

43

63

137

8192

50

1.1

48

116

50

52

53

85

116

16384

57

0.9

55

122

57

57

59

75

122

32768

72

0.9

69

132

71

72

74

94

132

63000

104

1.0

102

208

104

105

107

138

208

100000

148

3.4

145

260

147

149

162

176

260

500000

506

3.3

504

829

506

507

513

656

829

1048576

998

8.9

993

1652

997

1001

1010

1652

1652

1548576

1459

13.9

1452

2348

1459

1462

1468

2348

2348

4194304

4216

57.1

4174

6289

4213

4232

4248

6289

6289

10485760

11495

203.4

11436

16688

11484

11506

11530

16688

16688


Perftest Scripts

To produce these tests, we executed RTI Perftest for C++98. The exact commands used can be found here:

Publisher Side

 1sudo /set_lat_mode.sh
 2
 3echo EXECUTABLE IS $1
 4export executable=$1
 5
 6echo OUTPUT PATH IS $2
 7export output_folder=$2
 8
 9export exec_time=30
10export nic=172.16.0.1
11export peer=172.16.0.2
12export pub_string="-pub \
13        -transport TCP\
14        -peer 0@tcpv4_lan://$peer:7400 \
15        -nic $nic \
16        -noPrint \
17        -noOutputHeaders \
18        -exec $exec_time \
19        -initialBurst 100\
20        -noXML\
21        -latencyTest"
22
23mkdir -p $output_folder
24
25echo ">> UNKEYED BE"
26export my_file=$output_folder/lat_shmem_pub_unkeyed_be.csv
27touch $my_file
28for DATALEN in 32 64 128 256 512 1024 8192 63000; do
29    export command="\
30    $executable -best -datalen $DATALEN $pub_string"
31    echo $command
32    $command >> $my_file;
33    sleep 3;
34done
35sleep 5;
36
37echo ">> UNKEYED REL"
38export my_file=$output_folder/lat_shmem_pub_unkeyed_rel.csv
39touch $my_file
40for DATALEN in 32 64 128 256 512 1024 8192 63000 100000 500000 1048576 1548576 4194304 10485760; do
41    export command="\
42    $executable -datalen $DATALEN $pub_string"
43    echo $command
44    $command >> $my_file;
45    sleep 3;
46done
47sleep 5;
48
49echo ">> KEYED BE"
50export my_file=$output_folder/lat_shmem_pub_keyed_be.csv
51touch $my_file
52for DATALEN in 32 64 128 256 512 1024 8192 63000; do
53    export command="\
54    $executable -best -keyed -instances 100000 -datalen $DATALEN $pub_string"
55    echo $command
56    $command >> $my_file;
57    sleep 3;
58done
59sleep 5;
60
61echo ">> KEYED REL"
62export my_file=$output_folder/lat_shmem_pub_keyed_rel.csv
63touch $my_file
64for DATALEN in 32 64 128 256 512 1024 8192 63000; do
65    export command="\
66    $executable -keyed -instances 100000 -datalen $DATALEN $pub_string"
67    echo $command
68    $command >> $my_file;
69    sleep 3;
70done

Subscriber Side

 1sudo /set_lat_mode.sh
 2
 3echo EXECUTABLE IS $1
 4export executable=$1
 5
 6echo OUTPUT PATH IS $2
 7export output_folder=$2
 8
 9export nic=172.16.0.2
10export peer=172.16.0.1
11export sub_string="-sub \
12        -transport TCP\
13        -peer 0@tcpv4_lan://$peer:7400 \
14        -nic $nic \
15        -noPrint \
16        -noOutputHeaders \
17        -noXML"
18
19mkdir -p $output_folder
20
21echo ">> UNKEYED BE"
22export my_file=$output_folder/lat_shmem_sub_unkeyed_be.csv
23touch $my_file
24for DATALEN in 32 64 128 256 512 1024 8192 63000; do
25    export command="\
26    $executable -best $sub_string -datalen $DATALEN"
27    echo $command
28    $command >> $my_file;
29    sleep 10;
30done
31sleep 5;
32
33echo ">> UNKEYED REL"
34export my_file=$output_folder/lat_shmem_sub_unkeyed_rel.csv
35touch $my_file
36for DATALEN in 32 64 128 256 512 1024 8192 63000 100000 500000 1048576 1548576 4194304 10485760; do
37    export command="\
38    $executable $sub_string -datalen $DATALEN"
39    echo $command
40    $command >> $my_file;
41    sleep 10;
42done
43sleep 5;
44
45echo ">> KEYED BE"
46export my_file=$output_folder/lat_shmem_sub_keyed_be.csv
47touch $my_file
48for DATALEN in 32 64 128 256 512 1024 8192 63000; do
49    export command="\
50    $executable -best -keyed -instances 100000 $sub_string -datalen $DATALEN"
51    echo $command
52    $command >> $my_file;
53    sleep 10;
54done
55sleep 5;
56
57echo ">> KEYED REL"
58export my_file=$output_folder/lat_shmem_sub_keyed_rel.csv
59touch $my_file
60for DATALEN in 32 64 128 256 512 1024 8192 63000; do
61    export command="\
62    $executable -keyed -instances 100000 $sub_string -datalen $DATALEN"
63    echo $command
64    $command >> $my_file;
65    sleep 10;
66done

Test Hardware

The following hardware was used to perform these tests:

Linux Nodes

Processor: Intel® Xeon® E-2186G 3.8GHz, 12M cache, 6C/12T, turbo (95W)
RAM: 16GB 2666MT/s DDR4 ECC UDIMM
NIC 1: Intel X550 Dual Port 10GbE BASE-T Adapter, PCIe Full Height
NIC 2: Intel Ethernet I350 Dual Port 1GbE BASE-T Adapter, PCIe Low Profile
OS: Ubuntu 18.04 -- gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

Switch

Dell Networking S4048T-ON, 48x 10GBASE-T and 6x 40GbE QSFP+ ports, IO to PSU air, 2x AC PSU, OS9

Keyed, TCP 10Gbps, C++98

The graph below shows the one-way latency without load between a Publisher and a Subscriber running in two Linux nodes in a 10Gbps network. The numbers are for best-effort as well as strict reliable reliability scenarios.

Note

We use the median (50th percentile) instead of the average in order to get a more stable measurement that does not account for spurious outliers. We also calculate the average value and other percentile values, which can be seen in the Detailed Statistics section below.

Detailed Statistics

The following tables contain the raw numbers presented by RTI Perftest. These numbers are the exact output with no further processing.

  • Best Effort

Sample Size (Bytes)

Avg (μs)

Std (μs)

Min (μs)

Max (μs)

50% (μs)

90% (μs)

99% (μs)

99.99% (μs)

99.9999% (μs)

32

29

1.1

27

215

28

29

31

46

215

64

28

1.1

26

210

28

29

31

45

210

128

29

1.1

27

217

29

30

32

46

217

256

29

1.1

27

212

29

30

31

48

212

512

31

1.4

28

225

29

32

34

49

225

1024

33

1.7

29

220

31

34

36

57

220

2048

36

1.4

33

222

36

37

39

60

222

4096

42

1.6

39

231

42

43

45

69

231

8192

53

1.7

49

237

52

54

56

82

237

16384

59

1.4

57

248

59

61

63

90

248

32768

73

1.4

71

259

73

74

77

110

259

63000

106

1.3

104

196

106

107

111

138

196

  • Reliable

Sample Size (Bytes)

Avg (μs)

Std (μs)

Min (μs)

Max (μs)

50% (μs)

90% (μs)

99% (μs)

99.99% (μs)

99.9999% (μs)

32

30

1.1

29

224

30

31

34

48

224

64

31

1.1

29

218

30

32

34

51

218

128

31

1.1

29

221

30

32

34

51

221

256

31

1.3

29

223

31

32

35

60

223

512

33

1.4

30

221

32

34

36

53

221

1024

35

1.5

32

234

34

36

38

56

234

2048

37

1.5

35

227

37

39

42

63

227

4096

43

1.5

40

234

43

45

48

72

234

8192

55

1.5

52

242

54

56

58

86

242

16384

61

1.4

58

251

61

62

64

89

251

32768

75

1.5

73

263

75

77

79

104

263

63000

107

1.3

105

201

107

108

111

147

201


Perftest Scripts

To produce these tests, we executed RTI Perftest for C++98. The exact commands used can be found here:

Publisher Side

 1sudo /set_lat_mode.sh
 2
 3echo EXECUTABLE IS $1
 4export executable=$1
 5
 6echo OUTPUT PATH IS $2
 7export output_folder=$2
 8
 9export exec_time=30
10export nic=172.16.0.1
11export peer=172.16.0.2
12export pub_string="-pub \
13        -transport TCP\
14        -peer 0@tcpv4_lan://$peer:7400 \
15        -nic $nic \
16        -noPrint \
17        -noOutputHeaders \
18        -exec $exec_time \
19        -initialBurst 100\
20        -noXML\
21        -latencyTest"
22
23mkdir -p $output_folder
24
25echo ">> UNKEYED BE"
26export my_file=$output_folder/lat_shmem_pub_unkeyed_be.csv
27touch $my_file
28for DATALEN in 32 64 128 256 512 1024 8192 63000; do
29    export command="\
30    $executable -best -datalen $DATALEN $pub_string"
31    echo $command
32    $command >> $my_file;
33    sleep 3;
34done
35sleep 5;
36
37echo ">> UNKEYED REL"
38export my_file=$output_folder/lat_shmem_pub_unkeyed_rel.csv
39touch $my_file
40for DATALEN in 32 64 128 256 512 1024 8192 63000 100000 500000 1048576 1548576 4194304 10485760; do
41    export command="\
42    $executable -datalen $DATALEN $pub_string"
43    echo $command
44    $command >> $my_file;
45    sleep 3;
46done
47sleep 5;
48
49echo ">> KEYED BE"
50export my_file=$output_folder/lat_shmem_pub_keyed_be.csv
51touch $my_file
52for DATALEN in 32 64 128 256 512 1024 8192 63000; do
53    export command="\
54    $executable -best -keyed -instances 100000 -datalen $DATALEN $pub_string"
55    echo $command
56    $command >> $my_file;
57    sleep 3;
58done
59sleep 5;
60
61echo ">> KEYED REL"
62export my_file=$output_folder/lat_shmem_pub_keyed_rel.csv
63touch $my_file
64for DATALEN in 32 64 128 256 512 1024 8192 63000; do
65    export command="\
66    $executable -keyed -instances 100000 -datalen $DATALEN $pub_string"
67    echo $command
68    $command >> $my_file;
69    sleep 3;
70done

Subscriber Side

 1sudo /set_lat_mode.sh
 2
 3echo EXECUTABLE IS $1
 4export executable=$1
 5
 6echo OUTPUT PATH IS $2
 7export output_folder=$2
 8
 9export nic=172.16.0.2
10export peer=172.16.0.1
11export sub_string="-sub \
12        -transport TCP\
13        -peer 0@tcpv4_lan://$peer:7400 \
14        -nic $nic \
15        -noPrint \
16        -noOutputHeaders \
17        -noXML"
18
19mkdir -p $output_folder
20
21echo ">> UNKEYED BE"
22export my_file=$output_folder/lat_shmem_sub_unkeyed_be.csv
23touch $my_file
24for DATALEN in 32 64 128 256 512 1024 8192 63000; do
25    export command="\
26    $executable -best $sub_string -datalen $DATALEN"
27    echo $command
28    $command >> $my_file;
29    sleep 10;
30done
31sleep 5;
32
33echo ">> UNKEYED REL"
34export my_file=$output_folder/lat_shmem_sub_unkeyed_rel.csv
35touch $my_file
36for DATALEN in 32 64 128 256 512 1024 8192 63000 100000 500000 1048576 1548576 4194304 10485760; do
37    export command="\
38    $executable $sub_string -datalen $DATALEN"
39    echo $command
40    $command >> $my_file;
41    sleep 10;
42done
43sleep 5;
44
45echo ">> KEYED BE"
46export my_file=$output_folder/lat_shmem_sub_keyed_be.csv
47touch $my_file
48for DATALEN in 32 64 128 256 512 1024 8192 63000; do
49    export command="\
50    $executable -best -keyed -instances 100000 $sub_string -datalen $DATALEN"
51    echo $command
52    $command >> $my_file;
53    sleep 10;
54done
55sleep 5;
56
57echo ">> KEYED REL"
58export my_file=$output_folder/lat_shmem_sub_keyed_rel.csv
59touch $my_file
60for DATALEN in 32 64 128 256 512 1024 8192 63000; do
61    export command="\
62    $executable -keyed -instances 100000 $sub_string -datalen $DATALEN"
63    echo $command
64    $command >> $my_file;
65    sleep 10;
66done

Test Hardware

The following hardware was used to perform these tests:

Linux Nodes

Processor: Intel® Xeon® E-2186G 3.8GHz, 12M cache, 6C/12T, turbo (95W)
RAM: 16GB 2666MT/s DDR4 ECC UDIMM
NIC 1: Intel X550 Dual Port 10GbE BASE-T Adapter, PCIe Full Height
NIC 2: Intel Ethernet I350 Dual Port 1GbE BASE-T Adapter, PCIe Low Profile
OS: Ubuntu 18.04 -- gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

Switch

Dell Networking S4048T-ON, 48x 10GBASE-T and 6x 40GbE QSFP+ ports, IO to PSU air, 2x AC PSU, OS9