TCP

The following tests have been performed by executing RTI Perftest C++98 Publisher and a Subscriber between two nodes, connected to a switch via Ethernet. The communication has been restricted to a single interface and the transport has been set to TCP.

Find the information about the hardware, network and command-line parameters after each of the tests.

Unkeyed, TCP 10Gbps, C++98

The graph below shows the one-way latency without load between a Publisher and a Subscriber running in two Linux nodes in a 10Gbps network. The numbers are for best-effort as well as strict reliable reliability scenarios.

Note

We use the median (50th percentile) instead of the average in order to get a more stable measurement that does not account for spurious outliers. We also calculate the average value and other percentile values, which can be seen in the Detailed Statistics section below.

Detailed Statistics

The following tables contain the raw numbers presented by RTI Perftest. These numbers are the exact output with no further processing.

  • Best Effort

Sample Size (Bytes)

Avg (μs)

Std (μs)

Min (μs)

Max (μs)

50% (μs)

90% (μs)

99% (μs)

99.99% (μs)

99.9999% (μs)

32

26

1.2

24

148

25

26

32

38

148

64

26

1.6

24

138

25

26

34

41

138

128

26

1.7

24

109

25

26

35

42

109

256

27

1.0

25

47

27

27

30

36

47

512

27

2.0

25

108

27

28

35

43

108

1024

28

1.7

26

119

29

29

36

43

119

8192

48

1.7

46

137

48

49

56

63

137

63000

102

2.4

99

177

101

104

112

122

177

  • Reliable

Sample Size (Bytes)

Avg (μs)

Std (μs)

Min (μs)

Max (μs)

50% (μs)

90% (μs)

99% (μs)

99.99% (μs)

99.9999% (μs)

32

28

1.8

26

112

27

29

36

44

112

64

28

2.5

25

117

27

31

37

47

117

128

29

2.2

26

109

28

30

38

46

109

256

28

2.2

26

178

28

30

37

46

178

512

29

1.8

26

90

28

30

37

44

90

1024

30

1.9

28

112

30

31

39

45

112

8192

50

2.2

47

86

50

51

59

67

86

63000

105

3.3

101

183

104

110

117

129

183

100000

150

3.8

144

311

148

156

162

181

311

500000

592

53.3

506

959

597

672

709

781

959

1048576

1214

108.7

1007

1760

1196

1350

1450

1514

1760

1548576

1759

161.3

1468

2514

1731

1965

2125

2514

2514

4194304

4533

472.6

4107

6708

4218

5344

5527

6708

6708

10485760

11346

248.0

11258

17502

11326

11375

11410

17502

17502

Perftest Scripts

To produce these tests, we executed RTI Perftest for C++98. The exact commands used can be found here:

Publisher Side

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
sudo /set_lat_mode.sh

echo EXECUTABLE IS $1
export executable=$1

echo OUTPUT PATH IS $2
export output_folder=$2

export exec_time=30
export nic=172.16.0.1
export peer=172.16.0.2
export pub_string="-pub \
        -transport TCP\
        -peer 0@tcpv4_lan://$peer:7400 \
        -nic $nic \
        -noPrint \
        -noOutputHeaders \
        -exec $exec_time \
        -initialBurst 100\
        -noXML\
        -latencyTest"

mkdir -p $output_folder

echo ">> UNKEYED BE"
export my_file=$output_folder/lat_shmem_pub_unkeyed_be.csv
touch $my_file
for DATALEN in 32 64 128 256 512 1024 8192 63000; do
    export command="\
    $executable -best -datalen $DATALEN $pub_string"
    echo $command
    $command >> $my_file;
    sleep 3;
done
sleep 5;

echo ">> UNKEYED REL"
export my_file=$output_folder/lat_shmem_pub_unkeyed_rel.csv
touch $my_file
for DATALEN in 32 64 128 256 512 1024 8192 63000 100000 500000 1048576 1548576 4194304 10485760; do
    export command="\
    $executable -datalen $DATALEN $pub_string"
    echo $command
    $command >> $my_file;
    sleep 3;
done
sleep 5;

echo ">> KEYED BE"
export my_file=$output_folder/lat_shmem_pub_keyed_be.csv
touch $my_file
for DATALEN in 32 64 128 256 512 1024 8192 63000; do
    export command="\
    $executable -best -keyed -instances 100000 -datalen $DATALEN $pub_string"
    echo $command
    $command >> $my_file;
    sleep 3;
done
sleep 5;

echo ">> KEYED REL"
export my_file=$output_folder/lat_shmem_pub_keyed_rel.csv
touch $my_file
for DATALEN in 32 64 128 256 512 1024 8192 63000; do
    export command="\
    $executable -keyed -instances 100000 -datalen $DATALEN $pub_string"
    echo $command
    $command >> $my_file;
    sleep 3;
done

Subscriber Side

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
sudo /set_lat_mode.sh

echo EXECUTABLE IS $1
export executable=$1

echo OUTPUT PATH IS $2
export output_folder=$2

export nic=172.16.0.2
export peer=172.16.0.1
export sub_string="-sub \
        -transport TCP\
        -peer 0@tcpv4_lan://$peer:7400 \
        -nic $nic \
        -noPrint \
        -noOutputHeaders \
        -noXML"

mkdir -p $output_folder

echo ">> UNKEYED BE"
export my_file=$output_folder/lat_shmem_sub_unkeyed_be.csv
touch $my_file
for DATALEN in 32 64 128 256 512 1024 8192 63000; do
    export command="\
    $executable -best $sub_string -datalen $DATALEN"
    echo $command
    $command >> $my_file;
    sleep 10;
done
sleep 5;

echo ">> UNKEYED REL"
export my_file=$output_folder/lat_shmem_sub_unkeyed_rel.csv
touch $my_file
for DATALEN in 32 64 128 256 512 1024 8192 63000 100000 500000 1048576 1548576 4194304 10485760; do
    export command="\
    $executable $sub_string -datalen $DATALEN"
    echo $command
    $command >> $my_file;
    sleep 10;
done
sleep 5;

echo ">> KEYED BE"
export my_file=$output_folder/lat_shmem_sub_keyed_be.csv
touch $my_file
for DATALEN in 32 64 128 256 512 1024 8192 63000; do
    export command="\
    $executable -best -keyed -instances 100000 $sub_string -datalen $DATALEN"
    echo $command
    $command >> $my_file;
    sleep 10;
done
sleep 5;

echo ">> KEYED REL"
export my_file=$output_folder/lat_shmem_sub_keyed_rel.csv
touch $my_file
for DATALEN in 32 64 128 256 512 1024 8192 63000; do
    export command="\
    $executable -keyed -instances 100000 $sub_string -datalen $DATALEN"
    echo $command
    $command >> $my_file;
    sleep 10;
done

Test Hardware

The following hardware was used to perform these tests:

Linux Nodes

Processor: Intel® Xeon® E-2186G 3.8GHz, 12M cache, 6C/12T, turbo (95W)
RAM: 16GB 2666MT/s DDR4 ECC UDIMM
NIC 1: Intel X550 Dual Port 10GbE BASE-T Adapter, PCIe Full Height
NIC 2: Intel Ethernet I350 Dual Port 1GbE BASE-T Adapter, PCIe Low Profile
OS: Ubuntu 18.04 -- gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

Switch

Dell Networking S4048T-ON, 48x 10GBASE-T and 6x 40GbE QSFP+ ports, IO to PSU air, 2x AC PSU, OS9

Keyed, TCP 10Gbps, C++98

The graph below shows the one-way latency without load between a Publisher and a Subscriber running in two Linux nodes in a 10Gbps network. The numbers are for best-effort as well as strict reliable reliability scenarios.

Note

We use the median (50th percentile) instead of the average in order to get a more stable measurement that does not account for spurious outliers. We also calculate the average value and other percentile values, which can be seen in the Detailed Statistics section below.

Detailed Statistics

The following tables contain the raw numbers presented by RTI Perftest. These numbers are the exact output with no further processing.

  • Best Effort

Sample Size (Bytes)

Avg (μs)

Std (μs)

Min (μs)

Max (μs)

50% (μs)

90% (μs)

99% (μs)

99.99% (μs)

99.9999% (μs)

32

28

1.6

26

221

28

30

35

44

221

64

29

1.4

27

220

29

30

35

42

220

128

29

1.5

27

225

29

30

35

47

225

256

30

1.4

27

223

30

31

34

42

223

512

30

2.0

27

222

29

31

39

46

222

1024

32

1.9

29

230

31

33

41

48

230

8192

51

2.2

48

255

51

53

60

69

255

63000

106

2.4

102

307

105

107

115

130

307

  • Reliable

Sample Size (Bytes)

Avg (μs)

Std (μs)

Min (μs)

Max (μs)

50% (μs)

90% (μs)

99% (μs)

99.99% (μs)

99.9999% (μs)

32

31

2.5

28

224

30

33

40

49

224

64

31

2.7

28

222

31

34

41

54

222

128

32

2.5

29

234

31

33

41

50

234

256

31

2.6

28

233

31

34

41

50

233

512

32

2.4

29

239

32

34

41

51

239

1024

34

2.6

30

248

33

36

43

54

248

8192

53

2.3

50

261

53

55

62

73

261

63000

108

3.6

103

317

107

114

120

137

317

Perftest Scripts

To produce these tests, we executed RTI Perftest for C++98. The exact commands used can be found here:

Publisher Side

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
sudo /set_lat_mode.sh

echo EXECUTABLE IS $1
export executable=$1

echo OUTPUT PATH IS $2
export output_folder=$2

export exec_time=30
export nic=172.16.0.1
export peer=172.16.0.2
export pub_string="-pub \
        -transport TCP\
        -peer 0@tcpv4_lan://$peer:7400 \
        -nic $nic \
        -noPrint \
        -noOutputHeaders \
        -exec $exec_time \
        -initialBurst 100\
        -noXML\
        -latencyTest"

mkdir -p $output_folder

echo ">> UNKEYED BE"
export my_file=$output_folder/lat_shmem_pub_unkeyed_be.csv
touch $my_file
for DATALEN in 32 64 128 256 512 1024 8192 63000; do
    export command="\
    $executable -best -datalen $DATALEN $pub_string"
    echo $command
    $command >> $my_file;
    sleep 3;
done
sleep 5;

echo ">> UNKEYED REL"
export my_file=$output_folder/lat_shmem_pub_unkeyed_rel.csv
touch $my_file
for DATALEN in 32 64 128 256 512 1024 8192 63000 100000 500000 1048576 1548576 4194304 10485760; do
    export command="\
    $executable -datalen $DATALEN $pub_string"
    echo $command
    $command >> $my_file;
    sleep 3;
done
sleep 5;

echo ">> KEYED BE"
export my_file=$output_folder/lat_shmem_pub_keyed_be.csv
touch $my_file
for DATALEN in 32 64 128 256 512 1024 8192 63000; do
    export command="\
    $executable -best -keyed -instances 100000 -datalen $DATALEN $pub_string"
    echo $command
    $command >> $my_file;
    sleep 3;
done
sleep 5;

echo ">> KEYED REL"
export my_file=$output_folder/lat_shmem_pub_keyed_rel.csv
touch $my_file
for DATALEN in 32 64 128 256 512 1024 8192 63000; do
    export command="\
    $executable -keyed -instances 100000 -datalen $DATALEN $pub_string"
    echo $command
    $command >> $my_file;
    sleep 3;
done

Subscriber Side

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
sudo /set_lat_mode.sh

echo EXECUTABLE IS $1
export executable=$1

echo OUTPUT PATH IS $2
export output_folder=$2

export nic=172.16.0.2
export peer=172.16.0.1
export sub_string="-sub \
        -transport TCP\
        -peer 0@tcpv4_lan://$peer:7400 \
        -nic $nic \
        -noPrint \
        -noOutputHeaders \
        -noXML"

mkdir -p $output_folder

echo ">> UNKEYED BE"
export my_file=$output_folder/lat_shmem_sub_unkeyed_be.csv
touch $my_file
for DATALEN in 32 64 128 256 512 1024 8192 63000; do
    export command="\
    $executable -best $sub_string -datalen $DATALEN"
    echo $command
    $command >> $my_file;
    sleep 10;
done
sleep 5;

echo ">> UNKEYED REL"
export my_file=$output_folder/lat_shmem_sub_unkeyed_rel.csv
touch $my_file
for DATALEN in 32 64 128 256 512 1024 8192 63000 100000 500000 1048576 1548576 4194304 10485760; do
    export command="\
    $executable $sub_string -datalen $DATALEN"
    echo $command
    $command >> $my_file;
    sleep 10;
done
sleep 5;

echo ">> KEYED BE"
export my_file=$output_folder/lat_shmem_sub_keyed_be.csv
touch $my_file
for DATALEN in 32 64 128 256 512 1024 8192 63000; do
    export command="\
    $executable -best -keyed -instances 100000 $sub_string -datalen $DATALEN"
    echo $command
    $command >> $my_file;
    sleep 10;
done
sleep 5;

echo ">> KEYED REL"
export my_file=$output_folder/lat_shmem_sub_keyed_rel.csv
touch $my_file
for DATALEN in 32 64 128 256 512 1024 8192 63000; do
    export command="\
    $executable -keyed -instances 100000 $sub_string -datalen $DATALEN"
    echo $command
    $command >> $my_file;
    sleep 10;
done

Test Hardware

The following hardware was used to perform these tests:

Linux Nodes

Processor: Intel® Xeon® E-2186G 3.8GHz, 12M cache, 6C/12T, turbo (95W)
RAM: 16GB 2666MT/s DDR4 ECC UDIMM
NIC 1: Intel X550 Dual Port 10GbE BASE-T Adapter, PCIe Full Height
NIC 2: Intel Ethernet I350 Dual Port 1GbE BASE-T Adapter, PCIe Low Profile
OS: Ubuntu 18.04 -- gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

Switch

Dell Networking S4048T-ON, 48x 10GBASE-T and 6x 40GbE QSFP+ ports, IO to PSU air, 2x AC PSU, OS9