Shared Memory

The following tests have been performed by executing the RTI Perftest C++98 benchmark application between two applications within the same node. The communication has been set to use Shared Memory (SHMEM).

Find the information about the hardware, network and command-line parameters after each of the tests.

Unkeyed, Shared Memory, C++98

Linux x64Linux ARM v8

Minimum LatencyMaximum Throughput

The graph below shows the one-way latency without load between a Publisher and a Subscriber running in two processes within a single node. The numbers are for best-effort as well as strict reliable reliability scenarios.

Note

We use the median (50th percentile) instead of the average in order to get a more stable measurement that does not account for spurious outliers. We also calculate the average value and other percentile values, which can be seen in the Detailed Statistics section below.

Detailed Statistics

The following tables contain the raw numbers presented by RTI Perftest. These numbers are the exact output with no further processing.

• Best Effort

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)
32	9	0.6	8	124	8	9	10	17	124
64	8	0.6	8	48	8	9	10	17	48
128	8	0.4	8	48	8	9	9	12	48
256	8	0.6	8	49	8	9	10	17	49
512	8	0.5	8	47	8	9	9	12	47
1024	8	0.6	8	49	8	9	9	12	49
2048	9	0.4	8	49	9	9	10	17	49
4096	9	0.4	8	58	9	9	10	18	58
8192	9	0.5	9	51	9	10	10	16	51
16384	11	0.6	10	49	10	11	11	21	49
32768	13	0.6	12	155	13	13	14	30	155
63000	16	0.7	15	67	16	17	18	47	67

• Reliable

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)
32	10	1.1	9	48	10	12	14	17	48
64	10	1.1	9	48	9	12	14	20	48
128	10	1.3	8	91	9	12	17	20	91
256	10	1.0	9	48	10	12	14	18	48
512	10	1.1	9	55	10	12	17	20	55
1024	10	1.0	9	48	10	12	14	18	48
2048	10	1.0	9	97	10	13	14	18	97
4096	11	1.1	9	51	10	13	14	19	51
8192	11	1.0	9	49	11	13	15	21	49
16384	12	0.9	11	51	12	14	15	24	51
32768	15	1.2	13	125	14	16	19	36	125
63000	19	1.1	17	71	18	19	23	55	71
100000	28	3.8	25	130	27	28	46	83	130
500000	91	11.1	78	523	90	92	147	354	523
1048576	268	66.2	161	1077	285	294	530	742	1077
1548576	410	86.0	248	1570	434	446	639	1111	1570
4194304	1291	195.2	846	4250	1361	1385	1405	4250	4250
10485760	2692	446.6	2454	10956	2586	3547	3687	10956	10956

Perftest Scripts

To produce these tests, we executed RTI Perftest for C++98. The exact script used can be found here:

echo EXECUTABLE IS $1
export executable=$1
export folder_base="$(dirname "${executable}")"/../../..

echo OUTPUT PATH IS $2
export output_folder=$2
export pub_sub="pub"
export lat_thr="lat"
export num_reps="1 2 3 4"
export dataLens="32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000"
export domain="2"

if [[ -z "$3" ]]; then
    echo "You need a third argument with publisher or subscriber"
    exit -1
else
    if [[ "$3" == "publisher" ]]; then
        echo "Publisher"
        export pub_sub="pub"
        
    elif [[ "$3" == "subscriber" ]]; then
        echo "Subscriber"
        export pub_sub="sub"
    else
        echo "It must be either publisher or subscriber"
        exit -1
    fi
fi

if [[ -z "$4" ]]; then
    echo "You need a forth argument with lat or thr"
    exit -1
else
    if [[ "$4" == "thr" ]]; then
        echo "Throughput test"
        export ${lat_thr}_thr="thr"

    elif [[ "$4" == "lat" ]]; then
        echo "Latency test"
        export ${lat_thr}_thr="lat"
    else
        echo "It must be either lat or thr"
        exit -1
    fi
fi

sudo /set_${lat_thr}_mode.sh
sleep 10

export exec_time=20

export pub_string="-pub \
        -transport SHMEM \
        -noPrint \
        -exec $exec_time"

if [[ ${lat_thr} == "lat" ]]; then
    export pub_string="$pub_string \
        -latencyTest"
fi

export sub_string="-sub \
        -transport SHMEM \
        -noPrint"

if [[ "$pub_sub" == "pub" ]]; then
    echo "Publisher side"
    export commands_string=${pub_string}
    export core=0
else
    echo "Subscriber side"
    export commands_string=${sub_string}
    export core=1
fi

mkdir -p $output_folder
cd $folder_base

export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_rel.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in 32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000 100000 500000 1048576 1548576 4194304 10485760; do
        export command="taskset -c $core \
        $executable -domain $domain -datalen $DATALEN $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_be.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in ${dataLens}; do
        export command="taskset -c $core \
        $executable -domain $domain -datalen $DATALEN -best $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

export my_file=$output_folder/${lat_thr}_${pub_sub}_keyed_rel.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in ${dataLens}; do
        export command="taskset -c $core \
        $executable -domain $domain -datalen $DATALEN -keyed -instances 100000 $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

export my_file=$output_folder/${lat_thr}_${pub_sub}_keyed_be.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in ${dataLens}; do
        export command="taskset -c $core \
        $executable -domain $domain -datalen $DATALEN -keyed -instances 100000 -best $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

if [[ ${lat_thr} == "thr" ]]; then

    if [[ "$pub_sub" == "pub" ]]; then
        echo "Publisher side"
        export commands_string="${commands_string} -batchSize 0"
    fi

    export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_be_noBatch.csv
    touch $my_file
    export extra_args=""
    for index in ${num_reps}; do
        for DATALEN in ${dataLens}; do
            export command="taskset -c $core \
            $executable -domain $domain -datalen $DATALEN -best $commands_string $extra_args"
            echo $command ---- $index
            $command >> $my_file;
            if [[ "$domain" == "1" ]]; then
                export domain="2"
            else
                export domain="1"
            fi
            export extra_args=" -noOutputHeaders "
        done
    done

    export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_rel_noBatch.csv
    touch $my_file
    export extra_args=""
    for index in ${num_reps}; do
        for DATALEN in ${dataLens}; do
            export command="taskset -c $core \
            $executable -domain $domain -datalen $DATALEN $commands_string $extra_args"
            echo $command ---- $index
            $command >> $my_file;
            if [[ "$domain" == "1" ]]; then
                export domain="2"
            else
                export domain="1"
            fi
            export extra_args=" -noOutputHeaders "
        done
    done

fi

Test Hardware

The following hardware was used to perform these tests:

Linux Nodes

Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 20.04 -- gcc 9.3.0

Switch

Dell 2048 -- 10Gbps switch

Keyed, Shared Memory, C++98

Linux x64Linux ARM v8

Minimum LatencyMaximum Throughput

The graph below shows the one-way latency without load between a Publisher and a Subscriber running in two processes within a single node. The numbers are for best-effort as well as strict reliable reliability scenarios.

Note

We use the median (50th percentile) instead of the average in order to get a more stable measurement that does not account for spurious outliers. We also calculate the average value and other percentile values, which can be seen in the Detailed Statistics section below.

Detailed Statistics

The following tables contain the raw numbers presented by RTI Perftest. These numbers are the exact output with no further processing.

• Best Effort

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)
32	11	0.7	9	162	11	11	13	19	162
64	11	0.7	9	158	10	11	12	18	158
128	11	0.7	9	165	11	11	13	20	165
256	11	0.6	9	164	11	11	13	20	164
512	11	0.6	9	165	11	11	13	19	165
1024	11	0.6	10	164	11	12	13	20	164
2048	11	0.6	10	164	11	12	13	20	164
4096	11	0.7	10	169	11	12	13	20	169
8192	12	0.7	10	166	12	12	14	21	166
16384	13	0.7	11	168	13	14	15	22	168
32768	15	0.8	14	168	15	16	17	24	168
63000	19	0.8	18	174	19	20	21	29	174

• Reliable

Sample Size (Bytes)	Avg (μs)	Std (μs)	Min (μs)	Max (μs)	50% (μs)	90% (μs)	99% (μs)	99.99% (μs)	99.9999% (μs)
32	13	1.0	11	170	12	14	17	22	170
64	13	1.0	11	171	12	14	17	22	171
128	13	1.0	11	170	12	14	17	22	170
256	13	1.0	11	170	12	14	17	22	170
512	13	1.0	11	170	12	14	17	22	170
1024	13	1.0	11	170	12	14	17	22	170
2048	13	1.0	11	169	12	14	17	22	169
4096	13	0.9	12	172	13	14	17	22	172
8192	14	1.0	12	173	13	15	18	22	173
16384	15	1.0	13	227	14	15	19	25	227
32768	17	1.0	16	179	17	18	22	27	179
63000	22	1.6	19	171	21	22	32	38	171

Perftest Scripts

To produce these tests, we executed RTI Perftest for C++98. The exact script used can be found here:

echo EXECUTABLE IS $1
export executable=$1
export folder_base="$(dirname "${executable}")"/../../..

echo OUTPUT PATH IS $2
export output_folder=$2
export pub_sub="pub"
export lat_thr="lat"
export num_reps="1 2 3 4"
export dataLens="32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000"
export domain="2"

if [[ -z "$3" ]]; then
    echo "You need a third argument with publisher or subscriber"
    exit -1
else
    if [[ "$3" == "publisher" ]]; then
        echo "Publisher"
        export pub_sub="pub"
        
    elif [[ "$3" == "subscriber" ]]; then
        echo "Subscriber"
        export pub_sub="sub"
    else
        echo "It must be either publisher or subscriber"
        exit -1
    fi
fi

if [[ -z "$4" ]]; then
    echo "You need a forth argument with lat or thr"
    exit -1
else
    if [[ "$4" == "thr" ]]; then
        echo "Throughput test"
        export ${lat_thr}_thr="thr"

    elif [[ "$4" == "lat" ]]; then
        echo "Latency test"
        export ${lat_thr}_thr="lat"
    else
        echo "It must be either lat or thr"
        exit -1
    fi
fi

sudo /set_${lat_thr}_mode.sh
sleep 10

export exec_time=20

export pub_string="-pub \
        -transport SHMEM \
        -noPrint \
        -exec $exec_time"

if [[ ${lat_thr} == "lat" ]]; then
    export pub_string="$pub_string \
        -latencyTest"
fi

export sub_string="-sub \
        -transport SHMEM \
        -noPrint"

if [[ "$pub_sub" == "pub" ]]; then
    echo "Publisher side"
    export commands_string=${pub_string}
    export core=0
else
    echo "Subscriber side"
    export commands_string=${sub_string}
    export core=1
fi

mkdir -p $output_folder
cd $folder_base

export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_rel.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in 32 64 128 256 512 1024 2048 4096 8192 16384 32768 63000 100000 500000 1048576 1548576 4194304 10485760; do
        export command="taskset -c $core \
        $executable -domain $domain -datalen $DATALEN $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_be.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in ${dataLens}; do
        export command="taskset -c $core \
        $executable -domain $domain -datalen $DATALEN -best $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

export my_file=$output_folder/${lat_thr}_${pub_sub}_keyed_rel.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in ${dataLens}; do
        export command="taskset -c $core \
        $executable -domain $domain -datalen $DATALEN -keyed -instances 100000 $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

export my_file=$output_folder/${lat_thr}_${pub_sub}_keyed_be.csv
touch $my_file
export extra_args=""
for index in ${num_reps}; do
    for DATALEN in ${dataLens}; do
        export command="taskset -c $core \
        $executable -domain $domain -datalen $DATALEN -keyed -instances 100000 -best $commands_string $extra_args"
        echo $command ---- $index
        $command >> $my_file;
        if [[ "$domain" == "1" ]]; then
            export domain="2"
        else
            export domain="1"
        fi
        export extra_args=" -noOutputHeaders "
    done
done

if [[ ${lat_thr} == "thr" ]]; then

    if [[ "$pub_sub" == "pub" ]]; then
        echo "Publisher side"
        export commands_string="${commands_string} -batchSize 0"
    fi

    export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_be_noBatch.csv
    touch $my_file
    export extra_args=""
    for index in ${num_reps}; do
        for DATALEN in ${dataLens}; do
            export command="taskset -c $core \
            $executable -domain $domain -datalen $DATALEN -best $commands_string $extra_args"
            echo $command ---- $index
            $command >> $my_file;
            if [[ "$domain" == "1" ]]; then
                export domain="2"
            else
                export domain="1"
            fi
            export extra_args=" -noOutputHeaders "
        done
    done

    export my_file=$output_folder/${lat_thr}_${pub_sub}_unkeyed_rel_noBatch.csv
    touch $my_file
    export extra_args=""
    for index in ${num_reps}; do
        for DATALEN in ${dataLens}; do
            export command="taskset -c $core \
            $executable -domain $domain -datalen $DATALEN $commands_string $extra_args"
            echo $command ---- $index
            $command >> $my_file;
            if [[ "$domain" == "1" ]]; then
                export domain="2"
            else
                export domain="1"
            fi
            export extra_args=" -noOutputHeaders "
        done
    done

fi

Test Hardware

The following hardware was used to perform these tests:

Linux Nodes

Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 20.04 -- gcc 9.3.0

Switch

Dell 2048 -- 10Gbps switch