1.3.3. RTI Persistence Service

1.3.3.1. Unkeyed, UDPv4 10Gbps Network

Linux x64 (10Gbps Network)

UnkeyedKeyed

Direct CommunicationNo Direct Communication

Minimum LatencyMaximum Throughput (No Batching)

The graph below shows the one-way latency without load between a Publisher and a Subscriber running in two Linux nodes in a 10Gbps network. The numbers are for best-effort as well as strict reliable reliability scenarios.

Note

We use the median (50th percentile) instead of the average in order to get a more stable measurement that does not account for spurious outliers. We also calculate the average value and other percentile values, which can be seen in the Detailed Statistics section below.

Detailed Statistics

The following tables contain the raw numbers presented by RTI Perftest. These numbers are the exact output with no further processing.

• Reliable

How to read the latency tables and plots:

• Avg: Mean of the avg values for N runs per sample size.
• Std: Standard deviation of the avg values across N runs. High values suggest inconsistency across runs.
• Plot representation: for each sample size, all N individual values are plotted as circles. The line connects the average of the N values (‘Avg’ in the table) across all sample sizes.

Note: each row represents a different sample size, and the values are aggregated across N runs for that size.

Sample Size	Avg	P-50	Std	Min	Max	P-99	P-9999
32	16.99	16.8	1.11	16.03	87.25	19.69	48.51
256	18	17.59	6.01	16.77	4944	30.97	54.41
512	20.3	19.72	3.19	17.44	116.47	37.03	62.63
1024	26.02	26.48	12.33	18.45	4748.38	45.47	74.71
8192	44.88	39.51	13.6	36.21	4703.96	79.11	123.24
63000	187.54	187.56	2.5	121.29	221.98	191.42	200.03

Perftest Scripts

To produce these tests, we executed RTI Perftest for C++98 and Persistence Service. The exact script used can be found here:

#!/bin/bash
filename=$0

# This is in order to know where is the script and where is the repo base folder
script_location=$(cd "$(dirname "$filename")" || exit 255; pwd)
repo_location=`cd "$script_location/../.."; pwd`

################################################################################

# We will use some colors to improve visibility of errors and information
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
NC='\033[0m'
INFO_TAG="${GREEN}[INFO]:${NC}"
WARNING_TAG="${YELLOW}[WARNING]:${NC}"
ERROR_TAG="${RED}[ERROR]:${NC}"

# Use key-based SSH only and fail fast instead of prompting for a password.
export ssh="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR -o BatchMode=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o KbdInteractiveAuthentication=no -o NumberOfPasswordPrompts=0 -o ConnectTimeout=8 -o ConnectionAttempts=1"

################################################################################

# Default Lab to use.
export LAB="LINUX_X64"

# TEST RELATED CONSTANTS

# Some default values for certain variables like openssl
export openssl_1_version="openssl-1.1.1s"
export openssl_3_version="openssl-3.0.12"
export openssl_309_version="openssl-3.0.9"
export openssl_3011_version="openssl-3.0.11"
export openssl_350_version="openssl-3.5.0"
export openssl_355_version="openssl-3.5.5"
export wolfssl_version="wolfssl-5.5.1"
export JAVA_LINUX_BIN_PATH="/local/apps/Java/PLATFORMSDK/linux64/jdk-11.0.7+10/bin"
export DOTNET6_LINUX_BIN_PATH="/local/apps/dotnet/dotnet-sdk-5.0.103-linux-x64"

export middleware="pro"
export TEST_KIND="throughput"
export KEYED_UNKEYED="unkeyed"
export MULTICAST_UNICAST="unicast"
export timeout_value=200
export machines_to_use=""

################################################################################
### FUNCTIONS

### Cleaning funtion:

function clean_machines () {

    if ! validate_ssh_connectivity_all_machines; then
        echo -e "$ERROR_TAG Aborting cleanup due to SSH connectivity failures."
        return 1
    fi

    echo -e "${YELLOW}Clean Machines${NC}: Launching cleanup in parallel for ${number_machines} machine(s)..."

    for ((i = 0 ; i < number_machines ; i++)); do
        (
            $ssh "${machines[$i]}" pkill perftest_cpp >/dev/null 2>&1 || true
            # stdout and stderr in tcsh to /dev/null
            $ssh "${machines[$i]}" killall -s KILL /bin/bash >/dev/null 2>&1 || true

            if [[ "${machines[$i]}" == "$tcpdump_machine" && "$TCP_DUMP" == "1" ]]; then
                $ssh "${machines[$i]}" sudo pkill tcpdump >/dev/null 2>&1 || true
            fi
        ) &
    done
    wait

    echo -e "${YELLOW}Clean Machines${NC}: Base cleanup ${GREEN}Done.${NC}"

    if [[ "$PERSISTENCE_SERVICE" == "1" || "$RECORDING_SERVICE" == "1" || "$ROUTING_SERVICE" == "1" ]]; then
        echo -e "${YELLOW}Clean Machines${NC}: Cleaning infrastructure services in parallel..."
        for ((i = 0 ; i < number_machines ; i++)); do
            (
                $ssh "${machines[$i]}" pkill rtipersistenc >/dev/null 2>&1 || true
                $ssh "${machines[$i]}" pkill rtirecording >/dev/null 2>&1 || true
                $ssh "${machines[$i]}" pkill rtirouting >/dev/null 2>&1 || true
                # stdout and stderr in tcsh to /dev/null
                $ssh "${machines[$i]}" killall -s KILL /bin/bash >/dev/null 2>&1 || true
            ) &
        done
        wait
        echo -e "${YELLOW}Clean Machines${NC}: Infrastructure services cleanup ${GREEN}Done.${NC}"

    fi
    if [[ "${THR_LAT_MODE}" == "THR" ]]; then
        for ((i = 0 ; i < number_machines ; i++)); do
            echo -e "$INFO_TAG ssh ${machines[$i]} sudo /set_thr_mode.sh &"
            $ssh "${machines[$i]}" sudo /set_thr_mode.sh &
        done
        # Wait for all the background processes to finish
        wait
    elif [[ "${THR_LAT_MODE}" == "LAT" ]]; then
        for ((i = 0 ; i < $number_machines ; i++)); do
            echo -e "$INFO_TAG ssh ${machines[$i]} sudo /set_lat_mode.sh &"
            $ssh "${machines[$i]}" sudo /set_lat_mode.sh &
        done
        # Wait for all the background processes to finish
        wait
    fi
}

### Reboot Machines:

function reboot_machines () {
    for ((i = 0 ; i < $number_machines ; i++)); do
        command="$ssh ${machines[$i]} sudo reboot"
        echo -e "$INFO_TAG Executing $command"
        $command
    done
}


# trap ctrl-c and call ctrl_c()
trap 'ctrl_c' SIGINT

function ctrl_c() {
    echo -e " ${RED}TRAPPED CTRL-C !!!!!!!!!!! ${NC}"
    clean_machines || echo -e "$WARNING_TAG Cleanup skipped due to SSH connectivity failures."
    killall -s KILL /bin/bash
}

### Check if I can access via SSH to a machine

function check_ssh_machine() {
    command="$ssh $1 exit"
    $command
    if [[ "$?" == "0" ]]; then
        echo "true"
    else
        echo "false"
    fi
}

function validate_ssh_connectivity_all_machines() {
    local failed_machines=()
    local failed_machines_file
    failed_machines_file=$(mktemp)

    echo -e "$INFO_TAG Checking SSH connectivity for all selected machines..."
    for ((i = 0 ; i < number_machines ; i++)); do
        (
            if [[ "$(check_ssh_machine "${machines[$i]}")" != "true" ]]; then
                echo "${machines[$i]}" >> "$failed_machines_file"
            fi
        ) &
    done
    wait

    while IFS= read -r failed_machine; do
        failed_machines+=("$failed_machine")
    done < "$failed_machines_file"
    rm -f "$failed_machines_file"

    if [[ "${#failed_machines[@]}" -gt 0 ]]; then
        echo -e "$ERROR_TAG SSH connectivity check failed for:"
        for machine in "${failed_machines[@]}"; do
            echo -e " - ${machine}"
        done
        return 1
    fi

    echo -e "$INFO_TAG SSH connectivity check passed."
    return 0
}

function append_ld_library_path_export() {
    local script_file=$1
    local lib_path=$2

    if [[ "$NO_LD_PATH_CHECKS" == "1" ]]; then
        echo -e "export LD_LIBRARY_PATH=${lib_path}:\$LD_LIBRARY_PATH;" >> "$script_file"
    else
        echo -e "if [[ -d \"${lib_path}\" ]]; then export LD_LIBRARY_PATH=${lib_path}:\$LD_LIBRARY_PATH; fi" >> "$script_file"
    fi
}

### Calculate the perftest executable name and check if exists

function calculate_perftest_binary() {
    if [[ "$LANGUAGE" == "c++98" ]]; then
        export perftest_exec="${PERFTESTHOME}/bin/${PLATFORM}/${RELEASE_DEBUG}/perftest_cpp"
    elif [[ "$LANGUAGE" == "c++11" ]]; then
        export perftest_exec="${PERFTESTHOME}/bin/${PLATFORM}/${RELEASE_DEBUG}/perftest_cpp11"
    elif [[ "$LANGUAGE" == "java" ]]; then
        export perftest_exec="${PERFTESTHOME}/bin/${RELEASE_DEBUG}/perftest_java.sh"
    elif [[ "$LANGUAGE" == "cs" ]]; then
        export perftest_exec="${PERFTESTHOME}/bin/${RELEASE_DEBUG}/perftest_cs"
    fi

    if [[ ! -f $perftest_exec ]]; then
        echo -e "$ERROR_TAG: The Perftest binary does not exist: $perftest_exec"
        exit -1
    fi

    if [[ "$RECORDING_SERVICE" == "1" ]]; then
        export recording_wrapper_exec="${PERFTESTHOME}/resource/recording_service/recordingservice_wrapper.sh"
    fi

    if [[ "$ROUTING_SERVICE" == "1" ]]; then
        export routing_service_exec="${PERFTESTHOME}/resource/routing_service/routingservice_wrapper.sh"
    fi

    if [[ "$PERSISTENCE_SERVICE" == "1" ]]; then
        export persistence_service_exec="${PERFTESTHOME}/resource/persistence_service/persistenceService_wrapper.sh"
    fi

}

### Calculate the folder where to store the results

function calculate_store_folder() {

    export store_point="${PERFTESTNUMBERSHOME}/results/${middleware}/${BRANCH}/network_performance/core/${LANGUAGE}/${TRANSPORT}"
}

calculate_store_folder

################################################################################

while [ "$1" != "" ]; do
    case $1 in
        --nddshome)
            export NDDSHOME=$2
            shift
            ;;
        --perftest-home)
            export PERFTESTHOME=$2
            shift
            ;;
        --perftest-numbers-home)
            export PERFTESTNUMBERSHOME=$2
            shift
            ;;
        --transport)
            export TRANSPORT=$2
            shift
            ;;
        --arch1 | --platform)
            export PLATFORM=$2
            shift
            ;;
        --arch2)
            export ARCH2=$2
            shift
            ;;
        --branch)
            export BRANCH=$2
            shift
            ;;
        --id)
            export BUILD_ID=$2
            shift
            ;;
        --openssl-libs)
            export OPENSSLPATHLIB=$2
            shift
            ;;
        --multicast)
            export MULTICAST_UNICAST="multicast"
            ;;
        --unicast)
            export MULTICAST_UNICAST="unicast"
            ;;
        --best-effort | --be)
            export BEST_EFFORT="1"
            ;;
        --reliable | --rel)
            export BEST_EFFORT=""
            ;;
        --num-publishers)
            export publishers=$2
            shift
            ;;
        --num-subscribers)
            export subscribers=$2
            shift
            ;;
        --output-folder)
            export results_output=$2
            shift
            ;;
        --execution-time)
            export execution_time=$2
            shift
            ;;
        --data-length | --datalen)
            export datalen=$2
            shift
            ;;
        --dry)
            export DRY=1
            ;;
        --clean)
            export CLEAN_MACHINES=1
            ;;
        --reboot)
            reboot_machines
            exit 0
            ;;
        --thr-mode)
            export THR_LAT_MODE="THR"
            ;;
        --lat-mode)
            export THR_LAT_MODE="LAT"
            ;;
        --leave-pub-alone)
            export PUB_ALONE=1
            ;;
        --no-positive-acks)
            export NO_POS_ACKS=1
            ;;
        --security-gov)
            export SECURITY=1
            ;;
        --cft)
            export CFT=1
            ;;
        --instances)
            export INSTANCES=$2
            shift
            ;;
        --latency-test | --latency)
            export TEST_KIND="latency"
            ;;
        --throughput)
            export TEST_KIND="throughput"
            ;;
        --keyed)
            export KEYED_UNKEYED="keyed"
            ;;
        --unkeyed)
            export KEYED_UNKEYED="unkeyed"
            ;;
        --batch-size)
            export BATCH_SIZE=$2
            shift
            ;;
        --routing-service)
            export ROUTING_SERVICE=1
            ;;
        --recording-service)
            export RECORDING_SERVICE=1
            ;;
        --persistence-service | --ps)
            export PERSISTENCE_SERVICE=1
            ;;
        --domain-pub)
            export DOMAIN_PUB=$2
            shift
            ;;
        --domain-sub)
            export DOMAIN_SUB=$2
            shift
            ;;
        --domain-rec)
            export DOMAIN_REC=$2
            shift
            ;;
        --domain-rs)
            export DOMAIN_RS=$2
            shift
            ;;
        --domain-ps)
            export DOMAIN_PS=$2
            shift
            ;;
        --timeout)
            export timeout_value=$2
            shift
            ;;
        --durability)
            export DURABILITY=$2
            shift
            ;;
        --ps-no-direct-communication)
            export NO_DIRECT_COMMUNICATION=1
            ;;
        --test-name)
            export test_name=$2
            shift
            ;;
        --lab | --lab-file)
            export LAB_FILE=$2
            shift
            ;;
        --lab-folder)
            export LAB_FILE="$2/lab.json"
            shift
            ;;
        --machines)
            export machines_to_use=$2
            shift
            ;;
        --low-jitter-settings)
            export low_jitter_settings="1"
            ;;
        --no-ld-path-checks)
            export NO_LD_PATH_CHECKS="1"
            ;;
        --taskset)
            export TASKSET_LIST=$2
            shift
            ;;
        *)
            echo -e "unknown parameter \"$1\""
            exit -1
            ;;
    esac
    shift
done

################################################################################

function validate_input() {


    if [[ "$LAB_FILE" != "" ]]; then
        if [ -f "$LAB_FILE" ]; then
            echo -e "$INFO_TAG Loading lab file $LAB_FILE"
            export machines=($(jq -r 'to_entries[] | .key' "$LAB_FILE"))
            export main_interface=($(jq -r 'to_entries[] | .value.main_interface.interface' $LAB_FILE))
            export main_interface_ip=($(jq -r 'to_entries[] | .value.main_interface.ip' $LAB_FILE))
            export platforms=($(jq -r 'to_entries[] | .value.architecture' $LAB_FILE))
            export number_machines=${#machines[@]}

            if [[ "$machines_to_use" != "" ]]; then
                if ! [[ "$machines_to_use" =~ ^[0-9]+$ ]]; then
                    echo -e "$ERROR_TAG: --machines must be a positive integer."
                    exit -1
                fi

                if [[ "$machines_to_use" -lt "2" ]]; then
                    echo -e "$ERROR_TAG: --machines must be >= 2."
                    exit -1
                fi

                if [[ "$machines_to_use" -gt "$number_machines" ]]; then
                    echo -e "$ERROR_TAG: --machines ($machines_to_use) cannot be greater than machines in lab file ($number_machines)."
                    exit -1
                fi

                machines=("${machines[@]:0:$machines_to_use}")
                main_interface=("${main_interface[@]:0:$machines_to_use}")
                main_interface_ip=("${main_interface_ip[@]:0:$machines_to_use}")
                platforms=("${platforms[@]:0:$machines_to_use}")
                export number_machines=${#machines[@]}
            fi

        else
            echo -e "$ERROR_TAG: The lab file $lab_file does not exist."
            exit -1
        fi
    fi

    if [[ "$CLEAN_MACHINES" == "1" ]]; then
        if ! clean_machines; then
            exit -1
        fi
        exit 0
    fi

    # Lets fill up the test information first, either with the defaults or the data.

    if [[ "$PERFTESTNUMBERSHOME" == "" ]]; then
        export PERFTESTNUMBERSHOME=$repo_location
    fi

    if [[ "$PERFTESTHOME" == "" ]]; then
        echo -e "$ERROR_TAG: We need a perftest home"
        exit -1
    elif [ ! -d "$PERFTESTHOME" ]; then
        echo -e "$ERROR_TAG: The Perftest home $PERFTESTHOME does not exist."
        exit -1
    fi

    if [[ "$NDDSHOME" == "" ]]; then
        echo "Setting NDDSHOME to PERFTESTHOME"
        export NDDSHOME=$PERFTESTHOME
    fi

    if [[ "$TRANSPORT" == "" ]]; then
        export TRANSPORT="UDPv4_10Gbps"
    fi

    if [[ "$LANGUAGE" == "" ]]; then
        export LANGUAGE="c++98"
    fi

    if [[ "$RELEASE_DEBUG" == "" ]]; then
        export RELEASE_DEBUG="release"
    fi

    if [[ "$PLATFORM" == "" ]]; then
        export PLATFORM="x64Linux4gcc8.5.0"
    fi

    if [[ "$ARCH2" != "" ]]; then
        echo -e "$WARNING_TAG: --arch2 will be ignored and $PLATFORM will be used instead"
    fi
    export ARCH2=$PLATFORM

    if [[ "$OPENSSLPATHLIB" == "" ]]; then
        export OPENSSL1PATHLIB="${NDDSHOME}/third_party/${openssl_1_version}/${PLATFORM}/${RELEASE_DEBUG}/lib"
        export OPENSSL3PATHLIB="${NDDSHOME}/third_party/${openssl_3_version}/${PLATFORM}/${RELEASE_DEBUG}/lib"
        export OPENSSL3011PATHLIB="${NDDSHOME}/third_party/${openssl_3011_version}/${PLATFORM}/${RELEASE_DEBUG}/lib"
        export OPENSSL309PATHLIB="${NDDSHOME}/third_party/${openssl_309_version}/${PLATFORM}/${RELEASE_DEBUG}/lib"
        export OPENSSL350PATHLIB="${NDDSHOME}/third_party/${openssl_350_version}/${PLATFORM}/${RELEASE_DEBUG}/lib"
        export OPENSSL355PATHLIB="${NDDSHOME}/third_party/${openssl_355_version}/${PLATFORM}/${RELEASE_DEBUG}/lib"
        export WOLFSSLPATH_LIB="${NDDSHOME}/third_party/${wolfssl_version}/${PLATFORM}/${RELEASE_DEBUG}/lib"
    fi

    if [[ "$BEST_EFFORT" == "1" ]]; then
        export BEST_EFFORT=" -best "
        export be_string="be"
    else 
        export be_string="rel"
    fi

    calculate_perftest_binary
    calculate_store_folder

    if [[ "$TASKSET_LIST" != "" ]]; then
        export taskset_prefix="taskset -c $TASKSET_LIST"
    else
        export taskset_prefix=""
    fi

    if [[ "$publishers" == "" ]]; then
        export publishers=1
    fi

    if [[ "$subscribers" == "" ]]; then
        export subscribers=1
    fi

    if [[ "${MULTICAST_UNICAST}" == "multicast" ]]; then
        export USE_MULTICAST="-multicast"
    fi
    if [[ "$results_output" == "" ]]; then
        export results_output="${repo_location}/results/pro/${BRANCH}/${BUILD_ID}/network_performance"

        if [[ "$ROUTING_SERVICE" == "1" ]]; then
            export results_output="${results_output}/infrastructure_services/routing_service"
        elif [[ "$RECORDING_SERVICE" == "1" ]]; then
            export results_output="${results_output}/infrastructure_services/recording_service"
        elif [[ "$PERSISTENCE_SERVICE" == "1" ]]; then
            export ps_communication_mode="direct"
            if [[ "$NO_DIRECT_COMMUNICATION" == "1" ]]; then
                export ps_communication_mode="no_direct"
            fi
            export results_output="${results_output}/infrastructure_services/persistence_service/${ps_communication_mode}"
        else
            export results_output="${results_output}/core/${LANGUAGE}/N-M"
        fi

        export results_output="${results_output}/${TEST_KIND}/${be_string}/${KEYED_UNKEYED}/${MULTICAST_UNICAST}"
    fi

    if [[ "$execution_time" == "" ]]; then
        export execution_time=30
    fi

    if [[ "$datalen" == "" ]]; then
        export datalen=100
    fi

    if [[ "$NO_POS_ACKS" == "1" ]]; then
        export no_positive_acks="-noPositiveAcks"
        export results_output="${results_output}_no_positive_acks"
    fi

    if [[ "$KEYED_UNKEYED" == "keyed" ]]; then
        export keyed_args="-keyed"
    fi

    if [[ "$INSTANCES" != "" ]]; then
        export instances_args="-instances $INSTANCES"
    elif [[ "$KEYED_UNKEYED" == "keyed" ]]; then
        export instances_args="-instances 200"
    fi

    if [[ "$CFT" != "" ]]; then
        export results_output="${results_output}_cft"
    fi

    export LAT_THR_SCRIPT="/set_thr_mode.sh"
    export latency_count="-latencyCount 10000"
    if [[ "$TEST_KIND" == "latency" ]]; then
        export latency_test="-latencyTest"
        export latency_count=""
        export LAT_THR_SCRIPT="/set_lat_mode.sh"
    fi

    if [[ "$CFT" == "1" ]]; then
        export instances_args="-instances $subscribers"
    fi

    export PATH_TO_GOVERNANCE_FILES_FOLDER=$PERFTESTHOME/resource/secure

    if [[ "$SECURITY" == "1" ]]; then
        export security_params="-secureGovernanceFile $PATH_TO_GOVERNANCE_FILES_FOLDER/signed_PerftestGovernance_RTPSEncrypt.xml"
    fi

    echo -e "$INFO_TAG: TEST INFO:\n"
    echo -e " - Perftest home = $PERFTESTHOME"
    echo -e " - Number of machines = $number_machines"
    echo -e " - Number of publishers = $publishers"
    echo -e " - Number of subscribers = $subscribers"
    echo -e " - OpenSSL Libraries path = $OPENSSLPATHLIB $OPENSSL1PATHLIB $OPENSSL3PATHLIB"

    echo

    export full_test_name="${TRANSPORT}_${be_string}_${publishers}_${subscribers}_${number_machines}"
    if [[ "$test_name" != "" ]]; then
        export full_test_name="${test_name}"
        return
    fi

}

function main() {

    echo -e "\n=== LAUNCHING PERFORMANCE TESTS ===\n"

    # This function will fill up the environmental variables needed to test
    validate_input

    export temporal_output=$repo_location/.test/output
    export temporal_output_files=$repo_location/.test/output_files
    export temporal_scripts=$repo_location/.test/generated_scripts
    rm -rf ${temporal_scripts}/*
    rm -rf ${temporal_output}/*
    rm -rf ${temporal_output_files}/*
    mkdir -p $temporal_output
    mkdir -p $temporal_output_files
    mkdir -p $temporal_scripts
    mkdir -p $results_output

    echo -e "\n$INFO_TAG: Cleaning machines before starting the test:"
    if ! clean_machines; then
        exit -1
    fi

    # Generate scripts to run for every perftest

    echo -e "\n$INFO_TAG: GENERATING SCRIPTS:"
    for ((i = 0 ; i < $number_machines ; i++)); do
        script_file="${temporal_scripts}/${full_test_name}_${machines[$i]}.sh"
        output_file="${temporal_output}/${full_test_name}_${machines[$i]}.txt"
        echo "Creating $script_file"
        touch $script_file
        chmod +x $script_file
        echo -e "#!/bin/bash" >> $script_file
        if [[ "$THR_LAT_MODE" != "" && "$low_jitter_settings" != "1" ]]; then
            echo -e "sudo $LAT_THR_SCRIPT" >> $script_file
            echo -e "sleep 5;" >> $script_file
        fi
        echo -e "cd $PERFTESTHOME;" >> $script_file
        echo -e "export NDDSHOME=${NDDSHOME};" >> $script_file
        append_ld_library_path_export "$script_file" "${NDDSHOME}/lib/${PLATFORM}"
        append_ld_library_path_export "$script_file" "${NDDSHOME}/lib/${PLATFORM}/openssl-1.1.1"
        append_ld_library_path_export "$script_file" "${NDDSHOME}/lib/${PLATFORM}/openssl-3.0"
        append_ld_library_path_export "$script_file" "${NDDSHOME}/lib/${PLATFORM}/openssl-3.5"
        append_ld_library_path_export "$script_file" "${NDDSHOME}/lib/${PLATFORM}/wolfssl-5.5"
        if [[ "$OPENSSLPATHLIB" == "" ]]; then
            append_ld_library_path_export "$script_file" "${OPENSSL1PATHLIB}"
            append_ld_library_path_export "$script_file" "${OPENSSL3PATHLIB}"
            append_ld_library_path_export "$script_file" "${OPENSSL309PATHLIB}"
            append_ld_library_path_export "$script_file" "${OPENSSL3011PATHLIB}"
            append_ld_library_path_export "$script_file" "${OPENSSL350PATHLIB}"
            append_ld_library_path_export "$script_file" "${OPENSSL355PATHLIB}"
            append_ld_library_path_export "$script_file" "${WOLFSSLPATH_LIB}"
        else
            append_ld_library_path_export "$script_file" "${OPENSSLPATHLIB}"
        fi
        echo -e "export PERFTEST_BIN=${perftest_exec};" >> $script_file

        if [[ "$RECORDING_SERVICE" == "1" ]]; then
            echo -e "export RECORDING_WRAPPER=${recording_wrapper_exec};" >> $script_file
        fi

        if [[ "$ROUTING_SERVICE" == "1" ]]; then
            echo -e "export ROUTING_WRAPPER=${routing_service_exec};" >> $script_file
        fi

        if [[ "$PERSISTENCE_SERVICE" == "1" ]]; then
            echo -e "export PERSISTENCE_SERVICE_WRAPPER=${persistence_service_exec};" >> $script_file
        fi

        echo -e "touch ${output_file} \\" >> $script_file
    done
    echo -e ""

    if [[ "$PUB_ALONE" == "1" ]]; then
        if [[ "$publishers" -gt "1" ]]; then
            echo -e "$ERROR_TAG: If you want to have the publisher alone, you can only have one"
            exit -1
        fi
    fi

    for ((i = 0 ; i < $number_machines ; i++)); do
        machine_status[$i]="None"
    done

    carried_index=0
    for ((i = 0 ; i < $publishers ; i++)); do
        index=$(($i%$number_machines))
        # echo "i is $i -- index $index"
        carried_index=$i
        export script_file="${temporal_scripts}/${full_test_name}_${machines[$index]}.sh"
        temporal_ouput_file_stderr="${temporal_output_files}/stderr_pub_${full_test_name}_${machines[$index]}_$i.txt"
        temporal_ouput_file_stout="${temporal_output_files}/stdout_pub_${full_test_name}_${machines[$index]}_$i.csv"
        export no_headers="-noOutputHeaders"
        if [[ "$NO_POS_ACKS" == "1" ]]; then
            export no_positive_acks_string="${no_positive_acks} -initialBurst 80000"
        fi
        if [[ "${DOMAIN_PUB}" != "" ]]; then
            export domain_pub="-domain ${DOMAIN_PUB}"
        fi
        if [[ "$DURABILITY" != "" ]]; then
            export durability="-durability $DURABILITY"
        fi
        if [[ "$NO_DIRECT_COMMUNICATION" == "1" ]]; then
            export no_direct_communication="-noDirectCommunication"
        fi
        if [[ "$BATCH_SIZE" != "" ]]; then
            export batch_size="-batchSize $BATCH_SIZE"
        fi

        export command="${taskset_prefix} \$PERFTEST_BIN $no_headers -pub -datalen $datalen ${latency_test} ${cft_related_params_pub} \
            ${keyed_args} ${instances_args} ${domain_pub} ${durability} ${no_direct_communication} ${batch_size} \
            -transport UDPv4 -nic ${main_interface[$index]} -pid $i -numSubscribers $subscribers ${security_params} \
            -exec $execution_time ${latency_count} -noPrint -noXML ${no_positive_acks_string} ${USE_MULTICAST} \
            ${BEST_EFFORT} 2>${temporal_ouput_file_stderr} 1>${temporal_ouput_file_stout} \\"
        machine_status[$carried_index]="Perftest Publisher"
        echo -e "&" >> $script_file
        echo $command >> $script_file
    done

    carried_index=$(($carried_index+1))
    for ((i = 0 ; i < $subscribers ; i++)); do
        carried_i=$(($i+$carried_index))
        index=$(( ($i+$carried_index) % ($number_machines-1) ))
        index=$(($index+1))
        # echo "i is $carried_i -- index $index"
        export script_file="${temporal_scripts}/${full_test_name}_${machines[$index]}.sh"
        temporal_ouput_file_stderr="${temporal_output_files}/stderr_sub_${full_test_name}_${machines[$index]}_$i.txt"
        temporal_ouput_file_stout="${temporal_output_files}/stdout_sub_${full_test_name}_${machines[$index]}_$i.csv"
        export no_headers="-noOutputHeaders"
        if [[ "${DOMAIN_SUB}" != "" ]]; then
            export domain_sub="-domain ${DOMAIN_SUB}"
        fi
        if [[ "$CFT" == "1" ]]; then
            export cft_related_params_sub="-cft $i:$i"
        fi
        if [[ "$DURABILITY" != "" ]]; then
            export durability="-durability $DURABILITY"
        fi
        if [[ "$NO_DIRECT_COMMUNICATION" == "1" ]]; then
            export no_direct_communication="-noDirectCommunication"
        fi
        export command="${taskset_prefix} \$PERFTEST_BIN $no_headers -sub -transport UDPv4 -nic ${main_interface[$index]} -sid $i ${cft_related_params_sub} \
            ${keyed_args} ${instances_args} ${domain_sub} ${durability} ${no_direct_communication} \
            -numPublishers $publishers -noPrint ${security_params} -noXML ${USE_MULTICAST} ${no_positive_acks} \
            ${BEST_EFFORT} 2>${temporal_ouput_file_stderr} 1>${temporal_ouput_file_stout} \\"
        machine_status[$index]="Perftest Subscriber"
        echo -e "&" >> $script_file
        echo $command >> $script_file
    done

    if [[ "$RECORDING_SERVICE" == "1" ]]; then
        carried_index=$(($carried_index+1))
        for ((i = 0 ; i < 1 ; i++)); do
            carried_i=$(($i+$carried_index))
            index=$(( ($i+$carried_index) % ($number_machines-1) ))
            index=$(($index+1))
            # echo "i is $carried_i -- index $index"
            if [[ "${DOMAIN_REC}" != "" ]]; then
                export domain_recs="-domain ${DOMAIN_REC}"
            fi
            if [[ "$KEYED_UNKEYED" == "keyed" ]]; then
                export keyed_args="-keyed"
            fi
            export script_file="${temporal_scripts}/${full_test_name}_${machines[$index]}.sh"
            temporal_ouput_file_stderr="${temporal_output_files}/stderr_recording_${full_test_name}_${machines[$index]}_$i.txt"
            temporal_ouput_file_stout="${temporal_output_files}/stdout_recording_${full_test_name}_${machines[$index]}_$i.csv"
            execution_time_recording=$(($execution_time+20))
            export command="\$RECORDING_WRAPPER -executionTime $execution_time_recording -nic ${main_interface[$index]} $domain_recs \
                ${keyed_args} ${BEST_EFFORT} 2>${temporal_ouput_file_stderr} 1>${temporal_ouput_file_stout} \\"
            echo -e "&" >> $script_file
            echo $command >> $script_file
            machine_status[$index]="recording"
        done
    fi

    if [[ "$ROUTING_SERVICE" == "1" ]]; then
        carried_index=$(($carried_index+1))
        for ((i = 0 ; i < 1 ; i++)); do
            carried_i=$(($i+$carried_index))
            index=$(( ($i+$carried_index) % ($number_machines-1) ))
            index=$(($index+1))
            # echo "i is $carried_i -- index $index"
            if [[ "${DOMAIN_RS}" != "" ]]; then
                export domain_rs="-domain ${DOMAIN_RS}"
            fi
            if [[ "$KEYED_UNKEYED" == "keyed" ]]; then
                export keyed_args="-keyed"
            fi
            export script_file="${temporal_scripts}/${full_test_name}_${machines[$index]}.sh"
            temporal_ouput_file_stderr="${temporal_output_files}/stderr_routing_${full_test_name}_${machines[$index]}_$i.txt"
            temporal_ouput_file_stout="${temporal_output_files}/stdout_routing_${full_test_name}_${machines[$index]}_$i.csv"
            execution_time_routing=$(($execution_time+40))
            export command="\$ROUTING_WRAPPER -executionTime $execution_time_routing -nic ${main_interface[$index]} $domain_rs \
                ${keyed_args} ${BEST_EFFORT} 2>${temporal_ouput_file_stderr} 1>${temporal_ouput_file_stout} \\"
            echo -e "&" >> $script_file
            echo $command >> $script_file
            machine_status[$index]="routing"
        done
    fi


    if [[ "$PERSISTENCE_SERVICE" == "1" ]]; then
        carried_index=$(($carried_index+1))
        for ((i = 0 ; i < 1 ; i++)); do
            carried_i=$(($i+$carried_index))
            index=$(( ($i+$carried_index) % ($number_machines-1) ))
            index=$(($index+1))
            if [[ "${DOMAIN_PS}" != "" ]]; then
                export domain_ps="-domain ${DOMAIN_PS}"
            fi
            if [[ "$KEYED_UNKEYED" == "keyed" ]]; then
                export keyed_args="-keyed"
            fi
            export script_file="${temporal_scripts}/${full_test_name}_${machines[$index]}.sh"
            temporal_ouput_file_stderr="${temporal_output_files}/stderr_ps_${full_test_name}_${machines[$index]}_$i.txt"
            temporal_ouput_file_stout="${temporal_output_files}/stdout_ps_${full_test_name}_${machines[$index]}_$i.csv"
            execution_time_routing=$(($execution_time+40))
            export command="\$PERSISTENCE_SERVICE_WRAPPER -executionTime $execution_time_routing -nic ${main_interface[$index]} $domain_ps \
                ${keyed_args} ${BEST_EFFORT} 2>${temporal_ouput_file_stderr} 1>${temporal_ouput_file_stout} \\"
            echo -e "&" >> $script_file
            echo $command >> $script_file
            machine_status[$index]="persistence"
        done
    fi

    for ((i = 0 ; i < $number_machines ; i++)); do
        script_file="${temporal_scripts}/${full_test_name}_${machines[$i]}.sh"
        output_file="${temporal_output}/${full_test_name}_${machines[$i]}.txt"
        echo -e ";" >> $script_file
        echo ${output_file}
        echo -e "echo -e \"FINISHED $?\" >> ${output_file}" >> $script_file
        echo -e "echo \"${machines[$i]} OK [${machine_status[$i]}]\"" >> $script_file
    done

    # If using dry we do not execute anything
    if [[ "$DRY" == "1" ]]; then
        echo -e "\n$WARNING_TAG Dry test, leaving now, scripts are written.\n"
        exit 0
    fi

    echo -e "\n$INFO_TAG: LAUNCHING TEST:"
    # ssh to each of the machines executing the command!
    for ((i = 0 ; i < $number_machines ; i++)); do
        script_file="${temporal_scripts}/${full_test_name}_${machines[$i]}.sh"
        export ssh_command="$ssh ${machines[$i]} \"${script_file}\" &"
        echo -e $ssh_command

        $ssh ${machines[$i]} "${script_file}" &
    done
    echo -e "\n"

    sleep 5
    echo -e "\n$INFO_TAG Test started!!! $full_test_name"

    # Every machine will write a file in $script_location/output, with a single
    # file, once they finish, if the number of lines is = to the number of participants
    # that means discovery was completed.
    sleep 1
    export found=`cat ${temporal_output}/${full_test_name}_*.txt | wc -l`
    sleep 5
    i=5
    while [ "$found" != "$number_machines" ]; do
        ((i=i+10))
        sleep 10
        echo -e "${GREEN}TEST: $full_test_name ${NC} -- Time: $i s (Found = $found, waiting for $number_machines)."
        export found=`cat ${temporal_output}/${full_test_name}_*.txt | wc -l`
        if [[ "$i" -gt "$timeout_value" ]]; then
            echo -e "$ERROR_TAG: Timeout"
            clean_machines || echo -e "$WARNING_TAG Cleanup skipped due to SSH connectivity failures."
            exit 111
        fi
    done
    sleep 2

    # rm -rf  "${results_output}/pub"
    # rm -rf  "${results_output}/subs"
    mkdir -p "${results_output}/pub"
    mkdir -p "${results_output}/subs"
    export result_file_pubs="${results_output}/pub/${full_test_name}.csv"
    export result_file_subs="${results_output}/subs/${full_test_name}.csv"
    touch $result_file_pubs
    touch $result_file_subs

    echo "Sample Size (Bytes), Ave (μs), Std (μs), Min (μs), Max (μs), 50% (μs), 90% (μs), 99% (μs), 99.99% (μs), 99.9999% (μs)" >> $result_file_pubs
    echo "Sample Size (Bytes), Total Samples, Avg Samples/s,    Avg Mbps, Lost Samples, Lost Samples (%)" >> $result_file_subs
    echo -e "$INFO_TAG: Result files are in : $result_file_pubs"
    cat $temporal_output_files/stdout_pub* >> $result_file_pubs
    cat $temporal_output_files/stdout_sub* >> $result_file_subs

    awk 'NR==1 {print $0 ", command-line"; next} {print $0 ",command-line not provided"}' "$result_file_pubs" > "${result_file_pubs}.tmp" && mv "${result_file_pubs}.tmp" "$result_file_pubs"
    awk 'NR==1 {print $0 ", command-line"; next} {print $0 ",command-line not provided"}' "$result_file_subs" > "${result_file_subs}.tmp" && mv "${result_file_subs}.tmp" "$result_file_subs"

    if [[ "$RECORDING_SERVICE" == "1" ]]; then
        rm -rf  "${results_output}/recording"
        mkdir -p "${results_output}/recording"
        export result_file_recording="${results_output}/recording/${full_test_name}.csv"
        touch $result_file_recording
        cat $temporal_output_files/stdout_recording* >> $result_file_recording
    fi

    if [[ "$ROUTING_SERVICE" == "1" ]]; then
        rm -rf  "${results_output}/routing"
        mkdir -p "${results_output}/routing"
        export result_file_routing="${results_output}/routing/${full_test_name}.csv"
        touch $result_file_routing
        cat $temporal_output_files/stdout_routing* >> $result_file_routing
    fi

    if [[ "$PERSISTENCE_SERVICE" == "1" ]]; then
        rm -rf  "${results_output}/persistence"
        mkdir -p "${results_output}/persistence"
        export result_file_persistence="${results_output}/persistence/${full_test_name}.csv"
        touch $result_file_persistence
        cat $temporal_output_files/stdout_ps* >> $result_file_persistence
    fi

    # Merge infrastructure services results into consolidated CSV files
    if [[ "$ROUTING_SERVICE" == "1" || "$RECORDING_SERVICE" == "1" || "$PERSISTENCE_SERVICE" == "1" ]]; then
        echo -e "$INFO_TAG Merging infrastructure services results..."
        export merge_script="${repo_location}/scripts/tools/merge_infrastructure_services_results.py"
        if [[ -f "$merge_script" ]]; then
            python3 "$merge_script" "${results_output}"
        else
            echo -e "$WARNING_TAG Merge script not found: $merge_script"
        fi
    fi

    echo -e "$INFO_TAG Test finished Successfully"

}

main

Test Software

The following software was used to perform these tests:

RTI Connext DDS 7.7.0 Host and Target Libraries for x64 Linux (x64Linux4gcc8.5.0)

Test Hardware

The following hardware was used to perform these tests:

Linux Nodes

Dell R340 Servers (13 Units)
Processor: Intel Xeon E-2278G (3.4-5GHz, 8c/16t, 16MB cache, 2 memory channels @2666MHz)
RAM: 4x 16GB 2666MHz DIMM (64GB RAM)
HD: 480GB SATA SSD
NIC 1: Intel 710 dual port 10Gbps SFP
OS: Ubuntu 24.04 -- gcc 9.3.0

Switch

Dell 2048 -- 10Gbps switch (10Gbps and 1Gbps interfaces)