A Comprehensive Performance Evaluation of Different Kubernetes CNI Plugins for Edge-based and Containerized Publish/Subscribe Applications

The growing number of data- and latency-sensitive Internet of Things (IoT) applications is posing significant challenges for the edge and cloud deployment of publish/subscribe services, which are required by these applications. Two independently developed technologies show promise in addressing these challenges. First, Kubernetes (K8s) provides a de-facto standard for container orchestration that can manage and scale distributed applications in the cloud. Second, OMG’s Data Distribution Service (DDS), a standardized real-time, data-centric and peer-to-peer publish/subscribe middleware, is being used in thousands of critical systems around the world. However, the feasibility of running DDS applications within K8s for latency-sensitive edge computing, and specifically the performance overhead of K8s’ network virtualization on DDS applications is not yet well-understood. To address this, in this paper we evaluate the performance overhead of several container network interface (CNI) plugins including Flannel, WeaveNet and Kube-Router installed on a hybrid (ARM+AMD) edge/cloud K8s cluster. The paper reports results from a comprehensive set of experiments conducted to measure and analyze the performance (throughput, latency, and CPU/memory usage) of containerized DDS applications from the perspectives of virtualization overhead, reliability (DDS Reliable and BestEffort QoS), transport mechanisms (UDP unicast and multicast), and security. The insights derived from this study provide concrete guidance to developers of DDS-based applications in choosing the right virtual network plugin and configurations when hosting their real-time IoT applications in real-world containerized environments.