Monitoring network traffic in K8S Multi Node Cluster

5 min readSep 7, 2022

ARTH — Task 25

Task Description:
⚜️ Research Based Task 👨🏻‍💻
Task Description📄
📌 Monitoring the network traffic In K8S Master Node while Ingress Path based Routing using tcpdump command and observe the network packet header.

Objective

Tracking network traffic on a Kubernetes (k8s) cluster, capturing request-response tuple and associated metadata. Streaming the captured data out of the cluster to be analysed further.

Since we’re planning on tracing each network request (OSI Layer 7), there are a few considerations that form a part of our requirements. They are:

Considerations

Lightweight →Low memory and CPU footprint are critical.
Scalable → New nodes are added and removed from a cluster, and the solution should scale accordingly.
Language and Framework agnostic → Containers might be created in any number of languages and frameworks, this should not affect/influence our tracking.
Rapid → The scale of all network data in a server would be massive. Our solution must be able to track and report that data to us within considerable time limits.
Comfortable → Users should feel comfortable/safe/assured when deploying this on their production clusters.
Plug and Play → Low onboarding time and cost. No code changes to attach hooks.

Based on the above details, we chose Pixie for this task. Pixie is an open source observability platform built by New Relic, one of the leaders in the APM space.

The fact that Pixie is a Cloud Native Computing Foundation project and New Relic uses it for its Kubernetes observability lends credibility to the project. Alleviating the rapid and comfortable considerations made above.

What is Pixie?

In their own words:

“Open source Kubernetes observability for developers
Auto-instrumented. Scriptable. Kubernetes native.”

For us, it means easy deployments, custom data points tracking (if required), and native capture of requests and responses made on the cluster.

So, how does Pixie do that? And what makes it special?

Pixie uses eBPF technology to run event-driven scripts at the kernel level. Bypassing all code requirements, making the Linux kernel its first-class citizen.

This makes Pixie language and framework agnostic, and since eBPF works on any system running the Linux kernel, Pixie has a wide install base.

Note: This also means Pixie will not run on Windows servers, though the volume of production k8s clusters running on Windows servers might be pretty low.

Full list of supported Pixie environments and cluster types: Install | Requirements.

[Optional] Further resources on eBPF technology:

What Is eBPF and Why Does It Matter for Observability?

BPF performance analysis at Netflix

But still, why Pixie? What are its advantages?

Pixie’s Edge Module is deployed at a node level. So, irrespective of the pods and services running, only a single edge module deployment is required per node. Addressing lightweight and scalable.

Pixie deployment on a k8s cluster. Image from Pixie’s Website

Pixie has a very simple deployment and installation. Once authenticated, only the px deploy command needs to be run. Addresses plug and play.
As soon as it is deployed, Pixie is ready to report on Network data. No other setup is required.
Pixie provides an easy-to-use API (in Python and Go) for extracting/fetching/streaming that data via the internet. It also supports in-transit e2e encryption of the data.

We go ahead with Pixie as it addresses most of our questions while providing additional features which make development easier.

In the next sections, we’ll discuss pixie specific things, such as:

Installation
What all data does Pixie track and its impact on the cluster?
Finally, streaming data out from the cluster using Pixie’s API.

Installation

Installing Pixie is pretty easy, there are only a few steps to follow. Pixie’s documentation: Comprehensive Install Guide for Pixie Community Edition.

If you already have a Kubernetes cluster available you can skip this section and go straight to the installation steps for Pixie. Here, we’re going to install and run a minikube cluster for local testing.

Steps to set up a local Kubernetes environment:

Step 1: Install minikube.

Step 2: Increase the configured CPU and memory limits. This is required as we’ll be deploying Pixie and a microservice for request simulation.

minikube config set cpus 4
minikube config set memory 8192

If you already have minikube installed and running, then you need to perform minikube delete before your changes are materialized.

To check if the changes took place correctly run the below commands.

minikube config get memory
# Output: 8192minikube config get cpus
# Output: 4

Step 3: Start minikube with the hyperkit driver.

minikube start --driver=hyperkit

Image by author

Steps to install Pixie on a cluster:

Step 1: The main interface for Pixie to interact with the cluster is its CLI. Install it via:

# Copy and run command to install the Pixie CLI.
bash -c "$(curl -fsSL https://withpixie.ai/install.sh)"

Step 2: Pixie (CLI as well) requires an account to work and authenticate with the cluster. To authenticate Pixie CLI, run:

px auth login

Step 3: Once login is successful. Deploy Pixie on the cluster.

px deploy

That’s it ✅ for Pixie’s installation. We’ve set up a cluster and installed Pixie on it. But, we can’t track traffic if there isn’t any so we deploy a demo application. Sock Shop Weaveworks.