Security
eBPF Observability with Cilium Tetragon on Kubernetes
July 20269 min read
Photo by Wei, Heng (Civil engineer) via Wikimedia Commons (Public domain)
No. That is the whole point of eBPF. You load a sandboxed program into a running kernel and attach it to a hook point without compiling a module, patching kernel source, or rebooting. A kernel verifier statically checks the program for unsafe behavior before it is allowed to run, and a JIT compiler turns it into native machine code.
Cilium is an eBPF-based networking, security, and observability layer for Kubernetes, focused on connectivity and network policy. Tetragon is a sub-project under Cilium focused specifically on security observability and runtime enforcement: process execution, network connections, and file access. You can run Tetragon without running Cilium as your CNI.
Because filtering and enforcement happen inside the kernel using eBPF rather than by copying every event to a user-space agent, overhead is typically reported at under one percent. The exact figure depends on how many TracingPolicies you load and how noisy the monitored events are. Filtering in-kernel is what keeps it cheap.
It can do both. In observe mode a TracingPolicy uses a Post action and reports events. Switch the action to Sigkill and Tetragon terminates the offending process synchronously inside the kernel, before the operation completes. This synchronous enforcement closes the time-of-check-to-time-of-use gap that asynchronous user-space agents leave open.
You need a reasonably modern Linux kernel. Full features, including CO-RE via BTF type information, depend on newer kernels; older distro kernels give a reduced feature set. Always check the kernel running on your nodes before planning a rollout, especially on managed Kubernetes where you do not control the node image.
Photo by Wei, Heng (Civil engineer) via Wikimedia Commons (Public domain)
Key Takeaway
eBPF lets you run sandboxed programs inside the Linux kernel without loading modules or rebooting. Tetragon, a CNCF project from the Cilium team, uses it to observe process execution, network connections, and file access on Kubernetes, and can enforce policy synchronously in-kernel with under one percent overhead.
For years, getting real visibility into what a container actually does at runtime meant one of two bad options: bolt a sidecar agent onto every pod, or ship a kernel module and pray it does not panic the node on the next kernel upgrade. On a Kubernetes cluster I run on a self-hosted VPS, neither felt acceptable. eBPF changed that calculation, and Tetragon is the tool I reach for when I want to see every process that spawns, every outbound connection, and every sensitive file touched, without a single application change.
This post explains what eBPF is, how it attaches to the kernel safely without modules, and how Tetragon turns that machinery into practical process and network visibility on Kubernetes. Everything below is drawn from running it in anger, not from the marketing page.
eBPF, short for extended Berkeley Packet Filter, is a technology built into the Linux kernel that lets you run small sandboxed programs inside kernel space. Think of it as a lightweight virtual machine embedded in the kernel. You write a program, the kernel loads it, and it executes when a specific event fires. Crucially, you do this without modifying kernel source code, without compiling a kernel module, and without a reboot. That last point is why it took over observability and security tooling in the cloud native world.
An eBPF program is event-driven. It runs when the kernel or an application crosses a hook point. Predefined hooks include system calls, function entry and exit, kernel tracepoints, and network events. When no predefined hook fits, you attach a kernel probe (kprobe) or a user probe (uprobe) to place your program almost anywhere in the kernel or in a user-space application.
The reason eBPF is safe to run in the kernel while a hand-written module is not comes down to three stages: a verifier, a JIT compiler, and the sandbox itself. Your program is compiled to a generic bytecode and handed to the kernel. Before it is allowed to run, the verifier statically analyzes every possible execution path.
A buggy kernel module can crash the whole node. A buggy eBPF program is rejected at load time by the verifier, or it simply returns without harming the kernel. That difference is the entire reason this technology is trusted in production. Do not disable the verifier or run unverified programs from untrusted sources.
Writing raw eBPF is not something most engineers want to do to answer basic questions like which binaries ran inside a pod. Tetragon is the layer that removes that friction. It is a CNCF project maintained by the Cilium team (originally Isovalent, now part of Cisco), and it runs as a DaemonSet on Kubernetes. Each Tetragon agent loads eBPF programs on its node and streams enriched events tagged with Kubernetes identity: namespace, pod, container, and labels.
Out of the box, before you write any policy, Tetragon already collects process execution and exit events for every container. That alone gives you an auditable trail of every command run in the cluster, correlated to the workload that ran it. Installation is a two-minute Helm exercise.
# Install Tetragon as a DaemonSet
helm repo add cilium https://helm.cilium.io
helm repo update
helm install tetragon cilium/tetragon -n kube-system
kubectl rollout status -n kube-system ds/tetragon -w
# Stream process events in real time, human-readable
kubectl exec -ti -n kube-system ds/tetragon -c tetragon -- \
tetra getevents -o compact
# Or filter the export stream by pod / process
kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon \
-c export-stdout -f | tetra getevents -o compact --pod xwingThe default process events are useful, but the real power comes from a TracingPolicy: a custom Kubernetes resource that tells Tetragon exactly which kernel events to observe, and optionally what to do when they fire. You can watch a specific system call like execve, monitor connections to a TCP port, or flag any read of a sensitive path such as the file that holds password hashes. Because the filtering happens in the kernel, you are not paying to ship millions of irrelevant events to user space.
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: monitor-sensitive-file-reads
spec:
kprobes:
- call: "security_file_permission"
syscall: false
args:
- index: 0
type: "file"
- index: 1
type: "int"
selectors:
- matchArgs:
- index: 0
operator: "Equal"
values:
- "/etc/shadow"
- index: 1
operator: "Equal"
values:
- "4" # MAY_READ
# Swap the action below for "Sigkill" to enforce, not just observe
matchActions:
- action: PostThat policy watches every attempt to read the shadow password file across the whole cluster. Left as Post, it observes and reports. Change the action to Sigkill and Tetragon kills the offending process synchronously, inside the kernel, before the read completes. Because the enforcement is in-kernel and synchronous, it closes the time-of-check-to-time-of-use gap that user-space agents suffer from: there is no window between detecting the event and acting on it.
Start every TracingPolicy in observe-only mode (action Post). Run it for a few days, read the events, and confirm you are not about to Sigkill a legitimate process. Only then flip to enforcement. eBPF gives you a very sharp knife; the verifier stops you cutting the kernel, not yourself.
eBPF is not free of downsides. It is worth being honest about where it wins and where it costs you, so you go in with the right expectations.
| Concern | eBPF / Tetragon | Traditional agent / module |
|---|---|---|
| Deployment | One DaemonSet, no app changes | Sidecar per pod or kernel module per node |
| Overhead | Typically under 1 percent, filtering in-kernel | Higher, copies events to user space first |
| Safety | Verifier rejects unsafe programs at load | A bad module can panic the whole node |
| Enforcement | Synchronous, in-kernel, no TOCTOU gap | Often asynchronous, exploitable timing window |
| Kernel dependency | Needs a recent kernel and BTF for full features | Module must be rebuilt per kernel version |
The one caveat that bites people is kernel version. eBPF features and CO-RE (compile once, run everywhere via BTF type information) depend on a reasonably modern kernel. On old distro kernels you get a reduced feature set. Check your node kernel before you plan a rollout, especially on managed clusters where you do not control the image.
If you need a low-overhead audit trail of process execution and network activity across a Kubernetes cluster, or synchronous runtime enforcement without instrumenting applications, eBPF via Tetragon is the strongest option available today. It is what I run for cluster-level visibility. What it is not is a full SIEM or a replacement for image scanning and admission control; it observes and enforces at runtime, so pair it with build-time and deploy-time controls rather than treating it as the only layer.