A while back I wrote a comparison of Kubernetes CNIs, the usual Calico versus Flannel versus cloud-native tour. Rereading it now, it feels like a review of horse breeds published in 1910. The interesting question in Kubernetes networking is no longer "which CNI", it's "eBPF or not", and increasingly the answer is eBPF whether you chose it or not, because the cloud providers are making the choice for you. GKE Dataplane V2 is Cilium. EKS and AKS both offer Cilium-powered dataplanes. The default is shifting under our feet.

Having now migrated a real production cluster onto Cilium, here's my updated view: what eBPF actually is, why it won, and, importantly, when you should still ignore all of this.

What eBPF actually is

Strip the hype and eBPF is this: a way to run small, sandboxed programs inside the Linux kernel, attached to events, a packet arriving, a syscall firing, a socket opening, without writing a kernel module and without rebooting.

The part that makes this safe enough for production is the verifier. Before the kernel accepts an eBPF program, it statically proves the program terminates (bounded loops only), touches only memory it's allowed to touch, and can't crash the kernel. It's genuinely restrictive, people porting non-trivial logic hit its limits constantly, but that restrictiveness is the entire deal. You get kernel-speed execution with userspace-ish safety guarantees.

For networking, this means packet processing logic can live at the earliest possible point in the stack, sometimes in the NIC driver itself via XDP, instead of a packet traversing the whole kernel networking stack to reach a decision that could have been made in the first microsecond.

Why Cilium replaced kube-proxy and iptables

To appreciate what Cilium removes, remember what kube-proxy in iptables mode actually does: every Service becomes a chain of iptables rules, and every packet to a Service IP walks through those chains sequentially. With 50 Services this is invisible. With 5,000 Services and heavy churn, you get two problems: per-packet lookup cost that grows with rule count, and, worse, rule updates that require rewriting giant rule sets while pods churn. On busy clusters, iptables restore operations can take seconds, during which your dataplane is stale.

Cilium's eBPF approach replaces the sequential chain walk with hash table lookups in eBPF maps: O(1) regardless of whether you have 50 Services or 50,000. Updates modify a map entry instead of rewriting rule text. Service-to-backend selection, load balancing, and network policy all happen in eBPF, and with kubeProxyReplacement enabled, kube-proxy is simply gone from the node.

There's also a conceptual win: Cilium's network policy operates on identities, labels resolved to numeric identities carried with the traffic, rather than IP addresses. In a cluster where pod IPs are recycled every few minutes, policies that reason about identities rather than IPs are both faster and less prone to the race conditions that IP-based policy engines fight forever.

The observability win is bigger than the performance win

Here's the part I underestimated: Hubble. Because every flow already passes through Cilium's eBPF programs, you get flow-level visibility, source pod, destination, port, protocol, L7 verb if you enable it, verdict (forwarded/dropped, and why dropped), essentially for free. No sidecars, no packet capture, no instrumentation of the application.

The first week after our migration, a client's intermittent 500s, previously the subject of two inconclusive investigation attempts, turned out to be a NetworkPolicy dropping traffic from a CronJob whose pods only existed for 40 seconds at a time. hubble observe --verdict DROPPED found it in ten minutes. Nobody had ever seen the drop because nobody had ever caught the pod alive. That single debugging session justified the migration to the client more than any latency graph.

The same mechanism powers the sidecar-less service mesh story. The traditional mesh injects an Envoy sidecar into every pod: an extra container, extra memory, extra latency hops, and an upgrade nightmare multiplied by your pod count. Cilium's mesh mode does mTLS and L7 handling with per-node components instead of per-pod ones. I won't claim feature parity with Istio for the long tail of traffic-management exotica, it isn't there, but for the 80% case of "encrypt everything, give me L7 metrics, enforce some policies", the resource and operational savings are real. And Istio itself now ships ambient mode going sidecar-less too, which tells you which way the industry believes this goes.

When boring VPC-native CNIs are still right

Now the cold water. A good chunk of clusters I see should not run Cilium, and I've told clients so.

If you're on EKS with the AWS VPC CNI and your cluster has 30 nodes, 40 Services, no network policies beyond "deny from the internet", and a team of three that doesn't include a networking person, the VPC CNI is the correct choice. Pods get real VPC IPs, security groups work the way your cloud team already understands, AWS support will actually help you, and there is no eBPF datapath to reason about at 3 a.m. The iptables scaling problems I described start mattering in the hundreds-of-Services range, not at 40.

The honest checklist for whether eBPF CNI earns its complexity: thousands of Services or serious pod churn, real network policy requirements (especially L7), a need for flow-level observability you can't get otherwise, multi-cluster networking, or you're paying a measurable sidecar tax today. Zero or one of those: stay boring. Two or more: it's probably time.

Notes from a real migration

The cluster move that informs this post: a roughly 60-node EKS cluster, VPC CNI to Cilium, motivated by NetworkPolicy needs and observability, done as a blue/green cluster swap rather than an in-place CNI replacement. In-place CNI migration is possible and I have read the guides; I have also read enough postmortems to happily pay for a second node group instead.

What bit us, so it doesn't bite you:

  • Health probe surprises. With kube-proxy replacement, a handful of workloads relying on subtle externalTrafficPolicy and NodePort behaviours behaved differently. Test your ingress path properly, not just pod-to-pod.
  • The verifier is real. A colleague's custom Tetragon-adjacent policy hit program complexity limits. If you plan to write your own eBPF rather than just consume Cilium's, budget learning time.
  • Kernel versions matter. Features arrive gated on kernel versions; our oldest node AMI was the blocker for two of them. Standardise AMIs before, not during.
  • Turn on Hubble from day one. We initially deferred it "for later" and then needed it in week one anyway (see the CronJob story above).
  • MTU and encapsulation choices deserve an afternoon. We ran native routing mode since the VPC could carry pod IPs; if you tunnel, understand the overhead you're accepting.

Total migration effort was about three weeks of one engineer, most of it validation rather than configuration. The dataplane itself has been the quietest part of that platform since.

The pattern from my original CNI post still holds: choose the most boring networking that meets your actual requirements. What's changed is that eBPF is rapidly becoming the boring option, when GKE ships it as the default dataplane, "exotic" is no longer the right word. Give it two more years and running iptables-based kube-proxy will be the choice you have to justify.