Last spring a client asked us to review their EKS spend: around €38k/month for a workload that, on inspection, needed maybe a third of that. No exotic waste. No forgotten GPU nodes. Just the same three problems I find almost everywhere: resource requests nobody had ever measured, node types nobody had ever questioned, and a bill nobody outside finance ever saw. Four months later they were at €16k/month with better reliability than before.
Kubernetes doesn't make infrastructure expensive. It makes waste invisible, because the unit you pay for (nodes) is two abstraction layers away from the unit engineers think in (pods). Closing that gap is the whole game.
Requests are reservations, and reservations are money
The number one cost driver in every cluster I've audited is misconfigured resource requests. Not limits. Requests. The scheduler reserves requested CPU and memory whether the pod uses it or not. Request 2 CPU, use 100m, and you're paying for 2 CPU. Multiply by 60 deployments whose requests were copy-pasted from the first service ever written in 2021, and you have that €38k bill.
The tell is the gap between allocation and usage:
# What's reserved
kubectl describe nodes | grep -A5 "Allocated resources"
# What's actually used
kubectl top pods -A --sort-by=cpu
At that client, cluster-wide CPU utilization was 11% while the cluster autoscaler kept adding nodes, because allocation was at 85%. The autoscaler doesn't scale on usage. It scales on requests. Garbage requests in, garbage nodes out.
The rightsizing workflow
Rightsizing isn't a one-off spreadsheet exercise; it's a loop. Ours looks like this:
- Collect real usage. Two weeks minimum of Prometheus data (
container_cpu_usage_seconds_total,container_memory_working_set_bytes), covering at least one traffic peak and, ideally, one month-end or batch cycle. Rightsizing off a quiet week is how you cause the incident that discredits the whole effort. - Get recommendations. Deploy VPA in recommendation-only mode (
updateMode: "Off") and let it publish targets without touching anything. Goldilocks gives you a dashboard over VPA if you want the friendly view; KRR does the same job straight from Prometheus with no in-cluster agent. - Adjust with policy, not vibes. My defaults: memory request at ~P99 of the working set plus 15-20% headroom, memory limit somewhat above that (OOM kills are the failure you're buying insurance against); CPU request around P95, and for most workloads no CPU limit at all: CPU is compressible, throttling latency-sensitive services to protect nothing is a self-inflicted wound.
- Ship it like code. Requests live in the Helm values / kustomize overlays, changes go through PRs, and the diff is visible. Then re-run the loop quarterly, because services drift.
The first pass at that client cut allocatable demand by ~40% before we touched anything else.
Bin packing: node shapes matter
Once requests reflect reality, the next question is whether your pods actually pack onto your nodes. A cluster of 4xlarge nodes running pods that request 3 CPU each strands a core per node. Pods requesting 2.5 GB on nodes with 7.5 GB allocatable strand 2.5 GB each. This is a knapsack problem, and the fix is usually unglamorous: pick one or two node shapes whose ratios match your aggregate workload ratio (total requested CPU : total requested memory), rather than whatever instance type was fashionable when the cluster was built.
Or let Karpenter do it (more below).
Spot is free money if you do the handling
Spot/preemptible instances run 60-90% cheaper, and for stateless replicated workloads there is very little excuse not to use them. The engineering that makes spot boring instead of scary:
- Terminate gracefully. Handle SIGTERM, drain connections, keep
terminationGracePeriodSecondshonest. You need this anyway. Spot just makes the negligence visible. - Spread the risk. Multiple instance types and AZs in the spot pool; simultaneous reclaim of your whole fleet becomes vanishingly unlikely when you diversify across 10+ pools.
- Set a PodDisruptionBudget so voluntary drains can't take a service below quorum.
- Keep the stateful stuff on-demand. Databases, anything with leader election that reacts badly to churn, singleton jobs: on-demand or reserved. This isn't a purity contest.
Client mix that works repeatedly: on-demand/reserved for the control layer and stateful sets, spot for the ~70% of workloads that are stateless replicas. That alone was another €8k/month at the client above.
Autoscaling: four layers, in order
People reach for autoscalers before fixing requests, which just automates the waste. In order of leverage:
| Layer | Tool | Scales | Reach for it when |
|---|---|---|---|
| Pod, horizontal | HPA | Replica count | CPU/memory tracks load |
| Pod, event-driven | KEDA | Replicas incl. to zero | Queues, cron windows, custom metrics |
| Node, reactive | Cluster Autoscaler | Node count in fixed groups | Simple, homogeneous node groups |
| Node, provisioning | Karpenter | Node count and shape | You want bin packing solved for you |
KEDA deserves a special mention for anything queue-driven: scaling workers on SQS depth instead of CPU, and to zero overnight, turned one client's batch fleet from a fixed cost into a marginal one. And Karpenter has genuinely changed the node-management conversation on AWS: it picks instance types per pending pod batch, consolidates underused nodes, and handles spot diversification, which retires most of the manual node-shape work from two sections ago.
The cultural half: nobody fixes a bill they can't see
Everything above is the easy 50%. The durable half is making cost visible to the people who create it.
The pattern that works is showback per namespace: deploy OpenCost or Kubecost, tag namespaces to teams, and publish a simple monthly view: each team's compute cost, trend, and top three workloads. Not chargeback with real invoices and politics. Just visibility, in the same dashboards engineers already look at.
The effect is embarrassingly reliable. At the €38k client, the first monthly report showed one team that a staging namespace they'd stopped using cost €1,900/month. It was gone by Thursday. No ticket, no mandate, no meeting. Engineers are perfectly happy to fix waste; they've just never been shown where it is. The inverse is also true: with no showback, every optimization we shipped would have eroded within a year, because the feedback loop that created the waste would still be open.
The point
That client's journey from €38k to €16k was, in order: measure and fix requests (−40% allocation), reshape nodes and adopt Karpenter, move stateless workloads to spot, add KEDA for the queue workers, and stand up per-namespace showback so it stays fixed. Nothing on that list is a product you buy or a trick. It's choices, made with data, by people who can see the bill.
Your Kubernetes bill is a choice. If nobody in engineering can see it, the choice is being made by default, and defaults are expensive.