FinOps · Kubernetes
Cutting EKS Costs with Karpenter and Spot
TL;DR
- Two levers cut your EKS compute bill: fit (Karpenter launches nodes that match your pods) and price (Spot costs up to 90% less than On-Demand, per AWS).
- Spot can be taken back with a two-minute warning. The real question is which workloads can lose a node and not notice.
- We set it up in a fixed order: sort workloads, fix resource requests, one wide Spot pool with On-Demand fallback, interruption queue, guardrails, then break it on purpose.
- Most failures come from the same few causes: too few instance types, single replicas, and requests that are far above real usage.
On EKS, most of your Kubernetes bill is EC2 nodes. Two things decide how big it is: how well your pods fit on the nodes you pay for, and what you pay for each node. Karpenter helps with the first. Spot Instances help with the second. Used together, with a few safety rules, they are one of the biggest compute savings an EKS team can make without a rewrite.
This is how we set it up, in the order we do it, and where it goes wrong. For the wider picture of where Kubernetes money leaks, start with why Kubernetes wastes 40% of your budget. This post goes deep on one part of it.
Two levers, not one
Right-sizing and cheaper capacity are two different problems. It helps to keep them apart.
- Karpenter fixes the fit. It looks at pods that are waiting, launches nodes that match what they ask for, and removes nodes that are no longer needed.
- Spot fixes the price. AWS says Spot Instances can cost up to 90% less than On-Demand, because they use spare capacity that AWS can take back.
Pull only the price lever and you run wasteful nodes cheaply. Pull only the fit lever and you still pay full price for every node. Pull both, in the right order, and the savings stack.
What Spot really means
A Spot Instance is spare EC2 capacity sold at a big discount. The catch is that AWS can take it back. When that happens, you get a two-minute warning before the instance is interrupted. AWS also sends an earlier signal, called a rebalance recommendation, when an instance is at higher risk.
Two minutes is plenty for an app that is built to move. It is nowhere near enough for one that is not. So the useful question is never "is Spot safe?". It is "which of my workloads can lose a node and not notice?"
What Karpenter does for Spot
Pending podson your EKS cluster
Karpenterpicks the nodes
Spot firstup to 90% cheaperOn-Demandthe fallback- It picks from many instance types. When pods are waiting, Karpenter asks EC2 for capacity across the instance types you allow. On Spot it uses AWS's price-capacity-optimized strategy, which looks at spare capacity first and price second. That favors pools that are less likely to be interrupted.
- It listens for interruption warnings. With an interruption queue set up (an SQS queue fed by EventBridge rules), Karpenter hears the two-minute notice, starts a replacement, and drains the old node before it disappears.
- It prefers Spot when you allow both. If a NodePool allows several capacity types, Karpenter tries reserved capacity first, then Spot, then On-Demand. That makes "Spot with On-Demand fallback" a simple, safe default.
- It consolidates. It removes or replaces nodes that cost more than they need to. Replacing one Spot node with a cheaper Spot node is called spot-to-spot consolidation. It needs at least 15 instance types to choose from, and at the time of writing it is an opt-in feature gate. Check the docs for your version.
AWS's own write-up of Grover, a Berlin rental company, describes running about 80% Spot in production, with 25% higher Spot usage across its EKS clusters after adopting Karpenter. They kept stateful and critical workloads on On-Demand, and protected long jobs with a do-not-disrupt annotation. That post does not give dollar savings, and we are not going to promise you one either.
How we set it up, in order
The order matters more than any single setting. Skipping ahead is how teams end up with a cheap cluster that pages someone at 3am.
Sort your workloads
List what can lose a node without anyone noticing: stateless, two or more replicas, shuts down cleanly. That list decides everything else.
Fix requests before nodes
Karpenter sizes nodes from what pods request. If requests are 3x real usage, you buy oversized nodes cheaply, which is still waste. We right-size requests first.
One wide pool, Spot first
Allow both Spot and On-Demand in the pool, and allow many instance families and sizes. Do not pin one instance type.
Turn on the interruption queue
Set up the SQS queue and EventBridge rules so Karpenter hears every two-minute warning and moves pods early.
Add guardrails
Pod disruption budgets, replicas spread across zones, a disruption budget on the pool, the do-not-disrupt annotation for long jobs, and a CPU limit plus a billing alarm as a ceiling.
Break it on purpose
AWS Fault Injection Service can send a real Spot interruption on demand. We test in staging before any production workload moves.
Here is a starting shape for the pool in step 03. Treat it as a sketch, not something to paste in: instance families, limits, and disruption settings depend on your workloads and your Karpenter version.
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: general
spec:
template:
spec:
nodeClassRef:
group: karpenter.k8s.aws
kind: EC2NodeClass
name: default
requirements:
# Spot first, On-Demand as the fallback
- key: karpenter.sh/capacity-type
operator: In
values: ["spot", "on-demand"]
# wide on purpose, not pinned to one type
- key: karpenter.k8s.aws/instance-category
operator: In
values: ["c", "m", "r"]
- key: karpenter.k8s.aws/instance-generation
operator: Gt
values: ["4"]
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized
consolidateAfter: 1m
budgets:
# never replace more than 10% of nodes at once
- nodes: "10%"
# a ceiling on what this pool can grow to
limits:
cpu: 1000Three things teams skip, all from AWS's own Karpenter guidance:
- Do not run the Karpenter controller on nodes Karpenter manages. Put it on Fargate or a small managed node group.
- Pin your AMIs in production. Letting untested images roll out automatically is a risk to your workloads.
- Pick one interruption handler. AWS advises against running Karpenter's interruption handling alongside Node Termination Handler.
Which workloads go where
| Workload | Capacity | Why |
|---|---|---|
| Stateless web and API, two or more replicas | Spot, with a PodDisruptionBudget | Losing one pod at a time is fine, and it is replaced fast |
| CI runners, batch jobs, queue workers | Spot | Safe to retry if a node goes away |
| Databases and single-replica services | On-Demand | One interruption is an outage |
| Long jobs that cannot resume | On-Demand, or do-not-disrupt | An interruption throws away hours of work |
| Cluster add-ons like CoreDNS and your ingress controller | On-Demand baseline | Everything else depends on them |
What goes wrong, and the fix
| What you see | Usual cause | Fix |
|---|---|---|
| Many pods restart at once | Single replica, no PodDisruptionBudget, or every replica on one node | Two or more replicas, a PDB, and topology spread across zones |
| Pods stuck in Pending | Instance list too narrow, so the Spot pool ran dry | Widen families, sizes, and zones. Keep On-Demand as the fallback |
| Nodes keep getting replaced | Consolidation is too eager for your workload | Raise consolidateAfter, add a disruption budget, protect long jobs |
| Spot-to-spot consolidation never happens | Feature gate is off, or fewer than 15 instance types allowed | Enable the gate on your version and widen the list |
| Random out-of-memory kills after consolidation | Memory limits set well above memory requests | Set memory requests equal to limits. Consolidation packs by requests only |
| The bill barely moved | Requests far above real usage, so nodes are still too big | Right-size requests first, then let Karpenter pack |
What to expect
We are not giving one Spot savings number here, because it depends on your mix: how much of your load is safe for Spot, and how much you keep as a steady On-Demand or Savings Plan baseline. What we can point to is a case we have already published. A Series C SaaS team cut its EKS bill from $96K to $61K a month with a right-sizing rollout, a Karpenter migration, and scheduled scale-down for non-production. It took six weeks with zero customer-facing incidents. The details are in our Kubernetes cost guide.
For a quick estimate of your own, plug your monthly spend into the free Kubernetes Waste Calculator.
Quick answers
Is Spot safe for production on EKS?
For the right workloads, yes. Stateless services with two or more replicas, a PodDisruptionBudget, and clean shutdown handling lose a node without anyone noticing. Databases, single-replica services, and jobs that cannot resume should stay on On-Demand.
How much can Karpenter and Spot save?
AWS says Spot can cost up to 90% less than On-Demand per instance. Your total saving is lower, because you keep a steady baseline on On-Demand and because right-sizing matters as much as price. Our cost guide describes recovering 30 to 50% of compute spend in 4 to 8 weeks with Karpenter, VPA, and a deliberate Spot and On-Demand mix.
Do I need to change my application?
Usually not the code. Apps should run more than one replica and shut down cleanly within the grace period when they receive SIGTERM. Both are worth doing on any cluster.
Karpenter or Cluster Autoscaler for Spot?
Cluster Autoscaler scales the node groups you defined up front. Karpenter chooses instance types for each batch of pending pods and consolidates nodes afterward, which is why it fits Spot better. For how EKS compares with the other managed Kubernetes services, see EKS vs GKE vs AKS.
