Zero-downtime Kubernetes upgrades: a practical playbook.

Most teams don't lose sleep over the Kubernetes control plane anymore — managed offerings and well-tested upgrade tooling have made that part fairly routine. What still causes incidents is everything downstream of it: the workloads, the drain behavior, and the assumptions baked into manifests that were written once and never revisited.

What actually breaks

In our experience, upgrade-related incidents rarely come from the version bump itself. They come from three recurring gaps:

• No disruption budget, or a meaningless one. A Deployment with no PodDisruptionBudget lets the node drain evict every replica at once if the scheduler lines them up on the same node pool.
• Connection draining that's shorter than real client timeouts. Pods report Ready long before in-flight requests and long-lived connections have actually finished.
• Removed or changed API versions. Admission webhooks, CRDs and controllers pinned to a deprecated apiVersion fail silently until the exact moment the old version is gone.

None of these show up in a staging cluster that receives no real traffic. They show up the first time a node pool rotates under load.

Start with a real PodDisruptionBudget

A disruption budget is the single highest-leverage change most teams can make before their next upgrade. At minimum:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-pdb
spec:
  minAvailable: 80%
  selector:
    matchLabels:
      app: checkout

Pair it with a terminationGracePeriodSeconds that's actually longer than your slowest real request, and a preStop hook that stops accepting new connections before the process exits. Readiness probes should fail during that window — that's what tells the load balancer to stop routing traffic before the pod disappears.

Order the rollout — don't just click upgrade

The sequence matters more than the tooling. The order we run, regardless of whether the cluster is managed (EKS/AKS/GKE) or self-hosted:

1. Upgrade the control plane first, one minor version at a time, and confirm API compatibility for every webhook and CRD before touching nodes.
2. Surge a new node pool alongside the old one rather than upgrading nodes in place — this keeps a full-capacity fallback available for the entire rollout.
3. Cordon and drain one node at a time, watching disruption budgets do their job, not a scripted force-delete.
4. Canary-validate on the new node pool before draining the rest — error rate, p99 latency and saturation, not just pod status.
5. Decommission the old pool only once the new one has carried production traffic cleanly through at least one full deploy cycle.

$ kubectl drain node-eu-1 \
    --ignore-daemonsets \
    --delete-emptydir-data \
    --timeout=300s

Keep the old node pool's scale-down as a separate, deliberate step — not something that happens automatically the moment the drain command returns.

Validation gates before calling it done

"All pods Running" is not a validation gate. What we actually check before marking an upgrade complete:

• Error rate and latency on the real service dashboards, compared against the pre-upgrade baseline — not just synthetic health checks.
• A diff of Kubernetes events during the window, looking specifically for FailedScheduling, BackOff and admission webhook rejections.
• Confirmation that autoscalers (HPA/cluster autoscaler) settled back to their normal steady state rather than thrashing.
• A warm rollback path — the previous node pool and control-plane version still reachable — kept alive until the above checks are clean.

None of this requires exotic tooling. It requires treating the rollout as a change with defined success criteria, not a version number you bump and walk away from.