⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 1 of 207 in Kubernetes
Senior DevOps / SRE Kubernetes Amazon EKS & Upgrades Classic Scenario

Q: Walk me through an EKS upgrade zero downtime from v1.34 to v1.36+ — how did we avoid downtime?

Production runbook strategy to avoid downtime during an EKS upgrade zero downtime across Kubernetes minor versions (v1.34 → v1.35 → v1.36 → v1.37) using sequential control plane bumps, blue green managed node groups, PodDisruptionBudgets, and live continuous telemetry monitoring.

#Kubernetes #Amazon EKS #EKS Upgrade Zero Downtime #Cluster Upgrade #PDB #High Availability #AWS #Managed Node Groups
🎙️ Candidate Opening & Architectural Context
"In one of my production environments, we resolved Challenge 1 zero-downtime cluster version upgrades by migrating microservices across Kubernetes v1.34, v1.35, and v1.36. We achieved an EKS upgrade zero downtime by sequentially upgrading the AWS control plane, refreshing cluster add-ons, and replacing compute with blue/green managed node groups guarded by PodDisruptionBudgets, preStop hooks, and multi-AZ replicas."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: In one of my production environments, we resolved Challenge 1 zero-downtime cluster version upgrades by migrating microservices across Kubernetes v1.34, v1.35, and v1.36. We achieved an EKS upgrade zero downtime by sequentially upgrading the AWS control plane, refreshing cluster add-ons, and replacing compute with blue/green managed node groups guarded by PodDisruptionBudgets, preStop hooks, and multi-AZ replicas.

1️⃣

Pre-checks & Compatibility Verification: 5. Zero-Downtime Cluster Upgrade Runbook

Before initiating any cluster changes, execute these structured EKS cluster upgrade steps following the official aws eks upgrade documentation:

  • EKS Upgrade Insights: Queried AWS automated insights to detect deprecated APIs and cluster readiness blockers across Amazon EKS control planes.
  • API Deprecation Scans: Scanned Helm charts and manifests using Pluto and kubent to eliminate removed Kubernetes APIs against eks upgrade release notes. Verified no workloads relied on deprecated ingress or controller APIs.
  • Add-on Compatibility Matrices: Cross-referenced version support matrices for CoreDNS, kube-proxy, VPC CNI, and the AWS EBS CSI driver across both source and target minor versions.
  • AWS Load Balancer Controller: Verified TargetGroupBinding CRDs, IAM roles for service accounts (IRSA), and ingress controller annotations to prevent webhook rejection during admission control.
  • Workload Headroom & PDBs: Verified that critical deployments ran at least two replicas with active PodDisruptionBudgets. Confirmed sufficient compute headroom in target subnets to avoid IP exhaustion.
  • Telemetry Readiness: Verified real-time dashboards in Prometheus, Grafana, and CloudWatch Container Insights before proceeding.
  • Practitioner Resources: For engineers mastering core administration fundamentals, review our guide on What is the Best Kubernetes Learning Path for Beginners? (2026 Guide). When inspecting cluster compute capacity prior to migrations, consult Why is a Kubernetes Node marked as NotReady: Complete Production Runbook.
Pro Tip: Once the pre-checks were clean, we tested the complete upgrade in a lower environment first (Dev → Staging/UAT). We never started directly with Production.
2️⃣

How to Upgrade One Version at a Time: Upgrading EKS Cluster Without Downtime

Kubernetes minor version upgrades must be performed sequentially. To upgrade EKS cluster without downtime and learn how to upgrade EKS cluster safely, you cannot skip minor versions:

v1.33→v1.34→Validate→v1.35→Validate→v1.36→Validate→v1.37
# 1. Check current cluster version and health status
aws eks describe-cluster --name prod-core-eks --query "cluster.{Version:version,Status:status}" --output table

# 2. Trigger sequential control plane bump (e.g., v1.34 to v1.35)
aws eks update-cluster-version --name prod-core-eks --kubernetes-version 1.35

# 3. Monitor control plane status until Active
aws eks wait cluster-active --name prod-core-eks

# 4. Sequentially upgrade core managed cluster add-ons
aws eks update-addon --cluster-name prod-core-eks --addon-name vpc-cni --addon-version v1.18.1-eksbuild.1 --resolve-conflicts OVERWRITE
aws eks update-addon --cluster-name prod-core-eks --addon-name kube-proxy --addon-version v1.35.0-eksbuild.1
aws eks update-addon --cluster-name prod-core-eks --addon-name coredns --addon-version v1.11.3-eksbuild.1
  • Sequential Version Execution: To execute an aws upgrade eks version procedure, follow explicit sequential hops. We documented the eks upgrade from 1.32 to 1.33 (eks upgrade from 1 32 to 1 33), the eks upgrade from 1.33 to 1.34 (eks upgrade from 1 33 to 1 34), and earlier eks 1 31 to 1 32 upgrade baselines. Skipping versions (such as v1.34 directly to v1.36) is blocked by AWS.
  • EKS Control Plane Isolation: AWS manages the EKS control plane upgrade across multiple availability zones. The Kubernetes API server remains operational throughout, ensuring uninterrupted scheduling during sequential control plane bumping.
  • Add-on Synchronization: Upgraded VPC CNI, CoreDNS, and kube-proxy immediately after the control plane reached Active status and before modifying worker nodes.
  • Version Skew Verification: Ensured that worker node kubelets stayed within the supported two-minor-version skew relative to the upgraded API server.
Advertisement
3️⃣

Replace Worker Nodes (Blue/Green Node Groups): Workload Migration

For worker nodes, we execute zero-downtime blue-green cluster migration patterns across blue green node groups with an EKS optimized AMI (eks optimized ami) rather than performing in-place node termination:

⚡ Worker Node Upgrade Strategy Comparison: In-Place Rolling vs Blue/Green vs Karpenter
Strategy DimensionBlue/Green Node Groups (Used Here)In-Place Rolling MNGKarpenter Node Expiry
Workload RiskLowest: Old and new node pools run in parallel before cutoverModerate: Replaces nodes in-place within the same groupLow: Provisions fresh compute based on node drift
Rollback SpeedNear-instant: Uncordon blue nodes and redirect trafficSlow: Requires rebuilding older AMI instances in-placeFast: Revert NodePool manifest to previous AMI family
Capacity OverheadTemporary 100% compute overhead during migration windowMinimal: Controlled by maxUnavailable / maxSurge settingsJust-in-time provisioning matching exact pod resource requests
Stateful WorkloadsRequires single-AZ node affinity for EBS volume reattachmentManaged sequentially per instance lifecycle hookHandled through disruption budgets and topology constraints
Provision Green Node Group→Validate Ready Status→Cordon Blue Nodes→Drain Nodes Gradually→Verify Workload Telemetry
  • Provision Green Node Pool: Created a new managed node group configured with the latest EKS-optimized AMI matching the target Kubernetes minor version.
  • Cordon Old Node Fleet: Once green worker nodes registered as Ready, we marked each old node unschedulable using: kubectl cordon -l alpha.eksctl.io/nodegroup-name=blue-ng. Cordoning the old node ensures new pods only land on green instances.
  • Orderly Node Eviction: Drained one blue node at a time using kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --force, respecting PodDisruptionBudgets.
  • Stateful Endpoint Anchoring: Used zero-downtime safe cutover using external-name anchoring for stateful services with external dependencies.
4️⃣

How Did We Avoid Downtime (STAR Action): How to Achieve EKS Upgrade Zero Downtime

To upgrade eks cluster with zero downtime, application workloads must be architected for resilience against node eviction and network routing convergence:

🛡️ Multiple Replicas & Topology Spread
Every critical service maintained at least three replicas distributed across separate Availability Zones using topologySpreadConstraints.
⚖️ Strict PodDisruptionBudgets
Enforced PDBs with minAvailable: 2 or maxUnavailable: 1, guaranteeing that eviction routines could not compromise application capacity.
⏱️ PreStop Hooks & Graceful Shutdown
Implemented container preStop hooks with a 15-second sleep to allow kube-proxy iptables rules and AWS ALB target groups to deregister terminating pods.
🩺 Readiness Probes
Configured accurate HTTP readiness probes to ensure new pods on green nodes handled production requests only after full initialization.
5️⃣

Continuous Telemetry Monitoring: Verifying Real-Time System Health

Throughout continuous telemetry monitoring and the eks upgrade process, telemetry must confirm zero dropped HTTP requests, healthy target registrations, and normal latency envelopes:

  • Pending Pod Warnings: Monitored kube-scheduler metrics to verify that pods evicted from blue nodes immediately found capacity on green nodes without entering prolonged pending loops.
  • ALB Target Health: Monitored AWS Application Load Balancer target group health checks to verify zero unhealthy host registrations during pod evictions.
  • Application Latency: Tracked p95 and p99 latency percentiles in Prometheus to confirm that node migrations caused no processing degradation.
  • HTTP 5xx Error Tracking: Maintained an error budget alert in Grafana; zero 502 Bad Gateway responses were recorded during node draining.
  • Container Crash Loop Prevention: If application pods terminate unexpectedly due to configuration drift, review our triage runbook on What is CrashLoopBackOff Error in Kubernetes? (2026 Guide) and explore related scenarios in What are Kubernetes Scenario Interview Questions? (2026 Guide).
6️⃣

Validate Before Removing Anything: End-to-End Smoke Tests & Clean Teardown

Once all workloads migrated to the green node group, we maintained the blue pool in a cordoned state while running verification tests to validate before removing anything:

Customer Authentication→Database Query Latency→Payment Processing→Decommission Blue Pool
  • Smoke Test Verification: Executed automated end-to-end integration tests covering core business workflows under live user traffic.
  • Clean Node Pool Deletion: After a four-hour validation window with clean telemetry, safely deleted the old blue managed node group.
  • Subsequent Minor Version Iteration: Repeated the identical runbook sequence for the next minor release step (v1.35 → v1.36).
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Zero downtime in AWS was not achieved by simply issuing an upgrade command. It was guaranteed by architectural preparation: multi-AZ replica distribution, PodDisruptionBudgets, preStop sleep hooks for ALB deregistration, and parallel blue/green node group replacement."
⚡ 60-Second Elevator Pitch Talking Points
  • Conducted rigorous pre-flight audits using EKS Upgrade Insights, Pluto, and kubent to eliminate deprecated APIs prior to cluster modification.
  • Upgraded the control plane sequentially one minor version at a time (v1.34 → v1.35 → v1.36) with zero API downtime, followed immediately by core managed add-ons.
  • Executed a blue/green worker node migration by provisioning fresh node groups with the target EKS-optimized AMI, then cordoning and draining legacy nodes one by one.
  • Enforced strict high availability guardrails: multi-AZ topology spreads, PodDisruptionBudgets, and container preStop hooks to prevent 502 Bad Gateway errors during traffic shift.
  • Tracked continuous telemetry across ALB target group health, pending pods, and HTTP error rates to verify zero customer-facing disruption.
  • Validated end-to-end transaction flows under production load before safely decommissioning the legacy node group in this zero-downtime deployments devops interview.
🎬 60-Second Video Walkthrough
Watch the Animated Breakdown & Practice Live

Mastered this procedure? Subscribe to @naveedkumbhar01 on YouTube for daily 60-second SRE incident runbooks, Kubernetes triage shortcuts, and high-impact interview elevator pitches.

🔔 Subscribe on YouTube (Free) ▶ Watch Short (60s) ↗
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Draining cluster worker nodes without auditing PodDisruptionBudget availability and replica headroom Unconditional node draining without checking PDB maxUnavailable evicts quorum-critical pods simultaneously, causing cascading service outages.
  2. Skipping deprecated API manifest validation tools prior to initiating cluster upgrades Kubernetes regularly removes beta APIs; deploying modern cluster versions with outdated manifest APIs causes immediate deployment and controller rejections.
  3. Architecting zero-downtime workload migration without preStop lifecycle hooks Without a preStop sleep hook, kubelet terminates pods before ingress controllers and kube-proxy can remove their endpoints, resulting in dropped in-flight requests and 502 errors.
  4. Overlooking hands-on zero-downtime deployments for a stateful service with EBS volumes AWS EBS volumes are bound to a single Availability Zone; scheduling stateful pods on new nodes in a different AZ causes volume multi-attach or volume-not-found failures.
  5. Can you upgrade Amazon EKS worker nodes before the control plane? No, Kubernetes strictly enforces that worker node kubelet versions cannot be newer than the control plane API server. Worker nodes upgraded prematurely fail to register with the control plane.
  6. What happens if an Amazon EKS control plane upgrade fails? AWS manages EKS control plane upgrades with automated health checks across multiple availability zones. If health checks fail during a minor upgrade, AWS rolls back the control plane automatically without dropping API availability or terminating worker nodes.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →