Master 175+ battle-tested scenario-based Kubernetes interview questions for Senior DevOps, Cloud, and SRE engineers. Includes incident runbooks, STAR talking points, and CLI commands.
Production runbook strategy for upgrading Amazon EKS clusters across minor versions (v1.34 → v1.35 → v1.36 → v1.37) with zero application downtime using sequent...
Step-by-step diagnostic workflow for pods stuck in Pending state: decoding kube-scheduler events, capacity exhaustion, taints/tolerations, PVC binding, and auto...
Exhaustive triage process for debugging CrashLoopBackOff: decoding container exit codes (137 OOMKill, 1 app error, 143 SIGTERM), fetching previous container log...
Systematic networking approach for when pods report Running/Ready but the Kubernetes ClusterIP/NodePort/LoadBalancer Service cannot be reached....
Fast-response production incident runbook for rolling back broken Kubernetes deployments across native kubectl, GitOps (ArgoCD/Flux), and handling database migr...
Exhaustive diagnostic methodology for intermittent Kubernetes failures: node kernel pressure, CPU throttling, CoreDNS latency (ndots:5), CNI IP exhaustion, and ...
Clear decision matrix distinguishing Kubernetes Deployment, StatefulSet, and DaemonSet controllers, covering pod identity, network identity, persistent volume l...
Demystifying the Kubernetes Ingress architecture: distinguishing the Ingress Resource (manifest), Ingress Controller (reverse proxy daemon), and ClusterIP Servi...
Comprehensive diagnostic runbook for isolating the 4 primary root causes of ImagePullBackOff: image name/tag typos, missing or expired registry credentials, Doc...
Practical command toolkit for inspecting Kubernetes pod logs and cluster events: previous container crashes, multi-container pods, real-time log streaming with ...
Definitive architectural guide for configuring CPU and memory requests/limits: how kube-scheduler uses requests, how the Linux kernel enforces limits, CFS CPU t...
Deep dive into Kubernetes HPA: metrics collection pipeline via Metrics Server and Custom Metrics API, the exact mathematical autoscaling formula, stabilization ...
End-to-end investigative procedure for identifying and resolving runaway CPU and memory saturation in Kubernetes clusters: isolating nodes vs pods, diagnosing m...
The definitive lifecycle of a Kubernetes resource: tracing 'kubectl apply' from client OpenAPI validation, API server authentication & admission w...
First run kubectl describe pod <pod-name> and look at the Events section at the bottom. Common reasons for Pending:...
CrashLoopBackOff means the container starts, crashes, and Kubernetes keeps restarting it with increasing delay....
OOMKilled means the container exceeded its memory limit and the kernel killed it....
This is typical RollingUpdate behavior when new pods fail healthchecks....
1. Check pod labels vs service selector — kubectl describe service <name> shows the selector. kubectl get pod --show-labels shows pod lab......
1. Don't panic — check if pods already rescheduled. Kubernetes evicts pods from NotReady nodes after pod-eviction-timeout (default 5 min)......
HPA needs metrics-server to be installed and working. Without it, HPA can't read CPU/memory metrics and shows <unknown> in kubectl get hpa....
Since nothing changed in the spec, suspect external changes:...
If a pod has multiple containers, you need to specify which one:...
Init containers run sequentially before the main container. If one fails, the pod stays in Init:0/1 or similar state....
- Deployment — for stateless apps. Pods are interchangeable. Any pod can handle any request. Use for web servers, APIs, workers....
Not ideal. A Deployment doesn't guarantee stable pod identity or ordered startup/shutdown, which matters for clustered databases (Postgre......
Environment variables are loaded at pod start time. Changing a ConfigMap doesn't restart running pods, so they keep the old values....
Two approaches:...
Use Pod Anti-Affinity:...
Configure RollingUpdate strategy:...
A DaemonSet ensures one pod runs on every node (or a subset of nodes). When a new node joins the cluster, the DaemonSet automatically pla......
Common reasons:...
- Job — run a task once to completion. E.g., database migration on deploy, one-time data processing, sending a batch of emails....
Set concurrencyPolicy: Forbid in the CronJob spec. This skips the new run if the previous one is still running....
- ClusterIP — only accessible inside the cluster. Default type. Used for internal service-to-service communication....
Every LoadBalancer service creates a new cloud load balancer = new cost + new IP address. For 10 services, that's 10 load balancers....
1. Check the Ingress resource — kubectl describe ingress <name> — verify the path and service name are correct....
Use NetworkPolicy. By default, all pods can talk to all other pods. NetworkPolicy lets you restrict this....
A headless service has clusterIP: None. Instead of a single virtual IP, DNS queries for a headless service return the actual pod IPs dire......
1. Baseline test — kubectl exec -it <pod-a> -- curl -o /dev/null -s -w "%{time_total}" http://<service>:<port> to measure actual lat...
1. kubectl get pods -n kube-system | grep coredns — is CoreDNS running?...
- PV (PersistentVolume) — the actual storage resource. Could be an AWS EBS volume, NFS share, local disk. Created by a cluster admin or d......
1. No matching PV — check if a PV exists with matching storageClassName, accessMode, and enough capacity: kubectl get pv....
By setting the Reclaim Policy on the PV or StorageClass:...
This is a common scenario when a node dies without gracefully releasing its volumes. The PV shows Terminating or the pod shows volume att......
RBAC controls are namespace-scoped. Check:...
1. Create a ServiceAccount: kubectl create serviceaccount my-app -n my-namespace...
This is a serious situation. If you're locked out of the API server entirely:...
HPA is reactive — it waits for metrics to breach thresholds before scaling. By then you've already had a slowdown....
HPA has a stabilization window to prevent thrashing. Tune it:...
The kube-scheduler evaluates all nodes for each pod. At 50 nodes it shouldn't be slow unless:...
Set securityContext on the pod/container:...
StatefulSets don't support zero-downtime rolling updates as cleanly as Deployments because each pod has unique state....
Tools: ArgoCD or Flux — both are CNCF projects....
Default Kubernetes Secrets problems:...
Several approaches:...
Yes — this is a supported temporary state during upgrades. Kubernetes supports N-2 version skew between control plane and nodes. A 1.27 c......
Two main approaches:...
A PodDisruptionBudget (PDB) limits how many pods of a deployment can be voluntarily disrupted at the same time. "Voluntary disruption" in......
kubectl drain does two things:...
The scheduler goes through two phases:...
A LimitRange sets default and maximum resource requests/limits for pods in a namespace. If a pod doesn't specify resources, LimitRange fi......
Soft multi-tenancy in Kubernetes (hard isolation requires separate clusters):...
- kubectl create — imperative. Creates the resource. Fails if it already exists. Good for one-time resource creation....
1. Wrong port or path — probe is checking a different port/endpoint than the app actually serves....
- Liveness probe — "is this container still alive?" If it fails, Kubernetes restarts the container. Use for detecting deadlocks or infini......
1. API server receives the pod spec, validates it, stores it in etcd. Pod status: Pending....
1. Identify the policy: kubectl get networkpolicy -A — list all NetworkPolicies across namespaces....
etcd is the key-value store that is Kubernetes' "brain." Every cluster state (pod specs, node info, secrets, configmaps, events) is store......
Use Kubernetes Secrets:...
yaml...
- Request — the guaranteed amount. Scheduler uses this to decide which node has enough room....
1. Guaranteed — requests == limits for all containers. Both CPU and memory. Highest priority. Evicted last....
Containers in the same pod share:...
- emptyDir — temporary directory created when pod starts, deleted when pod is removed. Shared between containers in the pod. Good for scr......
1. Pod is unschedulable for a reason other than resources — e.g., node affinity requires a specific label that no node type has. CA won't......
bash...
Helm is a package manager for Kubernetes. A Helm chart bundles all the Kubernetes YAML for an application (Deployment, Service, ConfigMap......
kubelet monitors node memory usage against eviction thresholds:...
When a pod is deleted, Kubernetes sends SIGTERM to the container and waits terminationGracePeriodSeconds (default: 30s) for the app to sh......
Use an init container that polls for the ConfigMap:...
This spreads pods across availability zones rather than just across nodes. If your cluster spans 3 AZs (us-east-1a, 1b, 1c), this anti-af......
In-tree volume plugins (like the old AWS EBS plugin built into kubelet) had problems:...
Admission webhooks intercept API requests before they're stored in etcd. Mutating webhooks can modify the request (add/change fields). Va......
Image not found or wrong tag. Check image name/tag. For private registry, ensure imagePullSecret is configured....
All pods failing readiness. Check probe config and app logs....
kubectl scale deployment <name> --replicas=5 or update spec.replicas....
Check firewall/security group rules allow the NodePort (30000-32767) range....
CNI plugin may not support NetworkPolicy. Check CNI (Flannel doesn't, Calico does)....
Upgrade control plane first (API server, etcd, scheduler). Then drain, upgrade, uncordon worker nodes one by one....
No LoadBalancer IP assigned yet. On bare-metal, need MetalLB. On cloud, wait 1-2 min for cloud LB provisioning....
kubectl logs -l app=<label> or use label selector....
metrics-server not installed or pods have no resource requests set....
etcdctl snapshot save backup.db. Restore: stop API server, restore snapshot, restart....
From backup/GitOps. There's no undo in kubectl. This is why GitOps (ArgoCD/Flux) matters — re-apply the Git state....
Force delete: kubectl delete pod <name> --grace-period=0 --force. Usually caused by finalizers or stuck volumes....
NetworkPolicy = L3/L4 (IP/port). Service mesh (Istio/Linkerd) = L7 (HTTP routing, mTLS, retries, circuit breaking). Use both for layered ......
Add annotation with configmap hash: checksum/config: {{ include (print .Template.BasePath "/configmap.yaml") . | sha256sum }} in Helm. Or......
successfulJobsHistoryLimit may be 0 or 1 and old jobs were cleaned. Adjust to keep history....
If webhook server is down, set failurePolicy: Ignore on the webhook or delete the MutatingWebhookConfiguration object....
kubectl auth can-i create pods --as=system:serviceaccount:<namespace>:<sa-name>...
get = brief summary table. describe = full detail including events. Use describe for debugging....
kubectl debug node/<node-name> -it --image=ubuntu or use nodeName field in pod spec....
kube-proxy watches Services/Endpoints. Creates iptables DNAT rules: traffic to ClusterIP is redirected to one of the pod IPs using a roun......
TopologySpreadConstraints gives fine-grained control over pod distribution (e.g., max skew of 1 between zones). Anti-affinity is binary. ......
Node must have GPU + GPU device plugin installed. Pod requests: resources.limits: nvidia.com/gpu: 1. Scheduler finds a node with availabl......
Scheduler and controller-manager use Lease objects in etcd. Only the leader processes work. Others watch. If leader fails to renew its le......
Finalizer is a string in metadata.finalizers. Prevents object deletion until the finalizer is removed. Use case: ensure external resource......
Uses owner references. When a parent object (Deployment) is deleted, GC deletes owned objects (ReplicaSets → Pods) in cascade. --cascade=......
kubectl debug -it <pod> --image=ubuntu --share-processes --copy-to=debug-pod — creates a copy of the pod with an extra debug container....
The API server supports a watch query param. Client gets a stream of events (ADDED/MODIFIED/DELETED) instead of polling. Informers use th......
-f applies a single file or directory of raw YAML. -k runs Kustomize, applying base + overlays, generating configs, and applying the merg......
1) Snapshot PVC data (Velero). 2) Deploy workload in new cluster from same Git source. 3) Restore data snapshot to new cluster PVCs. 4) T......
KEDA (Kubernetes Event-Driven Autoscaling) scales pods based on external event sources: Kafka topic lag, RabbitMQ queue depth, HTTP reque......
Use a Service with the correct port and protocol annotation. For Ingress: you need an ingress controller that supports gRPC (nginx-ingres......
DaemonSets support kubectl rollout undo daemonset/<name> similar to Deployments. DaemonSet keeps rollout history (configurable via revisi......
Same as any pod: kubectl describe job <n>, check the created pod's events. Common issues: image pull error, no nodes with enough resource......
Use Vault Agent Injector (Vault installed in K8s). Annotate the pod:...
The control loop pattern: watch current state → compare with desired state → take action to reconcile. Controllers (Deployment controller......
Without restriction, any pod can query the EC2 metadata endpoint and potentially steal the node's IAM role credentials. Block it with Net......
In K8s 1.21+, ServiceAccount tokens are bound tokens — time-limited (default 1 hour), audience-specific, automatically rotated by the kub......
Helm chart = templated K8s YAML for stateless deployment. No runtime intelligence. Good for most stateless apps....
Use two Deployments with the same Service label selector but different replica counts:...
kube-proxy runs on every node (as a DaemonSet) and maintains network rules (iptables or ipvs) that implement Services. It watches the API......
Store the nginx.conf in a ConfigMap. Mount it as a volume in the Deployment. All pods get the same config from the same source. When conf......
Both spread pods across topology domains (nodes, zones)....
Use the gRPC Health Checking Protocol. Your service implements grpc.health.v1.Health/Check. In the probe:...
In iptables mode: kube-proxy creates a chain of iptables rules for each Service. With thousands of Services, rule traversal becomes linea......
Use GitOps (ArgoCD) with a shared Git repository. Both clusters sync from the same manifests repo. Each cluster has its own ArgoCD instan......
CRI is the API between kubelet and the container runtime. kubelet doesn't care what runtime is used as long as it speaks CRI....
Use KEDA (Kubernetes Event-Driven Autoscaling). KEDA supports 50+ built-in scalers:...
Node is under resource pressure. kubelet evicts lower-priority pods to free resources. Check: kubectl describe pod <n> — reason will say ......
Options:...
The "infra container" or "sandbox container" that holds the network namespace for the pod. All containers in the pod join its network nam......
Always: pulls from registry every pod start (ensures latest changes). IfNotPresent: uses local cache if image tag exists. Production: use......
Same as regular containers under initContainers[].resources. Init containers don't run simultaneously, so effective pod request = max(ini......
Combines multiple volume sources (secrets, configmaps, serviceAccountToken, downward API) into a single directory mount. Useful when the ......
kubectl get node <n> --show-labels or kubectl describe node <n>. Add labels: kubectl label node <n> key=value....
Exposes pod/node metadata (pod name, namespace, labels, annotations, resource limits) to the container as env vars or volume files. Usefu......
Set PodDisruptionBudgets on all critical workloads. kubectl drain --ignore-daemonsets --delete-emptydir-data. The drain respects PDBs — w......
Shows what would change if you applied a manifest, compared to what's currently running. Like terraform plan for Kubernetes....
LimitRange with defaultRequest and default limits OR use OPA/Gatekeeper/Kyverno policy that rejects pods without resource limits....
VPA adjusts CPU/memory requests of running pods based on actual usage. Use VPA for: workloads where you know they need more resources but......
kubectl port-forward service/<name> 8080:80 — forwards local port 8080 to the service's port 80. Works through the API server tunnel. Kil......
Shows real-time CPU/memory usage of nodes and pods. Requires metrics-server to be installed. kubectl top nodes and kubectl top pods....
Three levels: Privileged (no restrictions), Baseline (blocks most dangerous capabilities — no privileged, no hostPath), Restricted (most ......
Lightweight K8s object used for leader election by control plane components (scheduler, controller-manager) and custom controllers. The h......
kubectl get events -n <namespace> --sort-by='.lastTimestamp'. Events are a great first stop when debugging — they capture all resource st......
Service mesh (Istio, Linkerd) adds a sidecar proxy to every pod, enabling: mTLS, traffic policies, retries, circuit breaking, distributed......
Use Fluentd or Filebeat as a DaemonSet. They read container logs from /var/log/containers/ on each node and ship to Elasticsearch. Altern......
Extended Berkeley Packet Filter — runs sandboxed programs in the Linux kernel. In K8s: Cilium uses eBPF instead of iptables for service r......
K8s deletes all resources in the namespace in dependency order. The namespace stays in Terminating until all resources are deleted. If a ......
Control plane nodes are tainted with node-role.kubernetes.io/control-plane:NoSchedule. Add a toleration to your pod: tolerations: [{key: ......
By default, K8s has NO network isolation. All pods can talk to all other pods in the cluster, across namespaces. This is intentional (for......
A helper container that runs alongside the main app container in the same pod, sharing its network and volumes. Examples: Istio proxy (En......
Use downward API:...
Federation v1 (deprecated) tried to manage multiple clusters from a single control plane. It was complex and unreliable. Federation v2 (K......
Use cert-manager with ClusterIssuer: selfsigned. Or: openssl req -x509 -nodes -newkey rsa:2048 -out tls.crt -keyout tls.key, then kubectl......
Plugins that intercept API requests AFTER authentication/authorization but BEFORE persistence in etcd. Two types: Mutating (modify the re......
kubectl logs <pod> -c <container-name>. Get container names: kubectl get pod <pod> -o jsonpath='{.spec.containers[].name}'....
When combined with a label selector, it deletes resources that were previously applied with kubectl apply but are no longer in the curren......
Force all outbound traffic through a central point (useful for IP whitelisting at third-party APIs). With Istio: configure an EgressGatew......
client: validates locally using the cached schema. server: sends to the API server which validates (including webhook admission controlle......
With Istio: EnvoyFilter or RateLimitPolicy using a Redis-backed rate limit service. With nginx-ingress: nginx.ingress.kubernetes.io/limit......
Architectural runbook for scaling Istio and Envoy service discovery across 1,000+ microservices without control-plane xDS push storms, memory bloat, or OOM cras...
How to design multi-tier Kubernetes liveness, readiness, and startup probes that detect internal thread deadlocks, downstream connection pool starvation, and da...
Systematic triage guide for diagnosing why a Kubernetes Horizontal Pod Autoscaler fails to trigger replica scale-outs despite external Prometheus dashboards ale...
How to professionally articulate your current Kubernetes production version (v1.29/v1.30+), enforce supported version skew across control plane and worker nodes...
Real-world incident response narrative: diagnosing a sudden 5xx error spike caused by an invalid downstream endpoint in a ConfigMap feature flag, rolling back i...
Root-cause analysis and resolution for intermittent HTTP 502 Bad Gateway errors during traffic peaks: diagnosing upstream timeouts, CPU throttling, readiness pr...
Systematic triage when a Helm release fails due to resource constraints: diagnosing scheduling bottlenecks (node CPU/RAM starvation) vs namespace ResourceQuota ...
Modern strategy for publishing and sharing internal Helm charts across engineering teams via OCI registries (ECR, Harbor, GHCR) with semantic versioning, values...
Comprehensive automated testing pipeline for enterprise Helm charts: template linting, schema validation, Go template unit tests with helm-unittest, and integra...
Production engineering framework for building highly available, resiliently autoscaling Kubernetes workloads: Multi-AZ node spreading, PodDisruptionBudgets (PDB...
Deep dive into what actually happens when a Kubernetes Pod 'restarts': differentiating in-place container restarts by Kubelet from Pod deletion/resche...