⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 103 of 207 in Kubernetes
Staff SRE / Principal Architect [L3] Kubernetes Advanced Scenarios Staff SRE Scenario [L3]
🎯 Target Role / Context: Kubernetes Scenario #93 (L3) · Advanced Scenarios

Q: Describe leader election in Kubernetes control plane components.

Leader election in the Kubernetes control plane lets several replicas of kube-controller-manager and kube-scheduler run for high availability while only one actively makes decisions.

#Kubernetes #Advanced Scenarios #L3 #Container Orchestration #K8s #etcd
🎙️ Candidate Opening & Architectural Context
"<strong>Leader election in the Kubernetes control plane lets several replicas of kube-controller-manager and kube-scheduler run for high availability while only one actively makes decisions.</strong> Replicas compete to hold a <code>Lease</code> object in <code>kube-system</code>, and the holder must renew it regularly or another replica takes over. Interviewers ask because it explains control plane failover, split-brain avoidance and why two controller-managers do not fight each other. A complete answer separates this from etcd's Raft consensus and from the API server, which runs active-active without a leader."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: <strong>Leader election in the Kubernetes control plane lets several replicas of kube-controller-manager and kube-scheduler run for high availability while only one actively makes decisions.</strong> Replicas compete to hold a <code>Lease</code> object in <code>kube-system</code>, and the holder must renew it regularly or another replica takes over. Interviewers ask because it explains control plane failover, split-brain avoidance and why two controller-managers do not fight each other. A complete answer separates this from etcd's Raft consensus and from the API server, which runs active-active without a leader.

1️⃣

Mental model: which components elect a leader

Only components that make cluster-wide decisions need a single active instance.

  • kube-scheduler: one active scheduler assigns pods, so two active would double-bind pods.
  • kube-controller-manager: one active instance runs all built-in controllers. The cloud-controller-manager does the same.
  • kube-apiserver: not leader-elected. All API servers serve traffic simultaneously behind a load balancer, since they are stateless over etcd.
  • etcd: has its own leader through the Raft consensus protocol, unrelated to the Lease mechanism described here.
  • Standby behaviour: non-leader replicas keep running, hold caches and watch the Lease, but perform no scheduling or reconciliation. This is why failover is fast: informers are already warm.
2️⃣

Hands-on: leader election in Kubernetes control plane components via Lease

Election uses the coordination.k8s.io Lease API and optimistic concurrency from the API server, with etcd as the source of truth.

kubectl -n kube-system get lease
kubectl -n kube-system get lease kube-scheduler -o yaml
# spec:
#   holderIdentity: control-plane-1_8f1c...   <- current leader
#   leaseDurationSeconds: 15
#   renewTime: "2026-10-25T10:00:03.000000Z"
#   leaseTransitions: 4                       <- number of leader changes

# Quick view of both leaders
kubectl -n kube-system get lease kube-scheduler kube-controller-manager \
  -o custom-columns=NAME:.metadata.name,HOLDER:.spec.holderIdentity
  • Acquire: each replica tries to create or update the Lease and write its identity into spec.holderIdentity. Optimistic concurrency through resourceVersion means only one write wins.
  • Renew: the leader updates renewTime periodically. Followers watch and retry on a fixed interval.
  • Expire: if the lease is not renewed within the lease duration, a follower acquires it and becomes leader. The old leader stops work if it cannot renew in time.
  • Where: Leases are named kube-scheduler and kube-controller-manager in kube-system.
  • Optimistic locking: updating the Lease uses resourceVersion in the update request, so two replicas racing for an expired lease cannot both succeed. The loser receives a conflict and stays a follower.
Advertisement
3️⃣

Timing parameters and failover behaviour

Three flags control how fast failover happens, and the relationship between them matters.

# Static pod flags (kubeadm: /etc/kubernetes/manifests/kube-scheduler.yaml)
- --leader-elect=true
- --leader-elect-lease-duration=15s
- --leader-elect-renew-deadline=10s
- --leader-elect-retry-period=2s

# Worst-case failover is roughly lease duration after the last renew
  • --leader-elect: enables election and defaults to true for both components.
  • --leader-elect-lease-duration: how long non-leaders wait before taking over, default 15s.
  • --leader-elect-renew-deadline: how long the leader keeps retrying to renew before giving up, default 10s, and it must be shorter than the lease duration.
  • --leader-elect-retry-period: how often candidates attempt acquisition or renewal, default 2s. Verify defaults for your version with --help.
  • Failover estimate: a crashed leader is replaced after roughly the lease duration plus the retry period, because followers only notice expiry at their next retry. Lowering the values speeds failover but raises API traffic and risk of flapping.
4️⃣

Failure modes and troubleshooting

Most issues show up as no leader, flapping leaders or a leader that is alive but unhealthy.

  • Frequent transitions: a rising leaseTransitions count points to slow API server or etcd, CPU starvation or network delays, which stop renewals.
  • Stale holder: a renewTime far in the past with no new holder suggests all replicas are down or cannot reach the API server.
  • Clock skew: timestamps come from the holder, but expiry is judged by observed local time, so election is relatively robust. Still, keep NTP healthy.
  • Safety limit: election does not guarantee a deposed leader stops instantly, so controllers must tolerate brief overlap, using resourceVersion conflicts on writes.
  • Check control plane health: compare the Lease holder with kubectl get pods -n kube-system -o wide to find which node runs the active scheduler, and read that pod's logs when scheduling decisions look wrong.
Pro Tip: If the scheduler or controller-manager seems idle, find the holder first. A healthy standby doing nothing is expected, but a missing or stale leader is an outage.
5️⃣

Production practice and custom controllers

The same mechanism is available to your own operators.

  • client-go: the leaderelection package with a Lease lock is what most operators use, and controller-runtime enables it with LeaderElection: true.
  • Run replicas: two or three control plane nodes spread across failure domains, with a standby for every elected component.
  • Monitor: alert on leader changes, stale leases and the leader_election_master_status metric where exposed.
  • RBAC: custom controllers need get, create and update on leases in their namespace.
  • Upgrades and maintenance: when draining or upgrading control plane nodes one at a time, check the Lease holder first. Restarting the current leader triggers a short failover, so schedule it outside peak scheduling activity and confirm a new holder appears.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Check kubectl get lease in kube-system to see who leads right now; a climbing leaseTransitions count is an early warning of API server or etcd slowness."
⚡ 60-Second Elevator Pitch Talking Points
  • Scheduler and controller-manager run several replicas but only the holder of a Lease in kube-system is active; the API server is active-active and etcd uses Raft.
  • The leader renews the lease every few seconds and a follower takes over if it expires, with 15s, 10s and 2s as default timings.
  • I inspect holderIdentity and leaseTransitions, alert on flapping, and use the same Lease pattern in custom operators.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Saying the API server has a leader API servers are stateless and serve simultaneously; only etcd has Raft and only two components use Leases.
  2. Mixing up etcd Raft with Lease election They are separate mechanisms with different failure semantics.
  3. Setting renew deadline longer than lease duration The leader could keep acting after others consider the lease expired.
  4. Ignoring leaseTransitions Repeated changes reveal control plane resource or latency problems before a full outage.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →