Q: Describe leader election in Kubernetes control plane components.
Leader election in the Kubernetes control plane lets several replicas of kube-controller-manager and kube-scheduler run for high availability while only one actively makes decisions.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: <strong>Leader election in the Kubernetes control plane lets several replicas of kube-controller-manager and kube-scheduler run for high availability while only one actively makes decisions.</strong> Replicas compete to hold a <code>Lease</code> object in <code>kube-system</code>, and the holder must renew it regularly or another replica takes over. Interviewers ask because it explains control plane failover, split-brain avoidance and why two controller-managers do not fight each other. A complete answer separates this from etcd's Raft consensus and from the API server, which runs active-active without a leader.
Mental model: which components elect a leader
Only components that make cluster-wide decisions need a single active instance.
- kube-scheduler: one active scheduler assigns pods, so two active would double-bind pods.
- kube-controller-manager: one active instance runs all built-in controllers. The cloud-controller-manager does the same.
- kube-apiserver: not leader-elected. All API servers serve traffic simultaneously behind a load balancer, since they are stateless over etcd.
- etcd: has its own leader through the Raft consensus protocol, unrelated to the Lease mechanism described here.
- Standby behaviour: non-leader replicas keep running, hold caches and watch the Lease, but perform no scheduling or reconciliation. This is why failover is fast: informers are already warm.
Hands-on: leader election in Kubernetes control plane components via Lease
Election uses the coordination.k8s.io Lease API and optimistic concurrency from the API server, with etcd as the source of truth.
kubectl -n kube-system get lease
kubectl -n kube-system get lease kube-scheduler -o yaml
# spec:
# holderIdentity: control-plane-1_8f1c... <- current leader
# leaseDurationSeconds: 15
# renewTime: "2026-10-25T10:00:03.000000Z"
# leaseTransitions: 4 <- number of leader changes
# Quick view of both leaders
kubectl -n kube-system get lease kube-scheduler kube-controller-manager \
-o custom-columns=NAME:.metadata.name,HOLDER:.spec.holderIdentity
- Acquire: each replica tries to create or update the Lease and write its identity into
spec.holderIdentity. Optimistic concurrency throughresourceVersionmeans only one write wins. - Renew: the leader updates
renewTimeperiodically. Followers watch and retry on a fixed interval. - Expire: if the lease is not renewed within the lease duration, a follower acquires it and becomes leader. The old leader stops work if it cannot renew in time.
- Where: Leases are named
kube-schedulerandkube-controller-managerinkube-system. - Optimistic locking: updating the Lease uses
resourceVersionin the update request, so two replicas racing for an expired lease cannot both succeed. The loser receives a conflict and stays a follower.
Timing parameters and failover behaviour
Three flags control how fast failover happens, and the relationship between them matters.
# Static pod flags (kubeadm: /etc/kubernetes/manifests/kube-scheduler.yaml)
- --leader-elect=true
- --leader-elect-lease-duration=15s
- --leader-elect-renew-deadline=10s
- --leader-elect-retry-period=2s
# Worst-case failover is roughly lease duration after the last renew
- --leader-elect: enables election and defaults to true for both components.
- --leader-elect-lease-duration: how long non-leaders wait before taking over, default 15s.
- --leader-elect-renew-deadline: how long the leader keeps retrying to renew before giving up, default 10s, and it must be shorter than the lease duration.
- --leader-elect-retry-period: how often candidates attempt acquisition or renewal, default 2s. Verify defaults for your version with
--help. - Failover estimate: a crashed leader is replaced after roughly the lease duration plus the retry period, because followers only notice expiry at their next retry. Lowering the values speeds failover but raises API traffic and risk of flapping.
Failure modes and troubleshooting
Most issues show up as no leader, flapping leaders or a leader that is alive but unhealthy.
- Frequent transitions: a rising
leaseTransitionscount points to slow API server or etcd, CPU starvation or network delays, which stop renewals. - Stale holder: a
renewTimefar in the past with no new holder suggests all replicas are down or cannot reach the API server. - Clock skew: timestamps come from the holder, but expiry is judged by observed local time, so election is relatively robust. Still, keep NTP healthy.
- Safety limit: election does not guarantee a deposed leader stops instantly, so controllers must tolerate brief overlap, using resourceVersion conflicts on writes.
- Check control plane health: compare the Lease holder with
kubectl get pods -n kube-system -o wideto find which node runs the active scheduler, and read that pod's logs when scheduling decisions look wrong.
Production practice and custom controllers
The same mechanism is available to your own operators.
- client-go: the
leaderelectionpackage with a Lease lock is what most operators use, and controller-runtime enables it withLeaderElection: true. - Run replicas: two or three control plane nodes spread across failure domains, with a standby for every elected component.
- Monitor: alert on leader changes, stale leases and the leader_election_master_status metric where exposed.
- RBAC: custom controllers need get, create and update on leases in their namespace.
- Upgrades and maintenance: when draining or upgrading control plane nodes one at a time, check the Lease holder first. Restarting the current leader triggers a short failover, so schedule it outside peak scheduling activity and confirm a new holder appears.
- Scheduler and controller-manager run several replicas but only the holder of a Lease in kube-system is active; the API server is active-active and etcd uses Raft.
- The leader renews the lease every few seconds and a follower takes over if it expires, with 15s, 10s and 2s as default timings.
- I inspect holderIdentity and leaseTransitions, alert on flapping, and use the same Lease pattern in custom operators.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Saying the API server has a leader API servers are stateless and serve simultaneously; only etcd has Raft and only two components use Leases.
- Mixing up etcd Raft with Lease election They are separate mechanisms with different failure semantics.
- Setting renew deadline longer than lease duration The leader could keep acting after others consider the lease expired.
- Ignoring leaseTransitions Repeated changes reveal control plane resource or latency problems before a full outage.