⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 14 of 207 in Kubernetes
Senior DevOps / SRE Kubernetes Control Plane & Orchestration Internals Core Architecture
🎯 Target Role / Context: Senior / Staff Kubernetes Architecture Interview Question

Q: What really happens under the hood when you run 'kubectl apply -f deployment.yaml'? Walk me through the entire journey from client CLI to running container.

The definitive lifecycle of a Kubernetes resource: tracing 'kubectl apply' from client OpenAPI validation, API server authentication & admission webhooks, etcd persistence, controller reconciliation, scheduler scoring, kubelet CRI execution, to CNI network namespace plumbing.

#Kubernetes #Architecture #Control Plane #kubectl apply #API Server #etcd #Scheduler #kubelet #CNI #Internals
🎙️ Candidate Opening & Architectural Context
"Most Kubernetes engineers run 'kubectl apply' daily, but behind that single command lies an entire distributed orchestration engine. Kubernetes operates as a Desired State System: you declare the desired state, and a chain of independent control loops continuously works to reconcile reality to match that state."
📊 Architecture Runbook & Flow Diagram
What Really Happens Under the Hood When You Run 'kubectl apply -f deployment.yaml'?
🔍 Click diagram to open full-resolution high-res image in new tab
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: Most Kubernetes engineers run 'kubectl apply' daily, but behind that single command lies an entire distributed orchestration engine. Kubernetes operates as a Desired State System: you declare the desired state, and a chain of independent control loops continuously works to reconcile reality to match that state.

1️⃣

Client-Side Processing & Validation (kubectl)

Before any network request reaches the cluster, kubectl performs client-side inspection:

  • Client-Side Validation: kubectl verifies YAML syntax and checks resource fields against the locally cached OpenAPI / Swagger schema from the cluster (~/.kube/cache/discovery).
  • Three-Way Strategic Merge Patch: Reads the kubectl.kubernetes.io/last-applied-configuration annotation, compares it with the live cluster state and the new local YAML, and computes a JSON Strategic Merge Patch.
  • HTTP REST Request: Formats the payload into JSON and dispatches an HTTP POST/PATCH request to the API server: POST /apis/apps/v1/namespaces/default/deployments with TLS client certificates or OIDC Bearer tokens.
2️⃣

API Server: Authentication, Authorization & Admission Control

The kube-apiserver is the single front door to the cluster. The request traverses 4 sequential filters:

Authentication (Who are you?)→Authorization / RBAC (Can you do this?)→Mutating Webhooks (Inject defaults)→Object Schema Validation→Validating Webhooks (Accept or Reject)
  • 1. Authentication: Validates TLS cert, ServiceAccount token, or Entra/AWS IAM OIDC identity to establish the caller's username and groups.
  • 2. Authorization (RBAC): Evaluates ClusterRoles/RoleBindings to verify if the subject has create / patch permissions on deployments in the target namespace.
  • 3. Mutating Admission Webhooks: Plugins and webhooks (e.g. Istio sidecar injector, Vault agent injector) modify the object or set default values.
  • 4. Schema Validation: Enforces API schema rules, required fields, and immutability constraints.
  • 5. Validating Admission Webhooks: Webhooks (e.g. Kyverno, OPA Gatekeeper) run final compliance checks (e.g. enforcing non-root execution or image registry whitelisting) and reject invalid manifests.
Advertisement
3️⃣

etcd: State Persistence & Consensus

The single source of truth commits the change:

  • Once admission passes, kube-apiserver serializes the Deployment object into protocol buffers and writes it into etcd at key /registry/deployments/default/my-app.
  • etcd commits the write across a quorum of nodes using the Raft consensus algorithm.
  • The API server returns an HTTP 201 Created or 200 OK response back to the client CLI.
  • Important: At this exact second, NO containers or pods exist yet! Only the desired state is recorded.
4️⃣

Deployment Controller & ReplicaSet Controller (kube-controller-manager)

The control loops take over asynchronously:

API Server Watch Notification→Deployment Controller→Create ReplicaSet→ReplicaSet Controller→Create Unassigned Pods
  • Deployment Controller: Watches the API server for Deployment changes. Detects the new Deployment and creates a child ReplicaSet object with pod template hash.
  • ReplicaSet Controller: Detects the new ReplicaSet desiring e.g. 3 replicas. Compares desired replicas (3) with existing replicas (0).
  • It issues 3 API requests to create 3 Pod objects. Crucially, these Pod objects have no assigned node (spec.nodeName: '') and enter the Pending state.
5️⃣

kube-scheduler: Node Filtering & Scoring

Matching unbound pods to healthy worker nodes:

  • Watch Loop: kube-scheduler continuously watches the API server for Pods where spec.nodeName == ''.
  • Phase 1 (Filtering / Predicates): Eliminates ineligible nodes that lack sufficient CPU/memory requests, have untolerated taints, or fail nodeSelector / affinity constraints.
  • Phase 2 (Scoring / Priorities): Scores remaining nodes based on image locality, topology spread, and resource fragmentation (least/most requested).
  • Binding: Scheduler picks the highest-scoring node and sends a Binding API call to the API server, setting spec.nodeName: 'worker-node-2'.
6️⃣

Kubelet & Container Runtime (CRI containerd)

The local node agent brings the container to life:

  • Kubelet Watch: The kubelet daemon running on worker-node-2 observes that a Pod has been assigned to its node name.
  • Container Runtime Interface (CRI): Kubelet calls containerd via gRPC.
  • Image Pull: containerd checks if the image exists in local cache; if not, pulls it from registry using node IAM credentials or imagePullSecrets.
  • Pod Sandbox Creation: Kubelet instructs containerd to create the Pod Sandbox (pause container) establishing Linux namespaces (IPC, UTS, PID, Network).
7️⃣

CNI Plugin: Network Namespace & Pod IP Allocation

Plumbing the container into the cluster network:

  • Kubelet invokes the CNI plugin (AWS VPC CNI, Calico, Flannel, Cilium).
  • The CNI creates a virtual ethernet pair (veth), moves one interface into the pod's network namespace as eth0, and connects the other interface to the host bridge or routing table.
  • Allocates a dedicated Pod IP from the subnet CIDR and configures routing and MTU.
  • Once networking is ready, containerd starts the application containers inside the pod sandbox.
8️⃣

Readiness Probes & Service Endpoints Routing

Connecting the running pod to live client traffic:

  • Kubelet continuously executes configured startupProbe and readinessProbe.
  • Once probes return HTTP 200 / Success, kubelet reports pod status as Ready to the API server.
  • The Endpoints Controller detects the Ready pod and appends the Pod IP to the Service's Endpoints / EndpointSlices.
  • kube-proxy (or Cilium eBPF) on every node updates local iptables/IPVS rules, enabling load balancers and clients to route live traffic to the new pod!
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Kubernetes is a Desired State System. Running 'kubectl apply' does not start containers directly; it commits desired state into etcd via the API server. An asynchronous chain of independent control loops (Deployment Controller -> ReplicaSet Controller -> Scheduler -> Kubelet -> CRI -> CNI -> Endpoints Controller) works continuously to bring physical reality to match your declared state."
⚡ 60-Second Elevator Pitch Talking Points
  • 1. kubectl: Computes strategic 3-way merge patch and sends HTTP POST to kube-apiserver.
  • 2. API Server: Authenticates caller, checks RBAC, runs Mutating/Validating admission webhooks, and writes desired state to etcd via Raft consensus.
  • 3. Deployment & ReplicaSet Controllers: Detect the change via API watches and create unbound Pod definitions (spec.nodeName empty).
  • 4. kube-scheduler: Filters nodes (predicates) and scores nodes (priorities), then writes a Binding object assigning the pod to a node.
  • 5. Kubelet: Detects the assigned pod, instructs CRI (containerd) to create the pause container sandbox, and pulls the image.
  • 6. CNI: Configures pod network namespace, veth pair, and assigns the Pod IP.
  • 7. Service Ingress: Once readiness probes pass, Endpoints Controller adds Pod IP to Service Endpoints, and kube-proxy updates iptables/IPVS.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Over-granting cluster-admin permissions to pod ServiceAccounts instead of scoping namespace Roles when configuring Really Happens Hood Over-granting cluster-wide permissions violates least-privilege principles and allows compromised pods to escalate privileges across namespaces.
  2. Referencing mutable latest container tags across production Kubernetes Deployment specs when configuring Really Happens Hood Deployments referencing :latest risk inconsistent versions across replica pods and break deterministic rollback capabilities.
  3. Omitting pod readiness probes, causing traffic to hit uninitialized application containers when configuring Really Happens Hood Readiness probes gate traffic delivery; missing probes cause Kubernetes to route user requests to pods before dependencies or caches are loaded.
  4. Executing diagnostic triage without analyzing chronological cluster events and describe outputs when configuring Really Happens Hood Cluster events provide sequential timestamps of scheduler decisions, probe failures, and image pull timeouts essential for rapid root-cause isolation.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →