Q: What really happens under the hood when you run 'kubectl apply -f deployment.yaml'? Walk me through the entire journey from client CLI to running container.

Before any network request reaches the cluster, kubectl performs client-side inspection:

• Client-Side Validation: kubectl verifies YAML syntax and checks resource fields against the locally cached OpenAPI / Swagger schema from the cluster (~/.kube/cache/discovery).
• Three-Way Strategic Merge Patch: Reads the kubectl.kubernetes.io/last-applied-configuration annotation, compares it with the live cluster state and the new local YAML, and computes a JSON Strategic Merge Patch.
• HTTP REST Request: Formats the payload into JSON and dispatches an HTTP POST/PATCH request to the API server: POST /apis/apps/v1/namespaces/default/deployments with TLS client certificates or OIDC Bearer tokens.

The kube-apiserver is the single front door to the cluster. The request traverses 4 sequential filters:

• 1. Authentication: Validates TLS cert, ServiceAccount token, or Entra/AWS IAM OIDC identity to establish the caller's username and groups.
• 2. Authorization (RBAC): Evaluates ClusterRoles/RoleBindings to verify if the subject has create / patch permissions on deployments in the target namespace.
• 3. Mutating Admission Webhooks: Plugins and webhooks (e.g. Istio sidecar injector, Vault agent injector) modify the object or set default values.
• 4. Schema Validation: Enforces API schema rules, required fields, and immutability constraints.
• 5. Validating Admission Webhooks: Webhooks (e.g. Kyverno, OPA Gatekeeper) run final compliance checks (e.g. enforcing non-root execution or image registry whitelisting) and reject invalid manifests.

The single source of truth commits the change:

• Once admission passes, kube-apiserver serializes the Deployment object into protocol buffers and writes it into etcd at key /registry/deployments/default/my-app.
• etcd commits the write across a quorum of nodes using the Raft consensus algorithm.
• The API server returns an HTTP 201 Created or 200 OK response back to the client CLI.
• Important: At this exact second, NO containers or pods exist yet! Only the desired state is recorded.

The control loops take over asynchronously:

• Deployment Controller: Watches the API server for Deployment changes. Detects the new Deployment and creates a child ReplicaSet object with pod template hash.
• ReplicaSet Controller: Detects the new ReplicaSet desiring e.g. 3 replicas. Compares desired replicas (3) with existing replicas (0).
• It issues 3 API requests to create 3 Pod objects. Crucially, these Pod objects have no assigned node (spec.nodeName: '') and enter the Pending state.

Matching unbound pods to healthy worker nodes:

• Watch Loop: kube-scheduler continuously watches the API server for Pods where spec.nodeName == ''.
• Phase 1 (Filtering / Predicates): Eliminates ineligible nodes that lack sufficient CPU/memory requests, have untolerated taints, or fail nodeSelector / affinity constraints.
• Phase 2 (Scoring / Priorities): Scores remaining nodes based on image locality, topology spread, and resource fragmentation (least/most requested).
• Binding: Scheduler picks the highest-scoring node and sends a Binding API call to the API server, setting spec.nodeName: 'worker-node-2'.

The local node agent brings the container to life:

• Kubelet Watch: The kubelet daemon running on worker-node-2 observes that a Pod has been assigned to its node name.
• Container Runtime Interface (CRI): Kubelet calls containerd via gRPC.
• Image Pull: containerd checks if the image exists in local cache; if not, pulls it from registry using node IAM credentials or imagePullSecrets.
• Pod Sandbox Creation: Kubelet instructs containerd to create the Pod Sandbox (pause container) establishing Linux namespaces (IPC, UTS, PID, Network).

Plumbing the container into the cluster network:

• Kubelet invokes the CNI plugin (AWS VPC CNI, Calico, Flannel, Cilium).
• The CNI creates a virtual ethernet pair (veth), moves one interface into the pod's network namespace as eth0, and connects the other interface to the host bridge or routing table.
• Allocates a dedicated Pod IP from the subnet CIDR and configures routing and MTU.
• Once networking is ready, containerd starts the application containers inside the pod sandbox.

Connecting the running pod to live client traffic:

• Kubelet continuously executes configured startupProbe and readinessProbe.
• Once probes return HTTP 200 / Success, kubelet reports pod status as Ready to the API server.
• The Endpoints Controller detects the Ready pod and appends the Pod IP to the Service's Endpoints / EndpointSlices.
• kube-proxy (or Cilium eBPF) on every node updates local iptables/IPVS rules, enabling load balancers and clients to route live traffic to the new pod!