Q: What is the Kubernetes watch mechanism and how do informers use it?
The Kubernetes watch mechanism is a long-lived HTTP request that streams change events (ADDED, MODIFIED, DELETED) for resources starting from a given resourceVersion, and informers build a local cache and event handlers on top of.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: <strong>The Kubernetes watch mechanism is a long-lived HTTP request that streams change events (ADDED, MODIFIED, DELETED) for resources starting from a given <code>resourceVersion</code>, and informers build a local cache and event handlers on top of it.</strong> Controllers rely on this instead of polling, which keeps API server load low and reaction time fast. Interviewers ask because every controller, scheduler and operator is built on it, and failures (410 Gone, dropped watches, stale caches) explain odd cluster behaviour. A good answer walks through list-then-watch, the reflector, the store and work queue, and recovery behaviour.
Mental model: the watch protocol
A watch is a GET with watch=true that the server keeps open and answers with a stream of events.
- Event stream: the response is a chunked stream of JSON objects with a
type(ADDED, MODIFIED, DELETED, BOOKMARK, ERROR) and the object. - resourceVersion: an opaque version taken from etcd. Clients should treat it as opaque, pass it back to resume, and never compare numerically.
- List then watch: a client lists to get state plus the list's resourceVersion, then watches from that version so no change is missed.
- Server cache: the API server serves most watches from an in-memory watch cache, so many clients do not each hit etcd directly.
- Efficiency: one watch replaces thousands of list calls. Each event contains the full changed object, so clients rarely need an extra GET to know the new state.
Hands-on: the Kubernetes watch mechanism with kubectl and raw HTTP
You can observe the same stream controllers receive.
# Human view with event types
kubectl get pods -n prod -w --output-watch-events
# Raw stream
kubectl proxy --port=8001 &
RV=$(curl -s localhost:8001/api/v1/namespaces/prod/pods | jq -r .metadata.resourceVersion)
curl -N "localhost:8001/api/v1/namespaces/prod/pods?watch=true&resourceVersion=$RV&allowWatchBookmarks=true"
# {"type":"MODIFIED","object":{"kind":"Pod", ...}}
- kubectl:
kubectl get pods -wis list followed by watch. Add--output-watch-eventsto show event types. - Raw API: use
kubectl proxyand curl with?watch=trueto see JSON events. - Resume from a version: pass
resourceVersionfrom a previous list so you get only newer events. - Bookmarks:
allowWatchBookmarks=trueasks the server to send periodic BOOKMARK events so clients can keep an up-to-date resourceVersion cheaply. - Selectors: add
-l app=my-appor--field-selectortokubectl get -wto see how a scoped watch only delivers matching events, and note that a label change can appear as ADDED or DELETED for that selector.
How informers use watch
An informer, as in client-go, packages list-watch, caching and event dispatch so controllers do not reimplement it.
// Sketch with client-go (Go)
factory := informers.NewSharedInformerFactory(clientset, 10*time.Minute) // resync
podInformer := factory.Core().V1().Pods().Informer()
podInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: func(obj interface{}) { queue.Add(keyOf(obj)) },
UpdateFunc: func(_, obj interface{}) { queue.Add(keyOf(obj)) },
DeleteFunc: func(obj interface{}) { queue.Add(keyOf(obj)) },
})
factory.Start(stopCh)
cache.WaitForCacheSync(stopCh, podInformer.HasSynced)
- Reflector: performs the initial list and then the watch, pushing changes into a queue and re-listing when needed.
- DeltaFIFO and indexer: deltas update a thread-safe local store with optional indexes, so controllers read from memory (listers) rather than the API server.
- Event handlers:
OnAdd,OnUpdateandOnDeleteusually enqueue an object key to a rate-limited work queue, which deduplicates and retries. - Shared informers: one informer per resource type is shared by many controllers in a process, so there is a single watch connection per type.
- Work queue semantics: the queue stores keys, not objects, so ten updates to one pod collapse into one reconcile that reads the latest cached state. This is what makes controllers level-triggered rather than edge-triggered.
Failure modes and recovery
Watches break. Informers are designed to heal, but you need to understand the symptoms.
- 410 Gone: if the requested resourceVersion is too old (compacted from the watch history), the server returns 410 and the reflector must re-list, which costs a full list.
- Dropped connections: timeouts, load balancer idle limits and API server restarts close watches. The reflector re-establishes from its last version.
- Stale or slow consumers: a slow handler delays processing, so keep handlers fast and push heavy work to workers.
- Resync: a periodic resync replays cached objects to handlers and does not contact the API server, it helps repair missed reconciliation, not cache staleness.
- API server side: when many watchers exist, API Priority and Fairness and watch cache sizing decide who is slowed first, and very large clusters tune
--watch-cache-sizes. Check the documentation for your version.
Production practice for controller authors and operators
Watching efficiently is what keeps large clusters healthy.
- Scope the watch: use namespaces, label or field selectors so the controller caches only what it needs, reducing memory and API load.
- Avoid polling: never loop on list calls from many clients, since it multiplies etcd and API server work.
- Observe: monitor
apiserver_registered_watchers, workqueue depth and retries, and relist counts for controllers. - Set sensible resync: a long or zero resync period is usually fine, and API Priority and Fairness protects the server from noisy clients.
- Watch is a long-lived HTTP stream of change events from a resourceVersion, served mostly from the API server's watch cache.
- Informers wrap it: a reflector lists and watches, a DeltaFIFO updates a local indexed cache, and handlers enqueue keys to a work queue for reconciliation.
- I handle 410 Gone with a re-list, scope watches with selectors, and monitor watchers and queue depth.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Polling the API with repeated list calls It multiplies load on the API server and etcd, and is slower than a watch.
- Treating resourceVersion as a number to compare It is opaque, and ordering or arithmetic on it is unsupported.
- Doing heavy work inside informer event handlers It blocks event processing and makes the cache fall behind.
- Assuming lister reads are always current Caches are eventually consistent, so logic must tolerate stale data.