⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 123 of 207 in Kubernetes
Senior DevOps / SRE [L2] Kubernetes Additional Kubernetes Scenarios (Q101-Q200) Production Scenario [L2]
🎯 Target Role / Context: Kubernetes Scenario #113 (L2) · Additional Kubernetes Scenarios (Q101-Q200)

Q: How do you implement health checks for a gRPC service in Kubernetes?

Use the gRPC Health Checking Protocol. Your service implements grpc.health.v1.Health/Check. In the probe:

#Kubernetes #L2 #Container Orchestration #K8s
🎙️ Candidate Opening & Architectural Context
"To implement <strong>health checks for a gRPC service in Kubernetes</strong>, make the server implement the standard <code>grpc.health.v1.Health</code> service and point a native <code>grpc</code> probe at it. The native probe is beta since 1.24 and GA since 1.27, so on current clusters you no longer need an exec probe with a helper binary. Interviewers use this question to check that you can separate liveness, readiness and startup semantics and avoid probes that cause restart storms."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: To implement <strong>health checks for a gRPC service in Kubernetes</strong>, make the server implement the standard <code>grpc.health.v1.Health</code> service and point a native <code>grpc</code> probe at it. The native probe is beta since 1.24 and GA since 1.27, so on current clusters you no longer need an exec probe with a helper binary. Interviewers use this question to check that you can separate liveness, readiness and startup semantics and avoid probes that cause restart storms.

1️⃣

Mental model: the gRPC health protocol and three probe types

Kubernetes asks one question per probe; your server needs to answer each honestly.

  • Protocol: the health service exposes Check(service) returning SERVING or NOT_SERVING. An empty service name means overall server health; named services let you report per-component status.
  • Readiness: should this pod receive traffic now? It can go NOT_SERVING during warm-up, dependency loss, or shutdown draining.
  • Liveness: is the process wedged beyond recovery? Keep it cheap and independent of downstream dependencies, or a database blip will restart every replica.
  • Startup: protects slow-starting servers by disabling liveness and readiness until it first succeeds.
  • Why not a plain TCP probe: a TCP check only proves the port accepts connections; it cannot tell that the gRPC handlers are blocked, that the server is draining, or that a dependency needed for readiness is down.
Pro Tip: A restart cannot fix a broken dependency. Keep dependency checks in readiness, not liveness.
2️⃣

Hands-on: gRPC health check in Kubernetes with a native probe

Declare the grpc handler on each probe; the kubelet speaks the health protocol directly.

containers:
  - name: app
    image: registry.example.com/my-grpc:1.0.0
    ports:
      - containerPort: 50051
    startupProbe:
      grpc:
        port: 50051
      failureThreshold: 30
      periodSeconds: 2
    readinessProbe:
      grpc:
        port: 50051
        service: my.pkg.Greeter
      periodSeconds: 5
      timeoutSeconds: 2
    livenessProbe:
      grpc:
        port: 50051
      periodSeconds: 10
      failureThreshold: 3
  • Fields: grpc.port is required and must be a number (named ports are not supported); grpc.service is optional and defaults to the empty string.
  • Plaintext only: the built-in probe does not support TLS or authentication. If the health port requires mTLS, expose a separate plaintext health port or use an exec fallback.
  • Timing: the probe fails on the first non-SERVING response or timeout; tune timeoutSeconds (default 1) and failureThreshold for realistic latency.
  • Different services: use a distinct service value for readiness than liveness if you register more than one health name.
  • Probe type per goal: use startup to absorb slow warm-up, readiness for traffic gating, and liveness only for deadlock detection, each with its own thresholds.
Advertisement
3️⃣

Server side: register the health service and flip status

The probe is only as good as the status your code reports.

import (
  "google.golang.org/grpc"
  "google.golang.org/grpc/health"
  healthpb "google.golang.org/grpc/health/grpc_health_v1"
)

s := grpc.NewServer()
hs := health.NewServer()
healthpb.RegisterHealthServer(s, hs)
hs.SetServingStatus("", healthpb.HealthCheckResponse_SERVING)
// on SIGTERM: hs.Shutdown(); then s.GracefulStop()
  • Register it: in Go, google.golang.org/grpc/health provides a ready-made server; call SetServingStatus for each service name.
  • Drain on shutdown: on SIGTERM, set status to NOT_SERVING first so readiness fails and endpoints are removed, then wait briefly and call graceful stop.
  • Reflection: optional, but helpful because grpcurl can then discover services without proto files.
  • Other languages: Java, Python, C++ and others ship equivalent health service implementations; use the official one rather than a custom RPC.
  • Per-service status: report NOT_SERVING for a named service while its dependency is unavailable, so only the pods that truly cannot serve are removed from rotation.
4️⃣

Verify and troubleshoot failing gRPC probes

Reproduce the probe yourself before changing thresholds.

kubectl -n prod port-forward pod/my-grpc-abc 50051:50051 &
grpcurl -plaintext -d '{"service":"my.pkg.Greeter"}' localhost:50051 grpc.health.v1.Health/Check
kubectl -n prod describe pod my-grpc-abc | grep -iE 'probe|unhealthy'
  • Call it manually: grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check through a port-forward. Expect {"status": "SERVING"}.
  • Read the events: kubectl describe pod <pod> shows messages such as Liveness probe failed with the gRPC status.
  • Unimplemented error: the server never registered the health service, or the service name in the probe is unknown.
  • Old clusters: before 1.24 (or if the feature is not available) use the grpc_health_probe binary in an exec probe, shipped in the image.
  • Reading the failure message: events show rpc error: code = Unimplemented or DeadlineExceeded; the first points to missing registration, the second to a slow handler or a too-small timeout.
5️⃣

Production practice: tuning and rollout safety

Probes shape rollouts, so tune them as part of release engineering.

  • Startup probe first: allow a generous failureThreshold times periodSeconds budget instead of a large initialDelaySeconds.
  • Rollouts: readiness gates the rolling update; a wrong probe stalls or silently blocks the deployment.
  • Load balancer probes: cloud load balancers have their own health checks and may need HTTP/2 or TCP checks rather than your Kubernetes probe.
  • Alert on probe failures: watch the restart counter and the not-ready pod count.
  • Mesh and proxy checks: if a sidecar intercepts traffic, confirm the probe path still reaches the app container, since some meshes rewrite probes.
  • Capacity during rollouts: set maxUnavailable and minReadySeconds so new pods prove they stay healthy before old ones are removed.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: use the native grpc probe on Kubernetes 1.27 or newer, keep liveness independent of dependencies, and flip the health status to NOT_SERVING on SIGTERM to drain traffic cleanly."
⚡ 60-Second Elevator Pitch Talking Points
  • I implement grpc.health.v1 in the server and use native grpc startup, readiness and liveness probes.
  • I verify with grpcurl through port-forward and kubectl describe, checking the service name and port.
  • I automate probe tuning in manifests, keep dependencies out of liveness, and drain via NOT_SERVING on shutdown.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Making liveness depend on the database or a downstream service A shared dependency outage then restarts every healthy pod and turns a partial failure into a full outage.
  2. Pointing the grpc probe at a named port or a TLS-only port The native probe needs a numeric port and does not support TLS or auth, so it fails permanently.
  3. Skipping the startup probe and using a huge initialDelaySeconds It delays readiness for every restart; a startup probe adapts to the real start time.
  4. Forgetting to register the health service in the server Probes return Unimplemented and the pod is killed or never becomes Ready.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →