⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 56 of 207 in Kubernetes
Senior DevOps / SRE [L2] Kubernetes Advanced Scenarios Production Scenario [L2]
🎯 Target Role / Context: Kubernetes Scenario #44 (L2) · Advanced Scenarios

Q: How do you handle configuration that differs between environments (dev, staging, prod) in Kubernetes?

Two main approaches:

#Kubernetes #Advanced Scenarios #L2 #Container Orchestration #K8s #Ingress
🎙️ Candidate Opening & Architectural Context
"Handle <strong>configuration that differs between environments</strong> by building one immutable image and injecting everything environment-specific from outside it: ConfigMaps, Secrets, and per-environment overlays (Kustomize) or values files (Helm), all stored in Git and applied by a GitOps controller. Interviewers ask this because the wrong answer, rebuilding images per environment or copying whole manifests, causes drift and untested production artifacts. A strong answer separates non-secret config from secrets and explains how config changes trigger pod restarts."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: Handle <strong>configuration that differs between environments</strong> by building one immutable image and injecting everything environment-specific from outside it: ConfigMaps, Secrets, and per-environment overlays (Kustomize) or values files (Helm), all stored in Git and applied by a GitOps controller. Interviewers ask this because the wrong answer, rebuilding images per environment or copying whole manifests, causes drift and untested production artifacts. A strong answer separates non-secret config from secrets and explains how config changes trigger pod restarts.

1️⃣

Principles: Build Once, Configure Per Environment

Start from the rule that the artifact tested in staging is byte-for-byte the artifact that runs in prod.

  • Image: identical across dev, staging and prod, referenced by tag or preferably digest. Environment names must never be baked into the image.
  • Non-secret config: ConfigMaps consumed as environment variables or mounted files, differing in values such as log level, feature flags, replica counts and hostnames. Keep the ConfigMap for values the application can safely log, and put anything that grants access, such as tokens and passwords, in a Secret so RBAC can restrict who may read it.
  • Secrets: Kubernetes Secrets are only base64 encoded, so do not commit them as plain manifests. Use an external secret store synced in, or encrypt them in Git.
  • Structure: keep a shared base and thin per-environment differences, so a change to a common field is made once and reviewed once.
  • Twelve-factor view: store config in the environment, not the code, which is what lets one build serve dev, staging and prod without modification.
Pro Tip: Cluster per environment (strong isolation) versus namespace per environment (cheaper, weaker isolation) is a separate decision. The config pattern works for both, but prod should at least be a separate cluster or account for blast radius reasons.
2️⃣

Hands-on: Configuration That Differs Between Environments with Kustomize

Kustomize patches a common base without templates, which keeps the diff between environments small and readable.

# overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
namespace: prod
replicas:
  - name: my-app
    count: 4
images:
  - name: registry.example.com/my-app
    newTag: "1.4.2"
configMapGenerator:
  - name: my-app-config
    behavior: merge
    literals:
      - LOG_LEVEL=warn
      - FEATURE_NEW_CHECKOUT=false

# Render and diff
kubectl kustomize overlays/prod | head -40
kubectl diff -k overlays/prod
kubectl apply -k overlays/prod
  • Layout: base/ holds the Deployment, Service and a configMapGenerator; overlays/dev, overlays/staging, overlays/prod each hold one kustomization.yaml. Keep overlays small: if an overlay patches more than a handful of fields, the base probably has the wrong abstraction.
  • Generator behaviour: configMapGenerator appends a content hash to the ConfigMap name, so changing a value renames the ConfigMap and automatically triggers a rollout of pods that reference it.
  • Preview before applying: kubectl kustomize overlays/prod renders the final YAML, and kubectl diff -k overlays/prod shows what would change in the cluster.
Advertisement
3️⃣

Helm Values Files and Secret Handling

If the team ships a Helm chart, the same idea is expressed with layered values files.

# values-prod.yaml
replicaCount: 4
image:
  tag: "1.4.2"
env:
  LOG_LEVEL: warn
ingress:
  host: app.example.com

helm template my-app ./chart -n prod -f values.yaml -f values-prod.yaml | kubectl apply --dry-run=server -f -
helm upgrade --install my-app ./chart -n prod -f values.yaml -f values-prod.yaml --atomic --timeout 5m
  • Layering: helm upgrade --install my-app ./chart -n prod -f values.yaml -f values-prod.yaml. Later files override earlier ones, so values.yaml holds safe defaults.
  • Render check: helm template my-app ./chart -f values.yaml -f values-prod.yaml and helm lint catch mistakes before they reach a cluster.
  • Secrets option A: External Secrets Operator reads from AWS Secrets Manager or Vault and creates the Kubernetes Secret, so Git only holds a reference.
  • Secrets option B: SOPS or Sealed Secrets lets you commit encrypted values and decrypt in-cluster or in the GitOps tool.
4️⃣

How Config Changes Reach Running Pods

A ConfigMap update does not always restart anything, which surprises people in production.

  • Environment variables: read once at container start. Editing a ConfigMap does not change them until the pod is recreated.
  • Mounted files: the kubelet refreshes ConfigMap volumes eventually, but not for subPath mounts, and the application must re-read the file.
  • Hash-suffixed names: the Kustomize generator approach, or a Helm checksum annotation on the pod template, forces a rolling restart on every config change.
  • Immutable ConfigMaps: setting immutable: true prevents accidental in-place edits and reduces API server watch load.
  • Checksum annotation in Helm: add checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }} to the pod template annotations so the Deployment rolls whenever the rendered ConfigMap changes.
5️⃣

Verification, Drift Control and GitOps

The goal is that what Git says is what runs.

  • GitOps controller: Argo CD or Flux apply each environment directory and report drift. Manual kubectl edit in prod gets reverted or flagged.
  • Promotion: promote by changing the image tag in the next overlay via pull request, with CI running kubectl kustomize and a schema validator such as kubeconform.
  • Runtime check: kubectl get cm my-app-config -n prod -o yaml and kubectl exec deploy/my-app -n prod -- env | grep LOG_LEVEL confirm the value the container actually sees.
  • Policy checks: run kubeconform and a policy engine such as Kyverno or conftest in CI so a prod overlay cannot, for example, drop resource limits or enable debug flags by accident.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Let Kustomize configMapGenerator or a Helm checksum annotation rename or re-hash config so every config change triggers a rollout, instead of relying on pods to notice edits."
⚡ 60-Second Elevator Pitch Talking Points
  • One image everywhere, with a shared base and thin per-environment overlays or values files in Git.
  • Secrets come from an external store or are encrypted in Git, never plain base64 in a repo.
  • I render and diff in CI, apply via Argo CD or Flux, and use hashed ConfigMap names so config changes roll pods.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Rebuilding the image per environment The artifact promoted to prod is then not the one that was tested, and bugs hide in the differences.
  2. Committing Secret manifests with base64 values Base64 is encoding, not encryption, so anyone with repo read access gets production credentials.
  3. Copy-pasting whole manifests per environment Copies drift apart over time, and a fix applied to one environment is forgotten in the others.
  4. Assuming a ConfigMap edit restarts pods Env vars are read at start, so the cluster keeps running old values until something recreates the pods.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →