Q: Explain Kubernetes ConfigMaps and Secrets: How do you manage them across environments?
Best practices for Kubernetes ConfigMaps and Secrets across Dev, Staging, and Production environments: base64 pitfalls, External Secrets Operator (ESO), HashiCorp Vault, Kustomize overlays, and dynamic reloads.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Configuration and secret management is one of the most critical aspects of Kubernetes operations. In an interview, I explain the architectural differences between Kubernetes ConfigMaps and Secrets, why storing base64 secrets in Git is an immediate security red flag, and detail how we implement multi-environment separation using Kustomize overlays and External Secrets Operator (ESO).
Kubernetes ConfigMaps and Secrets: Core Architectural Differences
ConfigMaps store non-sensitive configuration data (URLs, log levels, thread pools) in plaintext key-value pairs or files. Secrets store sensitive credentials (passwords, API tokens, TLS certificates) encoded in base64 and stored in etcd.
# Creating a native ConfigMap and Secret
kubectl create configmap app-config --from-literal=LOG_LEVEL=info
kubectl create secret generic app-secrets --from-literal=DB_PASSWORD=SuperSecret123- Base64 is NOT Encryption: Base64 is merely an encoding scheme. Any engineer with read access can decode secrets instantly with
echo.| base64 -d - etcd Encryption at Rest: Production clusters must enable
EncryptionConfigurationusing AWS KMS, GCP KMS, or Vault to encrypt secrets stored in etcd.
Injection Patterns: Environment Variables vs Volume Mounts
Configurations can be injected as container environment variables or mounted as filesystem volumes. Volume mounts provide superior security and operational flexibility.
spec:
containers:
- name: web
image: app:v1
volumeMounts:
- name: config-vol
mountPath: /etc/config
readOnly: true
volumes:
- name: config-vol
configMap:
name: app-config- Environment Variables (envFrom): Injected at container startup. Modifying the ConfigMap does NOT update the running container without a rolling restart.
- Volume Mounts (mountPath): Mounted via symlinks. The kubelet periodically syncs updates into the container filesystem without requiring pod restarts.
Multi-Environment Management with Kustomize Overlays
To maintain environment parity between Dev, Staging, and Production, we use Kustomize overlays. A common base configuration defines default parameters, while environment overlays inject specific configurations.
# Directory structure:
# base/
# deployment.yaml
# kustomization.yaml
# overlays/
# dev/
# config.env
# kustomization.yaml
# prod/
# config.env
# kustomization.yaml- DRY Architecture: Base manifests remain identical; only environment-specific config files differ.
- configMapGenerator: Kustomize automatically appends content hashes (e.g.
app-config-g8h7f2) to ConfigMap names, triggering automatic rolling updates when configurations change.
Enterprise GitOps Secret Management: External Secrets Operator (ESO)
Storing plaintext or base64 secrets in Git violates security compliance. We deploy the External Secrets Operator (ESO) to synchronize sensitive credentials from AWS Secrets Manager or HashiCorp Vault into native Kubernetes Secrets.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: database-credentials
namespace: production
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secretsmanager
kind: ClusterSecretStore
target:
name: db-secret
data:
- secretKey: password
remoteRef:
key: prod/aurora/db
property: password- Zero Secrets in Git: Developers commit only declarative
ExternalSecretmanifests referencing secret names. - Automated Rotation: ESO polls the cloud secret store at a configured
refreshIntervaland updates the native Kubernetes Secret automatically upon credential rotation.
Kubernetes ConfigMaps and Secrets Tooling Comparison Matrix
Architectural comparison of production secret management solutions for Kubernetes ConfigMaps and Secrets:
- ConfigMap vs Secret: ConfigMaps hold non-sensitive settings in plaintext; Secrets hold credentials in base64 and should be encrypted in etcd via KMS.
- Injection Methods: Prefer volume mounts over environment variables to avoid credential leakage in process dumps and enable live updates.
- Multi-Environment Parity: Use Kustomize overlays (dev/stage/prod) with configMapGenerator hash suffixes to guarantee seamless rolling restarts.
- GitOps Secret Synchronization: Use External Secrets Operator (ESO) with AWS Secrets Manager or HashiCorp Vault to keep secrets completely out of Git.
- Immutable & Reloader: Set immutable: true to prevent configuration drift and leverage Reloader annotations for automated zero-downtime rollouts.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Committing base64-encoded Kubernetes Secrets directly to Git repositories Base64 is reversible encoding, not encryption; anyone with repository read access can decode your database and API credentials.
- Relying on environment variables for credentials that rotate periodically Environment variables are evaluated once at container startup; rotated secrets will not take effect until pods are manually deleted.
- Neglecting etcd encryption at rest for Kubernetes Secrets Without KMS envelope encryption enabled, Kubernetes Secrets are stored in plaintext inside etcd, exposing secrets to anyone with etcd backup access.
- Using a single shared ConfigMap across all environments Coupling Dev and Prod configurations leads to disastrous accidents where production services connect to staging databases during rollouts.