Q: How do you manage secrets securely in Kubernetes?
Enterprise standard operating model for Kubernetes secrets management: avoiding base64 Git commits, deploying External Secrets Operator (ESO) with cloud secret stores, and mounting secrets securely via tmpfs memory volumes.
#Kubernetes #Secrets #External Secrets Operator #AWS Secrets Manager #Azure Key Vault #HashiCorp Vault
🎙️ Candidate Opening & Architectural Context
"Kubernetes native Secret objects are only base64-encoded plain text—they are NOT encrypted by default. In enterprise production, we NEVER store secrets in Git or YAML files. We sync them dynamically using External Secrets Operator (ESO)."
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
The Native Secret Trap & Plaintext in Git
Why native Kubernetes secrets fail enterprise compliance:
- Base64 encoding is not encryption:
echo 'cGFzc3dvcmQ=' | base64 -dtakes 1 millisecond. - If developer commits a Secret manifest to Git, the secret is permanently recorded in Git history.
- In etcd, secrets are stored unencrypted unless EncryptionAtRest (using AWS KMS or Azure Key Vault KMS plugin) is explicitly enabled on the API server.
2️⃣
Production Standard: External Secrets Operator (ESO)
How External Secrets Operator bridges cloud vaults to Kubernetes:
Cloud Secret Store (AKV/AWS SM/Vault)→SecretStore CRD (IAM/Workload Identity)→ExternalSecret Manifest (Git Safe)→Kubernetes Secret (Auto-Generated)
- Source of Truth: Secrets are maintained and rotated inside AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault.
- SecretStore CRD: Configures authentication to the cloud vault using AWS IRSA or Azure Workload Identity (passwordless).
- ExternalSecret CRD: A safe Git-committable manifest that specifies which remote secret key to fetch and how often to refresh (e.g.
refreshInterval: 1h). - Auto-Reconciliation: ESO continuously syncs the remote secret into an in-cluster native Kubernetes Secret automatically.
3️⃣
Secure Pod Consumption: Volume Mounts vs Env Vars
How the pod should consume the secret securely:
- Avoid Plaintext Env Vars: Environment variables (
envFrom.secretRef) can leak into application error crash dumps, child process forks, and/proc/<pid>/environ. - Preferred Practice (Volume Mounts): Mount secrets as file volumes into
/etc/secrets. In Kubernetes, secret volumes are backed by tmpfs (RAM only), meaning they are never written to physical node disk storage.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Never store secrets in Git. Keep the source of truth in AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault. Use External Secrets Operator (ESO) with Workload Identity to sync them into in-memory Kubernetes Secrets, and mount them as tmpfs file volumes rather than environment variables."
⚡ 60-Second Elevator Pitch Talking Points
- Kubernetes Secrets are just base64 plain text; storing them in Git is a critical security vulnerability.
- Single Source of Truth: Store secrets in AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault with automated rotation.
- Syncing Engine: Deploy External Secrets Operator (ESO). The Git repo only contains ExternalSecret manifests pointing to secret paths.
- Authentication: ESO authenticates to cloud vaults via AWS IRSA or Azure Workload Identity (zero hardcoded cloud keys).
- In-Pod consumption: Mount secrets as tmpfs RAM-backed file volumes instead of environment variables to prevent leak in crash logs.
Advertisement