Q: How do you manage secrets securely in Kubernetes?
Enterprise standard operating model for Kubernetes secrets management: avoiding base64 Git commits, deploying External Secrets Operator (ESO) with cloud secret stores, and mounting secrets securely via tmpfs memory volumes.
Master cloud-native security and container runtime defense: Prepare with KodeKloud's CKS course with live browser-based terminal sandboxes.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Kubernetes native Secret objects are only base64-encoded plain text—they are NOT encrypted by default. In enterprise production, we NEVER store secrets in Git or YAML files. We sync them dynamically using External Secrets Operator (ESO).
The Native Secret Trap & Plaintext in Git
Why native Kubernetes secrets fail enterprise compliance:
- Base64 encoding is not encryption:
echo 'cGFzc3dvcmQ=' | base64 -dtakes 1 millisecond. - If developer commits a Secret manifest to Git, the secret is permanently recorded in Git history.
- In etcd, secrets are stored unencrypted unless EncryptionAtRest (using AWS KMS or Azure Key Vault KMS plugin) is explicitly enabled on the API server.
Production Standard: External Secrets Operator (ESO)
How External Secrets Operator bridges cloud vaults to Kubernetes:
- Source of Truth: Secrets are maintained and rotated inside AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault.
- SecretStore CRD: Configures authentication to the cloud vault using AWS IRSA or Azure Workload Identity (passwordless).
- ExternalSecret CRD: A safe Git-committable manifest that specifies which remote secret key to fetch and how often to refresh (e.g.
refreshInterval: 1h). - Auto-Reconciliation: ESO continuously syncs the remote secret into an in-cluster native Kubernetes Secret automatically.
Secure Pod Consumption: Volume Mounts vs Env Vars
How the pod should consume the secret securely:
- Avoid Plaintext Env Vars: Environment variables (
envFrom.secretRef) can leak into application error crash dumps, child process forks, and/proc/<pid>/environ. - Preferred Practice (Volume Mounts): Mount secrets as file volumes into
/etc/secrets. In Kubernetes, secret volumes are backed by tmpfs (RAM only), meaning they are never written to physical node disk storage.
- Kubernetes Secrets are just base64 plain text; storing them in Git is a critical security vulnerability.
- Single Source of Truth: Store secrets in AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault with automated rotation.
- Syncing Engine: Deploy External Secrets Operator (ESO). The Git repo only contains ExternalSecret manifests pointing to secret paths.
- Authentication: ESO authenticates to cloud vaults via AWS IRSA or Azure Workload Identity (zero hardcoded cloud keys).
- In-Pod consumption: Mount secrets as tmpfs RAM-backed file volumes instead of environment variables to prevent leak in crash logs.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Revoking compromised credentials immediately without analyzing forensic audit trails for active exploitation when configuring Managing Secrets Securely Deactivating a compromised access key prevents abuse while preserving forensic identity for CloudTrail investigations to assess blast radius.
- Terminating compromised instances immediately during incident triage without isolating volatile RAM state when configuring Managing Secrets Securely Terminating compromised instances immediately destroys volatile memory evidence (RAM dumps) required to establish intrusion timelines.
-
Injecting plaintext secrets into container environment variables rather than fetching from dynamic secret vaults when configuring Managing Secrets Securely
Container environment variables are visible via
docker inspector Kubernetes API queries, leaking sensitive API tokens to developers. - Failing to enforce automated credential rotation for third-party SaaS integration tokens when configuring Managing Secrets Securely Static long-lived API tokens shared across internal services represent the highest-probability vector for supply-chain credential compromise.