Q: How do you handle secrets in Kubernetes? What are the problems with default Kubernetes Secrets?
Default Kubernetes Secrets problems:
#Kubernetes #Advanced Scenarios #L2 #Container Orchestration #K8s #etcd
🎙️ Candidate Opening & Architectural Context
""In our production Kubernetes clusters running microservices on EKS/AKS, this was a classic operational challenge. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix.""
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Initial Diagnostics & Root Cause Analysis
Default Kubernetes Secrets problems:
- They're only base64 encoded, not encrypted. Anyone with etcd access can read them.
- They're stored in etcd in plaintext by default (unless etcd encryption is enabled).
- RBAC can restrict who reads them, but it's easy to accidentally over-grant.
- Encrypt etcd at rest — enable
EncryptionConfigurationin the API server.
2️⃣
Remediation & Permanent Safeguards
Better approaches: Production recommendation: External Secrets Operator + AWS Secrets Manager or HashiCorp Vault.
- External secrets — use External Secrets Operator to sync secrets from AWS Secrets Manager, HashiCorp Vault, or GCP Secret Manager into Kubernetes Secrets.
- Vault Agent Injector — inject secrets directly into pods as files without storing in etcd at all.
- Sealed Secrets (Bitnami) — encrypt secrets before storing in Git. Safe to commit.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: They're only base64 encoded, not encrypted. Anyone with etcd access can read them.."
⚡ 60-Second Elevator Pitch Talking Points
- They're only base64 encoded, not encrypted. Anyone with etcd access can read them.
- They're stored in etcd in plaintext by default (unless etcd encryption is enabled).
- RBAC can restrict who reads them, but it's easy to accidentally over-grant.
Advertisement