Senior DevOps / SRE CI/CD Security & Compliance DevSecOps

Q: How would you securely manage secrets?

Enterprise secrets management architecture: eliminating secrets from Git and CI logs, integrating HashiCorp Vault / AWS Secrets Manager with Kubernetes, and using SOPS/External Secrets Operator.

#Security #HashiCorp Vault #AWS Secrets Manager #SOPS #External Secrets #CI/CD

🎙️ Candidate Opening & Architectural Context

"Secure secrets management requires defense-in-depth across three stages: in Git repositories (at rest), during CI/CD execution (in transit), and inside production runtime environments."

🛠️ Production Runbook & Step-by-Step Resolution

1️⃣

Pillar 1: Never Commit Plaintext Secrets to Git

Enforce strict pre-commit and repository-level prevention:

Pre-commit Hooks & Secret Scanning: Run git-secrets, TruffleHog, or GitHub Secret Scanning to block commits containing API keys, private certs, or AWS tokens.
Encrypted In-Repo Secrets (GitOps): If secrets must be stored in Git for GitOps, use Mozilla SOPS (encrypted with AWS KMS / GCP KMS) or Bitnami Sealed Secrets (asymmetric public key encryption).

2️⃣

Pillar 2: Centralized Secrets Store of Record

Use dedicated secret management services:

Store canonical secrets in AWS Secrets Manager, AWS SSM Parameter Store, or HashiCorp Vault.
Enable automated rotation for database passwords and API tokens.
Enforce IAM least privilege and audit every secret read with AWS CloudTrail / Vault audit logs.

3️⃣

Pillar 3: Runtime Injection into Kubernetes

How applications consume secrets securely in production:

External Secrets Operator (ESO): Best practice — synchronizes secrets from AWS Secrets Manager / Vault into native Kubernetes Secrets declaratively.
Secrets Store CSI Driver: Mounts secrets directly from AWS Secrets Manager into container memory volumes without creating native K8s Secret objects.
Cluster Security: Enable etcd encryption-at-rest using KMS provider to ensure raw secrets aren't stored unencrypted on disk.

4️⃣

Pillar 4: CI/CD Pipeline Hygiene

Protecting secrets during automated builds:

Use OIDC (OpenID Connect): GitHub Actions / GitLab CI authenticates to AWS via IAM roles without long-lived static AWS access keys!
Mask all environment variables in pipeline logs.
Scoped permissions: Pipeline runners should only have access to secrets needed for build/deploy, never admin credentials.

💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)

"Store secrets in AWS Secrets Manager / Vault, inject into K8s via External Secrets Operator, encrypt in Git using SOPS, use OIDC for CI/CD instead of static keys, and encrypt etcd at rest."

⚡ 60-Second Elevator Pitch Talking Points

Repository level: Block secrets with TruffleHog / pre-commit hooks; use SOPS/KMS if storing in GitOps repos.
Storage of record: AWS Secrets Manager / HashiCorp Vault with automated rotation and CloudTrail audit logging.
Kubernetes integration: Use External Secrets Operator (ESO) or Secrets Store CSI driver; enable etcd encryption-at-rest.
CI/CD pipeline: Eliminate static AWS keys using OIDC federated IAM roles; mask all secrets in logs.
Access control: Enforce strict RBAC on K8s secrets; disable automountServiceAccountToken where not needed.