Q: Your security team discovers that engineers have committed GCP service account JSON private keys into git repositories. Walk me through how you completely eliminate JSON keys in GKE using Workload Identity Federation without causing downtime for running microservices.
Step-by-step production runbook for eliminating compromised JSON service account keys in Google Kubernetes Engine (GKE) by binding Kubernetes Service Accounts directly to Google IAM Service Accounts with Workload Identity.
#GCP #GKE #Workload Identity #IAM #Security #Kubernetes
🎙️ Candidate Opening & Architectural Context
"In our enterprise banking cluster running on GKE, an audit revealed that 40+ microservices relied on mounted service account JSON keys. We needed to transition all workloads to Workload Identity with zero service interruptions."
Advertisement
⚡ Recommended Practice Lab
Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Enable Workload Identity on GKE Cluster & Node Pools
Ensure cluster and node pools have GKE Workload Identity enabled via Google Cloud CLI:
- Cluster Update: Executed
gcloud container clusters update prod-cluster --workload-pool=PROJECT_ID.svc.id.goog. - Node Pool Metadata: Updated node pools with
--workload-metadata=GKE_METADATAto enforce the GKE metadata server.
Pro Tip: Enabling Workload Identity on node pools requires a rolling recreation or metadata update. Always test node drainage behavior.
2️⃣
Create IAM Bindings between K8s SA and Google Service Account
Establish trust relationship using roles/iam.workloadIdentityUser:
- IAM Policy Binding: Bound the K8s service account in the cluster namespace to the Google Service Account (GSA).
- Command:
gcloud iam service-accounts add-iam-policy-binding GSA_NAME@PROJECT.iam.gserviceaccount.com --role roles/iam.workloadIdentityUser --member 'serviceAccount:PROJECT.svc.id.goog[default/app-ksa]'.
Pro Tip: The format of the member string must strictly follow serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME].
3️⃣
Annotate K8s ServiceAccount & Perform Rolling Pod Restart
Connect the Kubernetes SA to the Google Service Account via annotation:
- Annotate Manifest:
kubectl annotate serviceaccount app-ksa iam.gke.io/gcp-service-account=GSA_NAME@PROJECT.iam.gserviceaccount.com. - Remove Volume Mounts: Removed
GOOGLE_APPLICATION_CREDENTIALSenvironment variables and secret volume mounts from deployment specs. - Rolling Rollout: Executed
kubectl rollout restart deployment/app-deployment.
Pro Tip: Google client libraries (Cloud Storage, BigQuery, PubSub) automatically detect the GKE Metadata Server and authenticate seamlessly.
4️⃣
Revoke Legacy Keys & Enforce Org Policy Constraint
Permanently prevent future service account key creation across the GCP Organization:
- Key Deletion: Audited and permanently deleted all static JSON keys via
gcloud iam service-accounts keys delete. - Org Policy: Enforced
constraints/iam.disableServiceAccountKeyCreationorganization-wide to prevent regression.
Pro Tip: Enforcing the org policy ensures no engineer or third-party tool can ever generate a downloadable JSON key again.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"GKE Workload Identity eliminates the single largest credential leak attack vector in GCP by replacing static keys with short-lived OAuth 2.0 access tokens generated automatically by the GKE metadata server."
⚡ 60-Second Elevator Pitch Talking Points
- Enabled Workload Identity on the GKE cluster and node pools using the project workload pool.
- Bound Kubernetes Service Accounts to Google Service Accounts using roles/iam.workloadIdentityUser.
- Annotated K8s service accounts and removed volume mounts and GOOGLE_APPLICATION_CREDENTIALS env vars.
- Enforced constraints/iam.disableServiceAccountKeyCreation org policy to permanently block static keys.
Advertisement