⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All AWS & Cloud Architecture Interview Questions Scenario 126 of 179 in AWS & Cloud Architecture
Senior DevOps / SRE GCP & Cloud GKE Security & IAM Security Hardening
🎯 Target Role / Context: Senior Cloud Security & SRE Classic Question

Q: Your security team discovers that engineers have committed GCP service account JSON private keys into git repositories. Walk me through how you completely eliminate JSON keys in GKE using Workload Identity Federation without causing downtime for running microservices.

Step-by-step production runbook for eliminating compromised JSON service account keys in Google Kubernetes Engine (GKE) by binding Kubernetes Service Accounts directly to Google IAM Service Accounts with Workload Identity.

#GCP #GKE #Workload Identity #IAM #Security #Kubernetes
🎙️ Candidate Opening & Architectural Context
"In our enterprise banking cluster running on GKE, an audit revealed that 40+ microservices relied on mounted service account JSON keys. We needed to transition all workloads to Workload Identity with zero service interruptions."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: In our enterprise banking cluster running on GKE, an audit revealed that 40+ microservices relied on mounted service account JSON keys. We needed to transition all workloads to Workload Identity with zero service interruptions.

1️⃣

Enable Workload Identity on GKE Cluster & Node Pools

Ensure cluster and node pools have GKE Workload Identity enabled via Google Cloud CLI:

  • Cluster Update: Executed gcloud container clusters update prod-cluster --workload-pool=PROJECT_ID.svc.id.goog.
  • Node Pool Metadata: Updated node pools with --workload-metadata=GKE_METADATA to enforce the GKE metadata server.
Pro Tip: Enabling Workload Identity on node pools requires a rolling recreation or metadata update. Always test node drainage behavior.
2️⃣

Create IAM Bindings between K8s SA and Google Service Account

Establish trust relationship using roles/iam.workloadIdentityUser:

  • IAM Policy Binding: Bound the K8s service account in the cluster namespace to the Google Service Account (GSA).
  • Command: gcloud iam service-accounts add-iam-policy-binding GSA_NAME@PROJECT.iam.gserviceaccount.com --role roles/iam.workloadIdentityUser --member 'serviceAccount:PROJECT.svc.id.goog[default/app-ksa]'.
Pro Tip: The format of the member string must strictly follow serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME].
Advertisement
3️⃣

Annotate K8s ServiceAccount & Perform Rolling Pod Restart

Connect the Kubernetes SA to the Google Service Account via annotation:

  • Annotate Manifest: kubectl annotate serviceaccount app-ksa iam.gke.io/gcp-service-account=GSA_NAME@PROJECT.iam.gserviceaccount.com.
  • Remove Volume Mounts: Removed GOOGLE_APPLICATION_CREDENTIALS environment variables and secret volume mounts from deployment specs.
  • Rolling Rollout: Executed kubectl rollout restart deployment/app-deployment.
Pro Tip: Google client libraries (Cloud Storage, BigQuery, PubSub) automatically detect the GKE Metadata Server and Authenticate directly.
4️⃣

Revoke Legacy Keys & Enforce Org Policy Constraint

Permanently prevent future service account key creation across the GCP Organization:

  • Key Deletion: Audited and permanently deleted all static JSON keys via gcloud iam service-accounts keys delete.
  • Org Policy: Enforced constraints/iam.disableServiceAccountKeyCreation organization-wide to prevent regression.
Pro Tip: Enforcing the org policy ensures no engineer or third-party tool can ever generate a downloadable JSON key again.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"GKE Workload Identity eliminates the single largest credential leak attack vector in GCP by replacing static keys with short-lived OAuth 2.0 access tokens generated automatically by the GKE metadata server."
⚡ 60-Second Elevator Pitch Talking Points
  • Enabled Workload Identity on the GKE cluster and node pools using the project workload pool.
  • Bound Kubernetes Service Accounts to Google Service Accounts using roles/iam.workloadIdentityUser.
  • Annotated K8s service accounts and removed volume mounts and GOOGLE_APPLICATION_CREDENTIALS env vars.
  • Enforced constraints/iam.disableServiceAccountKeyCreation org policy to permanently block static keys.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Engineering bespoke in-house developer tools instead of adopting standardized platform Golden Paths when configuring Migrating Long Lived Bespoke internal tools create long-term maintenance debt and fragmented workflows, whereas standardized Golden Paths accelerate delivery.
  2. Overlooking infrastructure cost visibility and FinOps allocation tagging across multi-tenant environments when configuring Migrating Long Lived Without granular resource tagging, engineering organizations cannot attribute cloud infrastructure costs to specific teams or business units.
  3. Failing to define explicit Service Level Objectives (SLOs) and Error Budgets to balance feature velocity with reliability when configuring Migrating Long Lived Without agreed SLOs, engineering and product teams lack an objective data-driven framework to prioritize technical debt over new features.
  4. Neglecting automated continuous drift detection between declarative configuration and deployed cloud state when configuring Migrating Long Lived Unmanaged drift accumulates over time, causing future automated deployments to fail or overwrite emergency manual adjustments.
Advertisement
Want more AWS & Cloud Architecture scenarios?
Explore our complete collection of scenario-based AWS & Cloud Architecture interview runbooks.
Browse All AWS & Cloud Architecture Questions →