⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All AWS & Cloud Architecture Interview Questions Scenario 145 of 179 in AWS & Cloud Architecture
Senior DevOps / SRE Azure & Cloud AKS Security & IAM Security Hardening

Q: Your AKS cluster relies on deprecated aad-pod-identity, which intercepts node metadata server requests via NMI iptables rules, causing intermittent 30-second token timeouts under high pod churn. How do you migrate to Azure AD Workload Identity with zero service downtime?

Step-by-step production migration from deprecated aad-pod-identity (NMI/MIC) to Azure AD / Entra ID Workload Identity using OIDC federation and Kubernetes Service Account annotations.

#Azure #AKS #Workload Identity #Entra ID #IAM #Key Vault
🎙️ Candidate Opening & Architectural Context
"aad-pod-identity modified node iptables to redirect metadata requests through Node Managed Identity (NMI) daemonsets, introducing severe latency and race conditions. We migrated our 600-pod cluster to Entra ID Workload Identity utilizing native OIDC federation."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: aad-pod-identity modified node iptables to redirect metadata requests through Node Managed Identity (NMI) daemonsets, introducing severe latency and race conditions. We migrated our 600-pod cluster to Entra ID Workload Identity utilizing native OIDC federation.

1️⃣

Enable OIDC Issuer and Workload Identity on AKS Cluster

Configure the cluster to act as a cryptographically verifiable OIDC token issuer:

  • Cluster Update: Executed az aks update --resource-group rg-prod --name aks-prod --enable-oidc-issuer --enable-workload-identity.
  • Retrieve OIDC URL: Extracted issuer URL via az aks show -n aks-prod -g rg-prod --query 'oidcIssuerProfile.issuerUrl' -otsv (e.g., https://eastus.oic.prod-aks.azure.com/...).
Pro Tip: Enabling the OIDC issuer allows Azure Entra ID to validate JSON Web Tokens (JWT) signed by the Kubernetes API server.
2️⃣

Create User-Assigned Managed Identity & Federated Credential

Establish cryptographic trust between Azure Entra ID and the Kubernetes Service Account:

  • Create Identity: Created identity az identity create --name payment-identity --resource-group rg-prod.
  • Federated Credential: Bound identity to K8s Service Account: az identity federated-credential create --name k8s-fed-cred --identity-name payment-identity --resource-group rg-prod --issuer $OIDC_URL --subject system:serviceaccount:payments:payment-sa --audience api://AzureADTokenExchange.
Pro Tip: The subject string must match exactly: system:serviceaccount:<NAMESPACE>:<KSA_NAME>.
Advertisement
3️⃣

Annotate Kubernetes ServiceAccount & Pod Specs

Inject projected service account tokens into running application pods:

  • Annotate SA: Added annotation azure.workload.identity/client-id: and label azure.workload.identity/use: 'true'.
  • Admission Webhook: The Azure Workload Identity mutating webhook automatically injects the federated token volume mount and environment variables (AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_FEDERATED_TOKEN_FILE).
Pro Tip: Application Azure SDKs (DefaultAzureCredential) automatically detect these environment variables and exchange the local K8s token for an Azure AD access token.
4️⃣

Validate Authentication & Safely Uninstall aad-pod-identity

Confirm pods successfully access Azure Key Vault and remove obsolete daemonsets:

  • Token Exchange Validation: Verified pod startup logs and confirmed successful access to Azure Key Vault secrets without node metadata calls.
  • Clean Helm Release: Uninstalled legacy daemonsets via helm uninstall aad-pod-identity -n kube-system and removed iptables redirection.
Pro Tip: Removing NMI daemonsets restores raw node metadata performance and removes all iptables interference on port 169.254.169.254.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Azure AD Workload Identity eliminates fragile iptables node metadata interception by using standard OIDC token federation between Kubernetes and Entra ID."
⚡ 60-Second Elevator Pitch Talking Points
  • Enable --enable-oidc-issuer and --enable-workload-identity on the AKS cluster.
  • Create a User-Assigned Managed Identity and establish an Entra ID federated credential.
  • Annotate the Kubernetes ServiceAccount with the Azure client ID for automatic token injection.
  • Decommission legacy aad-pod-identity NMI daemonsets to eliminate node metadata timeouts.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Engineering bespoke in-house developer tools instead of adopting standardized platform Golden Paths when configuring Eliminating Secrets Migrating Bespoke internal tools create long-term maintenance debt and fragmented workflows, whereas standardized Golden Paths accelerate delivery.
  2. Overlooking infrastructure cost visibility and FinOps allocation tagging across multi-tenant environments when configuring Eliminating Secrets Migrating Without granular resource tagging, engineering organizations cannot attribute cloud infrastructure costs to specific teams or business units.
  3. Failing to define explicit Service Level Objectives (SLOs) and Error Budgets to balance feature velocity with reliability when configuring Eliminating Secrets Migrating Without agreed SLOs, engineering and product teams lack an objective data-driven framework to prioritize technical debt over new features.
  4. Neglecting automated continuous drift detection between declarative configuration and deployed cloud state when configuring Eliminating Secrets Migrating Unmanaged drift accumulates over time, causing future automated deployments to fail or overwrite emergency manual adjustments.
Advertisement
Want more AWS & Cloud Architecture scenarios?
Explore our complete collection of scenario-based AWS & Cloud Architecture interview runbooks.
Browse All AWS & Cloud Architecture Questions →