Q: Your AKS cluster relies on deprecated aad-pod-identity, which intercepts node metadata server requests via NMI iptables rules, causing intermittent 30-second token timeouts under high pod churn. How do you migrate to Azure AD Workload Identity with zero service downtime?
Step-by-step production migration from deprecated aad-pod-identity (NMI/MIC) to Azure AD / Entra ID Workload Identity using OIDC federation and Kubernetes Service Account annotations.
#Azure #AKS #Workload Identity #Entra ID #IAM #Key Vault
🎙️ Candidate Opening & Architectural Context
"aad-pod-identity modified node iptables to redirect metadata requests through Node Managed Identity (NMI) daemonsets, introducing severe latency and race conditions. We migrated our 600-pod cluster to Entra ID Workload Identity utilizing native OIDC federation."
Advertisement
⚡ Recommended Practice Lab
Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Enable OIDC Issuer and Workload Identity on AKS Cluster
Configure the cluster to act as a cryptographically verifiable OIDC token issuer:
- Cluster Update: Executed
az aks update --resource-group rg-prod --name aks-prod --enable-oidc-issuer --enable-workload-identity. - Retrieve OIDC URL: Extracted issuer URL via
az aks show -n aks-prod -g rg-prod --query 'oidcIssuerProfile.issuerUrl' -otsv(e.g.,https://eastus.oic.prod-aks.azure.com/...).
Pro Tip: Enabling the OIDC issuer allows Azure Entra ID to validate JSON Web Tokens (JWT) signed by the Kubernetes API server.
2️⃣
Create User-Assigned Managed Identity & Federated Credential
Establish cryptographic trust between Azure Entra ID and the Kubernetes Service Account:
- Create Identity: Created identity
az identity create --name payment-identity --resource-group rg-prod. - Federated Credential: Bound identity to K8s Service Account:
az identity federated-credential create --name k8s-fed-cred --identity-name payment-identity --resource-group rg-prod --issuer $OIDC_URL --subject system:serviceaccount:payments:payment-sa --audience api://AzureADTokenExchange.
Pro Tip: The subject string must match exactly: system:serviceaccount:<NAMESPACE>:<KSA_NAME>.
3️⃣
Annotate Kubernetes ServiceAccount & Pod Specs
Inject projected service account tokens into running application pods:
- Annotate SA: Added annotation
azure.workload.identity/client-id:and labelazure.workload.identity/use: 'true'. - Admission Webhook: The Azure Workload Identity mutating webhook automatically injects the federated token volume mount and environment variables (
AZURE_CLIENT_ID,AZURE_TENANT_ID,AZURE_FEDERATED_TOKEN_FILE).
Pro Tip: Application Azure SDKs (DefaultAzureCredential) automatically detect these environment variables and exchange the local K8s token for an Azure AD access token.
4️⃣
Validate Authentication & Safely Uninstall aad-pod-identity
Confirm pods successfully access Azure Key Vault and remove obsolete daemonsets:
- Token Exchange Validation: Verified pod startup logs and confirmed successful access to Azure Key Vault secrets without node metadata calls.
- Clean Helm Release: Uninstalled legacy daemonsets via
helm uninstall aad-pod-identity -n kube-systemand removed iptables redirection.
Pro Tip: Removing NMI daemonsets restores raw node metadata performance and removes all iptables interference on port 169.254.169.254.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Azure AD Workload Identity eliminates fragile iptables node metadata interception by using standard OIDC token federation between Kubernetes and Entra ID."
⚡ 60-Second Elevator Pitch Talking Points
- Enable --enable-oidc-issuer and --enable-workload-identity on the AKS cluster.
- Create a User-Assigned Managed Identity and establish an Entra ID federated credential.
- Annotate the Kubernetes ServiceAccount with the Azure client ID for automatic token injection.
- Decommission legacy aad-pod-identity NMI daemonsets to eliminate node metadata timeouts.
Advertisement