Q: A pipeline suddenly gets "403 Forbidden" while accessing Azure Key Vault. No YAML or secret changes were made. What would you investigate?
Investigation runbook for resolving unexpected HTTP 403 Forbidden errors when Azure DevOps Service Connections attempt to retrieve secrets from Azure Key Vault.
🛠️ Production Runbook & Step-by-Step Resolution
Check Service Principal Secret / Certificate Expiration
In Microsoft Entra ID (Azure AD), check the App Registration associated with the Azure DevOps Service Connection. Client secrets expire by default after 1-2 years.
az ad app credential list --id <app-id> --query "[].{EndDate:endDateTime, KeyId:keyId}"Inspect Key Vault Networking Firewall Rules
Check if Azure Key Vault has 'Enabled from selected networks' enabled. If a teammate enabled the firewall or changed the virtual network rule, the pipeline agent's public IP will be blocked.
az keyvault show --name kv-production-eus --query "properties.networkAcls"Verify Key Vault Permission Model (RBAC vs. Access Policy)
Check whether the Key Vault was migrated from Access Policies to Azure RBAC (Key Vault Secrets User). If permissions were granted via Access Policy but someone enabled Azure role-based access control, all previous policies are invalidated.
- Check Entra ID (Azure AD) App Registration client secret expiration date.
- Verify Key Vault networking firewall rules: ensure agent VNet or public IP is whitelisted.
- Check Key Vault permission model: confirm whether it uses Azure RBAC or Vault Access Policies.
- Verify Service Connection authorization in Azure DevOps Project Settings.