Q: A pipeline suddenly gets "403 Forbidden" while accessing Azure Key Vault. No YAML or secret changes were made. What would you investigate?
Investigation runbook for resolving unexpected HTTP 403 Forbidden errors when Azure DevOps Service Connections attempt to retrieve secrets from Azure Key Vault.
Master cloud-native security and container runtime defense: Prepare with KodeKloud's CKS course with live browser-based terminal sandboxes.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: A 403 Forbidden with zero pipeline changes points to authentication credential expiration, Key Vault network firewall IP restrictions, or Azure RBAC role assignment changes.
Check Service Principal Secret / Certificate Expiration
In Microsoft Entra ID (Azure AD), check the App Registration associated with the Azure DevOps Service Connection. Client secrets expire by default after 1-2 years.
az ad app credential list --id <app-id> --query "[].{EndDate:endDateTime, KeyId:keyId}"
Inspect Key Vault Networking Firewall Rules
Check if Azure Key Vault has 'Enabled from selected networks' enabled. If a teammate enabled the firewall or changed the virtual network rule, the pipeline agent's public IP will be blocked.
az keyvault show --name kv-production-eus --query "properties.networkAcls"
Verify Key Vault Permission Model (RBAC vs. Access Policy)
Check whether the Key Vault was migrated from Access Policies to Azure RBAC (Key Vault Secrets User). If permissions were granted via Access Policy but someone enabled Azure role-based access control, all previous policies are invalidated.
- Check Entra ID (Azure AD) App Registration client secret expiration date.
- Verify Key Vault networking firewall rules: ensure agent VNet or public IP is whitelisted.
- Check Key Vault permission model: confirm whether it uses Azure RBAC or Vault Access Policies.
- Verify Service Connection authorization in Azure DevOps Project Settings.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Revoking compromised credentials immediately without analyzing forensic audit trails for active exploitation when configuring Azure Pipeline Suddenly Deactivating a compromised access key prevents abuse while preserving forensic identity for CloudTrail investigations to assess blast radius.
- Terminating compromised instances immediately during incident triage without isolating volatile RAM state when configuring Azure Pipeline Suddenly Terminating compromised instances immediately destroys volatile memory evidence (RAM dumps) required to establish intrusion timelines.
-
Injecting plaintext secrets into container environment variables rather than fetching from dynamic secret vaults when configuring Azure Pipeline Suddenly
Container environment variables are visible via
docker inspector Kubernetes API queries, leaking sensitive API tokens to developers. - Failing to enforce automated credential rotation for third-party SaaS integration tokens when configuring Azure Pipeline Suddenly Static long-lived API tokens shared across internal services represent the highest-probability vector for supply-chain credential compromise.