Q: A production secret was accidentally committed to a GitHub repository and developers have already cloned the commit. What are your exact step-by-step actions?
Complete enterprise DevSecOps incident response runbook for a production credential leaked into a Git repository that has already been cloned.
Master cloud-native security and container runtime defense: Prepare with KodeKloud's CKS course with live browser-based terminal sandboxes.
🛠️ Production Runbook & Step-by-Step Resolution
Step 1: Immediate Credential Revocation & Emergency Rotation
Do NOT waste time deleting the commit. Immediately log into the cloud/identity provider (AWS IAM, Database, Stripe, GitHub PAT) and revoke the compromised key or generate a replacement credential. Update production services with the new secret via Vault, AWS Secrets Manager, or Kubernetes Secrets.
# Example: Instantly deactivate compromised AWS IAM Access Key
aws iam update-access-key --access-key-id AKIAIOSFODNN7EXAMPLE --status Inactive
aws iam delete-access-key --access-key-id AKIAIOSFODNN7EXAMPLE
Step 2: Inspect CloudTrail & Audit Logs for Unauthorized Exploitation
Search security audit logs (AWS CloudTrail, database query logs, API access logs) from the timestamp the commit was pushed to the moment the key was revoked. Search for any unauthorized API calls or IP addresses originating outside corporate CIDR blocks.
# CloudTrail lookup for actions taken by the compromised key
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAIOSFODNN7EXAMPLE \
--start-time $(date -u -v-24H +%Y-%m-%dT%H:%M:%SZ)
Step 3: Scrub Git Commit History and Force-Push
Use `git-filter-repo` (recommended over legacy BFG or filter-branch) to purge the secret from all commits, tags, and tree objects. Force-push the cleansed history and notify all developers to re-clone the repository to prevent pushing old commit history back.
# Purge the sensitive file or string from entire git history
pip install git-filter-repo
git filter-repo --replace-text <(echo 'regex:AKIA[0-9A-Z]{16}==>REDACTED')
git push origin --force --all
git push origin --force --tags
Step 4: Install Preventative Shift-Left Pre-Commit & CI Gates
Deploy automated pre-commit secret detection tools (`gitleaks`, `trufflehog`) across developer workstations and enforce GitHub Secret Scanning with Push Protection on all organization repositories so commits with raw keys are blocked at `git push` time.
- Treat the secret as compromised immediately; never waste time scrubbing Git before rotating.
- Deactivate and rotate the credential in AWS/Vault/Database within 2 minutes.
- Query CloudTrail and audit logs for the compromised Key ID to confirm whether external attackers executed API calls.
- Scrub repository history using git-filter-repo and install Gitleaks push protection to permanently prevent future leaks.