⚡ ~/naveed Interview Prep
⚡ Portfolio Home ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Security & DevSecOps Interview Questions Scenario 51 of 54 in Security & DevSecOps
Senior DevSecOps / SRE Security Secret Governance & Credential Leaks Security Breach Triage

Q: A production secret was accidentally committed to a GitHub repository and developers have already cloned the commit. What are your exact step-by-step actions?

Complete enterprise DevSecOps incident response runbook for a production credential leaked into a Git repository that has already been cloned.

#Security #Git #Secret Rotation #git-filter-repo #Credential Leak #Audit Logging
🎙️ Candidate Opening & Architectural Context
"The moment a secret is pushed to a remote Git repository—especially if already cloned—it must be treated as 100% compromised. Rewriting Git history or deleting the commit does NOT protect you because the plaintext credential is already stored on external workstations, GitHub internal caches, and potentially scrape bots. The golden rule is: Rotate First, Invalidate Second, Clean Git History Last."
Advertisement
🛡️
⚡ Recommended DevSecOps Track

Master cloud-native security and container runtime defense: Prepare with KodeKloud's CKS course with live browser-based terminal sandboxes.

Prepare with KodeKloud's CKS Course →

🛠️ Production Runbook & Step-by-Step Resolution

1

Step 1: Immediate Credential Revocation & Emergency Rotation

Do NOT waste time deleting the commit. Immediately log into the cloud/identity provider (AWS IAM, Database, Stripe, GitHub PAT) and revoke the compromised key or generate a replacement credential. Update production services with the new secret via Vault, AWS Secrets Manager, or Kubernetes Secrets.

# Example: Instantly deactivate compromised AWS IAM Access Key
aws iam update-access-key --access-key-id AKIAIOSFODNN7EXAMPLE --status Inactive
aws iam delete-access-key --access-key-id AKIAIOSFODNN7EXAMPLE
2

Step 2: Inspect CloudTrail & Audit Logs for Unauthorized Exploitation

Search security audit logs (AWS CloudTrail, database query logs, API access logs) from the timestamp the commit was pushed to the moment the key was revoked. Search for any unauthorized API calls or IP addresses originating outside corporate CIDR blocks.

# CloudTrail lookup for actions taken by the compromised key
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAIOSFODNN7EXAMPLE \
  --start-time $(date -u -v-24H +%Y-%m-%dT%H:%M:%SZ)
Advertisement
3

Step 3: Scrub Git Commit History and Force-Push

Use `git-filter-repo` (recommended over legacy BFG or filter-branch) to purge the secret from all commits, tags, and tree objects. Force-push the cleansed history and notify all developers to re-clone the repository to prevent pushing old commit history back.

# Purge the sensitive file or string from entire git history
pip install git-filter-repo
git filter-repo --replace-text <(echo 'regex:AKIA[0-9A-Z]{16}==>REDACTED')
git push origin --force --all
git push origin --force --tags
4

Step 4: Install Preventative Shift-Left Pre-Commit & CI Gates

Deploy automated pre-commit secret detection tools (`gitleaks`, `trufflehog`) across developer workstations and enforce GitHub Secret Scanning with Push Protection on all organization repositories so commits with raw keys are blocked at `git push` time.

Pro Tip: Security Law: Secret deletion in Git is cosmetic. The only real fix is immediate revocation and rotation at the identity provider.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Assume the credential is breached. Prioritize: 1) Revoke and rotate the secret at the provider, 2) Audit access logs for compromise, 3) Purge Git history with git-filter-repo, 4) Enable GitHub Push Protection."
⚡ 60-Second Elevator Pitch Talking Points
  • Treat the secret as compromised immediately; never waste time scrubbing Git before rotating.
  • Deactivate and rotate the credential in AWS/Vault/Database within 2 minutes.
  • Query CloudTrail and audit logs for the compromised Key ID to confirm whether external attackers executed API calls.
  • Scrub repository history using git-filter-repo and install Gitleaks push protection to permanently prevent future leaks.
Advertisement
Want more Security & DevSecOps scenarios?
Explore our complete collection of scenario-based Security & DevSecOps interview runbooks.
Browse All Security & DevSecOps Questions →