DevOps Situational Interview Questions: How to Ace Real-World Production Scenarios

❌ The Weak Candidate Signal

"I'd SSH into the server, check the application logs, find the error stack trace, fix the code bug, and deploy a hotfix."

Why it fails: Assumes single-node access, ignores user blast radius, attempts code debugging during active customer pain, and neglects rollback safety.

✅ The Staff SRE Signal

"First, I scope blast radius: is this all checkouts or one payment gateway? Next, I check deploy correlation: did errors start precisely at release time? If the database schema is additive, I roll back the app immediately to restore revenue while debugging offline. I broadcast an incident update in Slack within 3 minutes."

The Friday 16:45 Deploy Spiking 5xx Errors with Database Migrations

• Scope: Is checkout completely down or failing on specific payment methods? Does the error graph align with the exact pod deployment timestamp?
• First Check: Check whether the database migration was purely additive (e.g., adding a nullable column) or destructive (renaming/dropping columns).
• The Decision: If additive, immediately trigger kubectl rollout undo deployment/checkout-service. The older pods will happily run against the new schema. If the migration was destructive, rolling back will crash older pods too—so toggle the feature flag or fix forward.
• Risk Accepted: We lose the new features until Monday, but we restore revenue immediately.

"🚨 INCIDENT UPDATE [SEV-2]: Checkout 5xx rate spiked to 3.2% following v2.31 deploy. Migration verified additive. Initiating application rollback to v2.30 now. Next update at 17:05."

Primary PostgreSQL Database Disk at 95% Growing 1% Every 10 Minutes

• Scope: At 1% per 10 minutes, we have ~50 minutes before total write failure and database crash.
• Immediate Safe Move: Trigger an online storage volume increase (e.g., in AWS RDS or EBS volume modification) immediately. Storage expansion takes 15–30 minutes, which beats the countdown.
• First Triage Query: Check whether an abandoned replication slot is retaining WAL archives:

  SELECT 
    slot_name, 
    active, 
    pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_bytes 
  FROM pg_replication_slots;

• The Trade-Off: If an inactive CDC slot holds 80GB of WAL and storage cannot expand in time, dropping the slot saves the primary database from crashing, but breaks downstream sync. Say this trade-off out loud!

Destructive Terraform Plan Showing Replacement of Production RDS

• Immediate Stance: Refuse approval. Replacement means terminating the live production instance, resulting in catastrophic downtime.
• First Check: Search the plan for the exact line marked # forces replacement. Typical triggers include changing KMS keys, moving subnets, or renaming resources within Terraform modules.
• The Fix: If a resource was moved into a module, use moved { from = ... to = ... } or terraform state mv to update state mapping without recreation.
• Systemic Follow-up: Enforce lifecycle { prevent_destroy = true } on all stateful resources, as detailed in our guide on How to Review Terraform Plans Without Destroying Production.

Leaked AWS Access Key Committed to a Public GitHub Repository

• Assume Compromise: Automated scraper bots scan GitHub within 30 seconds. Do not waste time force-pushing or deleting git history.
• Immediate Disruption: Deactivate the key in AWS IAM immediately (aws iam update-access-key --status Inactive). Accepting the risk that dependent services break is far better than letting an attacker spin up 50 GPU instances.
• CloudTrail Audit: Search CloudTrail across all regions for calls made by that accessKeyId. Specifically check for secondary backdoor IAM users or temporary STS credentials created.
• Systemic Prevention: Enforce GitHub secret scanning with push protection, and migrate from long-lived keys to short-lived OIDC federation.

Leadership Pressure to Skip Pipeline Security Scans for a Committed Release

• Avoid Extremes: Neither blindly disable the scanner ("ship it and fix next sprint") nor stonewall ("absolutely not"). Perform reachability analysis.
• Reachability Triage: Is the vulnerable package reachable in production, or is it a dev dependency / false positive?
• The Formal Exception Path: If unreachable or low-risk, issue a time-limited exception waiver (e.g., 72 hours) signed off by the CISO or Security Officer. Record the waiver in the deployment ticket.
• Never Turn Off the Gate: Never disable the scanner globally. Disabling scanners is how temporary bypasses become permanent security debt.