Q: You have a cron job script taking database backups at 2 AM every night. One night, the backup takes 25 hours instead of 2 hours because the DB grew. At 2 AM the next night, the cron job fires again. Now two heavy backups are running simultaneously, locking the DB. How do you prevent this in the script?
Cron jobs must be designed to be self-aware of overlapping executions.
#General DevOps #General DevOps — Scenario-Based Interview Questions #L2 #DevOps #SRE #Architecture
🎙️ Candidate Opening & Architectural Context
""Senior DevOps is about building reliable automated feedback loops between code commit and production observability. The interviewer is testing: Idempotency, locking mechanisms (flock/pid files).. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third.""
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Production Solution & Architecture
Cron jobs must be designed to be self-aware of overlapping executions. This is solved using File Locking. The script must attempt to acquire an exclusive lock before running the heavy operation. If it cannot get the lock, it immediately exits. In Linux, the standard tool is flock: 0 2 * * * /usr/bin/flock -n /var/run/backup.lock /path/to/backup_script.sh The -n (non-blocking) flag tells flock to immediately fail if the lock file is currently held by a previous, still-running instance of the script. This guarantees only one backup can ever run at a time.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Cron jobs must be designed to be self-aware of overlapping executions.."
⚡ 60-Second Elevator Pitch Talking Points
- Immediate Triage: Cron jobs must be designed to be self-aware of overlapping executions.
- Run targeted verification commands before modifying configuration.
- Automate permanent guardrails (CI check, alerts, IaC policy) to prevent recurrence.
Advertisement