What common mistakes should candidates avoid when answering: Ansible Q15: An in-place database schema migration fails halfway through a playbook How do you use block rescue and always to implement automated rollback [L3]?

1. Omitting a fail module task at the end of the rescue block: If all tasks in rescue succeed, Ansible considers the play successful (green), allowing broken builds to progress downstream. 2. Attempting rollbacks using ignore_errors: yes: Blindly ignores failures and proceeds with dependent tasks, corrupting live database states. 3. Neglecting lockfile cleanup in the always block: Leaves orphaned mutex lockfiles on target hosts, preventing future deployments until manual SSH intervention.

Staff Platform Architect [L3] Ansible Error Handling & Automated Rollback Staff Scenario [L3]

🎯 Target Role / Context: Staff DevOps / Platform Architect Scenario #15 · Chaos Engineering

Q: An in-place database schema migration task fails halfway through a playbook. How do you use Ansible block, rescue, and always directives to implement automated rollback runbooks and send alert webhooks?

Implement try-catch-finally error handling in Ansible using block, rescue, and always. When a critical database migration task in a block fails, the rescue section triggers automated rollbacks, and always executes telemetry and notification webhooks.

#Ansible #block #rescue #always #Error Handling #Rollback

🎙️ Candidate Opening & Architectural Context

""In enterprise CI/CD pipelines, automated failure handling is the difference between an unnoticeable self-healing deployment and a 3 AM Sev-1 outage. Ansible's block, rescue, and always directives provide robust exception handling analogous to try-catch-finally in modern programming languages.""

⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

1️⃣

Encapsulate Risky Tasks in a block

Group the forward migration operations inside a logical block structure:

- name: Database Schema Migration Block
block:
- name: Create database snapshot
- name: Apply Flyway / Liquibase database schema migration
- name: Run post-migration sanity tests

2️⃣

Trigger Automated Rollback with rescue

The rescue section executes exclusively if any task inside the preceding block returns an error:

rescue:
- name: Revert schema to previous migration version
command: /opt/bin/migrate -rollback
- name: Post alert to Slack incident channel
uri: url="{{ slack_webhook }}" method=POST body_format=json ...
- name: Fail explicitly to halt deployment pipeline
fail: msg='Schema migration failed! Automated rollback successfully executed.'

3️⃣

Guarantee Cleanup Operations with always

Tasks declared in always run unconditionally regardless of whether the block succeeded or failed:

always:
- name: Remove temporary lock files and credentials
file: path=/tmp/db_migration.lock state=absent
- name: Publish execution metric to CloudWatch / Datadog

💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)

"Use block, rescue, and always to build production-grade try-catch-finally automation. Execute migrations in block, trigger automated rollbacks and alerts in rescue, and guarantee cleanup of locks and temp files in always."

⚡ 60-Second Elevator Pitch Talking Points

Ansible's 'block' groups tasks logically, functioning as a 'try' block in software engineering.
The 'rescue' block functions as a 'catch' handler, executing only if any task inside the block fails.
Use rescue to trigger automated rollbacks (e.g. reverting DB schemas, restoring config backups) and paging SREs via webhook.
The 'always' block functions as a 'finally' handler, executing unconditionally regardless of success or failure.
Use always to clean up temporary credential files, release mutex lockfiles, and export pipeline metrics.
Always include an explicit 'fail' task at the end of rescue to ensure the CI/CD pipeline registers the build failure.

⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

Omitting a fail module task at the end of the rescue block If all tasks in rescue succeed, Ansible considers the play successful (green), allowing broken builds to progress downstream.
Attempting rollbacks using ignore_errors: yes Blindly ignores failures and proceeds with dependent tasks, corrupting live database states.
Neglecting lockfile cleanup in the always block Leaves orphaned mutex lockfiles on target hosts, preventing future deployments until manual SSH intervention.

Q: An in-place database schema migration task fails halfway through a playbook. How do you use Ansible block, rescue, and always directives to implement automated rollback runbooks and send alert webhooks?

🛠️ Production Runbook & Step-by-Step Resolution

Encapsulate Risky Tasks in a block

Trigger Automated Rollback with rescue

Guarantee Cleanup Operations with always

Get 1 DevOps interview question in your inbox every week

📚 Related Questions & Incident Scenarios

Ansible Q3: You need to rotate database credentials across 80 microservices How do you integrate Ansible Vault with automated CI/CD pipelines without exposing keys in console logs [L3]

Ansible Q7: A junior engineer uses the shell module for every task in an enterprise playbook What are the security risks when should you use command vs shell and how do you enforce idempotency [L1]

Ansible Q4: Why is an Nginx restart handler not firing even though a template configuration task notified it and how do you prevent broken configurations from reloading [L2]

Ansible Q6: When would you use the serial keyword versus forks in Ansible playbooks and how does serial enable rolling zero-downtime canary deployments [L3]

Ansible Q11: Playbook execution against 500 servers takes 45 minutes on Gathering Facts How do you optimize fact gathering and implement fact caching [L2]

Ansible Q1: You have an architecture running 50 identical EC2 instances You need to deploy a rapid emergency patch Using Ansible what is the problem with running the playbook against the inventory sequentially and how do you speed it up [L2]