⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All AWS & Cloud Architecture Interview Questions Scenario 78 of 179 in AWS & Cloud Architecture
Senior DevOps / SRE [L2] AWS Cost & Architecture Production Scenario [L2]
🎯 Target Role / Context: AWS Scenario #78 (L2) · Cost & Architecture

Q: A CloudFormation stack is in `UPDATE_ROLLBACK_FAILED` state. How do you recover?

A CloudFormation stack in UPDATE_ROLLBACK_FAILED means an update failed and then the automatic rollback also failed on one or more resources.

#AWS #Cost & Architecture #L2 #Cloud #Infrastructure #Terraform State
🎙️ Candidate Opening & Architectural Context
"A CloudFormation stack in <strong>UPDATE_ROLLBACK_FAILED</strong> means an update failed and then the automatic rollback also failed on one or more resources; you recover by fixing the underlying cause and running <code>continue-update-rollback</code>, skipping a resource only as a last resort. The stack is frozen: you cannot update it until the rollback completes. Interviewers use this to test calm, ordered diagnosis: read the stack events, identify why a resource could not return to its previous state, repair that cause, retry, and understand the drift risk that skipping creates."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: A CloudFormation stack in <strong>UPDATE_ROLLBACK_FAILED</strong> means an update failed and then the automatic rollback also failed on one or more resources; you recover by fixing the underlying cause and running <code>continue-update-rollback</code>, skipping a resource only as a last resort. The stack is frozen: you cannot update it until the rollback completes. Interviewers use this to test calm, ordered diagnosis: read the stack events, identify why a resource could not return to its previous state, repair that cause, retry, and understand the drift risk that skipping creates.

1️⃣

What UPDATE_ROLLBACK_FAILED means

Rollback tries to restore every changed resource to its previous configuration; one of those restores failed.

  • State: the stack is stuck; allowed actions are continue-update-rollback or deleting the stack. A normal update-stack is rejected.
  • Typical trigger: something outside CloudFormation changed, such as a resource deleted by hand, an IAM role removed, or a dependency in use.
  • Two failures: the original update failure (the first UPDATE_FAILED) is not the one blocking you; the blocker is the resource failing during rollback.
  • Priority: do not delete the stack in a hurry, as that may destroy production resources.
  • Contrast with other states: UPDATE_ROLLBACK_COMPLETE is a healthy, updatable state after a successful rollback; UPDATE_FAILED with rollback disabled is different again, and you must handle each correctly.
2️⃣

Hands-on: UPDATE_ROLLBACK_FAILED, find the failing resource

Read events in reverse chronological order for the status reason.

aws cloudformation describe-stack-events --stack-name my-stack \
  --query "StackEvents[?ResourceStatus=='UPDATE_FAILED'].[Timestamp,LogicalResourceId,ResourceType,ResourceStatusReason]" \
  --output table

aws cloudformation describe-stacks --stack-name my-stack \
  --query 'Stacks[0].[StackStatus,StackStatusReason]'
  • Locate failed resources: filter events for UPDATE_FAILED that occurred during rollback and read ResourceStatusReason.
  • Common reasons: resource does not exist, not authorized or role assume failure, resource in use or dependency conflict, S3 bucket not empty, or service stabilization timeout such as an ECS service that cannot reach a steady state.
  • Nested stacks: open the child stack that failed; the parent only reports the nested stack failure.
  • CloudTrail: search for manual changes to that resource around the time of the incident.
  • Console route: the Stack actions menu offers Continue update rollback with an advanced section listing which failed resources you may skip, which is handy when you cannot recall logical IDs.
  • Read in order: sort events by timestamp and look for the earliest rollback-phase failure, because later failures often cascade from it.
Advertisement
3️⃣

Fix the cause, then continue the rollback

Make the resource restorable and retry.

aws cloudformation continue-update-rollback --stack-name my-stack
aws cloudformation wait stack-rollback-complete --stack-name my-stack

# Last resort when the resource cannot be repaired: skip it
aws cloudformation continue-update-rollback --stack-name my-stack \
  --resources-to-skip MyBrokenResourceLogicalId
# nested stack resources use the form NestedStackLogicalId.ResourceLogicalId
  • Resource deleted manually: recreate it with the same physical identity where possible, or restore from backup, so rollback can proceed.
  • Permissions: if the service role or your principal lost permissions, restore them or supply --role-arn with a role that has the needed access.
  • In use or not empty: detach the dependency or empty the bucket (carefully), then retry.
  • Retry: run continue-update-rollback and wait for UPDATE_ROLLBACK_COMPLETE.
  • Service role considerations: if the stack was created with a service role, rollback uses it, so a deleted or trust-broken role is a frequent blocker.
4️⃣

Skipping resources and what it costs

Skipping tells CloudFormation to mark the resource as successfully rolled back without touching it.

  • Result: the skipped resource is marked as complete, but its real configuration may not match the previous template, so the stack is out of sync.
  • After skipping: run drift detection (aws cloudformation detect-stack-drift), reconcile manually or by importing, and only then deploy.
  • Which resources: only resources in a failed state can be skipped; others are not eligible.
  • Retry the original change: once the stack is UPDATE_ROLLBACK_COMPLETE, correct the template or parameters that caused the first failure and deploy again, preferably using a change set.
  • Document the skip: record which resources were skipped and why in the incident notes, so the later reconciliation is deliberate.
5️⃣

Verification and prevention

Confirm health and stop it recurring.

  • Verify: stack status is UPDATE_ROLLBACK_COMPLETE, critical resources respond, and drift detection is clean.
  • Change sets and previews: review replacement and delete actions before executing updates.
  • Stack policy and termination protection: shield critical resources from accidental updates and the stack from deletion.
  • Stop console edits: restrict manual changes to managed resources through IAM and detect them with scheduled drift detection.
  • Rollback triggers: attach CloudWatch alarms so bad updates roll back automatically in a controlled way.
  • Pipelines: make CI deploy through aws cloudformation deploy with change sets and fail fast, so a stuck stack is noticed immediately rather than at the next release.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Find the rollback-time UPDATE_FAILED event first, repair that resource, and only use --resources-to-skip as a last resort followed by drift detection."
⚡ 60-Second Elevator Pitch Talking Points
  • UPDATE_ROLLBACK_FAILED means the rollback itself failed on a resource, leaving the stack frozen until rollback completes.
  • I read describe-stack-events for the failing resource and reason, fix the cause such as a deleted resource or missing permission, then run continue-update-rollback.
  • I skip resources only as a last resort, run drift detection afterwards, and prevent repeats with change sets, stack policies and restricted console access.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Deleting the stack to get unstuck It can destroy production resources that were fine; recover the rollback instead.
  2. Reaching for --resources-to-skip immediately It leaves the stack out of sync with reality and hides the real defect.
  3. Debugging only the parent of a nested stack The actual failure is in the child stack's events.
  4. Redeploying the same template after recovery without finding the original failure The first update failure will happen again and roll back again.
Advertisement
Want more AWS & Cloud Architecture scenarios?
Explore our complete collection of scenario-based AWS & Cloud Architecture interview runbooks.
Browse All AWS & Cloud Architecture Questions →