Q: A CloudFormation stack is in `UPDATE_ROLLBACK_FAILED` state. How do you recover?
A CloudFormation stack in UPDATE_ROLLBACK_FAILED means an update failed and then the automatic rollback also failed on one or more resources.
Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: A CloudFormation stack in <strong>UPDATE_ROLLBACK_FAILED</strong> means an update failed and then the automatic rollback also failed on one or more resources; you recover by fixing the underlying cause and running <code>continue-update-rollback</code>, skipping a resource only as a last resort. The stack is frozen: you cannot update it until the rollback completes. Interviewers use this to test calm, ordered diagnosis: read the stack events, identify why a resource could not return to its previous state, repair that cause, retry, and understand the drift risk that skipping creates.
What UPDATE_ROLLBACK_FAILED means
Rollback tries to restore every changed resource to its previous configuration; one of those restores failed.
- State: the stack is stuck; allowed actions are
continue-update-rollbackor deleting the stack. A normalupdate-stackis rejected. - Typical trigger: something outside CloudFormation changed, such as a resource deleted by hand, an IAM role removed, or a dependency in use.
- Two failures: the original update failure (the first UPDATE_FAILED) is not the one blocking you; the blocker is the resource failing during rollback.
- Priority: do not delete the stack in a hurry, as that may destroy production resources.
- Contrast with other states: UPDATE_ROLLBACK_COMPLETE is a healthy, updatable state after a successful rollback; UPDATE_FAILED with rollback disabled is different again, and you must handle each correctly.
Hands-on: UPDATE_ROLLBACK_FAILED, find the failing resource
Read events in reverse chronological order for the status reason.
aws cloudformation describe-stack-events --stack-name my-stack \
--query "StackEvents[?ResourceStatus=='UPDATE_FAILED'].[Timestamp,LogicalResourceId,ResourceType,ResourceStatusReason]" \
--output table
aws cloudformation describe-stacks --stack-name my-stack \
--query 'Stacks[0].[StackStatus,StackStatusReason]'
- Locate failed resources: filter events for
UPDATE_FAILEDthat occurred during rollback and readResourceStatusReason. - Common reasons: resource does not exist, not authorized or role assume failure, resource in use or dependency conflict, S3 bucket not empty, or service stabilization timeout such as an ECS service that cannot reach a steady state.
- Nested stacks: open the child stack that failed; the parent only reports the nested stack failure.
- CloudTrail: search for manual changes to that resource around the time of the incident.
- Console route: the Stack actions menu offers Continue update rollback with an advanced section listing which failed resources you may skip, which is handy when you cannot recall logical IDs.
- Read in order: sort events by timestamp and look for the earliest rollback-phase failure, because later failures often cascade from it.
Fix the cause, then continue the rollback
Make the resource restorable and retry.
aws cloudformation continue-update-rollback --stack-name my-stack
aws cloudformation wait stack-rollback-complete --stack-name my-stack
# Last resort when the resource cannot be repaired: skip it
aws cloudformation continue-update-rollback --stack-name my-stack \
--resources-to-skip MyBrokenResourceLogicalId
# nested stack resources use the form NestedStackLogicalId.ResourceLogicalId
- Resource deleted manually: recreate it with the same physical identity where possible, or restore from backup, so rollback can proceed.
- Permissions: if the service role or your principal lost permissions, restore them or supply
--role-arnwith a role that has the needed access. - In use or not empty: detach the dependency or empty the bucket (carefully), then retry.
- Retry: run
continue-update-rollbackand wait forUPDATE_ROLLBACK_COMPLETE. - Service role considerations: if the stack was created with a service role, rollback uses it, so a deleted or trust-broken role is a frequent blocker.
Skipping resources and what it costs
Skipping tells CloudFormation to mark the resource as successfully rolled back without touching it.
- Result: the skipped resource is marked as complete, but its real configuration may not match the previous template, so the stack is out of sync.
- After skipping: run drift detection (
aws cloudformation detect-stack-drift), reconcile manually or by importing, and only then deploy. - Which resources: only resources in a failed state can be skipped; others are not eligible.
- Retry the original change: once the stack is
UPDATE_ROLLBACK_COMPLETE, correct the template or parameters that caused the first failure and deploy again, preferably using a change set. - Document the skip: record which resources were skipped and why in the incident notes, so the later reconciliation is deliberate.
Verification and prevention
Confirm health and stop it recurring.
- Verify: stack status is
UPDATE_ROLLBACK_COMPLETE, critical resources respond, and drift detection is clean. - Change sets and previews: review replacement and delete actions before executing updates.
- Stack policy and termination protection: shield critical resources from accidental updates and the stack from deletion.
- Stop console edits: restrict manual changes to managed resources through IAM and detect them with scheduled drift detection.
- Rollback triggers: attach CloudWatch alarms so bad updates roll back automatically in a controlled way.
- Pipelines: make CI deploy through
aws cloudformation deploywith change sets and fail fast, so a stuck stack is noticed immediately rather than at the next release.
- UPDATE_ROLLBACK_FAILED means the rollback itself failed on a resource, leaving the stack frozen until rollback completes.
- I read describe-stack-events for the failing resource and reason, fix the cause such as a deleted resource or missing permission, then run continue-update-rollback.
- I skip resources only as a last resort, run drift detection afterwards, and prevent repeats with change sets, stack policies and restricted console access.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Deleting the stack to get unstuck It can destroy production resources that were fine; recover the rollback instead.
- Reaching for --resources-to-skip immediately It leaves the stack out of sync with reality and hides the real defect.
- Debugging only the parent of a nested stack The actual failure is in the child stack's events.
- Redeploying the same template after recovery without finding the original failure The first update failure will happen again and roll back again.