How can I get my CloudFormation stack to update if it's stuck in the UPDATE_ROLLBACK_FAILED state?

3 minute read
1

My AWS CloudFormation stack is stuck in the UPDATE_ROLLBACK_FAILED state.

Short description

For CloudFormation stacks in the UPDATE_ROLLBACK_FAILED state, use either the ContinueUpdateRollback action to recover the stack or the DeleteStack action to delete the stack. To retry the rollback, first resolve any errors, and then use ContinueUpdateRollback to perform the rollback operation.
Note: To resolve the error, you might need to raise limits, change permissions, or modify other settings.

For information on common errors that cause update rollback failures, see Update rollback failed.

In some cases, the error doesn't resolve when you retry the rollback. When this happens, you must skip the failed resources when you perform the ContinueUpdateRollback action. When the stack is in the UPDATE_ROLLBACK_COMPLETE state, you can update the stack as usual.

Resolution

To roll the stack back to a working state, use either the CloudFormation console or AWS Command Line Interface (AWS CLI).
Note: If you receive errors when you run AWS CLI commands, then see Troubleshoot AWS CLI errors. Make sure that you use the most recent AWS CLI version.

CloudFormation console

To roll the stack back and skip failed resources, complete the following steps:

  1. Open the CloudFormation console.
  2. From the navigation pane, choose Stacks.
  3. From the Stack name column, select the stack that's stuck in UPDATE_ROLLBACK_FAILED status.
  4. Choose Stack Actions, and then choose Continue update rollback.
  5. In the Continue update rollback dialog, expand Advanced troubleshooting.
  6. In the Resources to skip - optional section, select the resources that you want to skip.
  7. Choose Continue update rollback.

To roll the stack back and not skip resources, complete the following steps:

  1. Open the CloudFormation console.
  2. From the navigation pane, choose Stacks.
  3. From the Stack name column, select the stack that's stuck in UPDATE_ROLLBACK_FAILED status.
  4. Choose Stack Actions, and then choose Continue update rollback.

AWS CLI

To roll the stack back in the AWS CLI, run the continue-update-rollback command:

$ aws cloudformation continue-update-rollback --stack-name awsstackname123 --resources-to-skip awsfaultyresource123

Note: Replace awsstackname123 with the name of your stack. Replace awsfaultyresource123 with the logical IDs of your resources.

Note that CloudFormation sets the status of specified resources to UPDATE_COMPLETE. Then, CloudFormation continues to roll back the stack. After the rollback completes, the state of the skipped resources isn't consistent with the state of the resources in the stack template.

Before you perform another stack update, you must update the stack or the resources to be consistent with each other. If you don't do this, then subsequent stack updates can fail, and the stack becomes unrecoverable.

For more information on rolling back nested stacks to a working state, see Using ResourcesToSkip to recover a nested stacks hierarchy.

AWS OFFICIAL
AWS OFFICIALUpdated 3 months ago
8 Comments

What to do in cases where it doesn't work, loops between 'Only the resources failed during UpdateRollback are allowed to be skipped' and 'The following resource(s) failed to update'.

replied 2 years ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 2 years ago

Hello, when I do this, I get Currently in UPDATE_COMPLETE with reason: null for the resource when I do not skip it and "Nested stack resources can only be skipped when their embedded stack statuses are one of [DELETE_COMPLETE, DELETE_IN_PROGRESS, DELETE_FAILED]" when I select it to be skipped.

What can I do?

Jonas
replied a year ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied a year ago

Hello, I aslo meet the error: null for the resource when I do not skip it and "Nested stack resources can only be skipped when their embedded stack statuses are one of [DELETE_COMPLETE, DELETE_IN_PROGRESS, DELETE_FAILED]" when I select it to be skipped. If I tried CLI, I had error: An error occurred (ValidationError) when calling the ContinueUpdateRollback operation: 1 validation error detected: Value '[monitoringServer, ec2-sap-ecc-landscape-shared-monitoringServer-6NE49OJUSLVE.ec2-sap-ecc-landscape-shared-monitoringServer-6NE49OJUSLVE]' at 'resourcesToSkip' failed to satisfy constraint: Member must satisfy constraint: [Member must satisfy regular expression pattern: [a-zA-Z0-9]+|[a-zA-Z][-a-zA-Z0-9]*.[a-zA-Z0-9]+] How can I can solve this ?

Kas
replied 8 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 8 months ago

Hey, what can we do incase we missed to select the resources under the 'Resources to skip - optional' section and performed the 'Continue update rollback' ? We are now getting an error at 'UPDATE_ROLLBACK_COMPLETE' with the ''Continue update rollback' greyed out. Any way to revert this and attempt the option again ? Or is there a cli command we can use to fetch the optional resource and run the above command mentioned in wiki to manually trigger 'Continue update rollback' ? Appreciate your help on this, thanks!

replied 5 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 5 months ago