How can I stop my Amazon ECS service from failing to stabilize in CloudFormation?

Short description

A service created in Amazon ECS fails to stabilize if it isn't in the state specified by the AWS CloudFormation template. To confirm that a service launched the desired number of tasks with the desired task definition, AWS CloudFormation makes repeated DescribeService API calls. These calls check the status of the service until the desired state is met. The calling process can take up to three hours. Then, AWS CloudFormation times out, and returns the "Service ARN did not stabilize" message. While AWS CloudFormation checks the status of the service, the stack with the service remains in the CREATE_IN_PROGRESS state or the UPDATE_IN_PROGRESS state and can't be updated.

To avoid a DescribeService API call timeout, manually force the state of the Amazon ECS service resource in AWS CloudFormation into a CREATE_COMPLETE state. Set the desired count of the service to zero in the Amazon ECS console to stop running tasks. AWS CloudFormation then considers the update as successful, because the number of tasks equals the desired count of zero.

Important: As a best practice, don't manually force AWS CloudFormation into a CREATE_COMPLETE state because a forced stop can cause a production outage.

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

Resolution

Verify resource creation

Complete the following steps:

1. In your AWS CloudFormation template, create an AWS::ECS::Service resource. For example:

   Resources:
   ECSServiceA:
    Type: AWS::ECS::Service
   Properties:
     DesiredCount: 1
     Cluster: awsExampleECSCluster
     LaunchType: EC2
     ServiceName: "MyNginxService2"
     TaskDefinition: NginxTask:1

2. Open the AWS CloudFormation console, and then select your stack.

3. Choose the Events tab, and then verify that your resource is being created.

Update the desired count of the service

You can update the desired count of the service to your original value with either the AWS CLI or the Amazon ECS console.

AWS CLI

Complete these steps:

1. To describe the service and list the service events, run the AWS CLI describe-services command:

   aws ecs describe-services --cluster awsExampleECSCluster --services MyNginxService2

2. To update the desired count of the service, run the AWS CLI update-service command:

   aws ecs update-service --cluster awsExampleECSCluster --service MyNginxService2 --desired-count 0

3. Update --desired-count to your original value.

Amazon ECS console

Complete the following steps:

1. Open the Amazon ECS console.

2. In the navigation pane, choose Clusters. Then, select the cluster that contains the Amazon ECS service that you created.

3. On the Clusters page, choose the cluster that contains the Amazon ECS service that you created.

4. On the page for the cluster that you selected, in the Service Name column, choose your service.

5. Choose the Events tab, and then choose Update.

6. On the Configure service page, for Number of tasks, enter 0.

7. Choose Next step to step through to the end of the Update Service wizard, and then choose Update Service.

   The service now reaches a steady state and transitions the Amazon ECS service resource in AWS CloudFormation to CREATE_COMPLETE or UPDATE_COMPLETE.

   Important: Sync your AWS CloudFormation stack with the Amazon ECS service properties after you fix the issue with the underlying tasks. To perform a sync, manually change the desired count (DesiredCount) back to the original value from your template.

Related information

What is Amazon Elastic Container Service?

services-stable