How do I delete ECS tasks and ENIs that remain after deleting an ECS service?

Short description

When ECS tasks remain after service deletion, a typical cause is that the IAM role required by ECS to stop tasks has been deleted or has an incorrect configuration. For example, this occurs when an ECS service using VPC Lattice is created with an Infrastructure as Code tool (CDK, CloudFormation, Terraform, etc.) and the IAM role is deleted before the ECS service.

In this case, recreating the IAM role with the correct configuration allows the task termination process to resume, and the ENIs are automatically deleted.

Resolution

Rule out common causes first

For common causes of delayed ECS task termination, see the following article. If the content of that article does not apply, proceed with the steps below.

• Why have my Amazon ECS tasks been stopped or are delayed in stopping during a scaling event?

In particular, check whether the following apply:

• The task definition has a long stopTimeout value configured
• The ELB target group has a long deregistration delay configured

Verify the ECS service status

Even if the service appears to be deleted in the management console, it might still exist in a DRAINING state. Run the following AWS CLI command to check the service status:

aws ecs describe-services --cluster <cluster-name> --services <service-name> --region <region>

If the status field in the output shows DRAINING, ECS is attempting to stop the tasks but the termination process has not completed.

Check the service events

The service events (the messages in the events field) included in the output of the aws ecs describe-services command might contain errors.

If an IAM-related error is recorded, proceed to the "Verify or recreate the IAM role" step.

If no error is recorded, verify the IAM role configuration in the next step.

Identify the IAM role used by the ECS service

ECS might require the IAM role configured on the service to stop tasks. Review all IAM roles (roleArn fields) included in the output of the aws ecs describe-services command.

The following example output shows an ECS infrastructure role configured in vpcLatticeConfigurations within deployments. This is a partial excerpt of the output.

{
    "services": [
        {
            "serviceName": "my-service",
            "status": "DRAINING",
            "deployments": [
                {
                    "id": "ecs-svc/1234567890123456789",
                    "status": "PRIMARY",
                    "vpcLatticeConfigurations": [
                        {
                            "roleArn": "arn:aws:iam::123456789012:role/MyECSInfrastructureRole",
                            "targetGroupArn": "arn:aws:vpc-lattice:ap-northeast-1:123456789012:targetgroup/tg-01234567890abcdef",
                            "portName": "app-port"
                        }
                    ]
                }
            ]
        }
    ]
}

Verify or recreate the IAM role

Run the following command to check whether the identified IAM role exists:

aws iam get-role --role-name <role-name>

If the role does not exist, a NoSuchEntity error is returned. If the IAM role has been deleted, recreate it with the same name and atatch the correct trust policy and IAM policies.

If the role exists, verify that its configuration (trust policy and attached policies) is correct. If the configuration is incorrect, update the trust policy or attach the required policies.

For the trust policy and attached policies required for an ECS infrastructure role, see the following documentation:

• Amazon ECS infrastructure IAM role

Wait for task termination and ENI deletion

After recreating the IAM role, tasks won't stop immediately. Over time, ECS resumes the task termination process, and the ENIs are automatically deleted along with the tasks.

Note: If the issue persists after following these steps, consider contacting AWS Support for further assistance.