How do I resolve Amazon ECS tasks that fail to pull images from Amazon ECR?

Short description

You might receive the CannotPullContainerError for the following reasons:

• You use a private subnet with no route to the internet, so your launch type doesn't have access to the Amazon ECR endpoint.
• Your Amazon ECR repository policy restricts access to repository images.
• Your Amazon ECS task execution or instance role doesn't have the correct permissions to pull images.
• The requested image URI doesn't exist.
• There's not enough space on your disk for the image.
• You exceeded the pull rate quota.
• There's a missing networking path between your task and the Amazon ECR endpoint.
• Your Docker credentials are outdated.

Resolution

Determine the cause of the issue

Use the AWSSupport-TroubleshootECSTaskFailedToStart runbook to determine the issue. You must run the runbook in the same AWS Region where your Amazon ECS cluster resources are located. Also, use the most recently failed task ID. If the failed task is part of an Amazon ECS service, then use the most recently failed task in the service. The failed task must be visible in ECS:DescribeTasks during the automation. By default, stopped ECS tasks are visible for 1 hour after they enter the Stopped state.

Based on the output of the automation, use one of the following manual troubleshooting steps.

Make sure that your private subnet has a route to a NAT gateway, or use PrivateLink

You can use a private subnet to run a task with an Amazon Elastic Compute Cloud (Amazon EC2) or AWS Fargate launch type. For both launch types, make sure that the private subnet has a route to a NAT gateway in the route table. Configure the NAT gateway in your VPC to route requests to the internet.

Or, use AWS PrivateLink with Amazon ECR virtual private cloud (VPC) endpoints and Amazon Simple Storage Service (Amazon S3) gateway endpoints.

The VPC endpoint security group must allow incoming connections on port 443 from the private subnet or the Amazon ECS task VPC.

Note: The ECS_IMAGE_PULL_INACTIVITY_TIMEOUT value in Windows is set to a default of 3 minutes. If you're pulling a large image, then the image pull might not complete in the 3-minute time period and the request times out.

Check your Amazon ECR repository policy

If your Amazon ECR repository policy restricts access to repository images, then you receive a 403 Forbidden error. Check your Amazon ECR repository policy to make sure that there aren't restrictions on the repository images. For examples of access policies, see Private repository policy examples in Amazon ECR.

Check the permissions of your task execution or container instance IAM role

If you use the Amazon EC2 launch type, then make sure that the container instance AWS Identity and Access Management (IAM) role has permission to access the Amazon ECR repository. The AmazonEC2ContainerRegistryReadOnly policy provides the minimum required permissions to pull images.

If you use the Fargate launch type, then make sure that the AmazonECSTaskExecutionRolePolicy is attached to the task execution role to pull an image.

Make sure that you're using the correct image URI

To confirm that you're using the correct image name in the URI, check the image parameter in the container definitions section of your task definition.

Note: To pull by tag, use the registry/repository[:tag] image name format. To pull by digest, use the registry/repository[@digest] format that you can copy directly from the Amazon ECR repository.

Check your disk space

If there isn't enough disk space to store the image that you pull, then you receive the "CannotPullContainerError: no space left on device" error message. To resolve this issue, see write /var/lib/docker/tmp/GetImageBlob111111111: no space left on device.

Check whether you exceeded the pull rate quota

If you use Docker Hub to pull the image and exceed the image pull quota, then you receive the "CannotPullContainerError: toomanyrequests: You have reached your pull rate limit" error message. For information about pull rate quotas, see Pull rate limit on the Docker Docs website. To resolve this issue, see ERROR: toomanyrequests: Too Many Requests or You have reached your pull rate limit.

Check your networking configuration

If there's a missing networking path between your task and the Amazon ECR endpoint, then you receive an error that's similar to the following examples:

"Cannotpullcontainererror: error response from daemon"

-or-

"Client.Timeout exceeded while awaiting headers"

To resolve this issue, check whether your task has access to the Amazon ECR repository through a NAT gateway or PrivateLink.

When you create an Amazon ECS service, configure the following settings under Networking:

• For tasks in public subnets, for Public IP, activate Auto-assign public IP.
• For tasks in private subnets, deactivate Auto-assign public IP, and then configure a NAT gateway or PrivateLink.

Update your Docker login credentials

If your Docker credentials are outdated, then you receive the "repository does not exist or may require 'docker login'" error message. To update your Docker login credentials for private repository authentication, see Using an authorization token.

Related information

CannotPullContainer task errors in Amazon ECS