I want to resolve the "cannotpullcontainererror" error so that I can start my Amazon Elastic Container Service (Amazon ECS) tasks on AWS Fargate.
Short description
The "cannotpullcontainererror" error can prevent your Amazon ECS tasks on Fargate from starting. For these tasks, your Amazon Virtual Private Cloud (Amazon VPC) networking configurations must allow ECS to access the repository where the image is stored. Without the correct networking, the image can't be pulled by Amazon ECS on Fargate and the container can't start.
Resolution
To resolve the "cannotpullcontainererror" error, complete the following tasks.
Confirm that your VPC allows your Amazon ECS infrastructure to reach the image repository
The route tables associated to the subnets that your task is created in must allow your Amazon ECS infrastructure to reach the repository endpoint. The endpoint can be reached through an internet gateway, NAT gateway, or VPC endpoints.
For configurations that don't use AWS PrivateLink, complete the following steps:
- Open the Amazon VPC console.
- In the navigation pane, choose Subnets.
- Select the subnet that your ECS Fargate task is using.
- Choose the Route Table tab.
- In the Destination column, confirm that the default route (0.0.0.0/0) of the route table allows public internet access. This access can be either through a NAT gateway or an internet gateway.
Important: The NAT gateway or internet gateway must be the target of the default route. For example route tables, see Example routing options. If you don't use a NAT gateway or internet gateway, then make sure that your custom configuration allows public internet access.
If you use an internet gateway (public subnets), then confirm that the task has a public IP assigned to it. When you create the task or service, set Auto-assign public IP to Enabled under the VPC and security groups section.
For configurations that use PrivateLink, confirm that the security groups for your VPC endpoints allow the Fargate infrastructure to use them.
Note: Amazon ECS tasks hosted on Fargate that use version 1.3.0 or earlier require the com.amazonaws.region.ecr.dkr Amazon Elastic Container Registry (Amazon ECR) VPC endpoint. These tasks also require the Amazon Simple Storage Service (Amazon S3) gateway endpoint. Amazon ECS tasks hosted on Fargate that use version 1.4.0 or later require the com.amazonaws.region.ecr.dkr and com.amazonaws.region.ecr.api Amazon ECR VPC endpoints. These tasks also require the Amazon S3 gateway endpoint.
For configurations that use PrivateLink, complete the following steps:
- Open the Amazon VPC console.
- In the navigation pane, choose Endpoints.
- Select the endpoint from the list of endpoints, and then choose the Subnets tab. The VPC endpoints com.amazonaws.region.ecr.dkr and com.amazonaws.region.ecr.api for Amazon ECR display in the list of subnets and associated with the Fargate subnets. Amazon S3 gateway is also on the list of subnets.
Note: If a subnet isn't listed, choose Manage Subnets. Next, select the subnet based on its Availability Zone. Then, choose Modify Subnets.
- Choose the Policy tab, and then confirm that the correct policy requirements are met.
- Confirm that the security group attached to com.amazonaws.region.ecr.api and com.amazonaws.region.ecr.dkr VPC endpoints allows incoming connections on port 443 from the Amazon ECS tasks for Fargate. To confirm this setting, select the endpoint from the list.
- Choose the Security Groups tab.
- For Group ID, choose the security group ID.
- Choose the Inbound rules tab, and then confirm that you can see the rule that allows 443 connections from your ECS tasks on Fargate.
Check the VPC DHCP Option Set
Complete the following steps:
- Open the Amazon VPC console.
- In the navigation pane, choose Your VPCs.
- Select the VPC that contains your Fargate task.
- On the Details tab, note the setting for DHCP options set.
- In the navigation pane, choose DHCP Options Sets.
- Select the DHCP options set that you noted.
- Choose Actions, and then choose View details.
- Confirm that Domain name servers is set to AmazonProvidedDNS. If it isn't set to AmazonProvidedDNS, then configure conditional DNS forwarding.
Check the task execution role permissions
Complete the following steps:
- Open the AWS Identity and Access Management (IAM) console.
- In the navigation pane, choose Roles.
- Select the task execution role that your Fargate tasks are using.
- Confirm that the task execution role has the permissions to pull an image from Amazon ECR.
Check that the image exists
Complete the following steps:
- Open the Amazon ECR console.
- Select the Amazon ECR repository that your Fargate task pulls the image from.
- Confirm that the URI and the tag in Amazon ECR are the same as what's specified in the task definition. If the image is missing, then check if a lifecycle policy manages the images.
Note: If you don't use Amazon ECR, then make sure that you see image:tag in the specified image repository.
Check the ephemeralStorage allocated to the Fargate task
By default, the Fargate tasks are allocated 20GiB of ephemeral storage. If your image size is too large, then you see an error similar to the following:
"CannotPullContainerError: ref pull has been retried 1 time(s): failed to extract layer no space left on device: unknown"
Note: Because this storage is used for temporary files, logs, and other processes, you might get the error even if the image is smaller in size.
To resolve the issue, edit the allocation on the task definition. For more information, see Fargate task ephemeral storage for Amazon ECS
Note: Fargate tasks that use platform version 1.4.0 or later receive a minimum of 20 GiB of ephemeral storage. The total amount of ephemeral storage can be increased up to a maximum of 200 GiB.