Why can't I pull images from Amazon ECR when I use Amazon EKS?

Short description

You can't pull images from Amazon ECR for one of the following reasons:

• You can't communicate with Amazon ECR endpoints.
• You don't have the appropriate permissions in the instance's AWS Identity and Access Management (IAM) role for your worker nodes.
• You haven't created interface virtual private cloud (VPC) endpoints.

Based on your use case, use one or more of the following resolutions.

Resolution

Troubleshoot the communication between your worker nodes and Amazon ECR endpoints

If your worker nodes can't communicate with the Amazon ECR endpoints, then you receive the following error message:

"Failed to pull image 'ACCOUNT.dkr.ecr.REGION.amazonaws.com/imagename:tag': rpc error: code = Unknown desc = Error response from daemon: Get https://ACCOUNT.dkr.ecr.REGION.amazonaws.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"

To resolve this error, confirm the following:

• The subnet for your worker node has a route to the internet and the route table associated with your subnet is configured correctly.
• The security group associated with your worker node allows outbound internet traffic.
• The ingress and egress rule for your network access control lists (ACLs) allows access to the internet.

Update the instance role of your worker nodes

The following error occurs in your Amazon EKS Pod when the instance's IAM role for your worker nodes doesn't have permission to pull images from Amazon ECR:

"Warning Failed 14s (x2 over 28s) kubelet, ip-000-000-000-000.us-west-2.compute.internal Failed to pull image 'ACCOUNT.dkr.ecr.REGION.amazonaws.com/imagename:tag': rpc error: code = Unknown desc = Error response from daemon: Get https://ACCOUNT.dkr.ecr.REGION.amazonaws.com/v2/imagename/manifests/tag: no basic auth credentials Warning Failed 14s (x2 over 28s) kubelet, ip-000-000-000-000.us-west-2.compute.internal Error: ErrImagePull Normal BackOff 2s (x2 over 28s) kubelet, ip-000-000-000-000.us-west-2.compute.internal Back-off pulling image 'ACCOUNT.dkr.ecr.REGION.amazonaws.com/imagename:tag' Warning Failed 2s (x2 over 28s) kubelet, ip-000-000-000-000.us-west-2.compute.internal Error: ImagePullBackOff"

To resolve this error, confirm that your worker nodes use the AmazonEC2ContainerRegistryReadOnly IAM managed policy. Or, update the Amazon Elastic Compute Cloud (Amazon EC2) instance profile of your worker nodes with the following IAM permissions:

"ecr:GetAuthorizationToken","ecr:BatchCheckLayerAvailability","ecr:GetDownloadUrlForLayer","ecr:GetRepositoryPolicy","ecr:DescribeRepositories","ecr:ListImages","ecr:DescribeImages","ecr:BatchGetImage","ecr:GetLifecyclePolicy","ecr:GetLifecyclePolicyPreview","ecr:ListTagsForResource","ecr:DescribeImageScanFindings"

Important: It's a best practice to use the AmazonEC2ContainerRegistryReadOnly policy and avoid duplicate policies.

The updated IAM role for the instance gives your worker nodes the permission to access Amazon ECR and pull images through the kubelet. The kubelet fetches and periodically refreshes Amazon ECR credentials. For more information, see Images on the Kubernetes website.

Validate your repository policies

To confirm that your repository policies are correct, complete the following steps:

1. Open the Amazon ECR console for your primary AWS account.
2. Navigate to the AWS Region that contains the Amazon ECR repository.
3. On the navigation pane, choose Repositories, and then choose the repository that you want to check.
4. On the navigation pane, choose Permissions. Then, check if your repository has the correct permissions. The following example policy allows a specific IAM user to describe the repository and the images in the repository:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ECR Repository Policy",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789012:user/username"
            },
            "Action": [
                "ecr:DescribeImages",
                "ecr:DescribeRepositories"
            ],
            "Resource": "arn:aws:ecr:REGION:cross-aws-account-id:repository/repo-name"
        }
    ]
}

Note: Replace 123456789012 with your account ID and username with your IAM user or role name. Replace REGION with your Region, cross-aws-account-id with your account ID, and repo-name with the name of your Amazon ECR repository.

Configure your repository policies to allow cross-account Amazon ECR access

If you don't have access to container images in another account, then the kubelet fails and you receive the following error message:

"Failed to pull image 'cross-aws-account-id:.dkr.ecr.REGION.amazonaws.com/repo-name:image-tag': rpc error: code = Unknown desc = Error response from daemon: pull access denied for arn:aws:ecr:REGION:cross-aws-account-id:repository/repo-name, repository does not exist or may require 'docker login': denied: User: arn:aws:sts::<aws-account-containing-eks-cluster>:assumed-role/<node-instance-role-for-worker-node> is not authorized to perform: ecr:BatchGetImage on resource: arn:aws:ecr:REGION:cross-aws-account-id:repository/repo-name"

To resolve this error, use the following policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789012:role/eksctl-cross-account-ecr-access-n-NodeInstanceRole"
            },
            "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:DescribeImages",
                "ecr:BatchGetImage",
                "ecr:GetLifecyclePolicy",
                "ecr:GetLifecyclePolicyPreview",
                "ecr:ListTagsForResource",
                "ecr:DescribeImageScanFindings"
            ],
            "Resource": "*"
        }
    ]
}

Note: Replace 123456789012 with your account ID and eksctl-cross-account-ecr-access-n-NodeInstanceRole with your node instance's IAM role. Use the Amazon Resource Name (ARN) of the instance's IAM role in the Amazon ECR policy, not the instance's profile ARN.

The policy allows an instance's IAM role in one account to describe and pull container images from an Amazon ECR repository in another account.

Create interface VPC endpoints

To pull images from Amazon ECR, you must create and configure interface VPC endpoints for Amazon ECS.

Validate the configuration of your Pod execution role

You receive the following error message when your AWS Fargate CoreDNS Pod is stuck in the ImagePullBackOff state when you retrieve images from Amazon hosted repositories:

"Warning Failed 27s (x2 over 40s) kubelet Failed to pull image '151284513677.dkr.ecr.eu-central-1.amazonaws.com/coredns:latest': rpc error: code = Unknown desc = failed to pull and unpack image '151284513677.dkr.ecr.eu-central-1.amazonaws.com/coredns:latest': failed to resolve reference '151284513677.dkr.ecr.eu-central-1.amazonaws.com/coredns:latest': pulling from host 151284513677.dkr.ecr.eu-central-1.amazonaws.com failed with status code [manifests latest]: 401 Unauthorized"

To resolve this error, configure your Pod execution role to use the AmazonEKSFargatePodExecutionRolePolicy.

Example policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789012:role/eksctl-cross-account-ecr-access-n-NodeInstanceRole"
            },
            "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:DescribeImages",
                "ecr:BatchGetImage",
                "ecr:GetLifecyclePolicy",
                "ecr:GetLifecyclePolicyPreview",
                "ecr:ListTagsForResource",
                "ecr:DescribeImageScanFindings"
            ],
            "Resource": "*"
        }
    ]
}

Note: Replace 123456789012 with your account ID and eksctl-cross-account-ecr-access-n-NodeInstanceRole with your node instance's IAM role.