How do I troubleshoot an ECS task that doesn’t start because of problems with an EFS filesystem DNS name?

Short description

When you reference your EFS file system and container mount point in your ECS task definition, ECS mounts the file system in your container. When you use the ECS task definition with an EFS volume mount to run an ECS task, the task fails with the following error:

"ResourceInitializationError: failed to invoke EFS utils commands to set up EFS volumes: stderr: Failed to resolve "fs-<id>.efs.<region>.amazonaws.com" - check that your file system ID is correct, and ensure that the VPC has an EFS mount target for this file system ID. See https://docs.aws.amazon.com/console/efs/mount-dns-name for more detail. Attempting to lookup mount target ip address using botocore. Failed to import necessary dependency botocore, please install botocore first."

The error occurs for the following reasons:

• You use a custom domain name and servers in your virtual private cloud (VPC) instead of the AmazonProvidedDNS.
• Your VPC DNS Hostnames is turned off.
• Your EFS doesn't have mount targets or the targets aren't in the same availability zone as your ECS tasks.

When the DNS name doesn't resolve an EFS file system, ECS uses efs-util tools to install botocore and retrieve the mount target IP address. Because botocore isn't installed on ECS optimized AMIs by default, the task fails.

Resolution

You have a custom domain name and servers in your VPC

For tasks that run on Amazon EC2

Note: These steps apply to only Amazon EC2 instances.

1.    Use SSH to connect to the container instance. To install botocore, run the following command:

 sudo pip3 install botocore --upgrade

Install botocore during the launch of the instance as part of the UserData you configure in your Auto Scaling group for container instances.

2.    Add the following permissions to the AWS Identity and Access Management (IAM) role for the container instance:

 elasticfilesystem:DescribeMountTargets
 ec2:DescribeAvailabilityZones

For tasks that run on Fargate

To allow your custom domain name servers to resolve AWS resources, configure conditional DNS forwarding.

Note: For AWS Fargate instances, you can't access the instance to install botocore and must use the DNS name resolution.

Your VPC DNS Hostnames is turned off

To review DNS attributes, complete the steps in DNS attributes. If DNS Hostnames is turned off, then turn it on, and then check if the ECS tasks are running.

Your EFS doesn't have mount targets or the targets aren't in the same availability zone as your ECS tasks

Your EFS file system must have mount targets and the targets must be in the same availability zone as the ECS tasks. For more information, see Creating and managing mount targets and security groups.

Related information:

Using botocore to retrieve the mount target IP address when the DNS name cannot be resolved on the GitHub website