How do I troubleshoot issues with my connection between Amazon ECS and Amazon S3?

Resolution

Analyze your CloudWatch logs to identify the cause of Amazon S3 access issues

First, set up access to CloudWatch Logs. Then, use CloudWatch Logs Insights to identify the cause of your issue.

Set up CloudWatch Logs access

To configure Amazon CloudWatch Logs for your Amazon ECS task definition, complete the following steps:

1. Open the Amazon ECS console.
2. In the navigation pane, choose Task definitions.
3. Select your task definition, and then add the following code to the task definition:

   {
       "containerDefinitions": [
           {
               "name": "my-container",
               "image": "my-image:latest",
               "logConfiguration": {
                   "logDriver": "awslogs",
                   "options": {
                       "awslogs-group": "/ecs/my-task",
                       "awslogs-region": "region-code",
                       "awslogs-stream-prefix": "ecs"
                   }
               }
           }
       ]
   }

   Note: Replace my-container with your container instance name, my-image:latest with your image name and tag, and region-code with your AWS Region.
4. Update your Amazon ECS service to use the latest version of the task definition.

Use CloudWatch Logs Insights to identify the cause of your issue

To search for general Amazon S3 errors, run the following query:

filter @message like /S3|AccessDenied|NoSuchBucket/

To search for specific bucket access issues, run the following query:

filter @message like /my-bucket-name/

Note: Replace my-bucket-name with your bucket name.

Use this information to identify whether the issue relates to AWS Identity and Access Management (IAM) permissions, network connectivity, bucket configuration, or application issues. Then, complete the related troubleshooting steps.

Verify your IAM role configuration

Verify that the task role for the Amazon ECS task has the necessary Amazon S3 permissions, for example the AmazonS3ReadOnlyAccess policy for read-only operations. Review the AWS CloudTrail logs to identify denied actions that are related to Amazon S3. For example, if your Amazon ECS task must read objects from the my-app-data bucket, then attach the following custom policy to the task:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::my-app-data",
                "arn:aws:s3:::my-app-data/*"
            ]
        }
    ]
}

Note: The preceding policy allows the task to list the contents of and retrieve objects from the my-app-data bucket.

Check your network configuration

Tasks in public subnets

For tasks in public subnets, check the network access control list (network ACL) of the virtual private cloud (VPC) that your instance is in. The network ACL must allow outbound traffic on port 443. Also, verify that the security groups that are associated with your task allow outbound HTTPS traffic on port 443 to the Amazon S3 AWS managed prefix lists.

Tasks in private subnets

For tasks in private subnets, verify that a NAT gateway is associated with the subnet's route table. The NAT gateway creates an internet path to reach the Amazon S3 endpoint. If you use a VPC endpoint for Amazon S3, then check the route table that's associated with your VPC for the Amazon S3 gateway endpoint. Verify that the route table includes a route for the Amazon S3 AWS managed prefix list that directs traffic to the Amazon S3 gateway endpoint. This route makes sure that requests to Amazon S3 aren't sent over the public internet.

When you use an Amazon S3 interface endpoint, verify that the security groups that are attached to the endpoint allow inbound HTTPS traffic on port 443. Also, make sure that the endpoint's private DNS setting and the Enable DNS hostname and Enable DNS support DNS settings of the VPC are activated. Finally, verify that the application uses the s3.region.amazonaws.com endpoint, not the s3.amazonaws.com global endpoint.

Test network connectivity from Amazon ECS tasks to Amazon S3 endpoints

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Verify that your network configuration, including firewalls and security groups, allows traffic to the Amazon S3 endpoints.

Note: You can use ECS Exec to run the following commands.

To test the HTTPS connection to the Amazon S3 endpoint on port 443, run the following telnet or curl command:

telnet s3.region-code.amazonaws.com 443

curl -v https://s3.region-code.amazonaws.com

Note: Replace region-code with your Region. You might need to install the telnet command.

To test whether the container instance can authenticate and perform Amazon S3 operations, run the ls AWS CLI command:

aws s3 ls

If the preceding test fails, then review your security groups, network ACLs, and firewall rules to check for blocked ports.

Related information

How do I troubleshoot 403 Access Denied errors from Amazon S3?

How can I access other AWS services from my Amazon ECS tasks on Fargate?

How do I troubleshoot a connection error when I run the "cp" or "sync" commands on my Amazon S3 bucket?