How do I troubleshoot the AWS Glue error "VPC S3 endpoint validation failed for SubnetId"?

Short description

You get the "VPC S3 endpoint validation failed for SubnetId" error when the following conditions are true:

• You're using an AWS Glue connection with your AWS Glue job or AWS Glue crawler.
• The subnet that's configured for your AWS Glue connection doesn't have an Amazon Virtual Private Cloud (VPC) endpoint for Amazon Simple Storage Service (Amazon S3) or a NAT gateway set up.

AWS Glue creates elastic network interfaces in the AWS Glue connection's subnet to connect to your data sources and Amazon S3. The elastic network interfaces are assigned a private IP address from the IP address range within the subnet. Because there are no public IP addresses, AWS Glue can't use the internet to directly connect to your data sources or Amazon S3 within your VPC. For more information, see Setting up network access to data stores.

Resolution

Configure your AWS Glue connection's subnet to have an Amazon S3 VPC gateway endpoint or a route to a NAT gateway in the route table. For more information, see Error: Could not find S3 endpoint or NAT gateway for subnetId in VPC.

If your AWS Glue job must access the public internet, confirm that you configured the subnet with a NAT gateway.

Set up an S3 VPC gateway endpoint

Complete the following steps:

1. Open the Amazon VPC console.
2. In the navigation pane, choose Endpoints.
3. Choose Create endpoint.
4. For Service name, select com.amazonaws.us-east-1.s3. 
   Note: Replace us-east-1 with your AWS Region.
5. For the Type column, choose Gateway.
6. For VPC, select the VPC where you want to create the endpoint.
   Note: For Configure route tables, a route to the S3 VPC endpoint is automatically added.
7. For Policy, keep the default option Full Access.
8. Choose Create endpoint.

For more information, see Gateway endpoints.

If you use a resource-based policy, then allow your AWS Glue job or crawler's AWS Identity and Access Management (IAM) role to access the required S3 resources.

Set up a NAT gateway

Before you set up the NAT gateway, you must allocate an Elastic IP address. 

Also, The VPC that you use for the AWS Glue connection must have at least one public subnet. For more information, see Example: VPC with servers in private subnets and NAT.

Complete the following steps:

1. Open the Amazon VPC console.
2. In the navigation pane, choose NAT gateways.
3. Choose Create NAT gateway.
4. For Name, enter the name of the gateway that you want to create.
5. For Subnet, select a public subnet in the VPC that you specified in the AWS Glue connection.
6. For Connectivity type, keep the default option Public.
7. Choose Create NAT gateway.

After you set up the gateway, modify the route table that's associated with the AWS Glue connection's subnet. Then, add a route to the NAT gateway.

Complete the following steps:

1. In the navigation pane, choose Route tables.
2. Select the route table that's associated with the subnet that you want to modify.
3. Choose Actions, and then choose Edit routes.
4. On the Edit routes page, choose Add route.
5. Configure the new route to send all subnet traffic to the NAT gateway:
   For Destination, enter 0.0.0.0/0.
   For Target, choose NAT gateway, and then select your NAT gateway.
6. Choose Save changes.
7. Test your AWS Glue connection.
8. Run your AWS Glue job or AWS Glue crawler again.