How do I troubleshoot the "Unable to execute HTTP request... connect timed out" error in AWS Glue?
My AWS Glue job fails with the error "Unable to execute HTTP request... connect timed out".
Short description
When you run an AWS Glue job with a connection attached to it, it might fail when it tries to connect to another service endpoint. For example, it might try to connect to a service endpoint such as Amazon Simple Storage Service (Amazon S3) and AWS Key Management Service (AWS KMS). The job fails if the networking route to this service endpoint is missing in the subnet that's used in the AWS Glue connection. The "Unable to execute HTTP request" or "failed: connect timed out" error message appears in AWS Glue job logs.
You get this error for the following reasons:
- The subnet that's configured in your AWS Glue connection contains an internet gateway or no route in the route table for the failed service endpoint.
- When you run your AWS Glue job, it interacts with the AWS Glue APIs. This happens if you use any operations that require listing AWS Glue Data Catalog tables, creating new tables, or reading from one. When your job needs to access the AWS Glue API, it sends an HTTP REST API request. Similarly, you might make other service API calls to Amazon S3, Amazon Simple Notification Service (Amazon SNS), or AWS Secret Manager. When you make the API call, then the job tries to connect with these service endpoints and the request fails with a timed-out error. This happens because although the job is able to send the request, it didn't receive the response within the timeout period.
AWS Glue creates elastic network interfaces in the subnet specified in the AWS Glue connection. For more information, see Setting up network access to data stores.
Resources in AWS Glue jobs have private IP addresses only. This means that jobs are unable to connect to other resources using an internet gateway. The AWS Glue job tries to send requests to the public AWS service endpoint, but these requests come from a private IP address. So the endpoint is unable to send a response back and the job fails with "connect timed out" error message.
Resolution
The error message that you find in logs looks similar to one of following, depending on your use-case:
AWS Glue REST API:
AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to glue.ap-southeast-1.amazonaws.com:443
Amazon S3 request:
Unable to execute HTTP request: Connect to s3BucketName.us-east-1.amazonaws.com:443 [s3BucketName.s3.us-east-1.amazonaws.com/xx.xxx.xx.xxx ] failed: connect timed out
AWS KMS request:
Unable to execute HTTP request: Connect to kms.us-east-1.amazonaws.com:443 [kms.us-east-1.amazonaws.com/xx.xxx.xx.xxx ] failed: connect timed out
Confirm that the traffic between the AWS Glue job and the service that the job is trying to make an API request to is allowed. You can check this by using a virtual private cloud (VPC) endpoint, or by using a NAT gateway.
Using a VPC endpoint
Use the Amazon Virtual Private Cloud (Amazon VPC) console to create an interface VPC endpoint for the service that you get the error for. For example, the AWS Glue job might fail and throw an error while trying to access the AWS Glue API. Create the interface endpoint for AWS Glue in the same VPC subnet that's configured for your AWS Glue connection. This confirms that the traffic that's supposed to connect to the AWS Glue endpoint does so using this interface.
- Open the Amazon VPC console.
- In the navigation pane, choose Endpoints.
- Choose Create endpoint.
- For Service category, choose AWS services.
- For Service name, choose the service that you're connecting to. For more information, see AWS services that integrate with AWS PrivateLink.
- For VPC, choose the VPC that you want to access the AWS service from.
- To create an interface endpoint for Amazon S3, you must clear Additional settings and Enable DNS name. This is because Amazon S3 doesn't support private DNS for interface VPC endpoints.
- For Subnets, choose one subnet per Availability Zone (AZ) that you want to access the AWS service from.
- For Security group, choose a security group to associate with the endpoint network interfaces. The security group rules must allow resources that will use the VPC endpoint. The VPC endpoint communicates with the AWS service, which then communicates with the endpoint network interface.
- For Policy, choose Full access. This allows all operations by all principals on all resources over the VPC endpoint. Or, choose Custom to attach a VPC endpoint policy. This policy controls the permissions that principals have to perform actions on resources over the VPC endpoint. This option is available only if the service supports VPC endpoint policies. For more information, see VPC endpoint policies.
- (Optional) To add a tag, choose Add new tag, and then enter the tag key and the tag value.
- Choose Create endpoint.
Using a NAT gateway
An AWS NAT Gateway connects resources privately and securely using the private internet. Create a private subnet and point your AWS Glue connection towards a private subnet. At the same time, create a NAT gateway in one of the public subnets of your AWS Glue VPC. Configure the route table of your AWS Glue connection subnet so that if traffic is routed to internet, it does so using this NAT Gateway. For more information, see How do I set up a NAT gateway for a private subnet in Amazon VPC? and NAT gateways.
Note: If you are using a NAT gateway, then traffic to your service goes through the internet.
Additional troubleshooting
- Check that the connection security group and subnet network ACL allow AWS Glue to send traffic when making the API request.
- To test the connectivity issue, launch an EC2 instance in same VPC, subnet, and security group that you used when creating the AWS Glue connection. Then run the following command to confirm connectivity with the service endpoint:
telnet <glue endpoint url> 443 dig <glue endpoint url>
Examples:
$ telnet glue.us-east-1.amazonaws.com 443 $ dig glue.us-east-1.amazonaws.com
Related information
Relevant content
- asked a year agolg...
- Accepted Answerasked 2 years agolg...
- Accepted Answerasked a year agolg...
- asked 8 months agolg...
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 10 months ago
- AWS OFFICIALUpdated 2 years ago