AWS Glue is throwing this error - Failed to create any executor tasks

0

Running a Glue Job to write data to a database and read from S3 bucket. The job is failing with below error:

Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:38097) caused by Failed to create any executor tasks

Unable to find the cause. Have tried to print the dataframe, looks fine to me, failing on write statement. Please help.

asked 2 years ago4030 views
1 Answer
0

The error that you are facing might be because your server is unreachable. This type of error generally occurs due to lack of free IP addresses in the subnet that is being used by your Glue job.

Whenever a Glue job is using a connection, Glue creates ENIs within the subnet so as to establish communication between your job and the data source. The executors use these ENIs to communicate among themselves. The number of ENIs created by Glue depends on the number of worker nodes deployed by your job. Each ENI requires an IP address. Therefore on a higher level, there must be enough number of free IP addresses available in your subnet such that each worker node gets one allocated and completes it's task successfully.

So please check the number of free IP addresses available in your subnet. You can refer this https://aws.amazon.com/premiumsupport/knowledge-center/glue-specified-subnet-free-addresses/ for more information.

profile pictureAWS
SUPPORT ENGINEER
Chaitu
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions