- Newest
- Most votes
- Most comments
I had this exact problem/error.
I solved the problem by upgrading the job to use Glue 3.0 with Spark 3.1. Glue version 2.0 with Spark version 2.4 gets this error. I am using Python as the ETL language. My connection is JDBC to a SQL Server database. The Crawler works just fine so the connection and database and tables are in good order.
Add a little more detail for @enricu :
I have the older configuration running in production but have not touched it. It still runs ok. I was replicating that job in a lower environment to test changes to the ETL. So if you need to make changes to an existing implementation you will probably need to upgrade. This may or may not be impactful depending on your implementation.
Job that errors:
Name <JobName>
IAM role <JobRole>
Type Spark
Glue version 2.0
Python version 3
Spark version 2.4
ETL language python
New Job that does not experience that error:
Name <JobName>
IAM role <JobRole>
Type Spark
Glue version 3.0
Python version 3
Spark version 3.1
ETL language python
Edited by: mystic on Sep 22, 2021 4:40 PM
Right now I have same problems as your. So please tell me if you passed it and how do you do? Thank you.
I'm having the same issue... The thing is that it worked before. I made a few changes to my CF and after that it started failing
After going through all of the logs ("All logs" link in the runs list), I've found the real cause of the issue, which was a typo in SQL statement. Try to go through all of the logs, so you might find more details about this issue
Relevant content
- asked a year ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 3 years ago
In my case, I'm using Glue 4.0 and I'm getting the same problem. However it started occurring when I've tried to consume parquet files, that AWS DMS generated. It works fine for files, that I generate with Python pandas library and dataframe