1개 답변
- 최신
- 최다 투표
- 가장 많은 댓글
1
To identify the exact error messages, searching through the ERROR logs with GlueExceptionAnalysisListener typically would lead me to the exact line that errors and the Error message. Please try that and let us know if you are still unable to identify the error
답변함 2년 전
Hi,
I only see that the GlueExceptionAnalysisListener was registered in the error logs. No other exception is logged until the job times out.
2022-10-03 21:44:07,711 INFO [Thread-12] spark.SparkContext (Logging.scala:logInfo(57)): Registered listener com.amazonaws.services.glueexceptionanalysis.GlueExceptionAnalysisListener.
Also, note that I am using MySQL v8.
I started fresh with a new job where the script was generated by Glue Studio. It ran fine without any issues. However, as soon as I added get_connection() call, the job started timing out.
Please refer to Glue connection page where it mentions how to use the Glue Connection https://docs.aws.amazon.com/glue/latest/dg/connection-properties.html#connection-properties-jdbc
Note that MySQL connections are possible for Amazon RDS instances only. If you are using other MySQL database, you dont have to use Glue Connections. You would need to upload the JDBC driver to S3. In the AWS Glue job console, Choose the AWS Glue job and edit the Job details tab, under Libraries > Referenced File Path and enter the S3 URI of JDBC file.
If you are able to get the job to work without connections and the data looks right, you do not have to add a Glue connection.