Error Category: UNCLASSIFIED_ERROR

Question

I'm trying to run glue job that join 3 mysql tables as data source and insert that data into redshift as data target, but facing this error that says

"Error Category: UNCLASSIFIED_ERROR; An error occurred while calling z:com.amazonaws.services.glue.DynamicFrame.apply. YEAR"

This error happen when I use Glue version 4.0 then I change it to Glue version 3.0 and similar error still happen, that also says

"An error occurred while calling z:com.amazonaws.services.glue.DynamicFrame.apply. YEAR"

Any idea to fix this error?

Thanks in advice.

Answer

Hello,

This seems to a compatibility issue in spark version. Please check if it works in Spark 2 version. Please check the **[execution log](https://docs.aws.amazon.com/glue/latest/dg/monitor-continuous-logging-view.html)** for more insight about this issue. If it is due to changes in Spark 3.1 (glue 3.0+) versions particularly when handling timestamps in parquet files, then you have to add the below configs to correct the timestamp handler in higher version in Job parameters under advanced properties,

```
   key: --conf  
   value: spark.sql.legacy.parquet.int96RebaseModeInRead=CORRECTED --conf spark.sql.legacy.parquet.int96RebaseModeInWrite=CORRECTED --conf spark.sql.legacy.parquet.datetimeRebaseModeInRead=CORRECTED
```

References:

`We recommend to set the following parameters when reading/writing parquet data that contains timestamp columns. Setting those parameters can resolve the calendar incompatibility issue that occurs during the Spark 2 to Spark 3 upgrade, for both the AWS Glue Dynamic Frame and Spark Data Frame. Use the CORRECTED option to read the datetime value as it is; and the LEGACY option to rebase the datetime values with regard to the calendar difference during reading.` More details **[here](https://docs.aws.amazon.com/glue/latest/dg/migrating-version-30.html#migrating-version-30-from-20)**

Error Category: UNCLASSIFIED_ERROR

Relevant content