We are trying to read a CSV file to process the data using AWS Glue and we are getting an error message as below:
Py4JJavaError: An error occurred while calling o91.schema.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 4 times, most recent failure: Lost task 0.3 in stage 3.0 (TID 8) executor 3): com.amazonaws.services.glue.util.FatalException: Unable to parse file:
We have already verified the codification, it is UTF-8 so there should be no problem. When we read the CSV using from_options function it was able show the data but printSchema() raised an error. We could not read the data using from_catalog function.
Appreciate any help on resolving this