AWS Glue Spark UI Parsing error

Question

Hi Experts,
I have been experiencing issues with my AWS Glue Pyspark Job and hence enabled SPARK UI logging feature. However, when the job completed, I clicked on the Spark UI tab on Job run and it is failing with the below error information. I am not sure how to take it forward from here. Please share your inputs if any.

We encountered an issue trying to parse your Spark logs.
Spark UI log parsing has failed for an unknown reason.
Error details
ErrorType: RequestLogParsingStatus
ErrorCode: N/A
RequestID: N/A
StatusCode:
FileLocation: >

Answer

Hello. I've recorded this 6 minutes video showing how to get Spark UI working.

In the video I cover all the steps needed including:
* VPC setup
* S3 buckets creation
* Glue connection creation (to ensure no public traffic is used)
* Glue job creation (and association with Glue connection)
* Spark UI usage

Let me know if you want to access the recording.

Please accept this answer. Otherwise, reply this question with more information about your glue job for us to troubleshoot.

References:
https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-ui-jobs.html
https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-ui.html

Regards,
Pablo Silva

AWS Glue Spark UI Parsing error

Conteúdo relevante