Command failed with exit code 10

Question

Code works in glue notebook but fails in glue job ( tried both glue 3.0 and 4.0)
The line where it fails is,

`df.toPandas().to_csv(,index=False)`

no detail message in glue logs

`2023-05-19 19:08:37,793 ERROR [main] glue.ProcessLauncher (Logging.scala:logError(73)): Unknown error from Python: Error Traceback is not available.`

Obviously the data frame is large as 500MB specifically, but it succeeds in glue notebook. Wondering if there is any subtle differences internally in glue notebook and glue job that is not obvious or some kind of bug.
P.S: write to S3 using Databricks technology works too.

Accepted Answer

More likely this error can happen when dealing with large datasets while they move back and forth between Spark tasks and pure Python operations. Data needs to be serialized between Spark's JVMs and Python's processes. So, in this regard my suggestion is to consider processing your datasets in separate batches. In other words, process less data per Job Run so that the Spark-to-Python data serialization doesn't take too long or fail.

I can also understand that, your Glue job is failed but the same code is working in Glue notebook. But, in general there is no such difference when using spark-session in Glue job or Glue notebook. To compare, you can run your Glue notebook as a Glue job. To get more understanding of this behavior, I would suggest you to please open a support case with AWS using the [link here](https://console.aws.amazon.com/support/home#/case/create)

Further, you can try below workaround for **df.toPandas()** using below spark configuration in Glue job. You can pass it as a key value pair in Glue job parameter.

> Key : --conf

> Value : spark.sql.execution.arrow.pyspark.enabled=true, --conf spark.driver.maxResultSize=0

Answer

Excellent response. I was able to get around the issue by adding the spark configuration/Glue job  parameter --conf mentioned. Thanks a lot.

Answer

Very likely you are running out of memory by converting toPandas(),  why don't you just save the csv using the DataFrame API?, even if you coalesce it to generate a single file (so it's single thread processing), it won't run out of memory.

Command failed with exit code 10

Relevant content