Problems with Glue writing Iceberg outputs to S3

Hello Thomas,

From your description, it looks like the job is failing at the transformation stage, since it reports failure at ShuffleMapStage. I believe you are aware of the working of a Spark ETL job, where the task is distributed among worker nodes, and when any of your transformation like join() or coalesce() or any functions which require a shuffle of data between these nodes is called, this requires data to be moved and sometimes may fail due to the complexity of the transfer.

I could see you are using G.2X which is quite good for memory intensive operations, however, depending on your use case, you should plan the Worker-type and number of workers. I would encourage you to have a look at the Glue Job Metrics of your job run (Refer this to learn to view your metrics) and check the

1. glue.{driver,.....}.system.cpuSystemLoad
2. glue.{driver,.....}.jvm.heap.usage

to check the overall CPU consumption and memory consumption.

If all your worker nodes have even distribution of CPU and memory, this is good and if they are at around the maximum, i.e for cpu around 0.9 and for memory >50% usage constantly for a long execution time, then you need to scale horizontally or vertically, either by increasing the number of workers (horizontal scaling) or increase the existing configuration to use G.4X, etc (vertical scaling) to support the complexity. I would recommend having a look at Monitoring Jobs for DPU capacity planning for more information on this.

Depending on your script, you might need some optimisations that should be done at code level as well, for example if you are using join() in your code, you should use techniques like Broadcase Hints etc and optimise them.

I hope the above information serves you well. Please feel free to reach out to us with a Support case if you need more assistance in issues such as this.

Thanks for your detailed answer, Shrivathsa!

My issue is, that the monitoring plots do not look suspicious at all to me. Or do I overlook something critical?

In the meanwhile I deactivated the Flex mode and changed to

spark_config = pyspark.conf.SparkConf()
spark_config.set("spark.shuffle.storage.path", f"s3://{shuffle_data_bucket}/")
spark_config.set("spark.shuffle.sort.io.plugin.class", "com.amazonaws.spark.shuffle.io.cloud.ChopperPlugin")
...
spark_context = pyspark.context.SparkContext(conf=spark_config)
glue_context = awsglue.context.GlueContext(spark_context)

but it does not change the behaviour. Also there is no useful hint in the error logs of the job. To me it looks like a shuffle object, which has been written in the first part of the job, can later not be read properly.