Configure AWS Glue Spark shuffle plugin with Amazon S3 in the code

Question

Can I specify the S3 bucket where you write shuffle files with the "AWS Glue Spark shuffle plugin with Amazon S3" (https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-shuffle-manager.html) also in the (PySpark) code of my Glue job or only via the job parameter `--conf spark.shuffle.glue.s3ShuffleBucket=s3://`. It looks like
```python
spark_config = pyspark.conf.SparkConf()
spark_config.set("spark.shuffle.glue.s3ShuffleBucket", f"s3://{shuffle_data_bucket}/")
...
spark_context = pyspark.context.SparkContext(conf=spark_config)
glue_context = awsglue.context.GlueContext(spark_context)
```
does not do the job as with other Glue/Spark settings.

Accepted Answer

I found that it works as explaned here: https://docs.aws.amazon.com/glue/latest/dg/cloud-shuffle-storage-plugin.html.
```python
spark_config.set("spark.shuffle.storage.path", f"s3://{shuffle_data_bucket}/")
spark_config.set("spark.shuffle.sort.io.plugin.class", "com.amazonaws.spark.shuffle.io.cloud.ChopperPlugin")
```

Configure AWS Glue Spark shuffle plugin with Amazon S3 in the code

相關內容