1 Answer
- Newest
- Most votes
- Most comments
0
I found that it works as explaned here: https://docs.aws.amazon.com/glue/latest/dg/cloud-shuffle-storage-plugin.html.
spark_config.set("spark.shuffle.storage.path", f"s3://{shuffle_data_bucket}/") spark_config.set("spark.shuffle.sort.io.plugin.class", "com.amazonaws.spark.shuffle.io.cloud.ChopperPlugin")
answered 5 months ago
Relevant content
- asked 2 months ago
- asked 5 months ago
- Accepted Answerasked 2 years ago
- asked 23 days ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated a year ago