Can I specify the S3 bucket where you write shuffle files with the "AWS Glue Spark shuffle plugin with Amazon S3" (https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-shuffle-manager.html) also in the (PySpark) code of my Glue job or only via the job parameter `--conf spark.shuffle.glue.s3ShuffleBucket=s3://`. It looks like ```python spark_config = pyspark.conf.SparkConf() spark_config.set("spark.shuffle.glue.s3ShuffleBucket", f"s3://{shuffle_data_bucket}/") ... spark_context = pyspark.context.SparkContext(conf=spark_config) glue_context = awsglue.context.GlueContext(spark_context) ``` does not do the job as with other Glue/Spark settings.

Configure AWS Glue Spark shuffle plugin with Amazon S3 in the code

Can I specify the S3 bucket where you write shuffle files with the "AWS Glue Spark shuffle plugin with Amazon S3" (https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-shuffle-manager.html) also in the (PySpark) code of my Glue job or only via the job parameter --conf spark.shuffle.glue.s3ShuffleBucket=s3://<shuffle-bucket>. It looks like

spark_config = pyspark.conf.SparkConf()
spark_config.set("spark.shuffle.glue.s3ShuffleBucket", f"s3://{shuffle_data_bucket}/")
...
spark_context = pyspark.context.SparkContext(conf=spark_config)
glue_context = awsglue.context.GlueContext(spark_context)

does not do the job as with other Glue/Spark settings.

トピック

分析

タグ

AWS Glue

言語

English

Thomas Mueller

質問済み 5ヶ月前304ビュー

1回答

新しい順
投票が多い順
コメントが多い順

これらの回答は役に立ちましたか？コミュニティがあなたの知識を活用できるように、正解に賛成票を投じてください。

承認された回答

I found that it works as explaned here: https://docs.aws.amazon.com/glue/latest/dg/cloud-shuffle-storage-plugin.html.

spark_config.set("spark.shuffle.storage.path", f"s3://{shuffle_data_bucket}/")
spark_config.set("spark.shuffle.sort.io.plugin.class", "com.amazonaws.spark.shuffle.io.cloud.ChopperPlugin")

Thomas Mueller

回答済み 5ヶ月前

エキスパート

Giovanni Lauria

レビュー済み 2ヶ月前

Configure AWS Glue Spark shuffle plugin with Amazon S3 in the code

関連するコンテンツ