Can Glue Spark timezone be changed?

0

We have upstream systems that all use Central US time zone, but our pyspark/sparkSQL jobs in Glue is UTC and current_timestamp() is giving UTC time. Can we direct glue to use a different timezone? We tried adding a configuration to SparkConf: ("spark.sql.session.timeZone", "America/Chicago")

We also tried adding --java-options -Duser.timezone="America/Chicago" from the dashboard Run with Parameters feature.

Neither had the effect of updating spark's timezone. Any help here?

質問済み 1年前967ビュー
1回答
0

A timestamp doesn't have a timezone, by definition is based on UTC. The timezone you are configuring comes into play when you parse a date or you format that timestamp into a string.

If you do a "show()" on a timestamp column, you should see it in the timezone configured, if not maybe it's not correctly configured, notice that properly spark.sql.session.timeZone has to be set for SparkSession, not context.

profile pictureAWS
エキスパート
回答済み 1年前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ