2 個答案
- 最新
- 最多得票
- 最多評論
1
That sounds like a detect, when using the same Delta sink on a regular non streaming job, the code looks like the second one you describe. Will open a ticket to the streaming team.
0
No, it is correct. In the same Visual Job add a parent data node (Kafka in my case). Once this is in place you will see that the script changes to something like:
s3 = glueContext.write_dynamic_frame.from_options(
frame=df,
connection_type="s3",
format="delta",
connection_options={
"path": "s3://...",
"partitionKeys": [],
},
format_options={"compression": "snappy"},
transformation_ctx="s3",
)
I made the same test you did and is how discovered that this version works:
S3bucket_node3_df.write.format("delta").options(**additional_options).mode("append").save()
已回答 10 個月前
The streaming is treating delta (as well as hudi and iceberg) like one of the basic formats; anyway, the bug is reported
相關內容
- AWS 官方已更新 1 年前
- AWS 官方已更新 2 年前
- AWS 官方已更新 2 年前
I've run into a very similar error trying to write to an iceberg table from a Glue streaming job.