2 Risposte
- Più recenti
- Maggior numero di voti
- Maggior numero di commenti
1
That sounds like a detect, when using the same Delta sink on a regular non streaming job, the code looks like the second one you describe. Will open a ticket to the streaming team.
0
No, it is correct. In the same Visual Job add a parent data node (Kafka in my case). Once this is in place you will see that the script changes to something like:
s3 = glueContext.write_dynamic_frame.from_options(
frame=df,
connection_type="s3",
format="delta",
connection_options={
"path": "s3://...",
"partitionKeys": [],
},
format_options={"compression": "snappy"},
transformation_ctx="s3",
)
I made the same test you did and is how discovered that this version works:
S3bucket_node3_df.write.format("delta").options(**additional_options).mode("append").save()
con risposta 10 mesi fa
The streaming is treating delta (as well as hudi and iceberg) like one of the basic formats; anyway, the bug is reported
Contenuto pertinente
- AWS UFFICIALEAggiornata 2 anni fa
- AWS UFFICIALEAggiornata un anno fa
- AWS UFFICIALEAggiornata un anno fa
I've run into a very similar error trying to write to an iceberg table from a Glue streaming job.