UPSERT RDS tables using AWS Glue Studio Visual and alternatively using Notebook
Similar to RedShift or Snowflake tables is there a way to perform UPSERT for RDS DBs or non RS/SF DB/tables using Glue Visual? I know Spark Dataframe through JDBS connections only support Insert / Overwrite, have seen multiple re:Post on this topic, but all are couple of years old. are there any updates to this functionality in the recent times?
If not with Glue Visual, using Notebook, I see an article (https://medium.com/@thomaspt748/how-to-upsert-data-into-relational-database-using-spark-7d2d92e05bb9) explaining how to achieve this using broadcast variables and foreachPartition call approach, is there any better way or alternative to this approach?
- Newest
- Most votes
- Most comments
Not out of the box (with the exceptions you mention), you normally do that writing into a temporary table and then issuing a MERGE/UPSERT query into the final table using a separate JDBC connection (so you need a custom code node).
Relevant content
- asked 2 years ago
- asked 8 months ago
AWS OFFICIALUpdated 2 months ago- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 3 years ago
