AWS Glue visual job

Question

Hi
I was trying to work on a simple problem where I am taking data from 3 csv files in s3(source) and after combining them, I am appending them to a table in postgre sql where my database is by establishing a connection in my glue catalog with that database.  But I am facing an error stating that this table already exist and the data is not getting attached to it. Kindly help me if there is anything wrong with this approach

Shubham_P · Answer

Hello,

There is not any issue in above approach. You can read the data from multiple sources and can perform transformation like join and then write to the existing relational database table without any issue.

By default Glue dynamic frame use append mode and add the data to an existing table. Make sure that all the columns should be present in the database table before as job would not be able to add new column and can fail with column not found in schema.

I would suggest please verify the schema of your data after join operation with the database table schema.

If you still face any issue, we require your sample data and script. Hence Please feel free to reach out to AWS Premium Support with sample data and script and we will be happy to help.

AWS Glue visual job

Relevant content