AWS Glue Job detects schema changes, but they don't appear in Redshift.

0

Hello, we have an S3 bucket with various CSV files and an AWS Glue crawler to update the Data Catalog and finally an AWS Glue job to move the data to RedShift. The handling of data and target table is set to drops and recreate target table.

When we add a new column to the original S3 files (in this example col5), the crawler does detect the new schema, and the output schema of the job also detects the new schema in every node (as shown in the following images):

Crawler schema

Glue visual ETL job and output schema

However, when we run the Glue job, the new column won't appear in the RedShift database (next image). It does detect new rows, but not the new columns.

RedShift table

We noticed that if we start a data preview session and run the job again, the new column does show in RedShift.

Thanks in advance.

Isabel
gefragt vor 9 Monaten124 Aufrufe
Keine Antworten

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen