AWS Glue Job detects schema changes, but they don't appear in Redshift.

0

Hello, we have an S3 bucket with various CSV files and an AWS Glue crawler to update the Data Catalog and finally an AWS Glue job to move the data to RedShift. The handling of data and target table is set to drops and recreate target table.

When we add a new column to the original S3 files (in this example col5), the crawler does detect the new schema, and the output schema of the job also detects the new schema in every node (as shown in the following images):

Crawler schema

Glue visual ETL job and output schema

However, when we run the Glue job, the new column won't appear in the RedShift database (next image). It does detect new rows, but not the new columns.

RedShift table

We noticed that if we start a data preview session and run the job again, the new column does show in RedShift.

Thanks in advance.

Isabel
已提問 9 個月前檢視次數 124 次
沒有答案

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南