AWS Glue Job detects schema changes, but they don't appear in Redshift.

0

Hello, we have an S3 bucket with various CSV files and an AWS Glue crawler to update the Data Catalog and finally an AWS Glue job to move the data to RedShift. The handling of data and target table is set to drops and recreate target table.

When we add a new column to the original S3 files (in this example col5), the crawler does detect the new schema, and the output schema of the job also detects the new schema in every node (as shown in the following images):

Crawler schema

Glue visual ETL job and output schema

However, when we run the Glue job, the new column won't appear in the RedShift database (next image). It does detect new rows, but not the new columns.

RedShift table

We noticed that if we start a data preview session and run the job again, the new column does show in RedShift.

Thanks in advance.

Isabel
asked 8 months ago121 views
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions