AWS Glue Job detects schema changes, but they don't appear in Redshift.

0

Hello, we have an S3 bucket with various CSV files and an AWS Glue crawler to update the Data Catalog and finally an AWS Glue job to move the data to RedShift. The handling of data and target table is set to drops and recreate target table.

When we add a new column to the original S3 files (in this example col5), the crawler does detect the new schema, and the output schema of the job also detects the new schema in every node (as shown in the following images):

Crawler schema

Glue visual ETL job and output schema

However, when we run the Glue job, the new column won't appear in the RedShift database (next image). It does detect new rows, but not the new columns.

RedShift table

We noticed that if we start a data preview session and run the job again, the new column does show in RedShift.

Thanks in advance.

Isabel
preguntada hace 9 meses124 visualizaciones
No hay respuestas

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas