AWS Glue Job detects schema changes, but they don't appear in Redshift.

0

Hello, we have an S3 bucket with various CSV files and an AWS Glue crawler to update the Data Catalog and finally an AWS Glue job to move the data to RedShift. The handling of data and target table is set to drops and recreate target table.

When we add a new column to the original S3 files (in this example col5), the crawler does detect the new schema, and the output schema of the job also detects the new schema in every node (as shown in the following images):

Crawler schema

Glue visual ETL job and output schema

However, when we run the Glue job, the new column won't appear in the RedShift database (next image). It does detect new rows, but not the new columns.

RedShift table

We noticed that if we start a data preview session and run the job again, the new column does show in RedShift.

Thanks in advance.

Isabel
질문됨 9달 전124회 조회
답변 없음

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠