Glue ETL job to load data from S3 to Redshift failed

0

I am running a Glue ETL job to load data from parquet file from S3 into Redshift table. Enter image description here

I am able to get data preview while configuring connections but job fails with below error.

Error Category: UNCLASSIFIED_ERROR; Failed Line Number: 31; An error occurred while calling o112.pyWriteDynamicFrame. Exception thrown in awaitResult:

Please let me issue and provide solution.

Ani
asked 5 months ago322 views
2 Answers
1
Accepted Answer

I would recommend you switch to the script and edit it to add a few things:

  1. print out the schema of the dynamic frame of the source. See if that matches your Redshift schema.
  2. Look for nested fields/structures - you might need a Resolve choice
  3. use an Apply mapping transform if your schema doesn't match Redshift
  4. check permissions of the IAM role of your job
answered 5 months ago
profile picture
EXPERT
reviewed 5 months ago
  • Thank you. Issue was in schema difference between Source and Redshift.

0

The data preview won't really insert into Redshift, which is what is really failing, likely a permission issue.
Look in the log for a cause, also looking into the Redshift error table might help (if it got that far): https://docs.aws.amazon.com/redshift/latest/dg/r_STL_LOAD_ERRORS.html

profile pictureAWS
EXPERT
answered 5 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions