How to read the schema only from Glue Data Catalog

0

Hi Team,

I am using below code and its giving me the columns from the data, but my expectation to get the columns from Glue Data Catalog. glueContext.create_dynamic_frame.from_catalog(database = "xyz", table_name = "abc")

Step 1 : Kept 5 columns in a file in S3 and ran the crawler. Glue Data Catalog captured all 5 columns as schema Step 2 : Executed a Glue Pyspark script job with glueContext.create_dynamic_frame.from_catalog and loaded data Iceberg table with merge statement. Step 3 : Removed one column from step1 and Kept 4 columns in a file in S3 as delta file(cdc) and ran the crawler. Glue Data Catalog captured all 4 columns Step 4 : Executed the same Pyspark script job with glueContext.create_dynamic_frame.from_catalog and the job failed with an error : AnalysisException: cannot resolve colunm_name in MERGE command.

Could you please help on this

Thank you in advance !!

Bharath
asked 9 months ago83 views
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions