How to read the schema only from Glue Data Catalog

0

Hi Team,

I am using below code and its giving me the columns from the data, but my expectation to get the columns from Glue Data Catalog. glueContext.create_dynamic_frame.from_catalog(database = "xyz", table_name = "abc")

Step 1 : Kept 5 columns in a file in S3 and ran the crawler. Glue Data Catalog captured all 5 columns as schema Step 2 : Executed a Glue Pyspark script job with glueContext.create_dynamic_frame.from_catalog and loaded data Iceberg table with merge statement. Step 3 : Removed one column from step1 and Kept 4 columns in a file in S3 as delta file(cdc) and ran the crawler. Glue Data Catalog captured all 4 columns Step 4 : Executed the same Pyspark script job with glueContext.create_dynamic_frame.from_catalog and the job failed with an error : AnalysisException: cannot resolve colunm_name in MERGE command.

Could you please help on this

Thank you in advance !!

Bharath
gefragt vor 9 Monaten87 Aufrufe
Keine Antworten

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen