AWS Glue create_dynamic_frame.from_catalog Failed to find data source: UNKNOWN

Question

Here is my code snippet:

[transaction_DyF = glueContext.create_dynamic_frame.from_catalog(
                               database = source_db,
                               table_name = source_tbl,
                               push_down_predicate = pushDownPredicate,
                               transformation_ctx = source_tbl)

Error Message:

Error: An error occurred while calling o103.getDynamicFrame.
: java.lang.ClassNotFoundException: Failed to find data source: UNKNOWN

Answer

Calling the `create_dynamic_frame.from_catalog` is supposed to return a dynamic frame that is created using a Data Catalog database and table provided. However, in this case it is likely that the table input classification is UNKNOWN, when the crawler detects no pattern in files to classify it, a table with a classification UNKNOWN is created.

Check the table properties in Data Catalog for classification. See below link on creating a custom classifier.

* https://aws.amazon.com/premiumsupport/knowledge-center/glue-crawler-unknown-classification/ 
* https://docs.aws.amazon.com/glue/latest/dg/console-classifiers.html

AWS Glue create_dynamic_frame.from_catalog Failed to find data source: UNKNOWN

Relevant content