HIVE_UNSUPPORTED_FORMAT: Unable to create input format

0

backgoround: querying cloudtrail logs via athena, however the database and table table is created using glue crawler but running into error HIVE_UNSUPPORTED_FORMAT: Unable to create input format

posta un anno fa1226 visualizzazioni
1 Risposta
0

Hello,

Please note that there are multiple causes of this error.

a. The AWS Glue crawler can't classify the data format

  1. The data is stored in Amazon Simple Storage Service (Amazon S3).

  2. You run an AWS Glue crawler with a built-in classifier to detect the table schema. The crawler returns a classification of UNKNOWN. c. At least one column is detected, but the schema is incorrect.

When you query the table from Athena, the query fails with the error "HIVE_UNKNOWN_ERROR: Unable to create input format".

Resolution -

Use a data type that is supported by a built-in classifier. If the data format can't be classified by a built-in classifier [1], then consider using a custom classifier [2].

============

b. Athena doesn't support the data format

  1. The data is stored in Amazon S3.

  2. You run a crawler to create the table. The crawler classifies the table in a format that Athena doesn't support, such as ion or xml.

  3. When you query the table from Athena, the query fails with the error "HIVE_UNKNOWN_ERROR: Unable to create input format".

Resolution -

To resolve this error, use a data format that Athena supports.

============

c. One or more of the AWS Glue table definition properties are empty

  1. The AWS Glue table isn't created in Athena or by an AWS Glue crawler. The table is created using any other method. For example, the table is created manually on the AWS Glue console.

  2. When you query the table from Athena, the query fails with the error "HIVE_UNKNOWN_ERROR: Unable to create input format".

This error occurs because one or more of the following properties in the AWS Glue table definition are empty:

  1. Input format

  2. Output format

  3. Serde name

Resolution -

Confirm that these properties are set correctly for the SerDe and data format. Keep in mind that the SerDe that you specify defines the table schema. The SerDe can override the DDL configuration that you specify in Athena when you create your table.

To update the table definition properties, do the following:

  1. Open the AWS Glue console.

  2. Select the table that you want to update.

  3. Choose Action, and then choose View details.

  4. Choose Edit table.

  5. Update the settings for Input format, Output format, or Serde name.

  6. Choose Apply.

References:

[1] https://docs.aws.amazon.com/glue/latest/dg/add-classifier.html#classifier-built-in

[2] https://docs.aws.amazon.com/glue/latest/dg/add-classifier.html#classifier-defining

AWS
TECNICO DI SUPPORTO
con risposta un anno fa

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande