Glue Catalog Delta Tables: Error "TABLE_REDIRECTION_ERROR" and "INVALID_GLUE_SCHEMA"

1

Hello everyone,

1. Context

We have a Delta Lake where we write our tables in S3 in Delta format, and we use the Glue Catalog for queries in Athena. Tables are created both with and without the use of Glue crawlers. The crawlers create them using the Delta Native Table format.

2. Problems

Our pipelines have been running smoothly for months. However, since September 15, 2023 (last Friday), we have started to receive several errors in Athena queries. In summary, there are two prominent errors:

  1. TABLE_REDIRECTION_ERROR
  2. INVALID_GLUE_SCHEMA

Regarding the INVALID_GLUE_SCHEMA error, the first occurrence we have recorded was on September 15, 2023, at 16:35:59.552.

INVALID_GLUE_SCHEMA: Delta Lake table schema in Glue does not match the most recent schema of the Delta Lake transaction log. Please ensure that you have the correct schema defined in Glue.

As for the TABLE_REDIRECTION_ERROR error, the first occurrence we have recorded was on September 15, 2023, at 22:46:52.188.

TABLE_REDIRECTION_ERROR: Table 'awsdatacatalog.silver.core_customers' redirected to 'awsdatacatalog$delta-lake-aws.silver.core_customers', but the target table 'awsdatacatalog$delta-lake-aws.silver.core_customers' does not exist

For an error to occur, a simple query like "SELECT * FROM catalog.table LIMIT 10" is sufficient. Sometimes the query is executed successfully, while other times it results in an error. We didn't make any changes on our end during the creation of the Delta tables.

However, we are noticing an increase in the frequency of this type of error. Currently, it is necessary to execute the query about five times for only one of them to work correctly.

3. Attachments

Enter image description here

Can someone help us? Thank you.

preguntada hace 8 meses510 visualizaciones
1 Respuesta
0

Hi!

For this issue is better to open a support ticket for Amazon Athena. They can see internally how the query is run and see why sometimes the queries run and sometimes not.

Bests!

profile pictureAWS
respondido hace 8 meses
profile picture
EXPERTO
revisado hace 5 meses

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas