Glue Catalog Delta Tables: Error "TABLE_REDIRECTION_ERROR" and "INVALID_GLUE_SCHEMA"

Hello everyone,

1. Context

We have a Delta Lake where we write our tables in S3 in Delta format, and we use the Glue Catalog for queries in Athena. Tables are created both with and without the use of Glue crawlers. The crawlers create them using the Delta Native Table format.

2. Problems

Our pipelines have been running smoothly for months. However, since September 15, 2023 (last Friday), we have started to receive several errors in Athena queries. In summary, there are two prominent errors:

1. TABLE_REDIRECTION_ERROR
2. INVALID_GLUE_SCHEMA

Regarding the INVALID_GLUE_SCHEMA error, the first occurrence we have recorded was on September 15, 2023, at 16:35:59.552.

INVALID_GLUE_SCHEMA: Delta Lake table schema in Glue does not match the most recent schema of the Delta Lake transaction log. Please ensure that you have the correct schema defined in Glue.

As for the TABLE_REDIRECTION_ERROR error, the first occurrence we have recorded was on September 15, 2023, at 22:46:52.188.

TABLE_REDIRECTION_ERROR: Table 'awsdatacatalog.silver.core_customers' redirected to 'awsdatacatalog$delta-lake-aws.silver.core_customers', but the target table 'awsdatacatalog$delta-lake-aws.silver.core_customers' does not exist

For an error to occur, a simple query like "SELECT * FROM catalog.table LIMIT 10" is sufficient. Sometimes the query is executed successfully, while other times it results in an error. We didn't make any changes on our end during the creation of the Delta tables.

However, we are noticing an increase in the frequency of this type of error. Currently, it is necessary to execute the query about five times for only one of them to work correctly.

3. Attachments

Can someone help us? Thank you.