Glue Catalog Delta Tables: Error "TABLE_REDIRECTION_ERROR" and "INVALID_GLUE_SCHEMA"

1

Hello everyone,

1. Context

We have a Delta Lake where we write our tables in S3 in Delta format, and we use the Glue Catalog for queries in Athena. Tables are created both with and without the use of Glue crawlers. The crawlers create them using the Delta Native Table format.

2. Problems

Our pipelines have been running smoothly for months. However, since September 15, 2023 (last Friday), we have started to receive several errors in Athena queries. In summary, there are two prominent errors:

  1. TABLE_REDIRECTION_ERROR
  2. INVALID_GLUE_SCHEMA

Regarding the INVALID_GLUE_SCHEMA error, the first occurrence we have recorded was on September 15, 2023, at 16:35:59.552.

INVALID_GLUE_SCHEMA: Delta Lake table schema in Glue does not match the most recent schema of the Delta Lake transaction log. Please ensure that you have the correct schema defined in Glue.

As for the TABLE_REDIRECTION_ERROR error, the first occurrence we have recorded was on September 15, 2023, at 22:46:52.188.

TABLE_REDIRECTION_ERROR: Table 'awsdatacatalog.silver.core_customers' redirected to 'awsdatacatalog$delta-lake-aws.silver.core_customers', but the target table 'awsdatacatalog$delta-lake-aws.silver.core_customers' does not exist

For an error to occur, a simple query like "SELECT * FROM catalog.table LIMIT 10" is sufficient. Sometimes the query is executed successfully, while other times it results in an error. We didn't make any changes on our end during the creation of the Delta tables.

However, we are noticing an increase in the frequency of this type of error. Currently, it is necessary to execute the query about five times for only one of them to work correctly.

3. Attachments

Enter image description here

Can someone help us? Thank you.

질문됨 8달 전510회 조회
1개 답변
0

Hi!

For this issue is better to open a support ticket for Amazon Athena. They can see internally how the query is run and see why sometimes the queries run and sometimes not.

Bests!

profile pictureAWS
답변함 8달 전
profile picture
전문가
검토됨 5달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠