Athena engine v3 cannot read any Iceberg v2 table written by Glue 5.1 / Spark 3.5.6

Question

Hi,

I am trying to integrate Athena with iceberg tables created by my ingestion pipeline in Glue 5.1. I am encountering INTERNAL_ERROR_QUERY_ENGINE error and no matter what i try this error  would not resolve. I am however able to query iceberg using spark so i don't believe the  iceberg table files to be invalid or corrupt.

Details are as below:

Region: ap-southeast-5
Workgroup: mdp-poc-analytics
Engine version: Athena engine version 3 (selected + effective)
Service: Amazon Athena (engine v3 / Trino 435 Iceberg integration)

SYMPTOM
Every query against any table in the mdp_poc_bronze database (Iceberg v2, written by AWS Glue 5.1 / Spark 3.5.6-amzn-1) fails with:
  - [ErrorCode: INTERNAL_ERROR_QUERY_ENGINE]
  - ErrorType: 234
  - DataScannedInBytes: 0
  - EngineExecutionTimeInMillis: ~80,000 (deterministic 80-second timeout)
  - Retryable: false

The same failure occurs for all query types: SELECT COUNT(*), raw SELECT, all metadata views ($history, $snapshots, $manifests, $partitions, $files). Trivial queries against the same workgroup succeed in 132ms: SELECT 1, SHOW TABLES IN mdp_poc_bronze.

CONTROL: SPARK CAN READ THE TABLE
The same Iceberg table is readable from Spark 3.5.6-amzn-1 in 81 seconds via the same glue_catalog catalog-impl. The Spark job reads the full table, prints the schema, and reads all 13 snapshots / 13 history / 13 manifest entries without error. This proves the table is well-formed at the Iceberg layer and the bug is in Athena engine v3's metadata-resolution code path.

REPRODUCER
1.  Create a Glue 5.1 / Spark 3.5.6-amzn-1 job that writes an Iceberg v2 table to S3 using the AWS-managed Glue catalog (GlueCatalog catalog-impl): --conf spark.sql.catalog.glue_catalog=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.glue_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
2.  Use microsecond-precision timestamp columns (Iceberg timestamptz type)
3.  Add a partition spec with day-transformed timestamp columns (e.g., partitionedBy(F.days("ts"))) (1/5)
4.  Run a Glue job that performs one or more append operations via df.writeTo(...).append()
5.  After the table is created, query it from any Athena workgroup on engine v3 in the same region

Every query fails with the symptoms above.

PROOF ARTIFACTS
Failed query IDs (user-side, original $history query):
  - 1a806f97-5fc3-466a-96d7-1b11e16147ae (history, 64s)
  - 14cdd92c-b8a5-4e54-b017-ac38c396a944
  - 6295dcba-366e-44b5-9d94-aac7235c9d67
  - 7d68afb4-d3b3-437c-8098-95fb1100e013
  - 1adec2d8-8080-4a05-9601-1ecfad5488af
  - 3b371722-de2e-4253-b005-a0b81595b8cc
  - 8b83d47c-fdd2-4be3-ba2a-fe66de7268d7
  - 8c005a85-16ac-45d1-96fe-194abf7a5b85
  - 23a3b278-3031-451d-97c2-6c11cca4c1eb
  - 0fb9b592-c639-4884-a85f-33123b3cd59b
  - bb191d66-7305-4013-8acb-fcf9bfa33c4e
  - c673d59a-9dd5-47da-af1a-5481de0c10c1
  - 07399c4f-6ced-4205-baf1-545ed12b5048
  - ff840015-87ec-4798-a58b-9a351c8a1b39
  - 23f07220-12c8-45f0-8414-5b64625d5559
  - 1141b2a0-f183-40de-8cde-c34985e1cd73
  - 53dfa08c-2bd3-416e-944a-8ab5627bbdd3
  - 7ef87351-2d98-45f2-ad4f-22c456747021

Failed query IDs (deploy-role, post-permission-fix):
  - 862ada18-45b8-46a0-b620-7378efaf205f ($snapshots, 64s, LF AccessDenied initially)
  - 1d882eb6-394d-42f6-976c-c8079c83af0c ($snapshots, 64s after LF grant)
  - 9ad53368-d16d-421e-9cc9-ea3c1b49c328 ($manifests, 65s)
  - 07d9b2ba-33f7-49b6-804a-07469332d943 ($partitions, 65s)
  - ba53ec24-585e-48dc-b2c6-90e3d147d397 ($files, 65s)
  - c02efc87-edbe-43b5-a631-84ff34176423 (raw SELECT, 80s)
  - 11cd64e2-ccbb-4969-877a-794707a28fe7 (raw SELECT on FRESH table, 80s)
  - acaa4fe1-f7e6-4205-bdce-e08e492d5df6 (raw SELECT on FRESH table, 80s)
  - 6ff0093a-ef69-4fb6-ab2d-19c8704628a2 (raw SELECT on Athena-DDL table, 80s)
  - 230ce37d-8363-4364-b0ab-f14d7cc1fecb (raw SELECT on Athena-DDL table, 80s)
  - 046212c4-146d-4359-9667-d4049c16fe43 ($history, 64s)
  - d4a19d00-f247-41d0-b164-5654da58a2c9 (raw SELECT on hybrid-written table, 80s) (2/5)
  - 61a1681c-4ccf-45db-a387-5b25359bd3a9 (raw SELECT on empty Athena-DDL table, 80s)

Successful Spark read of the same table (Glue job run):
  - jr_24d68e0f0e0a5a17bca86aaa89159bb121eb7113a74291e12891dd3b38c5c6dd (mdp-poc-iceberg-reader-smoke, 81s, all 6 tests passed)

KEY OBSERVATIONS
The bug is independent of:
  - metadata history length (fresh table with 2 snapshots fails identically to 13-snapshot table)
  - schema transitions (single schema vs schema-id 0→1 transition both fail)
  - LF grants (deploy role with lakeformation:GetDataAccess clears the gate, then fails identically)
  - LF registration (same query against non-LF-registered table succeeds)
  - microsecond vs nanosecond precision (all timestamps are microsecond, confirmed via PyArrow)
  - partition spec (empty spec and day-transformed spec both fail)
  - writer config (with/without handle-timestamp-without-timezone and table-default.format-version=2 both fail)
  - catalog storage descriptor completeness (empty Athena-DDL table fails identically to worker-created table)
  - deprecated table properties (after migrating write.object-storage.path to write.data.path, the same failure persists)

Takahiro Nozaki · Answer

The official Athena documentation "[Query Apache Iceberg tables - Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html)" explicitly states the following:

> Iceberg version support – Athena supports Apache Iceberg version 1.4.2.

Glue 5.1 is a relatively new version (released in 2025), and the Iceberg library used in conjunction with Spark 3.5.6 likely utilizes an internal version newer than 1.4.2 (such as the 1.6.x or 1.7.x series).

This aligns very well with the results of the isolation tests you conducted:

Tables not registered with Lake Formation → Success (Direct access path to the Glue Catalog)
Tables registered with Lake Formation → Consistent failure after a 64–80 second timeout (Metadata resolution path via Lake Formation)
Readable without issues from Spark (using the Iceberg 1.6/1.7 series reader implementation)

In other words, the hypothesis that "the Athena v3 Iceberg reader (based on the somewhat older 1.4.2 version) cannot correctly interpret metadata structures written by the newer Iceberg library (such as new manifest formats, partition statistics, or files related to puffin/deletion vectors)" is highly consistent with your experimental results.

* Why does behavior differ based on Lake Formation registration?

There is another relevant statement in the official documentation:

> Iceberg restricted metadata – Lake Formation does not evaluate the Iceberg metadata tables. Hence, the Iceberg metadata tables are restricted if there are any Lake Formation row or cell filters present on the base table or if you do not have permissions to view all columns in the base table.

While this does not appear to be the direct cause (since your case results in `INTERNAL_ERROR_QUERY_ENGINE` rather than `AccessDeniedException`), it confirms that the query path via Lake Formation follows a different metadata resolution code path than direct access to the Glue Catalog. As you have verified, the fact that the failure persists even after correctly granting LF permissions strongly suggests that the issue is not permission-related; rather, it is highly likely that a specific code path within LF itself is failing to parse the new Iceberg metadata structure.

* Why the consistent 64–80 second duration?

This consistent (deterministic) timeout duration suggests that the issue is not merely network latency or throttling. Instead, it implies that an internal process—such as a retry loop or a recursive metadata resolution routine—is encountering a structure it cannot parse, resulting in an endless loop (or a fixed number of retries) that ultimately manifests as a timeout. This pattern is characteristic of an internal hang or crash caused by version incompatibility.

Collin · Answer

ok understood. So basically my most viable option would be to downgrade glue to a version that have the iceberg metadata format that is supported by Athena v3 engine and Lake Formation? which would mean that i would loose the ability to use materialized views since it is only in glue 5.1?

Collin

answered 2 months ago

Takahiro Nozaki · Answer

* Athena v3 itself is capable of reading that Iceberg version (as evidenced by the fact that it works successfully with tables not registered in Lake Formation).
* The issue lies solely in the "Lake Formation integration" aspect; specifically, the separate metadata resolution mechanism used for Lake Formation-registered tables is failing to properly handle that version's Iceberg metadata structure.

Therefore, it is more accurate to understand the situation not as "Athena v3 being incompatible with that Iceberg version," but rather as "the combination of Athena v3 and Lake Formation not functioning correctly with that version's metadata."

Collin · Answer

So in a nutshell, Athena v3 engine does not support the iceberg version that is tied to glue 5.1?

Athena engine v3 cannot read any Iceberg v2 table written by Glue 5.1 / Spark 3.5.6

Add your answer

Relevant content