- Newest
- Most votes
- Most comments
There doesn't appear to be a documented hard limit on the maximum number of columns in an Iceberg table in AWS Athena. Your experimental findings suggest that while you can create and insert into tables with up to 7500 columns, you're encountering issues with update operations at this scale.
The error "ATHENA INTERNAL_ERROR_QUERY_ENGINE" you're experiencing with updates on very wide tables (7500+ columns) suggests you may be hitting an implementation limit in Athena's query engine when processing update operations on tables with an extremely high column count.
This appears to be a practical limitation rather than a documented restriction. Iceberg schema updates are designed to be metadata-only changes that don't modify data files, but the sheer number of columns may be exceeding what Athena's implementation can efficiently process during update operations.
For optimal performance and reliability, you might want to consider restructuring your data model to use fewer columns, perhaps by using nested structures (like structs) where appropriate, though be aware that there are some limitations with operations like partitioning on nested fields in Athena's current Iceberg implementation.
Sources
Evolve Iceberg table schema - Amazon Athena
S3 tables schema evolution and data loss | AWS re:Post
Athena Iceberg partitioning on nested ROW fields | AWS re:Post
Relevant content
- AWS OFFICIALUpdated 6 months ago
