Skip to content

Maximum number of columns in an Iceberg table?

0

Hi all, I am unable to find any documentation about maximum number of columns in an Iceberg table.

By way of experiment, I figured that it is somewhere between 7500 and 8500. Having created a table with 7500 columns with a partition on one of the columns, I am able to successfully insert into the table. The delete also succeeds (DELETE FROM tablex WHERE col1='X').

However, any updates in Athena (update tablex SET col2='Y' WHERE col1='X') are failing with error "ATHENA INTERNAL_ERROR_QUERY_ENGINE".

Any help much appreciated.

asked a month ago100 views
1 Answer
0

There doesn't appear to be a documented hard limit on the maximum number of columns in an Iceberg table in AWS Athena. Your experimental findings suggest that while you can create and insert into tables with up to 7500 columns, you're encountering issues with update operations at this scale.

The error "ATHENA INTERNAL_ERROR_QUERY_ENGINE" you're experiencing with updates on very wide tables (7500+ columns) suggests you may be hitting an implementation limit in Athena's query engine when processing update operations on tables with an extremely high column count.

This appears to be a practical limitation rather than a documented restriction. Iceberg schema updates are designed to be metadata-only changes that don't modify data files, but the sheer number of columns may be exceeding what Athena's implementation can efficiently process during update operations.

For optimal performance and reliability, you might want to consider restructuring your data model to use fewer columns, perhaps by using nested structures (like structs) where appropriate, though be aware that there are some limitations with operations like partitioning on nested fields in Athena's current Iceberg implementation.
Sources
Evolve Iceberg table schema - Amazon Athena
S3 tables schema evolution and data loss | AWS re:Post
Athena Iceberg partitioning on nested ROW fields | AWS re:Post

answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.