Athena Iceberg Table is growing infinitely with the same number of rows.
0
I have an Iceberg Table which holds constant number of rows.
Every hour I update some of this rows using MERGE command.
After each MERGE I'm executing VACUUM command.
Table is created with following properties.
TBLPROPERTIES (
'table_type'='iceberg',
'vacuum_max_snapshot_age_seconds'='60'
);
Initially this table is about a 1GB. In week or so it grows up to 50+ GB.
What I'm doing wrong? How to avoid this growth and delete unnecessary garbage?
asked a month ago501 views
1 Answer
- Newest
- Most votes
- Most comments
2
Can you:
- Verify that the VACUUM command is executing successfully after each MERGE operation?
- Verify that it's actually deleting old snapshots and associated data files?
- Look at the size and nature of the updates you're making every hour? If each update is adding a significant amount of data, this could contribute to the rapid growth of the table.
Relevant content
- Accepted Answerasked 5 months ago
- Accepted Answerasked 3 months ago
- asked 4 months ago
AWS OFFICIALUpdated 2 years ago- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 months ago
Thank you very much for your suggestion.
SELECT * FROM "my_table$snapshotsquery returns a single row with following content.SELECT * FROM "my_table$files";returns 308 rows. Unfortunately I don't know how to check if any of this files associated with an old snapshot.Could you verify if versioning is enabled on your bucket? If so, this might be contributing to the increase in size. You can check more about Versioning workflows.
Bucket Versioning is disabled.