Athena Iceberg Table is growing infinitely with the same number of rows.
0
I have an Iceberg Table which holds constant number of rows.
Every hour I update some of this rows using MERGE command.
After each MERGE I'm executing VACUUM command.
Table is created with following properties.
TBLPROPERTIES (
'table_type'='iceberg',
'vacuum_max_snapshot_age_seconds'='60'
);
Initially this table is about a 1GB. In week or so it grows up to 50+ GB.
What I'm doing wrong? How to avoid this growth and delete unnecessary garbage?
preguntada hace un mes521 visualizaciones
1 Respuesta
- Más nuevo
- Más votos
- Más comentarios
2
Can you:
- Verify that the VACUUM command is executing successfully after each MERGE operation?
- Verify that it's actually deleting old snapshots and associated data files?
- Look at the size and nature of the updates you're making every hour? If each update is adding a significant amount of data, this could contribute to the rapid growth of the table.
Contenido relevante
OFICIAL DE AWSActualizada hace 2 años- OFICIAL DE AWSActualizada hace 3 años
Thank you very much for your suggestion.
SELECT * FROM "my_table$snapshotsquery returns a single row with following content.SELECT * FROM "my_table$files";returns 308 rows. Unfortunately I don't know how to check if any of this files associated with an old snapshot.Could you verify if versioning is enabled on your bucket? If so, this might be contributing to the increase in size. You can check more about Versioning workflows.
Bucket Versioning is disabled.