Delete old parquet files of overwritten Iceberg table

Question

I am trying to write a pyspark dataframe to S3 and the AWS data catalog using the Iceberg format and the pyspark.sql.DataFrameWriterV2 with the createOrReplace function. When I write the same dataframe twice one after another, I see that all parquet files on S3 exist twice with slightly different names (hashes) in each partition, however, when I read the table with SQL, i get the expected number of rows, which corresponds to the number of rows in the dataframe. Is there a way to automatically delete the overwritten/superseded parquet files?

Answer

That's normal Iceberg behavior, it keeps the old files in case you want to get the data as it was in the past.  
In the configuration you can tell it when to expire or you can force it, see:
https://iceberg.apache.org/docs/latest/maintenance/
https://iceberg.apache.org/docs/latest/configuration/

Delete old parquet files of overwritten Iceberg table

Relevanter Inhalt