Redshift Materialized view on Kinesis Streaming Ingestion not deleting records after retention period

Question

I created a Materialised view on Redshift on top of Kinesis Streaming Ingestion with Auto Refresh Yes I set the retention period of my Kinesis Data Stream to 1 day. Even after 2 days I can able to see the older records through the Materialized view in Redshift. I need to know the reason why the records are still visible even after the retention period.

I checked with arrival_timestamp to validate it is older record or not.

Accepted Answer

The retention period applied only to the Kinesis Data Stream and the Redshift MV will retain the data as incremental refresh is only pulling in new records since last fetch. This is by design as performing analytics on Redshift will most likely need the entire data set. If you need only latest data which matches the data stream then you will need to full refresh the MV.

Redshift Materialized view on Kinesis Streaming Ingestion not deleting records after retention period

Relevanter Inhalt