How to Delete all Dynamodb records except last 1 year (dynamodb table does not have a TTL attribute set)

0

Hello , I need to purge all data from dynamodb table except the last 1 year of data . I do not have a TTL attribute set in the table, what is the best approach to proceed? IT will cost a lot if I write a TTL attribute for every record as per my knowledge although expiring items from TTL is free! I have an attribute called "created_on_date" in the table though !

1 Answer
2

You can still enable TTL on created_on_date providing that attribute is an epoch time in seconds.

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/howitworks-ttl.html

If its not and epoch timestamp in seconds then unfortunately you do not have any other method to approach this that will not involve reading all the items in the table and then either adding TTL or directly deleting.

I would consider backfilling a TTL attribute for all items in the table, which you can compute based on created_on_date. This will ensure you will not run into this situation again. Depending on the amount of data in your table you can either run a script on Lambda/EC2/Local etc... to perform this. For large tables which scale beyond several GB's then you may need to use something such as EMR - this is documented in two blogs:

  1. https://aws.amazon.com/blogs/database/backfilling-an-amazon-dynamodb-time-to-live-ttl-attribute-with-amazon-emr
  2. https://aws.amazon.com/blogs/database/part-2-backfilling-an-amazon-dynamodb-time-to-live-attribute-using-amazon-emr/
profile pictureAWS
EXPERT
answered 2 years ago
profile pictureAWS
EXPERT
Chris_G
reviewed 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions