Delete Billions of objects from S3 Bucket

0

Hi, I have a use-case to delete billions of object from s3 storage. The data is in folder inside the bucket which have storage class "Glacier Deep Archive". Is there any efficient and cost effective way? Is there cost associated with it to delete the objects from s3 Glacier Deep Archive storage? If i implemented the s3 lifecycle on all object in the bucket but in my case i have to implement on specific folder inside the bucket. If i apply Prefix rule using one or more filters. Is there any cost associated with it? I have billions object in the folder of bucket.

asked a month ago119 views
2 Answers
0

There are no Delete API charges with S3. The efficient and recommended option to delete millions/billions of objects is through a Lifecycle Policy that asynchronously removes the files from the bucket. As you mentioned objects resides in GDA, please take a note of minimum storage duration else customer will end up incurring early delete charges.

S3 Glacier Deep Archive has a minimum storage duration of 180 days. Objects deleted prior to the minimum storage duration incur a pro-rated charge equal to the storage charge for the remaining days. Objects that are deleted, overwritten, or transitioned to a different storage class before the minimum storage duration will incur the normal storage usage charge plus a pro-rated storage charge for the remainder of the minimum storage duration. Check S3 pricing document: https://aws.amazon.com/s3/pricing/ for more information.

AWS
answered a month ago
  • What is the formula to calculate early delete charges?

0

Hi There

The best way to go about this would be to set up a lifecycle policy that expires and deletes all objects in the bucket. The cost will depend on how long the objects have been stored in the DEEP_ARCHIVE class. There is a minimum time that objects need to be stored, and you will be charged an Early Deletion fee.

Amazon S3 Glacier storage classes have a minimum storage duration charge. If you delete an archive before meeting the minimum storage duration, you are charged a prorated early deletion fee.

Amazon S3 Glacier minimum storage duration depends on the storage class used. For a summary of minimum storage duration for each storage class, see Performance across the S3 storage classes.. For Deep Archive, the minimum is 180 days.

For information about Amazon S3 Glacier pricing and how early deletion fees are calculated, see Amazon S3 pricing.

Please also take a look at Cost COnsiderations for additional things to consider. DELETE and CANCEL requests are free.

profile pictureAWS
EXPERT
Matt-B
answered a month ago
profile pictureAWS
EXPERT
reviewed a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions