How do I restore a large volume of Amazon S3 objects that are in the S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive storage class?

Resolution

To restore a large volume of Amazon S3 Glacier storage class objects, use either Amazon S3 Batch Operations or a custom AWS Command Line Interface (AWS CLI) command.

Note: If you receive errors when you run AWS CLI commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Use S3 Batch Operations

Create an S3 Batch Operations job to restore all the objects. You can run a Restore job on a custom list of objects or an Amazon S3 inventory report.

Prerequisites:

• Review the Amazon S3 pricing for Batch Operations.
• You have an AWS Identity and Access Management (IAM) user or role that has permissions to restore an object. The IAM user or role must also have a trust policy with Amazon S3 batch operations.
• You have a .csv file or an Amazon S3 inventory report to use as the manifest of the objects that you want to restore. Store the manifest file in an S3 bucket. For more information, see Creating a manifest file.

To use the Amazon S3 console to create a batch operation job and initiate a restore, complete the following steps:

1. Open the Amazon S3 console.
2. In the navigation pane, choose Batch operations.
3. Choose Create job.
4. For Region, select the AWS Region where you want to create the job.
5. Under Choose manifest, enter the following:
   For Manifest format, select S3 inventory report or CSV as your file format.
   For Path to manifest object, enter the S3 path to the manifest file, for example s3://awsexamplebucket/manifest.csv.
6. Choose Next.
7. Under Choose operation, enter the following:
   For Operation, choose Restore.
   For Restore source, choose Glacier Flexible Retrieval or Glacier Deep Archive.
   For Number of days that the restored copy is available, enter the number of days.
   For Restore tier, choose either Bulk retrieval or Standard retrieval.
   Note: S3 batch operations don't support the Expedited retrieval tier.
8. Choose Next.
9. Under Configure additional options, enter the following:
   (Optional) For Description, enter a description.
   For Priority, enter a number for the job's priority.
   For Generate completion report, keep this option selected.
   For Completion report scope, choose Failed tasks only or All tasks.
   For Path to completion report destination, enter the path where you want to send the report.
   For Permission, choose Choose from existing IAM roles, and then select the IAM role that has the required permissions.
10. Choose Next.
11. On the Review page, review the details of the job, and then choose Create job.
12. Select the job, and then choose Confirm and run.
13. (Optional) If you selected Generate completion report, then review the report after the job completes. You can find the report at the Path to completion report destination that you specified.

For descriptions of each job status, see Job statuses.

Important: The Restore job only initiates the request to restore objects. After you initiate the request, S3 Batch Operations reports the job as complete for the object. After the batch operation job restores the objects, Amazon S3 doesn't update the job or notify you. However, you can use Amazon S3 Event Notifications to receive notifications when the objects are available in Amazon S3. 

For more information about failed jobs, see Tracking job failure.

Use a custom AWS CLI command

You can run the AWS CLI restore-object command to restore your Amazon S3 Glacier objects. However, the restore-object command can only restore one S3 Glacier object at a time and doesn't support the bulk restore action.

To restore bulk data from the S3 Glacier storage classes, use the following custom command for your operating system (OS).

Note: It's a best practice to test the custom scripts in a non-production environment before you use them in your production environment. If you have too many objects, then the command might time out. Use the Prefix parameter in the command to decrease the number of objects.

For a Linux or Unix-based system, run the following list-objects command to restore all the S3 Glacier objects in the bucket:

aws s3api list-objects --bucket bucket-name --prefix your-prefix --query 'Contents[?StorageClass==`GLACIER`][Key]' --output text | xargs -I {} sh -c "aws s3api restore-object --bucket bucket-name --key \"{}\" --restore-request Days=5,GlacierJobParameters={Tier=Standard} || true"

Note: Replace bucket-name with your S3 bucket name, your-prefix with your S3 folder path, and StorageClass value with either GLACIER or DEEP_ARCHIVE.

For a Windows-based system, complete the following steps:

1. Run the following list-objects command to list all the S3 Glacier objects in the bucket:

   aws s3api list-objects --bucket bucket-name --prefix your-prefix --query "Contents[?StorageClass==`GLACIER`][Key]" --output text > list.txt

   The list of objects is saved in a file that's named list.txt.

2. Run the restore-object command to restore the S3 Glacier objects:

   for /F "tokens=*" %i in (list.txt) do @aws s3api restore-object --bucket bucket-name --key "%i" --restore-request Days=5,GlacierJobParameters={Tier=Standard} || true"

Note: The preceding custom AWS CLI command incurs additional charges for the list and data retrieval requests. Because the list-objects-v2 API is a paginated operation, multiple API calls retrieve the entire dataset of results.

Related information

Understanding archive retrieval options

Performing object operations in bulk with Batch Operations

Managing S3 Batch Operations jobs