Using S3 Batch to perform an expedited restore

0

Hi, I have very little experience with AWS Batch and am hoping someone can guide me with a prospective redevelopment approach.

I'm looking to use S3 Batch to replace an existing restore/copy process which is not proving scalable. I intended to use S3 Batch using a S3InitiateRestoreObject operation to restore from Glacier Flexible Storage, but then found out that this supports STANDARD and BULK retrieval tiers, but not the EXPEDITED retrieval tier according to this documentation.

It's very important that I also support the EXPEDITED retrieval tier as well. I'm wondering whether I could use LambdaInvoke instead to perform an EXPEDITED restore? I appreciate that might be quite pricey since I'd be paying for the lambda and the expedited restore, but the first question is: Would it work? And in any case, for what reason is the EXPEDITED retrieval tier not included when using a S3InitiateRestoreObject operation?

질문됨 9달 전333회 조회
2개 답변
2

Yes, that right, S3 batch operations don't support the Expedited retrieval tier.

In your use case, I'd prefer to choose custom CLI script. Please refer this re:Post Knowledge Center Article if you haven't already gone through this.

The above mentioned article tells how you can work on retrieval of s3 objects from tier which S3 batch doesn't support.

Additional Reference:

s3 batch Create Job CLI

Hope you find this useful.

Abhishek

profile pictureAWS
전문가
답변함 9달 전
1
수락된 답변

AWS S3 Batch Operations is a powerful service that can perform large-scale operations on S3 objects, such as copying, restoring, or applying tags. However, as you've noted, the S3InitiateRestoreObject operation currently only supports STANDARD and BULK retrieval tiers, not the EXPEDITED retrieval tier.

The reason for this limitation is not explicitly stated in the AWS documentation, but it could be due to the fact that expedited retrievals are designed for urgent situations where you need your data back in less than 5 minutes. This is a more costly operation, and it may not be suitable for large-scale batch operations where cost-effectiveness is often a priority.

As for your question about using AWS Lambda to perform an EXPEDITED restore, yes, it is technically possible. You can write a Lambda function that uses the AWS SDK to call the restore_object method with the Expedited retrieval option. This Lambda function can then be invoked as part of your S3 Batch Operations job.

Here's a basic example of what the Lambda function could look like in Python:

import boto3

def lambda_handler(event, context):
    s3 = boto3.client('s3')

    for record in event['Records']:
        bucket = record['s3']['bucket']['name']
        key = record['s3']['object']['key']

        s3.restore_object(
            Bucket=bucket,
            Key=key,
            RestoreRequest={
                'Days': 1,
                'GlacierJobParameters': {
                    'Tier': 'Expedited'
                }
            }
        )

This function would be triggered for each object in your S3 Batch Operations job, and it would initiate an expedited restore for that object.

Keep in mind that using Lambda in this way will indeed incur additional costs, both for the Lambda invocations and for the expedited retrievals. Also, expedited retrievals are not always available if AWS is experiencing peak demand, so you should have a fallback plan in place.

profile picture
답변함 9달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠