S3 sync handling of downloads beyond 36 hours or session token expiry

Question

When downloading large files (e.g. a few TB of data) via S3 CLI, the session token expires beyond 36 hours. I understand from here that it's not possible to extend or refresh temporary credentials.

I would like to know if s3 sync can resume the same downloading job with a new set of temporary credentials.

Answer

Yes with new credentials any object that has already been transferred won't be retransferred. However if one object is particularly large and will not complete within 36 hours even though the s3 sync commmand will use multipart upload you cannot resume from failed uploads in this scenario - see docs. You could run a multi part upload on large file which you can resume using the lower level cli commands - see here.

You might also want to consider using AWS Datasync if this is a process which you will be running regularly.

EXPERT

Peter_G

answered 2 years ago

You might also want to consider using AWS Datasync if this is a process which you will be running regularly.

S3 sync handling of downloads beyond 36 hours or session token expiry

Relevant content