S3 sync handling of downloads beyond 36 hours or session token expiry
When downloading large files (e.g. a few TB of data) via S3 CLI, the session token expires beyond 36 hours. I understand from here that it's not possible to extend or refresh temporary credentials.
I would like to know if s3 sync can resume the same downloading job with a new set of temporary credentials.
- 新しい順
- 投票が多い順
- コメントが多い順
Yes with new credentials any object that has already been transferred won't be retransferred. However if one object is particularly large and will not complete within 36 hours even though the s3 sync commmand will use multipart upload you cannot resume from failed uploads in this scenario - see docs. You could run a multi part upload on large file which you can resume using the lower level cli commands - see here.
You might also want to consider using AWS Datasync if this is a process which you will be running regularly.
関連するコンテンツ
- 質問済み 5年前
AWS公式更新しました 8ヶ月前
