S3 sync handling of downloads beyond 36 hours or session token expiry
When downloading large files (e.g. a few TB of data) via S3 CLI, the session token expires beyond 36 hours. I understand from here that it's not possible to extend or refresh temporary credentials.
I would like to know if s3 sync can resume the same downloading job with a new set of temporary credentials.
- Mais recentes
- Mais votos
- Mais comentários
Yes with new credentials any object that has already been transferred won't be retransferred. However if one object is particularly large and will not complete within 36 hours even though the s3 sync commmand will use multipart upload you cannot resume from failed uploads in this scenario - see docs. You could run a multi part upload on large file which you can resume using the lower level cli commands - see here.
You might also want to consider using AWS Datasync if this is a process which you will be running regularly.
Conteúdo relevante
AWS OFICIALAtualizada há 10 meses- Como utilizar um token de MFA para autenticar o acesso aos meus recursos da AWS por meio da AWS CLI?AWS OFICIALAtualizada há 2 anos