S3 sync handling of downloads beyond 36 hours or session token expiry
When downloading large files (e.g. a few TB of data) via S3 CLI, the session token expires beyond 36 hours. I understand from here that it's not possible to extend or refresh temporary credentials.
I would like to know if s3 sync can resume the same downloading job with a new set of temporary credentials.
- Más nuevo
- Más votos
- Más comentarios
Yes with new credentials any object that has already been transferred won't be retransferred. However if one object is particularly large and will not complete within 36 hours even though the s3 sync commmand will use multipart upload you cannot resume from failed uploads in this scenario - see docs. You could run a multi part upload on large file which you can resume using the lower level cli commands - see here.
You might also want to consider using AWS Datasync if this is a process which you will be running regularly.
Contenido relevante
OFICIAL DE AWSActualizada hace 2 años- OFICIAL DE AWSActualizada hace 2 años