Upload large files to S3 via CLI

Question

We are trying to use the CLI on Linux to upload a file larger than 110GB to AWS S3, however, we are getting the following error.

```
An error occurred (EntityTooLarge) when calling the PutObject operation: Your proposed upload exceeds the maximum allowed size
```

```
aws s3api put-object \
        --bucket $bucket \
        --key "$key" \
        --body "$body" \
        --sse-customer-algorithm AES256 \
        --sse-customer-key "$customer-key" \
        --sse-customer-key-md5 "$customer-key-md5"
```

Another obstacle we have is that we are encrypting the file with SSE C.

How can we upload this?

AWS TAM · Answer

To upload a file larger than 110GB to Amazon S3 using the AWS CLI, you can use the multipart upload feature. However, there are some limitations on the maximum object size that you should be aware of:

1. The maximum object size that can be uploaded in a single PUT operation is 5GB. For objects larger than 100MB, it is recommended to use the multipart upload feature.

2. The maximum size of an individual part in a multipart upload is 5GB. This means that for a file larger than 110GB, you will need to split it into multiple parts and upload them separately.

3. To perform a multipart upload using the AWS CLI, you can use the following steps:

```
   aws s3 mb s3://your-bucket-name
   aws s3 cp /path/to/large/file.zip s3://your-bucket-name/file.zip --recursive --multipart-upload --chunk-size 5GB
   ```

- The `--multipart-upload` option enables the multipart upload feature.
   - The `--chunk-size` option specifies the size of each part, which should be less than or equal to 5GB.

4. You can monitor the progress of the multipart upload using the `aws s3 ls` command:

```
   aws s3 ls s3://your-bucket-name/file.zip --human-readable --summarize
   ```

5. If the upload is interrupted for any reason, you can resume the upload from the last successful part using the `aws s3 cp` command with the `--continue` option:

```
   aws s3 cp /path/to/large/file.zip s3://your-bucket-name/file.zip --continue
   ```

For more detailed information on uploading large files to Amazon S3, you can refer to the

[Amazon S3 Developer Guide](https://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html).

[Upload large files to S3 | AWS re:Post](https://repost.aws/selections/KPZHjZ7hWnRs2APnSxxrHN3w/practical-guide-to-upload-large-files-to-s3)

ThePokyWizard · Answer

And how do I encrypt with SSE C?

Yaniv Rozenboim · Answer

You need to use multipart upload as there is a [limit on a single upload of 5 GB size.](https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html).

You can use the high-level `aws s3 cp` command instead of the low-level `aws s3api put-object` command to have the multipart done automatically.

Please refer to this [link](https://repost.aws/knowledge-center/s3-multipart-upload-cli) for more information on the exact usage.

Upload large files to S3 via CLI

相關內容