AWS S3 tar.gz compressed files issue after download from S3

Question

Hello,  
  
I would like to check with you guys here in AWS forum the issue that I discovered a while ago. Basically I'm trying to test our backup recovery process. Below is my simple flow of our backing up process.  
  
**Backup Flow**  
ec2 -> **uploads backup files compressed using tar.gz to S3** -> S3(backup bucket).  
_NOTE: I used AWS CLI for moving files from ec2 to s3._  
  
However when I tried to download the tar.gz file from S3, considering that on the AWS Console S3 file list it's has a correct file extension of tar.gz e.g. backup1.tar.gz BUT after the download, I've noticed that my backup file turns to backup1.tar (not tar.gz) so when I tried to decompress the backup its not usable or readable etc.  
I'm not sure if it's on my OS level error but overall it will make a huge confusion to the user if the backup process is working fine or not.

nateprewitt · Answer

Hi anjalitrambadia, DSimon,  
  
We took a quick look over the sync issues and wanted to get some more information.  
  
If you wouldn't mind creating an issue on the AWS-CLI's Github (https://github.com/aws/aws-cli/), that will help us better track this. Including the version of your aws cli (aws --version) and a brief set up steps for reproduction will be really useful.  
  
Thanks,  
AWS-CLI team  
  
Edited by: nateprewitt on May 12, 2020 6:11 PM

anjalitrambadia · Answer

Hello All,  
I have the same issue, anyone has any solution or suggestion?

DavidZemon · Answer

I'm having the same problems...  
  
I did `aws s3 sync some/local/folder s3://my-special-bucket/some/local/folder`  
and some of the files that got uploaded were .tar.gz files. Well, AWS decided that those should receive the metadata `Content-Type: applicaton/x-tar` instead of `Content-Type: application/x-gzip`. Highly aggravating. I had to write  a script to list every file in my bucket and then fix the metadata for any file with the .tar.gz extension.

DSimon · Answer

I have yet to see anyone explain how to download a tar.gz from an S3 bucket without AWS changing the format to a .tar and changing the config of the files. I need help to figure out how to down load a large (1.7 GB) tar.gz file.

dastafurov · Answer

You should use --content-encoding "application/x-compressed-tar" option during S3 uploading:  
aws s3 cp  --content-encoding "application/x-compressed-tar".......

AWS S3 tar.gz compressed files issue after download from S3

Relevant content