I am using the following command to download files from S3 to my local server
"aws s3 cp s3://bucket-name/dir-name/ . --recursive --debug"
Sometimes I will get the files downloaded successfully.
If I run the same command a few times, sometimes I will get error.
With --debug flag, this is the output:
GET
/
encoding-type=url&list-type=2&prefix=2022-04-18%2F
host:glue-bucket-interactions-staging.s3.ca-central-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220423T041824Z
host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2022-04-22 22:18:24,445 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20220423T041824Z
20220423/ca-central-1/s3/aws4_request
184d4f7de08e4ea90234c5717ce78cfd7c31c01cfe854a3d11fa94381f9ab1c3
2022-04-22 22:18:24,445 - MainThread - botocore.auth - DEBUG - Signature:
87d083e547678eebd416afb6691a541b05fa930456c5bb57bb3f9a650cf8c276
2022-04-22 22:18:24,445 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://glue-bucket-interactions-staging.s3.ca-central-1.amazonaws.com/?list-type=2&prefix=2022-04-18%2F&encoding-type=url, headers={'User-Agent': b'aws-cli/2.5.4 Python/3.9.11 Linux/3.10.0-1160.45.1.el7.x86_64 exe/x86_64.rhel.7 prompt/off command/s3.cp', 'X-Amz-Date': b'20220423T041824Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAVLZENK7ICR7W3PXG/20220423/ca-central-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=87d083e547678eebd416afb6691a541b05fa930456c5bb57bb3f9a650cf8c276'}>
2022-04-22 22:18:24,445 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/v2/2.5.4/dist/awscli/botocore/cacert.pem
2022-04-22 22:18:24,446 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (3): glue-bucket-interactions-staging.s3.ca-central-1.amazonaws.com:443
2022-04-22 22:18:24,448 - MainThread - botocore.endpoint - DEBUG - Exception received when sending HTTP request.
Traceback (most recent call last):
File "urllib3/connection.py", line 174, in _new_conn
File "urllib3/util/connection.py", line 95, in create_connection
File "urllib3/util/connection.py", line 85, in create_connection
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "awscli/botocore/httpsession.py", line 358, in send
File "urllib3/connectionpool.py", line 785, in urlopen
File "urllib3/util/retry.py", line 525, in increment
File "urllib3/packages/six.py", line 770, in reraise
File "urllib3/connectionpool.py", line 703, in urlopen
File "urllib3/connectionpool.py", line 386, in _make_request
File "urllib3/connectionpool.py", line 1040, in _validate_conn
File "urllib3/connection.py", line 358, in connect
File "urllib3/connection.py", line 186, in _new_conn
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7fad81a9d7c0>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "awscli/botocore/endpoint.py", line 199, in _do_get_response
File "awscli/botocore/endpoint.py", line 271, in _send
File "awscli/botocore/httpsession.py", line 387, in send
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://glue-bucket-interactions-staging.s3.ca-central-1.amazonaws.com/?list-type=2&prefix=2022-04-18%2F&encoding-type=url"
2022-04-22 22:18:24,448 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fad81af65e0>>
2022-04-22 22:18:24,448 - MainThread - botocore.retries.standard - DEBUG - Max attempts of 3 reached.
2022-04-22 22:18:24,448 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2022-04-22 22:18:24,449 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fad81af6670>>
2022-04-22 22:18:24,449 - MainThread - awscli.customizations.s3.results - DEBUG - Exception caught during command execution: Could not connect to the endpoint URL: "https://glue-bucket-interactions-staging.s3.ca-central-1.amazonaws.com/?list-type=2&prefix=2022-04-18%2F&encoding-type=url"
Traceback (most recent call last):
File "urllib3/connection.py", line 174, in _new_conn
File "urllib3/util/connection.py", line 95, in create_connection
File "urllib3/util/connection.py", line 85, in create_connection
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "awscli/botocore/httpsession.py", line 358, in send
File "urllib3/connectionpool.py", line 785, in urlopen
File "urllib3/util/retry.py", line 525, in increment
File "urllib3/packages/six.py", line 770, in reraise
File "urllib3/connectionpool.py", line 703, in urlopen
File "urllib3/connectionpool.py", line 386, in _make_request
File "urllib3/connectionpool.py", line 1040, in _validate_conn
File "urllib3/connection.py", line 358, in connect
File "urllib3/connection.py", line 186, in _new_conn
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7fad81a9d7c0>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "awscli/customizations/s3/s3handler.py", line 149, in call
File "awscli/customizations/s3/fileinfobuilder.py", line 31, in call
File "awscli/customizations/s3/filegenerator.py", line 142, in call
File "awscli/customizations/s3/filegenerator.py", line 322, in list_objects
File "awscli/customizations/s3/utils.py", line 412, in list_objects
File "awscli/botocore/paginate.py", line 252, in iter
File "awscli/botocore/paginate.py", line 329, in _make_request
File "awscli/botocore/client.py", line 304, in _api_call
File "awscli/botocore/client.py", line 620, in _make_api_call
File "awscli/botocore/client.py", line 640, in _make_request
File "awscli/botocore/endpoint.py", line 101, in make_request
File "awscli/botocore/endpoint.py", line 155, in _send_request
File "awscli/botocore/endpoint.py", line 199, in _do_get_response
File "awscli/botocore/endpoint.py", line 271, in _send
File "awscli/botocore/httpsession.py", line 387, in send
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://glue-bucket-interactions-staging.s3.ca-central-1.amazonaws.com/?list-type=2&prefix=2022-04-18%2F&encoding-type=url"
fatal error: Could not connect to the endpoint URL: "https://glue-bucket-interactions-staging.s3.ca-central-1.amazonaws.com/?list-type=2&prefix=2022-04-18%2F&encoding-type=url"
2022-04-22 22:18:24,450 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.