It seems the S3 is using a special signing method to authenticate API calls (like PutObject). This information is send on headers, but these headers appear to be getting lost or altered when passed through an nginx reverse proxy. My test code works without the proxy, but when the endpoint URL is supplied, I get the following error:
api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method
Here are some example request headers from the Go SDK - the credentials come from an STS call, but they work outside of the proxy. The proxy endpoint is provided to the client via a ResolveEndpoint function:
SDK 2024/03/29 20:03:35 DEBUG Request
PUT /archive_24c00b22-1b96-47bf-bb20-9e72bea75c62.gz?x-id=PutObject HTTP/1.1
Host: s3.flare.dev.vaultara.com
User-Agent: aws-sdk-go-v2/1.26.0 os/macos lang/go#1.20.4 md/GOOS#darwin md/GOARCH#arm64 api/s3#1.53.0 ft/s3-transfer
Content-Length: 6162208
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 32dc6d8a-3379-4722-8a72-983eb42b8b5f
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=ASIA5C37OGSDFWFCIEVW/20240330/us-west-2/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-server-side-encryption, Signature=489b96d389658c823e2f3ff26dc3f5facb6625ad8aaef1306506347199bb6a5e
Content-Type: application/octet-stream
Expect: 100-continue
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20240330T000335Z
X-Amz-Security-Token: IQoJb3JpZ2luX2VjEOD//////////wEaCXVzLXdlc3QtMiJHMEUCIQDINYn3NYLW3/eUnzH0p2eyGcqp4ehxl+KjYpvxg2B2HQIgEaLAiBdSVvKlLknC5WU7hGDkXvrqvIbGjmzqRzvEGY4qugII+f//////////ARAFGgw4OTk1MjYwNDY4NTQiDHeSSLAGVaiVyg+r2iqOAgVTVlpBV9LFOYDrWaBp18YKa+6f3TasO+rPIezdA22hhMn+pj9o7ctN25Bpk+rsoxwlHzhYwxf/reSWan3wm1owqFFZ254NQA7099eRq/1U1zsfbH2hMq+x0wwbfYC/SaIBNCih1/QLrew7DCVsCo2LDbFHyJLd4hq50Z3DMMkH7U7YKnjiwK/0CkCh/5o7hACio6DxSCXB8TlLiMt4Iby1DZVa77U8cjMczwcfamQfhtEUkkViDhMqK2b8MymmY6+gpCDJWEKxVjHx5hr0lbCRv9TZ8TrCnV1BJOyMQRtT75Bx3olXggJbUdbywTsavcAGE2ArBuM4rrGTcGOjg6oHTnD50YU+cGPkivfjFTDXrZ2wBjqdAWWOLiddRPdt455h2tlGxkhjyhNpdQ81u0Jvm7nUS5O7J70H5i7ygNriEiJpew6g3YU1WaTLu8TNrkSpeb/cDrDDgbTcvP7Kd4D75BYpDAtiAhuV4vKgwC8tYXubaHMwcUCvDOfhZc9fuiYZNnxN3aZ7b+1KS1+2jtMZSEma3EiYgXlJ8E8WP/XeFy0UKGv7UyP07Zd9y+NTaOYBLN4=
X-Amz-Server-Side-Encryption: aws:kms
The proxy config looks like this. I'm assuming that all other headers for the request are being passed through, as proxy_pass_request_headers
defaults to on
:
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header CF-Connecting-IP $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
server {
listen 443 ssl;
server_name "s3.dev.example.com";
location / {
proxy_pass "https://s3.us-west-2.amazonaws.com";
}
proxy_set_header Host s3.us-west-2.amazonaws.com;
}
Any ideas about what could be missing? Could Cloudtrail logs shed some light on what needs to change here?
Thanks in advance for any help.
Thanks for the direction. However, this doesn't seem to do anything. First. my assumption is that nginx is passing these through automatically because of the default:
And even setting these explicitly:
Has no discernible effect. The larger challenge here is that while I can see the request headers from the SDK (as indicated my first post), I can't see what's arriving at the S3 API endpoint. So I can't tell what is missing. I don't think CloudTrail is going to help here as this isn't making it to the API (I don't think). VPC logs? Any other suggestions?
Thanks again.
Furthermore, it doesn't look like
X-Amz-Credential
orX-Amz-Signature
are being sent at all. These, however, are:I think I have learned what could be happening here. According to this: https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html The signature is made up of various headers, i.e.:
My understanding is that in order to proxy to any API, the
Host:
header must be changed or the server won't respond. However, theHost:
header of the originating request is that of the proxy server, where is is then rewritten to the upstream service - see config above.Is this even possible without resigning the request?
Thanks.