Generation of presigned URL to S3 with temporary credentials - InvalidAccessKeyId

Question

I have been working on adding support for presigned URLs in aws-iot-device-sdk-embedded-C and I believe I'm very close.

I started from the demo to download from S3 with a temporary credentials user profile, and I adjusted my signature comparing with the CLI (aws s3 presign) with same data.

The signature is now OK but still while the temporary AccessKeyId is able to download the file, when I use the (printed to terminal) presigned URL I get instead <Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message>

I'm working on https://github.com/giuspen/aws-iot-device-sdk-embedded-C/blob/GP_http_demo_s3_download_test_signature2/demos/http/http_demo_s3_download/http_demo_s3_download.c

I'm puzzled, is there an additional procedure to record the AccessKeyId in order to generate the presigned URL?

EXAMPLE LOG:

[INFO] [DEMO] [http_demo_s3_download.c:1752] HTTP Client Synchronous S3 download demo using temporary credentials fetched from iot credential provider:
c10nfdpw0ux517.credentials.iot.eu-central-1.amazonaws.com
[INFO] [DEMO] [http_demo_s3_download.c:910] Establishing a TLS session with c10nfdpw0ux517.credentials.iot.eu-central-1.amazonaws.com:443.
[INFO] [DEMO] [http_demo_s3_download.c:766] AWS IoT credentials will expire after this timestamp: 2023-10-02T13:30:09Z.
[INFO] [DEMO] [http_demo_s3_download.c:650] AWS IoT credential provider response: HTTP/1.1 200 OK
content-type: application/json
content-length: 1165
date: Mon, 02 Oct 2023 12:30:09 GMT
x-amzn-RequestId: fd0bb5c9-754f-9486-6cfa-2529899e17c8
{"credentials":{"accessKeyId":"ASIAVBKNXEL5HP2B4VXX","secretAccessKey":"****************************************","sessionToken":"IQoJb3JpZ2luX2VjEA0aDGV1LWNlbnRyYWwtMSJHMEUCIFsKZ0d7JLu+P1S0AfpaXUGuW5/agg0/3+joQt+D2oirAiEA+fCfmAqo9zt8Xb5CcuJj7He77UQQcgq/HMaZNJljQKcq3gMIFhAAGgwzNDY0NDQ3MzUyMjYiDCk8oHzTPPEymfx9Kiq7AwvmDkvcH5RPb9RWtAoci/WO/UhI4D1CQ5kg2X7KEUbpdk/ig8W6hSMhEcFWYEfMNpryhicDB/XoJzpjfvPxccOewA6Rd/y+2qFzNOOc2Cba3GtovdGctYJBtvex6du367kJoYks6PtaKpxFXyA6oEHWQ2X0yy9Ai+bZdPX9woHEJOQ7x9Orr6msJo47wSFgF9R5JlPk2n1mvLuI1kQlzPbgakTSwkSYb0hR91Sbd9V9kWQzQsK1GI3TktJ9YkPM0EIP5FMlLtwBPa0IYNMpHT1mz5LnXg/Imoepb6sZtsfW/M1LhkbgL//8sTeE5HHUXIL0RbRRSEQSctFPETIozfmKDp+yGNeOgFXDKwuY7aeKTzw1URnKHSM9mN5d5BVCdIjFu4obUgOLrL1tSVK3ai+54d1pXplNkcT1RDuZUKPvdj1eNDb46Ykd70LbtuQCHZNJQLeIxOQ/XbH3slf49UXp6Gwr8ENk888U+w8W3Emi1uZ6OkM4X00DKZaUksQCH1/my7tAwOP/RZNypXiQj6d33COeu2W92NvTt74xH8XrUEBOxxAQuDTasOTf8SK4E+y7Q9GoB0tb9CQiMNHv6qgGOpoB3LgWRNLccumU7fynfuXkdXoi3utGH7UpWqs8zz2mImgSJccDFg8NEyT9x+yhdcitTiJ7lugAFNgKg1qTpphL6i0CRDExX6JyYYwXU1l3BxSxLMSehYlLWpJV+/SCOVqur9ps+7dC3M3+hjuSG3jusANCGbbINzink/mQmXSHDnGL8zcbV6YpLPEH+DVXuz4cFW3FT2jbkjuV4g==","expiration":"2023-10-02T13:30:09Z"}}.
[INFO] [DEMO] [http_demo_s3_download.c:961] Establishing a TLS session with otahalo.s3.eu-central-1.amazonaws.com:443.
[INFO] [DEMO] [http_demo_s3_download.c:1478] Getting file object size from host...
[INFO] [DEMO] [http_demo_s3_download.c:1652] Received successful response from server (Status Code: 206).
[INFO] [DEMO] [http_demo_s3_download.c:1698] The file is 58 bytes long.
[INFO] [DEMO] [http_demo_s3_download.c:1176] Downloading bytes 0-57, out of 58 total bytes, from otahalo.s3.eu-central-1.amazonaws.com...:  
[INFO] [DEMO] [http_demo_s3_download.c:1206] Response Body:
{
"filename"="update-bundle-imx8mnevk-v01-00-01.raucb"
}
[INFO] [DEMO] [http_demo_s3_download.c:1307] Getting presigned URL...
[INFO] [DEMO] [http_demo_s3_download.c:1410] ota_temp_url=
https://otahalo.s3.eu-central-1.amazonaws.com/ota-v01-00-01.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAVBKNXEL5HP2B4VXX%2F20231002%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20231002T123009Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEA0aDGV1LWNlbnRyYWwtMSJHMEUCIFsKZ0d7JLu%2BP1S0AfpaXUGuW5%2Fagg0%2F3%2BjoQt%2BD2oirAiEA%2BfCfmAqo9zt8Xb5CcuJj7He77UQQcgq%2FHMaZNJljQKcq3gMIFhAAGgwzNDY0NDQ3MzUyMjYiDCk8oHzTPPEymfx9Kiq7AwvmDkvcH5RPb9RWtAoci%2FWO%2FUhI4D1CQ5kg2X7KEUbpdk%2Fig8W6hSMhEcFWYEfMNpryhicDB%2FXoJzpjfvPxccOewA6Rd%2Fy%2B2qFzNOOc2Cba3GtovdGctYJBtvex6du367kJoYks6PtaKpxFXyA6oEHWQ2X0yy9Ai%2BbZdPX9woHEJOQ7x9Orr6msJo47wSFgF9R5JlPk2n1mvLuI1kQlzPbgakTSwkSYb0hR91Sbd9V9kWQzQsK1GI3TktJ9YkPM0EIP5FMlLtwBPa0IYNMpHT1mz5LnXg%2FImoepb6sZtsfW%2FM1LhkbgL%2F%2F8sTeE5HHUXIL0RbRRSEQSctFPETIozfmKDp%2ByGNeOgFXDKwuY7aeKTzw1URnKHSM9mN5d5BVCdIjFu4obUgOLrL1tSVK3ai%2B54d1pXplNkcT1RDuZUKPvdj1eNDb46Ykd70LbtuQCHZNJQLeIxOQ%2FXbH3slf49UXp6Gwr8ENk888U%2Bw8W3Emi1uZ6OkM4X00DKZaUksQCH1%2Fmy7tAwOP%2FRZNypXiQj6d33COeu2W92NvTt74xH8XrUEBOxxAQuDTasOTf8SK4E%2By7Q9GoB0tb9CQiMNHv6qgGOpoB3LgWRNLccumU7fynfuXkdXoi3utGH7UpWqs8zz2mImgSJccDFg8NEyT9x%2ByhdcitTiJ7lugAFNgKg1qTpphL6i0CRDExX6JyYYwXU1l3BxSxLMSehYlLWpJV%2B%2FSCOVqur9ps%2B7dC3M3%2BhjuSG3jusANCGbbINzink%2FmQmXSHDnGL8zcbV6YpLPEH%2BDVXuz4cFW3FT2jbkjuV4g%3D%3D%22%2C%22expiration%22%3A%222023-10-02T13%3A30%3A09Z%22%7D%7D&X-Amz-Signature=fd138ce95b8bf62db8ba1a0cf1aea3147a48e4c95231b44e2c4a6a30aebd52b7
[INFO] [DEMO] [http_demo_s3_download.c:1856] Demo iteration 1 is successful.
[INFO] [DEMO] [http_demo_s3_download.c:1875] Demo completed successfully.

NOW If I use the presigned URL:

<Error>
  <Code>InvalidAccessKeyId</Code>
  <Message>The AWS Access Key Id you provided does not exist in our records.</Message>
  <AWSAccessKeyId>ASIAVBKNXEL5HP2B4VXX</AWSAccessKeyId>
  <RequestId>1ZXWN5V7S7N41VHX</RequestId>
  <HostId>3lXR0Y18JE21ljldrxdoXZrZ1PCR6kC5MAEp9G+RcVms+fA1kRpn0S8aVJ9mFP/6ydlvCXHlFs0=</HostId>
</Error>

Giuseppe · Accepted Answer

I was able with help to fix the bug as discussed on https://github.com/aws/aws-iot-device-sdk-embedded-C/issues/1887 and I will soon make a proper demo for aws-iot-device-sdk-embedded-C

Greg_B · Answer

Hi Giuseppe. I'm not sure what's wrong. Maybe I misunderstand your goals and intentions, but I think you should consider an alternate approach.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html

> the object owner may share objects with others by creating a presigned URL

I don't think the IoT device should have the rights to create a presigned URL to download the object. Why not instead grant the device the ability to download the object directly? That's how AWS IoT Greengrass works, incidentally. It uses it's X.509 certificate to get temporary credentials from the AWS IoT Core credentials provider. And the corresponding IAM role grants the device permissions to get certain S3 objects.

There are two major approaches:
1. The device uses credential provider to exchange it's X.509 certificate for IAM credentials that let it access the S3 object. That's what the [HTTP download demo in the C-SDK](https://github.com/aws/aws-iot-device-sdk-embedded-C/blob/main/demos/http/http_demo_s3_download/README.md) does.
2. The device obtains a presigned URL (usually from an MQTT subscription). That's what the HTTP upload demo in the C-SDK does.

You seem to be blending the two approaches.

Generation of presigned URL to S3 with temporary credentials - InvalidAccessKeyId

Relevant content