We are planning on ingesting messages over MQTT using IoT Core, where the payload will be binary (more specifically, protobuf). Data will be forwarded using a rule to Kinesis. In the rule, we also want to add principal() to the message. Whether the message is still binary or not is not important.
We first looked into using the decode function for protobuf messages, as described here: https://docs.aws.amazon.com/iot/latest/developerguide/binary-payloads.html#binary-payloads-protobuf. However, this requires storing the protobuf .desc file in S3, and I noticed through cloudtrail that it makes frequent requests (maybe every request? Not sure) to S3, which would incur a high cost at our message load.
Next, we decided to just pass it on in binary through kinesis, and do the decoding in the kinesis consumer. However, in the documentation it's mentioned that "If you send a raw binary payload, AWS IoT Core routes it downstream to an Amazon S3 bucket through an S3 action. The raw binary payload is then encoded as base64 and attached to JSON.". Am I understanding it correctly that if we send a binary payload we cannot avoid incurring S3 charges? Would our only option be to base64 encode (or similar) the payload before sending it from our sensor device?
AWS 공식업데이트됨 일 년 전
Thank you, something like this is indeed what we had in mind. I have to ask though, was it a typo when you said "Amazon S3 is involved in this flow", and it actually should have said isn't, or is S3 actually involved in this? Assume you can ignore everything that happens after Kinesis, my question is only in regards to whether IoT Core itself will make calls to S3. To be even more precise, what I'm really looking for is clarification of the sentence "If you send a raw binary payload, AWS IoT Core routes it downstream to an Amazon S3 bucket through an S3 action." found in the documentation..
I meant to say "Amazon S3 is not involved" (corrected now). As for the documentation, that statement is utterly confusing. Thanks you for bringing to our attention. Binary payloads sent to AWS IoT Core are not written to S3 unless you use a specific S3 action in the rule. What I think that the documentation was supposed tos state is that to store a binary payload (eg a jpeg encoded image) to S3 you can use
SELECT * from 'topic/subtopicand an S3 action.