I was looking into a bug recently where it seems like the lambda for processing MQTT messages from an IoT source just dropped a single message somehow. Our MQTT client (using the Java AWS IoT SDK) logged the message being sent successfully to MQTT (with a unique ID), but the logs in AWS from the lambda handling MQTT messages don't show anything with this ID, or any errors from around that time. Lots of other messages go through successfully around that time, it's just one that gets dropped.
I noticed that around the time the message should have been received in AWS, the lambda spun up two more execution environments to deal with load. The log group for one of these new instances shows it processing a bunch of messages successfully, but the log group for the other one doesn't have anything. I'm thinking maybe the message got sort of routed to the new instance getting spun up, but then that instance failed to actually start for some reason, and so the message got dropped?
I'm wondering if this explanation sounds plausible and is maybe an expected behavior for lambdas (and something we just have to work around in our application logic) or if I should be digging deeper into something that might have gont wrong in MQTT or in our SDK client.
