Possible bug with SQS Lambda consumer and filters?

Question

I have configured a queue with a lambda consumer. The Lambda trigger is configured with a filter to process only certain messages from the queue:
```
"FilterCriteria": {
  "Filters": [
    {
      "Pattern": "{\"body\":{\"action\":[\"sample1\"]}}"
    }
  ]
}
```

When sending a message matching the filter to the queue, no problem, the message gets comsumed by the lambda function and is removed from the queue.

When sending a message not matching the filter `{"action":"testing"}`, the message isn't consumed by the lambda function (this is expected), but the message is deleted from the queue and no more available for any other consumer.

This gets even worse when we configure a maxConcurrency for the Lambda function: Lambda will consume some of the message and some messages (matching the filter) won't be consumed and will still be deleted in SQS.

Did I stumble upon a bug, or did I miss something in how the filter is supposed to work?

Thanks,
Daniel

Accepted Answer

First of let us understand one lambda event filters is not designed to filter same message multiple times and after every successful single/batch invocation the messages are cleared unless you explicitly delete them if there is an abrupt error handling.

This is needed to avoid reprocessing the unmatched messages again and it may clog your batch size and the lambda filter runs endlessness not finding suitable records to process in the head.

Then why do we have this filter you may ask, this is to make sure we are processing the **right message** saving our compute time and space. Since SQS could not filter by itself.

Also SQS is more suitable for **one consumer kind architecture**, if you would like to have another consumer it is better to have multiple SQS queues.

To have separate queue you could initially create an event bridge event with the same payload and then use **event bridge rule **to move the message to right SQS and them use filter pattern with single/batch invocation strategy.

Possible bug with SQS Lambda consumer and filters?

Relevant content