Does AWS SQS allow consumers to filter and retrieve only specific messages?

0

I have multiple consumers that read messages from an AWS SQS. Each consumer is specifically interested in messages that contain a unique value. Every consumer will have at least one message they are interested in, which includes a unique key-value pair such as JobId = HkjshkHkh88HK. No consumer will be interested in messages intended for other consumers, as they are only looking for the key-value pair associated with their unique ID. How can I implement this scenario efficiently using a single AWS SQS and multiple consumers?

Sachin
已提問 1 年前檢視次數 4448 次
4 個答案
0

You can't do that. The messages will be distributed across all consumers without a way to filter the messages.

You could use EventBridge and create multiple rules, with different filters that will distribute to different queues.

profile pictureAWS
專家
Uri
已回答 1 年前
  • Multiple queues are not considered as an option in the proposed architecture. In the current architecture, it is necessary to have a single AWS SQS that is consumed by several consumers, more than 100 in most cases.

0

Hi, if you have a single queue, all readers can read from this queue wihtout acknowledging in a non-destructive manner. Only the reader interested in it will delete the message after he successfully got it.

This scenario implies useless reads by all readers except 1. So, these additional API calls have to be accounted for. Of course, 1queue per reader would be better from API call perspective. It all boils down to how many of such queues you need.

From https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/step-receive-delete-message.html

Amazon SQS doesn't automatically delete a message after retrieving it 
for you, in case you don't successfully receive the message (for example, 
if the consumers fail or you lose connectivity). To delete a message, you 
must send a separate request which acknowledges that you've successfully received 
and processed the message. Note that you must receive a message before you can 
delete it.
profile pictureAWS
專家
已回答 1 年前
  • What is the meaning of a non-destructive manner in this case?

    When a consumer reads a message from AWS SQS without acknowledging it, will the other consumers also be able to read the same message? Wouldn't that message become invisible to all other consumers while one consumer is already reading it?

0

I don't believe SQS is the right service for this use case. You can consider following alternatives :

  • Amazon SNS that allows message filtering at the subscriber but it might or might not fit your use case as it is based on a pub/sub methodology and not a Producer/Consumer
  • Amazon MSK : you can benefit from full power of Kafka to create topics and direct each consumer to consume from its topic. You can deploy Amazon MSK as well in a serverless manner as well as cluster manner.
  • You can drive a serverless consumer using step-function that receives all the messages and triggers the right consumer based on the message categorization (this means that the consumers are better to be Lambda based). In this way you are pushing the consumption logic to the step function and not SQS.
AWS
專家
已回答 1 年前
profile picture
專家
已審閱 1 年前
0

To me it seems that Kinesis Data Streams are a more applicable solution here. It allows for multiple readers/consumers for each message; each of which can decide whether to process the message or ignore it.

I'd note that the two services (Kinesis and SQS) have different limits in terms of data and message volumes but it might be worth considering.

profile pictureAWS
專家
已回答 1 年前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南