By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

How to use SQS for bulk processing?

0

Relatively new to AWS and I think I'm missing something here. We have two scenarios where we need to control when items in the queue get processed.

  1. Updating an external system that times out after 15 minutes of inactivity. There's not enough volume to keep the connection open all day, so we want to hold all updates in a queue and process them overnight. We still need to pull and process them one at a time, but if we can do this all at once then the system won't time out. Order doesn't matter.

  2. Event attendance/speakers. We need to hold info on attendees and speakers in the queue until the event itself has been created. Otherwise there's no event to connect those items to. The event creation is currently a manual process so it could be a couple hours or a couple days. Not entirely sure how this process will be started, but we'll get to that.

In both cases my original design was to send the updates to an SQS queue and hold them there until the Lambda function is ready to process them. I have since learned that's not how it works and the Lambda function just continually pulls from the queue. What am I missing? Isn't that the purpose of a queue, to hold items? It seems to act more like a load balancer than a queue.

How should I set this up so it works as desired?

Topics
ServerlessApplication Integration
Tags
Amazon Simple Queue Service
Language
English
Seven77
asked 5 months ago242 views
1 Answer
0

Hello.

I have since learned that's not how it works and the Lambda function just continually pulls from the queue. What am I missing?

Have you set up SQS for your Lambda trigger?
In this case, as you know, when a message is sent to an SQS queue, Lambda pulls the message and begins processing it.
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-lambda-function-trigger.html

It is possible to set a delay until the message is processed on the SQS queue side, but the maximum delay is 15 minutes, so it cannot be used if you want to process it at a specific date and time.
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-delay-queues.html

If you want to perform processing at a specific time, I think it would be a good idea to configure Lambda to run using the EventBridge scheduler.
https://docs.aws.amazon.com/lambda/latest/dg/with-eventbridge-scheduler.html

profile picture
EXPERT
Riku_Kobayashi
answered 5 months ago
  • Seven77
    5 months ago

    Thanks. The scheduler will work great, but that also means self managing the queue, so it feels like I'm not using it as it's designed. Is the need to control when or how fast queue items process an unusual requirement? Is there a better way to do this like saving the items off to DynamoDB and processing them from there?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content