By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Continuously Trigger ECS Tasks While Ensuring Parallel Task Limit is Maintained

0

I have an ECS service running on Fargate that registers messages in an SQS queue. An EventBridge rule triggers an ECS task whenever a message is added to the queue. The service must handle request spikes efficiently. However, the CIDR range limits the available IPs to 500, with around 100 reserved for another service in the same cluster. For scenarios where around 20,000 requests are sent simultaneously, I need a specified number of ECS tasks to be triggered in parallel. As tasks complete, SQS messages should be consumed continuously, triggering new tasks to maintain parallel processing. What would be the best possible design / AWS components to us in this scenario?

Topics
ServerlessContainersApplication Integration
Tags
ServerlessAWS FargateAmazon Simple Queue Service
Language
English
Greeshma
asked a month ago54 views
1 Answer
0

To manage a large number of simultaneous requests with your ECS service while staying within your IP limits, you can use a combination of AWS services and careful design. Keep using SQS for message queuing and EventBridge to trigger ECS tasks, but focus on configuring your ECS tasks to scale dynamically based on queue depth. This way, you can handle spikes efficiently.

You should monitor and manage your IP usage to avoid hitting limits, which might involve optimizing how tasks are placed and scaled. Additionally, implementing event-driven processing will help maintain parallel task execution, ensuring that new tasks are triggered as others complete. Set up monitoring and alerts to keep track of performance and resource use, and make adjustments as needed to stay within your constraints.

profile picture
EXPERT
Giovanni Lauria
answered a month ago
  • Greeshma
    25 days ago

    Thanks, Giovanni. To monitor and manage IP usage, I was initially thinking about using lambda functions. However, my tasks are going to run for hours and lambda might not be the right way to go about. Are there ways to configure in the service that more than, say 400 parallel tasks, are not triggered by the service at a time? Or what would you suggest?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content