Master and Slave Instances Architecture

Question

Hi Dears,

I have a qustion on how to build master EC2 that can communicate with other EC2s based on need and send tasks to them based on a certian code/ group of tasks.  Where Master EC2 will take data from RDS db and send data with task to each EC2 based on scudualing and aknowledgamet from each one.

I appriclate your help dears!
Thanks
Basem

Accepted Answer

I would probably put the tasks on SQS, then have the "Slaves" (worker nodes) pull the messages from SQS.

If tasks contain a lot of data, save it to S3, and have the SQS message include a path (key) to the object so that the worker can find it.

This will be more robust (retry etc.), you can scale your EC2s based on SQS queue length, and you won't have to deal with service discovery (CloudMap/ALB) and firewalls while staying secure.

Good luck!

//Carl

Answer

Do you want to think of using AWS Batch or serverless compute (StepFunction or a series of Lambda functions) for such use-case?

https://docs.aws.amazon.com/batch/latest/userguide/what-is-batch.html

https://aws.amazon.com/step-functions/

An example serverless batch processing architecture (may or may not be suitable for your use-case, but just an fyi) -
https://aws.amazon.com/blogs/compute/creating-aws-serverless-batch-processing-architectures/

Master and Slave Instances Architecture

Relevant content