- Más nuevo
- Más votos
- Más comentarios
I would probably put the tasks on SQS, then have the "Slaves" (worker nodes) pull the messages from SQS.
If tasks contain a lot of data, save it to S3, and have the SQS message include a path (key) to the object so that the worker can find it.
This will be more robust (retry etc.), you can scale your EC2s based on SQS queue length, and you won't have to deal with service discovery (CloudMap/ALB) and firewalls while staying secure.
Good luck!
//Carl
Do you want to think of using AWS Batch or serverless compute (StepFunction or a series of Lambda functions) for such use-case?
https://docs.aws.amazon.com/batch/latest/userguide/what-is-batch.html
https://aws.amazon.com/step-functions/
An example serverless batch processing architecture (may or may not be suitable for your use-case, but just an fyi) - https://aws.amazon.com/blogs/compute/creating-aws-serverless-batch-processing-architectures/
Contenido relevante
- OFICIAL DE AWSActualizada hace un año
- OFICIAL DE AWSActualizada hace 2 años
- OFICIAL DE AWSActualizada hace 3 meses