- Mais recentes
- Mais votos
- Mais comentários
Thank you for sharing the details of your IoT architecture. Here are my thoughts on optimizing it for high message volume while maintaining reliability:
Optimization Suggestions:
- Enable enhanced fan-out on Kinesis to parallelize processing by Lambda. This spreads load across shards.
- Optimize Lambda function to avoid throttling. Allocate sufficient memory/concurrency.
- Use multiple smaller SQS queues with Lambda consumers instead of one large queue for better scalability.
Scalability Considerations:
- The architecture can scale to handle 25 million messages daily. Monitor throughput on each service.
- Watch for bottlenecks at Lambda and SQS. Scale out Lambda functions and use SQS auto-scaling.
- Kinesis shards and Lambda concurrency may need to increase based on load.
Reliability of SQS:
- SQS provides high reliability for decoupled processing. Messages are persisted redundantly.
- Configure visibility timeout and dead letter queue to handle failed processing.
Need for SQS:
- SQS adds value by decoupling ingestion from processing. This prevents data loss if consumers go down.
- However, Kinesis analytics and destinations may also provide decoupling. SQS may be redundant.
Alternatives:
- Consider using Kinesis Data Firehose for delivery to S3 then analytics/destinations for processing.
- Or Kafka streams could provide distributed processing without need for SQS queue.
I don't have more details, but take a look in this article: 7 patterns for IoT data ingestion and visualization- How to decide what works best for your use case. Your use case sounds similar to Pattern 5, but I leave to you to decide.
Some related resources:
Hi,
This very recent post will answer many of your questions above re perf., throughput, etc: https://aws.amazon.com/blogs/compute/introducing-faster-polling-scale-up-for-aws-lambda-functions-configured-with-amazon-sqs/
And this pair of blogs will provide answers to other questions re. reliability, security, etc. via guidance from Well-Architected Framework: https://aws.amazon.com/blogs/compute/implementing-aws-well-architected-best-practices-for-amazon-sqs-part-2/
Best,
Didier
Conteúdo relevante
- AWS OFICIALAtualizada há 3 anos
- AWS OFICIALAtualizada há um ano
- AWS OFICIALAtualizada há um ano
- AWS OFICIALAtualizada há um ano