Elegant way to transfer data from IoT Core things to ECS container service

Hello,

Use AWS IoT Rule to send data to an Amazon SQS queue, then have your ECS container service poll and process messages from SQS. This approach decouples your system, ensuring scalability and simplicity.

Hi Ize_hedgehog,

Please go through below steps once it will be helpfull to resolve your issue.

Using SQS with ECS Integration

Benefits:

• Decoupling: SQS decouples the IoT data ingestion from the processing logic in ECS, allowing both systems to scale independently.
• Reliability: SQS provides reliable message delivery, ensuring that no data is lost during the transfer from IoT Core to ECS.
• Flexibility: This setup can handle varying data loads efficiently, and ECS tasks can process messages as they arrive or in batches.
• Simplicity: Compared to Kafka or direct HTTP integrations, using SQS is relatively straightforward to set up and manage.

Implementation Steps:

Create an SQS Queue:

• • Go to the AWS Management Console and navigate to Amazon SQS.
• Create a new queue (standard or FIFO based on your needs).

Set Up an IoT Rule to Route Messages to SQS:

• In AWS IoT Core, create a rule to capture the telemetry data or events from your IoT devices.
• Configure the rule to send the data to the SQS queue you created.

Example IoT Rule SQL:

SELECT * FROM 'iot/topic'

Set the action to "Send a message to an Amazon SQS queue."

Implement an SQS Poller in ECS:

• Develop a service within your ECS tasks to poll the SQS queue for new messages.
• The service can use the AWS SDK to read messages from the queue and process them.

Example code snippet for polling SQS (Python using Boto3):

import boto3

sqs = boto3.client('sqs')

def poll_sqs(queue_url):
    while True:
        messages = sqs.receive_message(QueueUrl=queue_url, MaxNumberOfMessages=10)
        if 'Messages' in messages:
            for message in messages['Messages']:
                process_message(message)
                sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=message['ReceiptHandle'])

def process_message(message):
    # Your processing logic here
    print("Received message: ", message['Body'])

queue_url = 'https://sqs.<region>.amazonaws.com/<account_id>/<queue_name>'
poll_sqs(queue_url)

Security and Monitoring:

• Ensure that your ECS tasks have the necessary IAM permissions to interact with the SQS queue.
• Set up CloudWatch alarms and logs for monitoring the SQS queue and ECS tasks to ensure they are operating correctly and to detect any issues promptly.

Optional - Use Lambda for Event Handling:

• For specific real-time event handling, you can also trigger AWS Lambda functions directly from IoT rules.
• Lambda functions can process the data immediately or forward it to other services like ECS or DynamoDB.

Hi. Some links for your consideration:

• https://iotatlas.net/en/best_practices/aws/data_ingest/#decouple-ingest-from-processing-with-queues-and-streams
• https://iotatlas.net/en/best_practices/aws/data_ingest/#batch-incoming-data-using-iot-core-rules-before-sending-downstream-to-other-aws-services
• https://iotatlas.net/en/best_practices/aws/data_ingest/#use-iot-cores-basic-ingest-to-save-on-messaging-costs
• https://iotatlas.net/en/best_practices/aws/data_ingest/#save-ingested-data-to-durable-storage-before-processing
• https://docs.aws.amazon.com/wellarchitected/latest/iot-lens/tradeoffs.html
• https://docs.aws.amazon.com/wellarchitected/latest/iot-lens/cost-effective-resources.html

For your ECS consumption, the two answers already here (recommending SQS for decoupling), are good answers. However, since SQS is priced by messages, and IoT messages are generally small, I think Kinesis Data Streams will probably be cheaper. There's an existing rule action you can use: https://docs.aws.amazon.com/iot/latest/developerguide/kinesis-rule-action.html.

Potentially can bootstrap ‘Basic Ingest’ in a way to be ready to put some data into DynamoDB (or with help of Lambda), but obviously it makes the entire project very coupled, e.g. firmware dependent.

Basic Ingest will increase the coupling (and reduce your flexibility), but it can be a very large cost saving.

I like how ‘Shadow’ feature of IoT Core interfaces and divides mine IoT Fleet from my compute service, so seek for the same but more advanced.

For fast/frequently changing data, shadows can be expensive. They're best suited for slow/rarely changing status, configuration and command/control.