Eliminate the S3 Claim-Check Pattern with SQS 1 MiB Messages

Background

Until August 2025, SQS had a 256 KB message limit. For larger payloads (order events, audit logs, IoT batches), teams used the Claim-Check Pattern: store payload in S3, send the S3 key in SQS, consumer fetches from S3. This adds latency, cost, IAM complexity, and cleanup overhead.

With the 1 MiB payload increase, most of those workarounds are no longer needed.

Prerequisites

• An SQS queue (Standard or FIFO)
• No configuration change needed — all queues automatically support 1 MiB

Before: Claim-Check Pattern (256 KB limit)

Producer sends payload to S3, then puts S3 key in SQS:

import boto3, json, uuid

s3 = boto3.client('s3')
sqs = boto3.client('sqs')

def publish_order_event(order_data):
    # Step 1: Upload to S3
    key = f"events/{uuid.uuid4()}.json"
    s3.put_object(Bucket='event-payloads', Key=key, Body=json.dumps(order_data))

    # Step 2: Send reference to SQS
    sqs.send_message(
        QueueUrl=QUEUE_URL,
        MessageBody=json.dumps({"s3_key": key, "bucket": "event-payloads"})
    )

Consumer fetches from S3:

def process(record):
    ref = json.loads(record['body'])
    response = s3.get_object(Bucket=ref['bucket'], Key=ref['s3_key'])
    order = json.loads(response['Body'].read())
    process_order(order)
    s3.delete_object(Bucket=ref['bucket'], Key=ref['s3_key'])  # Cleanup

What this requires: S3 bucket + lifecycle policy + IAM for S3 on both producer and consumer + cleanup logic.

After: Direct 1 MiB Messages

Producer sends the full payload directly:

import boto3, json

sqs = boto3.client('sqs')

def publish_order_event(order_data):
    sqs.send_message(
        QueueUrl=QUEUE_URL,
        MessageBody=json.dumps({"type": "order.created", "payload": order_data})
    )

Consumer processes directly:

def process(record):
    message = json.loads(record['body'])
    process_order(message['payload'])  # Already there — no S3 fetch

What this requires: Just SQS. That's it.

What You Eliminate

Component	Before	After
S3 Bucket	Required	Gone
S3 Lifecycle Rules	Required	Gone
S3 IAM (producer + consumer)	Required	Gone
S3 GET/PUT latency	+100-400ms	Gone
Cleanup logic	Required	Gone
DLQ debugging	Hard (payload in S3)	Easy (payload in message)

Cost Comparison (1M messages/month at 500 KB average)

	Claim-Check	Direct 1 MiB
SQS	$0.40	$3.20 (8 chunks × $0.40/M)
S3 PUT + GET	$5.40	$0.00
Total	$5.80	$3.20

45% cheaper and architecturally simpler.

When to Still Use Claim-Check

Keep S3 for these scenarios:

• Payload > 1 MiB — still exceeds the new limit
• SNS fan-out — payload is copied per subscriber (expensive at 1 MiB × N)
• Long-term retention — SQS messages are deleted after processing
• Multiple consumers need same payload — S3 is read-many

Migration Tip

Handle both formats during transition:

def process(record):
    message = json.loads(record['body'])

    if 's3_key' in message:  # Old claim-check format
        payload = fetch_from_s3(message['s3_key'])
    else:  # New direct format
        payload = message['payload']

    process_payload(payload)

Lambda Memory Note

Larger messages use more memory. Adjust your Lambda consumer:

Message Size	Recommended Memory
< 256 KB	128-256 MB
256 KB - 1 MiB	512 MB - 1 GB

Conclusion

If your messages are between 256 KB and 1 MiB, you can now remove S3 from the message path entirely reducing latency, cost, IAM complexity, and operational overhead. The claim-check pattern isn't dead, but for the majority of event-driven systems, the simplest answer is now: just put it in the message.

Additional Resources

• Amazon SQS increases maximum message payload size to 1 MiB
• Amazon SQS Message Quotas
• Claim-Check Pattern (AWS Prescriptive Guidance)