Skip to content

Does AWS ElastiCache Redis Serverless support Celery as a Redis broker? Workers connect but never consume tasks.

0

I’m attempting to use Amazon ElastiCache Serverless for Redis (Redis 7.1, cluster‑mode enabled) as the message broker for Celery 5.3+ running on Amazon ECS Fargate. Celery powers asynchronous background tasks and periodic tasks (via Celery Beat) in a Django application.

The Problem

Celery workers:

  • Successfully connect to the Redis Serverless endpoint
  • Register themselves
  • Create the expected queue keys (verified via redis-cli) …but they never consume tasks.

Symptoms:

  • Tasks—including periodic Beat tasks—accumulate in the list (LLEN enterprise grows from 0→ 90).
  • Running BRPOP celery 5 manually via redis-cli returns instantly, confirming basic connectivity.
  • As a result, all asynchronous processing is effectively broken.

Environment / Configuration

  • Python packages: celery>=5.3,<6 kombu>=5.3,<6 redis>=5.0,<6
  • Celery is configured to use redis.cluster.RedisCluster.
  • Because ElastiCache Serverless disables commands like CLIENT LIST and triggers CROSSSLOT errors, I apply a custom redis_monkeypatch.py.

Behavior:

  • Without the patch → workers crash on startup with CROSSSLOT.
  • With the patch → workers start normally, but the consumption loop never activates.

Core Questions

  1. Is Celery + ElastiCache Serverless an officially supported or known‑working combination? I cannot find any AWS documentation confirming whether Celery works with the Serverless (cluster‑mode) flavor of Redis.
  2. If it is possible, what is the correct configuration?
  • Are specific broker_transport_options required?
  • Are there Celery/Kombu settings needed to make RedisCluster compatible with Serverless restrictions?
  • Are there known limitations with pub/sub or blocking list operations in Serverless mode?

Any guidance, shared experience or workarounds from others using Celery with ElastiCache Serverless would be extremely helpful.

2 Answers
0

AWS does not officially document or guarantee support for Celery with ElastiCache Serverless for Redis (cluster‑mode enabled). While ElastiCache Serverless is API‑compatible with Redis OSS, its restrictions (disabled commands, cluster‑mode sharding, CROSSSLOT errors) mean the Celery Redis broker implementation is not fully compatible out of the box. The official AWS documentation only confirms Redis OSS API compatibility, but does not list Celery as a supported integration.

https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/WhatIs.deployment.html

https://aws.amazon.com/documentation-overview/redis/

EXPERT

answered 9 months ago

0

ElastiCache Serverless for Redis runs in cluster‑mode and enforces the same command and key‑slot constraints as standard cluster configurations. Celery’s Redis broker implementation is not cluster‑aware—it assumes a single‑node topology and performs multi‑key list and pub/sub operations that can trigger CROSSSLOT errors or silently hang when keys land on different hash slots. That explains why your workers connect and write queue data but never consume tasks. AWS documentation for ElastiCache Serverless notes that commands like CLIENT LIST, MONITOR, and some blocking operations are restricted, which further breaks Celery’s polling loop. —Taz

answered 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.