Why adding consumers won't fix your consumer lag on Amazon MSK (head-of-line blocking)

Symptoms

A consumer lag alarm fires on your Amazon MSK cluster. You add consumer instances, but the lag does not go down. You add partitions, and it still does not help. Looking closer, the lag is not spread evenly across partitions; it is exploding on exactly one partition.

The cause of this pattern is not throughput. This article explains why it happens, why scaling out does not work, and what does.

The three causes of lag apart

Lag is computed as:

lag = logEndOffset - committed offset

There are three broad reasons this number grows, and you can distinguish them from the kafka-consumer-groups.sh --describe output pattern.

kafka-consumer-groups.sh --bootstrap-server <bootstrap-brokers> \
  --command-config client.properties --describe --group my-group

Observed pattern	Cause
Even lag across all partitions, CURRENT-OFFSET advancing	Insufficient throughput (slow, but processing)
Lag exploding on one partition, its CURRENT-OFFSET frozen	A slow/problematic message (stuck)
CONSUMER-ID column is empty	Consumer failure (no live members)

Tip: Run describe twice with some time in between and check whether CURRENT-OFFSET is moving. That is the decisive signal that separates "slow but processing" from "completely stuck."

If the cause is the first one (throughput), adding consumers is the right fix. This article is about the second one: CURRENT-OFFSET frozen at a specific value.

Why one slow message stalls an entire partition

In Kafka, the partition is the unit of ordering. Within a consumer group, one partition is assigned to exactly one consumer, and that consumer processes offsets in order. The committed offset is a single watermark: it means "everything below this number is done." If processing of offset 50 has not finished, the consumer cannot advance the commit even if it has already finished 51-100.

It is like a checkout line. If the customer at the front takes an hour, that entire line stops. The other lines (other partitions) keep flowing. That is exactly why describe shows the "one partition exploding" pattern.

Three reasons scaling out does not help

1. Adding consumers does not change who owns the stuck partition: still exactly one consumer. A partition is assigned to only one consumer within the group, so newly added consumers just redistribute the healthy partitions and cannot touch the stuck one.

2. Adding partitions does not relocate existing messages. The problematic message stays in its original partition. Adding partitions only spreads future incoming messages, it is irreversible, and it changes key hashing, which affects ordering for messages with the same key. It is not a button to press under pressure.

3. The problem spreads to the whole group. When a consumer spends too long on one message, the gap between poll() calls exceeds max.poll.interval.ms (default 5 minutes) and the consumer is evicted from the group. Eviction triggers a rebalance (during which the entire group pauses), the next consumer that inherits the stuck partition resumes from the committed offset, meets the same problematic message, and gets evicted 5 minutes later. A "one partition problem" turns into a rebalance storm on a 5-minute cycle that degrades the whole group.

Measured: the group stays up, and the stuck partition's gap is precisely predictable

I measured this on an MSK Provisioned cluster (Apache Kafka 3.8) with a 3-partition topic and three consumers (A/B/C), placing a message that takes 180 seconds to process at a specific offset in partition 0. To speed up the reproduction, max.poll.interval.ms was lowered from the default 5 minutes to 60 seconds.

Observed timeline:

00:35:53  Consumer A starts processing the problematic message in partition 0 (blocks 180s)
00:36:03  describe: lag grows only on partition 0; partitions 1/2 stay at lag 0
00:36:56  (after ~60s + a few seconds of rebalance) consumer B takes over partition 0 and resumes
00:37:00  describe: partition 0 lag cleared
00:38:53  A wakes up late and attempts to commit -> rejected with CommitFailedError

Three things this measurement shows:

1. No group-wide outage: during the 180 seconds A was stuck, consumption on partitions 1/2 was completely unaffected. Head-of-line blocking is confined to the affected partition.
2. The gap on the stuck partition equals max.poll.interval.ms + rebalance time. With the 60-second test setting it measured about 63 seconds; with the production default (5 minutes) the gap is about 5 minutes. This gap repeating is the contagion loop described above.
3. Zombie commits are blocked: when the evicted consumer A later attempted to commit, it was rejected with CommitFailedError. Because the group had already rebalanced, the protocol prevents a lagging instance from rolling back the committed offset that another consumer has advanced. The stale process itself still needs to be restarted for cleanup.

Remedies that work

If you need to unblock right now — skip the problematic message

Stop all consumers in the group, then move the committed offset past the problematic message with kafka-consumer-groups.sh --reset-offsets --shift-by 1. Before skipping, pull the message with kafka-console-consumer.sh and keep it for later analysis.

To prevent recurrence structurally — the pause/resume pattern

When you encounter a long-running message, call consumer.pause(partition) and hand the actual processing to a separate worker thread. The main thread keeps calling poll() to maintain group membership (while paused, poll() returns no records but still serves as the liveness signal). When processing finishes, commit and resume. This is the only structure in which a message that takes an hour does not cause eviction and rebalancing.

Interim mitigation — configuration tuning

Raising max.poll.interval.ms and lowering max.poll.records to 1 stops the eviction loop. However, the partition itself is still stalled, and detection of genuinely dead consumers becomes slower by the same amount. It is a trade-off, not a fix.

Root fix — processing timeout with a DLQ

Put an application-level timeout on message processing. On timeout, send the message to a retry topic or a dead-letter queue (DLQ), commit, and move on. This eliminates head-of-line blocking by design.

Supporting measure — static membership

If deployments or pod restarts trigger frequent rebalances in your environment, set group.instance.id (static membership, KIP-345) to reduce unnecessary rebalancing.

Note: sometimes the broker is a contributor

The first suspect for lag is the consumer side, but when broker CPU (CpuUser + CpuSystem) exceeds 60%, fetch responses slow down and contribute directly to lag. This does happen in practice on small instance types. Along with the describe pattern check, review broker CPU, disk throughput, and network utilization in CloudWatch.

Summary

• The first step in lag response is identifying the cause. Run describe twice and see whether CURRENT-OFFSET moves. Even lag with advancing offsets means scale out; one frozen partition means the scenario in this article.
• One slow message stalls its entire partition because of partition ordering and the single-watermark commit semantics. Adding consumers or partitions does not fix it.
• Left alone, it spreads to the whole group through repeated max.poll.interval.ms evictions and rebalances.
• Unblock now with --reset-offsets, and prevent recurrence structurally with the pause/resume pattern or a processing timeout with a DLQ.

Related information

• Amazon MSK Developer Guide — Best practices
• Apache Kafka documentation — consumer configs (max.poll.interval.ms, max.poll.records)
• KafkaConsumer pause/resume (Flow Control)
• KIP-345 (Static Membership)
• Knowledge Center — Troubleshoot consumer lag on MSK clusters