Kafka Rebalance Storm & Practical Rebalance Strategies

TL;DR: On a dynamic consumer group (autoscaling + failing pods), rebalances happen constantly. With the classic protocol + eager strategy (KafkaJS default), every rebalance stops almost ALL consumers from processing — the famous "stop-the-world" principle. Result: message lag piles up, consumption time grows unstoppably. This post reproduces that scenario on a self-built lab and compares three strategies with real measured numbers.

1. The Problem: Rebalance Storm on a Dynamic Environment

A realistic scenario

Say you have a consumer group consuming a topic with 25 partitions across 5 consumer instances (each pod ideally 5 partitions). This is a CDC system syncing near-real-time at high volume — we don't want processing to stop just because one consumer has a problem.

On production, rebalances are triggered constantly for three main reasons:

1. A consumer instance fails (timeout, server crash) → it stops processing and leaves the group → rebalance.
2. HPA (Horizontal Pod Autoscaler) spawns a new pod → the new pod joins the group → rebalance.
3. HPA reclaims an extra pod → the pod leaves the group → rebalance.

Result: rebalances fire frequently, and each one stops the processing pods. The more rebalances, the more consumption time grows → messages pile up.

[25 partitions] → [consumer group]
                       │
  ┌────────┬──────────┼──────────┬──────────┐
  │ Pod 1  │  Pod 2   │ Pod 3    │ Pod 4    │  Pod 5
  │ 5 parts│  5 parts │ 5 parts │ 5 parts  │  5 parts
  └────────┴──────────┴──────────┴──────────┘
        ↑ a pod dies / HPA spawn-reclaims pod → REBALANCE → stops everything

Why is it so bad with KafkaJS?

Many teams use NestJS with built-in Kafka support, built on KafkaJS — the most popular Kafka library for Node.js. By default:

• Protocol: classic
• Strategy: roundRobin (eager)

With the classic protocol + eager strategy, any change in group membership (consumer joining/leaving) or topic metadata triggers a complete halt:

Stop-the-world: All consumers revoke their partitions, a leader computes a new assignment, partitions are redistributed — and only then does processing resume.

The classic protocol uses client-side logic to compute partition assignments — meaning the decision lives in the consumers; the more consumers/partitions, the longer consensus takes.

And a crucial limitation:

KafkaJS hasn't been maintained for ~2 years (consider it dropped). NestJS also has no plan to switch to another stable Kafka library for built-in support.

This effectively blocks the "just upgrade the library" path.

2. The Way Forward

From this problem, the Kafka team introduced improved strategies:

1. Cooperative-Sticky: Keeps unaffected partitions, revokes only the ones needing reassignment. Consumers keep processing during rebalance.
2. KIP-848 (Consumer Protocol): Completely redesigns the consumer-broker interaction — rebalance is handled server-side, faster and more stable.

A big blocker: KafkaJS doesn't support these newer strategies (because it's unmaintained). So you need a newer Confluent-maintained library.

3. Moving to Confluent Kafka for JavaScript

confluent-kafka-javascript is built on top of two famous Kafka libraries:

• KafkaJS
• Node Rdkafka (bindings for librdkafka)

Strengths:

• Maintained by Confluent — the same company behind the Chorus Kafka platform, so no worries about maintenance or new features.
• Migration supported easily from both older libraries.
• Full KIP-848 support and the newer strategies.

4. Reproducing It On a Lab & Real Measured Numbers

I built a small lab to prove the story myself, instead of just trusting theory.

Lab setup

Each strategy ran 3 independent times with the same scenario. The numbers below are real measured results from the evidence logs.

4.1 Summary results (broker ground truth)

How to read this: The whole story is in one table.

• With eager, killing one consumer moved 76% of partitions (19/25), and one new pod joining reshuffled 84% of them. This is the rebalance storm signature: one small event → global chaos.
• With cooperative and KIP-848, only 20% of partitions (exactly the 5 the dead consumer held) had to move. The other pods were never touched → stop-the-world = 0 ms.

4.2 Classic Protocol + Eager Strategy (Round Robin)

'group.protocol': 'classic'
'partition.assignment.strategy': 'roundrobin'

Measured behavior when a pod leaves the group:

• All pods stop immediately when one pod leaves/joins.
• Partition ownership reshuffles almost completely: from steady-state [pod1: 0,5,10,15,20], [pod2: 1,6,11,16,21], [pod3: 2,7,12,17,22], [pod4: 3,8,13,18,23], [pod5: 4,9,14,19,24] → after pod3 dies, ownership is recomputed from scratch (pod1 takes 0,4,8,12,16,20,24, etc.).
• 8 rebalance starts for just one kill + one join (each surviving pod sees 2 rebalances per phase).
• Client-side rebalance handling.
• Rebalance takes longer with more consumer instances.

Conclusion: On every leave/join, all consumers in the group stop listening and stop processing messages. With dynamic HPA, lag accumulates uncontrollably. High risk for production.

4.3 Classic Protocol + Cooperative Strategy

'group.protocol': 'classic'
'partition.assignment.strategy': 'cooperative-sticky'

Measured behavior when a pod leaves:

• Only 5/25 partitions (exactly those the dead consumer held) are revoked and reassigned.
• The other pods keep their partitions and keep processing — measured stop-the-world = 0 ms.
• Each surviving pod sees only 1 rebalance per phase (vs 2 for eager).
• ⚠️ But: offset commit is paused (~250 ms) until rebalance completes.
• Still client-side rebalance.

Conclusion: Stop-the-world is mitigated — consumers keep unaffected partitions processing, only the touched part is affected. However commit pauses between phases and it's still client-side. Clearly safer than eager.

4.4 Consumer Protocol (KIP-848) + Uniform Strategy

'group.protocol': 'consumer'
'group.remote.assignor': 'uniform'

Measured behavior when a pod leaves:

• Only 5/25 partitions move (same as cooperative), but the assignment is computed server-side immediately.
• Stop-the-world = 0 ms; pods outside the touched partitions keep processing.
• Commit gap ~253 ms (comparable to cooperative in this lab).
• Server-side rebalance: the broker computes assignments, no consumer consensus.

Conclusion: Preferred for production. Server-side rebalance → fast and stable. KIP-848 is designed to become the default protocol in a future Apache Kafka release (likely 5.0).

4.5 An important finding: detection latency is dominated by session timeout, not the protocol

This is what the lab taught me that theory doesn't spell out:

The paradox: KIP-848 rebalances "cheaper" (only 20% of partitions move) but detects a hard-killed consumer more slowly — because:

• Classic uses a client-side session timeout (default 30s).
• KIP-848 uses broker-side group.consumer.session.timeout.ms (set to 45s in the lab).

This is a real trade-off: to make KIP-848 detect faster, you must lower group.consumer.session.timeout.ms on the broker. Otherwise a hard-killed pod takes ~45s to notice — longer than eager.

Lesson: Don't just pick a protocol and forget timeouts. Measure detection time, not just rebalance time.

5. Comparison Summary

6. Recommendations by Use Case

There's no single "best" strategy — it depends on your system's characteristics:

Case A: High-volume CDC, near-real-time sync

1–2 million messages/day, near-real-time sync, large partitions, many consumer instances, HPA, some message types process longer.

• Recommendation: Consumer Protocol (KIP-848) (preferred)
• Because: we don't want to stop all consumers if one fails; KIP-848 rebalances server-side → fast and only affects the touched part. Remember to lower group.consumer.session.timeout.ms so dead pods are detected quickly.

Case B: Many partitions (>10), long processing, near-real-time, HPA

• Recommendation: Cooperative or Consumer Protocol (KIP-848) (preferred)
• Because: allows continuing to process messages on unaffected partitions; only delays messages on affected partitions.

Case C: Few partitions (<10), fast processing, no real-time need, HPA

• Recommendation: Classic with Eager (Round Robin)
• Because: with few partitions, revoke + reassign completes fast (few consumers/partitions, less consensus time). If no real-time need, stopping all consumers is acceptable.

Case D: Few partitions, fast messages, no real-time, no HPA

• Recommendation: Classic with Eager (Round Robin)
• Because: no frequent rebalance, eager is simple and good enough.

7. Other Factors Affecting Rebalance

The above describes an ideal scenario. Real-world rebalance time is also affected by:

• A pod terminated without properly disconnecting its Kafka connection (orphaned connections) → the broker must wait for a timeout to notice, slowing rebalance.
• A large consumer group timeout configuration → slower detection (as found in section 4.5).
• And a few more.

Check these before blaming the strategy.

8. Conclusion

A rebalance storm on a dynamic environment (HPA + pod crashes) is a real risk for near-real-time Kafka consumers, especially with the classic protocol + eager strategy — KafkaJS's default.

The lab numbers proved it clearly:

• Eager: killing one consumer → 76% of partitions reshuffled, 8 rebalance starts for just 2 events.
• Cooperative / KIP-848: only 20% of partitions move, 0 ms stop-the-world for untouched pods.
• But: KIP-848 detects dead pods more slowly unless you lower session.timeout — a trade-off to keep in mind.

Three tools, ordered by modernity:

1. Classic Eager: simple, but stop-the-world — not suited for production HPA.
2. Classic Cooperative: keeps unaffected partitions processing — mitigates stop-the-world but still client-side.
3. Consumer Protocol (KIP-848): the future of Kafka — server-side rebalance, fast, stable.

With KafkaJS now unmaintained, moving to the Confluent library is the right production step.

My "goblin" takeaway: Don't just trust benchmarks — reproduce the rebalance scenario in a test environment with your partition/consumer counts, and measure both stop-the-world and detection time. Your system's actual numbers are the truth.

Based on research from the Apache Kafka KIP-848 docs and experiments reproduced on a self-built Kafka lab (Confluent Kafka 8.3.1, KafkaJS + Confluent Kafka for JavaScript).