If you are running kafka production workloads at any considerable length of time you have probably faced rebalancing challenges. You need to consider rebalncing your cluster for a number of reasons such as: hot spots or hot brokers across your cluster, needing to rollout changes at cluster level, specific distribution of workloads or topic partitions on a certain group of brokers etc…

A common tool used for these sorts of activities is Cruise Control. I plan to write a lot about Cruise Control. At first glance it comes across like a basic, powerful tool. If you use it a lot to move things around and have a none simplistic setup, things can pile up pretty quickly. So here is the first article on broker demottion.

Kafka Cruise Control (CC) demotes a broker before you terminate it: reorder every partition’s replica list so that broker sits last, then run a preferred election so it stops leading anything. No data moves. It’s a pointer swap and a controller call.

So a routine demotion of two brokers had no business taking as long as it did. Each batch sat for minutes before the next one started (viewable from live logs), and nothing about the operation should have touched the disk or the network at all.

The obvious suspect

CC ramps concurrency up gradually — ours was configured to grow from a floor of 8 to a cap of 24 inter-broker partition movements at a time. Watching the batch, concurrency only ever reached 12. Turning the dial on the concurrency up was the first idea that came to mind: give it more batches, it’ll keep climbing, and the slowness is just the adjuster being conservative.

It wasn’t. The ramp reaching half its ceiling was incidental — the real delay showed up identically whether concurrency was at 8 or 12 or 24. Whatever was slow, it wasn’t the movement concurrency.

Where it actually was

Every inter-broker execution phase in CC’s executor calls a ReplicationThrottleHelper — win or lose, data or no data. That helper’s job is to write a dynamic broker config (leader.replication.throttled.rate / follower.replication.throttled.rate) through the admin client before the batch starts, and clear it again after. Both of those are round trips to the controller, and CC waits for confirmation that the config change actually propagated before it moves on.

For a batch that’s really moving gigabytes, that wait is the cost of doing it safely. For a demotion — a replica-list reorder plus a preferred election, zero bytes in flight — the throttle value is throttling nothing. (there is a small caveat here and thats the safety belt of not overloading the controller broker with 1000s of leadership switch requests - however I will leave that for another article) CC set it and cleared it anyway, every single batch, and paid the full set-and-wait-for-controller wait each time for a number that never did any work.

The fix

A few open pull requests against the upstream LinkedIn Cruise Control project were chasing variants of this same problem. The one that mattered here was the simplest version of it: skip the throttle helper entirely when there’s no data to move.

int numTotalPartitionMovements = _executionTaskManager.numRemainingInterBrokerPartitionMovements();
long totalDataToMoveInMB = _executionTaskManager.remainingInterBrokerDataToMoveInMB();
Long replicationThrottle = _replicationThrottle;
if (totalDataToMoveInMB == 0 && replicationThrottle != null) {
    LOG.info("Skipping replication throttle for this execution: no data to move.");
    replicationThrottle = null;
}
ReplicationThrottleHelper throttleHelper = new ReplicationThrottleHelper(_adminClient, replicationThrottle,
    currentDeadBrokersWithReplicas);

We build our Cruise Control image by pulling a pinned upstream tag and compiling it ourselves, so this went in as a small patch file applied to the vendored source during the Docker build — not a config change, an actual code patch on top of the tag.

Here is the PR that applies the fix - feel free to bump up :)