Rebalancing of Partitions in Kafka Brokers

[Shreya21Mishra]

Hi. I am looking for options to automatically rebalance partitions on Kafka brokers and customise the criteria for rebalancing, other than relying solely on rack awareness

1. Are there any configurable parameters or annotations in Strimzi that enable automatic partition rebalancing among Kafka brokers?
2. Is there any way to set the rebalancing criteria to broker or the disks rather than rackawarness?
3. Also, is there an automatic intra broker rebalancing i.e., rebalancing between the different disks of a broker?

[ppatierno]

There is nothing "automated" in the sense that the Strimzi operator looks the cluster for you and decide to rebalance partitions but we have integration with Cruise Control. Did you have a look at the doc? There is an entire chapter on it https://strimzi.io/docs/operators/latest/deploying#cruise-control-concepts-str

[ppatierno]

Just when the proposal is ready. When it's returned by the CC and the KafkaRebalance status is updated with its information.

[Shreya21Mishra]

@ppatierno
If I have a KafkaRebalance already in ready state, will the cruise control automatically detect the any skewness and create the proposal and rebalance the cluster

For example I already have the above, does this mean it will automatically keep generating proposals and rebalancing the cluster whenever necesary?

[ppatierno]

Nope. I think you should read again the doc and pointing here where it's not clear. It also mention a "refresh" annotation in order to make a new proposal from the same KafkaRebalance resource. As I already mentioned, there is no automation on CC generating proposals or doing rebalancing without user interaction.

[Shreya21Mishra]

@ppatierno Can I do partition rebalance between 2 specific brokers?

[ppatierno]

This could be a more Cruise Control related question. AFAICS from the Cruise Control wiki here https://github.com/linkedin/cruise-control/wiki/REST-APIs#trigger-a-workload-balance there is just a parameter named destination_broker_ids where you can "specify brokers to move replicas to" which is all brokers by default. Not sure it covers what you are asking for.
Said that, it's anyway not configurable via the Strimzi KafkaRebalance custom resource, so there is no support for it right now.