Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Redundant set/unset throttling during the rebalance #1972

Open
CCisGG opened this issue Mar 2, 2023 · 6 comments · May be fixed by #2214
Open

Redundant set/unset throttling during the rebalance #1972

CCisGG opened this issue Mar 2, 2023 · 6 comments · May be fixed by #2214
Assignees
@CCisGG
Labels
functionality A feature request. good first issue A good fit as a first issue in the project.

Comments

@CCisGG
Copy link
Contributor

CCisGG commented Mar 2, 2023 •
edited
Loading

Today, Cruise Control are setting/resetting replication rate throttles on a per-task base. E.g. it set throttling before a task executes:

cruise-control/cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/executor/Executor.java

Line 1415 in 155bf3c

throttleHelper.setThrottles(tasksToExecute.stream().map(ExecutionTask::proposal).collect(Collectors.toList()));

Then clear the throttling for the broker after task finished and if there is no ongoing-task on the same broker:

cruise-control/cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/executor/Executor.java

Line 1437 in 155bf3c

throttleHelper.clearThrottles(completedTasks, inProgressTasks);

Note that each set/clear throttle will be an individual Kafka admin request, will may take a couple of seconds.

The problem is: there can be a lot of tasks in the backlog that is involving the same broker. It doesn't make senses to set/clear the throttling for the same broker again and again on every task execution batch. From local unit test running, it takes ~20 seconds for adminClient to handle each set/clear throttle request.

If we set the throttling for broker at the beginning of the execution, and only clean up the throttling after all tasks finished execution, that should speed up the rebalance.

Note: this is not the concurrency throttling, but the replication bytes throttling. This throttle is set by Cruise Control sending request to kafka brokers to cap the max replication bytes rate.
@CCisGG CCisGG self-assigned this Mar 2, 2023
@CCisGG CCisGG added functionality A feature request. good first issue A good fit as a first issue in the project. labels Mar 2, 2023
@AKARSHITJOSHI
Copy link

@CCisGG I would like to contribute on this.

@CCisGG
Copy link
Contributor Author

CCisGG commented Mar 21, 2023

@CCisGG Thank you Akarshit! I'd recommend to take a look at this https://github.com/linkedin/cruise-control/blob/migrate_to_kafka_2_5/CONTRIBUTING.md before start coding.

@p-palanisami
Copy link

Hey, I am new to Open Source Contributor. @AKARSHITJOSHI Are you still looking into this? If not I wouldd like to take it.

@the-sky7
Copy link

the-sky7 commented Oct 8, 2024

Hi @CCisGG, thanks for sharing the context. looking into it, will appreciate any help or guidance

@aswinayyolath
Copy link
Contributor

Hi @the-sky7, have you had a chance to make any progress on this? If not, I’d be happy to take a look

@aswinayyolath aswinayyolath linked a pull request Oct 27, 2024 that will close this issue
Open
6 tasks
@aswinayyolath
Copy link
Contributor

@CCisGG When you have a moment, could you please take a look at this PR? I would greatly appreciate your insights and any feedback you might have. Thank you very much for your time!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@CCisGG CCisGG

Labels
functionality A feature request. good first issue A good fit as a first issue in the project.
Projects
None yet
Milestone
No milestone
5 participants
@CCisGG @AKARSHITJOSHI @aswinayyolath @p-palanisami @the-sky7