KAFKA-8449: Restart tasks on reconfiguration under incremental cooperative rebalancing#6850
Conversation
|
jk8 tests green. jdk11 has one failure in a not relevant test case ( |
rhauch
left a comment
rhauch
left a comment
There was a problem hiding this comment.
Thanks, @kkonstantine! These changes look good, but I have a few pretty minor suggestions.
There was a problem hiding this comment.
Minor: it would help to have a comment here to explain why the tasks to restart are removed from the running tasks.
There was a problem hiding this comment.
Minor: this log line is in a different location in the similar updateConfigsWithEager(...) method. Would it make sense to move this log message to just before the member.requestRejoin() call?
There was a problem hiding this comment.
Moved it to both before member.requestRejoin. I changed it because it was even after we set this variable to false, which made the log message a bit confusing.
There was a problem hiding this comment.
Would it make sense to move this log message to just before the member.requestRejoin() call?
…eager and cooperative Also include the group generation in the log message when the worker receives its assignment
|
Thanks @rhauch ! |
rhauch
left a comment
rhauch
left a comment
There was a problem hiding this comment.
LGTM. Thanks, @kkonstantine!
…ative rebalancing (#6850) Restart task on reconfiguration under incremental cooperative rebalancing, and keep execution paths separate for config updates between eager and cooperative. Include the group generation in the log message when the worker receives its assignment. Author: Konstantine Karantasis <konstantine@confluent.io> Reviewer: Randall Hauch <rhauch@gmail.com>
* AK_REPO/2.3: KAFKA-8155: Add 2.2.0 release to system tests (apache#6597) KAFKA-8404: Add HttpHeader to RestClient HTTP Request and Connector REST API (apache#6791) KAFKA-8473: Adjust Connect system tests for incremental cooperative rebalancing (apache#6872) KAFKA-8475: Temporarily restore SslFactory.sslContext() helper KAFKA-8449: Restart tasks on reconfiguration under incremental cooperative rebalancing (apache#6850) KAFKA-8426; Fix for keeping the ConfigProvider configs consistent with KIP-297 (apache#6750) KAFKA-8425: Fix for correctly handling immutable maps (KIP-421 bug) (apache#6795) MINOR: Reordering the props modification with configs construction
…ative rebalancing (apache#6850) Restart task on reconfiguration under incremental cooperative rebalancing, and keep execution paths separate for config updates between eager and cooperative. Include the group generation in the log message when the worker receives its assignment. Author: Konstantine Karantasis <konstantine@confluent.io> Reviewer: Randall Hauch <rhauch@gmail.com>
2 participants
With incremental cooperative rebalancing tasks are not restarted unless they are reassigned to different worker. Therefore, to load config updates due to connector reconfiguration the currently running tasks that get new configs need to be recorded and considered for restart when connectors and tasks are started after a rebalance.
Committer Checklist (excluded from commit message)