KAFKA-8473: Adjust Connect system tests for incremental cooperative rebalancing#6872
Conversation
|
@rhauch this PR aims to resolve unpredictability in test runs that depends on the order of worker departures (and whether a leader leaves the group first or last). I'd appreciate a review. |
rhauch
left a comment
rhauch
left a comment
There was a problem hiding this comment.
Thanks, @kkonstantine! Overall this looks good, and it will be nice to exercise the different rebalance protocols in the system tests. However, I do have a few questions.
There was a problem hiding this comment.
Does this need to be so large by default?
There was a problem hiding this comment.
So, we are trying to strike a balance here. We don't want this to be too high, because tests will start failing since tasks will remain unassigned, but we also want to exercise the delay logic because workers do indeed come back. Usually with class loading and stuff it takes less than 30 seconds for a worker to come back and join the group. But to guard against flakiness I'm making this 60sec.
In two places the delay will need to increase but only for tests setting protocol to compatible. In one the wait time increased from 70sec to 150sec and another case from 30sec to 120sec. Also, this applies to worst case scenarios when a real failure happens. In normal times, the assertions will take a bit less than that, but indeed more than 60sec.
Would you prefer another value?
There was a problem hiding this comment.
Your description was very helpful. No need to change the value.
|
Thanks @rhauch I had to rebase to remove the change on |
rhauch
left a comment
rhauch
left a comment
There was a problem hiding this comment.
LGTM. Thanks, @kkonstantine!
…ebalancing (#6872) Author: Konstantine Karantasis <konstantine@confluent.io> Reviewer: Randall Hauch <rhauch@gmail.com>
* AK_REPO/2.3: KAFKA-8155: Add 2.2.0 release to system tests (apache#6597) KAFKA-8404: Add HttpHeader to RestClient HTTP Request and Connector REST API (apache#6791) KAFKA-8473: Adjust Connect system tests for incremental cooperative rebalancing (apache#6872) KAFKA-8475: Temporarily restore SslFactory.sslContext() helper KAFKA-8449: Restart tasks on reconfiguration under incremental cooperative rebalancing (apache#6850) KAFKA-8426; Fix for keeping the ConfigProvider configs consistent with KIP-297 (apache#6750) KAFKA-8425: Fix for correctly handling immutable maps (KIP-421 bug) (apache#6795) MINOR: Reordering the props modification with configs construction
…ebalancing (apache#6872) Author: Konstantine Karantasis <konstantine@confluent.io> Reviewer: Randall Hauch <rhauch@gmail.com>
2 participants
As predicted by the new incremental cooperative rebalancing protocol, when a worker leaves the group a delay is triggered, controlled by the scheduled.rebalance.max.delay.ms property, during which the tasks detected as lost due to the workers departure are not reassigned. After this delay, workers are joining for another rebalancing round to reschedule the lost tasks.
System tests that perform worker bounces need to account for this delay.
This PR is also enabled connect system tests on distributed workers for both eager and incremental cooperative rebalancing protocols.
Tested by repeated runs of system tests.
Committer Checklist (excluded from commit message)