Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Debug #1

Closed
wants to merge 2 commits into from
Closed

Debug #1

wants to merge 2 commits into from

Conversation

zywillc
Copy link
Owner

@zywillc zywillc commented Oct 14, 2022

No description provided.

twmb added a commit to twmb/franz-go that referenced this pull request Oct 20, 2022
@twmb
kgo: work around KIP-814 limitations
d361be6 
I noticed fork zywillc#1 attempting to
fix a problem that I didn't know existed. After a full day of looking
into it, as it turns out, KIP-345's implementation had no solution for
restarting leaders while changing interests, and KIP-814 further doesn't
close the gap. As a client, we can leverage a little bit of KIP-814 to
close the gap ourselves.

We only do this for well known balancers because we cannot be sure if
custom balancers make weird choices depending on time, number of
invocations, instance id / member id oridering, etc., so we cannot
rely on balance plans to be the same from one run to the next.

We do rely on this for our own balancers. Technically this may not be
true if multiple member-id-only members are restarting at once. That's
ok. We expect instance id using groups to be filled with instance id
members and thus always have the same sort ordering in our balancers.

We could add another optional interface to opt in to this enhanced
re-check behavior, but I'm not sure how common it is for somebody to:
(a) write a custom balancer
(b) use instance IDs
(c) change topic interests across restarts
so we'll just leave the option out for now as it'd be complicated to
discover.

Regardless, there's a lot of documentation on what's being done and why.
twmb added a commit to twmb/franz-go that referenced this pull request Oct 20, 2022
@twmb
kgo: work around KIP-814 limitations
b18341d 
I noticed fork zywillc#1 attempting to
fix a problem that I didn't know existed. After a full day of looking
into it, as it turns out, KIP-345's implementation had no solution for
restarting leaders while changing interests, and KIP-814 further doesn't
close the gap. As a client, we can leverage a little bit of KIP-814 to
close the gap ourselves.

We only do this for well known balancers because we cannot be sure if
custom balancers make weird choices depending on time, number of
invocations, instance id / member id oridering, etc., so we cannot
rely on balance plans to be the same from one run to the next.

We do rely on this for our own balancers. Technically this may not be
true if multiple member-id-only members are restarting at once. That's
ok. We expect instance id using groups to be filled with instance id
members and thus always have the same sort ordering in our balancers.

We could add another optional interface to opt in to this enhanced
re-check behavior, but I'm not sure how common it is for somebody to:
(a) write a custom balancer
(b) use instance IDs
(c) change topic interests across restarts
so we'll just leave the option out for now as it'd be complicated to
discover.

Regardless, there's a lot of documentation on what's being done and why.

Also see KAFKA-13435 and the unaddressed KAFKA-12759
@zywillc zywillc closed this Oct 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@twmb twmb twmb left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@zywillc @twmb