Skip to content

[Membership] Addressing recovery from contention on membership writes #9376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Mar 24, 2025
Merged

[Membership] Addressing recovery from contention on membership writes #9376

merged 6 commits into from
Mar 24, 2025

Conversation

@rkargMsft
Copy link
Contributor

@rkargMsft rkargMsft commented Mar 9, 2025 •
edited
Loading

This alleviates overloading clustering storage when it is performing slower than normal request rates.

Microsoft Reviewers: Open in CodeFlow

This appears to be the only two places with writes that would have retries when encountering slow membership backends that doesn't have exponential backoffs.

@rkargMsft rkargMsft mentioned this pull request Mar 9, 2025
Closed
@rkargMsft rkargMsft changed the title Backing off Periodic Membership refresh Addressing recovery from contention on membership writes Mar 11, 2025
@rkargMsft rkargMsft marked this pull request as ready for review March 12, 2025 21:04
@rkargMsft rkargMsft changed the title Addressing recovery from contention on membership writes [Membership] Addressing recovery from contention on membership writes Mar 13, 2025
dmorganMsft
dmorganMsft approved these changes Mar 21, 2025
View reviewed changes
Copy link
Contributor

@dmorganMsft dmorganMsft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would love to see this in the next release! This helped when using Cassandra clustering with a multi-dc deployment where the LWTs are particularly slow

rkargMsft and others added 6 commits March 20, 2025 19:20
@rkargMsft @ReubenBond
Backing off Periodic Membership refresh
e33b70b 
This alleviates overloading clustering storage when it is performing slower than normal request rates.
@rkargMsft @ReubenBond
Backoff for IAmAliveUpdate calls
878abae
@rkargMsft @ReubenBond
Queueing SuspectOrKill and Kill requests
dadfbd4 
This helps avoid contention and increases the chance that one silo will perform the operation.
@rkargMsft @ReubenBond
Adding timeout on test waiting
3bb6c2b
@rkargMsft @ReubenBond
changing waiting point
accb3cd
@ReubenBond
Add cancellation to ProcessSuspectOrKillLists
78ac08f
@ReubenBond
Copy link
Member

@rkargMsft if my changes look good to you, I'm ready to merge.

@rkargMsft
Copy link
Contributor Author

@ReubenBond LGTM

@ReubenBond ReubenBond merged commit c723211 into dotnet:main Mar 24, 2025
24 of 25 checks passed
@rkargMsft rkargMsft deleted the patch-1 branch March 24, 2025 19:05
@github-actions github-actions bot locked and limited conversation to collaborators Apr 24, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@ReubenBond ReubenBond ReubenBond approved these changes

+1 more reviewer

@dmorganMsft dmorganMsft dmorganMsft approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@rkargMsft @ReubenBond @dmorganMsft