Fix rebuild channel hang #1335
Merged
Fix rebuild channel hang #1335
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tiagolobocastro
requested review from
mtzaurus,
dsavitskiy,
Abhinandan-Purkait and
dsharma-dc
dsavitskiy
approved these changes
Mar 20, 2023
mtzaurus
approved these changes
Mar 20, 2023
When creating a rebuild job we were reusing the same comms type which is used to do bi-dir communication between the rebuild frontend and backend. In some situations this could lead into blocking stats because the receiver channel was kept alive by the frontend even after the frontend terminates. Force closing the channel should in theory help but it seems the queue is not drained on close. See: async-channel/issues/23 Instead, let's split the channels into two types meaning only the backend has the receiver side. Signed-off-by: Tiago Castro <[email protected]>
tiagolobocastro
force-pushed
the
rebuild-chan
branch
from
a14c7e6
to
c20056b
Compare
When rebuild completes, right before the nexus completes the rebuild callback it's possible that we get a stats request and in such case we have no stats to give. To address this, then the rebuild backend completes it now sets the final rebuild stats in the states, which will allow the stats to be retrieved. Signed-off-by: Tiago Castro <[email protected]>
tiagolobocastro
force-pushed
the
rebuild-chan
branch
from
c20056b
to
fa55f62
Compare
|
bors merge |
|
Build succeeded: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fix(rebuild): don't clone receiver channel
When creating a rebuild job we were reusing the same comms type which is used to do bi-dir
communication between the rebuild frontend and backend.
In some situations this could lead into blocking stats because the receiver channel was kept
alive by the frontend even after the frontend terminates.
Force closing the channel should in theory help but it seems the queue is not drained on close.
See: async-channel/issues/23
Instead, let's split the channels into two types meaning only the backend has the receiver side.
Signed-off-by: Tiago Castro [email protected]
fix(rebuild): set final rebuild stats
When rebuild completes, right before the nexus completes the rebuild callback it's possible
that we get a stats request and in such case we have no stats to give.
To address this, then the rebuild backend completes it now sets the final rebuild stats in
the states, which will allow the stats to be retrieved.
Signed-off-by: Tiago Castro [email protected]