Merged
Conversation
✅ Docs preview readyThe preview is ready to be viewed. View the preview File Changes 0 new, 2 changed, 0 removedBuild ID: 47fe91528b4560d3b32621a2 URL: https://www.apollographql.com/docs/deploy-preview/47fe91528b4560d3b32621a2 |
This comment has been minimized.
This comment has been minimized.
BrynCooke
force-pushed
the
bryn/connection_cancellation_race
branch
from
4c3f00c to
dac90b0
Compare
BrynCooke
force-pushed
the
bryn/connection_cancellation_race
branch
2 times, most recently
from
3d56e5a to
bd2106d
Compare
Previously we were using Notify to indicate to connections that they should shut down when a reload happens. However, this creates a race condition where if a connection is being established at the moment that shutdown is called the new connection will never recieve the shutdown notification. This PR switches to CancellationToken which once cancelled will always yield cancelled immediatey.
BrynCooke
force-pushed
the
bryn/connection_cancellation_race
branch
from
bd2106d to
618eaf7
Compare
bnjjj
approved these changes
Sep 2, 2025
Member
|
@BrynCooke is this a candidate for |
Contributor
Author
|
@abernix I think we could backport it. Will ping you separately. |
| let received_first_request = $received_first_request; | ||
| tokio::pin!(connection); | ||
| tokio::select! { | ||
| // the connection finished first |
Member
There was a problem hiding this comment.
the order doesn't matter unless biased; is used: https://docs.rs/tokio/latest/tokio/macro.select.html#fairness
did you change this only for readability or did you intend to change the priority of each branch?
Contributor
Author
There was a problem hiding this comment.
Yeah, I had actually changed it because I had thought that this was important. I'll revert the ordering.
BrynCooke
commented
Sep 4, 2025
Also revert the change in order for the select.
goto-bus-stop
approved these changes
Sep 5, 2025
BrynCooke
added a commit
that referenced
this pull request
Sep 5, 2025
Co-authored-by: bryn <bryn@apollographql.com> (cherry picked from commit efa9c72)
10 tasks
Member
|
@mergify backport 2.6.2 |
Contributor
✅ Backports have been createdDetails
|
BrynCooke
added a commit
that referenced
this pull request
Sep 5, 2025
Co-authored-by: bryn <bryn@apollographql.com> (cherry picked from commit efa9c72)
10 tasks
BrynCooke
added a commit
that referenced
this pull request
Sep 5, 2025
abernix
added a commit
that referenced
this pull request
Sep 5, 2025
Merged
Merged
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously we were using Notify to indicate to connections that they should shut down when a reload happens. However, this creates a race condition where if a connection is being established at the moment that shutdown is called the new connection will never receive the shutdown notification. Notify only notifies current waiters and will not immediately resolve if new waiters are added.
This PR switches to CancellationToken which once cancelled will always yield cancelled immediately.
This would manifest for users as ever growing memory use over hot reloads as a connection could potentially hold onto a pipeline forever.
Note: there is no unit test
The reason for this is that to trgger this it requires an enormous number of iterations, it's not possible to trigger it as part of a standard CI run.
I have reproduced by introducing sleeps at critical points which allows reproducing of a connection being created after the point where the the connection has shut down.
Checklist
Complete the checklist (and note appropriate exceptions) before the PR is marked ready-for-review.
Exceptions
Note any exceptions here
Notes
Footnotes
It may be appropriate to bring upcoming changes to the attention of other (impacted) groups. Please endeavour to do this before seeking PR approval. The mechanism for doing this will vary considerably, so use your judgement as to how and when to do this. ↩
Configuration is an important part of many changes. Where applicable please try to document configuration examples. ↩
A lot of (if not most) features benefit from built-in observability and
debug-level logs. Please read this guidance on metrics best-practices. ↩Tick whichever testing boxes are applicable. If you are adding Manual Tests, please document the manual testing (extensively) in the Exceptions. ↩