Connection cancellation race (backport #8169)

[BrynCooke]

Previously we were using Notify to indicate to connections that they should shut down when a reload happens. However, this creates a race condition where if a connection is being established at the moment that shutdown is called the new connection will never receive the shutdown notification. Notify only notifies current waiters and will not immediately resolve if new waiters are added.

This PR switches to CancellationToken which once cancelled will always yield cancelled immediately.

This would manifest for users as ever growing memory use over hot reloads as a connection could potentially hold onto a pipeline forever.

Note: there is no unit test
The reason for this is that to trgger this it requires an enormous number of iterations, it's not possible to trigger it as part of a standard CI run.
I have reproduced by introducing sleeps at critical points which allows reproducing of a connection being created after the point where the the connection has shut down.

---

Checklist

Complete the checklist (and note appropriate exceptions) before the PR is marked ready-for-review.

• PR description explains the motivation for the change and relevant context for reviewing
• PR description links appropriate GitHub/Jira tickets (creating when necessary)
• Changeset is included for user-facing changes
• Changes are compatible¹
• Documentation² completed
• Performance impact assessed and acceptable
• Metrics and logs are added³ and documented
• Tests added and passing⁴
  • Unit tests
  • Integration tests
  • Manual tests, as necessary

Exceptions

Note any exceptions here

Notes

[ROUTER-1427]: https://apollographql.atlassian.net/browse/ROUTER-1427?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

---

This is an automatic backport of pull request #8169 done by Mergify.

Footnotes

1. It may be appropriate to bring upcoming changes to the attention of other (impacted) groups. Please endeavour to do this before seeking PR approval. The mechanism for doing this will vary considerably, so use your judgement as to how and when to do this. ↩

2. Configuration is an important part of many changes. Where applicable please try to document configuration examples. ↩

3. A lot of (if not most) features benefit from built-in observability and debug-level logs. Please read this guidance on metrics best-practices. ↩

4. Tick whichever testing boxes are applicable. If you are adding Manual Tests, please document the manual testing (extensively) in the Exceptions. ↩

[apollo-librarian]

✅ Docs preview ready

The preview is ready to be viewed. View the preview

File Changes

0 new, 11 changed, 0 removed

* graphos/routing/(latest)/cloud/migrate-to-dedicated.mdx
* graphos/routing/(latest)/configuration/envvars.mdx
* graphos/routing/(latest)/configuration/cli.mdx
* graphos/routing/(latest)/configuration/yaml.mdx
* graphos/routing/(latest)/customization/custom-binary.mdx
* graphos/routing/(latest)/customization/overview.mdx
* graphos/routing/(latest)/observability/telemetry/instrumentation/selectors.mdx
* graphos/routing/(latest)/operations/file-upload.mdx
* graphos/routing/(latest)/security/persisted-queries.mdx
* graphos/routing/(latest)/security/request-limits.mdx
* graphos/routing/(latest)/graphos-features.mdx

Build ID: 1f341c106939643e41c475ee
Build Logs: View logs

URL: https://www.apollographql.com/docs/deploy-preview/1f341c106939643e41c475ee