Skip to content

fix: drain outbound connections during shutdown - #3596

Closed
GrapeBaBa wants to merge 4 commits into
libp2p:mainfrom
GrapeBaBa:fix/graceful-shutdown-dial-drain-tested
Closed

fix: drain outbound connections during shutdown#3596
GrapeBaBa wants to merge 4 commits into
libp2p:mainfrom
GrapeBaBa:fix/graceful-shutdown-dial-drain-tested

Conversation

@GrapeBaBa

Copy link
Copy Markdown
Contributor

Description

Prevent shutdown from finishing while outbound dial jobs or TCP sockets still own referenced native handles.

  • mark the connection manager stopped before service teardown
  • abort and await active dial jobs
  • track outbound TCP sockets and wait for their close events
  • avoid resetAndDestroy after graceful socket shutdown has started, since libuv does not allow mixing uv_shutdown with uv_tcp_close_reset

This was observed while investigating a Lodestar shutdown hang: https://gist.github.com/nflaig/5f41cfc50f38baf5046a034162943dc3

Tested version

This draft intentionally points to commit c20c703, the exact js-libp2p source used to build Lodestar image sha256:2c2bcfbe1a9adf72cb9f73dfc605fb3a149e283d8aeae6b0325bbefd5e1b13f9.

The affected compiled libp2p and TCP files in the running image were compared byte-for-byte with the artifacts built from this commit. All hashes match.

The image has completed 44 consecutive effective mainnet shutdown cycles without a hang so far. The 50-cycle soak is still running.

This branch is intentionally not merged with newer main commits yet, so the PR head remains identical to the tested version.

Verification

  • Added real TCP regression coverage for outbound socket close and the reset/graceful-close race.
  • At this exact commit, the affected package lint, doc check, build, and 39 Node.js tests passed.
  • No tests were rerun while opening this draft PR.

Change checklist

  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation if necessary (this includes comments as well)
  • I have added tests that prove my fix is effective or that my feature works

@GrapeBaBa

Copy link
Copy Markdown
Contributor Author

Superseded by #3597. A controlled 50-cycle mainnet shutdown soak showed that only the minimal socket reset guard was required: unmodified Lodestar, 50 clean exits, 0 hangs.

@GrapeBaBa GrapeBaBa closed this Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant