Skip to content

fix: drain outbound connections during shutdown - #3594

Closed
GrapeBaBa wants to merge 5 commits into
libp2p:mainfrom
GrapeBaBa:fix/graceful-shutdown-dial-drain
Closed

fix: drain outbound connections during shutdown#3594
GrapeBaBa wants to merge 5 commits into
libp2p:mainfrom
GrapeBaBa:fix/graceful-shutdown-dial-drain

Conversation

@GrapeBaBa

Copy link
Copy Markdown
Contributor

Description

Prevent shutdown from finishing while outbound dial jobs or TCP sockets still own referenced native handles.

  • mark the connection manager stopped before service teardown
  • abort and await active dial jobs
  • track outbound TCP sockets and wait for their close events
  • avoid resetAndDestroy after a graceful socket shutdown has started, since libuv does not allow mixing uv_shutdown with uv_tcp_close_reset

This was observed while investigating a Lodestar shutdown hang: https://gist.github.com/nflaig/5f41cfc50f38baf5046a034162943dc3

A real failure left a Node.js worker in Environment::CleanupHandles with an inactive but referenced TCP handle. A minimal Node.js reproduction using socket.end followed by socket.resetAndDestroy also hangs with reset EINVAL, while closing the same socket with socket.destroy exits normally.

Verification

  • Added real TCP regression coverage for outbound socket close and the reset/graceful-close race.
  • Before syncing this branch with the latest main, the affected package lint, doc check, build, and 39 Node.js tests passed.
  • The merged head has not been rerun locally. This is a draft PR and CI is expected to validate the latest main integration.
  • The patched Lodestar image has completed 23 consecutive mainnet shutdown cycles without a hang so far. The 50-cycle soak is still running.

Notes & open questions

The reset guard preserves resetAndDestroy for ordinary aborts. It uses destroy only when writableEnded shows that graceful shutdown has already begun and a TCP reset can no longer be requested safely.

Change checklist

  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation if necessary (this includes comments as well)
  • I have added tests that prove my fix is effective or that my feature works

@GrapeBaBa

Copy link
Copy Markdown
Contributor Author

Superseded by #3596, whose head points exactly to the source used by the running 50-cycle Lodestar validation image.

@GrapeBaBa GrapeBaBa closed this Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant