fix: bound network worker termination so shutdown can complete by nflaig · Pull Request #9790 · ChainSafe/lodestar

nflaig · 2026-08-07T20:55:00Z

Graceful shutdown hangs and the process has to be force-killed:

Aug-07 20:13:03.049 []        info:  Stopping gracefully
Aug-07 20:13:05.065 [network] debug: terminating network worker   <- last shutdown progress
...                                  chain keeps ticking slots for another 54s
dockerd: "Container failed to exit within 1m0s of signal 15 - using the force"

terminateWorkerThread awaits Thread.terminate() outside the timeout race, so the retryCount * retryMs budget is unreachable. The budget is 3s, the hang was 56s, and there is no Worker thread failed to terminate, retrying... in the logs, i.e. it never returned from the first call.

Why terminate never resolves. gdb stacks captured from a live wedged process show the worker is not blocked, it is spinning:

Thread 73 (LWP 1524870 "WorkerThread"):     <- state R, on CPU
#2  uv_run (loop=0x7fd473dc6938, mode=UV_RUN_ONCE)   deps/uv/src/unix/core.c:434
#3  node::Environment::CleanupHandles()
#4  node::Environment::RunCleanup()
#5  node::FreeEnvironment(node::Environment*)
#6  node::worker::Worker::Run()

CleanupHandles() ends in while (handle_cleanup_waiting_ != 0 || request_waiting_ != 0 || !handle_wrap_queue_.IsEmpty()) uv_run(event_loop(), UV_RUN_ONCE);. A libuv handle on the worker's loop never closes, so the loop never exits and the thread never dies. Which handle is still open is not identified.

What it costs. BeaconNode.close() closes the network before chain.persistToDisk(), so the hang means the finalized state is never archived and the db is never closed cleanly. On the affected node checkpoint_states/ was empty for 5 days and a restart fell back to a db state 319 slots behind the head it had at shutdown.

race Thread.terminate() against the timeout so the retryCount * retryMs budget is enforced
return a boolean instead of throwing, so a failed termination does not abort the rest of BeaconNode.close()
bound getApi().close(), an unbounded RPC into the same worker that runs before the archive
log getActiveResourcesInfo() when the network core closes, so the next stuck shutdown can be diagnosed from a log line rather than gdb

Scope. This keeps a stuck worker from costing us the state archive. It does not stop the worker getting stuck, and it does not make the process exit promptly: process.exit() joins every worker via stop_sub_worker_contexts(), confirmed in the same capture, so a stuck shutdown still runs to the process manager's stop timeout.

Thread 1 (LWP 1524136 "MainThread"):
#2  uv_thread_join                     deps/uv/src/unix/thread.c:295
#3  node::worker::Worker::JoinThread()
#4  node::Environment::stop_sub_worker_contexts()
#5  node::DefaultProcessExitHandlerInternal(...)

I tried to fix that here too, by surfacing the failed termination and hard exiting from the CLI. It did not work - on both wedges that occurred during validation the flag read false at the CLI even though the worker had set it, and the process still waited for the docker timeout. That is dropped from this PR rather than shipped unproven, and unref() went with it since process.exit() joins regardless of refcounting.

Testing. 101 mainnet shutdowns on this branch, each after soaking at ~200 peers with 90-160 live inbound QUIC connections for at least 5 minutes:

99/101 archived the finalized state and logged Beacon node closed, including all 7 where the worker failed to terminate
clean shutdowns complete in 5.9-9.1s
the 7 stuck ones still archived and closed internally in ~9s before being force-killed on the docker timeout

The build without this change hung at terminating network worker and lost the archive on all 4 shutdowns observed.

NETWORK_CORE_CLOSE_TIMEOUT_MS is 5s. At the original 3s it tripped on 35 of 101 shutdowns, so the measurement was censored. Raising it to 10s temporarily made it uncensored: 21 shutdowns closed the core in 2.0-4.1s and none hit the bound, so 5s clears the observed max with margin while still bounding an await that sits in front of the archive. Note the 35 censored runs mean it is not established that this RPC always resolves, which is the argument for bounding it at all.

Root cause notes, gdb captures and the handle-walk tooling: https://gist.github.com/nflaig/b266d89c03cdd2c76338823afed5b2c0

AI Assistance Disclosure

Investigation and patch developed with Claude Code. Cause traced from debug logs and gdb captures of a live wedged process, validated on a mainnet node as above.

🤖 Generated with Claude Code

`terminateWorkerThread` awaited `Thread.terminate(worker)` outside the timeout race. `Thread.terminate` resolves to Node's `Worker.terminate()`, which cannot preempt a worker stuck inside a synchronous native (napi) call, so that await can hang indefinitely — making the intended `retryCount * retryMs` budget and the throw unreachable. Graceful shutdown then hangs (observed ~5 min until SIGKILL) instead of failing bounded. Move the `Thread.terminate()` call inside the `Promise.race` so both the terminate call and the termination-event wait are bounded by the timeout. Add a unit test covering the success path and the stuck-terminate case (throws within retryCount * retryMs instead of hanging forever). 🤖 Generated with AI assistance

…meout Complete the bounded-terminate fix so a stuck network worker yields a clean shutdown, not just a bounded one: - Return `false` instead of throwing when the worker can't be terminated. Throwing aborts the rest of `BeaconNode.close()` (the AbortController that stops the clock/chain timers, `chain.persistToDisk()`, and `db.close()`), leaving the process to exit via unhandledRejection with an unclean DB close. - `unref()` the network worker when it can't be terminated: a still-running worker is ref'd and keeps the main event loop alive, so bounding the terminate alone would still prevent the process from exiting on its own. On nodes hitting the hang every shutdown (glamsterdam-devnet-6) this turns each shutdown from a ~5 min zombie / unclean crash into a clean bounded exit. 🤖 Generated with AI assistance Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Address Gemini review on #9582: the `afterEach(vi.useRealTimers())` cleanup was vestigial (fake timers were never enabled in `beforeEach`). Enable them and drive the per-retry `sleep(retryMs)` timeouts with `vi.advanceTimersByTimeAsync`, so the bounded-shutdown test is deterministic on CI instead of depending on real-time delays and a wall-clock assertion. Kept `resolves.toBe(false)` (the function returns false rather than throwing since 6ec9952). 🤖 Generated with AI assistance Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

`WorkerNetworkCore.close()` awaits `getApi().close()`, an RPC into the network worker, without a bound. `BeaconNode.close()` closes the network before calling `chain.persistToDisk()`, so if the worker is wedged the call never settles and shutdown never reaches the archival step - the finalized state is not written and the next start has to replay from an older state. Race it against a 3s timeout and log instead of throwing, terminating the worker right below tears the core down anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

chatgpt-codex-connector · 2026-08-07T20:55:04Z

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

github-actions · 2026-08-07T21:11:01Z

Performance Report

✔️ no performance regression detected

Full benchmark results

Benchmark suite	Current: `6524b9e`	Previous: `4741558`	Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc	903.01 us/op	881.74 us/op	1.02
getPubkeys - validatorsArr - req 1000 vs - 250000 vc	39.759 us/op	40.710 us/op	0.98
BLS verify - blst	721.70 us/op	708.09 us/op	1.02
BLS verifyMultipleSignatures 3 - blst	1.3182 ms/op	1.3508 ms/op	0.98
BLS verifyMultipleSignatures 8 - blst	2.1052 ms/op	2.1470 ms/op	0.98
BLS verifyMultipleSignatures 32 - blst	6.6993 ms/op	6.7211 ms/op	1.00
BLS verifyMultipleSignatures 64 - blst	12.681 ms/op	12.980 ms/op	0.98
BLS verifyMultipleSignatures 128 - blst	24.644 ms/op	25.433 ms/op	0.97
BLS deserializing 10000 signatures	617.62 ms/op	634.26 ms/op	0.97
BLS deserializing 100000 signatures	6.0620 s/op	6.4417 s/op	0.94
BLS verifyMultipleSignatures - same message - 3 - blst	665.89 us/op	722.03 us/op	0.92
BLS verifyMultipleSignatures - same message - 8 - blst	791.41 us/op	941.93 us/op	0.84
BLS verifyMultipleSignatures - same message - 32 - blst	1.3754 ms/op	1.4650 ms/op	0.94
BLS verifyMultipleSignatures - same message - 64 - blst	2.1676 ms/op	2.3273 ms/op	0.93
BLS verifyMultipleSignatures - same message - 128 - blst	3.8722 ms/op	4.0082 ms/op	0.97
BLS aggregatePubkeys 32 - blst	16.768 us/op	17.740 us/op	0.95
BLS aggregatePubkeys 128 - blst	59.685 us/op	63.483 us/op	0.94
getSlashingsAndExits - default max	44.524 us/op	47.229 us/op	0.94
getSlashingsAndExits - 2k	328.30 us/op	339.13 us/op	0.97
proposeBlockBody type=full, size=empty	609.23 us/op	660.94 us/op	0.92
isKnown best case - 1 super set check	157.00 ns/op	166.00 ns/op	0.95
isKnown normal case - 2 super set checks	159.00 ns/op	164.00 ns/op	0.97
isKnown worse case - 16 super set checks	156.00 ns/op	166.00 ns/op	0.94
validate api signedAggregateAndProof - struct	1.4329 ms/op	1.5224 ms/op	0.94
validate gossip signedAggregateAndProof - struct	1.4321 ms/op	1.5199 ms/op	0.94
batch validate gossip attestation - vc 640000 - chunk 32	103.61 us/op	107.28 us/op	0.97
batch validate gossip attestation - vc 640000 - chunk 64	91.137 us/op	96.494 us/op	0.94
batch validate gossip attestation - vc 640000 - chunk 128	84.133 us/op	89.737 us/op	0.94
batch validate gossip attestation - vc 640000 - chunk 256	80.653 us/op	86.922 us/op	0.93
bytes32 toHexString	274.00 ns/op	291.00 ns/op	0.94
bytes32 Buffer.toString(hex)	170.00 ns/op	174.00 ns/op	0.98
bytes32 Buffer.toString(hex) from Uint8Array	229.00 ns/op	246.00 ns/op	0.93
bytes32 Buffer.toString(hex) + 0x	168.00 ns/op	172.00 ns/op	0.98
Return object 10000 times	0.20380 ns/op	0.21550 ns/op	0.95
Throw Error 10000 times	3.1310 us/op	3.4578 us/op	0.91
toHex	88.703 ns/op	101.73 ns/op	0.87
Buffer.from	79.891 ns/op	87.283 ns/op	0.92
shared Buffer	54.715 ns/op	60.522 ns/op	0.90
fastMsgIdFn sha256 / 200 bytes	1.4020 us/op	1.4840 us/op	0.94
fastMsgIdFn h32 xxhash / 200 bytes	152.00 ns/op	153.00 ns/op	0.99
fastMsgIdFn h64 xxhash / 200 bytes	196.00 ns/op	198.00 ns/op	0.99
fastMsgIdFn sha256 / 1000 bytes	4.4950 us/op	4.7680 us/op	0.94
fastMsgIdFn h32 xxhash / 1000 bytes	236.00 ns/op	247.00 ns/op	0.96
fastMsgIdFn h64 xxhash / 1000 bytes	247.00 ns/op	248.00 ns/op	1.00
fastMsgIdFn sha256 / 10000 bytes	40.056 us/op	42.266 us/op	0.95
fastMsgIdFn h32 xxhash / 10000 bytes	1.2380 us/op	1.3090 us/op	0.95
fastMsgIdFn h64 xxhash / 10000 bytes	809.00 ns/op	847.00 ns/op	0.96
send data - 1000 256B messages	4.2245 ms/op	4.3099 ms/op	0.98
send data - 1000 512B messages	5.2943 ms/op	5.4165 ms/op	0.98
send data - 1000 1024B messages	5.4742 ms/op	5.5881 ms/op	0.98
send data - 1000 1200B messages	6.2800 ms/op	6.5300 ms/op	0.96
send data - 1000 2048B messages	17.335 ms/op	23.886 ms/op	0.73
send data - 1000 4096B messages	27.449 ms/op	55.138 ms/op	0.50
send data - 1000 16384B messages	179.82 ms/op	308.85 ms/op	0.58
send data - 1000 65536B messages	585.58 ms/op	1.2519 s/op	0.47
enrSubnets - fastDeserialize 64 bits	706.00 ns/op	765.00 ns/op	0.92
enrSubnets - ssz BitVector 64 bits	249.00 ns/op	266.00 ns/op	0.94
enrSubnets - fastDeserialize 4 bits	97.000 ns/op	104.00 ns/op	0.93
enrSubnets - ssz BitVector 4 bits	244.00 ns/op	263.00 ns/op	0.93
prioritizePeers score -10:0 att 32-0.1 sync 2-0	192.86 us/op	199.95 us/op	0.96
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25	218.00 us/op	228.83 us/op	0.95
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5	320.18 us/op	333.05 us/op	0.96
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75	563.07 us/op	595.04 us/op	0.95
prioritizePeers score 0:0 att 64-1 sync 4-1	664.96 us/op	698.00 us/op	0.95
array of 16000 items push then shift	1.2031 us/op	1.2865 us/op	0.94
LinkedList of 16000 items push then shift	7.1990 ns/op	7.0270 ns/op	1.02
array of 16000 items push then pop	63.682 ns/op	66.719 ns/op	0.95
LinkedList of 16000 items push then pop	5.7600 ns/op	6.1040 ns/op	0.94
array of 24000 items push then shift	1.7925 us/op	1.9119 us/op	0.94
LinkedList of 24000 items push then shift	6.9170 ns/op	6.8230 ns/op	1.01
array of 24000 items push then pop	92.491 ns/op	93.731 ns/op	0.99
LinkedList of 24000 items push then pop	5.8680 ns/op	6.0340 ns/op	0.97
intersect bitArray bitLen 8	3.7520 ns/op	3.8640 ns/op	0.97
intersect array and set length 8	28.225 ns/op	29.451 ns/op	0.96
intersect bitArray bitLen 128	22.907 ns/op	23.308 ns/op	0.98
intersect array and set length 128	505.24 ns/op	496.87 ns/op	1.02
bitArray.getTrueBitIndexes() bitLen 128	910.00 ns/op	913.00 ns/op	1.00
bitArray.getTrueBitIndexes() bitLen 248	1.6230 us/op	1.6610 us/op	0.98
bitArray.getTrueBitIndexes() bitLen 512	3.4440 us/op	3.4100 us/op	1.01
Full columns - reconstruct all 6 blobs	232.19 us/op	110.75 us/op	2.10
Full columns - reconstruct half of the blobs out of 6	90.430 us/op	63.940 us/op	1.41
Full columns - reconstruct single blob out of 6	30.982 us/op	33.218 us/op	0.93
Half columns - reconstruct all 6 blobs	370.51 ms/op	377.45 ms/op	0.98
Half columns - reconstruct half of the blobs out of 6	186.07 ms/op	188.23 ms/op	0.99
Half columns - reconstruct single blob out of 6	67.870 ms/op	66.066 ms/op	1.03
Set add up to 64 items then delete first	1.5566 us/op	1.6577 us/op	0.94
OrderedSet add up to 64 items then delete first	2.3880 us/op	2.5131 us/op	0.95
Set add up to 64 items then delete last	1.7665 us/op	1.8525 us/op	0.95
OrderedSet add up to 64 items then delete last	2.6475 us/op	2.7547 us/op	0.96
Set add up to 64 items then delete middle	1.7551 us/op	1.8626 us/op	0.94
OrderedSet add up to 64 items then delete middle	4.2909 us/op	4.2183 us/op	1.02
Set add up to 128 items then delete first	3.4629 us/op	3.7093 us/op	0.93
OrderedSet add up to 128 items then delete first	5.3594 us/op	6.2141 us/op	0.86
Set add up to 128 items then delete last	3.3460 us/op	3.5712 us/op	0.94
OrderedSet add up to 128 items then delete last	5.1333 us/op	5.2977 us/op	0.97
Set add up to 128 items then delete middle	3.4375 us/op	3.5905 us/op	0.96
OrderedSet add up to 128 items then delete middle	10.585 us/op	11.470 us/op	0.92
Set add up to 256 items then delete first	7.0370 us/op	7.3395 us/op	0.96
OrderedSet add up to 256 items then delete first	11.145 us/op	11.511 us/op	0.97
Set add up to 256 items then delete last	6.6575 us/op	7.2698 us/op	0.92
OrderedSet add up to 256 items then delete last	10.758 us/op	11.014 us/op	0.98
Set add up to 256 items then delete middle	6.7036 us/op	7.4635 us/op	0.90
OrderedSet add up to 256 items then delete middle	33.421 us/op	34.741 us/op	0.96
runFastConfirmationRules vc:100000 bc:96 eq:0	4.7128 ms/op	4.5362 ms/op	1.04
runFastConfirmationRules vc:600000 bc:96 eq:0	33.901 ms/op	35.073 ms/op	0.97
runFastConfirmationRules vc:1000000 bc:96 eq:0	55.274 ms/op	57.916 ms/op	0.95
runFastConfirmationRules vc:600000 bc:320 eq:0	33.385 ms/op	33.944 ms/op	0.98
runFastConfirmationRules vc:100000 bc:96 eq:1000	1.0458 s/op	1.0845 s/op	0.96
pass gossip attestations to forkchoice per slot	2.4344 ms/op	2.5715 ms/op	0.95
forkChoice updateHead vc 100000 bc 64 eq 0	390.00 us/op	426.85 us/op	0.91
forkChoice updateHead vc 600000 bc 64 eq 0	2.2931 ms/op	2.5174 ms/op	0.91
forkChoice updateHead vc 1000000 bc 64 eq 0	3.8530 ms/op	4.2331 ms/op	0.91
forkChoice updateHead vc 600000 bc 320 eq 0	2.3295 ms/op	2.5814 ms/op	0.90
forkChoice updateHead vc 600000 bc 1200 eq 0	2.3645 ms/op	2.6067 ms/op	0.91
forkChoice updateHead vc 600000 bc 7200 eq 0	2.7167 ms/op	2.9094 ms/op	0.93
forkChoice updateHead vc 600000 bc 64 eq 1000	2.3263 ms/op	2.6275 ms/op	0.89
forkChoice updateHead vc 600000 bc 64 eq 10000	2.4032 ms/op	2.7165 ms/op	0.88
forkChoice updateHead vc 600000 bc 64 eq 300000	6.4534 ms/op	6.5254 ms/op	0.99
computeDeltas 1400000 validators 0% inactive	11.586 ms/op	12.460 ms/op	0.93
computeDeltas 1400000 validators 10% inactive	11.020 ms/op	12.054 ms/op	0.91
computeDeltas 1400000 validators 20% inactive	10.397 ms/op	11.543 ms/op	0.90
computeDeltas 1400000 validators 50% inactive	8.4417 ms/op	9.2567 ms/op	0.91
computeDeltas 2100000 validators 0% inactive	17.449 ms/op	19.183 ms/op	0.91
computeDeltas 2100000 validators 10% inactive	16.560 ms/op	18.109 ms/op	0.91
computeDeltas 2100000 validators 20% inactive	15.644 ms/op	17.117 ms/op	0.91
computeDeltas 2100000 validators 50% inactive	10.304 ms/op	11.928 ms/op	0.86
altair processAttestation - 250000 vs - 7PWei normalcase	1.7958 ms/op	1.5775 ms/op	1.14
altair processAttestation - 250000 vs - 7PWei worstcase	2.5438 ms/op	2.3604 ms/op	1.08
altair processAttestation - setStatus - 1/6 committees join	98.601 us/op	104.23 us/op	0.95
altair processAttestation - setStatus - 1/3 committees join	189.80 us/op	200.02 us/op	0.95
altair processAttestation - setStatus - 1/2 committees join	274.61 us/op	279.91 us/op	0.98
altair processAttestation - setStatus - 2/3 committees join	351.84 us/op	366.06 us/op	0.96
altair processAttestation - setStatus - 4/5 committees join	490.11 us/op	501.76 us/op	0.98
altair processAttestation - setStatus - 100% committees join	577.76 us/op	595.14 us/op	0.97
altair processBlock - 250000 vs - 7PWei normalcase	3.7183 ms/op	2.9440 ms/op	1.26
altair processBlock - 250000 vs - 7PWei normalcase hashState	15.922 ms/op	15.390 ms/op	1.03
altair processBlock - 250000 vs - 7PWei worstcase	20.242 ms/op	21.200 ms/op	0.95
altair processBlock - 250000 vs - 7PWei worstcase hashState	40.700 ms/op	40.478 ms/op	1.01
phase0 processBlock - 250000 vs - 7PWei normalcase	1.2657 ms/op	1.3285 ms/op	0.95
phase0 processBlock - 250000 vs - 7PWei worstcase	16.526 ms/op	17.730 ms/op	0.93
altair processEth1Data - 250000 vs - 7PWei normalcase	285.90 us/op	295.77 us/op	0.97
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:16	3.1000 us/op	6.0560 us/op	0.51
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:220	19.110 us/op	29.740 us/op	0.64
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:43	5.3370 us/op	7.2250 us/op	0.74
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:19	3.4340 us/op	7.2640 us/op	0.47
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1021	82.376 us/op	92.985 us/op	0.89
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11778	1.2961 ms/op	1.3658 ms/op	0.95
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384	1.7077 ms/op	1.7718 ms/op	0.96
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384	1.7120 ms/op	1.7740 ms/op	0.97
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384	3.4712 ms/op	3.6286 ms/op	0.96
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384	1.9390 ms/op	2.0267 ms/op	0.96
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384	3.7734 ms/op	3.9261 ms/op	0.96
Tree 40 250000 create	297.95 ms/op	314.22 ms/op	0.95
Tree 40 250000 get(125000)	88.608 ns/op	97.934 ns/op	0.90
Tree 40 250000 set(125000)	943.91 ns/op	1.0291 us/op	0.92
Tree 40 250000 toArray()	10.952 ms/op	10.272 ms/op	1.07
Tree 40 250000 iterate all - toArray() + loop	10.838 ms/op	9.5615 ms/op	1.13
Tree 40 250000 iterate all - get(i)	33.926 ms/op	36.514 ms/op	0.93
Array 250000 create	2.0058 ms/op	2.0610 ms/op	0.97
Array 250000 clone - spread	626.56 us/op	644.05 us/op	0.97
Array 250000 get(125000)	0.28300 ns/op	0.29800 ns/op	0.95
Array 250000 set(125000)	0.28600 ns/op	0.30200 ns/op	0.95
Array 250000 iterate all - loop	56.104 us/op	58.048 us/op	0.97
phase0 afterProcessEpoch - 250000 vs - 7PWei	48.101 ms/op	39.794 ms/op	1.21
Array.fill - length 1000000	2.0023 ms/op	2.3343 ms/op	0.86
Array push - length 1000000	8.3414 ms/op	7.4785 ms/op	1.12
Array.get	0.19757 ns/op	0.20325 ns/op	0.97
Uint8Array.get	0.23348 ns/op	0.24430 ns/op	0.96
phase0 beforeProcessEpoch - 250000 vs - 7PWei	16.410 ms/op	11.374 ms/op	1.44
altair processEpoch - mainnet_e81889	265.84 ms/op	256.37 ms/op	1.04
mainnet_e81889 - altair beforeProcessEpoch	14.678 ms/op	70.023 ms/op	0.21
mainnet_e81889 - altair processJustificationAndFinalization	6.0950 us/op	5.1940 us/op	1.17
mainnet_e81889 - altair processInactivityUpdates	3.3535 ms/op	3.4781 ms/op	0.96
mainnet_e81889 - altair processRewardsAndPenalties	19.643 ms/op	19.629 ms/op	1.00
mainnet_e81889 - altair processRegistryUpdates	562.00 ns/op	596.00 ns/op	0.94
mainnet_e81889 - altair processSlashings	138.00 ns/op	147.00 ns/op	0.94
mainnet_e81889 - altair processEth1DataReset	135.00 ns/op	145.00 ns/op	0.93
mainnet_e81889 - altair processEffectiveBalanceUpdates	2.0027 ms/op	1.1701 ms/op	1.71
mainnet_e81889 - altair processSlashingsReset	687.00 ns/op	729.00 ns/op	0.94
mainnet_e81889 - altair processRandaoMixesReset	1.2770 us/op	1.0640 us/op	1.20
mainnet_e81889 - altair processHistoricalRootsUpdate	137.00 ns/op	147.00 ns/op	0.93
mainnet_e81889 - altair processParticipationFlagUpdates	443.00 ns/op	456.00 ns/op	0.97
mainnet_e81889 - altair processSyncCommitteeUpdates	112.00 ns/op	122.00 ns/op	0.92
mainnet_e81889 - altair afterProcessEpoch	40.374 ms/op	41.701 ms/op	0.97
capella processEpoch - mainnet_e217614	753.14 ms/op	767.13 ms/op	0.98
mainnet_e217614 - capella beforeProcessEpoch	59.685 ms/op	62.151 ms/op	0.96
mainnet_e217614 - capella processJustificationAndFinalization	6.2040 us/op	5.4050 us/op	1.15
mainnet_e217614 - capella processInactivityUpdates	14.756 ms/op	13.143 ms/op	1.12
mainnet_e217614 - capella processRewardsAndPenalties	84.103 ms/op	88.164 ms/op	0.95
mainnet_e217614 - capella processRegistryUpdates	4.4750 us/op	4.5640 us/op	0.98
mainnet_e217614 - capella processSlashings	139.00 ns/op	147.00 ns/op	0.95
mainnet_e217614 - capella processEth1DataReset	135.00 ns/op	139.00 ns/op	0.97
mainnet_e217614 - capella processEffectiveBalanceUpdates	12.236 ms/op	6.3047 ms/op	1.94
mainnet_e217614 - capella processSlashingsReset	664.00 ns/op	715.00 ns/op	0.93
mainnet_e217614 - capella processRandaoMixesReset	1.1740 us/op	1.2470 us/op	0.94
mainnet_e217614 - capella processHistoricalRootsUpdate	141.00 ns/op	149.00 ns/op	0.95
mainnet_e217614 - capella processParticipationFlagUpdates	462.00 ns/op	466.00 ns/op	0.99
mainnet_e217614 - capella afterProcessEpoch	104.02 ms/op	109.57 ms/op	0.95
phase0 processEpoch - mainnet_e58758	310.38 ms/op	294.30 ms/op	1.05
mainnet_e58758 - phase0 beforeProcessEpoch	61.766 ms/op	62.972 ms/op	0.98
mainnet_e58758 - phase0 processJustificationAndFinalization	6.2960 us/op	5.4470 us/op	1.16
mainnet_e58758 - phase0 processRewardsAndPenalties	33.386 ms/op	15.509 ms/op	2.15
mainnet_e58758 - phase0 processRegistryUpdates	2.2400 us/op	2.2970 us/op	0.98
mainnet_e58758 - phase0 processSlashings	139.00 ns/op	144.00 ns/op	0.97
mainnet_e58758 - phase0 processEth1DataReset	135.00 ns/op	136.00 ns/op	0.99
mainnet_e58758 - phase0 processEffectiveBalanceUpdates	1.5474 ms/op	961.38 us/op	1.61
mainnet_e58758 - phase0 processSlashingsReset	844.00 ns/op	884.00 ns/op	0.95
mainnet_e58758 - phase0 processRandaoMixesReset	1.2930 us/op	1.3290 us/op	0.97
mainnet_e58758 - phase0 processHistoricalRootsUpdate	138.00 ns/op	382.00 ns/op	0.36
mainnet_e58758 - phase0 processParticipationRecordUpdates	1.0280 us/op	1.0920 us/op	0.94
mainnet_e58758 - phase0 afterProcessEpoch	34.273 ms/op	34.535 ms/op	0.99
phase0 processEffectiveBalanceUpdates - 250000 normalcase	986.26 us/op	1.0131 ms/op	0.97
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5	1.6001 ms/op	1.7221 ms/op	0.93
altair processInactivityUpdates - 250000 normalcase	14.901 ms/op	10.748 ms/op	1.39
altair processInactivityUpdates - 250000 worstcase	10.684 ms/op	11.232 ms/op	0.95
phase0 processRegistryUpdates - 250000 normalcase	2.3000 us/op	2.3670 us/op	0.97
phase0 processRegistryUpdates - 250000 badcase_full_deposits	144.64 us/op	141.09 us/op	1.03
phase0 processRegistryUpdates - 250000 worstcase 0.5	75.349 ms/op	61.642 ms/op	1.22
altair processRewardsAndPenalties - 250000 normalcase	14.966 ms/op	16.115 ms/op	0.93
altair processRewardsAndPenalties - 250000 worstcase	14.496 ms/op	15.898 ms/op	0.91
phase0 getAttestationDeltas - 250000 normalcase	5.1476 ms/op	5.6006 ms/op	0.92
phase0 getAttestationDeltas - 250000 worstcase	5.2115 ms/op	5.5190 ms/op	0.94
phase0 processSlashings - 250000 worstcase	62.580 us/op	63.788 us/op	0.98
altair processSyncCommitteeUpdates - 250000	11.463 ms/op	10.303 ms/op	1.11
BeaconState.hashTreeRoot - No change	175.00 ns/op	181.00 ns/op	0.97
BeaconState.hashTreeRoot - 1 full validator	77.434 us/op	77.842 us/op	0.99
BeaconState.hashTreeRoot - 32 full validator	876.27 us/op	924.75 us/op	0.95
BeaconState.hashTreeRoot - 512 full validator	8.5703 ms/op	8.3148 ms/op	1.03
BeaconState.hashTreeRoot - 1 validator.effectiveBalance	98.961 us/op	103.37 us/op	0.96
BeaconState.hashTreeRoot - 32 validator.effectiveBalance	1.3957 ms/op	1.5677 ms/op	0.89
BeaconState.hashTreeRoot - 512 validator.effectiveBalance	20.209 ms/op	18.996 ms/op	1.06
BeaconState.hashTreeRoot - 1 balances	86.641 us/op	90.494 us/op	0.96
BeaconState.hashTreeRoot - 32 balances	716.11 us/op	811.48 us/op	0.88
BeaconState.hashTreeRoot - 512 balances	7.4150 ms/op	6.4559 ms/op	1.15
BeaconState.hashTreeRoot - 250000 balances	139.37 ms/op	113.94 ms/op	1.22
aggregationBits - 2048 els - zipIndexesInBitList	21.776 us/op	20.230 us/op	1.08
regular array get 100000 times	23.017 us/op	23.366 us/op	0.99
wrappedArray get 100000 times	22.718 us/op	23.412 us/op	0.97
arrayWithProxy get 100000 times	17.342 ms/op	16.547 ms/op	1.05
ssz.Root.equals	25.905 ns/op	82.522 ns/op	0.31
byteArrayEquals	21.338 ns/op	21.597 ns/op	0.99
Buffer.compare	8.6170 ns/op	8.9160 ns/op	0.97
processSlot - 1 slots	10.416 us/op	9.5160 us/op	1.09
processSlot - 32 slots	2.2033 ms/op	2.1438 ms/op	1.03
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei	7.6867 ms/op	4.0507 ms/op	1.90
getCommitteeAssignments - req 1 vs - 250000 vc	1.6842 ms/op	1.6821 ms/op	1.00
getCommitteeAssignments - req 100 vs - 250000 vc	3.4466 ms/op	3.4583 ms/op	1.00
getCommitteeAssignments - req 1000 vs - 250000 vc	3.7639 ms/op	3.7190 ms/op	1.01
findModifiedValidators - 10000 modified validators	784.54 ms/op	620.09 ms/op	1.27
findModifiedValidators - 1000 modified validators	560.09 ms/op	446.99 ms/op	1.25
findModifiedValidators - 100 modified validators	369.58 ms/op	301.47 ms/op	1.23
findModifiedValidators - 10 modified validators	187.09 ms/op	270.65 ms/op	0.69
findModifiedValidators - 1 modified validators	188.55 ms/op	170.89 ms/op	1.10
findModifiedValidators - no difference	191.08 ms/op	148.89 ms/op	1.28
migrate state 1500000 validators, 3400 modified, 2000 new	2.6464 s/op	3.0606 s/op	0.86
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei	3.6500 ns/op	3.4900 ns/op	1.05
state getBlockRootAtSlot - 250000 vs - 7PWei	349.11 ns/op	278.55 ns/op	1.25
computeProposerIndex 100000 validators	1.2818 ms/op	1.3403 ms/op	0.96
getNextSyncCommitteeIndices 1000 validators	2.7337 ms/op	2.8640 ms/op	0.95
getNextSyncCommitteeIndices 10000 validators	24.011 ms/op	25.454 ms/op	0.94
getNextSyncCommitteeIndices 100000 validators	83.801 ms/op	84.715 ms/op	0.99
computeProposers - vc 250000	546.14 us/op	558.46 us/op	0.98
computeEpochShuffling - vc 250000	37.629 ms/op	38.950 ms/op	0.97
getNextSyncCommittee - vc 250000	9.3075 ms/op	9.4615 ms/op	0.98
nodejs block root to RootHex using toHex	95.863 ns/op	97.311 ns/op	0.99
nodejs block root to RootHex using toRootHex	60.702 ns/op	63.463 ns/op	0.96
nodejs fromHex(blob)	701.03 us/op	738.97 us/op	0.95
nodejs fromHexInto(blob)	596.03 us/op	639.94 us/op	0.93
nodejs block root to RootHex using the deprecated toHexString	425.58 ns/op	463.95 ns/op	0.92
nodejs byteArrayEquals 32 bytes (block root)	24.627 ns/op	26.247 ns/op	0.94
nodejs byteArrayEquals 48 bytes (pubkey)	35.464 ns/op	37.797 ns/op	0.94
nodejs byteArrayEquals 96 bytes (signature)	32.843 ns/op	36.162 ns/op	0.91
nodejs byteArrayEquals 1024 bytes	39.260 ns/op	45.713 ns/op	0.86
nodejs byteArrayEquals 131072 bytes (blob)	1.6676 us/op	1.7829 us/op	0.94
browser block root to RootHex using toHex	137.71 ns/op	146.13 ns/op	0.94
browser block root to RootHex using toRootHex	125.10 ns/op	130.31 ns/op	0.96
browser fromHex(blob)	1.4871 ms/op	1.7265 ms/op	0.86
browser fromHexInto(blob)	597.81 us/op	642.45 us/op	0.93
browser block root to RootHex using the deprecated toHexString	434.51 ns/op	332.50 ns/op	1.31
browser byteArrayEquals 32 bytes (block root)	26.713 ns/op	28.344 ns/op	0.94
browser byteArrayEquals 48 bytes (pubkey)	38.121 ns/op	40.004 ns/op	0.95
browser byteArrayEquals 96 bytes (signature)	71.072 ns/op	75.283 ns/op	0.94
browser byteArrayEquals 1024 bytes	717.08 ns/op	765.40 ns/op	0.94
browser byteArrayEquals 131072 bytes (blob)	90.874 us/op	96.465 us/op	0.94

by benchmarkbot/action

Every other line in `WorkerNetworkCore.close()` is debug, and until the pending `connection.closed()` promises in js-libp2p-quic are fixed this path is hit on every shutdown of a node with live QUIC connections. Warning on each restart is noise for something the operator can not act on: shutdown still completes, state is archived, the db is closed and the unref-ed worker dies with the process. `terminateWorkerThread` returns `false`, so the caller owns how severe a failed termination is rather than the helper hardcoding a level. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2.3s is the normal close time, 3s was cutting it too fine and tripped the timeout on most restarts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Measured 1.85-4.03s across 17 mainnet shutdowns, including on a build where the worker later wedged, so the call is not the hang risk the comment claimed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Captured gdb stacks from a wedged worker show it spinning in `Environment::CleanupHandles()` on `uv_run(UV_RUN_ONCE)`, state R, because a libuv handle on its loop never closes. It is not blocked in a native call that V8 can not preempt, and unref does not let the process exit, `process.exit()` still joins the thread. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

5 of 17 measured shutdowns were censored at the old 3s bound so the tail is unknown, and the only uncensored observation above it was 4.03s. Cutting the close short leaves libp2p handles open, and an unclosed handle is exactly what makes the worker spin in CleanupHandles, so err on the generous side. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`process.exit()` joins every worker via `stop_sub_worker_contexts()` regardless of refcounting, confirmed by gdb stacks of the main thread, so unref-ing the worker changes nothing on this path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The 10s bound made the measurement uncensored: 20 mainnet shutdowns closed the core in 2.0-4.1s and none hit the bound. 5s clears the observed max with margin. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`Worker.terminate()` never resolves when the worker spins in `Environment::CleanupHandles()` waiting on a libuv handle that never closes. Logging `getActiveResourcesInfo()` right after libp2p stops names the handle types still holding that loop, so the next occurrence can be diagnosed from a log line instead of gdb. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

21 measurements not 20, and the leaked handle was never attributed to libp2p, only that an unclosed libuv handle is what keeps the worker spinning. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

nflaig

LGTM, this was mostly done by claude debugging on my mainnet node, I don't like that we have to add mitigations like this but this issue has been causing a lot of user facing problems and also has been a problem on devnets. The changes in this PR ensure that we always archive the state correctly, there are still few edge cases where the process was not closing correctly and the process manager (docker in my case) had to force exit the process, this is not nice but at least we have the state archived and from my testing (rather claude testing) this was really rare.

The worker is not blocked in a native call, it spins in CleanupHandles, and the resources logged on close are candidates for the handle that fails to close rather than the established cause. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

spiral-ladder

LGTM, even if we don't fix the underlying issue bounding worker termination seems sane anyway

Picks up ChainSafe/js-libp2p-quic#66, released as 2.1.3. `Connection::closed()` only resolves once quinn reports the connection closed, so a stalled connection driver leaves it pending forever, and with it the napi deferred and threadsafe function backing that promise. libp2p will eventually drop such a peer on its own and call `abort()`, but before 2.1.3 the promise stayed pending even then, so each affected connection leaked those resources for the lifetime of the process. 2.1.3 settles it on `abort()` as well, which turns a permanent leak into none. To be clear about what it does not do: it does not make libp2p notice a dead peer any sooner. By the time `abort()` is called libp2p has already concluded the connection is gone and called `onTransportClosed()` itself. This is resource hygiene, not a peer state fix. **This is not a fix for the shutdown hang.** I originally wrote that patch believing the pending promises were what kept the network worker from terminating. They are not - gdb stacks from a live wedged worker show it spinning in Node's `Environment::CleanupHandles()` on a libuv handle that never closes, and the wedge rate was identical with and without the patch (4 in 20 shutdowns vs 1 in 5 on 2.1.2). #9790 is the mitigation for that, and the underlying handle is still unidentified. On the timing of `onTransportClosed()`: `abort()` runs on every locally initiated close, not just at shutdown, so it is worth being precise about what changes. libp2p already calls `onTransportClosed()` itself immediately after `sendClose()` (`abstract-multiaddr-connection.js`), and the method guards every state transition, so it is idempotent. The settled promise therefore produces a redundant call at a moment the framework was transitioning anyway, rather than an genuinely earlier notification. A remote initiated close does not call `abort()` at all and is unaffected. The change only has an observable effect in the stalled driver case it was written for. **Testing** 2.1.3 is byte-identical in behaviour to the build I ran on a mainnet node for 20 shutdowns at ~200 peers with 90-160 live inbound QUIC connections, no regression observed. Locally, install resolves cleanly, the native addon loads and exposes the unchanged API surface, and typecheck passes. **AI Assistance Disclosure** Dependency bump and validation with Claude Code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Graceful shutdown hangs and the process has to be force-killed: ``` Aug-07 20:13:03.049 [] info: Stopping gracefully Aug-07 20:13:05.065 [network] debug: terminating network worker <- last shutdown progress ... chain keeps ticking slots for another 54s dockerd: "Container failed to exit within 1m0s of signal 15 - using the force" ``` `terminateWorkerThread` awaits `Thread.terminate()` outside the timeout race, so the `retryCount * retryMs` budget is unreachable. The budget is 3s, the hang was 56s, and there is no `Worker thread failed to terminate, retrying...` in the logs, i.e. it never returned from the first call. **Why terminate never resolves.** gdb stacks captured from a live wedged process show the worker is not blocked, it is spinning: ``` Thread 73 (LWP 1524870 "WorkerThread"): <- state R, on CPU #2 uv_run (loop=0x7fd473dc6938, mode=UV_RUN_ONCE) deps/uv/src/unix/core.c:434 #3 node::Environment::CleanupHandles() #4 node::Environment::RunCleanup() #5 node::FreeEnvironment(node::Environment*) #6 node::worker::Worker::Run() ``` `CleanupHandles()` ends in `while (handle_cleanup_waiting_ != 0 || request_waiting_ != 0 || !handle_wrap_queue_.IsEmpty()) uv_run(event_loop(), UV_RUN_ONCE);`. A libuv handle on the worker's loop never closes, so the loop never exits and the thread never dies. Which handle is still open is not identified. **What it costs.** `BeaconNode.close()` closes the network before `chain.persistToDisk()`, so the hang means the finalized state is never archived and the db is never closed cleanly. On the affected node `checkpoint_states/` was empty for 5 days and a restart fell back to a db state 319 slots behind the head it had at shutdown. - race `Thread.terminate()` against the timeout so the `retryCount * retryMs` budget is enforced - return a boolean instead of throwing, so a failed termination does not abort the rest of `BeaconNode.close()` - bound `getApi().close()`, an unbounded RPC into the same worker that runs before the archive - log `getActiveResourcesInfo()` when the network core closes, so the next stuck shutdown can be diagnosed from a log line rather than gdb **Scope.** This keeps a stuck worker from costing us the state archive. It does not stop the worker getting stuck, and it does not make the process exit promptly: `process.exit()` joins every worker via `stop_sub_worker_contexts()`, confirmed in the same capture, so a stuck shutdown still runs to the process manager's stop timeout. ``` Thread 1 (LWP 1524136 "MainThread"): #2 uv_thread_join deps/uv/src/unix/thread.c:295 #3 node::worker::Worker::JoinThread() #4 node::Environment::stop_sub_worker_contexts() #5 node::DefaultProcessExitHandlerInternal(...) ``` I tried to fix that here too, by surfacing the failed termination and hard exiting from the CLI. It did not work - on both wedges that occurred during validation the flag read false at the CLI even though the worker had set it, and the process still waited for the docker timeout. That is dropped from this PR rather than shipped unproven, and `unref()` went with it since `process.exit()` joins regardless of refcounting. **Testing.** 101 mainnet shutdowns on this branch, each after soaking at ~200 peers with 90-160 live inbound QUIC connections for at least 5 minutes: - **99/101 archived the finalized state and logged `Beacon node closed`**, including all 7 where the worker failed to terminate - clean shutdowns complete in 5.9-9.1s - the 7 stuck ones still archived and closed internally in ~9s before being force-killed on the docker timeout The build without this change hung at `terminating network worker` and lost the archive on all 4 shutdowns observed. `NETWORK_CORE_CLOSE_TIMEOUT_MS` is 5s. At the original 3s it tripped on 35 of 101 shutdowns, so the measurement was censored. Raising it to 10s temporarily made it uncensored: 21 shutdowns closed the core in 2.0-4.1s and none hit the bound, so 5s clears the observed max with margin while still bounding an `await` that sits in front of the archive. Note the 35 censored runs mean it is not established that this RPC always resolves, which is the argument for bounding it at all. Root cause notes, gdb captures and the handle-walk tooling: https://gist.github.com/nflaig/b266d89c03cdd2c76338823afed5b2c0 **AI Assistance Disclosure** Investigation and patch developed with Claude Code. Cause traced from debug logs and gdb captures of a live wedged process, validated on a mainnet node as above. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: lodekeeper <lodekeeper@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Picks up ChainSafe/js-libp2p-quic#66, released as 2.1.3. `Connection::closed()` only resolves once quinn reports the connection closed, so a stalled connection driver leaves it pending forever, and with it the napi deferred and threadsafe function backing that promise. libp2p will eventually drop such a peer on its own and call `abort()`, but before 2.1.3 the promise stayed pending even then, so each affected connection leaked those resources for the lifetime of the process. 2.1.3 settles it on `abort()` as well, which turns a permanent leak into none. To be clear about what it does not do: it does not make libp2p notice a dead peer any sooner. By the time `abort()` is called libp2p has already concluded the connection is gone and called `onTransportClosed()` itself. This is resource hygiene, not a peer state fix. **This is not a fix for the shutdown hang.** I originally wrote that patch believing the pending promises were what kept the network worker from terminating. They are not - gdb stacks from a live wedged worker show it spinning in Node's `Environment::CleanupHandles()` on a libuv handle that never closes, and the wedge rate was identical with and without the patch (4 in 20 shutdowns vs 1 in 5 on 2.1.2). #9790 is the mitigation for that, and the underlying handle is still unidentified. On the timing of `onTransportClosed()`: `abort()` runs on every locally initiated close, not just at shutdown, so it is worth being precise about what changes. libp2p already calls `onTransportClosed()` itself immediately after `sendClose()` (`abstract-multiaddr-connection.js`), and the method guards every state transition, so it is idempotent. The settled promise therefore produces a redundant call at a moment the framework was transitioning anyway, rather than an genuinely earlier notification. A remote initiated close does not call `abort()` at all and is unaffected. The change only has an observable effect in the stalled driver case it was written for. **Testing** 2.1.3 is byte-identical in behaviour to the build I ran on a mainnet node for 20 shutdowns at ~200 peers with 90-160 live inbound QUIC connections, no regression observed. Locally, install resolves cleanly, the native addon loads and exposes the unchanged API surface, and typecheck passes. **AI Assistance Disclosure** Dependency bump and validation with Claude Code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

nflaig · 2026-08-09T20:09:16Z

For anyone picking this up later, the investigation behind this change is written up here:

Close-out, validation results and how to resume: https://gist.github.com/nflaig/5f41cfc50f38baf5046a034162943dc3
Analysis and background: https://gist.github.com/nflaig/b266d89c03cdd2c76338823afed5b2c0
Raw logs and native stacks from a wedged worker: https://gist.github.com/nflaig/639b19c041dfa656cee5ef337afbacc5

Short version of what this does and does not fix. Validated over 20 mainnet shutdowns on v1.46.0-rc.1: state was archived 20/20, against 4/4 losses on a build without it. 18/20 completed cleanly in 6.1-9.1s. The remaining 2 hit the known gap, the network worker fails to terminate and the process runs to the stop timeout, but those still archived state, which was the point.

The underlying cause is unfixed. The worker spins in Environment::CleanupHandles() because one of handle_cleanup_waiting_, request_waiting_ or !handle_wrap_queue_.IsEmpty() never clears, so Worker.terminate() never resolves and the main thread blocks in uv_thread_join. Which of the three is unknown. Leading theory is an in-flight outbound TCP connect (ConnectWrap), and the cheapest next test is nsenter -t <pid> -n ss -tan state syn-sent on a live wedge, where the healthy baseline is 0.

Eight refuted hypotheses are listed in the close-out so nobody repeats them.

wemeetagain · 2026-08-12T08:33:25Z

🎉 This PR is included in v1.46.0 🎉

Applies libp2p/js-libp2p#3597 as a local `pnpm patch` until it is released. This fixes the network worker shutdown hang, the underlying handle that #9790 mitigated but did not identify. ## Root cause `TCPSocketMultiaddrConnection.sendReset()` calls `socket.resetAndDestroy()` unconditionally. When the writable side has already ended that does not tear the handle down, so the socket stays alive as an active `TCPSocketWrap` while libp2p considers it closed. Nothing else holds a reference, so nothing ever closes it. Node's worker teardown then spins forever, because `Environment::CleanupHandles()` runs until every handle is closed: ```cpp while (handle_cleanup_waiting_ != 0 || request_waiting_ != 0 || !handle_wrap_queue_.IsEmpty()) { uv_run(event_loop(), UV_RUN_ONCE); } ``` The thread never exits, so `Worker.terminate()` never resolves, so the main thread blocks in `uv_thread_join` from `process.exit()` until the process manager kills it. The patch guards the case the reset cannot handle: ```js sendReset (): void { if (this.socket.writableEnded) { this.socket.destroy() return } this.socket.resetAndDestroy() } ``` ## Why the exact-version pin `@libp2p/tcp` is declared as `^11.0.13` and `patchedDependencies` keys are version exact, so an 11.0.14 release would resolve past the patch and silently drop the fix. The `overrides` pin prevents that, same as the existing `sigstore` patch. It does mean no `@libp2p/tcp` bump until the patch is dropped. ## Testing Validated on a mainnet node against a baseline hang rate of 4 in 36 shutdowns (11.1%). 70 consecutive shutdowns, synced with >=200 peers and a 5 minute soak each: | criterion | result | | --- | --- | | hangs | 0 / 70 | | `TCPSocketWrap` present after `libp2p.stop()` | 0 / 70 | | worker terminate | 0.027-0.072s, hang signature is 3.000s | | exit code | 0 on all 70, never SIGKILLed | | shutdown duration | mean 7.20s, max 9.8s | | state archived | 70 / 70 | Plus one shutdown at 9.07h uptime, since orphaned socket counts grew with uptime: terminate 0.180s, no `TCPSocketWrap`, exit 0. Counted independently from the raw node logs as well as from the test harness. The last unpatched shutdown on the same machine, three minutes before deploying the patched build, hung at 3.001s with `activeResources=MessagePort=1,TCPSocketWrap=6,Timeout=1`. Observing 70 consecutive clean shutdowns if the bug were still present has probability 0.00026. Fisher exact against the measured baseline gives p = 0.012. The `activeResources` logging added by #9790 is what made this diagnosable, the presence of `TCPSocketWrap` separated 5 hangs from 32 clean shutdowns perfectly (Fisher p = 0.0000023). Longest uptime tested was 9.07h. The worst case observed, 28 orphaned sockets, took ~38h to accumulate, so this validates the mechanism rather than proving an upper bound. Full write-up: https://gist.github.com/nflaig/5f41cfc50f38baf5046a034162943dc3 ## When to remove Drop the patch and the `overrides` pin together once js-libp2p#3597 is released and `@libp2p/tcp` is bumped to a version containing it. ## AI Assistance Disclosure Investigation, patch and validation with Claude Code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

lodekeeper and others added 4 commits August 7, 2026 21:45

nflaig requested a review from a team as a code owner August 7, 2026 20:55

github-project-automation Bot added this to Lodestar Team Coordination Aug 7, 2026

github-project-automation Bot moved this to In Progress in Lodestar Team Coordination Aug 7, 2026

nflaig marked this pull request as draft August 7, 2026 21:20

nflaig and others added 5 commits August 8, 2026 08:27

chore: raise network core close timeout to 5s and trim comments

8417a2c

2.3s is the normal close time, 3s was cutting it too fine and tripped the timeout on most restarts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

chore: drop comments duplicating the log line below them

5cbeffb

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

chore: say blocked in a native call instead of wedged

fc0f0da

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

chore: drop vague anyway from network core close error log

47adfb1

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

chore: correct network core close timeout rationale

4f3f1c2

Measured 1.85-4.03s across 17 mainnet shutdowns, including on a build where the worker later wedged, so the call is not the hang risk the comment claimed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

nflaig marked this pull request as ready for review August 8, 2026 10:16

chore: name the retry budget explicitly in terminate comment

fbde19e

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

nflaig mentioned this pull request Aug 8, 2026

fix: settle connection.closed() when the connection is aborted ChainSafe/js-libp2p-quic#66

Merged

nflaig and others added 3 commits August 8, 2026 17:50

nflaig force-pushed the nflaig/bound-network-worker-shutdown branch from e6658a8 to 2438f1e Compare August 9, 2026 08:50

nflaig marked this pull request as draft August 9, 2026 08:51

chore: set network core close timeout to 5s

a30c154

The 10s bound made the measurement uncensored: 20 mainnet shutdowns closed the core in 2.0-4.1s and none hit the bound. 5s clears the observed max with margin. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

nflaig marked this pull request as ready for review August 9, 2026 08:53

nflaig and others added 4 commits August 9, 2026 10:04

chore: correct close timeout comment

289c102

21 measurements not 20, and the leaked handle was never attributed to libp2p, only that an unclosed libuv handle is what keeps the worker spinning. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

chore: drop false precision from close timeout comment

2976ace

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

chore: make the close timeout comment readable

10d9049

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

nflaig commented Aug 9, 2026

View reviewed changes

nflaig changed the title ~~fix: bound network worker termination to prevent shutdown hang~~ fix: bound network worker termination to protect state archival Aug 9, 2026

nflaig changed the title ~~fix: bound network worker termination to protect state archival~~ fix: continue shutdown when the network worker can not be terminated Aug 9, 2026

nflaig changed the title ~~fix: continue shutdown when the network worker can not be terminated~~ fix: bound network worker termination so shutdown can complete Aug 9, 2026

nflaig changed the title ~~fix: bound network worker termination so shutdown can complete~~ fix: continue shutdown when the network worker can not be terminated Aug 9, 2026

nflaig changed the title ~~fix: continue shutdown when the network worker can not be terminated~~ fix: bound network worker termination so shutdown can complete Aug 9, 2026

spiral-ladder approved these changes Aug 9, 2026

View reviewed changes

nflaig merged commit eabf12a into unstable Aug 9, 2026
25 checks passed

nflaig deleted the nflaig/bound-network-worker-shutdown branch August 9, 2026 10:01

github-project-automation Bot moved this from In Progress to Done in Lodestar Team Coordination Aug 9, 2026

nflaig mentioned this pull request Aug 9, 2026

deps: bump @chainsafe/libp2p-quic to 2.1.3 #9792

Merged

nflaig mentioned this pull request Aug 9, 2026

fix: force process exit when a worker thread can not be terminated #9793

Closed

This was referenced Aug 9, 2026

[BASELINE-01] Pin unstable and run the baseline capability audit krisoshea-eth/lodestar#6

Open

[MET-01] Add metrics server and introduce metrics for [CLI-01, SIGN-01, API-01] krisoshea-eth/lodestar#46

Closed

nflaig mentioned this pull request Aug 14, 2026

deps: patch @libp2p/tcp to fix network worker shutdown hang #9825

Merged

Navigation Menu

Uh oh!

Uh oh!

fix: bound network worker termination so shutdown can complete - #9790

fix: bound network worker termination so shutdown can complete#9790
nflaig merged 20 commits into
unstablefrom
nflaig/bound-network-worker-shutdown

nflaig commented Aug 7, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Aug 7, 2026

Uh oh!

github-actions Bot commented Aug 7, 2026 •

edited

Loading

Uh oh!

nflaig left a comment

Uh oh!

spiral-ladder left a comment

Uh oh!

Uh oh!

nflaig commented Aug 9, 2026

Uh oh!

wemeetagain commented Aug 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Uh oh!

Conversation

nflaig commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Aug 7, 2026

Uh oh!

github-actions Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Performance Report

Uh oh!

nflaig left a comment

Choose a reason for hiding this comment

Uh oh!

spiral-ladder left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nflaig commented Aug 9, 2026

Uh oh!

wemeetagain commented Aug 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nflaig commented Aug 7, 2026 •

edited

Loading

github-actions Bot commented Aug 7, 2026 •

edited

Loading