Milagro crypto bls - #4
Merged
Merged
Conversation
…nd added shardAndCommittee.js that was added in the current spec
…stallizedState.js and shardAndCommittee.js
wemeetagain
pushed a commit
that referenced
this pull request
Aug 2, 2019
Serialize implementation and tests
1 task
nflaig
added a commit
that referenced
this pull request
Aug 9, 2026
Graceful shutdown hangs and the process has to be force-killed: ``` Aug-07 20:13:03.049 [] info: Stopping gracefully Aug-07 20:13:05.065 [network] debug: terminating network worker <- last shutdown progress ... chain keeps ticking slots for another 54s dockerd: "Container failed to exit within 1m0s of signal 15 - using the force" ``` `terminateWorkerThread` awaits `Thread.terminate()` outside the timeout race, so the `retryCount * retryMs` budget is unreachable. The budget is 3s, the hang was 56s, and there is no `Worker thread failed to terminate, retrying...` in the logs, i.e. it never returned from the first call. **Why terminate never resolves.** gdb stacks captured from a live wedged process show the worker is not blocked, it is spinning: ``` Thread 73 (LWP 1524870 "WorkerThread"): <- state R, on CPU #2 uv_run (loop=0x7fd473dc6938, mode=UV_RUN_ONCE) deps/uv/src/unix/core.c:434 #3 node::Environment::CleanupHandles() #4 node::Environment::RunCleanup() #5 node::FreeEnvironment(node::Environment*) #6 node::worker::Worker::Run() ``` `CleanupHandles()` ends in `while (handle_cleanup_waiting_ != 0 || request_waiting_ != 0 || !handle_wrap_queue_.IsEmpty()) uv_run(event_loop(), UV_RUN_ONCE);`. A libuv handle on the worker's loop never closes, so the loop never exits and the thread never dies. Which handle is still open is not identified. **What it costs.** `BeaconNode.close()` closes the network before `chain.persistToDisk()`, so the hang means the finalized state is never archived and the db is never closed cleanly. On the affected node `checkpoint_states/` was empty for 5 days and a restart fell back to a db state 319 slots behind the head it had at shutdown. - race `Thread.terminate()` against the timeout so the `retryCount * retryMs` budget is enforced - return a boolean instead of throwing, so a failed termination does not abort the rest of `BeaconNode.close()` - bound `getApi().close()`, an unbounded RPC into the same worker that runs before the archive - log `getActiveResourcesInfo()` when the network core closes, so the next stuck shutdown can be diagnosed from a log line rather than gdb **Scope.** This keeps a stuck worker from costing us the state archive. It does not stop the worker getting stuck, and it does not make the process exit promptly: `process.exit()` joins every worker via `stop_sub_worker_contexts()`, confirmed in the same capture, so a stuck shutdown still runs to the process manager's stop timeout. ``` Thread 1 (LWP 1524136 "MainThread"): #2 uv_thread_join deps/uv/src/unix/thread.c:295 #3 node::worker::Worker::JoinThread() #4 node::Environment::stop_sub_worker_contexts() #5 node::DefaultProcessExitHandlerInternal(...) ``` I tried to fix that here too, by surfacing the failed termination and hard exiting from the CLI. It did not work - on both wedges that occurred during validation the flag read false at the CLI even though the worker had set it, and the process still waited for the docker timeout. That is dropped from this PR rather than shipped unproven, and `unref()` went with it since `process.exit()` joins regardless of refcounting. **Testing.** 101 mainnet shutdowns on this branch, each after soaking at ~200 peers with 90-160 live inbound QUIC connections for at least 5 minutes: - **99/101 archived the finalized state and logged `Beacon node closed`**, including all 7 where the worker failed to terminate - clean shutdowns complete in 5.9-9.1s - the 7 stuck ones still archived and closed internally in ~9s before being force-killed on the docker timeout The build without this change hung at `terminating network worker` and lost the archive on all 4 shutdowns observed. `NETWORK_CORE_CLOSE_TIMEOUT_MS` is 5s. At the original 3s it tripped on 35 of 101 shutdowns, so the measurement was censored. Raising it to 10s temporarily made it uncensored: 21 shutdowns closed the core in 2.0-4.1s and none hit the bound, so 5s clears the observed max with margin while still bounding an `await` that sits in front of the archive. Note the 35 censored runs mean it is not established that this RPC always resolves, which is the argument for bounding it at all. Root cause notes, gdb captures and the handle-walk tooling: https://gist.github.com/nflaig/b266d89c03cdd2c76338823afed5b2c0 **AI Assistance Disclosure** Investigation and patch developed with Claude Code. Cause traced from debug logs and gdb captures of a live wedged process, validated on a mainnet node as above. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: lodekeeper <lodekeeper@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
nflaig
added a commit
that referenced
this pull request
Aug 9, 2026
Graceful shutdown hangs and the process has to be force-killed: ``` Aug-07 20:13:03.049 [] info: Stopping gracefully Aug-07 20:13:05.065 [network] debug: terminating network worker <- last shutdown progress ... chain keeps ticking slots for another 54s dockerd: "Container failed to exit within 1m0s of signal 15 - using the force" ``` `terminateWorkerThread` awaits `Thread.terminate()` outside the timeout race, so the `retryCount * retryMs` budget is unreachable. The budget is 3s, the hang was 56s, and there is no `Worker thread failed to terminate, retrying...` in the logs, i.e. it never returned from the first call. **Why terminate never resolves.** gdb stacks captured from a live wedged process show the worker is not blocked, it is spinning: ``` Thread 73 (LWP 1524870 "WorkerThread"): <- state R, on CPU #2 uv_run (loop=0x7fd473dc6938, mode=UV_RUN_ONCE) deps/uv/src/unix/core.c:434 #3 node::Environment::CleanupHandles() #4 node::Environment::RunCleanup() #5 node::FreeEnvironment(node::Environment*) #6 node::worker::Worker::Run() ``` `CleanupHandles()` ends in `while (handle_cleanup_waiting_ != 0 || request_waiting_ != 0 || !handle_wrap_queue_.IsEmpty()) uv_run(event_loop(), UV_RUN_ONCE);`. A libuv handle on the worker's loop never closes, so the loop never exits and the thread never dies. Which handle is still open is not identified. **What it costs.** `BeaconNode.close()` closes the network before `chain.persistToDisk()`, so the hang means the finalized state is never archived and the db is never closed cleanly. On the affected node `checkpoint_states/` was empty for 5 days and a restart fell back to a db state 319 slots behind the head it had at shutdown. - race `Thread.terminate()` against the timeout so the `retryCount * retryMs` budget is enforced - return a boolean instead of throwing, so a failed termination does not abort the rest of `BeaconNode.close()` - bound `getApi().close()`, an unbounded RPC into the same worker that runs before the archive - log `getActiveResourcesInfo()` when the network core closes, so the next stuck shutdown can be diagnosed from a log line rather than gdb **Scope.** This keeps a stuck worker from costing us the state archive. It does not stop the worker getting stuck, and it does not make the process exit promptly: `process.exit()` joins every worker via `stop_sub_worker_contexts()`, confirmed in the same capture, so a stuck shutdown still runs to the process manager's stop timeout. ``` Thread 1 (LWP 1524136 "MainThread"): #2 uv_thread_join deps/uv/src/unix/thread.c:295 #3 node::worker::Worker::JoinThread() #4 node::Environment::stop_sub_worker_contexts() #5 node::DefaultProcessExitHandlerInternal(...) ``` I tried to fix that here too, by surfacing the failed termination and hard exiting from the CLI. It did not work - on both wedges that occurred during validation the flag read false at the CLI even though the worker had set it, and the process still waited for the docker timeout. That is dropped from this PR rather than shipped unproven, and `unref()` went with it since `process.exit()` joins regardless of refcounting. **Testing.** 101 mainnet shutdowns on this branch, each after soaking at ~200 peers with 90-160 live inbound QUIC connections for at least 5 minutes: - **99/101 archived the finalized state and logged `Beacon node closed`**, including all 7 where the worker failed to terminate - clean shutdowns complete in 5.9-9.1s - the 7 stuck ones still archived and closed internally in ~9s before being force-killed on the docker timeout The build without this change hung at `terminating network worker` and lost the archive on all 4 shutdowns observed. `NETWORK_CORE_CLOSE_TIMEOUT_MS` is 5s. At the original 3s it tripped on 35 of 101 shutdowns, so the measurement was censored. Raising it to 10s temporarily made it uncensored: 21 shutdowns closed the core in 2.0-4.1s and none hit the bound, so 5s clears the observed max with margin while still bounding an `await` that sits in front of the archive. Note the 35 censored runs mean it is not established that this RPC always resolves, which is the argument for bounding it at all. Root cause notes, gdb captures and the handle-walk tooling: https://gist.github.com/nflaig/b266d89c03cdd2c76338823afed5b2c0 **AI Assistance Disclosure** Investigation and patch developed with Claude Code. Cause traced from debug logs and gdb captures of a live wedged process, validated on a mainnet node as above. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: lodekeeper <lodekeeper@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Locally, I forgot to switch branches. Instead of cherry-picking the commits and adding those commits to the master branch, I am going to merge this branch with the master.
I will personally make sure this never happens again.