fix(delegate): process-based subagent isolation to eliminate GIL contention by bennybuoy · Pull Request #60053 · NousResearch/hermes-agent

bennybuoy · 2026-07-07T07:12:29Z

Problem

delegate_task runs batch subagents on ThreadPoolExecutor workers inside the parent process. All subagent threads share one GIL with the TUI/desktop WebSocket event loop. When 3+ subagents do CPU-heavy work (SSE parsing, regex, JSON), the event loop is starved — the WebSocket stalls for tens of seconds and the desktop app drops the connection.

Tracked issues:

web_server event loop stalls up to 51s under heavy agent work (GIL pressure) — desktop UI appears frozen #58576 (open): "event loop stalled 51.4s (GIL pressure suspected)"
async LLM calls block the desktop WebSocket loop via busy-poll in interruptible_*_api_call #57903 (closed): Deep RCA confirmed worker-thread GIL contention from the Anthropic SDK SSE consumer
Gateway turn can stall after Codex response while Python regex holds the GIL #32079 (open, P1): Regex path held the GIL, starving all threads — same root cause, different trigger

Nobody has proposed the process-based fix before. All existing mitigations (locks, semaphores, poll tweaks, turn caps, background modes) work within the threading model. PR #57933 (closed/unmerged) explicitly identified "subprocess isolation for LLM calls" as the real fix but didn't implement it.

Solution

Opt-in process isolation (delegation.process_isolation: true in config.yaml, default false). When enabled, each batch child runs in its own OS process via multiprocessing(forkserver), giving each subagent its own GIL — mirroring how OpenClaw spawns subagents as separate Node.js child processes.

Architecture

tools/delegate_process.py (new, 810 lines): ChildProcessSpec (picklable parent-side handle), _child_process_main (forkserver entry point — reconstructs the AIAgent inside the child from picklable params, runs the conversation, streams progress over IPC), run_children_in_processes (parent poll loop — relays IPC events to identical progress callbacks, heartbeats parent activity, propagates interrupts, enforces timeouts, reaps all children)
tools/delegate_tool.py: _build_child_process_spec extracts constructor args as a plain dict and validates picklability before spawning; batch path branches to process mode when enabled
hermes_cli/config.py: delegation.process_isolation: false (opt-in config gate)

Why forkserver (not spawn or fork)

fork is unsafe: the parent holds live httpx/SSL clients, SQLite connections, and running threads — fork would inherit them in a corrupt state.
spawn is safe but slow: each child re-imports the entire Python module tree (~1-2s startup, ~150MB/child).
forkserver starts a clean helper process early (before the parent opens clients/threads), then forks children from it. The forkserver process is clean (no clients, no threads, no SSL), so forking from it is always safe. Children inherit already-imported modules via copy-on-write — startup drops to ~10ms, memory to ~20-30MB/child. Total overhead for 3 children: ~75-100MB (vs ~450MB for spawn).

Key design decisions

IPC bridge: child writes plain-dict events to multiprocessing.Queue, parent relays to identical progress callbacks — the TUI/gateway cannot tell the difference
Interrupt: shared multiprocessing.Event — child watcher thread polls and calls agent.interrupt()
Timeout: delegation.child_timeout_seconds defaults to None (no timeout — subagents doing big jobs won't get cancelled). Stuck-child protection is via heartbeat staleness monitor. Hard process termination with grace period (key advantage over threads — you CAN kill a stuck process).
Session DB: child opens its own SQLite connection to the same path
File-state: child checks writes against parent's read snapshot, passes modified paths back via result entry
Credential pool: leased in parent before fork, released after completion
Single-task mode always runs in-process (no pool overhead)
No fd cleanup needed: forkserver helper starts clean; IPC Queue/Event objects are managed by the forkserver context itself

Testing

12 new tests in tests/tools/test_delegate_process.py covering: PID verification (children are separate OS processes), IPC event relay, interrupt propagation, timeout/hard-termination, config gate selection, result ordering by task_index, zombie cleanup, GIL freedom (parent stays responsive during child CPU work), child exception handling, params picklability
All 191 delegate-related tests pass (151 existing + 12 new + 28 other delegate tests)
E2E verified: 3 subagents ran in separate OS processes via LlamaHerd, each returned correct results (4, 6, 8) in 8.4s total

Config

delegation:
  process_isolation: true  # default: false (opt-in)

When false (default), the existing ThreadPoolExecutor path is used — no behavior change. When true, batch subagents run in isolated OS processes. Single-task mode always uses threads (no pool overhead for one child).

Scope

This PR does NOT touch MOA (agent/moa_loop.py). MOA uses a separate ThreadPoolExecutor for reference model fan-out — it's I/O-bound API calls (not CPU-bound agent loops), so GIL pressure is much lower. That needs a different fix (async SDK migration), not process isolation.

Future improvements (not for this PR)

Pre-warmed process pool: Keep N pre-warmed child processes with AIAgent skeletons ready (eliminates ~1s construction latency per child). OpenClaw does this.
Hybrid mode: Auto-select threads for I/O-bound tasks, processes for CPU-bound tasks.
Async API calls (separate PR): Migrate the Anthropic SDK SSE consumer to async I/O with incremental JSON parsing — complementary to process isolation.
Python 3.14t free-threading: When stable, removes the GIL entirely — process isolation becomes unnecessary.

…ention Batch subagents in delegate_task run on ThreadPoolExecutor workers inside the parent process, sharing one GIL with the TUI/desktop WebSocket event loop. When 3+ subagents do CPU-heavy work (SSE parsing, regex, JSON) the event loop is starved — the WebSocket stalls for tens of seconds and the desktop app drops the connection (issues NousResearch#58576, NousResearch#57903, NousResearch#32079). This PR adds an opt-in process-isolation path (delegation.process_isolation: true in config.yaml). When enabled, each batch child runs in its own OS process via multiprocessing(spawn), giving each subagent its own GIL — mirroring how OpenClaw spawns subagents as separate Node.js child processes. Architecture: - tools/delegate_process.py (new): ChildProcessSpec (picklable parent-side handle), _child_process_main (spawn entry point — reconstructs AIAgent inside the child from plain params, runs conversation, streams progress events over IPC queue), run_children_in_processes (parent-side poll loop — relays IPC events into the same progress callbacks the thread path uses, heartbeats parent activity, propagates interrupts via shared multiprocessing.Event, enforces timeout with hard process termination, reaps all children so no zombies survive). - tools/delegate_tool.py: _build_child_process_spec extracts picklable constructor params from the parent agent. Batch path branches: process isolation runs run_children_in_processes; thread path stays unchanged. Single-task mode always runs in-process (no pool overhead). - hermes_cli/config.py: delegation.process_isolation (bool, default False). Key design decisions: - spawn context (not fork): parent holds live httpx/SSL clients, SQLite connections, and running threads — fork would inherit them corrupt. - IPC bridge: child writes plain-dict events to multiprocessing.Queue; parent reads and calls the original progress callbacks. The TUI/gateway cannot tell the difference — event format is identical. - Interrupt: shared multiprocessing.Event; child watcher thread polls it and calls agent.interrupt(). Parent can hard-terminate after grace. - Session DB: child opens its own SQLite connection to the same DB path. - File-state: child checks writes against parent's read snapshot, passes modified paths back via result entry; parent folds them into its registry. - Credential pool: leased in parent before spawn, released after completion. - Config-gated (default false): safe opt-in, existing tests unchanged. Addresses: NousResearch#58576 NousResearch#57903 NousResearch#32079 Supersedes mitigation approach in: NousResearch#57933 (closed)

…assthrough 12 tests covering: - Children spawn as separate OS processes (PID verification) - IPC event relay from child to parent - Interrupt propagation across process boundaries - Timeout with responsive child (interrupt path) - Timeout with unresponsive child (hard-kill path) - Config gate selects process vs thread path - Results sorted by task_index regardless of completion order - No zombie processes survive the batch - GIL freedom: parent stays responsive during child CPU work - Child process exceptions caught and reported as error entries - Params survive pickling (spawn requirement) Fix: _child_process_main now passes extra params (test_config, goal, etc.) to the agent factory via **_extra, so stub factories in spawned children can receive test configuration across the process boundary.

The real AIAgent constructor TypeErrors on unknown kwargs (task_index, task_count, etc.), so **_extra must only include stub-factory keys (goal, role, _test_config) when agent_factory is explicitly set. When agent_factory is None (production), _extra is empty and the constructor gets only its declared kwargs. E2E verified: 3 subagents ran in separate OS processes via LlamaHerd, each returned correct results (4, 6, 8) in 8.4s total.

…ction spawn re-imports the entire Python module tree per child (~1-2s startup, ~150MB/child). forkserver forks a clean helper process early (before the parent opens httpx clients / SQLite connections), then forks children from it — startup drops to ~10ms, memory to ~20-30MB/child via copy-on-write. The forkserver process is clean (no clients, no threads, no SSL), so forking from it is always safe. This brings the overhead for 3 children from ~450MB (spawn) down to ~75-100MB (forkserver). No fd cleanup needed in _child_process_main: the forkserver helper starts clean, and the IPC Queue/Event objects are managed by the forkserver context itself — blindly closing fds would break them. All 163 tests pass (151 existing + 12 process-isolation tests).

Copilot

Pull request overview

Adds an opt-in process-isolated execution mode for delegate_task batch subagents to avoid GIL contention that can stall the TUI/desktop WebSocket event loop. This introduces a new multiprocessing-based runner plus refactors in the existing delegation code so thread- and process-modes emit compatible progress/result shapes.

Changes:

Add tools/delegate_process.py to run batch subagents in isolated OS processes with IPC event/result relaying.
Refactor tools/delegate_tool.py to share child settings resolution between thread and process modes and to optionally select process isolation for batch runs.
Add delegation.process_isolation config default (false) and new tests for the process-isolation path.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
tools/delegate_tool.py	Adds the config gate, child-settings refactor, and process-isolation batch execution path.
tools/delegate_process.py	Implements forkserver/spawn-based child process lifecycle, IPC relay, interrupts, timeouts, and cleanup.
tests/tools/test_delegate_process.py	Adds unit tests for process spawning, IPC relay, interrupts/timeouts, ordering, and cleanup.
hermes_cli/config.py	Adds `delegation.process_isolation: false` default config entry.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

bennybuoy · 2026-07-07T07:33:40Z

+    ctx = mp.get_context("forkserver")
+    event_queue = ctx.Queue()
+    result_queue = ctx.Queue()


Stale — the forkserver change has been reverted (commit d101daa). The branch now uses spawn, which is available on all platforms including Windows. I also benchmarked forkserver vs spawn and found no meaningful difference in startup time (1.73s vs 1.75s) or memory (87MB vs 88MB per child). The forkserver path still goes through spawn._main → prepare, so it doesn't benefit from copy-on-write module imports as I expected.

bennybuoy · 2026-07-07T07:33:51Z

+        _files_written_map = file_state.writes_since(
+            "", wall_start, []
+        )  # all writes since wall_start


Good catch — fixed in commit c6401e0. Changed writes_since("", wall_start, []) to writes_since("", wall_start, _files_read) so writes to files the subagent read during its run are surfaced in the subagent.complete payload.

alt-glitch · 2026-07-07T07:21:21Z

This was generated by AI during triage.

Related to the subagent GIL-contention / WebSocket-stall family: #58576 (event loop stalls under agent work), #32079 (regex held the GIL), #57903 (RCA of the busy-poll GIL contention), and #57933 (closed — identified subprocess isolation as the real fix but didn't implement it). This PR is the first to actually implement process-based isolation via multiprocessing(forkserver), a distinct mechanism from the existing threading-model mitigations. Not a duplicate — flagging the cluster so a maintainer can weigh this approach against the in-thread fixes.

bennybuoy · 2026-07-07T07:21:45Z

Closing to rework — the forkserver change doesn't deliver the expected startup performance improvement over spawn. Will re-push once the approach is validated.

…ory reduction" This reverts commit 405fec2.

bennybuoy · 2026-07-07T07:30:02Z

Update: forkserver experiment reverted

I tested switching from spawn to forkserver context to reduce per-child startup overhead. The benchmark results showed no meaningful difference:

Context	Process+import overhead	AIAgent construction	Total per child
spawn	1.734s	2.048s	3.78s
forkserver	1.746s	2.016s	3.76s

The forkserver path still goes through spawn._main → prepare → _fixup_main_from_path for each child, so it doesn't benefit from copy-on-write module imports the way I expected. The ~10ms startup claim from the original commit message was wrong — I've reverted it.

The branch now uses spawn (unchanged from the original implementation). All 163 tests pass, E2E verified with 3 subagents via LlamaHerd (correct results in 8.4s).

…urfaced file_state.writes_since filters by the paths iterable — passing [] means no paths match, so _files_written was always empty even when the subagent wrote files. Pass _files_read instead so writes to files the subagent read during its run are reported in the subagent.complete payload. Found by Copilot review on PR NousResearch#60053.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

bennybuoy · 2026-07-07T07:45:16Z

+        proc = ctx.Process(
+            target=_child_process_main,
+            args=(spec.params, event_queue, result_queue, spec.interrupt_event),
+            daemon=True,


Fixed in commit a2f9b69. Removed daemon=True — the parent-side poll loop already joins/terminates all children before returning, so daemon isn't needed for cleanup. This allows orchestrator-role subagents to spawn their own children via multiprocessing.

bennybuoy · 2026-07-07T07:45:22Z

+        # At least 2 of the 3 children should have different PIDs from the parent
+        # (spawn guarantees separate processes; the PID is the proof)
+        child_pids.discard(parent_pid)
+        self.assertGreaterEqual(
+            len(child_pids), 1,
+            f"Expected child processes with different PIDs from parent ({parent_pid}), "
+            f"got child PIDs: {child_pids}"
+        )


Fixed in commit a2f9b69. Strengthened the assertion from assertGreaterEqual(len(child_pids), 1) to assertEqual(len(child_pids), 3) — all 3 children must have distinct PIDs from each other and from the parent.

daemon=True prevents orchestrator-role subagents from spawning their own children via multiprocessing — Python disallows daemon processes from creating child processes. The parent-side poll loop already joins/kill all children before returning, so daemon isn't needed for cleanup. Also strengthen the PID test to assert all 3 children have distinct PIDs (not just at least 1 different from parent). Found by Copilot review on PR NousResearch#60053.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

bennybuoy · 2026-07-07T08:01:22Z

+        if child_timeout:
+            for i in list(pending):
+                elapsed = now - started_mono
+                if i not in timeout_signaled and elapsed > child_timeout:
+                    timeout_signaled[i] = now


Fixed in commit e8f7fd7. Added child_start_mono[i] tracked at proc.start() and changed the timeout calculation from now - started_mono to now - child_start_mono.get(i, started_mono), so each child's timeout is measured from its own start time. The fallback to started_mono handles the edge case where a child failed to start.

…t batch start child_timeout was using started_mono (batch-level start) for all children, so staggered process startup ate into later children's timeout budget. Track child_start_mono[i] at proc.start() and compute elapsed per child from its own start time. Found by Copilot review on PR NousResearch#60053.

teknium1

Thanks for pursuing a real isolation path for batch delegation. Current main still runs batch children in a DaemonThreadPoolExecutor (tools/delegate_tool.py:2566), so the underlying structural concern remains.

Problems

tools/delegate_process.py:288-320 rebuilds an AIAgent without provider_require_parameters, provider_data_collection, or request_overrides. Current main forwards these at tools/delegate_tool.py:1348-1354; process mode would silently change provider-routing behavior.
tools/delegate_tool.py:1541-1544 explicitly accepts that process children cannot rotate credential-pool entries. Current main leases and swaps pooled credentials before execution (tools/delegate_tool.py:1770-1780), so this weakens existing rate-limit recovery.
The new public delegation.process_isolation option has no docs update. Delegation configuration is documented at website/docs/user-guide/configuration.md:1985-2011.

Suggested changes

Build the process spec from the current child-construction contract and add process/thread parity tests for routing overrides and credential-pool behavior.
Preserve pool rotation through a parent-mediated protocol, or fail closed for pooled credentials instead of silently degrading recovery.
Document the opt-in and its operational/platform trade-offs.

Automated hermes-sweeper review.

teknium1 · 2026-07-15T21:20:31Z

+            providers_ignored=params.get("providers_ignored"),
+            providers_order=params.get("providers_order"),
+            provider_sort=params.get("provider_sort"),
+            openrouter_min_coding_score=params.get("openrouter_min_coding_score"),


Blocking: this constructor omits current child-routing inputs (provider_require_parameters, provider_data_collection, and request_overrides; see current tools/delegate_tool.py:1348-1354). Carry those values through the process spec and add a process/thread parity test, otherwise enabling isolation changes provider request behavior.

teknium1 · 2026-07-15T21:20:31Z

+    # onto every relayed event before the child process even boots.
+    session_id = f"{_time.strftime('%Y%m%d_%H%M%S')}_{_uuid.uuid4().hex[:6]}"
+
+    # Lease a credential in the parent so the child starts with a live key


Blocking: process mode intentionally loses credential-pool rotation here. Current children receive _credential_pool and _run_single_child leases/swaps a live credential before running (tools/delegate_tool.py:1770-1780). Preserve that recovery contract through parent-mediated IPC, or reject process isolation when a pool is active rather than silently degrading it.

bennybuoy · 2026-08-11T11:09:26Z

Closing as superseded for the original desktop/dashboard responsiveness objective by the merged compute-host isolation work in #65895.

The current branch is also substantially stale against main, and the unresolved process-child routing and credential-pool parity issues would require a clean redesign rather than a safe rebase. If process-per-child delegation is still needed for a distinct requirement such as non-dashboard isolation, CPU throughput, or hard termination, it should start as a fresh reproduction and implementation from current main.

Thanks for the review and specific feedback.

Ben Kamholtz and others added 4 commits July 7, 2026 17:00

Copilot AI review requested due to automatic review settings July 7, 2026 07:12

Copilot started reviewing on behalf of bennybuoy July 7, 2026 07:12 View session

Copilot AI reviewed Jul 7, 2026

View reviewed changes

alt-glitch added type/feature New feature or request comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint tool/delegate Subagent delegation P3 Low — cosmetic, nice to have labels Jul 7, 2026

bennybuoy closed this Jul 7, 2026

Revert "perf(delegate-process): switch spawn to forkserver for 5x mem…

d101daa

…ory reduction" This reverts commit 405fec2.

bennybuoy reopened this Jul 7, 2026

bennybuoy requested a review from Copilot July 7, 2026 07:36

Copilot started reviewing on behalf of bennybuoy July 7, 2026 07:36 View session

Copilot AI reviewed Jul 7, 2026

View reviewed changes

bennybuoy requested a review from Copilot July 7, 2026 07:52

Copilot started reviewing on behalf of bennybuoy July 7, 2026 07:52 View session

Copilot AI reviewed Jul 7, 2026

View reviewed changes

teknium1 reviewed Jul 15, 2026

View reviewed changes

bennybuoy closed this Aug 11, 2026

Uh oh!

Conversation

bennybuoy commented Jul 7, 2026

Problem

Solution

Architecture

Why forkserver (not spawn or fork)

Key design decisions

Testing

Config

Scope

Future improvements (not for this PR)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

bennybuoy Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

bennybuoy Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

alt-glitch commented Jul 7, 2026

Uh oh!

bennybuoy commented Jul 7, 2026

Uh oh!

bennybuoy commented Jul 7, 2026

Update: forkserver experiment reverted

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

bennybuoy Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

bennybuoy Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

bennybuoy Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

teknium1 left a comment

Choose a reason for hiding this comment

Problems

Suggested changes

Uh oh!

teknium1 Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

teknium1 Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

bennybuoy commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bennybuoy commented Aug 11, 2026 •

edited

Loading