feat(code-execution): remote kernel host — session persistence for docker/ssh/modal (closes #96873) - #96991
Merged
Conversation
…cker/ssh/modal backends (closes #96873)
Contributor
૮ >ﻌ< ა ci reviewran on 89d1dfc — feat(code-execution): remote kernel host — session persisten
|
This was referenced Aug 28, 2026
mrkillbob
pushed a commit
to mrkillbob/hermes-agent
that referenced
this pull request
Aug 28, 2026
…cker/ssh/modal backends (closes NousResearch#96873) (NousResearch#96991)
teknium1
pushed a commit
that referenced
this pull request
Sep 3, 2026
Local session kernels sweep idle-expired entries and enforce a process-wide cap (DEFAULT_MAX_SESSION_KERNELS) on every call (tools/code_kernel.py's _reap_unlocked / _evict_over_cap_unlocked). The new remote kernel host (#96991) never got the same treatment: _REMOTE_KERNELS only shrinks lazily when a specific key is revisited and found dead, so an owner that opens kernels for several distinct (env_type, task_env_id) combinations (or delegated children) and never revisits some of them accumulates host-side bookkeeping entries for the life of the gateway process. Note this is narrower than the local case: the remote runner already self-reaps on its own idle timeout, and SSH/Docker connections are independently bounded by their own transport-level lifecycles (SSH ControlPersist, Docker's session-scoped idle-timeout in terminal_tool.py) — so nothing here leaks a live remote connection. What's missing is purely the host-side dict/cap bookkeeping symmetry with local kernels. Adds _reap_unlocked/_evict_over_cap_unlocked mirroring the local implementation, reusing the same max_session_kernels config as an independent cap on _REMOTE_KERNELS.
melon-xf
added a commit
to melon-xf/hermes-agent
that referenced
this pull request
Sep 3, 2026
…cker/ssh/modal backends (closes NousResearch#96873) (NousResearch#96991)
kshitijk4poor
added a commit
that referenced
this pull request
Sep 8, 2026
The docs batch (#105782-#105787, tracking #105788) fixed the website docs but flagged two in-code strays it could not touch: - hermes_cli/tips.py:121 still said delegate_task "spawns up to 3 concurrent sub-agents"; the default has been 10 since the v33 config migration (config_defaults.py:1256, config_migrations.py:613). - hermes_cli/config_defaults.py:1778 said "remote backends run per-call", contradicted by tools/code_kernel_remote.py (session kernels on remote backends, failing open to per-call only when the backend cannot spawn one) since #96991. Comment-only changes: tips.py list entry updated, config_defaults.py comment rewritten to match the actual kernel lifecycle. No behaviour change. Validated: test_tips.py 6 passed; ruff, check-windows-footguns, and check_compat_pointers all clean. Refs #105788 (crumb noted in the issue body).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #96873 (maintainer-directed): remote terminal backends get the same always-on session-kernel semantics local received in #94647/#96787. The maintainer's framing drove the design: remote never needed anything special — it needed the three things local gets from owning a child process, rebuilt on the run-to-completion
env.execute()primitive.Design (tools/code_kernel_remote.py, ~450 lines)
nohup … & echo PID:$!), PID recorded; every later cell probeskill -0first. Runner self-reaps afterkernel_idle_timeoutof silence (no host-side reaper thread needed for a process we don't own).cells/cell_req_NNNNNN.json→cells/cell_res_NNNNNN.json(atomic tmp+rename both directions), sibling to the existing tool-RPC file protocol which is reused unchanged. The host-side_rpc_poll_loopis armed per cell with the calling thread's context — the remote analogue of local's CellAuthority (per-cell tool authority; stale rpc req/res files swept before each cell).kernel.note, fresh respawn. Cell timeout = kill kernel + report state loss (same contract as local).Same invariants as local: owner resolution imported from
tools.code_kernel._resolve_owner— one resolver, so the::child::{id}delegated-child qualifier (the #94647 review fix) cannot drift; same generated tool stubs; same output pipeline (truncation → ANSI strip → secret redaction).reset=truehonored. Session-close disposal wired into the sameclear_sessionhook as local kernels.Fail-open: spawn failure (can't fork, no nohup, read-only tmp) falls back to the untouched per-call path with a log note — a degraded remote host never blocks execution. Hosts that can't sustain a background process keep exactly today's behavior.
Verification
Live E2E on real detached processes (contract-faithful env harness: every
env.execute()is a real bash run-to-completion subprocess; real nohup, real files, real kill — only the network hop is absent). All six probes green:reused: true) · exception keeps kernel alive ·reset=truewipesstate_lost: true, fresh kernel, no hangISOLATEDUnit tests (9, CI-runnable, no bash needed): scripted contract-fake env pins spawn parsing, single-spawn reuse, fail-open on no-PID, reset respawn, death detection +
state_lostnote, timeout kill, child/sibling owner isolation, owner-scoped disposal, and both_execute_remotedispatch branches (kernel-result and fall-open-to-per-call).Existing suites: 86 passed; 1 pre-existing failure (TestRpcTokenAuthorization, predates this branch). One existing remote test updated from index-pinned to search-based command assertions (the kernel attempt now precedes the per-call commands).
Refs: #94647 (kernel machinery + isolation fix), #96787 (always-on + corrected remote framing), #88637 (@z80dev — lifecycle model).