Skip to content

feat(code-execution): remote kernel host — session persistence for docker/ssh/modal (closes #96873) - #96991

Merged
teknium1 merged 1 commit into
mainfrom
remote-kernel-host
Aug 28, 2026
Merged

teknium1 merged 1 commit into
mainfrom
remote-kernel-host

Conversation

@teknium1

Copy link
Copy Markdown
Collaborator

Implements #96873 (maintainer-directed): remote terminal backends get the same always-on session-kernel semantics local received in #94647/#96787. The maintainer's framing drove the design: remote never needed anything special — it needed the three things local gets from owning a child process, rebuilt on the run-to-completion env.execute() primitive.

Design (tools/code_kernel_remote.py, ~450 lines)

  1. Process that outlives one env.execute(): kernel runner started detached (nohup … & echo PID:$!), PID recorded; every later cell probes kill -0 first. Runner self-reaps after kernel_idle_timeout of silence (no host-side reaper thread needed for a process we don't own).
  2. Cell protocol over files: cells/cell_req_NNNNNN.jsoncells/cell_res_NNNNNN.json (atomic tmp+rename both directions), sibling to the existing tool-RPC file protocol which is reused unchanged. The host-side _rpc_poll_loop is armed per cell with the calling thread's context — the remote analogue of local's CellAuthority (per-cell tool authority; stale rpc req/res files swept before each cell).
  3. Death detection, never a hang: every wait is bounded by the cell timeout; a failed liveness probe (ssh drop, container restart, OOM, idle self-exit) reads as kernel died: state lost, reported in kernel.note, fresh respawn. Cell timeout = kill kernel + report state loss (same contract as local).

Same invariants as local: owner resolution imported from tools.code_kernel._resolve_owner — one resolver, so the ::child::{id} delegated-child qualifier (the #94647 review fix) cannot drift; same generated tool stubs; same output pipeline (truncation → ANSI strip → secret redaction). reset=true honored. Session-close disposal wired into the same clear_session hook as local kernels.

Fail-open: spawn failure (can't fork, no nohup, read-only tmp) falls back to the untouched per-call path with a log note — a degraded remote host never blocks execution. Hosts that can't sustain a background process keep exactly today's behavior.

Verification

Live E2E on real detached processes (contract-faithful env harness: every env.execute() is a real bash run-to-completion subprocess; real nohup, real files, real kill — only the network hop is absent). All six probes green:

  • state persists across cells (reused: true) · exception keeps kernel alive · reset=true wipes
  • kill the kernel between cells → next call: state_lost: true, fresh kernel, no hang
  • delegated-child isolation on remote: parent-planted global → child sees ISOLATED
  • cell timeout → kernel killed, loss reported, next call respawns clean

Unit tests (9, CI-runnable, no bash needed): scripted contract-fake env pins spawn parsing, single-spawn reuse, fail-open on no-PID, reset respawn, death detection + state_lost note, timeout kill, child/sibling owner isolation, owner-scoped disposal, and both _execute_remote dispatch branches (kernel-result and fall-open-to-per-call).

Existing suites: 86 passed; 1 pre-existing failure (TestRpcTokenAuthorization, predates this branch). One existing remote test updated from index-pinned to search-based command assertions (the kernel attempt now precedes the per-call commands).

Refs: #94647 (kernel machinery + isolation fix), #96787 (always-on + corrected remote framing), #88637 (@z80dev — lifecycle model).

@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

૮ >ﻌ< ა ci review

ran on 89d1dfc — feat(code-execution): remote kernel host — session persisten

⚠️ Warnings

OSV vulnerability scan · View job

6 known vulnerabilities found in pinned dependencies.

How to fix:

Review the findings in the Security tab. Update the affected dependencies if a patched version is available.


debug info

CI timings

CI timings · View report · View job

Wall time 3m45s vs 3m57s (-5.1%). 5 job(s) slower, 7 faster,

  • Python tests / Run tests: -27.0s
  • OS-specific tests / macOS-only tests: -26.0s
  • OS-specific tests / Windows-only tests: -16.0s
  • Detect affected areas: +13.0s
  • Check contributors / check-attribution: +10.0s

@alt-glitch alt-glitch added type/feature New feature or request comp/tools Tool registry, model_tools, toolsets tool/code-exec execute_code sandbox backend/docker Docker container execution backend/ssh SSH remote execution backend/modal Modal.com cloud execution P3 Low — cosmetic, nice to have sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data labels Aug 28, 2026
@teknium1
teknium1 merged commit 5f75ec1 into main Aug 28, 2026
39 checks passed
@teknium1
teknium1 deleted the remote-kernel-host branch August 28, 2026 08:39
mrkillbob pushed a commit to mrkillbob/hermes-agent that referenced this pull request Aug 28, 2026
teknium1 pushed a commit that referenced this pull request Sep 3, 2026
Local session kernels sweep idle-expired entries and enforce a
process-wide cap (DEFAULT_MAX_SESSION_KERNELS) on every call
(tools/code_kernel.py's _reap_unlocked / _evict_over_cap_unlocked). The
new remote kernel host (#96991) never got the same treatment:
_REMOTE_KERNELS only shrinks lazily when a specific key is revisited and
found dead, so an owner that opens kernels for several distinct
(env_type, task_env_id) combinations (or delegated children) and never
revisits some of them accumulates host-side bookkeeping entries for the
life of the gateway process.

Note this is narrower than the local case: the remote runner already
self-reaps on its own idle timeout, and SSH/Docker connections are
independently bounded by their own transport-level lifecycles (SSH
ControlPersist, Docker's session-scoped idle-timeout in terminal_tool.py)
— so nothing here leaks a live remote connection. What's missing is
purely the host-side dict/cap bookkeeping symmetry with local kernels.

Adds _reap_unlocked/_evict_over_cap_unlocked mirroring the local
implementation, reusing the same max_session_kernels config as an
independent cap on _REMOTE_KERNELS.
melon-xf added a commit to melon-xf/hermes-agent that referenced this pull request Sep 3, 2026
kshitijk4poor added a commit that referenced this pull request Sep 8, 2026
The docs batch (#105782-#105787, tracking #105788) fixed the website docs but
flagged two in-code strays it could not touch:

- hermes_cli/tips.py:121 still said delegate_task "spawns up to 3 concurrent
  sub-agents"; the default has been 10 since the v33 config migration
  (config_defaults.py:1256, config_migrations.py:613).
- hermes_cli/config_defaults.py:1778 said "remote backends run per-call",
  contradicted by tools/code_kernel_remote.py (session kernels on remote
  backends, failing open to per-call only when the backend cannot spawn one)
  since #96991.

Comment-only changes: tips.py list entry updated, config_defaults.py comment
rewritten to match the actual kernel lifecycle. No behaviour change.

Validated: test_tips.py 6 passed; ruff, check-windows-footguns, and
check_compat_pointers all clean. Refs #105788 (crumb noted in the issue body).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend/docker Docker container execution backend/modal Modal.com cloud execution backend/ssh SSH remote execution comp/tools Tool registry, model_tools, toolsets P3 Low — cosmetic, nice to have sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state tool/code-exec execute_code sandbox type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants