Skip to content

fix(bot-mode): service DMs use the active Hermes installation (salvage #100673) - #111298

Merged
teknium1 merged 3 commits into
mainfrom
fix/botmode-dm-matrix
Sep 15, 2026
Merged

teknium1 merged 3 commits into
mainfrom
fix/botmode-dm-matrix

Conversation

@teknium1

@teknium1 teknium1 commented Sep 14, 2026 •

Copy link
Copy Markdown
Collaborator

Bot messages no longer launch an unrelated hermes from a service PATH when the sending interpreter has its own adjacent entrypoint.

Live repro

Live repro: real Linux Electron + production backend and real message_agent/quiet CLI subprocesses; deterministic loopback inference, disposable HOME/profiles/runtime. Base 40f2702b22a343cbc95647efd0bdffe8e0b3d9e9.

Case Base Fixed
Default canonical Desktop owner live, Beta/Gamma unowned; default → quiet Beta → Gamma → replies Pass with worktree PATH shim Pass
Same case with stale PATH hermes and correct interpreter-adjacent launcher sent ACK, stale launcher exits 2, zero Beta inputs Adjacent launcher used, each named input once, both completion follow-ups, stale launcher never invoked
Incoming Beta DM into live Alpha Desktop, no reload Pass after sender card mounts and is expanded Pass

Final rebuilt committed native run: 2 passed. Held provider response proves Beta remains a distinct CLI owner after its initial turn; retained process checkpoint is scoped to matrix-beta, not default. Default Desktop lease remains unchanged, Gamma retains matrix-gamma, and opening Gamma renders the attributed message.

The previous “incoming card needs reload” observation was a probe race: it tried to expand before asynchronous reconciliation mounted the card. The original final DOM already had Message from beta / show message; bounded mount wait + expand works without reload. No speculative renderer fix.

Validation

  • Focused canonical runner: 73 passed, 1 Windows-only skipped across bot DM, relay paths, one-shot linger.
  • Added resolver invariant red on base: expected adjacent entrypoint, received bare hermes.
  • Ruff, Windows-footgun scan, subprocess-stdin check, compat check and full branch git diff --check: passed.
  • Independent review: no production defect; embedded patch whitespace finding fixed and final native suite rebuilt/rerun.
  • Broad tests/tools run was interrupted before completion; observed three failures also reproduce with base production source: delegate timeout cleanup and two Modal snapshot fixtures. Not claiming the full directory green.

Scope and related reports

Related #105460: native default-live/named-unowned matrix refutes #105323's premise that --in ~ selects the default profile DB. It selects cwd; explicit Bot Chat title resolves in the target profile DB.

Related #109750 / #109767: ordinary Linux nested quiet-CLI path works on main with correctly scoped completion ownership. Native Windows is not verified: available host is Linux, no Windows executable/mount/VM or configured native host was discovered. The long-running legacy-child 420-second failure was additionally tested and did not reproduce on current Linux (see measured probe below); Windows remains unverified.

Supersedes #100673 for the proven stale-launcher class; overlaps #108632's alternative executable selection. No merge/close requested automatically. No paid model inference.

Real-wallclock long-child control

An ordinary Desktop → quiet Beta CLI → quiet Gamma CLI chain held Gamma inference for 451.947 real seconds, with no clock acceleration or timeout override. Sequential/concurrent tool budgets remained 420 seconds; quiet notification linger was the existing 600 seconds. Native Electron test passed in 9.0 minutes. Both CLI owners remained live at 305, 425 and 450 seconds; default Desktop lease was unchanged. Default and Beta tool acknowledgements took 0.716s and 0.521s, with exactly one successful completion each after 482.376s and 466.459s respectively. Gamma had exactly one input and answer. The parent independently reran the receipt verifier over /tmp/botmode-dm-long/native. This demonstrates asynchronous message_agent lifecycle, not blocking a2a_call or native Windows parity.

Native screenshots

Alpha live incoming message, expanded without reload:

Alpha incoming message without reload

Gamma named canonical chat after nested CLI delivery:

Gamma named target

Infographic

Bot messages use the right runtime

Combined campaign verification

All four exact PR heads (#111240, #111283, #111273, #111298) were locally integrated onto main 1a990f30628c25fb83d29c4d3b3d18dcb085406e in union ee20e99def270fb4c160018358a938e2a344c822; parent verified zero missing commits from every head. Additive test-fixture conflict resolution preserves both group scripting and held-response behavior. 13 native Linux Electron tests passed, covering group handoffs/queue/Stop/late/error, cron owner deferral and custom-root fallback, nested/stale-launcher delivery, parallel side chats, and busy-DM FIFO. 602 Bot Mode unit tests passed; 155 Python invariants passed, 3 skipped; full Desktop build and typecheck/lint passed (179 existing warnings, no errors). Real backend and tool execution with scripted loopback inference; late-observation clocks accelerated only in the named harvest test. Native Windows remains unverified. Evidence and conflict-resolution patch: /tmp/botmode-campaign/union/verification.json, conflicts.patch, and neighboring logs/screenshots. This is local integration proof, not a merge to main.

liuhao1024 and others added 3 commits September 14, 2026 16:00
…deliveries

Bot-to-bot message_agent delivery builds both transport argvs (local
teammate chat and peer dm) with a bare "hermes" as argv[0]. Since #96631
the delivery runner spawns under terminal_tool's isolated host-local
environment, which does not inherit the gateway's PATH — so on
docker/service installs (venv at /opt/hermes/.venv) every delivery exits
with FileNotFoundError: 'hermes'.

Resolve the CLI with bot_relay._hermes_cli() (#93590) — the venv sibling
of this interpreter, then shutil.which, then the bare name — at both
argv construction sites. The turn-lock matcher in _delivery_lock()
already matches argv[0] by basename, so absolute paths lock exactly as
before.

Fixes #100662
…ership

Retain executable Electron evidence for the named-unowned and live-owner
paths; stale PATH fails on base and passes with the contributor fix.
Clarify that --in selects cwd rather than the profile database.
@github-actions

github-actions Bot commented Sep 14, 2026 •

Copy link
Copy Markdown

૮ >ﻌ< ა ci review

ran on bca2079 — test(bot-mode): keep the reproducible mock patch whitespace-

⚠️ Warnings

OSV vulnerability scan · View job

76 known vulnerabilities found in pinned dependencies.

How to fix:

Review the findings in the Security tab. Update the affected dependencies if a patched version is available.


debug info

CI timings

CI timings · View report · View job

Wall time 6m13s vs 5m48s (+7.2%). 6 job(s) slower, 8 faster, 1 unchanged.

  • OS-specific tests / Windows-only tests: -36.0s
  • Python tests / Run tests: +26.0s
  • OS-specific tests / macOS-only tests: -15.0s
  • Python lints / Windows footguns (blocking): +12.0s
  • Docs Site / docs-site-checks: +7.0s

@alt-glitch alt-glitch added type/bug Something isn't working comp/tools Tool registry, model_tools, toolsets P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Sep 14, 2026
@alt-glitch

Copy link
Copy Markdown

This was generated by AI during triage.

Related: salvage of #100673 (same fix, authorship preserved). Competing open fix for the same bare-hermes argv problem: #108632. Reviewers should pick one; #108631 is the closed predecessor.

@kvnloo kvnloo left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve — the serialization, frozen-boundary watermarks, and failure visibility all check out, and the e2e spec covers the exact reported scenarios.

Two nits, both non-blocking:

  • In queueGroupChatDrive's catch, the drive-level failed event is tagged with the closure's thread (the send that created the drive), not the nextThread whose rounds actually threw — a later queued thread's failure gets attributed to the first thread. Would need nextThread hoisted out of the loop to fix.
  • In the collapsed summary, [...unresolvedFailures.values()].at(-1) follows Map first-insertion order, so A-fails, B-fails, A-fails-again shows B's older failure. Delete-then-set on re-failure would make it recency-ordered.

One design question worth your call: the only reason argv shape matters here is that _delivery_lock / _local_delivery_home parse the child argv to recover the target profile, which was already known at construction time. Would you take a follow-up that passes --profile through _delivery_command (next to the existing --profile-home) and deletes the argv parsing? Then the launcher becomes a free variable — bare name, sibling path, or sys.executable -m hermes_cli.main all work with zero parser updates.

That unlocks a cleaner launcher too: sibling hermes if present, else -m against the running interpreter, dropping the which()/bare tiers — which() can resolve to a different install than the gateway, which is the exact wrong-install failure this PR fixes. Happy to sketch the diff if the direction appeals.

@kvnloo

kvnloo commented Sep 15, 2026

Copy link
Copy Markdown

Following up with the full triage of the hybrid, since the question above deserves a concrete answer.

Proposal: stop parsing argv, resolve the launcher as a list. Two independent changes in tools/bot_mode_dm.py + tools/bot_relay.py, ~40 lines, no new surface:

1. Pass --profile as runner metadata. _delivery_command (bot_mode_dm.py:516-532) already carries --profile-home; add --profile <name> the same way, parse it in _delivery_main (:595-613), and thread it into _run_delivery. _delivery_lock's argv parse becomes acquire_turn_lock(root, profile) directly; _local_delivery_home's fallback goes away (profile is always present). _run_local_turn's argv.index("-p") (:391-396) is untouched and already shape-agnostic.

2. List-returning launcher. New bot_relay.delivery_launcher() -> list[str]: [<sibling hermes>] when a hermes entrypoint sits beside sys.executable, else [sys.executable, "-m", "hermes_cli.main"]. Drop the shutil.which("hermes")/bare tiers. C1/C2/C3 build argv through one helper: [*delivery_launcher(), "-p", profile, ...].

Why this beats both current options:

  • It deletes the bug class, not the symptom. The fix: bind bot delivery to gateway Python #108632 conflict (taught _delivery_lock the -m shape, forgot _local_delivery_home) is what happens every time the launcher changes while parsers read argv. With metadata, the launcher is a free variable forever — no parser ever needs updating again.
  • -m is the only shape that provably launches the active installation. The sibling script is a pip shim with no env setup, so behavior is identical minus one exec hop; but on source-tree runs, embedded interpreters, and pipx/uv installs there is no sibling at all, and fix(bot-mode): service DMs use the active Hermes installation (salvage #100673) #111298's which()/bare fallback can resolve to a different hermes than the gateway — the exact failure class this PR fixes. -m holds everywhere the gateway itself runs.
  • Nothing else reads the shape. The canonical process matchers already accept -m (gateway/status.py looks_like_gateway_command_line, hermes_cli/profiles.py _is_hermes_argv); delivery children never match the gateway subcommand filter, so no fleet-update interaction. Windows skips the .exe launcher chain.

Costs, honestly: the sibling tier exists only as a transition courtesy for the old parsers; once step 1 lands they have no consumers and could go. Tests change shape: assert on delivery_launcher() output and on --profile plumbing instead of pinning _hermes_cli against PATH. And the -m tier needs the hermes_cli.main module importable from the delivery child's environment — true wherever the gateway runs, which is the only place delivery children spawn.

Either PR can carry this; #111298's structure (salvage authorship, native matrix evidence) is the natural base. If you want the smallest increment instead: step 1 alone, keeping _hermes_cli() as-is, still kills the parser class — the -m tier can follow without touching parsers ever again.

@teknium1
teknium1 merged commit b8bf484 into main Sep 15, 2026
40 checks passed
@teknium1
teknium1 deleted the fix/botmode-dm-matrix branch September 15, 2026 00:04
@teknium1

Copy link
Copy Markdown
Collaborator Author

Thanks @kvnloo. We landed the verified interpreter-adjacent resolver repair in b8bf484. The native stale-PATH case and nested replies passed, including a 451.947-second hold. The no-adjacent-entrypoint fallback remains the existing resolver contract; removing its PATH tiers and carrying explicit profile metadata is a separate follow-up design, not claimed by this patch. We have not closed #108632 as redundant. The two group-summary nits posted here are also on #111283 and are being verified there before its merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/tools Tool registry, model_tools, toolsets P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants