fix(gateway): keep a chat's media and quick text in arrival order by dokterdok · Pull Request #140454 · NousResearch/hermes-agent

dokterdok · 2026-10-17T10:05:52Z

What does this PR do?

The problem. On a messaging platform, if you send a photo or video and then a separate text ("what is this?"), the text can run first as its own turn, without the media. The agent answers about an image it hasn't seen, and the media then runs as a second turn. This happens idle or while a turn is running, on any adapter, and needs no added delay: delivery back to back is enough. A large video or a slow disk widens the window.

Why.

prepare_native copies, hashes and fsyncs native media in a worker thread (gateway/session_envelope.py), inside SessionAuthority.admit_native and before the ordered admission write. Keeping that off the owner loop is right: a large file would otherwise freeze every chat.
tracked_write(ordered=True) orders writes only from the moment they are called. Nothing held the chat's place during the capture, so a text that needs no capture reached the write first and took the earlier FIFO slot.
Adapter messaging reaches admit_native through session_ingress.admit_message on the idle, busy and burst paths, so all of them were affected. Relay is not: it already admits one chat's messages one at a time, on its reader or in per-chat lanes. Webhook deliveries each get their own session.

The fix. Three parts, one per invariant.

Arrival order. admit_native holds a per-route asyncio.Lock (SessionAuthority._route_turn) from entry until the admission write returns or raises. The key is the session key (session_store._generate_session_key(event.source)), computed before the first await. A later input of the same chat waits for an earlier one that is still capturing. The lock map is refcounted and drops a route once nobody holds or waits on it. An event without a routable source gets no lock and is refused by prepare_native as before.
Stop and reset still reach every earlier input. A messaging input waiting for its route, or still capturing, has no row yet for /stop to cancel. Each route entry therefore carries a stop fence. _hm_stop_owner_turn (the hook behind /stop, /new and /reset, busy, idle or confirmed) bumps it before it lists the queued rows to cancel, including when the chat has no session yet. An input that entered admission before the bump is refused inside its own admission transaction (stale_generation, field route). The ingress drops it silently, as main drops a cancelled queued row. A FIFO paused on a lost turn keeps its queued followers through /stop and /new (its pause notice promises they stay saved), so there the fence is not bumped either and in-flight input queues as on main.
One chat's wait never stalls a transport reader. Signal, IRC and Mattermost await each busy input's durable acceptance (dispatch_shared_busy) before reading the next frame. When the input queues behind an earlier input of its chat, its FIFO position is fixed, so the reader is released then. Other chats and /stop keep flowing. The input is not acknowledged early: _gateway_accepted is still set only by the commit, and a later refusal reaches the chat as it did when it happened after commit.

Invariants.

Within one route, inputs commit to the FIFO in the order they entered admit_native, whatever capture each needs.
A messaging input that entered admission before a /stop, /new or /reset of its chat never commits after it.
Another route never waits, and neither does a transport reader, beyond what main already waits for (an input's own capture).

Design choice.

A lock, not a ticket handed on in finally. asyncio.Lock is FIFO, and a cancelled waiter gives up only its own place. With a ticket released in finally, a cancelled middle input lets a third overtake the first while it is still capturing (['x3-text', 'x1-video']).
Held from entry, not just around the capture. The role re-check awaits before the capture, and the route reservation after it. Holding from entry to commit covers all three.
The fence is a counter on the route entry, checked in the write transaction. No new state outlives the in-flight admissions. A route with nothing in flight has nothing to fence. /stop bumps the fence before it lists the queued rows to cancel. An admission is then either refused or committed early enough to be listed and cancelled; only the instant inside its commit stays racy, as on main.
The fence and the reader release apply to messaging ingress only (a context variable set by admit_message). Relay and webhook producers keep main's behaviour; neither waits on the lock in practice.
Capture stays off the loop. Moving it back would restore the order by freezing every chat behind one large file.

Before / after (session authority on; "first version" is this PR before the review round):

Case	Before (`main`)	After
Video or photo while idle, then a quick text, in that order at the gateway (Telegram's photo buffering: Known limits)	text first, media as a second turn	media, then text
Both while a turn runs, dispatched concurrently	text first	media, then text
Middle input cancelled while the first still captures	the third commits before the first	third after first
`/stop` or `/new` while an earlier video captures and a text waits	the text committed first and is cancelled; a capturing busy input survives	both refused, silently (first version: both survived, after `/new` onto the fresh conversation)
`/stop` or `/new` on a FIFO paused on a lost turn, while a video captures and a text waits	the text stays queued, ahead of the video	both stay queued, in order (before the final pass: both dropped)
Serial reader (Signal, IRC, Mattermost): busy text behind a capturing video, then another chat, then `/stop`	all read	all read (first version: the reader stalled until the capture ended)
Another chat's text during a slow capture	commits at once	unchanged
A single text, nothing earlier in flight for the chat	commits	unchanged (uncontended lock)
Busy same-sender photo or text bursts	merged	unchanged

Related Issue

Fixes #140453

Refs #134889

Overlap:

fix(gateway): honor external drain at durable input admission #140135, fix(gateway): resume cold persisted messaging sessions #139802 and fix(gateway): restore messaging sessions after restart #139668 edit gateway/session_authority.py near SessionAuthority.__init__ / authorize. Textual adjacency only: this PR adds one attribute at the end of __init__ and changes admit_native.
fix(gateway): keep busy follow-ups from being dropped, merged across senders or reordered #129126 (busy follow-ups dropped, merged across senders or reordered) works in the adapter's pending slot and busy merge, and touches gateway/run_inbound.py elsewhere. No hunk overlap; this PR orders the durable FIFO behind it.
fix(telegram): replay held inbound in arrival order #138795 (Telegram: replay held inbound in arrival order) and [Bug]: Telegram text+photo sends can start a text-only turn before the image is attached #33270 (Telegram text then photo starts a text-only turn) are adapter-side orderings. Neither is fixed here; see Known limits.

Type of Change

🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

gateway/session_authority.py: admit_native runs inside _route_turn(event), a refcounted per-route asyncio.Lock held from entry to commit, whose entry also carries the stop fence. fence_route(route) bumps it, and the admission transaction checks it. The messaging_admission context variable marks messaging input and carries the busy caller's acceptance future, released once the input queues behind an earlier one.
gateway/session_ingress.py: admit_message sets messaging_admission around admit_native and drops a fenced input silently. The dispatch_shared_busy docstring states the new release point.
gateway/run_inbound.py: _hm_stop_owner_turn fences the route before its no-session early return; otherwise it lists the rows, fences the route unless the FIFO is paused, and lists them again to cancel the queued ones.
tests/gateway/test_native_admission_arrival_order.py (new):
- test_media_then_quick_text_keep_arrival_order[idle|busy]: the real runner and adapter path with the real capture held on an Event. An idle video then a busy text, and both while a turn runs. The text's dispatch returns while the video still captures, without the text being committed or acknowledged.
- test_a_cancelled_middle_input_does_not_let_the_next_overtake: video held, the second input cancelled while it waits; the third commits after the first, and the route entries are gone afterwards.
- test_a_held_capture_never_delays_another_chat_nor_outlives_its_cancel: another chat commits within 1 s during the held capture, and cancelling the capturing input releases the next one.
- test_stop_fences_inputs_still_capturing_or_waiting[existing|first-message|paused]: through admit_message, a video held in capture and a text waiting behind it; the real _hm_stop_owner_turn leaves nothing queued and both callers return None, also when the chat has no session yet. On a FIFO paused on a lost turn both stay queued, in order.
tests/gateway/test_native_fifo_context_exhaustion.py: test_admission_reserved_on_the_old_route_commits_on_the_fresh_session set up two same-chat admissions reserving the old route at once. The second now waits behind the first at admission, so the test expects that; the retry still commits on the fresh session, and the second queues behind it there.

How to Test

scripts/run_tests.sh tests/gateway/test_native_admission_arrival_order.py tests/gateway/test_native_fifo_context_exhaustion.py

Fail first. With gateway/ at 0ab43d4f67, six of the seven new cases fail: idle t1 already committed ahead of the video, busy likewise, cancelled middle ['x3-text', 'x1-video'], both /stop cases leave the capturing video queued, and the paused case queues ['x2-text', 'x1-video']. The cross-chat and cancel case passes on both and pins that the lock adds no new waiting. The updated exhaustion test fails on main too: the second admission reserves the old route alongside the first. Against this PR's first version (before the review round), the two adapter-path cases fail on the reader check and both /stop cases leave ['x1-video', 'x2-text'] queued. Against the version after the review round, the paused case fails: both inputs are dropped.
On this branch: 13 passed over the two files. The touched modules' suites give 470 passed, 2 skipped, 0 failed over 74 files: every admit_native caller, native media capture, lease, ownership and redelivery, busy burst merge and the shared busy receive loop, busy durable acceptance and marker hand-off, messaging /stop and /new (cancel queued, owner turn, chat scope, thread siblings, reset, and /reset out of a paused FIFO), pre/post admission hooks and the relay reader, webhook and relay producers, Signal, IRC and Mattermost admission, and the session_authority suites. Fifteen files that drive /stop, /new and /reset (confirmed resets and /reset out of a paused FIFO among them) give 60 passed, 3 skipped (Linux-only), 0 failed.
Probes (kept out of the PR): the review round's probes all pass on this branch. A serial-reader probe (idle video held in capture, then the chat's follow-up, another chat's text and /stop, read one at a time) reads all four, leaves nothing of the stopped chat queued and runs the other chat. On main the follow-up is committed and started before the video. A paused-FIFO probe (serial reader, the head unknown, an idle video held in capture, its follow-up, then /stop) keeps the follow-up queued, as main does.
scripts/check --base 0ab43d4f67: 12 checks ok; code health 0 blocking, 0 advisory.

Independent review: An independent review round found 4 P2s (two regressions in this fix's first version, two gaps main shares). The two regressions are fixed above, each with a regression test that fails on the first version: /stop, /new and /reset now reach inputs still waiting or capturing, and serial readers no longer stall. The two shared gaps, adapter-side work before admission, are known limits below. A final review-and-fix pass then reviewed the whole change and found one more P2 and no P1: on a FIFO paused on a lost turn, where main's /stop and /new keep the queued followers, the new fence dropped a message still in flight. It is fixed above, with the paused case as its regression test. The rest of what it checked (fencing of idle, busy, capturing and waiting input; no post-stop input refused; reader release, acceptance and error reporting; lock cleanup on refusal and cancellation; no cross-chat waits) held, and its smaller findings are known limits below.

Known limits (shared with main unless stated):

Adapter work before admission can still reorder. The order point is entry to admit_native.
- An idle first message awaits the adapter's on_processing_start before its handler (Discord adds a 👀 reaction by default; Telegram, Slack, Matrix and Signal can too). The pre_gateway_dispatch and post_gateway_admission hooks also run before it. A busy follow-up skips the reaction, so one sent within that round trip can go first. With Discord's default 0.6 s text batching, a photo then a text 50 ms later overtakes only when the reaction takes longer than about 0.65 s. With batching off, the whole reaction interval is exposed.
- Adapter buffering and downloads sit before it too. Telegram's defaults buffer photos and albums for 0.8 s but short text for 0.18 s, so a photo and a text sent within about 0.6 s are reordered before they reach the gateway, idle or busy. This PR doesn't fix that Telegram case. Discord and Slack download attachments before dispatch.
- Follow-up: reserve the route's admission position at arrival (adapter dispatch and batching enqueue, on the owning profile's canonical route), and release it at commit, refusal, consumption or cancellation; a merged batch keeps its first input's position.
Relay and webhook inputs are not fenced by /stop, as on main: a relay message still in flight when /stop arrives can commit after it.
A busy input's own capture still holds a serial reader until it commits, as on main.
A capture that never returns (a hung disk) holds the chat's later inputs, where main let them commit ahead. The earlier input hangs on main as well, and /stop still reaches the waiting inputs.
A restart that begins during a capture refuses the chat's inputs waiting behind it (runtime_draining, reported to the chat), as it refuses the capturing input itself on main. On main a text behind it had already committed and ran after the restart.
On a FIFO paused on a lost turn, /new leaves input still in flight alone, as it leaves the queued input. One still waiting behind a capture then commits to the new conversation, where main had already queued a text on the paused one.

Still needs a live retest:

WhatsApp: a large video, then a quick text, idle and while a turn runs;
Signal or IRC: a large video, then a text, another chat's message and /stop, to check the reader keeps reading;
/new while a video is still being captured, on an ordinary chat and on one paused after a restart;
Discord with reactions on and off, and Telegram photo then text, to size the adapter-side windows above.

Checklist

Code

I've read the Contributing Guide
My commit messages follow Conventional Commits
I searched for existing PRs to make sure this isn't a duplicate. Related PRs are listed under Overlap above
My PR contains only changes related to this fix/feature (no unrelated commits)
I've run pytest tests/ -q and all tests pass. Not ticked: I ran the related suites above, not the whole tree; CI is the full run
python scripts/check passes
I've added tests for my changes
I've tested on my platform: macOS (Python 3.14)

Documentation & Housekeeping

Documentation: N/A; the rules are in the _route_turn, fence_route and messaging_admission comments
cli-config.yaml.example: N/A
CONTRIBUTING.md / AGENTS.md: N/A
Cross-platform: N/A (asyncio only, no file or process handling)
Tool descriptions/schemas: N/A

A photo or video followed quickly by a separate text ("what is this?") on a messaging platform made the text run first, without the media, and the media then ran as its own turn. Native media capture (copy, SHA-256, fsyncs) runs in a worker thread inside admit_native, before the ordered admission write, and nothing held the chat's place while it ran: the text reached the write first and took the earlier FIFO slot. SessionAuthority.admit_native now takes a per-route asyncio.Lock, keyed by the session key and computed before any await, and holds it until the admission write returns or raises. A chat's later input waits for an earlier one still capturing; other chats never wait. asyncio.Lock is FIFO and a cancelled waiter gives up only its own place, so a cancelled middle input never lets a later one overtake. The lock map is refcounted and drops a route once idle. A messaging input waiting for its route, or still capturing, has no row for /stop to cancel. Each route entry carries a stop fence that _hm_stop_owner_turn (/stop, /new, /reset) bumps before it lists the queued rows to cancel, also when the chat has no session yet; an input that entered admission before it is refused in its own admission transaction and dropped silently, as a cancelled queued row is. A FIFO paused on a lost turn keeps its queued followers through /stop and /new, so the fence is not bumped there either. Serial readers (Signal, IRC, Mattermost) await a busy input's durable acceptance before reading on. Once the input queues behind an earlier input of its chat its position is fixed, so dispatch_shared_busy releases the reader then; acceptance itself is still set only by the commit. The exhaustion re-home test now expects the chat's second late input to wait behind the first at admission instead of reserving the old route alongside it. Refs NousResearch#134889 Fixes NousResearch#140453

Refs #134889 Fixes #140520 (cherry picked from commit d689cb8) [teknium: resolved a conflict in SessionAuthority.admit_native with the MSG-23 route lock (#140454): the capture lease and its abandon cleanup now run inside _route_turn.]

dokterdok · 2026-10-18T18:41:38Z

Landed via #140973 (merged as 54809183676), salvaged with my commits kept as authored. Closing.

* fix(plugins): fire post_compaction on every committed compaction path The salvaged hook fired from ContextCompressor.compress(), which is not where a compaction commits: compress_context can still refuse or roll back the candidate (snapshot stale, commit fence, split failure), the gateway's /compress runs its own compressor and commits elsewhere, and Codex-native, micro-compaction and proactive-prune rewrites never call compress() at all. session_id was also the compressor's bound id, which is stale after rotation. Fire it from the compaction-record seam (agent/compaction_events.py) that already sees exactly one committed record per rewrite on every agent path, and from the gateway session.mutate compress commit. Payload: session_id (the session holding the compacted transcript), old_session_id, kind, reason, before/after token estimates, messages_removed. Bounded by plugins.hook_callback_timeout, since it runs inside the commit window. Tests trimmed to one invariant (fires once per committed compaction, never for an aborted one, raising plugin isolated); docs in hooks.md and the plugin guide. * fix(plugins): post_compaction survives concurrent sessions and a failed projection Key the bounded dispatcher's running-callback gate by session_id, so one session's slow callback no longer skips another session's compaction. Fire the gateway /compress hook in a finally after the receipt commits, so a failed runtime projection cannot swallow it; exact retries (applied=False) still never refire. Review findings from Greptile on #141017. * feat(plugins): ctx.post_message — post one message to the current conversation and edit it in place A plugin can now post a status message into the conversation it is working for and update that same message later, on any channel, with no per-platform code (#133306). - ctx.post_message(text) -> PluginMessage; handle.edit(text). Async twins ctx.apost_message / handle.aedit for code on the gateway loop. - The conversation is the session bound to the calling tool call, slash command or hook; the plugin never names a chat. No bound session (timer, plugin thread, cron) or no live gateway -> ok=False with an error code, never an exception. - Messaging chats go through the answering adapter's send/edit_message. Adapters without edit support, or a refused edit, get a fresh send and the handle follows the new message (edited=False, fallback=True). - Local CLI/TUI/Desktop sessions get a keyed notification.show that Desktop and the Ink TUI replace in place. - Refused in the out-of-process plugin host (handle is loop-bound). - Every post/edit is logged as plugin_message plugin=... op=post|edit. * fix(plugins): post_message review fixes — interim sends, full bound route, pending timeouts, raised-edit fallback - Mark every post/fallback send _interim_send so a relay stream open for the turn's answer is never sealed with plugin progress. - Rebuild a first-message source from every field the gateway binds (scope_id, user_id_alt, parent_chat_id, ...) so it keys like the live one. - A send past the 30 s wait keeps running as the handle's pending operation; its result still lands on the handle and the next edit waits for it, so a slow post never leaves an orphan message. - An edit_message that raises falls back to a fresh send like a refused one. * fix(plugins): post_message review round 2 — no loop block on pending edits, canonical first-message route, failed-edit fallback - Sync edit() refuses on the gateway loop before any wait on a pending operation (the wait would block the loop that operation needs). - A first-message route is rebuilt through _plugin_origin_source, which stamps the receiving bot provenance and checks the profile, like an inject_message origin. - Branch on SendResult.success, not on an empty error string. * fix(plugins): post_message first-message route carries the live inbound source The plugin-command dispatch binds the live SessionSource (receiving-bot provenance included) task-locally, so a post from a command that is the chat first message goes through the bot that received it instead of a source rebuilt from session env vars. * fix(desktop): reopen a gateway group by name after a restart, never fork it Group bindings live in memory, keyed by room_id. After a Desktop restart, opening a group by name (a roster row or a restored tab) missed the binding and fell through to GroupExecutionGate, whose only action was groups.create with a fresh UUID, so every click minted a same-name fork room while the original kept its history. On a miss, the gate now resolves the name (or the room id of a restored canonical: key) against groups.list and adopts the existing room. If the lookup fails, the gate fails closed with an error and a Retry button (the canonical-chat "retry, never mint" contract). "Start gateway group" is offered only when the authority answered and holds no room of that name, and never for a canonical: key. Refs #140805 Refs #140719 * fix(desktop): keep "All profiles" across a boot-time source restore selectConnection decided whether a call was the silent boot restore from `$activeConnectionId === null`. That only holds while nothing has published a source yet — main opens the local backend before the connection registry settles, so the restore that actually runs sees a live source, reads as a user-initiated switch, and collapses the persisted "All profiles" browse mode (#93197 assumed the null-id case was the only one). The preference is written back as false, so the next launch starts in single-profile scope too: the sidebar flashes every profile's sessions, then drops back to the gateway's. The boot path now says what it is: initializeConnectionsRegistry passes `restore: true` and selectConnection honours it (the null-id inference stays as the fallback for the first pass). The same flag guards the "pair already live" branch, which is otherwise reached only because browse mode made the pair look like a switch. Regression test: "a boot-time restore of the last-used source leaves 'All profiles' on" (store/connections.test.ts) — fails before this change (the restore sets $showAllProfiles to false), passes after it. (cherry picked from commit c612a2f4b022d34070804cd17b4e7705f43007bc) * fix(desktop): keep all profile sessions across paged merges Refs #134889 Fixes #140518 (cherry picked from commit 04801e9a65fdc0b0d67328398f2c474a78d6a73d) * fix(gateway): name every profile and the install in the authority's profiles.list After `hermes update` the Bots tab showed only `default`. The roster logged `answered=-` on every refresh. The gateway authority's profiles.list (gateway/session_group_controls.py) named only its own profile, the siblings this process multiplexes, and the hosted_rooms entries. It also dropped the `install_id` that the pre-authority tui_gateway handler returned. On a host that runs one gateway per profile (no multiplexing, no hosted_rooms entries), the answer was `default` alone with no install id. Desktop's mergeMultiSourceRoster skips every union agent the active source didn't return, so the other profiles disappeared from the roster. The roster now also discovers the root and every named profile on disk, metadata only, as the old handler did. A discovered row carries no session fields unless this socket already routes to that profile. The reply carries `install_id` again (the ProfilesListResult contract field). Refs #134889 (RT-38) Refs #140417 Refs #140719 * refactor(desktop): keep selectConnection under its complexity cap The boot-restore flag from the previous commit took selectConnection from CC 34 (its cap on main) to 36. This moves the restore check and the "pair already live" branch into two named helpers. Behaviour is unchanged, and connections.test.ts passes 34/34. Refs #134889 (RT-45) Refs #140719 * fix(gateway): catch only OSError around the roster's install id read_or_create_install_id raises OSError for an unreadable or unwritable root. A blind except would hide a real bug in the roster reply. Refs #134889 (RT-38) Refs #140719 * fix(desktop): count an overlapping registry source's sessions once applyRegistrySessionTotals added a registry scope's reported total minus the rows it fetched that were already in the merged list. That removes only the fetched duplicates, not the whole scope the primary base had already counted. With a remote registry primary that is also a pooled registry source, 650 sessions at limit=20 became 1,280. A scope that produced any duplicate is already counted, so it adds only the rows it actually added. Only a scope that added every row it fetched contributes its full backend total. Review finding by @kshitijk4poor on #140972 Refs #134889 (RT-42) Refs #140719 * fix(desktop): bind a reopened group to its own gateway and drop late lookups resolveCanonicalGroup published its binding itself, before the caller's cancelled check, and aliased it under the bare group name. A lookup still pending on connection A published A's room after the user had switched to B, and once "Team" resolved on A, opening "Team" from B's roster reused A's room without a lookup. Either way the workspace sent messages to the other gateway's room. The resolver is now read-only. The gate publishes the result with publishCanonicalGroup only when its effect is still current, and a name alias is keyed by (connection, profile, name). GroupChatWorkspace and the tab title read the binding through canonicalGroupBindingFor, which matches a name only on the source it was resolved on. A canonical: key still names its own route. Review finding by @kshitijk4poor on #140972 Refs #134889 (#140805) Refs #140719 * fix(desktop): page a deep Archived load in 500-row pages On the Electron merge path (remote profiles or registry gateways) every Archived page is answered by re-reading the primary prefix through offset + limit, so the 200-row cursor re-read that prefix once per 200 rows. That grows as N^2/200: 650 archived rows took 6 primary calls and sent 2,000 rows to the renderer. listEveryArchivedSession now pages at the backend cap of 500 rows (`le=500` on /api/profiles/sessions). 650 rows take two renderer requests and three primary reads. The cursor still advances by the page size, and the stop rules are unchanged. Review finding by @kshitijk4poor on #140972 Refs #134889 (RT-42) Refs #140719 * fix(desktop): translate the gateway group reopen states and Retry The opening, lookup-failure and missing-room messages added for #140805, and the gate's Retry button, were hardcoded English. The three messages are now `gatewayGroupOpen.*` in the core en catalog, with their locale gaps recorded in overlay-gaps.json. Retry reuses `common.retry`. The "Checking group driver…", legacy-room and "Start gateway group" strings already existed on main and are left for a follow-up. Review finding by @kshitijk4poor on #140972 Refs #134889 (#140805) Refs #140719 * fix(desktop): keep a later page's scan errors in a deep primary prefix readProfileSessionRows returned the first page with all rows spliced in, so for limit + offset > 500 the later pages' `errors` and `storage` were dropped. A profile whose scan failed at offset 500 looked like a complete read, and the renderer cleared its unread rows instead of keeping them and showing the failure. The reader now collects errors (deduplicated) and storage across every page. Review finding by @kshitijk4poor on #140972 Refs #134889 (RT-42) Refs #140719 * fix(desktop): keep live pins out of the Archived cursor on the merge path On the Electron merge path, composeUnifiedSessionResponse windowed every merged row, live pins included, while `total` counts archived rows only. With a page or more of live pins newer than the archived rows (200 pins and one old archived chat), the first window held only pins, the cursor passed `total`, and the Archived loader stopped before the archived row. On an archived-only page the window now covers archived rows only, and live pins ride after it, as the backend already does. The primary prefix read for archived=only keeps paging until it has `want` archived rows (or the backend runs out), so an older backend whose pins take cursor positions still supplies them. The new test runs the loader's own stop rule over both backend orderings. Review finding by @greptile-apps on #140972 Refs #134889 (RT-42) Refs #140719 * fix(gateway): skip an unreadable sibling profile.yaml and cache the roster's disk scan profiles.list disk discovery caught only OSError and YAMLError around a sibling's profile.yaml. A file saved in a legacy Windows code page (`display_name: André` in cp1252) raised UnicodeDecodeError, and the healthy default's whole roster failed. That row now degrades to basic metadata, and the unreadable file is logged with the profile name. The Bots pane polls profiles.list every 5 s, and every call re-listed profiles/ and re-parsed every sibling profile.yaml (~60 ms at 50 profiles). The listing is now cached by the mtimes of profiles/ and each entry, which a new profile, its identity marker or a tombstone bumps. A parsed profile.yaml is cached by (path, mtime_ns, size). An unreadable profiles/ drops the siblings, not the roster. The dead `except OSError` around get_install_id() is removed, since that call already returns None on OSError. Review finding by @dokterdok on #140972 Review finding by @kshitijk4poor on #140972 Refs #134889 (RT-38) Refs #140719 * refactor(desktop): return a registry source's first page with its rows fetchSessionRowsInPages returned rows and sent their first page through an onFirstPage callback. Each caller had to capture it in a `let` and reset it on the fallback read. It now returns `{ rows, first }`, or null when any page failed. Behavior is unchanged; "a failed later registry page does not contribute a stale first-page total" still pins the all-or-nothing rule. Review finding by @greptile-apps on #140972 Refs #134889 (RT-42) Refs #140719 * refactor(desktop): offset this lane's growth in main.ts and the en catalog The code-health ratchet blocked main.ts (+2), i18n/en.ts (+5) and i18n/types.ts (+6), which are already over target on main. The skillDeepLink and new gatewayGroupOpen copy move into en_deep_links.ts and types_deep_links.ts, following the en_clarify / types_clarify pattern. The registry merge and archived-only compose calls in mergeRemoteProfileSessions are written on fewer lines. No behavior change. Refs #134889 (#140805) Refs #140719 * fix(gateway): keep a chat's media and quick text in arrival order A photo or video followed quickly by a separate text ("what is this?") on a messaging platform made the text run first, without the media, and the media then ran as its own turn. Native media capture (copy, SHA-256, fsyncs) runs in a worker thread inside admit_native, before the ordered admission write, and nothing held the chat's place while it ran: the text reached the write first and took the earlier FIFO slot. SessionAuthority.admit_native now takes a per-route asyncio.Lock, keyed by the session key and computed before any await, and holds it until the admission write returns or raises. A chat's later input waits for an earlier one still capturing; other chats never wait. asyncio.Lock is FIFO and a cancelled waiter gives up only its own place, so a cancelled middle input never lets a later one overtake. The lock map is refcounted and drops a route once idle. A messaging input waiting for its route, or still capturing, has no row for /stop to cancel. Each route entry carries a stop fence that _hm_stop_owner_turn (/stop, /new, /reset) bumps before it lists the queued rows to cancel, also when the chat has no session yet; an input that entered admission before it is refused in its own admission transaction and dropped silently, as a cancelled queued row is. A FIFO paused on a lost turn keeps its queued followers through /stop and /new, so the fence is not bumped there either. Serial readers (Signal, IRC, Mattermost) await a busy input's durable acceptance before reading on. Once the input queues behind an earlier input of its chat its position is fixed, so dispatch_shared_busy releases the reader then; acceptance itself is still set only by the commit. The exhaustion re-home test now expects the chat's second late input to wait behind the first at admission instead of reserving the old route alongside it. Refs #134889 Fixes #140453 (cherry picked from commit 3a8ffdf2962706a2aad7a56119e9290752038d3b) * fix(gateway): drop a claimed chat turn when /new moves its route A /new that lands after the drain claimed a chat turn, but before that turn looked up its session, let the old turn resolve the route's fresh session and run there. No owner adopts its agent on the fresh session, so the Stop that /new sent was lost and /stop could not reach it. Main's /new cancelled the adapter task running the turn. A viewer's submit into a messaging session took the same path. execute_admission now records the claimed logical owner and generation on every ordinary turn of a chat route, native or a viewer's submit (internal completions keep their ownership proof; LOCAL and API turns carry nothing). The session resolver drops the turn when a Stop was latched for that claim, which /new and /reset send before they move the route, and the resolved session is not the owner or one of its compression continuations (the forward walk the claim's preflight uses). The turn settles interrupted, as the Stop would have, and never runs on the fresh conversation. A /resume, /branch or compression sends no Stop, so those turns still run where the route points, as on main. Refs #134889 Fixes #140548 (cherry picked from commit 3eae47f690a354b6e58b5121e786440742f49813) * fix(gateway): let a serial reader return once a busy burst input is held Signal and IRC await handle_message before they read their next input. A busy photo or queue-mode text was held for the burst window and the call returned only after the durable commit, so the burst's next input could arrive only once the burst had committed alone: two turns where main merged one, and the reader stalled about 0.38 s per busy input. An adapter now declares serial_receive_loop when it delivers a chat's next input only after handle_message returns and returning triggers no acknowledgement. For it, dispatch_busy_burst returns as soon as the input is held, as main returned once the input was merged into the pending slot. The burst still commits once and in order, and a failed commit is logged. Every other adapter keeps commit-before-return. The reader still never runs ahead of what it returned on: - a command first waits until every earlier input of that reader has committed (/stop, /new and /reset drop their own chat's held burst), so /restart cannot close admission under a held input and /new cannot miss one still committing; - a chat holds at most one uncommitted burst: the next one waits for that commit, as main's reader waited at its first commit. Flagged: Signal, IRC, email, Matrix, Mattermost, ntfy, Photon, SimpleX, WeCom, WhatsApp and Feishu. Buzz is left out: it saves its cursor only after the call returns. Refs #134889 Fixes #140538 (cherry picked from commit e72e05a1e6e415ff54426b205dcc6b6f408417cf) * fix(gateway): queue messaging slash-command turns behind the chat's waiting messages Under a session authority, plain text joins the chat's durable FIFO, but a slash command that becomes a turn skipped it: busy `/queue C`, the `/steer` fallback and mid-run skill commands went to the in-process pending slot, which the running turn drains inside itself, and idle rewrites (`/queue`, `/steer`, `/plan`, `/learn`, `/init`, `/blueprint`, skills, bundles) ran in-process. Either way they ran ahead of turns already admitted (A, C, B). Such a turn is now admitted like text once the command is resolved: inline on the idle path and the runner's busy path, through dispatch_shared_busy from a busy command, with the command's native or relay callback carried to the rebuilt event. Before its commit it is flagged `_fifo_only` so the busy preempt, the plaintext approval check and pending-prompt replies never read it, and its commit skips the ingress gates its command already passed. A /stop or /new after the command (a per-route epoch, captured before a slow skill rewrite) drops it before admission, as it clears the slot. Once admitted, no prompt reply (/update, clarify, slash-confirm) reads any executing admission. The `/queue` and `/steer` acks follow the row's commit, not the busy sender's release. Text that is still a command, executing admissions, internal events, profiles without an authority and turns carrying a one-turn model (`/moa`) keep the in-process path. Refs #134889 Fixes #140572 (cherry picked from commit e74f79c9382b38f7666466046848e2ac917c660e) * fix(gateway): release abandoned native media captures Refs #134889 Fixes #140520 (cherry picked from commit d689cb8411ff6649fb298b9dba867235bbe94d3f) [teknium: resolved a conflict in SessionAuthority.admit_native with the MSG-23 route lock (#140454): the capture lease and its abandon cleanup now run inside _route_turn.] * fix(gateway): make a Bot Chat follow its profile, whoever created it Desktop creates the Bot Chat with follow_profile_config, so each turn runs the profile's current model, endpoint and key. A Bot or cron delivery, a peer DM, a CLI titled create, and adoption of a pre-upgrade Bot Chat froze the route of that moment instead. Desktop then reuses the chat by title, and retiring the old key breaks it. Those producers now ask to follow the profile too. A safe-mode create stays frozen, and a creation an older build staged before the flag resumes as it was staged. Adopting a Bot Chat whose profile names no model keeps the saved model name on the receipt, so the chat can be opened and follows once the profile has a default. Chats that are not a Bot Chat are unchanged. A Bot Chat an earlier build already committed frozen follows from the first time a client opens it, also on a later open over the same connection when the first found a turn running. The resume refresh marks the policy as following inside its own config, so no other receipt changes shape and the creation request stays the exact-retry identity, and binds its private values again without the followed sections, so the frozen key never reaches the profile's current endpoint. A composer pick is held under adoption's rule, and a committed /reasoning stays the session's level. Safe mode, a request that chose false, and other titles stay frozen. A held pick ends once the named provider entry (or alias) its key comes from no longer serves the endpoint the pick was made on, so that entry's new key never reaches the old endpoint. A provider-only or endpoint-only pick on a new following session holds like a model pick. Fixes #140470 Refs #134889 (cherry picked from commit acd3e9420ab4d61b7d5147657c42b6fedaccb717) * fix(gateway): move an API-held Bot Chat title aside and mint the local chat A profile whose canonical "Bot Chat" title is held by a conversation no FIFO can own (an api_ row an older gateway or a plain POST /api/sessions minted, chat_id NULL) refused every delivery for good: - a peer DM or `hermes peer run` / POST /v1/runs got 502 bot_chat_not_local, "rename or delete it there". Neither is possible: `hermes sessions rename` refuses the canonical title by design, and the API server has no rename route (#140769); - a local bot_relay or cron delivery found the row by title, failed adoption (validate_legacy_row) and answered not_found (#140374). The Bot Chat door now moves such a row aside under the titled-create lock and mints the profile's local Bot Chat there: the old row keeps its id and history and becomes visible as "Bot Chat (API <id>)", which the user can find, rename or delete. A peer turn aimed at it is admitted under the peer's create grant (the same principal create_peer_bot_chat uses), so the owner mints the chat once and every later DM finds it by title. A bound local chat, and old CLI/TUI/Desktop history a native open adopts, are never moved; a submit-only principal (owner recovery) cannot create the replacement, so it never moves the title either. The refusal that remains names the remedy that now runs. Refs #134889 (MSG-21) Refs #140374 Refs #140769 Refs #140719 * refactor(gateway): move inbound voice transcription into its own mixin The lane's ordering fixes (MSG-23, MSG-27) grew gateway/run_inbound.py past its 2000-line cap. The STT enrichment and pending-voice echo methods move unchanged into gateway/run_inbound_transcription.py as GatewayInboundTranscriptionMixin. Refs #134889 (MSG-23, MSG-27) Refs #140719 * fix(gateway): keep a session's reasoning choice on a profile-following chat A session that follows its profile's config (every Bot Chat Desktop creates) acknowledged Desktop's reasoning selector and /reasoning <level>, then ran the profile's default level on the next turn: effective() kept the frozen level over the followed one only when the creation request named reasoning. A level sent with a model pick was lost the same way, and the resume refresh re-read the profile's level over it on any session. The policy's own config_json now records the session's choice, session_pins: {reasoning: true|false}. A session-scope level (the reasoning mutation, or a model pick that carries one) writes true; the new reset form of the same verb ({effort: null}) writes false and takes the profile's current level. effective() and the resume refresh both read it through session_pinned(), so a pick beats the profile's default and a reset beats the creation request and older pick receipts. The creation request stays the launch identity, and a new session's policy is unchanged. Every command that applies a level to a local session now goes through that owner verb: config.set reasoning in either scope (global still writes config.yaml) and slash.exec /reasoning, including reset, commit before the handler replies, so an owner restart keeps the latest choice. While a turn runs, the handler still applies the command for the next turn and the drain commits it at that turn's end. A messaging chat viewed from Desktop or Ink has no session policy and keeps the handler alone. The classic CLI sends an explicit level (/reasoning, a model pick's --reasoning) even when the session already runs it by inheritance; re-derived levels keep the old dedupe. Refs #140446 Refs #134889 (MP-16) Refs #140719 (cherry picked from commit 7121ec71787206261e4b64d953445aaac5859441) (cherry picked from commit 0b21fdca8b948c17553bb13aab42c7c690f7a227) * test(gateway): pin a later /reasoning on a following Bot Chat over the profile MSG-22 makes delivery-, peer- and cron-created Bot Chats follow their profile. Without a session pin, a /reasoning high committed after the follow was acknowledged and then lost to the profile's level on the next turn (red on 655aef56d89: 'low' != 'high', for a delivery-minted chat and for a frozen chat an open adopted). The fix is MP-16 (0b21fdca8b9, cherry-picked above from fix/pkg10-creds-a): prepare_reasoning and a model pick carrying reasoning now record session_pins.reasoning, which effective() reads through session_pinned(). This test pins that the two compose. Review finding by @kshitijk4poor on #140973 Refs #134889 (MSG-22) Refs #140719 * perf(gateway): parse a policy's follow flag once, not on every read follows_profile is read ~8-10 times a turn on the event loop (activate_policy, read_profile, current_toolsets, effective, run_turn and side agents). MSG-22 added a json.loads of config_json to it, and a frozen session's config_json is a whole profile config: ~435 us per read with a DEFAULT_CONFIG policy vs ~2 us on main. The policy is a frozen dataclass, so the result is cached per instance; replace() yields a fresh instance with its own cache. Test: test_follows_profile_parses_each_frozen_blob_once_per_policy (red: 10 parses for 10 reads; green: 1). Review finding by @kshitijk4poor on #140973 Refs #134889 (MSG-22) Refs #140719 * fix(gateway): keep an idle slash command's stop epoch from before its rewrite The idle path called _admit_command_turn without a stop_epoch, so the epoch was read after the off-loop skill rewrite. A /new or /stop landing during the rewrite bumped the epoch first, the old /skill turn passed the stale check, and its durable row committed into the fresh conversation. _hm_dispatch_idle_commands now reads the epoch before dispatch and passes it on, as _busy_skill_turn already did; stop_epoch is required. Test: test_reset_during_an_idle_skill_rewrite_drops_it[/new|/stop] (red: C admitted and run after the reset; green: dropped). Review finding by @kshitijk4poor on #140973 Review finding by @dokterdok on #140973 Refs #134889 (MSG-27) Refs #140719 * refactor(gateway): one per-route /stop and /new epoch, on the authority Two counters recorded the same event: the runner's _route_stop_epochs (never pruned) fenced slash-command turns and the authority's _route_turns[key][2] fenced inputs still capturing, under different conditions: the runner's was bumped even when a FIFO paused on a lost turn kept its followers, the authority's was skipped then. The authority now keeps one persistent per-route epoch (route_epoch / fence_route); _route_stop_epoch reads it and the runner dict is gone, so a command turn and a capturing input are fenced by the same /stop or /new, and a paused FIFO keeps both. The route-lock entry no longer carries a counter. Test: test_one_stop_epoch_per_route_fences_inputs_and_command_turns [existing|first-message|paused] (red before: no shared epoch; the paused case diverged). Review finding by @kshitijk4poor on #140973 Refs #134889 (MSG-23) Refs #134889 (MSG-27) Refs #140719 * test(gateway): MSG-23/MSG-26 timing bounds meet the 2 s floor test_a_held_capture_never_delays_another_chat_nor_outlives_its_cancel asserted completion within 1.0 s (twice); tests/AGENTS.md asks for >= 2 s. Both now wait up to 5 s through async_wait_until, and the held capture outlasts that bound (15 s, released in finally), so "done within the bound and chat X still capturing" is the ordering proof, not elapsed time. The serial-reader negative window (0.5 s) is 2 s. Review finding by @kshitijk4poor on #140973 Review finding by @dokterdok on #140973 Refs #134889 (MSG-23) Refs #140719 * chore(gateway): drop the _UNSET import the transcription move left behind 655aef56d89 moved the only _UNSET user (_transcribe_inbound_media's metadata default) into run_inbound_transcription.py, which imports it itself. run_inbound.py kept the import (ruff F401 on this head only; nothing imports _UNSET from run_inbound). Check: `ruff check --select F401 gateway/run_inbound.py` no longer reports _UNSET. Review finding by @kshitijk4poor on #140973 Refs #134889 (MSG-22) Refs #140719 * fix(gateway): an MCP reload keeps a Bot Chat's pending adoption After a busy first open left a frozen Bot Chat's adoption pending, reload.mcp's commit_refresh(mcp_only=True) discarded the pending entry on both exits, though an MCP-only refresh never runs follow_bot_chat. Warm reopens on that connection then returned early and the chat stayed on its old model, endpoint and key. An MCP-only refresh now leaves the pending set alone; only the adopt and full paths settle it. Kept the in-memory set rather than re-deriving it on every warm attach: that would run a profile read and follow_bot_chat off-loop on each Desktop session.activate, and the set is only a hint (a new connection already re-derives it from the stored policy). Test: test_an_mcp_reload_keeps_a_busy_first_opens_adoption_pending (red: still frozen after the warm reopen; green: follows the new route). Review finding by @kshitijk4poor on #140973 Review finding by @dokterdok on #140973 Refs #134889 (MSG-22) Refs #140719 * fix(gateway): name the local Bot Chat a moved-aside API turn landed in After MSG-21 moves an API-held "Bot Chat" title aside, deliver() lands the peer turn in the newly minted local chat, but the session-chat JSON and X-Hermes-Session-Id, the SSE headers and events, and the /v1/runs record (and its idempotency record) still named the requested, now "Bot Chat (API ...)" id. A client continuing by that id ran a plain API turn on the old row and split the conversation. Every answer now carries the delivery receipt's session_id; /v1/runs rewrites its record once admit_peer_turn returns, which also persists to the idempotency store. hermes peer dm's JSON reads that body field, so its report follows. Test: test_a_turn_after_the_move_aside_names_the_local_bot_chat [chat|stream|run] (red: all three named api_1_stale; green: local-...). Review finding by @kshitijk4poor on #140973 Review finding by @dokterdok on #140973 Refs #134889 (MSG-21) Refs #140769 Refs #140719 * fix(gateway): promise a move-aside only when the Bot Chat will be moved For a Bot Chat a peer may not adopt but that displaceable_bot_chat never moves (a Desktop row a served remote login stamped, another profile's row), the 502 said it "is moved aside ... on the next delivery". The next delivery was refused the same way. The hint is now chosen by displaceable_bot_chat: the move-aside promise for a row it would move, else "owned by another login or profile; open or rename it from the client that owns it". The reason code is unchanged. Test: test_a_protected_desktop_bot_chat_refusal_never_promises_a_move (red: "moved aside" in both refusals; green). Review finding by @dokterdok on #140973 Refs #134889 (MSG-21) Refs #140719 * fix(gateway): an open that adopts a frozen Bot Chat keeps its committed /model follow_bot_chat kept a legacy composer pick and a committed /reasoning, but not a /model the user committed while the chat was frozen: picks_on reported only reasoning and workspace, so adoption recorded no selection_profile, effective() read the pick as not held, and the next turn ran the profile's model although the profile never moved. On main the chat stayed frozen on the pick. picks_on now also reports committed model edits, and follow_bot_chat records selection_profile for one (when none is recorded), so it holds until the profile's model or provider moves, exactly like a composer pick. Test: test_a_model_committed_on_a_frozen_bot_chat_holds_when_an_open_adopts_it (red: 'original' after the open; green: 'picked' until the profile moves). Review finding by @dokterdok on #140973 Refs #134889 (MSG-22) Refs #140719 * fix(gateway): a busy burst's commit task ends when its row is durable On Signal and other serial readers: an idle video A with a slow capture, then queue-mode text B, then /restart. B's burst queued behind A's route reservation, and _route_turn resolved its acceptance future so the reader could go on (MSG-26). That also made dispatch_shared_busy return, so the burst's commit task, the thing /restart's barrier waits on (_shared_burst_commits), finished before B had a row. /restart closed admission, and releasing the capture refused both A and B with runtime_draining; B had no durable row to recover. "The reader may read on" and "the input is durably accepted" are now separate signals: admit_message resolves a busy_commit future once admit_native returns, and the burst's commit task calls dispatch_shared_busy(durable=True), which waits for it (or for the input to be refused, fenced or answered). The serial reader still returns as soon as the input is held (_accepted), so MSG-26 is unchanged; the chat's next burst, command barriers and /restart now wait for the commit, as main did. Test: test_restart_waits_for_a_held_text_queued_behind_a_slow_capture (red: no rows for V or B after /restart; green: both committed and run). Review finding by @dokterdok on #140973 Refs #134889 (MSG-23) Refs #134889 (MSG-26) Refs #140719 * refactor(gateway): move the voice helpers into the transcription mixin 655aef56d89 moved the transcription methods into GatewayInboundTranscriptionMixin but left _prepend_media_prefix (with its _EMPTY_TEXT_PLACEHOLDER), _untranscribed_audio_note and _transcribe_one_clip in run_inbound.py, though only the mixin calls them. They move unchanged; asyncio is imported for _transcribe_one_clip. No behaviour change: the voice/STT suites pass (14 files, 260 tests). Review finding by @kshitijk4poor on #140973 Refs #134889 (MSG-22) Refs #140719 * fix(gateway): label a moved-aside Bot Chat by its source, check it cheaply displaceable_bot_chat also moves a messaging row that took the canonical title, and _displace_bot_chat renamed every such row "Bot Chat (API <id>)". The new name now says where it came from: API for api_server/bot_peer_dm rows, else the platform ("Bot Chat (telegram <id>)"). The refusal hint names the same shape. It also runs on every delivery (twice per peer DM): a bound local chat, the common case, now answers from its own row before any compression-tip or session read. Tests: test_a_messaging_bot_chat_moved_aside_is_labelled_by_its_platform, test_a_local_bot_chat_is_never_displaceable_without_a_lineage_read (both red before; green). Review finding by @kshitijk4poor on #140973 Refs #134889 (MSG-21) Refs #140719 * refactor(gateway): one abandon-capture helper and one keyed FIFO lock The "abandon capture and release" block was pasted at admit_native, webhook_ingress._prepared and prepare_native's non-cancel path. The idempotent flag now lives in abandon_capture(scope), and all three call it. The cancel path keeps its deferred done-callback release. _route_turn and webhook_ingress._delivery_turn each re-implemented the ref-counted keyed asyncio.Lock. Both now use session_authority.keyed_turn(table, key, queued=...). The route's "position fixed" release runs in its queued hook, under the same condition as before (an earlier turn holds or awaits the key). Also: two hook tests stubbed the authority as a bare object(); _route_stop_epoch now reads the authority's route_epoch (b4c97d5a8b2), so the stubs carry that one method. No behaviour change: the MSG-23/24/26 suites, the webhook-ingress and capture-lease suites, and the pre/post gateway hook suites pass. Review finding by @kshitijk4poor on #140973 Refs #134889 (MSG-23) Refs #134889 (MSG-24) Refs #140719 * fix(desktop): align main model settings controls * refactor(gateway): keep api_server.py and _handle_runs under their health caps dd1a58b2a69 grew api_server.py past its main value (4611 lines) and _handle_runs to CC 21. The delivered-id read is now inline at its two call sites, and the /v1/runs retarget is its own helper (_retarget_to_delivered_chat). No behaviour change. Check: python3 scripts/check: 12 checks ok (health 0 blocking). Review finding by @kshitijk4poor on #140973 Refs #134889 (MSG-21) Refs #140719 * fix(gateway): a stamped local route keeps its conversation instead of an empty replacement A remote viewer of a Desktop session disconnected mid-turn, and the conversation was stuck for good (#140642). Three links, all reproduced on main through the real SessionStore, authority and SQLite: 1. The viewer's ws-orphan reaper (tui_gateway _finalize_session) stamped the row `ws_orphan_reap` while the owning gateway still ran its admitted turn. The stamp now shares one write txn with a check of the canonical ledger (session_admissions / worker_executions under the physical id, the key and the local logical id), and a busy ledger keeps the row open (agent.close() is told not to end it either). 2. The SessionStore's #54878 stale-route self-heal treated that stamped local row as stale, minted a replacement id and inserted its row, then the authority refused its peer record (`admission_conflict`). A local route the authority holds by receipt is never stale now: only the authority advances it, and its first admitted turn reopens a stamped row (reopen_local_session). 3. The restart prune repointed sessions.json at that empty, never-admitted row. For a receipt-held local route the prune now keeps the route, or follows the receipt's own target (a compression child), never a peer row. Installs already wedged heal on the next resume: restore_local_session drops a route entry naming an empty, never-admitted row outside the receipt's lineage, instead of answering `admission_conflict` forever. Tests: tests/gateway/test_local_route_orphan_reap_stamp.py (3) and tests/tui_gateway/test_reap_respects_canonical_ledger.py (5), red on main. Refs #140642 Refs #134889 (RT-43) Refs #140719 * fix(state): a canonical rewind refuses a target that is not a live user row session.mutate rewind took the client's target_message_id as is. Desktop lists rows that in-place compaction archived (include_compacted) with their row ids, so an edit, restore or regenerate above a compaction boundary named an archived row; the rewind then deactivated every active row with a larger id, which is the compaction summary and every later turn, leaving an empty live transcript. The target must now be an active user row of the physical session, as a branch boundary must be, or the rewind is refused with invalid_params naming target_message_id. A missing or non-user target, which used to raise past dispatch as an internal error, gets the same refusal. (cherry picked from commit 98f753c3b41c347f5ffedc0f017bd8d2d5cfd40c) * fix(gateway): show the whole history when resuming a compressed conversation After any compression, session.resume (Ink /resume, Desktop's warm re-focus and replay-gap re-snapshot, the one-shot chat view's --resume printout) returned the model's working set: the summary and the last exchange. Earlier turns looked deleted although they are stored. The resume snapshot is now the display history. local_history returns the display lineage (as the REST transcript and the legacy resume do), marks every row outside the current transcript's active set `archived: true`, and keeps every live row, so the rows without the mark are exactly the live transcript. Archived rows fill, newest first, what the live rows leave of a 3 MiB budget, so the reply stays under the clients' 8 MiB frame whenever main's live-only reply does; when older ones are left out the first row says `archived_truncated: true`. attach and the hosted-room history read and budget the lineage off the loop; under the event-stream lock only the live tip is re-read, and a rewind or compaction since the read triggers a re-read. Each logical message shows once: - the store's display dedupe also reads removed rows, and hides a message whose newest copy is removed, so a turn undone in a compression child does not come back from the parent's copy (copies share the display key or message_uid; a merge stands for the prompts it absorbed); - a micro-compaction merge (model_only) stays a live undo target but Ink's display_messages, Desktop hydration and the one-shot printer skip it: its originals are the archived rows they show. An archived row is display-only: - the canonical rewind refuses a target that is not an active row of the current transcript (before, it deactivated the whole live conversation, also reachable from a Desktop edit of an archived row); - the classic and Ink /undo and /retry selectors, the classic rewind_user_turn, /compress and /rollback history, and ACP's replay and token estimate skip it. Desktop's /undo and /retry and the owner's /rollback turn rewind follow in the next commit. Three tests that pinned the tip-only snapshot now assert the old rows are present and archived, and the live rows unchanged. Refs #134889 Refs #140449 (cherry picked from commit 81e4666af9d048e1df89c704a921ff55ee9b2ef6) * fix(gateway): rename and archive of a compressed local conversation authorize on its owner Session lists hand Desktop the listed row's physical id, which after a compression or reset is a child segment. `_owner_ref` remapped only `delete` to the lineage owner, so a rename, archive, pin or any other control mutation of that segment restored the CHILD as if it were an owner (`restore_local_session(child)`) and answered `storage_unavailable`. Reproduced warm and cold on main with a real authority, SQLite and a real two-step compression publish. Control and the session gate are now checked on the conversation's creation id for every operation. The write still names the listed row: only authorization moves, as delete's lineage write already did. Same class as LC-12 on the resume door (#140183). Test: tests/gateway/test_local_control_compression_child.py (warm + cold), red on main. Refs #134889 (LC-22) Refs #140719 * fix(gateway): a rewind by a compression segment's id keeps the owner's busy gate The LC-22 change authorized every control on the lineage owner while the write still targeted the listed segment. For rewind that skipped the owner's FIFO (the ledger keys admissions by owner id), its generation fence and its cached-agent eviction: a rewind by `child-2` applied with a turn queued on the owner, where the same rewind by the owner id answered `session_busy`. Owner authorization now covers only controls that leave the transcript alone (rename, archive, sidebar pin, delete). Rewind, reset, compress and policy picks keep the listed id, so a segment of a live owner is a named refusal again, as on main. The rename test now asserts the title lands on the listed segment, not anywhere in the lineage. Review finding by @kshitijk4poor on #140971 Refs #134889 (LC-22) Refs #140719 * fix(gateway): a resume's live rows keep stored order after a rotated compression `local_history` placed each live row at its first copy's display position, so after a rotated compression the carried tail came back before the new summary: `OLD_Q OLD_A TAIL_Q TAIL_A [SUMMARY]`. The CLI takes the live rows as its `conversation_history` (after `/compress` and `/rollback`) and as `/undo`'s prefix, which then lost the summary the store keeps; Ink's `display_messages`, built from the same rows, painted the summary after the tail. The merge now emits the live rows in `row_id` order and anchors each run of archived rows before the live row it preceded in display order, so every consumer (CLI, rollback, Ink, ACP) gets the stored sequence without a per-site sort. The tests compared `sorted(row_id)`, which hid this; they now compare the reply's own order and include the summary. Review finding by @kshitijk4poor on #140971 Review finding by @dokterdok on #140971 Refs #134889 (RT-39) Refs #140719 * fix(state): display reads take removed rows' identities, never their bodies RT-39's display reads dropped the active clause so the dedupe could hide a message whose newest copy was removed (rewound, or replaced by a newer copy). That fetched the full body of every `active=0, compacted=0` row (each in-place compaction leaves one per carried tail row) and discarded it; `assert_resume_safe` counts none of them, so nothing bounded it. The display set is read as before (`active OR compacted`), and the removed rows are read separately with only what the dedupe needs: role, timestamp, tool-call columns, uids and the stored `display_identity` (content only for a legacy row without one). The dedupe keys a removed row by its stored identity. `worker_history` (a worker's display read) now applies the same removed-copy rule, so the two readers agree. Review finding by @kshitijk4poor on #140971 Review finding by @dokterdok on #140971 Refs #134889 (RT-39) Refs #140719 * fix(gateway): resume's prepare window answers a delete and counts new rows `attach` and the hosted room's `_history` now await `prepare_history` (the lineage read, off the loop) before reading the live entry: - A delete committed in that window retired `authority.sessions[...]`, so the lookup raised `KeyError`, which `dispatch` does not catch: the viewer got an internal error. `attach` re-authorizes after the await; the room authorizes (restoring a cold session) before the prepare and again after. - A live row appended in that window passed the subset check and joined the reply without shrinking the archive, so the archive no longer fit what the live rows leave of `RESUME_HISTORY_BYTES` (a 2 MiB append gave a 9.9 MB reply with `display_projection`). The locked read now re-fits the prepared archive against the current live rows (no lineage re-read). Review finding by @kshitijk4poor on #140971 Review finding by @dokterdok on #140971 Refs #134889 (RT-39) Refs #140719 * fix(tui): a viewer's reap checks an adopted conversation's owner ledger `_end_unless_ledger_busy` added the row's `chat_id` to the ledger check only when it starts with `local-`. A compression segment of an adopted (pre-canonical) conversation carries the legacy owner id as `chat_id`, so a viewer's reap stamped that segment `ws_orphan_reap` mid-turn while the owner held a started or queued admission. The check now also adds `local_lineage_owner(conn, sid)`, which resolves both shapes through the receipt's lineage, as `_owner_ref` does. Review finding by @kshitijk4poor on #140971 Review finding by @dokterdok on #140971 Refs #134889 (RT-43) Refs #140719 * fix(desktop,gateway): never aim Desktop /undo, /retry or /rollback at an archived row The previous commit's resume snapshot also carries the rows a compression summarized away, marked `archived: true`. The owner refuses a rewind to one, so this commit is not needed for safety. It keeps two selectors choosing among live rows only, as Ink and the classic CLI already do: - Desktop's typed /undo and /retry (`isUserTurn` in slash-session-commands.ts): asked to go past the live turns, they said "error: invalid_params" instead of nothing to undo or retry. - The owner's /rollback turn rewind (`_rewind_last_turn` in gateway/session_tui_ops.py): with no live user turn left (the last one undone after a compression) it aimed the rewind at an archived turn, so a whole-tree restore that had already restored the files reported an error. Overlaps #140382 by file only: it also edits slash-session-commands.ts (the busy refusal text) and session_tui_ops.py (session.history), in other hunks. Neither change covers this, and the two merge cleanly. Refs #134889 Refs #140449 (cherry picked from commit 84527993e213d757f0d0297e97608ab2f4b7c473) * fix(desktop): skipping a model-only merge keeps later rows' ids `toChatMessages` filtered the micro-compaction merge rows out before its `forEach`, so every later message and tool-part id (keyed by the row's index) shifted by one against the stored position. The merge is now skipped in place. The classic CLI's resume print reads the shared `MODEL_ONLY_DISPLAY_METADATA_KEY` instead of an inline "model_only". Review finding by @kshitijk4poor on #140971 Refs #134889 (RT-39) Refs #140719 * refactor(gateway): resolve a mutation's lineage owner once `_owned_mutation` and `_authorize_control` each ran `_owner_ref`, two blocking reads that could disagree if a compression landed between them. The owner is resolved once in `_owned_mutation` and passed down. No behaviour change; the owner-auth scope fix is the earlier commit. Review finding by @kshitijk4poor on #140971 Refs #134889 (LC-22) Refs #140719 * fix(desktop): keep toChatMessages' callback under its complexity cap The in-place skip of a model-only merge added a branch to the callback (CC 80 against its main value 79). The painted rows are now paired with their stored index before the loop, which keeps the ids stable without a branch inside it. Review finding by @kshitijk4poor on #140971 Refs #134889 (RT-39) Refs #140719 * refactor(state): move get_latest_todo_result into hermes_state_todo hermes_state_messages.py has to stay under its 2,000-line cap together with the other package 10 lanes (with #140968 it came to 2,008). The latest-Todo-result search is one self-contained method with no callers inside the module, so it moves unchanged into a `SessionTodoResultMixin` in `hermes_state_todo.py`, which `SessionMessagesMixin` inherits (as it does `SessionDetachedResultsMixin`). Callers reach it as `SessionDB.get_latest_todo_result`, so no import path changes. No behaviour change. This branch's net growth in the file goes from +42 to -30 against main. Refs #134889 (RT-39) Refs #140719 * fix(gateway): don't stall relay on a parked profile; retry its other ingress A relay frame routed to a parked profile was refused runtime_draining with its ACK withheld. The park lasts until a rescan or a restart, and the connector drains its per-instance buffer in order, ack-gated, so one such frame held back every later buffered frame for every profile on that instance. A live frame has no redelivery, so it was lost silently. Producer admission now raises ParkedProfileRefusal for a parked target. Relay acknowledges the frame and tells the sender in-channel to resend, as a direct bot does, also when the park kept the profile's retiring authority (a turn outlived its Stop). Transient refusals still withhold the ACK. The notice is marked interim, so it never seals another turn's open stream in the same chat (relay Slack) or fills a pending answer slot (LINE). The rest of an unserved profile's ingress is now retryable too: - a coalesced webhook that profile_routes sends to a parked profile is answered 503 before its delivery id is recorded, not 202 and dropped; - a /p/<profile>/ webhook prefix with no serving authority in the live registry gets the same 503, whatever keeps it from being served; - the API doors under a parked /p/<profile>/ answer 503 runtime_draining with Retry-After instead of 409 profile_mismatch, and shared-listener callbacks answer 503 instead of 404. Once a hot-added profile's bots connect, queued native rows are bound in every serving ledger, not only that profile's: a bot_profile route runs a bot's messages on another profile. Fixes #140564 Refs #134889 (cherry picked from commit a88ea4e2de26fbc6b09bdd826b7b7a8972ef6d43) * fix(gateway): a confirmed downgrade stop exits with the service-restart code The downgrade watcher stops on a proven, post-update-lock move to a pre-runtime tree, and its docstring promises "a service manager that relaunches it starts the older release's own gateway". Under systemd (Restart=always) that holds on any exit. Under the shipped launchd plist it does not: KeepAlive SuccessfulExit=false parks exit 0, so the watcher's plain-success exit 0 left the gateway down until a human noticed — production receipt: an agent session checked out a pre-runtime branch under a live gateway on 2026-10-17 and the service stayed down ~2.5h until a manual start. A confirmed downgrade past a free update lock means the tree on disk is the operator's settled choice. Set the service-restart exit code before the plain stop: the supervisor relaunches and the settled tree serves. The stop itself stays a plain stop (no restart machinery, no restart kwargs) — the relaunch runs whatever release is on disk, never this process. The stop-failure hard exit(1) is unchanged (non-zero relaunches too). Deliberately flips the pinned no-exit-75 assertion in test_runtime_downgrade_exit.py to pin the new contract. (cherry picked from commit cfdea701cf9a95689099f6037f2f3c045945ce2f) * fix(gateway): say a downgrade stop relaunches; pin the exit code end to end On top of the cherry-picked exit-75 change: the module docstring and the logged reason still said the stop never asks for a restart, which now contradicts the exit code. Both now say the service manager starts the older release's own gateway. A new test drives the bootstrap's exit verdict and checks the process exits 75, so launchd's KeepAlive.SuccessfulExit=false relaunches it. Refs #134889 (GW-14) Refs #140418 Refs #140719 * fix(gateway): count /bg, /btw and preview-restart side agents as owner work A side agent runs on a detached thread that only a module-private set in gateway/session_tui_side.py knew about. Every owner work reader goes through uncounted_runtime_work (restart wait, stop drain, SessionDB close gate, idle exit) or stop/join_authority_work (profile retirement), so a running side agent counted as zero: a planned restart (hermes update, service restart, SIGUSR1) or idle exit proceeded within seconds, killed it, and the client never got background.complete / btw.complete. _spawn now registers each side agent in an authority-owned ledger before thread.start() and removes it in a finally around the whole thread target, re-publishing the work count on the owner loop, so a thread that dies on a BaseException (cancellation, exit) is never left counted. uncounted_runtime_work counts the ledger; stop_authority_work interrupts the built agent (/bg, preview restart) and join_authority_work waits for the thread. Restart timeouts still cap the wait, and new side agents are still refused with runtime_draining once a drain starts. Refs #134889 Fixes #140375 (cherry picked from commit 098ba77d706296945a1d3a0b0ee6c0963fa070b4) * fix(gateway): ignore secondary local adapters for idle exit Refs #134889 Fixes #140509 (cherry picked from commit 29e3d195564ef67427b9466f20db666e98b7cf2a) * fix(gateway): compare a managed worker's birth by its kernel start on macOS On macOS psutil shifts create_time() by abs(INIT_BOOT_TIME - boot_time()), using a boot-time reference frozen when psutil was imported in the calling process. After a clock step (NTP, sleep/wake) a long-lived gateway owner reads a different birth for a fresh worker than the worker reports itself, so verify_worker_identity's exact match refused every managed worker with permission_denied until a gateway restart. hermes_cli.process_identity.process_birth reads the unadjusted kernel start (create_time(monotonic=True)) on macOS and plain create_time() elsewhere (Linux and Windows apply no per-process reference). Every site that writes or checks a worker birth now uses it: the worker's hello, the reservation, the worker.adopt claim, the delegation dispatch check, the verified-interpreter signal, and the Kanban owner binding with its reclaim check. The match stays exact, so a forged birth is still refused. #140612 undoes the shift arithmetically (birth - diff). psutil applies abs(diff), so for a backward step that doubles the error instead of removing it, and the other birth sites stay exact on the drifting value. The new test emulates both step directions on every platform. Refs #134889 (KB-6) Refs #140580 Refs #140719 * fix(kanban): wait for kanban.run's admission answer past the RPC default The owner answers kanban.run only after it checks the claim and writes the owner_admitted marker in one BEGIN IMMEDIATE on the board, the cross-profile dispatch bus with a 120 s busy timeout. The submitter waited the 30 s RPC default. Behind another board writer it exited "Gateway connection failed" while the owner went on to admit the attempt; the later-bound worker's submitter-liveness watcher then ended it, leaving the admission unknown. The submitter now waits without a deadline (math.inf, the same knob /compress uses). A hang-up still fails the wait and nothing is resent. The dispatcher's runtime cap and claim reclaim still bound a stuck submitter, since they kill its pid. Probe: a write_txn on a board held by another writer waits out the writer (2.5 s here, up to the 120 s default busy timeout). Refs #134889 (KB-7) Refs #140719 * fix(logging): never take logging's module lock under a profile router lock test_runtime_listener_supervision hung on CI: uvicorn's dictConfig (build_listener_server) holds logging._lock and, in logging.shutdown, acquires each handler's lock. Meanwhile the QueueListener thread was inside _ProfileRoutingFileHandler.handle, which held the router's I/O lock across emit. Routing a record to a not-yet-opened profile home then built that home's file handler, and Handler.__init__ needs logging._lock. Each thread waited on the other's lock. The router now overrides handle() without its own I/O lock (each per-home handler takes its own lock to write; routing state keeps _profile_handlers_lock). It also builds a per-home handler with no router lock held, then publishes it with setdefault and closes a racing duplicate. The order is fixed rather than the timeout. Test: with logging._lock held as dictConfig holds it, a routed record that opens a new home's handler must leave the router's lock acquirable (red on main: the listener held it while blocked on the module lock). Refs #134889 (T23) Refs #140719 * fix(gateway): let profile retirement stop a running /btw GW-15 counts /btw as owner work, but its _spawn passed no holder, so stop_authority_work could not reach its fork while join_authority_work waited for it. A slow side question (the one-shot fallback allows 180 s) held retirement for the whole settle window, then the profile kept its reservation ("workers did not stop; ownership retained"). answer_side_question now publishes its cache-parity fork through on_fork, and /btw passes a holder (_SideQuestionStop) that interrupts it. Once stopped, the one-shot fallback is skipped; one already in flight is an uncancellable, read-only aux call, so a stopped /btw with no fork left stops counting as owner work and publishes no late answer. Review finding by @kshitijk4poor on #140967 Refs #134889 (GW-15) Refs #140719 * fix(webhook): don't 503 a coalesced delivery to a served /p/<profile>/ _coalesced_to_parked matched profile_routes on a hand-built webhook:<route> user and ignored the request's explicit profile, so a route reachable both bare and as /p/default/ with a profile_routes rule sending that user to a parked beta answered the /p/default/ delivery 503 and the provider retried it forever. An explicit /p/<profile>/ now skips the check (its own prefix check already ran), and the bare route resolves through the admission's own resolver (GatewayRunner._profile_name_for_source, gated on multiplex_profiles) instead of calling match_profile_route directly. A parked target has left the served set, so the resolver rejects it; that rejection's profile is what admission retries (_rejected_route_profile). Review finding by @kshitijk4poor on #140967 Refs #134889 (GW-17) Refs #140719 * fix(kanban): read process start fingerprints from the kernel birth on macOS KB-6 moved worker-birth sites to process_birth, but Kanban's submitter fingerprint (_process_fingerprint -> gateway.status get_process_start_time) still used psutil's adjusted create_time() on macOS (no /proc). After a clock step the embedded dispatcher and a fresh worker fingerprinted the same live submitter differently, and _watch_submitter exit(1)'d one step after admission. The same drift fed the delegation owner_started_at and the spawn ledger's _pid_alive_matches / _same_incarnation / _process_create_time. All of those now read process_birth: the kernel start on macOS, plain create_time() elsewhere. With no step (|drift| < 1 s) psutil returns the kernel start unchanged, so stored fingerprints and ledger rows keep matching; Linux keeps /proc ticks first. Tests: the KB-6 test no longer fakes macOS (psutil.MACOS and the platform create_time). It runs on the macOS lane (platforms("macos")) and moves only psutil's frozen boot-time reference, which is what a clock step makes stale; a new macOS test pins the Kanban fingerprint, the start time and both _pid_alive_matches modes across a +/-5 s step, and a host-independent test pins backward compatibility. Review finding by @kshitijk4poor on #140967 Review finding by @dokterdok on #140967 Refs #134889 (KB-6) Refs #140580 Refs #140719 * fix(kanban): bound kanban.run's admission wait by the board lock budget KB-7 waited for kanban.run's answer with no deadline, and the comment said the dispatcher's runtime cap and reclaim bound a stuck submitter. They don't once the owner_admitted marker commits: enforce_max_runtime and TTL reclaim both skip the run while owner_reclaim_paused is true, so an owner that stalled after admitting left the submitter waiting forever. The submitter now waits admission_answer_budget(): the longest the owner's admission write can wait on the board lock (write_txn retries BEGIN IMMEDIATE _BUSY_MAX_RETRIES times, each under the busy timeout; 6 x 120 s by default) plus a 60 s margin. That is still past the 30 s RPC default KB-7 fixed. A hang-up fails it sooner, as before. HERMES_KANBAN_BUSY_TIMEOUT_MS is kept in the submitter's env (a host tuning knob, not task identity) so the budget follows a tuned board. The comment now says what bounds the wait. Review finding by @kshitijk4poor on #140967 Refs #134889 (KB-7) Refs #140719 * test(gateway): give the parked relay test's redelivery wait a 10 s floor The connector stand-in resent an unacknowledged head after 1 s, then the test required exactly one b1 send. On a loaded runner the ack took longer, and it failed with ['b1','b1','b2','b2'] (dokterdok: 5 of 5 under load). tests/AGENTS.md asks for waits of at least 2 s. The wait is now 10 s. A withheld ack still never arrives, so b2 is never sent and the 10 s admission wait fails: the test still proves the frame is acknowledged. Review finding by @dokterdok on #140967 Review finding by @kshitijk4poor on #140967 Refs #134889 (GW-17) Refs #140719 * fix(logging): a closed profile router never reopens a per-home log Dropping the …

dokterdok mentioned this pull request Oct 17, 2026

Messaging: a text sent right after a photo or video runs first, without the media #140453

Closed

alt-glitch added comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists type/bug Something isn't working labels Oct 17, 2026

teknium1 mentioned this pull request Oct 18, 2026

fix(gateway): keep chat input in FIFO order and land Bot Chat deliveries whoever created the chat (package 10, lane 5) #140973

Merged

dokterdok closed this Oct 18, 2026

Navigation Menu

Repository navigation

fix(gateway): keep a chat's media and quick text in arrival order - #140454

fix(gateway): keep a chat's media and quick text in arrival order#140454
dokterdok wants to merge 1 commit into
NousResearch:mainfrom
dokterdok:fix/messaging-media-text-order

dokterdok commented Oct 17, 2026

Uh oh!

dokterdok commented Oct 18, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

dokterdok commented Oct 17, 2026

What does this PR do?

Related Issue

Type of Change

Changes Made

How to Test

Checklist

Code

Documentation & Housekeeping

Uh oh!

dokterdok commented Oct 18, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants