Skip to content

fix: prevent concurrent gateway updates from clobbering shared IPC state - #15539

Open
hharry11 wants to merge 1 commit into
NousResearch:mainfrom
hharry11:fix/gateway-update-concurrency-guard
Open

hharry11 wants to merge 1 commit into
NousResearch:mainfrom
hharry11:fix/gateway-update-concurrency-guard

Conversation

@hharry11

Copy link
Copy Markdown

Summary

Prevent duplicate gateway /update requests from starting a second detached update process while another update is still active.

The gateway update flow uses profile-global IPC files:

  • .update_pending.json
  • .update_pending.claimed.json
  • .update_output.txt
  • .update_exit_code
  • .update_prompt.json
  • .update_response

Before this change, a second /update call could overwrite the active update metadata and spawn another detached update process against the same shared IPC files. That could misroute progress/prompts/final notifications and clobber the original run state.

What changed

  • Add an early guard in GatewayRunner._handle_update_command()
  • If either .update_pending.json or .update_pending.claimed.json already exists, do not spawn another update process
  • Reuse the existing watcher so the current update continues streaming in its original chat
  • Return a user-facing message that an update is already running

Why this is a bug fix

The update IPC design is single-active-run per profile. The watcher and hermes update --gateway prompt bridge both assume a single shared set of marker files. The handler was missing the admission guard that enforces that invariant.

Tests

Added regression coverage for both active-marker states:

  • test_rejects_duplicate_update_when_pending_marker_exists
  • test_rejects_duplicate_update_when_claimed_marker_exists

Local/targeted result:

  • tests/gateway/test_update_command.py: 28 passed

Risk

Low.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery labels Apr 25, 2026
SGuibord pushed a commit to SGuibord/hermes-agent that referenced this pull request May 7, 2026
All tool_calls synthesis removed. The patch now only strips JSON-wrapped
text responses (content/text/message/response/answer keys + action:"text")
for display in Discord. It never fabricates tool_calls.

Root cause of ReAct output: Ollama bug NousResearch#15539 — system prompt + think:false
+ tools parser fails for gemma4, causing tool_calls to leak as plain JSON
text. Fix is at the model level (switching hermes-primary to phi4:14b which
has stable native tool calling in Ollama). Tool dispatch is Hermes' job;
this patch must not interfere with it.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for identifying the shared-IPC collision. Current main still has the underlying overwrite path, but the submitted guard cannot make concurrent admission safe.

Problems

  • gateway/run.py:7829 checks for a marker before the later write at PR lines 7843-7845. Two simultaneous handlers can both pass that check and both spawn an update process, so the reported race remains.
  • The handler moved to gateway/slash_commands.py in 619bd7827; current main still overwrites .update_pending.json at gateway/slash_commands.py:4531-4549.
  • The new tests at PR tests/gateway/test_update_command.py:218 and :257 pre-seed a marker, so they do not exercise the concurrent check-to-create gap.

Suggested changes

  • Port the change to gateway/slash_commands.py and atomically reserve the profile-wide update state before writing routing metadata or spawning the updater.
  • Add a synchronized two-caller regression proving only one updater launches and the winner's metadata remains intact.

Automated hermes-sweeper review.

Comment thread gateway/run.py
exit_code_path = _hermes_home / ".update_exit_code"
session_key = self._session_key_for_source(event.source)

if pending_path.exists() or claimed_path.exists():

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a TOCTOU check: two handlers can both observe no marker before either reaches the later replace(pending_path), then both launch updates. Reserve the profile-wide update state with an exclusive atomic operation and test two interleaving callers.

@teknium1 teknium1 added sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 12, 2026
briandevans added a commit to briandevans/hermes-agent that referenced this pull request Aug 14, 2026
`/update` wrote `.update_pending.json` and spawned a detached
`hermes update --gateway` with no admission control of any kind. Two
concurrent invocations — a user double-tapping because the update runs
for minutes with no acknowledgement, or two platforms on one multiplexed
gateway both triggering it — each wrote the marker and each spawned an
updater against the same checkout and virtualenv. The second write also
replaced the first requester's routing metadata, so that user never
learned their update finished. Both handlers also stage through the same
`.update_pending.tmp` path, so the losing rename raises FileNotFoundError
out of the handler.

Reserve the profile-wide slot with os.open(O_CREAT | O_EXCL) before any
routing metadata is written and before the updater is spawned. That
collapses observe-and-claim into a single atomic syscall, so exactly one
caller can ever win. The losing caller gets an "update already running"
reply and still gets `_schedule_update_notification_watch()`, so it
learns the outcome of the update that is actually running.

The reservation carries a TTL: an updater killed before it can write its
exit code (host reboot, OOM kill) would otherwise wedge /update for the
lifetime of the profile. Any failure between the claim and the spawn
releases the slot immediately so the user can retry.

Supersedes NousResearch#15539, which identified this collision. That guard patched
`gateway/run.py`, where the handler no longer lives after 619bd78, and
used a check-then-write `if pending_path.exists(): return` that both
callers can pass; its tests pre-seeded the marker, so they never
exercised the check-to-create gap.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants