Skip to content

fix(gateway): write restart markers atomically and avoid Windows runtime-lock read failures - #17842

Closed
johnncenae wants to merge 1 commit into
NousResearch:mainfrom
johnncenae:fix/gateway-atomic-restart-markers
Closed

fix(gateway): write restart markers atomically and avoid Windows runtime-lock read failures#17842
johnncenae wants to merge 1 commit into
NousResearch:mainfrom
johnncenae:fix/gateway-atomic-restart-markers

Conversation

@johnncenae

@johnncenae johnncenae commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

Summary

This hardens the gateway restart/control-plane persistence paths and fixes a Windows-specific runtime lock issue.

Changes:

  • switch restart marker writes to atomic_json_write()
    • .restart_failure_counts
    • .restart_notify.json
    • .restart_last_processed.json
  • switch gateway.status._write_json_file() to atomic JSON writes so runtime status, scoped locks, and takeover markers inherit the same safety
  • move the Windows byte-range lock off byte 0 to a high offset (1 MiB) so readers can still read the JSON metadata while another process holds the runtime lock
  • add regression coverage for the atomic write paths and the Windows lock offset behavior

Why

The gateway currently relies on small JSON control-plane files for restart recovery, dedup, and lock/state coordination. Writing those files in-place can leave partial JSON behind on interrupted writes, which degrades recovery behavior.

On Windows, locking byte 0 of gateway.lock caused mandatory-lock read failures when get_running_pid() or related readers tried to inspect the lock record. Locking a byte well past the JSON payload preserves mutual exclusion without blocking metadata reads.

Tests

Verified on Windows with:

uv run pytest tests/gateway/test_restart_notification.py tests/gateway/test_restart_resume_pending.py tests/gateway/test_status.py
----- 101 passed

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery labels Apr 30, 2026
@johnncenae johnncenae changed the title fix(gateway): write restart markers atomically and fix Windows lock c… fix(gateway): write restart markers atomically and avoid Windows runtime-lock read failures Apr 30, 2026
teknium1 added a commit that referenced this pull request May 1, 2026
Widen PR #17842's atomic-write fix to two sibling sites that exhibit the
same 'partial JSON on interrupted write' class of bug:

- gateway/platforms/feishu.py: dedup state (_dedup_state_path)
- gateway/platforms/helpers.py: ParticipatedThreadTracker save

Both are small recovery/coordination files that get rewritten frequently and
break cross-restart dedup if left partial.
@teknium1

teknium1 commented May 1, 2026

Copy link
Copy Markdown
Contributor

Merged via #18179 — your commit cherry-picked onto current main with rebase-merge so your authorship is preserved. Added a follow-up commit from me widening the same atomic-write fix to two sibling sites (feishu dedup state + ParticipatedThreadTracker) that had the identical partial-write bug class. Thanks for the fix — the Windows lock offset catch in particular was sharp.

nickdlkk pushed a commit to nickdlkk/hermes-agent that referenced this pull request May 11, 2026
Widen PR NousResearch#17842's atomic-write fix to two sibling sites that exhibit the
same 'partial JSON on interrupted write' class of bug:

- gateway/platforms/feishu.py: dedup state (_dedup_state_path)
- gateway/platforms/helpers.py: ParticipatedThreadTracker save

Both are small recovery/coordination files that get rewritten frequently and
break cross-restart dedup if left partial.
jsboige pushed a commit to jsboige/hermes-agent that referenced this pull request May 14, 2026
Widen PR NousResearch#17842's atomic-write fix to two sibling sites that exhibit the
same 'partial JSON on interrupted write' class of bug:

- gateway/platforms/feishu.py: dedup state (_dedup_state_path)
- gateway/platforms/helpers.py: ParticipatedThreadTracker save

Both are small recovery/coordination files that get rewritten frequently and
break cross-restart dedup if left partial.
dannyJ848 pushed a commit to dannyJ848/hermes-agent that referenced this pull request May 17, 2026
Widen PR NousResearch#17842's atomic-write fix to two sibling sites that exhibit the
same 'partial JSON on interrupted write' class of bug:

- gateway/platforms/feishu.py: dedup state (_dedup_state_path)
- gateway/platforms/helpers.py: ParticipatedThreadTracker save

Both are small recovery/coordination files that get rewritten frequently and
break cross-restart dedup if left partial.
gweeteve pushed a commit to gweeteve/hermes-agent that referenced this pull request Jun 2, 2026
Widen PR NousResearch#17842's atomic-write fix to two sibling sites that exhibit the
same 'partial JSON on interrupted write' class of bug:

- gateway/platforms/feishu.py: dedup state (_dedup_state_path)
- gateway/platforms/helpers.py: ParticipatedThreadTracker save

Both are small recovery/coordination files that get rewritten frequently and
break cross-restart dedup if left partial.
Seven74AI pushed a commit to Seven74AI/hermes-agent that referenced this pull request Jun 13, 2026
Widen PR NousResearch#17842's atomic-write fix to two sibling sites that exhibit the
same 'partial JSON on interrupted write' class of bug:

- gateway/platforms/feishu.py: dedup state (_dedup_state_path)
- gateway/platforms/helpers.py: ParticipatedThreadTracker save

Both are small recovery/coordination files that get rewritten frequently and
break cross-restart dedup if left partial.
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
Widen PR NousResearch#17842's atomic-write fix to two sibling sites that exhibit the
same 'partial JSON on interrupted write' class of bug:

- gateway/platforms/feishu.py: dedup state (_dedup_state_path)
- gateway/platforms/helpers.py: ParticipatedThreadTracker save

Both are small recovery/coordination files that get rewritten frequently and
break cross-restart dedup if left partial.
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
Widen PR NousResearch#17842's atomic-write fix to two sibling sites that exhibit the
same 'partial JSON on interrupted write' class of bug:

- gateway/platforms/feishu.py: dedup state (_dedup_state_path)
- gateway/platforms/helpers.py: ParticipatedThreadTracker save

Both are small recovery/coordination files that get rewritten frequently and
break cross-restart dedup if left partial.
melon-xf added a commit to melon-xf/hermes-agent that referenced this pull request Sep 3, 2026
Widen PR NousResearch#17842's atomic-write fix to two sibling sites that exhibit the
same 'partial JSON on interrupted write' class of bug:

- gateway/platforms/feishu.py: dedup state (_dedup_state_path)
- gateway/platforms/helpers.py: ParticipatedThreadTracker save

Both are small recovery/coordination files that get rewritten frequently and
break cross-restart dedup if left partial.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants