fix(utils): recover Windows renames contended by an open handle

[OutThisLife]

What does this PR do?

Consolidates five open PRs that all fix one bug: on Windows, atomic_replace() only falls back for EXDEV/EBUSY, so a rename onto a file that any other handle holds open raises and the write is lost. CPython opens files without FILE_SHARE_DELETE, so an ordinary reader is enough.

Two reported symptoms, same root cause:

• gateway/run.py::_persist_active_agents rewrites gateway_state.json at every turn boundary while gateway/status.py readers poll it — the exception is swallowed, status updates are dropped silently, and .tmp files accumulate.
• hermes_cli/auth.py::_save_auth_store lets it propagate, so the desktop app shows agent init failed: [WinError 5] Access is denied and the session never opens.

The fix classifies winerror 5/32/33 as contention candidates, retries the rename with jittered backoff (a retry that wins keeps the write fully atomic), and only rewrites in place when the handle outlives the budget.

Two things the cluster got wrong

1. The error code is 5, not 32. Every PR in this cluster that gated on ERROR_SHARING_VIOLATION (32) keyed on a code the bug does not produce. Measured on Windows 11 build 26200 / CPython 3.11:

TARGET open for read (in-process)     winerror=5     <- the actual bug
TARGET open by ANOTHER PROCESS        winerror=5     <- gateway_state.json
TARGET open for append/write          winerror=5
SOURCE (tmp) open for read            winerror=32    <- 32 needs the SOURCE held

ERROR_SHARING_VIOLATION is what a held source reports. A held target — the entire bug — reports ERROR_ACCESS_DENIED. @kevin182 reported exactly this on #57777.

2. You cannot classify denial up front. The natural narrowing is to probe the target with os.access and treat "writable" as contention. It does not work — os.replace needs delete-child rights on the parent directory, so a directory-level denial reports the target as writable:

parent dir ACL: deny delete-child+write   os.access(tgt,W)=True   -> misclassified as transient
target is a directory                     os.access(tgt,W)=True   -> misclassified as transient

So this PR does not guess. Contention and genuine denial take the same bounded path; a real denial exhausts the budget, fails the in-place rewrite too, and is re-raised unchanged with its pending temp file intact. A permissions bug stays diagnosable instead of being converted into a silent success.

Why in-place rewrite instead of shutil.copyfile

copyfile truncates the target to zero before refilling it. With a 4-thread poller during an 8 MB copy, readers observe a 0-byte file — for auth.json that is an empty credential store, i.e. this fallback can cause the very failure it is meant to prevent.

Writing through the existing file (single os.write + ftruncate) removes that window and, as a bonus, preserves the target's ACL — which os.replace does not, since the temp file's inherited ACL wins. Verified: no 0-byte read observed, ACL preserved. EXDEV/EBUSY keep the original copy fallback; they are a different failure and never clear on retry.

Budget sizing

Measured, not guessed. Sized so realistic holds stay atomic without punishing a permanently-held target (gateway_state.json pays this on every turn):

budget	held 100ms	held 200ms	persistent reader (100 writes)
6 × 20..400ms	atomic	atomic	129.5s (1295ms/write)
4 × 20..100ms	atomic	atomic	31.5s (315ms/write)
3 × 20..50ms	atomic	rewrite	15.0s

4 × 20..100ms keeps identical atomic coverage at ~4.4× better throughput under a persistent reader.

Related Issue

Fixes #57775

Supersedes #57777, #79793, #73807, #45022, #36921

Credit to every author in the cluster — this keeps @lEWFkRAD's retry-then-fallback architecture (the only shape that handles both transient and persistent readers), @ruochu88s's {5,32,33} winerror set, and @lost9999's jittered_backoff reuse. Also to @kevin182, whose same-process thread-collision analysis and WinError 5 evidence on #57777 is what identified the misclassification.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

• utils.py: _WINDOWS_CONTENDED_REPLACE_ERRORS ({5,32,33}), _is_contended_windows_replace_error(), jittered bounded retry, _rewrite_in_place() (no truncate window, ACL-preserving), _copy_fallback() extracted unchanged for EXDEV/EBUSY. A retry that returns EXDEV/EBUSY stops consuming the budget and switches to the copy fallback. jittered_backoff is imported lazily to keep utils free of a package-level agent dependency.
• tests/test_atomic_replace_symlinks.py: 12 tests. Cross-platform ones simulate winerror 5 and pin the state machine; the four real-handle tests are @pytest.mark.windows_only.

How to Test

scripts/run_tests.sh tests/test_atomic_replace_symlinks.py -q     # 20 passed, 4 windows_only skipped
python -m pytest tests/test_atomic_replace_symlinks.py -m windows_only -q   # on Windows: 4 passed

Manual repro (Windows) — raises PermissionError before, succeeds after:

import utils
utils.atomic_json_write("gateway_state.json", {"active_agents": 1})
with open("gateway_state.json") as reader:
    utils.atomic_json_write("gateway_state.json", {"active_agents": 2})

A note on CI coverage

The three Windows tests on #57777 gated with pytest.skip(os.name != "nt") and carried no marker. scripts/ci/list_os_marked_tests.py greps for the marker name to choose which files the Windows lane imports, so those tests ran on no host at all — the suite was green over zero real coverage, exactly the trap AGENTS.md documents. Fixed here with @pytest.mark.windows_only; confirmed the lane now selects the file.

Verified RED/GREEN on real Windows (11 build 26200, Python 3.11.15) — with utils.py reverted to main:

FAILED test_windows_real_held_read_handle_lands_the_write
FAILED test_windows_real_held_handle_reports_access_denied
FAILED test_windows_atomic_json_write_with_concurrent_reader
E  PermissionError: [WinError 5] Access is denied
3 failed, 1 passed

and with the fix applied: 4 passed. One of those tests pins the winerror-5 premise itself, so if CPython ever changes the code, it surfaces here rather than silently reintroducing the bug.

Also verified on real Windows: 200 consecutive writes against a permanently-held reader — 0 errors, no temp accumulation; symlinked targets still survive the contended path (#16743).

Checklist

Code

• I've read the Contributing Guide
• Conventional Commits (fix(utils):, test(utils):)
• Searched for existing PRs — found and superseded four
• Only changes related to this fix
• Tests pass — 20 passed / 4 skipped locally; 4 passed on native Windows. One pre-existing failure (test_atomic_roundtrip_yaml_save_restores_owner, ModuleNotFoundError: ruamel) reproduces identically on clean main. Ran all 27 test files touching the atomic helpers: 20 failures on this branch vs 21 on main, same ruamel set, none introduced.
• Added tests for the bug class
• Tested on Windows 11 build 26200, Python 3.11.15

Documentation & Housekeeping

• Docstrings updated
• cli-config.yaml.example — N/A
• CONTRIBUTING.md / AGENTS.md — N/A
• Cross-platform impact considered — Windows-gated behind _IS_WINDOWS; POSIX EACCES propagation and the no-retry path for permanent errors are both pinned by tests
• Tool descriptions/schemas — N/A

[github-actions]

૮ >ﻌ< ა ci review

_{running on f514d54 — test(utils): cover contended Windows renames on the Windows}

---

_{waiting for more jobs to start…}

⚠️ Warnings

OSV vulnerability scan · View job

3 known vulnerabilities found in pinned dependencies.

• CVE-2026-70608 in package-lock.json
• CVE-2026-56876 in package-lock.json
• CVE-2026-70606 in package-lock.json

How to fix:

Review the findings in the Security tab. Update the affected dependencies if a patched version is available.