Skip to content

fix(utils): retry transient Windows lock failures in atomic_replace - #79793

Closed
ruochu88s wants to merge 1 commit into
NousResearch:mainfrom
ruochu88s:pr/windows-atomic-replace-retry
Closed

fix(utils): retry transient Windows lock failures in atomic_replace#79793
ruochu88s wants to merge 1 commit into
NousResearch:mainfrom
ruochu88s:pr/windows-atomic-replace-retry

Conversation

@ruochu88s

Copy link
Copy Markdown
Contributor

What does this PR do?

On Windows, os.replace() intermittently fails when another process holds a
short-lived handle on the destination without FILE_SHARE_DELETE. Antivirus
scanners, search indexers, cloud sync clients (OneDrive/Dropbox), and a
just-closed editor all do this. The call fails with one of:

WinError 5  (access denied)
WinError 32 (sharing violation)
WinError 33 (lock violation)

None of these are covered by the existing EXDEV/EBUSY copy fallback, so the
exception propagates and the write is lost — even though a retry a few
milliseconds later would succeed.

atomic_replace() backs config writes, session state, and credential updates,
so a background scanner touching the file at the wrong moment surfaces to the
user as an unexplained save failure with a raw WinError in the traceback.

Why this approach

Retry only the three winerror codes that are genuinely transient, with a bounded
backoff (50ms/100ms/200ms/400ms/800ms/1s, ~2.5s total). Bounded is the important
part: a real ACL denial still propagates instead of hanging, which keeps a
permissions bug diagnosable rather than turning it into a stall.

The predicate is gated on os.name == "nt" so POSIX behaviour is untouched, and
EXDEV/EBUSY continue to take the existing copy-fallback path. Symlink
resolution is unchanged.

An alternative would be to widen the existing copy fallback to cover these
codes, but that changes replace semantics (the copy path is not atomic) for a
failure that is usually gone within milliseconds. Retrying preserves atomicity.

Related Issue

No existing issue — I searched open issues/PRs and found no report of this. Happy
to open a tracking issue first if maintainers prefer that order.

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • utils.py
    • Add _WINDOWS_ATOMIC_REPLACE_RETRY_DELAYS — bounded backoff schedule.
    • Add _WINDOWS_TRANSIENT_REPLACE_ERRORS = frozenset({5, 32, 33}).
    • Add _is_transient_windows_replace_error() — gated on os.name == "nt" and
      the winerror attribute, so it can never fire on POSIX.
    • atomic_replace(): wrap the os.replace() call in a bounded retry loop,
      logging each retry at DEBUG. Existing EXDEV/EBUSY handling and symlink
      resolution are unchanged.
  • tests/test_atomic_replace_symlinks.py — extend with the retry cases.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(utils):)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix (single commit)
  • I've run the relevant suite and all tests pass (see below)
  • I've added tests for my changes
  • I've tested on my platform: Windows 10
tests/test_atomic_replace_symlinks.py    10 passed, 0 failed

Coverage: each retried winerror code, retry exhaustion still raising, a
non-transient error propagating immediately without retries, POSIX being
unaffected, and symlink targets still resolved.

I verified the tests actually fail when the fix is reverted — emptying
_WINDOWS_TRANSIENT_REPLACE_ERRORS turns the suite red, confirming the
assertions are not vacuous.

Documentation & Housekeeping

  • I've updated relevant documentation — N/A (internal helper; the retry
    rationale is documented in an inline comment)
  • I've updated cli-config.yaml.example if I added/changed config keys — N/A
  • I've updated CONTRIBUTING.md or AGENTS.md — N/A
  • I've considered cross-platform impact — yes: the retry predicate requires
    os.name == "nt", so POSIX takes exactly the previous code path
  • I've updated tool descriptions/schemas — N/A

Screenshots / Logs

With the fix, a transient lock is retried instead of surfacing:

DEBUG utils: atomic_replace: transient Windows lock 32 for
      <tmp> -> <target>; retrying in 0.05s (1/6)

On Windows, `os.replace()` intermittently fails when another process holds a
short-lived handle on the destination without FILE_SHARE_DELETE. Antivirus
scanners, search indexers, cloud sync clients, and a just-closed editor all do
this. The call fails with one of:

    WinError 5  (access denied)
    WinError 32 (sharing violation)
    WinError 33 (lock violation)

None of these are covered by the existing EXDEV/EBUSY copy fallback, so the
exception propagates and the write is lost even though a retry microseconds
later would succeed. Because `atomic_replace()` backs config writes, session
state, and credential updates, a background scanner touching the file at the
wrong moment surfaces as an unexplained save failure.

Fix: retry those three winerror codes with a bounded backoff
(50ms/100ms/200ms/400ms/800ms/1s, ~2.5s total). Retries are deliberately
bounded so a genuine ACL denial still propagates instead of hanging; the
predicate is also gated on os.name == "nt" so POSIX behavior is untouched.
EXDEV/EBUSY continue to take the existing copy fallback path, and symlink
resolution is unchanged.

Tests: 10 tests covering each retried code, exhaustion still raising,
non-transient errors propagating immediately, POSIX being unaffected, and
symlink targets still resolved. Verified the suite fails when the retry
predicate is reverted.
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint platform/windows Native Windows-specific behavior or breakage P2 Medium — degraded but workaround exists duplicate This issue or pull request already exists sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows labels Aug 6, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Duplicate of #36921: both implement bounded retry for transient Windows atomic_replace permission/lock failures. #57777 is related because it additionally adds a copy-fallback path.

@OutThisLife

Copy link
Copy Markdown
Collaborator

Superseded by #84852.

Your winerror set {5, 32, 33} was the correct classification in this cluster — verified on real Windows that a held target reports 5, not 32, so PRs gating on ERROR_SHARING_VIOLATION alone missed the actual bug. #84852 uses that set and credits you as a co-author.

The one gap: retry alone doesn't rescue a reader that outlives the budget (a desktop auth-init holds auth.json past 100ms), so #84852 pairs your classification with a fallback for the persistent case. Your bounded-retry rationale — a real ACL denial must surface rather than hang — is preserved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint duplicate This issue or pull request already exists P2 Medium — degraded but workaround exists platform/windows Native Windows-specific behavior or breakage sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants