Skip to content

fix(logging): tolerate rename failure during multi-process log rollover - #46089

Closed
JohnC1009 wants to merge 1 commit into
NousResearch:mainfrom
JohnC1009:fix/logging-rollover-winerror32-clean
Closed

fix(logging): tolerate rename failure during multi-process log rollover#46089
JohnC1009 wants to merge 1 commit into
NousResearch:mainfrom
JohnC1009:fix/logging-rollover-winerror32-clean

Conversation

@JohnC1009

Copy link
Copy Markdown
Contributor

Problem

On Windows, multiple Hermes processes (gateway service, dashboard service, cron/profile workers, and the CLI) all write to the same log file (e.g. agent.log). When the file crosses maxBytes, whichever process emits next calls doRollover(), which does os.rename(base -> base.1). On Windows you cannot rename a file that another process holds open, so the losing process raises:

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'agent.log' -> 'agent.log.1'

logging.Handler.handleError then dumps the full traceback to stderr (--- Logging error ---) on every emit until the file finally gets rotated. This is non-fatal but produces a wall of tracebacks attached to harmless INFO lines (e.g. model-switch / vision auto-detect logging).

_ManagedRotatingFileHandler already has WatchedFileHandler-style inode-watching in emit() to recover when a peer wins the rotation race — but doRollover() itself had no guard, so the losers crash instead of recovering.

Fix

Wrap _ManagedRotatingFileHandler.doRollover() so an OSError on the rename is caught: the process re-opens its stream on baseFilename and carries on. Whichever process wins the rename performs the real rotation; the existing emit() inode-watch then reopens every other process onto the fresh inode. Worst case the file grows slightly past maxBytes until a rollover succeeds — far better than spamming a traceback per log line.

In stdlib doRollover, the stream is closed and set to None before the os.rename, and the final _open() never runs when the rename raises — so the if self.stream is None: reopen recovery path is correct (baseFilename still exists because the rename failed).

Verification

  • Reproduced the exact failure: stock stdlib RotatingFileHandler.doRollover() raises PermissionError WinError 32 when a peer holds the file open.
  • Patched handler under the same conditions: doRollover() returns cleanly, the stream stays alive, writes keep working, and a later unblocked rollover correctly creates agent.log.1.
  • hermes_logging.py imports cleanly; lint passes.

Scope: single file, hermes_logging.py (+19/-1).

On Windows, multiple Hermes processes (gateway, dashboard, cron workers,
CLI) write the same agent.log. When the file crosses maxBytes, the loser
of the rotation race hits os.rename(base -> base.1) while a peer holds the
file open -> WinError 32, which logging.handleError dumps as a traceback on
every emit until rotation finally succeeds.

Wrap _ManagedRotatingFileHandler.doRollover() so an OSError on rename is
caught: re-open the stream on baseFilename and carry on. Whichever process
wins the rename rotates; the existing emit() inode-watch reopens the rest
onto the fresh inode. Worst case the file grows slightly past maxBytes
until a rollover succeeds, instead of spamming '--- Logging error ---'.
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists labels Jun 14, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the focused Windows rollover investigation. This is already covered on current main by a stronger cross-process solution.

  • Automated hermes-sweeper review verified hermes_logging.py:64-69 aliases the Windows handler to concurrent_log_handler.ConcurrentRotatingFileHandler, which serializes rollover with a cross-process lock rather than allowing competing os.rename() calls.
  • Commit 394cdf48ce2702de224dc95ae73f663076d043bd (fix(logging): alias RotatingFileHandler to concurrent-log-handler, fix(logging): alias RotatingFileHandler to concurrent-log-handler (salvage #44921) #46794) implemented this exact WinError 32 rollover protection.
  • The Windows-only dependency is wired in pyproject.toml:140; the fix is contained in release v2026.6.19.

Closing as implemented on main.

@teknium1 teknium1 closed this Jul 14, 2026
@teknium1 teknium1 added the sweeper:implemented-on-main Sweeper: behavior already present on current main label Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:implemented-on-main Sweeper: behavior already present on current main type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants