Skip to content

fix(update): remove stale .git/shallow.lock before fetch (#75133) - #75168

Closed
RelaxJonh wants to merge 1 commit into
NousResearch:mainfrom
RelaxJonh:fix/stale-shallow-lock-75133
Closed

fix(update): remove stale .git/shallow.lock before fetch (#75133)#75168
RelaxJonh wants to merge 1 commit into
NousResearch:mainfrom
RelaxJonh:fix/stale-shallow-lock-75133

Conversation

@RelaxJonh

Copy link
Copy Markdown
Contributor

Summary

Fixes #75133.

When an update is interrupted (network timeout, app force-close, system sleep), Git leaves behind a stale .git/shallow.lock in shallow-cloned repositories. On the next update:

  1. git fetch fails with fatal: Unable to create '.git/shallow.lock': File exists
  2. hermes update exits with an error
  3. The Tauri/Electron GUI misinterprets this as a concurrent process: "UPDATE DIDN'T FINISH. Hermes is still running"
  4. The .hermes-update-in-progress lock file is left orphaned, trapping users in an unrecoverable update loop

Fix

Before running git fetch on shallow repos, check for and remove stale .shallow.lock files. This is safe because the lock is only meaningful during an active fetch — a stale lock from a dead process has no holder.

     depth_args = ["--depth", "1"] if is_shallow else []

+    # Remove stale .git/shallow.lock left by interrupted updates (#75133).
+    if is_shallow:
+        _shallow_lock = os.path.join(_m().PROJECT_ROOT, ".git", "shallow.lock")
+        if os.path.exists(_shallow_lock):
+            try:
+                os.remove(_shallow_lock)
+            except OSError:
+                pass
+
     if branch == "main":

Testing

  1. touch ~/.hermes/hermes-agent/.git/shallow.lock
  2. Run hermes update
  3. Verify update proceeds without "Hermes is still running" error

…h#75133)

When an update is interrupted (network timeout, app force-close, system
sleep), Git leaves behind a stale .git/shallow.lock in shallow-cloned
repositories.  The next fetch fails with "Unable to create '.git/shallow.lock':
File exists", which the GUI misinterprets as a concurrent process and traps
the user in an unrecoverable update loop.

Fix: before running git fetch on shallow repos, check for and remove stale
.shallow.lock files.  This is safe because the lock is only meaningful
during an active fetch — a stale lock from a dead process has no holder.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cli CLI entry point, hermes_cli/, setup wizard area/install-update Installer, updater, packaging, wheels, doctor sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Jul 31, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracing the stale-lock failure. The report identifies a current unrecovered fetch path, but this patch needs rework before it can address it safely.

Problems

  • The added cleanup is only in _cmd_update_check, reached by the --check return at hermes_cli/main.py:8991-8999. Normal apply updates fetch separately in _cmd_update_impl at hermes_cli/update_cmd.py:3295-3300, which this PR does not change.
  • --check returns before shared UpdateLock acquisition (hermes_cli/main.py:8991-9020). The added existence-only unlink can therefore remove a shallow.lock belonging to a concurrent fetch.
  • The PR changes only hermes_cli/update_cmd.py and adds no regression coverage for either stale or live shallow locks.

Suggested changes

  • Put recovery on the applying fetch path, and make check/apply use a safe ownership or liveness rule rather than deleting any existing lock.
  • Add isolated shallow-repository tests for stale-lock recovery and preservation of a live lock.

Automated hermes-sweeper review.

Comment thread hermes_cli/update_cmd.py
_shallow_lock = os.path.join(_m().PROJECT_ROOT, ".git", "shallow.lock")
if os.path.exists(_shallow_lock):
try:
os.remove(_shallow_lock)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This --check path returns before UpdateLock is acquired (hermes_cli/main.py:8991-9020), so existence alone cannot establish that this lock is stale. It can unlink a lock held by a concurrent fetch; use a shared ownership/liveness guard instead of deleting solely because the path exists.

@teknium1 teknium1 added the sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform label Jul 31, 2026
teknium1 added a commit that referenced this pull request Aug 15, 2026
…with compare-API status

Follow-up on the cherry-picked gitlock work (#80501 by @RGerrish, covering
the #75133 / #75168 wedge first reported and fixed by @RelaxJonh):

- Drop the PR's ancestor-check halves in banner.py, update-count.ts and
  main.ts: superseded by the compare-API status recovery that landed in
  #86257/#86331 (ahead_by == 0 already reports local-ahead as up to date).
  The salvaged update_cmd.py check path keeps main's compare-API structure
  instead of the PR's tip-SHA-plus-ancestry print.
- Keep and wire clear_stale_git_locks() at the remaining wedge sites the
  original PR targeted: hermes update apply, hermes update --check, and the
  passive banner check.
- Add the desktop counterpart (electron/gitlock.ts) so checkUpdates() heals
  the same wedge instead of reporting fetch-failed forever; mirrored
  age + git-process guards; vitest coverage.

E2E verified: real --depth 1 clone with an aged .git/shallow.lock reproduces
"Unable to create '.git/shallow.lock': File exists"; clear_stale_git_locks
removes it and the fetch succeeds; a fresh lock (in-flight fetch) is
preserved.
teknium1 added a commit that referenced this pull request Aug 15, 2026
…with compare-API status

Follow-up on the cherry-picked gitlock work (#80501 by @RGerrish, covering
the #75133 / #75168 wedge first reported and fixed by @RelaxJonh):

- Drop the PR's ancestor-check halves in banner.py, update-count.ts and
  main.ts: superseded by the compare-API status recovery that landed in
  #86257/#86331 (ahead_by == 0 already reports local-ahead as up to date).
  The salvaged update_cmd.py check path keeps main's compare-API structure
  instead of the PR's tip-SHA-plus-ancestry print.
- Keep and wire clear_stale_git_locks() at the remaining wedge sites the
  original PR targeted: hermes update apply, hermes update --check, and the
  passive banner check.
- Add the desktop counterpart (electron/gitlock.ts) so checkUpdates() heals
  the same wedge instead of reporting fetch-failed forever; mirrored
  age + git-process guards; vitest coverage.

E2E verified: real --depth 1 clone with an aged .git/shallow.lock reproduces
"Unable to create '.git/shallow.lock': File exists"; clear_stale_git_locks
removes it and the fetch succeeds; a fresh lock (in-flight fetch) is
preserved.
@teknium1

Copy link
Copy Markdown
Contributor

Thanks @RelaxJonh — earliest fix for this wedge (#75133), and the diagnosis in your PR body (interrupted fetch → stale .git/shallow.lock → GUI misreads it as a concurrent process) was exactly right. The fix landed via #86928, which salvages the wider variant from #80501: same self-heal, plus an age guard (10 min) and a git-process check so an in-flight fetch's lock is never removed, applied at all four wedge sites (update apply, --check, passive banner check, desktop checkUpdates()). Closing with credit — your report drove this.

@teknium1 teknium1 closed this Aug 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/install-update Installer, updater, packaging, wheels, doctor comp/cli CLI entry point, hermes_cli/, setup wizard P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Stuck on "Hermes is still running" caused by stale .git/shallow.lock after interrupted update

3 participants