Repository navigation
fix(kanban): discard partial worktrees when git worktree add fails - #126020
Open
liuhao1024 wants to merge 1 commit into
Open
liuhao1024 wants to merge 1 commit into
liuhao1024 wants to merge 1 commit into
Conversation
A killed or failed 'git worktree add' left a partial checkout behind (the .git pointer lands before the files do), and the next dispatch passed _ensure_git_worktree's common-dir reuse shortcut straight into the incomplete tree — workers ran on a partial checkout and most such tasks even reported completed (NousResearch#126004). - Both failure paths (non-zero exit, subprocess timeout) now run a best-effort discard: worktree remove --force, rmtree, then prune last so an admin registration the remove could not clear is still dropped. - The add timeout rises from a hard 60s (which killed full checkouts of large monorepos mid-flight under AV scanning) to 600s, overridable via HERMES_KANBAN_WORKTREE_TIMEOUT. - _record_task_failure keeps 4000 chars instead of 500 so the real git fatal line survives into last_failure_error and task_runs.error.
12 of 19 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
A failed or killed
git worktree addleft a partial checkout behind, and the next dispatch silently reused it. Git writes the worktree's.gitpointer before checking out files, so the leftover directory passed_ensure_git_worktree's common-dir reuse shortcut and workers ran inside an incomplete tree — in the reporter's data, 5 of 6 affected tasks even reportedcompletedon a partial checkout.This PR:
subprocess.TimeoutExpired): best-effortgit worktree remove --force, thenrmtree, thengit worktree prunelast — the order matters, becausepruneonly drops registrations whose directory is gone, so it must run after the directory is removed to clear admin metadata thatremovecould not (damaged state). A retry now performs a real, completeworktree add.HERMES_KANBAN_WORKTREE_TIMEOUT. The 60s cap killed full checkouts of large monorepos mid-flight (the reporter's 41.6k-file repo under Defender real-time scanning exceeded it);worktree addis a local checkout with no network hang risk._record_task_failure, so the real git fatal line survives intotasks.last_failure_error/task_runs.error/ event payloads instead of being cut off by progress noise.Related Issue
Fixes #126004
Type of Change
Changes Made
hermes_cli/kanban_db_workspace.py: added_worktree_add_timeout()(env-overridable 600s default) and_discard_partial_worktree()(best-effort remove → rmtree → prune);_ensure_git_worktreenow cleans up on both the non-zero-exit and timeout paths before raising, preserving the original error.hermes_cli/kanban_db_dispatch.py:_record_task_failurecaps the error text at 4000 chars instead of 500.tests/hermes_cli/test_kanban_worktree_failure_cleanup.py: new coverage — partial discard + successful retry after a non-zero exit and after a timeout; timeout env override is honored; failure text keeps 4000 chars.How to Test
python -m pytest tests/hermes_cli/test_kanban_worktree_failure_cleanup.py -v— Observed result: 4 passed (partial checkout discarded on both failure paths, retry completes a real checkout,HERMES_KANBAN_WORKTREE_TIMEOUT=123reaches the git call,last_failure_errorlength is 4000).python -m pytest tests/hermes_cli/test_kanban_worktree_isolation.py tests/hermes_cli/test_kanban_worktree_teardown.py tests/hermes_cli/test_kanban_worktree_unicode_path.py -q— Observed result: 19 passed (existing worktree behavior unchanged).python -m pytest tests/hermes_cli/test_kanban_worker_exit_trailer.py tests/hermes_cli/test_kanban_blocked_sticky.py tests/hermes_cli/test_kanban_review_lifecycle_complete.py -q— Observed result: 42 passed (_record_task_failurecallers unaffected).ruff checkon all three changed files — all checks passed.Note:
tests/hermes_cli/test_kanban_db.py::test_infrastructure_spawn_refusal_never_charges_the_cardfails identically on a clean upstream/main checkout of this host (verified viagit stashon the base commit) — pre-existing environment failure, unrelated to this change. All other 52 tests in that file pass.Checklist
Code
fix(scope):,feat(scope):, etc.)pytest tests/ -qand all tests passDocumentation & Housekeeping
docs/, docstrings) — or N/Acli-config.yaml.exampleif I added/changed config keys — or N/A (HERMES_KANBAN_WORKTREE_TIMEOUTis an env knob like the existingHERMES_KANBAN_WORKSPACES_ROOT, documented in the helper's docstring)CONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — or N/AFor New Skills
N/A
Screenshots / Logs