Skip to content

worktree: Reload git state when a watcher rescan covers a repository - #61541

Merged
eholk merged 2 commits into
mainfrom
preserve-git-dir-scan-id
Jul 24, 2026
Merged

worktree: Reload git state when a watcher rescan covers a repository#61541
eholk merged 2 commits into
mainfrom
preserve-git-dir-scan-id

Conversation

@eholk

@eholk eholk commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Objective

When the OS file watcher loses sync (e.g. its event queue overflows under heavy fs churn), it drops pending events and reports a single Rescan event for the watched root. Git changes hidden behind such a rescan were silently lost, leaving the git panel and branch indicator stale until something else touched .git.

Contributes to #13176. May fix #60102, though that report predates #60660 and may already be addressed by it on nightly (see Related PRs below).

Two bugs caused this, and they masked each other:

  1. A rescan never triggered a git reload. The Rescan event's path is the worktree root, not something inside .git, so it never populated dot_git_abs_paths in process_events — and since .git is excluded from entry scanning, the rescan produced no .git events either. The dropped git changes were simply never picked up.
  2. Re-scanning reset git_dir_scan_id to 0. The snapshot diff detects git changes by comparing scan ids, so re-inserting the repository with a fresh id could wipe out a bump made earlier in the same scan cycle, swallowing the corresponding UpdatedGitRepositories signal.

The masking is why these fixes land together rather than as two PRs: the root-rescan case in test_dot_git_dir_event_does_not_suppress_children only passed on main because the buggy scan-id reset made the snapshot diff fire spuriously. Fixing either bug alone turns that (currently green) test red.

Solution

  • process_events: a Rescan event now schedules a git state reload for every repository whose git directory (dot_git, common_dir, or repository_dir) lies under the rescanned path, covering linked worktrees and gitfile repositories, including git dirs watched outside the worktree root.
  • insert_git_repository_for_path: carry the existing git_dir_scan_id forward when re-inserting a repository instead of resetting it to 0. Deliberately not bumped either: re-insertion is snapshot bookkeeping, not evidence of a git change — bumping would trigger spurious full reloads on non-lossy paths (explicit refreshes, path-prefix scans). Only update_git_repositories claims that git state changed.
  • changed_repos: a debug_assert enforcing that git_dir_scan_id never regresses, so future violations of this invariant fail loudly in tests instead of manifesting as a stale git panel.

Testing

New fault-injection infrastructure and tests:

  • FakeFs::simulate_watcher_overflow models the kernel's watch queue overflowing: buffered (undelivered) events are discarded and replaced by a single Rescan for the given root, mirroring FSEvents kFSEventStreamEventFlagMustScanSubDirs, inotify IN_Q_OVERFLOW, and Windows ERROR_NOTIFY_ENUM_DIR.
  • test_watcher_overflow_rescan_reloads_git_state: a git change whose events are lost to an overflow must still be picked up via the rescan (reproduces bug 1; fails on main).
  • test_git_update_in_same_batch_as_rescan_is_not_lost: a git event processed in the same batch as a rescan must not lose its scan-id bump to the repository re-insertion (reproduces bug 2; fails on main).
  • test_random_git_updates_with_watcher_overflows: randomized property test (100 iterations) asserting that every git state change is eventually signaled via UpdatedGitRepositories under random event batching, delays, and overflows. Fails on main within the first few seeds.
  • test_random_worktree_changes now also injects watcher overflows, extending its existing convergence property to rescan reconciliation of worktree entries (this already passed; the injection guards it going forward).

Full worktree and fs suites pass. Verified the directed tests exercise the intended code paths via trace logging (the same-batch test hits update_git_repositories stamping followed by re-insertion, distinct from the overflow test where no .git event arrives at all).

Related PRs

The recent stale-git-state reports trace back to three distinct mechanisms that share one symptom. This PR addresses the third:

Self-Review Checklist:

  • I've reviewed my own diff for quality, security, and reliability
  • Unsafe blocks (if any) have justifying comments
  • The content adheres to Zed's UI standards (UX/UI and icon guidelines)
  • Tests cover the new/changed behavior
  • Performance impact has been considered and is acceptable

Release Notes:

  • Fixed the git panel and branch indicator showing stale state after heavy file-system activity caused the file watcher to lose events

When the OS file watcher loses sync (e.g. its event queue overflows), it
drops pending events and reports a single Rescan event for the watched
root. Two bugs caused git changes hidden behind such a rescan to be
silently lost, leaving the git panel and branch indicator stale until
something else touched .git:

1. A rescan only re-scanned worktree entries. Since the Rescan event's
   path is the worktree root and .git is excluded from entry scanning,
   no git reload was ever triggered, even though the dropped events may
   have included changes inside .git. Now a Rescan event schedules a git
   state reload for every repository whose git directory lies under the
   rescanned path.

2. Re-scanning re-inserts the repository entry with git_dir_scan_id: 0.
   The snapshot diff detects git changes by comparing scan ids, so the
   reset could wipe out a bump made earlier in the same scan cycle,
   swallowing the corresponding UpdatedGitRepositories signal. The scan
   id is now carried forward on re-insertion, and a debug assertion in
   changed_repos enforces that it never regresses.

The two bugs masked each other in
test_dot_git_dir_event_does_not_suppress_children: the root-rescan case
only passed because the buggy scan id reset made the diff fire
spuriously. Fixing either bug alone turns that test red, which is why
both fixes land together.

Adds FakeFs::simulate_watcher_overflow, which models the kernel's watch
queue overflowing (buffered events are discarded and replaced by a
Rescan for the given root), and uses it in:

- test_watcher_overflow_rescan_reloads_git_state: a git change whose
  events are lost to an overflow must still be picked up via the rescan.
- test_git_update_in_same_batch_as_rescan_is_not_lost: a git event
  processed in the same batch as a rescan must not lose its scan id
  bump to the repository re-insertion.
- test_random_git_updates_with_watcher_overflows: randomized property
  test asserting every git state change is eventually signaled via
  UpdatedGitRepositories under random batching, delays, and overflows.
- test_random_worktree_changes: now also injects watcher overflows,
  extending its convergence property to rescan reconciliation.
@cla-bot cla-bot Bot added the cla-signed The user has signed the Contributor License Agreement label Jul 23, 2026
@zed-community-bot zed-community-bot Bot added the staff Pull requests authored by a current member of Zed staff label Jul 23, 2026

@RemcoSmitsDev RemcoSmitsDev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey eholk, thanks for making these 2 fixes. Sorry I haven't pushed this forward, but I couldn't consistently reproduce this issue anymore, sometime not at all. So was hard to say if it was really fixed except by the tests that I wrote similar to yours.

But these 2 fixes makes total sense to me, since I had the same fixes😀.

@Anthony-Eid Anthony-Eid left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree with Remco, awesome work

@eholk
eholk added this pull request to the merge queue Jul 24, 2026
@eholk

eholk commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review and all the investigation, @RemcoSmitsDev. It definitely made it a lot easier to pick this up and push this forward. To be honest, I hadn't directly reproed these issues either but I've definitely seen git sync issues and the tests looked good so this should be a significant step forward.

Merged via the queue into main with commit 64672ee Jul 24, 2026
36 checks passed
@eholk
eholk deleted the preserve-git-dir-scan-id branch July 24, 2026 16:54
mdz-axo added a commit to mdz-axo/zed-kask that referenced this pull request Jul 25, 2026
Upstream changes (zed-industries/zed main, 27 commits):
- agent: Add agent.compaction_model setting for context compaction (zed-industries#60012)
- agent: Show effort selector for anthropic compatible providers (zed-industries#61579)
- acp: Update agent-client-protocol SDK to 2.0.0 (zed-industries#61570)
- client: Extract proxy handshakes into new proxy_handshake crate (zed-industries#61427)
- collab: Fix multiworkspace location out of sync bugs (zed-industries#61598)
- editor: Fix sticky header drag cancels autoscroll (zed-industries#53592)
- editor: Fix crash when copying and pasting using multiple cursors (zed-industries#61545)
- editor: Skip untitled buffers when saving a multi-buffer (zed-industries#61380)
- gpui: Fix images not being drawn with rounded corners with ObjectFit::Cover (zed-industries#61383)
- gpui: Fix deadlock in performance profiler and reenable it (zed-industries#61584)
- git_ui: Prevent Git panel bindings in repository selector (zed-industries#61282)
- language_model: Add explicit OpenAI conversation compaction and fix Anthropic compaction (zed-industries#61370)
- markdown: Fix squashed Mermaid diagrams in markdown preview (zed-industries#61260)
- Opus 5 BYOK Support (zed-industries#61596)
- repl: Show add-cell controls in empty notebooks (zed-industries#61329)
- search: Escape seeded buffer search query in regex mode (zed-industries#57748)
- settings: Fix VS Code import appending duplicate file associations (zed-industries#61355)
- settings: Split VSCode and Zed keymap files (zed-industries#61532)
- Treat blank spawn_agent session IDs as absent (zed-industries#60893)
- worktree: Reload git state when a watcher rescan covers a repository (zed-industries#61541)
- Plus 7 more minor fixes.

Merge fixes:
- crates/agent/src/thread.rs: replay_tool_call used 'message_ix' (undefined)
  after auto-merge; renamed to 'owning_message_ix' (the parameter name).
- Cargo.toml: Removed stale workspace members hkask-wallet and hkask-git-cas
  (both directories deleted in prior commits but workspace entries remained).
- kask/crates/hkask-regulation/src/wallet_manager.rs: Stubbed consume() and
  settle_rjoules() on WalletBudgetPort — these were API-key encumbrance
  operations from the deleted hkask-wallet crate; regulation tracks per-agent
  gas balances, not per-key encumbrances.
- kask/crates/hkask-regulation/src/wallet_gas_calibrator.rs: Fixed test to
  use crate::agent_wallet_store::WalletStore instead of hkask_storage::WalletStore.
- kask/crates/hkask-regulation/Cargo.toml: Added tokio macros feature to
  dev-dependencies for #[tokio::test].
- kask/crates/kask_bridge/Cargo.toml: Added futures dependency (needed by
  context_injector.rs for futures::executor::block_on).
- kask/crates/kask_bridge/src/context_injector.rs: Fixed futures_util::executor
  to futures::executor (futures-util doesn't include executor module).

Release Notes:

- N/A
0arm pushed a commit to 0arm/zed that referenced this pull request Jul 26, 2026
…ed-industries#61541)

# Objective

When the OS file watcher loses sync (e.g. its event queue overflows
under heavy fs churn), it drops pending events and reports a single
`Rescan` event for the watched root. Git changes hidden behind such a
rescan were silently lost, leaving the git panel and branch indicator
stale until something else touched `.git`.

Contributes to zed-industries#13176. May fix zed-industries#60102, though that report predates
zed-industries#60660 and may already be addressed by it on nightly (see Related PRs
below).

Two bugs caused this, and they masked each other:

1. **A rescan never triggered a git reload.** The `Rescan` event's path
is the worktree root, not something inside `.git`, so it never populated
`dot_git_abs_paths` in `process_events` — and since `.git` is excluded
from entry scanning, the rescan produced no `.git` events either. The
dropped git changes were simply never picked up.
2. **Re-scanning reset `git_dir_scan_id` to 0.** The snapshot diff
detects git changes by comparing scan ids, so re-inserting the
repository with a fresh id could wipe out a bump made earlier in the
same scan cycle, swallowing the corresponding `UpdatedGitRepositories`
signal.

The masking is why these fixes land together rather than as two PRs: the
root-rescan case in `test_dot_git_dir_event_does_not_suppress_children`
only passed on `main` because the buggy scan-id reset made the snapshot
diff fire spuriously. Fixing either bug alone turns that (currently
green) test red.

## Solution

- `process_events`: a `Rescan` event now schedules a git state reload
for every repository whose git directory (`dot_git`, `common_dir`, or
`repository_dir`) lies under the rescanned path, covering linked
worktrees and gitfile repositories, including git dirs watched outside
the worktree root.
- `insert_git_repository_for_path`: carry the existing `git_dir_scan_id`
forward when re-inserting a repository instead of resetting it to 0.
Deliberately *not* bumped either: re-insertion is snapshot bookkeeping,
not evidence of a git change — bumping would trigger spurious full
reloads on non-lossy paths (explicit refreshes, path-prefix scans). Only
`update_git_repositories` claims that git state changed.
- `changed_repos`: a `debug_assert` enforcing that `git_dir_scan_id`
never regresses, so future violations of this invariant fail loudly in
tests instead of manifesting as a stale git panel.

## Testing

New fault-injection infrastructure and tests:

- `FakeFs::simulate_watcher_overflow` models the kernel's watch queue
overflowing: buffered (undelivered) events are discarded and replaced by
a single `Rescan` for the given root, mirroring FSEvents
`kFSEventStreamEventFlagMustScanSubDirs`, inotify `IN_Q_OVERFLOW`, and
Windows `ERROR_NOTIFY_ENUM_DIR`.
- `test_watcher_overflow_rescan_reloads_git_state`: a git change whose
events are lost to an overflow must still be picked up via the rescan
(reproduces bug 1; fails on `main`).
- `test_git_update_in_same_batch_as_rescan_is_not_lost`: a git event
processed in the same batch as a rescan must not lose its scan-id bump
to the repository re-insertion (reproduces bug 2; fails on `main`).
- `test_random_git_updates_with_watcher_overflows`: randomized property
test (100 iterations) asserting that every git state change is
eventually signaled via `UpdatedGitRepositories` under random event
batching, delays, and overflows. Fails on `main` within the first few
seeds.
- `test_random_worktree_changes` now also injects watcher overflows,
extending its existing convergence property to rescan reconciliation of
worktree entries (this already passed; the injection guards it going
forward).

Full `worktree` and `fs` suites pass. Verified the directed tests
exercise the intended code paths via trace logging (the same-batch test
hits `update_git_repositories` stamping followed by re-insertion,
distinct from the overflow test where no `.git` event arrives at all).

## Related PRs

The recent stale-git-state reports trace back to three distinct
mechanisms that share one symptom. This PR addresses the third:

- **Events never generated** — zed-industries#60660 (merged, in this PR's base):
Linux's non-recursive watcher missed nested `refs/` directories, so
external commits/fetches produced no events at all. That PR (together
with zed-industries#60590, which explicitly rescans after Zed-initiated reset/fetch
and touches only `git_store.rs`) fixed zed-industries#60348. No overlap with this PR;
a merge against current `main` is clean, and the refs-watching tests are
disjoint from the overflow/rescan tests added here.
- **Events coalesced** — zed-industries#59876 (open, complementary): FSEvents can
merge `.git` child events into a bare `.git` `Changed` event; the signal
arrives, in a shape Zed ignores. @RemcoSmitsDev's review comment there
describes this PR's failure mode and calls the two fixes complementary;
this is effectively the follow-up promised in that comment. Both PRs
touch the same region of `process_events`, so whichever lands second
needs a trivial rebase, and zed-industries#59876 flips the bare-`.git` expectation in
`test_dot_git_dir_event_does_not_suppress_children` Case 2, which this
PR preserves.
- **Events dropped** — this PR: the watcher generated events but lost
them to a queue overflow, and the resulting `Rescan` did not reach the
git reload path. zed-industries#59976 and zed-industries#60098 (merged) reduced how often this
happens; this PR makes git state recover correctly when it does.

## Self-Review Checklist:

- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments
- [x] The content adheres to Zed's UI standards
([UX/UI](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
and
[icon](https://github.com/zed-industries/zed/blob/main/crates/icons/README.md)
guidelines)
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable

Release Notes:

- Fixed the git panel and branch indicator showing stale state after
heavy file-system activity caused the file watcher to lose events
jolutz pushed a commit to jolutz/zed that referenced this pull request Aug 8, 2026
…ed-industries#61541)

# Objective

When the OS file watcher loses sync (e.g. its event queue overflows
under heavy fs churn), it drops pending events and reports a single
`Rescan` event for the watched root. Git changes hidden behind such a
rescan were silently lost, leaving the git panel and branch indicator
stale until something else touched `.git`.

Contributes to zed-industries#13176. May fix zed-industries#60102, though that report predates
zed-industries#60660 and may already be addressed by it on nightly (see Related PRs
below).

Two bugs caused this, and they masked each other:

1. **A rescan never triggered a git reload.** The `Rescan` event's path
is the worktree root, not something inside `.git`, so it never populated
`dot_git_abs_paths` in `process_events` — and since `.git` is excluded
from entry scanning, the rescan produced no `.git` events either. The
dropped git changes were simply never picked up.
2. **Re-scanning reset `git_dir_scan_id` to 0.** The snapshot diff
detects git changes by comparing scan ids, so re-inserting the
repository with a fresh id could wipe out a bump made earlier in the
same scan cycle, swallowing the corresponding `UpdatedGitRepositories`
signal.

The masking is why these fixes land together rather than as two PRs: the
root-rescan case in `test_dot_git_dir_event_does_not_suppress_children`
only passed on `main` because the buggy scan-id reset made the snapshot
diff fire spuriously. Fixing either bug alone turns that (currently
green) test red.

## Solution

- `process_events`: a `Rescan` event now schedules a git state reload
for every repository whose git directory (`dot_git`, `common_dir`, or
`repository_dir`) lies under the rescanned path, covering linked
worktrees and gitfile repositories, including git dirs watched outside
the worktree root.
- `insert_git_repository_for_path`: carry the existing `git_dir_scan_id`
forward when re-inserting a repository instead of resetting it to 0.
Deliberately *not* bumped either: re-insertion is snapshot bookkeeping,
not evidence of a git change — bumping would trigger spurious full
reloads on non-lossy paths (explicit refreshes, path-prefix scans). Only
`update_git_repositories` claims that git state changed.
- `changed_repos`: a `debug_assert` enforcing that `git_dir_scan_id`
never regresses, so future violations of this invariant fail loudly in
tests instead of manifesting as a stale git panel.

## Testing

New fault-injection infrastructure and tests:

- `FakeFs::simulate_watcher_overflow` models the kernel's watch queue
overflowing: buffered (undelivered) events are discarded and replaced by
a single `Rescan` for the given root, mirroring FSEvents
`kFSEventStreamEventFlagMustScanSubDirs`, inotify `IN_Q_OVERFLOW`, and
Windows `ERROR_NOTIFY_ENUM_DIR`.
- `test_watcher_overflow_rescan_reloads_git_state`: a git change whose
events are lost to an overflow must still be picked up via the rescan
(reproduces bug 1; fails on `main`).
- `test_git_update_in_same_batch_as_rescan_is_not_lost`: a git event
processed in the same batch as a rescan must not lose its scan-id bump
to the repository re-insertion (reproduces bug 2; fails on `main`).
- `test_random_git_updates_with_watcher_overflows`: randomized property
test (100 iterations) asserting that every git state change is
eventually signaled via `UpdatedGitRepositories` under random event
batching, delays, and overflows. Fails on `main` within the first few
seeds.
- `test_random_worktree_changes` now also injects watcher overflows,
extending its existing convergence property to rescan reconciliation of
worktree entries (this already passed; the injection guards it going
forward).

Full `worktree` and `fs` suites pass. Verified the directed tests
exercise the intended code paths via trace logging (the same-batch test
hits `update_git_repositories` stamping followed by re-insertion,
distinct from the overflow test where no `.git` event arrives at all).

## Related PRs

The recent stale-git-state reports trace back to three distinct
mechanisms that share one symptom. This PR addresses the third:

- **Events never generated** — zed-industries#60660 (merged, in this PR's base):
Linux's non-recursive watcher missed nested `refs/` directories, so
external commits/fetches produced no events at all. That PR (together
with zed-industries#60590, which explicitly rescans after Zed-initiated reset/fetch
and touches only `git_store.rs`) fixed zed-industries#60348. No overlap with this PR;
a merge against current `main` is clean, and the refs-watching tests are
disjoint from the overflow/rescan tests added here.
- **Events coalesced** — zed-industries#59876 (open, complementary): FSEvents can
merge `.git` child events into a bare `.git` `Changed` event; the signal
arrives, in a shape Zed ignores. @RemcoSmitsDev's review comment there
describes this PR's failure mode and calls the two fixes complementary;
this is effectively the follow-up promised in that comment. Both PRs
touch the same region of `process_events`, so whichever lands second
needs a trivial rebase, and zed-industries#59876 flips the bare-`.git` expectation in
`test_dot_git_dir_event_does_not_suppress_children` Case 2, which this
PR preserves.
- **Events dropped** — this PR: the watcher generated events but lost
them to a queue overflow, and the resulting `Rescan` did not reach the
git reload path. zed-industries#59976 and zed-industries#60098 (merged) reduced how often this
happens; this PR makes git state recover correctly when it does.

## Self-Review Checklist:

- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments
- [x] The content adheres to Zed's UI standards
([UX/UI](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
and
[icon](https://github.com/zed-industries/zed/blob/main/crates/icons/README.md)
guidelines)
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable

Release Notes:

- Fixed the git panel and branch indicator showing stale state after
heavy file-system activity caused the file watcher to lose events
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed The user has signed the Contributor License Agreement staff Pull requests authored by a current member of Zed staff

Projects

None yet

3 participants