Back off retries when file watch registration keeps failing - #62489
Open
etiennechatreaux wants to merge 3 commits into
Open
Back off retries when file watch registration keeps failing#62489etiennechatreaux wants to merge 3 commits into
etiennechatreaux wants to merge 3 commits into
Conversation
poll_path_until_created retried a failing registration at the base poll interval (2s) forever, spamming the log and hammering an already-degraded process (see the fd exhaustion in zed-industries#62486). Double the delay after each failed attempt, capped at 60s; the create-poll cadence is unchanged.
|
We require contributors to sign our Contributor License Agreement, and we don't have @etiennechatreaux on file. You can sign our CLA at https://zed.dev/cla. Once you've signed, post a comment here that says '@cla-bot check'. |
`TestAppContext::executor()` already returns a `BackgroundExecutor` by value, so the extra `.clone()` tripped `-D clippy::redundant-clone` on every clippy_* CI job for the fs lib test target. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LKrrp8U9ptLJTYM6kujA1Y
Author
|
The Verified locally with the CI command on the pinned toolchain ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Objective
Part of #62486: when watch registration keeps failing (in that issue, FSEvent stream creation failing after fd exhaustion on macOS),
poll_path_until_createdretries the same path every 2 seconds forever — no backoff, no cap. On an already-degraded process this spams the log (267 warnings in the 10 minutes before log rotation killed the log) and keeps hammering a resource that is already exhausted.This does not fix the watcher leak that causes the exhaustion in the first place (see the retention chain posted on the issue); it stops the retry loop from making a degraded process worse. Related: #59571 takes a complementary route for the same error signature (classifying the macOS FSEvent failure into the native watch-limit cooldown); this PR bounds the per-path retry loop itself, for any registration error on any platform.
Solution
Double the retry delay after each failed registration attempt, capped at 60 seconds. The base interval stays
poll_interval()(2s by default), so the common case — polling for a path that doesn't exist yet — is unchanged, and the first sleep is identical to before. The delay only grows on theErrarm, and the warning now says when the next retry happens.To make the failure path testable, the registration function is now a parameter of
poll_path_until_created(it previously calledregister_existing_pathdirectly, which goes through theglobal_watcher()static, so a fake backend couldn't be injected into the poll task).Testing
failed_watch_registration_retries_with_backoff: drives the poll task with an injected registration function that always fails, and asserts the exact backoff schedule (2s, 4s, 8s, … capped at 60s) with a single attempt per interval, on the deterministic test executor.cargo test -p fs --lib fs_watcher::tests: 16 passed.pending_path_is_registered_once_createdcovers the success path (first poll still fires atpoll_interval()).Self-Review Checklist:
Release Notes: