Add a "retries" option to gpui::test macro and use it in flaky tests - #96
Merged
Conversation
ascarter
pushed a commit
to ascarter/zed
that referenced
this pull request
Jan 21, 2025
* Start adding docs for configurations and adapters * Update debugger setting files with settings, attach config, & themes I also ran prettier and typos on it too." --------- Co-authored-by: Anthony Eid <hello@anthonyeid.me>
jolutz
pushed a commit
to jolutz/zed
that referenced
this pull request
Aug 8, 2026
…tests Add a "retries" option to gpui::test macro and use it in flaky tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #94
Closes #85
Closes #87
Closes #69
This is an experiment. Flaky tests have been a problem in almost every software project I've ever worked on. Any time you're testing async code, it's extremely difficult to nail down the timing of everything perfectly. There are always intermittent issues, and they often only occur on CI. It can be extremely time-consuming and demoralizing to try to track down every issue, but leaving tests flaky causes random build failures which is also bad.
The thought here is this: What if we accept that we're going to have some flaky tests and simply retry them. This lets us keep the tests in place without having to spend tons of time figuring out why they're flaky. This could be dangerous if we overuse it. Sometimes a flaky tests points to a real systemic problem. But I'd like us to have a means of choosing to take on debt related to these flaky tests rather than either deleting them or stopping everything to fix them immediately.
Now, if you want to add retry support to any tests, simply add a
retriesparameter to its test declaration:We'll see if this ends up being a bad idea.