-
Notifications
You must be signed in to change notification settings - Fork 295
fix: flaky test test_transformers_image_embedder_other #5130
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Greptile Summary
This PR fixes a flaky test in the transformers image embedding functionality by adding retry logic to handle transient failures during Hugging Face model instantiation. The change specifically targets the test_transformers_image_embedder_other
test in tests/ai/test_transformers.py
, which has been experiencing intermittent failures in CI environments.
The implementation adds a retry mechanism that attempts model instantiation up to 5 times with 5-second delays between attempts. The retry logic only catches OSError
exceptions (which are typical for network/IO failures when downloading models) and re-raises the original exception if all retries are exhausted. A time
import was added to support the sleep functionality between retry attempts.
This approach addresses the external dependency flakiness without modifying the core AI functionality. The test structure remains the same, with the retry logic wrapping only the problematic descriptor.instantiate()
call. This is a pragmatic solution for dealing with transient external service issues that are outside the control of the Daft codebase, specifically related to Hugging Face's service reliability during model downloads.
Confidence score: 4/5
- This PR is safe to merge with minimal risk as it only adds defensive retry logic to a flaky test
- Score reflects a well-implemented retry mechanism with appropriate exception handling and reasonable retry parameters
- No files require special attention as this is a straightforward test stability improvement
Context used:
Rule - Import statements should be placed at the top of the file rather than inline within functions or methods. (link)
1 file reviewed, no comments
I do wonder if we should slap the retries onto the |
@desmondcheongzx do you know why most of the tests are not running? I'm not super familiar with the |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5130 +/- ##
=======================================
Coverage 71.87% 71.88%
=======================================
Files 953 953
Lines 130522 130524 +2
=======================================
+ Hits 93816 93821 +5
+ Misses 36706 36703 -3 🚀 New features to boost your workflow:
|
…5130) ## Changes Made This test keeps failing in CI. Adding a retry here so that it succeeds. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes Eventual-Inc#123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review)
Changes Made
This test keeps failing in CI. Adding a retry here so that it succeeds.
Related Issues
Checklist
docs/mkdocs.yml
navigation