-
-
Notifications
You must be signed in to change notification settings - Fork 11.4k
[CI] Pooling models mteb test disable enforce_eager #26408
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Signed-off-by: wang.yuqi <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request aims to re-enable optimized code paths for certain tests by removing enforce_eager=True, with the goal of verifying a fix for a flaky test. The changes in tests/models/language/pooling_mteb_test/mteb_utils.py align perfectly with this objective. However, a similar change in tests/models/language/generation_ppl_test/ppl_utils.py seems out of scope and could potentially introduce instability in a different test suite. My review includes a comment highlighting this concern.
Signed-off-by: wang.yuqi <[email protected]>
|
https://buildkite.com/vllm/ci/builds/34315/steps/canvas?sid=0199cc47-93cd-48de-9a15-04250b1a3b55 Can be reproduced locally @Isotr0py Could you please help to do further investigation. |
Signed-off-by: wang.yuqi <[email protected]> Signed-off-by: xuebwang-amd <[email protected]>
Signed-off-by: wang.yuqi <[email protected]> Signed-off-by: Dhruvil Bhatt <[email protected]>
Signed-off-by: wang.yuqi <[email protected]>
Signed-off-by: wang.yuqi <[email protected]>
Signed-off-by: wang.yuqi <[email protected]> Signed-off-by: xuebwang-amd <[email protected]>
Signed-off-by: wang.yuqi <[email protected]> Signed-off-by: 0xrushi <[email protected]>
Signed-off-by: wang.yuqi <[email protected]>
Purpose
Related to #22862
Let's re-disable enforce_eager and check if the flaky test has been fixed.
The mteb test has been separated from Language Models Test in #24634, so the Nightly Test keeps failing, it won't confuse anyone.
cc @DarkLight1337 @robertgshaw2-redhat @zou3519
Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.