Skip to content

Enable accuracy test for PR labeled with "*accuracy-test"#1040

Merged
Yikun merged 5 commits intovllm-project:mainfrom
Yikun:pr/953
Jun 3, 2025
Merged

Enable accuracy test for PR labeled with "*accuracy-test"#1040
Yikun merged 5 commits intovllm-project:mainfrom
Yikun:pr/953

Conversation

@Yikun
Copy link
Copy Markdown
Member

@Yikun Yikun commented May 31, 2025

What this PR does / why we need it?

This PR enable accuracy test for PR labeled with "*accuracy-test" and workflow_dispatch.

Only one model test running for each type test to reduce excution time.

  • The dense test costs about 25mins to complete (gsm8k 7mins, mmlu 3h24mins, cEval 18mins)
  • The vl test costs about 40mins to complete

In futute, we might consider enable all job test as nightly schedule job.

Below is mainly changes:

  • the dense/vl accuracy test will be triggered by lableling accuracy-test and ready-for-test
  • the dense accuracy test will be triggered by lableling dense-accuracy-test and ready-for-test
  • the vl accuracy test will be triggered by lableling vl-accuracy-test and ready-for-test
  • accuracy test will also be triggered by workflow_dispatch
  • Support V1 and V0 for qwen and V0 for VL

For PR test we also generate summary in test summary.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Closes: #953

@Yikun Yikun added ready-for-test start test by label for PR dense-accuracy-test labels May 31, 2025
@Yikun Yikun added the accuracy-test enable all accuracy test for PR label Jun 1, 2025
@Yikun Yikun added accuracy-test enable all accuracy test for PR and removed accuracy-test enable all accuracy test for PR labels Jun 1, 2025
@Yikun Yikun added accuracy-test enable all accuracy test for PR and removed accuracy-test enable all accuracy test for PR labels Jun 1, 2025
@Yikun Yikun added accuracy-test enable all accuracy test for PR and removed accuracy-test enable all accuracy test for PR labels Jun 2, 2025
@Yikun Yikun added accuracy-test enable all accuracy test for PR dense-accuracy-test and removed accuracy-test enable all accuracy test for PR labels Jun 2, 2025
@Yikun Yikun force-pushed the pr/953 branch 2 times, most recently from 94c553c to f35efe9 Compare June 2, 2025 16:09
@Yikun Yikun force-pushed the pr/953 branch 2 times, most recently from b16c182 to f92fea7 Compare June 2, 2025 16:15
@Yikun Yikun added accuracy-test enable all accuracy test for PR and removed vl-accuracy-test accuracy-test enable all accuracy test for PR labels Jun 2, 2025
zhangxinyuehfad and others added 5 commits June 3, 2025 11:08
Signed-off-by: hfadzxy <starmoon_zhang@163.com>
Signed-off-by: Yikun Jiang <yikunkero@gmail.com>
Signed-off-by: Yikun Jiang <yikunkero@gmail.com>
Signed-off-by: Yikun Jiang <yikunkero@gmail.com>
Signed-off-by: Yikun Jiang <yikunkero@gmail.com>
@Yikun
Copy link
Copy Markdown
Member Author

Yikun commented Jun 3, 2025

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready read for review ready-for-test start test by label for PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants