Fix accuracy test config --config-list-file#2163
Fix accuracy test config --config-list-file#2163wxsIcey wants to merge 2 commits intovllm-project:mainfrom wxsIcey:wxs_0801
Conversation
Signed-off-by: Icey <1790571317@qq.com>
wxsIcey
added
accuracy-test
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. |
wxsIcey
added
accuracy-test
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2163 +/- ##
==========================================
- Coverage 76.89% 76.67% -0.22%
==========================================
Files 107 107
Lines 11883 11968 +85
==========================================
+ Hits 9137 9177 +40
- Misses 2746 2791 +45
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| if single_config: | ||
| metafunc.parametrize("config_filename", | ||
| [Path(single_config).resolve()]) | ||
| if metafunc.config.getoption("--config-list-file"): |
There was a problem hiding this comment.
Why reorder this?
the original thoughts is that --config has higer priority than --config-list-file
There was a problem hiding this comment.
Because the --config option has default settings, if --config is checked first, the logic for --config-list-file will definitely not be executed.
| parser.addoption( | ||
| "--report-dir", | ||
| action="store", | ||
| default="./benchmarks/accuracy", |
There was a problem hiding this comment.
it's better to keep current path, ./, otherwise the ./benchmarks/accuracy is required.
There was a problem hiding this comment.
The code has ensured the creation of the ./benchmarks/accuracy directory: https://github.com/vllm-project/vllm-ascend/blob/main/tests/e2e/singlecard/models/test_lm_eval_correctness.py#L88
2 participants
What this PR does / why we need it?
This PR fix accuracy test related to #2073, users can now perform accuracy tests on multiple models simultaneously and generate different report files by running:
Does this PR introduce any user-facing change?
no
How was this patch tested?