Enable pytest and yaml style accuracy test#2073
Enable pytest and yaml style accuracy test#2073Yikun merged 36 commits intovllm-project:mainfrom wxsIcey:new_acc_test
Conversation
github-actions
bot
added
documentation
wxsIcey
added
accuracy-new-test
ready-for-test
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2073 +/- ##
==========================================
+ Coverage 74.41% 75.05% +0.63%
==========================================
Files 100 103 +3
Lines 11208 11355 +147
==========================================
+ Hits 8340 8522 +182
+ Misses 2868 2833 -35
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
wxsIcey
added
accuracy-new-test
ready-for-test
Yikun
left a comment
Yikun
left a comment
There was a problem hiding this comment.
LGTM, let's replace: https://github.com/vllm-project/vllm-ascend/blob/main/.github/workflows/accuracy_test.yaml and fix nits.
.github/actionlint.yaml
Outdated
| - linux-aarch64-a2-1 | ||
| - linux-aarch64-a2-2 |
| env_config): | ||
| env = Environment(loader=FileSystemLoader('.')) | ||
| template = env.get_template( | ||
| "./tests/e2e/singlecard/models/report_template.md") |
There was a problem hiding this comment.
Could we use relative path to make sure we can execute the pytest in any path?
| # test will be triggered when tag '*-accuracy-test' & 'ready-for-test' or workflow_dispatch job | ||
| if: >- | ||
| ${{ | ||
| contains(github.event.pull_request.labels.*.name, 'accuracy-new-test') && |
There was a problem hiding this comment.
| contains(github.event.pull_request.labels.*.name, 'accuracy-new-test') && | |
| contains(github.event.pull_request.labels.*.name, 'accuracy-test') && |
| help="Path to the model config YAML file", | ||
| ) | ||
| parser.addoption( | ||
| "--report_output", |
There was a problem hiding this comment.
I think we can change this to report_path and the name should be same as yaml.model_name, then --config-list-file will make sense when specify report_path
|
Let's merge this first, and do some fixes in a separate PR when adding |
4 participants
What this PR does / why we need it?
This PR enabled pytest and yaml style accuracy test, users now can enable accuracy test by running:
Closes: #1970
Does this PR introduce any user-facing change?
no
How was this patch tested?