[CI] Speed up quantization test group - #53291
Conversation
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
fxmarty-amd
changed the title
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
|
Documentation preview: https://vllm--53291.org.readthedocs.build/en/53291/ |
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
| excepted_value=10.6, | ||
| ), | ||
| AccuracyTestConfig( | ||
| model_name="fxmarty/qwen_1.5-moe-a2.7b-mxfp4", excepted_value=12.45 |
There was a problem hiding this comment.
This test case is dropped as covered by other gsm8k yaml configs.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
@fxmarty-amd How much is the speedup? Can you share here a mi300/mi355 before and after table? |
|
/amd-ci run |
|
✅ Triggered Buildkite AMD CI #12348 for commit |
It was ongoing, done now, see I could break it down per-test, if necessary. |
|
Remaining slow tests (some addressed in #53585, others I'd prefer to do later): |
|
✅ Triggered Buildkite AMD CI #12364 for commit |
BowenBao
left a comment
BowenBao
left a comment
There was a problem hiding this comment.
Changes seem fair to me. The unittests affected mostly are doing simple checks that don't need vllm_runner, and the ones that need are covered by model eval tests.
My only concern is over robustness of load_model_without_vllm_runner
There was a problem hiding this comment.
nice, we have more coverage with shorter test time!
There was a problem hiding this comment.
actually, the nvfp4/mxfp4 tests
122.48s call tests/quantization/test_quark.py::test_nvfp4_wikitext_correctness[1]
103.76s call tests/quantization/test_quark.py::test_ocp_mx_wikitext_correctness[tp_size:1-config:AccuracyTestConfig(model_name='fxmarty/qwen1.5_moe_a2.7b_chat_w_fp4_a_fp6_e2m3', excepted_value=11.3)]
97.75s call tests/quantization/test_quark.py::test_ocp_mx_wikitext_correctness[tp_size:1-config:AccuracyTestConfig(model_name='fxmarty/qwen1.5_moe_a2.7b_chat_w_fp6_e3m2_a_fp6_e3m2', excepted_value=10.6)]
89.16s call tests/quantization/test_quark.py::test_ocp_mx_wikitext_correctness[tp_size:1-config:AccuracyTestConfig(model_name='fxmarty/qwen_1.5-moe-a2.7b-mxfp4', excepted_value=12.45)]
are moved to respectively:
Line 1426 in 796822d
Line 3379 in 796822d
and now use gsm8k instead of wikitext, that is not available in https://github.com/vllm-project/vllm/tree/main/tests/evals, unfortunately.
Alternatively, we could have a test_quark_eval.py that uses actual lm-eval lib + wikitext, and register it as part of the groups above.
For the record,
cat > tests/evals/gsm8k/configs/tests_refactor.txt << 'EOF'
Qwen-1.5-MOE-W-MXFP4-A-MXFP6.yaml
Qwen-1.5-MOE-W-MXFP6-A-MXFP6.yaml
Qwen3-30B-A3B-NVFP4-quark.yaml
EOF
and
pytest -s -v tests/evals/gsm8k/test_gsm8k_correctness.py --config-list-file=tests/evals/gsm8k/configs/tests_refactor.txt
takes:
- on 1xMI350:
3 passed, 14 warnings in 345.70s (0:05:45) - on 1xMI300:
3 passed, 14 warnings in 429.37s (0:07:09)
There was a problem hiding this comment.
this coverage is moved to model eval test
There was a problem hiding this comment.
Yes, it is moved to test_online.py parametrized test
| dtype="bfloat16", | ||
| enforce_eager=True, | ||
| ) as llm: | ||
| output = llm.generate_greedy(["The capital of France is"], max_tokens=4) |
There was a problem hiding this comment.
This is not checking generate output for reasonable content anyways (only does assert output), seems fair to simplify the test for shorter run time.
There was a problem hiding this comment.
Yes.
In this PR, we call a single forward with attention/KV cache instead of calling generate. The generate output was not checked in the modified tests.
Note that we use monkeypatch.setattr(Attention, "forward", lambda _, q, k, v: q.contiguous()) to be able to call model forward without KV cache allocation.
I think for simple integration tests it is reasonable
| model.load_weights = load_weights | ||
|
|
||
|
|
||
| def load_model_without_vllm_runner( |
There was a problem hiding this comment.
Would this potentially break if vllm runner is updated?
There was a problem hiding this comment.
Yes, right, it duplicates DefaultModelLoader.load_model, which is not ideal.
Simplified in 9ed1b8f to rely purely on DefaultModelLoader.load_model instead, which this load_model_without_vllm_runner function wraps.
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
|
Let's retrigger the CI once approved by maintainers. |
|
/ci run |
|
✅ Triggered Buildkite CI #86387 for commit |
|
/amd-ci run |
|
✅ Triggered Buildkite AMD CI #12479 for commit |
|
@fxmarty-amd this nan failure definitely seems related |
AndreasKaratzas
left a comment
AndreasKaratzas
left a comment
There was a problem hiding this comment.
Let's wait for green CI first -- @fxmarty-amd check the failure on NV that is there.
|
I am not able to reproduce this failure on H100 and H200, this passes locally. It definitely seem related given that monkeypatch.setattr(Attention, "forward", lambda _, q, k, v: q.contiguous())
input_ids = torch.tensor([1, 2, 3, 4], device=DEVICE_TYPE)
positions = torch.arange(input_ids.numel(), device=DEVICE_TYPE)
with (
set_current_vllm_config(vllm_config),
set_forward_context(None, vllm_config, num_tokens=input_ids.numel()),
):
hidden_states = model(input_ids, positions, None)
logits = model.compute_logits(hidden_states)
assert torch.isfinite(logits).all()Could we try Alternatively, is there an easy way to rerun in the exact same docker environment as CI? I'll try to look for that. |
|
This pull request has merge conflicts that must be resolved before it can be |
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
|
I can not reproduce either on H200 using: export IMAGE=public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:8b4d51de5567cf952282d25dd0655c3ce47c25b3
docker pull "$IMAGE"
docker run --rm -td --init --name felix_repro_h200 \
--gpus all \
--volume /dev/shm:/dev/shm \
--env HF_TOKEN \
--env NCCL_CUMEM_HOST_ENABLE=0 \
--env PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False \
--env VLLM_USAGE_SOURCE=ci-test \
--env VLLM_USE_V2_MODEL_RUNNER=0 \
--env VLLM_GPU_SYNC_CHECK=error \
-v $(pwd)/scratch/felmarty:/felmarty \
"$IMAGE" \
/bin/bash
pip uninstall vllm
apt install curl wget
pip install setuptools_scm setuptools_rust tblib
pip install torchao==0.17.0 --index-url https://download.pytorch.org/whl/cu130
pip install conch-triton-kernels
git config --global --add safe.directory /felmarty/repos/vllm
VLLM_USE_PRECOMPILED=1 pip install -vvv --editable . --no-build-isolation
VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest tests/quantization/test_quark.py -vvvvv -s -k "test_quark_int8_w8a8"
VLLM_TEST_FORCE_LOAD_FORMAT=auto \
pytest tests/quantization/ -vvvvv -s \
--ignore quantization/test_blackwell_moe.py \
-k "not test_compressed_tensors_w4a8_fp8" \
--shard-id=3 --num-shards=4(from ) |
|
/ci run |
|
✅ Triggered Buildkite CI #86639 for commit |
Signed-off-by: Felix Marty <Felix.Marty@amd.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com>
Signed-off-by: Felix Marty <Felix.Marty@amd.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com>
4 participants
Disclosure
AI assistance was used. The changes were reviewed and tested manually.
Purpose
Speed up quantization test group, this PR fixes #51142.
After all models are cached / compilation done (i.e., second run):
main(9f295fe)main(d9fbe52)main(d9fbe52)main(d9fbe52)Common changes to many tests:
hf_overrideonnum_hidden_layers(to a smaller number, e.g. 3)Three accuracy tests are move to
vllm/.buildkite/test-amd.yaml
Line 1438 in 79bb395
vllm/.buildkite/test-amd.yaml
Line 3378 in 79bb395
Remaining:
test_cpu_offload_*=> to be done in an other PR.test_online_quant_peak_meminfp8.py=> likely deprecate/remove online quantization as part of--quantization fp8in a separate PR, see [Cleanup] Remove online quantization support infp8.pyin favor of online shorthands #53585See e.g. on
main+ MI355 https://buildkite.com/vllm/amd-ci/builds/12275/list?sid=01a01e66-70c8-42cb-8d84-bfea236686e4&tab=output that has:Test Plan
vllm/.buildkite/test-amd.yaml
Line 3824 in 0ecc284
run locally.
Additional gsm8k tests, run through:
and
Test Results
VLLM_TEST_FORCE_LOAD_FORMAT=auto HIP_VISIBLE_DEVICES=1 nohup pytest -vvvvv -s --durations=50 tests/quantization/ --ignore tests/quantization/test_blackwell_moe.pyon 1xMI350, after all model download:512 passed, 67 skipped, 28 warnings in 1317.00s (0:21:56)515 passed, 71 skipped, 23 warnings in 3369.39s (0:56:09)