Skip to content

[CI/Build] Temporary fix to LM Eval Small Models #28324

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Nov 9, 2025
Merged

[CI/Build] Temporary fix to LM Eval Small Models #28324

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bc767d4
update tests
zhewenl Nov 7, 2025
d418338
update test
zhewenl Nov 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .buildkite/test-pipeline.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1224,7 +1224,7 @@ steps:
- pytest -v -s tests/compile/test_fusions_e2e.py::test_tp2_attn_quant_allreduce_rmsnorm
- pytest -v -s tests/distributed/test_context_parallel.py
- CUDA_VISIBLE_DEVICES=1,2 VLLM_ALL2ALL_BACKEND=deepep_high_throughput VLLM_USE_DEEP_GEMM=1 VLLM_LOGGING_LEVEL=DEBUG python3 examples/offline_inference/data_parallel.py --model Qwen/Qwen1.5-MoE-A2.7B --tp-size=1 --dp-size=2 --max-model-len 2048
- pytest -v -s tests/v1/distributed/test_dbo.py
- pytest -v -s tests/v1/distributed/test_dbo.py

##### B200 test #####
- label: Distributed Tests (B200) # optional
Expand Down
5 changes: 4 additions & 1 deletion tests/evals/gsm8k/configs/Qwen1.5-MoE-W4A16-CT.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,7 @@ model_name: "nm-testing/Qwen1.5-MoE-A2.7B-Chat-quantized.w4a16"
accuracy_threshold: 0.45
num_questions: 1319
num_fewshot: 5
max_model_len: 4096
max_model_len: 4096
# Duo stream incompatabilbe with this model: https://github.com/vllm-project/vllm/issues/28220
env:
VLLM_DISABLE_SHARED_EXPERTS_STREAM: "1"
4 changes: 3 additions & 1 deletion tests/evals/gsm8k/test_gsm8k_correctness.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,9 +62,11 @@ def test_gsm8k_correctness_param(config_filename, tp_size):
str(tp_size),
]

env_dict = eval_config.get("env", None)

# Launch server and run evaluation
with RemoteOpenAIServer(
eval_config["model_name"], server_args, max_wait_seconds=480
eval_config["model_name"], server_args, env_dict=env_dict, max_wait_seconds=480
) as remote_server:
server_url = remote_server.url_for("v1")

Expand Down