Skip to content

[Test] Add some gsm8k configs for hybrid models. #35406

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tdoublep wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[Test] Add some gsm8k configs for hybrid models. #35406

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions tests/evals/gsm8k/configs/hybrid/Qwen3-Next-FP8-TP4-MTP-Align.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
model_name: "Qwen/Qwen3-Next-80B-A3B-Instruct-FP8"
Copy link
Copy Markdown
Collaborator

@robertgshaw2-redhat robertgshaw2-redhat Feb 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we just have this one? or is it also useful to test the non spec decoding / non-prefix caching case

Copy link
Copy Markdown
Member Author

@tdoublep tdoublep Feb 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's useful yeah. I put the 3 configs in a "hybrid" folder so as not to pollute what's there.

Alternatively, if we want to keep the number of configs to a minimum, maybe it could be useful to be able to pass additional overrides when passing the configs to pytest (if that isn't possible already).

accuracy_threshold: 0.85
num_questions: 1319
num_fewshot: 5
server_args: >-
--max-model-len 4096
--tensor-parallel-size 4
--speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'
--enable-prefix-caching
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot Feb 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

The configuration enables prefix caching (--enable-prefix-caching) for the Qwen3Next model. Hybrid models like Qwen3Next do not support prefix caching, which will cause a ValueError during engine initialization. Please remove this argument.

8 changes: 8 additions & 0 deletions tests/evals/gsm8k/configs/hybrid/Qwen3-Next-FP8-TP4-MTP.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
model_name: "Qwen/Qwen3-Next-80B-A3B-Instruct-FP8"
accuracy_threshold: 0.85
num_questions: 1319
num_fewshot: 5
server_args: >-
--max-model-len 4096
--tensor-parallel-size 4
--speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'
7 changes: 7 additions & 0 deletions tests/evals/gsm8k/configs/hybrid/Qwen3-Next-FP8-TP4.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
model_name: "Qwen/Qwen3-Next-80B-A3B-Instruct-FP8"
accuracy_threshold: 0.85
num_questions: 1319
num_fewshot: 5
server_args: >-
--max-model-len 4096
--tensor-parallel-size 4
3 changes: 3 additions & 0 deletions tests/evals/gsm8k/configs/hybrid/models-h100.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Qwen3-Next-FP8-TP4.yaml
Qwen3-Next-FP8-TP4-MTP.yaml
Qwen3-Next-FP8-TP4-MTP-Align.yaml