[arathi-hlab]

Motivation

sglang.test.run_eval (SGLang's legacy simple-evals scorers) is deprecated in favor of sgl-eval (#41280), and every backend's test extra already pins sgl-eval. XPU's [test] extra did not include sgl-eval, and its GSM8K nightly tests still scored through run_eval. lm-eval itself is not deprecated (CUDA's extra keeps it for lm_eval_kit), so XPU keeps it and adds sgl-eval alongside, matching CUDA.

Modifications

Deps

• python/pyproject_xpu.toml [test]: keep lm-eval[api]>=0.4.9.2, add sgl-eval==0.1.3 (same pin as pyproject.toml) and antlr4-python3-runtime==4.9.3, which sgl-eval's math_verify grader requires. Verified in both nightly images: lm-eval 0.4.13 and sgl-eval 0.1.3 import together, and pip check reports no conflict between them.
• nightly-test-intel.yml (4 jobs) and nightly-test-intel-wheel.yml: install sgl-eval (covering the window before the nightly image rebuild) instead of lm-eval==0.4.9. That pin downgraded the image's lm-eval 0.4.13 to 0.4.9, which fails to import with the image's transformers 5.19 (no attribute AutoModelForVision2Seq); the image's own lm-eval is kept.

GSM8K tests → sgl-eval

• simple_eval_gsm8k_xpu_mixin.py: score with run_sgl_eval; add optional temperature / top_p / chat_template_kwargs.
• test_xpu_dcp_gsm8k.py: gsm8k_backend = "sgl_eval", sampled thinking (temperature 1.0, top_p 0.95, 8192 tokens), since greedy thinking loops.
• Llama-3.1-8B: max_tokens 512 → 1024, threshold 0.80 → 0.77.
• est_time updated to measured runtimes: DCP 900 → 1800, Gemma-4 2400 → 3000, Nemotron-3 2400 → 4800, Qwen3.5-9B 2400 → 900, Qwen3.5-35B 2400 → 3300.
• nightly-test-intel.yml: the overlay step also copies the in-tree mixin into the image, because the kernel-main jobs import it from the image venv rather than the checkout.

Qwen3.5-9B and Qwen3.5-35B-A3B (20 examples each). sgl-eval's gsm8k is 0-shot with a strict \boxed{} grader ("answer (and only answer)"). With greedy thinking, both models reach the right answer and then loop re-checking the box format until max_tokens.

• Qwen3.5-9B: thinking disabled (chat_template_kwargs={"enable_thinking": False}), greedy, max_tokens 8192 → 2048.
• Qwen3.5-35B-A3B: sampled thinking (temperature 1.0, top_p 0.95, max_tokens 16384, --max-total-tokens 131072). With thinking off it answers with a bare \boxed{N} and no working (0.42).
• Thresholds are the lowest score measured on B70, to be raised only on new data: 9B 0.95 (3 runs × 3 classes, 0.95–1.0), 35B 0.85 (runs of 0.85 and 0.90). The recurring misses are gsm8k-12 (every model answers 12, expected 13) and 1–2 thinking loops that reach the 16384 cap.

Accuracy (B70, image latest-kernel-main 6a96c63a, oneAPI DLE 2026.1.3)

Test	Suite	run_eval (10-27 nightly)	sgl-eval (this PR)	Threshold
Qwen3-30B-A3B	4-gpu	0.98	0.98 (1003 s)	0.90
Gemma-4-26B-A4B	4-gpu	0.98	0.95 (2905 s)	0.90
Nemotron-3-Nano-30B-A3B	4-gpu	0.97	0.835 (4596 s)	0.72
Llama-3.1-8B-Instruct	2-gpu	0.85	0.81 / 0.82	0.77
DCP Qwen3.5-4B (TP2/DCP2)	2-gpu	0.885	0.87 (1663 s)	0.80
Qwen3.5-9B (20 ex, thinking off)	4-gpu	0.96	0.95–1.0 (×9, ~65–97 s each; file ~430 s) ¹	0.95
Qwen3.5-35B-A3B (20 ex, sampled thinking)	4-gpu	0.96	0.85 / 0.90 (2651 s / 2577 s) ¹	0.85

¹ Measured 2026-10-29 on image latest-kernel-main e0a3e853866e.

Other XPU suites don't need changes: the 8-gpu suite registers no tests; the wheel nightly and PR CI suites use neither lm-eval nor sgl-eval; test_spec_eagle_intel_xpu.py (1-gpu) goes through the shared spec kit, which still calls run_eval (shared with CUDA, left for upstream).

Note: sgl-eval==0.1.3 caps datasets<5 and pandas<3, so installing it moves the image's datasets 5.1.0 → 4.8.5 and pandas 3.1.0 → 2.3.3 (the same as CUDA's [test] extra). Nothing broke in the runs above.

Checklist

• Migrated tests re-run on XPU (B70) with this branch's mixin overlaid on the latest image
• Scorer routing checked for every mixin test class (all → run_sgl_eval; the logged sgl-eval gsm8k generation: line matches each test's settings)
• pre-commit passes

---

CI States

Latest PR Test (Base): ⏳ Run #40997251822
Latest PR Test (Extra): ❌ Run #40997251848
Latest PR Test (AMD ROCm 10): ⏳ Run #40997251860

[mingfeima]

/rerun-failed-ci