[Thangnguyenvn98]

What

LM Eval TurboQuant KV Cache runs four GSM8K accuracy evals — one per kv-cache-dtype config — serially in a single H200 MIG job measuring 39–51m of pytest against a 55m timeout. This replaces it with four content-named jobs, one config each, so the timeline and job names say exactly what ran. Same per-config split as the merged (H200) LM Eval Large Models change (#57965).

Old

tests/evals/gsm8k/configs/
└── models-turboquant.txt        # 4 configs, one job: 39–51m pytest, timeout 55m

New

lm-eval-turboquant-k8v4    (H200 MIG 18GB) LM Eval TurboQuant k8v4    models-turboquant-k8v4.txt
lm-eval-turboquant-t4nc    (H200 MIG 18GB) LM Eval TurboQuant t4nc    models-turboquant-t4nc.txt
lm-eval-turboquant-k3v4nc  (H200 MIG 18GB) LM Eval TurboQuant k3v4nc  models-turboquant-k3v4nc.txt
lm-eval-turboquant-t3nc    (H200 MIG 18GB) LM Eval TurboQuant t3nc    models-turboquant-t3nc.txt

Old → new key: lm-eval-turboquant-kv-cache → the four lm-eval-turboquant-* keys. models-turboquant.txt is removed; the four YAML configs are untouched, each in exactly one list. Timeout drops 55 → 30 (~1.7× the slowest measured config).

Notes

• Same cases, unchanged configs. The four Qwen3-4B-TQ-*.yaml configs are not modified; coverage is 4/4 configs exactly once, verified locally. models-turboquant.txt's only CI consumer was this job.
• Per-config jobs over parallelism: 4. An earlier revision of this PR used shard-N list files; revised to named per-config jobs to match [CI] Split (H200) LM Eval Large Models into per-model jobs #57965 — timeline-readable names and per-config flake isolation (retrying one config no longer re-runs the other three), at the cost of four block/job entries instead of one job with replicas.
• Resource note: the four MIG jobs can run concurrently; each still fits a single MIG slice.
• No AMD mirror exists on this step and .buildkite/test-amd.yaml is untouched; only the TurboQuant lane changes.

Validation

Scoped build #89258 measured the four configs individually (12.4–15.9m pytest, 14.4–17.2m wall each, all green) — those per-config durations carry over unchanged since the configs and commands are identical; only the job boundary moved. A fresh scoped build on the revised head is linked in the comments.

Checks run locally: yaml.safe_load on the edited pipeline YAML; a script verifying each list file resolves to an existing config and that the four TurboQuant configs are claimed exactly once (matching the old models-turboquant.txt); no remaining references to the old key or list file. pytest --collect-only is not runnable locally (no GPU/torch env); the scoped build exercises all four jobs.

Contribution checklist

• Not a duplicate: no open PR touches this job or these configs (searched turboquant, lm eval shard, and this branch name).
• This change is AI-assisted (noted per the contribution policy); commit trailers carry the attribution.

[Thangnguyenvn98]

Validation

Scoped build #89258 on the PR head 1cc70c8 (VLLM_CI_ONLY_STEP_KEYS=["lm-eval-turboquant-kv-cache"]) — all 4 shards green, one kv-cache-dtype config each:

Shard	Config	pytest	Wall
1	Qwen3-4B-TQ-k8v4	12.4m	14.4m
2	Qwen3-4B-TQ-t4nc	15.3m	17.2m
3	Qwen3-4B-TQ-k3v4nc	15.7m	16.9m
4	Qwen3-4B-TQ-t3nc	15.9m	17.2m

Slowest shard 17.2m wall vs the unsharded job's ~48m p90 baseline (30d green runs) and inside the ~20m prediction in the PR body; every shard sits well under the new 30m timeout. The trade is the standard one for shard replicas: total GPU-minutes rise (4 × ~17m ≈ 69m vs ~45m) in exchange for ~2.8× faster time-to-signal and no more living one slow run from the 55m timeout.

AI assistance used (Kimi Code) for the validation run and this summary.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[Thangnguyenvn98]

Rebased onto latest main (2bdbbc80), new head 30aa0f13; diff unchanged (same 6 files). No new build was run for the rebase — the validation above stands on 1cc70c8, and the merge-time /ci run covers the rebased head.

[Thangnguyenvn98]

Revised per the approach in #57965: the parallelism: 4 shard is replaced with four content-named jobs (lm-eval-turboquant-k8v4 / -t4nc / -k3v4nc / -t3nc), one config list each, so the timeline and job names say exactly what ran. Same four configs, unchanged, once each; timeout 55 → 30. PR body rewritten to match; branch squashed to a single commit on latest main (bdecc005).

New scoped validation build: #90488 with VLLM_CI_ONLY_STEP_KEYS set to the four new step keys. Per-config durations measured in #89258 (12.4–15.9m pytest each) carry over since only the job boundary moved.

[Thangnguyenvn98]

Validation (per-config split, final head)

Scoped build #90488 on head bdecc005 (VLLM_CI_ONLY_STEP_KEYS = the four new job keys) — all four jobs green, one config each:

job	pytest	running wall	non-pytest overhead	main overhead cause
TurboQuant k8v4	10.5m	11.9m	1.4m	image pull (~45s) + checkout/container start
TurboQuant t4nc	12.6m	14.0m	1.4m	image pull (~46s) + checkout/container start
TurboQuant k3v4nc	13.7m	15.4m	1.7m	image pull (~44s) + checkout/container start
TurboQuant t3nc	13.6m	15.0m	1.4m	image pull (~33s) + checkout/container start

Coverage. Each job ran exactly its one config case (4 × 1 passed, 0 skipped) — the same four cases the unsharded job runs on main (4 passed, build #89221), once each.

vs predictions. All four jobs came in faster than the #89258 measurements quoted in the PR body (12.4–15.9m pytest / 14.4–17.2m wall). Slowest job: 13.7m pytest / 15.4m wall — inside the ~20m target and well under the 30m timeout.

Worker-minutes. Baseline 41.6m wall (89221, unsharded; recent green range 41.6–52.7m) → 56.3m summed after the split, +14.7m aggregate, within the ≤20 extra budget. ~3.5m of the increase is the three repeated container setups (~1.4–1.7m per job vs 2.2m paid once before); the rest is per-case runtime variance, which runs ±3m on this lane (per-case times in the PR body were measured at 12.4–14.6m across five main builds).

AI assistance used (Kimi Code) for the validation run and this summary.