[Thangnguyenvn98]

What

(H200) LM Eval Large Models runs six GSM8K accuracy evals — DeepSeek-R1/V3.2 in TP and DP forms plus two Nemotron-3-Super-120B variants — serially in a single 8xH200 job measuring 41–75 minutes on recent main builds. This replaces it with five content-named jobs, one config each for the four DeepSeek evals and both Nemotron variants together, so the timeline and job names say exactly what ran.

Old

lm-eval-large-models-8xh200   "(H200) LM Eval Large Models"      38.4–71.0m pytest / 41–75m wall
└── models-h200.txt
    ├── DeepSeek-R1-TP.yaml                       12.8–18.6m
    ├── DeepSeek-R1-DP.yaml                        6.6–14.9m
    ├── DeepSeek-V3.2-TP.yaml                      7.8–13.2m
    ├── DeepSeek-V3.2-DP.yaml                      4.1–12.0m
    ├── Nemotron-3-Super-120B-A12B-BF16.yaml       4.2– 8.6m
    └── Nemotron-3-Super-120B-A12B-FP8.yaml        2.8– 4.8m
mirror: amd-lm-eval-large-models-8xh200  "(MI300) LM Eval Large Models"  15.2–17.7m pytest

New

lm-eval-deepseek-r1-tp-8xh200      (H200) LM Eval DeepSeek R1 TP      models-h200-deepseek-r1-tp.txt      18.6m worst
lm-eval-deepseek-r1-dp-8xh200      (H200) LM Eval DeepSeek R1 DP      models-h200-deepseek-r1-dp.txt      14.9m worst
lm-eval-deepseek-v32-tp-8xh200     (H200) LM Eval DeepSeek V3.2 TP    models-h200-deepseek-v32-tp.txt     13.2m worst
lm-eval-deepseek-v32-dp-8xh200     (H200) LM Eval DeepSeek V3.2 DP    models-h200-deepseek-v32-dp.txt     12.0m worst
lm-eval-nemotron-3-super-8xh200    (H200) LM Eval Nemotron 3 Super    models-h200-nemotron-3-super.txt    13.4m worst
amd-lm-eval-large-models           (MI300) LM Eval Large Models       models-mi3xx.txt (unchanged)        15.2–17.7m

Old → new keys: lm-eval-large-models-8xh200 → the five lm-eval-*-8xh200 keys; the mirror's generated key amd-lm-eval-large-models-8xh200 → amd-lm-eval-large-models (standalone step, same command, same env, same source_file_dependencies; the mirror block was dropped because its coverage no longer matches any single split job). models-h200.txt is removed; the six YAML configs are untouched, each in exactly one list.

Expected runtime and overhead

Predictions from per-case log timestamps on main builds 89998 / 90039 / 90097 / 90154 (worst-of-4 per config; measured, wall estimated):

job	pytest (meas. worst)	running wall (est.)	non-pytest overhead (est.)	main overhead cause
DeepSeek R1 TP	18.6m	~21m	~2.5m	container start + image pull
DeepSeek R1 DP	14.9m	~17.5m	~2.5m	container start + image pull
DeepSeek V3.2 TP	13.2m	~16m	~2.5m	container start + image pull
DeepSeek V3.2 DP	12.0m	~14.5m	~2.5m	container start + image pull
Nemotron 3 Super	13.4m	~16m	~2.5m	container start + image pull
MI300 (unchanged)	15.2–17.7m	—	—	—

• Slowest job: 18.6m pytest, inside the 20m target (was 71.0m worst / 120m timeout; timeouts drop 120 → 35).
• Aggregate: ~72m worker-minutes before → 82m after (est.), **+10 vs the ≤20 budget**. Model loading and eval stay inside pytest per case; the increase is four repeated container setups (~1.2–4.7m measured on the unsharded job).
• Per-case timings vary run to run (e.g. DeepSeek-R1-TP 12.8–18.6m), so job walls carry ±3m uncertainty.

Notes

• Same cases, unchanged configs. The six YAMLs are not modified; coverage is 6/6 configs exactly once, verified locally. models-h200.txt's only CI consumer was this job (the qwen4_exp and gpt_oss suites use their own config directories).
• Five block steps instead of one. Each job is optional: true with its own block step, so manually running this lane takes five unblocks instead of one. The alternative (parallelism: 5 with shard-N lists, one click) was dropped in favor of timeline-readable job names; happy to switch if reviewers prefer the single unblock.
• Resource note: the five 8-GPU jobs can run concurrently when unblocked (40 H200s vs 8 before); each still fits one node.
• Per-config jobs also isolate flakes: retrying the Nemotron job no longer re-runs DeepSeek.

AI assistance: this change was prepared with AI assistance (Kimi Code). Not a duplicate: no open PR touches this job or these configs (checked open PRs for lm_eval/gsm8k/models-h200). Local verification: pipeline generator render shows the five named jobs on the H200 queue and the standalone MI300 job with its command unchanged; all list entries resolve to existing config files; yaml.safe_load passes on the edited YAML. GPU validation via the scoped Buildkite builds linked in the comments.

[Thangnguyenvn98]

Validation — builds #90228, #90261, baseline #90243

Commit a287ff6444 (refs/pull/57965/head, on main 8b98b7d). Scoped to lm-eval-large-models-8xh200 + amd-lm-eval-large-models-8xh200. Two validation runs, both green: all 5 shards + the mirror passed on the first attempt, no empty shards. Coverage reconciles exactly with the unsharded baseline: 6/6 configs pass on the primary lane; the mirror lane runs 3 passed / 5 platform-skipped, identical to main. 90228's numbers were polluted by cold weight caches on first-touch nodes; 90261 (warm) is the cleaner measurement and agrees with 90228 on the structural overhead below.

Measured per replica, 90261 (pytest | running wall | non-pytest overhead | main overhead cause):

shard	config	pytest	wall	overhead	main overhead cause
0	DeepSeek-R1-TP	17.2m	18.0m	0.8m	container start
1	DeepSeek-R1-DP	17.6m	18.4m	0.8m	container start
2	DeepSeek-V3.2-TP	20.2m	21.0m	0.8m	container start
3	DeepSeek-V3.2-DP	18.2m	19.0m	0.8m	container start
4	Nemotron-3-Super BF16+FP8	13.3m	14.1m	0.8m	container start
mirror	models-mi3xx list	16.8m	21.9m	5.1m	image pull + ROCm setup

• Baseline (same-day, unsharded, build 90243 on main): 69.7m pytest / 70.7m wall. W_after = 90.5m → +19.8 extra worker-minutes vs the ≤20 budget (passes, with no headroom).
• Slowest replica: 20.2m pytest (shard 2) — 0.2m over the 20m target; historical worst for that config is 13.2m (prediction was ≤18.6m). Rationale for 5 shards stands: any pairing of the four large configs exceeds 20m (e.g. DeepSeek-V3.2-TP + -DP would be ~31m), and a single config's runtime cannot be reduced by adding replicas.
• Where the extra ~20m goes (fully attributed): DeepGEMM JIT warmup is cached per container. The unsharded job pays it 5.0/2.0/0/0m across its four sequential DeepSeek configs; sharded, every DeepSeek shard pays a cold 4–5m (measured 5.0/5.0/4.0/4.0) — ≈ +11m of repeated compilation. The rest is 4 extra container starts (≈ +3m) and per-config weight loads no longer benefiting from the previous config's page cache (R1-DP +2.6m, V3.2-DP +6.2m on 90261; same-day baseline on the same agent shows 12.6m/9.4m startups vs 15.2m/15.6m sharded).
• Mirror lane: unchanged command, pinned to parallelism: 1; 16.8m pytest is inside its own recent unsharded range (15.2–17.7m), well under its 65m timeout.
• The ~5m first-case DeepGEMM warmup is also paid by the unsharded job on every run — a shared/persistent JIT cache for these jobs would save more than this PR's entire overhead; happy to file that as a follow-up if there's interest.

Verdict: aggregate cost is within budget (+19.8 of 20); the slowest-shard target is marginally exceeded (20.2m vs 20m) solely because of the repeated per-shard DeepGEMM warmup. Requesting an explicit ruling on that 0.2m overage rather than relaxing the target silently — the alternatives (4 shards, or pairing configs) all put a shard past 23m.

[Thangnguyenvn98]

Validation — build #90304 (named per-model jobs)

Commit 73577f84c6 (refs/pull/57965/head, rebased onto main 382970e). Scoped to the five new jobs + amd-lm-eval-large-models. All six passed on the first attempt — no retries, no empty selections. Coverage reconciles exactly with the unsharded baseline: each named job ran its expected config(s) and only those (1 passed per DeepSeek job, 2 passed for Nemotron, mirror 3 passed / 5 platform-skipped — identical to main).

Measured per job (pytest | running wall | non-pytest overhead | main overhead cause):

job	pytest	wall	overhead	main overhead cause
DeepSeek R1 TP	18.2m	20.4m	2.2m	container start + image pull
DeepSeek R1 DP	18.6m	20.9m	2.3m	container start + image pull
DeepSeek V3.2 TP	20.4m	21.2m	0.8m	container start
DeepSeek V3.2 DP	18.4m	20.7m	2.3m	container start + image pull
Nemotron 3 Super	12.6m	13.5m	0.9m	container start
(MI300) LM Eval Large Models	15.2m	22.5m	7.3m	image pull + ROCm setup

• Baseline (same-day unsharded, build 90243 on main): 69.7m pytest / 70.7m wall → Σwall 96.7m, +26.0 extra worker-minutes this run (+19.8 on the earlier identical-workload run 90261; run-to-run swing is pool/load-dependent). The structural, layout-inherent part is ≈ +14m: per-job DeepGEMM JIT warmup (the unsharded job amortizes it 5/2/0/0m across the four sequential DeepSeek configs; each named job pays a cold 4–5m) plus four repeated container setups.
• Slowest job: DeepSeek V3.2 TP at 20.4m pytest — 0.4m over the 20m target on this run (18.6m worst in the pre-split history; the overage is the cold DeepGEMM warmup, same finding as the first validation).
• Mirror lane unchanged in behavior: 15.2m pytest, inside its own recent range (15.2–17.7m), far under its 65m timeout.
• Same request as on the shard-layout validation: the runtime target is marginally exceeded only because of the repeated per-job DeepGEMM warmup. A shared/persistent JIT cache for these eval jobs would recover ~11m of the overhead (and ~7m/run on today's unsharded main job) — happy to file that as a follow-up.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.