[Thangnguyenvn98]

What

:nvidia: (B200) LM Eval Humming Activation FP8/INT8 TEMPORARY runs 16 GSM8K activation-quant configs (9 FP8 + 7 INT8) as two sequential pytest invocations in one optional pod, well over the 20-minute target. Following the per-model-job convention from #57965, this replaces it with five content-named jobs, one config list each, so the timeline and job names say exactly what ran.

Old

lm-eval-humming-b200-act   "(B200) LM Eval Humming Activation FP8/INT8 TEMPORARY"
├── config-act-fp8.txt   (9 configs)
└── config-act-int8.txt  (7 configs)

New

lm-eval-humming-b200-qwen35-35b          (B200) LM Eval Humming Qwen3.5 35B               config-b200-qwen35-35b.txt
lm-eval-humming-b200-qwen3-30b-act-fp8   (B200) LM Eval Humming Qwen3 30B Activation FP8  config-b200-qwen3-30b-act-fp8.txt
lm-eval-humming-b200-qwen3-30b-act-int8  (B200) LM Eval Humming Qwen3 30B Activation INT8 config-b200-qwen3-30b-act-int8.txt
lm-eval-humming-b200-gpt-oss             (B200) LM Eval Humming GPT-OSS                   config-b200-gpt-oss.txt
lm-eval-humming-b200-small-models        (B200) LM Eval Humming Small Models              config-b200-small-models.txt

Old → new key: lm-eval-humming-b200-act → the five lm-eval-humming-b200-* keys. config-act-fp8.txt and config-act-int8.txt stay unchanged, because the (H100) activation job still runs both. The 16 YAML configs are untouched; coverage is multiset-identical to the two lists on current main, including the AWQ config that runs in both the FP8 and INT8 commands today.

Expected runtime and overhead

Predictions from per-config log timestamps on a recent passing main run, as a share of the old job's pytest time:

job	pytest (share of old)	non-pytest overhead	main overhead cause
Qwen3.5 35B	~32%	<1m	pod setup + checkout
Qwen3 30B Activation FP8	~29%	<1m	pod setup + checkout
Qwen3 30B Activation INT8	~18%	<1m	pod setup + checkout
GPT-OSS	~11%	<1m	pod setup + checkout
Small Models	~11%	<1m	pod setup + checkout

• Slowest job stays under the 20-minute target; timeouts drop 50 → 30.
• Aggregate: about +3 worker-minutes versus the unsharded job (four repeated pod setups), well inside the ≤20 budget.
• The 30B family is the only one split further, along the activation-dtype axis the old label already used; together it would sit just under 20m with no headroom for new configs.

Notes

• Same 16 configs, unchanged. Multiset equality against main's config-act-fp8.txt + config-act-int8.txt checked locally after rebasing onto [Quantization] Enable humming wNaM asymmetric quant (zero_point) with compressed-tensors #46528, which renamed the Qwen3-4B RTN config to Qwen3-4B-mixed-quant-RTN-no-w4a4-humming.yaml; the Small Models list uses the new name.
• Convention: content-named single-replica jobs with per-job config lists, as in [CI] Split (H200) LM Eval Large Models into per-model jobs #57965 — no parallelism, no Shard %N, no indexed shard lists.
• Five block steps instead of one. Each job is optional: true with its own block step (same tradeoff as [CI] Split (H200) LM Eval Large Models into per-model jobs #57965).
• No test files move; these steps have no AMD mirror; no other consumer references lm-eval-humming-b200-act. The (H100) activation job is split separately in a stacked follow-up.

Verification done locally (no GPU)

• check_test_tethering.py --all — pass.
• yaml.safe_load on lm_eval.yaml; multiset coverage check against main; all referenced config YAMLs exist.
• GPU validation via a scoped Buildkite build; results will be posted as a comment.

AI assistance disclosure

This change was prepared with AI assistance (Kimi Code, Claude Code). I reviewed the diff and the evidence cited above.

🤖 Generated with Claude Code

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[Thangnguyenvn98]

Reworked to follow the #57965 convention: dropped the parallelism: 2 + indexed shard-list design in favor of five content-named single-replica jobs, one config list each. The old job and the now-orphaned config-act-int8.txt are removed; config-act-fp8.txt stays for the H100 job. Body updated with the new layout and predictions.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Thangnguyenvn98.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[Thangnguyenvn98]

Validation

Scoped build 90987 on 49369030cf (rebased onto main after #46528), keys = the five new lm-eval-humming-b200-* jobs. Every job's log shows HEAD is now at 49369030cf.

job	configs passed	pytest (share of old)	non-pytest overhead	main overhead cause
Qwen3.5 35B	3 / 3	~31%	<1m	pod setup + checkout
Qwen3 30B Activation FP8	5 / 5	~29%	<1m	pod setup + checkout
Qwen3 30B Activation INT8	3 / 3	~19%	<1m	pod setup + checkout
GPT-OSS	2 / 2	~11%	<1m	pod setup + checkout
Small Models	3 / 3	~11%	<1m	pod setup + checkout

• Coverage: 16 passed, 0 skipped — the same 16 configs the unsharded job runs from config-act-fp8.txt + config-act-int8.txt on main.
• Runtime: slowest job well under the 20-minute target, matching the prediction within a minute per job.
• Cost: about +3.4 worker-minutes versus the unsharded baseline (predicted ~+2.9), inside the ≤20 budget. The increase is the four extra pod setups.
• GPT-OSS's first attempt hit a Buildkite stack_error before it was assigned an agent (it never ran); the automatic retry passed. Not related to this change.

🤖 Generated with Claude Code

[Thangnguyenvn98]

Closing in favor of automatic runtime sharding: CI now plans each step's shards when the pipeline is generated, from main's per-file timings (automatic_shard: true), so hand-split jobs are no longer the approach. This step isn't a fit for automatic sharding yet (see #61999 for why); if its runtime warrants it, it will be enrolled through automatic_shard once its commands allow.