[CI] Shard Humming A100 eval

[khluu]

Why

lm-eval-humming-f16-a100 took 64.3 minutes in full CI build 83851. Its 14 model configurations ran serially even though each evaluation is independent.

This does not duplicate an open pull request. Open PRs were searched by the exact step key and Humming/A100 sharding terms before implementation.

What changed

• run the step with three Buildkite jobs
• partition all 14 configurations into three explicit A100 lists using observed build 83851 per-config durations
• keep each configuration in exactly one shard

The measured baseline loads are 20.20, 21.47, and 21.60 minutes of test work, leaving headroom below the 30-minute target.

Validation

• baseline model evaluation: 14/14 configurations passed in build 83851
• config partition check: 14/14 unique entries, no omissions or duplicates, all referenced YAML files exist
• selected-step pipeline generation: three parallel A100 jobs with the image-build dependency
• targeted build 83894: all three shards passed; the executed tests exactly match each shard list and cover all 14/14 configurations with no omissions or duplicates
• targeted shard wall times: 24.09, 22.07, and 24.44 minutes (2.37-minute max/min spread; 1.11x imbalance)
• outer job/bootstrap + teardown overhead around pytest: 0.82, 0.81, and 0.82 minutes per shard
• critical path: 24.44 minutes versus 64.27 minutes in build 83851 (2.63x faster)
• observed total GPU time: 70.59 versus 64.27 GPU-minutes in the single baseline run (+9.8%)
• uvx pre-commit run --files ...: passed
• git diff --check: passed

AI assistance was used. The submitter will review and validate the change before marking it ready.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.