Remove decommissioned B300 runner fleet - #2818
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit e471c95. Configure here.
| - b300-nv_14 | ||
| - b300-nv_15 | ||
| - b300-nv_16 | ||
| - b300-nv_17 |
There was a problem hiding this comment.
Removed B300 hardware breaks sweep generation
High Severity
Removing the cluster:b300-nv hardware profile and the b300/cluster:b300-nv labels while live nvidia-master.yaml configs still target those runners makes matrix generation abort. Agentic DRAM-offload rows require available-cpu-dram-mib, and multinode B300 rows resolve gpus-per-node from that same metadata. A documented full sweep of nvidia-master.yaml, or any changelog or model-prefix selection that includes those keys, now fails before other configs are emitted.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit e471c95. Configure here.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Beyond the inline finding, I also checked utils/calc_success_rate.py, which derives its hardware labels from cluster: prefixed keys in runners.yaml — removing cluster:b300-nv there just silently drops b300 from success-rate tracking rather than crashing, so it's not an additional blocking issue.
Extended reasoning...
The PR removes the b300 runner fleet from configs/runners.yaml but leaves 16 live runner: b300 references in configs/nvidia-master.yaml, which the bug hunter already flagged as a CONFIRMED inline finding (dangling reference causing a crash in generate_sweep_configs.py's hardware lookup path). I independently verified this by grepping the current repo state: 16 matches for runner: b300 in nvidia-master.yaml, and no b300/cluster:b300-nv keys remain in runners.yaml. Since a confirmed finding already exists and will be posted inline, I'm not repeating it here.
I additionally traced a related but distinct code path suggested in the ruled-out candidates: utils/calc_success_rate.py builds its HARDWARE_LABELS list by scanning labels keys in runners.yaml prefixed with cluster: (see load_hardware_labels()). Removing cluster:b300-nv means this script will simply stop tracking/reporting success-rate stats for b300 hardware — it does not raise an exception or otherwise break, since the label list is just used to build regex match patterns and initialize per-hardware counters. This is a soft behavioral change (silently dropping b300 from reporting), not a crash, so it doesn't add a new blocking concern beyond what's already flagged inline.
Given a CONFIRMED finding already exists pointing to an active dangling-reference bug that would break sweep-config generation for the still-referenced dsr1-fp4-b300-dynamo-trt/dsr1-fp8-b300-dynamo-trt master config entries, I am not approving this PR. I'm posting a brief defer note only because I verified an additional, distinct code path that a human reviewer might otherwise want checked, and can report it as ruled out.
| @@ -257,9 +221,6 @@ hardware: | |||
| cluster:b200-nscale: | |||
There was a problem hiding this comment.
🔴 Removing the cluster:b300-nv hardware entry (and b300/cluster:b300-nv labels) leaves 16 live runner: b300 entries in configs/nvidia-master.yaml (e.g. dsr1-fp4-b300-dynamo-trt, dsr1-fp8-b300-dynamo-trt) with no hardware fallback, so generating a full/unfiltered sweep now crashes instead of producing a matrix. Fix: either keep a b300/cluster:b300-nv hardware entry until every master config referencing it is retargeted, or retarget/remove those 16 runner: b300 master-config entries in the same change. [also at: configs/runners.yaml:224 - Removing the b300 label and cluster:b300-nv hardware entries leaves configs/nvidia-master.yaml's active…]
Extended reasoning...
generate_sweep_configs.py resolves node counts via multinode_node_count -> worker_node_count -> scheduling_gpus_per_node(hardware_label='b300', runner_data) (utils/matrix_logic/generate_sweep_configs.py:171-198). recipe_node_count() returns None here because the referenced CONFIG_FILE paths (e.g. recipes/trtllm/b300-fp4/8k1k/mtp/ctx10_gen1_dep8_batch256_eplb0_mtp1.yaml) don't exist under benchmarks/multi_node/srt-slurm-recipes in this repo (only gb300-fp4 paths exist), per the comment 'Some srt-slurm recipes live only in the runtime image.' So it falls through to worker_node_count, which calls scheduling_gpus_per_node('b300', ...). Before this diff, hardware.get('b300') was already None, but the family fallback matched the single hardware entry cluster:b300-nv (family 'b300') and returned gpus-per-node=8. After this diff, no hardware key has family 'b300', so matches is empty and it raises ValueError("Cannot resolve gpus_per_node for 'b300'"). This affects all 16 runner: b300 master-config entries (nvidia-master.yaml lines 452, 658, 1039, 1117, 1135, 1186, 1288, 1303, 1318,…
Verification: normal; acknowledged in diff (note's claim does not hold as protection). The diff removes the cluster:b300-nv hardware entry (configs/runners.yaml, formerly ~257: gpus-per-node 8) and the b300/cluster:b300-nv labels, but 16 active runner: b300 master entries remain in configs/nvidia-master.yaml (lines 452, 658, 1039, 1117, 1135, 1186, 1288, 1303, 1318, 1370, 1469, 1728, 1745, 5175, 5238,…
Adds an LMCache 0.5.5rc2 DRAM offload arm to kimik3-fp4-b300-vllm-agentic-dspark at concurrency 56 and 70, and moves the recipe to cluster:b300-dsxe after cluster:b300-nv was decommissioned in #2818. The resolved DRAM budget is unchanged at 2249 GB. Concurrency 56 and 70 are where the offload tier carries the run. In the sweep from run 33472235399 the GPU prefix cache hit rate holds 90.4-96.3% from conc 1 through 48, so nearly all prefix reuse is served out of HBM. At conc 56 GPU hit falls to 88.0% and at conc 70 to 59.9%. Under DCP the LMCache chunk must be a multiple of block size x DCP size; Kimi-K3's hybrid KDA/MLA groups resolve a local block of 1536, so DCP 8 gives 12288, matching the validated Kimi-K3 DCP8 e2e run 33463265881. cp_kv_cache_interleave_size stays at vLLM's default of 1: the pinned image defines the option but carries none of the auto-resolution that sets it to the local block size. The cumem allocator is disabled on this arm only. The MP connector shares KV tensors with its out-of-process server over CUDA IPC, and cuMem memory has no legacy IPC handle, so _share_cuda_() fails with cudaErrorInvalidValue during initialize_from_config. The L1 read-lock TTL is raised to 3600s. The lock is taken at LOOKUP and verified at RETRIEVE, which are separate async operations; at conc 70 a queued request exceeded the 300s default (measured 458s), the lock expired, and unsafe_read returned KEY_NOT_READABLE, which killed the engine and failed 630 requests. Not append-only: the incumbent points were measured on the retired cluster, so both arms are re-measured together on b300-dsxe.
Adds an LMCache 0.5.5rc2 DRAM offload arm to kimik3-fp4-b300-vllm-agentic-dspark at concurrency 56 and 70, where the offload tier carries the run: GPU prefix cache hit rate holds 90.4-96.3% from conc 1 through 48 (run 33472235399) but falls to 88.0% at conc 56 and 59.9% at conc 70. Also moves the recipe off two dead dependencies. cluster:b300-nv was decommissioned in #2818, so the runner is now cluster:b300-dsxe; the resolved DRAM budget is unchanged at 2249 GB. The pinned image nightly-dev-x86_64-cu13.0.1-5894fdf was garbage-collected from Docker Hub and 404s at enroot import, so it is replaced with nightly-dev-x86_64-cu13-3696c77, the newest live x86_64 cu13 build, whose arm64 twin backs the GB300 DCP8 recipes. Under DCP the LMCache chunk must be a multiple of block size x DCP size; Kimi-K3's hybrid KDA/MLA groups resolve a local block of 1536, so DCP 8 gives 12288, matching the validated Kimi-K3 DCP8 e2e run 33463265881. cp_kv_cache_interleave_size stays at vLLM's default of 1: the image defines the option but carries none of the auto-resolution that sets it to the local block size. The cumem allocator is disabled on this arm only. The MP connector shares KV tensors with its out-of-process server over CUDA IPC, and cuMem memory has no legacy IPC handle, so _share_cuda_() fails with cudaErrorInvalidValue during initialize_from_config. The L1 read-lock TTL is raised to 3600s. The lock is taken at LOOKUP and verified at RETRIEVE, which are separate async operations; at conc 70 a queued request exceeded the 300s default (measured 458s), the lock expired, and unsafe_read returned KEY_NOT_READABLE, killing the engine and failing 630 requests. Not append-only: the incumbent points were measured on the retired cluster, so both arms are re-measured together on b300-dsxe.
1 participant
Summary
b300-nv_01throughb300-nv_17runner inventorycluster:b300-nvrunner mapping and runner hardware profileValidation
python3 -m pytest utils/matrix_logic/test_validation.py utils/matrix_logic/test_generate_sweep_configs.py -q(246 passed)configs/runners.yamlsuccessfullygit diff --checkNote
Low Risk
Config-only removal of decommissioned runners; no auth, data, or runtime logic changes, though any matrix still pointing at
b300orcluster:b300-nvwill need retargeting.Overview
Removes the decommissioned B300 CI runner inventory from
configs/runners.yamlso jobs can no longer target hosts that no longer exist.The change drops the
b300label listingb300-nv_01–b300-nv_17, thecluster:b300-nvlabel mapping for the same nodes, and thecluster:b300-nvhardware profile (available-cpu-dram-mib,gpus-per-node). B300 workload/benchmark configs elsewhere are intentionally unchanged so they can be retargeted in a follow-up.Reviewed by Cursor Bugbot for commit e471c95. Bugbot is set up for automated code reviews on this repo. Configure here.