Repository navigation
fix(launch): request explicit Slurm memory for single-node jobs and fit host pools to measured DRAM - #3804
Conversation
…DRAM Port of #3615 onto the current srt driver. Every single-node srun step gets --mem = available-cpu-dram-mib / gpus-per-node * GPU_COUNT through SRT_SRUN_OPTIONS, so a point using half the node can no longer take more than half its host DRAM. The sbatch job requests the whole available-cpu-dram-mib only when slurm.srt-slurm.single-node-exclusive is true; on shared nodes it requests the same GPU share, so it leaves the rest of the node to other jobs. Clusters without available-cpu-dram-mib keep their previous behavior. Co-authored-by: Cam Quilici <cjquilici@gmail.com>
Three single-node AgentX DRAM arms pinned host KV pools at or within a few GiB of their TP4 srun step's new --mem share of node DRAM: - minimaxm3-fp4-b200-vllm-agentic-mtp TP4 SimpleCPU: dram-utilization 1.0 -> 0.96, cpu_bytes_to_use and TOTAL_CPU_DRAM_GB 1082 -> 1038 GB (headroom 0.08 -> 41.1 GiB under 1,031,960 MiB). - minimaxm3-fp4-b200-trtllm-agentic-mtp TP4: per-rank host_cache_size 250 -> 240 GiB (headroom 7.8 -> 47.8 GiB under 1,031,960 MiB). - qwen3.5-fp8-mi355x-sglang-agentic-mtp TP4 conc 64-80: per-rank hicache-size 400 -> 395 GB (headroom 21.5 -> 40.1 GiB under 1,547,890 MiB). Also trims the MiniMax-M3 B200 vLLM master-config comment to its rationale.
|
Thanks for the contribution!
中文感谢你的贡献！
|
Slurm's RealMemory on b200-nscale and b300-dsxe is below available-cpu-dram-mib, so sbatch rejected the whole-node --mem. Exclusive jobs now take all the memory Slurm has, and a step that spans the whole node uses the job's memory instead of a share cap.
Measured on each cluster's GPU nodes as the lower of RealMemory - MemSpecLimit and MemTotal less 16 GiB: h100-dgxc and b300-dsxe reserve 128 GiB through MemSpecLimit, b200-nscale configures RealMemory 15.5 GiB under MemTotal, and the AMD clusters set RealMemory to MemTotal.
The lower available-cpu-dram-mib on h100-dgxc, b200-nscale and mi300x-amd lowers the matrix total-cpu-dram-gb of their single-node DRAM offload points, and the single-node binder rejected every recipe variant still naming the old TOTAL_CPU_DRAM_GB. Each variant now names the new budget, and host pools sized from it follow their recipe's rule: - DSv4 B200 vLLM: SimpleCPU bytes per rank = budget / 8, 243.375 -> 241.5 GB. - MiniMax-M3 B200 vLLM: cpu_bytes_to_use = budget, 1038 -> 1030 GB. - MiniMax-M3 H100 vLLM: Mooncake segment = (budget GiB - 414) / 8 - 30, 119 -> 105 GB. - MiniMax-M3 MI300X vLLM: LMCache L1 = 8 x budget / 8, 1296 -> 1280 GB. - Qwen3.5 B200 SGLang: hicache-size = (budget - TP) * 15 / TP / 31, 104 -> 103. - Qwen3.5 H100 SGLang: hicache-size = budget / TP8 / 2 pools, 108 -> 100. GLM-5.2 B200's 169 GB/rank pool (~1,662 GB with the DSA indexer) and the MiniMax-M3 B200 TRT-LLM per-rank pools, which are not derived from the budget, still fit. Every pool stays 32+ GiB under its srun step's share of node DRAM.
Validation OOM-killed the Qwen3.5 MI355X FP8 TP4 CONC 64 HiCache point inside its srun step cap: 4 x 395 GB per-rank pools also exceeded the 3 TB policy share of four GPUs (1500 GB). The MiniMax-M3 B300 TRT-LLM TP2/TP4 host caches (388.6 GB per rank) exceeded theirs too (750/1500 GB). - Qwen3.5 MI355X FP8: dram-utilization 0.80 -> 1.0, hicache-size at CONC 64-80 395 -> 374 GB per rank (budget 1199 -> 1499 GB). - MiniMax-M3 B300 TRT-LLM: dram-utilization 0.8 -> 1.0, TP2/TP4 host_cache_size 388554555392 -> 374500000000/374750000000 bytes; TOTAL_CPU_DRAM_GB 599/1199/2399 -> 749/1499/2999. The budgets of two formula-sized pools now name the pool itself, as the refactor stack's '@dram' references expect: MiniMax-M3 MI300X 0.80 -> 0.7972 (TOTAL_CPU_DRAM_GB 1284 -> 1280, the LMCache L1) and MiniMax-M3 H100 0.80 -> 0.449 (1607 -> 902, eight 105 GiB Mooncake segments).
A step capped at its GPUs' share of node DRAM leaves no room for the
engine and page cache once its host pools fill the share:
qwen3.5-fp8-mi355x-sglang-agentic-mtp TP4 HiCache failed inside the step
cgroup ("Requesting 150.27 GB but only have 15.04 GB free") with its
pools at the DRAM policy limit.
Exclusive single-node jobs keep requesting --mem=0 and no longer cap
their srun steps, so pools must fit the node. On shared nodes the job and
its step still get the serving GPUs' share, which pools must fit.
…nother cluster 8c9da40 took h100-dgxc's 1,915,904 MiB from login-h100, which fronts the DSXE H100 cluster (partitions batch_1/batch_all, /data), not the DGX Cloud cluster runners.yaml calls h100-dgxc (partition hpc-gpu-1, /mnt/nfs). The 128 GiB MemSpecLimit that commit's message attributes to h100-dgxc is the DSXE cluster's; h100-dgxc was not measured. Restore 2,063,837 MiB and undo the H100 values derived from the wrong one: - Qwen3.5 H100 SGLang: hicache-size 100 -> 108 GB, TOTAL_CPU_DRAM_GB 1607 -> 1731. - MiniMax-M3 H100 vLLM: dram-utilization 0.449 -> 0.80, Mooncake segment 105 -> 119 GB, TOTAL_CPU_DRAM_GB 902 -> 1731. Every h100-dgxc and h100 matrix row and H100 recipe again matches main. CONFIGS.md gives the Slurm rule as how to measure the field, not as what every recorded value is. runners.yaml h100-dgxc has no registered runners; the DSXE H100 runners (cluster:h100-dsxe) have no runners.yaml entry.
Condense the driver's DRAM cap comment, the available-cpu-dram-mib doc and the MiniMax-M3 B200 dram-utilization comment; drop the test comment that repeated the driver's and name the test for all three of its cases; fix the MI300X LMCache recipe comment, whose L1 is now the whole 1280 GB budget rather than shares of 1284 GB. 精简驱动中的 DRAM 上限注释、available-cpu-dram-mib 文档和 MiniMax-M3 B200 的 dram-utilization 注释；删除重复驱动注释的测试注释，并按三种情形重命名该 测试；修正 MI300X LMCache 配方注释：其 L1 现为完整的 1280 GB 预算，而非 1284 GB 的分摊份额。
|
Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package. 中文大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要，改动通常也很小。请让你的 agent 清理 PR，使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。 |
|
Claude finished @adibarra's task in 1m 36s —— View job Review of PR #3804
Summary: The driver change looks right. Exclusive jobs get
Note: The PR body's headroom table says the minimaxm3 b200 vLLM pool goes to 1038 GB. The recipe actually sets 1030 GB, which is what 2,047,536 MiB × 0.96 × 4/8 works out to. The recipe is right; only the body is out of date. I couldn't run the tests: running |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| agentic-coding: | ||
| - dram-utilization: 0.80 | ||
| # CONC 64-80 HiCache pools take the 3 TB policy share per GPU. | ||
| - dram-utilization: 1.0 |
There was a problem hiding this comment.
🔴 BLOCKING: The master config files changed, but inferencex-e2e/perf-changelog.yaml was not updated. When you change inferencex-e2e/configs/amd-master.yaml or inferencex-e2e/configs/nvidia-master.yaml, you need to add an entry to inferencex-e2e/perf-changelog.yaml describing the change.
Why it matters: Without the entry, no sweep runs for this PR. These edits change results: dram-utilization changes on qwen3.5-mi355x, minimaxm3-mi300x, minimaxm3-b300-trtllm and minimaxm3-b200-vllm, plus the recipe pool sizes. The new --mem request is also never tested on a cluster. The PR body says this was deliberate, but merging these changes would leave the published curves out of sync with the configs that produced them.
Fix: Add an entry at the end of perf-changelog.yaml that lists the affected config keys and links #3804. Then apply full-sweep-fail-fast.
| step_mib = job_mib = config.gpu_share_mib(run.cluster, request.gpu_count) | ||
| if run.srt.single_node_exclusive and job_mib is not None: | ||
| step_mib, job_mib = None, 0 |
There was a problem hiding this comment.
🟡 WARNING: This change sends a real non-zero --mem only to the shared clusters (single-node-exclusive: false): h100-cw and h200-cw, both at available-cpu-dram-mib: 1_998_848. Those two values were not re-measured here. The PR re-measured b200-nscale, b300-dsxe and mi3xx, which all take --mem=0 now, so the new values never reach sbatch there.
Why it matters: A TP8 point on h100-cw or h200-cw now asks for --mem=1998848M on both the job and the srun step. Your Risk section already describes what happens if that is above RealMemory - MemSpecLimit: sbatch fails with "Memory specification can not be satisfied", and every TP8 point on those clusters fails before it starts. TP4 requests half that, which leaves more margin.
Fix: Run scontrol show node on a GPU node in each cluster and check that RealMemory - MemSpecLimit ≥ 1_998_848. If it isn't, re-measure both values the same way you did for the other clusters. Alternatively, run one TP8 smoke on h100-cw or h200-cw before merging.
There was a problem hiding this comment.
Beyond the inline findings (none reported this run), I also checked the gpu_share_mib floor-to-zero scenario flagged as a candidate issue in config.py:179 — it would require available-cpu-dram-mib/gpus-per-node*GPU_COUNT to floor to 0, but every cluster's available-cpu-dram-mib in runners.yaml is in the hundreds of thousands to millions of MiB against gpus-per-node of 4 or 8, so this cannot occur with current config data.
Extended reasoning...
The diff adds a gpu_share_mib() DRAM-capping calculation and wires it into srun/sbatch mem options for single-node srt-slurm jobs, with matching test coverage (test_single_node_jobs_cap_dram_only_on_shared_nodes) that exercises launch() and asserts on emitted SRT_SRUN_OPTIONS and sbatch directives per the repo's test-quality conventions. No security-sensitive surface (auth, crypto, permissions) is touched; it is cluster-config and memory-accounting logic plus config/doc value updates matching the PR's documented headroom table.
# Conflicts: # inferencex-e2e/configs/nvidia-master.yaml
Main now keeps infx free of explanatory comments and bans header comments in configs. Remove the driver's DRAM cap comment, which CONFIGS.md already documents, and the MiniMax-M3 B200 vLLM master-config header.
# Conflicts: # inferencex-e2e/infx/launch/drivers/srt/config.py
Main's native single-node power telemetry (#3781) stages the cluster's GPU exporter for every non-eval single-node point and stops preparation when the cluster record has none, so the DRAM-cap driver test's ad hoc cluster now needs default_gpu_exporter to reach submission.
# Conflicts: # inferencex-e2e/configs/runners.yaml
… budget b200-nscale runs single-node jobs whole-node (--mem=0), so the TP4 pools no longer need headroom under a per-GPU step cap. The vLLM SimpleCPU pool and the TRT-LLM per-rank host cache now take the full TP4 budget (dram-utilization 1.0, 1073 GB), and the TRT TP4 rows report the budget their pool actually uses instead of a nominal 0.8 one.
adibarra
changed the title
1 participant
Summary
Replaces #3615 (Cam). Single-node srt-slurm jobs now make an explicit Slurm memory request from
runners.yaml,available-cpu-dram-mibrecords what Slurm can actually allocate, and the AgentX host KV pools fit their DRAM budgets.Launcher
slurm.srt-slurm.single-node-exclusive: true, the default; every cluster except h100-cw): the job requests--mem=0, all the memory Slurm has on the node. Steps get no memory cap.single-node-exclusive: false; today only h100-cw): the job and its srun step request the serving GPUs' share,available-cpu-dram-mib * GPU_COUNT / gpus-per-nodeMiB, so a job no longer takes the whole node's memory.available-cpu-dram-mibgets no memory request, as before. Every cluster inrunners.yamlhas one.The code change is
gpu_share_mibandsrun_optionsininfx/launch/drivers/srt/config.py, thememsbatch directive, and the call inrun_single_node.Cluster DRAM figures
available-cpu-dram-mibis the lower of Slurm'sRealMemory - MemSpecLimitandMemTotalless 16 GiB for the OS, over the cluster's GPU nodes (documented inconfigs/CONFIGS.md). Measured on 2026-10-15:Not measured: gb200-nv and gb300-nv (no single-node DRAM points here) and h100-cw (no login access; Slurm accepted its share on two cluster runs, see Verification).
Recipe and budget changes
These change what 11 config keys run. Pools now equal or fit their reported
total-cpu-dram-gb:minimaxm3-fp4-b200-vllm-agentic-mtpminimaxm3-fp4-b200-trtllm-agentic-mtpdram-utilization0.8 → 1.0; per-rankhost_cache_size250 GiB → 268.25 GB, so the reported budget (858 → 1073 GB) matches the 1073 GB pool. It previously reported 865 GB for a 1074 GB pool. TP8 keeps 200 GiB per rank; budget 1731 → 1717 GB.minimaxm3-fp4-b300-trtllm-agentic-mtpdram-utilization0.8 → 1.0; per-rank host cache 388.6 → 374.5/374.75 GB, equal to the TP2/TP4 budgets (749/1499 GB) under the 3 TB policy cap. Previously the pools exceeded their reported budgets.qwen3.5-fp8-mi355x-sglang-agentic-mtpdram-utilization0.8 → 1.0; TP4 c64-80hicache-size400 → 374 GB per rank, within the 1500 GB TP4 share of the 3 TB policy. The 400 GB pools did not fit and were OOM-killed.minimaxm3-fp8-mi300x-vllm-agentic-mtpdram-utilization0.7972).qwen3.5-fp{4,8}-b200-sglang-agentic-mtp,dsv4-fp4-b200-vllm-agentic-mtp,dsv4-fp4-b200-sglang-agentic-hicache-mtp,glm5.2-fp{4,8}-b200-sglang-agentic-mtpTOTAL_CPU_DRAM_GBfollows the b200-nscale figure (for example 1731 → 1717, 865 → 858); derived pools are recomputed by each recipe's own rule (for example qwenhicache-size104 → 103 GB).Main currently cannot build recipes for 8 of these keys:
82fc21bd6lowered b200-nscale from 2_063_920 to 2_048_000 MiB, so their recipes'TOTAL_CPU_DRAM_GBno longer matches the matrix and launch fails with "Expected exactly one matching single-node SRT recipe". This PR updates them. Still broken on main and untouched here:qwen3.5-fp8-h100-sglang-agentic,qwen3.5-fp8-h100-sglang-agentic-mtp,minimaxm3-fp8-h100-vllm-agentic-mtp(sized for the retired h100-dgxc) andqwen3.8next-fp4-gb300-sglang-agentic-mtp.On whole-node clusters the pools take their full budget; there is no per-GPU step cap to leave headroom under.
The 31-line header comment above
minimaxm3-fp4-b200-vllm-agentic-mtpis removed under the current comment rules.There is no
perf-changelog.yamlentry; one will be added when results for these keys should be collected.Verification
test_single_node_jobs_cap_dram_only_on_shared_nodes(fake Slurm, 8000 MiB, 8 GPUs per node,GPU_COUNT=4): whole-node gives job--mem=0and no step cap; shared gives job and step4000M; no figure gives no request.--mem=1998848Mon job and step) passed twice; h200-dgxc (whole-node,#SBATCH --exclusive,#SBATCH --mem=0) passed with 40/40 requests (run 38746233127); mi355x and b200-nscale single-node AgentX passed on the integration branch.