Repository navigation
Conversation
…hare of node DRAM
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| run.cluster.available_cpu_dram_mib if request.env.get("KV_OFFLOADING") == "dram" else None | ||
| ) | ||
| step_mib = node_mib and node_mib * request.gpu_count // run.cluster.gpus_per_node | ||
| if (options := config.srun_options(run.backend.settings, step_mib)) is not None: |
There was a problem hiding this comment.
🔴 This driver change caps srun/sbatch memory for single-node DRAM-offload jobs but the PR adds no inferencex-e2e/perf-changelog.yaml entry, though AGENTS.md says every change that can affect benchmark performance requires one. Capping srun steps to available-cpu-dram-mib / gpus-per-node * GPU_COUNT can newly OOM-kill DRAM-offload points that previously ran uncapped, which is exactly the kind of benchmark-affecting change the changelog exists to record. Fix: append a new entry to inferencex-e2e/perf-changelog.yaml describing this driver-level memory cap, preserving all existing bytes/whitespace and appending only at the tail, per AGENTS.md:125. [also at: inferencex-e2e/perf-changelog.yaml:1 - This PR changes srun/sbatch memory limits for every existing DRAM-offload (KV_OFFLOADING=dram) job cluster-wide, which can newly OOM-kill runs that previously completed, but ships with no perf-changelog.yaml entry.]
Why this was flagged
AGENTS.md:125 states every change that can affect benchmark performance requires a new append-only inferencex-e2e/perf-changelog.yaml entry. This PR (inferencex-e2e/infx/launch/drivers/srt/init.py:54-58, config.py:140-141,167-175) changes the sbatch mem directive and srun --mem option for every KV_OFFLOADING=dram single-node point, which can OOM-kill steps that previously had no memory cap. git diff d265460..HEAD touches only init.py, config.py and test_srt_config.py; perf-changelog.yaml is unmodified. No safeguard (CI check) is visible in this checkout enforcing the changelog rule, so the operational effect ships without the required record of the behavior change.
Verification: normal (mandatory repo invariant violated by this change). git diff d265460..HEAD --name-only shows only infx/launch/drivers/srt/init.py, config.py, and tests/launch/test_srt_config.py; inferencex-e2e/perf-changelog.yaml is NOT in the diff, and its tail still ends at PR #3602 with no new entry. init.py:54-58 computes node_mib/step_mib passed into srun_options and SrtJob; config.py:140-141 emits an sbatch mem directive and config.py:167-175 injects options["mem"] into SRT_SRUN_OPTIONS.
| node_mib = ( | ||
| run.cluster.available_cpu_dram_mib if request.env.get("KV_OFFLOADING") == "dram" else None | ||
| ) | ||
| step_mib = node_mib and node_mib * request.gpu_count // run.cluster.gpus_per_node | ||
| if (options := config.srun_options(run.backend.settings, step_mib)) is not None: | ||
| run.env["SRT_SRUN_OPTIONS"] = options | ||
| if rc := submit.bind_point(run, checkout, root / "arguments"): | ||
| return rc |
There was a problem hiding this comment.
🟡 (optional) On clusters like h100-cw and h200-cw (available-cpu-dram-mib set plus srt-slurm.single-node-exclusive: false in runners.yaml), a DRAM-offload single-node job now gets sbatch --mem=M via mem_mib=node_mib (init.py:75, config.py:140-141), monopolizing the node's memory even though use_exclusive_sbatch_directive stays false. Requesting the whole node's DRAM without --exclusive blocks Slurm from co-scheduling any other job on that node even when GPUs/CPUs are free, defeating the cluster's intentional non-exclusive sharing for every KV_OFFLOADING=dram job there. Fix: scale the sbatch mem request to the job's own GPU share (like step_mib already does) or only apply the full node-DRAM --mem directive when single_node_exclusive is true.
Why this was flagged
Trigger: a KV_OFFLOADING=dram single-node point on h100-cw or h200-cw, which set available-cpu-dram-mib (runners.yaml:306,361) and srt-slurm.single-node-exclusive: false (runners.yaml:323,379). init.py:54-57,75 sets node_mib=run.cluster.available_cpu_dram_mib unconditionally and passes it as SrtJob.mem_mib with no dependency on single_node_exclusive. config.py:140-141 then always adds default_sbatch_directives.mem=f'{node_mib}M' when job.mem_mib is set, while use_exclusive_sbatch_directive (config.py:133-135) is computed separately and stays false for these clusters. Base behavior: before this diff no mem directive was ever emitted, so such nodes could host multiple concurrent single-node jobs by GPU/gres sharing; after merging, one job's full-node --mem request prevents Slurm from allocating any other job to that node until it finishes, silently serializing throughput the non-exclusive setting was meant to allow.
Verification: Nit; acknowledged in diff (partially). init.py:54-57,75 sets node_mib = run.cluster.available_cpu_dram_mib for any KV_OFFLOADING=dram single-node point and passes it as SrtJob.mem_mib with no dependency on exclusivity; config.py:140-141 always emits the sbatch mem directive, while config.py:133-134 sets use_exclusive_sbatch_directive = srt.single_node_exclusive independently.
cquil11
changed the title
…it host pools to measured DRAM (#3804) * fix(launch): cap single-node srun steps at their GPUs' share of node DRAM Port of #3615 onto the current srt driver. Every single-node srun step gets --mem = available-cpu-dram-mib / gpus-per-node * GPU_COUNT through SRT_SRUN_OPTIONS, so a point using half the node can no longer take more than half its host DRAM. The sbatch job requests the whole available-cpu-dram-mib only when slurm.srt-slurm.single-node-exclusive is true; on shared nodes it requests the same GPU share, so it leaves the rest of the node to other jobs. Clusters without available-cpu-dram-mib keep their previous behavior. Co-authored-by: Cam Quilici <cjquilici@gmail.com> * fix(agentx): keep single-node DRAM pools 32 GiB under the srun step cap Three single-node AgentX DRAM arms pinned host KV pools at or within a few GiB of their TP4 srun step's new --mem share of node DRAM: - minimaxm3-fp4-b200-vllm-agentic-mtp TP4 SimpleCPU: dram-utilization 1.0 -> 0.96, cpu_bytes_to_use and TOTAL_CPU_DRAM_GB 1082 -> 1038 GB (headroom 0.08 -> 41.1 GiB under 1,031,960 MiB). - minimaxm3-fp4-b200-trtllm-agentic-mtp TP4: per-rank host_cache_size 250 -> 240 GiB (headroom 7.8 -> 47.8 GiB under 1,031,960 MiB). - qwen3.5-fp8-mi355x-sglang-agentic-mtp TP4 conc 64-80: per-rank hicache-size 400 -> 395 GB (headroom 21.5 -> 40.1 GiB under 1,547,890 MiB). Also trims the MiniMax-M3 B200 vLLM master-config comment to its rationale. * fix(srt): request --mem=0 for exclusive single-node jobs Slurm's RealMemory on b200-nscale and b300-dsxe is below available-cpu-dram-mib, so sbatch rejected the whole-node --mem. Exclusive jobs now take all the memory Slurm has, and a step that spans the whole node uses the job's memory instead of a share cap. * fix(runners): set available-cpu-dram-mib to what Slurm can allocate Measured on each cluster's GPU nodes as the lower of RealMemory - MemSpecLimit and MemTotal less 16 GiB: h100-dgxc and b300-dsxe reserve 128 GiB through MemSpecLimit, b200-nscale configures RealMemory 15.5 GiB under MemTotal, and the AMD clusters set RealMemory to MemTotal. * fix(recipes): match TOTAL_CPU_DRAM_GB to measured node DRAM The lower available-cpu-dram-mib on h100-dgxc, b200-nscale and mi300x-amd lowers the matrix total-cpu-dram-gb of their single-node DRAM offload points, and the single-node binder rejected every recipe variant still naming the old TOTAL_CPU_DRAM_GB. Each variant now names the new budget, and host pools sized from it follow their recipe's rule: - DSv4 B200 vLLM: SimpleCPU bytes per rank = budget / 8, 243.375 -> 241.5 GB. - MiniMax-M3 B200 vLLM: cpu_bytes_to_use = budget, 1038 -> 1030 GB. - MiniMax-M3 H100 vLLM: Mooncake segment = (budget GiB - 414) / 8 - 30, 119 -> 105 GB. - MiniMax-M3 MI300X vLLM: LMCache L1 = 8 x budget / 8, 1296 -> 1280 GB. - Qwen3.5 B200 SGLang: hicache-size = (budget - TP) * 15 / TP / 31, 104 -> 103. - Qwen3.5 H100 SGLang: hicache-size = budget / TP8 / 2 pools, 108 -> 100. GLM-5.2 B200's 169 GB/rank pool (~1,662 GB with the DSA indexer) and the MiniMax-M3 B200 TRT-LLM per-rank pools, which are not derived from the budget, still fit. Every pool stays 32+ GiB under its srun step's share of node DRAM. * fix(recipes): keep single-node DRAM pools within the 3 TB policy share Validation OOM-killed the Qwen3.5 MI355X FP8 TP4 CONC 64 HiCache point inside its srun step cap: 4 x 395 GB per-rank pools also exceeded the 3 TB policy share of four GPUs (1500 GB). The MiniMax-M3 B300 TRT-LLM TP2/TP4 host caches (388.6 GB per rank) exceeded theirs too (750/1500 GB). - Qwen3.5 MI355X FP8: dram-utilization 0.80 -> 1.0, hicache-size at CONC 64-80 395 -> 374 GB per rank (budget 1199 -> 1499 GB). - MiniMax-M3 B300 TRT-LLM: dram-utilization 0.8 -> 1.0, TP2/TP4 host_cache_size 388554555392 -> 374500000000/374750000000 bytes; TOTAL_CPU_DRAM_GB 599/1199/2399 -> 749/1499/2999. The budgets of two formula-sized pools now name the pool itself, as the refactor stack's '@dram' references expect: MiniMax-M3 MI300X 0.80 -> 0.7972 (TOTAL_CPU_DRAM_GB 1284 -> 1280, the LMCache L1) and MiniMax-M3 H100 0.80 -> 0.449 (1607 -> 902, eight 105 GiB Mooncake segments). * fix(srt): cap single-node steps only on shared nodes A step capped at its GPUs' share of node DRAM leaves no room for the engine and page cache once its host pools fill the share: qwen3.5-fp8-mi355x-sglang-agentic-mtp TP4 HiCache failed inside the step cgroup ("Requesting 150.27 GB but only have 15.04 GB free") with its pools at the DRAM policy limit. Exclusive single-node jobs keep requesting --mem=0 and no longer cap their srun steps, so pools must fit the node. On shared nodes the job and its step still get the serving GPUs' share, which pools must fit. * fix(runners): restore h100-dgxc DRAM; earlier value was measured on another cluster 8c9da40 took h100-dgxc's 1,915,904 MiB from login-h100, which fronts the DSXE H100 cluster (partitions batch_1/batch_all, /data), not the DGX Cloud cluster runners.yaml calls h100-dgxc (partition hpc-gpu-1, /mnt/nfs). The 128 GiB MemSpecLimit that commit's message attributes to h100-dgxc is the DSXE cluster's; h100-dgxc was not measured. Restore 2,063,837 MiB and undo the H100 values derived from the wrong one: - Qwen3.5 H100 SGLang: hicache-size 100 -> 108 GB, TOTAL_CPU_DRAM_GB 1607 -> 1731. - MiniMax-M3 H100 vLLM: dram-utilization 0.449 -> 0.80, Mooncake segment 105 -> 119 GB, TOTAL_CPU_DRAM_GB 902 -> 1731. Every h100-dgxc and h100 matrix row and H100 recipe again matches main. CONFIGS.md gives the Slurm rule as how to measure the field, not as what every recorded value is. runners.yaml h100-dgxc has no registered runners; the DSXE H100 runners (cluster:h100-dsxe) have no runners.yaml entry. * refactor(srt): tighten single-node DRAM test, comments and docs Condense the driver's DRAM cap comment, the available-cpu-dram-mib doc and the MiniMax-M3 B200 dram-utilization comment; drop the test comment that repeated the driver's and name the test for all three of its cases; fix the MI300X LMCache recipe comment, whose L1 is now the whole 1280 GB budget rather than shares of 1284 GB. 精简驱动中的 DRAM 上限注释、available-cpu-dram-mib 文档和 MiniMax-M3 B200 的 dram-utilization 注释；删除重复驱动注释的测试注释，并按三种情形重命名该 测试；修正 MI300X LMCache 配方注释：其 L1 现为完整的 1280 GB 预算，而非 1284 GB 的分摊份额。 * chore: drop comments that break the code comment rules Main now keeps infx free of explanatory comments and bans header comments in configs. Remove the driver's DRAM cap comment, which CONFIGS.md already documents, and the MiniMax-M3 B200 vLLM master-config header. * test(srt): give the DRAM-cap test cluster a GPU exporter Main's native single-node power telemetry (#3781) stages the cluster's GPU exporter for every non-eval single-node point and stops preparation when the cluster record has none, so the DRAM-cap driver test's ad hoc cluster now needs default_gpu_exporter to reach submission. * fix(recipes): size minimaxm3 b200 host pools to their whole-node DRAM budget b200-nscale runs single-node jobs whole-node (--mem=0), so the TP4 pools no longer need headroom under a per-GPU step cap. The vLLM SimpleCPU pool and the TRT-LLM per-rank host cache now take the full TP4 budget (dram-utilization 1.0, 1073 GB), and the TRT TP4 rows report the budget their pool actually uses instead of a nominal 0.8 one. --------- Co-authored-by: Cam Quilici <cjquilici@gmail.com>
This branch has not been deployed
1 participant
Summary
Single-node srt-slurm runs now get their host memory from
runners.yamland Slurm enforces it, instead of relying on the engine honoringTOTAL_CPU_DRAM_GB.--mem = available-cpu-dram-mib / gpus-per-node * GPU_COUNTthroughSRT_SRUN_OPTIONS. The cgroup OOM-kills a step that exceeds its GPUs' share.--mem=<available-cpu-dram-mib>M, so the step request always fits the allocation (e.g. on clusters with a smallDefMemPerCPU).This applies by default to every single-node point on clusters that declare
available-cpu-dram-mib.TOTAL_CPU_DRAM_GBis unchanged and still sizes the engines' offload pools. It is the same share scaled bydram-utilization, so it stays under the cap.