Repository navigation
Conversation
…ency calibration First of four layers split out of #35335. This layer only measures: it records, during synthetic server warmup, what a residency planner needs to decide a placement, and defines the record types that carry it. - ComponentResidencyManager tracks a warmup phase key per stage setup / use / prefetch / full-weight transition and records the allocator peak and active components per phase (WarmupPhasePeak). - The pipeline stages record their denoising iteration counts against the default workload (record_stage_iterations), and the perf metrics keep per-stage step timings. - GPUWorker records one WarmupMemoryRecord per warmup request (LayerwiseUsageTracker counts per-layer uses while it runs). - auto_residency.py carries the record/target/report types and the estimators that extrapolate a probe to the default workload (peaks per phase, layer uses per iteration, stage timings). - The warmup request builder gains the full-shape calibration probe, its step floor, OOM degradation and --warmup-sampling-params; the gate that says whether a deployment can be calibrated lives in auto_tune. - Control-request structs move from entrypoints/utils.py to entrypoints/control_requests.py (a pure move that later layers build on). Planning, applying and validating a placement follow in the next layers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Second of four layers split out of #35335: the planner as a library of pure functions, with no runtime wiring yet. - collect_residency_targets enumerates every complete placement a component could take (resident, component offload, layerwise with N resident layers and M pinned layers) with its signed cost on each named resource: per-phase device headroom, transition-time headroom, host pin budget, host repack space. - plan_auto_residency solves the joint placement over all ranks' reports (consensus candidates, binding phase constraints, VRAM reserve, latency-equivalence tie-breaks), backed by placement_budget's optimize_placement. - Sizing helpers for modules and layerwise managers, the H2D savings ranking used by the residency hint, and the plan summary formatters. - host_memory_budget: HostPinBudget release/for_local_worker, pin reserve, storage-deduplicated tensor sizing; layerwise_offload: layer byte estimation helpers the planner sizes unconfigured managers with. Applying, validating and rolling back a plan follow in the next layer. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mickqian
requested review from
AgainstEntropy,
BBuf and
yichiche
as code owners
September 4, 2026 03:40
mickqian
added
run-ci
This was referenced Sep 4, 2026
Merged
… in every loop record_stage_iterations had to be called by hand inside 14 denoising loops, each redoing the same arithmetic, and a new model that forgot it silently fell back to "iterations == steps". Now PipelineStage records the count itself after forward for warmup requests: a stage declares default_workload_iterations(batch, num_inference_steps) once, the base class evaluates it at the probe's step count (measured) and at the default workload's (target). - DenoisingStage: steps. Cosmos3 and MOVA: steps. DMD: the fixed distilled schedule length. MiniMax-H3: steps - 1. LTX-2 refiner: its distilled sigma table, which also retires the stage_target_num_inference_steps bookkeeping in batch.extra. - Loops whose length is only known inside them (block-wise causal DMD, Helios chunks, SANA-WM refiners) keep the explicit call; an explicit record always wins over the formula. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…p_routing The structs moved to entrypoints/control_requests.py in this layer, but these two files kept importing them from entrypoints/utils.py: the server failed to import and the unit-test collection errored. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ent supports test_dp_routing's warmup fan-out and auto-residency merge tests exercise the third layer's scheduler client; they return with it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ts' into HEAD # Conflicts: # python/sglang/multimodal_gen/test/unit/test_dp_routing.py
…-measurements # Conflicts: # python/sglang/multimodal_gen/test/test_utils.py
…ts' into feat/residency-warmup-measurements # Conflicts: # python/sglang/multimodal_gen/test/unit/test_dp_routing.py
…nto feat/residency-planner-solver
…-planner-solver # Conflicts: # python/sglang/multimodal_gen/test/unit/test_dp_routing.py
…he bounded shape The 2-rank IPC all-to-all caches staging buffers keyed by message size and evicts by count only, so the full-shape probe left ~5 GiB of buffers sized for 1280x720x81f behind on each rank; wan2_2_t2v_a14b_2gpu then served at 20.3 GiB allocated against main's 15.1 GiB with the same placement. Release the staging cache together with the allocator pool before the request that follows the probe. Probe lightening also walked below the bounded warmup shape down to 16x16x1f, which models reject; stop at the first shape at or below the smallest measured record instead.
…und the warmup probe
…ites around the warmup probe" This reverts commit dddfc33.
added 4 commits
September 6, 2026 19:51
…r-solver # Conflicts: # python/sglang/multimodal_gen/runtime/managers/memory_managers/auto_residency.py # python/sglang/multimodal_gen/runtime/managers/memory_managers/layerwise_offload.py # python/sglang/multimodal_gen/test/unit/test_auto_residency.py
3 tasks done
…r-solver # Conflicts: # python/sglang/multimodal_gen/runtime/managers/memory_managers/layerwise_offload.py
Collaborator
Author
|
/tag-and-rerun-ci |
This was referenced Sep 21, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second of four stacked PRs split out of #35335. Base: the measurement layer. This layer is a library of pure functions: it plans, but nothing calls it yet.
What it adds
collect_residency_targetslists every complete placement a component could take (resident, component offload, layerwise with N resident layers and M pinned layers) with its signed cost on each named resource: per-phase device headroom, transition-time headroom, host pin budget, host repack space.plan_auto_residencypicks one complete placement per component across all ranks' reports: consensus candidates, binding phase constraints, a VRAM reserve, latency-equivalence tie-breaks, backed byplacement_budget.optimize_placement._EstimatedLayerwiseManagerfor components that were never configured layerwise), the H2D-savings ranking the residency hint reuses, and the plan summary formatters.host_memory_budget:HostPinBudget.release/for_local_worker, the pin reserve, storage-deduplicated tensor sizing.layerwise_offload: the layer byte estimation helpers the planner sizes unconfigured managers with.Tests
test_placement_budget(the optimizer),test_host_memory_budget, and the solver tests intest_auto_residency(candidate frontiers, phase constraints, pins, consensus, promotions on GB10- and H100-shaped reports). All pure, no GPU.Stack
Every layer's tree is a prefix of #35335's; the union is byte-identical to the original branch merged with main.
CI States
Latest PR Test (Base): ✅ Run #37804302290
Latest PR Test (Extra): ❌ Run #37804301772
Latest PR Test (AMD ROCm 10): ❌ Run #37804302462