feat(planner): rate-bound (Little's Law) decode scale-down projection - #9386
Closed
tedzhouhk wants to merge 1 commit into
Closed
feat(planner): rate-bound (Little's Law) decode scale-down projection#9386tedzhouhk wants to merge 1 commit into
tedzhouhk wants to merge 1 commit into
Conversation
…-down
Replaces the linear "scale KV by N/(N-1)" projection used in
``_decode_load_decision``'s SLA check with the closed-form fixed point
of the steady-state Little's Law equation. The new projection captures
catastrophic saturation as ``itl_curr`` approaches the rate-bound
capacity ``N * intercept``, which the linear extrapolation misses.
## Math
The decode regression decomposes ITL into a fixed cost plus a
load-dependent variable cost:
itl_curr = intercept + V_curr
where ``V_curr = c_req * num_req + c_kv * kv`` is the variable portion.
At steady state Little's Law ties per-worker concurrency to the product
of arrival rate and time-in-system. Time-in-system for a decode token
is ~ITL, so the variable load on the survivor after N -> N-1 scales by:
1. ``N/(N-1)``: arrival rate per worker after losing a worker.
2. ``itl_post / itl_curr``: longer ITL means each request lingers
longer in the batch, further inflating concurrency.
The variable cost on the survivor becomes:
V_post = (itl_post / itl_curr) * (N / (N-1)) * V_curr
Substituting ``itl_post = intercept + V_post`` and solving the linear
fixed point in ``itl_post`` yields:
itl_post = (N-1) * intercept * itl_curr / (N * intercept - itl_curr)
The denominator goes to zero as ``itl_curr -> N * intercept``; past
that, one fewer worker physically cannot sustain the offered load.
## Why this matters
For the customer's logs (regression intercept ~17.9 ms, current ITL
~30.5 ms at 2-worker steady state with 177K KV each, N=2):
- Linear "scale kv" extrapolation: predicts ``itl_post ~ 43 ms``
(barely above 32 ms threshold).
- Rate-bound closed form: predicts ``itl_post ~ 104 ms``.
Both refuse against the 32 ms threshold in this case, but for
borderline scenarios near the rate-bound capacity the linear
approximation under-predicts and lets unsafe scale-downs through.
## Changes
- ``DecodeRegressionModel.intercept_seconds`` (new property) and
``estimate_post_consolidation_itl`` (closed-form helper) -- both
carry the math derivation in their docstrings.
- ``_decode_load_decision`` queries the closed form first; falls back
to the previous linear projection when the closed form is
unavailable (non-positive intercept from noisy fit, unfitted
regression).
- Hard cache feasibility check remains unchanged -- still independent
of the SLA model.
- Tests:
* ``TestEstimatePostConsolidationItl`` unit tests for the closed
form (saturation, sub-saturation, fallback paths, N-sensitivity).
* ``TestDecodeConsolidationAwareScaleDown`` integration tests
rewritten to exercise the rate-bound regime explicitly:
below-saturation permit, SLA breach refusal, rate-bound
saturation refusal, cache fail-safe.
All 380 planner unit tests pass.
Note: prefill / agg-prefill / agg-decode keep their existing
projections. The rate-bound derivation is specific to decode's
batched-iteration physics; prefill's queue-induced TTFT separation
remains the right model there.
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
WalkthroughThis PR introduces a closed-form rate-bound ITL projection method for decode worker consolidation decisions, replacing simpler linear extrapolation. The regression model now exposes fitted intercept values and post-consolidation ITL estimates; the load-scaling consolidation check uses these to validate SLA feasibility with fallback to linear estimation when unavailable. Tests updated to cover rate-bound scenarios explicitly. ChangesRate-bound ITL Consolidation Check
🎯 3 (Moderate) | ⏱️ ~22 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days. |
Contributor
|
This PR has been closed due to inactivity. If you believe this PR is still relevant, please feel free to reopen it with additional context or information. |
github-actions
Bot
deleted the
hzhou/planner-rate-bound-decode-consolidation
branch
June 21, 2026 10:41
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the linear
post_kv = (sched + queued) * N/(N-1)projection used in_decode_load_decisionfor the post-consolidation SLA check (merged via #9294) with the closed-form fixed point of the steady-state Little's Law equation. The new projection captures catastrophic saturation asitl_currapproaches the rate-bound capacityN * intercept— a regime the linear extrapolation under-predicts.Math
The decode regression decomposes ITL into a fixed cost plus a load-dependent variable cost:
where
V_curr = c_req * num_req + c_kv * kvis the variable portion (load × regression slopes).At steady state, Little's Law ties per-worker concurrency to the product of arrival rate and time-in-system. For decode, time-in-system ≈ ITL, so the variable load on the survivor after
N → N-1scales by:N/(N-1)— arrival rate per worker after losing a worker (cluster offered load is invariant).itl_post / itl_curr— longer ITL means each request lingers longer in the batch, further inflating concurrency.Hence on the survivor:
Substituting
itl_post = intercept + V_postand solving the linear fixed point:The denominator vanishes as
itl_curr → N × intercept— the rate-bound capacity limit beyond which one fewer worker physically cannot sustain the offered request rate.Why this matters
For the customer's logs (regression
ITL = 7.15e-5·kv + 17.89, current ITL ~30.5 ms at 2-worker steady state with 177K KV each, N=2):itl_postscale kv(merged #9294)Both refuse against the 32 ms threshold in this case, but for borderline scenarios near the rate-bound capacity the linear approximation under-predicts and lets unsafe scale-downs through. The closed form is also dimensionally correct (no extra regression query needed post-consolidation).
Changes
DecodeRegressionModel.intercept_seconds(new property) — exposes the regression's fitted intercept.DecodeRegressionModel.estimate_post_consolidation_itl(new method) — closed-form Little's-Law projection; full derivation in the docstring._decode_load_decision— queries the closed form first; falls back to the previous linear projection when the closed form is unavailable (non-positive intercept from a noisy fit, or regression unfitted).Test plan
TestEstimatePostConsolidationItlunit tests for the closed form: unfitted/N<2 returnsNone, below-intercept returns intercept, at/past saturation returns+inf, sub-saturation matches the formula exactly, higher N is more permissive at the same load.TestDecodeConsolidationAwareScaleDownintegration tests rewritten for the rate-bound regime: below-saturation permits, SLA breach refuses, rate-bound saturation refuses, cache fail-safe still fires.Scope notes
Prefill / agg-prefill / agg-decode keep their existing projections. The rate-bound derivation is specific to decode's batched-iteration physics — prefill's queue-induced TTFT separation (queue scales with consolidation but the new request's own
avg_islcompute does not) remains the right model there.🤖 Generated with Claude Code
Summary by CodeRabbit
Refactor
Tests