Repository navigation
[Bugfix][Core] Fail requests the KV cache pool can never hold instead of retrying them forever - #52530
[Bugfix][Core] Fail requests the KV cache pool can never hold instead of retrying them forever#52530malaiwah wants to merge 2 commits into
Conversation
… of retrying them forever On a hybrid Mamba + full-attention model in `--mamba-cache-mode align`, a request near the KV pool ceiling is admitted, chunk-prefilled to ~99% of the pool, descheduled when the last chunk cannot be allocated, and then never scheduled again: zero output tokens, the prefill discarded, and the HTTP request never returns. Startup sizes the pool from each spec's `max_memory_usage_bytes` but compares against all blocks, while `BlockPool` permanently reserves one as the null block, so a pool built at exactly the startup minimum is one usable block short of one `max_model_len` request. Runtime admission uses the per-step allocators, which ask for one block less than the pool is short by on a first prefill, so it admits a request no future pool state can hold. Add `max_servable_num_tokens()` in `kv_cache_utils`: the longest sequence a pool of `num_blocks` blocks can hold for one request, computed with the same `_estimate_max_model_len_from_groups` helper the startup check reports as "the estimated maximum model length", evaluated against `num_blocks - 1` blocks. The scheduler evaluates it once at construction, logs an error when it is below `max_model_len`, and fails an over-long request at admission as `FINISHED_IGNORED` instead of queueing it forever. Behaviour is unchanged whenever the pool covers `max_model_len`. Fixes vllm-project#52520. Signed-off-by: Michel Belleau <michel.belleau@malaiwah.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment Once the PR is approved or has the If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
brianosaurus
left a comment
brianosaurus
left a comment
There was a problem hiding this comment.
The null-block accounting is the right diagnosis. BlockPool pops one block as
the null block at core/block_pool.py:190, so the pool really does have one
fewer usable block than the startup check prices, and
usable_blocks = max(num_blocks - 1, 0) matches that exactly.
Two things I checked so the next reviewer doesn't have to:
- The deferred cleanup does not hang.
unservable_reqssit in no queue, but
the PR adds them toget_num_unfinished_requests, which feeds
has_unfinished_requests(sched/interface.py:176) and thenhas_requests,
soEngineCore.stepdoes not take its early return andupdate_from_output
is reached. Sound. - The
max_servable >= max_model_lenshort-circuit in_is_unservablekeeps
this free on the common path.
One scoping question. _is_unservable uses request.num_tokens + 1, which at
admission is the prompt plus one output slot, but a generate request grows to
prompt + max_tokens. With max_model_len=8192 and
max_servable_num_tokens=8000:
prompt=7000, max_tokens=1000
frontend 7000 + 1000 = 8000 <= 8192 accepted
_is_unservable 7000 + 1 = 7001 <= 8000 admitted
at token 8000 needs a block the pool does not have
That request is descheduled mid-generation and re-prefills into the same wall,
which is the symptom this PR removes for prefill. Is the decode case out of
scope deliberately, on the grounds that the gap between max_servable and
max_model_len is only ever a block or two? If it can be wider, bounding on
num_tokens + max_tokens for generate requests would close it.
|
Thank you - and the decode case is a real gap, not a deliberate exclusion. You are right about the arithmetic: I would rather measure that than argue it, so I am reproducing your exact example on unmodified upstream
On the gap width, your framing is the right one: if I will report the measurement here either way, including if it shows the decode case is benign and this PR should stay scoped to admission. Thanks also for checking the deferred-cleanup path against |
…x_model_len The admission gate in the previous commit bounds a request on `num_tokens + 1`, but a generate request grows to `prompt + max_tokens`. A request that clears both the frontend's `prompt + max_tokens <= max_model_len` check and admission can still walk into a block the pool does not have, during decode. Measured on unmodified main at 4d2a68d, CPU only, on the harness this branch was gated on: block_size 16, max_model_len 1024, a 69-block pool (max_servable_num_tokens = 1008), prompt=1000 and max_tokens=24. The sequence reaches 1009 tokens at output token 9 and is preempted there; 1009 is then longer than the pool can re-prefill, so the next 3987 scheduler steps scheduled zero tokens, produced no output and left all 68 blocks free. `vllm:num_preemptions_total` increments once and never again. The decode case collapses into the prefill case of vllm-project#52520 rather than having a failure mode of its own. Bounding admission on `num_tokens + max_tokens` would close it, but would also refuse requests that stop long before the wall: the same pool runs prompt=1000/max_tokens=8 to completion in 12 steps. Treat the pool's ceiling as what it is instead -- a length cap. `check_stop` now takes `max_servable_num_tokens`, which equals `max_model_len` on every pool that covers its window, so the common path is byte-for-byte unchanged; where they differ the request stops on the ceiling and returns the tokens it produced with finish_reason `length`, exactly as if `max_model_len` had been set to what the pool can serve. Signed-off-by: Michel Belleau <michel.belleau@malaiwah.com>
|
Measured: it livelocks. The decode case is the same defect, not a deliberate exclusion, and it is now fixed in this PR. Reproduction on unmodified upstream
|
| run | pool | request | result |
|---|---|---|---|
| decode-over-ceiling | 69 blocks | prompt 1000, max_tokens 24 | never finishes, 9/24 tokens, stalled from step 13 to step 3999 |
| control, one spare block | 70 blocks | prompt 1000, max_tokens 24 | finishes 24/24 in 28 steps, 0 preemptions |
| control, under the ceiling | 69 blocks | prompt 1000, max_tokens 8 | finishes 8/8 in 12 steps, 0 preemptions |
One correction to the signature I expected. vllm:num_preemptions_total increments once and then stops — it does not climb. The request is preempted at token 1009, and 1009 > 1008 means it is now too long for its own re-prefill, so it collapses straight into the state #52520 describes: scheduled=0 forever, waiting non-empty, pool idle, one EngineCoreEventType.PREEMPTED event ever. The decode case does not have a distinct failure mode; it walks into the prefill case.
The fix, and why not the bound you suggested
num_tokens + max_tokens at admission does catch this request. It also refuses every request that asks for generous headroom and stops long before the wall — and the third row of the table above is that request: prompt=1000, max_tokens=8 runs to completion on the very same 69-block pool. Rejecting on max_tokens trades a livelock for a false rejection on the common path, and callers routinely send max_tokens an order of magnitude above what the model actually emits.
So the bound belongs at the point of exhaustion, and it is one line:
- stopped = check_stop(request, self.max_model_len)
+ stopped = check_stop(request, self.max_servable_num_tokens)max_servable_num_tokens <= max_model_len holds by construction — _estimate_max_model_len_from_groups binary-searches with right = original_max (kv_cache_utils.py:1960 on main) — and the scheduler already falls back to max_model_len itself when there is no per-block bound to price. So on every pool that covers its window this is byte-for-byte the call it was. Where they differ, the request stops on the pool's ceiling and returns what it produced with finish_reason: length, exactly as if max_model_len had been set to what the pool can actually serve. In the repro it now finishes 8/24 in 12 steps with zero preemptions instead of freezing at 9/24 forever.
I considered failing it with an error at exhaustion instead, and rejected that too: by the time the ceiling is hit the client has already been streamed 8 tokens, and length is the truthful reason — the sequence hit a context limit, just a lower one than advertised. The operator gets the actionable version once, at startup, from the logger.error that already fires; it now also names the generation ceiling.
That makes it one bound with three call sites — startup sizing, admission, and the length cap — which is the shape #40946 settled on.
Gap width
You asked whether the gap can be wider than a block or two. I measured rather than argued, and the answer corrects something I wrote earlier in this thread.
The gap is exactly one block, never more, and it is ((max_model_len - 1) mod block_size) + 1 tokens — i.e. however much of its last block max_model_len occupies. It is also zero as soon as the pool has one block more than the startup minimum. Fourteen configurations, on unmodified main:
block max_model_len nspec need smallest_ok servable gap_tok gap_blk | +1blk_servable +1blk_gap
16 1024 3 69 69 1008 16 1.00 | 1024 0
16 1024 0 66 69 1008 16 1.00 | 1024 0
16 1000 3 68 68 992 8 0.50 | 1000 0
16 1009 3 69 69 1008 1 0.06 | 1009 0
16 4096 3 261 261 4080 16 1.00 | 4096 0
32 1024 3 37 37 992 32 1.00 | 1024 0
64 1024 3 21 21 960 64 1.00 | 1024 0
128 1024 3 13 13 896 128 1.00 | 1024 0
128 8192 3 69 69 8064 128 1.00 | 8192 0
128 8192 16 82 78 8192 0 0.00 | 8192 0
16 8192 16 530 526 8192 0 0.00 | 8192 0
256 8192 8 42 38 8192 0 0.00 | 8192 0
256 8192 0 34 37 7936 256 1.00 | 8192 0
512 32768 8 74 70 32768 0 0.00 | 32768 0
Contributors, with what each one is actually worth:
- The null block — the entire gap.
BlockPoolpops one block atcore/block_pool.py:190, and_check_enough_kv_cache_memoryis handed the byte count for all blocks (kv_cache_utils.py:2193). Since that check also refuses any pool below the minimum, the pool is short by exactly one block or by nothing at all. One block, never two. - Block rounding — decides how much of that block is lost.
_max_memory_usage_bytes_from_groupssumscdiv(spec_bytes, page_size)per group (kv_cache_utils.py:1933) andFullAttentionSpec.max_memory_usage_bytesiscdiv(max_model_len, block_size) * page_size(kv_cache_interface.py:305), so the lost block costs whatevermax_model_lenoccupies in its last block: the fullblock_sizewhen it divides evenly, less otherwise. Rows 3 and 4 above aremax_model_len1000 and 1009 atblock_size16 — 8 tokens and 1 token. - The speculative reserve — not a contributor. I was wrong about this above.
MambaSpec.max_memory_usage_bytescharges2 + num_speculative_blocksblocks in align mode (kv_cache_interface.py:694), but it charges the same amount on both sides of the comparison, so it moves the requirement and the ceiling together. Rows 1 and 2 arenum_speculative_blocks3 and 0 at the same block size, and both gaps are 16. - One spare block closes it completely, because the estimator's binary search is capped at
max_model_len(kv_cache_utils.py:1960). Every row's right-hand columns are 0.
So the gap is bounded by block_size rather than unbounded — but block_size is 128 or 256 in plenty of real deployments, and the block_size=128, max_model_len=8192 row lands at 8064 vs 8192, which is within touching distance of the 8000/8192 you wrote by hand. Your read on the magnitude was right; the mechanism is one block, not an accumulation of three.
One thing I checked and deliberately did not change
The running-request path asks the allocator for 1 + len(spec_token_ids) slots, so a draft proposal can reach past the ceiling before the length cap can fire. I injected drafts into the same harness on stock main: the request preempts twice around the ceiling and then finishes, because preemption clears spec_token_ids (sched/scheduler.py:1389) and the retry decodes one token at a time straight into the cap. Bounded and self-correcting, identical before and after this PR. I left scheduler.py:605 alone rather than widen the diff on something I could not show was broken.
Tests
Two new cases in the existing file, same style:
test_generation_stops_at_the_bound_instead_of_stalling_mid_decode— asserts the request clears both length checks, then must finishFINISHED_LENGTH_CAPPEDatSERVABLE_LENwithnum_preemptions == 0and nothing left unfinished. With only thecheck_stopline reverted: 1 failed, 6 passed,assert RequestStatus.PREEMPTED == RequestStatus.FINISHED_LENGTH_CAPPED.test_generation_is_not_capped_when_the_pool_covers_the_window— the same request with one block of headroom must still produce all 24 tokens, so the cap cannot silently truncate generation on the common path. This one passes both before and after, by design.
On unmodified main the file is 5 failed, 2 passed; on this branch, 7 passed. tests/v1/core/ is 2 failed, 502 passed, 2 errors — the same four GPU-requiring cases as before, unchanged.
Thanks again for tracing unservable_reqs through to has_unfinished_requests; the decode path deliberately does not use that deferred set — it finishes through the ordinary stopped path in _update_request_with_output — but knowing the first one was sound is what let me leave it alone and put this in a different place.
|
Thank you for measuring this rather than taking my word for it. The 4000-step You are right that my suggested bound was the wrong fix. The third control row Thanks also for going back and correcting the gap width. Nothing further from |
|
Closing in favor of the maintainer-requested path. #47272 has merged as The pre-#47272 CPU reproduction and decode trace remain valid historical evidence for the defect as it existed, but this implementation is superseded. Our downstream clean-runtime rebuild will adopt #47272 and remove the #52530 hand-port. If a request-side failure independently survives on a post-#47272 tree, it should be demonstrated there and filed as a new narrowly scoped change rather than extending this branch. |
Purpose
Fixes #52520.
The symptom
On a hybrid Mamba + full-attention model with
--mamba-cache-mode align, a requestwhose length is near the KV pool ceiling is accepted, chunk-prefilled to ~99 % of
the pool, descheduled when the last chunk cannot be allocated — and then never
scheduled again. Zero output tokens, the whole prefill discarded, and the only
observable is
vllm:num_requests_waiting_by_reason{reason="capacity"} = 1, which isindistinguishable from a healthy server that is momentarily full. The HTTP request
never returns.
The engine boots happily into this state:
_check_enough_kv_cache_memorypasses.The mechanism
Two bounds describe the same quantity — how many blocks one
max_model_lenrequestcan hold at once — and they are computed by different code that does not agree.
Startup sizes the pool from each spec's
max_memory_usage_bytes(
vllm/v1/core/kv_cache_utils.py:2189-2198). For a hybrid Mamba model in align modethat is
cdiv(max_model_len, block_size)blocks for the attention group plus2 + num_speculative_blocksfor the Mamba group(
vllm/v1/kv_cache_interface.py:687-696). It compares that against all theblocks the pool will have. But
BlockPoolpermanently reserves one block as the nullblock (
vllm/v1/core/block_pool.py:188-191), so onlynum_blocks - 1are everavailable to a request. A pool built at exactly the startup minimum is one usable
block short of one
max_model_lenrequest, and startup does not notice.Runtime admission (
vllm/v1/core/kv_cache_manager.py:475-491, reached withscheduler_reserve_full_isl— defaultTrue) asks the coordinator for the samequantity, but through the per-step allocators. For the Mamba align group that path
returns
1 + num_speculative_blockson a first prefill(
vllm/v1/core/single_type_kv_cache_manager.py:1529-1540), one block below the peakthe spec itself states, because a later step allocates the next state block before
remove_skipped_blocksretires the one from two steps ago. So admission asks for oneblock less than the pool is short by, and admits.
The request then chunk-prefills until the final chunk needs the block that does not
exist. It is the only running request, so it preempts itself
(
vllm/v1/core/sched/scheduler.py:676-688):num_computed_tokensback to 0, allblocks freed, requeued. On the next step the same gate now also has to count the
blocks the discarded prefill left in the prefix cache as evictable, so it refuses —
and keeps refusing. The request is stuck, and the prefill is gone.
Measured on unmodified
mainat4d2a68d64d9e05921ed5c4099146e768a92d71d5, CPU only,no model weights: hybrid full-attention + Mamba(align),
block_size=16,max_model_len=1024,num_speculative_blocks=3,max_num_batched_tokens=256. Thestartup bound is
cdiv(1024,16)=64 + (2+3)=5 = 69blocks, andget_kv_cache_configsaccepts a 69-block pool:One block of headroom is the entire difference. Note that the prefill dies at exactly
1008 tokens, which is
(69 - 1 - 5) * 16— the sequence length the pool can actuallyserve.
The fix
One function computes the bound, and it is the function the startup error message
already uses:
max_servable_num_tokens(vllm_config, kv_cache_groups, num_blocks)inkv_cache_utils.py— the longest sequence a pool ofnum_blocksblocks can holdfor one request. It is
_estimate_max_model_len_from_groups(the helper behind"the estimated maximum model length" that
_check_enough_kv_cache_memoryprints)evaluated against
num_blocks - 1blocks, i.e. against the capacityBlockPoolwill actually hand out. It returns
Nonefor group layouts that cannot be pricedper block — exactly the layouts whose pool was not sized per block either — so it
never invents a bound the startup check did not apply.
Schedulerevaluates it once at construction. When it is belowmax_model_lentheconfiguration cannot serve its own window, which is logged at
ERRORwith thenumber the operator needs. A request past the bound can never be scheduled at any
future time, however much drains, so
add_requestfails it immediately instead ofqueueing it: no prefill is spent, nothing is preempted, and the client gets a
terminal response rather than a request that never returns.
So there is one bound with two call sites, which is the point:
_estimate_max_model_len_from_groupsis called by the startup check(
kv_cache_utils.py:2234, via_check_enough_kv_cache_memory) and now bymax_servable_num_tokens, which is what the scheduler admits against. Neither sidecan grow its own formula without the other seeing it.
The failure uses
RequestStatus.FINISHED_IGNORED, whichvllm/v1/request.pyalreadydocuments as the status for "requests whose prompt lengths are longer than the
model's length cap", mapping to
finish_reason="length".FinishReason.ERRORisdocumented as a retryable internal error always converted to a 500, which is the
wrong contract here. Returning a literal 400 would mean exposing the pool-derived
bound to
input_processor._validate_prompt_len, wheremax_model_lenis validated;happy to do that in a follow-up if maintainers prefer it — it is a frontend/engine-core
plumbing change and I kept it out of this PR.
Behaviour is unchanged whenever the pool covers
max_model_len, which is everycorrectly-sized deployment: the bound is then
>= max_model_len,_is_unservableshort-circuits on the first comparison, and nothing is rejected. The 70-block row
above is byte-for-byte identical before and after.
Relationship to #47272 — scoped around it, not a duplicate
I read #47272 ("Reserve the KV null block when validating
max_model_len") beforewriting any code, and this PR is deliberately scoped around it.
startup check validates against, so a pool that cannot serve
max_model_lenisrefused at boot with the existing message naming the estimated maximum model
length. That is the right place for it and this PR does not touch that code path.
Applied to the reproduction above, [Bugfix][Core] Reserve the KV null block when validating max_model_len #47272 makes the 69-block pool illegal at
startup — which is a better outcome than anything the scheduler can do, and is why
I am not proposing an alternative to it.
the startup bound, and gives the engine an answer for a request it cannot serve
other than retrying it. That is needed independently of [Bugfix][Core] Reserve the KV null block when validating max_model_len #47272, because the
scheduler currently has no concept of "this request can never be scheduled": it
only distinguishes "not now". If [Bugfix][Core] Reserve the KV null block when validating max_model_len #47272 lands, this PR's check becomes dormant on
correctly-sized pools (bound
>= max_model_len) and remains the backstop for thepaths [Bugfix][Core] Reserve the KV null block when validating max_model_len #47272 does not cover — notably
_auto_fit_max_model_len, which runs before[Bugfix][Core] Reserve the KV null block when validating max_model_len #47272's reservation and can still land on the boundary.
instead of [Bugfix][Core] Reserve the KV null block when validating max_model_len #47272.
Other prior art I checked: #40946 (merged) is the design precedent — it fixed the
same class of defect (startup pool sizing and runtime admission using different
formulas) for SWA/chunked-local by putting the bound in one method called from both
sites, and this PR follows that shape. #35541 (closed not-planned/stale) asked for
"abort requests that can never fit in the KV cache block pool with a clear error
rather than hanging indefinitely" — this is that, for the mechanism above. #40384
is why the numbers here are stated in blocks and tokens rather than inverted from the
GPU KV cache sizelog line, which on a hybrid model isnum_blocks // len(kv_cache_groups) * min_block_size.Scope of the claim
The field trace in #52520 came from a downstream fork build on a hybrid Qwen3.5
model (261,794-token request,
--enable-prefix-caching --mamba-cache-mode align, MTPdepth 3, pool within ~1 % of the window; 63 full re-prefills over 656 s with zero
output tokens). The mechanism above is upstream code, and everything in this PR is
demonstrated on stock
mainat4d2a68d64d9e05921ed5c4099146e768a92d71d5by theCPU-only unit test below — no GPU, no model weights, no fork code involved.
One part of the original report I could not reproduce on
mainand am thereforenot claiming:
vllm:num_preemptions_totalstaying at0.0across the re-prefills.On
mainthe self-preemption path does incrementrequest.num_preemptionsand recordEngineCoreEventType.PREEMPTED, whichIterationStatscounts intovllm:num_preemptions_total. So there is no accounting fix here, and I am not filingone. (With this PR the point is moot for this defect: nothing is preempted, because
nothing is admitted, and the operator gets two
ERRORlines naming the limit.)The decode case, added in the second commit
Review raised it and I measured it on the same stock-
mainharness: the admissiongate bounds a request on
num_tokens + 1, but a generate request grows toprompt + max_tokens. Withblock_size16,max_model_len1024 and the 69-blockpool above (
max_servable_num_tokens1008),prompt=1000, max_tokens=24clears thefrontend (
1000 + 24 <= 1024,vllm/renderers/params.py:210,442) and clearsadmission (
1000 + 1 <= 1008), and then reaches 1009 tokens at output token 9. Onmainit is preempted there, is now too long to re-prefill, and the next 3987scheduler steps schedule zero tokens, produce no output and leave all 68 blocks
free — the same terminal state as the prefill case, reached through decode.
Bounding admission on
num_tokens + max_tokenswould close it, but would refuserequests that stop well before the wall: on that same pool,
max_tokens=8runs tocompletion in 12 steps. So the ceiling is applied where it belongs, as a length cap:
check_stoptakesmax_servable_num_tokensinstead ofmax_model_len. Those areequal on every pool that covers its window —
_estimate_max_model_len_from_groupssearches with
right = original_max(vllm/v1/core/kv_cache_utils.py:1960), and thescheduler falls back to
max_model_lenwhen there is no per-block bound — so thecommon path is unchanged. Where they differ the request stops on the ceiling and
returns its tokens with
finish_reason: length, exactly as ifmax_model_lenhadbeen set to what the pool can serve.
That is one bound with three call sites: startup sizing, admission, and the length
cap.
The spec-decode lookahead can also reach past the ceiling, because the running path
asks for
1 + len(spec_token_ids)slots. Measured on stockmainwith draftsinjected: it preempts twice and then finishes, because preemption clears
spec_token_idsand the retry decodes into the cap. Bounded and self-correcting, sosched/scheduler.py:605is left alone.Test Plan
pytest tests/v1/core/test_kv_pool_unservable_requests.py # surrounding suite, unchanged behaviour: pytest tests/v1/core/tests/v1/core/test_kv_pool_unservable_requests.pyis CPU-only(
pytestmark = pytest.mark.cpu_test), built the same way astests/v1/core/test_mamba_align_chunk_split.py: a realSchedulerover a hand-builthybrid
KVCacheConfig, driven withschedule()/update_from_output(). No modelweights, no GPU. Seven cases:
test_max_servable_num_tokens_reserves_the_null_block— the bound is(69 - 1 - 5) * 16 = 1008at the startup minimum, and>= max_model_lenwith oneblock of headroom.
test_unservable_request_is_failed_not_prefilled_and_retried— the regressiontest. A 1023-token request on the 69-block pool: nothing is scheduled, it is
finished as
FINISHED_IGNORED,num_computed_tokens == 0,num_preemptions == 0, and no unfinished request is left behind.test_request_at_the_bound_is_failed_because_it_needs_an_output_slot— the boundis a sequence length, so a prompt of exactly 1008 is already too long (it was the
one length that prefilled fully on
mainand then stalled on its first decodestep).
test_servable_request_at_the_same_pool_still_runs— 1007 tokens on the same69-block pool prefills fully, decodes, and finishes. The bound must not
over-reject.
test_pool_with_headroom_serves_the_whole_window— 1023 tokens on a 70-block poolbehaves exactly as before this PR.
test_generation_stops_at_the_bound_instead_of_stalling_mid_decode— the decoderegression test.
prompt=1000, max_tokens=24on the 69-block pool clears bothlength checks, then must finish
FINISHED_LENGTH_CAPPEDat 1008 tokens withnum_preemptions == 0and nothing left unfinished.test_generation_is_not_capped_when_the_pool_covers_the_window— the same requestwith one block of headroom must still produce all 24 tokens, so the length cap
cannot silently truncate generation on the common path.
Test Result
The new file, checked out on top of unmodified
mainat4d2a68d6(the fix reverted,the test kept), fails:
{'0': 256}ismainadmitting the unservable request and scheduling the first256-token prefill chunk of the sequence it will then throw away. The two that pass
before and after are the two "must still work" directions — they are there to catch
over-rejection and over-capping, and they must never have gone red.
Reverting only the one-line length cap and keeping everything else isolates the
decode case:
With the fix:
Surrounding suite, same command on
4d2a68d6and on this branch:The same two failures and the same two setup errors occur on both, all environmental
in my CPU-only sandbox and unrelated to this change: failures in
test_scheduler.py::test_async_scheduling_pp_allows_rescheduling_with_output_placeholdersand
test_reset_prefix_cache_e2e.py::test_reset_prefix_cache_e2e, and setup errors ontest_concurrent_partial_prefillandtest_prefix_cache_stats_is_recorded— all fourneed a GPU-backed engine core. The +7 are the new tests.
Two existing tests build a
Schedulerwithobject.__new__and set its attributes byhand; they each gain one line for the new
unservable_reqsset and one formax_servable_num_tokens(
tests/v1/core/test_scheduler.py,tests/v1/core/test_async_scheduler.py).Lint gates on the changed files:
AI assistance was used to investigate and draft this change; every changed line was
reviewed, and the reproduction, the tests and the lint gates were run locally as
described above.
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model. — none needed: no new model, flag, or user-facing configuration.