[Restore] Reconcile audited #292 history with latest main - #352
Merged
Merged
Conversation
added 5 commits
August 25, 2026 09:03
…nder async scheduling With async scheduling / PP, `num_computed_tokens` optimistically includes tokens of steps whose output is not yet processed; their attention windows still read the blocks just below the optimistic boundary, and rejected speculative tokens can roll it back. Freeing recycling-aware KV (sliding window, chunked-local, prefix-anchored SWA gap eviction) on that optimistic basis can release blocks an in-flight step still reads. Track `Request.num_in_flight_tokens` in the scheduler and free on `num_computed_tokens - num_in_flight_tokens` in `allocate_slots` and in the pre-connector prune. Thread the renamed `max_in_flight_tokens` bound (`max_concurrent_batches * max_num_batched_tokens`) through the KV cache manager/coordinator and the per-request admission caps so startup pool sizing reserves for overlapping batches. Tests: in-flight accounting, sliding-window/chunked-local/prefix-anchored frees deferred until the in-flight step settles (and unchanged under sync scheduling), admission caps under overlapping batches, and the connector-finish prune on the settled basis. Signed-off-by: kkobold <sabbaghdanilo@gmai.com>
Some models are only correct when served with a specific decoding recipe; DeepSeek-OCR's documentation makes its anti-repetition logits processor mandatory for long documents and requires `skip_special_tokens=False` with greedy decoding. Serving without the recipe produces documented looping with no visible error. Add a serving-layer registry keyed by model architecture (extensible by out-of-tree plugins at import time): registered sampling defaults flow through the existing `default_sampling_params` channel (with `skip_special_tokens` now tri-state on the chat request so a model default can apply), recipe `extra_args` (the processor parameters) are filled in for requests that omit them, and client-supplied `vllm_xargs` overrides are validated against registered bounds — out-of-range values return HTTP 400 instead of reaching the engine. Startup fails loudly when a recipe-required logits processor is not loaded. DeepSeek-OCR's recipe is registered in-tree. Signed-off-by: kkobold <sabbaghdanilo@gmai.com>
…CR prompt The DeepSeek-OCR checkpoint reports model_type "deepseek_vl_v2", so the chat endpoint resolved the DeepSeek-VL2 fallback template, which inserts chat role markers and silently changes the OCR prompt relative to the documented recipe — a quality failure with no error. Add an architecture-keyed fallback map that takes precedence over the model_type map (public registration for plugins included) and key DeepseekOCRForCausalLM to the raw-concatenation OCR template; genuine DeepSeek-VL2 chat checkpoints keep their template. Golden tests pin the resolution and the byte-exact rendered prompts for both documented OCR modes, with a negative control proving the goldens discriminate. Signed-off-by: kkobold <sabbaghdanilo@gmai.com>
Degeneration patterns longer than the n-gram processor's ban window (e.g. slow-drift empty-table rows) can still bloat output by thousands of junk tokens. Registered models may now declare repetition_detection defaults, applied when the request leaves the field unset; a request-supplied value always wins. DeepSeek-OCR registers a conservative profile grounded in the observed failure shape (min_count=8 spares legitimate dense tables). Signed-off-by: kkobold <sabbaghdanilo@gmai.com>
…ving profile Plumb the official resolution modes (tiny/small/base/large/gundam) through DeepseekOCRProcessor as a per-request image_mode selector, with crop bound plumbing (min_crops/max_crops) and an opt-in multi-image crop safeguard. Token counting moves to a single shared arithmetic (count_image_tokens_for) used by both the processor pixel path and the model's placeholder counter, making the producer/counter contract hold by construction for every mode, crop cap, and image count. The serving defaults registry gains per-image-count profiles (sampling_defaults_multi_image / extra_args_defaults_multi_image); the chat completion path counts request image items and selects the profile before merging request overrides. Signed-off-by: kkobold <sabbaghdanilo@gmai.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…-20260828-1334 [Restore] Reconcile #352 with audited latest main
yangzhuxinyzx
changed the title
Contributor
Author
yangzhuxinyzx
marked this pull request as ready for review
August 28, 2026 05:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Preserve the recoverable history and authorship of archived #292 while reconciling it with the audited latest
main.Final audit
main, with settled-token freeing and admission sizing retained.main.3a8f3a51c1eba9eaa9b8df174330e3f181500496remains in ancestry, preserving authorship and review provenance.Test Result
37 passed.327e4a37fa55830f58c5e18d86e6c6a3af068e9b.a6d9af1056de0db77294f2209f86c39e24eb5def.a6d9af1056de0db77294f2209f86c39e24eb5def.Decision
This is an intentional zero-source-difference merge: useful generic fixes stay in
main, rejected identity defaults stay out, and GitHub records the restored contribution as merged rather than closed.Repair chain: #383.