Repository navigation
fix: allow concurrent agents to prefill during generation - #1
Open
krunkosaurus wants to merge 78 commits into
Open
krunkosaurus wants to merge 78 commits into
krunkosaurus wants to merge 78 commits into
Conversation
…-Lab#10) and spark_doctor diagnostic script
…AI-Lab#43) The OpenAI layer backs an omitted max_tokens with max_model_len - prompt (~1M on this recipe), so a single client that omits max_tokens can decode effectively forever and grow KV until it preempts every other session. DEFAULT_MAX_NEW_TOKENS (default 65536) installs a server-side bound via --override-generation-config {"max_new_tokens": N} on head AND worker (HF generation_config semantics: max_new_tokens == vLLM max_tokens). Scope note (measured on 2x GB10 TP=2, vLLM build in ghcr image): admission in this vLLM is chunk-based and NOT gated on max_tokens, so this flag does not change admission behavior; long-context concurrency is governed by the effective KV pool (1.41M nominal, ~830K effective with mamba/SWA/draft overhead; a 258K-token request reads ~31% kv_cache_usage_perc). The num_requests_waiting_by_reason{reason="capacity"} gauge is a catch-all for ALL waiting requests in this build (loggers.py sets it from num_waiting_reqs), not a KV-capacity diagnostic. - DEFAULT_MAX_NEW_TOKENS in .env (default 65536, empty = old behavior); explicit client max_tokens always wins - passed to head (-e block) and worker (serve_env loop) - documented in .env.example + README env table; static-anchor tests
…tokens # Conflicts: # start.sh
Field notes from a 4x TP=4 deployment of this kit lineage: prevention (swappiness=0 while KEEPING swap, residual-swap cycling — swapped-out pages being touched later triggered our mid-serve wedge), detection (/health stays 200 throughout; poll /metrics for pending>0 with frozen token counters), and recovery (docker kill + swap cycle + a trivial CUDA probe distinguishes the cosmetic 96%-utilization reading from a truly wedged driver — both incidents recovered without a power cycle). Extends the failure class documented in tonyd2wild OPEN-PROBLEMS MiaAI-Lab#4 with the recovery path that was unknown there. Also ships scripts/spec-accept-gate.sh: vllm#53030 can silently pin per-position acceptance at exactly 1.00 under CUDA graphs; the gate reads the decay curve from /metrics and fails on the pinned signature. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tokens # Conflicts: # README.md # start.sh
…tokens # Conflicts: # README.md
…README Four DGX Sparks, this image/overlay and weights, DFlash2 draft TP=4, 1M context: 24 one-knob experiments with a ten-minute production-mix bench after every relaunch and hard gates, then a 30-min soak on the winner (195 requests, 0 errors). Adds the tuned serve knobs to .env.tp4.example with the measured reason for each, and a short measured paragraph under the experimental TP=4 section. Full recipe and every receipt: https://github.com/punkjazz-labs/glm-5.3-flash-exl3-4x-dgx-spark Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Um6EfS81CU5BXMxYw94pkL
…aft off until fixed Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Um6EfS81CU5BXMxYw94pkL
…two fixes and a repro protocol
…-flight invariant behind the default 4, the in-flight budget, headroom note
…tokens # Conflicts: # README.md # start.sh
…tokens # Conflicts: # start.sh
…y, string tool-args guard) — live in production boots since pre-09-08
…tokens # Conflicts: # files/chat_template.jinja # start.sh
The only kernel that JIT-compiled during live traffic in a 5 h agent serve was BuildPrefillChunkMetadataKernel (jit_monitor WARN, hours after boot): the warmup ladder only exercised <=248-token prompts, so the first long request paid the compile on the user's path. - PREFILL_S=(3584 7168 14336 65536) long-prefill rungs; prefill does not affect the DFlash decode BLOCK shapes (decode is one query). - stage tokenize/completions payloads in files: 64k-token prompts exceed ARG_MAX when passed as a curl argument. - ignore the staged .json payload files in the end-of-run tally. Verified on 2x DGX Spark GB10: prior serve's only mid-serve JIT disappears with the new rungs; boot warmup reports 24/24 requests.
Default to stock chunked prefill and upgrade the legacy overlay default. Retain explicit decode protection and add progress regressions plus a live canary that rejects serialized HTTP-success responses.
Describe the existing per-request prefill cap for sharing a small batch budget, and distinguish the per-request context limit from combined resident KV capacity.
…tokens # Conflicts: # README.md # tests/bench_decode.py
An 11.9 MB video attached in a chat killed the engine on 2026-09-14. The
client decomposed it into ~33 separate full-resolution *image* items —
vLLM never splits one video into multiple encoder items, so more than one
encoder index always means the client sent images. At the checkpoint's
max_image_tokens=8000 each frame cost ~7.2k tokens, the prompt reached
236,544 tokens of vision encode, Node 0 fell to 44 MB free, and the
kernel took the worker with it:
Out of memory: Killed process 1539229 (VLLM::Worker_TP)
Worker proc VllmWorker-0 died unexpectedly (exit code: None)
EngineCore encountered a fatal error
SKIP_MM_PROFILING=1 is mandatory on this UMA kit (the max-size dummy MM
profile OOMs it), so nothing is reserved for the vision tower and every
multimodal token is encoded out of memory the model has already spent.
LIMIT_MM is only a validation ceiling, and image:100 x 8000 tokens was
effectively the whole context window.
Add three caps. All are tunable, and an empty value means stock vLLM:
MM_IMAGE_TOKENS=2048 --mm-processor-kwargs {"max_image_tokens":N}
LIMIT_MM image 100 -> 48 worst case 48 x 2048 = 98k tokens, not 800k
MM_PROCESSOR_CACHE_GB=1 --mm-processor-cache-gb (vLLM default 4 GiB
of host RAM, which on UMA the model cannot have)
VIDEO_NUM_FRAMES= --media-io-kwargs; empty = vLLM's 32. Left at
stock: the crash arrived over the image path.
Measured on the checkpoint's own processor, one 1920x1080 frame costs
2691 tokens uncapped, 2040 at cap 2048, 1008 at 1024 — so this barely
touches ordinary images and only bites oversized ones. A 33-frame
attachment now costs 67,320 tokens and serves; past 48 items the API
returns HTTP 500 "At most N image(s) may be provided in one prompt",
which is an error instead of a dead engine.
The stock 8000 also exceeds MAX_NUM_BATCHED_TOKENS=7168, which is vLLM's
encoder cache size, so a single full-res image could never fit the cache.
The flag is verified to reach the processor with vLLM's own predicate,
supports_kw(MuseGlimmerProcessor.__call__, "max_image_tokens",
allow_var_kwargs=True) -> True. Unknown --mm-processor-kwargs keys are
filtered out with a warning rather than raising, so this cannot cost a boot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017chDHP8k2EaYRmpiNoAhPN
fix(vision): cap per-image tokens so a chat video cannot OOM the host
…s/pr51 # Conflicts: # start.sh
Keeps both sides of the README path table: main's TP3/TP4/NFS rows and the kernel-lab rows added by this branch. No other integration change.
Maintainer decision on MiaAI-Lab#75: retain kernel_lab/, its CPU oracles/tests and docs/exl3-sm121-kernel-lab.md in this PR as development-only work, and remove every production Dockerfile addition from it. The ARG/ENV runtime stamp and the kernel_lab COPY are gone, so the Dockerfile is byte-identical to main's (pinned EXLLAMAV3_COMMIT build argument, compile/overlay layers and late recipe-stamp conventions untouched). README.md, kernel_lab/README.md and the engineering log now describe the lab as a source checkout that the production Dockerfile neither copies nor installs, drop the candidate-image build recipe and the in-image stamp claim, and keep measured provenance fail-closed and declarative: --runtime-commit or the clean checkout's Git identity, the environment override only when it is exactly the loaded source, and a developer-provided backend must set its real EXLLAMAV3_COMMIT before the metadata module is imported, since the pinned fallback describes only the production pin. Historical author results stay attributed and no speedup is claimed for the untimed SM121 sweep. No code changes: kernel_lab/exl3/**, tests/test_exl3_kernel_lab.py are unchanged from the author's head.
…vidence
A missing spec_decode_num_drafts_total family fell through `|| echo 0` into the
<100-drafts SKIP, so "cannot judge" was reported as "not enough traffic"; a
matched accepted_tokens_per_pos_total series without a position label made the
position list fail under pipefail and took the run down rc 1 with no output at
all; and ${v%%.*} read the exponent form (1.5e+03) as its leading digit, so a
live counter looked almost empty and a healthy start was reported as SKIP.
- parse the value token with awk instead of truncating the sample tail: the
optional trailing timestamp is ignored, finite non-negative decimal and
scientific counts are read as counts, anything else is refused
- require the drafts denominator to be present and readable (rc 2) instead of
defaulting it to zero, and require one numeric position label per matched
per-position series (rc 2) instead of judging a silently partial curve
- judge pos0 from the value already parsed in the loop, removing the second
unguarded extraction that could die mid-run
- duplicate label sets stay rc 2, valid low traffic stays rc 0 SKIP and the
PASS/FAIL verdicts stay rc 0/1
- tests: inert fixtures for the missing family, unreadable drafts and position
values, unlabelled (and partly unlabelled) per-position series, and
scientific counts with timestamps
…nly admission docs: historical image4 rejection and current LIMIT_MM admission semantics
…pt construction fix(warmup): cover long-prefill metadata kernels in boot shape warmup
…d causal caveat docs(tp4): dated 4-Spark measurements and corrected stall attribution
skip starves newcomers for the whole decode; CHUNK=0 admits them into shared TP=2 steps and collapses incumbent tok/s. Fair time-shares decode-only stretches with bounded prefill turns, stays opt-in, and migrates v1 images via a versioned installer. Co-authored-by: Cursor <cursoragent@cursor.com>
v4 priced large chunks linearly off 128-token samples and spent credit on tiny steps (~70 tok/s). v5 fits a fixed-plus-per-token cost, saves credit for the largest step that fits MAX_STEP_MS, and was measured on the reporter recipe: 2k newcomers get a first token in ~12 s during the essay instead of waiting for the whole decode. Co-authored-by: Cursor <cursoragent@cursor.com>
.env.example already used fair; .env.tp3.example and .env.tp4.example still overrode it (unset / off). Match start-tp3.sh and start-tp4.sh so every launcher and example ships the same default. Revert with skip. Co-authored-by: Cursor <cursoragent@cursor.com>
…ll-scheduler feat(scheduler): fair v5 mixed-prefill as the TP2 default
…overage main() formatted ttft_s/tok_s with .2f/.1f on values that stay None when a successful stream carries no usage block, so the lane print raised TypeError before the JSON receipt was written. Absent measurements now print as "unavailable" and the receipt keeps None instead of a substituted 0. The module header claimed a non-streaming comparison and required-argument validation that the detector never performed; it now names the blank/malformed check it actually runs and states both absences. Drop the incidental padded-prompt and tool-schema tests, and add the missing-usage receipt regression.
…ion labels at a boundary
1e300 is a finite double but not an exact integer: sample() printed a 300-digit
count and the `[ -lt 100 ]` compare silently returned false (bash warned on
stderr, then took the non-SKIP branch), so an unmeasured curve was reported
PASS. 1e999 overflowed the double to +Inf and the position ratio came out
infinite, FAILing instead of refusing the sample. A bare `position="N"` also
matched inside `notposition="0"`, whose value was then judged as the position-0
series.
- bound the value in sample() to 2^53, the last integer awk's doubles hold
exactly (an overflowing exponent reaches +Inf, which fails the same bound), so
the bash compares below only ever see exact integers
- match position labels as `[{,]position="N"` in the unlabelled guard, the
position list and the per-position series selection, so a longer label name
ending in `position` is no longer a position
- pin both with inert fixtures: 1e300 drafts / 1e999 and 1e300 position values
-> rc 2 unreadable, `notposition="0"` alone or beside a genuine position 0
-> rc 2 unlabelled, `notposition="4"` beside a genuine label -> rc 0 with no
`pos4` line
fair v5 stays the TP=2/TP=4 default. TP=3 goes back to CHUNK=0 in start-tp3.sh and .env.tp3.example so a copied example overrides a fair .env. Opt in with GLM53_MIXED_PREFILL_CHUNK=fair. Co-authored-by: Cursor <cursoragent@cursor.com>
…rdev/pr70-cleanup-7c8b
fair v5 stays the TP=2 default only. TP=4 goes back to skip in start-tp4.sh and .env.tp4.example so a copied example overrides a fair .env. Opt in with GLM53_MIXED_PREFILL_CHUNK=fair. Co-authored-by: Cursor <cursoragent@cursor.com>
docs: GB10 UVM livelock runbook + spec-acceptance gate script
…rdev/pr41-cleanup-7c8b
…l-off fix: do not default fair mixed-prefill on TP3 or TP4
…rdev/pr41-cleanup-7c8b
…bench-and-spark-doctor feat(tests): add tool-call concurrency benchmark harness (Issue MiaAI-Lab#10) and spark_doctor diagnostics
…rdev/pr75-cleanup-7c8b
…rnel-lab Start model-agnostic EXL3 SM121 kernel lab
… canary unchanged PR180's validator work re-expressed against main's versioned installer instead of forking a second helper/gate definition. - overlay/patch_scheduler_decode_floor.py: byte-identical to the MiaAI-Lab#80 branch (installer v6). It validates an existing patch by unpatch/re-patch round-trip (a marker alone is never trusted), refuses drift/partial/duplicate/decorated patches without a write, is idempotent on re-apply, migrates v1-v5 images, and accepts the opt-in warm/deadline gate forms so a supported gate-v2 scheduler is not rejected at container start. - start.sh / .env.example / README.md: main's defaults kept (fair on TP=2, skip on TP=4, FP8 off). The TP=2 CHUNK=0 default flip is held as a gated candidate; it is not landed here. - tests/check_concurrent_agents.py: unchanged from the reviewed head (full completion URL, fail-closed assessment, socket cancel, no API key in receipts). - tests/test_prefill_concurrency.py: rewritten to the installed policy (the old copy exec'd the v1 HELPER literal and asserted the pre-fair launcher defaults). Patch-integrity/migration assertions moved to tests/test_scheduler_decode_floor.py, which now runs the v0-v5 matrix, re-apply idempotence, drift and marker-only refusal against the same installer. - docs/concurrent-agents.md and CHANGELOG: shipped defaults, the migration contract and the v6 acceptance stated explicitly.
…round-trip v1-v5 The v6 installer removed everything between `class _Glm53MixedPrefill:` and the next anchor without checking what it removed, and only a v6 image was round-trip verified: v1-v5 were unpatched and rewritten on the strength of their markers. The migration tests built their legacy fixtures as `class ... pass`, so a real legacy image was never exercised. - registry of the published helper texts (v1 f3043c9, cross-attested byte-identical by cad980f7263:HELPER_V1; v2 9a23b2d; v5 a9bbbad), keyed by sha256 and length, validated BEFORE anything is removed. v3/v4 have no published producer (180725a introduced their markers only), so their site is validated structurally: one undecorated class plus its binding and policy function, the trailer that version's own frozen sites call, and the members those sites need. v6 is compared against the installer's own text. - one exact byte range is removed and the whole file is round-trip checked: re-adding that span and re-applying the version's frozen sites must reproduce the input byte-for-byte. Missing, duplicated, decorated, drifted, marker-only and unpublished variants are refused with no write; the unversioned "0"-default build is refused because that baked default is a held direction. - the fabricated `class ... pass` helpers are replaced by the real published texts in tests/fixtures/legacy_scheduler_helpers.py, and every migration must land on exactly the bytes a fresh install of the pristine scheduler produces. Refusal and idempotence cases cover drift, duplication, decoration, a required-member gap and unrelated bytes after the helper. The v1 strip also left (or dropped) a byte relative to the pristine scheduler; the byte-identity assertion catches that now. v6 defaults are unchanged: the warm bypass and the deadline stay 0, and `fair` is untouched.
…s, bound-case test Same installer and migration matrix as the MiaAI-Lab#80 cleanup branch (byte-identical): a v3/v4 marker is refused without touching the source because no authenticated producer of those helper bodies was recovered, while v1/v2/v5 still migrate to v6 and re-verify by byte-identical round trip. Remove the shadowed duplicate _helper_text() and V4_PAIRS definitions, the unused HELPER alias, and the v3 pair table plus the structural member list that existed only for the removed path. Fair and gate defaults are unchanged. tests/test_prefill_concurrency.py: the capped bound case now proves the newcomer is offered min(cap, remaining) tokens on every step under a decoding peer, instead of only proving eventual completion. Docs: the migration wording and the CPU-coverage paragraph state the v3/v4 refusal.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description and Motivation
With
GLM53_MIXED_PREFILL_CHUNK=skip, an agent processing a new prompt can receive zero prefill tokens for the entire time another agent generates. This also affects requests already admitted to the running list. A client can exhaust its idle timeout despite available sequence slots and KV memory.Default the launcher, example configuration, and scheduler overlay to stock chunked prefill (
0) so new prompts can use the remaining step budget during generation. Preserve explicitskip/-1and positive caps. Reapplying the overlay also upgrades the known legacy default in existing images. The launcher propagates the same setting to both TP=2 ranks.This favors concurrent prompt progress and can reduce an individual stream's decode throughput during mixed work. The docs explain that tradeoff, migration of existing
.envfiles, the existing--long-prefill-token-thresholdoption for sharing a small batch budget, and the difference between a per-request context limit and total resident KV capacity.Add CPU regressions and a single live canary that requires a short request to answer before a long response finishes. The canary checks completed streams, the expected short answer, and overlapping content timestamps; serialized HTTP 200 responses do not pass.
Related Issues
Related to MiaAI-Lab#6. Upstream submission: MiaAI-Lab#180. The mixed-prefill throughput tradeoff remains available as explicit decode protection; this change addresses the starvation caused by making that policy the default.
Testing
Validated the exact published head,
14b6a9f6d4b6256ef4a51dade72581c4cf4a3175:bash -npasses separately forstart.sh,stop.sh, anddownload.sh; changed Python files compile;git diff --checkpasses.Live runtime validation used two DGX Sparks at clean upstream commit
f906ee990596486e10ddbe381efa6f0e496f77e3, built with the original Dockerfile into imagesha256:232573f48556c301b516ebc86a651501ed4f5c69aa72ef957168ff5a6ecd4fdf. The runtime applies this PR's concurrent-prefill policy through explicit configuration; the published PR source is covered by the regressions above.The recipe uses
GLM53_MIXED_PREFILL_CHUNK=0,LONG_PREFILL_TOKEN_THRESHOLD=512, a 1,024-token batch budget, a 1,000,000-token per-request limit, four sequence slots, GPU memory utilization 0.87, FP8 KV, DFlash k=7 with draft TP=2, and the upstream FAT/grouped/temp-row/rightsize performance settings. The engine reports 2,070,680 total KV-cache tokens. A separate LiteLLM stream-lifetime backport is installed, and GLM-route, proxy, and Hermes stale/read timeouts are 1,800 seconds.OKin 9.013 seconds, with first content at 8.837 seconds. B answered 13.014 seconds before A's final content.OKin 1.455 seconds while the large prompt was still processing.These are functional canaries and an operational endurance qualification of the whole deployed recipe, not a controlled throughput comparison or a guarantee of bounded latency under arbitrary load. Hermes normally compresses around 500K tokens; the separate capacity canary tests near-1M API requests.
Checklist:
Update README.md).