Stage 2 + Branch B: RL output contracts, prefill cost geometry (report), spec-decode geometry harness - #12
Merged
yinli-systems merged 45 commits intoSep 18, 2026
Conversation
added 8 commits
September 17, 2026 19:52
…patch vLLM's --return-sampling-mask (distribution replay for RL rollouts) packs the surviving support as a full-vocabulary bitmap every step, copies ceil(V/8) bytes per row to the host, and unpacks V bits per row with np.unpackbits before np.nonzero. The feature already requires top_k > 0, so the support is bounded by top_k; the bitmap path pays for the vocabulary regardless. - src/l20_stack/ops/triton_support_pack.py: one pass over the processed logits emits ascending support token IDs [B, K] with counts and an explicit overflow flag (never silent truncation), and the same pass computes the processed log-normalizer, the sampled token's processed logprob and, optionally, every support token's logprob. Same finite-logit keep rule as upstream. Empty-tile-safe online log-sum-exp; empty rows yield NaN like torch.log_softmax and do not disturb neighbours. - tests/test_support_pack_gpu.py: parity with a dense reference after top-k filtering at vocab 32000/151936 and K 64/256, overflow flagging, inactive rows, +inf exclusion and ragged last tile, and tensor-contract rejection. - integrations/vllm/vllm-v0.29.0-compact-sampling-mask.patch: upstream-shaped git diff adding a compact layout to SamplingMaskTensors, sized from the batch's largest top_k read from host-side SamplingStates (no device sync), with VLLM_SAMPLING_MASK_COMPACT / _MAX_K to force or cap it; tolists becomes a ragged slice and raises on overflow. install_compact_sampling_mask .py applies/reverts it on an installed 0.29.0. - scripts/benchmark_support_pack.py: whole-path microbenchmark (GPU pack, D2H, host CSR build) of the verbatim upstream bitmap kernel vs the compact kernel with CSR equality asserted every trial. - scripts/measure_vllm_feature_cost.py: serving A/B harness that starts one vllm serve per server condition (native, mask, mask_bitmap, mask_compact) and replays a fixed-length rollout workload on the token-in/token-out endpoint (the only route that returns sampling_mask) or the streaming completions route, interleaving request conditions across rounds. - docs/stage1-feature-cost-notes.md: where the compatibility penalty sits in the v0.29.0 code path.
…ce comparer With per-request seeds, the bit-packed and compact sampling-mask layouts must return identical token sequences and masks; the comparer turns that into a checkable end-to-end conformance result rather than a statistical one.
…s package files only The patch now carries the tests/v1/test_outputs.py additions (compact tolist, compact-vs-bitpacked CSR equality after top-k/top-p, overflow raising) so it can be submitted upstream unchanged. Installed wheels have no tests tree, so the installer filters the diff to vllm/ sections; the patch test checks that filter.
…zer for the feature-cost harness
…k and serving A/B Two clean-commit artifacts on the L20 (vLLM 0.29.0, FlashInfer 0.6.18): - benchmarks/results/l20-support-pack-path/: whole output path (GPU pack, D2H, host CSR build) of the verbatim upstream bitmap kernel vs compact packing, identical CSR on every shape; 39x (B=8) to 457x (B=256) per step, almost entirely host time (np.unpackbits over the vocabulary: 145 ms/step at B=256, V=151936). - benchmarks/results/l20-vllm-sampling-mask-ab/: serving A/B with one vllm serve per condition, 128 x 256-token rollouts at concurrency 64 on the token-in/token-out endpoint. --return-sampling-mask caps both Qwen2.5-0.5B and Qwen3-4B at ~1.6K tok/s on this host; the compact layout restores 0.77x (0.5B) and 0.97x (4B) of native, streaming ITL 39.6 ms -> 3.6 ms. Under VLLM_BATCH_INVARIANT=1 bitmap and compact return identical token sequences for 128/128 seeded requests; mask agreement (123/128) equals the bitmap server's own run-to-run agreement, and the differing entries are top-p tail tokens at prefill or wave boundaries. Disabling the FlashInfer sampler and processed-logprobs mode measured no cost at this scale; logprobs=1 costs 0.72x (0.5B) / 0.94x (4B) on the API server, not in the engine. Result index, artifact catalog, experiment status and README headline point at both artifacts; tests/test_benchmark_protocol.py checks raw-file hashes, clean provenance, the headline ratios and the equivalence contract.
…LM #54901 Upstream vllm-project/vllm#54901 (merged 2026-09-04, shipped in 0.29.1) diagnosed the same host-side np.unpackbits bottleneck and landed the same top_k-bounded compact layout before this work was done against v0.29.0. README, artifact READMEs, notes, status ledger and installer now state that; the patch is for 0.29.0 only. The residual-mask wording no longer asserts a cause for the five differing masks; it reports the bitmap-vs-bitmap control and states that the experiment does not attribute them to the compact layout.
yinli-systems
changed the title
added 21 commits
September 17, 2026 20:59
…x (4B) / 0.86x (0.5B) of native
…ruction, not detokenization or the kernel Skip-tokenizer A/B (no change), per-process CPU split (+1.3 s API / +0.35 s engine core per 32,768 tokens), and a three-way path microbenchmark that quantifies upstream #54901's residual bitmap copy (0.66 ms vs 0.32 ms per step at batch 256). Recorded as a diagnostic artifact; no fix implemented.
…serving effect on L20 Upper-bound variant that never produces or copies the bitmap equals the upstream compact path within round-to-round noise on Qwen2.5-0.5B and Qwen3-4B at concurrency 64 and 256. The residual GPU/D2H work is hidden by the async output pipeline; a conditional-fallback design is not pursued.
…_logprobs server conditions
…mask+logprob contract 0.94x -> 0.98x (4B), exact Upper-bound experiment on the API-side logprob cost that turned out to be the feature: when a request sets logprobs=0 and flat_logprobs=True, emit choices[].token_logprobs straight from FlatLogprobs instead of building per-token OpenAI-style objects. Values identical to the object path under batch-invariant mode (max abs diff 0.0, 16/16 sequences). Env-gated patch against vLLM main recorded with the runs; needs a request-level field before an upstream proposal.
…tream-shaped patch
…, stdev in summaries
…kip engine kernels)
…x native/mask, with API/engine CPU split and bytes per request
…=5 points upper bound), detokenisation is API CPU but not the limiter, engine-core transport is what remains
…cted long prefills per token budget
…ining logprob cost (upper bound +15/+8 points on 0.5B at c256/c64); D2H and tolists are free
… to 8192 holds decode ITL under a 32k prefill; step cost nonlinear in chunk size
…-0.96x of native, 4B at 0.99x, exact; patch and runs recorded
…es across shifted cells
…ep cost by KV-depth bucket
…r context/batch/budget shift; point predictor 56% false-safe at 32k, KV-read + online refit removes bias, online quantile removes tail
…V-distribution control)
…ent step costs; per-request attention-work term explains it; live geometry controller beats aggregate controller 1.5-2.8x on TTFT at zero violations, best fixed budget by only 7-8%
…d, still <15%) and default-FCFS burst control (geometry irrelevant there); scope narrowed to mixed-depth multi-request steps
…g, live and FCFS figures
…ill (frozen artifact 17478ac)
…sult / boundary + four figures)
…h size with draft/verify traces
…e; long-batch cap
…composition); join carries draft_ms
yinli-systems
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges the Stage 1 → Stage 2 → Branch B line of work into
main(47 commits, every artifact produced on a clean tree with aprovenanceblock).What lands
Stage 1 (PR #11, contained here) — all-masked-tile NaN fix in top-logprobs, hardened contracts, L20 remeasurement.
Stage 2 — output contracts for RL rollouts (L20)
benchmarks/results/l20-vllm-sampling-mask-ab/,l20-mask-bitmap-removal-upper-bound/).token_logprobsfast path (l20-logprobs-feature-cost/,l20-logprob-engine-decomposition/,l20-flat-token-logprobs/) — evidence behind upstream [Frontend][Core] Sampled-token logprob fast path for /inference/v1/generate (return_token_logprobs) vllm-project/vllm#57442 (0.95–0.96× of native at c64–256 on 0.5B, 0.99× on 4B).Branch B — prefill cost geometry (frozen at 17478ac)
benchmarks/results/l20-prefill-interference/,l20-step-cost-shift/(Phase 0, superseded claim recorded), and the final artifactl20-prefill-cost-geometry/: measurement contract, same-aggregate/different-cost result (up to 1.9×), M0/M1/M2 predictors under explicit OOD splits, offline replay, live env-gated controller A/B (campaigns 17–22), negative results and scope.docs/when-token-budgets-lie.md.scripts/measure_prefill_interference.py,step_trace_join.py,analyze_measurement_contract.py,analyze_step_cost_v2.py,replay_prefill_controller.py,analyze_live_controller.py,plot_prefill_geometry.py; experiment patches under the artifact'spatches/.Third line — speculative-decoding cost geometry (in progress, harness only)
scripts/measure_spec_geometry.py,scripts/analyze_spec_geometry.py(campaigns 23–24 running; results will come in a separate PR).Index/catalog/status (
benchmarks/results/README.md,artifact-catalog.json,docs/experiment-status.md) updated for every artifact. CPU-safe checks green on every commit;pytest tests384 passed / 20 skipped locally.