[Spec Decode] Reuse EAGLE3 CPU query offsets with Triton attention - #56147
venkywonka wants to merge 2 commits into
Conversation
Cache the invariant CPU offsets for each serial-draft batch size in MRV1. Keep private clones for other builders and M-RoPE, and clear cached storage when attention backends are initialized again. Co-authored-by: OpenAI Codex (GPT-6 Astra) <noreply@openai.com> Signed-off-by: venkywonka <23023424+venkywonka@users.noreply.github.com>
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment Once the PR is approved or has the If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
|
Validation update: a separate MiniMax runner-v1 GPU diagnostic completed one request at TP8/PP1, draft TP8/K5, capture width 6: HTTP 200, 1,239 prompt tokens, 136 completion tokens. Runner-v1/base-proposer setup was observed. This supersedes the earlier statement that no runner-v1 model request had run. Exact-source analysis supports getter reachability, but there is no direct observation of the changed getter, its cache eligibility, or a cache hit. This does not qualify the CPU-offset cache or establish a benefit. The completed runner-v2 full-workload runs remain outside this PR's path; the existing 31 CPU tests and 38 model-free native cases retain their original scope. Full standalone runner-v1 qualification and human review remain pending; keeping this draft. |
Signed-off-by: Venky <23023424+venkywonka@users.noreply.github.com>
|
Retained GPU validation update: a real Llama-3.1-8B/EAGLE3 run on one RTX PRO 6000 Blackwell GPU exercised runner-v1 offset caching with K3 and PIECEWISE graphs. Changing batches generated 1,408 tokens per round before and after backend reset. The getter recorded 331 calls / 330 cache hits before reset and 331 calls / 324 hits afterward, with no probe errors. Reset emptied the cache, generation repopulated it, and prompt/output token IDs matched across reset. Draft-proposal graph replay was observed; recorded postflight and assigned-GPU process checks passed. The current public production source is byte-identical to that September 15 run; only a test fixture differs. This is retained path-specific GPU evidence, not a fresh run of the amended whole head, baseline parity, full accuracy validation or a performance claim. I updated the description to distinguish it from the separate MiniMax diagnostic. Keeping the PR draft while the isolated-head lane is prepared. |
|
Standalone GPU model update — September 16, 2026 (EDT): cache/path/reset passed; baseline token parity failed. This remains a draft. The current public candidate and its baseline both completed real Llama-3.1-8B-Instruct/EAGLE3 K3 generation on one RTX PRO 6000 Blackwell GPU, runner v1, BF16 and PIECEWISE graphs. Seven batches (1, 2, 4, 8, 4, 2, 1) generated 64 tokens per request; candidate execution repeated after backend reinitialization. That is 44 candidate plus 22 baseline requests, 4,224 checked output tokens. The changed getter/cache was directly observed: 330 / 324 cache hits before/after reset, 993 / 993 metadata builds, and 1,974 / 1,974 draft graph replays. Reset cleared the cache; subsequent generation repopulated it. Candidate prompt/output tokens match exactly across reset. Source/native checks and assigned-GPU cleanup passed; native artifacts were reused, not rebuilt. The unchanged baseline comparator nevertheless failed: 9 of 22 paired requests differ, across 398 output-token positions. All prompt-token sequences match. Candidate instrumentation differs from the baseline, and identical prompts also differ at some batch positions within the baseline; this does not yet isolate a cache-PR regression. The mismatch is not waived. Matched-instrumentation comparison is the remaining causal gap. No full model-quality, performance, positive speculative-rejection-count or general non-regression claim is made. No additional GPU run was used to publish this result. AI assistance was used; human review and CI success are not claimed. |
|
Fresh matched, uninstrumented GPU candidate/parent comparison completed: 44 requests and 2,816 checked output tokens per arm on the same assigned GPU, with the batch sequence repeated twice. Both model processes exited 0 and the allocation was released. Exact output-token parity still fails: 14 of 44 paired requests differ at 670 token positions, across six of the 14 grouped workload records. Prompt tokens match. Each arm reproduces its own second repeat exactly, but the fresh parent also differs from the retained earlier parent on three of seven grouped records. The result therefore does not clear the parity failure or establish its cause. The 16 result-manifest files and raw output comparisons were independently checked. The earlier 9/22-request, 398-position failure remains preserved as a separate run, not overwritten or reclassified. These are correctness diagnostics, not qualified timing or full-model accuracy results. Keeping this PR draft while investigating the retained first divergences; no tolerance relaxation or additional benchmark replay is being claimed. |
|
Fresh bounded GPU diagnostic completed on the current public source: one SM120 GPU, EAGLE3 K3, greedy generation, and the unchanged batch sequence 1/2/4/8. All 15 checked requests and 960 generated tokens completed; the model step exited normally and owned GPU processes/allocation were cleaned up. The observer was limited to the existing batch-8 window and 128-sample cap. It directly recorded both required batch-8 continuation indices, with 37 samples each. All 128 recorded CPU/GPU query-offset pairs agree, every recorded continuation offset matches the expected sequence, and the probe reported no errors. This is real cache/metadata-path evidence, not proof outside the sampled states. The previously divergent witness now matches the retained baseline's complete 64-token response. However, the full 15-request prefix still differs from the retained matched-baseline prefix on 6 requests and 235 token positions. That is a historical-output comparison across different observer schedules, not a fresh matched A/B or proof of observer causation. The earlier 14/44-request, 670-position candidate/base parity failure remains unresolved. Raw outputs, source/native identity, retained references and artifact integrity were independently checked. Compatible native binaries were reused; no source patch, tolerance relaxation or extra baseline run was used. Keeping this PR draft while localizing the remaining divergence; no performance, full-accuracy or upstream-CI pass is claimed. AI assistance was used. |
|
Fresh matched-admission GPU attempt — September 17, 2026 UTC: The current public candidate passed its source/runtime preflight on a freshly allocated SM120 GPU, initialized the Llama/EAGLE3 K3 model with Triton attention, and entered request generation through the acknowledged pause/enqueue/resume sequence. The candidate then exited with The complete 44-request candidate matrix was not obtained; the baseline arm and final comparison did not run. This failure does not isolate a cache implementation defect and does not clear the earlier 14-of-44-request, 670-token-position parity mismatch. No new correctness or performance pass is claimed. The raw failure result, public-source/preflight identity and owned cleanup/accounting were independently checked. Cleanup exited successfully, the assigned GPU had no remaining workload processes and zero used device memory, and the dedicated allocation was released after 3 minutes 20 seconds. The next bounded step is to reproduce the actual request-ID contract mismatch in CPU tests before correcting the harness, without relaxing output identity or token-parity assertions. This PR remains draft. AI assistance was used; human approval and upstream CI success are not claimed. |
|
GPU validation update — September 17, 2026 UTC: The request-ID harness defect was reproduced in bounded CPU tests and corrected using the production mapping between internal enqueue IDs and external output IDs. The original identity/cardinality, ordering and exact-token assertions remain intact. The focused nine-test suite passed in both the owner and independent review; these tests use unchanged extracted production methods with CPU fixtures, not a full vLLM repository import or GPU qualification. A fresh same-GPU candidate/baseline pair then completed the entire original batch sequence 1/2/4/8/4/2/1 twice: 44 checked requests and 2,816 output tokens per source, or 88 checked requests and 5,632 tokens across both sources, excluding warmup. Both model processes exited normally. Initial admission layouts match, and each source reproduces its own second repeat exactly. Cross-source exact-token parity still fails: 12 of 44 paired requests differ at 428 output-token positions, across six grouped workload records. The retained earlier 14-of-44-request, 670-position failure is a separate run and remains preserved. The changed candidate cache getter was actually observed with 664 cache hits, 1,992 metadata builds and 3,964 draft graph replays; each arm retained 128 observer samples with no reported probe errors. Positive path execution does not clear the output mismatch or establish its cause. The public candidate source, raw request/token counts, comparison, postflight and assigned-GPU cleanup were independently checked. Compatible prebuilt native extensions were reused. The assigned GPU reported zero used memory at cleanup, and the dedicated allocation was released after 7 minutes 37 seconds. This PR remains draft. The next step is retained-evidence causal isolation and a focused regression test for any demonstrated defect, not another unchanged replay or a relaxed comparator. No performance, full-model accuracy, universal non-regression, human approval or upstream-CI pass is claimed. AI assistance was used. |
|
GPU validation update — September 17, 2026 UTC: A fresh SM120 candidate run completed the original batch sequence 1/2/4/8/4/2/1 twice: 44 checked requests and 2,816 generated tokens, excluding warmup. Both repeats were exact, and all candidate outputs matched the previous candidate run. The intended path recorded 664 cache hits, 1,992 metadata builds and 3,964 draft graph replays; all 128 sampled CPU/GPU query-offset observations agreed, with no recorded probe errors. Returned top-two/selected-token score data was retained for diagnosis. The paired diagnosis remains incomplete. After the successful candidate process, the coordinator could not immediately read its completion receipt and stopped before launching the baseline. A later read on the same frontend found the successful receipt; all retained result hashes verified. This is a validation-orchestration receipt-visibility failure, not an observed model failure or a new candidate/base comparator result. The previous 12-of-44-request, 428-token-position cross-source mismatch remains unresolved. The public candidate and native runtime identities passed preflight/postflight; compatible prebuilt native extensions were reused. Owned cleanup passed, the assigned GPU had no remaining workload processes or used device memory, and the allocation was released. The next fix is a bounded completion-receipt read with CPU regression tests for delayed visibility, without relaxing source, output-parity or cleanup checks. This PR remains draft. No paired-correctness, performance, full-model accuracy or upstream-CI pass is claimed. AI assistance was used. |
|
Validation update: the latest fresh, same-GPU EAGLE3 + Triton-attention candidate/baseline pair completed the original ascending/descending batch matrix with repeat controls: 44 requests and 2,816 generated tokens per source. Both execution arms and within-arm repeat checks passed; the intended cache path was exercised. Owned cleanup and allocation release were verified. Cross-source output parity still fails. The original admission comparator failed; the retained token comparison reports differences in 20 of 44 requests, across 600 token positions. The additional returned top-two target-score diagnostic is INCONCLUSIVE and does not override that failure. Returned-score ties alone do not establish a harmless difference or identify the cause. Earlier failed comparisons remain retained. The completion-receipt controller issue was corrected with CPU regression RED/GREEN coverage, allowing both arms and the comparisons to finish. No production change or validator/tolerance relaxation was made for this attempt. This PR remains draft pending resolution of the output difference. These results are targeted model-path correctness evidence, not full quality-suite or performance qualification. |
|
GPU validation update — September 17, 2026 UTC: A fresh one-SM120 EAGLE3 + Triton-attention run of the current public candidate completed a same-process cache intervention: cached A1, forced-cache-miss B, then cached A2 after restoring the original cache dictionary. Each pass exercised batch sizes 1/2/4/8/4/2/1, totaling 66 checked requests and 4,224 generated tokens, with 15 separate warmup requests. All three pairwise comparisons passed exact output-token parity, with zero differing requests or token positions. The real observer recorded 996 offset-getter calls, 664 cache hits, 2,988 metadata builds and 5,946 draft CUDA-graph replays, with no observer errors. The forced-miss pass and restoration checks passed; the public getter itself was unchanged. This is intended-path GPU execution evidence, not a mocked numerical test. The private intervention controller also has independently run focused CPU coverage and a genuine predecessor-hook negative control, not a claimed production regression fix. The model step, strict comparison and owned cleanup exited successfully; the assigned GPU reported zero remaining memory usage before release. The dedicated allocation was released after 4 minutes 36 seconds. This does not clear the earlier candidate-versus-parent failure of 20/44 requests and 600 token positions: no parent-source arm ran in this intervention. The bounded conclusion is that forcing cache misses had no observed output effect in this same-process matrix. The PR remains draft while the retained cross-source discrepancy is investigated; no broad quality, performance or upstream-CI pass is claimed. No production change or tolerance relaxation was made. AI assistance was used. |
|
GPU validation update — September 17, 2026 UTC: A fresh one-SM120 Llama-3.1-8B-Instruct/EAGLE3 run compared the current public candidate with its true parent, using BF16, K3, PIECEWISE graphs and Triton attention. Each source completed the original batch matrix 1/2/4/8/4/2/1 twice, with 64 generated tokens per request: 44 checked requests and 2,816 tokens per source; 88 requests and 5,632 tokens combined, plus separate warmups. The unchanged strict candidate/parent comparison passed exact prompt- and output-token parity, with zero differing requests or token positions. Both within-source repeats also passed. Candidate instrumentation recorded 664 offset-getter calls, 664 cache hits, 1,992 metadata builds and 3,964 draft CUDA-graph replays, with no original probe errors. The parent recorded the same metadata-build/replay counts and no cache-getter calls, as expected. Both model steps, the original comparison, source/native checks and owned cleanup succeeded; the assigned GPU reported zero remaining memory usage before release. A separate optional target-row diagnostic was inconclusive, not successful: its observer rejected an unexpected attention-metadata layout in both sources and retained no complete target transactions. That diagnostic failure does not change the original comparison result, but it does not explain or erase the earlier 20/44-request, 600-position mismatch. The previously reported same-process cached/forced-miss/restored-cache parity and separate backend-reset results retain their original scope. This is real intended-path, paired GPU correctness evidence, not full model-quality evaluation, performance validation or universal non-regression. No production change or tolerance relaxation was made. Readiness is being reconciled against the existing acceptance coverage; no upstream-CI pass is claimed. AI assistance was used. |
|
GPU validation follow-up — September 17, 2026 UTC: The current public candidate and true parent completed the revised, same-GPU SM120 Llama-3.1-8B-Instruct/EAGLE3 diagnostic: BF16/K3, Triton attention, PIECEWISE graphs, original batch matrix 1/2/4/8/4/2/1 twice. 44 checked requests and 2,816 generated tokens per source; 88 requests and 5,632 tokens combined, plus separate warmups. Strict output-token parity still FAILED: 16 of 44 paired requests differ across 400 positions. Prompt-token equality and both within-source repeats passed. The preceding diagnostic's 24/44 requests and 952 differing positions, the earlier passing pair, and other failures remain separate retained outcomes—not an improvement claim or a waiver. The corrected observer captured two complete target transactions per source at an actually reached compiled-attention boundary. The selected first-layer attention query/output rows match byte-for-byte in both transactions, but final selected hidden states/logits differ. The selected request's full output is equal in this pair: these captures do not include the actual first emitted mismatch, and the historical token-level witness was not reproduced. This is bounded tensor evidence, not full KV/weight/other-row equality, an identified production defect, or cache exoneration. The candidate recorded 662 getter calls, 661 cache hits and 3,952 draft graph replays, with no original probe errors. Source/native/result checks passed; native artifacts were reused, not rebuilt. Both model steps exited zero; owned cleanup found no assigned-GPU processes and zero GPU memory, and the allocation was released. The paired allocation lasted under eight minutes. Remains draft. Production code and acceptance tolerances are unchanged; no full quality/performance, universal non-regression, or upstream-CI success is claimed. AI assistance was used. |
|
Validation update, September 17:
This is not a new production RED/GREEN or GPU-parity pass. The retained cross-source token mismatch remains unresolved; the PR stays draft. |
|
Retained GPU negative-control follow-up: two already-completed uncached true-parent runs differ on 24/44 requests and 952 output-token positions; both within-run repeats are exact. Production source, recorded model/native identities, GPU, prompts, sampling and declared compilation settings match. The previously investigated batch-8 token split also occurs between these unchanged-parent runs, so it is not unique to the cached candidate. Observer helpers, independent process/compilation state and physical KV block/slot mappings differ. This does not identify a production cause or exonerate the cache change. The latest candidate/parent 16/44-request, 400-position FAIL remains retained, and the PR remains draft. This was offline analysis of existing GPU results: no additional GPU execution, source changes, tolerance relaxation or performance claim. AI assistance was used. |
|
New bounded real-weight EAGLE3/Triton-attention diagnostic on one RTX PRO 6000 Blackwell GPU: 66 checked requests / 4,224 generated tokens across cached → exact-parent-expression → restored-cache phases. Prompt and generated token IDs match exactly across all three phases for batch sizes 1, 2, 4, 8, 4, 2, 1. The cache observer records 993 offset calls / 661 hits, with the original cache/getter restored afterward and no probe errors. This is a same-process intervention on the candidate, not execution of the full parent source. The earlier divergent witness was not reproduced, so the diagnostic correctly remains inconclusive and its comparator exits nonzero. It does not clear the earlier candidate/parent parity failure or establish general numerical equivalence, accuracy or performance. The model step completed successfully, retained result-file checksums were verified, assigned-GPU cleanup passed and the allocation was released. No production change, tolerance relaxation or automatic benchmark retry was made. The PR remains draft while the earlier parity discrepancy is unresolved. |
|
Keeping this PR as a separate draft while the other changes are consolidated The latest bounded paired SM120 test completed 44 requests and 2,816 output Both model steps completed and owned resources were released. This is actual |
1 participant
Retained true-parent control — September 17, 2026
Offline comparison of two already-completed uncached true-parent GPU runs found 24/44 requests and 952 output-token positions differ, despite identical production source, recorded model/native identities, GPU, prompts, sampling and declared compilation settings. Each run repeats its own output exactly. The previously investigated batch-8 token split also occurs between these unchanged-parent runs, so it is not unique to the cached candidate.
Observer helpers, separate process/compilation state and physical KV block/slot mappings differ; their causal contribution is unresolved. This is a useful negative control, not cache exoneration or a production root cause. The latest candidate/parent 16/44-request, 400-position FAIL remains unchanged, and the PR stays draft. No new GPU run, production change, tolerance relaxation or performance claim was made. AI assistance was used.
Latest paired GPU result — September 17, 2026, 13:07 UTC
Draft: strict candidate/parent output-token parity still FAILED. The current public candidate and true parent completed a revised same-SM120-GPU Llama-3.1-8B-Instruct/EAGLE3 diagnostic, BF16/K3 with Triton attention and PIECEWISE graphs. The unchanged batch matrix 1/2/4/8/4/2/1 ran twice: 44 checked requests and 2,816 generated tokens per source; 88 requests and 5,632 tokens combined. 16 of 44 paired requests differ across 400 output-token positions. Prompt equality and both within-source repeats passed; successful model exits do not waive the strict failure.
Two complete target transactions per source now include selected first-layer attention query/output rows at the actual compiled-attention host boundary. These selected rows match byte-for-byte in both transactions, while final selected hidden states/logits differ. The selected request's full output is equal in this pair, so these captures do not include the actual first emitted mismatch; the historical token-level witness was not reproduced. This is bounded tensor evidence, not full KV/weight/other-row equality, an identified production defect, or cache exoneration. Candidate execution recorded 662 getter calls, 661 hits and 3,952 draft graph replays, with no original probe errors.
Source/native/result checks passed; native artifacts were reused, not rebuilt. Both model steps completed successfully; owned cleanup found no assigned-GPU processes and zero GPU memory before allocation release. The paired allocation lasted under eight minutes. Production source and numerical acceptance are unchanged. The earlier 24/44-request, 952-position failure and earlier passing/failing pairs remain below as independent history, not an improvement claim. No full quality/performance, universal non-regression, review-readiness or upstream-CI success is asserted. AI assistance was used.
Retained paired diagnostic — September 17, 2026, 10:49 UTC
This pair's strict candidate/parent output-token comparison FAILED. Both current-public-candidate and true-parent model runs completed on the same SM120 GPU, using Llama-3.1-8B-Instruct/EAGLE3, BF16/K3, Triton attention and PIECEWISE graphs. Each ran the original 1/2/4/8/4/2/1 batch matrix twice: 44 checked requests and 2,816 generated tokens per source, 88 requests and 5,632 tokens combined. 24 of 44 paired requests differed across 952 output-token positions. Prompt-token comparison and both within-source repeats passed; successful model exits do not waive the failed comparison.
The corrected private observer retained two complete target transactions per source, including the actual 32-target-layer plus one-draft-layer metadata layout. The candidate recorded 662 getter calls, 661 cache hits and 3,952 draft CUDA-graph replays. Selected hidden state and logits differ before sampling for the captured differing target step, while recorded forward-input, dispatch and sampling controls agree. This is useful localization, not a proven production root cause: KV contents, per-layer intermediate tensors and in-memory weight equality remain unestablished.
Source/native/result checks and owned cleanup passed, with no assigned-GPU processes and zero reported GPU memory before release. The earlier strict pair PASS and earlier failures remain retained below; no pair substitutes for another. No production fix, relaxed tolerance, full accuracy/performance, universal non-regression, review-readiness or upstream-CI success is claimed. See the September 17 GPU-validation follow-up comments. AI assistance was used.
Retained validation history
Retained earlier passing pair — September 17, 2026 UTC
The current public candidate and its true parent each completed real one-SM120 Llama-3.1-8B-Instruct/EAGLE3 generation, BF16/K3 with PIECEWISE graphs and Triton attention. Each arm ran the original batch matrix 1/2/4/8/4/2/1 twice: 44 checked requests and 2,816 generated tokens per source; 88 requests and 5,632 tokens combined. The unchanged strict comparison passed exact prompt/output-token parity, with zero differing requests or positions; both within-source repeats passed.
The changed path was directly observed: 664 getter calls and 664 cache hits, 1,992 metadata builds and 3,964 draft CUDA-graph replays, with no original probe errors. The true parent had the same build/replay counts and no cache getter. Model steps, source/native identity checks and owned cleanup passed. Separate retained cache-reset and cached/forced-miss/restored-cache tests keep their original scope; this pair did not rerun backend reset.
The optional target-row diagnostic was INCONCLUSIVE because both observers rejected an unexpected attention-metadata layout. It is not a new qualification requirement and is not evidence of a production defect. The fresh strict pair PASS also does not explain or erase the earlier cross-source failures, including 20/44 requests and 600 positions. The PR remains draft while that contradictory evidence is reconciled. No full model-quality, performance, universal non-regression, human-approval or upstream-CI claim is made. AI assistance was used. The detailed update is in the September 17 GPU-validation comment; earlier results remain below as history, not the latest result.
Retained validation history — September 16–17, 2026
New standalone result (EDT): path/reset PASS; baseline parity FAIL
The current public candidate and its baseline both completed real Llama-3.1-8B-Instruct/EAGLE3 K3 generation on one RTX PRO 6000 Blackwell GPU, runner v1, BF16 and PIECEWISE graphs. Seven batches (1, 2, 4, 8, 4, 2, 1) generated 64 tokens per request; candidate execution repeated after backend reinitialization: 44 candidate plus 22 baseline requests, 4,224 checked output tokens.
The retained validation helper's instrumented candidate/reset mode and baseline run mode both exited zero. Its unchanged comparison mode exited one: 9 of 22 paired requests differ, across 398 output-token positions, despite identical prompt-token sequences. Candidate instrumentation differs from baseline, and identical prompts also differ at some batch positions within baseline. These observations do not isolate a cache-PR regression; the failed comparison is not waived. Matched-instrumentation comparison remains the causal gap, and this PR remains a draft.
The changed getter/cache was directly observed: 330 / 324 cache hits, 993 / 993 metadata builds, and 1,974 / 1,974 draft graph replays before/after reset. Reset cleared the cache and subsequent generation repopulated it; candidate prompt/output tokens match exactly across reset. Source/native checks and assigned-GPU cleanup passed. Compatible native artifacts were reused, not rebuilt. This establishes standalone path/reset execution, not full model quality, performance, a positive speculative-rejection count or universal non-regression. The historical evidence below retains its original scope.
Draft; not qualified by the latest combined MiniMax workload runs. The completed TP4/PP2 and TP8/PP1 MiniMax serving logs both select model runner v2. Their source routes through
EagleSpeculator/AutoRegressiveSpeculator, not the runner-v1EagleProposer/SpecDecodeBaseProposerchanged here. Consequently, neither those full accuracy/serving results nor other PRs' observed kernel launches demonstrate use of this CPU-offset cache.The existing 31 CPU tests and 38 model-free native cases below retain their original scope. A subsequent, separate MiniMax runner-v1 diagnostic completed one real GPU request at TP8/PP1, draft TP8/K5, capture width 6: HTTP 200, 1,239 prompt tokens and 136 completion tokens. Runner-v1/base-proposer setup was observed. Exact-source analysis supports getter reachability, but the trace does not record the changed getter's invocation, cache eligibility, or a cache hit. This is not a full accuracy/serving run, a cache-benefit measurement, or qualification of this PR. Human review and standalone runner-v1 qualification remain pending; neither minimum-stack necessity nor dispensability is established.
Retained real GPU cache/reset evidence — September 16, 2026
A retained September 15 run provides direct, path-specific evidence beyond the separate MiniMax diagnostic: real Llama-3.1-8B-Instruct with its EAGLE3 draft on one RTX PRO 6000 Blackwell GPU, runner v1, BF16, K3 and PIECEWISE graphs. Seven batches (1, 2, 4, 8, 4, 2, 1) generated 64 tokens per sequence, totaling 1,408 tokens per round, before and after backend reinitialization.
The changed offset getter was directly observed: 331 calls / 330 cache hits before reset and 331 calls / 324 hits afterward, with no probe errors. Reset returned an empty cache and subsequent real generation repopulated it. Prompt and output token IDs matched exactly across reset; graph replay calls were observed inside draft proposal. Source/native checks and the recorded postflight passed, and the assigned-GPU process observation was empty afterward.
All non-test source files in the current public snapshot are byte-identical to that retained model run; only the metadata-fixture test file differs. This is historical runtime evidence plus source equivalence, not a new execution of the amended whole head, baseline-versus-candidate token parity, full model-quality evaluation, MiniMax performance recovery, or universal non-regression. The run is instrumented correctness evidence, not a timing result. The PR remains a draft; fresh isolated-head validation is being prepared and no human review is claimed.
Purpose
After the first EAGLE3 draft pass in model runner v1, each request contributes one query token. The proposer currently allocates a fresh CPU
[0, ..., batch_size]tensor on every proposal. This change reuses an owned tensor for each batch size with the exact Triton attention builder. Other builders and multidimensional positions retain private clones, and backend initialization clears the cache.The metadata-builder API and dynamic attention metadata rebuilding are unchanged. This applies to model runner v1 (
VLLM_USE_V2_MODEL_RUNNER=0or an existing fallback); it does not change the default runner-v2 path. The change was rebased onto main8c87c333b84c85908b1d11f0044457692277c6f3.Non-duplication: #46849 and #54790 optimize runner-v2 draft chains, #53301 reuses metadata across cache groups, #55292 refactors builder capabilities, and #55978 removes other runner-v2 loop overhead. Their actual diffs and focused CPU-offset searches were checked; none reuses this runner-v1 CPU offset tensor.
Validation:
VLLM_TARGET_DEVICE=cpu .venv/bin/python -m pytest tests/v1/spec_decode/test_llm_base_proposer.py tests/v1/worker/test_gpu_autoregressive_speculator.py tests/v1/spec_decode/test_eagle_draft_attn_metadata.py -q: 31 passed on the rebased head..venv/bin/pre-commit run --files vllm/v1/spec_decode/llm_base_proposer.py tests/v1/spec_decode/test_llm_base_proposer.py: passed..venv/bin/pre-commit run mypy-3.12 --hook-stage manual --files vllm/v1/spec_decode/llm_base_proposer.py tests/v1/spec_decode/test_llm_base_proposer.py: passed.The CPU tests exercise
propose()through real Triton metadata construction, covering nonuniform first-pass queries, changing batch sizes and draft lengths, fresh dynamic metadata, fallback storage, three/four-channel M-RoPE, and backend reinitialization. Model forward and GPU position updates are substituted for these CPU tests. The earlier workstation CPU run used Python 3.12.13 and Torch 2.13.0+cpu; native extensions were unbuilt in that environment. The tests report the existing missing generated-version and Torch deprecation warnings.Validation limits: the retained bounded runner-v1 Llama/EAGLE3 K3 graph/reset run is described above. Full model evaluation including real-sampler GSM8K, matched serving A/B measurements, and execution of the amended whole head are not established by it. Actual Austin native initialization and model-free correctness results are recorded below. No serving throughput, model-quality, or per-patch Pareto improvement is claimed. This PR remains a draft for human review and completion of GPU/model validation.
AI assistance: GPT-6 Astra xhigh assisted implementation, source comparison, testing and independent review. Human review of every changed line and relevant human-run tests remain pending.
Austin native validation, September 9–10, 2026:
e8ff0a4099bd5e9e2a8c7c21b1ff2a25d33f4a3fand baseline8c87c333b84c85908b1d11f0044457692277c6f3on an actual ComputeLab Austin RTX PRO 6000 Blackwell Server Edition, compute capability 12.0. Runtime: Python 3.12.13, Torch 2.13.0+cu130, CUDA 13.0, driver 595.58.03.e81a71acdfa363029263dc13fc6f42c9834534cd9b4c82ad3a70c68411264caf). Native/build inputs are unchanged by this PR. Both installed environments pass dependency checks; all 199 non-vLLM package versions match. Actual imported source commits and loaded native hashes were verified.torch.ops._C.silu_and_mulcheck against an FP32 reference cast to BF16, with matching output and loaded-native hashes.This establishes model-free native correctness. The captured graph above contains the EAGLE update kernel; it is not a full draft-model graph. No model weights, GSM8K evaluation, HTTP serving benchmark, or performance measurement ran in this phase. Full model and performance checks remain pending; the PR remains a draft.
CI status: the upstream
pre-run-checkjobs 34403567716 and 34403623764 failed the repository's contributor-eligibility gate (label/merged-PR threshold), so their pre-commit jobs did not run. The logs do not identify a code/test failure. Local pre-commit, the focused CPU tests, and the Austin native results above are separate evidence. No eligibility labels or workflow rules were changed.Validation-harness review, September 10 (off-node; source head unchanged): private controls assert observed draft K, eager mode with zero draft replay, and the assigned worker/source/arm/job/step/UUID. Live-origin collection uses a shared, reviewed interface with immutable startup/run receipts; continuation requires complete collection and closed descriptors. Full GSM8K authorization is checked separately, and errors/interruption retain failed status before safe cleanup.
.venv/bin/python -m pytest slop/eagle_metadata_cache/austin/test_phase_gates.py slop/eagle_metadata_cache/austin/test_origin_hook.py -q: 32 passed, zero failures/skips (19 control/lifecycle cases and 13 origin-hook cases). Expected pre-fix failures are preserved. These are private harness checks, not additional Austin model or serving passes. Matched Llama-3.1-8B/EAGLE3 base/head model checks, full1319 GSM8K, live library-origin collection, and host/HTTP A/B remain pending. MiniMax baseline results are not reused for this fixture.Recovery reconciliation, September 11: the local, fork, and PR heads remain
e8ff0a4099bd5e9e2a8c7c21b1ff2a25d33f4a3f. The historical 38-case native suite ran on that candidate; baseline8c87c333b84c85908b1d11f0044457692277c6f3has a separate passing native preflight, not another 38-case suite. The shared allocation has terminated with failure and all prior execution/continuation grants are retired. This recovery performed off-node receipt and denial-guard checks only; it did not rerun GPU tests or start a model.This recovery snapshot predates the retained runner-v1 Llama/EAGLE3 K3/reset run documented above. That later run observes cache hits, graph replay and across-reset token parity only; it does not establish the other variants, baseline/no-spec parity, full1319 GSM8K comparisons, full process-mapped library closure or host/HTTP A/B. Scheduler or reboot records do not establish direct process/descriptor closure. No shared MiniMax baseline or compound C12/C14 result is used as this PR's validation or individual speedup. There are no source changes or new commits from recovery; the PR remains a draft pending fresh resource authorization and the remaining tests.
Fixture compatibility update
The existing draft-metadata fixture now supplies the builder's optional speculative configuration and two-request scheduler capacity. Only these two fixture fields change; production offset caching and the existing test assertions remain unchanged. This belongs in the existing PR rather than a duplicate test-fix PR.
Changed-file
pre-commit run --files, syntax compilation, andgit diff --checkpassed using existing tools without dependency installation. Normal commit hooks also passed. The earlier Mac imported baseline/candidate attempts stopped during collection because NumPy was missing: zero behavior tests executed in those attempts.September 17 update: all eight public proposer regression cases now pass through normal Torch/NumPy/pytest imports against the current public source and unchanged test fixtures: eight executed, zero failures, errors or skips. Coverage includes kernel-block sizing, deterministic draft-layer ordering, CPU-offset reuse through real Triton metadata construction while batch size and draft length change, fallback storage, three/four-channel M-RoPE and backend reinitialization. Existing test substitutions for model forward and GPU position updates remain unchanged.
This used a CPU-only allocation of two CPUs and 8 GiB for 66 seconds, with no GPUs allocated. CUDA availability and initialization remained false before and after pytest; imported source/test identities and all 17 unchanged native-carrier hashes were checked, and tracked source remained unchanged afterward. No dependency installation, native rebuild, model load, GPU kernel execution or serving benchmark was performed in this phase. This is a current-candidate imported test pass, not a new pre-fix RED/GREEN result, and it does not resolve the separate retained GPU cross-source token-parity failure. The PR remains a draft.
AI assistance was used for this update, reusing the existing independent static fixture review. Human line-by-line review and relevant human-run tests are not confirmed by this update; the PR remains a draft.
Assembly alternative: omit this PR
The offset-off alternative means omitting this PR's V1 CPU-offset cache from an assembly and retaining the baseline per-call clone. It is not an additive mainline fix, proven sufficient stack, historical C1 performance recovery, or no-regression result. The active PR retains its offset cache unchanged and remains open; no removal patch is added here. Combined workload results do not isolate this PR's benefit or establish that omitting it reproduces either historical leader.