[nxphi47]

RL changes the target policy while a fixed speculative draft continues proposing from its original distribution. PrimeRL also needs to update both models together: reloading only the policy leaves the draft stale, and an unsafe draft reload can overwrite shared target weights or leave CUDA graphs pointing at obsolete tensors.

This PR adds an optional path to train a draft from the live policy's detached features, synchronize both models into inference, and save/resume/export them together. It also exposes upstream standalone draft training through PrimeRL's TOML CLI. Validation covers Qwen3-4B and a separately trained DSpark for Qwen3-30B-A3B-Thinking-2507; the model-specific evidence and gaps are listed below.

Accuracy update: two completed default-vLLM0.30 Avg@16 evaluations now give pooled Avg@32: 609/960 (63.44%) ordinary decoding versus 603/960 (62.81%) DSpark, a difference of −0.625 percentage points, paired-question bootstrap 95% interval [−3.44, +2.08]. The first set showed a −3.96-point drop; the independent-seed replication reversed direction to +2.71 points. The initial drop did not replicate, and the combined result is consistent with sampling variation. This does not prove equivalence: the interval still permits a 3.44-point drop and no noninferiority margin was specified. Exact target-weight checks, scored greedy tests and complete-serving distribution checks found no verification bypass. Optional batch-invariant execution eliminates the tested greedy numerical divergence; its separate completed Avg@16 scores 295/480 ordinary versus 318/480 DSpark (+4.79 points, interval [+1.88, +7.92]). This exploratory diagnostic is not pooled with default results or adopted in the RL overlay.

Branch/runtime status: the last main merge incorporates d5f29c072 in 667d0acf5; 1279ebe6f adds the scoped example fusion setting; 710dd6e20 fixes the subsequently identified DSpark replica compilation-cache defect. The branch uses upstream vLLM 0.30.0, retains main's dependency and submodule pins, and adapts the example configs to the current source-group schema. The full benchmark and training matrix below was originally run with 0.29.0; post-merge validation is reported separately.

Router fix and 30B validation update

Long-response math runs exposed slow backend health responses: the router could mark a still-generating vLLM worker unhealthy and fail rollouts with HTTP 503. Commit 31520387b adds inference.router.health_check_timeout_secs, constrained to a positive integer, with the existing five-second default preserved. Both the local inference launcher and the shared SLURM router template pass it to vllm-router --health-check-timeout-secs. This is a six-line implementation change across three files, not only an example TOML change, and does not modify vLLM sources.

The full 30B workload uses 60 seconds. A side-by-side health-probe check returned HTTP 200 in all 106 ready samples with 60 seconds, versus 16 HTTP 503 responses from the original five-second router. This mitigates false worker exclusions; it does not establish the underlying API event-loop latency cause, and it delays detection of a truly unresponsive backend.

The independent serving-speed fix is an upstream configuration override, max_cudagraph_capture_size=1024, for 128 requests and seven draft tokens: target verification schedules up to 128 × (7 + 1) = 1024 positions. The old cap of 512 forced eager fallback for those batches. The draft's own 896-position batch also exceeded 512. No verifier or acceptance rule changed.

For the tested 30B TP4 configuration, compose this after the base RL config and the overlay selecting the separately trained compatible draft:

[inference.router]
health_check_timeout_secs = 60

[inference.vllm.compilation_config]
max_cudagraph_capture_size = 1024

[inference.vllm.compilation_config.pass_config]
fuse_allreduce_rms = false

The router and graph values above are explicit workload overrides, not new global defaults. Backend selection remains vLLM's default auto (MNNVL on these H200s). A larger concurrency ceiling needs corresponding graph and KV-cache capacity.

Latest validation: on 2026-10-05, 207 existing configuration tests passed on an allocated compute node; Ruff lint/format and Git whitespace checks passed. Direct checks verify the default of 5, the override of 60, rejection of nonpositive values, the local command argument, and SLURM template rendering with Bash syntax validation. The complete repository suite was not rerun for these six lines. Earlier supervised tests and CI results below retain their original scope.

Upstream vLLM or a custom fork?

The runtime is the official upstream vLLM release wheel. This PR does not introduce a vLLM fork, custom wheel, or edits to installed vLLM sources.

Component	Implementation boundary
DSpark architecture, proposals, verification and sampling	Existing upstream vLLM support, enabled with speculative_config
Draft models, losses and standalone trainer	Upstream vllm-project/speculators, pinned to ffcccb8; sibling hs-connectors uses the same revision
Standalone attention compilation	PrimeRL worker wraps the upstream FlexAttention function with torch.compile(dynamic=False) before invoking the upstream trainer; no installed source is edited
Live policy/draft weight updates	PrimeRL-owned vLLM worker extensions, admin routes and existing filesystem/NCCL transports
Joint FSDP training	PrimeRL adapter around upstream draft model/loss APIs
DSpark compilation-cache compatibility	PrimeRL runtime patch preserves the target replica's cache index when upstream constructs the draft parallel config; no installed source or kernel is edited

Cache fix validation (710dd6e20): two configuration regression cases pass, including a run with no GPUs exposed. Ruff and diff checks pass. CPU CI passed 678 tests (1 skipped, 191 deselected); slim installation, Ruff and CodeQL also passed. GPU CI remains skipped for this draft; the GPU checks here were supervised compute-node runs. Cold and same-cache TP2×DP2 serving completed 96 requests /32,768 tokens, including 8,192 scored greedy tokens, all target argmax, with zero request failures or preemptions. Both DP engines served requests, and complete per-engine counters agree with responses. Cached logs show all four distinct draft AOT paths. Backend selection remains auto→MNNVL, fusion is disabled, and AOT/compilation/CUDA graphs remain enabled. These short functional checks do not replace the matched TP1 throughput result. Artifacts: dspark-dp-cache-validation.json, startup-audit-dp-index-{cold,cached}.log, and raw startup-dp-index-* responses/counters.

The integration calls vLLM's layerwise reload helpers and an internal per-engine RPC API. It also invokes the draft's fused-KV buffer rebuild hook after reload finalization. These are version-sensitive integration points, so using upstream wheels does not imply compatibility with every vLLM release.

The initial fusion workaround commit changes only examples/specdecode/dspark.toml, examples/specdecode/rl.toml, and their documentation. It disables the graph-level all-reduce/RMSNorm fusion pass; ordinary all-reduce can independently select FlashInfer MNNVL, and cached startup still failed with this flag disabled. Disabling AOT serialization also failed because the regular compilation cache uses the same replica index.

The follow-up investigation identified a separate upstream configuration bug. Independent dense-model data-parallel replicas keep distinct data_parallel_index values after their process-group rank resets to zero. DSpark's replace(parallel_config, ...) reconstructs that init=False field from the zero rank. Both replicas then read/write the same draft compilation paths, whose generated functions contain device-specific CUDA guards. Metadata captured target→draft tensor-device transitions cuda:0→cuda:2 and cuda:3→cuda:1 in two failing workers. The target compilation paths were correctly separated. The new PrimeRL plugin wrapper preserves the original cache index after upstream construction. It is a runtime vLLM compatibility patch, not a fork, custom wheel, kernel replacement or verification change. Use a fresh cache root to avoid artifacts already produced with colliding indices; cache cleanup is not automatic. This DP2 failure cannot explain the separate TP1 accuracy comparisons.

The benchmarks use the default auto backend selection and default AOT settings, with fusion disabled. A diagnostic explicit trtllm override was stopped during compilation before requests and is not adopted. auto selected MNNVL on the tested H200s; its kernel name trtllm_mnnvl_allreduce_fusion refers to FlashInfer communication inside vLLM, not the TensorRT-LLM inference engine. Compilation and CUDA graphs remain enabled. An eager-only diagnostic passed but is not adopted.

The optional specdecode dependency group installs Speculators and its connector. datasets>=4.7.0 avoids the old VideoReader import failure encountered with the installed torchvision version. The UV override replaces an upstream build-machine file:// connector dependency with the pinned Git source; tool.uv.package=true ensures the project CLI entrypoint is installed.

Features shipped

Feature	Behavior and validation
Speculative inference	Composable baseline/DSpark configs; full AIME 2025 baseline, normal-draft and zero-draft runs
Offline draft training	uv run specdecode @ ... trains from saved target hidden states using upstream Speculators
Online draft training	The same bridge requests missing hidden states from a running extraction server
Joint RL + draft training	Optional trainer.speculator; draft joins the policy FSDP mesh and optimizer, with detached targets and an optional separate learning rate
Frozen-backbone training	Freeze the policy before FSDP setup and update only the draft; verified using real batches from the upstream loader
Live weight synchronization	Combined policy/draft filesystem and NCCL broadcasts, plus a filesystem draft-only update endpoint
Checkpoint save/load/resume	Joint DCP state includes both models, optimizer, scheduler and progress; frozen, joint GPU-optimizer and CPU-offload resumes tested
HF export	BF16 target + speculator/ directory; FP8 target export keeps the draft in BF16
Metrics	Draft loss/acceptance estimates, GPU-optimizer draft gradient norm, and inference acceptance/throughput in file and W&B monitors

Online standalone training here means on-demand feature extraction; joint RL training directly reuses the current trainer forward pass.

Design and code structure

flowchart LR
    P[FSDP policy forward] -->|detached layer features| D[Upstream draft model and loss on FSDP]
    P --> O[Shared optimizer and scheduler]
    D --> O
    O --> C[Joint DCP checkpoint]
    O --> W[Policy and speculator weight namespaces]
    W --> T[Filesystem or NCCL transport]
    T --> V[PrimeRL vLLM worker extension]
    V --> I[Upstream target and draft inference]
    C --> E[HF target export plus speculator directory]

Loading

Code	Responsibility
configs/specdecode.py, trainer/RL configs	Optional draft configuration, upstream option validation, inference checkpoint consistency, unsupported-combination checks
specdecode/train.py, specdecode/worker.py	Resolve TOML/CLI, record config snapshots, launch local torchrun workers, explicitly compile draft FlexAttention and invoke the upstream trainer
specdecode/training.py	Discover requested target layers from the checkpoint, capture features, prepare masks, call upstream draft loss, refresh verifier projections, aggregate metrics
trainer/rl/train.py, trainer/optim	Attach the independently sharded draft, add its loss/learning-rate group, reuse optimizer/checkpoint/broadcast lifecycle
utils/weights.py, transports/weights/*	Separate the speculator. namespace and omit reconstructed verifier-only tensors when appropriate
inference/vllm/worker/*, inference/vllm/server.py	Route updates, preserve shared target storage and captured tensor addresses, expose per-worker fingerprints
orchestrator/inference_metrics.py, checkpoint converters	Pooled interval metrics and separate target/draft exports

The joint adapter captures the checkpoint-requested policy layer features and final pre-normalization hidden state. Features are detached, so draft loss does not backpropagate through the policy. It aligns token targets, masks anchors that cross packed-document boundaries, and gathers context-parallel features before the draft forward. Each CP rank therefore processes the full draft sequence.

Frozen verifier embedding/head/norm projections are refreshed from the live policy before training and before publishing updated weights. The optional draft learning rate uses the existing scheduler. Policy RL loss formulas and the optimization_dtype/reduce_dtype settings are unchanged.

Reload correctness requires more than copying parameter values:

• Target and draft can share embedding/head storage. All aliases are inspected, including tied names, so draft-only updates preserve target-owned parameters.
• Draft weights are collected before calling the draft loader, which may rebuild projections from the complete checkpoint. vLLM still performs the TP sharding.
• Layerwise reload is finalized before rebuilding fused KV buffers. Rebuilt tensor values are copied into the original tensors so captured CUDA-graph addresses remain valid.
• The existing pause/update/resume lifecycle publishes both models together. Draft-only replacement invalidates the prefix cache; ordinary RL requests retain policy-version cache salts.
• NCCL updates are not automatically retried after an error because the one-way transfer may already have been consumed.
• GET /weight_fingerprint?target=model|draft gathers hashes, shapes, nonzero counts and addresses from every DP/TP worker. This is an expensive idle-worker diagnostic, not a training hot-path operation.

Example commands and configs

Run model workloads inside compute allocations. Use a one-GPU allocation for an inference server or standalone trainer, separate GPUs for online extraction/training, and six GPUs for the joint RL example. The AIME config uses the Prime runtime and requires its normal credentials. Do not resync dependencies while workers are running.

uv sync --all-extras --all-packages

Inference and AIME evaluation

# Start a target-only server on the allocated compute host.
uv run inference @ examples/specdecode/inference.toml

# In another terminal on that host:
uv run eval @ examples/specdecode/aime25.toml --run.name aime25-baseline

# Stop the baseline server before starting DSpark on the same ports/GPU.
uv run inference @ examples/specdecode/inference.toml @ examples/specdecode/dspark.toml
uv run eval @ examples/specdecode/aime25.toml --run.name aime25-dspark

inference.toml pins the target and serves router/backend ports 18200/18201. dspark.toml pins the draft and requests seven speculative tokens. aime25.toml selects all 30 questions ×8 completions, temperature 0.6, a 31,000-token completion limit and concurrency 16. Point --client.base-url at the compute host if the evaluator runs elsewhere.

Offline and online standalone training

First prepare tokenized target responses with upstream speculators prepare-data, containing input_ids, loss_mask and seq_len. Put the dataset at outputs/specdecode/data, or override train.data.data_path. Training data is an input to the example and is not bundled in the PR.

# Extraction server on its own allocated GPU; router port 18300.
uv run inference @ examples/specdecode/extract-hidden-states.toml

# Offline: generate and validate cached features, then train.
uv run speculators generate-offline-data \
  --preprocessed-data outputs/specdecode/data \
  --endpoint http://localhost:18300/v1 \
  --validate-outputs --fail-on-error
uv run specdecode @ examples/specdecode/qwen3_train_dspark.toml

# Online: request missing features from the running extraction server.
uv run specdecode @ examples/specdecode/qwen3_train_dspark.toml \
  @ examples/specdecode/online.toml

These are alternative training modes; use distinct --output-dir values for separate experiments. The file backend normally reads <data_path>/hidden_states; set [train.backend].hidden_states_path for another location. The example extraction layers match this DSpark checkpoint: 1, 9, 17, 25, 33 and final pre-normalization layer 36. The default standalone recipe runs four updates; --dry-run validates options without loading models. Upstream manages standalone checkpoint saving and resume under train.trainer.save_path.

Frozen draft training and joint RL

# One-GPU FSDP component smoke; the checked-in recipe uses synthetic token batches.
uv run torchrun --standalone --nproc_per_node=1 \
  -m prime_rl.trainer.rl.train @ examples/specdecode/frozen-trainer.toml

# Six-GPU RL smoke: two CP trainer ranks, TP2 x DP2 inference, NCCL updates.
uv run rl @ examples/specdecode/rl.toml \
  --output-dir outputs/specdecode --run.name joint-dspark

# Resume both models and optimizer/progress after the three-step run.
uv run rl @ examples/specdecode/rl.toml \
  --output-dir outputs/specdecode --run.name joint-dspark \
  --resume.step 3 --max-steps 4

The joint recipe uses 4096-token sequences, 3072-token completions, batch size 8/group size 2, i3_math, and three updates. The key optional section is:

[trainer.speculator]
name = "RedHatAI/Qwen3-4B-speculator.dspark"
lr = 0.00001
loss_weight = 1.0
freeze_backbone = false

[trainer.speculator.training]
max_anchors = 16
loss_fn = '{"ce": 0.1, "tv": 0.9}'

Set freeze_backbone = true to train only the draft. The checked-in frozen smoke is synthetic; the stronger real-batch validation below used the upstream offline loader's first two batches.

Export and serve a trained checkpoint

uv run python tools/convert_dcp_to_bf16.py \
  outputs/specdecode/joint-dspark/checkpoints/step_4 \
  outputs/specdecode/joint-dspark-hf

uv run python tools/convert_dcp_to_fp8.py \
  outputs/specdecode/joint-dspark/checkpoints/step_4 \
  outputs/specdecode/joint-dspark-fp8

For inference, set vllm.model to the exported root and vllm.speculative_config.model to its speculator/ directory, with method = "dspark" and num_speculative_tokens = 7. The examples set vllm.compilation_config.pass_config.fuse_allreduce_rms = false; this does not guarantee successful multi-GPU startup on 0.30.0. A filesystem worker also accepts a standalone draft checkpoint through POST /update_weights?target=draft with {"weight_dir":"/path/to/draft"} between /pause and /resume. Keep VLLM_SERVER_DEV_MODE disabled: vLLM's development update route has a different request schema and conflicts with this endpoint.

More usage detail: docs/specdecode.md. All eight recipes are under examples/specdecode.

Results and validation performed

The original validation matrix used H200 GPUs on one compute node, upstream vLLM 0.29.0, Torch 2.13.0+cu130 and Transformers 5.6.2. The implementation was exercised before being recorded in commit 3b6c27c9e, on base f962e9c10. Post-merge checks use upstream vLLM 0.30.0 and main's dependency versions. Original 0.29 results are retained separately from the new 0.30 measurements below.

Post-merge validation: vLLM 0.30.0

The regenerated lockfile installs successfully with uv sync --all-extras --all-packages --locked. All 235 focused tests passed, and Ruff lint/format passed all 24 changed Python files. After the standalone worker fix in 55a279fa1, GitHub CPU CI also passed 676 tests (1 skipped, 191 deselected); Ruff, slim-config installation and all CodeQL checks passed. GPU integration CI remains skipped on the draft PR; the GPU checks below were run manually in compute allocations. The merge retains main's vLLM/router/CUTLASS versions and verifier/renderer submodule pins.

Supervised CP2 + TP2×DP2 joint RL completed updates 1–2 and resumed checkpoint 2→3 with NCCL policy/draft synchronization. Draft gradient norms were 0.5447, 1.0958 and 0.3678; losses were 0.417862, 0.417463 and 0.339368. All three checkpoints saved. The first restart failed during FlashInfer initialization; a retry with the scoped example setting passed, but later startup failures showed that setting is insufficient as a workaround. All eight specdecode config cases were rechecked and passed after the setting changed.

The BF16 export exactly matches all 398 target +62 draft checkpoint tensors. Fresh loading exactly matches the final NCCL policy/draft hashes on all four workers. Draft-only zero/restore preserves the target and captured addresses, restores every owned tensor and gives identical greedy probe outputs. Combined policy+draft filesystem reload also restores all four workers exactly, including addresses. Initial run, resume.

Accuracy audit: the installed upstream standard rejection kernel passed 655,360 draws across point-mass, adversarial, matching, masked-target and temperature-0.6 proposal distributions, within a preset six-binomial-standard-error tolerance plus 2/N. Masked target tokens never appeared; matching proposals accepted 100%. All 4,096 deliberately wrong greedy proposals were replaced by the target argmax. Code inspection confirms target-ratio acceptance and corrected residual sampling; synthetic acceptance and adaptive verification are disabled in the follow-up evaluation. These checks do not establish end-to-end accuracy equivalence. Completed full-length AIME30×16 and greedy-prefix results are reported below.

Matched default-settings TP1 benchmark: neither 0.30 repeat reproduced 5K+. The warmed repeat delivered 4,937.36 output tokens/s, versus 5,087.80 on 0.29 (−2.96%). The first run, including first-request sampling-kernel JIT, delivered 4,564.84 tokens/s (−10.28%). Both results are retained:

Qwen3-4B TP1, full AIME 2025, 30 questions × 8	vLLM 0.29 original	vLLM 0.30 first	vLLM 0.30 warmed
Output tokens / trace wall seconds	4,151,990 / 816.0678	4,104,820 / 899.2250	4,088,901 / 828.1547
Output tokens/s	5,087.80	4,564.84	4,937.36
Correct / 240	151	154	154
Pass@8 / 30	25	24	24
Draft acceptance / mean length	53.66% / 4.7561	53.81% / 4.7669	53.51% / 4.7454
Truncations	30	30	33
Errors / preemptions	0 / 0	0 / 0	0 / 0

The two 0.30 runs use the same live server without changed settings; complete engine counter deltas match trace token totals exactly. The pinned model/draft, TP1/DP1, 32K context, concurrency 16, sampling and PrimeRL eval recipe match the original, with fusion disabled and no backend/AOT override. The new runs use another physical H200 because the original node is occupied. The merged-main software stack also changes dependencies and the PrimeRL wrapper (including enable_scale_out API registration). These limited stochastic repeats do not establish a statistically significant or vLLM-only regression. The original TP1 engine already resolved fusion disabled, so the difference cannot be attributed to turning off that pass. Detailed artifacts: dspark-030-default-aime25/summary.json and dspark-030-default-warm-aime25/summary.json under outputs/specdecode-dev.

Multi-GPU fusion performance, preliminary: on the same four H200s/checkpoint (TP2×DP2, concurrency16), three60-request×2048-token rounds gave medians 13,915.48tok/s enabled versus 13,863.50 disabled: 0.37% lower with fusion disabled. Ranges were13,906.05–14,163.14 and13,742.83–13,960.74 respectively; no preemptions. This difference is small relative to round variation. The final disabled cached restart failed in vllm.all_reduce → FlashInferAllReduce.all_reduce → trtllm_mnnvl_allreduce_fusion, so that phase has no throughput result. Longer600-request rounds were cancelled before requests to prioritize the matched default-settings TP1 benchmark. These short-output four-GPU rates are not directly comparable to full-AIME TP1 rates.

A further installed-kernel test passed 131,072 three-token sequences with prefix-dependent target and adversarial proposal distributions: all64 joint probabilities met the preset tolerance, exercising rejection at each depth and all-accepted bonus sampling. The documentation now explains the rejection guarantee, seed limitations, Avg@16 versus pass@16, and data preparation for a target without a pretrained draft.

Expanded accuracy check: both default-settings AIME Avg@16 sets completed at the full 31K completion limit. The first uses request seeds 1729 + 1009*replicate + question_index; the independent replication uses 25000 + 1009*replicate + question_index. Each set has 480 completions per mode. All settings, questions and scoring are otherwise unchanged. Paired-question bootstrap intervals use 100,000 draws and seed7331.

Default vLLM0.30 evaluation	Ordinary	DSpark	DSpark minus ordinary, percentage points	Paired-question 95% interval
First Avg@16	310/480, 64.58%	291/480, 60.63%	−3.96	[−7.50, −0.83]
Independent Avg@16	299/480, 62.29%	312/480, 65.00%	+2.71	[−1.46, +6.88]
Pooled Avg@32	609/960, 63.44%	603/960, 62.81%	−0.625	[−3.44, +2.08]

The first negative result is retained, but it did not replicate. Both pooled modes pass25/30 questions at least once; truncations total126 ordinary /129 DSpark. All1,920 completions succeeded without preemptions. Raw records, per-question counts and bootstrap results are in aime16-030-* and accuracy-030-{analysis,seed25000-analysis,avg32-analysis}.json. No persistent regression or accuracy equivalence has been demonstrated.

A separate full-length Avg@16 diagnostic with VLLM_BATCH_INVARIANT=1 completed using the original seeds: 295/480 (61.4583%) ordinary versus 318/480 (66.2500%) DSpark, a difference of +4.7917 points, paired-question bootstrap 95% interval [+1.8750, +7.9167]. Pass@16 is 26/30 versus 25/30, with 65 versus 60 truncations. All scored records completed and no preemptions occurred. This is an exploratory diagnostic, reported separately from default results and the matched throughput benchmark. It does not establish an accuracy improvement from batch invariance or DSpark. The same nominal seed does not couple random draws across execution modes; numerical execution changes proposal/acceptance paths and subsequent draws. Artifact: accuracy-030-batch-invariant-analysis.json.

The batch-invariant ordinary arm was deliberately moved to a longer allocation after 46 completed records because its original allocation could not be extended. Those records and seeds were preserved; unfinished requests were intentionally cancelled and retried under the same config on another H200 in the same node, where the remaining 434 records completed. Its complete timing is invalid, so no speed ratio is reported. Neither default-settings replication arm nor the batch-invariant DSpark arm was restarted by this move. All diagnostic servers and allocations have been released.

A separate greedy test on all30 questions, capped at4,096 tokens, found 0/30 bitwise-identical complete prefixes, with first differences after12–1,263 tokens. Temperature zero rules out sampling randomness as that difference's explanation. All 290 inspected target tensors match exactly. An isolated scored follow-up found all49,152 DSpark tokens and49,152 ordinary tokens were target argmax tokens, with no verification failure observed. At first divergences, the top-two score gap had median0.02250, range0.00050–0.16278. Ordinary decoding itself changed on5/6 control prompts when batch size changed. Fresh identical-prefix one-token probes matched29/30; the remaining case flipped two close target scores. This supports numerical execution differences, without proving their effect on stochastic correctness. Artifacts: greedy-030-comparison.json, accuracy-030-target-equality.json, greedy-030-scored/analysis.json. Read-only fingerprint diagnostics overlapped Avg@16 timing; they did not overlap the completed30×8 throughput benchmark.

Complete DSpark counters give 53.74% draft acceptance and mean acceptance length 4.7617. The direct HTTP Avg@16 harness differs from the matched30×8 benchmark above; their throughput values must not be substituted for each other.

Expanded long-run validation: trained a fresh five-layer DSpark for Qwen/Qwen3-30B-A3B-Thinking-2507. Corpus generation and preparation completed: 8,192 unique i3_math questions, 142,140,970 completion tokens and 143,171,059 total tokens. All 8,192 requests succeeded, with zero errors, aborts or preemptions; 1,164 length-limited responses are retained. Normalized exact AIME matches are excluded. The validated question-disjoint split is 7,372 training / 820 validation, with sequences 819–31,673 tokens.

The proper run completed all five epochs / 5,864 updates, with held-out validation and model/optimizer/scheduler checkpoints for every epoch. The selected epoch-5 checkpoint is checkpoints/4; all 62 tensors are finite. All 5,864 W&B training rows have finite losses, with zero failed records. Completed standalone W&B run.

Completed epoch	Cumulative updates	Held-out loss	Soft acceptance estimate	Greedy expected acceptance length
1	1,173	0.632584	41.7151%	2.833740
2	2,342	0.569627	49.3312%	3.290792
3	3,513	0.536077	53.1601%	3.538906
4	4,686	0.514670	55.2703%	3.693625
5	5,864	0.508600	55.8439%	3.739154

These are teacher-forced validation estimates, not live serving acceptance. The selected trained draft fed the real math RL run with the supplied model and training/evaluation sources, 400 steps, batch 64/group 8, and four trainer/four inference H200 GPUs. Diagnostic checkpoints did not feed that run.

Full 30B RL completed: corrected job 1894 ran from 2026-10-01 03:56:57 to 2026-10-02 16:23:27 UTC and exited successfully. The trainer completed update 400 and its joint policy/draft broadcast at 16:16:01 UTC, then saved the final checkpoint. Checkpoints exist at updates 100, 200, 300 and 400. The file-metric audit finds all 400 completed updates with finite recorded policy/draft losses and timings. The run was not error-free: of 78,142 indexed training episodes, 2,792 failed (889 provider-error and 1,903 harness-error stop conditions); 156 of 5,040 indexed evaluation episodes failed. Root-cause attribution of those later failures and their impact on final scores remain unaudited. This validates the full real-rollout training/broadcast/checkpoint lifecycle at 32K with CP4/EP4 and native CPU optimizer offload; resuming a 30B RL checkpoint and a complete final-quality/error audit are still pending. Completed RL W&B run.

Matched 30B serving benchmark, same four H200s and frozen policy/draft, vLLM 0.30 with fusion off:

Mode	Complete-batch output tokens/s	Sustained output tokens/s
Ordinary decoding	6,050.34	7,381.01
DSpark, graph cap 512	5,592.90	5,988.10
DSpark, graph cap 1024	10,006.62	11,593.52

The fixed configuration improves complete-batch speed by 65.4% and sustained speed by 57.1% over ordinary decoding in this benchmark. It uses 256 held-out math prompts, concurrency 128, natural EOS up to 31K output tokens, and the production token/logprob workload. Zero request errors or preemptions were recorded. This direct-backend benchmark excludes the router and does not measure an entire RL training run.

Observed early RL speed: W&B completion timestamps for updates 2–20 give 14.20 updates/hour for corrected spec versus 8.08 for mathcispo (1.76×). Trainer forward/backward averages 204 versus 198 seconds, while residual batch waits average 37 versus 237 seconds. The corresponding serving rates are about 10.68K versus 6.74K output tokens/s (1.59×). These are observational windows, not an isolated speculation effect: losses, policies, nodes and runtime versions differ, and the historical CISPO window includes failed rollouts. A later wall-hour snapshot gives 15 versus 12 completed updates (1.25×) at different policy ages. Neither ratio is claimed as the whole-run speedup.

Forward/backward overlaps generation; its share of trainer elapsed time does not measure inference idle time. off_policy/mean measures rollout version age, while TARGET_LAG=1 separately gates new admission. In the inspected window, the off-policy limit of 8 was not reached, existing requests continued generating while admission was gated, and acknowledged weight-sync pauses averaged about five seconds in both runs. For updates/hour, filter W&B history to rows containing time/forward_backward, then use step and _timestamp; unfiltered time/step also contains orchestrator records.

The corrected run's initial AIME evaluation was 199/240 (82.92%), pass@8 28/30, all at policy v0, versus CISPO's 193/240. The paired-question difference is +2.5 points, 95% interval [−0.42, +6.25], consistent with sampling variation rather than proof of equivalence. In-flight requests retain KV caches across live weight updates; fixed-policy speculative-sampling correctness does not by itself prove distribution equivalence across such updates. Earlier graph-corrected attempts encountered false router health exclusions and a separate host-memory/node failure; they are excluded from healthy end-to-end speed claims.

Standalone performance bug fixed: activation-checkpoint recomputation can call upstream FlexAttention outside the compiled model, reaching Torch2.13's eager fallback. The PrimeRL standalone worker now compiles that attention function explicitly before invoking the upstream trainer. An isolated H20032K/512-anchor attention forward/backward check reduced median time from371.87ms to4.72ms and peak allocated memory from102.10GiB to0.312GiB; outputs and Q/K/V gradients passed the predeclared rtol=0.03, atol=0.005 comparison. These are one-attention measurements, not whole-model speedups or vLLM serving results. The fix passed212 existing configuration/draft tests, Ruff, a real-feature one-GPU train/checkpoint/resume from steps2→4, and four-GPU train/checkpoint at32K. All62 checkpoint tensors were finite;39 changed on resume, and target-owned projections were unchanged. No eager-attention warning occurred. The eight-record diagnostic dataset (repeated for the four-GPU test) establishes runtime compatibility, not held-out learning quality. The completed five-epoch job used its original upstream launcher; its learning results do not measure the subsequent PrimeRL worker compilation fix.

A separate 30B trainer configuration check completed two updates at32K with CP4/EP4, compilation, activation offloading and native full CPU optimizer offload, using a fresh compatible draft and two synthetic packed microbatches per update. Losses were finite; peak GPU memory42.73GiB. No real rollout, weight synchronization or checkpoint saving was exercised by this check. This earlier trainer-only check is separate from the subsequently completed standalone training and full RL run above.

The 30B standalone interface also passed on eight real generated responses: two updates, held-out validation, and model/optimizer/scheduler checkpoint saving. Its diagnostic checkpoint then passed cold and cached TP4/DP1 serving startup with default backend selection and AOT enabled, fusion disabled. Both phases completed 16 requests × 512 greedy tokens; all 16,384 tokens matched target argmax, with no request failures or preemptions. Logs confirm automatic MNNVL selection and cached AOT loading on all four workers. This clears that tested 30B topology, not the separate 4B TP2×DP2 failure or real RL weight synchronization. The diagnostic weights did not feed the full RL run.

Original full AIME 2025: 30 questions ×8 samples, vLLM 0.29.0

Mode	Accuracy	Pass@8	Output tokens/s	Draft acceptance	Acceptance length	Truncated	Errors
Target only	160/240, 66.67%	25/30	1424.23	—	—	29	0
DSpark	151/240, 62.92%	25/30	5087.80	53.66%	4.7561	30	0
Zero owned draft weights	159/240, 66.25%	24/30	1211.93	0.0092%	1.00065	25	0

DSpark achieved 3.572× baseline throughput. Baseline and normal DSpark ran sequentially on the same physical GPU; zero-draft used another H200 on the same node. Throughput is completed output tokens divided by first dispatch to last arrival, including evaluation overhead and excluding server startup/compilation. All three runs had zero preemptions. These are single-run end-to-end measurements.

Accuracy equivalence is not established. Normal DSpark's paired question-bootstrap difference from baseline was −3.75 percentage points, 95% interval [−8.75,+0.42]. Zero-draft's difference was −0.42 points, interval [−5.42,+4.17]. Matching pass@8 or overlapping intervals is not proof of noninferiority.

Original exact weight and training checks, vLLM 0.29.0

Check	Observed result
TP1 and TP2×DP2 draft-only filesystem zero/restore	Every owned draft tensor zeroed; target unchanged; exact restoration; captured addresses stable; four greedy probes identical before/zero/after
NCCL versus fresh checkpoint load, TP2×DP2	Policy and draft hashes/shapes/dtypes/nonzero counts matched exactly on all four workers; DP replicas agreed
Combined filesystem policy+draft round trip, TP2×DP2	Actual frozen-training broadcast changed both models on every worker, then joint weights restored exactly including addresses
Standalone online and offline	Two updates plus validation/checkpoint saving in each mode; offline extraction validated 10/10 samples
Frozen FSDP, real upstream batches	All 398 target tensors bitwise unchanged; 62 draft tensors changed; a separate frozen run resumed step 2→3
Joint math RL	Three updates plus resume through update 4; nonzero finite draft gradients and changed weights reached all four inference workers
Native CPU-offload component	CP2, draft activation checkpointing, two updates plus same-mode resume 2→3; finite model weights with both models changed
BF16 export and reload	All 398 target +62 exported draft tensors exactly matched checkpoint BF16 casts; fresh vLLM loaded both
FP8 export	Target quantization completed; all 62 BF16 draft tensors matched the BF16 export; FP8 serving not tested

Joint draft gradient norms at updates 1–4 were 0.3504, 0.5466, 0.3903, 0.4580. Steady update 2/3 forward/backward took 4.626/2.975s, or 4811/5374 training tokens/s. The short run validates working gradients and synchronization, not a learning trend. CPU-offload validation was a trainer component test with synthetic batches; the complete RL run used the GPU optimizer.

Matched held-out validation used one fixed 512-token batch, seed 42, and eight anchors:

Draft checkpoint	Loss	Acceptance length
Original	0.582735	3.07007
Standalone online	0.582254	3.07765
Standalone offline	0.583141	3.06801
Frozen FSDP, real upstream batches	0.582318	3.08476

The small dataset contained 10 chunks /8420 tokens from two capped target responses. This comparison establishes functional compatibility, not improved draft quality or generalization.

W&B: joint updates 1–3, resumed update 4. Raw trace/checkpoint artifacts, fingerprint JSONs, helper scripts and the detailed report are retained locally under outputs/specdecode-dev/ and are not included in this PR. The original validation jobs finished and their compute allocations were released.

Automated checks executed

235 focused cases passed after merging main: 207 config cases, 8 weight utilities, 5 draft-mask/optimizer cases, 5 inference-ratio cases and 10 existing client cases. The original pre-merge run passed 228 cases. Ruff lint/format passed all 24 changed Python files. PR diff whitespace checks pass. The complete repository suite was not run.

GitHub checks also passed: CPU tests and slim installation (676 passed, 1 skipped, 191 deselected), Ruff, and CodeQL. GPU CI was skipped for this draft; GPU evidence comes from the supervised compute-node runs described above.

To repeat the focused checks in a compute environment:

uv run pytest -q \
  tests/unit/test_configs.py \
  tests/unit/utils/test_weights.py \
  tests/unit/train/test_specdecode.py \
  tests/unit/orchestrator/test_inference_metrics.py \
  tests/unit/orchestrator/test_clients.py

The follow-up VLLM_BATCH_INVARIANT=1 diagnostic produced30/30 identical1,536-token greedy prefixes across ordinary and DSpark modes, all six256-token batch controls identical across concurrency16/6/1, and30/30 identical next-token replays with exactly equal shared top-five log probabilities. All49,152 tokens per mode followed the target argmax. This supports a numerical explanation for the default greedy discrepancy; it does not prove full stochastic accuracy equivalence. The optional mode is not adopted in the benchmark or RL configuration. Both independent default-settings Avg@16 sets and their pooled Avg@32 are reported above.

An independent audit of all960 scored answers reproduces all310/291 correct results using direct integer comparison; the13 fractional answers are all unequal to their gold answer. No scoring discrepancy was found (accuracy-030-scorer-audit.json).

A complete-serving short-sequence sampling test passed under default vLLM0.30 settings with fusion disabled:32,768 four-token requests (131,072 output tokens), two sampling recipes, and every one of256 enumerated outcomes within the predeclared six-standard-error plus2/N tolerance. No zero-reference-probability outputs, request errors or preemptions occurred. Counters confirm20,442 accepted draft tokens across the two DSpark cases. The test uses a constrained four-digit vocabulary and mode-specific reference probabilities; it detects substantial short-sequence sampling bias but does not prove long-context accuracy equivalence. A completed batch-invariant repeat adds32,768 requests/131,072 tokens, passes the same checks, and makes all measured reference probabilities exactly equal across modes. It remains a diagnostic. Artifacts: serving-distribution-030/analysis.json and raw reference/sample records.

The proper30B data preparation also now converts raw assistant-token masks into DSpark next-token prediction masks and excludes seven tail positions per response before packing, matching the joint RL adapter. The corrected mask matches the adapter exactly on12 real packed records, including four31K responses (prediction-mask-check.json). Commitb8d59eb97 documents this custom-data contract; git diff --check passed. This custom-data mismatch was caught before full draft optimization; raw generated tokens/responses and the supplied RL model/train/eval choices are unchanged. The earlier eight-record checkpoint remains a compatibility probe only.

Pending gaps / before this is ready to merge

• The runtime cache-index fix is specific to the tested vLLM 0.30 DSpark path. An upstream fix is not included in the installed wheel, and existing cache artifacts produced with colliding replica indices are not repaired automatically. Cold and same-cache restart passed on the tested TP2×DP2 deployment; use a fresh cache root instead of reusing corrupted artifacts. Other model families and topologies still need validation.
• Validation covers Qwen3-4B and Qwen3-30B-A3B-Thinking-2507 with compatible DSpark drafts. Model/loss construction uses upstream interfaces rather than a Qwen-only guard, but MTP, DFlash and other families need their own feature-alignment and training/reload tests.
• Joint LoRA, multimodal token batches and NIXL draft routing are explicitly rejected. Joint SFT draft training is not implemented; standalone specdecode is separate from the SFT entrypoint.
• CP currently gathers full features and repeats draft-sequence computation per CP rank. The full 400-update real-rollout 30B run at 32K with CP4/EP4 completed, but multi-node training and other parallel topologies remain unvalidated. A matched same-batch measurement of added draft-training cost is still pending.
• Preserve optimizer offload mode across resume. GPU→CPU-offload checkpoint conversion failed because their optimizer-state layouts differ; same-mode GPU and CPU resumes passed.
• FP8 export passed; FP8 serving and quantized draft reload did not receive GPU validation.
• Accuracy equivalence remains unproven. The first default0.30 Avg@16 drop did not replicate, and pooled Avg@32 is consistent with sampling variation, but its interval still permits a 3.44-point drop. The separate batch-invariant Avg@16 is positive (+4.79 points) but exploratory; greedy/default execution differs numerically without an observed verification bypass.
• Upstream standalone trainer max_steps does not stop its outer epoch loop at the pinned revision: a 30B preflight configured for two updates continued into later epochs. Bounded examples use one epoch; the proper five-epoch run has no step cap. No upstream fix is included in this PR.
• GPU round trips and the full CPU-offload RL run were supervised compute-node checks, not an automated integration CI suite. Full-size 30B checkpoint resume, the final quality/error audit, and a controlled whole-RL speed comparison remain pending. The larger-concurrency overlay has only been dry-run validated, not benchmarked.
• The router timeout is a mitigation for slow health responses, not a fix for the underlying API latency. The workload-specific graph cap must be revisited if request concurrency or speculative width changes.