Card-absorbed DFlash speculation: Laguna XS 2.1 FP8 + vLLM 0.25.1 matrix - #651
Merged
Conversation
Vendor speculation schemes should be a card declaration, not an engine fork. Poolside publishes their Laguna models with a separate block-parallel DFlash speculator and documents serving it through a custom llama.cpp branch; vLLM 0.25.0+ serves both the Laguna architecture and its DFlash drafter natively, so Skulk absorbs the whole scheme through the existing card-driven speculative fields: - vllm_spec_method widens to Literal["mtp", "dflash"]; new vllm_spec_draft_repo names a separate speculator repo, mapped to the --speculative-config "model" key. The card validator enforces the pairing both ways (dflash requires a draft repo; a draft repo under mtp contradicts its in-checkpoint drafter contract) so custom cards fail loudly at load rather than opaquely at serve time. The draft repo resolves through vLLM's own Hugging Face cache at engine start; staging drafts through the model store is a follow-up. - New card poolside--Laguna-XS-2.1-FP8 (33.4B MoE, ~35 GB FP8, 262k context, OpenMDW-1.1 license) paired with the vendor-published Laguna-XS-2.1-DFlash-FP8 drafter at the vendor-recommended depth 15 (block size 16). Tool calling off pending #638, thinking declared with the template's default-off toggle, matching the #649 slate precedent. - Validated vLLM matrix bumps 0.24.0+cu129 -> 0.25.1+cu129 in install.sh: 0.25.1 is the floor for the DFlash drafter architectures (the dflash method itself predates it; laguna_dflash.py landed upstream 2026-07-03 in vllm-project/vllm#46853 and first shipped in v0.25.0). The cu129-variant-wheel/ninja/index-pin rationale is unchanged. Docs: architecture-reference.md (engine facts + installer matrix), speculative-decoding.md and vllm-engine.md (DFlash section), CLAUDE.md vllm bullet. Tests: serve-arg emission for the dflash config shape, card-validator pairing rejections, bundled-invariant note. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Extends Skulk’s card-driven vLLM speculative-decoding configuration to support vendor-published “draft model” schemes (DFlash) as pure model-card declarations, and updates the validated vLLM installer matrix accordingly. This keeps engine code generic while enabling Laguna XS FP8 + DFlash via a bundled card.
Changes:
- Expand model-card runtime schema to support
vllm_spec_method = "dflash"plusvllm_spec_draft_repo, with load-time validation for consistent field pairing. - Wire the new draft-repo field through the vLLM runner’s
--speculative-configemission and add targeted unit tests. - Add the first bundled Laguna XS 2.1 FP8 card using DFlash and update docs +
install.shto vLLM0.25.1+cu129.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| website/docs/vllm-engine.md | Documents DFlash as a card-declared speculative scheme for vLLM. |
| website/docs/speculative-decoding.md | Adds DFlash explanation and how the same card fields cover separate-drafter schemes. |
| website/docs/architecture-reference.md | Updates installer vLLM matrix to 0.25.1+cu129 and documents dflash + draft repo behavior. |
| src/skulk/worker/runner/vllm/tests/test_vllm_runner.py | Tests --speculative-config JSON emission for dflash including the model key. |
| src/skulk/worker/runner/vllm/runner.py | Adds spec_draft_repo plumbing into vllm serve args. |
| src/skulk/shared/tests/test_model_cards.py | Adds tests for runtime speculative-field pairing validation at card load time. |
| src/skulk/shared/tests/test_bundled_model_cards.py | Notes pairing is enforced by the card model validator (parse-time failure). |
| src/skulk/shared/models/model_cards.py | Extends runtime config schema (dflash, vllm_spec_draft_repo) and enforces pairing via model validator. |
| resources/inference_model_cards/poolside--Laguna-XS-2.1-FP8.toml | New bundled Laguna XS 2.1 FP8 vLLM card using dflash + vendor draft repo. |
| install.sh | Bumps --with-vllm install to vllm==0.25.1+cu129 and updates wheel index URL. |
| CLAUDE.md | Updates vLLM matrix and documents dflash + vllm_spec_draft_repo mapping. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c84f25c581
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ncy, deep-spec scheduler sizing)
Fresh A100-80GB pod validation of this branch surfaced three defects; all
fixed here, each proven by the same pod run that found it:
- install.sh paired vllm==0.25.1+cu129 with --torch-backend=cu128, a
leftover from the 0.24.0 matrix. vllm 0.25.1 requires torchcodec>=0.14,
which the cu128 torch index does not carry (tops out at 0.11.1+cu128),
so resolution fails outright on a fresh box. Now cu129 (validated: the
cu129 index carries torchcodec 0.15.0+cu129 and the install completes).
- install.sh --with-vllm was not idempotent: a re-run died at `uv venv`
("virtual environment already exists"). Now --allow-existing, matching
the installer's documented idempotency.
- The runner could not bring up deep speculative depths: vLLM budgets
draft slots out of --max-num-batched-tokens, and with its defaults
(2048 batched tokens, 256 seqs) the scheduler budget goes non-positive
at depth 9 ("max_num_scheduled_tokens is set to -1536" observed live
with the depth-15 Laguna card). build_vllm_serve_args now emits
--max-num-batched-tokens = max(8192, 2048 + 256 * (depth - 1)) for
carded depths >= 8; shallow MTP depths keep vLLM's defaults untouched
(the exact shape the #649 cards validated under). 8192 is the
fresh-box-validated value for the Laguna depth-15 card.
Also recorded from the same run: measured DFlash numbers on the card
comment (1.35x single-stream on A100-80GB, acceptance length 3.44; the
A100 lacks native FP8 so the target runs Marlin W8A16, below the vendor's
1.7-2.6x on native-FP8 hardware) and the CUDA >= 12.8 toolchain
requirement for the DFlash NVRTC JIT (12.4 headers lack __nv_fp8_e8m0) in
install.sh, nvidia-cuda-nodes.md, vllm-engine.md, and
architecture-reference.md.
Validation: basedpyright 0 errors, ruff clean, nix fmt no changes,
pytest 2750 passed. New serve-arg tests cover the depth-15 sizing, the
no-flag shallow shape, and linear scaling past the floor.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
Fresh-box pod validation (round 1) + fixes pushedValidated on a fresh RunPod A100 80GB PCIe (~2.3 h, branch @ c84f25c). The run found three defects; all are fixed in e6be74d and each fix was proven by the same pod session that found it. Defects found and fixed:
What proved out (with the fixes/workarounds applied on-pod):
Local gates on the fix commit: basedpyright 0 errors, ruff clean, nix fmt no changes, pytest 2750 passed. |
The capability resolver has no laguna family branch, so the card's thinking declaration resolved with thinking_format "none" in /v1/models metadata while the model actually reasons behind <think> token delimiters. The [reasoning] section states the model truth directly (token_delimited, toggle-capable), matching the resolved shape of the Qwen3.6 vllm cards. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ad182d4ed2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The card's DFlash runtime JIT-compiles its speculator through NVRTC (CUDA 12.8+), which has no ROCm path in vLLM 0.25.1. Advertising vllm-rocm let placement route the instance to an AMD node that would only fail at engine start; the card now declares vllm-cuda alone until a ROCm validation pass proves the pairing (or cards it off per backend). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Extends the card-driven vLLM speculative-decoding fields so a vendor-published speculation scheme is absorbed as a card declaration, with no engine code or vendor fork, and ships the first card using it.
runtime.vllm_spec_methodwidens from"mtp"to"mtp" | "dflash". Newruntime.vllm_spec_draft_reponames a separately published speculator repo and maps to the--speculative-config"model"key. A card model validator enforces the pairing both ways:dflashwithout a draft repo, or a draft repo undermtp(whose drafter lives inside the target checkpoint), fails loudly at card load instead of opaquely at serve time.poolside--Laguna-XS-2.1-FP8: Poolside's 33.4B MoE coding/agentic model, official FP8 release (~35 GB, fits 48 GB-class GPUs), 262k declared context (capped to 32k at the placement stamp like every vllm placement), paired with the vendor-publishedLaguna-XS-2.1-DFlash-FP8block-parallel drafter at the vendor-recommended depth 15 (block size 16). License verified: OpenMDW-1.1 (permissive). Tool calling declared off pending vllm runner: implement tool calling (tool dispatch currently raises) #638; thinking declared with the template's default-off toggle, matching the feat: the wowsers vLLM slate: Qwen3.6 + Gemma 4 QAT + Ornith cards, native-MTP speculation, vLLM 0.24 matrix #649 slate precedent.0.24.0+cu129to0.25.1+cu129ininstall.sh. Verified against upstream tags directly: thedflashmethod and the Laguna target architecture exist at v0.24.0, but the Laguna DFlash drafter (DFlashLagunaForCausalLM) landed upstream in Add Laguna XS.2.1 DFlash drafter support vllm-project/vllm#46853 (2026-07-03) and first shipped in v0.25.0; the cu129 variant wheel for 0.25.1 exists on wheels.vllm.ai. The variant-wheel/ninja/index-pin rationale is unchanged.The draft repo resolves through vLLM's own Hugging Face cache at engine start (the target model still stages through the Skulk model store); staging the draft through the store as a pinned companion is a follow-up.
Why
Poolside's own README routes llama.cpp users to a custom vendor branch for DFlash support. The point of the card-driven speculative fields is that Skulk absorbs a new vendor scheme generically: vLLM serves the whole pairing natively, and the card records the method, depth, and drafter as declarative model truth.
Docs
architecture-reference.md(engine facts + installer matrix),speculative-decoding.md+vllm-engine.md(DFlash sections),CLAUDE.mdvllm bullet.Validation
uv run basedpyright: 0 errors.uv run ruff check: clean.nix fmt: no changes.uv run pytest: 2750 passed, 1 skipped.🤖 Generated with Claude Code