Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build-vllm-audio.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ env:
IMAGE_NAME: docker.io/openimage/vllm-openai-audio
# Deterministic upstream release the patch series is generated against.
# Bump this in lockstep with fork/scripts/refresh-patches.sh on each release.
DEFAULT_BASE_TAG: v0.25.1
DEFAULT_BASE_TAG: v0.26.0

jobs:
# ---------------------------------------------------------------------------
Expand Down
39 changes: 22 additions & 17 deletions FORK.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,17 +15,20 @@ fork-owned file lives under `fork/` (plus one CI workflow). You can always
**0001 — [#47953](https://github.com/vllm-project/vllm/pull/47953): Restrict
embedding-width share guard to EAGLE drafts.**
Fixes a Gemma-4 MTP boot crash (`mat1 and mat2 ... 6400/10752` at
`pre_projection`). v0.25.1 carries the #43957 regression; without this, V1 + MTP
won't boot.
`pre_projection`). v0.26.0 still carries the #43957 regression; without this,
V1 + MTP won't boot.

**0002 — [#44993](https://github.com/vllm-project/vllm/pull/44993): Advance
grammar across reasoning boundary.**
Fixes structured-output `{{` / `{"{` corruption under reasoning + spec decode
(#43388). The grammar must advance at the true reasoning boundary; the
placeholder-derived delta window misses `</think>` when drafts are rejected.

Both are **pure-Python** upstream backports. Patch 0002 carries only the PR's
source changes (its test file is not present in the runtime image).
Both are **pure-Python** upstream backports, byte-identical to the merged
upstream commits. Patch 0002 carries only the PR's source changes (its test file
is not present in the runtime image). Both PRs merged *after* `v0.26.0` was cut,
so they are still not in any release — drop them once a release we rebase onto
contains them.

Each patch is filed with full context — impact, root cause, a **reproduce case**
to re-check relevance, validation, and ruled-out theories — under
Expand All @@ -49,7 +52,7 @@ Two things are deliberately decoupled:
patches. It is *not* what we build.
- **The image** is built from a pinned release tag — `DEFAULT_BASE_TAG` in
[`.github/workflows/build-vllm-audio.yml`](.github/workflows/build-vllm-audio.yml),
currently **`v0.25.1`**.
currently **`v0.26.0`**.

The patch files in `fork/patches/` are generated against that exact tag, which
is why they apply with no fuzz. If a patch ever fails to apply, the image build
Expand All @@ -58,22 +61,24 @@ ship an image whose patches silently did nothing.

## Lockstep with upstream releases

When vLLM cuts a new release (e.g. `v0.26.0`):
When vLLM cuts a new release (e.g. `v0.27.0`):

```bash
# 1. Rebase the patch series onto the new tag (verifies + regenerates).
fork/scripts/refresh-patches.sh v0.26.0
fork/scripts/refresh-patches.sh v0.27.0

# 2. Bump the base tag the image builds from.
# edit .github/workflows/build-vllm-audio.yml -> DEFAULT_BASE_TAG: v0.26.0
# edit .github/workflows/build-vllm-audio.yml -> DEFAULT_BASE_TAG: v0.27.0

# 3. Review the regenerated patches, commit, push. Pushing to main (or running
# the workflow) builds and publishes openimage/vllm-openai-audio:v0.26.0.
# the workflow) builds and publishes openimage/vllm-openai-audio:v0.27.0.
```

If `refresh-patches.sh` reports a patch no longer applies, rebase that patch by
hand — or, if the upstream PR has since merged into the release, drop it from
`fork/patches/series` entirely.
Before step 1, check whether each patch's upstream PR already landed in the new
release — `git merge-base --is-ancestor <merge-commit> <tag>`, with the merge
commit recorded in the patch's note. If it did, drop the patch from
`fork/patches/series` instead of rebasing it. Otherwise, if
`refresh-patches.sh` reports a patch no longer applies, rebase it by hand.

**CI hygiene.** This fork keeps only its own workflow
(`.github/workflows/build-vllm-audio.yml`); upstream's governance/lint workflows
Expand All @@ -87,22 +92,22 @@ The canonical integrated tree is the `fork/<tag>` branch (the release tag with
the patch series applied as discrete commits):

```bash
git fetch origin fork/v0.25.1
git log --oneline v0.25.1..origin/fork/v0.25.1 # exactly the two patches
git fetch origin fork/v0.26.0
git log --oneline v0.26.0..origin/fork/v0.26.0 # exactly the two patches
```

Or apply a single patch against a fresh checkout to inspect it in isolation:

```bash
git worktree add /tmp/v0.25.1 v0.25.1
cd /tmp/v0.25.1
git worktree add /tmp/v0.26.0 v0.26.0
cd /tmp/v0.26.0
git apply --check fork/patches/0001-restrict-embedding-width-guard-to-eagle-pr47953.patch
```

## The image

- **Registry / name:** `docker.io/openimage/vllm-openai-audio`
- **Tags:** the upstream base tag (e.g. `v0.25.1`) and `latest`.
- **Tags:** the upstream base tag (e.g. `v0.26.0`) and `latest`.
- **Drop-in:** entrypoint is inherited from `vllm/vllm-openai`, so it replaces
the stock image directly.
- **CI:** [`build-vllm-audio.yml`](.github/workflows/build-vllm-audio.yml) —
Expand Down
4 changes: 2 additions & 2 deletions fork/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,8 @@ against a new tag with `scripts/refresh-patches.sh <tag>`.
```bash
# from the repo root
docker build -f fork/docker/Dockerfile.audio \
--build-arg BASE_TAG=v0.25.1 \
-t openimage/vllm-openai-audio:v0.25.1 .
--build-arg BASE_TAG=v0.26.0 \
-t openimage/vllm-openai-audio:v0.26.0 .
```

The build fails loudly if any patch does not apply to `BASE_TAG` — that is the
Expand Down
4 changes: 2 additions & 2 deletions fork/docker/Dockerfile.audio
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,10 @@
# The entrypoint is inherited unchanged, so this is a drop-in replacement for
# vllm/vllm-openai:${BASE_TAG}.
#
# BASE_TAG MUST be a real upstream release tag (e.g. v0.25.1). The patch series
# BASE_TAG MUST be a real upstream release tag (e.g. v0.26.0). The patch series
# is generated against that tag; bumping BASE_TAG without refreshing the series
# (fork/scripts/refresh-patches.sh) will fail the build by design.
ARG BASE_TAG=v0.25.1
ARG BASE_TAG=v0.26.0
FROM vllm/vllm-openai:${BASE_TAG}

# `pip show` reads metadata WITHOUT importing vllm, so this stays fast and needs
Expand Down
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# Fork patch 0001 — Restrict embedding-width share guard to EAGLE drafts
# Upstream PR: https://github.com/vllm-project/vllm/pull/47953
# Generated against tag: v0.25.1
# Generated against tag: v0.26.0
# Apply from the vLLM package root (site-packages/ or repo root): patch -p1 < this-file
#
diff --git a/vllm/v1/spec_decode/llm_base_proposer.py b/vllm/v1/spec_decode/llm_base_proposer.py
index 756c5f3b3..45b463737 100644
index f8b52d079..6ee442af5 100644
--- a/vllm/v1/spec_decode/llm_base_proposer.py
+++ b/vllm/v1/spec_decode/llm_base_proposer.py
@@ -1468,9 +1468,13 @@ class SpecDecodeBaseProposer:
@@ -1477,9 +1477,13 @@ class SpecDecodeBaseProposer:
"Sharing target model embedding weights with the draft model."
)

Expand Down
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# Fork patch 0002 — Advance grammar across reasoning boundary (source only)
# Upstream PR: https://github.com/vllm-project/vllm/pull/44993
# Generated against tag: v0.25.1
# Generated against tag: v0.26.0
# Apply from the vLLM package root (site-packages/ or repo root): patch -p1 < this-file
#
diff --git a/vllm/v1/core/sched/scheduler.py b/vllm/v1/core/sched/scheduler.py
index 8e01035ae..d079a100e 100644
index b77d01410..d979bd7d1 100644
--- a/vllm/v1/core/sched/scheduler.py
+++ b/vllm/v1/core/sched/scheduler.py
@@ -1635,7 +1635,9 @@ class Scheduler(SchedulerInterface):
@@ -1729,7 +1729,9 @@ class Scheduler(SchedulerInterface):
request.status = RequestStatus.FINISHED_STOPPED
stopped = True

Expand All @@ -19,7 +19,7 @@ index 8e01035ae..d079a100e 100644
assert struct_output_request is not None
grammar = struct_output_request.grammar
diff --git a/vllm/v1/structured_output/__init__.py b/vllm/v1/structured_output/__init__.py
index 34f775257..27a4b26f8 100644
index 939a387ee..7619a7fc2 100644
--- a/vllm/v1/structured_output/__init__.py
+++ b/vllm/v1/structured_output/__init__.py
@@ -7,7 +7,6 @@ from concurrent.futures import Future, ThreadPoolExecutor
Expand Down Expand Up @@ -48,7 +48,7 @@ index 34f775257..27a4b26f8 100644
class StructuredOutputManager:
"""Engine-level manager for structured output requests."""

@@ -368,7 +363,11 @@ class StructuredOutputManager:
@@ -379,7 +374,11 @@ class StructuredOutputManager:
return request.structured_output_request.reasoning_ended
return True

Expand All @@ -61,7 +61,7 @@ index 34f775257..27a4b26f8 100644
if not request.use_structured_output:
return False

@@ -391,37 +390,40 @@ class StructuredOutputManager:
@@ -402,37 +401,36 @@ class StructuredOutputManager:
if structured_req.reasoning_ended:
return True

Expand All @@ -88,9 +88,7 @@ index 34f775257..27a4b26f8 100644
+ start = len(all_token_ids) - len(new_token_ids)
+ delta_ids: Iterable[int] = new_token_ids
+ else:
+ delta_from = (
+ request.num_computed_tokens - request.num_output_placeholders
+ )
+ delta_from = request.num_computed_tokens - request.num_output_placeholders
+ start = (
+ delta_from
+ if delta_from >= 0
Expand Down Expand Up @@ -121,9 +119,7 @@ index 34f775257..27a4b26f8 100644
- )
- return True
+ # Record the boundary so the scheduler can exclude reasoning tokens.
+ end_index = self._find_reasoning_end_index(
+ reasoner, all_token_ids, start
+ )
+ end_index = self._find_reasoning_end_index(reasoner, all_token_ids, start)
+
+ structured_req.reasoning_end_token_index = end_index
+ return True
Expand Down
28 changes: 18 additions & 10 deletions fork/patches/notes/0001-gemma4-mtp-boot-crash.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@
| **Patch file** | [`../0001-restrict-embedding-width-guard-to-eagle-pr47953.patch`](../0001-restrict-embedding-width-guard-to-eagle-pr47953.patch) |
| **Upstream PR** | <https://github.com/vllm-project/vllm/pull/47953> |
| **File touched** | `vllm/v1/spec_decode/llm_base_proposer.py` |
| **Applied on** | `v0.25.1` |
| **Upstream status** | Open, mergeable. Related: issue #47794, sibling PR #47833 ("Always share target embeddings for MTP"). |
| **Drop this patch when** | #47953 (or an equivalent such as #47833) lands in a release we rebase onto — verify with the reproduce below, then remove it from `../series`. |
| **Applied on** | `v0.26.0` |
| **Upstream status** | **Merged** 2026-07-21 as `b2b8f679d` — *after* `v0.26.0` was cut (2026-07-20), so it is in no release yet. Related: issue #47794, sibling PR #47833 ("Always share target embeddings for MTP"). |
| **Drop this patch when** | `b2b8f679d` is an ancestor of the tag we rebase onto (`git merge-base --is-ancestor b2b8f679d <tag>`) — expected in the release after `v0.26.0`. Then remove it from `../series`. |

## Why it hurts us (impact)

Expand Down Expand Up @@ -42,11 +42,13 @@ MTP drafts, which is wrong for Gemma-4:
4. `pre_projection` is `Linear(2 * backbone_hidden_size = 10752, ...)`, so the
**6400**-wide input cannot be multiplied by the **10752**-wide weight. Crash.

This is a **`0.25.x`-only regression**: before #43957 (e.g. the pre-regression
nightly `34b560b72`, `v0.23.1rc1.dev786`), vLLM shared the target embedding
unconditionally for MTP, so `combined = 10752` and it worked. The `pre_projection`
weight is genuinely **10752**-wide in the checkpoint, so the correct fix is to
**share** the embedding, not to resize anything.
This regression is present in every release from `0.25.0` through **`v0.26.0`**
(fixed on `main` only by #47953, merged after the `v0.26.0` cut): before #43957
(e.g. the pre-regression nightly `34b560b72`, `v0.23.1rc1.dev786`), vLLM shared
the target embedding unconditionally for MTP, so `combined = 10752` and it
worked. The `pre_projection` weight is genuinely **10752**-wide in the
checkpoint, so the correct fix is to **share** the embedding, not to resize
anything.

**The fix** restricts the width guard to EAGLE drafts. EAGLE draft modules define
`has_own_embed_tokens`; MTP drafts do not (the code already branches on
Expand All @@ -71,8 +73,9 @@ VLLM_USE_V2_MODEL_RUNNER=0 vllm serve <gemma-4-31B-it-FP8-block> \
--max-model-len 8192
```

- **Stock `v0.25.1`:** crashes at boot; the log contains `Keeping separate
embedding weights` and the `6400`/`10752` shape error.
- **Stock `v0.26.0`:** crashes at boot; the log contains `Keeping separate
embedding weights` and the `6400`/`10752` shape error. (The guard is byte-for-byte
unchanged since `v0.25.1`, where this was observed directly.)
- **With patch 0001:** boots; the log contains only `Sharing target model
embedding weights with the draft model`, and the server serves a completion.

Expand All @@ -87,6 +90,11 @@ Applied on `vllm/vllm-openai:v0.25.1` with the 31B main and its MTP draft: befor
patch is effective. Confirmed on H100 (SM90) and L40S (SM89). The built fork image
was re-validated the same way on **2026-07-18**.

**2026-07-25 — `v0.26.0` rebase, static verification only.** `_maybe_share_embeddings`
is byte-identical in `v0.25.1` and `v0.26.0`, so the crash carries over unchanged;
the regenerated patch matches upstream's merged commit `b2b8f679d` exactly and
applies to `v0.26.0` with no fuzz. Not re-run on hardware.

## Ruled out (do not re-explore)

An early hypothesis that the MTP draft was inheriting the fp8-block main's
Expand Down
15 changes: 11 additions & 4 deletions fork/patches/notes/0002-structured-output-reasoning-corruption.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@
| **Patch file** | [`../0002-advance-grammar-across-reasoning-boundary-pr44993.patch`](../0002-advance-grammar-across-reasoning-boundary-pr44993.patch) |
| **Upstream PR** | <https://github.com/vllm-project/vllm/pull/44993> |
| **Files touched** | `vllm/v1/core/sched/scheduler.py`, `vllm/v1/structured_output/__init__.py` |
| **Applied on** | `v0.25.1` (source changes only; the PR's test file is not needed in the runtime image) |
| **Upstream status** | Open, mergeable. Closes issues #43388, #34650, #48228. Related: #41967 (Gemma-4 + MTP dropping the first tool-call arguments — same root cause in the tool-call path). |
| **Drop this patch when** | #44993 lands in a release we rebase onto — verify with the reproduce below, then remove it from `../series`. |
| **Applied on** | `v0.26.0` (source changes only; the PR's test file is not needed in the runtime image) |
| **Upstream status** | **Merged** 2026-07-24 as `0416dab27` — *after* `v0.26.0` was cut (2026-07-20), so it is in no release yet. Closes issues #43388, #34650, #48228. Related: #41967 (Gemma-4 + MTP dropping the first tool-call arguments — same root cause in the tool-call path). |
| **Drop this patch when** | `0416dab27` is an ancestor of the tag we rebase onto (`git merge-base --is-ancestor 0416dab27 <tag>`) — expected in the release after `v0.26.0`. Then remove it from `../series`. |

## Why it hurts us (impact)

Expand Down Expand Up @@ -77,7 +77,8 @@ VLLM_USE_V2_MODEL_RUNNER=0 vllm serve google/gemma-4-E2B-it \
Drive it with structured (native) JSON requests that also enable thinking, at
concurrency around 16.

- **Stock `v0.25.1`:** roughly 75–100% of responses corrupt with `{{`.
- **Stock `v0.26.0`:** roughly 75–100% of responses corrupt with `{{`. (Measured
on `v0.25.1`; both touched paths are unchanged in `v0.26.0`.)
- **With patch 0002:** 0% corrupt.

**Relevance check:** if stock (unpatched) shows 0% corruption here, the upstream
Expand All @@ -90,6 +91,12 @@ On Gemma-4-E2B, applying both coupled changes moved corruption from
with no errors. Because the fault is in the shared grammar-bitmask path and not in
any model-specific code, the fix carries to the 31B production model.

**2026-07-25 — `v0.26.0` rebase, static verification only.** `should_advance` and
the scheduler call site are byte-identical in `v0.25.1` and `v0.26.0`, so the bug
carries over unchanged; the regenerated patch matches upstream's merged commit
`0416dab27` exactly, and the patched files are byte-identical to `upstream/main`.
Not re-run on hardware.

## Ruled out (do not re-explore)

- **Parallel-vs-serial bitmask path** — the parallel threshold is 128 and the bug
Expand Down