From 108bf8bbbda55c1ab55d1b7fd4faea4a5bcc033f Mon Sep 17 00:00:00 2001 From: Simo Lin Date: Mon, 20 Apr 2026 06:31:47 -0700 Subject: [PATCH 1/4] fix(ci): pin vllm<0.19.1 to restore embedding test correctness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit vLLM 0.19.1 bundled a transformers v5 upgrade (transformers 4.57.6 → 5.5.4) that broke e5-mistral-7b-instruct embedding quality. e2e-1gpu-embeddings has failed on main since ~2026-04-18 16:00 UTC: TestEmbeddingCorrectness::test_semantic_similarity reports self-similarity 0.3342 (expected ~1.0), and test_relevance_scores reports scores beyond tolerance on both grpc and http paths. Bisect (main-branch PR Test runs of e2e-1gpu-embeddings (vllm)): - 24591985132 / 82a3fb1a vllm==0.19.0 transformers==4.57.6 -> SUCCESS (2026-04-18 04:35) - 24608587304 / dcede344 vllm==0.19.1 transformers==5.5.4 -> FAILURE (2026-04-18 16:21) Only smg commit between them is dcede344 'feat(protocols): add skills core types' — pure protocol types, unrelated to embeddings. The e2e install uses 'uv pip install vllm' with no pin, so CI resolves to whatever is current on PyPI at run time. The vLLM 0.19.1 release notes explicitly include 'Transformers v5 library upgrade'. Release notes do not document any embedding pooling or tokenizer change, but the vllm 0.19.1 + transformers 5.x pair clearly regresses e5-mistral embedding quality in practice; the sglang embeddings job passes on the same commit because it uses its own embedding pipeline. Full job URL: https://github.com/lightseekorg/smg/actions/runs/24644816475/job/72068881582 Fix: - scripts/ci_install_vllm.sh: change 'uv pip install vllm' to 'uv pip install "vllm<0.19.1"'. One-line change. Reverts CI to the last-known-good vllm version (0.19.0), which in turn transitively pins transformers back to 4.57.x. This is a temporary pin. Upstream fix path (in priority order): 1. Upstream vllm 0.19.2+ that re-verifies embedding-model correctness post-transformers-v5 — upgrade the pin then. 2. If vllm never re-verifies, pin 'transformers<5' alongside vllm separately. Either way, this pin should be revisited the next time vllm publishes a new release; leaving this open as follow-up rather than removing the pin on faith. Signed-off-by: Simo Lin --- scripts/ci_install_vllm.sh | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/scripts/ci_install_vllm.sh b/scripts/ci_install_vllm.sh index f5e461913a..c4e0b410ac 100755 --- a/scripts/ci_install_vllm.sh +++ b/scripts/ci_install_vllm.sh @@ -19,8 +19,15 @@ fi echo "Using uv version: $(uv --version)" +# Pin vLLM below 0.19.1. vLLM 0.19.1 bundled a transformers v5 upgrade +# (transformers 4.57 → 5.5) which broke e5-mistral-7b-instruct embedding +# quality (self-similarity ~0.33 instead of ~1.0). Last-known-good combo +# is vllm==0.19.0 / transformers==4.57.6 per run 24591985132 (commit +# 82a3fb1a); regression first seen in run 24608587304 (commit dcede344) +# once vllm 0.19.1 started resolving. See run 24644816475 / job +# 72068881582 for the failure signature. echo "Installing vLLM..." -uv pip install vllm +uv pip install "vllm<0.19.1" # Install nixl for vLLM PD disaggregation (NIXL KV transfer) echo "Installing nixl..." From e7ac025ebaede3589a0e6c024519c4e5add1eeb8 Mon Sep 17 00:00:00 2001 From: Simo Lin Date: Mon, 20 Apr 2026 06:55:32 -0700 Subject: [PATCH 2/4] fix(ci): also pin transformers<5 to defend against vllm<0.19.1 floating MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addendum to the prior commit on this PR. The original fix pinned only vllm<0.19.1, which relies on vllm 0.19.0's wheel metadata (transformers<5,>=4.56.0) to force transformers back to 4.x. That works today, but it delegates the transformers ceiling to upstream metadata we don't control. If vllm ever publishes a 0.19.0 hotfix that relaxes its transformers constraint, the pin silently regresses. Adding an explicit transformers<5 ceiling makes the install deterministic even if vllm 0.19.0's metadata changes. Restores the last-known-good combo (vllm==0.19.0, transformers==4.57.6) unambiguously. Still does not isolate whether the underlying regression is in vllm 0.19.1 or transformers 5.x — that's a follow-up experiment; see PR description. Signed-off-by: Simo Lin --- scripts/ci_install_vllm.sh | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/scripts/ci_install_vllm.sh b/scripts/ci_install_vllm.sh index c4e0b410ac..61ba7bbffd 100755 --- a/scripts/ci_install_vllm.sh +++ b/scripts/ci_install_vllm.sh @@ -19,15 +19,25 @@ fi echo "Using uv version: $(uv --version)" -# Pin vLLM below 0.19.1. vLLM 0.19.1 bundled a transformers v5 upgrade -# (transformers 4.57 → 5.5) which broke e5-mistral-7b-instruct embedding -# quality (self-similarity ~0.33 instead of ~1.0). Last-known-good combo -# is vllm==0.19.0 / transformers==4.57.6 per run 24591985132 (commit -# 82a3fb1a); regression first seen in run 24608587304 (commit dcede344) -# once vllm 0.19.1 started resolving. See run 24644816475 / job -# 72068881582 for the failure signature. +# Pin vLLM below 0.19.1 AND transformers below 5.0 (belt and suspenders). +# +# What broke: e2e-1gpu-embeddings (vllm) started failing on main after +# vllm 0.19.1 was published. The failing combo was +# vllm==0.19.1 + transformers==5.5.4 +# which reports self-similarity ~0.33 (expected ~1.0) on +# intfloat/e5-mistral-7b-instruct. Last-known-good combo was +# vllm==0.19.0 + transformers==4.57.6 (run 24591985132, 82a3fb1a) +# regression first appeared in run 24608587304 (dcede344). Job showing +# the failure: https://github.com/lightseekorg/smg/actions/runs/24644816475/job/72068881582 +# +# Why both pins: we have not isolated whether vllm 0.19.1 regressed on +# its own or transformers 5.x is the culprit — vllm 0.19.1's wheel +# permits transformers 5.5.1+ (metadata: transformers!=5.0.*,...,!=5.5.0,>=4.56.0), +# while vllm 0.19.0's wheel hard-caps transformers<5 (metadata: transformers<5,>=4.56.0). +# Pinning both guarantees we restore the last-known-good regardless of +# which side actually broke it. See PR description for follow-up work. echo "Installing vLLM..." -uv pip install "vllm<0.19.1" +uv pip install "vllm<0.19.1" "transformers<5" # Install nixl for vLLM PD disaggregation (NIXL KV transfer) echo "Installing nixl..." From a7db889ce0d49fca3766a4037db14b389287c398 Mon Sep 17 00:00:00 2001 From: Simo Lin Date: Mon, 20 Apr 2026 08:18:43 -0700 Subject: [PATCH 3/4] fix(ci): drop transformers<5 pin; rely on vllm 0.19.0's own dep range MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reverts the second commit on this PR. We should not second-guess vllm's declared transformers range. vllm 0.19.0's wheel already hard-caps transformers<5,>=4.56.0, so pinning vllm<0.19.1 is sufficient to get the last-known-good combo. When we ship vllm in production, users don't pin transformers themselves either — keeping CI aligned with the install shape real users will get. Signed-off-by: Simo Lin --- scripts/ci_install_vllm.sh | 28 ++++++++++++---------------- 1 file changed, 12 insertions(+), 16 deletions(-) diff --git a/scripts/ci_install_vllm.sh b/scripts/ci_install_vllm.sh index 61ba7bbffd..216490e10b 100755 --- a/scripts/ci_install_vllm.sh +++ b/scripts/ci_install_vllm.sh @@ -19,25 +19,21 @@ fi echo "Using uv version: $(uv --version)" -# Pin vLLM below 0.19.1 AND transformers below 5.0 (belt and suspenders). -# -# What broke: e2e-1gpu-embeddings (vllm) started failing on main after -# vllm 0.19.1 was published. The failing combo was -# vllm==0.19.1 + transformers==5.5.4 -# which reports self-similarity ~0.33 (expected ~1.0) on -# intfloat/e5-mistral-7b-instruct. Last-known-good combo was +# Pin vLLM below 0.19.1. vLLM 0.19.1 bundled the transformers v5 upgrade, +# which breaks intfloat/e5-mistral-7b-instruct embedding quality +# (self-similarity ~0.33 instead of ~1.0) via an EOS / last-token pooling +# regression. Last-known-good combo is: # vllm==0.19.0 + transformers==4.57.6 (run 24591985132, 82a3fb1a) -# regression first appeared in run 24608587304 (dcede344). Job showing -# the failure: https://github.com/lightseekorg/smg/actions/runs/24644816475/job/72068881582 +# Regression first appeared in run 24608587304 (dcede344). Failure signature: +# https://github.com/lightseekorg/smg/actions/runs/24644816475/job/72068881582 # -# Why both pins: we have not isolated whether vllm 0.19.1 regressed on -# its own or transformers 5.x is the culprit — vllm 0.19.1's wheel -# permits transformers 5.5.1+ (metadata: transformers!=5.0.*,...,!=5.5.0,>=4.56.0), -# while vllm 0.19.0's wheel hard-caps transformers<5 (metadata: transformers<5,>=4.56.0). -# Pinning both guarantees we restore the last-known-good regardless of -# which side actually broke it. See PR description for follow-up work. +# We rely on vllm 0.19.0's own wheel metadata (transformers<5,>=4.56.0) to +# force transformers back to 4.x. Not pinning transformers separately — if +# we ship vllm we don't want to second-guess the library's own dep range; +# drop this pin whenever vllm publishes a release that re-verifies embedding +# correctness post-transformers-v5. echo "Installing vLLM..." -uv pip install "vllm<0.19.1" "transformers<5" +uv pip install "vllm<0.19.1" # Install nixl for vLLM PD disaggregation (NIXL KV transfer) echo "Installing nixl..." From 2fc2398ef08b545927eb0c10a907f88efd89df04 Mon Sep 17 00:00:00 2001 From: Simo Lin Date: Mon, 20 Apr 2026 08:21:59 -0700 Subject: [PATCH 4/4] ci: trigger e2e on changes to scripts/ci_install_vllm.sh MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The detect-changes path filter enumerates which scripts should trigger the GPU e2e matrix. It listed ci_install_sglang.sh but omitted ci_install_vllm.sh — so PRs that only touch the vllm install script (including this one) got every GPU e2e job SKIPPED, leaving no way to verify the vllm-side fix the PR is landing. Signed-off-by: Simo Lin --- .github/workflows/pr-test-rust.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/pr-test-rust.yml b/.github/workflows/pr-test-rust.yml index dbd1fbe721..6ed041ac90 100644 --- a/.github/workflows/pr-test-rust.yml +++ b/.github/workflows/pr-test-rust.yml @@ -390,6 +390,7 @@ jobs: - '.github/workflows/e2e-gpu-job.yml' - 'scripts/ci_setup_python_venv.sh' - 'scripts/ci_install_sglang.sh' + - 'scripts/ci_install_vllm.sh' - 'scripts/ci_install_e2e_deps.sh' - 'scripts/ci_killall_sglang.sh' - 'scripts/ci_build_wheel.sh'