[Model] Streamline EmbeddingGemma 2 config resolution and harmonize Triton prefill attention by lucianommartins · Pull Request #60289 · vllm-project/vllm

lucianommartins · 2026-10-06T19:34:36Z

Overview

This fast-follow pull request addresses review feedback and refines the initial EmbeddingGemma 2 (google/embeddinggemma-2) integration across three focused areas:

Native Sequence Length Resolution: Relaxes get_sentence_transformer_tokenizer_config in vllm/transformers_utils/config.py:1142 to require only "max_seq_length". This enables vLLM's standard ModelConfig pipeline to automatically derive max_model_len: 8192 from sentence_bert_config.json without ad-hoc disk reads or manual CLI flags.
Technical Debt Elimination: Removes 86 lines of redundant manual config inspection and scheduler clamping in vllm/model_executor/models/config.py (EmbeddingGemma2ModelConfig), letting vLLM's built-in scheduler derivation handle token limits natively.
Harmonized Triton Prefill Attention: Caps Triton prefill attention block sizing to 32 for architectures with large head dimensions (head_dim >= 512) in vllm/v1/attention/ops/triton_prefill_attention.py. This eliminates shared memory overflow on NVIDIA Ampere and Ada hardware while composing seamlessly with upstream PR [Perf][ROCm][Attention] Narrow the Triton prefill-attention KV tile on RDNA3/RDNA4 #58225 (AMD RDNA3/4 KV tile narrowing).

Claims

Zero-Config CLI Serving (Roger Wang's Question): Users no longer need to pass --max-model-len 8192 on the CLI. Running vllm serve google/embeddinggemma-2 --runner pooling automatically resolves max_model_len to 8192 from sentence_bert_config.json.
Elimination of 86 Lines of Scheduler Clamping: Deletes manual filesystem reads and invasive scheduler mutations (max_num_batched_tokens, max_num_encoder_input_tokens, encoder_cache_size) from EmbeddingGemma2ModelConfig, restoring canonical vLLM config flow.
Prevention of Triton Prefill Attention OOR (MatthewBonanni's Feedback): Fixes OutOfResources: shared memory on NVIDIA A100 / RTX 3090 / RTX 4090 GPUs when executing bidirectional prefill with head_dim = 512. Reduces shared memory consumption from 294,912 bytes (> 166,912 byte hardware limit) to 98,304 bytes (96 KB).
Seamless Composition with PR [Perf][ROCm][Attention] Narrow the Triton prefill-attention KV tile on RDNA3/RDNA4 #58225: Composes cleanly with PR [Perf][ROCm][Attention] Narrow the Triton prefill-attention KV tile on RDNA3/RDNA4 #58225's _prefer_narrow_kv_tile() logic. When Lk >= 512, BLOCK is 32, so BLOCK_N = min(32, 32) evaluates to 32 on AMD RDNA and NVIDIA alike. When Lk < 512, PR [Perf][ROCm][Attention] Narrow the Triton prefill-attention KV tile on RDNA3/RDNA4 #58225's platform-dependent narrowing is 100% unaltered.
Zero Regressions: All 58/58 embeddings unit & e2e tests, 25/25 kernel attention tests (including mocked ROCm RDNA platform tests), and model-specific unit tests pass without error.

Validation

1. Test Suite Execution & Passing Evidence

A. Kernel Attention Suite (including PR #58225 RDNA mock tests)

pytest tests/kernels/attention/test_triton_prefill_attention.py -v

Result: 25/25 passed in 1.48s.

test_kv_tile_width_is_gated_by_platform[rdna] -> PASSED (BLOCK_N = 32)
test_kv_tile_width_is_gated_by_platform[rocm-not-rdna] -> PASSED (BLOCK_N = 128)
test_kv_tile_width_is_gated_by_platform[not-rocm] -> PASSED (BLOCK_N = 128)
test_kv_tile_width_is_gated_by_platform[rdna-float32] -> PASSED (BLOCK_N = 32)
test_rdna_narrows_the_kv_tile_and_nothing_else -> PASSED
All variable sequence length, causal, and sliding-window prefill attention numerical parity tests -> PASSED

B. End-to-End Embeddings Test Suite

pytest tests/vllm/unittests/test_embeddings_e2e.py -v

Result: 58/58 passed.

All pooling heads, normalization modes, and multi-prompt batching tests pass.

C. Multimodal Pooling & Model Config Unit Tests

pytest tests/models/multimodal/pooling/test_embedding_gemma2.py -v

Result: 21/21 passed.

test_config_capping_and_explicit_limits: Verifies default TRITON_ATTN backend selection and preservation of explicit overrides.
test_sentence_transformer_tokenizer_config_max_seq_length: Verifies get_sentence_transformer_tokenizer_config correctly loads configs containing only max_seq_length (both synthetically and from real model checkpoints).

2. Verified CLI Serving Experience

Users on nightly or latest release can now launch EmbeddingGemma 2 with a clean, standard CLI command:

# Clean launch: no manual --max-model-len required!
vllm serve google/embeddinggemma-2 --runner pooling

Engine Startup Log Verification:

INFO 10-06 19:25:10 config.py:1140] Found sentence-transformers tokenize configuration.
INFO 10-06 19:25:10 config.py:382] Using max_model_len=8192 from sentence-transformers configuration.
INFO 10-06 19:25:10 config.py:302] EmbeddingGemma2: using attention backend AttentionBackendEnum.TRITON_ATTN
INFO 10-06 19:25:11 scheduler.py:112] Initializing SchedulerConfig: max_num_batched_tokens=8192, max_num_seqs=256

Details

1. Triton Prefill Attention Block Sizing & Harmonization with PR #58225 (For @MatthewBonanni)

A. Root Cause: Shared Memory Exhaustion on NVIDIA Hardware

In vllm/v1/attention/ops/triton_prefill_attention.py, context_attention_fwd computes prefill attention tiles for encoder/bidirectional attention. For 16-bit dtypes (torch.bfloat16, torch.float16) on CUDA capability >= 80, get_block_size(q.dtype) defaults to BLOCK = 128.

EmbeddingGemma 2 features full attention layers with a head dimension of head_dim = 512 (Lk = 512). With BLOCK_M = 128 and BLOCK_N = 128:

Each query tile Q of shape [BLOCK_M, head_dim] in BF16 requires:
128 * 512 * 2 bytes = 131,072 bytes (128 KB)
Each key tile K of shape [BLOCK_N, head_dim] in BF16 requires:
128 * 512 * 2 bytes = 131,072 bytes (128 KB)
Each value tile V of shape [BLOCK_N, head_dim] in BF16 requires:
128 * 512 * 2 bytes = 131,072 bytes (128 KB)

Even without multi-stage buffering, the shared memory requirement for the tiles and reduction accumulators reaches 294,912 bytes.
However, on NVIDIA Ampere (A100-SXM4-80GB), the maximum dynamic shared memory per thread block is 166,912 bytes (163 KB). Attempting to launch the kernel throws:

triton.compiler.errors.OutOfResources: out of resource: shared memory,
requested: 294912b, available: 166912b

B. The Resolution: `if Lk >= 512: BLOCK = min(BLOCK, 32)`

By capping BLOCK to 32 when Lk >= 512:

Q tile (32 * 512 * 2 bytes) = 32,768 bytes (32 KB)
K tile (32 * 512 * 2 bytes) = 32,768 bytes (32 KB)
V tile (32 * 512 * 2 bytes) = 32,768 bytes (32 KB)
Total Shared Memory Footprint: 32 KB + 32 KB + 32 KB = 98,304 bytes (96 KB)

96 KB fits comfortably within the 166,912 byte limit on A100, RTX 3090, and RTX 4090, allowing kernel launch and execution with high occupancy.

C. Harmonious Composition with PR #58225

Upstream PR #58225 introduced _prefer_narrow_kv_tile() to narrow the KV tile width (BLOCK_N) on AMD RDNA architectures (e.g. gfx1100, gfx1101, gfx1200, gfx1201) while preserving BLOCK_M = 128.

In context_attention_fwd:

    BLOCK = get_block_size(q.dtype)

    Lq, Lk, _ = q.shape[-1], k.shape[-1], v.shape[-1]
    if Lk >= 512:
        BLOCK = min(BLOCK, 32)

    ...
    # BLOCK_M, num_warps and num_stages stay at the shared defaults; min()
    # leaves dtypes whose default tile is already 32, such as float32, alone.
    BLOCK_N = min(BLOCK, 32) if _prefer_narrow_kv_tile() else BLOCK

The table below demonstrates how the two mechanisms interact:

Architecture / Platform	Head Dimension (`Lk`)	`BLOCK` (Initial)	`Lk >= 512` Cap	`BLOCK_M`	`_prefer_narrow_kv_tile()`	`BLOCK_N` Formula	Final `BLOCK_N`
NVIDIA Ampere/Ada	128 (Llama/Mistral)	128	No change	128	`False`	`BLOCK`	128
AMD CDNA (MI200/MI300)	128 (Llama/Mistral)	128	No change	128	`False`	`BLOCK`	128
AMD RDNA3/4 (gfx11/12)	128 (Llama/Mistral)	128	No change	128	`True`	`min(128, 32)`	32 (PR #58225)
NVIDIA Ampere/Ada	512 (EmbeddingGemma 2)	128	Capped to 32	32	`False`	`BLOCK`	32 (Fixed!)
AMD CDNA (MI200/MI300)	512 (EmbeddingGemma 2)	128	Capped to 32	32	`False`	`BLOCK`	32 (Fixed!)
AMD RDNA3/4 (gfx11/12)	512 (EmbeddingGemma 2)	128	Capped to 32	32	`True`	`min(32, 32)`	32 (Harmonized!)

Key Takeaways:

When Lk < 512: BLOCK is unaltered, so PR [Perf][ROCm][Attention] Narrow the Triton prefill-attention KV tile on RDNA3/RDNA4 #58225 behaves identically to upstream main.
When Lk >= 512: BLOCK = 32, so min(BLOCK, 32) trivially evaluates to min(32, 32) = 32. Both BLOCK_M and BLOCK_N become 32 on all platforms, guaranteeing correctness and hardware safety.

2. Native Sequence Length Auto-Detection & Removal of Manual Clamping (For @rogerwang)

A. The Underlying Issue in `get_sentence_transformer_tokenizer_config`

In vllm/transformers_utils/config.py:

# Before:
if all(k in encoder_dict for k in ("max_seq_length", "do_lower_case")):
    return encoder_dict
return None

In SentenceTransformers, sentence_bert_config.json stores metadata about sentence embedding models. While legacy BERT checkpoints included "do_lower_case", modern autoregressive and bidirectional models (such as google/embeddinggemma-2, modern Qwen, and modern Llama embeddings) do not specify "do_lower_case".

Because "do_lower_case" was missing from sentence_bert_config.json, all(...) evaluated to False. The loader silently discarded the config and returned None. As a consequence:

vLLM fell back to config.json, which specified max_position_embeddings: 262144.
Users were required to pass --max-model-len 8192 explicitly to avoid allocating massive KV and scheduler buffers.

B. The Fix

We relaxed the condition in vllm/transformers_utils/config.py:1142 to:

# After:
if "max_seq_length" in encoder_dict:
    return encoder_dict
return None

Since max_seq_length is the sole parameter needed by vLLM to bound max_model_len, this allows vLLM's existing create_model_config logic:

st_config = get_sentence_transformer_tokenizer_config(model)
if st_config is not None:
    max_model_len = st_config.get("max_seq_length", max_model_len)

to automatically discover max_model_len = 8192 during engine initialization.

C. Removal of 86 Lines of Ad-Hoc Clamping Code

Because model_config.max_model_len is now correctly set to 8192 from the beginning of engine startup, vLLM's standard scheduler initialization logic natively sets:

scheduler_config.max_num_batched_tokens = 8192
scheduler_config.max_num_encoder_input_tokens = 8192
scheduler_config.encoder_cache_size = 8192

We were able to completely delete the 86 lines of manual JSON parsing, warning loggers, and scheduler clamping from EmbeddingGemma2ModelConfig.verify_and_update_config in vllm/model_executor/models/config.py. The class now cleanly delegates to Gemma4Config and only ensures attention_config.backend defaults to TRITON_ATTN if unassigned.

Pull Request Checklist

I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).
AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and eliminates technical debt.
Testing and Validation: Verified with 58/58 embeddings unit tests, 25/25 kernel prefill attention tests (including mocked RDNA platform tests), and model-specific unit tests.
Code Quality and Style: Clean diff adhering strictly to vLLM's coding guidelines; passed ruff and mypy.
Pull Request Contents: Detailed rationale addressing maintainer review comments and technical questions.

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

MatthewBonanni · 2026-10-06T20:38:10Z

It looks like the HF config doesn't include a max_seq_length? Is this something that only existed in a local version?
https://huggingface.co/google/embeddinggemma-2/blob/main/config.json
https://huggingface.co/google/embeddinggemma-2/blob/main/sentence_bert_config.json

lucianommartins · 2026-10-06T21:20:25Z

Hi @MatthewBonanni -

yes - on the official HF repo (google/embeddinggemma-2), neither sentence_bert_config.json nor config.json specifies max_seq_length, with text_config.max_position_embeddings set to 262,144 (inherited from the Gemma text backbone).

in eager PyTorch (SentenceTransformers), an uncapped 262K position embedding works fine because it only allocates memory for the actual tokens in each batch. But in vLLM, which pre-allocates KV cache and scheduler buffers based on max_model_len, an unguided 262K context causes severe GPU OOMs.

the model's intended context length is 8K (as documented in the model card). To handle this cleanly:

in transformers_utils/config.py: I kept if "max_seq_length" in encoder_dict: to benefit any modern sentence-transformer models that do specify it
in EmbeddingGemma2ModelConfig.verify_and_update_config: I added a clean 4-line architectural default: if max_model_len > 8192 and the user did not pass an explicit limit, default it to 8192.

this automatically caps unconstrained 262K contexts to 8,192 out of the box, while cleanly honoring explicit user overrides like --max-model-len 16384 without any scheduler-hack overhead.

…ean up attention ops Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>

…local path, and revert triton diff Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>

lucianommartins · 2026-10-07T12:05:26Z

hey @MatthewBonanni @Isotr0py @ywang96 - could we move forward with this fast follow for the embeddinggemma-2 implementation?

MatthewBonanni

LGTM, thanks!

MatthewBonanni · 2026-10-07T14:11:22Z

/ci run

github-actions · 2026-10-07T14:11:32Z

✅ @lucianommartins, CI is now available for this PR.

/ci run starts upstream CI; /amd-ci run starts AMD CI only.
Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append --allow-stale to a run command to test an outdated branch at your own risk.
/ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
/amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
/ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

github-actions · 2026-10-07T14:11:57Z

✅ Triggered Buildkite CI #93324 for commit a19afc63811c.

vllm-agent · 2026-10-07T16:29:04Z

CI selector (shadow): 151 test steps (206 jobs) instead of 65 (97 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today's rules. How it works.

Feedback welcome: reply here if it would skip a step this change needs, or runs something unrelated.

steps (jobs)	Today's rules	Selector	Would skip	Would add
NVIDIA, CPU and others	65 (97)	151 (206)	4 (4)	90 (113)
AMD mirrors	58 (85)	134 (190)	4 (4)	80 (109)

Selector would run (151)

amd-kernels-mi355
amd-lm-eval-small-models-harness
amd-qwen3-next-mtp-async-eplb-accuracy
async-engine-inputs-utils-worker
basic-correctness ×2
basic-correctness-cpu-offload
basic-correctness-cumem
basic-correctness-prefetch-offload
basic-correctness-sleep-mode
basic-models-test-other-cpu
basic-models-tests-extra-initialization ×14
basic-models-tests-initialization
basic-models-tests-other
batch-invariance-b200
batch-invariance-h100
benchmarks-cli-test
cpu-language-generation-and-pooling-model-tests ×3
cpu-multimodal-config
cpu-params-env-tokenizers-parser
cpu-reasoning-renderers
cpu-tool-parsers
crosslayer-kv-layout-distributed-nixlconnector-pd-accuracy-tests-4-gpus
cudagraph
distributed-compile-comm-4-gpus
distributed-compile-rpc-tests-2-gpus
distributed-compile-unit-tests-2xh100
distributed-dp-tests-2-gpus
distributed-dp-tests-4-gpus
distributed-flashinfer-nixlconnector-pd-accuracy-4-gpus
distributed-model-tests-2-gpus ×3
distributed-mooncakeconnector-pd-accuracy-4-gpus
distributed-nixlconnector-pd-accuracy-4-gpus
distributed-tests-8xh100
distributed-torchrun-examples-4-gpus
distributed-torchrun-shutdown-tests-2-gpus
dp-ep-distributed-nixlconnector-pd-accuracy-tests-4-gpus
e2e-core-1-gpu
e2e-core-large-memory
e2e-scheduling-1-gpu
e2e-scheduling-accuracy-1-gpu
elastic-ep-scaling-test
engine
engine-1-gpu
entrypoints-integration-api-server ×4
entrypoints-integration-api-server-generate
entrypoints-integration-api-server-openai-chat_completion
entrypoints-integration-api-server-openai-completion
entrypoints-integration-llm
entrypoints-integration-multimodal
entrypoints-integration-pooling
entrypoints-integration-responses-api
entrypoints-integration-speech_to_text
entrypoints-unit-tests
examples
extract-hidden-states-integration
extract-hidden-states-integration-2-gpus
fault-tolerance-e2e-2xh100
fusion-and-compile-unit-tests-2xb200
fusion-e2e-config-sweep-h100
fusion-e2e-quick-h100
fusion-e2e-tp2-ar-rms-config-sweep-h100
fusion-e2e-tp2-ar-rms-dynamo-partition-amd
fusion-e2e-tp2-ar-rms-inductor-partition-amd
fusion-e2e-tp2-asynctp-config-sweep-h100
fusion-e2e-tp2-b200
fusion-e2e-tp2-quick-h100
gemm-rs-ar-2xb200
hybrid-ssm-nixlconnector-pd-accuracy-tests-4-gpus
hybrid-ssm-nixlconnector-pd-prefix-cache-2-gpus
kernels-attention-test ×7
kernels-b200 ×3
kernels-deepgemm-test-h100
kernels-mamba-test
kernels-root-misc-test-b200
kimi-k3-prefix-cache-4xb200 ×2
kimi-k3-unit-tests-b200
kv-offload-large
kv-offload-medium
kv-offload-small
language-models-tests-extra-standard ×2
language-models-tests-granite-l4-compatibility
language-models-tests-hybrid ×2
language-models-tests-standard
lm-eval-dspark-watermark-2xh100
lm-eval-small-models
lm-eval-turboquant-k3v4nc
lm-eval-turboquant-k8v4
lm-eval-turboquant-t3nc
lm-eval-turboquant-t4nc
lm-eval-watermarking
lora ×4
lora-tp-distributed ×4
metrics-tracing-2-gpus
model-executor
mooncake-ec-tcp-e2e-2-gpus
mrcr-eval-small-models
multi-modal-accuracy-eval-small-models
multi-modal-models-standard-1-qwen2
multi-modal-models-standard-2-qwen3-gemma
multi-modal-models-standard-3-llava-qwen2-vl
multi-modal-models-standard-4-other-whisper
multi-modal-processor ×4
multi-modal-processor-cpu ×4
multiconnector-nixl-offloading-pd-accuracy-2-gpus
multiconnector-nixl-offloading-pd-edge-cases-2-gpus
nixlconnector-pd-edge-cases-2-gpus
openai-api-correctness
pipeline-context-parallelism-4-gpus
plugin-tests-2-gpus
push-nixlconnector-pp-prefill-pd-accuracy-4-gpus
pytorch-compilation-dynamic-shapes
pytorch-compilation-passes-unit-tests
pytorch-compilation-unit-tests
pytorch-compilation-unit-tests-h100
pytorch-fullgraph-cudagraph-l4-compatibility
pytorch-fullgraph-test
quantization ×4
quantized-models-test
quantized-moe-test-b200
qwen4-exp-unit-tests-cpu
rayexecutorv2-4-gpus
regression
replayssm-e2e
rust-frontend-core-correctness
rust-frontend-distributed
rust-frontend-openai-coverage
rust-frontend-serve-admin-coverage
rust-frontend-tool-use
samplers-multimodal-beam-search
samplers-test
scale-out-ec-e2e-2-gpus
sharded-rdt-weight-transfer
spec-decode-draft-model ×4
spec-decode-eagle-1-deepseek-qwen
spec-decode-eagle-2-llama3-qwen-vl-other
spec-decode-mtp-deepseek-mimo
spec-decode-mtp-gemma4
spec-decode-mtp-qwen3-5
spec-decode-ngram-suffix
spec-decode-speculators
v1-attention-b200 ×2
v1-attention-h100-mi300 ×2
v1-core
v1-executor-worker
v1-kv-connectors ×4
v1-kv-offload
v1-logits-oracle
v1-metrics-lmeval
v1-others-cpu
v1-sample
v1-spec-decode

Would skip (today's rules run them) (4)

ascend-npu-test
kernels-fla-ops-test-b200
kernels-mhc-test-b200
rl-entrypoints-tests

Would add (today's rules do not run them) (90)

amd-kernels-mi355 (Python record)
batch-invariance-b200 (Python record)
batch-invariance-h100 (Python record)
crosslayer-kv-layout-distributed-nixlconnector-pd-accuracy-tests-4-gpus (Python record)
cudagraph (Python record)
distributed-compile-comm-4-gpus (Python record)
distributed-compile-rpc-tests-2-gpus (Python record)
distributed-compile-unit-tests-2xh100 (Python record)
distributed-dp-tests-2-gpus (Python record)
distributed-dp-tests-4-gpus (Python record)
distributed-flashinfer-nixlconnector-pd-accuracy-4-gpus (Python record)
distributed-mooncakeconnector-pd-accuracy-4-gpus (Python record)
distributed-nixlconnector-pd-accuracy-4-gpus (Python record)
distributed-tests-8xh100 (Python record)
distributed-torchrun-examples-4-gpus (Python record)
distributed-torchrun-shutdown-tests-2-gpus (Python record)
dp-ep-distributed-nixlconnector-pd-accuracy-tests-4-gpus (Python record)
e2e-core-1-gpu (Python record)
e2e-core-large-memory (Python record)
e2e-scheduling-1-gpu (Python record)
e2e-scheduling-accuracy-1-gpu (Python record)
elastic-ep-scaling-test (Python record)
engine (code map)
engine-1-gpu (Python record)
entrypoints-unit-tests (Python record)
examples (Python record)
extract-hidden-states-integration (Python record)
extract-hidden-states-integration-2-gpus (Python record)
fault-tolerance-e2e-2xh100 (Python record)
fusion-and-compile-unit-tests-2xb200 (Python record)
fusion-e2e-config-sweep-h100 (Python record)
fusion-e2e-tp2-ar-rms-config-sweep-h100 (Python record)
fusion-e2e-tp2-ar-rms-dynamo-partition-amd (Python record)
fusion-e2e-tp2-ar-rms-inductor-partition-amd (Python record)
fusion-e2e-tp2-asynctp-config-sweep-h100 (Python record)
gemm-rs-ar-2xb200 (Python record)
hybrid-ssm-nixlconnector-pd-accuracy-tests-4-gpus (Python record)
hybrid-ssm-nixlconnector-pd-prefix-cache-2-gpus (Python record)
kernels-attention-test ×7 (Python record)
kernels-b200 ×3 (Python record)
kernels-deepgemm-test-h100 (Python record)
kernels-mamba-test (code map)
kimi-k3-prefix-cache-4xb200 ×2 (Python record)
kimi-k3-unit-tests-b200 (Python record)
kv-offload-large (Python record)
kv-offload-medium (Python record)
kv-offload-small (Python record)
lm-eval-dspark-watermark-2xh100 (Python record)
lm-eval-small-models (Python record)
lm-eval-turboquant-k3v4nc (Python record)
lm-eval-turboquant-k8v4 (Python record)
lm-eval-turboquant-t3nc (Python record)
lm-eval-turboquant-t4nc (Python record)
lm-eval-watermarking (Python record)
lora ×4 (Python record)
lora-tp-distributed ×4 (Python record)
mooncake-ec-tcp-e2e-2-gpus (Python record)
mrcr-eval-small-models (Python record)
multi-modal-accuracy-eval-small-models (Python record)
multiconnector-nixl-offloading-pd-accuracy-2-gpus (Python record)
multiconnector-nixl-offloading-pd-edge-cases-2-gpus (Python record)
nixlconnector-pd-edge-cases-2-gpus (Python record)
openai-api-correctness (Python record)
plugin-tests-2-gpus (Python record)
push-nixlconnector-pp-prefill-pd-accuracy-4-gpus (Python record)
quantization ×4 (code map)
quantized-models-test (Python record)
quantized-moe-test-b200 (Python record)
qwen4-exp-unit-tests-cpu (code map)
rayexecutorv2-4-gpus (code map)
replayssm-e2e (Python record)
rust-frontend-core-correctness (Python record)
rust-frontend-distributed (Python record)
rust-frontend-openai-coverage (Python record)
rust-frontend-serve-admin-coverage (Python record)
rust-frontend-tool-use (Python record)
samplers-multimodal-beam-search (Python record)
samplers-test (Python record)
scale-out-ec-e2e-2-gpus (Python record)
sharded-rdt-weight-transfer (Python record)
spec-decode-draft-model ×4 (Python record)
spec-decode-eagle-1-deepseek-qwen (Python record)
spec-decode-eagle-2-llama3-qwen-vl-other (Python record)
spec-decode-mtp-deepseek-mimo (Python record)
spec-decode-mtp-gemma4 (Python record)
spec-decode-mtp-qwen3-5 (Python record)
spec-decode-ngram-suffix (Python record)
spec-decode-speculators (Python record)
v1-attention-b200 ×2 (Python record)
v1-attention-h100-mi300 ×2 (Python record)

AMD mirrors: would skip (4)

kernels-fla-ops-test-b200
kernels-mhc-test-b200
platform-tests
rl-entrypoints-tests

AMD mirrors: would add (80)

crosslayer-kv-layout-distributed-nixlconnector-pd-accuracy-tests-4-gpus (Python record)
cudagraph (Python record)
distributed-compile-comm-4-gpus (Python record)
distributed-compile-rpc-tests-2-gpus (Python record)
distributed-compile-unit-tests-2xh100 (Python record)
distributed-dp-tests-2-gpus (Python record)
distributed-dp-tests-4-gpus (Python record)
distributed-flashinfer-nixlconnector-pd-accuracy-4-gpus (Python record)
distributed-mooncakeconnector-pd-accuracy-4-gpus (Python record)
distributed-nixlconnector-pd-accuracy-4-gpus (Python record)
distributed-tests-8xh100 (Python record)
distributed-torchrun-examples-4-gpus (Python record)
distributed-torchrun-shutdown-tests-2-gpus (Python record)
dp-ep-distributed-nixlconnector-pd-accuracy-tests-4-gpus (Python record)
e2e-core-1-gpu (Python record)
e2e-core-large-memory (Python record)
e2e-scheduling-1-gpu (Python record)
e2e-scheduling-accuracy-1-gpu (Python record)
engine (code map)
engine-1-gpu (code map)
entrypoints-unit-tests (Python record)
examples (Python record)
extract-hidden-states-integration (Python record)
extract-hidden-states-integration-2-gpus (Python record)
fusion-and-compile-unit-tests-2xb200 (Python record)
fusion-e2e-config-sweep-h100 (Python record)
fusion-e2e-tp2-asynctp-config-sweep-h100 (Python record)
gemm-rs-ar-2xb200 (Python record)
hybrid-ssm-nixlconnector-pd-accuracy-tests-4-gpus (Python record)
hybrid-ssm-nixlconnector-pd-prefix-cache-2-gpus (Python record)
jit-monitor-no-runtime-jit (Python record)
kernels-attention-test ×7 (Python record)
kernels-flashmla-test-h100 (Python record)
kernels-mamba-test (code map)
kernels-moe-test ×5 (Python record)
kernels-quantization-test ×6 (Python record)
kimi-k3-unit-tests-b200 (Python record)
kv-offload-large (Python record)
kv-offload-medium (Python record)
kv-offload-small (Python record)
lm-eval-dspark-watermark-2xh100 (Python record)
lm-eval-turboquant-k3v4nc (Python record)
lm-eval-turboquant-k8v4 (Python record)
lm-eval-turboquant-t3nc (Python record)
lm-eval-turboquant-t4nc (Python record)
lm-eval-watermarking (Python record)
lora ×4 (Python record)
lora-tp-distributed ×4 (Python record)
mooncake-ec-tcp-e2e-2-gpus (Python record)
multi-modal-accuracy-eval-small-models (Python record)
multiconnector-nixl-offloading-pd-accuracy-2-gpus (Python record)
multiconnector-nixl-offloading-pd-edge-cases-2-gpus (Python record)
nixlconnector-pd-edge-cases-2-gpus (Python record)
nixlconnector-pd-spec-decode-acceptance-2-gpus (Python record)
openai-api-correctness (Python record)
plugin-tests-2-gpus (Python record)
push-nixlconnector-pp-prefill-pd-accuracy-4-gpus (Python record)
quantization ×4 (code map)
quantized-models-test (Python record)
quantized-moe-test-b200 (Python record)
rayexecutorv2-4-gpus (code map)
rust-frontend-core-correctness (Python record)
rust-frontend-distributed (Python record)
rust-frontend-openai-coverage (Python record)
rust-frontend-serve-admin-coverage (Python record)
rust-frontend-tool-use (Python record)
samplers-multimodal-beam-search (Python record)
samplers-test (Python record)
scale-out-ec-e2e-2-gpus (Python record)
sharded-rdt-weight-transfer (Python record)
spec-decode-draft-model ×4 (Python record)
spec-decode-eagle-1-deepseek-qwen (Python record)
spec-decode-eagle-2-llama3-qwen-vl-other (Python record)
spec-decode-mtp-deepseek-mimo (Python record)
spec-decode-mtp-gemma4 (Python record)
spec-decode-mtp-qwen3-5 (Python record)
spec-decode-ngram-suffix (Python record)
spec-decode-speculators (Python record)
v1-attention-b200 ×2 (Python record)
v1-attention-h100-mi300 ×2 (Python record)

_{3 changed files · base c83935b300 · head a19afc6381 · Python record: build 93039 at 1e5d0ea888 · kernel record: table 43b4aae (build 93244), map 43b4aae · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 126 optional steps the selector would also run}

lucianommartins · 2026-10-07T17:13:27Z

hey @MatthewBonanni @ywang96 - I couldn't find what failed on the buildkite/ci/pr... is it due to th stale sync with the head?

other than that, all the checks passed... let's merge it? :D

…riton prefill attention (vllm-project#60289) Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com> Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com> Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>

lucianommartins requested review from LucasWilkinson and MatthewBonanni as code owners October 6, 2026 19:34

claude Bot reviewed Oct 6, 2026

View reviewed changes

mergify Bot added pooling multi-modality Related to multi-modality (#4194) labels Oct 6, 2026

khluu mentioned this pull request Oct 6, 2026

[CI] Require transformers 5.19 for the EmbeddingGemma2 registry test #60295

Merged

6 tasks done

MatthewBonanni reviewed Oct 6, 2026

View reviewed changes

Comment thread vllm/v1/attention/ops/triton_prefill_attention.py

lucianommartins force-pushed the lucianommartins/eg2 branch from 152012d to 3b0fa40 Compare October 6, 2026 20:27

MatthewBonanni reviewed Oct 6, 2026

View reviewed changes

Comment thread tests/models/multimodal/pooling/test_embedding_gemma2.py Outdated

lucianommartins added 2 commits October 6, 2026 21:21

refactor(models): streamline EmbeddingGemma2 config resolution and cl…

8ce5aa7

…ean up attention ops Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>

refactor(models): clean up EmbeddingGemma2 config resolution, remove …

dae6406

…local path, and revert triton diff Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>

lucianommartins force-pushed the lucianommartins/eg2 branch from 68b90df to dae6406 Compare October 6, 2026 21:21

MatthewBonanni approved these changes Oct 7, 2026

View reviewed changes

MatthewBonanni added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 7, 2026

Merge branch 'main' into lucianommartins/eg2

a19afc6

MatthewBonanni requested a review from noooop as a code owner October 7, 2026 14:11

MatthewBonanni enabled auto-merge (squash) October 7, 2026 14:12

khluu disabled auto-merge October 7, 2026 21:56

khluu merged commit b803edb into vllm-project:main Oct 7, 2026
155 of 156 checks passed

lucianommartins deleted the lucianommartins/eg2 branch October 7, 2026 22:13

kasperg3 mentioned this pull request Oct 8, 2026

[Model][LoRA] Support LoRA for EmbeddingGemma2Model #60632

Open

5 of 6 tasks

vuquoctrungbk mentioned this pull request Oct 9, 2026

[Bug]: EmbeddingGemma 2 fails at startup on sm_86 and sm_120: triton_prefill_attention needs 163840 B shared memory, limit is 101376 B #60847

Open

1 task done

Navigation Menu

Repository navigation

[Model] Streamline EmbeddingGemma 2 config resolution and harmonize Triton prefill attention - #60289

[Model] Streamline EmbeddingGemma 2 config resolution and harmonize Triton prefill attention#60289
khluu merged 3 commits into
vllm-project:mainfrom
lucianommartins:lucianommartins/eg2

lucianommartins commented Oct 6, 2026 •

edited by github-actions Bot

Loading

Uh oh!

claude Bot left a comment

Uh oh!

Uh oh!

Uh oh!

MatthewBonanni commented Oct 6, 2026

Uh oh!

lucianommartins commented Oct 6, 2026

Uh oh!

lucianommartins commented Oct 7, 2026

Uh oh!

MatthewBonanni left a comment

Uh oh!

MatthewBonanni commented Oct 7, 2026

Uh oh!

github-actions Bot commented Oct 7, 2026

Uh oh!

github-actions Bot commented Oct 7, 2026

Uh oh!

vllm-agent commented Oct 7, 2026

Uh oh!

lucianommartins commented Oct 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

lucianommartins commented Oct 6, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Claims

Validation

1. Test Suite Execution & Passing Evidence

A. Kernel Attention Suite (including PR #58225 RDNA mock tests)

B. End-to-End Embeddings Test Suite

C. Multimodal Pooling & Model Config Unit Tests

2. Verified CLI Serving Experience

Details

1. Triton Prefill Attention Block Sizing & Harmonization with PR #58225 (For @MatthewBonanni)

A. Root Cause: Shared Memory Exhaustion on NVIDIA Hardware

B. The Resolution: if Lk >= 512: BLOCK = min(BLOCK, 32)

C. Harmonious Composition with PR #58225

2. Native Sequence Length Auto-Detection & Removal of Manual Clamping (For @rogerwang)

A. The Underlying Issue in get_sentence_transformer_tokenizer_config

B. The Fix

C. Removal of 86 Lines of Ad-Hoc Clamping Code

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

Uh oh!

Uh oh!

MatthewBonanni commented Oct 6, 2026

Uh oh!

lucianommartins commented Oct 6, 2026

Uh oh!

lucianommartins commented Oct 7, 2026

Uh oh!

MatthewBonanni left a comment

Choose a reason for hiding this comment

Uh oh!

MatthewBonanni commented Oct 7, 2026

Uh oh!

github-actions Bot commented Oct 7, 2026

Uh oh!

github-actions Bot commented Oct 7, 2026

Uh oh!

vllm-agent commented Oct 7, 2026

CI selector (shadow): 151 test steps (206 jobs) instead of 65 (97 jobs)

Uh oh!

lucianommartins commented Oct 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lucianommartins commented Oct 6, 2026 •

edited by github-actions Bot

Loading

B. The Resolution: `if Lk >= 512: BLOCK = min(BLOCK, 32)`

A. The Underlying Issue in `get_sentence_transformer_tokenizer_config`