Fix decode bucket generation for hybrid models with mismatched block sizes

[yangulei]

Problem

For hybrid models like Qwen3.5 (GDN + attention), _align_hybrid_block_size() sets block_size=640 (unified KV-cache page for mamba/attention alignment), while HPU kernels use attn_block_size=128.

The decode bucket generation (introduced by f24f3f9) uses the formula:

max_decode_blocks = ceil(max_model_len / block_size) * max_num_seqs
                  = ceil(262144 / 640) * 45 = 18450

But the runtime decode path (_create_decode_input_data) computes num_blocks using attn_block_size=128, producing values up to ceil(262144/128) * 45 = 92160.

This causes hundreds of "Configuration was not warmed-up" warnings and costly HPU graph recompilation on every decode step.

Root Cause

Two different block_size semantics coexist:

• self.block_size = 640: KV-cache management page size (unified for hybrid mamba/attention)
• self.attn_block_size = 128: HPU attention kernel page size (what hardware actually uses)

Decode bucket generation used block_size but should use attn_block_size to match the runtime.

Fix

Temporarily scope bucketing_manager.block_size to attn_block_size during decode bucket generation in warmup_model(), then restore the original value so prompt fallback paths remain unaffected.

Testing

• Verified with Qwen3.5-35B-A3B on 4x Gaudi3 (TP=4, max_model_len=262144, max_num_seqs=45)
• Decode buckets now correctly cover runtime num_blocks range
• No more "Configuration was not warmed-up" warnings during serving

Signed-off-by: Youlei Yang youlei.yang@intel.com

[Copilot]

Pull request overview

Adjusts Gaudi bucketing warmup to generate decode buckets using the HPU attention kernel block granularity (attn_block_size) for hybrid models where it differs from the KV-cache management block_size, preventing “not warmed-up” warnings and repeated HPU graph recompilation during decode.

Changes:

• Temporarily overrides bucketing_manager.block_size to attn_block_size when generating decode buckets in warmup_model().
• Restores the original bucketing_manager.block_size afterward to avoid impacting prompt fallback behavior.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

[adobrzyn]

Finding 1 🟡 Medium · tests/unit_tests/test_decode_bucket_hybrid.py:L122-141

The test file defines a local _generate_seq_lengths() that reimplements the production method instead of importing/patching the real HPUModelRunner._generate_seq_lengths. As written, the tests will pass even if the production method in vllm_gaudi/v1/worker/hpu_model_runner.py regresses (e.g., someone re-introduces the unconditional cap). The same risk applies to _MockModelRunner — it only encodes the assumed invariants, not the real ones.

Suggestion: Either (a) instantiate the real HPUModelRunner with mocks and call runner._generate_seq_lengths(...), or (b) import the function (or extract it into a pure helper shared by production and tests). Otherwise this file behaves more like documentation than a regression test.

---

[- Reviewed by Awesome ChlOpus]

[kamil-kaczor]

lgtm

[github-actions]

✅ CI Passed

All checks passed successfully against the following vllm commit:
0a54df28471be07b3d668ea21c5e411569d3baea

[github-actions]

✅ CI Passed

All checks passed successfully against the following vllm commit:
0a54df28471be07b3d668ea21c5e411569d3baea