[FIX_FOR_VLLM_CUSTOM=9a70c233cd5d3ef3b14d79c582a6f9ddfa03cf80] forward skip_padding kwarg, drop stale manager.touch() (+3 more)

[pawel-olejniczak]

This PR consolidates 5 hourly-CI fixes against vllm@9a70c233cd5d3ef3b14d79c582a6f9ddfa03cf80.

Bug 1: forward skip_padding kwarg through HPU MoE router factory override

• State machine id: create_fused_moe_router_unexpected_kwarg
• Commit: 65cc65e

Root cause

upstream vllm#56079 added a skip_padding: bool = False kwarg to create_fused_moe_router()/GroupedTopKRouter to invalidate sequence-parallel padding rows in grouped MoE routing. The vllm-gaudi override mirrors this factory's signature and did not accept the new kwarg, so every call site now raises: TypeError: create_fused_moe_router() got an unexpected keyword argument 'skip_padding'

Culprit

Regression introduced by PR #56079.

Fix

accept skip_padding on the override (signature parity) and forward it to GroupedTopKRouter; the guard it enables only fires under VLLM_MOE_SKIP_PADDING + DeepEP-v2-kernel forward context, which HPU never exercises, so this is a no-op on HPU beyond restoring the call signature.

Bug 2: drop stale manager.touch() assertions in offloading connector test

• State machine id: offloading_connector_touch_assertion
• Commit: c5c5c55

Root cause

upstream vllm#51787 ("Track cache recency once per request") removed OffloadingConnectorScheduler._touch and every manager.touch() call site, replacing per-call cache-recency tracking with req_context.set_offload_key_position() tracked once per request. vllm-gaudi's vendored copy of the unit test still asserted the old manager.touch() calls, so it now fails: AssertionError: Expected 'touch' to have been called.

Culprit

Probable culprit: PR #51787 - pinned by symbol archaeology, bisect not run.
Candidate range: compare.

Fix

mirror upstream's own post-#51787 test update (its analogous tests/v1/kv_connector/unit/offloading_connector/test_scheduler.py) by dropping the removed-call assertions and asserting touch is no longer called for the same scenario.

Bug 3: add missing release_kv_cache_memory to MultiModelEngineClient

• State machine id: online_model_swap_multimodelengineclient_abstract
• Commit: 7ba9ac9

Root cause

upstream vllm#44890 added a new abstract method release_kv_cache_memory to EngineClient; MultiModelEngineClient never implemented it, so instantiation raised TypeError.

Culprit

Probable culprit: PR #44890 - pinned by symbol archaeology, bisect not run.
Candidate range: compare.

Fix

delegate to the wrapped AsyncLLM instance, mirroring the existing sleep/wake_up passthrough pattern.

Bug 4: skip is_sleeping assignment on read-only Executor property

• State machine id: online_model_swap_multimodelengineclient_abstract
• Commit: 198d7e9

Root cause

upstream vllm#44890 (same PR as the MultiModelEngineClient abstract-method break) turned Executor.is_sleeping into a read-only property computed from sleeping_tags. gaudi_reconfigure_engine's _reset_executor_sleep_state still assigned model_executor.is_sleeping directly, raising AttributeError: property 'is_sleeping' of 'UniProcExecutor' object has no setter during online model swap.

Culprit

Probable culprit: PR #44890 - pinned by symbol archaeology, bisect not run.
Candidate range: compare.

Fix

feature-detect whether is_sleeping is a class-level property before assigning it; clearing sleeping_tags already makes the property read False on the new executor.

Bug 5: implement apply_lora_full_linear on PunicaWrapperHPU

• State machine id: hpu_unit_tests_engine_core_init_failed
• Commit: 18d4414

Root cause

upstream vllm PR 53555 ("[LoRA] Support modules_to_save for sequence classification") added a new abstract method apply_lora_full_linear to PunicaWrapperABC and implemented it for CPU, GPU and XPU. PunicaWrapperHPU never implemented it, so it is no longer a concrete class and get_punica_wrapper() raises TypeError: Can't instantiate abstract class PunicaWrapperHPU without an implementation for abstract method 'apply_lora_full_linear' while loading any LoRA model on HPU, which the engine surfaces as the generic "Engine core initialization failed".

Culprit

Probable culprit: PR #53555 - pinned by symbol archaeology, bisect not run.
Candidate range: compare.

Fix

implement the hook with plain torch ops, mirroring the CPU and GPU versions: route rows through the prompt-level sampler_indices mapping and hand off to the inherited _select_full_linear_output helper for the bias and module-enabled selection. Multiplying against every adapter and gathering the routed row afterwards keeps the weight operand statically shaped for HPU graph compilation. The new method only references _select_full_linear_output at call time, and its only caller (ClassificationHeadWithLoRA) postdates PR 53555, so the change stays compatible with older VLLM_STABLE_COMMIT builds.

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

Review effort: Lite
Findings: None

What changed in this PR

This PR updates vLLM-Gaudi compatibility with newer upstream vLLM APIs and removes obsolete offloading scheduler assertions.

Changes:

• Adds skip_padding support to the HPU fused MoE router factory.
• Avoids assigning to the new read-only is_sleeping property and adds release_kv_cache_memory delegation.
• Updates the offloading connector test to reflect removal of manager.touch().

File	Description
vllm_gaudi/​v1/​engine/​core_patch.py	Handles executor sleep-state reset across old and new vLLM APIs.
vllm_gaudi/​ops/​hpu_fused_moe.py	Accepts and forwards the upstream skip_padding router argument.
vllm_gaudi/​entrypoints/​openai/​multi_model_api_server.py	Delegates KV-cache memory release to the wrapped engine.
tests/​unit_tests/​kv_offload/​offloading_connector/​test_scheduler.py	Removes assertions for the deleted manager.touch() behavior.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

✅ CI Passed

All checks passed successfully against the following vllm commit:
9a70c233cd5d3ef3b14d79c582a6f9ddfa03cf80