Skip to content

[Bugfix] HybridAttnBackend declares forward() twice, killing the linear-attention path - #34303

Closed
noron12234 wants to merge 1 commit into
sgl-project:mainfrom
noron12234:fix/hybrid-attn-duplicate-forward
Closed

noron12234 wants to merge 1 commit into
sgl-project:mainfrom
noron12234:fix/hybrid-attn-duplicate-forward

Conversation

@noron12234

@noron12234 noron12234 commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Motivation

HybridAttnBackend declares forward() twice. Python binds the later
definition, so the earlier one is dead code:

line origin signature
154 [Kimi] Support kimi-k3 (#32541), 2026-08-04 q, k, v, layer, forward_batch, save_kv_cache, *, mixed_qkv, a, b, **kwargs
224 [feat] Support different attention backends for prefill and decode, 2025-07-28 q, k, v, layer, forward_batch, save_kv_cache, **kwargs

#32541 added the linear-attention-aware forward() at the top of the class but
did not remove the pre-existing one further down, so the 2025 definition is the
one that is actually bound.

The effect is that the linear-attention dispatch added by #32541 never runs.
A caller passing mixed_qkv / a / b — as
hybrid_linear_attn_backend.py does — falls into **kwargs on the old
definition and is then handed to backend.forward() alongside q=None, k=None, v=None, instead of taking the if mixed_qkv is not None: branch.

Modifications

  • Remove the stale 2025 forward(). The kimi-k3 one is a strict superset: same
    parameters for the regular path, plus the linear-attention branch.
  • Add two unit tests to
    test/registered/unit/model_executor/model_runner_components/test_attention_backend_setup.py
    (already a CPU test module, and it already imports HybridAttnBackend)
    asserting that forward() routes mixed_qkv/a/b to the selected backend
    and that the plain q/k/v path is unchanged. A future duplicate
    definition fails these instead of silently shadowing.

No behaviour change for the regular attention path; the only difference is
positional vs keyword argument passing to backend.forward().

Accuracy Tests

Not applicable — this restores a dispatch branch that is currently unreachable,
it does not change kernel or model math.

Verification

black and isort clean on both touched files.

I was not able to run the full test module locally: importing sglang.srt on
this machine fails at transformers config registration
(ValueError: 'qwen3_asr' is already used by a Transformers config), which is a
version-pinning issue in my environment and unrelated to this change. I
verified the dispatch by executing the real, unmodified HybridAttnBackend
class body with stubbed imports:

forward definitions in class: 1
linear path  -> prefill {'mixed_qkv': 'mixed', 'a': 'a', 'b': 'b'}
regular path -> ('q', 'k', 'v')

Please run the added tests in CI.

Checklist

  • Format your code according to the Format code with pre-commit.
  • Add unit tests according to the Run and add unit tests.
  • Update documentation according to Write documentations.
  • Provide accuracy and speed benchmark results.
  • Follow the SGLang code style guidance.

CI States

Latest PR Test (Base): ❌ Run #33099494579
Latest PR Test (Extra): ❌ Run #33099494294
Latest PR Test (AMD ROCm 7.2): ❌ Run #33099494501

…ar-attention path

HybridAttnBackend has two forward() definitions. The second one wins, so the
first is dead:

  line 154  added by [Kimi] Support kimi-k3 (sgl-project#32541), 2026-08-04
            accepts mixed_qkv / a / b and routes them to the linear-attention
            backend
  line 224  the original from [feat] Support different attention backends for
            prefill and decode, 2025-07-28
            only forwards q / k / v

sgl-project#32541 added the new forward() at the top of the class but did not remove the
pre-existing one below, so Python binds the 2025 version. A caller passing
mixed_qkv/a/b now lands in **kwargs and is handed to backend.forward() with
q=k=v=None instead of taking the linear-attention branch.

Drop the stale definition and keep the kimi-k3 one, which is a strict superset.

Also add two unit tests to test_attention_backend_setup.py covering both
dispatch paths, so a future duplicate definition fails CI instead of silently
shadowing.

Signed-off-by: Lin Junrong <noron12334@gmail.com>
@noron12234
noron12234 force-pushed the fix/hybrid-attn-duplicate-forward branch from 75a5700 to d7df79d Compare August 27, 2026 17:39
@noron12234

Copy link
Copy Markdown
Contributor Author

Rebased onto current main; the branch was well over a thousand commits behind and every red check was an aggregator (pr-gate, call-gate, *-finish) from a stale run rather than a leaf test failure. Cherry-pick applied clean onto the current tip, 2 files. black --check clean on both. Fresh CI should give a real signal now.

@noron12234

Copy link
Copy Markdown
Contributor Author

Closing this. I am cleaning up the PRs I have open and keeping only the ones a maintainer has already picked up; nobody has needed to look at this one, and leaving it open just adds to your queue.

The branch stays on my fork, so nothing is lost if the fix turns out to be worth having. Sorry for the noise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant