Skip to content

Fail fast on extra_buffer for a non-mamba arch - #39430

Closed
ispobock wants to merge 4 commits into
mainfrom
fix-extra-buffer-nonmamba-validation
Closed

ispobock wants to merge 4 commits into
mainfrom
fix-extra-buffer-nonmamba-validation

Conversation

@ispobock

@ispobock ispobock commented Sep 14, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

--mamba-radix-cache-strategy extra_buffer (or extra_buffer_lazy) on a model with no mamba state arms the mamba ping-pong track buffers, which are never allocated, so the scheduler crashes at decode:

track_index = req.kv.mamba_ping_pong_track_buffer[...].item()
TypeError: 'NoneType' object is not subscriptable

The default strategy is auto, and the auto -> extra_buffer resolution is already gated on the model architecture, so auto never reaches this on a non-mamba arch. Only an explicit extra_buffer / extra_buffer_lazy does, which is a misconfiguration. handle_mamba_radix_cache already returns early for a non-mamba arch, so nothing validates the explicit strategy before it reaches the runtime. This rejects it at resolution with a clear message instead:

ValueError: --mamba-radix-cache-strategy=extra_buffer needs a model with mamba state, but Qwen2ForCausalLM has none. Use 'auto' (default) or 'no_buffer'.

auto and no_buffer stay inert on a non-mamba arch. This is an alternative to #37474, which gates the extra-buffer predicates on a new architecture-derived leaf rather than failing fast.

Test

test/registered/unit/server_args/test_mamba_extra_buffer_arch_guard.py, CPU. The raises case fails on main and passes with this change. The auto/no_buffer case passes on both.

Repro on Qwen2.5-0.5B-Instruct (non-mamba), single GPU. Before: launching with --mamba-radix-cache-strategy extra_buffer starts, then the scheduler dies with the NoneType error above. After: the launch exits at startup with the ValueError above, before the server comes up.


CI States

Latest PR Test (Base): 🚫 Run #35083432358
Latest PR Test (Extra): ❌ Run #35083431958
Latest PR Test (AMD ROCm 10): ❌ Run #35083432336

@ispobock ispobock added run-ci CI: run the baseline test suite on this PR bypass-fastfail labels Sep 14, 2026
@ispobock

Copy link
Copy Markdown
Collaborator Author

move to #37474

@ispobock ispobock closed this Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fastfail run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant