Repository navigation
Allow platforms to declare Mamba extra-buffer support - #36410
Conversation
|
/tag-and-rerun-ci |
|
Thanks! This branch now needs to be updated onto the latest |
6ca4a34 to
e4237f2
Compare
|
@BBuf @alexnails thanks for reviewing! Rebased onto master |
ed44692 to
d9cc76c
Compare
|
/tag-and-rerun-ci |
|
@alexnails thanks for reviewing this PR!! I rebased the branch. Are we ready to merge it? |
24c9161 to
a79200a
Compare
# Problem Mamba extra-buffer validation hardcodes the supported in-tree device types, so an out-of-tree platform cannot opt in even when it implements the required cache-state snapshots. # Root cause Built-in hardware facts are read through runtime_context.get_platform(), while current_platform owns out-of-tree plugin behavior. The platform interface had no capability for this Mamba cache behavior. # Fix Add a conservative SRTPlatform capability for out-of-tree plugins. Preserve the existing built-in checks through get_platform() and consult the capability only for an out-of-tree platform. Add CPU tests for every supported built-in fact and both out-of-tree capability outcomes. Test Plan: To be added
a79200a to
22b410e
Compare
|
All PR Test Base tests are passing, except for the H200 DeepSeek-v4 FlashInfer/EAGLE ones, which are unrelated to my change. |
Motivation
Mamba extra-buffer validation currently uses a fixed list of device types. This prevents out-of-tree platforms from enabling the feature even when they implement the required cache-state snapshot behavior, and it couples server argument validation to specific hardware names.
Modifications
SRTPlatform.support_mamba_cache_extra_buffer()as the platform capability used by server argument validation.Accuracy Tests
N/A. This changes platform capability validation only and does not modify model computation or numerical behavior.
Speed Tests and Profiling
N/A. This change is outside the inference hot path.
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #37397056745
Latest PR Test (Extra): ❌ Run #37397056334
Latest PR Test (AMD ROCm 10): ❌ Run #37397056631