[Attention][Core] Add generic prefix-anchored SWA on SM70 - #281
Merged
Merged
Conversation
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Port the prefix-anchored sliding-window cache and SM70 kernels to current main as an explicit model-agnostic AttentionConfig capability. Validate operator/runtime contracts up front, fail closed on missing metadata, preserve default-off SASS, and cover eager, piecewise, decode, prefill, gap eviction, and multi-request anchors. Co-authored-by: kkobold <sabbaghdanilo@gmai.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
yangzhuxinyzx
marked this pull request as ready for review
August 24, 2026 16:57
Contributor
Author
|
Audit note: the required |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Replacement for #239, rebased onto current
main. Adds prefix-anchored sliding-window attention as an explicit, default-off inference-engine capability. Admission is based only on hardware, operator, tensor/cache, graph, and concurrency contracts; it does not inspect model/checkpoint/architecture identity.The engine option is
AttentionConfig.prefix_anchored_decode_window. The prefix remains globally visible while generated-token KV outside the live window is masked and reclaimed. Unsupported combinations fail up front, and missing runtime metadata fails closed.Key audit fixes over #239
AttentionConfigand baseAttentionTest Plan / Result
pytest -q tests/config/test_prefix_anchored_swa.py tests/v1/attention/test_prefix_anchored_swa_mask.py tests/v1/core/test_single_type_kv_cache_manager.py: 32 passedCUDA_HOME=/usr MAX_JOBS=4 TORCH_CUDA_ARCH_LIST=7.0 python setup.py build_ext --inplace: passedpytest -q tests/kernels/attention/test_sm70_flash_v100_anchored_swa.pyon V100: 19 passedorigin/main: 55/55 decode and 64/64 paged-prefill default-off instances byte-identicalNo model end-to-end or throughput claim is made. Refs #239.