Skip to content

[TRTLLM-6371][feat] Restructure C++ KVCacheManager to better handle limited attention layers #7510

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 50 commits into
base: main
Choose a base branch
from
Open

[TRTLLM-6371][feat] Restructure C++ KVCacheManager to better handle limited attention layers #7510

Show file tree
Hide file tree
Changes from 35 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
9cc92f6
Draft
thorjohnsen Sep 3, 2025
28d6dd1
Code simplification
thorjohnsen Sep 9, 2025
8a06a28
Simplify eviction policy into true priority based eviction
thorjohnsen Sep 9, 2025
f30fa42
Remove personal info
thorjohnsen Sep 9, 2025
9d8c420
Remove superfluous method, add better comments
thorjohnsen Sep 9, 2025
6ceed6b
Fixes
thorjohnsen Sep 10, 2025
64f4504
Fixes
thorjohnsen Sep 10, 2025
5eaf9d5
Implement some lookup node methods
thorjohnsen Sep 10, 2025
800b05d
Move some method(s) to make PR diff easier to read
thorjohnsen Sep 10, 2025
21643e3
More fixes
thorjohnsen Sep 15, 2025
e0499cf
Faster and more flexible findNewContextBlock method
thorjohnsen Sep 23, 2025
4840a98
Fix last compile issue
thorjohnsen Sep 23, 2025
8647828
Bug fixes
thorjohnsen Sep 26, 2025
91445c5
Bug fixes
thorjohnsen Sep 26, 2025
76f317e
Turn debug printf into TLLM_LOG_DEBUG
thorjohnsen Oct 1, 2025
ca48af9
Bug fixes
thorjohnsen Oct 3, 2025
3399798
Add multi-node scanning for partial match
thorjohnsen Oct 8, 2025
78bcf4d
Bug fixes
thorjohnsen Oct 9, 2025
8836822
Add lookupBlocks method
thorjohnsen Oct 9, 2025
86ed0f8
Fix priority eviction, add lots of debug printouts
thorjohnsen Oct 10, 2025
42fe938
Fix unit test that relied on leaf block only eviction
thorjohnsen Oct 10, 2025
cef3334
Bug fixes and unit test adjustments
thorjohnsen Oct 14, 2025
ad2aa8f
Fix last unit test
thorjohnsen Oct 16, 2025
cfe0609
Remove superfluous arguments
thorjohnsen Oct 16, 2025
02abe39
Remove superfluous member variable
thorjohnsen Oct 17, 2025
f025cfb
Split setLookupNode into two methods to improve readability of code
thorjohnsen Oct 17, 2025
735fe3c
Move stuff around to simplify diff
thorjohnsen Oct 17, 2025
a678b91
Resolve merge conflicts
thorjohnsen Oct 17, 2025
787b681
Resolve build issue
thorjohnsen Oct 18, 2025
50a6f3c
Merge remote-tracking branch 'upstream/main' into user/tjohnsen/restr…
thorjohnsen Oct 18, 2025
a487b55
Fix remaining merge issues
thorjohnsen Oct 21, 2025
78c6253
Merge remote-tracking branch 'upstream/main' into user/tjohnsen/restr…
thorjohnsen Oct 21, 2025
bb421b9
precommit run
thorjohnsen Oct 21, 2025
f3e5c13
Manual precommit fix
thorjohnsen Oct 21, 2025
d427948
Address issue identified by coderabbit
thorjohnsen Oct 21, 2025
d57c3b4
Bug fixes
thorjohnsen Oct 22, 2025
3f696f6
Bug fixes, more debug printouts
thorjohnsen Oct 24, 2025
c8177d3
Fix last unit test failures after merge with main
thorjohnsen Oct 24, 2025
3802864
precommit run
thorjohnsen Oct 24, 2025
e3a1921
Fix spelling errors
thorjohnsen Oct 24, 2025
24dfa89
Merge remote-tracking branch 'upstream/main' into user/tjohnsen/restr…
thorjohnsen Oct 24, 2025
61f9702
Fix race condition that has been in code for months
thorjohnsen Oct 27, 2025
984a4d2
precommit run
thorjohnsen Oct 27, 2025
23c28fb
Merge remote-tracking branch 'upstream/main' into user/tjohnsen/restr…
thorjohnsen Oct 27, 2025
1014a3e
Update unit test to reflect last bug fix
thorjohnsen Oct 27, 2025
a8ad970
Merge remote-tracking branch 'upstream/main' into user/tjohnsen/restr…
thorjohnsen Oct 27, 2025
39e81e1
Merge remote-tracking branch 'upstream/main' into user/tjohnsen/restr…
thorjohnsen Oct 27, 2025
64ef5f3
Bug fix
thorjohnsen Oct 27, 2025
fd364d9
Bug fix
thorjohnsen Oct 29, 2025
3f22026
Merge remote-tracking branch 'upstream/main' into user/tjohnsen/restr…
thorjohnsen Oct 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Loading