Repository navigation
[#16251][feat] KV cache manager v1: add disk (L3) tier with per-request retention TTL #16252
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
nafis271
wants to merge
55
commits into
NVIDIA:main
Choose a base branch
from
deepinfra:disk-kv-tier-pr
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+2,826
−68
Open
Changes from 49 commits
Commits
Show all changes
55 commits
Select commit
Hold shift + click to select a range
cba9d21
[#16251][feat] KV-cache disk tier: config plumbing, poolless disk blo…
125aba4
[#16251][feat] disk tier: retained-only gate, per-request disk retent…
b301d7f
[#16251][feat] disk tier: re-stamp reused blocks, deadline-heap displ…
fc8e892
[#16251][feat] disk tier: trailing-default ctor params (non-breaking)…
a638e07
[#16251][test] disk tier: C++ unit test suite (DiskTier spill/gate/ex…
667f474
[#16251][feat] disk tier: protect-unexpired admission mode (refuse ne…
6158a3d
[#16251][fix] disk tier: fix spill-all eviction order (consume empty …
fe19f0b
[#16251][perf] disk tier: async per-slot store baseline (pre reserved…
f7c4fa0
[#16251][perf] disk tier: unstaged reserved-block-pool store (pin sou…
4288656
[#16251][chore] disk tier: profiling instrumentation (load-wait vs en…
629277e
[#16251][perf] disk tier: N background writer threads on the shared q…
331d678
[#16251][chore] disk tier: drop profiling instrumentation
c0df24d
[#16251][perf] batch_manager: detach disk-tier onboard from the sched…
938f8e6
[#16251][feat] batch_manager: reuse-count disk spill gate (TLLM_KV_DI…
ffcb1f5
[#16251][feat] disk tier: tensor-parallel support (per-rank disk dir …
09c80a9
[#16251][perf] disk tier: skip areBlocksReady scan when no disk read …
c128718
[#16251][fix] batch_manager: bind disk-tier reader thread to its CUDA…
1b507ac
[#16251][fix] batch_manager: correct disk-tier queue-integrity valida…
b54f55a
[#16251][fix] batch_manager: don't partial-reuse disk-resident blocks
e2864a8
[#16251][fix] batch_manager: hold a disk slot until its onboard read …
0f00346
[#16251][fix] batch_manager: fail loud on disk-tier I/O errors
85f61b4
[#16251][fix] pyexecutor: gate disk onboard-readiness sync to plain TP
5d29113
[#16251][fix] pyexecutor: re-check can_queue after parking onboard-pe…
b95254e
[#16251][perf] disk tier: move onboard slot-write wait off the schedu…
5eec1bf
[#16251][perf] disk tier: retention-aware write-pressure policy (opt-…
326420f
[#16251][perf] disk tier: stage onboard reads through a pinned host b…
04e2ab3
[#16251][perf] pyexecutor: skip onboard-readiness park + its TP colle…
b27c6c9
[#16251][fix] disk tier: close two silent-KV-corruption paths on onbo…
97eee6e
[#16251][fix] disk tier: reap read-pending releases before the disk-f…
a988406
[#16251][fix] serve: don't pass kv_cache_retention_config to the mm-e…
6d61f66
Merge branch 'main' into disk-kv-tier-pr
4e12a20
[#16251][fix] kv cache: shut down disk workers before releasing pools
ad59c79
[#16251][fix] disk tier: bound the write queue for retained spills
3727a82
[#16251][fix] disk tier: anchor retention deadline at commit, not all…
7177595
[#16251][fix] disk tier: contain async worker I/O failures
ad27775
[#16251][infra] apply clang-format to the review-fix files
ffcb625
[#16251][doc] fix stale disk-cache docstring and comment placement
5f0afb7
[#16251][fix] validate kv_cache_ttl_seconds and reject CP with the di…
c054a3c
[#16251][infra] dedup spill enqueue paths and scope test env vars
83ea50c
Merge branch 'main' into disk-kv-tier-pr
1a551b9
[#16251][chore] minor comment cleanup
a2c8628
[#16251][fix] erase disk deadline entries when their blocks leave the…
f611319
Merge branch 'main' into disk-kv-tier-pr
23c7130
[#16251][fix] refuse TLLM_KV_DISK_DROP_ON_PRESSURE with tensor parall…
492a0cf
[#16251][fix] anchor retention decisions to a rank-consistent clock
0855e55
Merge branch 'main' into disk-kv-tier-pr
46e4040
[#16251][chore] rename a parameter flagged by codespell
b995e3e
Merge branch 'main' into disk-kv-tier-pr
3ea3cd8
Merge branch 'main' into disk-kv-tier-pr
343512c
[#16251][chore] add disk write-pressure getter and fix onboard doc co…
4bcca7b
[#16251][fix] disk tier: fix per-manager slot collisions under spec d…
d65addb
Merge remote-tracking branch 'upstream/main' into mainmerge
cebfeb5
[#16251][fix] sync retention clock in disagg-gen path; annotate are_b…
a0d33fe
Merge branch 'main' into disk-kv-tier-pr
d980233
[#16251][fix] disk-tier: use getUInt64Env for env knobs; test cleanup
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@nafis271 Could you check if you can switch to kvCacheManagerV2 which already has this support? If you want more information about kvCacheManagerV2, please let us know. Thanks!
We are planning to migrate from V1 to V2 very soon.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for letting me know! The reason we implemented it on V1 is that when we wanted this explicit user-provided TTL cache at DeepInfra, most of our models ran on V1 and we could not migrate due to performance and compatibility issues with V2. But I understand it makes sense to move to V2 if that is the long term plan.