Repository navigation

[#16251][feat] KV cache manager v1: add disk (L3) tier with per-request retention TTL - #16252

Open

nafis271 wants to merge 55 commits into

NVIDIA:mainfrom

deepinfra:disk-kv-tier-pr

Open

[#16251][feat] KV cache manager v1: add disk (L3) tier with per-request retention TTL#16252
nafis271 wants to merge 55 commits into
NVIDIA:mainfrom
deepinfra:disk-kv-tier-pr

Commits on Jul 10, 2026
- [#16251][feat] KV-cache disk tier: config plumbing, poolless disk blocks, host->disk spill and disk->GPU onboard
  Md Nafis Ul Haque Shifat
  committed
  cba9d21
- [#16251][feat] disk tier: retained-only gate, per-request disk retention TTL, serve wiring
  Md Nafis Ul Haque Shifat
  committed
  125aba4
- [#16251][feat] disk tier: re-stamp reused blocks, deadline-heap displacement (empties first, exact TTL order, suffix-first FIFO), retention wiring at all serve entry points
  Md Nafis Ul Haque Shifat
  committed
  b301d7f
- [#16251][feat] disk tier: trailing-default ctor params (non-breaking), unit tests (C++ disk-tier suite + isEnqueued + python plumbing), disk-count getters
  Md Nafis Ul Haque Shifat
  committed
  fc8e892
- [#16251][test] disk tier: C++ unit test suite (DiskTier spill/gate/expiry/reuse/deadline + isEnqueued + placeholder level fix)
  Md Nafis Ul Haque Shifat
  committed
  a638e07
- [#16251][feat] disk tier: protect-unexpired admission mode (refuse new blocks when disk full of live TTLs; only expired evictable) + unit tests
  Md Nafis Ul Haque Shifat
  committed
  667f474
- [#16251][fix] disk tier: fix spill-all eviction order (consume empty slots before displacing cached blocks)
  Md Nafis Ul Haque Shifat
  committed
  6158a3d
- [#16251][perf] disk tier: async per-slot store baseline (pre reserved-pool unstaged)
  Md Nafis Ul Haque Shifat
  committed
  fe19f0b
- [#16251][perf] disk tier: unstaged reserved-block-pool store (pin source, hand out reserved slot)
  Md Nafis Ul Haque Shifat
  committed
  f7c4fa0
- [#16251][chore] disk tier: profiling instrumentation (load-wait vs enqueue-backpressure vs queue depth)
  Md Nafis Ul Haque Shifat
  committed
  4288656
- [#16251][perf] disk tier: N background writer threads on the shared queue (TLLM_KV_DISK_WRITERS)
  Md Nafis Ul Haque Shifat
  committed
  629277e
- [#16251][chore] disk tier: drop profiling instrumentation
  Md Nafis Ul Haque Shifat
  committed
  331d678
- [#16251][perf] batch_manager: detach disk-tier onboard from the scheduler thread
  Md Nafis Ul Haque Shifat
  committed
  c0df24d
- [#16251][feat] batch_manager: reuse-count disk spill gate (TLLM_KV_DISK_MIN_REUSE)
  Md Nafis Ul Haque Shifat
  committed
  938f8e6
- [#16251][feat] disk tier: tensor-parallel support (per-rank disk dir + cross-rank onboard readiness)
  Md Nafis Ul Haque Shifat
  committed
  ffcb1f5
- [#16251][perf] disk tier: skip areBlocksReady scan when no disk read is in flight
  Md Nafis Ul Haque Shifat
  committed
  09c80a9
- [#16251][fix] batch_manager: bind disk-tier reader thread to its CUDA device
  Md Nafis Ul Haque Shifat
  committed
  c128718
- [#16251][fix] batch_manager: correct disk-tier queue-integrity validators
  Md Nafis Ul Haque Shifat
  committed
  1b507ac
- [#16251][fix] batch_manager: don't partial-reuse disk-resident blocks
  Md Nafis Ul Haque Shifat
  committed
  b54f55a
- [#16251][fix] batch_manager: hold a disk slot until its onboard read completes
  Md Nafis Ul Haque Shifat
  committed
  e2864a8
- [#16251][fix] batch_manager: fail loud on disk-tier I/O errors
  Md Nafis Ul Haque Shifat
  committed
  0f00346
- [#16251][fix] pyexecutor: gate disk onboard-readiness sync to plain TP
  Md Nafis Ul Haque Shifat
  committed
  85f61b4
- [#16251][fix] pyexecutor: re-check can_queue after parking onboard-pending requests
  Md Nafis Ul Haque Shifat
  committed
  5d29113
- [#16251][perf] disk tier: move onboard slot-write wait off the scheduler into the reader
  Md Nafis Ul Haque Shifat
  committed
  b95254e
- [#16251][perf] disk tier: retention-aware write-pressure policy (opt-in TLLM_KV_DISK_DROP_ON_PRESSURE)
  Md Nafis Ul Haque Shifat
  committed
  5eec1bf
- [#16251][perf] disk tier: stage onboard reads through a pinned host buffer
  Md Nafis Ul Haque Shifat
  committed
  326420f
- [#16251][perf] pyexecutor: skip onboard-readiness park + its TP collective when the disk tier is off
  Md Nafis Ul Haque Shifat
  committed
  04e2ab3
- [#16251][fix] disk tier: close two silent-KV-corruption paths on onboard/release
  Md Nafis Ul Haque Shifat
  committed
  b27c6c9
- [#16251][fix] disk tier: reap read-pending releases before the disk-free check
  Md Nafis Ul Haque Shifat
  committed
  97eee6e
- [#16251][fix] serve: don't pass kv_cache_retention_config to the mm-encoder
  Md Nafis Ul Haque Shifat
  committed
  a988406
Commits on Jul 14, 2026
- Merge branch 'main' into disk-kv-tier-pr
  Md Nafis Ul Haque Shifat
  committed
  6d61f66
- [#16251][fix] kv cache: shut down disk workers before releasing pools
  Md Nafis Ul Haque Shifat
  committed
  4e12a20
- [#16251][fix] disk tier: bound the write queue for retained spills
  Md Nafis Ul Haque Shifat
  committed
  ad59c79
Commits on Jul 15, 2026
- [#16251][fix] disk tier: anchor retention deadline at commit, not allocation
  Md Nafis Ul Haque Shifat
  committed
  3727a82
- [#16251][fix] disk tier: contain async worker I/O failures
  Md Nafis Ul Haque Shifat
  committed
  7177595
- [#16251][infra] apply clang-format to the review-fix files
  Md Nafis Ul Haque Shifat
  committed
  ad27775
- [#16251][doc] fix stale disk-cache docstring and comment placement
  Md Nafis Ul Haque Shifat
  committed
  ffcb625
- [#16251][fix] validate kv_cache_ttl_seconds and reject CP with the disk tier
  Md Nafis Ul Haque Shifat
  committed
  5f0afb7
- [#16251][infra] dedup spill enqueue paths and scope test env vars
  Md Nafis Ul Haque Shifat
  committed
  c054a3c
- Merge branch 'main' into disk-kv-tier-pr
  Md Nafis Ul Haque Shifat
  authored
  83ea50c
Commits on Jul 20, 2026
- [#16251][chore] minor comment cleanup
  Md Nafis Ul Haque Shifat
  committed
  1a551b9
- [#16251][fix] erase disk deadline entries when their blocks leave the disk
  Md Nafis Ul Haque Shifat
  committed
  a2c8628
- Merge branch 'main' into disk-kv-tier-pr
  Md Nafis Ul Haque Shifat
  authored
  f611319
Commits on Jul 21, 2026
- [#16251][fix] refuse TLLM_KV_DISK_DROP_ON_PRESSURE with tensor parallelism
  Md Nafis Ul Haque Shifat
  committed
  23c7130
- [#16251][fix] anchor retention decisions to a rank-consistent clock
  Md Nafis Ul Haque Shifat
  committed
  492a0cf
- Merge branch 'main' into disk-kv-tier-pr
  Md Nafis Ul Haque Shifat
  authored
  0855e55
- [#16251][chore] rename a parameter flagged by codespell
  Md Nafis Ul Haque Shifat
  committed
  46e4040
Commits on Jul 28, 2026
- Merge branch 'main' into disk-kv-tier-pr
  Md Nafis Ul Haque Shifat
  committed
  b995e3e
- Merge branch 'main' into disk-kv-tier-pr
  Md Nafis Ul Haque Shifat
  authored
  3ea3cd8
Commits on Jul 29, 2026
- [#16251][chore] add disk write-pressure getter and fix onboard doc comment
  Md Nafis Ul Haque Shifat
  committed
  343512c
Commits on Aug 5, 2026
- [#16251][fix] disk tier: fix per-manager slot collisions under spec decode
  Md Nafis Ul Haque Shifat
  committed
  4bcca7b
- Merge remote-tracking branch 'upstream/main' into mainmerge
  Md Nafis Ul Haque Shifat
  committed
  d65addb
- [#16251][fix] sync retention clock in disagg-gen path; annotate are_blocks_ready
  Md Nafis Ul Haque Shifat
  committed
  cebfeb5
Commits on Aug 12, 2026
- Merge branch 'main' into disk-kv-tier-pr
  Md Nafis Ul Haque Shifat
  committed
  a0d33fe
- [#16251][fix] disk-tier: use getUInt64Env for env knobs; test cleanup
  Md Nafis Ul Haque Shifat
  committed
  d980233

Commits

Commits on Jul 10, 2026

Commits on Jul 14, 2026

Commits on Jul 15, 2026

Commits on Jul 20, 2026

Commits on Jul 21, 2026

Commits on Jul 28, 2026

Commits on Jul 29, 2026

Commits on Aug 5, 2026

Commits on Aug 12, 2026