Skip to content
Open
Show file tree
Hide file tree
Changes from 49 commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
cba9d21
[#16251][feat] KV-cache disk tier: config plumbing, poolless disk blo…
Jul 2, 2026
125aba4
[#16251][feat] disk tier: retained-only gate, per-request disk retent…
Jul 3, 2026
b301d7f
[#16251][feat] disk tier: re-stamp reused blocks, deadline-heap displ…
Jul 3, 2026
fc8e892
[#16251][feat] disk tier: trailing-default ctor params (non-breaking)…
Jul 3, 2026
a638e07
[#16251][test] disk tier: C++ unit test suite (DiskTier spill/gate/ex…
Jul 3, 2026
667f474
[#16251][feat] disk tier: protect-unexpired admission mode (refuse ne…
Jul 3, 2026
6158a3d
[#16251][fix] disk tier: fix spill-all eviction order (consume empty …
Jul 4, 2026
fe19f0b
[#16251][perf] disk tier: async per-slot store baseline (pre reserved…
Jul 6, 2026
f7c4fa0
[#16251][perf] disk tier: unstaged reserved-block-pool store (pin sou…
Jul 6, 2026
4288656
[#16251][chore] disk tier: profiling instrumentation (load-wait vs en…
Jul 6, 2026
629277e
[#16251][perf] disk tier: N background writer threads on the shared q…
Jul 6, 2026
331d678
[#16251][chore] disk tier: drop profiling instrumentation
Jul 6, 2026
c0df24d
[#16251][perf] batch_manager: detach disk-tier onboard from the sched…
Jul 7, 2026
938f8e6
[#16251][feat] batch_manager: reuse-count disk spill gate (TLLM_KV_DI…
Jul 7, 2026
ffcb1f5
[#16251][feat] disk tier: tensor-parallel support (per-rank disk dir …
Jul 8, 2026
09c80a9
[#16251][perf] disk tier: skip areBlocksReady scan when no disk read …
Jul 8, 2026
c128718
[#16251][fix] batch_manager: bind disk-tier reader thread to its CUDA…
Jul 8, 2026
1b507ac
[#16251][fix] batch_manager: correct disk-tier queue-integrity valida…
Jul 8, 2026
b54f55a
[#16251][fix] batch_manager: don't partial-reuse disk-resident blocks
Jul 8, 2026
e2864a8
[#16251][fix] batch_manager: hold a disk slot until its onboard read …
Jul 8, 2026
0f00346
[#16251][fix] batch_manager: fail loud on disk-tier I/O errors
Jul 8, 2026
85f61b4
[#16251][fix] pyexecutor: gate disk onboard-readiness sync to plain TP
Jul 8, 2026
5d29113
[#16251][fix] pyexecutor: re-check can_queue after parking onboard-pe…
Jul 8, 2026
b95254e
[#16251][perf] disk tier: move onboard slot-write wait off the schedu…
Jul 8, 2026
5eec1bf
[#16251][perf] disk tier: retention-aware write-pressure policy (opt-…
Jul 8, 2026
326420f
[#16251][perf] disk tier: stage onboard reads through a pinned host b…
Jul 8, 2026
04e2ab3
[#16251][perf] pyexecutor: skip onboard-readiness park + its TP colle…
Jul 8, 2026
b27c6c9
[#16251][fix] disk tier: close two silent-KV-corruption paths on onbo…
Jul 9, 2026
97eee6e
[#16251][fix] disk tier: reap read-pending releases before the disk-f…
Jul 9, 2026
a988406
[#16251][fix] serve: don't pass kv_cache_retention_config to the mm-e…
Jul 9, 2026
6d61f66
Merge branch 'main' into disk-kv-tier-pr
Jul 14, 2026
4e12a20
[#16251][fix] kv cache: shut down disk workers before releasing pools
Jul 14, 2026
ad59c79
[#16251][fix] disk tier: bound the write queue for retained spills
Jul 14, 2026
3727a82
[#16251][fix] disk tier: anchor retention deadline at commit, not all…
Jul 15, 2026
7177595
[#16251][fix] disk tier: contain async worker I/O failures
Jul 15, 2026
ad27775
[#16251][infra] apply clang-format to the review-fix files
Jul 15, 2026
ffcb625
[#16251][doc] fix stale disk-cache docstring and comment placement
Jul 15, 2026
5f0afb7
[#16251][fix] validate kv_cache_ttl_seconds and reject CP with the di…
Jul 15, 2026
c054a3c
[#16251][infra] dedup spill enqueue paths and scope test env vars
Jul 15, 2026
83ea50c
Merge branch 'main' into disk-kv-tier-pr
Jul 15, 2026
1a551b9
[#16251][chore] minor comment cleanup
Jul 20, 2026
a2c8628
[#16251][fix] erase disk deadline entries when their blocks leave the…
Jul 20, 2026
f611319
Merge branch 'main' into disk-kv-tier-pr
Jul 20, 2026
23c7130
[#16251][fix] refuse TLLM_KV_DISK_DROP_ON_PRESSURE with tensor parall…
Jul 21, 2026
492a0cf
[#16251][fix] anchor retention decisions to a rank-consistent clock
Jul 21, 2026
0855e55
Merge branch 'main' into disk-kv-tier-pr
Jul 21, 2026
46e4040
[#16251][chore] rename a parameter flagged by codespell
Jul 21, 2026
b995e3e
Merge branch 'main' into disk-kv-tier-pr
Jul 28, 2026
3ea3cd8
Merge branch 'main' into disk-kv-tier-pr
Jul 28, 2026
343512c
[#16251][chore] add disk write-pressure getter and fix onboard doc co…
Jul 29, 2026
4bcca7b
[#16251][fix] disk tier: fix per-manager slot collisions under spec d…
Aug 5, 2026
d65addb
Merge remote-tracking branch 'upstream/main' into mainmerge
Aug 5, 2026
cebfeb5
[#16251][fix] sync retention clock in disagg-gen path; annotate are_b…
Aug 5, 2026
a0d33fe
Merge branch 'main' into disk-kv-tier-pr
Aug 12, 2026
d980233
[#16251][fix] disk-tier: use getUInt64Env for env knobs; test cleanup
Aug 12, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions cpp/include/tensorrt_llm/batch_manager/evictionPolicy.h
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,14 @@ class BaseEvictionPolicy
virtual void releaseBlock(BlockPtr block, bool toFront) = 0;
/// @brief Get the amount of free blocks in the primary memory pool
virtual SizeType32 getNumFreeBlocks(SizeType32 cacheLevel) = 0;

/// @brief True when the block currently sits in a free queue (evictable right now).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nafis271 Could you check if you can switch to kvCacheManagerV2 which already has this support? If you want more information about kvCacheManagerV2, please let us know. Thanks!

We are planning to migrate from V1 to V2 very soon.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for letting me know! The reason we implemented it on V1 is that when we wanted this explicit user-provided TTL cache at DeepInfra, most of our models ran on V1 and we could not migrate due to performance and compatibility issues with V2. But I understand it makes sense to move to V2 if that is the long term plan.

/// Conservative default preserves prior behavior for policies that do not track it.
[[nodiscard]] virtual bool isEnqueued(BlockPtr const& block)
{
return true;
}

/// @brief Claim a free block. Called when the cache manager allocates or reuses a new block
virtual void claimBlock(BlockPtr block) = 0;
virtual void claimBlock(BlockPtr block, std::optional<executor::RetentionPriority> priority,
Expand Down Expand Up @@ -84,6 +92,7 @@ class LRUEvictionPolicy : public BaseEvictionPolicy
void releaseBlock(BlockPtr block, bool toFront) override;

SizeType32 getNumFreeBlocks(SizeType32 cacheLevel) override;
[[nodiscard]] bool isEnqueued(BlockPtr const& block) override;

void claimBlock(BlockPtr block) override;
void claimBlock(BlockPtr block, std::optional<executor::RetentionPriority> priority,
Expand Down
Loading
Loading