[None][feat] Enable disk cache config for KV cache v2 - #14845
Conversation
Expose disk cache size and path through the LLM API config and bridge them into the KV cache manager v2 disk tier. Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>
|
/bot run --disable-fail-fast |
📝 WalkthroughWalkthroughThis PR adds disk-backed KV-cache tier support to the KV cache system by extending ChangesDisk KV-cache tier support
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
tests/unittest/llmapi/test_llm_args.py (1)
345-353: ⚡ Quick winAdd coverage for invalid disk-cache paths.
tests/unittest/llmapi/test_llm_args.pyis still insufficient for the full validator contract described in this PR: this test only covers the “size set without path” branch, but not the “path does not exist / is not a directory” branch. Please add at least one negative case for a missing path and one for a regular file so the directory validation stays pinned down.🧪 Proposed test additions
def test_KvCacheConfig_disk_cache_validation(tmp_path): config = KvCacheConfig(disk_cache_size=2048, disk_cache_path=str(tmp_path)) assert config.disk_cache_size == 2048 assert config.disk_cache_path == str(tmp_path) with pytest.raises(ValidationError) as exc_info: KvCacheConfig(disk_cache_size=2048) assert "disk_cache_path" in str(exc_info.value) + + missing_dir = tmp_path / "missing" + with pytest.raises(ValidationError) as exc_info: + KvCacheConfig(disk_cache_size=2048, + disk_cache_path=str(missing_dir)) + assert "disk_cache_path" in str(exc_info.value) + + not_a_dir = tmp_path / "cache_file" + not_a_dir.write_text("x") + with pytest.raises(ValidationError) as exc_info: + KvCacheConfig(disk_cache_size=2048, + disk_cache_path=str(not_a_dir)) + assert "disk_cache_path" in str(exc_info.value)As per coding guidelines,
tests/**: Act as a QA engineer reviewing test changes and coverage for TensorRT-LLM.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/llmapi/test_llm_args.py` around lines 345 - 353, The test test_KvCacheConfig_disk_cache_validation only covers the "size set without path" case; add two additional negative test cases for KvCacheConfig: one where disk_cache_path points to a non-existent path (e.g., tmp_path / "does_not_exist") and one where disk_cache_path points to a regular file (create a file under tmp_path), and in each use pytest.raises(ValidationError) to assert the validator rejects the config and that "disk_cache_path" appears in the error message; update or add assertions within the same test function or as separate tests (referencing KvCacheConfig and test_KvCacheConfig_disk_cache_validation) to ensure both "missing path" and "path is not a directory" branches are covered.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tensorrt_llm/llmapi/llm_args.py`:
- Around line 2547-2556: Add a fast-fail validation in the Pydantic model that
defines disk_cache_size and disk_cache_path (the same model that has
use_kv_cache_manager_v2) so that if disk_cache_size is set to a positive value
or disk_cache_path is non-empty while use_kv_cache_manager_v2 is False, the
model raises a ValueError with a clear message. Implement this as a
`@root_validator` (or a validator that has access to multiple fields) in the same
class that declares disk_cache_size/disk_cache_path, referencing the symbols
disk_cache_size, disk_cache_path and use_kv_cache_manager_v2, and ensure the
same validation is applied for the other occurrence mentioned (the second block
around lines 2705-2716).
- Around line 2705-2716: The model_validator validate_disk_cache_config should
only enforce that disk_cache_path is set when disk_cache_size > 0 and must not
check filesystem state; remove the os.path.isdir(self.disk_cache_path) existence
check from validate_disk_cache_config (keep the ValueError for missing path) and
move the directory-existence and writability checks into the worker-side
disk-tier initialization in KVCacheManagerV2 (e.g., in its disk
provisioning/init method) so node-local filesystem validation happens where the
path is actually used.
---
Nitpick comments:
In `@tests/unittest/llmapi/test_llm_args.py`:
- Around line 345-353: The test test_KvCacheConfig_disk_cache_validation only
covers the "size set without path" case; add two additional negative test cases
for KvCacheConfig: one where disk_cache_path points to a non-existent path
(e.g., tmp_path / "does_not_exist") and one where disk_cache_path points to a
regular file (create a file under tmp_path), and in each use
pytest.raises(ValidationError) to assert the validator rejects the config and
that "disk_cache_path" appears in the error message; update or add assertions
within the same test function or as separate tests (referencing KvCacheConfig
and test_KvCacheConfig_disk_cache_validation) to ensure both "missing path" and
"path is not a directory" branches are covered.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: e6f47868-9bb1-4f1a-b821-2f49a448f5b8
📒 Files selected for processing (3)
tensorrt_llm/_torch/pyexecutor/resource_manager.pytensorrt_llm/llmapi/llm_args.pytests/unittest/llmapi/test_llm_args.py
|
PR_Github #51480 [ run ] triggered by Bot. Commit: |
|
PR_Github #51480 [ run ] completed with state
|
The disk cache feature added disk_cache_size and disk_cache_path to KvCacheConfig but did not update the KvCacheConfigV2 mock dataclasses in test files, causing AttributeError in all v2 test variants. Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>
|
/bot run --disable-fail-fast |
|
PR_Github #52253 [ run ] triggered by Bot. Commit: |
|
PR_Github #52253 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #52335 [ run ] triggered by Bot. Commit: |
|
PR_Github #52335 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #52679 [ run ] triggered by Bot. Commit: |
|
PR_Github #52679 [ run ] completed with state |
4 participants
Expose disk cache size and path through the LLM API config and bridge them into the KV cache manager v2 disk tier.
Summary by CodeRabbit
Release Notes
Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.