[None][feat] Enable KVCacheManagerV2 by default for Llama and Llama4

[erictsai-nv]

Dev Engineer Review

• LlamaForCausalLM and Llama4ForConditionalGeneration prefer KVCacheManagerV2.
• Explicit KV cache and transceiver settings retain precedence.
• Llama4 selects the Python transceiver for disaggregated NIXL serving and retains V2.
• Documentation lists Llama and Llama4 as V2-default models.
• No configuration or test-list files changed.
• No correctness or API consistency issues identified.

QA Engineer Review

• Modified tests/unittest/llmapi/test_llm_args.py.
• Added preference coverage for LlamaForCausalLM and Llama4ForConditionalGeneration.
• Tests verify V2 KV cache preference and Llama4 Python NIXL transceiver behavior.
• No integration test-list coverage was reported.
• Verdict: needs follow-up.

Description

Enables KVCacheManagerV2 by default for Llama and Llama4.

LlamaForCausalLM and Llama4ForConditionalGeneration declare get_preferred_kv_cache_manager_version() -> "V2", using the per-model preference hook introduced in #16060. LlamaForCausalLM already prefers the Python transceiver through #16787; this change adds the same preference to Llama4ForConditionalGeneration, so disaggregated serving over NIXL retains V2 instead of falling back to V1. The preferences are adopted only when the user leaves kv_cache_config.use_kv_cache_manager_v2 and cache_transceiver_config.transceiver_runtime at "auto"; explicit user settings continue to win.

The KV cache documentation now lists Llama and Llama4 among the model families that select V2 automatically.

Validation

Pre-PR GPU validation covered the existing Llama integration matrix across H100, H200, B200, B300, GB200, GB300, A10, A100, and L40S. Llama3 accuracy, end-to-end, speculative-decoding, disaggregated-serving, and performance cases passed across V1 and V2 arms without a V2-attributable correctness or performance regression. Llama4 V2 validation on B200 passed quickstart, TP2/PP2 multinode, 8-GPU chunked-prefill, and one-model Eagle3 cases.

Test Coverage

• tests/unittest/llmapi/test_llm_args.py::TestKvCacheManagerV2AutoResolution::test_registered_models_prefer_v2 now covers LlamaForCausalLM and Llama4ForConditionalGeneration.
• tests/unittest/llmapi/test_llm_args.py::TestKvCacheManagerV2AutoResolution::test_registered_models_keep_v2_on_nixl now resolves both Llama entry points in production order and verifies that the fully-"auto" NIXL path retains V2 with the Python transceiver.
• Existing mock-based tests in TestKvCacheManagerV2AutoResolution continue to cover explicit user overrides and automatic V1 demotion for incompatible routes.

PR Checklist

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d5b38f84-0e6d-472c-89a0-c620de108315

📥 Commits

Reviewing files that changed from the base of the PR and between 808627a and e870239.

📒 Files selected for processing (2)

• docs/source/features/kvcache.md
• tensorrt_llm/_torch/models/modeling_llama.py

🚧 Files skipped from review as they are similar to previous changes (1)

• docs/source/features/kvcache.md

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

Llama and Llama4 now prefer KV cache manager V2. Llama4 also explicitly prefers the Python transceiver runtime for disaggregated NIXL serving. Documentation and registered-model tests cover these preferences.

Changes

Llama runtime preference integration

Layer / File(s)	Summary
Model runtime preferences tensorrt_llm/_torch/models/modeling_llama.py	LlamaForCausalLM and Llama4ForConditionalGeneration prefer KV cache manager V2. Llama4 prefers the Python transceiver runtime.
Preference documentation and tests docs/source/features/kvcache.md, tests/unittest/llmapi/test_llm_args.py	The documentation and registered-model tests include the Llama and Llama4 preferences.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to e8702

Llama and Llama4 will select KVCacheManagerV2 automatically while explicit configuration remains honored. No actionable merge-blocking risk remains beyond normal checks and review.

Suggested reviewers: bowenfu, brnguyen2

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 57.14% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 2 files. (1 skipped: 1…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title follows the required [None][feat] format and clearly states that KVCacheManagerV2 is enabled by default for Llama and Llama4.
Description check	✅ Passed	The description explains the change, user-setting precedence, documentation updates, validation results, and relevant test coverage. The PR checklist is summarized by the final checked item, so the de…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Description check

Explanation

The description explains the change, user-setting precedence, documentation updates, validation results, and relevant test coverage. The PR checklist is summarized by the final checked item, so the description is sufficiently complete.

Full details: Docstring Coverage

Explanation

Docstring coverage is 57.14% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 2 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[brnguyen2]

Approving — the comments below are optional touch-ups, not blockers.

Two things worth resolving before this lands.

The transceiver preference on Llama4ForConditionalGeneration isn't conditional on V2 being chosen. _resolve_transceiver_runtime_auto runs first and only looks at the backend and the model hook, so a user who sets kv_cache_config.use_kv_cache_manager_v2=False (or hits one of the automatic V1 demotions, e.g. two-model spec dec) on NIXL disagg still gets the Python transceiver with a V1 manager. That combination is permitted by create_kv_cache_transceiver, but the validation you describe covers V2 disagg. Did any Llama4 disagg run exercise V1 + Python transceiver? If not, it'd be good to say so explicitly, since it's the new default for that config.

Second, this is a default flip for the most widely used architecture in the repo (every checkpoint declaring architectures: ["LlamaForCausalLM"], not just Llama 3/4), and the title carries [None]. A JIRA would make it easier to trace and revert if a V2-attributable regression shows up post-merge.

Test and doc coverage otherwise matches the pattern used by the earlier Gemma/GPT-OSS entries.

[erictsai-nv]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #70288 [ run ] triggered by Bot. Commit: e870239 Link to invocation

[tensorrt-cicd]

PR_Github #70288 [ run ] completed with state SUCCESS. Commit: e870239
/LLM/main/L0_MergeRequest_PR pipeline #57531 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation