[TRTLLM-14388][refactor] BREAKING: Force 2 model spec dec to fall back to 1 model - #17366
Conversation
|
/bot run --disable-fail-fast |
mikeiovine
added
the
api-breaking
mikeiovine
changed the title
f80563a to
1066a4f
Compare
|
PR_Github #64369 [ run ] triggered by Bot. Commit: |
|
PR_Github #64369 [ run ] completed with state
|
1066a4f to
e34db64
Compare
|
/bot run |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
WalkthroughEAGLE3 and MTP validation now enforces one-model operation and warns when disabled. AutoDeploy no longer selects two-model DraftTarget decoding. The shared-KV test no longer configures the removed option, and deprecated Eagle3 tests were deleted. ChangesSpeculative decoding configuration
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tensorrt_llm/llmapi/llm_args.py`:
- Around line 2104-2106: Regenerate the LLM-args golden manifest after the
EAGLE3 and MTP metadata or behavior changes by running
scripts/generate_llm_args_golden_manifest.py, then commit the resulting update
to llm_args_golden_manifest.json.
In `@tests/unittest/_torch/speculative/hw_agnostic/test_mtp.py`:
- Around line 1745-1754: Extend test_mtp_shared_kv_config to accept and pass
mtp_eagle_one_model=False, then assert the resulting configuration stores it as
True and spec_dec_mode.is_mtp_eagle_one_model() returns true. Preserve the
existing architecture parameterization and coverage for both architecture
values.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: f2525f3b-109b-4120-b803-c695bcbc826a
📒 Files selected for processing (3)
tensorrt_llm/llmapi/llm_args.pytests/unittest/_torch/speculative/hw_agnostic/test_mtp.pytests/unittest/_torch/speculative/test_eagle3.py
💤 Files with no reviewable changes (1)
- tests/unittest/_torch/speculative/test_eagle3.py
|
PR_Github #64690 [ run ] triggered by Bot. Commit: |
|
PR_Github #64690 [ run ] completed with state
|
Signed-off-by: Mike Iovine <miovine@nvidia.com>
e34db64 to
0393ec8
Compare
|
/bot run |
|
PR_Github #65055 [ run ] triggered by Bot. Commit: |
|
PR_Github #65055 [ run ] completed with state
|
|
/bot run |
|
PR_Github #65098 [ run ] triggered by Bot. Commit: |
|
PR_Github #65098 [ run ] completed with state |
|
/bot help |
GitHub Bot Help
Provide a user friendly way for developers to interact with a Jenkins server. Run See details below for each supported subcommand. Details
Launch build/test pipelines. All previously running jobs will be killed.
kill
Kill all running builds associated with pull request. skip
Skip testing for latest commit on pull request. reuse-pipeline
Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break. |
|
/bot run --only-multi-gpu-test |
|
PR_Github #65306 [ run ] triggered by Bot. Commit: |
|
PR_Github #65306 [ run ] completed with state
|
|
/bot run --only-multi-gpu-test |
|
PR_Github #65373 [ run ] triggered by Bot. Commit: |
|
PR_Github #65373 [ run ] completed with state |
|
/bot skip --comment "Single GPU and multi GPU passed on separate runs" |
|
PR_Github #65598 [ skip ] triggered by Bot. Commit: |
|
PR_Github #65598 [ skip ] completed with state |
…r exists) The waive entry pointed at unittest/_torch/modeling/test_modeling_gpt_oss.py::test_gpt_oss_two_model_eagle3_falls_back_to_v1, which was removed from main as obsolete in NVIDIA#17597 (commit 8398196). The 2-model Eagle3 fallback scenario it asserted was invalidated when NVIDIA#17366 forced 2-model spec-dec to fall back to 1-model, so the test was deleted rather than fixed. Waiving a nonexistent test fails the AST-based validate-test-lists pre-commit hook (check_test_list.py --validate) and gates the whole pipeline. NVBug 6606544 is moot now that the test is gone. Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>
…r exists) The waive entry pointed at unittest/_torch/modeling/test_modeling_gpt_oss.py::test_gpt_oss_two_model_eagle3_falls_back_to_v1, which was removed from main as obsolete in NVIDIA#17597 (commit 8398196). The 2-model Eagle3 fallback scenario it asserted was invalidated when NVIDIA#17366 forced 2-model spec-dec to fall back to 1-model, so the test was deleted rather than fixed. Waiving a nonexistent test fails the AST-based validate-test-lists pre-commit hook (check_test_list.py --validate) and gates the whole pipeline. NVBug 6606544 is moot now that the test is gone. Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>
3 participants
Description
First step of 2-model removal. Before removing the flags, make it so that all users are forced to use 1-model.
After this change, we can start to remove internal code piece by piece. Then, we can slowly delete tests and remove the API all together. Doing it step by step will avoid merge conflict difficulties.
Test Coverage
Removed some existing tests that were 2-model only. They no longer work on this change (usually because they relied on some misbehavior of 2-model). All this stuff has coverage elsewhere
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.Dev Engineer Review
False, the code warns, resets it toTrue, and continues with one-model behavior.DraftTargetDecodingConfiginto two-model mode.QA Engineer Review
test_mtp_shared_kv_config.one_modelparameter.mtp_eagle_one_modelconfiguration from test cases.test_deepseek_eagle3.test_eagle3_cuda_graph_padding.tests/integration/test_lists/,test-db/, orqa/.