[https://nvbugs/5470769][fix] fix disagg-serving accuracy test case

[reasonsolo]

Summary by CodeRabbit

• Tests

  • Standardized disaggregated serving test by explicitly setting instance counts, improving consistency and coverage of GPU budgeting scenarios.

• Chores

  • Updated accuracy reference data for Qwen3/Qwen3-30B-A3B by adding an explicit accuracy-only entry alongside the existing configuration, improving traceability of reported metrics.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

For guidance on mapping tests to stage names, see docs/source/reference/ci-overview.md
and the scripts/test_to_stage_mapping.py helper.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds an extra accuracy-only entry for Qwen3/Qwen3-30B-A3B in the MMLU reference YAML and updates a test to pass explicit ctx_instances=1 and gen_instances=1 to run_parallel_test. No public APIs changed.

Changes

Cohort / File(s)	Summary of changes
Accuracy references (MMLU) tests/integration/defs/accuracy/references/mmlu.yaml	Inserted a new list item under Qwen3/Qwen3-30B-A3B containing only accuracy: 79.53, preceding the existing FP8_BLOCK_SCALES entry with the same accuracy.
Disaggregated serving tests tests/integration/defs/accuracy/test_disaggregated_serving.py	test_mixed_ctx_gen_model now calls run_parallel_test(..., ctx_instances=1, gen_instances=1, ...); minor formatting adjustment.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

• [None][feat] adding support for disaggregated multi-instance tests #6674 — Similar update to pass ctx_instances/gen_instances to run_parallel_test after signature change.
• [TRTLLM-7014][chore] Add accuracy test for ctx and gen workers with different models #6741 — Related edits in accuracy tests and disaggregated serving utilities for separate ctx/gen models.
• [TRTLLM-6975][test] Add multi-turn test cases for VLM models #6749 — Adjusts MMLU accuracy reference entries for Qwen3 models, overlapping file changes.

Suggested labels

Community want to contribute

Suggested reviewers

• Shixiaowei02
• LarryXFly
• yilin-void
• Tabrizian
• pcastonguay

Tip

🔌 Remote MCP (Model Context Protocol) integration is now available!

Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats.

---

📜 Recent review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between ae89163 and 8aaf541.

📒 Files selected for processing (2)

• tests/integration/defs/accuracy/references/mmlu.yaml (1 hunks)
• tests/integration/defs/accuracy/test_disaggregated_serving.py (1 hunks)

🧰 Additional context used

📓 Path-based instructions (2)

**/*.py

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

**/*.py: Code must target Python 3.8+
Indent with 4 spaces; do not use tabs
Preserve module namespaces in imports: import the subpackage/module, not the symbol (from package.subpackage import foo; foo.SomeClass())
Naming: files snake_case; classes PascalCase; functions/methods snake_case; local variables snake_case (k_ prefix if starting with a number); globals G_ + UPPER_SNAKE_CASE; constants UPPER_SNAKE_CASE
Avoid shadowing outer-scope variables; initialize all externally visible members in init
Prefer docstrings for interfaces used outside a file; reserve comments for function-internal or file-local interfaces
Use Google-style docstrings for classes and functions; inline docstrings for attributes/variables are allowed
Avoid reflection when straightforward code suffices (e.g., prefer explicit parameters over dict(**locals()))
Use narrow except clauses (e.g., catch FileNotFoundError instead of bare except)
For duck-typing try/except, keep try body minimal and use else for the main logic

Files:

• tests/integration/defs/accuracy/test_disaggregated_serving.py

**/*.{cpp,cc,cxx,cu,h,hpp,hh,hxx,cuh,py}

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

Prepend NVIDIA copyright header with current year to all source files

Files:

• tests/integration/defs/accuracy/test_disaggregated_serving.py

🧠 Learnings (1)

📓 Common learnings

Learnt from: venkywonka
PR: NVIDIA/TensorRT-LLM#6029
File: .github/pull_request_template.md:45-53
Timestamp: 2025-08-27T17:50:13.243Z
Learning: For PR templates in TensorRT-LLM, avoid suggesting changes that would increase developer overhead, such as converting plain bullets to mandatory checkboxes. The team prefers guidance-style bullets that don't require explicit interaction to reduce friction in the PR creation process.

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

• GitHub Check: Pre-commit Check

🔇 Additional comments (2)

tests/integration/defs/accuracy/references/mmlu.yaml (1)

186-188: LGTM: Added baseline accuracy-only entry for Qwen3/Qwen3-30B-A3B.

Consistent with other models having a plain baseline alongside quantized variants; no YAML structure issues spotted.

tests/integration/defs/accuracy/test_disaggregated_serving.py (1)

910-913: LGTM: Explicit ctx/gen instance counts fix positional-arg pitfall.

Passing ctx_instances=1 and gen_instances=1 removes ambiguity in run_parallel_test and ensures correct GPU budgeting.

✨ Finishing Touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbit in a new review comment at the desired location with your query.
• PR comments: Tag @coderabbit in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbit gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbit read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbit help to get the list of available commands.

Other keywords and placeholders

• Add @coderabbit ignore or @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbit summary or @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbit or @coderabbitai title anywhere in the PR title to generate the title automatically.

Status, Documentation and Community

• Visit our Status Page to check the current availability of CodeRabbit.
• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[reasonsolo]

/bot run

[reasonsolo]

/bot run

[tensorrt-cicd]

PR_Github #17123 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #17123 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #12877 completed with status: 'FAILURE'

[reasonsolo]

/bot run

[tensorrt-cicd]

PR_Github #17185 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #17185 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #12919 completed with status: 'FAILURE'

[reasonsolo]

/bot run

[tensorrt-cicd]

PR_Github #17319 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #17319 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #13017 completed with status: 'FAILURE'

[reasonsolo]

/bot run

[tensorrt-cicd]

PR_Github #17340 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #17340 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #13033 completed with status: 'SUCCESS'