[None][chore] add spec_decoding configs in perf benchmark scripts and fix typos

[lancelly]

This PR does the following things:

1. Add spec_decoding configs
2. Fix setting of max_batch_size. Now max_batch_size could be set corretly in extra-llm-api-config.yaml

3. Fix setting of model_path for client command(It's empty before the fix).

Summary by CodeRabbit

• New Features
  • Added support for speculative decoding configuration options including decoding type, maximum draft length, speculative model directory, and model settings.
  • Configuration can now be read from YAML files and automatically included in generated reproduction scripts.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[lancelly]

/bot run

[coderabbitai]

📝 Walkthrough

Walkthrough

Extends ServerConfig to support speculative decoding configuration by adding four new parameters (decoding_type, max_draft_len, speculative_model_dir, eagle3_one_model). Updates YAML parsing to populate these fields and modifies config generation to conditionally emit speculative_config blocks when decoding_type is specified.

Changes

Cohort / File(s)

Change Summary

Speculative Config Support tests/scripts/perf-sanity/run_benchmark_serve.py

Added SPECULATIVE_CONFIG_METRICS constant mapping; extended ServerConfig.__init__ with speculative-related parameters (decoding_type, max_draft_len, speculative_model_dir, eagle3_one_model) with sensible defaults; updated generate_extra_llm_api_config to emit max_batch_size and conditional speculative_config block; extended parse_config_file to extract speculative settings from YAML and populate ServerConfig instances.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

• Review new SPECULATIVE_CONFIG_METRICS constant structure and field definitions
• Verify constructor parameter defaults and instance variable assignments are consistent
• Check conditional logic in generate_extra_llm_api_config for correct YAML emission (especially max_draft_len and eagle3_one_model conditional handling)
• Validate YAML parsing logic in parse_config_file correctly handles missing/optional speculative_config sections
• Ensure reproduce-script generation properly reflects new speculative settings

Pre-merge checks and finishing touches

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.
Description check	❓ Inconclusive	The PR description outlines the main changes (spec_decoding configs, max_batch_size fix, model_path fix) but lacks details on test coverage and doesn't fully follow the template structure.	Add a Test Coverage section explaining what tests validate these changes, and ensure description clearly explains the why and impact of each fix.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly indicates the PR adds spec_decoding configs to benchmark scripts and fixes issues, matching the core changes in the raw summary.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

Tip

📝 Customizable high-level summaries are now available in beta!

You can now customize how CodeRabbit generates the high-level summary in your pull requests — including its content, structure, tone, and formatting.

• Provide your own instructions using the high_level_summary_instructions setting.
• Format the summary however you like (bullet lists, tables, multi-section layouts, contributor stats, etc.).
• Use high_level_summary_in_walkthrough to move the summary from the description to the walkthrough section.

Example instruction:

"Divide the high-level summary into five sections:

1. 📝 Description — Summarize the main change in 50–60 words, explaining what was done.
2. 📓 References — List relevant issues, discussions, documentation, or related PRs.
3. 📦 Dependencies & Requirements — Mention any new/updated dependencies, environment variable changes, or configuration updates.
4. 📊 Contributor Summary — Include a Markdown table showing contributions:
   | Contributor | Lines Added | Lines Removed | Files Changed |
5. ✔️ Additional Notes — Add any extra reviewer context.
   Keep each section concise (under 200 words) and use bullet or numbered lists for clarity."

Note: This feature is currently in beta for Pro-tier users, and pricing will be announced later.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (2)

tests/scripts/perf-sanity/run_benchmark_serve.py (2)

218-224: Clarify intended use of SPECULATIVE_CONFIG_METRICS or remove if unused

SPECULATIVE_CONFIG_METRICS is defined but not referenced anywhere in this module; parsing of speculative_config is done directly in parse_config_file. If there is no external consumer importing this constant, consider either:

• Removing it, or
• Refactoring parsing to use this metadata to avoid future drift when fields change.

---

1-4: Add NVIDIA copyright header per TensorRT‑LLM guidelines

This Python file lacks the NVIDIA copyright header required for *.py/*.cpp/*.cu/*.h TensorRT‑LLM OSS files. Please prepend the standard NVIDIA copyright/licensing header (with the current year) above the shebang to align with the project guidelines.

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c87e81c and 9055fb5.

📒 Files selected for processing (1)

• tests/scripts/perf-sanity/run_benchmark_serve.py (6 hunks)

🧰 Additional context used

📓 Path-based instructions (2)

**/*.py

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

**/*.py: The code developed for TensorRT-LLM should conform to Python 3.8+
Indent Python code with 4 spaces; do not use tabs
Always maintain the namespace when importing in Python, even if only one class or function from a module is used (e.g., use from package.subpackage import foo and then foo.SomeClass() instead of from package.subpackage.foo import SomeClass)
Python filenames should use snake_case (e.g., some_file.py)
Python class names should use PascalCase (e.g., class SomeClass)
Python function and method names should use snake_case (e.g., def my_awesome_function():)
Python local variable names should use snake_case, with prefix k for variable names that start with a number (e.g., k_99th_percentile = ...)
Python global variables should use upper snake_case with prefix G (e.g., G_MY_GLOBAL = ...)
Python constants should use upper snake_case (e.g., MY_CONSTANT = ...)
Avoid shadowing variables declared in an outer scope in Python
Initialize all externally visible members of a Python class in the constructor
For Python interfaces that may be used outside a file, prefer docstrings over comments
Python comments should be reserved for code within a function, or interfaces that are local to a file
Use Google style docstrings for Python classes and functions, which can be parsed by Sphinx
Python attributes and variables can be documented inline with type and description (e.g., self.x = 5 followed by """<type>: Description of 'x'""" )
Avoid using reflection in Python when functionality can be easily achieved without reflection
When using try-except blocks in Python, limit the except clause to the smallest set of specific errors possible instead of catching all exceptions
When using try-except blocks in Python to handle multiple possible variable types (duck-typing), keep the body of the try as small as possible and use the else block to implement the logic

Files:

• tests/scripts/perf-sanity/run_benchmark_serve.py

**/*.{cpp,h,cu,py}

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

All TensorRT-LLM Open Source Software code files should contain an NVIDIA copyright header that includes the current year at the top

Files:

• tests/scripts/perf-sanity/run_benchmark_serve.py

🧬 Code graph analysis (1)

tests/scripts/perf-sanity/run_benchmark_serve.py (2)

tensorrt_llm/llmapi/llm_args.py (4)

• speculative_model_dir (1944-1945)
• model_dir (1670-1672)
• model_dir (1675-1679)
• model_name (1682-1683)

tests/integration/defs/perf/test_perf.py (3)

• get_model_dir (213-220)
• to_cmd (554-564)
• to_cmd (656-670)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)

• GitHub Check: Check PR Checklist Resolution
• GitHub Check: Pre-commit Check

🔇 Additional comments (3)

tests/scripts/perf-sanity/run_benchmark_serve.py (3)

261-264: Speculative decoding config is plumbed coherently end‑to‑end

The new speculative fields (decoding_type, max_draft_len, speculative_model_dir, eagle3_one_model) are consistently:

• Accepted in ServerConfig.__init__,
• Stored on the instance,
• Emitted into speculative_config only when decoding_type is set, and
• Populated from server_config_data['speculative_config'] in parse_config_file with safe defaults.

The gating of the whole block on decoding_type is a reasonable guard against partially‑specified speculative configs, and defaults line up across constructor and parsing. This looks correct and aligns with the PR objective of surfacing speculative decoding options in the generated configs.

Also applies to: 287-290, 343-355, 499-507

---

295-303: ServerConfig.to_cmd correctly reuses resolved model_path

Using self.model_path (resolved once in __init__ via get_model_dir with an os.path.exists fallback) to build the trtllm-serve command ensures:

• The server command and the generated repro script both use the same effective model path.
• model_path is no longer empty when commands are constructed.

This addresses the “empty model_path for client/server commands” issue cleanly.

---

312-312: Top‑level max_batch_size emission matches YAML and cuda_graph_config usage

Adding max_batch_size at the top level of extra-llm-api-config.yml (while retaining cuda_graph_config.max_batch_size) ensures that the YAML value propagated via parse_config_file is visible to components that read the top‑level key, matching the PR description about fixing max_batch_size handling.

[tensorrt-cicd]

PR_Github #26107 [ run ] triggered by Bot. Commit: 9055fb5

[tensorrt-cicd]

PR_Github #26107 [ run ] completed with state SUCCESS. Commit: 9055fb5
/LLM/main/L0_MergeRequest_PR pipeline #19824 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.