Draft: [TRTC-1934][feat] Initial SA recipe database. #9272

FrankD412 · 2025-11-18T17:59:55Z

This PR adds the basics of recipes dedicated for Semi-Analysis. These recipes are the starts of a exploring ways to allow TRT-LLM to configure itself instead of users having to tune the knobs to find decent performance.

@coderabbitai summary

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

For guidance on mapping tests to stage names, see docs/source/reference/ci-overview.md
and the scripts/test_to_stage_mapping.py helper.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

tensorrt_llm/configure/database/scenario_list.yaml

...atabase/deepseek-ai/DeepSeek-R1-0528/B200/deepseek_r1_0528_fp8_b200_trt_1k1k_tp8_conc16.yaml

anish-shanbhag · 2025-11-18T23:54:27Z

...atabase/deepseek-ai/DeepSeek-R1-0528/B200/deepseek_r1_0528_fp8_b200_trt_1k1k_tp8_conc16.yaml

+    enable_padding: true
+    max_batch_size: 256
+  enable_attention_dp: false
+  print_iter_log: true


Should we remove this since it's specific to logging?

Yeah -- can do.

anish-shanbhag · 2025-11-19T00:00:33Z

tensorrt_llm/configure/database.py

+DATABASE_LIST_PATH = Path(__file__).parent / "database" / "scenario_list.yaml"
+
+
+class RecipeRecord(BaseModel):


Instead of creating a separate schema for the recipe record, what if we make "partial" versions of the Constraints classes here which have all fields in those classes as optional (this can be easily done dynamically in Pydantic)

This would make it so any recipe can have an arbitrary subset of a set of Constraints upon which we filter

(this obviously can't be done until we get #9160 merged, but just wanted to see what folks think)

I'm ambivalent to the joining on my end. This was implemented primarily because #9160 isn't in yet, but the isolation does mean we're decoupled from the constraints classing even if those are shared. I can port to those if we want to wait on the other PR.

This would make it so any recipe can have an arbitrary subset of a set of Constraints upon which we filter

didn't quite catch that - you mean that the dataclass to hold the recipe attributes would benefit from inheriting from BaseConstraints? (I see BenchmarkConstraints inherits from too from there)

Would that let us use pydantic more effectively for mapping schemas? - if so that would make sense to me too - but curious to know what's the benefit here

@venkywonka As a concrete example, e.g. BenchmarkConstraints has these fields:

model gpu num_gpus isl osl concurrency

If we make the recipe attributes class something like PartialBenchmarkConstraints which is the exact same as BenchmarkConstraints but every field is Optional, then a hypothetical recipe could filter on a subset of these fields, like:

model: DSR1 concurrency: 200 # no gpu, isl, osl specified (this recipe applies to any gpu, isl, or osl)

Let me know if that makes sense

@FrankD412 sounds good - maybe let's try to merge either one of our PRs first, and then we can decide to either update the other PR or do a 3rd integration PR

@anish-shanbhag -- I'm fine if your MR gets in first and I think that would be beneficial to getting things in a working state.

didn't quite catch that - you mean that the dataclass to hold the recipe attributes would benefit from inheriting from BaseConstraints? (I see BenchmarkConstraints inherits from too from there)

I think the benefit of keeping them separate is kind of minimal, so I'm fine either way.

tensorrt_llm/configure/database.py

anish-shanbhag · 2025-11-19T00:19:55Z

tensorrt_llm/configure/database.py

@@ -0,0 +1,38 @@
+from pathlib import Path


Can we add a test to validate that all recipes in the DB are valid, e.g. what @venkywonka had in https://github.com/NVIDIA/TensorRT-LLM/pull/8990/files#diff-4bbcd72bf3493bac466a30030fb7124f2f22ab9952f62936ca04b67c3759c4f8R33

Good call -- I can add one yeah.

Signed-off-by: Frank Di Natale <[email protected]>

anish-shanbhag · 2025-11-19T21:01:19Z

tensorrt_llm/configure/database.py

+    """Recipe that describes a single scenario."""
+
+    constraints: RecipeConstraints = Field(description="Recipe constraints")
+    env_overrides: Dict[str, Any] = Field(description="Environment overrides", default_factory=dict)


I believe that in #9104 env_overrides is now a field within LlmArgs, so I don't think we need this as a separate field (it would just be part of config)

Yeah -- I saw that. Will modify this to align there. Thanks for keeping track 🙂

anish-shanbhag · 2025-11-19T21:02:44Z

tensorrt_llm/configure/database.py

+import yaml
+from pydantic import BaseModel, Field, RootModel
+
+DATABASE_LIST_PATH = Path(__file__).parent / "database" / "scenario_list.yaml"


Have we verified that the full recipe DB is included within the built TRTLLM wheel, meaning reading this path still works when we don't have the full TRTLLM repo stored locally?

fadi-sadi · 2025-11-20T15:31:06Z

...atabase/deepseek-ai/DeepSeek-R1-0528/B200/deepseek_r1_0528_fp8_b200_trt_1k1k_tp8_conc16.yaml

+constraints:
+  model: deepseek-ai/DeepSeek-R1-0528
+  gpu: B200_NVL
+  num_gpus: 8


I'm confused because based on the filename and the constraint this should have tp=8 but it isn't in the config. The configs should have tensor_parallel_size: 8.

Also since I already brought it up, maybe we should consider changing the filename format from ..._tp<num_gpus>... to ..._gpus<num_gpus>... since if the constraint says num_gpus=8 theoretically we could be using a different parallelism config other than tp=num_gpus

that's at least what I'm doing in !4

FrankD412 self-assigned this Nov 18, 2025

FrankD412 requested review from anish-shanbhag and venkywonka November 18, 2025 18:00

anish-shanbhag reviewed Nov 19, 2025

View reviewed changes

FrankD412 added 2 commits November 18, 2025 18:49

Initial SA recipe database.

b7665d3

Signed-off-by: Frank Di Natale <[email protected]>

Rename scenaro to constraints

d427dda

Signed-off-by: Frank Di Natale <[email protected]>

FrankD412 force-pushed the trtllm-configure/recipes branch from 3e33d70 to d427dda Compare November 19, 2025 02:49

FrankD412 added 4 commits November 18, 2025 19:59

Port over to RootModel

adc08b2

Signed-off-by: Frank Di Natale <[email protected]>

Update recipe classes.

b79365b

Signed-off-by: Frank Di Natale <[email protected]>

Update env to env_overrides.

414a1c0

Signed-off-by: Frank Di Natale <[email protected]>

Update to add config recipe + load.

f5203b5

Signed-off-by: Frank Di Natale <[email protected]>

venkywonka mentioned this pull request Nov 19, 2025

[TRTC-1943][feat] Env vars override support in LLM API #9104

Open

6 tasks

anish-shanbhag reviewed Nov 19, 2025

View reviewed changes

fadi-sadi reviewed Nov 20, 2025

View reviewed changes

		DATABASE_LIST_PATH = Path(__file__).parent / "database" / "scenario_list.yaml"


		class RecipeRecord(BaseModel):

Draft: [TRTC-1934][feat] Initial SA recipe database. #9272

Are you sure you want to change the base?

Draft: [TRTC-1934][feat] Initial SA recipe database. #9272

Uh oh!

Conversation

FrankD412 commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

GitHub Bot Help

kill

skip

reuse-pipeline

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

venkywonka Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anish-shanbhag Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fadi-sadi Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

FrankD412 commented Nov 18, 2025 •

edited

Loading

venkywonka Nov 20, 2025 •

edited

Loading

anish-shanbhag Nov 19, 2025 •

edited

Loading

fadi-sadi Nov 20, 2025 •

edited

Loading