[TRTLLM-9264][fix] Add accuracy/unit tests/doc for phi4mm

[Wanli-Jiang]

Features:

• Added MMMU accuracy for phi4mm image modality.
• Added unittests for phi4mm.
• Added doc for phi4mm.

Summary by CodeRabbit

Release Notes

• Documentation

  • Added comprehensive guide for running Phi-4-MultiModal model with TensorRT LLM, including offline inference, serving setup, and usage examples.

• New Features

  • Extended multimodal test infrastructure to support audio modality evaluation.

• Tests

  • Added accuracy tests for Phi-4-MultiModal model with benchmark results (53.67 on MMMU).
  • Updated test coverage for multimodal scenarios.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

For guidance on mapping tests to stage names, see docs/source/reference/ci-overview.md
and the scripts/test_to_stage_mapping.py helper.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

[coderabbitai]

📝 Walkthrough

Walkthrough

This pull request introduces support for the Phi-4-MultiModal (phi4-mm) model to TensorRT LLM, including documentation, accuracy testing infrastructure, and multimodal unit tests. The changes add audio modality support to the test framework, establish accuracy benchmarks, and consolidate multimodal test coverage while removing certain phi4 end-to-end test variants.

Changes

Cohort / File(s)	Summary
Documentation examples/models/core/phi/phi4-mm.md	New documentation detailing how to run Phi-4-MultiModal with TensorRT LLM, including offline batch inference, LoRA configuration, TRTLLM-serve setup, and usage examples.
Accuracy Testing Infrastructure tests/integration/defs/accuracy/references/mmmu.yaml tests/integration/defs/accuracy/test_llm_api_pytorch_multimodal.py	Added accuracy reference entry for microsoft/Phi-4-multimodal-instruct (53.67) and new test class TestPhi4MMFusedVisionLora with MMMU evaluation and sampling/kv-cache configuration.
End-to-End Test Modifications tests/integration/defs/test_e2e.py	Removed multiple phi4 multimodal test variants (phi4-multimodal-instruct, FP4, FP8 flavors) from parameterized test groups, simplified assertion blocks, and consolidated KV-cache and LoRA-related test logic.
Test List Updates tests/integration/test_lists/qa/llm_function_core.txt tests/integration/test_lists/qa/llm_function_nim.txt tests/integration/test_lists/test-db/l0_l40s.yml tests/integration/test_lists/test-db/l0_rtx_pro_6000.yml tests/integration/test_lists/waives.txt	Added Phi4MMFusedVisionLora test case, replaced multimodal Phi-4 test entries with Mistral/Gemma alternatives, added modeling_phi4mm test, removed 6000-series GPU test entries, and updated skip waiver entries.
Multimodal Test Framework tests/unittest/_torch/modeling/test_modeling_multimodal.py	Added audio modality support, introduced trust_remote_code and skip_hf_inference properties to base test classes, and added conditional logic to optionally skip HuggingFace inference during testing.
Phi4MM Unit Test Module tests/unittest/_torch/modeling/test_modeling_phi4mm.py	New comprehensive test module for Phi4MM with scenario dataclass, model configuration, weight mapping, HuggingFace integration controls, and multi-variant scenario generation (image/audio with CUDA graph, chunked prefill, KV cache reuse options).

Sequence Diagram(s)

sequenceDiagram
    participant Test as Test Class
    participant Setup as setUp()
    participant HFMock as HF Model (Mocked)
    participant TRTModel as TRT-LLM Model
    participant Scenario as Scenario Runner
    
    Test->>Setup: initialize
    alt skip_hf_inference = True
        Setup->>HFMock: create dummy (no weights)
    else skip_hf_inference = False
        Setup->>HFMock: load real HF model
    end
    
    Test->>Scenario: run_scenario_test
    
    alt Context Phase
        alt skip_hf_inference = True
            Scenario->>TRTModel: compute context
            Note over Scenario: skip HF ref computation
        else
            Scenario->>HFMock: get context ref
            Scenario->>TRTModel: compute context
            Scenario->>Scenario: compare outputs
        end
    end
    
    alt Generation Phase
        Scenario->>TRTModel: generate with past_key_values
        alt skip_hf_inference = False
            Scenario->>HFMock: get generation ref
            Scenario->>Scenario: validate against HF
        else
            Note over Scenario: skip HF validation
        end
    end

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

• End-to-end test removals: The extensive removal of phi4 multimodal test variants across multiple parameterized test groups in test_e2e.py requires careful validation to ensure consistency and that test coverage is appropriately consolidated rather than lost.
• Test framework extensions: The new skip_hf_inference and trust_remote_code properties in base multimodal test classes affect all downstream test implementations and require verification that the optional HF inference skipping logic is correctly integrated.
• New Phi4MM test module scope: The comprehensive test scaffolding with scenario generation, HF integration controls, and multi-variant modality support introduces substantial new testing infrastructure that needs validation for correctness and maintainability.
• Test list consolidation: Multiple test list files have been modified with removals and additions (Phi-4 variants replaced with Mistral/Gemma); verify that the intended test coverage shift is accurate and GPU-specific constraints are properly reflected.

Pre-merge checks and finishing touches

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 44.83% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.
Description check	❓ Inconclusive	PR description is minimal and vague; contains only feature bullets without explaining the issue, solution, or test coverage details.	Expand the Description section to explain what problem is being solved and how. Complete the Test Coverage section with specific test names and explain how they validate the changes.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly describes the main change: adding accuracy tests, unit tests, and documentation for the Phi-4 MultiModal model.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

Tip

📝 Customizable high-level summaries are now available in beta!

You can now customize how CodeRabbit generates the high-level summary in your pull requests — including its content, structure, tone, and formatting.

• Provide your own instructions using the high_level_summary_instructions setting.
• Format the summary however you like (bullet lists, tables, multi-section layouts, contributor stats, etc.).
• Use high_level_summary_in_walkthrough to move the summary from the description to the walkthrough section.

Example instruction:

"Divide the high-level summary into five sections:

1. 📝 Description — Summarize the main change in 50–60 words, explaining what was done.
2. 📓 References — List relevant issues, discussions, documentation, or related PRs.
3. 📦 Dependencies & Requirements — Mention any new/updated dependencies, environment variable changes, or configuration updates.
4. 📊 Contributor Summary — Include a Markdown table showing contributions:
   | Contributor | Lines Added | Lines Removed | Files Changed |
5. ✔️ Additional Notes — Add any extra reviewer context.
   Keep each section concise (under 200 words) and use bullet or numbered lists for clarity."

Note: This feature is currently in beta for Pro-tier users, and pricing will be announced later.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 0

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

tests/unittest/_torch/modeling/test_modeling_multimodal.py (1)

49-63: Align audio modality + HF skip behavior and avoid unnecessary HF work

A few small consistency points in the base multimodal harness:

1. Audio modality vs error messages / HF inputs

   • MultimodalScenario.__post_init__ and get_raw_test_inputs now accept "audio", but:
     • get_raw_test_inputs’s error message still lists only image, multiple_image, video, mixture_text_image, text.
     • get_hf_inputs has no "audio" branch and would raise for modality="audio" if skip_hf_inference were ever False.

   Consider:

   • Updating the error message to include "audio" so it stays in sync with valid_modalities, and
   • Either documenting that audio scenarios must override get_hf_inputs, or adding a minimal "audio" branch there when you actually need HF audio baselines.

2. Skip-HF mode still builds HF inputs

   In run_scenario_test (context + generation):

   hf_inputs = self.get_hf_inputs(scenario.modality, prompts, media)
   if self.skip_hf_inference:
       ref = None
   else:
       ref = self.hf_model.forward(...)
   ...
   past_key_values = ref.past_key_values if not self.skip_hf_inference else None
   ...
   if self.skip_hf_inference:
       ref = None
   else:
       ref = self.hf_model.forward(**gen_hf_inputs)

   When skip_hf_inference is True (e.g., Phi4MM), you still:

   • Instantiate HF processors and prepare hf_inputs, and
   • Build gen_hf_inputs, even though they’re never used.

   To make “skip HF inference” truly lightweight and avoid depending on HF processors for these models, you can gate the HF work:

•    # Compare context outputs
  

•    hf_inputs = self.get_hf_inputs(scenario.modality, prompts, media)
  

•    if self.skip_hf_inference:
  

•        ref = None
  

•    else:
  

•        ref = self.hf_model.forward(**hf_inputs, use_cache=True)
  

•    # Compare context outputs (optionally)
  

•    if self.skip_hf_inference:
  

•        ref = None
  

•    else:
  

•        hf_inputs = self.get_hf_inputs(scenario.modality, prompts, media)
  

•        ref = self.hf_model.forward(**hf_inputs, use_cache=True)
  

@@

•    gen_trtllm_inputs = self.get_trtllm_inputs(
  

•        trtllm_input_ids, multimodal_params_list, is_gen=True
  

•    )
  

•    past_key_values = ref.past_key_values if not self.skip_hf_inference else None
  

•    gen_trtllm_inputs = self.get_trtllm_inputs(
  

•        trtllm_input_ids, multimodal_params_list, is_gen=True
  

•    )
  

•    past_key_values = None
  

•    if not self.skip_hf_inference:
  

•        past_key_values = ref.past_key_values
  

@@

•        logits = self.run_trtllm_forward(gen_trtllm_inputs, scenario.use_cuda_graph)
  

•        if self.skip_hf_inference:
  

•            ref = None
  

•        else:
  

•            ref = self.hf_model.forward(**gen_hf_inputs)
  

•        logits = self.run_trtllm_forward(gen_trtllm_inputs, scenario.use_cuda_graph)
  

•        if not self.skip_hf_inference:
  

•            ref = self.hf_model.forward(**gen_hf_inputs)
  

  
  This keeps existing comparison behavior when HF is enabled, while fully decoupling HF in `skip_hf_inference` scenarios.
  
  
  
  
  
  

Also applies to: 125-160, 399-438, 572-588, 589-617, 660-673

🧹 Nitpick comments (3)

examples/models/core/phi/phi4-mm.md (1)

1-84: Tighten markdown style: fenced languages, bare URLs, and minor wording

Current doc is clear, but a few small tweaks will keep markdownlint happy and improve readability:

• Add languages to fenced code blocks (shell/YAML) at lines 12–18, 21–47, 49–79.
• Avoid bare URLs by wrapping them in markdown links.
• Slightly rephrase repetitive/awkward bullets.

For example:

-This document explains how to run Phi-4-Multimodal (phi4-mm) using TensorRT LLM and run on a single or multiple GPUs with pytorch backend. The original BF16 phi4-mm HF repository is https://huggingface.co/microsoft/Phi-4-multimodal-instruct, and we used ModelOpt to get the FP8 and NVFP4 versions.
+This document explains how to run Phi-4-Multimodal (phi4-mm) using TensorRT LLM on a single or multiple GPUs with the PyTorch backend. The original BF16 phi4-mm HF repository is [microsoft/Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct), and we used ModelOpt to get the FP8 and NVFP4 versions.
@@
-## Usage
-### Offline batch inference with LoRA support
-```
+## Usage
+### Offline batch inference with LoRA support
+```bash
 python examples/llm-api/quickstart_multimodal.py --model_dir <model_folder_path> --modality image --load_lora --auto_model_name Phi4MMForCausalLM
@@
-python examples/llm-api/quickstart_multimodal.py --model_dir <model_folder_path> --modality image_audio --load_lora --auto_model_name Phi4MMForCausalLM
-```
+python examples/llm-api/quickstart_multimodal.py --model_dir <model_folder_path> --modality image_audio --load_lora --auto_model_name Phi4MMForCausalLM
+```
@@
-### TRTLLM-serve with LoRA support
-```
+### TRTLLM-serve with LoRA support
+```bash
 cat > lora-extra-llm-api-config.yml<<EOF
@@
 EOF
@@
-trtllm-serve  \
+trtllm-serve  \
 <model_folder_path> \
@@
-```
+```
@@
-```
+```bash
 curl http://localhost:8000/v1/chat/completions \
@@
-  }' | jq
-```
+  }' | jq
+```
@@
-* About HF model downloading, please use `git clone [email protected]:microsoft/Phi-4-multimodal-instruct` (snapshot download will raise error when running the model) since we assumed some specific folder architectures for phi4-mm, see tensorrt_llm/_torch/models/modeling_phi4mm.py
+* For HF model downloading, use `git clone [email protected]:microsoft/Phi-4-multimodal-instruct` (snapshot download will raise an error when running the model) since we assume specific folder layouts for phi4-mm; see `tensorrt_llm/_torch/models/modeling_phi4mm.py`.
@@
-* About phi4-mm HF model, it is not compatible with the latest transformers even there are some codes related with phi4-mm. If you want to use the transformers codes, you can refer to https://huggingface.co/microsoft/Phi-4-multimodal-instruct/discussions/70.
+* The phi4-mm HF model is incompatible with the latest `transformers`, even though some related code exists there. For details, see the HF discussion: [microsoft/Phi-4-multimodal-instruct#70](https://huggingface.co/microsoft/Phi-4-multimodal-instruct/discussions/70).

tests/unittest/_torch/modeling/test_modeling_phi4mm.py (1)

16-211: Clarify Phi4MM config usage and HF-path expectations

A couple of structural points in this scaffold are worth tightening:

1. PHI4MM_CONFIG vs create_hf_config

   • PHI4MM_CONFIG carries test-specific overrides (e.g., num_blocks: 2, num_hidden_layers: 2 with comments # original: ...), but create_hf_config() currently ignores all of that except _name_or_path:

     hf_config = transformers.AutoConfig.from_pretrained(
         PHI4MM_CONFIG["_name_or_path"], trust_remote_code=self.trust_remote_code
     )
     self.Phi4MMConfig = type(hf_config)

   • As a result, the TRT-LLM model is built from whatever config.json lives under that directory (likely the full model), not the lightweight shape implied by PHI4MM_CONFIG.

   If the intent is to run a reduced-depth Phi4MM for unit tests, consider driving the HF config from PHI4MM_CONFIG instead, using the discovered config class:

• def create_hf_config(self):

•    hf_config = transformers.AutoConfig.from_pretrained(
  

•        PHI4MM_CONFIG["_name_or_path"], trust_remote_code=self.trust_remote_code
  

•    )
  

•    # Override the Phi4MMConfig class with the actual class from the config.
  

•    self.Phi4MMConfig = type(hf_config)
  

•    return hf_config
  

• def create_hf_config(self):

•    # Load remote config class, then instantiate from our test config dict.
  

•    base_config = transformers.AutoConfig.from_pretrained(
  

•        PHI4MM_CONFIG["_name_or_path"], trust_remote_code=self.trust_remote_code
  

•    )
  

•    self.Phi4MMConfig = type(base_config)
  

•    return self.Phi4MMConfig.from_dict(PHI4MM_CONFIG)
  

  
  That way, the comments (`# original: ...`) match what’s actually instantiated, and you avoid surprising memory usage.
  
  

2. HF inputs override and skip mode

   • skip_hf_inference is hard-coded to True, and the base harness now prints “(skipped HF inference)” instead of comparing outputs. Once the base run_scenario_test is updated to not call get_hf_inputs when skip_hf_inference is True (see previous file), the custom get_hf_inputs() here becomes effectively unused.
   • If you intend to eventually enable HF baselines for Phi4MM, it may be worth revisiting get_hf_inputs() (e.g., shape assumptions like ["image"][0]) at that time; for now, it’s fine to leave as-is or even drop it until HF support is ready.

3. Minor Ruff warning

   • In create_hf_model(self, pretrained_config), the argument is intentionally unused since HF inference is always skipped. If you want to silence linters while keeping the signature compatible with the base class, you can rename it to _pretrained_config.

Overall, the scaffold looks consistent with the base multimodal harness and the explicit “skip HF” decision; the main potential source of confusion is whether PHI4MM_CONFIG is meant to reflect the actual instantiated architecture.

Also applies to: 225-275, 296-313, 314-343

tests/integration/defs/test_e2e.py (1)

2590-2593: Phi4MM quickstart e2e test now only sanity-checks execution (no output validation)

The changes in this region do two things:

• Adjust the video skip conditions for existing multimodal tests (kv-cache reuse / chunked prefill) to skip certain model/modality combos earlier, which is fine and still guarded by pytest.skip.
• Simplify test_ptp_quickstart_multimodal_phi4mm so it only runs the quickstart script and prints "Sanity check passed!", without checking generated outputs or keyword match ratios.

Given the new dedicated MMMU accuracy harness and modeling/unit tests for Phi4MM, this lighter e2e check is reasonable if the goal is just “CLI flow doesn’t crash”. If you still want any minimal behavioral guard here, consider asserting that output is non-empty or contains at least one completion instead of only printing.

Also applies to: 2678-2680, 2751-2818

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 6e5384d and eda51c1.

📒 Files selected for processing (11)

• examples/models/core/phi/phi4-mm.md (1 hunks)
• tests/integration/defs/accuracy/references/mmmu.yaml (1 hunks)
• tests/integration/defs/accuracy/test_llm_api_pytorch_multimodal.py (1 hunks)
• tests/integration/defs/test_e2e.py (3 hunks)
• tests/integration/test_lists/qa/llm_function_core.txt (1 hunks)
• tests/integration/test_lists/qa/llm_function_nim.txt (0 hunks)
• tests/integration/test_lists/test-db/l0_l40s.yml (1 hunks)
• tests/integration/test_lists/test-db/l0_rtx_pro_6000.yml (0 hunks)
• tests/integration/test_lists/waives.txt (0 hunks)
• tests/unittest/_torch/modeling/test_modeling_multimodal.py (5 hunks)
• tests/unittest/_torch/modeling/test_modeling_phi4mm.py (1 hunks)

💤 Files with no reviewable changes (3)

• tests/integration/test_lists/test-db/l0_rtx_pro_6000.yml
• tests/integration/test_lists/waives.txt
• tests/integration/test_lists/qa/llm_function_nim.txt

🧰 Additional context used

🧠 Learnings (7)

📓 Common learnings

Learnt from: moraxu
Repo: NVIDIA/TensorRT-LLM PR: 6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.

📚 Learning: 2025-09-09T09:40:45.658Z

Learnt from: fredricz-20070104
Repo: NVIDIA/TensorRT-LLM PR: 7645
File: tests/integration/test_lists/qa/llm_function_core.txt:648-648
Timestamp: 2025-09-09T09:40:45.658Z
Learning: In TensorRT-LLM test lists, it's common and intentional for the same test to appear in multiple test list files when they serve different purposes (e.g., llm_function_core.txt for comprehensive core functionality testing and llm_function_core_sanity.txt for quick sanity checks). This duplication allows tests to be run in different testing contexts.

Applied to files:

• tests/integration/test_lists/test-db/l0_l40s.yml
• tests/integration/test_lists/qa/llm_function_core.txt
• tests/integration/defs/test_e2e.py
• tests/unittest/_torch/modeling/test_modeling_phi4mm.py

📚 Learning: 2025-09-17T02:48:52.732Z

Learnt from: tongyuantongyu
Repo: NVIDIA/TensorRT-LLM PR: 7781
File: tests/integration/test_lists/waives.txt:313-313
Timestamp: 2025-09-17T02:48:52.732Z
Learning: In TensorRT-LLM, `tests/integration/test_lists/waives.txt` is specifically for waiving/skipping tests, while other test list files like those in `test-db/` and `qa/` directories are for different test execution contexts (pre-merge, post-merge, QA tests). The same test appearing in both waives.txt and execution list files is intentional - the test is part of test suites but will be skipped due to the waiver.

Applied to files:

• tests/integration/test_lists/test-db/l0_l40s.yml
• tests/integration/test_lists/qa/llm_function_core.txt

📚 Learning: 2025-07-28T17:06:08.621Z

Learnt from: moraxu
Repo: NVIDIA/TensorRT-LLM PR: 6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.

Applied to files:

• tests/integration/test_lists/test-db/l0_l40s.yml
• tests/integration/test_lists/qa/llm_function_core.txt
• tests/integration/defs/accuracy/test_llm_api_pytorch_multimodal.py
• examples/models/core/phi/phi4-mm.md
• tests/integration/defs/test_e2e.py
• tests/unittest/_torch/modeling/test_modeling_phi4mm.py

📚 Learning: 2025-08-09T02:04:49.623Z

Learnt from: Fridah-nv
Repo: NVIDIA/TensorRT-LLM PR: 6760
File: tensorrt_llm/_torch/auto_deploy/models/quant_config_reader.py:81-98
Timestamp: 2025-08-09T02:04:49.623Z
Learning: In TensorRT-LLM's auto_deploy module, torch.dtype values in configuration dictionaries must be stored as string representations (e.g., "float16" instead of torch.float16) because OmegaConf.merge does not support torch.dtype types. These string representations are converted to actual torch.dtype objects in downstream code.

Applied to files:

• tests/integration/test_lists/qa/llm_function_core.txt

📚 Learning: 2025-08-26T09:49:04.956Z

Learnt from: pengbowang-nv
Repo: NVIDIA/TensorRT-LLM PR: 7192
File: tests/integration/test_lists/test-db/l0_dgx_b200.yml:56-72
Timestamp: 2025-08-26T09:49:04.956Z
Learning: In TensorRT-LLM test configuration files, the test scheduling system handles wildcard matching with special rules that prevent duplicate test execution even when the same tests appear in multiple yaml files with overlapping GPU wildcards (e.g., "*b200*" and "*gb200*").

Applied to files:

• tests/integration/test_lists/qa/llm_function_core.txt
• tests/integration/defs/test_e2e.py

📚 Learning: 2025-08-29T14:07:45.863Z

Learnt from: EmmaQiaoCh
Repo: NVIDIA/TensorRT-LLM PR: 7370
File: tests/unittest/trt/model_api/test_model_quantization.py:24-27
Timestamp: 2025-08-29T14:07:45.863Z
Learning: In TensorRT-LLM's CI infrastructure, pytest skip markers (pytest.mark.skip) are properly honored even when test files have __main__ blocks that call test functions directly. The testing system correctly skips tests without requiring modifications to the __main__ block execution pattern.

Applied to files:

• tests/integration/defs/test_e2e.py

🧬 Code graph analysis (3)

tests/integration/defs/accuracy/test_llm_api_pytorch_multimodal.py (2)

tests/integration/defs/accuracy/accuracy_core.py (1)

• LlmapiAccuracyTestHarness (949-960)

tensorrt_llm/evaluate/lm_eval.py (2)

• MMMU (663-716)
• evaluate (395-430)

tests/unittest/_torch/modeling/test_modeling_phi4mm.py (3)

tests/unittest/_torch/modeling/test_modeling_multimodal.py (9)

• MultimodalScenario (34-75)
• TestModelingMultimodal (78-710)
• get_model_config (80-81)
• get_trtllm_model_class (84-85)
• skip_hf_inference (109-114)
• get_weight_mapper_class (92-93)
• get_model_type (96-97)
• get_model_config_class (100-101)
• trust_remote_code (104-106)

tensorrt_llm/inputs/utils.py (1)

• default_multimodal_input_loader (606-769)

tensorrt_llm/llmapi/llm.py (1)

• prompt (86-87)

tests/unittest/_torch/modeling/test_modeling_multimodal.py (1)

tests/unittest/_torch/modeling/test_modeling_phi4mm.py (3)

• trust_remote_code (256-258)
• skip_hf_inference (261-266)
• create_hf_model (234-241)

🪛 LanguageTool

examples/models/core/phi/phi4-mm.md

[style] ~8-~8: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...LoRA inputs for different modalities. * We supported to set short/long RoPE, by se...

(ENGLISH_WORD_REPEAT_BEGINNING_RULE)

---

[style] ~84-~84: Consider using “incompatible” to avoid wordiness.
Context: ...4mm.py * About phi4-mm HF model, it is not compatible with the latest transformers even there...

(NOT_ABLE_PREMIUM)

🪛 markdownlint-cli2 (0.18.1)

examples/models/core/phi/phi4-mm.md

2-2: Bare URL used

(MD034, no-bare-urls)

---

12-12: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

21-21: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

49-49: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

84-84: Bare URL used

(MD034, no-bare-urls)

🪛 Ruff (0.14.5)

tests/unittest/_torch/modeling/test_modeling_phi4mm.py

234-234: Unused method argument: pretrained_config

(ARG002)

---

239-241: Avoid specifying long messages outside the exception class

(TRY003)

---

303-303: Avoid specifying long messages outside the exception class

(TRY003)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

• GitHub Check: Pre-commit Check

🔇 Additional comments (4)

tests/integration/test_lists/qa/llm_function_core.txt (1)

636-643: Phi4MM fused-vision LoRA accuracy test is correctly wired into core QA list

The added entry matches the new TestPhi4MMFusedVisionLora::test_auto_dtype class and follows existing accuracy test naming conventions; duplication across lists is expected for different test contexts.

tests/integration/test_lists/test-db/l0_l40s.yml (1)

16-31: New Phi4MM modeling unittest is correctly added to L0 L40s pre-merge set

The unittest/_torch/modeling -k "modeling_phi4mm" entry matches the new test module and aligns with the existing pattern for other multimodal modeling tests on L40s.

tests/integration/defs/accuracy/references/mmmu.yaml (1)

16-19: Phi-4-MM MMMU reference looks consistent; confirm value vs measured run

The new microsoft/Phi-4-multimodal-instruct entry follows the existing MMMU reference format and normalization (accuracy in 0–100). Just make sure 53.67 matches the actual MMMU run from the new TestPhi4MMFusedVisionLora harness before treating it as a golden reference.

tests/integration/defs/accuracy/test_llm_api_pytorch_multimodal.py (1)

197-216: Phi4MM fused-vision LoRA MMMU harness matches existing pattern; check stop token choice

The TestPhi4MMFusedVisionLora setup mirrors other MMMU tests (LLM + MMMU + shared SamplingParams/KvCacheConfig), and wiring into test lists/reference YAML looks coherent.

Two things to explicitly confirm:

• Using stop="<|USER|>" instead of <|endoftext|> matches Phi-4-MM’s chat template expectations for MMMU prompts.
• MAX_NUM_TOKENS = 25600 is compatible with your GPU budget on target CI machines when combined with MMMU.MAX_INPUT_LEN.

If those assumptions hold, the test looks good.

[Wanli-Jiang]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #25516 [ run ] triggered by Bot. Commit: eda51c1

[tensorrt-cicd]

PR_Github #25516 [ run ] completed with state SUCCESS. Commit: eda51c1
/LLM/main/L0_MergeRequest_PR pipeline #19322 completed with status: 'FAILURE'

[Wanli-Jiang]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #25633 [ run ] triggered by Bot. Commit: eda51c1

[tensorrt-cicd]

PR_Github #25633 [ run ] completed with state SUCCESS. Commit: eda51c1
/LLM/main/L0_MergeRequest_PR pipeline #19421 completed with status: 'FAILURE'

[Wanli-Jiang]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #25698 [ run ] triggered by Bot. Commit: 9d51d01

[tensorrt-cicd]

PR_Github #25698 [ run ] completed with state SUCCESS. Commit: 9d51d01
/LLM/main/L0_MergeRequest_PR pipeline #19480 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

[Wanli-Jiang]

/bot reuse-pipeline

[tensorrt-cicd]

PR_Github #25811 [ reuse-pipeline ] triggered by Bot. Commit: 59b2d3e

[tensorrt-cicd]

PR_Github #25811 [ reuse-pipeline ] completed with state SUCCESS. Commit: 59b2d3e
Reusing PR_Github #25698 for commit 59b2d3e