[https://nvbugs/6187185][fix] Apply the existing `low_memory_overrides()` helper in `TestNemotronV2.test_auto_ by tensorrt-cicd · Pull Request #14584 · NVIDIA/TensorRT-LLM

tensorrt-cicd · 2026-05-26T15:51:28Z

Summary

Root cause: On L40S 48GB, the YAML's max_batch_size=128 (Mamba state O(max_batch_size) ≈ 8.8GB) + 16.5GB BF16 weights + forward-pass peak leaves negative budget for the attention KV cache.
Fix: Apply the existing low_memory_overrides() helper in TestNemotronV2.test_auto_dtype when get_device_memory() < 80000 MiB, mirroring the precedent in TestNemotronNanoV3.test_accuracy; preserve max_num_tokens so the chunked-prefill variant keeps its 512-token override.
Automated fix generated by repair-bot

Test plan

Verify fix on the same GPU type as the original failure
Check for regressions in related tests

Links

Bug: https://nvbugs/6187185

Summary by CodeRabbit

Tests
- Enhanced GPU memory handling in auto-deployment tests to detect and manage configurations on devices with limited VRAM, preventing memory-related deployment failures on smaller GPUs.

…st_auto_dtype on <80GB GPUs The YAML config (nemotron-nano-9b-v2.yaml) sets max_batch_size=128 and free_gpu_memory_fraction=0.7 — values tuned for H100/B200 80GB+. On L40S 48GB, the Mamba state cache scales O(max_batch_size) and consumes ~8.8GB, leaving negative budget for the attention KV cache and tripping the 'Impossible to fit in any sequence in kvCache' assertion in get_max_atten_window_upper_bound(). Mirror the precedent in TestNemotronNanoV3.test_accuracy: when get_device_memory() < 80000 MiB, apply low_memory_overrides() to lower max_batch_size, free_gpu_memory_fraction, and align cuda_graph batch sizes. Preserve max_num_tokens (= 512 when chunked prefill is on). Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>

coderabbitai · 2026-05-26T15:55:49Z

📝 Walkthrough

Walkthrough

The test TestNemotronV2.test_auto_dtype now detects CUDA device memory and applies low-memory configuration overrides when available GPU memory is below 80GB, while preserving the existing chunked-prefill token budget.

Changes

Nemotron V2 Memory Configuration

Layer / File(s)	Summary
Low-memory configuration override `tests/integration/defs/accuracy/test_llm_api_autodeploy.py`	Device-memory check added to `test_auto_dtype` that conditionally applies `low_memory_overrides(...)` when CUDA memory is below 80GB, explicitly preserving the `max_num_tokens` setting from the existing config to avoid KV-cache oversubscription on smaller GPUs.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check	❓ Inconclusive	The title is incomplete and appears truncated, ending with 'test_auto_' without completing the method name mentioned in the PR objectives.	Complete the title to properly describe the fix, e.g., '[https://nvbugs/6187185][fix] Apply low_memory_overrides() in TestNemotronV2.test_auto_dtype'

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description check	✅ Passed	The description includes summary, test plan, and links sections but is missing several required checklist items from the template and lacks clear explanation of testing coverage.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

🧹 Nitpick comments (1)

tests/integration/defs/accuracy/test_llm_api_autodeploy.py (1)
503-503: QA list update is not needed for this change.

This modifies runtime config behavior only; test node id/name/paramization did not change, so tests/integration/test_lists/qa/llm_function_core.txt updates are unnecessary.

As per coding guidelines, “If a PR changes a test function/class name or its parameterization, update this node id list accordingly…”.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/defs/accuracy/test_llm_api_autodeploy.py` at line 503, The
change only alters runtime config and did not rename or re-parameterize the test
function test_auto_dtype (with its fixture enable_chunked_prefill), so revert
any edits made to the QA node id list file
tests/integration/test_lists/qa/llm_function_core.txt; ensure the node id list
remains unchanged and only runtime-related test logic in test_auto_dtype is
kept, without touching the QA list entries.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/integration/defs/accuracy/test_llm_api_autodeploy.py`:
- Line 503: The change only alters runtime config and did not rename or
re-parameterize the test function test_auto_dtype (with its fixture
enable_chunked_prefill), so revert any edits made to the QA node id list file
tests/integration/test_lists/qa/llm_function_core.txt; ensure the node id list
remains unchanged and only runtime-related test logic in test_auto_dtype is
kept, without touching the QA list entries.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8e46dd9e-6d0b-4e48-a3c8-2221fdfc3aac

📥 Commits

Reviewing files that changed from the base of the PR and between 1f8312d and 9fe2421.

📒 Files selected for processing (1)

tests/integration/defs/accuracy/test_llm_api_autodeploy.py

suyoggupta · 2026-05-26T16:17:01Z

/bot run

tensorrt-cicd · 2026-05-26T16:23:47Z

PR_Github #50364 [ run ] triggered by Bot. Commit: 9fe2421 Link to invocation

tensorrt-cicd · 2026-05-26T21:11:58Z

PR_Github #50364 [ run ] completed with state SUCCESS. Commit: 9fe2421
/LLM/main/L0_MergeRequest_PR pipeline #39890 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

jieli-matrix

plz use decorator in conftest instead of adding if-else branch inside the test function; skip test decorator can help to skip the test case before running it, which means better execution efficiency & clear code structure.

Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com>

nvchenghaoz · 2026-05-27T21:50:04Z

@jieli-matrix thanks for the review! Updated the PR. Could you please take a look?

nvchenghaoz · 2026-05-28T15:10:39Z

/bot run

tensorrt-cicd · 2026-05-28T15:16:33Z

PR_Github #50816 [ run ] triggered by Bot. Commit: 730eb23 Link to invocation

tensorrt-cicd · 2026-05-28T15:40:22Z

PR_Github #50816 [ run ] completed with state FAILURE. Commit: 730eb23
/LLM/main/L0_MergeRequest_PR pipeline #40286 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

nvchenghaoz · 2026-05-28T15:51:50Z

/bot run

tensorrt-cicd · 2026-05-28T15:57:31Z

PR_Github #50822 [ run ] triggered by Bot. Commit: 730eb23 Link to invocation

tensorrt-cicd · 2026-05-28T16:13:49Z

PR_Github #50822 [ run ] completed with state FAILURE. Commit: 730eb23
/LLM/main/L0_MergeRequest_PR pipeline #40292 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

nvchenghaoz · 2026-05-28T17:31:14Z

/bot run

tensorrt-cicd · 2026-05-28T17:37:35Z

PR_Github #50846 [ run ] triggered by Bot. Commit: 730eb23 Link to invocation

tensorrt-cicd · 2026-05-28T18:49:28Z

PR_Github #50846 [ run ] completed with state SUCCESS. Commit: 730eb23
/LLM/main/L0_MergeRequest_PR pipeline #40316 completed with status: 'SUCCESS'

CI Report

Link to invocation

tensorrt-cicd requested review from a team as code owners May 26, 2026 15:51

tensorrt-cicd requested a review from Fridah-nv May 26, 2026 15:51

tensorrt-cicd assigned suyoggupta May 26, 2026

github-actions Bot assigned tensorrt-cicd May 26, 2026

coderabbitai Bot reviewed May 26, 2026

View reviewed changes

suyoggupta approved these changes May 26, 2026

View reviewed changes

jieli-matrix requested changes May 27, 2026

View reviewed changes

Update test_llm_api_autodeploy.py

189f48e

Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com>

jieli-matrix approved these changes May 28, 2026

View reviewed changes

Merge branch 'main' into repair-bot-bug6187185

730eb23

nvchenghaoz enabled auto-merge (squash) May 28, 2026 18:42

nvchenghaoz merged commit 56df200 into NVIDIA:main May 28, 2026
8 checks passed

nvchenghaoz deleted the repair-bot-bug6187185 branch May 28, 2026 19:03

Uh oh!

Conversation

tensorrt-cicd commented May 26, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Links

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented May 26, 2026

Walkthrough

Changes

Estimated code review effort

❌ Failed checks (1 warning, 1 inconclusive)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

suyoggupta commented May 26, 2026

Uh oh!

tensorrt-cicd commented May 26, 2026

Uh oh!

tensorrt-cicd commented May 26, 2026

Uh oh!

jieli-matrix left a comment

Choose a reason for hiding this comment

Uh oh!

nvchenghaoz commented May 27, 2026

Uh oh!

nvchenghaoz commented May 28, 2026

Uh oh!

tensorrt-cicd commented May 28, 2026

Uh oh!

tensorrt-cicd commented May 28, 2026

Uh oh!

nvchenghaoz commented May 28, 2026

Uh oh!

tensorrt-cicd commented May 28, 2026

Uh oh!

tensorrt-cicd commented May 28, 2026

Uh oh!

nvchenghaoz commented May 28, 2026

Uh oh!

tensorrt-cicd commented May 28, 2026

Uh oh!

tensorrt-cicd commented May 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tensorrt-cicd commented May 26, 2026 •

edited by coderabbitai Bot

Loading