Skip to content

[TRTLLM-15264][test] Wire KDA disagg transfer tests into CI (cpu_only CPU stage + l0_b200 GPU) - #17484

Merged
brnguyen2 merged 2 commits into
NVIDIA:mainfrom
brnguyen2:k3-disagg-kda-cpu-only-marker
Aug 11, 2026
Merged

[TRTLLM-15264][test] Wire KDA disagg transfer tests into CI (cpu_only CPU stage + l0_b200 GPU)#17484
brnguyen2 merged 2 commits into
NVIDIA:mainfrom
brnguyen2:k3-disagg-kda-cpu-only-marker

Conversation

@brnguyen2

@brnguyen2 brnguyen2 commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Description

Follow-up to review feedback on #17334, wiring tests/unittest/disaggregated/test_kda_mamba_transfer.py into CI. Before this change the file was in no test list except the dir-level unittest/disaggregated entry that only the CPU stage consumes — and it was silently dropped there too, so none of its tests ran anywhere.

Two commits:

1. Mark the CPU-runnable tests cpu_only (T3 of TRTLLM-15264).
The CPU-Generic-* stages run pytest with -m cpu_only, and conftest.py's pytest_ignore_collect skips any file in unittest/disaggregated that contains no pytest.mark.cpu_only text. The two GPU-free tests — test_kda_peer_validation_synthetic_cpu and test_kda_peer_validation_allows_pipeline_parallel_layer_split (synthetic rank-info / page-table builders + MambaPolicy.validate_peer_compatible, no CUDA manager) — now carry the marker and run on the CPU stage.

2. Wire the GPU tests into l0_b200.
The remaining tests (test_kda_transfer NIXL loopback + the manager-backed peer-validation cases) build simulated TP ranks in one process on a single device, so a single-GPU Blackwell stage suffices; added to the Kimi K3 block of l0_b200 alongside the other single-GPU K3 KDA suites. The GPU stage's -m "not cpu_only" markexpr selects exactly these and skips the two cpu_only tests.

Net: the CPU subset runs on CPU-Generic-*, the GPU subset on l0_b200, with the single cpu_only marker partitioning them.

Test Coverage

No product code and no test logic changed — marker + test-list wiring only. Not run locally (dev box has no torch/CUDA stack); CI now executes the tests it previously skipped. If the GPU cases show a B200-specific issue (cf. test_kda_prefill_op.py, which NaNs on B200 but passes on GB300 — TRTLLM-15204), the file can move to l0_gb300_multi_gpus.

PR Checklist

  • PR title follows the [TRTLLM-15264][test] convention
  • pre-commit run on the touched files (incl. test-list AST validation)

Dev Engineer Review

  • Added pytest.mark.cpu_only to:
    • test_kda_peer_validation_synthetic_cpu
    • test_kda_peer_validation_allows_pipeline_parallel_layer_split
  • Added unittest/disaggregated/test_kda_mamba_transfer.py to tests/integration/test_lists/test-db/l0_b200.yml.
  • The marker matches the l0_cpu CI filter.
  • Manager-backed and NIXL transfer tests remain GPU-only.
  • The test path and list format are consistent.
  • No product code or test logic changed.
  • No correctness, performance, API, or regression concerns identified.

QA Engineer Review

  • Modified test functions:
    • test_kda_peer_validation_synthetic_cpu
    • test_kda_peer_validation_allows_pipeline_parallel_layer_split
  • Added unittest/disaggregated/test_kda_mamba_transfer.py to tests/integration/test_lists/test-db/l0_b200.yml.
  • The GPU tests are covered by the B200 CI test list.
  • The two CPU-only tests are selected by the l0_cpu CI sweep.
  • Verdict: sufficient

…only

The l0_cpu stage collects unittest/disaggregated but runs pytest with
-m cpu_only, so tests in test_kda_mamba_transfer.py without the marker are
never selected by the CPU sweep. Mark the two logic-only tests that use the
synthetic rank-info/page-table builders and MambaPolicy.validate_peer_compatible
(no CUDA cache manager): test_kda_peer_validation_synthetic_cpu and
test_kda_peer_validation_allows_pipeline_parallel_layer_split. The
manager-backed and NIXL-transfer tests in the same file stay GPU-only.

Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>
@brnguyen2
brnguyen2 requested a review from a team as a code owner August 11, 2026 02:06
@brnguyen2

Copy link
Copy Markdown
Collaborator Author

/bot run

@coderabbitai

coderabbitai Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

Two disaggregated validation tests are marked as CPU-only. The KDA Mamba transfer test module is added to the B200 pre-merge test list.

Changes

Test execution

Layer / File(s) Summary
CPU-only validation scheduling
tests/unittest/disaggregated/test_kda_mamba_transfer.py, tests/integration/test_lists/test-db/l0_b200.yml
The synthetic peer-validation matrix test and pipeline-parallel layer-split validation test use pytest.mark.cpu_only. The test module is added to the B200 pre-merge configuration, with CPU-only cases assigned to the CPU-Generic stage.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Suggested reviewers: bowenfu, schetlur-nv

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the main change: wiring KDA disaggregated transfer tests into CPU and B200 GPU CI stages.
Description check ✅ Passed The description explains the purpose, test partitioning, coverage, CI behavior, and checklist items with sufficient detail.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65166 [ run ] triggered by Bot. Commit: 60e6720 Link to invocation

test_kda_mamba_transfer.py was in no GPU test list, so its GPU tests
(test_kda_transfer NIXL loopback and the manager-backed peer-validation
cases) ran nowhere in CI. Add the file to the Kimi K3 block of l0_b200
alongside the other single-GPU K3 KDA suites; the tests build simulated
tp ranks in one process on a single device, so a single-GPU Blackwell
stage is sufficient. The GPU stage's 'not cpu_only' markexpr selects only
the GPU cases, leaving the two cpu_only peer-validation tests to the
CPU-Generic stage.

Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>
@brnguyen2

Copy link
Copy Markdown
Collaborator Author

/bot run

@brnguyen2 brnguyen2 changed the title [TRTLLM-15264][test] Mark CPU-runnable KDA peer-validation tests cpu_only [TRTLLM-15264][test] Wire KDA disagg transfer tests into CI (cpu_only CPU stage + l0_b200 GPU) Aug 11, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integration/test_lists/test-db/l0_b200.yml`:
- Around line 102-104: Add the standard NVIDIA copyright header at the beginning
of l0_b200.yml, preserving the existing test-list entries and formatting.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: cccb855e-f124-4aed-9f56-08ca5a5b469e

📥 Commits

Reviewing files that changed from the base of the PR and between 60e6720 and ff7acc7.

📒 Files selected for processing (1)
  • tests/integration/test_lists/test-db/l0_b200.yml

Comment thread tests/integration/test_lists/test-db/l0_b200.yml
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65170 [ run ] triggered by Bot. Commit: ff7acc7 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65166 [ run ] completed with state ABORTED. Commit: 60e6720

Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #65170 [ run ] completed with state SUCCESS. Commit: ff7acc7
/LLM/main/L0_MergeRequest_PR pipeline #52964 completed with status: 'SUCCESS'

CI Report

Link to invocation

@brnguyen2
brnguyen2 merged commit 48df89d into NVIDIA:main Aug 11, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants