Fix test_joint_config_validation_uses_gathered_backends when no GPUs are visible by akshay919191 · Pull Request #7719 · flashinfer-ai/flashinfer

akshay919191 · 2026-10-28T10:22:43Z

📌 Description

The test test_joint_config_validation_uses_gathered_backends was failing on
zero-GPU CI machines because encoding cuda:0 raised a device visibility
error before the dtype validation could run. This PR patches
torch.cuda.device_count() to return 1 via monkeypatch so the test
validates the actual logic (dtype checking) independently of host GPU count.

🔍 Related Issues

Closes #7683

🚀 Pull Request Checklist

pre-commit run --all-files passes
Tests updated — existing test now passes on zero-GPU machines

Summary by CodeRabbit

Tests
- Made a backend validation test independent of the machine’s available GPU count.

coderabbitai · 2026-10-28T10:30:12Z

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: defaults
Review profile: CHILL
Plan: Advanced
Run ID: 0ce38cfc-7b41-46ea-8539-2a4dd4ab4da9

Commits

Reviewing files that changed from the base of the PR and between ee552e8 and 4250dec.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

No actionable comments were generated in the recent review.

Recent review info

Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: defaults
Review profile: CHILL
Plan: Advanced
Run ID: cc3bfd03-36f0-421b-9884-e49e33eba0eb

Commits

Reviewing files that changed from the base of the PR and between 7ca063e and ee552e8.

Files selected for processing: 1

tests/comm/test_ulysses_communicator.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

Walkthrough

The config-validation test now fixes the visible CUDA device count at one before encoding cuda:0. Its gathered-backend validation assertions are unchanged.

Changes: Communicator config validation

Layer / File(s)	Summary
Control visible CUDA device count `tests/comm/test_ulysses_communicator.py`	The test uses `monkeypatch` to set `torch.cuda.device_count()` to one before encoding `cuda:0`. The existing validation assertions are unchanged.

Priority: Low

Estimated code review effort: 1 (Trivial) | ~3 minutes

Change: Bug fix · Severity of issue fixed: Low

Merge Risk: Minimal · previous review ee552

The targeted test change is low risk and addresses zero-GPU CI behavior.

Pre-merge checks: 5

Check name	Status	Explanation
Title check		The title clearly identifies the test and the zero-GPU failure it fixes. It accurately summarizes the main change.
Description check		The description explains the failure, the monkeypatch-based fix, the related issue, and reported validation results. It omits some template checklist details, such as full-scope CI justification and t…
Linked Issues check		Issue `#7683` requires the affected test to run when no CUDA devices are visible and reach the intended gathered-backend validation. The test now patches `torch.cuda.device_count()` to return 1 before e…
Out of Scope Changes check		The diff changes only `tests/comm/test_ulysses_communicator.py`. It adds the `monkeypatch` fixture and a targeted `torch.cuda.device_count()` patch for the test named in issue `#7683`. The change suppor…
Docstring Coverage		No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…

Full details: Description check

Explanation

The description explains the failure, the monkeypatch-based fix, the related issue, and reported validation results. It omits some template checklist details, such as full-scope CI justification and the complete pre-commit setup confirmations, but it is sufficiently complete for this focused test fix.

Full details: Linked Issues check

Explanation

Issue #7683 requires the affected test to run when no CUDA devices are visible and reach the intended gathered-backend validation. The test now patches torch.cuda.device_count() to return 1 before encoding cuda:0. This removes the host GPU-count dependency while preserving the existing validation cases.

Full details: Out of Scope Changes check

Explanation

The diff changes only tests/comm/test_ulysses_communicator.py. It adds the monkeypatch fixture and a targeted torch.cuda.device_count() patch for the test named in issue #7683. The change supports the issue objective and contains no unrelated implementation changes.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 1 files.

Finishing Touches: 💡 1

🛠️ Fix failing CI checks: 💡

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

akshay919191 · 2026-10-28T10:34:15Z

@kangbintNV could you authorize the CI run when you get a chance? First-time contributor

kangbintNV · 2026-10-29T11:08:29Z

@flashinfer-bot run

kangbintNV · 2026-10-29T11:11:51Z

/bot run tests/comm/test_ulysses_communicator.py

flashinfer-bot · 2026-10-29T11:13:00Z

GitLab MR !2502 has been created, and the CI pipeline #77571011 is currently running. I'll report back once the pipeline job completes.

akshay919191 · 2026-10-29T12:48:50Z

@kangbintNV rebased on main — ready for review.

flashinfer-bot · 2026-10-29T14:41:51Z

[FAILED] Pipeline #77571011 — 34/41 executed test jobs passed

Compared with nightly #77255012 (different CI configuration).

Unit Tests

GPU	CUDA 12.9	CUDA 13.0	CUDA 13.4	Notes
B200	✅ Pass	✅ Pass	✅ Pass	—
GB200	✅ Pass	✅ Pass	✅ Pass	—
GB300	✅ Pass	✅ Pass	✅ Pass	—
H100	—	✅ Pass	—	—
RTX Pro 6000 Blackwell	—	✅ Pass	—	—
Spark	—	✅ Pass	—	—
VR NVL72	—	—	✅ Pass	—

✅ Pass · 🟡 Old failure · ❌ New failure · ⏱ Test timeout · ⚠️ Infrastructure · ❔ Unknown or unclassified · — Not run

Multi-GPU and Multi-Node Tests — 5/12 passed

GPU	CUDA 12.9	CUDA 13.0	CUDA 13.4	Notes
B200 (multi-GPU)	❌ New	❌ New	—	PR-related: `tests.comm.test_ulysses_communicator` (4 failures; CUDA 12.9, CUDA 13.0)
GB200 (multi-GPU)	❌ New	❌ New	—	PR-related: `tests.comm.test_ulysses_communicator` (4 failures; CUDA 12.9, CUDA 13.0)
GB200 (multi-node)	✅ Pass	✅ Pass	—	—
GB300 (multi-GPU)	❌ New	❌ New	—	PR-related: `tests.comm.test_ulysses_communicator` (4 failures; CUDA 12.9, CUDA 13.0)
GB300 (multi-node)	✅ Pass	✅ Pass	—	—
VR NVL72 (multi-GPU)	—	—	❌ New	PR-related: `tests.comm.test_ulysses_communicator` (2 failures; CUDA 13.4)
VR NVL72 (multi-node)	—	—	✅ Pass	—

Failure details

PR-related regressions

tests.comm.test_ulysses_communicator — 14 failures on B200 / CUDA 12.9 / multi-GPU, B200 / CUDA 13.0 / multi-GPU, GB200 / CUDA 12.9 / multi-GPU, GB200 / CUDA 13.0 / multi-GPU, GB300 / CUDA 12.9 / multi-GPU, GB300 / CUDA 13.0 / multi-GPU, VR NVL72 / CUDA 13.4 / multi-GPU
- AssertionError: rank 0 failed: AssertionError: forced nvlink must fail when init faults assert 'AssertionError' == 'ok' - ok + AssertionError

…GPUs _encode_config range-checks 'cuda:0' against torch.cuda.device_count(), so test_joint_config_validation_uses_gathered_backends failed on jobs with no visible device (Spark CI). The property under test does not depend on hardware, so pin the device count in the test. Fixes flashinfer-ai#7683

akshay919191 · 2026-10-29T15:24:03Z

@kangbintNV I’ve force-pushed since pipeline #77571011 started (now a single commit, 4250dec, rebased on current main). The multi-GPU test_ulysses_communicator failures (“forced nvlink must fail when init faults”) are in tests this PR doesn’t touch; the diff only changes test_joint_config_validation_uses_gathered_backends (+5/-1). Could you re-run /bot run tests/comm/test_ulysses_communicator.py on the new head, and could you check whether the same tests fail on main?

kangbintNV · 2026-10-29T22:10:59Z

@flashinfer-bot run tests/comm/test_ulysses_communicator.py

kangbintNV · 2026-10-29T22:11:26Z

/bot run tests/comm/test_ulysses_communicator.py

flashinfer-bot · 2026-10-29T22:11:48Z

GitLab MR !2502 has been updated with latest changes, and the CI pipeline #77720163 is currently running. I'll report back once the pipeline job completes.

flashinfer-bot · 2026-10-29T22:59:46Z

[FAILED] Pipeline #77720163 — 30/32 executed test jobs passed

Compared with nightly #77547447 (different CI configuration).

Unit Tests

GPU	CUDA 12.9	CUDA 13.0	CUDA 13.4	Notes
B200	✅ Pass	✅ Pass	✅ Pass	—
GB200	✅ Pass	✅ Pass	✅ Pass	—
H100	—	✅ Pass	—	—
RTX Pro 6000 Blackwell	—	✅ Pass	—	—
Spark	—	✅ Pass	—	—
VR NVL72	—	—	✅ Pass	—

✅ Pass · 🟡 Old failure · ❌ New failure · ⏱ Test timeout · ⚠️ Infrastructure · ❔ Unknown or unclassified · — Not run

Multi-GPU and Multi-Node Tests — 10/12 passed

GPU	CUDA 12.9	CUDA 13.0	CUDA 13.4	Notes
B200 (multi-GPU)	✅ Pass	✅ Pass	—	—
GB200 (multi-GPU)	✅ Pass	✅ Pass	—	—
GB200 (multi-node)	✅ Pass	✅ Pass	—	—
GB300 (multi-GPU)	✅ Pass	✅ Pass	—	—
GB300 (multi-node)	❔ Unknown	❔ Unknown	—	Unknown: `script failed before producing a JUnit report` (2 jobs; CUDA 12.9, CUDA 13.0)
VR NVL72 (multi-GPU)	—	—	✅ Pass	—
VR NVL72 (multi-node)	—	—	✅ Pass	—

Failure details

Timeouts, infrastructure, or incomplete jobs

Unknown: script failed before producing a JUnit report — GB300 / CUDA 12.9 / multi-node, GB300 / CUDA 13.0 / multi-node
- Jobs: multi_node_test_gb300: [cu130], multi_node_test_gb300: [cu129]

akshay919191 · 2026-10-30T03:25:50Z

Re-run #77720163 on 4250dec: Spark and all multi-GPU jobs pass, including test_ulysses_communicator, and the GitHub “Test Results Summary” is green. The only non-green jobs are the two GB300 multi-node ones (“script failed before producing a JUnit report”). I can’t see those logs, and the PR only changes one single-process test, so I don’t think they’re related. Could you take a look or re-run them? Otherwise this should be ready for review.

kangbintNV · 2026-10-30T06:22:03Z

@akshay919191 don't worry, this is node env issue, I reran your failed GB300 (multi-node) config and multi_node_test_gb300: [cu129] & multi_node_test_gb300: [cu130] are getting passed.

akshay919191 · 2026-10-30T06:29:40Z

@kangbintNV thanks for the re-runs. All checks are green now, and the only blocker is code owner review. Could you point me to, or request, the right owner for tests/comm/?

kangbintNV · 2026-10-30T06:31:23Z

@Anerudhan could you please review and approve this focused test-only fix for issue #7683?

The exact testcase now passes on Spark CI with no visible CUDA devices (unit_test_spark: [cu130]).

akshay919191 requested review from Anerudhan, Aneureka, Jingfan-Sun, YangXu1990uiuc, aleozlx, bkryu, feih-nv, jimmyzho, nv-yunzheq, yyihuang and yzh119 as code owners October 28, 2026 10:29

flashinfer-bot added the op: comm label Oct 28, 2026

akshay919191 changed the title ~~Fix/ulysses communicator 7683~~ Fix test_joint_config_validation_uses_gathered_backends when no GPUs are visible Oct 28, 2026

flashinfer-bot added the run-ci label Oct 29, 2026

akshay919191 force-pushed the fix/ulysses-communicator-7683 branch from 7ca063e to ee552e8 Compare October 29, 2026 12:48

akshay919191 force-pushed the fix/ulysses-communicator-7683 branch from ee552e8 to 4250dec Compare October 29, 2026 15:20

flashinfer-bot added run-ci and removed run-ci labels Oct 29, 2026

bkryu approved these changes Nov 2, 2026

View reviewed changes

bkryu merged commit 27efa45 into flashinfer-ai:main Nov 2, 2026
24 of 25 checks passed

Conversation

akshay919191 commented Oct 28, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📌 Description

🔍 Related Issues

🚀 Pull Request Checklist

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

akshay919191 commented Oct 28, 2026

Uh oh!

kangbintNV commented Oct 29, 2026

Uh oh!

kangbintNV commented Oct 29, 2026

Uh oh!

flashinfer-bot commented Oct 29, 2026

Uh oh!

akshay919191 commented Oct 29, 2026

Uh oh!

flashinfer-bot commented Oct 29, 2026

Unit Tests

Multi-GPU and Multi-Node Tests — 5/12 passed

PR-related regressions

Uh oh!

akshay919191 commented Oct 29, 2026

Uh oh!

kangbintNV commented Oct 29, 2026

Uh oh!

kangbintNV commented Oct 29, 2026

Uh oh!

flashinfer-bot commented Oct 29, 2026

Uh oh!

flashinfer-bot commented Oct 29, 2026

Unit Tests

Multi-GPU and Multi-Node Tests — 10/12 passed

Timeouts, infrastructure, or incomplete jobs

Uh oh!

akshay919191 commented Oct 30, 2026

Uh oh!

kangbintNV commented Oct 30, 2026

Uh oh!

akshay919191 commented Oct 30, 2026

Uh oh!

kangbintNV commented Oct 30, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

akshay919191 commented Oct 28, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Oct 28, 2026 •

edited

Loading