Skip to content

fix(test): mock the DEEPGEMM_RUBIN pin in test_get_subdir_file_list - #4292

Closed
aleozlx wants to merge 2 commits into
flashinfer-ai:mainfrom
aleozlx:fix/artifacts-test-deepgemm-rubin-mock
Closed

aleozlx wants to merge 2 commits into
flashinfer-ai:mainfrom
aleozlx:fix/artifacts-test-deepgemm-rubin-mock

Conversation

@aleozlx

@aleozlx aleozlx commented Jul 31, 2026

Copy link
Copy Markdown
Member

📌 Description

tests/test_artifacts.py::test_get_subdir_file_list has been failing on main since #4280:

FileNotFoundError: [Errno 2] No such file or directory:
  '/tmp/pytest-of-root/pytest-0/test_get_subdir_file_list0/cubins/7ec7ac40b9fd48172651b77ff2ebe20d79decc39/deep-gemm/checksums.txt'

Root cause. #4280 added ArtifactPath.DEEPGEMM_RUBIN and appended it to the cubin_dirs
list walked by get_subdir_file_list(). The same PR registered responses mocks for the two
other new Rubin pins (TRTLLM_GEN_BMM_RUBIN, TRTLLM_GEN_GEMM_RUBIN) but not for the
deep-gemm one. Under @responses.activate an unregistered URL raises ConnectionError, so
download_file() exhausts its 4 retries and returns False; get_checksums() ignores that
return value and open()s a checksums.txt that was never written.

The artifact itself is fine — I fetched both deep-gemm manifests from artifactory (HTTP 200)
and the live SHA256 of 7ec7ac40…/deep-gemm/checksums.txt matches the pin in
CheckSumHash.DEEPGEMM_RUBIN (09e961d4…). Only the unit test's mock registry was stale, so no
production download path is affected.

Changes:

  • tests/test_artifacts.py: register the DEEPGEMM_RUBIN directory index and checksums.txt,
    mirroring the BMM/GEMM Rubin pattern (same index body, distinct per-pin hashes).
  • tests/test_artifacts.py: extend the existing per-pin-hash-collision assertion to a shared
    deep-gemm kernel name, so deep-gemm gets the same regression coverage the trtllm-gen pins have.
  • flashinfer/artifacts.py: get_checksums() now raises a RuntimeError naming the pin and the
    URL when the manifest download fails, instead of falling through to a bare FileNotFoundError
    on a local cache path. That failure mode is what made this read as a cache bug rather than
    "a pin was added without a mock".

🔍 Related Issues

Regression from #4280. Observed on unrelated PR CI, e.g.
https://github.com/flashinfer-ai/flashinfer/actions/runs/30607427968/job/91082694368

🚀 Pull Request Checklist

✅ Pre-commit Checks

  • I have installed pre-commit by running pip install pre-commit (or used your preferred method).
  • I have installed the hooks with pre-commit install.
  • I have run the hooks manually with pre-commit run --all-files and fixed any reported issues.

🧪 Tests

  • Tests have been added or updated as needed.
  • All tests are passing (unittest, etc.) — verification in progress on a dev box; draft until tests/test_artifacts.py is confirmed green.

Reviewer Notes

CPU-only change; no kernel or GPU behavior is touched. The get_checksums() error-path change is
the only non-test edit — it converts a silent-download-failure into a diagnosable error and does
not alter the success path.

AI-assisted (Claude Code): diagnosis and patch.

Summary by CodeRabbit

  • Bug Fixes
    • Improved checksum validation when artifact manifests are unavailable.
    • Errors now clearly identify the affected artifact and repository instead of producing a less informative file-not-found error.
    • Added fallback to cached checksum manifests when refreshing from a repository fails.
    • Prevented checksum collisions for shared kernel filenames across different DeepGEMM paths, including the Rubin variant.

flashinfer-ai#4280 added ArtifactPath.DEEPGEMM_RUBIN to the cubin_dirs list walked by
get_subdir_file_list(), and registered `responses` mocks for the two other
new Rubin pins (TRTLLM_GEN_BMM_RUBIN / TRTLLM_GEN_GEMM_RUBIN) but not for
the deep-gemm one. Under @responses.activate the unregistered URL raises
ConnectionError, so download_file() exhausts its retries and returns False,
and get_checksums() then open()s a checksums.txt that was never written:

  FileNotFoundError: .../cubins/7ec7ac40.../deep-gemm/checksums.txt

The artifact itself is fine -- the live manifest SHA matches the pin in
CheckSumHash.DEEPGEMM_RUBIN; only the unit test's mock registry was stale.

Changes:
- Register the DEEPGEMM_RUBIN directory index and checksums.txt, mirroring
  the BMM/GEMM Rubin pattern (same index body, distinct per-pin hashes).
- Extend the per-pin-hash-collision assertion to a shared deep-gemm kernel
  name, so deep-gemm gets the coverage the trtllm-gen pins already have.
- get_checksums(): raise a RuntimeError naming the pin and URL when the
  manifest download fails, instead of falling through to a bare
  FileNotFoundError on a local cache path that hides the real cause.

AI-assisted (Claude Code): diagnosis and patch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

get_checksums now reports unavailable checksum manifests with a descriptive error. Tests cover cached manifests and distinct checksums for shared filenames across regular and Rubin DeepGEMM paths.

Changes

Checksum validation

Layer / File(s) Summary
Manifest availability validation
flashinfer/artifacts.py, tests/test_artifacts.py
get_checksums raises a RuntimeError when a checksum manifest is unavailable locally and cannot be downloaded. Tests verify the error and cached-manifest fallback.
DeepGEMM checksum regression coverage
tests/test_artifacts.py
Tests mock DeepGEMM Rubin endpoints, add distinct Rubin checksum fixtures, and compare regular and Rubin checksums for shared filenames.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: aneureka, vinnie6167

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the primary fix: adding the missing DEEPGEMM_RUBIN mock for test_get_subdir_file_list.
Description check ✅ Passed The description covers the root cause, changes, related issue, checklist, test status, and reviewer notes.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

Adds the negative-path coverage the mock fix implies:

- test_get_checksums_unreachable_pin_raises: a pin whose manifest cannot be
  fetched must raise, and the error must name the pin. Stubs download_file
  instead of mocking HTTP so the test does not pay 4 rounds of exponential
  backoff (~70s), which is what made the original CI failure so slow.
- test_get_checksums_falls_back_to_cached_manifest: pins the case that *should*
  degrade gracefully -- a failed refresh with a manifest already on disk still
  works, so the new raise does not break offline / FLASHINFER_NO_DOWNLOAD setups.

AI-assisted (Claude Code).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
flashinfer/artifacts.py (1)

229-237: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Add a regression test for the missing-manifest error.

get_checksums now has a new failure branch at Lines [229]-[237]. The supplied tests cover successful manifests, but they do not exercise a failed download with no cached file. Add a test that asserts RuntimeError contains the artifact pin and manifest URL. This protects the new diagnostic contract.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@flashinfer/artifacts.py` around lines 229 - 237, Add a regression test for
the get_checksums failure path where download_file returns false and no checksum
manifest exists in the cache. Mock the download and filesystem dependencies as
needed, assert that RuntimeError is raised, and verify its message contains both
the artifact pin (subdir) and manifest URL.
tests/test_artifacts.py (1)

507-511: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Assert the expected checksum for each DeepGEMM path.

The added case only feeds the existing inequality assertion. That detects a bare-filename collision, but it does not prove that the regular path receives the regular manifest hash and the Rubin path receives the Rubin manifest hash. Add expected hashes to this case and assert both exact values. A swapped mapping could otherwise pass.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/test_artifacts.py` around lines 507 - 511, Update the DeepGEMM case in
the artifact-path test to include the expected regular and Rubin manifest
checksums, then assert each resolved path against its corresponding exact hash.
Keep the existing inequality assertion, and ensure the assertions distinguish
the regular mapping from the Rubin mapping so swapped values cannot pass.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@flashinfer/artifacts.py`:
- Around line 229-237: Add a regression test for the get_checksums failure path
where download_file returns false and no checksum manifest exists in the cache.
Mock the download and filesystem dependencies as needed, assert that
RuntimeError is raised, and verify its message contains both the artifact pin
(subdir) and manifest URL.

In `@tests/test_artifacts.py`:
- Around line 507-511: Update the DeepGEMM case in the artifact-path test to
include the expected regular and Rubin manifest checksums, then assert each
resolved path against its corresponding exact hash. Keep the existing inequality
assertion, and ensure the assertions distinguish the regular mapping from the
Rubin mapping so swapped values cannot pass.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ae502f02-aea5-4b84-92c9-0a9c1f866339

📥 Commits

Reviewing files that changed from the base of the PR and between dca5546 and 6f5e6ca.

📒 Files selected for processing (2)
  • flashinfer/artifacts.py
  • tests/test_artifacts.py

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/test_artifacts.py`:
- Around line 317-320: Update the RuntimeError assertion in the checksum test
around artifacts.get_checksums to also construct the expected manifest URI using
the same safe_urljoin expression used by the artifact implementation, then
assert that URI appears in str(excinfo.value) alongside DEEPGEMM_RUBIN.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 50c63583-a651-4f4d-997c-4f7dcb439fbe

📥 Commits

Reviewing files that changed from the base of the PR and between 6f5e6ca and 7b1981e.

📒 Files selected for processing (1)
  • tests/test_artifacts.py

Comment thread tests/test_artifacts.py
Comment on lines +317 to +320
with pytest.raises(RuntimeError) as excinfo:
artifacts.get_checksums([artifact_paths.DEEPGEMM_RUBIN])
# The pin must be named -- that is the whole point of the error.
assert artifact_paths.DEEPGEMM_RUBIN in str(excinfo.value)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Assert the manifest URL as well as the pin.

The RuntimeError contract includes both values. This test checks only DEEPGEMM_RUBIN. An error that omits the URL still passes. Build the expected URI with the same safe_urljoin expression as flashinfer/artifacts.py:222-249 and assert that it is present.

Proposed test assertion
     assert artifact_paths.DEEPGEMM_RUBIN in str(excinfo.value)
+    expected_uri = safe_urljoin(
+        artifacts.FLASHINFER_CUBINS_REPOSITORY,
+        safe_urljoin(artifact_paths.DEEPGEMM_RUBIN, "checksums.txt"),
+    )
+    assert expected_uri in str(excinfo.value)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
with pytest.raises(RuntimeError) as excinfo:
artifacts.get_checksums([artifact_paths.DEEPGEMM_RUBIN])
# The pin must be named -- that is the whole point of the error.
assert artifact_paths.DEEPGEMM_RUBIN in str(excinfo.value)
with pytest.raises(RuntimeError) as excinfo:
artifacts.get_checksums([artifact_paths.DEEPGEMM_RUBIN])
# The pin must be named -- that is the whole point of the error.
assert artifact_paths.DEEPGEMM_RUBIN in str(excinfo.value)
expected_uri = safe_urljoin(
artifacts.FLASHINFER_CUBINS_REPOSITORY,
safe_urljoin(artifact_paths.DEEPGEMM_RUBIN, "checksums.txt"),
)
assert expected_uri in str(excinfo.value)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/test_artifacts.py` around lines 317 - 320, Update the RuntimeError
assertion in the checksum test around artifacts.get_checksums to also construct
the expected manifest URI using the same safe_urljoin expression used by the
artifact implementation, then assert that URI appears in str(excinfo.value)
alongside DEEPGEMM_RUBIN.

@aleozlx

aleozlx commented Jul 31, 2026

Copy link
Copy Markdown
Member Author

/bot run tests/gemm

@flashinfer-bot

Copy link
Copy Markdown
Collaborator

GitLab MR !1090 has been created, and the CI pipeline #60450915 is currently running. I'll report back once the pipeline job completes.

@Aneureka Aneureka left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@aleozlx
aleozlx enabled auto-merge (squash) July 31, 2026 09:25
bkryu added a commit that referenced this pull request Aug 1, 2026
…s + CuTe-DSL MoE device guard) (#4301)

<!-- .github/pull_request_template.md -->

## 📌 Description

<!-- What does this PR do? Briefly describe the changes and why they’re
needed. -->

`main` is currently red on multiple test files, all stemming from #4280.
This PR bundles the fixes so CI can go green in a single run — the gate
requires one passing run, and none of these fixes pass in isolation
(each file still fails on the others), so they cannot land separately.

**Supersedes #4292** . Current PR includes the commit from #4292 and
adds the remaining MoE fix on top to pass the CI

## 🔍 Related Issues

<!-- Link any related issues here -->

- #4280
- #4292

## 🚀 Pull Request Checklist

Thank you for contributing to FlashInfer! Before we review your pull
request, please make sure the following items are complete.

### ✅ Pre-commit Checks

- [ ] I have installed `pre-commit` by running `pip install pre-commit`
(or used your preferred method).
- [ ] I have installed the hooks with `pre-commit install`.
- [ ] I have run the hooks manually with `pre-commit run --all-files`
and fixed any reported issues.

> If you are unsure about how to set up `pre-commit`, see [the
pre-commit documentation](https://pre-commit.com/).

## 🧪 Tests

- [ ] Tests have been added or updated as needed.
- [ ] All tests are passing (`unittest`, etc.).

## Reviewer Notes

<!-- Optional: anything you'd like reviewers to focus on, concerns, etc.
-->


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved error messages when checksum manifests cannot be downloaded
and are unavailable locally, including details about the affected
artifact.
* Preserved the use of cached checksum manifests when refresh attempts
fail, preventing unnecessary errors when valid local data is available.
* Improved reliability when validating artifacts across multiple
hardware and software configurations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Alex Yang <aleyang@nvidia.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@flashinfer-bot

Copy link
Copy Markdown
Collaborator

[SUCCESS] Pipeline #60450915: 18/18 executed test jobs passed

@jiahanc jiahanc added run-ci and removed run-ci labels Aug 3, 2026
@aleozlx

aleozlx commented Aug 3, 2026

Copy link
Copy Markdown
Member Author

closing in favor of #4301

@aleozlx aleozlx closed this Aug 3, 2026
auto-merge was automatically disabled August 3, 2026 21:56

Pull request was closed

aleozlx added a commit that referenced this pull request Aug 4, 2026
…s + CuTe-DSL MoE device guard) (#4301)

<!-- .github/pull_request_template.md -->

## 📌 Description

<!-- What does this PR do? Briefly describe the changes and why they’re
needed. -->

`main` is currently red on multiple test files, all stemming from #4280.
This PR bundles the fixes so CI can go green in a single run — the gate
requires one passing run, and none of these fixes pass in isolation
(each file still fails on the others), so they cannot land separately.

**Supersedes #4292** . Current PR includes the commit from #4292 and
adds the remaining MoE fix on top to pass the CI

## 🔍 Related Issues

<!-- Link any related issues here -->

- #4280
- #4292

## 🚀 Pull Request Checklist

Thank you for contributing to FlashInfer! Before we review your pull
request, please make sure the following items are complete.

### ✅ Pre-commit Checks

- [ ] I have installed `pre-commit` by running `pip install pre-commit`
(or used your preferred method).
- [ ] I have installed the hooks with `pre-commit install`.
- [ ] I have run the hooks manually with `pre-commit run --all-files`
and fixed any reported issues.

> If you are unsure about how to set up `pre-commit`, see [the
pre-commit documentation](https://pre-commit.com/).

## 🧪 Tests

- [ ] Tests have been added or updated as needed.
- [ ] All tests are passing (`unittest`, etc.).

## Reviewer Notes

<!-- Optional: anything you'd like reviewers to focus on, concerns, etc.
-->

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved error messages when checksum manifests cannot be downloaded
and are unavailable locally, including details about the affected
artifact.
* Preserved the use of cached checksum manifests when refresh attempts
fail, preventing unnecessary errors when valid local data is available.
* Improved reliability when validating artifacts across multiple
hardware and software configurations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Alex Yang <aleyang@nvidia.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit d020372)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants