Skip to content

testintel: fix severe issue in sellmvt_device (created by kliegeoi (@kliegeois)) - #13339

Merged
jiagaoxiang merged 11 commits into
ROCm:developfrom
jiagaoxiang:testintel/fix/cand-430c77788a2402dc4dfa26da9c0a1b5b-2
Oct 22, 2026
Merged

jiagaoxiang merged 11 commits into
ROCm:developfrom
jiagaoxiang:testintel/fix/cand-430c77788a2402dc4dfa26da9c0a1b5b-2

Conversation

@jiagaoxiang

@jiagaoxiang jiagaoxiang commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

This PR was created by kliegeoi (@kliegeois) via Test Gap Resolver.

Motivation

The SELL SpMV kernels compute a thread's row as sell_slice_size * sliceid + tidx. With 32-bit indices (J = int32_t) this is evaluated in uint32 and stored in int32, so near m = 2^31 it wraps negative and passes the row < m check. That happens in the padding slices of the launch grid (blocks_x * blocks_y >= nslices) and in the last real slice whenever m is not a multiple of the slice size. Padding slices then read past the end of sell_slice_offsets, sellmvn_device can write y at a negative index, and sellmvt_device reads x at a negative index.

Separately, sellmvt_device read x[row] for every padding thread (row >= m), past the end of x.

Technical Details

  • Compute row in int64_t in sellmvn_device, sellmvn_large_slice_device, sellmvt_device and sellmvt_large_slice_device.
  • sellmvt_device reads x[row] only when row < m, as sellmvt_large_slice_device already did.
  • Add a Resolved issues entry to the rocSPARSE changelog and update the copyright year.

Test Plan

No client test is added. Padding threads have start == end == 0, so the stray x read never changes the output, and reaching the overflow needs m close to 2^31 (8 GiB per single-precision vector plus a host reference over 2^31 rows). Instead:

  • Build rocsparse_sellmv_impl.cpp for gfx942 with the ROCm 10 compiler (clang 23).
  • On an MI325X, run a standalone kernel that calls sellmvn_device and sellmvn_large_slice_device with the launch configuration of sellmv_dispatch, on an empty matrix with 32-bit indices and m = 2^31 - 1, with a 2^31-element guard in front of y, once with the old header and once with the new one.

Test Result

  • rocsparse_sellmv_impl.cpp compiles without warnings.
  • Old header: writes land in front of y for slice sizes 3, 5, 7, 64, 100, 127 and 200 (between 6,424 and 423,664 writes per size). Slice size 2, which has no padding slices and no wrap, has none.
  • New header: no writes outside y for any slice size, and all 2^31 - 1 rows of y are written.

Submission Checklist

JIRA ID : AISPARSE-806

Candidate cand:430c77788a2402dc4dfa26da9c0a1b5b at fb79226.
@therock-pr-bot

therock-pr-bot Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

✅ All Checks Passed — Ready for Review

Check Status Details
📝 PR Description ✅ Pass —
⛔ Forbidden Files ✅ Pass —
🧪 Unit Test ⚠️ Warning Error: Source/code files changed without an accompanying unit test.
Expected: add at least one test file named like test_<name>.py / test_<name>.cpp (or <name>_test.*).
Current: code file(s) changed: projects/rocsparse/library/src/level2/sellmv_device.h; no test file found
🔎 pre-commit ✅ Pass —
🚫 Draft PR 🔜 To Be Enabled —
🚩 Feature Flag 🔜 To Be Enabled —
📊 Code Coverage 🔜 To Be Enabled —
🤖 therock-pr-bot ✅ Pass —

🎉 All checks passed! This PR is ready for review.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

@therock-pr-bot

therock-pr-bot Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

🎉 All checks passed! This PR is ready for review.

@kliegeois
kliegeois marked this pull request as ready for review October 8, 2026 18:38
@kliegeois
kliegeois requested a review from a team as a code owner October 8, 2026 18:38
@kliegeois
kliegeois enabled auto-merge (squash) October 8, 2026 21:25
@kliegeois
kliegeois disabled auto-merge October 9, 2026 13:43
@ntrost57

ntrost57 commented Oct 9, 2026

Copy link
Copy Markdown
Contributor
  • Changelog missing "resolved issues"
  • Copyright adjustment missing

Also, const J row = sell_slice_size * sliceid + tidx; (sellmv_device.h lines 63/136/198/254) is computed in uint32 and stored in int32 when J = int32_t. Near m = 2^31 it wraps negative and passes row < m. At m = 2^31-1 this happens for slice sizes 3, 5, 7, 64, 100 and 127. Padding slices then read past the end of sell_slice_offsets, and sellmvn_device can write y at a negative index. Fix by computing row in int64_t or by checking sliceid < nslices first.

With 32-bit indices, row = sell_slice_size * sliceid + tidx was computed
in uint32 and stored in int32. Near m = 2^31 it wrapped negative and
passed the row < m check, so padding slices read past the end of
sell_slice_offsets and sellmvn_device could write y at a negative index.
The last real slice wraps too when m is not a multiple of the slice size,
so the row index is now computed in int64_t in all four SELL kernels.

Add the changelog entry and the copyright year, and drop the test added
earlier in this PR: padding threads have start == end == 0, so it passed
with or without the fix.
@jiagaoxiang
jiagaoxiang requested a review from a team as a code owner October 9, 2026 19:18
@jiagaoxiang

jiagaoxiang commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor Author

@ntrost57 Thanks. Addressed in 6f18a21:

  • Added a Resolved issues entry to the rocSPARSE changelog.
  • Updated the copyright year in sellmv_device.h.
  • row is now computed in int64_t in all four kernels. I used that rather than a sliceid < nslices check because the last real slice wraps too when m is not a multiple of the slice size: at m = 2^31 - 1 and slice size 3, tidx = 2 in the last slice gives row = 2^31.
  • Removed the test (see the inline reply).

I checked it on an MI325X with a standalone kernel that calls sellmvn_device and sellmvn_large_slice_device with the launch configuration of sellmv_dispatch, on an empty matrix with 32-bit indices and m = 2^31 - 1, with a 2^31-element guard in front of y:

slice size padding slices writes in front of y, old header new header
2 0 0 0
3 2,141 6,424 0
5 8,170 40,852 0
7 9,361 65,532 0
64 4,416 282,624 0
100 3,753 375,352 0
127 3,335 423,664 0
200 (large-slice kernel) 1,309 261,952 0

All 2^31 - 1 rows of y are written in both runs. The writes beyond padding slices × slice size come from the last real slice (1 at slice size 3, 52 at 100). rocsparse_sellmv_impl.cpp also builds for gfx942 with clang 23 without warnings.

@jiagaoxiang

Copy link
Copy Markdown
Contributor Author

TestIntel PR Steward: Not merging yet. ntrost57's review comments were already addressed in 6f18a21: the changelog entry, the copyright year, row computed in int64_t, and the ineffective test removed. There are no failing checks, no conflicts, and no outstanding change requests.

GitHub still reports the PR as blocked. Two things likely explain it:

  • The approvals from jsandham and kliegeois (18:41 and 18:49 UTC) predate the latest push, so they may be stale.
  • The requested sparses-docs-reviewers team review is still pending.

A human needs to re-approve on the current head and clear the team review. After that the PR is ready to merge, and I'll merge it on the next poll once GitHub reports it mergeable.

@jiagaoxiang
jiagaoxiang requested a review from ntrost57 October 12, 2026 20:17
@jiagaoxiang

Copy link
Copy Markdown
Contributor Author

TestIntel PR Steward: ntrost57 approved the current head (316636e), and there are no failing or pending checks, conflicts or change requests. GitHub still reports the PR as blocked, most likely because a required review is outstanding, such as the sparses-docs-reviewers team. Once that is cleared and GitHub reports the PR as mergeable, I'll merge it on the next poll.

@jiagaoxiang

Copy link
Copy Markdown
Contributor Author

TestIntel PR Steward: ntrost57 approved the current head (316636e) and no checks are failing, but GitHub now reports merge conflicts with develop (dirty), so I can't merge. The conflict is probably in projects/rocsparse/CHANGELOG.md, where other entries were likely added nearby. The branch needs to be updated from develop, either by merging develop into it or by rebasing, with the conflicts resolved, and I can't do that from here. A new commit will invalidate the existing approval, so re-approval on the new head will be needed. After that I'll merge on the next poll once checks are green and GitHub reports the PR mergeable.

…da9c0a1b5b-2

Resolve the conflict in projects/rocsparse/CHANGELOG.md by keeping the Resolved issues entry that develop added and putting this PR's SELL SpMV entry after it.
@jiagaoxiang

Copy link
Copy Markdown
Contributor Author

TestIntel PR Steward: The branch now has a new head (0a27efc) and GitHub no longer reports a conflict with develop. The PR is not ready to merge yet, for two reasons:

  • The latest human approval (ntrost57) is on 316636e, so a human owner, member or collaborator needs to re-approve this head.
  • 4 checks are still pending on this head.

Once the checks pass and the head is re-approved, I'll merge on the next poll if GitHub reports the PR as mergeable.

@jiagaoxiang

Copy link
Copy Markdown
Contributor Author

Merged the latest develop into this branch in 0a27efc. The only conflict was in projects/rocsparse/CHANGELOG.md, where develop and this PR had each added a Resolved issues entry in the same place. The merge keeps both, with this PR's SELL SpMV entry last, so the PR's diff against develop is unchanged. I also resolved the outdated review thread on the removed test, since the develop ruleset requires every thread to be resolved. The existing approvals still stand.

@kliegeois @ntrost57 one blocker remains that I can't clear from here: the required Math CI Summary status has never been posted on this PR. None of its four commits has any Math CI status, while the rocSPARSE PRs merged this week (#12316, #12007, #12830, #11352) all got one, so Math CI doesn't seem to dispatch for PRs from this account. Could one of you dispatch Math CI for this PR, or ask a gardener to push it through once Multi-Arch CI on 0a27efc is green?

@jiagaoxiang

Copy link
Copy Markdown
Contributor Author

TestIntel PR Steward: ntrost57 and kliegeois have approved the current head (0a27efc), no checks are failing or pending, and there are no conflicts or change requests. The PR is ready except for one blocker: GitHub still reports it as blocked, most likely because the required Math CI Summary status has not been posted on this head. A human needs to dispatch Math CI for this PR, or a gardener needs to push it through. Once GitHub reports the PR as mergeable, I'll merge on the next poll.

@codecov

codecov Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop   #13339      +/-   ##
===========================================
- Coverage    75.93%   75.93%   -0.00%     
===========================================
  Files         2873     2873              
  Lines       477006   477006              
  Branches     71572    71572              
===========================================
- Hits        362207   362186      -21     
- Misses       90102    90135      +33     
+ Partials     24697    24685      -12     
Flag Coverage Δ *Carryforward flag
TensileLite-CPP 48.93% <ø> (ø) Carriedforward from 447caaf
TensileLite-Unit 83.65% <ø> (ø) Carriedforward from 447caaf
hipBLAS 90.62% <ø> (ø) Carriedforward from 447caaf
hipBLASLt 64.34% <ø> (ø) Carriedforward from 447caaf
hipCUB 82.68% <ø> (ø) Carriedforward from 447caaf
hipDNN 87.04% <ø> (ø) Carriedforward from 447caaf
hipFFT 42.00% <ø> (ø) Carriedforward from 447caaf
hipRAND 76.12% <ø> (ø) Carriedforward from 447caaf
hipSOLVER 69.67% <ø> (ø) Carriedforward from 447caaf
hipSPARSE 86.99% <ø> (ø) Carriedforward from 447caaf
rocBLAS 68.73% <ø> (ø) Carriedforward from 447caaf
rocFFT 48.10% <ø> (ø) Carriedforward from 447caaf
rocRAND 57.30% <ø> (ø) Carriedforward from 447caaf
rocSOLVER 76.74% <ø> (ø) Carriedforward from 447caaf
rocSPARSE 76.89% <ø> (-0.03%) ⬇️
rocThrust 93.15% <ø> (ø) Carriedforward from 447caaf

*This pull request uses carry forward flags. Click here to find out more.
see 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@jiagaoxiang

Copy link
Copy Markdown
Contributor Author

TestIntel PR Steward: The only failing check on the current head (2fa5511) is codecov/project/rocSPARSE: project coverage is 76.92% against an 80% target. This PR doesn't cause it. Codecov reports that all modified lines are covered, and the PR changes no tests. The change fixes a 32-bit row overflow in the SELL SpMV kernels, which needs m near 2^31 to reach, so it can't be exercised by a normal client test. I'm not adding tests or loosening anything to turn it green. A maintainer will need to waive or override this project-level check. kliegeois and ntrost57 have approved this head. The remaining blockers are the 18 pending checks and the required Math CI Summary status, which a human needs to dispatch. I'll merge on a later poll once checks are green and GitHub reports the PR as mergeable.

@jiagaoxiang

Copy link
Copy Markdown
Contributor Author

TestIntel PR Steward: The check Linux::release / Test gfx94X-dcgpu / Test rocsolver / Test rocsolver (shard 3/3) (gfx94X-dcgpu) failed on the current head (2fa5511). Its "Download and Unpack Artifacts" step timed out ("The action has timed out") while fetching artifacts, before any rocsolver test ran. That is an infrastructure flake, not a defect in this PR, which only touches the SELL SpMV kernels in rocSPARSE and the changelog. A human with write access needs to rerun that job (https://github.com/ROCm/rocm-libraries/actions/runs/38761018761/job/116549742379). I can't rerun jobs. The PR is approved on this head, so I'll merge on a later poll once checks are green and GitHub reports the PR as mergeable.

@jiagaoxiang

Copy link
Copy Markdown
Contributor Author

TestIntel PR Steward: The current head (2fa5511) is approved by kliegeois and ntrost57, but GitHub again reports merge conflicts with develop (dirty), so I can't merge. It is probably projects/rocsparse/CHANGELOG.md again, since other entries were added near this one. I can't merge or rebase the branch from here, so someone with push access needs to update it from develop and resolve the conflicts. A new commit will invalidate the existing approvals, so the PR needs re-approval on the new head. Two other blockers remain: the codecov/project/rocSPARSE check, which a maintainer needs to waive (explained in my earlier comment), and the required Math CI Summary status. I'll merge on a later poll once the conflicts are resolved, checks are green, the head is approved, and GitHub reports the PR as mergeable.

@jiagaoxiang

Copy link
Copy Markdown
Contributor Author

@kliegeois @ntrost57 this is ready except for Math CI. Math CI passed on 2fa5511, but after the Oct 16 merge from develop (3607aaf) its precheckin, static-analysis and codecov jobs never started, so Math CI Summary has been pending since Oct 17. 3607aaf only merges develop: the PR's diff against develop is the same two-file change that passed Math CI on 2fa5511.

Could one of you re-run Math CI for PR-13339 or ask a gardener to push it through, and then merge? Neither of my accounts has write access to this repository, so I can't merge it myself.

@jiagaoxiang

Copy link
Copy Markdown
Contributor Author

TestIntel PR Steward: jsandham has approved the current head (3607aaf), and no checks are failing or pending. GitHub still reports the PR as blocked. The likely cause is the required Math CI Summary status, which has not been posted on this head: Math CI's jobs did not start after the merge from develop. A human with write access needs to re-run Math CI for this PR or ask a gardener to push it through. The codecov/project/rocSPARSE project-level check may also need a maintainer waiver, as explained earlier. I'll merge on a later poll once GitHub reports the PR as mergeable.

@jiagaoxiang

Copy link
Copy Markdown
Contributor Author

TestIntel PR Steward: The check Linux::release / Test gfx94X-dcgpu / Test rocsolver / Test rocsolver (shard 3/3) (gfx94X-dcgpu) failed on the current head (7f3102f). The log shows a GPU memory access fault (HSA_STATUS_ERROR_MEMORY_FAULT) in rocsolver::stedc_mergeValues_copyD_kernel<float, int> while running checkin_lapack/SYGVDX.batched__float/30. This PR only changes the rocSPARSE SELL SpMV kernels and the changelog, and never runs rocsolver code, so it did not cause this failure. I can't rerun jobs, so a human with write access needs to rerun it (https://github.com/ROCm/rocm-libraries/actions/runs/39504715723/job/118976300734). If it fails again on rerun, it is probably a rocsolver or runner problem on the base branch. jsandham has approved this head. I'll merge on a later poll once checks are green and GitHub reports the PR as mergeable. The Math CI Summary status and the codecov/project/rocSPARSE waiver mentioned earlier are still needed.

@jiagaoxiang

Copy link
Copy Markdown
Contributor Author

TestIntel PR Steward: The check Linux::release / Test gfx94X-dcgpu / Test rocsolver / Test rocsolver (shard 3/3) (gfx94X-dcgpu) failed again on the current head (18e2b09). The log shows the same GPU memory access fault (HSA_STATUS_ERROR_MEMORY_FAULT) in rocsolver::stedc_mergeValues_copyD_kernel<float, int>, this time on checkin_lapack/SYGVDX.batched__float/30. This PR only changes the rocSPARSE SELL SpMV kernels and the changelog, so it does not cause this failure. I can't rerun jobs, so a human with write access needs to rerun it (https://github.com/ROCm/rocm-libraries/actions/runs/39555758755/job/119100486333). Because it has failed on more than one run, it is probably a rocsolver or runner problem on develop, and a rocsolver owner may need to look at it. jsandham has approved this head. I'll merge on a later poll once checks are green and GitHub reports the PR as mergeable. The Math CI Summary status and the codecov/project/rocSPARSE waiver mentioned earlier are still needed.

@jiagaoxiang

Copy link
Copy Markdown
Contributor Author

@eidenyoshida could you advise why the math CI didn't run?

@jiagaoxiang
jiagaoxiang merged commit 4131713 into ROCm:develop Oct 22, 2026
133 of 134 checks passed
@assistant-librarian

Copy link
Copy Markdown
Contributor

TheRock Submodule Bump Activity

Newest first

kliegeois added a commit that referenced this pull request Nov 5, 2026
…OCM-33114) (#13879)

JIRA ID : ROCM-33114.

## Summary
- Add device unit tests for ELL triangular solve and SELL SpMV. The YAML
coverage misses the slice-size boundaries and a partial last slice.
- ELL solve covers a lower non-unit diagonal, a unit diagonal whose
stored diagonal must be ignored, and a zero pivot. The zero-pivot output
is preset to a sentinel and its status must match the reported position.
Those cases are built only when the library has ELL TRSV.
- SELL SpMV covers the small-slice kernel (slice 32 and the 128 limit)
and the large-slice kernel (slice 129, and slice 300 for two passes per
slice), in both directions, each with a partial last slice; m = 600 also
leaves a surplus block in the launch grid. It also covers more than 8
entries per row in the small-slice kernel, slice size 1 (the CSR
fallback), non-zero beta in both directions, and a transpose case with n
> m. The beta = 0 cases start from a non-zero y.
- y is allocated with a sentinel tail, so a write past row m fails the
test. Out-of-bounds reads (for example by padding threads in the
transpose kernels) are not detectable by value checks and need a
sanitizer.
- Comparisons are exact: every value is a small integer or dyadic
fraction and partial sums stay below 2^24.
- This does not change `sellmv_device.h`. It is based on develop after
#13339, which guards the padding-thread read of `x` in the small-slice
transpose kernel.

## Test plan
- [x] Built `rocsparse-unit-test-device` for gfx1201; `EllSv.*:SellMv.*`
and the full binary pass.
- [x] Removing `row < m` from the non-transpose large-slice guard makes
`SellMv.none_slice_size_boundaries` fail.
- [ ] CI
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants