[SreecharanGundaboluAMD]

JIRA ID : ROCM-26011

Motivation

nn.Conv2d(96, 64, kernel_size=1) on MI350X with a [64, 96, 512, 512] fp32 input silently
returns weight.grad == 0.0 (every element exactly zero) after backward(), while
bias.grad and a mathematically-equivalent F.linear on the reshaped tensor both produce
correct, non-zero gradients. No warning or error is raised — the model trains, but the conv1x1
layer never receives gradients (ROCM-26011).

The root cause is in igemm_split_batch_size(), which decides how many dispatches to split a
convolution's batch across so no single dispatch addresses more than the ~4GB tensor limit.
For batch sizes in (max_n, 2*max_n), the split-count search returned a degenerate divisor of
1 ("no split needed") even though the tensor genuinely exceeded the 4GB limit. This let
ConvAsmImplicitGemmGTCDynamicWrwXdlopsNHWC launch a single dispatch that addressed more data
than the hardware limit allows, silently producing near-zero weight gradients instead of
failing loudly.

Technical Details

In projects/miopen/src/include/miopen/conv/asm_implicit_gemm.hpp, the split-multiplier
search used:

size_t m = std::ceil(n / max_n);

n and max_n are both integer types, so n / max_n truncates via integer division.

The fix replaces the float-ceil idiom with an integer-safe ceiling division:

size_t m = (static_cast<size_t>(n) + max_n - 1) / max_n;

Risk Assessment

Risk Level: 🟡 Medium

The diff is a single line, but it sits in a core MIOpen dispatch path (igemm_split_batch_size)
that gates numerical correctness — not just performance — for
ConvAsmImplicitGemmGTCDynamicWrwXdlopsNHWC across every batch size that falls in the affected
(max_n, 2*max_n) range.

Potential Side Effects

• Convolutions whose batch size previously fell in the buggy (max_n, 2*max_n) range will now
  be correctly split into multiple dispatches. This changes solver behavior (more dispatches,
  slightly different perf characteristics) for those specific shapes, but corrects silently
  wrong output to numerically correct output.

Mitigation Steps

• Previously failing shape passes.
• Change is minimal and localized to the split-count computation, with no change to the
  surrounding search algorithm's structure, keeping regression surface small.

Test Plan

• Reproduced the reported bug with a standalone repro matching the ticket's nn.Conv2d(96, 64, kernel_size=1) / [64, 96, 512, 512] fp32 shape, and additionally swept batch sizes spanning
  n=43..192 with multiple channel configurations to cover the full (max_n, 2*max_n) boundary
  range where the split-count search previously degenerated to m=1. Compared MIOpen
  ConvAsmImplicitGemmGTCDynamicWrwXdlopsNHWC weight-gradient output against a reference
  (F.linear-equivalent / naive) computation before and after the fix.

Test Result

• Before the fix, all swept shapes in the affected range produced weight.grad that was zero or
  otherwise numerically incorrect. After the fix, all 11+ verified shapes (n=43..192, multiple
  channel configs) match the reference computation to ~1e-6/1e-7 precision, including the exact
  shape from the reported ticket. No regressions observed in shapes outside the affected range,
  where the split-count computation was already correct.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

[therock-pr-bot]

✅ All Checks Passed — Ready for Review

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	⚠️ Warning	Error: Source/code files changed without an accompanying unit test. Expected: add at least one test file named like test_<name>.py / test_<name>.cpp (or <name>_test.*). Current: code file(s) changed: projects/miopen/src/include/miopen/conv/asm_implicit_gemm.hpp; no test file found
🔎 pre-commit	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—
🤖 therock-pr-bot	✅ Pass	—

🎉 All checks passed! This PR is ready for review.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

[therock-pr-bot]

🎉 All checks passed! This PR is ready for review.

[JonathanLichtnerAMD]

LGTM

[assistant-librarian]

TheRock Submodule Bump Activity

Newest first

• 2026-09-01 — Included in TheRock via ROCm/TheRock#7766.