Repository navigation
[MIOpen] Fix miopenStatusInternalError on depthwise NHWC Winograd conv (ROCM-26918) - #8722
Merged
adickin-amd merged 5 commits intoJun 24, 2026
Conversation
…v (ROCM-26918) Revert #6196 commented out the TransposedConvWinoRageRxS<2,3> registration in solver.cpp (keeping a ++id placeholder so later solver ids stay stable) but left that solver in the two Winograd Find solver lists in mlo_dir_conv.cpp. Under NORMAL find, Find selects the unregistered solver and then solver::Id::GetAlgo() (solver.cpp:198) throws miopenStatusInternalError, which PyTorch surfaces as a RuntimeError on CvT's depthwise NHWC fp16 3x3 group convolution. Remove TransposedConvWinoRageRxS<2,3> from both Find lists (GetWindogradSolvers and GetWindogradWrWSolvers) to complete the revert and make the Find lists consistent with the Id registry. No solver.cpp change is needed. Also add a regression-guard gtest (CPU_SolverFindListRegistry_NONE) that asserts every solver in the mlo_dir_conv.cpp Find containers is present in the Id registry, so this class of find-list/registry drift cannot silently ship again. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Keep the fix self-documenting without the internal ticket id; the defect is referenced in the PR instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
iq136boy
approved these changes
Jun 23, 2026
mattwill-amd
approved these changes
Jun 23, 2026
The production Find paths that triggered the original failure (ShrinkToFind10Results, EvaluateConvSolutions, FillFindReturnParameters in convolutionocl.cpp) call solver::Id::GetAlgo() on found solver ids without an IsValid() guard. Drop the IsValid() guard around the GetAlgo() call in the test so it exercises that exact unguarded path instead of skipping it for the very ids the test exists to catch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
issuing override merge. Confirmed with developer the change is low risk |
adickin-amd
deleted the
users/jlichtne/ROCM-26918-fix-incomplete-winograd-rage-revert
branch
June 24, 2026 14:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
PyTorch CvT / FAMBench training on gfx942 (MI325) threw
RuntimeError: miopenStatusInternalErroron a depthwise NHWC fp16 3x3 grouped convolution. The crash traced tosolver::Id::GetAlgo()insrc/solver.cppdoing a registry lookup for a solver that no longer existed in the registry.Root cause: PR #6196 ("Revert Integrate several Wino Rage kernels") removed
TransposedConvWinoRageRxS<2,3>from the solver Id registry (replacing it with a++id;placeholder to keep downstream solver Ids stable) but left that solver in both Winograd Find solver lists insrc/mlo_dir_conv.cpp(GetWindogradSolversandGetWindogradWrWSolvers). Under NORMAL find mode, Find selects the now-unregistered solver; the subsequentGetAlgo()call finds no registry entry and throwsmiopenStatusInternalError. Immediate and AI-heuristic find modes avoid the code path, which is why the bug only reproduced via PyTorch's DYNAMIC_HYBRID path (which falls back to NORMAL for uncached convolutions).Technical Details
Fix (
src/mlo_dir_conv.cpp): RemoveTransposedConvWinoRageRxS<2,3>fromGetWindogradSolvers()andGetWindogradWrWSolvers(), completing the revert started in PR #6196. The Rage kernels were originally reverted due to memory-access faults, so re-registration is not appropriate here.Guard (
src/include/miopen/mlo_internal.hpp+src/mlo_dir_conv.cpp): AddGetAllFindSolverDbIds()— a new exported helper that enumerates every solver across all eight Find containers (GetGemmSolvers,GetDirectSolvers,GetImplicitGemmSolvers,GetWindogradSolvers,GetImplicitGemmWrWSolvers,GetWindogradWrWSolvers,GetBwdWrW2DSolvers,GetFFTSolvers) and returns theirSolverDbIdstrings. This surfaces the complete Find-reachable solver set to tests without exposing internal container types.Regression test (
test/gtest/solver_find_list_registry.cpp):CPU_SolverFindListRegistry_NONE.AllFindSolversAreRegisterediterates every id returned byGetAllFindSolverDbIds(), assertssolver::Id::IsValid()is true for each, and assertsGetAlgo()does not throw — the exact call that crashed in production. The test is CPU-only (no GPU required) and runs in NONE find mode, so it catches find-list/registry drift at build time rather than in a field workload.No other behavior is changed. The solver Id numbering in the registry is not disturbed (the
++id;placeholder introduced by PR #6196 is left in place).Risk Assessment
Risk Level: 🟢 Low
Impacted Components
projects/miopen/src/mlo_dir_conv.cpp— two one-line removals from the Winograd Find solver listsprojects/miopen/src/include/miopen/mlo_internal.hpp— new exported declaration forGetAllFindSolverDbIds()projects/miopen/test/gtest/solver_find_list_registry.cpp— new CPU-only gtest; no production code pathPotential Side Effects
Removing
TransposedConvWinoRageRxS<2,3>from the Find lists means Find will never select it. This is intentional and was the desired outcome of PR #6196: the Rage kernels were reverted because they cause memory-access faults. Any workload that was silently falling back to another solver (due to applicability checks rejecting the Rage solver) will continue to do so unchanged. No performance regression is expected because the Rage solver was not usable in practice.Mitigation Steps
Test Plan
CPU_SolverFindListRegistry_NONE.AllFindSolversAreRegistered): builds without GPU, confirms every solver reachable through Find is registered and thatGetAlgo()does not throw.MIOPEN_FIND_MODE=NORMAL:Test Result
TransposedConvWinoRageRxS<2-3>as the unregistered solver, and PASSES after the fix.MIOpenDriverrepro command goes from exiting with code 5 (crash atsolver.cpp:198) to"Forward Convolution Verifies OK on GPU reference"(exit 0) after the fix.🤖 Generated with Claude Code