Repository navigation
[MIOpen] Fix use-after-free of kernel_id in InitInvokerFactoryNCHW - #7922
Merged
Merged
Conversation
InitInvokerFactoryNCHW captured the kernel_id string parameter by
pointer (kernel_id_ = &kernel_id) instead of by value. The
invoker_factory and its nested invoker lambda are stored in the
returned ConvSolution and executed long after InitInvokerFactoryNCHW
returns, by which point the referent of the `const std::string&
kernel_id` parameter has been destroyed. At invoke time the dangling
pointer was dereferenced via AddKernelToJsonAccumulator(*kernel_id2,
...) (gated on profiling + IsLoggingKernel), reading a freed
std::string and crashing with SIGSEGV. The accompanying
MIOPEN_LOG_I2("kernel_name = " << kernel_id2) also printed the stale
pointer address instead of the kernel name.
All three sibling invoker factories in this file already capture
kernel_id by value; only the NCHW forward path used a pointer (and the
earlier dereference-error fix corrected the other paths but missed this
one). Capture kernel_id by value here too so the lambda owns the string
for its full lifetime, drop the now-unnecessary dereference, and let the
log print the actual kernel name.
This is reached whenever an NCHW grouped CK xdlops convolution is
executed with kernel logging enabled (e.g. MIOPEN_PERFORMANCE_LOGS),
independent of data type or tensor size.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
BrianHarrisonAMD
approved these changes
Jun 1, 2026
BrianHarrisonAMD
left a comment
Contributor
There was a problem hiding this comment.
LGTM.
Minor comments, but I don't think it's blocking.
iq136boy
approved these changes
Jun 1, 2026
JonathanLichtnerAMD
approved these changes
Jun 1, 2026
JonathanLichtnerAMD
left a comment
Contributor
There was a problem hiding this comment.
LGTM. Would it be worth adding a test for this, i.e., it could either reproduce the crash or ASAN could detect the issue? I'll leave that as optional but having a test means that ASAN and other sanitizers will validate the code path against future changes.
yassinsolim
approved these changes
Jun 2, 2026
yassinsolim
left a comment
Member
There was a problem hiding this comment.
LGTM. The lifetime fix is correct: kernel_id is now owned by the factory/invoker instead of captured by pointer to a local. Brian’s move-consistency comments seem reasonable but non-blocking to me.
JonathanLichtnerAMD
added a commit
that referenced
this pull request
Jul 15, 2026
JIRA ID: ALMIOPEN-2312 ## Motivation `projects/miopen/CHANGELOG.md`'s "MIOpen 3.5.2 for ROCm 7.14.0" section only had 4 entries, but 68 commits touched `projects/miopen` between the `therock-7.13` tag and `release/therock-7.14`. Auditing that commit range (via `git log --oneline therock-7.13..remotes/origin/release/therock-7.14 -- projects/miopen`) found 18 additional user-facing changes — new gfx950/gfx1250 architecture support, SystemDB refreshes, AI-heuristic improvements, several correctness/crash bug fixes, and OpenCL solver removals — that were never documented. This PR backfills those entries so the 7.14.0 changelog accurately reflects what shipped. ## Technical Details Added 24 lines to the existing "MIOpen 3.5.2 for ROCm 7.14.0" section of `projects/miopen/CHANGELOG.md`, merging new bullets into the existing `Added`/`Changed`/`Resolved Issues` sub-sections (no new entries were duplicated or reworded) and adding a new `Removed` sub-section for solver removals. No existing content was altered. Each new bullet was traced to a specific commit + PR: | Entry | Commit | PR | |---|---|---| | Added gfx950 (MI350X/MI355X) SystemDB | `60d8e1e5175` | #8081 | | Added initial gfx1250 architecture support | `d3f057b2f22` | #7587 | | Added JSON-formatted performance logs (`MIOPEN_PERFORMANCE_LOGS`) | `ffbde871260` | #5736 | | Refreshed gfx942 SystemDB | `faf754a9ef6` | #8174 | | Refined 2D AI heuristics (Tunanet/KTN) for gfx942 & gfx950 | `c24ad415884` | #7109 | | Fixed thread-safety issue in AI-heuristic model caches | `cd957402309` | #9149 (cherry-pick of #9131) | | Fixed CK grouped-conv solvers not always registered in host | `c971d54d7ae` | #9105 | | Fixed legacy CK buffer resource descriptor (gfx1250) | `1dee011b8b0` | #8758 | | Removed stale gfx942 SystemDB naive-solver entries | `f42668f6316` | #7028 | | Fixed softmax NaN when `beta=0` | `fb8157c8512` | #8426 | | Fixed BN backward spatial variant 1 vectorized memory access bug | `1351a7e8b10` | #6765 | | Fixed use-after-free of `kernel_id` in `InitInvokerFactoryNCHW` | `394ebffb50d` | #7922 | | Fixed SoftMarginLoss forward reduce-kernel kernarg corruption | `1e65a31570f` | #8065, #8135 | | Fixed build with `MIOPEN_ENABLE_AI_KERNEL_TUNING=OFF` | `e6288c3ca61` | #6800 | | Fixed install RPATH for MIOpenDriver / CK backend libs | `ef8a4cf2f93` | #7485 | | Fixed dangling lambda causing log-level 6 segfault | `190f9791737` | #6953 | | Enabled Windows long-path support for `addkernels` | `1f9d69724cf` | #7387 | | Removed OCL solver `ConvOclDirectFwd1x1` | `126c720cadd` | #6491 | | Removed OCL solver `conv_ocl_dir2Dfwdgen` | `2d499ddf709` | #6633 | | Removed OCL solvers `ConvOclBwdWrW1x1`, `ConvOclBwdWrW2` | `b5d43a33dc6` | #6859 | | Removed OCL solvers `OpTensorLeadingOnes`, `Op5dTensorGeneric` | `8642c83b7de` | #7347 | The remaining ~40 commits in the 68-commit range were deliberately excluded (tests/CI/build/docs/internal refactors/1:1 OCL→HIP swaps with no behavior change, per each commit's own stated scope) and are not represented here. ## Risk Assessment **Risk Level: 🟢 Low** This is a documentation-only change to a single Markdown file. No code, build configuration, tests, or public API surface is touched. ### Impacted Components - `projects/miopen/CHANGELOG.md` only. ### Potential Side Effects None expected. This does not change any runtime behavior; it only documents changes that already shipped on `release/therock-7.14`. ### Mitigation Steps None required beyond normal review — a reviewer familiar with the 7.14 release can spot-check a few entries against the linked commit hashes/PR numbers for accuracy. ## Test Plan Not applicable (no code changes). Verification was done by cross-referencing every added bullet against a specific commit hash and PR number pulled from `git log`, and by diffing the edited section against `git show remotes/origin/release/therock-7.14:projects/miopen/CHANGELOG.md` to confirm no existing entry was duplicated or altered. ## Test Result All 21 new bullets map 1:1 to a real commit/PR in the `therock-7.13..release/therock-7.14` range for `projects/miopen`. No pre-existing changelog text was modified — `git diff` shows a pure addition (24 insertions, 0 deletions). Two borderline commits (`a67834413e8` "Standalone build", `7c7cd5774a9` "enable warnings about deprecated API/features") were deliberately left out pending author judgment on whether they warrant an entry.
pschang-phy
pushed a commit
that referenced
this pull request
Jul 17, 2026
JIRA ID: ALMIOPEN-2312 ## Motivation `projects/miopen/CHANGELOG.md`'s "MIOpen 3.5.2 for ROCm 7.14.0" section only had 4 entries, but 68 commits touched `projects/miopen` between the `therock-7.13` tag and `release/therock-7.14`. Auditing that commit range (via `git log --oneline therock-7.13..remotes/origin/release/therock-7.14 -- projects/miopen`) found 18 additional user-facing changes — new gfx950/gfx1250 architecture support, SystemDB refreshes, AI-heuristic improvements, several correctness/crash bug fixes, and OpenCL solver removals — that were never documented. This PR backfills those entries so the 7.14.0 changelog accurately reflects what shipped. ## Technical Details Added 24 lines to the existing "MIOpen 3.5.2 for ROCm 7.14.0" section of `projects/miopen/CHANGELOG.md`, merging new bullets into the existing `Added`/`Changed`/`Resolved Issues` sub-sections (no new entries were duplicated or reworded) and adding a new `Removed` sub-section for solver removals. No existing content was altered. Each new bullet was traced to a specific commit + PR: | Entry | Commit | PR | |---|---|---| | Added gfx950 (MI350X/MI355X) SystemDB | `60d8e1e5175` | #8081 | | Added initial gfx1250 architecture support | `d3f057b2f22` | #7587 | | Added JSON-formatted performance logs (`MIOPEN_PERFORMANCE_LOGS`) | `ffbde871260` | #5736 | | Refreshed gfx942 SystemDB | `faf754a9ef6` | #8174 | | Refined 2D AI heuristics (Tunanet/KTN) for gfx942 & gfx950 | `c24ad415884` | #7109 | | Fixed thread-safety issue in AI-heuristic model caches | `cd957402309` | #9149 (cherry-pick of #9131) | | Fixed CK grouped-conv solvers not always registered in host | `c971d54d7ae` | #9105 | | Fixed legacy CK buffer resource descriptor (gfx1250) | `1dee011b8b0` | #8758 | | Removed stale gfx942 SystemDB naive-solver entries | `f42668f6316` | #7028 | | Fixed softmax NaN when `beta=0` | `fb8157c8512` | #8426 | | Fixed BN backward spatial variant 1 vectorized memory access bug | `1351a7e8b10` | #6765 | | Fixed use-after-free of `kernel_id` in `InitInvokerFactoryNCHW` | `394ebffb50d` | #7922 | | Fixed SoftMarginLoss forward reduce-kernel kernarg corruption | `1e65a31570f` | #8065, #8135 | | Fixed build with `MIOPEN_ENABLE_AI_KERNEL_TUNING=OFF` | `e6288c3ca61` | #6800 | | Fixed install RPATH for MIOpenDriver / CK backend libs | `ef8a4cf2f93` | #7485 | | Fixed dangling lambda causing log-level 6 segfault | `190f9791737` | #6953 | | Enabled Windows long-path support for `addkernels` | `1f9d69724cf` | #7387 | | Removed OCL solver `ConvOclDirectFwd1x1` | `126c720cadd` | #6491 | | Removed OCL solver `conv_ocl_dir2Dfwdgen` | `2d499ddf709` | #6633 | | Removed OCL solvers `ConvOclBwdWrW1x1`, `ConvOclBwdWrW2` | `b5d43a33dc6` | #6859 | | Removed OCL solvers `OpTensorLeadingOnes`, `Op5dTensorGeneric` | `8642c83b7de` | #7347 | The remaining ~40 commits in the 68-commit range were deliberately excluded (tests/CI/build/docs/internal refactors/1:1 OCL→HIP swaps with no behavior change, per each commit's own stated scope) and are not represented here. ## Risk Assessment **Risk Level: 🟢 Low** This is a documentation-only change to a single Markdown file. No code, build configuration, tests, or public API surface is touched. ### Impacted Components - `projects/miopen/CHANGELOG.md` only. ### Potential Side Effects None expected. This does not change any runtime behavior; it only documents changes that already shipped on `release/therock-7.14`. ### Mitigation Steps None required beyond normal review — a reviewer familiar with the 7.14 release can spot-check a few entries against the linked commit hashes/PR numbers for accuracy. ## Test Plan Not applicable (no code changes). Verification was done by cross-referencing every added bullet against a specific commit hash and PR number pulled from `git log`, and by diffing the edited section against `git show remotes/origin/release/therock-7.14:projects/miopen/CHANGELOG.md` to confirm no existing entry was duplicated or altered. ## Test Result All 21 new bullets map 1:1 to a real commit/PR in the `therock-7.13..release/therock-7.14` range for `projects/miopen`. No pre-existing changelog text was modified — `git diff` shows a pure addition (24 insertions, 0 deletions). Two borderline commits (`a67834413e8` "Standalone build", `7c7cd5774a9` "enable warnings about deprecated API/features") were deliberately left out pending author judgment on whether they warrant an entry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Running any NCHW grouped CK xdlops convolution with kernel-name logging enabled (e.g.
MIOPEN_PERFORMANCE_LOGS=4) deterministically SIGSEGVs insideMIOpen's invoker. This was originally surfaced as a CK / large-tensor regression in a perf-eval run, but root-causing showed the fault is host-side in
MIOpen and reproduces independent of data type or tensor size — any NCHW grouped CK conv that runs
AddKernelToJsonAccumulatorwill crash.This PR fixes the use-after-free so the NCHW invoker matches the by-value capture used by the three sibling invoker factories in the same file.
Technical Details
In
projects/miopen/src/ck_impl/implicitgemm_ck_util.hpp,InitInvokerFactoryNCHWcaptured itsconst std::string& kernel_idparameter bypointer:
The
invoker_factory(and its nested invoker lambda) are stored in the returnedConvSolutionand run long afterInitInvokerFactoryNCHWhasreturned, by which time the referent of the
kernel_idparameter has been destroyed. The deref is gated onIsProfilingEnabled() && IsLoggingKernel(), so it fires whenever kernel-name JSON logging is on during execution.The other three invoker factories in this file (NHWC fwd, NHWC bwd/wrw) already capture
kernel_idby value (lines 690, 1138, 1210). An earlierdereference-error fix corrected the
*kernel_id2deref in some sibling paths but missed this NCHW one.This 3-line change:
kernel_idby value:kernel_id_ = kernel_id(line 981).kernel_id2 = std::move(kernel_id_)(line 991).AddKernelToJsonAccumulator(kernel_id2, elapsed, false)(line 1079).As a beneficial side-effect,
MIOPEN_LOG_I2("kernel_name = " << kernel_id2)now prints the actual kernel name instead of the stale pointer address.The remaining
&kernel_idcapture inFindConvPtrByID(line 165) is a synchronous, non-escapingstd::find_iflambda and is safe.Test Plan
MIOpenDriver conv -n 1 -c 512 -H 912 -W 1144 -k 512 -y 3 -x 3 -p 1 -q 1 -u 1 -v 1 -l 1 -j 1 -m conv -g 1 -F 1 -t 1 -iter 10) withMIOPEN_FIND_MODE=1 MIOPEN_FIND_ENFORCE=4 MIOPEN_LOG_LEVEL=7 MIOPEN_PERFORMANCE_LOGS=4on gfx950 and confirm it no longerSIGSEGVs.
MIOPEN_LOG_I2("kernel_name = …")log line now prints the kernel name string (e.g.DeviceGroupedConvFwdMultipleABD_Xdl_CShuffle<...>) rather than a0x7fff…stack address.Test Result
*kernel_id2dereferences and no escaping&kernel_idcaptures inimplicitgemm_ck_util.hpp; all fourinvoker factories now capture
kernel_idby value.Risk Assessment
Risk level: Low
projects/miopen/src/ck_impl/implicitgemm_ck_util.hpp, specifically the NCHW invoker-factory path used by theConvHipImplicitGemmGroup{Fwd,Bwd,Wrw}XdlopsandConvHipImplicitGemm3DGroup{Fwd,Bwd,Wrw}Xdlopssolvers when the problem is in default (NCHW/NCDHW)layout. No CK code, no kernel code, no public API changes. Other MIOpen layouts and solvers are unaffected.
a stale address.
std::stringcopy perInitInvokerFactoryNCHWcall (a one-time setup cost during solution build), then astd::moveintothe inner lambda — negligible vs the kernel launch and CK arg setup cost. Per-invocation cost on the hot path is unchanged.
review surface is small.
MIOPEN_PERFORMANCE_LOGSon any NCHW grouped CK conv, making the fix easy to verify pre-merge.Submission Checklist