[amd-ahyttine]

Motivation

This PR fixes "Segmentation fault (core dumped)" when running MIOpenDriver. Example:

"MIOPEN_FIND_MODE=2 MIOPEN_FIND_ENFORCE=1 MIOPEN_LOG_LEVEL=6 ./MIOpenDriver convfp16 --spatial_dim 2 --in_channels 24 --out_channels 12 --batchsize 4 --in_layout NHWC --out_layout NHWC --fil_layout NHWC --group_count 1 --forw 4 --pad_h 0 --pad_w 0 --in_h 112 --in_w 112 --fil_h 1 --fil_w 1 --dilation_h 1 --dilation_w 1 --conv_stride_h 1 --conv_stride_w 1 --mode conv --search 1 --time 1".

However when MIOPEN_LOG_LEVEL=5 it runs fine, no segmentation fault.

The problem is limited to NHWC layout, direction backwards weights and MIOPEN_LOG_LEVEL>=6.

Technical Details

The level-6 run goes much deeper into Info2 logging and appears to crash during candidate-selection debug logging.

Gdb confirms a host-side crash in memcpy while writing to an output stream, inside libMIOpenCKGroupedConv_gfx950.so during backward-weights invocation. That strongly points to a bad string/logging path in the CK grouped-conv runtime.

The root bug: one invoker lambda captures kernel_id by reference to a temporary outer lambda capture, so at log level 6 (Info2) the kernel_name log dereferences a dangling string and crashes in memcpy. I’m applying a one-line fix to capture that string by value.

                                 const std::vector<Kernel>&) mutable {
        return [&kernel_id2                  = kernel_id_,
        return [kernel_id2                   = kernel_id_,
                split_k2                     = split_k_,

Crash cause is a dangling reference in a log-only path.

In src/ck_impl/implicitgemm_ck_util.hpp, the NHWC WrW invoker lambda captured kernel_id like this: &kernel_id2 = kernel_id_.
That reference points to an outer lambda capture that no longer exists when the invoker runs later.
At MIOPEN_LOG_LEVEL=6, MIOPEN_LOG_I2("kernel_name = " << kernel_id2) tries to print that dangling string, causing a host segfault (memcpy in std::ostream), which matches your gdb backtrace.
At MIOPEN_LOG_LEVEL=5, that Info2 log is not executed, so the bug stays hidden.
I fixed it by capturing kernel_id2 by value instead of by reference:

Updated src/ck_impl/implicitgemm_ck_util.hpp:
return [&kernel_id2 = kernel_id_, ...]
to return [kernel_id2 = kernel_id_, ...]

Note that other occurrences of kernel_id2 in the file have the correct "kernel_id2= kernel_id_" version, instead of this faulty "&kernel_id2 = kernel_id_" version.

Test Plan

As validation test the command mentioned in the motivation (by AI).

Also rebuild MIOPenDriver and running 1000 command performance comparison that before the fix resulted in several segfaults and other crashes on MIOPEN_LOG_LEVEL=6.

Test Result

Validation done:
Rebuilt MIOpenDriver and libMIOpenCKGroupedConv_gfx950.so.
Re-ran your exact reproducer with MIOPEN_LOG_LEVEL=6.
Result: no segfault, run completes and verifies OK.

Rebuilt MIOPenDriver manually, all 1000 MIOpenDriver commands run now succesfully.

Submission Checklist

• [v] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

[vpietila-amd]

Looks good to me. Nice catch.

[JonathanLichtnerAMD]

LGTM