Skip to content
Merged
Show file tree
Hide file tree
Changes from 42 commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
84ca11a
Log kernel names when environment variables present
jdcampbe Feb 4, 2026
5d18a2f
JSON Logs
jdcampbe Feb 5, 2026
9cd0d40
Fix Typos
jdcampbe Feb 5, 2026
2677ef7
Fix Verbose Logging
jdcampbe Feb 6, 2026
3bd39f3
Compiler Fixes
jdcampbe Feb 6, 2026
349f14b
Add tuning flag to evaluate invokers
jdcampbe Feb 6, 2026
f90af96
Add Verification to Tuning
jdcampbe Feb 6, 2026
00aee5e
Update variable naming.
jdcampbe Feb 9, 2026
ad0cb4f
Add missing logging for WRW BWD
jdcampbe Feb 11, 2026
a59f311
Log find time in forward
jdcampbe Feb 12, 2026
bbe03ab
Find time
jdcampbe Feb 12, 2026
18a5248
Update variable name
jdcampbe Feb 12, 2026
96a210a
Add DB Key to performance logs
jdcampbe Mar 3, 2026
3afa61c
Update performance logs solution to include solution id
jdcampbe Mar 3, 2026
5dbaea8
Merge branch 'develop' into users/jascampb/miopen-log-kernel-names
jdcampbe Mar 3, 2026
1822993
Fix function arguments mismatch
jdcampbe Mar 3, 2026
ac0b54c
Fix scoping error
jdcampbe Mar 3, 2026
64ed4e1
Update db key to match system db
jdcampbe Mar 4, 2026
188d616
Log solution if solution id changes
jdcampbe Mar 4, 2026
4ea7785
Set time on if performance logs are active.
jdcampbe Mar 4, 2026
b3a6a1c
Set timing in different location so it is not overwritten
jdcampbe Mar 4, 2026
42875f7
Add Performance Configs
jdcampbe Mar 7, 2026
eab71b7
Merge branch 'develop' into users/jascampb/miopen-log-kernel-names
jdcampbe Mar 7, 2026
ecef3f6
Fix compile issues and missing solutions.
jdcampbe Mar 10, 2026
2fd2748
Add more complex phases to performance logs
jdcampbe Mar 11, 2026
7acda22
Modify Phase Tuning
jdcampbe Mar 13, 2026
031f950
Fix compiler errors
jdcampbe Mar 13, 2026
1b71d3c
Fix kernels not showing up
jdcampbe Mar 14, 2026
8b471de
Remove duplicate code
jdcampbe Mar 14, 2026
191e5c1
Merge branch 'develop' into users/jascampb/miopen-log-kernel-names
jdcampbe Mar 23, 2026
0e58662
Merge branch 'develop' into users/jascampb/miopen-log-kernel-names
jdcampbe Mar 23, 2026
43f18cc
Update readme
jdcampbe Mar 23, 2026
ffce808
Merge branch 'develop' into users/jascampb/miopen-log-kernel-names
jdcampbe Mar 24, 2026
ce71228
Add workspace size to logging
jdcampbe Mar 24, 2026
c0119c4
Treat PR issues
jdcampbe Mar 27, 2026
8b9c229
Merge branch 'develop' into users/jascampb/miopen-log-kernel-names
jdcampbe Mar 28, 2026
1feeda1
Clang formatting.
jdcampbe Mar 31, 2026
e4f24e5
Merge branch 'develop' into users/jascampb/miopen-log-kernel-names
jdcampbe Mar 31, 2026
d535ec9
Reorder invoker times gathering to avoid the changes to the sample list.
jdcampbe Apr 1, 2026
6a0867e
Merge branch 'develop' into users/jascampb/miopen-log-kernel-names
jdcampbe Apr 1, 2026
304a220
Merge branch 'develop' into users/jascampb/miopen-log-kernel-names
jdcampbe Apr 2, 2026
d7916ba
Merge branch 'develop' into users/jascampb/miopen-log-kernel-names
jdcampbe Apr 7, 2026
16a17c9
Update Perf DB Key
jdcampbe Apr 9, 2026
3dc5cd0
Merge remote-tracking branch 'origin/develop' into users/jascampb/mio…
jdcampbe Apr 27, 2026
d90e22f
Match kernel name.
jdcampbe Apr 28, 2026
1dc674f
Merge branch 'develop' into users/jascampb/miopen-log-kernel-names
jdcampbe Apr 30, 2026
70096de
Fix dereferencing error
jdcampbe Apr 30, 2026
f02f139
Apply clang-format to conv_driver.hpp
jdcampbe Apr 30, 2026
00a8aba
Removed unused variables from json call site.
jdcampbe Apr 30, 2026
b0e2b8f
Merge branch 'develop' into users/jascampb/miopen-log-kernel-names
jdcampbe May 4, 2026
7260944
Merge branch 'develop' into users/jascampb/miopen-log-kernel-names
jdcampbe May 6, 2026
0373345
Fix tidy errors on kernel tuning mode
jdcampbe May 8, 2026
b2cedd2
Fix clang tidy.
jdcampbe May 8, 2026
7fa1fb0
Merge branch 'develop' into users/jascampb/miopen-log-kernel-names
jdcampbe May 8, 2026
6a2b7e0
Merge branch 'develop' into users/jascampb/miopen-log-kernel-names
jdcampbe May 9, 2026
34decf0
Merge branch 'develop' into users/jascampb/miopen-log-kernel-names
jdcampbe May 11, 2026
0264aa3
Merge branch 'develop' into users/jascampb/miopen-log-kernel-names
jdcampbe May 12, 2026
441fd7a
Merge branch 'develop' into users/jascampb/miopen-log-kernel-names
jdcampbe May 13, 2026
ff01136
Merge branch 'develop' into users/jascampb/miopen-log-kernel-names
jdcampbe May 14, 2026
c700688
Merge branch 'develop' into users/jascampb/miopen-log-kernel-names
jdcampbe May 14, 2026
9db59a1
Merge branch 'develop' into users/jascampb/miopen-log-kernel-names
jdcampbe May 25, 2026
e30bd85
Add gtest for MIOPEN_PERFORMANCE_LOGS kernel logging
jdcampbe May 25, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 86 additions & 0 deletions projects/miopen/driver/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,3 +147,89 @@ MIOpenDriver example usages:
`./bin/MIOpenDriver *base_arg* -?` **OR** `./bin/MIOpenDriver *base_arg* -h (--help)`

Note: By default the CPU verification is turned on. Verification can be disabled using `-V 0`.

## Environment Variables

### Kernel Name and Execution Time Logging

The `MIOPEN_PERFORMANCE_LOGS` environment variable enables lightweight logging of kernel names and their execution times during MIOpenDriver runs. This is useful for debugging, performance analysis, and understanding which kernels are being executed under different configurations (e.g., different `MIOPEN_FIND_MODE` or `MIOPEN_FORCE` settings). It is meant for usage only with MIOpen driver commands and is untested for direct library usage.

**Logging Levels:**

The variable supports five levels with varying detail and scope:

- **Level 0** (default): No kernel logging
- **Level 1**: Log only executed solution with **total solution time** - excludes find/search
- **Level 2**: Log **all kernels** for executed solutions individually - includes transpose/transform kernels, excludes find/search
- **Level 3**: Log only **performance configs** per solution with **total solution time** - includes find/search
- **Level 4**: Log **all kernels** individually - includes transpose/transform kernels and find/search kernels

**Usage Examples:**

```bash
# Level 4: Log all available kernel/performance config/solution information
export MIOPEN_PERFORMANCE_LOGS=4
./bin/MIOpenDriver conv -W 32 -H 32 -c 3 -k 32 -x 5 -y 5 -p 2 -q 2
```

**JSON Output Format:**

Each solution outputs a JSON object with performance configs. Each config contains aggregated timing statistics and optionally individual kernel execution data:

```json
{
"solution": "ConvAsmImplicitGemmGTCDynamicFwdXdlopsNHWC",
"solver_id": 1234567890,
"phase": "execution",
"performance_configs": [
{
"config_name": "igemm_fwd_gtcx3_nhwc_bf16_bx0_ex1_bt128x128x32...",
"config_descriptor": "gemm_m_per_block=128 gemm_n_per_block=128...",
"exec_number": 1,
"time_executions_ms": [1.52, 1.49, 1.41, 1.38],
"time_ms": 1.42,
"time_std_ms": 0.056,
"time_min_ms": 1.38,
"time_max_ms": 1.52,
"number_of_transformations": 4,
"kernels": [
{
"kernel_name": "SubTensorOpWithScalar1d",
"time_executions_ms": [0.083, 0.104, 0.089],
"is_transformation": true
},
{
"kernel_name": "igemm_fwd_gtcx3_nhwc_bf16_bx0_ex1_bt128x128x32...",
"time_executions_ms": [0.893, 0.763, 0.724],
"is_transformation": false
}
]
}
]
}
```

**JSON Fields:**

**Solution Level:**
- `solution`: Name of the solver/algorithm
- `solver_id`: Numeric identifier for the solver
- `phase`: Either "execution" (actual computation) or "tuning" (find/search phase), "validation", "solver_tuning", or "unknown"
- `performance_configs`: Array of performance configuration results

**Performance Config Level:**
- `config_name`: Name of the main kernel or solution
- `config_descriptor`: Performance config parameters (optional)
- `exec_number`: Execution counter (starts at 1)
- `time_executions_ms`: Array of all execution times for this config
- `time_ms`: Mean execution time (computed using outlier removal)
- `time_std_ms`: Standard deviation of execution times
- `time_min_ms`: Minimum execution time
- `time_max_ms`: Maximum execution time
- `number_of_transformations`: Count of transformation/transpose kernels
- `kernels`: Array of individual kernel data (only present in levels 2 & 4, null in levels 1 & 3)

**Kernel Level (when included):**
- `kernel_name`: Full kernel name
- `time_executions_ms`: Array of execution times across multiple runs
- `is_transformation`: Boolean indicating if kernel is a transpose/transform operation
1,224 changes: 970 additions & 254 deletions projects/miopen/driver/conv_driver.hpp

Large diffs are not rendered by default.

135 changes: 97 additions & 38 deletions projects/miopen/driver/main.cpp
Original file line number Diff line number Diff line change
@@ -1,28 +1,5 @@
/*******************************************************************************
*
* MIT License
*
* Copyright (c) 2017 Advanced Micro Devices, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
*******************************************************************************/
// Copyright (c) Advanced Micro Devices, Inc., or its affiliates.
// SPDX-License-Identifier: MIT
#include "driver.hpp"
#include "registry_driver_maker.hpp"

Expand All @@ -48,13 +25,25 @@ int main(int argc, char* argv[])
exit(0); // NOLINT (concurrency-mt-unsafe)
}

const bool json_mode = miopen::IsPerformanceLoggingEnabled();
try
{

// show command
std::cout << "MIOpenDriver";
for(int i = 1; i < argc; i++)
std::cout << " " << argv[i];
std::cout << std::endl;
if(!json_mode)
{
std::cout << "MIOpenDriver";
for(int i = 1; i < argc; i++)
std::cout << " " << argv[i];
std::cout << std::endl;
}
else
{
std::cout << "{\"command\":\"MIOpenDriver";
for(int i = 1; i < argc; i++)
std::cout << " " << argv[i];
std::cout << "\"}" << std::endl;
}

std::shared_ptr<Driver> drv;
for(auto f : rdm::GetRegistry())
Expand All @@ -65,7 +54,15 @@ int main(int argc, char* argv[])
}
if(drv == nullptr)
{
MIOPEN_THROW(miopenStatusBadParm, "Incorrect BaseArg");
if(!json_mode)
{
MIOPEN_THROW(miopenStatusBadParm, "Incorrect BaseArg");
}
else
{
std::cout << "{\"error\":\"Incorrect BaseArg\"}" << std::endl;
exit(0); // NOLINT (concurrency-mt-unsafe)
}
}

drv->name = base_arg;
Expand All @@ -74,14 +71,33 @@ int main(int argc, char* argv[])
int rc = drv->ParseCmdLineArgs(argc, argv);
if(rc != 0)
{
std::cout << "ParseCmdLineArgs() FAILED, rc = " << rc << std::endl;
if(!json_mode)
{
std::cout << "ParseCmdLineArgs() FAILED, rc = " << rc << std::endl;
}
else
{
std::cout << "{\"error\":\"ParseCmdLineArgs() FAILED, rc = " << rc << "\"}"
<< std::endl;
}
return rc;
}

drv->GetandSetData();

rc = drv->AllocateBuffersAndCopy();

if(rc != 0)
{
std::cout << "AllocateBuffersAndCopy() FAILED, rc = " << rc << std::endl;
if(!json_mode)
{
std::cout << "AllocateBuffersAndCopy() FAILED, rc = " << rc << std::endl;
}
else
{
std::cout << "{\"error\":\"AllocateBuffersAndCopy() FAILED, rc = " << rc << "\"}"
<< std::endl;
}
return rc;
}

Expand All @@ -108,8 +124,18 @@ int main(int argc, char* argv[])
rc = drv->RunForwardGPU();
cumulative_rc |= rc;
if(rc != 0)
std::cout << "RunForwardGPU() FAILED, rc = " << "0x" << std::hex << rc << std::dec
<< std::endl;
{
if(!json_mode)
{
std::cout << "RunForwardGPU() FAILED, rc = " << "0x" << std::hex << rc
<< std::dec << std::endl;
}
else
{
std::cout << "{\"error\":\"RunForwardGPU() FAILED, rc = 0x" << std::hex << rc
<< std::dec << "\"}" << std::endl;
}
}
if(verifyarg) // Verify even if Run() failed.
cumulative_rc |= drv->VerifyForward();
}
Expand All @@ -119,22 +145,55 @@ int main(int argc, char* argv[])
rc = drv->RunBackwardGPU();
cumulative_rc |= rc;
if(rc != 0)
std::cout << "RunBackwardGPU() FAILED, rc = " << "0x" << std::hex << rc << std::dec
<< std::endl;
{
if(!json_mode)
{
std::cout << "RunBackwardGPU() FAILED, rc = " << "0x" << std::hex << rc
<< std::dec << std::endl;
}
else
{
std::cout << "{\"error\":\"RunBackwardGPU() FAILED, rc = 0x" << std::hex << rc
<< std::dec << "\"}" << std::endl;
}
}
if(verifyarg) // Verify even if Run() failed.
cumulative_rc |= drv->VerifyBackward();
}

// Flush any accumulated performance logging data before exiting
miopen::FinalizeJsonLogging();

return cumulative_rc;
}
catch(const miopen::Exception& ex)
{
std::cerr << "Error: " << ex.what() << std::endl;
// Flush any accumulated performance logging data before exiting
miopen::FinalizeJsonLogging();

if(!json_mode)
{
std::cerr << "Error: " << ex.what() << std::endl;
}
else
{
std::cout << "{\"error\":\"" << ex.what() << "\"}" << std::endl;
}
return EXIT_FAILURE;
}
catch(const std::exception& ex)
{
std::cerr << "Error: " << ex.what() << std::endl;
// Flush any accumulated performance logging data before exiting
miopen::FinalizeJsonLogging();

if(!json_mode)
{
std::cerr << "Error: " << ex.what() << std::endl;
}
else
{
std::cout << "{\"error\":\"" << ex.what() << "\"}" << std::endl;
}
return EXIT_FAILURE;
}
}
39 changes: 13 additions & 26 deletions projects/miopen/driver/random.hpp
Original file line number Diff line number Diff line change
@@ -1,32 +1,10 @@
/*******************************************************************************
*
* MIT License
*
* Copyright (c) 2025 Advanced Micro Devices, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
*******************************************************************************/
// Copyright (c) Advanced Micro Devices, Inc., or its affiliates.
// SPDX-License-Identifier: MIT
#ifndef GUARD_RANDOM_GEN_
#define GUARD_RANDOM_GEN_

#include <miopen/env.hpp>
#include <miopen/kernel_tuning_mode.hpp>

#include <cassert>
#include <iostream>
Expand All @@ -48,7 +26,16 @@ inline std::random_device::result_type get_default_seed()
auto seed = external_seed == 0
? std::random_device{}()
: static_cast<std::random_device::result_type>(external_seed);
std::cout << "PRNG seed: " << seed << "\n";

// Check if JSON mode is enabled in MIOPEN_PERFORMANCE_LOGS
if(miopen::IsPerformanceLoggingEnabled())
{
std::cout << "{\"prng_seed\":" << seed << "}" << std::endl;
}
else
{
std::cout << "PRNG seed: " << seed << "\n";
}
return seed;
}()};
return seed;
Expand Down
Loading
Loading