Skip to content

[MIOpen] Add gfx950 SystemDB for MI350X/MI355X - #8081

Merged
jdcampbe merged 9 commits into
developfrom
users/jascampb/systemdb-update-gfx950-05-2026
Jun 22, 2026
Merged

jdcampbe merged 9 commits into
developfrom
users/jascampb/systemdb-update-gfx950-05-2026

Conversation

@jdcampbe

@jdcampbe jdcampbe commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Motivation

MIOpen ships per-arch find/perf databases under src/kernels/, but no gfx950 SystemDB was packaged. Without it, gfx950 runs fall back to runtime tuning on every invocation. MI350X and MI355X both expose 256 CUs, so a single gfx950100 basename serves both per Handle::GetDbBasename() (hex(256) = 0x100, and the _<cu> suffix is omitted for >64 CU).

Technical Details

  • Adds projects/miopen/src/kernels/gfx950100.HIP.fdb.txt.bz2 and projects/miopen/src/kernels/gfx950100.db.txt.bz2.
  • Adds gfx950 to ALL_GPU_DATABASES in projects/miopen/CMakeLists.txt so install_gpu_databases() picks up the new bz2 files via the existing GLOB pattern.
  • No code changes outside the CMake list entry; basename selection, loader, and install paths are unchanged.

Test Plan

  • dbsync-test StaticFDBSync/5 round-trip on the staged find DB.
  • Confirm install_gpu_databases() packages both gfx950100.*.bz2 blobs in the install tree.
  • Smoke MIOpenDriver convolution runs on MI350X / MI355X to confirm the DBs are loaded (no fallback to runtime tuning for shapes present
    in the DB).

Test Result

dbsync-test StaticFDBSync/5 passes on the staged find DB. Runtime smoke on MI350X/MI355X pending; will update with results.

Risk Assessment

Risk level: Low

  • Impacted components: projects/miopen/src/kernels/gfx950100.HIP.fdb.txt.bz2, projects/miopen/src/kernels/gfx950100.db.txt.bz2,
    and the ALL_GPU_DATABASES list in projects/miopen/CMakeLists.txt. No source / kernel / solver code is touched. Other archs (gfx942,
    gfx90a, etc.) are unaffected.
    gfx90a, etc.) are unaffected.
  • Potential side effects:
    • Functional: gfx950 (MI350X/MI355X) now finds tuned entries instead of falling back to runtime search — strictly additive for those
      archs.
    • Performance: removes per-run tuning cost on covered shapes; no hot-path code change.
    • ABI/API: none.
  • Mitigation steps:
    • Change is data + one CMake list entry; trivially revertible by dropping the two bz2 files and the list entry.
    • Find DB was validated via the existing dbsync-test round-trip before staging.
    • No other arch's DB or install logic is affected.

Submission Checklist

jdcampbe and others added 2 commits May 25, 2026 18:24
Stages the cleaned find/perf DB pair for gfx950 (transposed Wino solvers
stripped, validated via dbsync-test StaticFDBSync/5). Both MI350X and
MI355X have 256 CUs, so a single basename gfx950100 covers both per
Handle::GetDbBasename() (hex(256)=0x100, no underscore for >64 CU).

Also adds gfx950 to ALL_GPU_DATABASES so install_gpu_databases() picks
up the new bz2 files via the existing GLOB pattern.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>

@yassinsolim yassinsolim left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don’t think this is ready as-is. The install mechanics and gfx950100 basename look correct, but the DB content is causing performance regressions in focused testing.

I tested the latest PR head (18d7536) on MI350X / gfx950-mi350x with an A/B where the only variable was the system DB path: current staged DB/model tree vs the PR DB overlay. Same current MIOpen/CK libs, same node, same shapes, fresh user DB/cache per run.

Focused examples on MI350X:

bf16 512x256x256 -> 256: ~0.39 ms baseline → ~1.10 ms with the PR DB
bf16 512x1024x1024 -> 256: ~5.92 ms baseline → ~18.39 ms with the PR DB
fp32 256x1024x1024 -> 256: ~12.8 ms baseline → ~51.5 ms with the PR DB
In the focused logs, MIOpen loads matching records from the new gfx950 DB for these cases. So the change is not only removing runtime tuning cost; it can also steer covered shapes to slower recorded entries.

I only have a reliable A/B result on MI350X so far, but since this DB is intended to cover both MI350X and MI355X via the same gfx950100 basename, I think the MI350X regressions are enough to hold this until the entries are refreshed. The CMake/install part looks fine to me; the issue is DB content quality.

jdcampbe and others added 2 commits June 8, 2026 13:09
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Updates gfx950100.{HIP.fdb,db}.txt.bz2 (MI350X/MI355X, 256 CU) with the
newer generated find/perf DB pair.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
@jdcampbe
jdcampbe requested a review from yassinsolim June 15, 2026 04:52

@yassinsolim yassinsolim left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-ran my A/B on the new head (ce73a60) and this is a lot better, the stuff I flagged before is gone. Couple of new ones to sort out, but we're close.

Setup was the same idea as last time: MI355X, only thing changing is the gfx950 DB. Baseline is 261cdb8 (this branch's libs with the DB taken out, so MIOpen just finds normally), candidate is ce73a60 (same libs + your DB). Same image, same node, same 172 shapes, iter=10, fresh user db/cache each run. Ratio = candidate/baseline, so >1 means the DB entry is slower than what a plain find lands on.

Across the 172 shapes the median is basically 1.0 (0.997) and 67 of them actually got faster. Only 6 are over 1.10x now, vs 42 on the old b84b2a6 DB (which also had that 18x case). So the cleanup clearly worked.

The ones I called out before are all good now:

bf16 512x256x256->256     0.69x   (was ~2.8x)
bf16 512x1024x1024->256   0.71x   (was ~3.1x)
fp32 256x1024x1024->256   0.90x   (was ~4x)

and the 9 MI355 residuals from last round are all 0.74–1.03x now (were 1.06–1.67x).

What's left, these are real DB hits (instant find, so it's the DB picking these, not a search):

bf16 508x2048x2048->256    5.66x    33 -> 186 ms   DB picks ConvHipImplicitGemmGroupFwdXdlops, GemmFwdRest is ~6x faster
fp32 1024x1448x1448->256   1.33x    94 -> 125 ms   DB picks GemmFwdRest, a find lands on something faster
fp32 1024x1024x1024->256   1.05x    48 -> 50 ms    minor

The bf16 508x2048x2048->256 one is the only thing I'd block on, it's a covered shape getting steered to a kernel ~6x slower than it should. Keys if it helps:

508-2048-2048-3x3-256-2048-2048-1-1x1-1x1-1x1-0-NCHW-BF16-F
2x508x2048x2048x1x3x3x1x256x1x1x1x0x1x1x0x1x1x0x0x1xNCHWxBF16xF

Can you re-tune those couple of entries (or just drop them so they fall back to find)? The 1024x1448x1448 fp32 one would be nice to grab too. Everything else looks good to me.

Two caveats so the numbers are in context: I ran this on the branch's current libs, which are ~10 days behind develop, so once you rebase I'd like to do one more quick pass before it merges (CK moves around enough that a couple shapes can shift). And the baseline is a full find, i.e. close to best-case, so these ratios are "DB entry vs the best you can do" rather than vs some weak default, the two I flagged are genuinely bad recordings, not noise.

jdcampbe and others added 3 commits June 19, 2026 02:24
Updates gfx950100.{HIP.fdb,db}.txt.bz2 (MI350X/MI355X, 256 CU) with the
retuned dbsyncfix2 merged find/perf DB pair, and adds gfx950/256 to the
CPU_DBSync_NONE Smoke suite so the StaticFDBSync test runs on gfx950
hardware in CI.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Updates gfx950100.{HIP.fdb,db}.txt.bz2 (MI350X/MI355X, 256 CU) with the
keyfix tuned final find/perf DB pair.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
@jdcampbe
jdcampbe requested a review from yassinsolim June 21, 2026 03:53
Updates gfx950100.{HIP.fdb,db}.txt.bz2 (MI350X/MI355X, 256 CU) with the
relayer tuned final find/perf DB pair.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
@jdcampbe
jdcampbe enabled auto-merge (squash) June 22, 2026 18:52

@yassinsolim yassinsolim left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-ran broader validation on the latest f8a1dd2 head. The original ByteDance blocker looks fixed, but I found two reproducible tiny-shape regressions that look related to the DB not including/selecting ConvDirectNaiveConvFwd.

Method:

  • candidate = f8a1dd2 image with system gfx950100 DB enabled
  • baseline = same image/libs with system find DB disabled, so this isolates the PR DB effect
  • repeated the two worst cases 3x

Regressions:

  1. MIOpenDriver convfp16 -n 1 -c 1542 -H 14 -W 14 -k 1542 -y 3 -x 3 -p 1 -q 1 -u 2 -v 2 -l 1 -j 1 -g 1542 -m conv -F 1

    • DB: GemmFwdRest, geo ~0.0269 ms
    • no system DB: ConvDirectNaiveConvFwd, geo ~0.00452 ms
    • DB is ~5.9x slower
  2. MIOpenDriver convfp16 -n 1 -c 1632 -H 12 -W 12 -k 1632 -y 3 -x 3 -p 1 -q 1 -u 1 -v 1 -l 1 -j 1 -g 1632 -m conv -F 1

    • DB: GemmFwdRest, geo ~0.0301 ms
    • no system DB: ConvDirectNaiveConvFwd, geo ~0.00638 ms
    • DB is ~4.7x slower

If excluding naive from gfx950 SystemDB is intentional policy, please confirm that these tiny-shape regressions are acceptable. Otherwise I think these entries should be dropped or retuned so they fall back to the faster no-DB path.

@jdcampbe
jdcampbe merged commit 60d8e1e into develop Jun 22, 2026
54 checks passed
@jdcampbe
jdcampbe deleted the users/jascampb/systemdb-update-gfx950-05-2026 branch June 22, 2026 20:16
JonathanLichtnerAMD added a commit that referenced this pull request Jul 15, 2026
JIRA ID: ALMIOPEN-2312

## Motivation

`projects/miopen/CHANGELOG.md`'s "MIOpen 3.5.2 for ROCm 7.14.0" section
only had 4 entries, but 68 commits touched `projects/miopen` between the
`therock-7.13` tag and `release/therock-7.14`. Auditing that commit
range (via `git log --oneline
therock-7.13..remotes/origin/release/therock-7.14 -- projects/miopen`)
found 18 additional user-facing changes — new gfx950/gfx1250
architecture support, SystemDB refreshes, AI-heuristic improvements,
several correctness/crash bug fixes, and OpenCL solver removals — that
were never documented. This PR backfills those entries so the 7.14.0
changelog accurately reflects what shipped.

## Technical Details

Added 24 lines to the existing "MIOpen 3.5.2 for ROCm 7.14.0" section of
`projects/miopen/CHANGELOG.md`, merging new bullets into the existing
`Added`/`Changed`/`Resolved Issues` sub-sections (no new entries were
duplicated or reworded) and adding a new `Removed` sub-section for
solver removals. No existing content was altered.

Each new bullet was traced to a specific commit + PR:

| Entry | Commit | PR |
|---|---|---|
| Added gfx950 (MI350X/MI355X) SystemDB | `60d8e1e5175` | #8081 |
| Added initial gfx1250 architecture support | `d3f057b2f22` | #7587 |
| Added JSON-formatted performance logs (`MIOPEN_PERFORMANCE_LOGS`) |
`ffbde871260` | #5736 |
| Refreshed gfx942 SystemDB | `faf754a9ef6` | #8174 |
| Refined 2D AI heuristics (Tunanet/KTN) for gfx942 & gfx950 |
`c24ad415884` | #7109 |
| Fixed thread-safety issue in AI-heuristic model caches | `cd957402309`
| #9149 (cherry-pick of #9131) |
| Fixed CK grouped-conv solvers not always registered in host |
`c971d54d7ae` | #9105 |
| Fixed legacy CK buffer resource descriptor (gfx1250) | `1dee011b8b0` |
#8758 |
| Removed stale gfx942 SystemDB naive-solver entries | `f42668f6316` |
#7028 |
| Fixed softmax NaN when `beta=0` | `fb8157c8512` | #8426 |
| Fixed BN backward spatial variant 1 vectorized memory access bug |
`1351a7e8b10` | #6765 |
| Fixed use-after-free of `kernel_id` in `InitInvokerFactoryNCHW` |
`394ebffb50d` | #7922 |
| Fixed SoftMarginLoss forward reduce-kernel kernarg corruption |
`1e65a31570f` | #8065, #8135 |
| Fixed build with `MIOPEN_ENABLE_AI_KERNEL_TUNING=OFF` | `e6288c3ca61`
| #6800 |
| Fixed install RPATH for MIOpenDriver / CK backend libs | `ef8a4cf2f93`
| #7485 |
| Fixed dangling lambda causing log-level 6 segfault | `190f9791737` |
#6953 |
| Enabled Windows long-path support for `addkernels` | `1f9d69724cf` |
#7387 |
| Removed OCL solver `ConvOclDirectFwd1x1` | `126c720cadd` | #6491 |
| Removed OCL solver `conv_ocl_dir2Dfwdgen` | `2d499ddf709` | #6633 |
| Removed OCL solvers `ConvOclBwdWrW1x1`, `ConvOclBwdWrW2` |
`b5d43a33dc6` | #6859 |
| Removed OCL solvers `OpTensorLeadingOnes`, `Op5dTensorGeneric` |
`8642c83b7de` | #7347 |

The remaining ~40 commits in the 68-commit range were deliberately
excluded (tests/CI/build/docs/internal refactors/1:1 OCL→HIP swaps with
no behavior change, per each commit's own stated scope) and are not
represented here.

## Risk Assessment

**Risk Level: 🟢 Low**

This is a documentation-only change to a single Markdown file. No code,
build configuration, tests, or public API surface is touched.

### Impacted Components

- `projects/miopen/CHANGELOG.md` only.

### Potential Side Effects

None expected. This does not change any runtime behavior; it only
documents changes that already shipped on `release/therock-7.14`.

### Mitigation Steps

None required beyond normal review — a reviewer familiar with the 7.14
release can spot-check a few entries against the linked commit hashes/PR
numbers for accuracy.

## Test Plan

Not applicable (no code changes). Verification was done by
cross-referencing every added bullet against a specific commit hash and
PR number pulled from `git log`, and by diffing the edited section
against `git show
remotes/origin/release/therock-7.14:projects/miopen/CHANGELOG.md` to
confirm no existing entry was duplicated or altered.

## Test Result

All 21 new bullets map 1:1 to a real commit/PR in the
`therock-7.13..release/therock-7.14` range for `projects/miopen`. No
pre-existing changelog text was modified — `git diff` shows a pure
addition (24 insertions, 0 deletions). Two borderline commits
(`a67834413e8` "Standalone build", `7c7cd5774a9` "enable warnings about
deprecated API/features") were deliberately left out pending author
judgment on whether they warrant an entry.
pschang-phy pushed a commit that referenced this pull request Jul 17, 2026
JIRA ID: ALMIOPEN-2312

## Motivation

`projects/miopen/CHANGELOG.md`'s "MIOpen 3.5.2 for ROCm 7.14.0" section
only had 4 entries, but 68 commits touched `projects/miopen` between the
`therock-7.13` tag and `release/therock-7.14`. Auditing that commit
range (via `git log --oneline
therock-7.13..remotes/origin/release/therock-7.14 -- projects/miopen`)
found 18 additional user-facing changes — new gfx950/gfx1250
architecture support, SystemDB refreshes, AI-heuristic improvements,
several correctness/crash bug fixes, and OpenCL solver removals — that
were never documented. This PR backfills those entries so the 7.14.0
changelog accurately reflects what shipped.

## Technical Details

Added 24 lines to the existing "MIOpen 3.5.2 for ROCm 7.14.0" section of
`projects/miopen/CHANGELOG.md`, merging new bullets into the existing
`Added`/`Changed`/`Resolved Issues` sub-sections (no new entries were
duplicated or reworded) and adding a new `Removed` sub-section for
solver removals. No existing content was altered.

Each new bullet was traced to a specific commit + PR:

| Entry | Commit | PR |
|---|---|---|
| Added gfx950 (MI350X/MI355X) SystemDB | `60d8e1e5175` | #8081 |
| Added initial gfx1250 architecture support | `d3f057b2f22` | #7587 |
| Added JSON-formatted performance logs (`MIOPEN_PERFORMANCE_LOGS`) |
`ffbde871260` | #5736 |
| Refreshed gfx942 SystemDB | `faf754a9ef6` | #8174 |
| Refined 2D AI heuristics (Tunanet/KTN) for gfx942 & gfx950 |
`c24ad415884` | #7109 |
| Fixed thread-safety issue in AI-heuristic model caches | `cd957402309`
| #9149 (cherry-pick of #9131) |
| Fixed CK grouped-conv solvers not always registered in host |
`c971d54d7ae` | #9105 |
| Fixed legacy CK buffer resource descriptor (gfx1250) | `1dee011b8b0` |
#8758 |
| Removed stale gfx942 SystemDB naive-solver entries | `f42668f6316` |
#7028 |
| Fixed softmax NaN when `beta=0` | `fb8157c8512` | #8426 |
| Fixed BN backward spatial variant 1 vectorized memory access bug |
`1351a7e8b10` | #6765 |
| Fixed use-after-free of `kernel_id` in `InitInvokerFactoryNCHW` |
`394ebffb50d` | #7922 |
| Fixed SoftMarginLoss forward reduce-kernel kernarg corruption |
`1e65a31570f` | #8065, #8135 |
| Fixed build with `MIOPEN_ENABLE_AI_KERNEL_TUNING=OFF` | `e6288c3ca61`
| #6800 |
| Fixed install RPATH for MIOpenDriver / CK backend libs | `ef8a4cf2f93`
| #7485 |
| Fixed dangling lambda causing log-level 6 segfault | `190f9791737` |
#6953 |
| Enabled Windows long-path support for `addkernels` | `1f9d69724cf` |
#7387 |
| Removed OCL solver `ConvOclDirectFwd1x1` | `126c720cadd` | #6491 |
| Removed OCL solver `conv_ocl_dir2Dfwdgen` | `2d499ddf709` | #6633 |
| Removed OCL solvers `ConvOclBwdWrW1x1`, `ConvOclBwdWrW2` |
`b5d43a33dc6` | #6859 |
| Removed OCL solvers `OpTensorLeadingOnes`, `Op5dTensorGeneric` |
`8642c83b7de` | #7347 |

The remaining ~40 commits in the 68-commit range were deliberately
excluded (tests/CI/build/docs/internal refactors/1:1 OCL→HIP swaps with
no behavior change, per each commit's own stated scope) and are not
represented here.

## Risk Assessment

**Risk Level: 🟢 Low**

This is a documentation-only change to a single Markdown file. No code,
build configuration, tests, or public API surface is touched.

### Impacted Components

- `projects/miopen/CHANGELOG.md` only.

### Potential Side Effects

None expected. This does not change any runtime behavior; it only
documents changes that already shipped on `release/therock-7.14`.

### Mitigation Steps

None required beyond normal review — a reviewer familiar with the 7.14
release can spot-check a few entries against the linked commit hashes/PR
numbers for accuracy.

## Test Plan

Not applicable (no code changes). Verification was done by
cross-referencing every added bullet against a specific commit hash and
PR number pulled from `git log`, and by diffing the edited section
against `git show
remotes/origin/release/therock-7.14:projects/miopen/CHANGELOG.md` to
confirm no existing entry was duplicated or altered.

## Test Result

All 21 new bullets map 1:1 to a real commit/PR in the
`therock-7.13..release/therock-7.14` range for `projects/miopen`. No
pre-existing changelog text was modified — `git diff` shows a pure
addition (24 insertions, 0 deletions). Two borderline commits
(`a67834413e8` "Standalone build", `7c7cd5774a9` "enable warnings about
deprecated API/features") were deliberately left out pending author
judgment on whether they warrant an entry.
JonathanLichtnerAMD added a commit that referenced this pull request Aug 15, 2026
The gfx942 leg failed StaticFDBSync with "invalid perf config" for 3 CK
grouped-wrw entries. Root cause is CK/DB version skew, not a develop bug: this
job links the CK preinstalled in rocm/miopen-dev:latest (built 2026-06-12),
while the gfx942 SystemDB was refreshed 2026-06-24 (#8174) against newer CK, so
the older container CK rejects perf configs the DB now stores. gfx950's DB
(#8081/#9111) is newer still. The real TheRock CI builds CK from the matching
source snapshot and is unaffected.

Scope the matrix to gfx90a -- the one arch with no TheRock test runner, so this
GPU-less job is its only dbsync coverage, and its DB matches the container CK
(reliable, green). gfx942/gfx950 stay covered by TheRock's source-CK dbsync;
gating them here would require building CK from the matching source.

ALMIOPEN-1779

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants