Repository navigation
[MIOpen] Add gfx950 SystemDB for MI350X/MI355X - #8081
Conversation
Stages the cleaned find/perf DB pair for gfx950 (transposed Wino solvers stripped, validated via dbsync-test StaticFDBSync/5). Both MI350X and MI355X have 256 CUs, so a single basename gfx950100 covers both per Handle::GetDbBasename() (hex(256)=0x100, no underscore for >64 CU). Also adds gfx950 to ALL_GPU_DATABASES so install_gpu_databases() picks up the new bz2 files via the existing GLOB pattern. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
yassinsolim
left a comment
There was a problem hiding this comment.
I don’t think this is ready as-is. The install mechanics and gfx950100 basename look correct, but the DB content is causing performance regressions in focused testing.
I tested the latest PR head (18d7536) on MI350X / gfx950-mi350x with an A/B where the only variable was the system DB path: current staged DB/model tree vs the PR DB overlay. Same current MIOpen/CK libs, same node, same shapes, fresh user DB/cache per run.
Focused examples on MI350X:
bf16 512x256x256 -> 256: ~0.39 ms baseline → ~1.10 ms with the PR DB
bf16 512x1024x1024 -> 256: ~5.92 ms baseline → ~18.39 ms with the PR DB
fp32 256x1024x1024 -> 256: ~12.8 ms baseline → ~51.5 ms with the PR DB
In the focused logs, MIOpen loads matching records from the new gfx950 DB for these cases. So the change is not only removing runtime tuning cost; it can also steer covered shapes to slower recorded entries.
I only have a reliable A/B result on MI350X so far, but since this DB is intended to cover both MI350X and MI355X via the same gfx950100 basename, I think the MI350X regressions are enough to hold this until the entries are refreshed. The CMake/install part looks fine to me; the issue is DB content quality.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Updates gfx950100.{HIP.fdb,db}.txt.bz2 (MI350X/MI355X, 256 CU) with the
newer generated find/perf DB pair.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
yassinsolim
left a comment
There was a problem hiding this comment.
Re-ran my A/B on the new head (ce73a60) and this is a lot better, the stuff I flagged before is gone. Couple of new ones to sort out, but we're close.
Setup was the same idea as last time: MI355X, only thing changing is the gfx950 DB. Baseline is 261cdb8 (this branch's libs with the DB taken out, so MIOpen just finds normally), candidate is ce73a60 (same libs + your DB). Same image, same node, same 172 shapes, iter=10, fresh user db/cache each run. Ratio = candidate/baseline, so >1 means the DB entry is slower than what a plain find lands on.
Across the 172 shapes the median is basically 1.0 (0.997) and 67 of them actually got faster. Only 6 are over 1.10x now, vs 42 on the old b84b2a6 DB (which also had that 18x case). So the cleanup clearly worked.
The ones I called out before are all good now:
bf16 512x256x256->256 0.69x (was ~2.8x)
bf16 512x1024x1024->256 0.71x (was ~3.1x)
fp32 256x1024x1024->256 0.90x (was ~4x)
and the 9 MI355 residuals from last round are all 0.74–1.03x now (were 1.06–1.67x).
What's left, these are real DB hits (instant find, so it's the DB picking these, not a search):
bf16 508x2048x2048->256 5.66x 33 -> 186 ms DB picks ConvHipImplicitGemmGroupFwdXdlops, GemmFwdRest is ~6x faster
fp32 1024x1448x1448->256 1.33x 94 -> 125 ms DB picks GemmFwdRest, a find lands on something faster
fp32 1024x1024x1024->256 1.05x 48 -> 50 ms minor
The bf16 508x2048x2048->256 one is the only thing I'd block on, it's a covered shape getting steered to a kernel ~6x slower than it should. Keys if it helps:
508-2048-2048-3x3-256-2048-2048-1-1x1-1x1-1x1-0-NCHW-BF16-F
2x508x2048x2048x1x3x3x1x256x1x1x1x0x1x1x0x1x1x0x0x1xNCHWxBF16xF
Can you re-tune those couple of entries (or just drop them so they fall back to find)? The 1024x1448x1448 fp32 one would be nice to grab too. Everything else looks good to me.
Two caveats so the numbers are in context: I ran this on the branch's current libs, which are ~10 days behind develop, so once you rebase I'd like to do one more quick pass before it merges (CK moves around enough that a couple shapes can shift). And the baseline is a full find, i.e. close to best-case, so these ratios are "DB entry vs the best you can do" rather than vs some weak default, the two I flagged are genuinely bad recordings, not noise.
Updates gfx950100.{HIP.fdb,db}.txt.bz2 (MI350X/MI355X, 256 CU) with the
retuned dbsyncfix2 merged find/perf DB pair, and adds gfx950/256 to the
CPU_DBSync_NONE Smoke suite so the StaticFDBSync test runs on gfx950
hardware in CI.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Updates gfx950100.{HIP.fdb,db}.txt.bz2 (MI350X/MI355X, 256 CU) with the
keyfix tuned final find/perf DB pair.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Updates gfx950100.{HIP.fdb,db}.txt.bz2 (MI350X/MI355X, 256 CU) with the
relayer tuned final find/perf DB pair.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
yassinsolim
left a comment
There was a problem hiding this comment.
Re-ran broader validation on the latest f8a1dd2 head. The original ByteDance blocker looks fixed, but I found two reproducible tiny-shape regressions that look related to the DB not including/selecting ConvDirectNaiveConvFwd.
Method:
- candidate = f8a1dd2 image with system gfx950100 DB enabled
- baseline = same image/libs with system find DB disabled, so this isolates the PR DB effect
- repeated the two worst cases 3x
Regressions:
-
MIOpenDriver convfp16 -n 1 -c 1542 -H 14 -W 14 -k 1542 -y 3 -x 3 -p 1 -q 1 -u 2 -v 2 -l 1 -j 1 -g 1542 -m conv -F 1- DB:
GemmFwdRest, geo ~0.0269 ms - no system DB:
ConvDirectNaiveConvFwd, geo ~0.00452 ms - DB is ~5.9x slower
- DB:
-
MIOpenDriver convfp16 -n 1 -c 1632 -H 12 -W 12 -k 1632 -y 3 -x 3 -p 1 -q 1 -u 1 -v 1 -l 1 -j 1 -g 1632 -m conv -F 1- DB:
GemmFwdRest, geo ~0.0301 ms - no system DB:
ConvDirectNaiveConvFwd, geo ~0.00638 ms - DB is ~4.7x slower
- DB:
If excluding naive from gfx950 SystemDB is intentional policy, please confirm that these tiny-shape regressions are acceptable. Otherwise I think these entries should be dropped or retuned so they fall back to the faster no-DB path.
JIRA ID: ALMIOPEN-2312 ## Motivation `projects/miopen/CHANGELOG.md`'s "MIOpen 3.5.2 for ROCm 7.14.0" section only had 4 entries, but 68 commits touched `projects/miopen` between the `therock-7.13` tag and `release/therock-7.14`. Auditing that commit range (via `git log --oneline therock-7.13..remotes/origin/release/therock-7.14 -- projects/miopen`) found 18 additional user-facing changes — new gfx950/gfx1250 architecture support, SystemDB refreshes, AI-heuristic improvements, several correctness/crash bug fixes, and OpenCL solver removals — that were never documented. This PR backfills those entries so the 7.14.0 changelog accurately reflects what shipped. ## Technical Details Added 24 lines to the existing "MIOpen 3.5.2 for ROCm 7.14.0" section of `projects/miopen/CHANGELOG.md`, merging new bullets into the existing `Added`/`Changed`/`Resolved Issues` sub-sections (no new entries were duplicated or reworded) and adding a new `Removed` sub-section for solver removals. No existing content was altered. Each new bullet was traced to a specific commit + PR: | Entry | Commit | PR | |---|---|---| | Added gfx950 (MI350X/MI355X) SystemDB | `60d8e1e5175` | #8081 | | Added initial gfx1250 architecture support | `d3f057b2f22` | #7587 | | Added JSON-formatted performance logs (`MIOPEN_PERFORMANCE_LOGS`) | `ffbde871260` | #5736 | | Refreshed gfx942 SystemDB | `faf754a9ef6` | #8174 | | Refined 2D AI heuristics (Tunanet/KTN) for gfx942 & gfx950 | `c24ad415884` | #7109 | | Fixed thread-safety issue in AI-heuristic model caches | `cd957402309` | #9149 (cherry-pick of #9131) | | Fixed CK grouped-conv solvers not always registered in host | `c971d54d7ae` | #9105 | | Fixed legacy CK buffer resource descriptor (gfx1250) | `1dee011b8b0` | #8758 | | Removed stale gfx942 SystemDB naive-solver entries | `f42668f6316` | #7028 | | Fixed softmax NaN when `beta=0` | `fb8157c8512` | #8426 | | Fixed BN backward spatial variant 1 vectorized memory access bug | `1351a7e8b10` | #6765 | | Fixed use-after-free of `kernel_id` in `InitInvokerFactoryNCHW` | `394ebffb50d` | #7922 | | Fixed SoftMarginLoss forward reduce-kernel kernarg corruption | `1e65a31570f` | #8065, #8135 | | Fixed build with `MIOPEN_ENABLE_AI_KERNEL_TUNING=OFF` | `e6288c3ca61` | #6800 | | Fixed install RPATH for MIOpenDriver / CK backend libs | `ef8a4cf2f93` | #7485 | | Fixed dangling lambda causing log-level 6 segfault | `190f9791737` | #6953 | | Enabled Windows long-path support for `addkernels` | `1f9d69724cf` | #7387 | | Removed OCL solver `ConvOclDirectFwd1x1` | `126c720cadd` | #6491 | | Removed OCL solver `conv_ocl_dir2Dfwdgen` | `2d499ddf709` | #6633 | | Removed OCL solvers `ConvOclBwdWrW1x1`, `ConvOclBwdWrW2` | `b5d43a33dc6` | #6859 | | Removed OCL solvers `OpTensorLeadingOnes`, `Op5dTensorGeneric` | `8642c83b7de` | #7347 | The remaining ~40 commits in the 68-commit range were deliberately excluded (tests/CI/build/docs/internal refactors/1:1 OCL→HIP swaps with no behavior change, per each commit's own stated scope) and are not represented here. ## Risk Assessment **Risk Level: 🟢 Low** This is a documentation-only change to a single Markdown file. No code, build configuration, tests, or public API surface is touched. ### Impacted Components - `projects/miopen/CHANGELOG.md` only. ### Potential Side Effects None expected. This does not change any runtime behavior; it only documents changes that already shipped on `release/therock-7.14`. ### Mitigation Steps None required beyond normal review — a reviewer familiar with the 7.14 release can spot-check a few entries against the linked commit hashes/PR numbers for accuracy. ## Test Plan Not applicable (no code changes). Verification was done by cross-referencing every added bullet against a specific commit hash and PR number pulled from `git log`, and by diffing the edited section against `git show remotes/origin/release/therock-7.14:projects/miopen/CHANGELOG.md` to confirm no existing entry was duplicated or altered. ## Test Result All 21 new bullets map 1:1 to a real commit/PR in the `therock-7.13..release/therock-7.14` range for `projects/miopen`. No pre-existing changelog text was modified — `git diff` shows a pure addition (24 insertions, 0 deletions). Two borderline commits (`a67834413e8` "Standalone build", `7c7cd5774a9` "enable warnings about deprecated API/features") were deliberately left out pending author judgment on whether they warrant an entry.
JIRA ID: ALMIOPEN-2312 ## Motivation `projects/miopen/CHANGELOG.md`'s "MIOpen 3.5.2 for ROCm 7.14.0" section only had 4 entries, but 68 commits touched `projects/miopen` between the `therock-7.13` tag and `release/therock-7.14`. Auditing that commit range (via `git log --oneline therock-7.13..remotes/origin/release/therock-7.14 -- projects/miopen`) found 18 additional user-facing changes — new gfx950/gfx1250 architecture support, SystemDB refreshes, AI-heuristic improvements, several correctness/crash bug fixes, and OpenCL solver removals — that were never documented. This PR backfills those entries so the 7.14.0 changelog accurately reflects what shipped. ## Technical Details Added 24 lines to the existing "MIOpen 3.5.2 for ROCm 7.14.0" section of `projects/miopen/CHANGELOG.md`, merging new bullets into the existing `Added`/`Changed`/`Resolved Issues` sub-sections (no new entries were duplicated or reworded) and adding a new `Removed` sub-section for solver removals. No existing content was altered. Each new bullet was traced to a specific commit + PR: | Entry | Commit | PR | |---|---|---| | Added gfx950 (MI350X/MI355X) SystemDB | `60d8e1e5175` | #8081 | | Added initial gfx1250 architecture support | `d3f057b2f22` | #7587 | | Added JSON-formatted performance logs (`MIOPEN_PERFORMANCE_LOGS`) | `ffbde871260` | #5736 | | Refreshed gfx942 SystemDB | `faf754a9ef6` | #8174 | | Refined 2D AI heuristics (Tunanet/KTN) for gfx942 & gfx950 | `c24ad415884` | #7109 | | Fixed thread-safety issue in AI-heuristic model caches | `cd957402309` | #9149 (cherry-pick of #9131) | | Fixed CK grouped-conv solvers not always registered in host | `c971d54d7ae` | #9105 | | Fixed legacy CK buffer resource descriptor (gfx1250) | `1dee011b8b0` | #8758 | | Removed stale gfx942 SystemDB naive-solver entries | `f42668f6316` | #7028 | | Fixed softmax NaN when `beta=0` | `fb8157c8512` | #8426 | | Fixed BN backward spatial variant 1 vectorized memory access bug | `1351a7e8b10` | #6765 | | Fixed use-after-free of `kernel_id` in `InitInvokerFactoryNCHW` | `394ebffb50d` | #7922 | | Fixed SoftMarginLoss forward reduce-kernel kernarg corruption | `1e65a31570f` | #8065, #8135 | | Fixed build with `MIOPEN_ENABLE_AI_KERNEL_TUNING=OFF` | `e6288c3ca61` | #6800 | | Fixed install RPATH for MIOpenDriver / CK backend libs | `ef8a4cf2f93` | #7485 | | Fixed dangling lambda causing log-level 6 segfault | `190f9791737` | #6953 | | Enabled Windows long-path support for `addkernels` | `1f9d69724cf` | #7387 | | Removed OCL solver `ConvOclDirectFwd1x1` | `126c720cadd` | #6491 | | Removed OCL solver `conv_ocl_dir2Dfwdgen` | `2d499ddf709` | #6633 | | Removed OCL solvers `ConvOclBwdWrW1x1`, `ConvOclBwdWrW2` | `b5d43a33dc6` | #6859 | | Removed OCL solvers `OpTensorLeadingOnes`, `Op5dTensorGeneric` | `8642c83b7de` | #7347 | The remaining ~40 commits in the 68-commit range were deliberately excluded (tests/CI/build/docs/internal refactors/1:1 OCL→HIP swaps with no behavior change, per each commit's own stated scope) and are not represented here. ## Risk Assessment **Risk Level: 🟢 Low** This is a documentation-only change to a single Markdown file. No code, build configuration, tests, or public API surface is touched. ### Impacted Components - `projects/miopen/CHANGELOG.md` only. ### Potential Side Effects None expected. This does not change any runtime behavior; it only documents changes that already shipped on `release/therock-7.14`. ### Mitigation Steps None required beyond normal review — a reviewer familiar with the 7.14 release can spot-check a few entries against the linked commit hashes/PR numbers for accuracy. ## Test Plan Not applicable (no code changes). Verification was done by cross-referencing every added bullet against a specific commit hash and PR number pulled from `git log`, and by diffing the edited section against `git show remotes/origin/release/therock-7.14:projects/miopen/CHANGELOG.md` to confirm no existing entry was duplicated or altered. ## Test Result All 21 new bullets map 1:1 to a real commit/PR in the `therock-7.13..release/therock-7.14` range for `projects/miopen`. No pre-existing changelog text was modified — `git diff` shows a pure addition (24 insertions, 0 deletions). Two borderline commits (`a67834413e8` "Standalone build", `7c7cd5774a9` "enable warnings about deprecated API/features") were deliberately left out pending author judgment on whether they warrant an entry.
The gfx942 leg failed StaticFDBSync with "invalid perf config" for 3 CK grouped-wrw entries. Root cause is CK/DB version skew, not a develop bug: this job links the CK preinstalled in rocm/miopen-dev:latest (built 2026-06-12), while the gfx942 SystemDB was refreshed 2026-06-24 (#8174) against newer CK, so the older container CK rejects perf configs the DB now stores. gfx950's DB (#8081/#9111) is newer still. The real TheRock CI builds CK from the matching source snapshot and is unaffected. Scope the matrix to gfx90a -- the one arch with no TheRock test runner, so this GPU-less job is its only dbsync coverage, and its DB matches the container CK (reliable, green). gfx942/gfx950 stay covered by TheRock's source-CK dbsync; gating them here would require building CK from the matching source. ALMIOPEN-1779 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Motivation
MIOpen ships per-arch find/perf databases under
src/kernels/, but no gfx950 SystemDB was packaged. Without it, gfx950 runs fall back to runtime tuning on every invocation. MI350X and MI355X both expose 256 CUs, so a singlegfx950100basename serves both perHandle::GetDbBasename()(hex(256) =0x100, and the_<cu>suffix is omitted for >64 CU).Technical Details
projects/miopen/src/kernels/gfx950100.HIP.fdb.txt.bz2andprojects/miopen/src/kernels/gfx950100.db.txt.bz2.gfx950toALL_GPU_DATABASESinprojects/miopen/CMakeLists.txtsoinstall_gpu_databases()picks up the newbz2files via the existing GLOB pattern.Test Plan
dbsync-testStaticFDBSync/5round-trip on the staged find DB.install_gpu_databases()packages bothgfx950100.*.bz2blobs in the install tree.in the DB).
Test Result
dbsync-test StaticFDBSync/5passes on the staged find DB. Runtime smoke on MI350X/MI355X pending; will update with results.Risk Assessment
Risk level: Low
projects/miopen/src/kernels/gfx950100.HIP.fdb.txt.bz2,projects/miopen/src/kernels/gfx950100.db.txt.bz2,and the
ALL_GPU_DATABASESlist inprojects/miopen/CMakeLists.txt. No source / kernel / solver code is touched. Other archs (gfx942,gfx90a, etc.) are unaffected.
gfx90a, etc.) are unaffected.
archs.
bz2files and the list entry.dbsync-testround-trip before staging.Submission Checklist