Skip to content

feat(miopen): Fix system-DB fallback so MI308 (gfx942) resolves to a usable DB - #8802

Merged
jdcampbe merged 14 commits into
developfrom
users/jascampb/mi308-db-identification-fix
Aug 25, 2026
Merged

jdcampbe merged 14 commits into
developfrom
users/jascampb/mi308-db-identification-fix

Conversation

@jdcampbe

@jdcampbe jdcampbe commented Jun 25, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

gfx942 encompasses multiple cards. The current system dbs don't cover all of the cards due to inexact ("closest compute-unit") fallback silently selected nothing, so MIOpen ran with no system tuning data and logged File is unreadable: ".../miopen_system_db/gfx942_64.HIP.fdb.txt". This PR fixes the root-cause bug that left the fallback's candidate list empty, and adds a small, scoped redirect so such parts fall back to the appropriate sibling gfx942 database (gfx942130) instead of the nearer-by-count but less suitable gfx942e4.
JIRA ID : ALMIOPEN-2121

Technical Details

System DBs are named an exact base naming system. When the exact basename file is absent, MIOpen runs an inexact search: scan the DB directory and pick the file for the same arch (DbId) with the numerically closest CU count.

Root-cause bug. The inexact search built its candidate list with fs::path::extension():

if(fs::is_regular_file(filepath) && filepath.extension() == path_suffix + ".fdb.txt")

std::filesystem::path::extension() returns only the final component after the last dot — .txt for gfx942e4.HIP.fdb.txt — so it can never equal the multi-dot .fdb.txt / .db.txt. On disk-based text DBs (every shipped DB) the candidate list was always empty, the inexact search returned an empty path, and no fallback DB was loaded. This was invisible on devices that ship their own DB because they short-circuit on the exact-match fs::exists() path, which never touches this filter.

Fix 1 — candidate collection. Match against the full filename with EndsWith(...) instead of extension(), in all four inexact loops:

  • src/find_db.cpp:163 (find-db, on-disk) and the embedded find-db loop (already used EndsWith, left as is at src/find_db.cpp:83).
  • src/include/miopen/execution_context.hpp:230 (perf-db, on-disk) and :153 (perf-db, embedded); added #include <miopen/stringutils.hpp>.

Fix 2 — scoped sibling redirect. Add GetSysDbSelectionCu() in src/include/miopen/handle.hpp:

inline int GetSysDbSelectionCu(const std::string& db_id, int real_cu_count)
{
    if(db_id == "gfx942" && real_cu_count <= 80)
        return 304;
    return real_cu_count;
}

Intentionally not changed. User DBs (*.udb.txt / *.ufdb.txt) keep the device's own basename and have no inexact fallback, so per-device tuned data is still read and written separately. The exact-match path is untouched — devices with their own DB short-circuit before any modified code runs. GetSysDbSelectionCu returns the real CU count unchanged for every other arch and for gfx942 parts above the threshold. The kdb path (arch-only fallback) is unaffected.

Test Plan

Build a multi-arch image from this branch and run a small smoke set (conv fwd/bwd/wrw x fp32/fp16/bf16) with MIOPEN_LOG_LEVEL=7 (to capture DB resolution) and MIOPEN_FIND_MODE=2, across:

  • mi308x (no dedicated DB) — the fallback target,
  • mi325x/mi300x — sanity,
  • a gfx950 — sanity.

For each, inspect the shard log for the find-db / perf-db resolution lines and confirm:

  • GOOD markers: Found exact ... database file for exact parts; inexact find database search then FindRecord ... in file ".../<expected>" for the fallback part.
  • No File is unreadable on a miopen_system_db path.

Test Result

Part find-db resolved perf-db used system-DB unreadable? Result
gfx942, low-CU (fallback) inexact -> gfx942130 gfx942130 none fallback now works
gfx942, 304 CU (sanity) exact gfx942130 gfx942130 none no regression
gfx950 (sanity) exact gfx950100 gfx950100 none no regression

The fallback part now logs inexact find database search, then FindRecord ... in file ".../gfx942130.HIP.fdb.txt", and the perf-db opens gfx942130.db.txt (the transient gfx942e4 "best candidate" line is superseded by gfx942130 at distance 0 from the biased target; only gfx942130 is actually opened). No File is unreadable on the system DB. Exact-match parts log Found exact ... database file and load their own DBs unchanged. Two additional same-DB sanity runs on a second node each (mi300x/mi325x and gfx950) are still in progress and are expected to match the corresponding sanity rows; will update if anything differs.

Risk Assessment

Risk level: Low

  • Impacted components: system-DB path resolution only — src/find_db.cpp, src/include/miopen/execution_context.hpp, src/include/miopen/handle.hpp (3 files, +40 / -11). Affects only the inexact ("closest-CU") fallback branch. Not touched: the exact-match path, user DBs, the kdb path, public API/ABI, kernels, and any non-gfx942 arch.
  • Potential side effects:
    • Functional: previously-broken fallback parts now load a usable system DB; low-CU gfx942 parts resolve to gfx942130 rather than gfx942e4 or nothing. No behavior change for any device that already resolved a DB.
    • Performance: negligible — one extra integer comparison per DB lookup, on the cold DB-open path only.
    • ABI/API: none.
  • Mitigation steps:
    • Change is small and localized to the inexact branch; the exact-match path that all DB-shipping devices use is provably unentered (short-circuits on fs::exists() before the modified code).
    • Defect and fix are easy to reproduce and verify pre-merge with MIOPEN_LOG_LEVEL=7 (see Test Plan); validated on real gfx942 and gfx950 hardware.
    • Fully revertible — no data files added, renamed, or removed.

Submission Checklist

…e DB

Two issues prevented gfx942 parts without their own system database from
falling back correctly:

1. The inexact (closest-CU) search built its candidate list with
   fs::path::extension(), which returns only the final ".txt" component and
   so never matched the multi-dot ".fdb.txt"/".db.txt" names. The candidate
   list was always empty on disk-based text DBs, so no fallback DB was ever
   selected. Exact-match devices were unaffected (they hit fs::exists), which
   hid the bug from every device that ships its own DB. Switch the filter to
   EndsWith on the full filename in the find-db and perf-db loops (embedded
   and on-disk).

2. Add GetSysDbSelectionCu(): low-CU gfx942 parts have no dedicated system DB
   and would otherwise land on the nearer-by-count sibling. Bias their
   selection to the higher-CU gfx942 database instead. System-DB selection
   only; user DBs keep the device's own basename, so per-device tuned data is
   still read and written separately.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@therock-pr-bot

therock-pr-bot Bot commented Jun 30, 2026 •

Copy link
Copy Markdown

✅ All Checks Passed — Ready for Review

Check Status Details
📝 PR Description ✅ Pass —
⛔ Forbidden Files ✅ Pass —
🧪 Unit Test ✅ Pass —
🔎 pre-commit ✅ Pass —
🚫 Draft PR 🔜 To Be Enabled —
🚩 Feature Flag 🔜 To Be Enabled —
📊 Code Coverage 🔜 To Be Enabled —
🤖 therock-pr-bot ✅ Pass —

🎉 All checks passed! This PR is ready for review.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

@therock-pr-bot

therock-pr-bot Bot commented Jun 30, 2026 •

Copy link
Copy Markdown

🎉 All checks passed! This PR is ready for review.

@jdcampbe jdcampbe changed the title [MIOpen] Fix system-DB fallback so MI308 (gfx942) resolves to a usable DB feat(miopen): Fix system-DB fallback so MI308 (gfx942) resolves to a usable DB Jul 8, 2026
Covers the low-CU gfx942 (MI308) system-DB redirect: threshold boundary
at 80 CUs, high-CU passthrough (228/304), non-gfx942 passthrough, and
empty db_id handling.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@cderb cderb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jdcampbe
jdcampbe enabled auto-merge (squash) August 24, 2026 22:30
@jdcampbe
jdcampbe merged commit b59bce8 into develop Aug 25, 2026
263 of 276 checks passed
@jdcampbe
jdcampbe deleted the users/jascampb/mi308-db-identification-fix branch August 25, 2026 07:07
JonathanLichtnerAMD added a commit that referenced this pull request Sep 2, 2026
## Summary

Backfills the **MIOpen 3.6.1 / ROCm 10.1.0** changelog section with
externally-visible changes that landed since the `release/therock-10.0`
branch but weren't yet documented. All 43 MIOpen commits in that range
were reviewed; CI/build-infra, test-only, docs, version-bump, and
already-documented changes were excluded.

21 entries added:

### Added
* [RNN] Fused single-layer LSTM forward-inference kernel, gated behind
`MIOPEN_DEBUG_RNN_FUSED_INFERENCE` (#8915)
* [Conv] Naive reference kernel enabling 3D INT8 forward convolution
(#10051)
* [Conv] `ConvHipConv` solver backed by vendored hipconv kernels (#9746)
* [Conv] float8 vector type support + deduplicated math kernels (#10130)
* [Conv] Validated gfx1151 and gfx1200 (Navi) SystemDBs (#10050)
* [Conv] `MIOPEN_DEBUG_DISABLE_SYSTEM_DB` /
`MIOPEN_DEBUG_DISABLE_USER_DB` env vars (#10728)
* [Conv] CK depthwise convolution on RDNA wave32 GPUs (#9373)

### Changed
* [Conv] Winograd Rage RxS f2x3 on gfx950 + gfx942 v4_6_1/v4_9_1 kernels
(#9014)
* [Conv] Updated 3D AI heuristics for gfx942 & gfx950 (#8954)

### Optimized
* [Conv] Single GEMM for point (1x1) output convolution, all directions,
2D/3D (#10912)
* [Conv] Naive backward-data kernel work group widened to 1024 (#10684)

### Resolved Issues
* [Conv] Batch-split miscalculation on large WrW tensors — ROCM-26011
(#11188)
* [Conv] Conv failure on XNACK-unsupported gfx9 APUs (gfx90c) (#11211)
* [BatchNorm] Device heap-overflow in fwd spatial `FinalMeanVariance` —
ALMIOPEN-2424 (#11093)
* [Conv] Deterministic `SetNextValue` polarity for grouped Bwd/Wrw CK
solvers (#10240)
* [LRN][LayerNorm] Grid alignment + OOB threads on gfx1250 (#10251)
* [Conv] Stale gfx950 grouped-wrw SystemDB entries (#11130)
* [Conv][Windows] System-DB path resolution from the MIOpen module
(#10838)
* [Conv] System-DB fallback so MI308X (gfx942) resolves a usable DB
(#8802)
* JSON perf logs (`MIOPEN_PERFORMANCE_LOGS`) dropping the last solver
during Find (#11166)
* Build failure against GCC 15 libstdc++ under C++20 (#9642)

## Notes

Deliberately excluded: internal/default-off changes (#10219, #10088,
#10649, #10231), a near-duplicate of an existing 10.0.0 entry (#10188),
and CI/build-infra/test-only/docs commits.

Docs-only change; no code impact.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants