Repository navigation
ci(miopen): prototype GPU-free dbsync via TheRock artifact-reuse + rocjitsu [ALMIOPEN-1779] - #10844
JonathanLichtnerAMD wants to merge 12 commits into
Conversation
…cjitsu Adds a workflow_dispatch prototype that fetches TheRock's per-PR source-CK build artifact for one arch and runs the MIOpen dbsync check (StaticFDBSync) under the rocjitsu KMD interposer on a CPU runner (no GPU). This is the scalable pattern for ALMIOPEN-1779: reuse the PR's own source-CK build (catches CK regressions, matches the SystemDB) instead of a frozen container CK, and validate any arch via MIOPEN_DEVICE_ARCH since dbsync is CPU-only (SKIP_KDB_TESTING=1, no kernel launch). Prototype target gfx950: TheRock builds it (source-CK artifact with the gfx950 CK plugin + DBs exists), and gfx950 dbsync currently runs nowhere -- excluded on TheRock (test_miopen.py TEST_TO_IGNORE gfx950-dcgpu, TODO rocm-libraries#2266) and on Jenkins (test_categories.yaml). If green, this re-enables gfx950 dbsync GPU-free and scales to any TheRock-built arch via a matrix over ARTIFACT_GROUP. Fetch reuses ROCm/TheRock's setup_test_environment action; artifact_run_id and the MIOpen fetch_artifact_args are dispatch inputs. Verification is a live dispatch. ALMIOPEN-1779 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
github-actions
Bot
added
github actions
project: none
✅ All Checks Passed — Ready for Review
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🎉 All checks passed! This PR is ready for review. |
…ld-essential, arch-matched rocjitsu config) Validated on gfx950 (run 31865606916): fetch + dist contents + GPU-free StaticFDBSync all work with the cdna4 config; surfaced real invalid-perf-config gfx950 grouped-wrw entries (the #2266 / PR #10043-predicted stale class). Key correction: CK IsApplicable does a HIP device call, so the rocjitsu config must match the arch (cdna4=gfx950).
✅ Validated live (GPU-free, source CK) — gfx950Drove the prototype via a temporary
Key correction baked into this PR: CK grouped-conv The finding: the run failed on real Scope caveat: this GPU-free path needs both (a) TheRock to build the arch and (b) a rocjitsu config for it. Today that's cdna3/gfx942 + cdna4/gfx950 only — gfx90a/gfx908 (no rocjitsu config, not built by TheRock) and RDNA/gfx1250 (no config yet) can't use it until configs exist. |
Matrix over the supported set (TheRock builds it AND rocjitsu has a KMD config): gfx942/cdna3 + gfx950/cdna4. artifact_run_id + fetch_artifact_args are shared inputs; family/arch/rj_config come from the matrix. gfx90a/gfx125X need rocjitsu configs (and, for gfx90a, a TheRock build) before they can join.
…dd job timeout Validated 2026-08-15 (run 31890772516): with the correctly-matched cdna3 config, StaticFDBSync/3 (gfx942, 304 CU) stalled -- ~200 find-db lines processed in 6 HOURS before the job was force-cancelled, vs 13s for the same DB locally on real hardware. db_sync.cpp spawns up to 32 threads each making a HIP device call (CK grouped-conv IsApplicable) through the rocjitsu interposer; something about that concurrency (or gfx942's much larger FDB) stalls under rocjitsu/cdna3 specifically. Not root-caused. gfx942 dbsync is already covered by TheRock's real GPU test job (gfx94X-dcgpu), so this GPU-free path isn't needed there -- not worth the CI cost until understood. Keep the validated gfx950 leg (passes fetch/inspect, runs in ~25s, surfaces real stale-DB findings for #2266). Add timeout-minutes: 20 as a safety net regardless.
|
Diagnostic-oriented addition to StaticFDBSync's thread fan-out. Default behavior is unchanged (min(hardware_concurrency(), 32)); when set, the env var overrides the thread count. Needed because CPU affinity restrictions (taskset, Docker --cpuset-cpus) do NOT reduce std::thread::hardware_concurrency() on Linux/glibc (verified empirically), so there was no way to test StaticFDBSync with fewer threads without a code change -- needed to investigate whether the gfx942 rocjitsu stall (run 31890772516) is a concurrency/interposer interaction. Validated locally: MIOPEN_DEVICE_ARCH=gfx90a on real gfx942 hardware, both default (3.98s) and MIOPEN_DBSYNC_MAX_THREADS=1 (32.9s) pass identically (2 passed, 7 skipped) -- confirms the override works and changes only speed, not correctness. ALMIOPEN-1779 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lls) Re-tested gfx942 against a fresh artifact (TheRock multi_arch_ci.yml run 31972143345, rocm-libraries pinned to a pre-rocThrust-regression base + the MIOPEN_DBSYNC_MAX_THREADS patch) across 4 separate dispatches: default (32-thread, the exact config that stalled 6h in run 31890772516), 4-thread, and 1-thread. None stalled -- 15-199s each, scaling sensibly with thread count. The original stall did not reproduce; presumed a one-off possibly tied to the earlier artifact's build pipeline rather than anything inherent to gfx942 or rocjitsu. gfx942 now reliably surfaces a real finding: 3 stale ConvHipImplicitGemmGroupWrwXdlops/ ConvHipImplicitGemm3DGroupWrwXdlops 'invalid perf config' entries in gfx942130.db.txt, identical regardless of CK build (old container CK, fresh source CK) -- genuinely stale SystemDB entries, same class as gfx950's #2266 issue, not a CK-version-skew artifact. Also generalized the artifact fetch: calls install_rocm_from_artifacts.py directly (not via the setup_test_environment composite action, which has no --run-github-repo input and silently queries the wrong repo when the artifact was built directly on ROCm/TheRock, e.g. via a multiarch-ci-sweep dispatch, rather than through a rocm-libraries PR's own TheRock CI run). New artifact_repo input selects the source repo explicitly. ALMIOPEN-1779 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
✅ gfx942 restored to the matrix — root cause of the earlier stall not confirmed, but does not reproducePer feedback that we need a clean passing run (not a retreat to gfx950-only), investigated the gfx942 stall from run 31890772516 properly before giving up on it: Findings from the investigation:
Re-tested against a fresh artifact (built by pinning rocm-libraries to a commit just before an unrelated TheRock rocThrust regression, via the gfx942 now reliably surfaces a real, legitimate finding: 3 stale Restored gfx942 to the matrix ( Honest caveat: the original stall's root cause was never confirmed — it simply didn't reproduce against this fresh artifact/build. Follow-up (separate from this PR): both gfx942 and gfx950 have real stale-DB findings that look like #2266-class cleanup work ( |
…#2266) StaticFDBSync[gfx950] has failed on 2 stale ConvHipImplicitGemmGroupWrwXdlops perf-config entries in the gfx950 SystemDB (fdb-keys 128-7-7-1x1-1888-7-7-128-... and 240-14-14-3x3-240-14-14-16-...) every time it's been run, on both the old container CK and fresh source-built CK -- genuinely stale entries, not a CK-version artifact. Same class of issue PR #10043 fixed for gfx942, but no gfx950 hardware was available to run the usual real-hardware MIOPEN_DBSYNC_CLEAN pass. Fixed via the rocjitsu GPU-free dbsync CI (ALMIOPEN-1779): fetched a source-CK TheRock gfx950 artifact, ran StaticFDBSync[gfx950] under rocjitsu with MIOPEN_DEVICE_ARCH=gfx950 MIOPEN_DBSYNC_CLEAN=1 (rewrites the find/perf DB files in place, stripping only entries that fail applicability/perf-config validation), then verified a fresh process re-reading the cleaned files now passes with 0 failures. Diffed the cleaned files against the committed versions: exactly the 2 known-bad fdb-keys changed, with only the ConvHipImplicitGemmGroupWrwXdlops entry removed from each -- every other solver's entry for those keys is untouched. Re-enables gfx950 dbsync in TheRock CI (test/therock/test_miopen.py), removing the TODO(rocm-libraries#2266) exclusion -- dbsync now runs and passes for gfx950 like every other family. ALMIOPEN-1779 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Added: gfx950 SystemDB fix + dbsync re-enablePer JL's ask, used this PR's rocjitsu CI to actually fix gfx950's dbsync issue (not just diagnose it) — no gfx950 hardware needed. Mechanism: `db_sync.cpp`'s `MIOPEN_DBSYNC_CLEAN=1` mode, on the "invalid perf config" failure this PR has consistently found for gfx950, calls `find_db_rw.Remove()`/`perf_db_rw.Remove()` instead of just failing — `RamDb::Remove()` writes the cleaned record back in place. Ran the existing rocjitsu job with that flag set → the 2 stale `ConvHipImplicitGemmGroupWrwXdlops` entries were stripped from `build/share/miopen/db/gfx950100.{HIP.fdb,db}.txt` → verified a fresh process (CLEAN off, re-reading the modified files from disk) now passes with 0 failures → downloaded the cleaned files, diffed against committed versions (confirmed exactly the 2 known-bad fdb-keys changed, only the bad solver entry removed from each, nothing else touched) → re-bzip2'd (round-trip verified) and replaced the committed blobs. Commit
Note: gfx942's analogous stale-entry issue is being fixed separately/urgently in #10043 (currently breaking CI) — not touched here. (This supersedes standalone PR #10900, which is now closed.) |
Converts the standalone dbsync-rocjitsu prototype into an automatic gate wired into the real per-PR flow: adds a dbsync-rocjitsu job to therock-ci.yml, matrixed over [gfx942, gfx950] (the rocjitsu-supported set), gated on `needs.setup.outputs.linux_projects` containing "miopen" -- the same signal that already gates whether MIOpen's own `Test miopen` job runs per family, so this only runs when TheRock's MIOpen CI would run. Unlike therock-test-packages.yml's configure_test_matrix (gated on test_runs_on != '', i.e. requires a real GPU runner), this job needs no GPU: it runs on a GPU-less runner and self-references github.run_id for the artifact (this job executes within the same GHA run as the build, so no external artifact_run_id input is needed for the automatic path). The standalone therock-dbsync-rocjitsu-prototype.yml (workflow_dispatch, explicit artifact_run_id/artifact_repo inputs) stays as-is for manual/ diagnostic re-runs against a pinned commit -- this is what made the gfx942 stall investigation and the gfx950 SystemDB fix (both already in this PR) possible, and remains useful going forward. Per JL: gfx90a is out of scope here (no rocjitsu functional-emulation config exists for it yet) -- Jenkins' Dbsync gfx90a stage is untouched. Once this gate has baked in on real PRs, a follow-up PR will remove the now-redundant Dbsync gfx942 Jenkins stage and the *DBSync* filter from test/therock/test_miopen.py's real-hardware test path. ALMIOPEN-1779 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The prior gating (contains(linux_projects, '"miopen"')) never matched: linux_projects' "projects_to_test" field is a comma-separated STRING per bundle (e.g. "hipblas,miopen,rocblas"), not a JSON array of individually quoted strings -- confirmed live (job showed as skipped on PR #10844's own CI run, run 32194353093, even though miopen was clearly in the build matrix). A naive contains(x, 'miopen') substring check would have the opposite problem: false-matching "miopenprovider". Adds a small "Determine if MIOpen dbsync gate should run" step to the setup job that parses linux_projects properly and checks "miopen" as an exact comma-delimited token across all bundles, exposed as needs.setup.outputs.miopen_selected. Verified locally against the real linux_projects value from that run (True) and a miopenprovider-only negative control (False). ALMIOPEN-1779 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…fact dbsync-rocjitsu only depended on `setup`, so it started as soon as setup finished -- racing ahead of the actual MIOpen build (which uploads the artifact this job fetches via github.run_id). Confirmed live (run 32194708858): both legs failed at the fetch step with "Found no artifacts matching [...] at s3://...", since the build hadn't uploaded yet. Adds `therock-ci-linux` to needs. It's a matrixed reusable-workflow job, so `needs:` can only depend on it as a whole (waits for all its legs, not just gfx942/gfx950's) -- coarser than ideal but the only granularity available, and mirrors how therock_ci_summary already depends on it the same way. The job's own `if:` (miopen_selected == 'true') doesn't check success(), so an unrelated leg failing elsewhere in therock-ci-linux won't block this job from running once all legs have completed. ALMIOPEN-1779 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…k-ci-linux needs: therock-ci-linux (added to fix the earlier race) has its own problem: that job is one reusable-workflow matrixed over EVERY family/project bundle in the PR (build AND test for all of them), so depending on it blocks dbsync-rocjitsu on unrelated components' GPU test runs finishing too. Confirmed live (run 32195745347): the job sat unstarted for a long stretch with unrelated Linux/Windows test legs still queued, nothing to do with MIOpen. Reverts to needs: setup only, and instead wraps the artifact fetch in a bounded retry loop (60s interval, 45 min budget) that retries on the "Found no artifacts matching [...]" failure until this run's own MIOpen build has uploaded -- decoupled from every other component's test schedule. Tested the retry loop's bash logic locally against a mock command (success-after-N-failures and always-fails/timeout paths both behave correctly). Job-level timeout raised to 60 min to cover the wait plus the KMD build + dbsync run itself (observed 15-200s). ALMIOPEN-1779 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…n polling
install_rocm_from_artifacts.py exits 0 as soon as ANY artifacts exist at this run's S3
prefix -- confirmed live (run 32204574647): both gfx942 and gfx950 legs reported "Fetched
artifact after ~2190s" (retry succeeded), but the download only contained "generic"
(architecture-independent) pieces (amd-llvm, base, core-hip, ...), none of the requested
miopen_* artifacts, because MIOpen's own project bundle for that family was still
`in_progress` -- other bundles sharing the same family (hipkernelprovider, rpp, hiptensor,
...) had simply finished uploading their shared generic pieces first. The subsequent
"Inspect artifact" step then failed on `test -f ./build/bin/miopen_gtest`.
Folds that exact check into the retry condition itself (`... && [ -f
"${OUTPUT_ARTIFACTS_DIR}/bin/miopen_gtest" ]`) so the loop keeps waiting until the file
MIOpen's own bundle produces is actually present, not just until the fetch script's own
(too-lenient) success heuristic passes. Verified the updated loop locally against a mock
that creates the file on the 3rd poll.
ALMIOPEN-1779
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Superseded by #10994 — same changes on a cleanly-named branch ( |
JIRA: ALMIOPEN-1779 (also re-enables rocm-libraries#2266 if green)
What / why
A
workflow_dispatchprototype that proves the scalable dbsync design: fetch TheRock's per-PR source-CK build artifact for one arch and run the MIOpen dbsync check (StaticFDBSync) under the rocjitsu KMD interposer on a CPU runner (no GPU).This fixes the two dead-ends we hit:
rocm/miopen-dev:latest) false-fails whenever a SystemDB is refreshed ahead of the image, and can never catch a PR's own CK change (the [CK] Disable test cases affected by compiler codegen bugs on gfx90a #6343 scenario).Reusing the PR's own source-CK artifact avoids both: it matches the SystemDB, catches PR CK regressions, and — since dbsync is CPU-only (
SKIP_KDB_TESTING=1, no kernel launch) — any arch can be validated viaMIOPEN_DEVICE_ARCHon a GPU-less runner.Why gfx950 is the prototype
gfx950CK plugin + DBs exists to reuse.test_miopen.pyTEST_TO_IGNOREgfx950-dcgpu,TODO(rocm-libraries#2266)) and on Jenkins (test_categories.yamlexclude_gpu_gfx950). So this adds net-new coverage and is the mechanism to re-enable gfx950 dbsync GPU-free.How it works
Fetch reuses ROCm/TheRock's
setup_test_environmentaction (astherock-test-component.ymldoes); rocjitsu build/run reuses the pattern fromcomponent-ci-miopen.yml. Steps: fetch artifact → assert CK plugin + DBs present → buildrocjitsu_kmd_shim→ runmiopen_gtest --gtest_filter='*StaticFDBSync*'withMIOPEN_DEVICE_ARCH=<arch>underLD_PRELOAD.Verification (needs a live dispatch)
Dispatch with a recent develop TheRock build
artifact_run_id+ the MIOpenfetch_artifact_args(from TheRock'sfetch_test_configurations.py). Expect the inspect step to find the gfx950 CK plugin + DBs, andStaticFDBSync[gfx950]to run under rocjitsu against source CK.Known unknowns (prototype — expect first-dispatch iteration)
fetch_artifact_argsvalue + whether the dist carries the per-arch CK plugin and all DBs (the inspect step checks this explicitly).no_rocmimage TheRock tests use (minus GPU mounts); rocjitsu build deps are apt-installed. May need tuning.DBSynctests ongfx950-dcgpu#2266) — a failure here may be a real source-CK repro of [miopen] re-enableDBSynctests ongfx950-dcgpu#2266, not a harness bug.Scope
Standalone/dispatch-only — does not touch PR CI. If green, generalize into a test-flow job with a matrix over TheRock-built arches (gfx950, gfx1250, future). gfx90a/gfx908 aren't built by TheRock, so they stay on Jenkins unless TheRock builds them.
🤖 Generated with Claude Code