[shumway]

ISSUE ID : AICK-2713

Summary

Optimize the exhaustive rocKE arch-domain reproducibility CTest by roughly 10x at the measured LLVM 23 configuration, without reducing its matrix, frequency, compiler isolation, or verdict strength.

Why

The gate validates every intrinsic declaration against every enrolled architecture, but its original implementation repeated work that belongs to broader logical axes:

• one opt process per declaration;
• repeated target-support discovery for every intrinsic;
• one compile/link sequence per expected-positive cell;
• speculative immediate-value sweeps after malformed default operands;
• serial reruns of deterministic expected compiler crashes.

The expensive part was process orchestration, not the coverage requirement. Expected-negative cells still need independent probes because a compiler fatal can terminate a combined module before later calls are evaluated.

Design decisions

Decision	Reason
Batch declaration discovery and recursively split failures	Recognition and canonicalization are architecture-independent
Preflight each compiler/architecture pair once	Target support is independent of the intrinsic
Use explicit ProbeSpec witnesses	Start from semantically valid operands instead of discovering them through speculative retries
Batch committed expected positives by architecture	A successful combined module proves every included call compiled, linked, and passed target assembly
Recursively bisect failed positive batches	Preserve exact changed-cell diagnostics
Keep expected negatives individual	A failing compiler may stop before evaluating later calls
Use the committed artifact only to schedule work	The expected answer never determines the measured verdict
Confirm unexpected outcomes serially	Separate real drift from host contention
Use bounded, quota-aware concurrency	Reduce wall time without unbounded process fan-out

Artifact generation remains per-cell. Optimized scheduling applies to reproduction and shares the same probe construction and result classification.

Improvements

Bug fixes

• Changed definitive verdicts receive contention-free confirmation before drift is reported.
• Non-definitive target preflights retry only after the parallel pool drains.
• Intrinsics with nonzero valid immediates no longer begin from malformed operands that can look like target rejection.

Cleaner design

• ProbeSpec records the valid witnesses used to construct a probe.
• Scheduling, positive-batch localization, target preflight, and retry selection are independently testable helpers.
• Generation and reproduction scheduling are distinct while sharing construction and classification.

Better test coverage

• Pins the complete scheduling partition so every measurable cell is assigned exactly once.
• Proves positive batches retain every call and still receive target-assembly validation.
• Proves recursive localization avoids singleton probes for healthy partitions.
• Pins explicit witness mappings and recorded witness fields.
• Covers declaration batching, alias deduplication, preflight, changed-verdict retry, and deterministic-crash policy.

Faster execution

• One batched declaration query replaces per-declaration discovery.
• One target preflight is performed per architecture.
• Expected positives share one module per architecture in a green run.
• Known-valid witnesses replace speculative value sweeps.
• Reproduced deterministic crashes and timeouts are not rerun.
• Independent work shares a bounded worker pool.

Performance

Measured scope: the focused host-only rocke_arch_domain_reproducibility CTest over the complete LLVM 23 column: 205 declaration keys across gfx11-generic, gfx1151, gfx12-generic, gfx1201, gfx1250, gfx90a, gfx942, and gfx950.

On the same host with AMD clang 23 build 8f497e0992fb7513f7f78a6f6b6f1056c375e961, the final implementation reduced focused-CTest wall time by approximately 91%—about an 11x speedup—relative to parent commit 784462004d, at this measured configuration. Both arms reproduced the committed artifact cell-for-cell. The result is sensitive to host load, filesystem/compiler page cache, CPU quota, and the selected worker count; it is not a general performance claim.

Replay:

ROCKE_LLVM_BIN=<matching-llvm23-bin> \
ctest --test-dir <configured-platform-build> \
  -R '^rocke_arch_domain_reproducibility$' --output-on-failure

Validation

• Fast arch-domain unit and structure tests
• Runner and CTest registration contract tests
• Full LLVM 23 reproduction check
• Full LLVM 22 reproduction check with the exact recorded ROCm 7.2.4 toolchain
• LLVM 20 fresh generation is byte-identical to develop with the exact recorded ROCm 7.1.1 toolchain
• Fresh LLVM 23 generation is byte-identical to the committed artifact
• Focused build-tree CTest
• Pre-commit over the PR range

LLVM 20's committed column has pre-existing evidence/coverage drift under its recorded toolchain: both develop and this branch fail --check with identical classification counts, while their freshly generated documents are byte-identical. This PR preserves that baseline behavior and does not attempt to add gfx1250 support to LLVM 20.

TheRock CI coverage note

TheRock CI for this repository currently uses LLVM 24. rocKE does not yet have a committed LLVM 24 arch-domain column, so rocke_arch_domain_reproducibility intentionally reports a CTest skip in those jobs rather than claiming reproduction coverage for the wrong compiler flavor. Those jobs validate registration, packaging, relocation, and skip semantics, but they do not execute the full compiler matrix added by this PR.

The full reproduction path for this change was validated locally with matching committed toolchains, including LLVM 23 from ROCm 10.0. LLVM 24 arch-domain and CI qualification support will follow in a separate PR after this change.

Deferred follow-up

Kept out of this PR intentionally:

• scheduler-specific provenance and generated-comment schema changes;
• operand-position witness modeling for future declarations with disjoint domains;
• typed plan/result objects and further orchestration refactoring;
• persistent or preforked compiler workers;
• lower-level LLVM/COMGR probing and safe cross-run caching;
• investigation and remediation of the pre-existing LLVM 20 committed-artifact drift.

[therock-pr-bot]

✅ All Checks Passed — Ready for Review

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🔎 pre-commit	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—
🤖 therock-pr-bot	✅ Pass	—

🎉 All checks passed! This PR is ready for review.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

[therock-pr-bot]

🎉 All checks passed! This PR is ready for review.

[Copilot]

🟡 Changes recommended

Changed crash evidence can bypass serial confirmation and cause load-dependent reproduction failures.

1 open issue

• Retry matching crash statuses when normalized evidence differs · New

What changed in this PR

This PR reduces compiler orchestration overhead in rocKE’s exhaustive arch-domain reproducibility gate while preserving per-cell coverage and target-assembly validation.

Changes:

• Batches declaration discovery and expected-positive probes, recursively isolating failures.
• Adds target preflights, explicit operand witnesses, bounded concurrency, and selective serial retries.
• Expands scheduler tests and documents the optimized reproduction process.

File	Description
dnn-providers/​hip-kernel-provider/​rocke/​TESTING.md	Documents scheduling, coverage, and retry behavior.
dnn-providers/​hip-kernel-provider/​rocke/​platform/​tools/​gen_arch_domain.py	Implements batching, preflights, witnesses, and retry scheduling.
dnn-providers/​hip-kernel-provider/​rocke/​platform/​tests/​tools/​test_arch_domain_artifact.py	Adds coverage for scheduling, batching, witnesses, and retries.
dnn-providers/​hip-kernel-provider/​rocke/​platform/​tests/​README.md	Explains reproduction scheduling and concurrency limits.

_{Review effort: Balanced}

[CongMa13]

No new issues found.

[CongMa13]

No new P0 or P1 issues were identified during this review. The changes are approved.

[cgmillette]

Automated review: Model - qwen3.6:35b-a3b-coding-mtp-q4_K_M, effort: max

PR #14355 — optimize arch-domain reproduction scheduling — 🟢 First-round (posted after the merge): no closer look needed — on a real compiler the scheduled reproduction agrees with the old per-cell one and names planted drift exactly; one 🟡: none of the new code runs in CI

Prior item	Status
Copilot (10-24): retry matching crash statuses when the evidence changes	🟢 FIXED in d743d131; the thread is resolved (read)
brockhargreaves-amd, CongMa13 APPROVED (10-26)	🟢 Both at head d743d131; merged 10-26
Bots / CI	🟢 Green at d743d131. As the description says, rocke_arch_domain_reproducibility skips in every TheRock and hipDNN lane, because CI runs LLVM 24 and there is no llvm24 column (read)
Our checks	🟢 Real-compiler equivalence (ran). This host has only LLVM 24, which the tool refuses, so a scratch shim presented it as llvm23. The test used the MFMA, WMMA and cvt key subsets: 376 cells covering ok, arch_absent, name_absent and toolchain_crash. On them, pre-merge develop's generator and this PR's generator write the same cells, and both the old per-cell check and the new scheduled check reproduce them. Planted drift was reported exactly and on its own (ran): one cell per copy, either ok → arch_absent; arch_absent → ok inside gfx950's batch of 21 real positives; or a compiler-crash cell marked ok inside gfx12-generic's batch of 6 positives. On these subsets the scheduled check ran 4.5–6.8× faster (ran). The unit tests pass: 69, against 51 before (ran). The merge with #14359 is clean, and rule_scan is clean (ran)
LLVM 24 (for the planned column)	🟢 A full LLVM 24 sweep stops on 29 cells whose diagnostic reads "… requires target feature '…'". Examples are permlane16, permlanex16 and mov.dpp8 on CDNA, and raw.ptr.buffer.load(.async).lds on RDNA and gfx1250. Neither the old nor the new _classify knows that wording, so both refuse to write the column (ran)

#	Sev	file:line	Finding
N1	🟡	platform/CMakeLists.txt:400	No CI lane runs any of this PR's code. The reproduction CTest skips under CI's LLVM 24. test_arch_domain_artifact.py holds the 69 host-only tests that pin the scheduling partition, batching, bisection and retry policy. It is excluded from the installed tree and has no build-tree registration, so neither TheRock's rocke_pytest nor the hipDNN superbuild runs it (read). The file was already uncovered before this PR, but the scheduler now decides verdicts on the strength of those tests. A build-tree CTest for the file (about 1 s) would put it in the hipDNN superbuild.

Utility: 🟢 Positive — the exhaustive reproduction gate gets much cheaper without dropping a cell, and it still names drift exactly, including inside a batch.

---

Legend: 🔴 blocking · 🟠 should-fix · 🟡 minor nit · 🟢 info/good · ⚪ retracted

First-round review, posted after the merge: a local qwen3.6 model made the independent first pass. The orchestrator (Claude Opus 5.5) verified it. The checks were generation and reproduction on a real compiler with pre-merge develop and this PR, on three key subsets, plus planted-drift controls. They also included the unit tests on both trees, CI-enrollment reads, a scratch merge with #14359 and the CI logs at head. The orchestrator owns this merge-worthiness call. It flags whether a closer look is needed; it is not an approval or a rejection.

[assistant-librarian]

TheRock Submodule Bump Activity

Newest first

• 2026-10-27 — Included in TheRock via ROCm/TheRock#9513.