fix(rocke): compare arch-domain baselines across compiler builds

dnn-providers/hip-kernel-provider/rocke/TESTING.md

@@ -430,15 +430,14 @@ This gate and §4.4 overlap on purpose and neither subsumes the other:
 ### 4.6 Does the measured arch-domain artifact reproduce?
 
 The warning gate consumes committed measurements; it does not prove that this
-host's toolchain reproduces them. When the host compiler identity matches the
-provenance recorded in the committed column, the
-`rocke_arch_domain_reproducibility` CTest runs
-`tools/gen_arch_domain.py --check` over the full intrinsic/architecture matrix
-and compares the result with that column. LLVM development builds with the same
-major can disagree about intrinsic availability and ABI, so a different compiler
-identity is explicitly unvalidated rather than compared as though it asked the
-same question. This is an artifact-reproduction gate backed by the host
-toolchain, not a Python unit test: it owns external compiler processes, a
+host's toolchain reproduces them. The `rocke_arch_domain_reproducibility` CTest
+runs `tools/gen_arch_domain.py --check` over the full intrinsic/architecture
+matrix for the host's LLVM flavor and compares the result with that column.
+Compiler identity is recorded and printed as provenance, but a different build
+within the same flavor neither skips the probes nor fails the comparison by
+itself. Changes in measurements still fail, as do changes in probe configuration
+or other artifact fields. This is an artifact-reproduction gate backed by the
+host toolchain, not a Python unit test: it owns external compiler processes, a
 dedicated timeout and resource policy, and an explicit CTest skip result.
 
 The gate is registered in both execution environments. A developer build runs
@@ -449,10 +448,9 @@ stages every input, CTest supplies artifact-relative paths, and the test never
 configures or reaches back into a source checkout.
 
 The result mapping distinguishes applicability from failure: no clang, no
-committed host-flavor column, a clang/resolved-flavor mismatch, or a compiler
-identity different from the column's provenance skips with a named reason. A
-partial or internally inconsistent LLVM installation fails. Flavor
-auto-detection errors, including unavailable COMGR evidence, also remain
+committed host-flavor column, or a clang/resolved-flavor mismatch skips with a
+named reason. A partial or internally inconsistent LLVM installation fails.
+Flavor auto-detection errors, including unavailable COMGR evidence, also remain
 failures during this migration.
 
 The reproduction scheduler preserves exhaustive coverage while avoiding
@@ -474,10 +472,9 @@ CTest is categorized as standard in both build-tree and installed manifests, so
 installed-artifact CI can schedule the reproduction result without adding the
 external process sweep to the quick or FFM pytest pools. A named skip is not
 reproduction coverage: check the CTest result and reason, especially when the
-compiler identity differs from the committed provenance, its major differs from
-rocKE's resolved flavor, or no committed column exists. Keeping one generator
-implementation behind two thin registrations preserves local coverage while
-avoiding duplicated test logic.
+compiler's major differs from rocKE's resolved flavor or no committed column
+exists. Keeping one generator implementation behind two thin registrations
+preserves local coverage while avoiding duplicated test logic.
 
 **Kernel-author routing.** Ordinary kernel work does not need a separate focused
 run beyond the full pre-merge `tests/run_all.py`; once that runner has configured

dnn-providers/hip-kernel-provider/rocke/platform/tests/tools/test_arch_domain_artifact.py

@@ -17,12 +17,12 @@
   removed while the artifact still looks authoritative.
 
 * **Regeneration** is the dedicated ``rocke_arch_domain_reproducibility`` CTest.
-  When the host has the exact compiler build recorded in the column, it
-  re-probes the toolchain and asserts the committed measurements reproduce
-  cell-for-cell. A different compiler build is explicitly unvalidated: an LLVM
-  major is not a stable intrinsic-availability contract. It is the only check
-  that can catch a wrong measurement rather than a missing one. Its external compiler processes,
-  timeout, resource policy, and skip result do not belong in the unit-test pool.
+  With a matching LLVM flavor, it re-probes the toolchain and compares the
+  committed measurements cell-for-cell, even when the compiler build differs.
+  Compiler identity is diagnostic provenance, not a reason to skip or fail the
+  comparison. It is the only check that can catch a wrong measurement rather
+  than a missing one. Its external compiler processes, timeout, resource policy,
+  and skip result do not belong in the unit-test pool.
 
 A host can only ever measure its own LLVM, so a flavor with no matching
 toolchain is skipped, never failed — "we did not get an answer" must not be
@@ -349,11 +349,7 @@ def test_jobs_parser_accepts_auto_and_positive_but_rejects_negative(self):
 
 
 class DriftTest(unittest.TestCase):
-    """What an applicable ``--check`` treats as drift.
-
-    The caller rejects a compiler-provenance mismatch before probing. Once the
-    exact recorded compiler is present, every artifact field is load-bearing.
-    """
+    """Every artifact field except compiler identity remains load-bearing."""
 
     def _doc(self, **toolchain) -> str:
         tc = {
@@ -374,14 +370,12 @@ def _doc(self, **toolchain) -> str:
             }
         )
 
-    def test_a_different_compiler_build_is_drift_when_compared(self):
+    def test_a_different_compiler_build_is_not_drift(self):
         other = "AMD clang version 22.0.0git (... roc-7.2.0 26014 def)"
-        why = G._drift(self._doc(), self._doc(clang=other))
-        self.assertIsNotNone(why)
-        self.assertIn("toolchain", why)
+        self.assertIsNone(G._drift(self._doc(), self._doc(clang=other)))
 
     def test_a_changed_measurement_is_drift_and_says_which(self):
-        fresh = json.loads(self._doc())
+        fresh = json.loads(self._doc(clang="AMD clang version 22.0.0git (other)"))
         fresh["keys"]["a.b"]["gfx942"]["status"] = G.STATUS_ARCH_ABSENT
         why = G._drift(self._doc(), json.dumps(fresh))
         self.assertIsNotNone(why)
@@ -395,9 +389,13 @@ def test_a_changed_probe_method_is_drift_even_with_identical_cells(self):
             ("probe_cflags", ["-nogpulib"]),
             ("probe_timeout_s", 5),
             ("generator", G.GENERATOR + 1),
+            ("flavor", "llvm23"),
+            ("arches", ["gfx950"]),
         ):
             with self.subTest(field=field):
-                why = G._drift(self._doc(), self._doc(**{field: value}))
+                why = G._drift(
+                    self._doc(), self._doc(clang="other build", **{field: value})
+                )
                 self.assertIsNotNone(why, field)
                 self.assertIn("toolchain", why)
 
@@ -415,9 +413,10 @@ def test_healthy_declarations_share_one_opt_process(self):
             "declare i32 @llvm.amdgcn.example(i32) #1\n"
         )
         completed = mock.Mock(returncode=0, stdout=output, stderr="")
-        with tempfile.TemporaryDirectory() as td, mock.patch.object(
-            G, "run_bounded", return_value=completed
-        ) as run:
+        with (
+            tempfile.TemporaryDirectory() as td,
+            mock.patch.object(G, "run_bounded", return_value=completed) as run,
+        ):
             verdicts = G._name_exists_many(
                 "opt", [("fabs", self._A), ("example", self._B)], Path(td)
             )
@@ -430,9 +429,10 @@ def test_healthy_declarations_share_one_opt_process(self):
     def test_alias_keys_share_one_declaration_query(self):
         output = "declare float @llvm.fabs.f32(float) #0\n"
         completed = mock.Mock(returncode=0, stdout=output, stderr="")
-        with tempfile.TemporaryDirectory() as td, mock.patch.object(
-            G, "run_bounded", return_value=completed
-        ) as run:
+        with (
+            tempfile.TemporaryDirectory() as td,
+            mock.patch.object(G, "run_bounded", return_value=completed) as run,
+        ):
             verdicts = G._name_exists_many(
                 "opt", [("fabs", self._A), ("fabs-alias", self._A)], Path(td)
             )
@@ -445,9 +445,10 @@ def test_a_failed_batch_is_bisected_to_singleton_answers(self):
         failed = mock.Mock(returncode=1, stdout="", stderr="bad combined module")
         a = mock.Mock(returncode=0, stdout=self._A + " #0\n", stderr="")
         b = mock.Mock(returncode=0, stdout=self._B + " #0\n", stderr="")
-        with tempfile.TemporaryDirectory() as td, mock.patch.object(
-            G, "run_bounded", side_effect=(failed, a, b)
-        ) as run:
+        with (
+            tempfile.TemporaryDirectory() as td,
+            mock.patch.object(G, "run_bounded", side_effect=(failed, a, b)) as run,
+        ):
             verdicts = G._name_exists_many(
                 "opt", [("fabs", self._A), ("example", self._B)], Path(td)
             )
@@ -625,9 +626,12 @@ def test_target_preflight_retries_transient_and_changed_answers_serially(self):
             "unsupported": G.STATUS_TARGET_UNSUPPORTED,
             "supported": G.STATUS_OK,
         }
-        with tempfile.TemporaryDirectory() as td, mock.patch.object(
-            G, "_target_preflight", return_value=(G.STATUS_OK, "")
-        ) as probe:
+        with (
+            tempfile.TemporaryDirectory() as td,
+            mock.patch.object(
+                G, "_target_preflight", return_value=(G.STATUS_OK, "")
+            ) as probe,
+        ):
             retried = G._retry_target_preflights(
                 "clang", path, Path(td), preflight, expected
             )
@@ -1287,8 +1291,9 @@ def test_a_disagreeing_opt_and_clang_write_nothing_and_fail(self):
         # Pin the flavor so the earlier clang-vs-rocke check passes on any host
         # and the run reaches the pairing check.
         for extra in ((), ("--check", "--unvalidated-exit-code", "77")):
-            with self.subTest(adapter=bool(extra)), mock.patch.dict(
-                os.environ, {"ROCKE_LLVM_FLAVOR": "llvm22"}
+            with (
+                self.subTest(adapter=bool(extra)),
+                mock.patch.dict(os.environ, {"ROCKE_LLVM_FLAVOR": "llvm22"}),
             ):
                 rc = self._run(tools, out, *extra)
                 self.assertEqual(rc, 2, "two vintages measured one column")
@@ -1321,34 +1326,88 @@ def test_ctest_can_translate_a_flavor_mismatch_to_a_skip(self):
         self.assertEqual(rc, 77)
         self.assertFalse(out.exists())
 
-    def test_ctest_skips_a_column_from_a_different_compiler_build(self):
-        """One LLVM major does not make development builds interchangeable."""
+    def _baseline_from_other_compiler(self) -> tuple[Path, dict[str, str]]:
+        """Measure with stub tools, then change only the recorded provenance."""
         out = Path(tempfile.mkdtemp()) / "column.json"
         self.addCleanup(shutil.rmtree, out.parent, True)
-        blessed = "AMD clang version 22.0.0git (blessed-build)"
-        host = "AMD clang version 22.0.0git"
-        out.write_text(json.dumps({"toolchain": {"clang": blessed}}))
         tools = {
             "clang": self._clang_with_accepting_assembler(self._CLANG_ACCEPTS),
             "opt": self._stub("opt", self._OPT_RESOLVES_AND_IDENTIFIES),
         }
+        with mock.patch.dict(os.environ, {"ROCKE_LLVM_FLAVOR": "llvm22"}):
+            self.assertEqual(self._run(tools, out, "--only", "fabs.f32"), 0)
+        doc = json.loads(out.read_text())
+        self.assertEqual(doc["keys"]["fabs.f32"]["gfx942"]["status"], G.STATUS_OK)
+        doc["toolchain"]["clang"] = "AMD clang version 22.0.0git (blessed-build)"
+        out.write_text(json.dumps(doc))
+        return out, tools
+
+    def test_ctest_compares_a_different_compiler_build_and_passes(self):
+        out, tools = self._baseline_from_other_compiler()
+        before = out.read_text()
         stdout = io.StringIO()
         with (
             mock.patch.dict(os.environ, {"ROCKE_LLVM_FLAVOR": "llvm22"}),
             mock.patch("sys.stdout", stdout),
+            mock.patch.object(G, "_probe", wraps=G._probe) as probe,
         ):
             rc = self._run(
                 tools,
                 out,
+                "--only",
+                "fabs.f32",
                 "--check",
                 "--unvalidated-exit-code",
                 "77",
             )
 
-        self.assertEqual(rc, 77)
-        self.assertIn("UNVALIDATED", stdout.getvalue())
-        self.assertIn(blessed, stdout.getvalue())
-        self.assertIn(host, stdout.getvalue())
+        self.assertEqual(rc, 0)
+        self.assertGreater(probe.call_count, 0, "skipped the measurement")
+        self.assertIn("semantically unchanged", stdout.getvalue())
+        self.assertIn("(blessed-build)", stdout.getvalue())
+        self.assertIn("this host : AMD clang version 22.0.0git", stdout.getvalue())
+        self.assertNotIn("UNVALIDATED", stdout.getvalue())
+        self.assertEqual(out.read_text(), before, "--check re-blessed provenance")
+
+    def test_ctest_different_compiler_build_still_fails_on_measurement_drift(self):
+        """A new compiler rejecting a previously accepted intrinsic is a failure."""
+        out, tools = self._baseline_from_other_compiler()
+        before = out.read_text()
+        tools["clang"] = self._clang_with_accepting_assembler(
+            """
+import sys
+if "--version" in sys.argv:
+    print("AMD clang version 22.0.0git")
+    sys.exit(0)
+if "@llvm.fabs.f32(" in open(sys.argv[-1]).read():
+    sys.stderr.write("error: Cannot select: intrinsic %llvm.fabs.f32\\n")
+    sys.exit(1)
+sys.exit(0)
+"""
+        )
+        stdout = io.StringIO()
+        with (
+            mock.patch.dict(os.environ, {"ROCKE_LLVM_FLAVOR": "llvm22"}),
+            mock.patch("sys.stdout", stdout),
+            mock.patch.object(G, "_probe", wraps=G._probe) as probe,
+        ):
+            rc = self._run(
+                tools,
+                out,
+                "--only",
+                "fabs.f32",
+                "--check",
+                "--unvalidated-exit-code",
+                "77",
+            )
+
+        self.assertEqual(rc, 1, "measurement drift must fail, not pass or skip")
+        self.assertGreater(probe.call_count, 0, "skipped the measurement")
+        self.assertIn("fabs.f32/gfx942: ok -> arch_absent", stdout.getvalue())
+        self.assertIn("(blessed-build)", stdout.getvalue())
+        self.assertIn("this host : AMD clang version 22.0.0git", stdout.getvalue())
+        self.assertNotIn("UNVALIDATED", stdout.getvalue())
+        self.assertEqual(out.read_text(), before, "--check changed the baseline")
 
     def test_ctest_can_translate_a_missing_column_to_a_skip(self):
         """A measured flavor with no baseline is unvalidated, not green."""
@@ -1725,9 +1784,11 @@ def test_skip_code_must_be_nonzero_and_check_only(self):
             ("--check", "--unvalidated-exit-code", "0"),
             ("--unvalidated-exit-code", "77"),
         ):
-            with self.subTest(args=args), mock.patch.object(
-                sys, "argv", [str(_TOOL), *args]
-            ), self.assertRaises(SystemExit):
+            with (
+                self.subTest(args=args),
+                mock.patch.object(sys, "argv", [str(_TOOL), *args]),
+                self.assertRaises(SystemExit),
+            ):
                 G.main()
 
 

dnn-providers/hip-kernel-provider/rocke/platform/tools/gen_arch_domain.py

@@ -1024,10 +1024,9 @@ def _decl_table(flavor: str) -> dict[str, str]:
 def _drift(committed: str, fresh: str) -> str | None:
     """Describe how a committed column differs from a fresh run, or None.
 
-    Exact reproduction is meaningful only with the compiler build recorded in
-    ``toolchain.clang``. The caller checks that provenance before probing, so a
-    different compiler build is unvalidated rather than drift. Once applicable,
-    every toolchain field remains part of the artifact: ``generator``,
+    ``toolchain.clang`` records provenance, not an expected measurement: another
+    compiler build must still be compared against the baseline. Every other
+    field remains part of the comparison. In particular, ``generator``,
     ``probe_cflags`` and ``probe_timeout_s`` say how the question was asked, and
     changing any of them means the committed answers were obtained by a method
     we no longer use. ``probe_cflags`` is the concrete example -- the same clang
@@ -1039,6 +1038,9 @@ def _drift(committed: str, fresh: str) -> str | None:
     except json.JSONDecodeError as exc:
         return f"committed column is not valid JSON: {exc}"
 
+    for doc in (a, b):
+        doc.get("toolchain", {}).pop("clang", None)
+
     if a.get("keys") != b.get("keys"):
         changed = [
             f"{k}/{arch}: {a['keys'][k][arch]['status']} -> {cell['status']}"
@@ -1470,12 +1472,9 @@ def main() -> int:
         )
         return 1
 
-    # A measured column is an artifact of one exact compiler build, not every
-    # compiler that reports the same LLVM major. Development snapshots within a
-    # flavor can change intrinsic availability and even ABI. Re-probing with a
-    # different build would compare two different questions and make ordinary
-    # CI depend on whichever compiler-runtime artifact it happened to receive.
-    # Report that honestly as unvalidated before starting the expensive sweep.
+    # Keep compiler provenance visible, but do not let a different build skip
+    # the comparison: changes in its intrinsic availability are what this gate
+    # must detect. The flavor and toolchain-consistency checks above still apply.
     if args.check and out.is_file():
         try:
             committed_doc = json.loads(out.read_text())
@@ -1485,12 +1484,11 @@ def main() -> int:
         blessed_identity = committed_doc.get("toolchain", {}).get("clang")
         if blessed_identity and blessed_identity != identity:
             print(
-                f"\nUNVALIDATED: {out.name} was measured with a different "
-                "compiler build; exact artifact reproduction is not applicable.\n"
+                f"\nNOTE: {out.name} was measured with a different "
+                "compiler build; comparing its measurements against the baseline.\n"
                 f"      blessed by: {blessed_identity}\n"
                 f"      this host : {identity}"
             )
-            return _unvalidated_exit(args.unvalidated_exit_code, 0)
 
     arches = sorted(args.arch) if args.arch else sorted(wired_arches())
     decls = _decl_table(flavor)