feat(hipdnn-integration-tests): no-reference and expected-rejection bundle metadata flags

dnn-providers/integration-tests/docs/README.md

@@ -53,6 +53,7 @@ other rows do not, so they are easy to miss.
 | `FATAL: --enforce-support-claims is active and …` | **yes** | Enforcement verified nothing. | [Hard stops](running-tests.md#hard-stops) |
 | `benchmark failed: …` | **yes** (that test failed) | Under `--benchmark`, the verified graph failed when it ran again, or the timer returned too few measurements. | [Time the engine](running-tests.md#4-time-the-engine---benchmark) |
 | `unclaimed_support` in the `SUPPORT CLAIM SUMMARY` | no | The engine accepts graphs that no sidecar claims. Record them. | [Updating support claims](adding-tests.md#updating-support-claims) |
+| `[ VERIFIER ] rejected(<stage>): <bundle>` / `Verified by: … rejected R` | no | An expected-rejection bundle passed: the engine rejected a graph that is invalid on purpose. This is the intended pass, not a skip. | [Expected-rejection bundles](running-tests.md#expected-rejection-bundles) |
 | `Skipped:` equal to (or close to) the total | no | Green, but little or nothing was tested. Read the skip reasons. | [Test coverage summary](running-tests.md#test-coverage-summary) |
 | `WARNING: Bundle tests are enabled but …` / `WARNING: No bundles could be loaded from …` | no | No bundles registered. Only compiled-in C++ tests ran. | [Troubleshooting](running-tests.md#troubleshooting) |
 | `No tests were found!!!` from CTest | no (exit 0) | Wrong directory, wrong label, or only disabled suites. | [Runs that look green but are not](running-tests.md#runs-that-look-green-but-are-not) |

dnn-providers/integration-tests/docs/adding-tests.md

@@ -43,7 +43,9 @@ A bundle of any kind is complete only when it has all three required parts
 3. **A support-claim sidecar** — `{Name}.support.json`, or the sweep's
    `support.json` — even if it claims nothing yet.
 
-Golden data is the only optional part.
+Golden data is the only optional part. An expected-rejection bundle is the
+exception: it has no support claim and no golden data
+([details](#add-an-expected-rejection-bundle)).
 
 ## End to end: a bundle for a new op
 
@@ -190,6 +192,94 @@ one of these:
 - **quarantine the bundle** until one of the above lands: exclude it in the
   provider's `test_categories_integration.yaml` with a comment that says why.
 
+### Disable a reference for a bundle
+
+Set `no_cpu_reference` or `no_gpu_reference` to `true` in the metadata of the
+bundle (`{Name}.meta.json`, or `cases[].metadata` of a sweep case) when that
+reference executor is known to be wrong for the bundle, or too slow to run on
+it. The harness never uses a disabled reference as an oracle
+([format](file-formats.md#disabling-a-reference-oracle--no_cpu_reference-and-no_gpu_reference)).
+
+```json
+{ "format_version": 1, "generator": "…", "reference_source": "…", "no_cpu_reference": true }
+```
+
+- **Disabling both requires golden data.** Add it first
+  ([Golden data with DVC](#golden-data-with-dvc)). A bundle with both flags and
+  no declared golden data fails to load. `verify_golden_bundles.py` reports it
+  as an error.
+- **A known gap is a different tool.** `knownReferenceGaps()`
+  (`ReferenceOpCoverage.cpp`) records an op that a reference executor cannot
+  handle. If a bundle has the flag for a lane and a gap entry for the same
+  lane, `hipdnn_golden_data_tests` fails that lane's test and names both. Delete
+  the gap entry.
+- **What a run prints.** In `auto` mode, a disabled reference appears in the
+  failure message as `(disabled by bundle metadata)`. With
+  `--verification-mode=cpu|gpu` for a disabled reference, the test skips. With
+  both references disabled and the golden data not pulled, the test skips with
+  `golden data not pulled; bundle disables both references`. In
+  `hipdnn_golden_data_tests`, the disabled lane registers a skip test that names
+  the flag. See [Bundle-disabled references](running-tests.md#bundle-disabled-references).
+
+### Add an expected-rejection bundle
+
+Use this to prove that an engine **rejects** a graph that is invalid on purpose.
+The test is never executed or compared
+([format](file-formats.md#expected-rejection-bundles)).
+
+1. **Choose one engine-visible invalid property.** The graph must load and
+   deserialize, and the engine must see the problem. Example: hipBLASLt
+   refuses a Matmul whose batch dimensions are neither equal, nor 1 on one
+   side, nor broadcast-compatible (they are invalid per the Matmul contract too),
+   so change the batch dimension of one input. A property that the JSON or the graph schema cannot even load is not
+   engine-visible: it is a red `Failed to load expected_rejection bundle …` test.
+2. **Derive it from a claimed valid case.** Copy the graph of a case that an
+   engine claims (here a hipBLASLt-claimed `quick/Matmul/Default` case) and
+   change only that property. The valid twin is the **positive control**: it
+   passes in the same lane, so the rejection comes from your change.
+3. **Place it** at
+   `integration-test-bundles/<tier>/<Op>/<Layout>/<DataType>/<Name>/<Name>.json`:
+   six path segments under the bundle root, as `verify_golden_bundles.py`
+   expects. Example:
+   `integration-test-bundles/quick/Matmul/ncl/fp32/ExpectedRejectionMatmulBatchMismatch/ExpectedRejectionMatmulBatchMismatch.json`.
+4. **Write `<Name>.meta.json`** with `format_version`, `generator` and
+   `reference_source` (the verifier requires both as non-empty strings) and
+   `expected_rejection: true`:
+
+   ```json
+   { "format_version": 1, "generator": "hand-authored", "reference_source": "none: expected_rejection bundle (never executed)", "expected_rejection": true }
+   ```
+
+5. **Add no sidecar and no golden data.** Do not hand-add a claim for the
+   bundle. A claim that names it fails the C++ test and
+   `verify_support_claims.py`. A `--write-support-claims` run skips it and writes
+   no claim. Golden data next to it is a load error.
+6. **Re-run CMake, then run the lane pinned to the engine** (an unpinned run
+   skips with `expected_rejection requires --test-engine`):
+
+   ```bash
+   build/bin/hipdnn_integration_tests --test-article <plugin> --test-engine HIPBLASLT_ENGINE \
+       --test-config dnn-providers/hipblaslt-provider/config/HIPBLASLT_ENGINE.toml \
+       --gtest_filter='quick_Matmul_ncl_fp32_ExpectedRejectionMatmulBatchMismatch.*'
+   ```
+
+   The test prints `[ VERIFIER ] rejected(<stage>): <path>`. For this example
+   hipBLASLt drops the graph from its ranked list (`ranked_list`). Lanes whose
+   engine has no Matmul support, such as MIOpen and hip-kernel, reject vacuously at
+   `ranked_list`.
+7. **Run the verifiers** on the bundle directory
+   ([Check the data itself](#check-the-data-itself)), and
+   `python dnn-providers/integration-tests/scripts/verify_support_claims.py`.
+
+What a failure means:
+
+| Failure | Meaning |
+|---|---|
+| `Engine X accepted expected-rejection bundle …` | X ranked the graph and built its plans. The chosen property is not one that X rejects, or X now accepts it. |
+| `expected-rejection bundle never reached an engine` | The graph failed `from_binary`. The invalidity is schema-level. Choose an engine-visible property. |
+| `Failed to load expected_rejection bundle …` | The JSON or graph does not load, or the metadata is malformed (a non-boolean `expected_rejection`, golden data declared next to it). |
+| A claim error | A sidecar claims the bundle. Remove the claim. |
+
 ### Finding existing cases
 
 ```bash
@@ -635,7 +725,8 @@ the harness (`hipdnn_integration_tests_unit_tests`).
 - [ ] New graph coverage is a bundle, not a C++ test.
 - [ ] Every new bundle has its graph, its metadata (with `generator` and
       `reference_source`) and a support sidecar, even an empty one. No tool
-      checks for a missing sidecar, so check by hand.
+      checks for a missing sidecar, so check by hand. An expected-rejection
+      bundle is the exception: it has no sidecar and no golden data.
 - [ ] `verify_golden_bundles.py` reports no errors for the bundle directories you
       added or changed ([Check the data itself](#check-the-data-itself)).
 - [ ] You re-ran CMake after you changed bundles or sidecars or pulled data.