feat(integration-tests): render the engine support matrix from sidecars

.pre-commit-config.yaml

@@ -49,7 +49,13 @@ exclude: |
     projects/hipdnn/.*\.bin$|
     # Generated hipDNN FlatBuffers headers
     projects/hipdnn/flatbuffers_sdk/include/hipdnn_flatbuffers_sdk/data_objects/.*_generated\.h|
-    # Reference data (auto-generated)
+    # Reference data (auto-generated). The generic hooks would fight the
+    # generators: some graph JSON is written without a final newline, which
+    # end-of-file-fixer would rewrite, and sweep manifests run past 1 MB, over
+    # check-added-large-files' limit. The exclusion also hides the support.json
+    # sidecars from the RFC 0015 hooks below; the real tree is checked instead
+    # by test_render_support_matrix.py, which runs as a ctest in
+    # hipdnn-superbuild-ci on every change under dnn-providers/.
     dnn-providers/integration-tests/integration-test-bundles/.*|
     # Generated files (see rocrand/tools/ _generator.cpp files)
     projects/rocrand/library/.*constants\.(cpp|h)|
@@ -156,7 +162,36 @@ repos:
         entry: python dnn-providers/integration-tests/scripts/verify_support_claims.py
         language: python
         pass_filenames: false
-        files: ^dnn-providers/integration-tests/(integration-test-bundles/.*|scripts/(test_)?verify_support_claims\.py)$
+        files: ^dnn-providers/integration-tests/(integration-test-bundles/.*|scripts/((test_)?verify_support_claims|bundle_discovery)\.py)$
+
+      # SUPPORT_MATRIX.md is generated, not committed (see
+      # dnn-providers/integration-tests/.gitignore for why), so there is no
+      # committed copy to check for staleness. Render and discard to catch
+      # errors in the script itself (import, syntax) and in the sidecars it
+      # reads: a malformed or orphaned sidecar, a claim on an unknown case,
+      # tier copies of a case holding different graphs, a bundle not under
+      # <tier>/<Op>/.
+      # `--output -` rather than /dev/null so the hook works on Windows too;
+      # pre-commit hides the stdout of a hook that passes.
+      # Scoped with `files:` rather than always_run: CI checks out only the
+      # directories a PR touches, so an always_run hook here would fail every
+      # PR outside dnn-providers/integration-tests.
+      # The script reads the tree through support_matrix_units.py, which uses
+      # bundle_discovery.py and verify_support_claims.py, so all of them count.
+      - id: render-support-matrix
+        name: Support matrix renders without errors (RFC 0015)
+        entry: python dnn-providers/integration-tests/scripts/render_support_matrix.py --output -
+        language: python
+        pass_filenames: false
+        files: ^dnn-providers/integration-tests/(integration-test-bundles/.*|scripts/(render_support_matrix|support_matrix_\w+|bundle_discovery|verify_support_claims)\.py)$
+
+      # SUPPORT_MATRIX_OVERVIEW.md *is* committed, so check it for staleness.
+      - id: check-support-matrix-overview
+        name: Support matrix overview is current (RFC 0015)
+        entry: python dnn-providers/integration-tests/scripts/render_support_matrix.py --overview-only --check
+        language: python
+        pass_filenames: false
+        files: ^dnn-providers/integration-tests/(integration-test-bundles/.*|scripts/(render_support_matrix|support_matrix_\w+|bundle_discovery|verify_support_claims)\.py|SUPPORT_MATRIX_OVERVIEW\.md)$
 
       - id: gtest-format-miopen
         name: GTest naming conventions (MIOpen)

dnn-providers/integration-tests/.gitignore

@@ -13,3 +13,12 @@
 # be added back under the old name, but guard against accidental recreation
 # (e.g. a stale script or muscle-memory path) getting committed again.
 integration_test_bundles/
+
+# The rendered engine support matrix (RFC 0015 §11). Generated, not committed:
+# it is large and grows with every bundle and every new gfx target,
+# which is both a large diff on every sidecar edit and close enough to the size
+# at which GitHub stops rendering a markdown file to be a problem. It is a pure
+# view of the committed .support.json sidecars, so anyone can rebuild it from a
+# plain checkout -- no GPU, no plugin, no DVC:
+#   python3 dnn-providers/integration-tests/scripts/render_support_matrix.py
+SUPPORT_MATRIX.md

dnn-providers/integration-tests/CMakeLists.txt

@@ -173,6 +173,37 @@ if(Python3_FOUND)
     )
     set(HIPDNN_IT_SUPPORT_SIDECAR_TEST hipdnn_support_sidecar_python_tests)
 
+    # pytest-style suites (fixtures, parametrize), so they run under -m pytest
+    # rather than as scripts; a bare `python file.py` would collect nothing.
+    add_test(
+        NAME hipdnn_bundle_discovery_python_tests
+        COMMAND ${Python3_EXECUTABLE} -m pytest -q -p no:cacheprovider
+                ${CMAKE_CURRENT_SOURCE_DIR}/scripts/test_bundle_discovery.py
+    )
+    set(HIPDNN_IT_BUNDLE_DISCOVERY_TEST hipdnn_bundle_discovery_python_tests)
+
+    add_test(
+        NAME hipdnn_support_matrix_renderer_python_tests
+        COMMAND ${Python3_EXECUTABLE} -m pytest -q -p no:cacheprovider
+                ${CMAKE_CURRENT_SOURCE_DIR}/scripts/test_render_support_matrix.py
+    )
+    set(HIPDNN_IT_SUPPORT_MATRIX_RENDERER_TEST hipdnn_support_matrix_renderer_python_tests)
+
+    add_test(
+        NAME hipdnn_support_features_python_tests
+        COMMAND ${Python3_EXECUTABLE} -m pytest -q -p no:cacheprovider
+                ${CMAKE_CURRENT_SOURCE_DIR}/scripts/test_support_features.py
+    )
+    set(HIPDNN_IT_SUPPORT_FEATURES_TEST hipdnn_support_features_python_tests)
+
+    # None of these suites uses a third-party plugin; don't let a broken one
+    # installed in the developer's environment fail them.
+    set_tests_properties(
+        hipdnn_bundle_discovery_python_tests hipdnn_support_matrix_renderer_python_tests
+        hipdnn_support_features_python_tests
+        PROPERTIES ENVIRONMENT "PYTEST_DISABLE_PLUGIN_AUTOLOAD=1"
+    )
+
     # Skips itself when PyTorch, which the generator needs, is not installed.
     add_test(
         NAME hipdnn_golden_generator_python_tests
@@ -327,6 +358,15 @@ if(EXISTS "${HIPDNN_IT_GTEST_CATEGORIES_YAML}"
     if(HIPDNN_IT_SUPPORT_SIDECAR_TEST)
         _apply_provider_ctest_category_labels(${HIPDNN_IT_SUPPORT_SIDECAR_TEST})
     endif()
+    if(HIPDNN_IT_BUNDLE_DISCOVERY_TEST)
+        _apply_provider_ctest_category_labels(${HIPDNN_IT_BUNDLE_DISCOVERY_TEST})
+    endif()
+    if(HIPDNN_IT_SUPPORT_MATRIX_RENDERER_TEST)
+        _apply_provider_ctest_category_labels(${HIPDNN_IT_SUPPORT_MATRIX_RENDERER_TEST})
+    endif()
+    if(HIPDNN_IT_SUPPORT_FEATURES_TEST)
+        _apply_provider_ctest_category_labels(${HIPDNN_IT_SUPPORT_FEATURES_TEST})
+    endif()
     if(HIPDNN_IT_GOLDEN_GENERATOR_TEST)
         _apply_provider_ctest_category_labels(${HIPDNN_IT_GOLDEN_GENERATOR_TEST})
     endif()

dnn-providers/integration-tests/README.md

@@ -337,6 +337,128 @@ run by hand behave alike; pass `--enforce-support-claims=false` to keep the summ
 but let a broken claim stay green. See
 [`docs/support-claim-enforcement.md`](docs/support-claim-enforcement.md).
 
+## Support Claims and the Support Matrix
+
+A bundle can carry a **support claim**: a sidecar asserting that a named engine
+supports that exact graph on a given `(arch, platform)`. Claims live beside the
+graph — `{Name}.support.json` for a single-graph bundle, one `support.json`
+keyed per `cases[].id` for a sweep — and the integration tests enforce them, so
+a claim that stops holding fails a test. The spec is
+[RFC 0015](../../projects/hipdnn/docs/rfcs/0015_EngineSupportClaims.md).
+
+Two scripts read them, and neither needs a GPU, a plugin, or a DVC pull — the
+`.json` and `.support.json` files are all in git:
+
+```bash
+# Validate every sidecar: schema, enforcement level, orphans, unknown case ids.
+# Also runs as the verify-support-claims pre-commit hook.
+python3 dnn-providers/integration-tests/scripts/verify_support_claims.py
+
+# Render the human-readable matrix of who supports what, where. With no
+# arguments this writes dnn-providers/integration-tests/SUPPORT_MATRIX.md.
+python3 dnn-providers/integration-tests/scripts/render_support_matrix.py
+```
+
+### Rendering the support matrix
+
+`SUPPORT_MATRIX.md` is **generated, not committed** — it is large and grows
+with every bundle and every gfx target, which would mean a large diff on every
+sidecar edit and a file near the size at which GitHub stops rendering markdown.
+Run the command above to produce your own copy; it is git-ignored, so it will
+not show up in `git status`. The matrix is a pure view of the sidecars, so two
+people rendering the same checkout get byte-identical files.
+
+The renderer is three files in `scripts/`: `support_matrix_units.py` collects
+one unit per graph from the bundle tree, `support_matrix_markdown.py` renders
+those units, and `render_support_matrix.py` is the command line over both.
+They read bundle JSON through the same `bundle_discovery.py` loader as
+`verify_support_claims.py`, so the two tools agree on which files are readable.
+
+`SUPPORT_MATRIX_OVERVIEW.md` **is committed** — it is the compact
+overview-only variant produced by `--overview-only`, small enough to render on
+GitHub. CI fails if it is stale; regenerate it with `--overview-only` in the
+same commit as the sidecar change.
+
+Bundle data is checked in CI, not at commit time. The top-level exclude in
+`.pre-commit-config.yaml` hides `integration-test-bundles/` from every hook,
+so the real tree is checked by `test_render_support_matrix.py` instead. It
+runs as the `hipdnn_support_matrix_renderer_python_tests` ctest in
+`hipdnn-superbuild-ci` on every change under `dnn-providers/`: the full matrix
+must render, and the committed overview must match a fresh render. To check
+locally before pushing, run that ctest, or run the script with
+`--overview-only --check`.
+
+Three pre-commit hooks give quick feedback when a commit changes the scripts
+themselves (or, for the last one, the overview file):
+
+| Hook | Fails when |
+|------|------------|
+| `verify-support-claims` | a sidecar is malformed, orphaned, or names an unknown case id |
+| `render-support-matrix` | the full matrix cannot be rendered — a malformed or orphaned sidecar, a claim on an unknown case, two tier copies of one bundle holding different graphs, or a bundle not under `<tier>/<Op>/`. Output is discarded |
+| `check-support-matrix-overview` | the committed `SUPPORT_MATRIX_OVERVIEW.md` differs from a fresh `--overview-only` render |
+
+Moving a bundle between tiers does not change the overview and needs no
+regenerate: a case is identified by
+its op and case id (or its path below the tier), not by its tier, and a case
+present in several tiers is counted once. The tier shows up only in the full
+matrix's provenance comments.
+
+Useful flags:
+
+| Flag | Effect |
+|------|--------|
+| `--output PATH` | write somewhere other than the default path; `-` means stdout |
+| `--bundles-dir PATH` | render a tree other than `integration-test-bundles/` |
+| `--max-case-ids N` | inline up to N sweep case ids per bundle in the provenance comments (0 = none, the default) |
+| `--overview-only` | emit only the per-target overview tables — no collapsible details, no per-variant drilldowns; writes to `SUPPORT_MATRIX_OVERVIEW.md` by default |
+| `--check` | compare against an existing file and exit 1 if they differ; for keeping a local copy in sync |
+
+Nothing is filtered: every `(arch, platform)` target gets its own section, and
+within it every engine is a column and every op family is a row, whether or not
+anything claims them. A family that vanished when unclaimed would be
+indistinguishable from one nobody has looked at.
+
+Reading it: each overview row is one op family. In the full
+`SUPPORT_MATRIX.md`, disclosure triangles expand a row into per-variant and then
+per-(variant, dtype) rows; the committed overview has the family rows only. A
+`—` means the graph is **unclaimed**, which is not the same as *known
+unsupported* — the engine may reject it, or nobody has written the claim yet.
+The Reading guide at the end of the full document spells out the rest,
+including the `[multi_batch]` / `[grouped]` / `[stride]` / `[dilation]` /
+`[padding]` variant tags.
+
+### Feature coverage per op
+
+The matrix counts graphs; it cannot show that an op has a feature no bundle
+touches. `scripts/support_features/` counts features instead: for each op, the
+share of its API features that at least one bundle graph exercises (*tested*),
+and the share that at least one claimed graph exercises on each arch. Features
+count singly, not in combination.
+
+```bash
+python3 dnn-providers/integration-tests/scripts/support_features/coverage.py
+```
+
+`feature_matrix.py` lays the same shares out like the support matrix: op
+groups by GPU and OS, then one table per engine, where a cell is the share of
+the op's features claimed there rather than the share of its graphs. The
+rendered document calls features *options*. Its *Options per op* section lists
+every option counted, split into those a graph exercises and those none does.
+
+```bash
+python3 dnn-providers/integration-tests/scripts/support_features/feature_matrix.py \
+    --output feature_matrix.md
+```
+
+The denominator is the op's API, not the bundles: each op module
+(`support_features/sdpa.py`, ...) catalogs the features of one node type in
+the flatbuffers schema, `support_features/registry.py` lists every node type
+in the schema, and `test_support_features.py` fails if a schema field is
+neither used by a feature nor listed with a reason it is not one. A graph that
+uses a value its catalog does not name — a new dtype, an uncatalogued head dim
+— is an error, not a silent omission, and both scripts render nothing until it
+is fixed. The tests run as the `hipdnn_support_features_python_tests` ctest.
+
 ## Test Tiers
 
 Tiers bound how long a run takes. They apply to both the C++ reference-executor
@@ -768,6 +890,9 @@ for the full workflow and tooling reference.
 - [`docs/support-claim-enforcement.md`](docs/support-claim-enforcement.md) —
   `.support.json` sidecars, the verdict set comparison, the `TestBody()`
   enforcement lifecycle, and how to read the claim summary.
+- [RFC 0015 — Per-Graph Engine Support Claims](../../projects/hipdnn/docs/rfcs/0015_EngineSupportClaims.md)
+  — the `.support.json` sidecar schema (§4), enforcement levels, and how the
+  support matrix is generated from them (§11).
 
 ## Project policies