feat(integration-tests): render the engine support matrix from sidecars by bghimireamd · Pull Request #11659 · ROCm/rocm-libraries

bghimireamd · 2026-09-03T14:55:15Z

JIRA ID : ALMIOPEN-2332

Based on #13747 (the two Batchnorm sweep-case moves), so this PR stays Python and docs only. Once #13747 merges, this PR retargets to develop.

Summary

render_support_matrix.py: builds the engine support matrix purely from committed data (RFC 0015 §11). It needs no GPU, no plugin and no DVC pull.
Full matrix: SUPPORT_MATRIX.md is generated, not committed (git-ignored). Run the script to get a local copy.
Overview: SUPPORT_MATRIX_OVERVIEW.md (--overview-only) is committed. CI fails when it is stale (the hipdnn_support_matrix_renderer_python_tests ctest re-renders it from the committed sidecars), so it is regenerated in the same commit as the sidecar change. There is no bot job.
bundle_discovery.py: the renderer and verify_support_claims.py share one Python copy of BundleDiscovery.hpp, so they agree on what a bundle is.
support_features/: feature coverage per op. The matrix counts graphs; this counts the share of each op's API features (catalogued from the flatbuffers schema) that some bundle graph exercises, and that some claimed graph exercises per arch. feature_matrix.py lays the same shares out like the support matrix.

python3 dnn-providers/integration-tests/scripts/render_support_matrix.py                  # full matrix
python3 dnn-providers/integration-tests/scripts/render_support_matrix.py --overview-only  # committed overview
python3 dnn-providers/integration-tests/scripts/support_features/coverage.py             # feature coverage per op
python3 dnn-providers/integration-tests/scripts/support_features/feature_matrix.py --output feature_matrix.md

Details

Sections and columns: every (arch, platform) target gets its own section, and every engine is a column. Every op is a row, claimed or not.
Variant tags: rows split by variant using feature tags read off the graph (multi_batch, grouped, stride, dilation, padding) and by dtype.
Bundles without a sidecar: they count in the denominator and in no numerator. That gap is what once reported Sdpa 12/12 for 35 graphs.
Tier-independent:
- Bundles are found by walking the tree recursively. The first path segment is the tier and the second is the op; nothing is hardcoded.
- A bundle's identity doesn't include its tier, so moving a case between tiers changes only the tier list in its provenance comment.
- Copies of the same bundle in two tiers are merged; copies holding different graphs are an error.
Fails loudly on:
- an orphan sidecar;
- a sidecar with the wrong shape;
- an unknown sidecar version;
- a bundle that doesn't sit under <tier>/<op>/;
- a sweep case naming a tensor uid its template doesn't have (mirrors buildCaseTensorMap());
- a graph using a value its feature catalog doesn't name, such as a new dtype or an uncatalogued head dim (support_features/).
Nothing hardcoded: arch headings are the bare gfx target, and the regenerate command is derived from where the script lives.

Test plan

193 renderer and bundle-discovery tests and 1715 feature-coverage tests pass, including:
- tier-move, deduplication, conflict, orphan, shape and version tests;
- an independent recount of every overview cell.
All three suites are registered as ctests (hipdnn_bundle_discovery_python_tests, hipdnn_support_matrix_renderer_python_tests, hipdnn_support_features_python_tests), so hipdnn-superbuild-ci runs them on Linux and Windows. A planted failure in the renderer suite makes its ctest fail.
Real-tree checks: the tree renders with 0 errors, two renders agree byte for byte, and the committed overview matches a fresh render. Removing one claim from a sidecar without regenerating fails the test with the regenerate command.
Pre-commit passes. The top-level exclude hides integration-test-bundles/ from every hook, so bundle data is checked by the ctest; the hooks fire on script changes.
CI

🤖 Generated with Claude Code

therock-pr-bot · 2026-09-03T14:59:39Z

✅ All Checks Passed — Ready for Review

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🔎 pre-commit	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—
🤖 therock-pr-bot	✅ Pass	—

🎉 All checks passed! This PR is ready for review.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

therock-pr-bot · 2026-09-03T14:59:42Z

🎉 All checks passed! This PR is ready for review.

codecov · 2026-09-03T15:03:41Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop   #11659   +/-   ##
========================================
  Coverage    75.95%   75.95%           
========================================
  Files         2880     2880           
  Lines       479025   479025           
  Branches     71958    71958           
========================================
+ Hits        363828   363830    +2     
+ Misses       90338    90337    -1     
+ Partials     24859    24858    -1

Flag	Coverage Δ		*Carryforward flag
TensileLite-CPP	`48.93% <ø> (ø)`		Carriedforward from 8f13da7
TensileLite-Unit	`83.68% <ø> (ø)`		Carriedforward from 8f13da7
hipBLAS	`90.62% <ø> (ø)`		Carriedforward from 8f13da7
hipBLASLt	`64.43% <ø> (ø)`		Carriedforward from 8f13da7
hipCUB	`82.68% <ø> (ø)`		Carriedforward from 8f13da7
hipDNN	`86.79% <ø> (+<0.01%)`	⬆️
hipFFT	`42.00% <ø> (ø)`		Carriedforward from 8f13da7
hipRAND	`76.12% <ø> (ø)`		Carriedforward from 8f13da7
hipSOLVER	`69.67% <ø> (ø)`		Carriedforward from 8f13da7
hipSPARSE	`86.99% <ø> (ø)`		Carriedforward from 8f13da7
rocBLAS	`68.73% <ø> (ø)`		Carriedforward from 8f13da7
rocFFT	`48.10% <ø> (ø)`		Carriedforward from 8f13da7
rocRAND	`57.30% <ø> (ø)`		Carriedforward from 8f13da7
rocSOLVER	`76.74% <ø> (ø)`		Carriedforward from 8f13da7
rocSPARSE	`76.92% <ø> (ø)`		Carriedforward from 8f13da7
rocThrust	`93.15% <ø> (ø)`		Carriedforward from 8f13da7

*This pull request uses carry forward flags. Click here to find out more.
see 1 file with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

therock-pr-bot · 2026-10-09T22:09:21Z

Pre-commit check failed

⛔ pre-commit failed

Please run locally:

python -m pip install pre-commit
pre-commit install
pre-commit run --all-files --show-diff-on-failure

This repo uses .pre-commit-config.yaml.

…laims (ALMIOPEN-2332) RFC 0015 §11 renderer, ported onto current develop. render_support_matrix.py joins every committed support sidecar to its graph and emits a combined markdown (or JSON) matrix of engine x (arch, platform) verdicts, with a committed overview that CI keeps fresh. bundle_discovery.py mirrors BundleDiscovery.hpp so the Python tools and the C++ harness agree on what a bundle is. The matrix is independent of bundle tiers, which move (ALMIOPEN-2851, retier_bundles.py). Tier and op come from the path (<tier>/<op>/...), with no tier or op names hardcoded. A unit's identity is its path below the tier plus case id (sweeps) or graph file (single bundles); copies in several tiers merge into one unit with claims unioned and every tier listed, and copies holding different graphs are an error, as are orphaned sidecars. Rendering develop's tree and the #13198 retiered tree gives identical output apart from tiers. The renderer and bundle-discovery pytest suites are registered as ctests, so hipdnn-superbuild-ci runs them on every dnn-providers/ change. They include real-tree checks: the full matrix renders and the committed overview matches a fresh render. Those checks carry the bundle data, because the top-level pre-commit exclude hides integration-test-bundles/ from every hook; the pre-commit hooks give quick feedback on script changes. Co-Authored-By: Claude <noreply@anthropic.com>

develop split the integration-tests README into docs/. Take its README and move the support-matrix and feature-coverage sections to docs/support-matrix.md, indexed in docs/README.md.

…read matrix output as UTF-8 (ALMIOPEN-2332) The develop merge brought GdnAttributes and KdaAttributes into the NodeAttributes union, so test_every_api_node_type_is_catalogued failed in both superbuild lanes. Add a shared linear_attention catalog with gdn and kda modules, and a Linear attention group in the feature matrix. test_output_writes_the_file read the UTF-8 matrix with the locale codec and failed on the Windows runner (cp1252).

…ix tests (ALMIOPEN-2332) The Windows superbuild failed hipdnn_support_features_python_tests: test_output_writes_the_file read feature_matrix.py's output with a bare read_text(), which decodes as cp1252 on Windows. The script writes UTF-8, and byte 0x9d of a non-ASCII character has no cp1252 mapping. Pass encoding="utf-8" wherever these tests read a file that is UTF-8: the renderer's and feature_matrix.py's output, and the flatbuffers schemas and C++ headers the catalog tests parse. The others passed on Windows only because their bytes happened to map in cp1252. Under LC_ALL=C with UTF-8 mode off, 50 tests failed before and all pass after. Co-Authored-By: Claude <noreply@anthropic.com>

… the feature matrix (ALMIOPEN-2332) The dashboard sorted archs into families by name prefix (gfx1 -> RDNA, gfx9 -> CDNA). That put the data-center gfx1250 under "RDNA (consumer)", silently dropped any arch matching neither prefix, and left the GPU targets row blank for an arch missing from GPU_INFO. GPU_INFO and FAMILIES become one table, GPUS: arch -> (family, generation, products), with gfx1250 as CDNA 5 / data center. A claimed arch missing from GPUS is now a defect: it is reported once, naming the first claiming graph, and the render fails with nothing written. A real-tree test checks every committed claim has a row. Output on the current tree is unchanged. Co-Authored-By: Claude <noreply@anthropic.com>

…OPEN-2332) feature_matrix.py and coverage.py wrote their emoji documents through sys.stdout, which raises UnicodeEncodeError on a cp1252 Windows console. The renderer's UTF-8 buffer write with its broken-pipe guard moves into support_matrix_units.write_stdout and all three CLIs use it. feature_matrix's "wrote" status line goes to stderr, like the renderer's. Co-Authored-By: Claude <noreply@anthropic.com>

…in bundle_discovery (ALMIOPEN-2332) Five tools each spelled the "integration-test-bundles" default, and three each derived X.support.json / X.meta.json from X.json by hand. They now share DEFAULT_BUNDLES_DIR and the sidecar_for_graph / meta_for_graph / graph_for_sidecar / is_single_graph_sidecar helpers, so a rename lands in one place. Co-Authored-By: Claude <noreply@anthropic.com>

… verifier CLI (ALMIOPEN-2332) support_matrix_units imported validate_*_schema from verify_support_claims, coupling the matrix collector to the verifier's command line. The validators, VALID_PLATFORMS and the sidecar SCHEMA_VERSION now live in support_claim_schema.py, which the verifier, the collector, the harvester and support_sidecar share. The pre-commit file patterns follow the move. Co-Authored-By: Claude <noreply@anthropic.com>

…LMIOPEN-2332) The collector's template expansion diverged from expandTemplateGraph() in IntegrationTestBundle.hpp in three ways: - a missing placeholder value read as None, where the harness throws and skips the case (INVALID_SWEEP_CASE); - placeholders were not scoped per tensor: inside an object with an integer uid, the harness reads that uid's values.tensors[] spec first, and dims / strides / data_type must come from it; - a values.tensors[] spec was overlaid on the tensor's literal fields, where the harness only feeds it to placeholders. Transcribe the C++ one function per function (placeholderField, lookupJsonPath, collectTemplateTensorUids, buildCaseTensorMap, resolvePlaceholder, expandTemplateNode), plus the golden.path check from resolveSweepGoldenDirectory(), in loadIntegrationTestBundle()'s order. warnUnusedSweepValues() only logs, so it is not mirrored. The rendered matrix, coverage, feature matrix and every unit digest over the committed tree are byte-identical before and after. Co-Authored-By: Claude <noreply@anthropic.com>

…LMIOPEN-2332) Both early-close pipe tests waited for the CLI to exit before reading its stderr pipe. A CLI that writes more than a pipe buffer of errors -- the renderer on a broken tree -- blocks on the full pipe and never exits, so the test hung instead of failing. Send stderr to a temporary file and bound every wait with a timeout. A renderer that floods stderr and exits 1 now fails the test in 0.1s. communicate() is not used: it would read the stdout handle the test has already handed to the reader and closed, which fails on Windows. Co-Authored-By: Claude <noreply@anthropic.com>

…ALMIOPEN-2332) Four copies stripped "Attributes" from a node type, three different ways: short_node_type() only off the end, node_type_label() every occurrence, feature_matrix.display_name() and coverage.resolve_node_type() the first. They agreed only because no node type has the token twice and no fused graph puts BatchnormInferenceAttributesVarianceExt after its first node. Keep node_type_label() and route the Variant column, the feature matrix and coverage --graphs through it. A parametrized test pins that every catalogued op reads the same in each view and that --graphs accepts it. The matrix, the coverage report and the feature matrix over the committed tree are byte-identical before and after. Co-Authored-By: Claude <noreply@anthropic.com>

…ms (ALMIOPEN-2332) harvest_support_claims.py --write changed sidecars but left the committed SUPPORT_MATRIX_OVERVIEW.md stale, so the next CI run failed the overview check. After writing any sidecar it now re-renders the overview through render_support_matrix.main(). If that render fails, the sidecars stay written, it names the regenerate command and exits 1. Co-Authored-By: Claude <noreply@anthropic.com>

…11 (ALMIOPEN-2332) §11 still said no matrix is committed and --check is not wired into CI. The full matrix is still generated on demand, but SUPPORT_MATRIX_OVERVIEW.md is committed and checked: by the check-support-matrix-overview hook, and, for sidecar-only commits the pre-commit exclude hides from every hook, by the hipdnn_support_matrix_renderer_python_tests ctest. Co-Authored-By: Claude <noreply@anthropic.com>

…2332) The scripts already pass encoding="utf-8" to every read_text() and write_text(); the tests did not, so on a host whose locale encoding is not UTF-8 they would write fixtures the code under test decodes differently. Pass the encoding in the tests too. Co-Authored-By: Claude <noreply@anthropic.com>

…(ALMIOPEN-2332) DTYPE_LABELS keyed FNUZ fp8 by names the serializer never writes ("e4m3_fnuz") and labelled it as OCP fp8, and had no entry for int4, fp4 or fp6, so the matrix and the feature catalogs named those dtypes differently. Key it by the serializer's DataType names, one per enumerator but UNSET, and give FNUZ fp8 its own label. TestDataTypeLabels locks the labels to data_types.fbs through Common.hpp, and every catalog's dtype map to the labels, so a dtype added to the schema fails a test instead of drifting. Co-Authored-By: Claude <noreply@anthropic.com>

…y (ALMIOPEN-2332) render_support_matrix.py writes `--output -` to stdout; feature_matrix.py wrote a file named "-". Read it as stdout in both. Co-Authored-By: Claude <noreply@anthropic.com>

…tifact (ALMIOPEN-2332) The full SUPPORT_MATRIX.md is git-ignored, so a reviewer had to render it locally to see it. The hipdnn-superbuild-ci Linux job now renders it, the overview and the feature matrix after the tests and uploads them as the hipdnn-support-matrix artifact. The renderers read only the committed sidecars, so the steps run even when the build or tests failed. The committed overview and its staleness check are unchanged. Co-Authored-By: Claude <noreply@anthropic.com>

bghimireamd · 2026-10-21T17:47:54Z

Re @adickin-amd's review:

Thanks, agreed on both.

Complexity: deferred to ALMIOPEN-2970, to start once the bundles are aligned and each op has a plan. It lists five hotspots with file:line. The main one is the Python template expansion, which copies the C++ expandTemplateNode and is the clearest candidate for sharing with the harness.

Drift: this PR now locks the matrix's dtype labels to the schema (718e8b6). Schema fields and enums were already locked by test_support_features.py. Layout/grouping and the arch table are still unlocked; the ticket tracks them.

Display: agreed. The ticket keeps display changes out of scope until someone using the matrix asks for one.

…IOPEN-2332/support-matrix-renderer-port

…velop's new bundles (ALMIOPEN-2332) #13418 added bundles under integration-test-bundles/, so the committed overview no longer matched a re-render of the merged tree. Co-Authored-By: Claude <noreply@anthropic.com>

…ort matrix overview (ALMIOPEN-2332) The team voted for the option-percentage layout (option 3), so SUPPORT_MATRIX_OVERVIEW.md is now rendered by support_features/feature_matrix.py, which writes it by default and gains --check. The pre-commit hook, harvest --write, the freshness test (moved to test_support_features.py) and the CI artifact follow it. The graph-count --overview-only render is kept but git-ignored as SUPPORT_MATRIX_GRAPH_OVERVIEW.md. Co-Authored-By: Claude <noreply@anthropic.com>

…IOPEN-2332/support-matrix-renderer-port

…MIOPEN-2332) rank:2d read as a 2-D tensor but meant two spatial dims after [N, C], and the same NCHW tensor showed as dims:4 on the norm rows. Rename the group to spatial (spatial:1d/2d/3d, and spatial:none for batchnorm's [N, C] floor). Labels only: the regenerated overview is identical modulo the rename. Co-Authored-By: Claude <noreply@anthropic.com>

…IOPEN-2332/support-matrix-renderer-port

adickin-amd · 2026-10-22T17:04:04Z

Bypass merging. Failures are on unrelated libraries.

Conflict in .pre-commit-config.yaml (#11659 expanded the comment on the reference-data exclusion): keep the generated ingestor_plan_generated.h exclusion and the expanded comment. No exclusion pattern changes. Committed on the user's request while gfx942 testing (full build, quick set, provider unit and integration tests; job 68147835) is still running. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

assistant-librarian · 2026-10-26T08:11:35Z

TheRock Submodule Bump Activity

Newest first

2026-10-26 — Included in TheRock via ROCm/TheRock#9478.

github-actions Bot added documentation github actions project: hipdnn project: hipdnn-integration-tests labels Sep 3, 2026

bghimireamd changed the base branch from develop to users/bghimire/ALMIOPEN-2335/support_claim_authoring September 3, 2026 14:57

bghimireamd changed the base branch from users/bghimire/ALMIOPEN-2335/support_claim_authoring to users/bghimire/ALMIOPEN-2425/initial-support-claim-population September 3, 2026 14:59

assistant-librarian Bot added the organization: ROCm label Sep 3, 2026

Base automatically changed from users/bghimire/ALMIOPEN-2425/initial-support-claim-population to develop September 15, 2026 23:23

bghimireamd force-pushed the users/bghimire/ALMIOPEN-2332/support-matrix-renderer branch from e29058e to a77f001 Compare October 9, 2026 22:04

bghimireamd force-pushed the users/bghimire/ALMIOPEN-2332/support-matrix-renderer branch 2 times, most recently from be1330a to fa457e3 Compare October 10, 2026 02:04

bghimireamd marked this pull request as ready for review October 10, 2026 02:08

bghimireamd requested review from a team as code owners October 10, 2026 02:08

bghimireamd force-pushed the users/bghimire/ALMIOPEN-2332/support-matrix-renderer branch from fa457e3 to 69ab724 Compare October 10, 2026 02:22