[EwanC]

Summary

Add --benchmark-reference to the integration-test binary. For each bundle test it times the reference that produces the expected outputs, in three phases, and reports them per test and as per-reference totals:

• prepare_ms: executor creation and applicability check, reference output allocation, variant-pack setup, and any input transfer the reference needs. Inputs are shared with the engine, so filling or loading them is not included. Zero for golden outputs.
• execute_ms: CPU/GPU reference compute, or loading the golden output blobs.
• materialize_ms: moving expected outputs to the validator's host or device.

The engine still runs once and its output is still compared; a mismatch fails the test as usual. Only engine execution and comparison are excluded from the reference timing. Each attempted reference prints [ REFERENCE BENCHMARK ] and records reference_benchmark_<cpu|gpu|golden>_* GoogleTest properties with status verified, mismatch, declined or error. Declined and errored attempts carry no timings. [ REFERENCE BENCHMARK SUMMARY ] totals verified references only and prints measured=0 when there are none. A failed device synchronize fails the test instead of falling back to the next reference.

Closes: #14456

Scope

• Bundle tests only. Tests built on the C++ graph harness are not measured: the BUILD_CPP_GRAPH_TESTS suites, which are being retired in favour of bundles, and the default-built ConvForwardSerializeRoundTrip and ResampleBackward tests. They run their reference as normal and print no reference line. docs/running-tests.md §5 says so.
• Off by default. Refused (exit 1) with --skip-graph-validation, --capture-bundles or --write-support-claims, which never run a reference, and when bundle tests are off (--no-bundles or HIPDNN_TEST_ALLOW_BUNDLES=0), where there is nothing to measure.
• A bundle whose engine declines or fails, or whose support claim stops it, records no reference timing.
• With the flag, golden blobs are read only when golden outputs verify the test. Without it every golden blob is read with the inputs, so a corrupt blob that a gpu or cpu run never compares against fails the run without the flag and passes with it.

CI coverage

• hipdnn_integration_tests_benchmark_reference_refusal (integration-tests, quick tier, no plugin): runs the binary with --benchmark-reference --skip-graph-validation and passes on the refusal message. It is in the build-tree CTest file and in the installed one, so the GPU jobs that run the install tree (TheRock) execute the refusal. On a runner with no HIP device the binary skips itself and the check passes without reaching the refusal.
• miopen-provider-external-integration-check-benchmark-reference-smoke (MIOpen lane, quick tier, build and install tree): runs quick_BatchnormFwdInference_Inference.* (5 bundles) with --benchmark-reference. It passes only when the summary reports at least one verified measurement, and fails on [ FAILED ], FATAL:, Fatal error: or Error: zero tests ran. CTest ignores the exit code of a check with a pass pattern, so a non-zero exit after the summary that prints none of these is not caught; a crash on a signal still fails. The opt-in BENCHMARK_REFERENCE_SMOKE_FILTER keyword on add_external_integration_test_target() registers it; only MIOpen opts in.
• At the previous revision TheRock ran the smoke from the installed tree on gfx942 Linux and gfx1101 Windows (5/5; 4 bundles verified by golden outputs, 1 by the GPU reference).

Testing

Example invocation (the quick_BatchnormFwdInference_* filter selected 11 bundle tests in the available MI210 build_release tree; the CI smoke filter above selects 5):

hipdnn_integration_tests --test-article libmiopen_plugin.so --test-engine MIOPEN_ENGINE \
  --test-config MIOPEN_ENGINE.toml --gtest_filter='quick_BatchnormFwdInference_*' \
  --verification-mode gpu --benchmark-reference --gtest_output=json:gpu.json

Quick tier (ctest -L quick, preset hipdnn-providers, 52 entries, serial) and the flag on each device:

Device	Quick tier	Flag runs (MIOpen engine, filter above)
gfx950 MI355X, Linux	52/52 passed (482 s)	gpu, cpu, auto, gpu with --validator cpu, and with --benchmark: 6/6 passed, measured=6 each
gfx1100, Linux (ROCm 10.3.0a20261103 from the nightly wheels)	52/52 passed (600 s)	same set: 6/6 passed, measured=6 each
gfx950 MI350X, Linux	52/52 passed	same set: 6/6 passed, measured=6 each
gfx90a MI210, Linux (available standalone build_release tree)	11/18 CTest entries passed; 4 failed, 3 skipped (566.54 s). This standalone tree is not the 52-entry hipdnn-providers preset.	gpu, cpu, auto, gpu with --validator cpu, and gpu with --benchmark: 11/11 passed, measured=11 each

The MI210 reference-benchmark runs used quick_BatchnormFwdInference_*, which selected 11 bundles in this build tree (rather than the 6 reported for the other flag runs). All five runs passed with no skipped or failed bundle tests. The measured GPU-reference totals were about 874 ms, CPU-reference totals 91 ms, auto-mode totals 65 ms, GPU-reference with the CPU validator totals 91 ms, and GPU-reference with --benchmark totals 201 ms. These totals depend on first use and device initialization.

The MI210 standalone quick-tier CTest failures were:

• hipdnn_integration_tests_unit_tests_quick_suite and hipdnn_integration_tests_unit_tests_ffm-quick_suite: TestReferenceOpCoverage.RaggedBundlesAreNotListedAsGaps failed because the discovered ragged-bundle set differed from the test's expected set.
• hipdnn_golden_data_tests_quick_suite and hipdnn_golden_data_tests_ffm-quick_suite: eight SDPA backward bundle cases failed output comparison against their CPU/GPU references.
• The GPU-reference quick suites and the benchmark-reference refusal check passed. Three golden-generator tests were skipped.

Additional testing:

• Refused combinations exit 1 with the refusal message: each of the three modes above, --no-bundles, and HIPDNN_TEST_ALLOW_BUNDLES=0. HIPDNN_TEST_ALLOW_BUNDLES=1 with --no-bundles runs and measures.
• Unit tests: hipdnn_integration_tests_unit_tests 1127 tests, 1125 passed, 2 skipped (no golden data on the node). The benchmark tests (89) also pass on gfx950, gfx1100 and gfx1101. They cover: the refusals with the flag on and off; the production timer waiting for queued device work, including as the HarnessDependencies default; a sync failure at each of the 8 reference and 4 golden synchronize points without fallback; golden outputs read inside the execute phase; the applicability and placement checks inside prepare_ms; host-to-device and device-to-host expected-output copies inside materialize_ms (device tests); an unreadable golden blob; a reference declining at execute; every per-phase property, mismatch samples, and verified-only totals over several samples. The MI210 quick-tier unit-test failure above was in a separate build_release run against a different bundle inventory.
• Mutation check on gfx90a: 9 targeted mutants (timer default, verified totals, prepare-window contents, the refusals, and main() ignoring the refusal) are all killed. The mutants that need the real synchronize are killed only by device tests, which skip on a host without a GPU.
• CTest checks, build and install tree on gfx90a: both pass with a GPU and pass through the no-device skip without one. The smoke fails with the flag dropped, with --no-bundles, and with a filter that matches no tests.
• Local result from an earlier revision, not rerun here and not reproduced in CI: the MIOpen quick lane (quick_*:Smoke/*:Quick/*, 706 tests) on gfx90a gave identical outcomes with and without the flag (251 passed, 455 skipped, 0 failed).
• Pre-commit passes on all changed files; clang-tidy reports no warnings on the changed translation units.

ASIC coverage

• Tested on Linux: gfx950 (MI355X, MI350X), gfx1100, gfx90a (MI210), with the MIOpen engine. Earlier CI: gfx942 Linux and gfx1101 Windows (smoke from the installed tree).
• Not tested with the flag: gfx125X (no hardware available), gfx1101 Linux (the available node could not run MIOpen), other Windows targets, the hipBLASLt and hip-kernel-provider engines, and golden mode on committed bundle data.
• PR CI covers the refusal check on the GPU jobs that run the installed CTest file, and the smoke on the MIOpen lane's architectures. On an architecture where MIOpen declines the five smoke bundles, the smoke reports measured=0 and fails.

Risk

3 of 5. The feature is off by default and only changes behaviour when the flag is passed. Two new quick-tier CTest entries and a change to the shared external-integration CMake helper (opt-in per lane; only MIOpen opts in) are the parts that run in every CI job.

[therock-pr-bot]

⏳ Waiting on Required Checks

⏳ Required checks have not all concluded yet. This bot run is exiting without failing so those checks can report independently.

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🔎 pre-commit	⏳ Pending	⏳ Still running…
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

⏳ Required checks are still running or have not started yet.
This PR is not ready for review until they complete.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

[therock-pr-bot]

⏳ Waiting for required checks to start or finish:

• ⏳ pre-commit