test(hipdnn): trim ffm-quick: skip tests that cannot run on FFM, mirror quick for miopen

dnn-providers/integration-tests/docs/adding-tests.md

@@ -649,5 +649,4 @@ the harness (`hipdnn_integration_tests_unit_tests`).
 - [ ] `python dnn-providers/integration-tests/scripts/verify_support_claims.py`
       passes when you run it by hand.
 - [ ] If you regenerated or renamed sweep cases, no sidecar names a case that no
-      longer exists. The `ffm-quick` ids of miopen-provider still exist
-      (`ctest --test-dir build/dnn-providers/miopen-provider -L ffm-quick -N`).
+      longer exists.

dnn-providers/integration-tests/docs/file-formats.md

@@ -207,8 +207,7 @@ suffix only when the readable part is not already unique.
 **Do not write cases by hand.** Generate them with `import_graph.py` (see
 [Adding Tests](adding-tests.md#add-a-bundle)). You can edit a field of an existing
 case, for example to add its `golden` pointer. Do not rename ids without a
-reason, because support-claim sidecars and the `ffm-quick` tier of
-miopen-provider name cases by id.
+reason, because support-claim sidecars name cases by id.
 
 #### `tensor_patches`
 
@@ -545,8 +544,6 @@ execution_settings:
   `DISABLED` and is never launched. Variants are generated for each `ex_gpu_*`
   label declared in the file, not for the arch that you build. To select one, use
   `ctest -L ex_gpu_<arch>`.
-- miopen-provider's `ffm-quick` lists exact test names (sweep case ids).
-  Regenerating a sweep can rename those ids and silently shrink that tier.
 
 ## Where each format is enforced
 

dnn-providers/integration-tests/docs/running-tests.md

@@ -313,7 +313,7 @@ Each lane YAML also defines an `ffm-quick` category (labels `ffm-quick`,
 `ffm-full`). This is the fast-feedback tier. It is not a superset of `quick`.
 It differs for each lane:
 
-- miopen lists one sweep case per op by exact name.
+- miopen selects `quick_*`, `Smoke/*` and `Quick/*`, the same as its quick tier.
 - HIP_MLOPS selects `quick_*`, `Smoke/*` and `Quick/*`.
 - ASM_SDPA selects the `*Sdpa*` subset of those.
 - hipBLASLt selects only `Smoke/*`, `Quick/*` and `TestCpu*GoldenReference*`,

dnn-providers/integration-tests/test_categories.yaml

@@ -86,7 +86,7 @@ test_categories:
       - "slow"
 
   ffm-quick:
-    description: "Fast-feedback mechanism (FFM) quick tier - currently mirrors the quick tier"
+    description: "Fast-feedback mechanism (FFM) quick tier - mirrors the quick tier minus the quick-tier SDPA golden-data GPU-reference cases"
     test_patterns:
       - "*"
     exclude:
@@ -97,6 +97,15 @@ test_categories:
       - "comprehensive_*"
       - "Full*"
       - "full_*"
+      # The GPU SDPA reference recomputes QK^T per output element, so cost tracks
+      # B*H*Sq*Skv*D. On the FFM simulator even the smallest quick-tier bundle
+      # (1x8x256x128, 36 ms on MI300X) took 329 s, hd192 took 484 s, and the
+      # 2x32x2048x128 bundle had not finished after 40 minutes.
+      # referenceShapeIsAffordable() gates only the CPU reference, on the
+      # assumption that the GPU reference runs these shapes in milliseconds;
+      # that holds on hardware, not on FFM. Real-GPU lanes still run these
+      # cases and the CpuRef lane still validates the same bundles on the host.
+      - "quick_SdpaFwd_*_GpuRef.*"
     labels:
       - "ffm-quick"
       - "ffm-full"

dnn-providers/miopen-provider/test_categories_integration.yaml

@@ -91,22 +91,17 @@ test_categories:
       - "all"
 
   ffm-quick:
-    description: "Fast-feedback mechanism (FFM) quick tier: one bundle case per operation for rapid signal"
+    description: "Fast-feedback mechanism (FFM) quick tier: same coverage as the quick tier (quick-tier bundle sweeps plus always-built C++ tests), with a 2 hour ctest timeout"
     test_patterns:
-      # One case per op, named exactly. These are sweep-case ids taken from
-      # integration-test-bundles/quick/<Op>/Default/sweep.json (and the SDPA
-      # single-graph bundle path); regenerating a sweep can rename them, in
-      # which case this tier silently shrinks. Verify with:
-      #   ctest -L ffm-quick -N
-      - "quick_ConvolutionFwd_Default.1_16_16_16_bfp16_nchw_dil1x1_postpad0x0_prepad0x0"
-      - "quick_BatchnormInference_Default.1_3_224_fp32_ncl"
-      - "quick_Pointwise_Default.2_4_8_8_fp16_nchw"
-      - "quick_Reduction_Default.1_16_8_8_bfp16_nchw_modeadd"
-      - "quick_RMSNorm_Default.2_3_4_4_bfp16_nchw_bcd247"
-      # Layernorm had no entry here at all, so the op had no fast-feedback
-      # signal; this is the smallest quick-tier layernorm bundle case.
-      - "quick_Layernorm_Default.2_2_3_2_bfp16_nhwc_normalized_dim_count1_6d6330"
-      - "quick_SdpaFwd_bhsd_bf16_hd128_nomask_batch_Small.Small"
+      # Deliberately identical to the quick tier above. This tier used to name
+      # six sweep-case ids explicitly (one per op); those ids are silently
+      # renamed whenever a sweep is regenerated, and the FFM run only ever
+      # exercised three of them (the other three skipped). Mirroring quick
+      # removes that maintenance trap and gives FFM the same bundle coverage
+      # every other tier gets.
+      - "quick_*"
+      - "Smoke/*"
+      - "Quick/*"
     exclude: *common_excludes
     labels:
       - "ffm-quick"

projects/hipdnn/backend/tests/CMakeLists.txt

@@ -322,4 +322,4 @@ add_dependencies(hipdnn_backend_tests
     ${HIPDNN_TEST_MALFORMED_VERSION_PLUGIN_TARGET}
 )
 
-add_hipdnn_test(hipdnn_backend_tests ${CMAKE_CURRENT_BINARY_DIR})
+add_hipdnn_test(hipdnn_backend_tests ${CMAKE_CURRENT_BINARY_DIR} GTEST_CATEGORIES)

projects/hipdnn/cmake/Tests.cmake

@@ -25,8 +25,11 @@ findandcheckllvmsymbolizer()
 
 # YAML-driven CTest categorisation. Build-tree tests get labels as they are
 # registered; install_hipdnn_ctest_files() bakes the same labels into the
-# installed CTestTestfile.cmake.
+# installed CTestTestfile.cmake. test_categories.yaml labels whole CTest entries by
+# name. A target that must run a gtest-filtered subset (add_hipdnn_test(...
+# GTEST_CATEGORIES)) also gets suites generated from test_categories_gtest.yaml.
 set(_HIPDNN_TEST_CATEGORIES_YAML "${PROJECT_SOURCE_DIR}/test_categories.yaml")
+set(_HIPDNN_GTEST_CATEGORIES_YAML "${PROJECT_SOURCE_DIR}/test_categories_gtest.yaml")
 set(_HIPDNN_SHARED_CTEST "${ROCM_LIBRARIES_ROOT}/shared/ctest/TestCategories.cmake")
 if(EXISTS "${_HIPDNN_SHARED_CTEST}" AND EXISTS "${_HIPDNN_TEST_CATEGORIES_YAML}")
     include("${_HIPDNN_SHARED_CTEST}")
@@ -176,9 +179,10 @@ function(_add_check_target_internal PREFIX_NAME TARGET_NAME LABEL VERBOSE COMMEN
     # Build the ctest command
     set(CTEST_CMD ${CMAKE_COMMAND} -E env ${CTEST_ENV} ${CMAKE_CTEST_COMMAND})
 
-    # Add label filter if specified
+    # Add label filter if specified. Anchored: -L is a regex search, so a bare `quick` would
+    # also select every test labelled `ffm-quick`.
     if(NOT "${LABEL}" STREQUAL "")
-        list(APPEND CTEST_CMD -L "${LABEL}")
+        list(APPEND CTEST_CMD -L "^${LABEL}$")
     endif()
 
     # Always add --output-on-failure
@@ -194,7 +198,10 @@ function(_add_check_target_internal PREFIX_NAME TARGET_NAME LABEL VERBOSE COMMEN
 
     # Create the target with prefix
     set(FULL_TARGET_NAME "${PREFIX_NAME}-${TARGET_NAME}")
-    add_custom_target(${FULL_TARGET_NAME} COMMAND ${CTEST_CMD} COMMENT "${COMMENT}" USES_TERMINAL)
+    # VERBATIM: the anchors in -L "^label$" are shell-special (cmd.exe treats ^ as its escape
+    # character), so the arguments must be quoted for the host shell.
+    add_custom_target(${FULL_TARGET_NAME} COMMAND ${CTEST_CMD} COMMENT "${COMMENT}" USES_TERMINAL
+                      VERBATIM)
     add_dependencies(${FULL_TARGET_NAME} ${PREFIX_NAME}-validate_test_names)
     message(VERBOSE "Created ${FULL_TARGET_NAME} target")
 endfunction() # _add_check_target_internal
@@ -292,6 +299,53 @@ function(finalize_test_targets prefix_name)
     # cmake-format: on
 endfunction() # finalize_test_targets
 
+# Generates the <TARGET>_<category>_suite entries of a GTEST_CATEGORIES target from
+# test_categories_gtest.yaml (see add_hipdnn_test).
+#
+# Without shared/ctest (a sparse or standalone checkout of projects/hipdnn) no test gets category
+# labels at all, so there is nothing to pair these suites with: skip them, as the label pass above
+# skips. A missing yaml next to a present shared/ctest would instead drop the target out of every
+# category the yaml defines, so that case is a warning.
+function(_hipdnn_add_gtest_category_suites TARGET WORKING_DIR)
+    if(NOT COMMAND apply_test_category_labels)
+        message(STATUS "hipDNN: shared/ctest not available; skipping the gtest-filter suites of ${TARGET}")
+        return()
+    endif()
+    if(NOT EXISTS "${_HIPDNN_GTEST_CATEGORIES_YAML}")
+        message(WARNING
+            "hipDNN: ${_HIPDNN_GTEST_CATEGORIES_YAML} not found; ${TARGET} gets no gtest-filter "
+            "suites and is not selected by the categories that file defines"
+        )
+        return()
+    endif()
+
+    # Same environment and fixtures as the plain entry. The installed-tree lines that the parser
+    # writes into the fragment carry the same ENVIRONMENT, which only differs from the plain
+    # installed entry in sanitizer builds.
+    set(_category_args "")
+    if(DEFINED TEST_ENVIRONMENT)
+        list(APPEND _category_args ENVIRONMENT ${TEST_ENVIRONMENT})
+    endif()
+    if(DEFINED TEST_ENVIRONMENT_MODIFICATION)
+        list(APPEND _category_args ENVIRONMENT_MODIFICATION ${TEST_ENVIRONMENT_MODIFICATION})
+    endif()
+    if(DEFINED HIPDNN_TEST_MIOPEN_CACHE_DIR)
+        list(APPEND _category_args FIXTURES_REQUIRED hipdnn_clear_miopen_test_cache)
+    endif()
+
+    # The parser appends the installed-tree add_test() lines to this fragment;
+    # install_hipdnn_ctest_files() copies it into the installed CTestTestfile.cmake.
+    set(_fragment "${CMAKE_CURRENT_BINARY_DIR}/${TARGET}_gtest_categories.ctest")
+    file(WRITE "${_fragment}" "")
+    set_property(GLOBAL APPEND PROPERTY HIPDNN_GTEST_CATEGORY_FRAGMENTS "${_fragment}")
+
+    apply_test_category_labels(
+        ${TARGET} "${_HIPDNN_GTEST_CATEGORIES_YAML}" "${WORKING_DIR}"
+        INSTALL_TEST_FILE "${_fragment}"
+        ${_category_args}
+    )
+endfunction() # _hipdnn_add_gtest_category_suites
+
 # ~~~
 # Records, configures, and registers a hipDNN gtest-based CTest test target. Assumes that the
 # test target is a gtest executable, setting up:
@@ -304,8 +358,14 @@ endfunction() # finalize_test_targets
 # Parameters:
 #   TARGET - Name of the test executable target (must already exist)
 #   WORKING_DIR - Working directory for test execution
+#   GTEST_CATEGORIES - Optional flag. Also generate <TARGET>_<category>_suite entries, each running
+#       the binary with the --gtest_filter that test_categories_gtest.yaml defines for that
+#       category, labelled as that file says. The target must then be listed under
+#       `exclude_tests` for the same categories in test_categories.yaml, or its plain entry would
+#       carry those labels too and run the tests the filter excludes.
 # ~~~
 function(add_hipdnn_test TARGET WORKING_DIR)
+    cmake_parse_arguments(ARG "GTEST_CATEGORIES" "" "" ${ARGN})
     # ROCm ships its own googletest, which hip::host puts on the include path. Ahead of the
     # fetched googletest it compiles against ROCm's headers but links the fetched archive.
     # Pinned per target, for both gtest and gmock, so library sources never see googletest.
@@ -381,6 +441,10 @@ function(add_hipdnn_test TARGET WORKING_DIR)
     if(DEFINED HIPDNN_TEST_MIOPEN_CACHE_DIR)
         set_tests_properties(${TARGET} PROPERTIES FIXTURES_REQUIRED hipdnn_clear_miopen_test_cache)
     endif()
+
+    if(ARG_GTEST_CATEGORIES)
+        _hipdnn_add_gtest_category_suites(${TARGET} "${WORKING_DIR}")
+    endif()
 endfunction() # add_hipdnn_test
 
 # Install CTest configuration files for direct test execution This should be called once at the end
@@ -431,6 +495,15 @@ function(install_hipdnn_ctest_files)
         )
     endif()
 
+    # Suites generated from test_categories_gtest.yaml. Appended after the label pass on
+    # purpose: that pass would label them through test_categories.yaml's `.*` patterns
+    # (quick, standard, ...), and their only labels are the ones the gtest yaml gives them.
+    get_property(gtest_category_fragments GLOBAL PROPERTY HIPDNN_GTEST_CATEGORY_FRAGMENTS)
+    foreach(_fragment IN LISTS gtest_category_fragments)
+        file(READ "${_fragment}" _fragment_contents)
+        file(APPEND "${INSTALLED_CTEST_FILE}" "\n${_fragment_contents}")
+    endforeach()
+
     # Install the generated CTestTestfile.cmake to HIPDNN_CTEST_FILE_INSTALL_PATH
     install(FILES "${INSTALLED_CTEST_FILE}" DESTINATION ${HIPDNN_CTEST_FILE_INSTALL_PATH}
             RENAME CTestTestfile.cmake

projects/hipdnn/docs/TESTING.md

@@ -48,6 +48,8 @@ cmake --build build --target hipdnn-standard-check
 
 `hipdnn-<category>-check` and `hipdnn-<category>-check-verbose` exist for each category in [`test_categories.yaml`](../test_categories.yaml): the tiers `quick`, `standard`, `comprehensive` and `full`, plus `ffm-quick`, `unit` and `integration`. They build and run the hipDNN core tests only; `hipdnn-check` runs all core tests. Provider and integration-tests targets are listed in [Building hipDNN: Superbuild Targets](./Building.md#superbuild-targets).
 
+`test_categories.yaml` labels whole CTest entries by name, so it cannot leave one gtest out of a binary. A target registered with `add_hipdnn_test(<target> <dir> GTEST_CATEGORIES)` also gets `<target>_<category>_suite` entries generated from [`test_categories_gtest.yaml`](../test_categories_gtest.yaml). Each runs the binary with that category's `--gtest_filter` and carries exactly the labels the file lists. Today only `hipdnn_backend_tests` opts in, and only for `ffm-quick`, where the file excludes tests the FFM simulator cannot pass. List such a target under `exclude_tests` for the same category in `test_categories.yaml`, so the plain entry does not carry the label and run the excluded tests. The `hipdnn-<category>-check` targets match their label exactly (`ctest -L '^<category>$'`), so `hipdnn-quick-check` does not also run the `ffm-quick` suite.
+
 **One provider.** Build the provider's preset (for example `miopen-provider`), then run the shared integration suite against it:
 
 ```bash

projects/hipdnn/test_categories.yaml

@@ -28,9 +28,14 @@ test_categories:
     test_patterns:
 
   ffm-quick:
-    description: "Fast-feedback mechanism (FFM) quick tier; currently mirrors the quick tier"
+    description: "Fast-feedback mechanism (FFM) quick tier; mirrors the quick tier"
     test_patterns:
       - ".*"
+    # Targets whose ffm-quick suite is generated from test_categories_gtest.yaml (so
+    # it can exclude individual gtests) carry the ffm labels on that suite only.
+    # Labelling the plain entry too would run the excluded tests under -L ffm-quick.
+    exclude_tests:
+      - "hipdnn_backend_tests"
     labels:
       - "ffm-full"
 

projects/hipdnn/test_categories_gtest.yaml

@@ -0,0 +1,38 @@
+# hipDNN gtest-filter test categories.
+#
+# test_categories.yaml labels whole CTest entries by name; it cannot drop a single
+# gtest from a binary. This file is the gtest-schema counterpart (shared/ctest/
+# parse_test_categories.py): each category becomes a CTest suite named
+# <target>_<category>_suite that runs the binary with a --gtest_filter built from
+# test_patterns/exclude. Labels are exactly the ones listed here; tiers do not
+# cascade.
+#
+# It applies to every gtest target that opts in with
+# add_hipdnn_test(... GTEST_CATEGORIES) (cmake/Tests.cmake), so one exclude list
+# serves all of them. A pattern that matches no test in a binary is harmless, but
+# qualify excludes as Suite.Test so they cannot match anything unintended.
+#
+# A target that opts in carries the labels of the categories below on these suites
+# only, so test_categories.yaml must list it under `exclude_tests` for the same
+# categories. Today that is ffm-quick.
+
+test_categories:
+  ffm-quick:
+    description: "Fast-feedback mechanism (FFM) quick tier - mirrors the quick tier minus tests that cannot pass on the simulator"
+    test_patterns:
+      - "*"
+    exclude:
+      - "*DISABLED*"
+      # Asserts that a stalled start event excludes a 20 ms host delay from the
+      # measured span. FFM does not honour the stall gate: stalled timing reported
+      # 100-112 ms against an expected < 5 ms.
+      - "TestGpuProfilingControlDescriptor.StallGateExcludesHostSubmissionDelay"
+    labels:
+      - "ffm-quick"
+      - "ffm-full"
+
+execution_settings:
+  default_timeout: 600
+  timeout_multiplier: 1
+  category_timeouts:
+    ffm-quick: 7200       # 2 hours; matches the integration-tests ffm-quick tier