Test Filter Standardization Proof of Concept - MIOpen by dileepr1 · Pull Request #3513 · ROCm/rocm-libraries

dileepr1 · 2025-12-22T15:44:53Z

The following motivation is adapted from #2411.
POC done on MIOpen.

Standardization of Test Filtering Strategy Across The ROCm Stack

Note that this is a proof of concept that will go through multiple iterations and improvements.
Problems

The ROCm stack consists of projects that use different test frameworks (googletest, catch2, pytest, tox, and ctest among others). Different commands must be learned and used to test across the stack.
Definitions of smoke test, sanity test, and what should be run pre-submit and on-submit vary from team to team and is not a problem isolated to ROCm.
While code coverage and test coverage can be useful values, the value of test results as a function of turnaround time should be emphasized given the ongoing constraints on compute test resources.
For some projects, disablement of flaky tests and the list of tests for a given set (e.g., pre-submit) is specified at the orchestration level and not in the project source, or the lists are not in a readable format to easily track when tests were added to or removed from filters and why.
Windows has been neglected in many cases with testing strategy.

Requirements

Solution must work on top of what ROCm projects already have in place with the current test frameworks already in use, but have the new standard provide the enhanced and standardized capabilities.
Instead of enforcing definitions of terms like smoke or sanity, define test sets and test suites based on test execution time which can be universally understood.
a. If a project only has five minutes to run tests on a resource, populate the five-minute test set with the list of tests for the project that would provide the most valuable signal within those time constraints.
b. These test sets should be owned and continuously maintained by the project teams, as new features get added, removed, or as feature set priorities change.
c. Comprehensive test sets for code coverage and test coverage are specified separately, with clear documentation on the test execution time for these sets.
The lists of tests that are in test sets and the current tests excluded from runs due to issues should be in a human-readable format in the source code of the project at a standardized location.
The solution must work on all supported OS distros of ROCm across Linux and Windows.
Compilation and packaging of tests must be supported on a GPU-less system, but local execution of these same commands should be functional on a GPU system.
Support should encompass ROCm core first, ROCm extras as optional, as well as the key frameworks important to customers (e.g., pytorch, JAX, etc.)

Specifications Proposal

As the majority of ROCm is based on CMake, and with CTest providing support for execution of commands to work with pytest, gtest, etc., CTest as the solution wrapper seems like the natural choice.
Test set definitions are human-readable files in a common directory in the project source (e.g., resources/tests/definitions/five-minutes.yml). No strong opinions on markup language choice or the actual path. These definitions are referred to as presets in CTest. https://cmake.org/cmake/help/latest/manual/ctest.1.html
In the build process, the build node should translate those test definitions from the human-readable files and converted to the presets that CTest understands through CMake invocations and produce the output that can be packaged and transferred to a test system.
Test systems should be able to run ctest --preset through those transferred test packages. Similar commands should work across ROCm projects and OS environments.
Proof of concept work will be done on two ROCm projects, each using a different test framework to verify the solution is flexible with its support.
The only changes that should occur on TheRock with this work is that the test invocation calls from TheRock should become a common string, regardless of ROCm project.

tony-davis · 2026-01-27T19:58:35Z

Hey team! 👋

I've rebased my optional metadata PR to target your branch (users/dravindr/ctest_miopen) instead of the abandoned #2411. The new PR is here: #4109

Important: This is a PR into YOUR PR - an amendment to your work. You can merge it directly into your branch without needing external approvals. It'll then come along when you merge #3513 into develop.

Quick context:

Documentation-only change to shared/ctest/README.md - No code modifications
Adds optional fields (notes, source_coverage, api_coverage, feature_tags) to the YAML schema
All fields are 100% optional and backward compatible - Parser naturally ignores them
Zero implementation cost for teams that don't need them

Why this matters:
Projects like hipBLASlt (with ~200+ test configurations) will need semantic metadata to enable AI-assisted test selection per-PR. Rather than having projects implement custom extensions (which defeats the standardization goal), these optional fields provide a standard way to add this metadata.

This aligns with the "AI-assisted test selection" capability mentioned as future work in the original proposal. Happy to discuss async to avoid blocking the MIOpen work!

Link: #4109

AaronStGeorge · 2026-01-27T21:47:10Z

I get the impression that TheRock seems to prefer TOML over YAML: BUILD_TOPOLOGY.toml. It would be nice to standardize.

[ROCm/composable_kernel commit: bbf0b1a]

JonathanLichtnerAMD

LGTM, thanks!

aalabsi-amd

Overall LGTM, I added some last improvements, considerations, minor stuff for you to look at and decide which ones make sense to apply in this PR and which not.
Thanks

Changed colors in mermaid diagram to make it readable in dark mode.

… check' to avoid long running time during build

## Add Optional Enhanced Metadata Fields to Test Categories This PR adds optional fields to the `test_categories.yaml` schema to enable richer test documentation and future AI-assisted test selection. ### Key Points ✅ - **All new fields are optional** - Zero burden on teams that don't want them - **100% backward compatible** - Existing YAML files work unchanged - **Minimal code changes** - Parser naturally ignores unknown fields (no code changes needed!) - **Incremental adoption** - Teams add fields only when they see value - **Clear documentation** - Enhanced README with examples and use cases ### Changes 1. **README.md**: Document optional enhancement fields with examples and use cases ### Examples **Minimal usage (works exactly as base PR):** ```yaml test_categories: smoke: description: "Quick sanity tests" test_patterns: ["*smoke*"] labels: ["smoke"] ``` **Enhanced usage (when team wants better documentation):** ```yaml test_categories: smoke: description: "Quick sanity tests" notes: "Run for any API changes. Critical path tests." source_coverage: ["library/src/api.cpp"] api_coverage: ["initialize", "cleanup"] feature_tags: ["fast", "critical-path"] test_patterns: ["*smoke*"] labels: ["smoke"] ``` ### Why Include Optional Fields in the Standard? 1. **Consistency** - Teams wanting richer metadata have a standardized structure 2. **Flexibility** - No mandatory adoption, teams use what they need 3. **Prevents divergence** - Projects like hipBLASlt that need extra metadata can adopt the standard as-is rather than creating project-specific extensions 4. **Future-proof** - Enables innovation (AI-assisted test selection, better documentation) without breaking changes ### Testing - ✅ Parser gracefully ignores optional fields (no code changes needed) - ✅ CTest execution behavior unchanged - ✅ Backward compatible with existing deployments ### Context This enables projects like hipBLASlt (with ~200+ test configurations) to adopt the standardized approach with rich test metadata, rather than implementing project-specific extensions that would defeat the standardization effort. This PR is a rebase of #3897 to target the new MIOpen branch (`users/dravindr/ctest_miopen` from PR #3513) instead of the abandoned PR #2411.

randyspauldingamd · 2026-03-05T22:04:31Z

+            # Write category test to CMake file and install file.
+            # =======================================================================
+            print("add_test(")
+            print(f"  NAME {target_name}-{category_name}-suite")


Wups, we missed this earlier. We only use underscores in test names, never dashes. Could you please change these for consistency (in all places these names are used)?

I've created the PR #5276 for this.

#3595) ## Motivation This PR is to integrate the Test Filter Standardization POC done in MIOpen to TheRock Once active, the MIOpen CI tests will be triggered through this. Other components will follow. ## Technical Details The POC in MIOpen is done here: ROCm/rocm-libraries#3513. This PR adds test_runner.py as a generic runner to call the required test categories from MIOpen, and can be expanded to other components. ## Test Plan Make sure that MIOpen tests are getting triggered and run fine through TheRock CI.

## Motivation This PR is created to implement the change requested in this [review comment](#3513 (review)) from PR #3513. The ctest entries created as part of Test Filter Standardization in the above PR is changed from the format {target_name}-{category_name}-suite to {target_name}_{category_name}_suite. Co-authored-by: Dileep Ravindranathan <dileep@dell-rack-13.amd.com> Co-authored-by: Randy Spaulding <167460321+randyspauldingamd@users.noreply.github.com>

The following motivation is adapted from ROCm#2411. POC done on MIOpen. **Standardization of Test Filtering Strategy Across The ROCm Stack** Note that this is a proof of concept that will go through multiple iterations and improvements. **Problems** 1. The ROCm stack consists of projects that use different test frameworks (googletest, catch2, pytest, tox, and ctest among others). Different commands must be learned and used to test across the stack. 2. Definitions of smoke test, sanity test, and what should be run pre-submit and on-submit vary from team to team and is not a problem isolated to ROCm. 3. While code coverage and test coverage can be useful values, the value of test results as a function of turnaround time should be emphasized given the ongoing constraints on compute test resources. 4. For some projects, disablement of flaky tests and the list of tests for a given set (e.g., pre-submit) is specified at the orchestration level and not in the project source, or the lists are not in a readable format to easily track when tests were added to or removed from filters and why. 5. Windows has been neglected in many cases with testing strategy. **Requirements** 1. Solution must work on top of what ROCm projects already have in place with the current test frameworks already in use, but have the new standard provide the enhanced and standardized capabilities. 2. Instead of enforcing definitions of terms like smoke or sanity, define test sets and test suites based on test execution time which can be universally understood. a. If a project only has five minutes to run tests on a resource, populate the five-minute test set with the list of tests for the project that would provide the most valuable signal within those time constraints. b. These test sets should be owned and continuously maintained by the project teams, as new features get added, removed, or as feature set priorities change. c. Comprehensive test sets for code coverage and test coverage are specified separately, with clear documentation on the test execution time for these sets. 4. The lists of tests that are in test sets and the current tests excluded from runs due to issues should be in a human-readable format in the source code of the project at a standardized location. 5. The solution must work on all supported OS distros of ROCm across Linux and Windows. 6. Compilation and packaging of tests must be supported on a GPU-less system, but local execution of these same commands should be functional on a GPU system. 7. Support should encompass ROCm core first, ROCm extras as optional, as well as the key frameworks important to customers (e.g., pytorch, JAX, etc.) **Specifications Proposal** 1. As the majority of ROCm is based on CMake, and with CTest providing support for execution of commands to work with pytest, gtest, etc., CTest as the solution wrapper seems like the natural choice. 2. Test set definitions are human-readable files in a common directory in the project source (e.g., resources/tests/definitions/five-minutes.yml). No strong opinions on markup language choice or the actual path. These definitions are referred to as presets in CTest. https://cmake.org/cmake/help/latest/manual/ctest.1.html 3. In the build process, the build node should translate those test definitions from the human-readable files and converted to the presets that CTest understands through CMake invocations and produce the output that can be packaged and transferred to a test system. 4. Test systems should be able to run ctest --preset through those transferred test packages. Similar commands should work across ROCm projects and OS environments. 5. Proof of concept work will be done on two ROCm projects, each using a different test framework to verify the solution is flexible with its support. 6. The only changes that should occur on TheRock with this work is that the test invocation calls from TheRock should become a common string, regardless of ROCm project. --------- Co-authored-by: Dileep Ravindranathan <dileep@dell-rack-13.amd.com> Co-authored-by: Tony Davis <tony.davis@amd.com>

## Motivation This PR is created to implement the change requested in this [review comment](ROCm#3513 (review)) from PR ROCm#3513. The ctest entries created as part of Test Filter Standardization in the above PR is changed from the format {target_name}-{category_name}-suite to {target_name}_{category_name}_suite. Co-authored-by: Dileep Ravindranathan <dileep@dell-rack-13.amd.com> Co-authored-by: Randy Spaulding <167460321+randyspauldingamd@users.noreply.github.com>

Implements AIROCBLAS-45. Adds rocBLAS to the same CTest test-filter setup as **#3513 (Test Filter Standardization – MIOpen)**, following that pattern and coordinating with the #3513 authors. ## What this PR does - Wires rocBLAS into the shared CTest flow from #3513: `shared/ctest` (TestCategories.cmake, parse_test_categories.py) and YAML-driven categories. - Sets `ROCM_LIBRARIES_ROOT` in rocBLAS and includes the shared module in `clients/gtest/CMakeLists.txt`. - Adds `projects/rocblas/clients/gtest/test_categories.yaml` with categories **quick**, **standard**, **comprehensive**, **full** and execution settings (timeouts, env). - Installs an install-time CTestTestfile to `bin/rocblas/` so TheRock (and other consumers) can run CTest from the install tree. - Documents rocBLAS in `shared/ctest/README.md` under Integrations. ## Test categories | Category | Purpose | Timeout | |----------------|----------------|---------| | quick | Smoke | 10 min | | standard | Precheckin | 2 hours | | comprehensive | Extended/nightly | 8 hours | | full | Stress/weekly | 24 hours | ## Context (#3513) Test filtering is being standardized across ROCm: time-based categories, human-readable YAML in tree, CTest labels for CI, and optional GPU/OS exclusions. MIOpen adopted this first; rocBLAS is the next project on the same path. ## Testing - **TheRock CI:** rocBLAS test job runs and passes (e.g. Linux gfx94X-dcgpu, Windows gfx1151). - **Local:** From a rocBLAS build, `ctest -L quick`, `ctest -L standard`, etc., run the corresponding suites with the configured timeouts. **Note:** TheRock CI will run rocBLAS tests only after [TheRock PR #3769](ROCm/TheRock#3769) is merged (that PR adds rocBLAS to the test matrix and artifact descriptor). --------- Co-authored-by: Dileep Ravindranathan <dileep@dell-rack-13.amd.com> Co-authored-by: root <root@dell-rack-13.amd.com> Co-authored-by: Joseph Macaranas <145489236+jayhawk-commits@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>

## Summary Adds YAML-driven CTest test categorization for the **miopen-provider** plugin so TheRock CI can drive `ctest -L {quick|standard|comprehensive|full}` from the installed `bin/miopen_plugin/` tree via the generic `test_runner.py` (replacing the legacy `test_miopenprovider.py` one-off). More details available at [test_filtering.md](https://github.com/ROCm/TheRock/blob/main/docs/development/test_filtering.md). Paired TheRock branch: [`users/dravindr/tr_miopen_prov`](https://github.com/ROCm/TheRock/tree/users/dravindr/tr_miopen_prov) (PR forthcoming) — switches the `miopenprovider` job's `test_script` to `test_runner.py`, adds `miopenprovider -> miopen_plugin` to `COMPONENT_DIR_MAPPING`, and broadens `artifact-miopenprovider.toml`'s test include to `bin/miopen_plugin/**`. Reference (canonical pattern): [`#3513`](#3513) + [`ROCm/TheRock#3097`](ROCm/TheRock#3097) (miopen). ## Changes ### `dnn-providers/miopen-provider/test_categories.yaml` (new) Added the test_categories.yaml for test selection ### `dnn-providers/miopen-provider/CMakeLists.txt` - Set `ROCM_LIBRARIES_ROOT` (guarded `if NOT DEFINED`) so `shared/ctest/TestCategories.cmake` is discoverable from the monorepo root. - After `finalize_test_targets(...)`, call `apply_test_category_labels()` once per test binary, writing both sets of `_quick_suite` / `_standard_suite` / `_comprehensive_suite` / `_full_suite` entries into a single install-time `CTestTestfile.cmake` with relative paths (`../miopen_plugin_tests`, `../miopen_plugin_integration_tests`). - Install the YAML-generated `CTestTestfile.cmake` to `bin/miopen_plugin/` when shared/ctest is available; otherwise fall back to the project's existing `install_provider_ctest_files()` so standalone / sparse checkouts that lack `shared/ctest/` keep working with their current unlabeled install file. ### `.github/actions/ci-env/action.yml` *(WIP — revert before merge)* Bumps the `therock-ref` output from the pinned 2026-05-29 SHA (`685f8c3b890063...`) to the paired TheRock branch `users/dravindr/tr_miopen_prov`. This file is the single source of truth for `therock-ci-linux.yml`, `therock-ci-windows.yml`, `therock-ci.yml`, `therock-test-component.yml`, and `therock-test-packages.yml`, so this one edit replaces what would otherwise be four parallel diffs. **Must be reverted to a pinned `ROCm/TheRock` SHA before merge**, once the paired TheRock PR lands and gives us a stable commit on `main`. ## Test plan - [x] CI build (Linux) on `users/dravindr/tr_miopen_prov` produces a `miopenprovider` artifact whose `bin/miopen_plugin/CTestTestfile.cmake` lists all 8 `*_suite` entries with labels. - [x] CI build (Windows) ditto. - [x] `therock-test-component.yml` / `therock-test-packages.yml` runs of the `miopenprovider` job invoke `test_runner.py` with `TEST_COMPONENT=miopenprovider` and `TEST_TYPE=quick` and pass. - [x] Revert `.github/actions/ci-env/action.yml` to a pinned `ROCm/TheRock` SHA once the paired TheRock PR is merged. Made with [Cursor](https://cursor.com) --------- Co-authored-by: Cursor <cursoragent@cursor.com>

…#13107) JIRA: ALMIOPEN-2602 ## Summary Jenkins runs every `miopen_gtest` case whose name contains `FP32`/`FP16`/`BFP16`/`TF32`/`NONE`. The GitHub CI tiers in `test_categories.yaml` are a whitelist, so about 21,000 cases in 191 suites were never selected by any tier. This PR adds the missing families and tiers them by measured time. It also quarantines the failures that the new coverage exposed: each exclusion is scoped to the arch/OS where it fails and carries a ticket. ## Changes (`projects/miopen/test/gtest/test_categories.yaml`) ### Coverage added | Where | Change | |---|---| | `positive_patterns` | **+30** prefixed families: `GPU_Adam`, `GPU_AmpAdam`, `GPU_AddLayerNorm`, `GPU_LayerNormBwdTest`, `GPU_T5LayerNormBwdTest`, `GPU_BNInputValidation`, `GPU_BN_Fwd_Spatial_Welford`, `GPU_BN_Spatial`, `GPU_Bn3DPeract`, `GPU_Bn3DSpatial`, `GPU_BnDerive`, `GPU_BnPeract`, `GPU_CbnaInference`, `GPU_na_*`, `GPU_Allocator`, `GPU_CandidateSelection`, `GPU_Find2Conv`, `GPU_FindDb`, `GPU_FindModeTrustVerify`, `GPU_GRU`, `GPU_LSTM`, `GPU_LstmMSRnn`, `GPU_RNNSeqApi`, `GPU_RNNVanilla`, `GPU_Pooling2d`, `GPU_Pooling3d`, `GPU_TensorReorder`, `GPU_TensorTransform`, `GPU_binaryTensorOps`, `GPU_reference_kernel` | | `positive_patterns` (unprefixed `TEST`/`TEST_F`) | **+16**: `GPU_BN_Spatial_FP32`, `GPU_CKValidatorIntegration_FP32`, `GPU_CkImplLoader_FP16`, `GPU_Conv{2D,3D}KernelTuning{AI,Solver}_FP32`, `GPU_ConvAIModelLoad_FP32`, `GPU_ConvNDKernelTuningUtils_FP32`, `GPU_ConvStridedTensors_FP32`, `GPU_HipBLASLtGEMMTest_{FP32,FP16,BFP16}`, `GPU_PerfConfig_HipImplicitGemm3DGroup`, `GPU_RunAIHeuristics_Integration_FP32`, `GPU_SoftmaxFind20` | | `positive_patterns` | **−2** dead patterns `*/GPU_GRUExtra_*` and `*/GPU_lstm_extra*` (they matched nothing; now covered by `GPU_GRU_*` / `GPU_LSTM_*`) | | `standard.exclude` | **+21** `Full/` excludes, so standard runs only the `Smoke/` and unprefixed instantiations of the new families | | `comprehensive.exclude` | **+12** full-only suites (> 200 s serial or untimed): BN find2/fusion tuning (`GPU_na_train_find2_*`, `GPU_na_fusion_test_bn_*`), `GPU_BN_Spatial_FP32`, `GPU_BN_Fwd_Spatial_Welford`, `GPU_LSTM_fused_infer`, `I8`/`I32` variants | | `comprehensive.exclude`, `full.exclude` | `Full/GPU_LstmMSRnn_FP32.TestFloat/1095` (ALMIOPEN-1926), the same index Jenkins filters on gfx94X | ### Quarantines (scoped to arch/OS) | Key | Pattern | Cases removed | Reason | Ticket | |---|---|---:|---|---| | `exclude_gpu_gfx110X` | `Full/GPU_LstmMSRnn_FP32*` | 1,535 (comprehensive and full) | Intermittent CPU/GPU mismatch at `rnn_seq_api.hpp:272`, bias-enabled cases only, a different index set on every attempt. 88 of 4,316 executions failed (2.0%) at 76 distinct indices. FP16 (0 / 3,312) and Linux gfx94X (0 / 2,302) are clean. | ALMIOPEN-1926 | | `exclude_gpu_gfx115X` | `Full/GPU_LstmMSRnn_FP32*` | 1,535 | Same signature on Windows gfx1151: 18 of 2,316 executions failed (run 39358709264) | ALMIOPEN-1926 | | `exclude_gpu_gfx115X` (Linux + Windows) | `Smoke/CPU_DBSync_NONE.StaticFDBSync/9` | 1 (every tier) | `Db file does not exist … gfx115128.HIP.fdb.txt` / `gfx115120.HIP.fdb.txt` on hosts whose CU count has no shipped db. Windows: 5 of 5 executions on 5 hosts, including develop nightly 38527180317, so it predates this PR. Linux: 12 of 12 on the 20-CU strixhalo hosts in run 39740910967 (passes on CS-RORDMZ hosts). Only the gfx115128 (28-CU) db ships: #13680 fixed its Windows install, but 20-CU hosts still have no gfx115120 db. | #13416 | | `exclude_gpu_gfx115X_windows` | `Full/GPU_GRU_Base_FP16.TestFloat16/batchSize_17_seqLength_23_inVecLen_13_hiddenSize_67_*biasMode_1_*` | 8 | 3 of these 8 bias-enabled cases failed verification at `gru_test.cpp:1985/2224/2473` (same bias-only signature); they passed on the rerun and in the next run | ALMIOPEN-1926 | | `exclude_gpu_gfx110X`, `exclude_gpu_gfx115X` (Linux + Windows) | `GPU_ConvAIModelLoad_FP32*` | 1 (every tier) | `no fdeep .tn.model files found`: AI heuristic models ship only for gfx908/gfx90a/gfx942/gfx950, and each install carries only the runner target's db files. Linux gfx1151 failed first (run 39358709264, reproduced locally with the CI binaries). Since #13680 (merged 10-29) Windows installs per-target db files too, so Windows gfx110X fails as well (run 40983714656, 2 of 2 attempts on 2 runners); gfx94X/gfx90a/gfx950 keep the test. | **ALMIOPEN-2966** | | `exclude_gpu_gfx115X_linux` | `CPU_PerfDb_NONE.*`, `GPU_BN_Spatial_FP32.MIOpen3900Regression` | 3 + 1 (every tier) | Killed by the OOM killer (SIGKILL / exit 137) on the 7.5 GB runners `strixhalo-12` and `strixhalo-10` (`MIOpen3900Regression` uses ~2.1 GB per tensor on a shared-memory APU); both pass on the 31–125 GB gfx1151 hosts. Only standard-tier cases are excluded (TheRock release tests run standard on these hosts). Out-of-memory kills in comprehensive/full are not excluded case by case, see open items. | **ALMIOPEN-2964** | Jenkins doesn't run Windows gfx110X/gfx115X or Linux gfx1151, so these quarantines need no Jenkins `add_gtest_negative_filter`. gfx94X keeps full `LstmMSRnn` coverage apart from `/1095`, which matches Jenkins's filter, and gets re-evaluated if a GitHub CI run sees it fail. ## CI runs on this PR | Run | Date | Commit | Tier | Archs tested | Result | |---|---|---|---|---|---| | [37362130564](https://github.com/ROCm/rocm-libraries/actions/runs/37362130564) | 10-05 | `7b72efe` | standard | gfx94X, gfx950, Win gfx110X | ✅ | | [37844419899](https://github.com/ROCm/rocm-libraries/actions/runs/37844419899) | 10-08 | `7b72efe` | standard¹ | gfx94X | ✅ | | [37960128662](https://github.com/ROCm/rocm-libraries/actions/runs/37960128662) | 10-09 | `7b72efe` | standard¹ | gfx94X | ✅ | | [38770867047](https://github.com/ROCm/rocm-libraries/actions/runs/38770867047) | 10-15 | `7b72efe` | standard | gfx94X, Win gfx110X | ✅ | | [38815363944](https://github.com/ROCm/rocm-libraries/actions/runs/38815363944) | 10-15 | `1d54535` | comprehensive | gfx94X, Win gfx110X | ❌ Win gfx110X `LstmMSRnn_FP32` flaky (both attempts); one runner lost (TheRock#3845) | | [38983909810](https://github.com/ROCm/rocm-libraries/actions/runs/38983909810) | 10-16 | `1d54535` | full | gfx94X, Win gfx110X | ❌ Win gfx110X `LstmMSRnn_FP32` flaky (both attempts) | | [39338174289](https://github.com/ROCm/rocm-libraries/actions/runs/39338174289) | 10-19 | `d9701b2` | full | Win gfx110X (completed before cancel) | ✅ gfx110X green after its quarantine | | [39358709264](https://github.com/ROCm/rocm-libraries/actions/runs/39358709264) | 10-20 | `e964061` | full | gfx94X, **gfx90a**, **gfx1151** Linux + Windows, Win gfx110X | ❌ gfx1151 only: `LstmMSRnn_FP32` (Win), `ConvAIModelLoad` (Linux), `StaticFDBSync/9` (Win), GRU FP16 flake (Win), OOM on strixhalo-12, hang on CS-RORDMZ-DT245, GPU `unspecified launch failure` and runner lost (Win) | | [39480689944](https://github.com/ROCm/rocm-libraries/actions/runs/39480689944) | 10-20 | `48eccae` | full | same as above | ❌ gfx1151 only: `StaticFDBSync/9` (Win), OOM on strixhalo-12 (Linux), hang on CS-RORDMZ-DT245 (Linux). gfx94X, gfx90a, gfx110X green. | | [39529845480](https://github.com/ROCm/rocm-libraries/actions/runs/39529845480) | 10-21 | `455cc52` | full | same as above | ❌ Linux gfx1151 only: OOM on strixhalo-10 in `GPU_BN_Spatial_FP32.MIOpen3900Regression` (now excluded under ALMIOPEN-2964) and `…/n8c192h256w512`, plus one container-setup failure. **No gtest failure in any lane.** gfx94X, gfx90a, gfx110X and Windows gfx1151 green. | | [39740910967](https://github.com/ROCm/rocm-libraries/actions/runs/39740910967) | 10-22 | `7012ede` | full | same as above | ❌ Linux gfx1151 only: OOM on strixhalo-10/-12 in **new** large cases (`na_fusion_test_bn_spatial_FP32 nchw_8_192_256_512`, `na_fusion_inference_test_FP32 nchw_1_16_4096_4096`) and `StaticFDBSync/9` on Linux (now excluded on all gfx115X); infra: container setup (halobox-b13), `BAD_COMMAND` on Windows host wshctr-halobox-b48-05 (passed on rerun). gfx94X, gfx90a, gfx110X and Windows gfx1151 green. | | [40053357276](https://github.com/ROCm/rocm-libraries/actions/runs/40053357276) | 10-24 | `ffcb4d4` | full | gfx94X, Win gfx110X | ✅ all MIOpen shards | | [40746486850](https://github.com/ROCm/rocm-libraries/actions/runs/40746486850) | 10-28 | `94c1c2b` | standard (opt-in labels removed) | gfx94X, Win gfx110X | ✅ all MIOpen shards | | [40901558044](https://github.com/ROCm/rocm-libraries/actions/runs/40901558044) | 10-29 | `98c7817` | standard (TEMP workflow change removed) | gfx94X, Win gfx110X | ❌ Linux gfx94X: all 17 FP8 + 17 BFP8 `RunHipBLASLtGEMM` cases (`hipblaslt_gemm.cpp:235`, error 0.12–0.17 vs 0.07). Not this PR: the attempt-2 rerun rebuilt only gfx950, whose `*_generic` MIOpen artifacts (built with the IEEE FP8 bias) replaced gfx94X's; tracked in #14375, fix in #14627. Windows green. | | [40983714656](https://github.com/ROCm/rocm-libraries/actions/runs/40983714656) | 10-30 | `7d71fd8` (develop merge) | standard | gfx94X, Win gfx110X | ❌ Windows gfx110X shard 3 only: `GPU_ConvAIModelLoad_FP32.BundledFdeepModelsLoad` (2 of 2 attempts) after #13680 moved Windows dbs to per-target packages; now excluded on gfx110X/gfx115X (ALMIOPEN-2966). Linux gfx94X green, including FP8/BFP8. | ¹ The `test_type:comprehensive` / `test_type:full` labels used then are not wired into the multi-arch workflow, so these runs ran standard. Labels used later: `test_filter:comprehensive`, `test_filter:full`, `ci:gfx90a`, `ci:gfx1151`. ## Timing All numbers are **per shard** (4 shards, serial ctest), as ctest wall minutes; the TheRock Test step is capped at **120 min**. ### Measured | OS / arch | Tier | develop | This PR | Δ | Runs (PR / develop) | |---|---|---:|---:|---:|---| | Linux gfx94X | standard | 26.8 (25.5–28.7) | 27.7 (26.8–29.2) | **+0.9** | 37362130564, 37844419899, 37960128662, 38770867047 / 38541202878, 38945603498, 38998133138 | | Linux gfx94X | comprehensive | 41.2 (38.8–42.7) | 57.4 (54.0–62.7) | **+16.2** | 38815363944 / nightly 38697600236 | | Linux gfx94X | full | not run² | 68.6 (63.9–72.5) | ≈ +27 | 38983909810, 39358709264, 39480689944 | | Linux gfx90a | full | not run | 83.6 (78.3–86.6) | – | 39358709264, 39480689944 | | Linux gfx950 | standard | not run | 15.4 (15.0–16.0) | – | 37362130564 | | Linux gfx1151 | full | not run | 28 (27.1–29.0)³ | – | 39358709264, 39480689944 | | Windows gfx110X | standard | 17.2 (10.9–19.5) | 17.1 (8.8–20.6) | within runner-host noise⁴ | 37362130564, 38770867047 / same develop PR runs | | Windows gfx110X | comprehensive | not run⁵ | 46.3 (42.0–50.3) | – | 38815363944 | | Windows gfx110X | full | not run | 51.9 (48.0–56.7) | – | 38983909810, 39338174289, 39358709264, 39480689944 | | Windows gfx1151 | comprehensive | 28.3 (20.0–42.8) | – | – | – / nightly 38697600236 | | Windows gfx1151 | full | not run | 53.2 (35.3–73.2)⁶ | – | 39480689944 | The slowest shard measured is gfx90a full at 86.6 min (72% of the cap). Every other lane is at or below 73 min. ² Develop's full tier selects the same cases as develop comprehensive (within 25 cases), so the delta is measured against the develop comprehensive time. ³ Healthy shards only; the other shards were OOM-killed (ALMIOPEN-2964) or hung at startup (ALMIOPEN-2965). ⁴ Windows gfx110X shards range from 9 to 21 min depending on the runner host (CS-RORDMZ vs azure scale-set). ⁵ The develop nightly runs Windows gfx1151, not gfx110X. ⁶ Large spread between hosts: one gfx1151 runner is about 2× slower than the others. ### Modelled delta vs develop (final yaml, cases added and per-shard minutes) `yaml_select.py diff --old git:<merge-base 4c5c20a> --new git:HEAD --gpu <arch> --os <os>`, using this PR's per-case timings measured on each arch. | OS / arch | standard | comprehensive | full | |---|---|---|---| | Linux gfx94X | +11,284 cases · **+1.5 min** | +25,714 · **+15.6 min** | +29,887 · **+23.8 min** | | Linux gfx90a | +11,284 · **+1.9 min** | +25,714 · **+18.3 min** | +29,887 · **+27.4 min** | | Windows gfx110X | +11,281 · **+1.4 min** | +24,068 · **+15.8 min** | +28,241 · **+21.8 min** | | Windows gfx1151 | +11,281 · **+1.5 min** | +24,060 · **+16.9 min** | +28,233 · **+23.2 min** | The model agrees with the measurements where a develop baseline exists: gfx94X comprehensive +15.6 modelled vs +16.2 measured; standard +1.5 vs +0.9. <details><summary>Largest additions per tier (Linux gfx94X, serial minutes)</summary> | Tier | Suites | |---|---| | standard | `Smoke/GPU_Find2Conv` 0.8, `Smoke/GPU_TensorReorder` 0.4, `Smoke/GPU_reference_kernel_{bwd,bww}_2d_*` 0.3–0.4 each | | comprehensive | `Full/GPU_Pooling3d` 9.0, `Full/GPU_LayerNormBwdTest` 7.8, `Full/GPU_Pooling2d` 7.0, `Full/GPU_BnPeract` 4.3, `Full/GPU_TensorTransform` 3.7 | | full only | `Full/GPU_na_fusion_test_bn_spatial` 10.8, `Full/GPU_na_train_find2_spatial` 10.2, `Full/GPU_na_*_peract` 5.8 each | </details> ### Where each tier runs after merge | Tier | Lanes | |---|---| | standard | rocm-libraries PR gate and develop postsubmit (Linux gfx94X, Windows gfx110X); TheRock postsubmit (gfx94X, gfx90a, gfx950, Windows gfx110X); TheRock release tests (Linux gfx1151, Windows gfx110X) | | comprehensive | rocm-libraries nightly (Linux gfx94X, Windows gfx1151); ASAN nightly (gfx94X) | | full | only on request, with a `test_filter:full` label | > [!NOTE] > **Standard-tier hot spot (pre-existing, not changed by this PR).** In this PR's standard runs on Linux gfx94X, `Full/GPU_Softmax_*` > is the most expensive group: **36.5 of 109.6 serial minutes (33%)**, about 9 min per shard. Breakdown per run: `Full/GPU_Softmax_FP32` > 18.5 min (1,008 cases), `_FP16` 7.1, `_BFP16` 7.0, `_NonContiguous_FP32` 3.9. It has been in standard since #3513. On gfx11/gfx12 > the FP32/FP16/BFP16 variants are already excluded as slow (TheRock#3362); `Full/GPU_Softmax_NonContiguous_FP32` still costs 5.1 min > (8%) on Windows gfx110X. > > Moving it to comprehensive (adding `Full/GPU_Softmax_*` to `standard.exclude`) is modelled at **−9.1 min per shard on gfx942** > (27.7 → about 18.6) and **−1.2 min per shard on Windows gfx110X**. Comprehensive and full keep every case, so the nightly still covers it. > Proposed as a follow-up rather than part of this PR. Next largest in standard: `Standard/GPU_Reduce_{FP32,FP16}` 3.9 + 3.1 min, > `Full/GPU_GroupConv3D_*` 1.4–2.2 min per dtype. ## ASAN validation of the added suites Every gtest suite this PR adds was built and run under full ASAN, both host-side and device-side, on gfx942 with XNACK on (`gfx942:xnack+`). Apart from one pair of test files that leak memory, they are clean. - **Setup.** This branch was built with `-fsanitize=address` against the rockrel *Multi-Arch Release ASAN* nightly #139 artifacts (run [40425965688](https://github.com/ROCm/rockrel/actions/runs/40425965688), TheRock `e73fd19`), using the same test environment as TheRock CI (`configure_asan_env.py`, `ROCM_PATH`, `HSA_XNACK=1`). It ran on MI300X: 28,354 selected cases in 4 shards, plus a canary run of the `Smoke/` and unprefixed suites first. - **Result.** **25,438 tests passed, with 0 gtest failures and 0 ASAN memory errors** (no overflow or use-after-free in MIOpen). - **One MIOpen problem, in the tests themselves:** `na_train_find2.cpp` and `na_inference_find2.cpp` call `miopenFindSolutions` but never `miopenDestroySolution`, so LeakSanitizer fails the process at exit. They are excluded from ASAN builds only, using `SKIP_TESTS` in `test/gtest/CMakeLists.txt` under `THEROCK_SANITIZER=ASAN`, next to the existing `find_2_conv.cpp` leak skip. This removes the 12 `*/GPU_na_{train,inference}_find2_*` suites (528 cases) from ASAN builds; normal CI keeps them. Tracked in **ALMIOPEN-3097**. - **Runtime issues, not MIOpen:** - **HIP start-up under ASAN with XNACK on (AILIROCR-587).** When two ASAN processes run on the same node, the HIP runtime hangs while initializing (`KernelBlitManager::initHeap`) or ASAN fires inside ROCr (`AqlQueue::HandleInsufficientScratch`). A single process started 5 of 5 times. The run therefore used one ASAN process per node. - **Bogus device-side leak report (ALMIOPEN-3098).** At exit, the device ASAN runtime lists about 1 TB of "leaks" on a 192 GB GPU, and the stack addresses are float data. It's informational and doesn't change the exit code. - **ROCr loader leak (AILIROCR-588).** A one-off 51-byte `strdup` leak in ROCr's `LoadCodeObject`, seen in the canary only. I didn't add an ASAN exclusion to `test_categories.yaml`. Its parser has no sanitizer-specific key, so any yaml key would also drop these suites from normal CI. ## Tickets | Ticket | Status | Scope | |---|---|---| | ALMIOPEN-1926 | referenced | `LstmMSRnn_FP32` (and GRU FP16) bias-only CPU/GPU mismatch: gfx110X, gfx115X, and the Jenkins gfx94X `/1026`, `/1095` | | **ALMIOPEN-2964** | **created for this PR** | 7.5 GB Linux gfx1151 runners `strixhalo-12` and `strixhalo-10` OOM-kill `miopen_gtest`; matches the `exclude_gpu_gfx115X_linux` OOM exclusions. The fix is to remove those hosts from the runner pool. | | **ALMIOPEN-2965** | **created for this PR** | `miopen_gtest` hangs before any output on Linux gfx1151 runners CS-RORDMZ-DT245 (3 of 3 jobs) and ctr-halobox-b13; no test starts, so there is no yaml exclusion | | **ALMIOPEN-2966** | **created for this PR** | `GPU_ConvAIModelLoad_FP32.BundledFdeepModelsLoad` fails instead of skipping when the system db has no `*.tn.model` (Linux gfx1151; Windows gfx110X/gfx1151 since #13680); repro steps on any GPU via `MIOPEN_SYSTEM_DB_PATH`; matches the `exclude_gpu_gfx110X` / `exclude_gpu_gfx115X` entries | | **ALMIOPEN-3097** | **created for this PR** | ASAN: `na_train_find2` / `na_inference_find2` tests leak `miopenSolution_t` (no `miopenDestroySolution`); matches the ASAN `SKIP_TESTS` entries | | **AILIROCR-587** | **created for this PR** | ASAN: HIP init on gfx942 hangs in `KernelBlitManager::initHeap` or trips ASAN in ROCr `AqlQueue::HandleInsufficientScratch` when two ASAN processes share a node | | **ALMIOPEN-3098** | **created for this PR** | ASAN: device-side leak report at exit lists ~1 TB of impossible allocations (stacks are float data) | | **AILIROCR-588** | **created for this PR** | ASAN: 51 B `strdup` leak in ROCr `ExecutableImpl::LoadCodeObject` (seen once) | | #13416 / #13680 | referenced | `StaticFDBSync` "Db file does not exist" on hosts whose CU count has no shipped db / Windows db install fix | | ROCm/TheRock#3845 | referenced | lost azure gfx1101 runner (38815363944 attempt 1) | ## Test plan - `python3 shared/ctest/parse_test_categories.py test_categories.yaml …` returns rc 0 (21 suites on Linux). - `yaml_select.py diff` after every quarantine: each change removes only the intended cases on the intended arch/OS; Linux gfx94X and gfx90a selections are unchanged by the RDNA quarantines. - CI: the runs above, read across all attempts. Run 39529845480 confirms the quarantines take effect (`LstmMSRnn_FP32` ran 0 times on gfx110X/gfx115X, gfx94X/gfx90a ran it 3,454 times each with no failures) and has no gtest failure in any lane. - ASAN: local build of this branch against rockrel ASAN nightly #139; all added suites run on MI300X gfx942:xnack+ (see *ASAN validation*). 25,438 passed, 0 failures, 0 ASAN errors. ## Open items - [x] Drop the TEMP workflow commit `4b8890f` before merge: removed in `98c7817` (workflow identical to develop's at the merge-base). - [ ] Linux gfx1151 out-of-memory kills: **no more per-case exclusions.** Each run kills a different large case (4 shapes across 3 runs; 82 cases of ≥ 32M elements; no size threshold separates killed cases from survivors). Fixed by ALMIOPEN-2964 (remove the 7.5 GB hosts). Linux gfx1151 comprehensive/full runs only when requested by label, never by default. - [ ] Final run: TEMP workflow change removed (`98c7817`) and labels `ci:gfx1151`, `ci:gfx90a`, `test_filter:full` removed; the PR gate runs standard on Linux gfx94X and Windows gfx110X. Run 40746486850 (`94c1c2b`) passed. 40901558044 hit the FP8 artifact overwrite (#14375) and 40983714656 hit `ConvAIModelLoad` on Windows gfx110X after #13680; that test is now excluded on gfx110X/gfx115X. Needs one green run with that change. - [ ] Re-enable the `na_*find2` suites under ASAN once ALMIOPEN-3097 is fixed (remove the two `SKIP_TESTS` entries). - [ ] After merge, check the first nightly (Linux gfx94X and Windows gfx1151 comprehensive) against the modelled +15.6 / +16.9 min per shard.

dileepr1 self-assigned this Dec 22, 2025

dileepr1 requested review from a team as code owners December 22, 2025 15:44

github-actions Bot added documentation project: miopen labels Dec 22, 2025

assistant-librarian Bot added the organization: ROCm label Dec 22, 2025

assistant-librarian Bot pushed a commit that referenced this pull request Jan 5, 2026

Fix large tensor grouped conv bwd data test (#3513)

bbf0b1a

aalabsi-amd reviewed Jan 6, 2026

View reviewed changes

dileepr1 force-pushed the users/dravindr/ctest_miopen branch from 12f7db2 to 8eb9bbf Compare January 23, 2026 14:20

dileepr1 mentioned this pull request Jan 26, 2026

Test Filter Standardization POC - Changes to integrate MIOpen ROCm/TheRock#3097

Closed

JonathanLichtnerAMD mentioned this pull request Jan 27, 2026

[MIOpen] Convert bn_3d_peract_test from ctest to Google Test #2447

Closed

4 tasks done

tony-davis mentioned this pull request Jan 27, 2026

Add optional enhanced metadata fields to test categories #4109

Merged

JonathanLichtnerAMD reviewed Jan 27, 2026

View reviewed changes

Comment thread projects/miopen/test/gtest/test_categories.yaml Resolved

JonathanLichtnerAMD reviewed Jan 28, 2026

View reviewed changes

Comment thread shared/ctest/README.md Resolved

Comment thread shared/ctest/README.md Resolved

ammallya pushed a commit that referenced this pull request Feb 3, 2026

Fix large tensor grouped conv bwd data test (#3513)

63bf307

[ROCm/composable_kernel commit: bbf0b1a]

dileepr1 requested review from JonathanLichtnerAMD, TristanZerrAMD and aalabsi-amd February 5, 2026 14:34

JonathanLichtnerAMD reviewed Feb 11, 2026

View reviewed changes

Comment thread projects/miopen/test/gtest/test_categories.yaml Resolved

JonathanLichtnerAMD approved these changes Feb 11, 2026

View reviewed changes

aalabsi-amd reviewed Feb 11, 2026

View reviewed changes

Comment thread projects/miopen/test/gtest/test_categories.yaml Resolved

aalabsi-amd reviewed Feb 11, 2026

View reviewed changes

Comment thread shared/ctest/README.md Resolved

dileepr1 removed the request for review from TristanZerrAMD February 12, 2026 17:49

aalabsi-amd requested review from randyspauldingamd February 12, 2026 22:19

aalabsi-amd approved these changes Feb 12, 2026

View reviewed changes

jayhawk-commits approved these changes Feb 16, 2026

View reviewed changes

Dileep Ravindranathan and others added 9 commits February 23, 2026 15:12

Changes in parser to improve clarity and readability

3ebf03e

Exclude slower tests as per #3362

f3e090c

Remove debugs and add empty string check

d136d7e

Add error handling in TestCategories.cmake

bb60a74

Using context managers (with statement) for file operations

dc0fa5a

Add input validation in TestCategories.cmake

7ee05a5

Made exceptions verbose.

fc83ef9

Changed colors in mermaid diagram to make it readable in dark mode.

Exclude tests added as part of Test Filter Standardization from 'make…

02ec0ee

… check' to avoid long running time during build

dileepr1 force-pushed the users/dravindr/ctest_miopen branch from cd38ed3 to 5cf141a Compare February 23, 2026 20:12

dileepr1 disabled auto-merge February 23, 2026 22:46

Remove the $ to help ninja build

31bdb07

dileepr1 merged commit 70b16b7 into develop Feb 24, 2026
36 of 38 checks passed

dileepr1 deleted the users/dravindr/ctest_miopen branch February 24, 2026 16:10

dileepr1 mentioned this pull request Feb 24, 2026

Changes to integrate Test Filter Standardization for miopen in TheRock ROCm/TheRock#3595

Merged

randyspauldingamd reviewed Mar 5, 2026

View reviewed changes

dileepr1 mentioned this pull request Mar 6, 2026

Test Filter Standardization Proof of Concept #2411

Closed

dileepr1 mentioned this pull request Mar 10, 2026

Use underscores instead of hyphens(-) in test names #5276

Merged

dileepr1 mentioned this pull request May 29, 2026

[miopen-provider] Add test filter standardization #7901

Merged

4 tasks done

aalabsi-amd mentioned this pull request Oct 21, 2026

test(miopen): add Jenkins-only gtest families to test_categories.yaml… #13107

Merged

1 of 5 tasks

Conversation

dileepr1 commented Dec 22, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tony-davis commented Jan 27, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AaronStGeorge commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JonathanLichtnerAMD left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aalabsi-amd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

randyspauldingamd Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dileepr1 Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

AaronStGeorge commented Jan 27, 2026 •

edited

Loading

randyspauldingamd Mar 5, 2026 •

edited

Loading