Repository navigation
test(miopen): add Jenkins-only gtest families to test_categories.yaml… - #13107
aalabsi-amd merged 15 commits into
Conversation
…, tiered by measured time Jenkins selects every miopen_gtest case whose name contains FP32/FP16/BFP16/TF32/NONE, while the GitHub CI tiers are a whitelist, so 21,004 cases in 191 suites were never selected by any tier. Add the 62 missing families and place them by per-test time (Jenkins build #1034 and TheRock gfx942 logs, median ms): - Smoke/ and unprefixed instantiations run in standard (about +3.4 min serial). - Full/ instantiations are excluded from standard; kept in comprehensive when timed and <= 200 s serial per suite. - Full/ suites over 200 s serial (BN find2/fusion tuning) or without timing (I8/I32 variants, new suites) are excluded from comprehensive (full only). Also drop the *GRUExtra* / *lstm_extra* patterns, which matched nothing (now covered by */GPU_GRU_* and */GPU_LSTM_*), and exclude the flaky Full/GPU_LstmMSRnn_FP32.TestFloat/1095 (ALMIOPEN-1926) from comprehensive and full, as Jenkins does on gfx94X.
✅ All Checks Passed — Ready for Review
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🎉 All checks passed! This PR is ready for review. |
aalabsi-amd
added
test_type:comprehensive
Pins therock-multi-arch-ci.yml to ROCm/TheRock users/aalabsi-amd/test-type-label-pin-498444ae (f4e1a35d = current pin 498444ae + test_type:* label support) so test_type:comprehensive and test_type:full labels select those tiers instead of standard.
…miopen-test-categories-jenkins-parity # Conflicts: # .github/workflows/therock-multi-arch-ci.yml
aalabsi-amd
added
test_filter:comprehensive
|
Failing indices per shard. Each count links to the shard's raw job log; GitHub's step viewer truncates these logs before the failures. In the raw log, search for Failing indices by shard
In total, 88 of 4,316 executions failed (2.0%), at 76 distinct indices. A re-run of the same commit fails a different set of indices. All failures report Linux gfx94X ran the same test in both runs with no failures. The same test fails intermittently on gfx942 in Jenkins, with the same bias-only pattern but at a rate about 100 times lower. That is tracked as ALMIOPEN-1926. As it stands, merging this PR would make nearly every Windows gfx110X full-category run fail on this test. Could |
aalabsi-amd
added
ci:gfx1151
Thanks for your comment, yes the purpose of this PR is catch the flaky and failing tests across archs, OSs and categories and exclude them before the merge. That's why I am using labels Full and comprehensive and arch specific ones to avoid merging and then coming across these issues later in nightly runs. |
7b43f48 to
04430e3
Compare
JonathanLichtnerAMD
left a comment
JonathanLichtnerAMD
left a comment
There was a problem hiding this comment.
LGTM, thanks!
|
I think we just need to revert the temporary commit (for the pin). |
|
|
#14169 fixes ALMIOPEN-1926, a data race in Could you drop both exclusions of I withdraw my 2026-10-18 suggestion to exclude |
yes this will be done outside this PR to not delay merging it anymore and there is a ticket tracking the work and I added your comment to it: https://amd-hub.atlassian.net/browse/ALMIOPEN-3053 |
…bs (ALMIOPEN-2966)
PR Pundit review(Testing PR Pundit Code Review agent. Not all comments may be relevant. This is a one-time automated pass: please address what is useful, no need to request a re-review. A quick reply per point (useful or not, and why) helps us calibrate the agent. Reach out to @peymanr for suggestions and feedback.)
1. 🟠 Should fix: GPU_ConvAIModelLoad_FP32.BundledFdeepModelsLoad fails on gfx1200/gfx1201 lanes: this PR excludes it only on gfx110X/gfx115X
This PR adds Who is affected: TheRock-style per-target CI/release test runs of the MIOpen standard/comprehensive/full tiers on gfx120X (and likely other archs without bundled AI models). Checked against the code by an independent pass. 2. 🟠 Should fix: One LstmMSRnn case silently skipped on gfx90a/gfx950/gfx120X: this PR adds a per-index exclude to arch-agnostic lists
The PR adds Who is affected: GitHub CI comprehensive/full runs on archs without a family-level LstmMSRnn exclude (e.g. gfx90a, gfx950, gfx120X) lose one test case. Checked against the code by an independent pass. Related past reviews: #5766 on 3. 🟠 Should fix: Linux gfx110X/gfx115X skip FP32 LstmMSRnn Full cases: this PR put a Windows-only flake in OS-agnostic keys
The Who is affected: Linux gfx110X/gfx115X CI in the comprehensive/full tiers, which no longer run Checked against the code by an independent pass. Related past reviews: #3513 on 4. 🟠 Should fix: ~10GB BN regression test OOM-killed in PR-tier CI on gfx1151: this PR added an unprefixed positive pattern
The new unprefixed positive pattern Who is affected: Standard, comprehensive and full GitHub CI runs on small or shared-memory GPUs other than Linux gfx115X, e.g. Windows gfx1151. Checked against the code by an independent pass. Related past reviews: #13403 on 5. 🟠 Should fix: CPU_PerfDb_NONE silently dropped from comprehensive/full gfx115X suites: this PR's tier labels contradict its comment
The Who is affected: Nightly (comprehensive) and weekly (full) MIOpen gtest runs on Linux gfx1150/gfx1151 lose Checked against the code by an independent pass. 6. 🟠 Should fix: ASan lane loses BN+activation Find-2.0 coverage: this PR skips whole files instead of destroying leaked solutions
Who is affected: Only ASan builds ( Checked against the code by an independent pass. Related past reviews: #6075 on 7. 🟠 Should fix: Non-bias FP32 multi-stream LSTM regressions on gfx942 no longer caught in Jenkins: this PR widened the filter to the whole family
Widening the Jenkins gfx94X filter from Who is affected: Jenkins Full runs on gfx94X ( Checked against the code by an independent pass. Related past reviews: #7780 on 8. 🟠 Should fix: Nightly comprehensive tier gains ~59 untimed Full suites: this PR adds broad family patterns excluded only from standard
The new Who is affected: Nightly comprehensive-tier MIOpen CI runs on all GPU targets. Checked against the code by an independent pass. Related past reviews: #3513 on 9. 🔵 Nit: gfx1151 db-sync case runs again and wrong arch gets skipped if list reorders: this PR uses a positional-index filter
The new exclusion Who is affected: Anyone who later edits the Checked against the code by an independent pass. 10. 🔵 Nit: GRU bias cases may still flake on Windows gfx1151: this PR pins one name prefix instead of a suite-level filter
The new Who is affected: Windows gfx1151 CI runs of the comprehensive/full MIOpen gtest tiers. Checked against the code by an independent pass. Related past reviews: #5766 on 11. 🔵 Nit: ConvAIModelLoad exclusion duplicated across two gfx11 keys, risking a missed removal: this PR skips the shared anchor
The Who is affected: Maintainers who later remove the ALMIOPEN-2966 exclusion; no runtime effect for test runs on gfx110X/gfx115X. Checked against the code by an independent pass. Shared from a pilot of PR Pundit, an agentic code reviewer that builds and tests the change and reads the project's past reviews. Reviewed at |
… and Windows gfx1151 Quarantine the MIOpen failures seen on TheRock bump PRs #9651/#9680 (issue #14874) after #13107 widened the TheRock tiers: - gfx120X: GPU_ConvAIModelLoad_FP32* (no bundled fdeep models, ALMIOPEN-2966) and Smoke/CPU_DBSync_NONE.StaticFDBSync/11 (no gfx120138 system db on 56-CU gfx1201 hosts, #13416). - Linux gfx120X: GPU_SoftmaxFind20_FP16.softmaxBackward_log_channel_mode_fp16 hangs to the ctest timeout on gfx1201 hosts. - Windows gfx115X: CPU_PerfDb_NONE.{RamDb,PlainTextDb}_AllTests hang in the multiprocess write phase.
TheRock Submodule Bump ActivityNewest first
|
… and Windows gfx1151 (#14882) JIRA-ID : ROCM-31323 ALMIOPEN-2966, ALMIOPEN-3129, ALMIOPEN-3130 ## Motivation The rocm-libraries bump into TheRock went red with #13107 (ROCm/TheRock#9651, `f4e24d5` → `8c41584`; still red on #9680). #13107 added Jenkins-only gtest families to the TheRock tiers, and TheRock tests gfx120X and Windows gfx1151, which the rocm-libraries PR gate doesn't cover. Tracking issue: #14874. Bump PR #9651 run [41113572856](https://github.com/ROCm/TheRock/actions/runs/41113572856) failed 4 of 4 attempts. #9680 run [41405101044](https://github.com/ROCm/TheRock/actions/runs/41405101044) also failed. Develop nightlies [41179300934](https://github.com/ROCm/rocm-libraries/actions/runs/41179300934) and [41283494663](https://github.com/ROCm/rocm-libraries/actions/runs/41283494663) were read across all attempts too. | # | Failure | Lane | Evidence | Action | |---|---|---|---|---| | 1 | `GPU_ConvAIModelLoad_FP32.BundledFdeepModelsLoad`: `conv_ai_nd_heuristics_test.cpp:962` "no fdeep .tn.model files found" | Linux + Windows gfx120X | 10 of 10 executions on 8 hosts (#9651 attempts 1–4, #9680) | **exclude** on `gfx120X` (ALMIOPEN-2966, already excluded on gfx110X/gfx115X) | | 2 | `GPU_SoftmaxFind20_FP16.softmaxBackward_log_channel_mode_fp16` hangs after "Testing solution functions..." → `miopen_gtest_standard_gfx120X_suite` ***Timeout 3600 s | Linux gfx1201 (CS-RORDMZ-DT87, -DT81) | 3 of 3 executions on Linux gfx1201. Passes in < 1.2 s on Linux gfx1200 and Windows gfx1201 (5 of 5) | **exclude** on `gfx120X_linux` (**ALMIOPEN-3130**). New with #13107 (`GPU_SoftmaxFind20_*`) | | 3 | `Smoke/CPU_DBSync_NONE.StaticFDBSync/11`: "Db file does not exist … gfx120138.HIP.fdb.txt" | Windows gfx120X, CS-RORDMZ-DT84 | 1 of 5 Windows gfx1201 executions. Only `gfx120140` dbs ship; the other hosts pass | **exclude** on `gfx120X` (#13416, same cause as `StaticFDBSync/9` on gfx115X). Existed before #13107 | | 4 | `CPU_PerfDb_NONE.{RamDb,PlainTextDb}_AllTests` hang in the multiprocess read or write phase ("Waiting for test processes...") until the ctest or step timeout | Windows gfx1151 | 6 hangs on 5 hosts since 10-31: nightly 41179300934 (RamDb), nightly 41283494663 (RamDb; PlainTextDb on attempts 1–3), #9680 (PlainTextDb). None in nightlies 40833587284 or 41018314721 | **exclude** on `gfx115X_windows` (**ALMIOPEN-3129**) | | 5 | Windows gfx1151 standard shard 3 ***Timeout 3600 s; `Smoke/GPU_HipGraphExistTest_FP32.HipGraphExist/2` took 596 s (7.5 s elsewhere) | #9680, windows-strix-halo-gpu-rocm-10 | runner-specific slowness on top of #13107's larger standard tier | not excluded: time budget, see below | | 6 | Nightly Windows gfx1151 comprehensive shards hit the 120-min step cap (killed in `Full/GPU_Pooling3d_FP16`, `Full/GPU_BnPeract_FP32`, `Full/GPU_Reduce_FP16`) | nightlies 10-31, 11-01 | passed on rerun | not excluded: time budget, see below | | 7 | Linux gfx110X shard 2 ctest timeout (azure scale-set runner) | #9651 attempt 1 | passed on attempts 2–4 | not excluded: flaky runner (TheRock#3845) | | 8 | gfx90a `CPU_PerfDb_NONE.MultiFileDb_AllTests` "terminate called recursively" | #9651 attempt 1 | passed on attempts 2–4, seen once | not excluded: one-off | | 9 | gfx950 shards 1 and 3: no tests ran, `Unexpected HTTP response: 429` | #9651 attempt 1 | infra | not excluded | ## Technical Details Only `projects/miopen/test/gtest/test_categories.yaml` changes. Each entry has the narrowest arch/OS scope that covers the failures, plus an inline ticket or issue reference: - `exclude_gpu_gfx120X`: changed from the bare `*gfx11_common_patterns` alias to a list that also holds `GPU_ConvAIModelLoad_FP32*` (ALMIOPEN-2966) and `Smoke/CPU_DBSync_NONE.StaticFDBSync/11` (#13416). - New `exclude_gpu_gfx120X_linux`: `GPU_SoftmaxFind20_FP16.softmaxBackward_log_channel_mode_fp16`, exact name (ALMIOPEN-3130). The other five `SoftmaxFind20` cases keep running. It only hangs on gfx1201, but the exclusion covers all Linux gfx120X. TheRock takes the arch from `AMDGPU_FAMILIES` (`gfx120X-all` → `gfx120X`), so gfx1200 and gfx1201 runners share one suite. An `exclude_gpu_gfx1201_*` key would build a suite that no runner selects. - `exclude_gpu_gfx115X_windows`: `CPU_PerfDb_NONE.RamDb_AllTests` and `CPU_PerfDb_NONE.PlainTextDb_AllTests` (ALMIOPEN-3129). These two spawn child processes (`DbMultiProcessReadTest`, `DbMultiProcessTest`). `MultiFileDb_AllTests` doesn't, so it stays. Linux gfx115X already excludes `CPU_PerfDb_NONE.*` (ALMIOPEN-2964). Jenkins gets no `add_gtest_negative_filter` change. It has no gfx120X or Windows stage, and these tests passed there before #13107 moved them into the TheRock tiers. Not addressed here: Windows gfx1151 shard time. After #13107, a standard shard runs about 6,550 cases, up from about 3,700. Windows gfx120X shard 3 takes 32–51 min against the 60-min ctest limit, and slow gfx1151 hosts push it over. Excluding single tests won't fix that. It needs a tier move (for example `Full/GPU_Softmax_*` out of standard, as proposed in #13107) or a timeout or shard change in a separate PR. ## Test Plan - `python3 shared/ctest/parse_test_categories.py test_categories.yaml miopen_gtest …`: rc 0, 25 suites. The generated `miopen_gtest_standard_gfx120X_suite` filter contains the three gfx120X entries on Linux. - `yaml_select.py diff --old git:origin/develop`, using the 40,957 case names from the logs above, per arch/OS (every tier): | GPU / OS | Removed (each tier) | |---|---| | gfx1201 Linux | 3: `ConvAIModelLoad`, `StaticFDBSync/11`, `SoftmaxFind20_FP16…channel_mode_fp16` | | gfx1201 Windows | 2: `ConvAIModelLoad`, `StaticFDBSync/11` | | gfx1151 Windows | 2: `CPU_PerfDb_NONE.{RamDb,PlainTextDb}_AllTests` | | gfx1151 Linux, gfx942 Linux, gfx1100 Windows | 0 | - After merge: the next TheRock bump should be green on gfx120X shards 1 and 3 and Windows gfx1151 shard 2. ## Test Result The selection diff above shows exactly the intended cases removed, with no change on other archs. ## Submission Checklist - [x] Each exclusion is scoped to arch/OS and cites a ticket or issue - [x] YAML parses and the generator returns rc 0 - [x] Jira for the gfx1201 `SoftmaxFind20` hang (ALMIOPEN-3130) and the Windows `PerfDb` multiprocess hang (ALMIOPEN-3129) - [ ] Re-enable each entry when its ticket closes - [ ] Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/GOVERNANCE.md#pull-requests.
…14169 (#15031) JIRA ID : ALMIOPEN-1926 ## Motivation #14169 (merged 2026-10-26, `cec66d06`) fixed the data race in `RNNDescriptor::RNNForwardMS` that made bias-enabled `Full/GPU_LstmMSRnn_FP32.TestFloat` cases fail intermittently. Every quarantine of this family still in the tree predates that fix: | Where | Pattern | Added by | Data it was based on | |---|---|---|---| | Jenkins `MIOPEN_TEST_GFX94X` negative filter | `Full/GPU_LstmMSRnn_FP32*` | #13107 | Jenkins gfx942, 2026-08-24..10-23 | | `comprehensive.exclude`, `full.exclude` | `Full/GPU_LstmMSRnn_FP32.TestFloat/1095` | #13107 | Jenkins per-index filter | | `exclude_gpu_gfx94X` | `Full/GPU_LstmMSRnn_FP32*` | #13107 | Jenkins gfx942 (above) | | `exclude_gpu_gfx110X` | `Full/GPU_LstmMSRnn_FP32*` | #13107 | Windows gfx110X runs of 10-15/10-16 | | `exclude_gpu_gfx115X` | `Full/GPU_LstmMSRnn_FP32*` | #13107 | Windows gfx1151 run of 10-20 | The #14169 author planned this cleanup as a follow-up once the fix was confirmed. While these quarantines stay, a regression in the LSTM multi-stream forward pass (reachable from `miopenRNNForward`, `miopenRNNForwardTraining` and `miopenRNNForwardInference`) goes untested on gfx942, gfx110X and gfx1151. ## Technical Details - `projects/miopen/test/gtest/test_categories.yaml`: - remove `/1095` from the comprehensive and full excludes; - remove `Full/GPU_LstmMSRnn_FP32*` from `exclude_gpu_gfx110X` and `exclude_gpu_gfx115X`; - remove the `exclude_gpu_gfx94X` key. Its only pattern was this family, so gfx94X goes back to the base suites (`-LE ex_gpu`), as before #13107. That's 21 suites instead of 25. - `projects/miopen/test/gtest/CMakeLists.txt`: remove the Jenkins `MIOPEN_TEST_GFX94X` block, which only held this filter. - **Kept:** `exclude_gpu_gfx115X_windows` `Full/GPU_GRU_Base_FP16…biasMode_1_*`. GRU doesn't use the multi-stream LSTM forward that #14169 changed, so the fix doesn't cover that flake. - **Kept:** `standard.exclude` `Full/GPU_LstmMSRnn_*`. That's tier placement (the Full/ cases run in comprehensive), not a quarantine. ## Test Plan 1. `parse_test_categories.py`: rc 0, 21 suites. 2. `yaml_select.py diff --old git:origin/develop`, using the 69,069 case names from `miopen_gtest --gtest_list_tests` of the post-fix build: | GPU / OS | standard | comprehensive / full | |---|---|---| | gfx942 Linux | 0 | +1,536 `Full/GPU_LstmMSRnn_FP32` | | gfx1100 Windows | 0 | +1,536 | | gfx1151 Windows / Linux | 0 | +1,536 | | gfx1201 Linux, gfx950 Linux | 0 | +1 (`/1095`; they already ran the rest) | 3. **Local A/B stress on gfx942 (MI300X).** Same method as #14169. The pre-fix build is TheRock CI artifacts of rl run 39358709264 (2026-10-20); the post-fix build is rl run 41582846076 (develop `9953cb5` + PR #14882). Two modes: (A) one process with `GPU_MAX_HW_QUEUES=16`; (B) 4 gtest shards sharing one GPU, as Jenkins does. Rounds alternate pre/post, each with a fresh user db and kernel cache. Script: `/scratch/users/aalabsi/tmp/lstm_ab/ab.sh`. 4. **CI on this PR**, labels `test:miopen` + `test_filter:comprehensive` + `ci:gfx1151`. This runs comprehensive (including all 1,536 `Full/GPU_LstmMSRnn_FP32` cases) on Linux gfx94X, Windows gfx110X and Windows/Linux gfx1151. These are the lanes that failed before. Read every attempt. 5. **Jenkins PR**: the gfx942 stages run the family again now that the filter is gone. ## Test Result **Local A/B stress, gfx942 (MI300X, banff-cyxtera-s70-4)**, Slurm jobs 68262769 (3 rounds) and 68262798 (20 rounds), 23 rounds per build per mode, pre/post alternating. Each round runs all 1,536 cases: 1,152 execute and 384 skip by design at `rnn_seq_api.cpp:61` in both builds. | Build | Mode A: 1 process, `GPU_MAX_HW_QUEUES=16` | Mode B: 4 shards on 1 GPU (Jenkins-like) | |---|---|---| | pre-fix (rl run 39358709264, 2026-10-20) | 0 failures / 26,496 | **150 failures / 26,496 (0.57%), in 23 of 23 rounds, 113 distinct indices**, all at `rnn_seq_api.hpp:272` (the ALMIOPEN-1926 signature) | | post-fix (rl run 41582846076, develop `9953cb5`) | 0 / 26,496 | **0 / 26,496, in 0 of 23 rounds** | At the pre-fix rate, 26,496 post-fix executions would give about 150 failures. Seeing 0 by chance is effectively impossible (p ≈ e^-150). Mode A didn't reproduce on this machine for either build; #14169 also needed mode B or many passes. A round takes 77–81 s for both builds, so the fix costs no measurable test time. Logs: `/scratch/users/aalabsi/tmp/lstm_ab/results-68262769/`, `…/results-68262798/`. **Jenkins develop, gfx942** (QE dashboard): 20 failures / 11,719 executions from 2026-09-01 to 10-26; 0 / 1,144 from 10-27 to 10-30, after which #13107 filtered the whole family. gfx90a: 0 / 11,615 before and 0 / 1,872 after; it was never quarantined. **CI on this PR:** pending. Windows gfx110X and gfx1151 are the open question: no Windows run of this family exists after the fix, and the stress test above covers gfx942 only. ## Submission Checklist - [x] Every removed entry cites ALMIOPEN-1926 and is covered by #14169 - [x] YAML parses; selection diff shows only the intended cases - [x] gfx942 A/B: pre-fix fails under stress (150 / 26,496), post-fix doesn't (0 / 26,496) - [ ] PR CI comprehensive: Windows gfx110X / gfx1151 and Linux gfx94X run `Full/GPU_LstmMSRnn_FP32` with 0 failures in every attempt - [ ] Jenkins gfx942 PR stages green with the filter removed - [ ] Remove the `test_filter:comprehensive` label before merge - [ ] Close ALMIOPEN-1926, or narrow it to the GRU FP16 flake
JIRA: ALMIOPEN-2602
Summary
Jenkins runs every
miopen_gtestcase whose name containsFP32/FP16/BFP16/TF32/NONE. The GitHub CI tiers intest_categories.yamlare a whitelist, so about 21,000 cases in 191 suites were never selected by any tier. This PR adds themissing families and tiers them by measured time. It also quarantines the failures that the new coverage exposed: each
exclusion is scoped to the arch/OS where it fails and carries a ticket.
Changes (
projects/miopen/test/gtest/test_categories.yaml)Coverage added
positive_patternsGPU_Adam,GPU_AmpAdam,GPU_AddLayerNorm,GPU_LayerNormBwdTest,GPU_T5LayerNormBwdTest,GPU_BNInputValidation,GPU_BN_Fwd_Spatial_Welford,GPU_BN_Spatial,GPU_Bn3DPeract,GPU_Bn3DSpatial,GPU_BnDerive,GPU_BnPeract,GPU_CbnaInference,GPU_na_*,GPU_Allocator,GPU_CandidateSelection,GPU_Find2Conv,GPU_FindDb,GPU_FindModeTrustVerify,GPU_GRU,GPU_LSTM,GPU_LstmMSRnn,GPU_RNNSeqApi,GPU_RNNVanilla,GPU_Pooling2d,GPU_Pooling3d,GPU_TensorReorder,GPU_TensorTransform,GPU_binaryTensorOps,GPU_reference_kernelpositive_patterns(unprefixedTEST/TEST_F)GPU_BN_Spatial_FP32,GPU_CKValidatorIntegration_FP32,GPU_CkImplLoader_FP16,GPU_Conv{2D,3D}KernelTuning{AI,Solver}_FP32,GPU_ConvAIModelLoad_FP32,GPU_ConvNDKernelTuningUtils_FP32,GPU_ConvStridedTensors_FP32,GPU_HipBLASLtGEMMTest_{FP32,FP16,BFP16},GPU_PerfConfig_HipImplicitGemm3DGroup,GPU_RunAIHeuristics_Integration_FP32,GPU_SoftmaxFind20positive_patterns*/GPU_GRUExtra_*and*/GPU_lstm_extra*(they matched nothing; now covered byGPU_GRU_*/GPU_LSTM_*)standard.excludeFull/excludes, so standard runs only theSmoke/and unprefixed instantiations of the new familiescomprehensive.excludeGPU_na_train_find2_*,GPU_na_fusion_test_bn_*),GPU_BN_Spatial_FP32,GPU_BN_Fwd_Spatial_Welford,GPU_LSTM_fused_infer,I8/I32variantscomprehensive.exclude,full.excludeFull/GPU_LstmMSRnn_FP32.TestFloat/1095(ALMIOPEN-1926), the same index Jenkins filters on gfx94XQuarantines (scoped to arch/OS)
exclude_gpu_gfx110XFull/GPU_LstmMSRnn_FP32*rnn_seq_api.hpp:272, bias-enabled cases only, a different index set on every attempt. 88 of 4,316 executions failed (2.0%) at 76 distinct indices. FP16 (0 / 3,312) and Linux gfx94X (0 / 2,302) are clean.exclude_gpu_gfx115XFull/GPU_LstmMSRnn_FP32*exclude_gpu_gfx115X(Linux + Windows)Smoke/CPU_DBSync_NONE.StaticFDBSync/9Db file does not exist … gfx115128.HIP.fdb.txt/gfx115120.HIP.fdb.txton hosts whose CU count has no shipped db. Windows: 5 of 5 executions on 5 hosts, including develop nightly 38527180317, so it predates this PR. Linux: 12 of 12 on the 20-CU strixhalo hosts in run 39740910967 (passes on CS-RORDMZ hosts). Only the gfx115128 (28-CU) db ships: #13680 fixed its Windows install, but 20-CU hosts still have no gfx115120 db.exclude_gpu_gfx115X_windowsFull/GPU_GRU_Base_FP16.TestFloat16/batchSize_17_seqLength_23_inVecLen_13_hiddenSize_67_*biasMode_1_*gru_test.cpp:1985/2224/2473(same bias-only signature); they passed on the rerun and in the next runexclude_gpu_gfx110X,exclude_gpu_gfx115X(Linux + Windows)GPU_ConvAIModelLoad_FP32*no fdeep .tn.model files found: AI heuristic models ship only for gfx908/gfx90a/gfx942/gfx950, and each install carries only the runner target's db files. Linux gfx1151 failed first (run 39358709264, reproduced locally with the CI binaries). Since #13680 (merged 10-29) Windows installs per-target db files too, so Windows gfx110X fails as well (run 40983714656, 2 of 2 attempts on 2 runners); gfx94X/gfx90a/gfx950 keep the test.exclude_gpu_gfx115X_linuxCPU_PerfDb_NONE.*,GPU_BN_Spatial_FP32.MIOpen3900Regressionstrixhalo-12andstrixhalo-10(MIOpen3900Regressionuses ~2.1 GB per tensor on a shared-memory APU); both pass on the 31–125 GB gfx1151 hosts. Only standard-tier cases are excluded (TheRock release tests run standard on these hosts). Out-of-memory kills in comprehensive/full are not excluded case by case, see open items.Jenkins doesn't run Windows gfx110X/gfx115X or Linux gfx1151, so these quarantines need no Jenkins
add_gtest_negative_filter. gfx94X keeps fullLstmMSRnncoverage apart from/1095, which matches Jenkins's filter, and getsre-evaluated if a GitHub CI run sees it fail.
CI runs on this PR
7b72efe7b72efe7b72efe7b72efe1d54535LstmMSRnn_FP32flaky (both attempts); one runner lost (TheRock#3845)1d54535LstmMSRnn_FP32flaky (both attempts)d9701b2e964061LstmMSRnn_FP32(Win),ConvAIModelLoad(Linux),StaticFDBSync/9(Win), GRU FP16 flake (Win), OOM on strixhalo-12, hang on CS-RORDMZ-DT245, GPUunspecified launch failureand runner lost (Win)48eccaeStaticFDBSync/9(Win), OOM on strixhalo-12 (Linux), hang on CS-RORDMZ-DT245 (Linux). gfx94X, gfx90a, gfx110X green.455cc52GPU_BN_Spatial_FP32.MIOpen3900Regression(now excluded under ALMIOPEN-2964) and…/n8c192h256w512, plus one container-setup failure. No gtest failure in any lane. gfx94X, gfx90a, gfx110X and Windows gfx1151 green.7012edena_fusion_test_bn_spatial_FP32 nchw_8_192_256_512,na_fusion_inference_test_FP32 nchw_1_16_4096_4096) andStaticFDBSync/9on Linux (now excluded on all gfx115X); infra: container setup (halobox-b13),BAD_COMMANDon Windows host wshctr-halobox-b48-05 (passed on rerun). gfx94X, gfx90a, gfx110X and Windows gfx1151 green.ffcb4d494c1c2b98c7817RunHipBLASLtGEMMcases (hipblaslt_gemm.cpp:235, error 0.12–0.17 vs 0.07). Not this PR: the attempt-2 rerun rebuilt only gfx950, whose*_genericMIOpen artifacts (built with the IEEE FP8 bias) replaced gfx94X's; tracked in #14375, fix in #14627. Windows green.7d71fd8(develop merge)GPU_ConvAIModelLoad_FP32.BundledFdeepModelsLoad(2 of 2 attempts) after #13680 moved Windows dbs to per-target packages; now excluded on gfx110X/gfx115X (ALMIOPEN-2966). Linux gfx94X green, including FP8/BFP8.¹ The
test_type:comprehensive/test_type:fulllabels used then are not wired into the multi-arch workflow, so these runs ran standard.Labels used later:
test_filter:comprehensive,test_filter:full,ci:gfx90a,ci:gfx1151.Timing
All numbers are per shard (4 shards, serial ctest), as ctest wall minutes; the TheRock Test step is capped at 120 min.
Measured
The slowest shard measured is gfx90a full at 86.6 min (72% of the cap). Every other lane is at or below 73 min.
² Develop's full tier selects the same cases as develop comprehensive (within 25 cases), so the delta is measured against the develop comprehensive time.
³ Healthy shards only; the other shards were OOM-killed (ALMIOPEN-2964) or hung at startup (ALMIOPEN-2965).
⁴ Windows gfx110X shards range from 9 to 21 min depending on the runner host (CS-RORDMZ vs azure scale-set).
⁵ The develop nightly runs Windows gfx1151, not gfx110X.
⁶ Large spread between hosts: one gfx1151 runner is about 2× slower than the others.
Modelled delta vs develop (final yaml, cases added and per-shard minutes)
yaml_select.py diff --old git:<merge-base 4c5c20a> --new git:HEAD --gpu <arch> --os <os>, using this PR's per-case timings measured on each arch.The model agrees with the measurements where a develop baseline exists: gfx94X comprehensive +15.6 modelled vs +16.2 measured;
standard +1.5 vs +0.9.
Largest additions per tier (Linux gfx94X, serial minutes)
Smoke/GPU_Find2Conv0.8,Smoke/GPU_TensorReorder0.4,Smoke/GPU_reference_kernel_{bwd,bww}_2d_*0.3–0.4 eachFull/GPU_Pooling3d9.0,Full/GPU_LayerNormBwdTest7.8,Full/GPU_Pooling2d7.0,Full/GPU_BnPeract4.3,Full/GPU_TensorTransform3.7Full/GPU_na_fusion_test_bn_spatial10.8,Full/GPU_na_train_find2_spatial10.2,Full/GPU_na_*_peract5.8 eachWhere each tier runs after merge
test_filter:fulllabelNote
Standard-tier hot spot (pre-existing, not changed by this PR). In this PR's standard runs on Linux gfx94X,
Full/GPU_Softmax_*is the most expensive group: 36.5 of 109.6 serial minutes (33%), about 9 min per shard. Breakdown per run:
Full/GPU_Softmax_FP3218.5 min (1,008 cases),
_FP167.1,_BFP167.0,_NonContiguous_FP323.9. It has been in standard since #3513. On gfx11/gfx12the FP32/FP16/BFP16 variants are already excluded as slow (TheRock#3362);
Full/GPU_Softmax_NonContiguous_FP32still costs 5.1 min(8%) on Windows gfx110X.
Moving it to comprehensive (adding
Full/GPU_Softmax_*tostandard.exclude) is modelled at −9.1 min per shard on gfx942(27.7 → about 18.6) and −1.2 min per shard on Windows gfx110X. Comprehensive and full keep every case, so the nightly still covers it.
Proposed as a follow-up rather than part of this PR. Next largest in standard:
Standard/GPU_Reduce_{FP32,FP16}3.9 + 3.1 min,Full/GPU_GroupConv3D_*1.4–2.2 min per dtype.ASAN validation of the added suites
Every gtest suite this PR adds was built and run under full ASAN, both host-side and device-side, on gfx942 with XNACK on (
gfx942:xnack+). Apart from one pair of test files that leak memory, they are clean.-fsanitize=addressagainst the rockrel Multi-Arch Release ASAN nightly Bump rocm-docs-core from 1.18.2 to 1.20.0 in /projects/rocthrust/docs/sphinx #139 artifacts (run 40425965688, TheRocke73fd19), using the same test environment as TheRock CI (configure_asan_env.py,ROCM_PATH,HSA_XNACK=1). It ran on MI300X: 28,354 selected cases in 4 shards, plus a canary run of theSmoke/and unprefixed suites first.na_train_find2.cppandna_inference_find2.cppcallmiopenFindSolutionsbut nevermiopenDestroySolution, so LeakSanitizer fails the process at exit. They are excluded from ASAN builds only, usingSKIP_TESTSintest/gtest/CMakeLists.txtunderTHEROCK_SANITIZER=ASAN, next to the existingfind_2_conv.cppleak skip. This removes the 12*/GPU_na_{train,inference}_find2_*suites (528 cases) from ASAN builds; normal CI keeps them. Tracked in ALMIOPEN-3097.KernelBlitManager::initHeap) or ASAN fires inside ROCr (AqlQueue::HandleInsufficientScratch). A single process started 5 of 5 times. The run therefore used one ASAN process per node.strdupleak in ROCr'sLoadCodeObject, seen in the canary only.I didn't add an ASAN exclusion to
test_categories.yaml. Its parser has no sanitizer-specific key, so any yaml key would also drop these suites from normal CI.Tickets
LstmMSRnn_FP32(and GRU FP16) bias-only CPU/GPU mismatch: gfx110X, gfx115X, and the Jenkins gfx94X/1026,/1095strixhalo-12andstrixhalo-10OOM-killmiopen_gtest; matches theexclude_gpu_gfx115X_linuxOOM exclusions. The fix is to remove those hosts from the runner pool.miopen_gtesthangs before any output on Linux gfx1151 runners CS-RORDMZ-DT245 (3 of 3 jobs) and ctr-halobox-b13; no test starts, so there is no yaml exclusionGPU_ConvAIModelLoad_FP32.BundledFdeepModelsLoadfails instead of skipping when the system db has no*.tn.model(Linux gfx1151; Windows gfx110X/gfx1151 since #13680); repro steps on any GPU viaMIOPEN_SYSTEM_DB_PATH; matches theexclude_gpu_gfx110X/exclude_gpu_gfx115Xentriesna_train_find2/na_inference_find2tests leakmiopenSolution_t(nomiopenDestroySolution); matches the ASANSKIP_TESTSentriesKernelBlitManager::initHeapor trips ASAN in ROCrAqlQueue::HandleInsufficientScratchwhen two ASAN processes share a nodestrdupleak in ROCrExecutableImpl::LoadCodeObject(seen once)StaticFDBSync"Db file does not exist" on hosts whose CU count has no shipped db / Windows db install fixTest plan
python3 shared/ctest/parse_test_categories.py test_categories.yaml …returns rc 0 (21 suites on Linux).yaml_select.py diffafter every quarantine: each change removes only the intended cases on the intended arch/OS; Linux gfx94Xand gfx90a selections are unchanged by the RDNA quarantines.
LstmMSRnn_FP32ran 0times on gfx110X/gfx115X, gfx94X/gfx90a ran it 3,454 times each with no failures) and has no gtest failure in any lane.
Open items
4b8890fbefore merge: removed in98c7817(workflow identical to develop's at the merge-base).98c7817) and labelsci:gfx1151,ci:gfx90a,test_filter:fullremoved; the PR gate runs standard on Linux gfx94X and Windows gfx110X. Run 40746486850 (94c1c2b) passed. 40901558044 hit the FP8 artifact overwrite ([Issue] Linux gfx94X-dcgpu: MIOpen HipBLASLt FP8/BFP8 GEMM tests fail with numerical error #14375) and 40983714656 hitConvAIModelLoadon Windows gfx110X after fix(miopen): install Windows system databases under share/miopen/db #13680; that test is now excluded on gfx110X/gfx115X. Needs one green run with that change.na_*find2suites under ASAN once ALMIOPEN-3097 is fixed (remove the twoSKIP_TESTSentries).