AIRUNTIME-2744: Require reason on populated test skip sections

projects/hip-tests/TESTING.md

@@ -81,7 +81,7 @@ When you change public HIP API behavior, run the contract tests for the touched
 
 A flaky or quarantined case is treated as unstable signal: it does not block merge, but it must have an owner, a tracking issue, and a removal or fix plan before it can be promoted back to trusted signal. Use `HIP_SKIP_TEST` for a genuine capability gap such as an unsupported device or runtime path, too few GPUs, or no image support, so the case skips cleanly instead of failing. Never leave a test that crashes the process. Any test that intentionally triggers a HIP error must consume the sticky thread-local error with `(void)hipGetLastError()` before returning so it does not leak into later cases in the same process.
 
-Disabling a test case should be rare and explicit. It is appropriate when a test cannot execute on a platform or architecture because a required capability, OS facility, graphics environment, or runtime mode is absent; when a known product or infrastructure defect has a tracking issue; or when keeping the test enabled would crash the process and hide later signal. Prefer runtime capability checks plus `HIP_SKIP_TEST` for environment-dependent behavior, and use the YAML skip fields only when a case should be hidden for a specific platform, architecture, sanitizer mode, or tracked known issue. The `disabled:` and `unsupported:` fields share the same syntax and produce the same skip; they differ only in intent. Do not disable a test only to make CI green. Every skipped case should record why it is skipped, where it is tracked when a `disabled:` entry points at a bug.
+Disabling a test case should be rare and explicit. It is appropriate when a test cannot execute on a platform or architecture because a required capability, OS facility, graphics environment, or runtime mode is absent; when a known product or infrastructure defect has a tracking issue; or when keeping the test enabled would crash the process and hide later signal. Prefer runtime capability checks plus `HIP_SKIP_TEST` for environment-dependent behavior, and use the YAML skip fields only when a case should be hidden for a specific platform, architecture, sanitizer mode, or tracked known issue. The `disabled:` and `unsupported:` fields share the same syntax and produce the same skip; they differ only in intent: `disabled:` is a temporary skip for a regression or tracked bug that is expected to be re-enabled, while `unsupported:` is a permanent skip where the platform, architecture, or backend cannot support the case. Do not disable a test only to make CI green. Every skipped case must record why it is skipped, and where it is tracked when the entry points at a bug, in the `reason:` field of the mapping form (`{targets: [...], reason: "..."}`). This is enforced in CI by `check_config.py`: any skip section with a non-empty `targets` list must carry a non-empty `reason`, and that reason must cite a real Jira reference from a known project key (AIRUNTIME, ROCM, SWDEV, EXSWHTEC, AIRADSW), e.g. `AIRUNTIME-2892`. When a skip has not yet been triaged, lead the reason with the tracking ticket, e.g. `AIRUNTIME-2744 - needs to be re-triaged`.
 
 ## Coverage
 

projects/hip-tests/catch/README.md

@@ -72,6 +72,15 @@ Every test case has its own entry. Currently supported options are:
 - disabled : Temporarily skip the case on the listed targets (a regression or tracked bug that is expected to be re-enabled)
 - unsupported : Permanently skip the case on the listed targets, where the platform, architecture, or backend cannot support it
 `disabled` and `unsupported` share the same syntax and produce the same skip; they differ only in intent (temporary regression versus permanent capability gap).
+Each takes the mapping form `{targets: [...], reason: "..."}`. A section with a
+non-empty `targets` list **must** carry a non-empty `reason`, and that reason **must**
+cite a real Jira reference from a known project key (**AIRUNTIME, ROCM, SWDEV, EXSWHTEC,
+AIRADSW**), e.g. `AIRUNTIME-2892`; both are enforced by `check_config.py` in CI. The known
+project keys are an allowlist; add a new one by editing the `JIRA_PROJECTS` tuple in
+`check_config.py`. The bare flat-list form (`disabled: [amd_wsl]`) is no longer accepted for a
+non-empty skip. An empty section (`targets: []`) is a no-op and needs no reason. If a
+skip has not yet been triaged, lead the reason with the tracking ticket, e.g.
+`AIRUNTIME-2744 - needs to be re-triaged`.
 The group name is automatically added as a tag for every case.
 Changing the configuration file will retrigger the build, so we have an up to date configuration every time.
 

projects/hip-tests/catch/config/check_config.py

@@ -3,11 +3,18 @@
 # SPDX-License-Identifier: MIT
 
 import argparse
+import re
 import sys
 
 from common import iter_group_configs, ERROR, RESET
 
 
+# Known Jira project keys accepted in a skip 'reason'. Add a new key here when
+# a new project starts tracking hip-tests skips.
+JIRA_PROJECTS = ("AIRUNTIME", "ROCM", "SWDEV", "EXSWHTEC", "AIRADSW")
+JIRA_RE = re.compile(r"(?:" + "|".join(JIRA_PROJECTS) + r")-\d+")
+
+
 def parse_args():
     parser = argparse.ArgumentParser(
         description="Check that every test case in the YAML configs has a "
@@ -27,6 +34,8 @@ def main():
 
     missing = []
     invalid_skip_fields = []
+    missing_reasons = []
+    missing_jira = []
 
     for group, cases in iter_group_configs(configs_path):
         for case_name, case_config in cases.items():
@@ -37,18 +46,31 @@ def main():
                     continue
                 value = case_config[field]
                 # A skip field is either a flat list of targets, or a mapping
-                # with a 'targets' list plus an optional 'reason' scalar. The
-                # reason is metadata and is intentionally NOT enforced yet —
-                # deferred to a later phase (AIRUNTIME-2744):
-                # https://amd-hub.atlassian.net/browse/AIRUNTIME-2744
-                # We only guard that the targets are a list, since a scalar
-                # breaks tag generation (a string is iterated char-by-char) and
-                # a malformed mapping fails the parser's list concatenation.
+                # with a 'targets' list plus a 'reason' scalar. We guard that the
+                # targets are a list, since a scalar breaks tag generation (a
+                # string is iterated char-by-char) and a malformed mapping fails
+                # the parser's list concatenation.
                 if isinstance(value, list):
+                    targets = value
+                    reason = ""
+                elif isinstance(value, dict) and isinstance(value.get("targets"), list):
+                    targets = value["targets"]
+                    reason = value.get("reason", "")
+                else:
+                    invalid_skip_fields.append(f"  {group}/{case_name}: '{field}'")
                     continue
-                if isinstance(value, dict) and isinstance(value.get("targets"), list):
-                    continue
-                invalid_skip_fields.append(f"  {group}/{case_name}: '{field}'")
+                # A populated skip section (non-empty targets) must record a
+                # non-empty string 'reason'; a non-scalar (list/mapping) or
+                # non-string reason is treated as missing. An empty section
+                # (targets: []) is a no-op and needs no reason.
+                if targets and not (isinstance(reason, str) and reason.strip()):
+                    missing_reasons.append(f"  {group}/{case_name}: '{field}'")
+                # A populated skip section with a real reason must cite a Jira
+                # reference (e.g. AIRUNTIME-2892 or ROCM-123, from a known
+                # project key); a reason with no such ref (or a digitless
+                # placeholder) does not count as a real ref.
+                elif targets and not JIRA_RE.search(reason):
+                    missing_jira.append(f"  {group}/{case_name}: '{field}'")
 
     if missing:
         print(
@@ -68,6 +90,24 @@ def main():
             print(f"[check_config] {ERROR}{entry}{RESET}", file=sys.stderr)
         sys.exit(1)
 
+    if missing_reasons:
+        print(
+            f"[check_config] {ERROR}ERROR: The following test cases have a 'disabled'/'unsupported' field with targets but no 'reason'. Use the mapping form ({{targets: [...], reason: \"...\"}}) and record why the case is skipped:{RESET}",
+            file=sys.stderr,
+        )
+        for entry in missing_reasons:
+            print(f"[check_config] {ERROR}{entry}{RESET}", file=sys.stderr)
+        sys.exit(1)
+
+    if missing_jira:
+        print(
+            f"[check_config] {ERROR}ERROR: The following test cases have a 'disabled'/'unsupported' 'reason' with no real Jira reference. The reason must cite a real Jira reference from a known project ({', '.join(JIRA_PROJECTS)}), e.g. AIRUNTIME-2892; a digitless placeholder like AIRUNTIME-XXXX does not count:{RESET}",
+            file=sys.stderr,
+        )
+        for entry in missing_jira:
+            print(f"[check_config] {ERROR}{entry}{RESET}", file=sys.stderr)
+        sys.exit(1)
+
 
 if __name__ == "__main__":
     main()

projects/hip-tests/catch/config/configs/TypeQualifiers.yaml

@@ -3,7 +3,8 @@ TypeQualifiers:
   Unit_hipManagedKeyword_SingleGpu: *level_default
   Unit_hipManagedKeyword_MultiGpu:
     <<: *level_default
-    # SWDEV-396615 mGPUs not considered correctly
-    disabled: [amd_windows, amd_wsl]
+    disabled:
+      targets: [amd_windows, amd_wsl]
+      reason: "AIRUNTIME-2744 - SWDEV-396615: multi-GPU case not considered correctly. gfx1201 (amd_windows) self-skips: GPU does not support managed memory. gfx1201 (amd_wsl) self-skips: GPU does not support managed memory. Needs re-triage."
   Unit_hipManagedKeyword_hipPointerGetAttributes_BeforeKernel: *level_default
   Unit_hipManagedKeyword_hipPointerGetAttributes_AfterKernel: *level_default

projects/hip-tests/catch/config/configs/multiproc.yaml

@@ -9,8 +9,9 @@ multiproc:
   Unit_hipDeviceGetPCIBusId_MaskedDevices: *level_default
   Unit_hipDeviceGetPCIBusId_CheckPciBusIDWithLspci:
     <<: *level_default
-    # Note: no valid pci bdf in wsl
-    disabled: [amd_wsl]
+    disabled:
+      targets: [amd_wsl]
+      reason: "AIRUNTIME-2892 - needs to be further investigated; no valid PCI BDF in WSL. gfx1201 (amd_wsl) FAIL: deviceMatchCount == deviceCount -> 0 == 1."
   Unit_hipDeviceTotalMem_MaskedDevices: *level_default
   Unit_hipDeviceGetAttribute_MaskedDevices: *level_default
   Unit_hipGetDeviceCount_MaskedDevices: *level_default
@@ -19,12 +20,8 @@ multiproc:
   Unit_hipIpcEventHandle_ParameterValidation: *level_default
   Unit_hipIpcMemAccess_Semaphores:
     <<: *level_default
-    # Below tests soft hang in stress test on 13/09/23
-    disabled: [amd_wsl]
   Unit_hipIpcMemAccess_ParameterValidation:
     <<: *level_default
-    # Below tests fail in stress test on 23/06/23
-    disabled: [amd_wsl]
   Unit_hipMalloc_ChildConcurrencyDefaultGpu: *level_default
   Unit_hipMalloc_ChildConcurrencyMultiGpu: *level_default
   Unit_malloc_CoherentTst: *level_default
@@ -35,23 +32,20 @@ multiproc:
   Unit_hipHostMalloc_WthEnv0Flg2: *level_default
   Unit_hipHostMalloc_WthEnv0Flg3:
     <<: *level_default
-    # Note: CONFIG_NUMA disabled
-    disabled: [amd_wsl]
   Unit_hipHostMalloc_WthEnv0Flg4: *level_default
   Unit_hipHostMalloc_WthEnv1: *level_default
   Unit_hipHostMalloc_WthEnv1Flg1: *level_default
   Unit_hipHostMalloc_WthEnv1Flg2: *level_default
   Unit_hipHostMalloc_WthEnv1Flg3:
     <<: *level_default
-    # Note: CONFIG_NUMA disabled
-    disabled: [amd_wsl]
   Unit_hipMemGetInfo_Functional_Scenario1: *level_default
   Unit_hipMemGetInfo_Functional_Scenario2: *level_default
   Unit_hipMemGetInfo_Functional_Scenario3: *level_default
   Unit_hipMemGetInfo_Functional_scenario4:
     <<: *level_default
-    # Test does not play well with Asan.
-    disabled: [asan]
+    disabled:
+      targets: [asan]
+      reason: "AIRUNTIME-2744 - test does not play well with ASAN. Needs re-triage."
   Unit_hipMemGetInfo_Functional_MultiDevice_Scenario5: *level_default
   Unit_hipMemGetInfo_SetHiddenFreeMemFromChild: *level_default
   Unit_hipMemGetInfo_VerifyHiddenFreeMemForAllGpu: *level_default

projects/hip-tests/catch/config/configs/performance.yaml

@@ -14,8 +14,9 @@ performance:
   Performance_hipExtLaunchKernel: *level_stress
   Performance_hipExtLaunchKernelGGL_QueryGPUFrequency:
     <<: *level_stress
-    # Following tests disabled as it should be a local perf test
-    disabled: [amd_linux, amd_windows, amd_wsl]
+    disabled:
+      targets: [amd_linux, amd_windows, amd_wsl]
+      reason: "AIRUNTIME-2744 - disabled because this should be run as a local perf test; needs to be re-triaged."
   Performance_hipMemcpy_DeviceToHost: *level_stress
   Performance_hipMemcpy_HostToDevice: *level_stress
   Performance_hipMemcpy_HostToHost: *level_stress
@@ -69,16 +70,18 @@ performance:
   Performance_hipMemcpy2D_HostToDevice: *level_stress
   Performance_hipMemcpy2D_HostToHost:
     <<: *level_stress
-    # Below tests are failing PSDB
-    disabled: [amd_windows]
+    disabled:
+      targets: [amd_windows]
+      reason: "AIRUNTIME-2744 - failing in PSDB; needs to be re-triaged."
   Performance_hipMemcpy2D_DeviceToDevice_DisablePeerAccess: *level_stress
   Performance_hipMemcpy2D_DeviceToDevice_EnablePeerAccess: *level_stress
   Performance_hipMemcpy2DAsync_DeviceToHost: *level_stress
   Performance_hipMemcpy2DAsync_HostToDevice: *level_stress
   Performance_hipMemcpy2DAsync_HostToHost:
     <<: *level_stress
-    # Below tests are failing PSDB
-    disabled: [amd_windows]
+    disabled:
+      targets: [amd_windows]
+      reason: "AIRUNTIME-2744 - failing in PSDB; needs to be re-triaged."
   Performance_hipMemcpy2DAsync_DeviceToDevice_DisablePeerAccess: *level_stress
   Performance_hipMemcpy2DAsync_DeviceToDevice_EnablePeerAccess: *level_stress
   Performance_hipMemcpy2DToArray_HostToDevice:
@@ -143,20 +146,24 @@ performance:
   Performance_hipMemsetD8Async: *level_stress
   Performance_hipMemsetD16:
     <<: *level_stress
-    # Below tests tests fail in PSDB
-    disabled: [nvidia_windows]
+    disabled:
+      targets: [nvidia_windows]
+      reason: "AIRUNTIME-2744 - fails in PSDB; needs to be re-triaged."
   Performance_hipMemsetD16Async:
     <<: *level_stress
-    # Below tests tests fail in PSDB
-    disabled: [nvidia_windows]
+    disabled:
+      targets: [nvidia_windows]
+      reason: "AIRUNTIME-2744 - fails in PSDB; needs to be re-triaged."
   Performance_hipMemsetD32:
     <<: *level_stress
-    # Below tests tests fail in PSDB
-    disabled: [nvidia_windows]
+    disabled:
+      targets: [nvidia_windows]
+      reason: "AIRUNTIME-2744 - fails in PSDB; needs to be re-triaged."
   Performance_hipMemsetD32Async:
     <<: *level_stress
-    # Below tests tests fail in PSDB
-    disabled: [nvidia_windows]
+    disabled:
+      targets: [nvidia_windows]
+      reason: "AIRUNTIME-2744 - fails in PSDB; needs to be re-triaged."
   Performance_hipMemset2D: *level_stress
   Performance_hipMemset2DAsync: *level_stress
   Performance_hipMemset3D: *level_stress

projects/hip-tests/catch/config/configs/unit/assertion.yaml

@@ -2,15 +2,16 @@
 assertion:
   Unit_StaticAssert_Positive_Basic_RTC:
     <<: *level_default
-    # Below tests fail in external CI for PR https://github.com/ROCm-Developer-Tools/hip-tests/pull/210
-    disabled: [amd_windows, amd_wsl]
+    disabled:
+      targets: [amd_windows]
+      reason: "AIRUNTIME-2892 - needs to be further investigated. Fails in external CI for hip-tests PR 210. gfx1201 (amd_windows) FAIL: error_count == expected_error_count -> 3 == 2."
     tags: [deviceLib]
   Unit_StaticAssert_Negative_Basic_RTC:
     <<: *level_default
     tags: [deviceLib]
   Unit_Assert_Positive_Basic_KernelPass: *level_default
   Unit_Assert_Positive_Basic_KernelFail:
     <<: *level_default
-    # === Below tests fail in external CI for PR https://github.com/ROCm-Developer-Tools/hip-tests/pull/210 ===
-    # Disabled for asan since this test intentionally enter a failed state which prevents freeing resources
-    disabled: [amd_windows, amd_wsl, asan]
+    disabled:
+      targets: [amd_windows, amd_wsl, asan]
+      reason: "AIRUNTIME-2744 - fails in external CI for hip-tests PR 210. Disabled for asan because this test intentionally enters a failed state, which prevents freeing resources. gfx1201 (amd_windows) self-skips: assertions are disabled in this build. gfx1201 (amd_wsl) self-skips: assertions are disabled in this build. Needs re-triage on asan."

projects/hip-tests/catch/config/configs/unit/atomics.yaml

@@ -233,9 +233,10 @@ atomics:
     tags: [deviceLib, ops]
   Unit_atomicAdd_system_Positive_Peer_GPUs:
     <<: *level_default
+    disabled:
+      targets: [amd_linux, amd_wsl]
+      reason: "AIRUNTIME-2744 - requires more than 1 GPU (peer); self-skips on single-device gfx1030 so not locally verifiable, and is in the Rock_Linux_Failures_on_gfx94X set. gfx1201 (amd_wsl) self-skips: required number of devices is not available. Needs re-triage on amd_linux."
     tags: [deviceLib, system, multigpu]
-    # Rock_Linux_Failures_on_gfx94X
-    disabled: [amd_linux, amd_wsl]
   Unit_atomicAdd_system_Positive_Host_And_GPU:
     <<: *level_default
     tags: [deviceLib, system]
@@ -295,35 +296,37 @@ atomics:
     tags: [deviceLib, system, multigpu]
   Unit_atomicExch_Positive_Same_Address_Compile_Time:
     <<: *level_default
-    # SWDEV-435667: Below tests failing randomly in stress test on 01/12/23
-    disabled: [amd_wsl, nvidia_linux]
+    disabled:
+      targets: [nvidia_linux]
+      reason: "AIRUNTIME-2744 - SWDEV-435667: fails randomly in stress test (observed 01/12/23). Needs re-triage on nvidia_linux."
     tags: [deviceLib, ops]
   Unit_atomicExch_Positive:
     <<: *level_default
     tags: [deviceLib, ops]
   Unit_atomicExch_Positive_Multi_Kernel:
     <<: *level_default
-    # SWDEV-435667: Below tests failing randomly in stress test on 01/12/23
-    disabled: [amd_wsl]
     tags: [deviceLib, ops]
   Unit_atomicExch_Negative_Parameters_RTC:
     <<: *level_default
     tags: [deviceLib, ops]
   Unit_atomicExch_system_Positive_Peer_GPUs:
     <<: *level_default
+    disabled:
+      targets: [amd_wsl]
+      reason: "AIRUNTIME-2744 - SWDEV-435667: fails randomly in stress test (observed 01/12/23). gfx1201 (amd_wsl) self-skips: required number of devices is not available. Needs re-triage."
     tags: [deviceLib, system, multigpu]
-    # SWDEV-435667: Below tests failing randomly in stress test on 01/12/23
-    disabled: [amd_wsl]
   Unit_atomicExch_system_Positive_Host_And_GPU:
     <<: *level_default
+    disabled:
+      targets: [nvidia_linux]
+      reason: "AIRUNTIME-2744 - SWDEV-435667: fails randomly in stress test (observed 01/12/23). Needs re-triage on nvidia_linux."
     tags: [deviceLib, system]
-    # SWDEV-435667: Below tests failing randomly in stress test on 01/12/23
-    disabled: [amd_wsl, nvidia_linux]
   Unit_atomicExch_system_Positive_Host_And_Peer_GPUs:
     <<: *level_default
+    disabled:
+      targets: [amd_wsl]
+      reason: "AIRUNTIME-2744 - SWDEV-435667: fails randomly in stress test (observed 01/12/23). gfx1201 (amd_wsl) self-skips: required number of devices is not available. Needs re-triage."
     tags: [deviceLib, system, multigpu]
-    # SWDEV-435667: Below tests failing randomly in stress test on 01/12/23
-    disabled: [amd_wsl]
   Unit_atomicExch_system_Negative_Parameters_RTC:
     <<: *level_default
     tags: [deviceLib, system]