From 6dbc1291e99923451ff7368c6cda606814150caa Mon Sep 17 00:00:00 2001 From: Tony Davis Date: Thu, 15 Jan 2026 17:18:17 -0500 Subject: [PATCH 1/2] Add optional enhanced metadata fields to test categories - Add documentation for optional fields (notes, source_coverage, api_coverage, feature_tags, dependencies) - Add clarifying comment to parser (no logic changes - YAML naturally ignores unknown fields) - Enhance level1 category as example, keep level2-level6 minimal to show backward compatibility All fields are optional and backward compatible. Teams can adopt incrementally. Enables richer test documentation and future AI-assisted test selection. --- shared/ctest/README.md | 62 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/shared/ctest/README.md b/shared/ctest/README.md index 83a3a5291af9..d94e64c6ac5f 100644 --- a/shared/ctest/README.md +++ b/shared/ctest/README.md @@ -126,6 +126,68 @@ execution_settings: - `category_timeouts`: Timeouts for specific categories (before multiplier is applied) ``` +### **Enhanced Structure (Optional Fields)** + +All fields below are **optional** and can be added incrementally. Teams can use them for richer test documentation and enable future capabilities like AI-assisted test selection: + +```yaml +test_categories: + category_name: + # Required fields (same as base) + description: "Human-readable description" + test_patterns: ["*pattern1*", "*pattern2*"] + labels: ["label1", "label2"] + + # Optional enhancement fields - add only if useful for your project + notes: | + Human-readable context about when to run these tests. + Can include historical context, gotchas, or guidance for developers and AI tools. + source_coverage: + - "library/src/file.cpp" + - "library/src/module.cpp:function_name" + api_coverage: + - "apiFunction1" + - "apiFunction2" + feature_tags: + - "performance-critical" + - "numerical-stability" + dependencies: + - "other_category" # Run this category after dependencies complete + + # Standard fields (from base) + exclude: ["*always_exclude*"] + exclude_windows: ["*linux_only*"] + exclude_linux: ["*windows_only*"] + +# Optional: Top-level context for AI/LLM tools +llm_context: + code_to_test_mapping_guidelines: | + Guidance for AI tools on how to map code changes to test categories. + Projects can use this for AI-assisted test selection. + +execution_settings: + default_timeout: 300 + category_timeouts: + category_name: 600 +``` + +**Optional Field Descriptions:** + +| Field | Purpose | Example Use | +|-------|---------|-------------| +| `notes` | Free-form text for context and documentation | "Run when epilogue changes. See bug #8765 for history" | +| `source_coverage` | Source files/functions tested by this category | `["library/src/gemm.cpp:matmul_kernel"]` | +| `api_coverage` | API functions tested by this category | `["hipblasLtMatmul", "hipblasLtMatmulAlgo"]` | +| `feature_tags` | Semantic tags for classification and filtering | `["performance-critical", "mixed-precision"]` | +| `dependencies` | Test categories that should run first | `["auxiliary"]` - run auxiliary tests before this category | +| `llm_context` | Top-level guidance for AI-assisted workflows | Instructions for AI tools on test selection logic | + +**Key Points:** +- All enhancement fields are **optional** - teams can ignore them entirely ✅ +- Projects can adopt incrementally: start with just `notes`, add more later ✅ +- Parser gracefully ignores unknown fields - no code changes needed ✅ +- Enables richer test documentation and future AI-assisted workflows ✅ + ### **GPU Exclusion with Hierarchical Matching** GPU-specific exclusions use hierarchical pattern matching with wildcard 'X': From dabf7693368e24f5243040aa08c0581507de1763 Mon Sep 17 00:00:00 2001 From: Tony Davis Date: Tue, 27 Jan 2026 15:02:48 -0500 Subject: [PATCH 2/2] Clarify that optional fields are for documentation/tooling only, not enforced by parser --- shared/ctest/README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/shared/ctest/README.md b/shared/ctest/README.md index d94e64c6ac5f..8da0338a9ffd 100644 --- a/shared/ctest/README.md +++ b/shared/ctest/README.md @@ -152,7 +152,7 @@ test_categories: - "performance-critical" - "numerical-stability" dependencies: - - "other_category" # Run this category after dependencies complete + - "other_category" # Metadata for related categories (not enforced by parser; for documentation/tooling) # Standard fields (from base) exclude: ["*always_exclude*"] @@ -179,12 +179,13 @@ execution_settings: | `source_coverage` | Source files/functions tested by this category | `["library/src/gemm.cpp:matmul_kernel"]` | | `api_coverage` | API functions tested by this category | `["hipblasLtMatmul", "hipblasLtMatmulAlgo"]` | | `feature_tags` | Semantic tags for classification and filtering | `["performance-critical", "mixed-precision"]` | -| `dependencies` | Test categories that should run first | `["auxiliary"]` - run auxiliary tests before this category | +| `dependencies` | Related test categories (documentation only) | `["auxiliary"]` - advisory metadata for downstream tooling | | `llm_context` | Top-level guidance for AI-assisted workflows | Instructions for AI tools on test selection logic | **Key Points:** - All enhancement fields are **optional** - teams can ignore them entirely ✅ - Projects can adopt incrementally: start with just `notes`, add more later ✅ +- **Parser does NOT process these fields** - they are for documentation and downstream tooling only ✅ - Parser gracefully ignores unknown fields - no code changes needed ✅ - Enables richer test documentation and future AI-assisted workflows ✅