From e3d35112441df871223303d2587c3d9920856827 Mon Sep 17 00:00:00 2001 From: Adam Dickin Date: Mon, 19 Oct 2026 10:29:06 -0600 Subject: [PATCH 1/5] docs(hipdnn): RFC 0021 per-op test plans (ALMIOPEN-2895) Adds RFC 0021: a per-op test plan methodology for the integration suite. It covers feature inventory, shape catalog and size ladder, tier rules, case purpose, trade-off register, oracle prerequisites, golden selection, and the test generator and fuzzing tool. JIRA ID : ALMIOPEN-2895 --- .../hipdnn/docs/rfcs/0021_PerOpTestPlans.md | 579 ++++++++++++++++++ 1 file changed, 579 insertions(+) create mode 100644 projects/hipdnn/docs/rfcs/0021_PerOpTestPlans.md diff --git a/projects/hipdnn/docs/rfcs/0021_PerOpTestPlans.md b/projects/hipdnn/docs/rfcs/0021_PerOpTestPlans.md new file mode 100644 index 000000000000..97b8d4a0d842 --- /dev/null +++ b/projects/hipdnn/docs/rfcs/0021_PerOpTestPlans.md @@ -0,0 +1,579 @@ +# RFC 0021: Per-Op Test Plans + +- Contributors: Adam Dickin + +## Table of Contents + +1. [Executive Summary](#1-executive-summary) +2. [Problem Statement](#2-problem-statement) +3. [Scope](#3-scope) +4. [Design Overview](#4-design-overview) +5. [Principles](#5-principles) +6. [Feature Inventory](#6-feature-inventory) +7. [Shape Catalog and Size Ladder](#7-shape-catalog-and-size-ladder) +8. [Sweep Construction and Tiers](#8-sweep-construction-and-tiers) +9. [Case Purpose](#9-case-purpose) +10. [Trade-off Register](#10-trade-off-register) +11. [Oracles and Reference Prerequisites](#11-oracles-and-reference-prerequisites) +12. [Golden Tests](#12-golden-tests) +13. [Authoring Process and Plan Template](#13-authoring-process-and-plan-template) +14. [Test Generator and Fuzzing Tool](#14-test-generator-and-fuzzing-tool) +15. [Required Infrastructure](#15-required-infrastructure) +16. [Rollout](#16-rollout) +17. [Alternatives Considered](#17-alternatives-considered) +18. [Risks](#18-risks) +19. [Open Questions and Future Work](#19-open-questions-and-future-work) +20. [Glossary](#20-glossary) + +## 1. Executive Summary + +The integration suite (`dnn-providers/integration-tests`) tests each op with +template-sweep bundles ([RFC 0011](0011_GoldenReferenceValidation.md)). It +records engine support as claims ([RFC 0015](0015_EngineSupportClaims.md)). It +has no rule for **which cases a bundle holds**. Today, bundles hold cases +captured from older C++ tests, then re-tiered. No case has a stated reason. +Nothing records what we left out. + +This RFC adds a **per-op test plan**: one reviewed document for each op family, +in `dnn-providers/integration-tests/docs/test-plans/`. All plans follow one +method: + +- Derive features from the graph API. Do not use engines or old tests. +- Take shapes from size sets (small, medium, model, stress) and hazard + breakpoints. Each tier has a size ceiling. +- Combine features one at a time, then all-pairs, then in targeted products. +- Give each case a purpose: `claim`, `probe` or `regression`. +- Record each product that we do not run in a trade-off register. +- Select a small golden set with fixed criteria. An independent third party + generates it. +- Write the bundles from the plan with one **test generator tool**. The same + tool makes tensor values and fuzzes (§14). + +The first plan, convolution forward +([`convolution-fwd.md`](../../../../dnn-providers/integration-tests/docs/test-plans/convolution-fwd.md), +ALMIOPEN-2895), is the worked example. Its audit motivates most of this RFC. + +## 2. Problem Statement + +The input space of an op is unbounded. Integer parameters have no limit, differ +for each dimension and multiply when combined. Tensor values are floats. Each +test set is a sample of that space. Today nobody chooses the sample. It is +inherited. + +An audit of the convolution-forward bundles on `develop` (`562a252cbab`) found: + +| Finding | Evidence | Consequence | +|---|---|---| +| **A degenerate default hid a dimension** | All 1,218 cases have output channels equal to groups (K = G). A dense conv then computes **one output channel**. The C++ case table (`src/common/ConvolutionCommon.hpp`) has the same pattern. | Channel and group indexing was never exercised. Only deriving K and G from the tensor dims showed it. | +| **Case count overstated coverage** | 1,218 cases cover 23 distinct geometries. 216 are skip-only (fused leaky ReLU, which every engine rejects). | The count suggests broad coverage. | +| **Toy sizes only** | The largest case is ~1 MFLOP and ~1 MB: spatial ≤ 16, channels ≤ 32, filters 1 or 3. | Engines choose solvers by size. Production paths are never selected. | +| **Engine-shaped coverage** | Every case is cross-correlation, symmetric padding and ReLU-family fusion. MIOpen accepts exactly these. | `CONVOLUTION` mode, asymmetric padding and non-ReLU fusions are untested. | +| **An oracle can be silently wrong** | The GPU conv reference ignores `conv_mode`. `auto` tries it before the CPU reference. | A correct `CONVOLUTION` result would fail. | +| **Re-tiering left gaps** | Nightly holds 2 and 1 cases for the two fused conv bundles. Full holds 444 each. | Nightly adds almost nothing for fused conv. | +| **Name-keyed config drifted** | The TOML skip `*ConvFwdBiasActiv*` matches C++ test names, never bundle names. Fused 3D fp32 has no tolerance override. | Skips and overrides stop applying with no error. | +| **No golden data** | No conv case has golden data. | Nothing independent checks the references. | + +These problems are not specific to convolution. Without a plan, the bundles of +each op depend on the tests that the op happened to have. + +## 3. Scope + +### 3.1 In scope + +- The **method** of every per-op plan (§5–§13). +- The **plan document**: location, template, and the rule that bundles and plan + stay in sync. +- **Golden selection** (§12), building on RFC 0011. +- The **test generator and fuzzing tool** (§14). +- The **infrastructure** that the method needs (§15) and its rollout (§16). + +### 3.2 Out of scope + +| Item | Where it belongs | +|---|---| +| Gating a new engine integration: per-case outcome reporting, running every kernel of an engine, fuzzing campaigns (engines, duration, schedule), the days-long gauntlet tier, review of agent-authored integrations | A separate engine-integration-gate proposal. It uses the plans' graphs and axes and the tool of §14. It returns bug reproducers as pinned regressions (§9). | +| Bundle format, verification modes, DVC layout | RFC 0011 | +| Claim schema, enforcement, harvesting | RFC 0015 | +| Fuzzing of the descriptor loader and matcher | RFC 0017, RFC 0018 | +| Kernel-internal testing | The project that owns the kernel | +| Release-train testing | Release readiness | + +## 4. Design Overview + +```mermaid +flowchart LR + API[Graph API
schemas + frontend validation] --> PLAN[Per-op test plan
docs/test-plans/<op>.md] + AUDIT[Audit of existing bundles] --> PLAN + PLAN --> GEN[Test generator tool
plan mode] + GEN --> B[Bundles per tier
sweep.json, support.json] + PLAN --> GOLD[Golden generator
third party] + GOLD --> B + B --> CI[CI tiers
quick / PR / nightly / full] + CI -- claims harvest --> B + PLAN --> FUZZ[Test generator tool
fuzz mode] + FUZZ -- minimized reproducers --> PLAN + GATE[Engine integration gate] -. campaigns .-> FUZZ +``` + +- **One plan for each op family**, for example `convolution-fwd.md`, `matmul.md` + or `sdpa-fwd.md`, in `dnn-providers/integration-tests/docs/test-plans/`. An + index (`README.md`) lists the plans and their bundles. +- **The plan is the source of truth.** The generator reads the declared axes, + legality rules, shape catalog and tier rules, and writes the bundles. Nobody + edits bundles by hand. +- **Traceability in both directions.** Each bundle case maps to a plan item. + Each plan item maps to at least one case. A PR that adds, moves or deletes + cases updates the plan in the same PR. +- **Working analysis stays out of the repo.** The audit, the gap list and the + work order are tracking material. The plan states the target. + +## 5. Principles + +| # | Principle | Practice | +|---|---|---| +| P1 | **Derive features from the API.** | Each field of the op schema (`flatbuffers_sdk/schemas/*.fbs`) is a feature. So is each rule in `pre_validate_node` and `infer_properties_node` of the frontend node. Tests and engine capabilities do not define the op. | +| P2 | **The plan does not depend on an engine.** | Engines join through claims. A feature that no engine supports stays as a probe. When support arrives, the run reports `UNCLAIMED_SUPPORT` (RFC 0015). The plan needs no edit. | +| P3 | **Divide each axis into classes. Then add boundaries and hazards.** | Classes: 1, odd, small, power of two, large, not a multiple of the vector width. Boundaries: output size 1, padding ≥ filter, remainders, values near the dtype limit. Hazards: powers of 2 ± 1, dims not divisible by 4 or 8, int32 index overflow. | +| P4 | **One feature at a time. Then all-pairs. Then targeted products.** | One-feature shapes show the cause of a failure. All-pairs finds most interaction faults. Run a complete product only where an interaction is plausible. The corpus and fuzz mode (§14) find three-way faults. Pin them. | +| P5 | **Size grows with the tier.** | Nightly tests each feature again at medium size. | +| P6 | **Each product that we do not run is a recorded decision.** | Use the trade-off register (§10). Never trim rejection probes. | +| P7 | **A case needs an oracle that models it.** | See §11. A case with no oracle is "not run", never green. | +| P8 | **Count distinct behaviors, not cases.** | Report distinct geometries, feature values and skip-only cases. | +| P9 | **Each case has a purpose.** | `claim`, `probe` or `regression` (§9). | +| P10 | **Covered means executed.** | A case that every engine skipped proves nothing. Quick needs one executed case for each feature × engine. | +| P11 | **Realism and coverage have different sources.** | Model layers and the benchmark corpus show what users run. Generated cases, hazards and probes reach what users have not run. | + +## 6. Feature Inventory + +A plan lists every feature of the op, in these groups: + +| Group | Features | +|---|---| +| Tensor | For each tensor: dtype, layout, packed or strided, **kind** (device tensor, compile-time constant, or runtime pass-by-value scalar per [RFC 0016](0016_RuntimePassByValueTensors.md)), **declared alignment and base offset**, 64-bit indexing | +| Op attributes | Each field and enum, including values that no engine supports | +| Geometry | Rank, each dimension, op parameters, output-shape rules, inferred or explicit outputs | +| Data types | Mixed precision, compute and intermediate types, graph-level or per-node types | +| Graph options | Override-shape ([RFC 0008](0008_OverridableTensorShapesDesign.md)), fused epilogues and chains, virtual or stored intermediates | +| Awkward combinations | Legal but hazardous graphs: inputs of different dtypes, tensors in different layouts, degenerate ratios | +| Values | The value domains of §14.3. Accumulation length. Determinism. | +| Execution | Workspace limits, plan reuse with new pointers, serialization round trip ([RFC 0009](0009_CompiledPlanSerialization.md)) | +| Invalid inputs | One frontend **unit test** for each validation rule. These are not sweep cases. | + +For each feature, the plan records the values to cover, the earliest tier, the +oracle readiness (§11) and the purpose of its cases (§9). + +**Legality is declared once for each op.** It copies the frontend validation. +The generator never writes an illegal graph. + +**Alignment is tested by placement.** The harness passes allocator pointers, +which are aligned to at least 256 bytes. A case that only *declares* a small +alignment tests nothing. An alignment case puts the base pointer at exactly the +declared alignment (N, not 2N), with a non-zero offset. It crosses this with a +leading stride that is not a multiple of the vector width. One case uses a +pointer *less* aligned than declared and expects `BAD_PARAM`. + +## 7. Shape Catalog and Size Ladder + +| Set | Purpose | Rule | +|---|---|---| +| **S** small | The feature works. | Defaults, with one feature changed. | +| **L** medium | The feature works at realistic size. | The features of S on a medium base. The generator halves batch, then channels, until the case fits the ceiling. | +| **M** model | Production solver paths. | Layers from named models (cite the source), plus the dnn-benchmarking corpus, normalized and deduplicated. | +| **H** hazard | Breakpoints. | Powers of 2 ± 1, dims not divisible by 4 or 8, group counts, non-dense strides. | +| **X** stress | Limits. | Huge batch, int32 overflow, very long accumulation. | + +**Defaults are non-degenerate.** Each dimension or ratio that can exceed 1 does +so in the default shape. + +**Per-tier ceilings.** These values come from convolution forward. Each plan +tunes them. + +| Tier | Ceiling for each case (fp32, x + w + y) | Sets | +|---|---|---| +| quick | ≤ 20 MFLOP, ≤ 1 MB | S | +| PR | ≤ 200 MFLOP, ≤ 16 MB | S, H, a few M | +| nightly | ≤ 8 GFLOP, ≤ 64 MB | S, L, M, H | +| full | none | all, including X | + +## 8. Sweep Construction and Tiers + +| Rule | Content | +|---|---| +| Core axes | Every tier covers them completely. Typically rank × layout × dtype. | +| Core cover | Every tier holds every core combination for the plain op and each fused topology. In quick, at least one engine executes each cell. | +| One at a time | S and L shapes isolate one feature. | +| All-pairs | By the end of nightly, each value of each axis meets each value of each other axis. A seeded generator makes this from the declared axes and legality. | +| Full | Complete products only where an interaction is plausible. | +| Probes | Each unsupported feature, awkward combination and hazard has a probe. Each tier has rejection probes. | +| Regressions | The reproducer of each escaped bug is a concrete case in the lowest tier that can afford it. | +| Concrete graphs | Each tier runs checked-in graphs only. Random sampling is fuzz mode (§14). | + +The tiers use the existing CTest labels and cascade. Each tier runs everything +below it. `full_*` runs only in full. + +| Tier | Bundle dir | Cadence | Budget (whole suite, cumulative) | +|---|---|---|---| +| quick | `quick/` | pre-commit | ~5 min | +| PR | `standard/` | PR gate | ~30 min | +| nightly | `comprehensive/` | nightly | ≤ 60 min | +| full | `full/` | weekly | none | + +**Estimate before you generate.** A plan states the case count, FLOP per case +and oracle cost for each tier and size class. The first draft for convolution +forward had ~14,000 full-tier cases. About 10,000 of them repeated one fused +epilogue over every geometry. The trimmed plan has ~3,800 cumulative cases and +still covers every pair. A cold-cache timing run fixes the final counts. Solver +search and compile time for each unique shape cost more than execution. + +## 9. Case Purpose + +Each generated case has a `purpose` in its sweep metadata: + +| Purpose | Expected outcome | Claims | +|---|---|---| +| `claim` | The claiming engine accepts and is correct. | `--write-support-claims` writes them (RFC 0015). | +| `probe` | A typed rejection, or a correct accept. | Never written. The published support matrix excludes probes. | +| `regression` | Same as the original bug case. | Same as the original case. | + +Today, a rejected probe and a probe that never ran both show as SKIP. The engine +integration gate will record rejections with a typed reason. This RFC requires +only the tag. With it, the support matrix can exclude probes, and reviewers can +tell probes from claims. + +## 10. Trade-off Register + +Each plan has a register: + +| Product not run | Cases | Covered instead by | Add it back when | +|---|---|---|---| + +The register of convolution forward, as an example: + +| Product not run | Cases | Covered instead by | Add it back when | +|---|---|---|---| +| Each activation × geometry × layout × dtype × both fusions | ~10,000 | Activation × core (complete), and activation × each S/L shape | A defect needs one activation and one geometry together | +| Extra dtypes × each shape × layout | ~1,660 | Dtype × core, and dtype paired with each S shape | An engine claims the dtype | +| Mixed layout, strides, alignment × each shape × dtype | ~580 | × core, and paired with each S shape | A layout bug depends on geometry | +| `CONVOLUTION` × each shape × layout × dtype | ~380 | Mode × core, and paired with each S shape | An engine supports it | + +Rejection probes are never in the register. + +## 11. Oracles and Reference Prerequisites + +Verification follows RFC 0011. In `auto` mode, the order is golden data, GPU +reference, CPU reference. Before you write cases for a feature, confirm that +each oracle **models** it. List each gap with its fix. Typical gaps: + +| Gap | Example | +|---|---| +| A mode that one reference rejects and another ignores | Conv `conv_mode`: the CPU reference rejects `CONVOLUTION`. The GPU reference always computes cross-correlation. | +| Asymmetric parameters tested in one reference only | Conv asymmetric padding | +| Types with no reference | fp8 and int8 conv | +| Special values | NaN and Inf need a validator that matches infinities. | +| Harness support | Exact-alignment placement, value-domain control, repeat-and-compare (§14.3) | + +A feature with no oracle stays in the plan and is marked "not run". A golden case +can cover it (§12, criterion G-B). + +## 12. Golden Tests + +### 12.1 Purpose + +Golden data does four jobs that a live reference cannot: + +- **Check the references.** A shared bug makes an engine and a reference agree. + A third party gives an independent answer. +- **Be the oracle where none exists.** The stored output checks a feature until + a reference catches up. +- **Pin semantics.** The documented semantics of the third party fix the meaning + of a parameter. +- **Anchor numerics.** Stored data does not move when reference code changes. + Each case has a calibrated tolerance. + +Golden tests are **not** a coverage tool. They make the judges of the sweep +trustworthy. + +### 12.2 Selection criteria + +A golden case meets at least one criterion. Most ops need 6–20 golden cases. + +| # | Criterion | +|---|---| +| G-A | **Baseline anchor**: one small case for each rank × layout family × dtype, for the plain op and each fused topology | +| G-B | **Oracle gap**: a feature that no hipDNN reference can check yet | +| G-C | **Semantics**: a parameter with two possible meanings | +| G-D | **Numerical regime**: a tiny output, a typical size, a long accumulation | +| G-E | **Escaped numerical bug**: the pinned reproducer | +| G-F | **Production anchor**: a few model layers that fit the size budget | + +A case is not golden if it meets no criterion and a live reference can check it. +A case is not golden if it exceeds 2 MiB or contains NaN or Inf. +`verify_golden_bundles.py` rejects both. A case is not golden if the third-party +semantics cannot map to hipDNN semantics. + +### 12.3 Generation + +- **Source.** An architecture-independent third party. PyTorch on CPU is the + default. A hipDNN reference never generates golden data. +- **Precision strategy.** Set it for each op in `meta.json`. Use precision + uplift for shallow ops and fp32. Use matched precision (the accumulation + contract of the engine) for deep ops and low-precision types (RFC 0011). +- **Provenance.** Pin the third-party version. Seed the inputs. Record + `forward_source` for backward ops in `meta.json`. +- **Tolerance.** Calibrate it for each case from the observed error of both + references, plus a margin. Never loosen it to make an engine pass. +- **Storage.** `golden/{CaseId}/tensors.dvc` on the `golden-data` remote. + +### 12.4 Confirmation + +`hipdnn_golden_data_tests --reference both` must pass before you commit a golden +case. A reference that cannot check a G-B case cannot confirm it. Use a second +independent computation (for example NumPy beside PyTorch) until the reference +is fixed. Never regenerate golden data from a hipDNN reference to resolve a +disagreement. + +### 12.5 Placement and lifecycle + +- Put one or two G-A anchors in quick. Put the rest in PR or nightly. +- A golden case is never the only coverage of a feature in a tier that some + lanes run without `dvc pull`. +- Regenerate golden data when semantics, third-party version or strategy + change. Review it like code. + +## 13. Authoring Process and Plan Template + +### 13.1 Process + +1. **Inventory and legality** from the API (§6). +2. **Audit** the bundles at the tip of the integration branch: + - map uids to roles through the template; + - derive the real parameters from the dims; + - count distinct behaviors, skip-only cases and per-tier cover. +3. **Shape catalog** (§7). +4. **Combination and tier rules** (§8). +5. **Estimate** counts, FLOP and oracle cost for each tier. +6. **Trade-off register** (§10). +7. **Oracle prerequisites** (§11). +8. **Golden selection** (§12). +9. **Generate, time, harvest and finalize:** + - generate the bundles and tag each case purpose; + - time a cold-cache run and fix the counts; + - harvest claims; + - update everything that uses case ids (FFM pins, TOML globs); + - update the plan in the same PR. + +### 13.2 Template + +```markdown +# Test Plan +Scope, bundles, out of scope. +## 1. The operation formula, tensors, shape rules, parameters, sources +## 2. Feature inventory per group: Id | Feature | Values | Oracle ready?; legality; rejection list +## 3. Shape catalog ceilings; S, L, M, H, X +## 4. Building the sweep rules; tier contents; counts by size class; probes and regressions; trade-off register +## 5. Golden reference Id | Case | Criterion | Precision strategy | Tier; generator; confirmation +## 6. Pass criteria verification, tolerances, claims; oracle and harness gaps +``` + +### 13.3 Review checklist + +- [ ] Each feature traces to an API field or validation rule. No section depends + on an engine. +- [ ] Each tensor has dtype, layout, stride, kind and alignment axes. Legality + is declared and the generator obeys it. +- [ ] Each axis has classes, boundaries and hazards. No default is degenerate. +- [ ] Each case has a purpose. Quick executes each core cell. Nightly retests + each feature at medium size. +- [ ] Each complete product is justified or in the register. Each tier has + rejection probes. +- [ ] Each oracle gap is listed with its fix. +- [ ] Counts, FLOP and oracle cost fit the tier budgets. +- [ ] Golden cases follow §12.2 and are confirmed. +- [ ] Bundle cases and plan items map in both directions. + +## 14. Test Generator and Fuzzing Tool + +### 14.1 Purpose + +One tool (working name `testgen`) reads one declaration of axes and legality for +each op. It has two modes: + +| Mode | Output | Where it runs | +|---|---|---| +| **Plan** | Deterministic bundles for the tiers (all-pairs, seeded rotation). We check in the output. | When a plan changes | +| **Fuzz** | Random legal graphs and tensors, run at once against engines and oracles | A developer machine, a scheduled non-gating job, the engine integration gate | + +The modes share the legality rules, so fuzz mode never writes an illegal graph. +A failing fuzz case becomes a plan case with no translation. + +### 14.2 Inputs + +- **Axes and legality** of the op (where they live: §19, question 1). +- **Shape catalog** of the plan (§7). Fuzz mode takes sizes from its ranges. A + size budget limits each case. +- **Seed.** The default is fixed. `RANDOM` picks a new seed and prints it. This + follows the `HIPDNN_GLOBAL_TEST_SEED` convention (`Seeds.hpp`). + +### 14.3 Tensor generation + +The test SDK has `Tensor::fillWithRandomValues(min, max, seed)`, `fillWithValue` +and `GraphTensorBundle::randomizeTensor`. The per-op tensor bundles fix the value +range in code. For example, the conv bundles fill x and w with uniform values in +[0, 1], and pass one seed to every tensor. The tool adds these **value domains** +(I3, I4): + +| Value domain | What it finds | +|---|---| +| Uniform, range set for each op and dtype | The default case | +| Zero-excluded | Hidden zero assumptions in division and normalization | +| All-zero, constant | Shortcut paths, bias-only paths | +| Large magnitude, near the dtype limit | Overflow, saturation | +| Subnormal | Flush-to-zero behavior | +| NaN and Inf injection (the case sets rate and position) | Special-value propagation. Needs a validator that matches infinities. | +| Index-unique ramp (each element is distinct) | Wrong indexing that random values can hide | +| Same sign, long accumulation | Worst-case rounding error | + +Rules: + +- The domain is an attribute of each tensor in the case, in the sweep metadata. +- The seed of each tensor comes from the case seed and the tensor uid. A new + tensor does not change the values of the other tensors. +- Placement (base alignment and offset, §6) is part of the case. +- Value fill runs in the C++ test SDK. The generator writes only the domain name + and its parameters. + +### 14.4 Fuzz mode + +Each iteration draws axis values from the legal space, builds the graph and +tensors, runs the graph on the selected engines, and checks the result with the +oracle chain (§11). It also checks these invariants: no crash, no hang, and the +same result when the run repeats or the plan is reused with new pointers. + +Each iteration has one typed outcome: `pass`, `mismatch`, `crash`, `timeout` or +`reject` (with the reason). A `reject` is not a failure. It does not count as +executed (P10). + +| Sampling strategy | Use | +|---|---| +| Uniform over the legal space | Broad search | +| Boundary-biased: draw the boundaries and hazards of P3 more often | Faults at limits | +| Corpus mutation: change one axis of a real model layer (set M) | Three-way faults near real use | + +A run has a count budget or a time budget, and prints its seed. A fault in a +feature that an oracle does not model (§11) is reported as "unverified". + +### 14.5 Failure handling + +1. Record the seed and the full case description (JSON). +2. **Minimize.** Reset one axis to its default, or halve one dimension. Keep the + change if the failure remains. Repeat until no change keeps the failure. +3. Write the result as a concrete sweep case with purpose `regression` (§9), in + the lowest tier that can afford it. +4. If two oracles disagree, report an oracle bug. Consider a G-B golden case. + +The case description is the permanent record. A seed reproduces a case only for +the same tool version. + +### 14.6 Boundaries + +- Fuzz mode is not a CI tier. The tiers run checked-in graphs only (§8). +- The engine integration gate decides campaign policy. This RFC defines the tool. +- The tool does not fuzz the descriptor loader or matcher (RFC 0017, RFC 0018). + It shares only the seed convention. + +## 15. Required Infrastructure + +| # | Capability | Status | Needed for | +|---|---|---|---| +| I1 | **Sweep generator** (plan mode, §14). Reads declared axes, legality, shape sets and tier rules. Writes `sweep.json` for each tier, with all-pairs and seeded rotation. | Missing. The conv sweeps came from C++ tests (`generator: capture-bundles`) or were written by hand (`generator: manual`). | §8 | +| I2 | **`purpose` field** in sweep case metadata. Support-matrix generation excludes probes. | Missing | §9 | +| I3 | **Exact-alignment and offset placement** in the harness | Missing. Pointers are allocator-aligned. | §6 | +| I4 | **Value-domain control** for each tensor, with seeds derived for each tensor (§14.3) | Missing. The tensor bundles fix ranges in code. | §6, §14 | +| I5 | **Repeat-and-compare** on bundles | Exists only inside providers | §6 | +| I6 | **Oracle fixes** for each plan (for example conv `conv_mode`) | Per op | §11 | +| I7 | **Golden generator for each op** in `reference-data-scripts/`, writing `meta.json` with criterion and strategy | Batchnorm and SDPA only | §12 | +| I8 | **Corpus intake** for M shapes | `migration-scripts/import_graph.py` is the start. | §7 | +| I9 | **Cold-cache timing** for each unique shape and arch | Ad hoc | §8 | +| I10 | **Fuzz mode** (§14.4, §14.5): sampling, invariant checks, minimizer, reproducer writer | Missing | §14 | + +Templates can also need new placeholders. Today, conv `conv_mode`, +`compute_data_type` and pointwise `operation` are hard-coded. These changes are +per op. The plan tracks them. + +## 16. Rollout + +| Phase | Content | Exit | +|---|---|---| +| 1 | This RFC. The methodology guide and index under `docs/test-plans/`. The reviewed convolution-forward plan (ALMIOPEN-2895). | Plan merged | +| 2 | I1, I2. Regenerate the conv-forward bundles from the plan. Harvest claims. Fix FFM pins and TOML globs. | Conv tiers green on supported engines. Each case traces to the plan. | +| 3 | I3–I7, as the conv plan needs them. The conv golden set. | Conv reference gaps closed or recorded | +| 4 | I10. Fuzz mode for conv forward, on a developer machine and in a scheduled non-gating job. | Fuzz finds a seeded fault. The minimizer writes a reproducer that runs as a `regression` case. | +| 5 | Plans for the other op families, in epic order (attention, normalization, then the rest). Each plan regenerates its bundles. | Each op family has a plan | + +Each phase lands on its own. Bundles without a plan keep working until the plan +of their op replaces them. + +## 17. Alternatives Considered + +| Alternative | Why rejected | +|---|---| +| **Keep curating bundles by hand** | It caused the K = G degeneracy, toy sizes and blind spots of §2. It does not scale to a full plan. | +| **Plans keyed to engine support** | They stay green easily but hide API features (§2) and need a rewrite when support changes. P2 and claims replace them. | +| **Full cross-products** | The conv-forward product is ~14,000 cases. Most repeat one check (§10). All-pairs, targeted products and a register replace them. | +| **Random sampling in the CI tiers** | Random cases make CI results change from run to run. Fuzz mode (§14) runs outside the tiers. Its reproducers become pinned cases. | +| **Golden data for every case** | It costs storage, has a 2 MiB limit for each case, cannot hold non-finite values, and adds nothing where a trusted live reference exists. Criteria (§12.2) replace it. | +| **Separate tools for bundles and fuzzing** | Two copies of the legality rules would drift. A fuzz failure would need translation into a plan case. One tool with two modes (§14) replaces them. | + +## 18. Risks + +| Risk | Mitigation | +|---|---| +| Plan and bundles drift | The generator writes the bundles. The same-PR rule. Reviewers check traceability in both directions. | +| Case counts exceed the budgets | Estimate first (§8). A cold-cache timing run fixes the counts. Per-tier ceilings. | +| Probes fill the suite with skips | A probe costs ~1 ms. The `purpose` tag keeps probes out of the support matrix. P10 requires executed cover. | +| A missed oracle gap gives wrong verdicts | Review step of §11. Golden G-B cases. Three-way confirmation. | +| Pairwise misses a three-way fault | Corpus shapes. Fuzz mode. Pinned regressions. | +| Case-id changes break name-keyed config | Step 9 of §13.1 updates FFM pins and TOML globs in the same PR. | +| The methodology guide and this RFC diverge | The guide (`docs/test-plans/methodology.md`) is the working form of §5–§13. Principle changes need an RFC amendment. | +| Fuzz mode reports false failures from a weak oracle | Check each minimized reproducer against both references. Report faults in unmodeled features as "unverified" (§14.4). | +| A seed stops reproducing a case after a tool change | The JSON case description is the permanent record (§14.5). | + +## 19. Open Questions and Future Work + +1. **Where axes and legality live.** A per-op Python module in + `reference-data-scripts/`, or a data file beside the plan? +2. **Precision strategy for fp16 and bf16 conv golden cases.** The plan proposes + matched precision. Confirm it against the accumulation contracts of all + claiming engines. +3. **Source of tolerance calibration.** Per-case `meta.json` (RFC 0011 level 2), + or dynamic tolerances? +4. **Full tier and unclaimed dtypes.** Should full-tier products run only on + claimed dtypes, to cut skip-only cases? +5. **The engine integration gate.** Its proposal, and the interface it takes from + the plans. +6. **Tool language split.** Plan mode can be Python, like + `reference-data-scripts/`. Value fill must run in the C++ harness. Does the + Python side drive a fuzz run through process calls, or do the two sides share + only the case-description format? +7. **Fuzz job placement.** Which machine and schedule run the non-gating job? + Where do its reports go? + +## 20. Glossary + +- **Plan.** A per-op document under `docs/test-plans/` that defines the test + sweep of the op. +- **Feature.** A schema field, enum value, validation rule or harness property + that can change behavior. +- **Core axes.** The axes that each tier covers completely. Typically rank × + layout × dtype. +- **All-pairs.** A set of cases in which each value of each axis meets each value + of each other axis at least once. +- **Probe, claim, regression.** The purposes of a case (§9). +- **S, L, M, H, X.** Small, medium, model, hazard and stress shape sets (§7). +- **Trade-off register.** The table in a plan of products that we do not run on + purpose. +- **Oracle.** Anything that produces the expected output: golden data, the GPU + reference or the CPU reference. +- **G-A … G-F.** Golden selection criteria (§12.2). +- **Test generator tool.** The tool of §14, with plan mode and fuzz mode. +- **Value domain.** The class of values that fills a tensor (§14.3). From b424a4193f029d96be57c4ae1b49035fdcd8eeae Mon Sep 17 00:00:00 2001 From: Adam Dickin Date: Mon, 19 Oct 2026 10:41:48 -0600 Subject: [PATCH 2/5] docs(hipdnn): drop first-plan content from RFC 0021 (ALMIOPEN-2895) The convolution-forward plan lands in a separate PR. Remove its link, worked-example register and rollout phases. Keep the convolution-forward audit as problem evidence. JIRA ID : ALMIOPEN-2895 --- .../hipdnn/docs/rfcs/0021_PerOpTestPlans.md | 40 ++++++------------- 1 file changed, 13 insertions(+), 27 deletions(-) diff --git a/projects/hipdnn/docs/rfcs/0021_PerOpTestPlans.md b/projects/hipdnn/docs/rfcs/0021_PerOpTestPlans.md index 97b8d4a0d842..4b9bc52496c7 100644 --- a/projects/hipdnn/docs/rfcs/0021_PerOpTestPlans.md +++ b/projects/hipdnn/docs/rfcs/0021_PerOpTestPlans.md @@ -49,10 +49,6 @@ method: - Write the bundles from the plan with one **test generator tool**. The same tool makes tensor values and fuzzes (§14). -The first plan, convolution forward -([`convolution-fwd.md`](../../../../dnn-providers/integration-tests/docs/test-plans/convolution-fwd.md), -ALMIOPEN-2895), is the worked example. Its audit motivates most of this RFC. - ## 2. Problem Statement The input space of an op is unbounded. Integer parameters have no limit, differ @@ -185,8 +181,7 @@ pointer *less* aligned than declared and expects `BAD_PARAM`. **Defaults are non-degenerate.** Each dimension or ratio that can exceed 1 does so in the default shape. -**Per-tier ceilings.** These values come from convolution forward. Each plan -tunes them. +**Per-tier ceilings.** Each plan tunes these starting values. | Tier | Ceiling for each case (fp32, x + w + y) | Sets | |---|---|---| @@ -219,11 +214,11 @@ below it. `full_*` runs only in full. | full | `full/` | weekly | none | **Estimate before you generate.** A plan states the case count, FLOP per case -and oracle cost for each tier and size class. The first draft for convolution -forward had ~14,000 full-tier cases. About 10,000 of them repeated one fused -epilogue over every geometry. The trimmed plan has ~3,800 cumulative cases and -still covers every pair. A cold-cache timing run fixes the final counts. Solver -search and compile time for each unique shape cost more than execution. +and oracle cost for each tier and size class. A full product can be very large. +For convolution forward, it is ~14,000 cases. About 10,000 of them repeat one +fused epilogue over every geometry. A cold-cache timing run fixes the final +counts. Solver search and compile time for each unique shape cost more than +execution. ## 9. Case Purpose @@ -247,15 +242,6 @@ Each plan has a register: | Product not run | Cases | Covered instead by | Add it back when | |---|---|---|---| -The register of convolution forward, as an example: - -| Product not run | Cases | Covered instead by | Add it back when | -|---|---|---|---| -| Each activation × geometry × layout × dtype × both fusions | ~10,000 | Activation × core (complete), and activation × each S/L shape | A defect needs one activation and one geometry together | -| Extra dtypes × each shape × layout | ~1,660 | Dtype × core, and dtype paired with each S shape | An engine claims the dtype | -| Mixed layout, strides, alignment × each shape × dtype | ~580 | × core, and paired with each S shape | A layout bug depends on geometry | -| `CONVOLUTION` × each shape × layout × dtype | ~380 | Mode × core, and paired with each S shape | An engine supports it | - Rejection probes are never in the register. ## 11. Oracles and Reference Prerequisites @@ -504,10 +490,10 @@ per op. The plan tracks them. | Phase | Content | Exit | |---|---|---| -| 1 | This RFC. The methodology guide and index under `docs/test-plans/`. The reviewed convolution-forward plan (ALMIOPEN-2895). | Plan merged | -| 2 | I1, I2. Regenerate the conv-forward bundles from the plan. Harvest claims. Fix FFM pins and TOML globs. | Conv tiers green on supported engines. Each case traces to the plan. | -| 3 | I3–I7, as the conv plan needs them. The conv golden set. | Conv reference gaps closed or recorded | -| 4 | I10. Fuzz mode for conv forward, on a developer machine and in a scheduled non-gating job. | Fuzz finds a seeded fault. The minimizer writes a reproducer that runs as a `regression` case. | +| 1 | This RFC. The methodology guide and index under `docs/test-plans/`. | RFC merged | +| 2 | I1, I2. The first per-op plan, in its own PR, regenerates its bundles from the plan. Harvest claims. Fix FFM pins and TOML globs. | Tiers green on supported engines. Each case traces to the plan. | +| 3 | I3–I7, as the first plan needs them. Its golden set. | Reference gaps closed or recorded | +| 4 | I10. Fuzz mode for the first op, on a developer machine and in a scheduled non-gating job. | Fuzz finds a seeded fault. The minimizer writes a reproducer that runs as a `regression` case. | | 5 | Plans for the other op families, in epic order (attention, normalization, then the rest). Each plan regenerates its bundles. | Each op family has a plan | Each phase lands on its own. Bundles without a plan keep working until the plan @@ -542,9 +528,9 @@ of their op replaces them. 1. **Where axes and legality live.** A per-op Python module in `reference-data-scripts/`, or a data file beside the plan? -2. **Precision strategy for fp16 and bf16 conv golden cases.** The plan proposes - matched precision. Confirm it against the accumulation contracts of all - claiming engines. +2. **Precision strategy for fp16 and bf16 golden cases.** We propose matched + precision. Confirm it against the accumulation contracts of all claiming + engines. 3. **Source of tolerance calibration.** Per-case `meta.json` (RFC 0011 level 2), or dynamic tolerances? 4. **Full tier and unclaimed dtypes.** Should full-tier products run only on From e7551c0928e1aaebe7b5b031c0543f7be085b9f5 Mon Sep 17 00:00:00 2001 From: Adam Dickin Date: Mon, 19 Oct 2026 15:54:12 -0600 Subject: [PATCH 3/5] docs(hipdnn): align RFC 0021 with ALMIOPEN-2947 Make exhaustive testing a separate tool with a hook only. Keep plans engine-independent. Add the level questions, the plan-to-file mapping, the rollout order (attention, convolution, normalization), the baseline golden policy, and the documents to update. JIRA ID : ALMIOPEN-2947 --- .../hipdnn/docs/rfcs/0021_PerOpTestPlans.md | 344 +++++++++--------- 1 file changed, 176 insertions(+), 168 deletions(-) diff --git a/projects/hipdnn/docs/rfcs/0021_PerOpTestPlans.md b/projects/hipdnn/docs/rfcs/0021_PerOpTestPlans.md index 4b9bc52496c7..66701a4176f4 100644 --- a/projects/hipdnn/docs/rfcs/0021_PerOpTestPlans.md +++ b/projects/hipdnn/docs/rfcs/0021_PerOpTestPlans.md @@ -11,13 +11,13 @@ 5. [Principles](#5-principles) 6. [Feature Inventory](#6-feature-inventory) 7. [Shape Catalog and Size Ladder](#7-shape-catalog-and-size-ladder) -8. [Sweep Construction and Tiers](#8-sweep-construction-and-tiers) +8. [Sweep Construction, Levels and Lanes](#8-sweep-construction-levels-and-lanes) 9. [Case Purpose](#9-case-purpose) 10. [Trade-off Register](#10-trade-off-register) 11. [Oracles and Reference Prerequisites](#11-oracles-and-reference-prerequisites) 12. [Golden Tests](#12-golden-tests) 13. [Authoring Process and Plan Template](#13-authoring-process-and-plan-template) -14. [Test Generator and Fuzzing Tool](#14-test-generator-and-fuzzing-tool) +14. [Exhaustive Mode and the Separate Tool](#14-exhaustive-mode-and-the-separate-tool) 15. [Required Infrastructure](#15-required-infrastructure) 16. [Rollout](#16-rollout) 17. [Alternatives Considered](#17-alternatives-considered) @@ -35,19 +35,25 @@ captured from older C++ tests, then re-tiered. No case has a stated reason. Nothing records what we left out. This RFC adds a **per-op test plan**: one reviewed document for each op family, -in `dnn-providers/integration-tests/docs/test-plans/`. All plans follow one -method: +in `dnn-providers/integration-tests/docs/test-plans/`. The plan tests the op. It +does not depend on an engine. We apply the plan to each engine through claims. +All plans follow one method: - Derive features from the graph API. Do not use engines or old tests. - Take shapes from size sets (small, medium, model, stress) and hazard - breakpoints. Each tier has a size ceiling. + breakpoints. Each level has a size ceiling. - Combine features one at a time, then all-pairs, then in targeted products. - Give each case a purpose: `claim`, `probe` or `regression`. - Record each product that we do not run in a trade-off register. +- Put each case in one of four levels (quick, PR, nightly, full). Each level + answers one question (§8). - Select a small golden set with fixed criteria. An independent third party generates it. -- Write the bundles from the plan with one **test generator tool**. The same - tool makes tensor values and fuzzes (§14). +- Write the bundles from the plan with a generator, so that bundles and plan + cannot drift. + +Exhaustive testing is **not** part of this RFC. It is a separate tool with its +own RFC. Each plan only declares the hook that the tool reads (§14). ## 2. Problem Statement @@ -79,15 +85,20 @@ each op depend on the tests that the op happened to have. - The **method** of every per-op plan (§5–§13). - The **plan document**: location, template, and the rule that bundles and plan stay in sync. -- **Golden selection** (§12), building on RFC 0011. -- The **test generator and fuzzing tool** (§14). -- The **infrastructure** that the method needs (§15) and its rollout (§16). +- The **four levels**, the question that each answers, and the **mapping** from + a plan to bundles, claims, TOML, tier YAML and CTest lanes (§8). +- **Golden policy and selection** (§12), building on RFC 0011. +- The **hook** that each plan declares for the exhaustive tool (§14). +- The **infrastructure** that the method needs (§15), its rollout and the + documents to update (§16). ### 3.2 Out of scope | Item | Where it belongs | |---|---| -| Gating a new engine integration: per-case outcome reporting, running every kernel of an engine, fuzzing campaigns (engines, duration, schedule), the days-long gauntlet tier, review of agent-authored integrations | A separate engine-integration-gate proposal. It uses the plans' graphs and axes and the tool of §14. It returns bug reproducers as pinned regressions (§9). | +| Exhaustive test generation, fuzzing, invalid-graph generation, engine-knob sweeps | A separate tool, with its own RFC. §14 states only the interface. | +| Gating a new engine integration: per-case outcome reporting, running every kernel of an engine, the days-long gauntlet tier, review of agent-authored integrations | A separate engine-integration-gate proposal | +| Writing the per-op plans and bundles | One PR for each op (§16) | | Bundle format, verification modes, DVC layout | RFC 0011 | | Claim schema, enforcement, harvesting | RFC 0015 | | Fuzzing of the descriptor loader and matcher | RFC 0017, RFC 0018 | @@ -100,22 +111,25 @@ each op depend on the tests that the op happened to have. flowchart LR API[Graph API
schemas + frontend validation] --> PLAN[Per-op test plan
docs/test-plans/<op>.md] AUDIT[Audit of existing bundles] --> PLAN - PLAN --> GEN[Test generator tool
plan mode] - GEN --> B[Bundles per tier
sweep.json, support.json] + PLAN --> GEN[Bundle generator] + GEN --> B[Bundles per level
sweep.json, support.json] PLAN --> GOLD[Golden generator
third party] GOLD --> B - B --> CI[CI tiers
quick / PR / nightly / full] + B --> CI[CI levels
quick / PR / nightly / full] CI -- claims harvest --> B - PLAN --> FUZZ[Test generator tool
fuzz mode] - FUZZ -- minimized reproducers --> PLAN - GATE[Engine integration gate] -. campaigns .-> FUZZ + PLAN -. exhaustive-mode hook .-> EXH[Exhaustive tool
separate RFC] + EXH -. minimized reproducers .-> PLAN ``` -- **One plan for each op family**, for example `convolution-fwd.md`, `matmul.md` - or `sdpa-fwd.md`, in `dnn-providers/integration-tests/docs/test-plans/`. An - index (`README.md`) lists the plans and their bundles. +- **One plan for each op family**, for example `sdpa-fwd.md`, + `convolution-fwd.md` or `matmul.md`, in + `dnn-providers/integration-tests/docs/test-plans/`. An index (`README.md`) + lists the plans and their bundles. +- **The plan is engine-independent.** It tests the op. It has no engine matrix. + Claims (`support.json`) and per-engine TOML apply the plan to each engine + (§8.4). - **The plan is the source of truth.** The generator reads the declared axes, - legality rules, shape catalog and tier rules, and writes the bundles. Nobody + legality rules, shape catalog and level rules, and writes the bundles. Nobody edits bundles by hand. - **Traceability in both directions.** Each bundle case maps to a plan item. Each plan item maps to at least one case. A PR that adds, moves or deletes @@ -128,10 +142,10 @@ flowchart LR | # | Principle | Practice | |---|---|---| | P1 | **Derive features from the API.** | Each field of the op schema (`flatbuffers_sdk/schemas/*.fbs`) is a feature. So is each rule in `pre_validate_node` and `infer_properties_node` of the frontend node. Tests and engine capabilities do not define the op. | -| P2 | **The plan does not depend on an engine.** | Engines join through claims. A feature that no engine supports stays as a probe. When support arrives, the run reports `UNCLAIMED_SUPPORT` (RFC 0015). The plan needs no edit. | +| P2 | **The plan does not depend on an engine.** | The plan tests the op. Engines join through claims. A feature that no engine supports stays as a probe. When support arrives, the run reports `UNCLAIMED_SUPPORT` (RFC 0015). The plan needs no edit. | | P3 | **Divide each axis into classes. Then add boundaries and hazards.** | Classes: 1, odd, small, power of two, large, not a multiple of the vector width. Boundaries: output size 1, padding ≥ filter, remainders, values near the dtype limit. Hazards: powers of 2 ± 1, dims not divisible by 4 or 8, int32 index overflow. | -| P4 | **One feature at a time. Then all-pairs. Then targeted products.** | One-feature shapes show the cause of a failure. All-pairs finds most interaction faults. Run a complete product only where an interaction is plausible. The corpus and fuzz mode (§14) find three-way faults. Pin them. | -| P5 | **Size grows with the tier.** | Nightly tests each feature again at medium size. | +| P4 | **One feature at a time. Then all-pairs. Then targeted products.** | One-feature shapes show the cause of a failure. All-pairs finds most interaction faults. Run a complete product only where an interaction is plausible. The corpus and the exhaustive tool (§14) find three-way faults. Pin them. | +| P5 | **Size grows with the level.** | Nightly tests each feature again at medium size. | | P6 | **Each product that we do not run is a recorded decision.** | Use the trade-off register (§10). Never trim rejection probes. | | P7 | **A case needs an oracle that models it.** | See §11. A case with no oracle is "not run", never green. | | P8 | **Count distinct behaviors, not cases.** | Report distinct geometries, feature values and skip-only cases. | @@ -151,11 +165,11 @@ A plan lists every feature of the op, in these groups: | Data types | Mixed precision, compute and intermediate types, graph-level or per-node types | | Graph options | Override-shape ([RFC 0008](0008_OverridableTensorShapesDesign.md)), fused epilogues and chains, virtual or stored intermediates | | Awkward combinations | Legal but hazardous graphs: inputs of different dtypes, tensors in different layouts, degenerate ratios | -| Values | The value domains of §14.3. Accumulation length. Determinism. | +| Values | Uniform, zero-excluded, all-zero, constant and large-magnitude fills. NaN, Inf, subnormals. Accumulation length. Determinism. | | Execution | Workspace limits, plan reuse with new pointers, serialization round trip ([RFC 0009](0009_CompiledPlanSerialization.md)) | | Invalid inputs | One frontend **unit test** for each validation rule. These are not sweep cases. | -For each feature, the plan records the values to cover, the earliest tier, the +For each feature, the plan records the values to cover, the earliest level, the oracle readiness (§11) and the purpose of its cases (§9). **Legality is declared once for each op.** It copies the frontend validation. @@ -181,45 +195,68 @@ pointer *less* aligned than declared and expects `BAD_PARAM`. **Defaults are non-degenerate.** Each dimension or ratio that can exceed 1 does so in the default shape. -**Per-tier ceilings.** Each plan tunes these starting values. +**Per-level ceilings.** Each plan tunes these starting values. -| Tier | Ceiling for each case (fp32, x + w + y) | Sets | +| Level | Ceiling for each case (fp32, x + w + y) | Sets | |---|---|---| | quick | ≤ 20 MFLOP, ≤ 1 MB | S | | PR | ≤ 200 MFLOP, ≤ 16 MB | S, H, a few M | | nightly | ≤ 8 GFLOP, ≤ 64 MB | S, L, M, H | | full | none | all, including X | -## 8. Sweep Construction and Tiers +## 8. Sweep Construction, Levels and Lanes + +### 8.1 Sweep rules | Rule | Content | |---|---| -| Core axes | Every tier covers them completely. Typically rank × layout × dtype. | -| Core cover | Every tier holds every core combination for the plain op and each fused topology. In quick, at least one engine executes each cell. | +| Core axes | Every level covers them completely. Typically rank × layout × dtype. | +| Core cover | Every level holds every core combination for the plain op and each fused topology. In quick, at least one engine executes each cell. | | One at a time | S and L shapes isolate one feature. | | All-pairs | By the end of nightly, each value of each axis meets each value of each other axis. A seeded generator makes this from the declared axes and legality. | | Full | Complete products only where an interaction is plausible. | -| Probes | Each unsupported feature, awkward combination and hazard has a probe. Each tier has rejection probes. | -| Regressions | The reproducer of each escaped bug is a concrete case in the lowest tier that can afford it. | -| Concrete graphs | Each tier runs checked-in graphs only. Random sampling is fuzz mode (§14). | +| Probes | Each unsupported feature, awkward combination and hazard has a probe. Each level has rejection probes. | +| Regressions | The reproducer of each escaped bug is a concrete case in the lowest level that can afford it. | +| Concrete graphs | Each level runs checked-in graphs only. Random sampling belongs to the exhaustive tool (§14). | -The tiers use the existing CTest labels and cascade. Each tier runs everything +### 8.2 The four levels + +Each level answers one question. The levels cascade: each level runs everything below it. `full_*` runs only in full. -| Tier | Bundle dir | Cadence | Budget (whole suite, cumulative) | -|---|---|---|---| -| quick | `quick/` | pre-commit | ~5 min | -| PR | `standard/` | PR gate | ~30 min | -| nightly | `comprehensive/` | nightly | ≤ 60 min | -| full | `full/` | weekly | none | +| Level | Question it answers | Bundle dir | CTest label (other labels) | Cadence | Budget (whole suite, cumulative) | +|---|---|---|---|---|---| +| quick | What must we test to catch gross breakage fast? | `quick/` | `quick` (`pre-commit`, `smoke`) | pre-commit | ~5 min | +| PR | What must we test on every PR? | `standard/` | `standard` (`pr`, `precheckin`) | PR gate | ~30 min | +| nightly | What should we test each day, with wider coverage for each engine? | `comprehensive/` | `comprehensive` (`nightly`, `extended`) | nightly | ≤ 60 min | +| full | Which wider sweeps must exist in the full suite? | `full/` | `full` (`all`) | weekly | none | **Estimate before you generate.** A plan states the case count, FLOP per case -and oracle cost for each tier and size class. A full product can be very large. +and oracle cost for each level and size class. A full product can be very large. For convolution forward, it is ~14,000 cases. About 10,000 of them repeat one fused epilogue over every geometry. A cold-cache timing run fixes the final counts. Solver search and compile time for each unique shape cost more than execution. +### 8.3 Mapping a plan to files + +| Plan item | File or setting | Who writes it | +|---|---|---| +| Op topology | `{level}/{Op}/{Topology}/graph.template.json` | The bundle generator | +| Cases: axes, shapes, level, purpose | `sweep.json` (cases and per-case metadata) in the directory of the level | The bundle generator | +| Golden cases | `golden/{CaseId}/tensors.dvc` (RFC 0011) | The golden generator | +| Claims: which engine supports which case | `support.json`, or `{Name}.support.json` for a single graph (RFC 0015) | Harvested with `--write-support-claims`. Never hand-written. | +| Tolerances, validator overrides, skips for an engine | `dnn-providers//config/.toml` | The provider, for each engine | +| Which suites each lane runs | Level YAML (`test_categories_integration.yaml`) in each provider. Its categories become CTest labels. | The provider | +| Cascade of levels | Each YAML category lists the patterns of the lower levels. CTest labels do not inherit. | The provider | + +### 8.4 From plan to engine + +The plan never names an engine. The same bundles run on every provider lane. +The claim of each engine decides which cases run for that engine. TOML records +the engine tolerances and known limits. Therefore, one plan covers any number of +engines, and a new engine needs no plan edit. + ## 9. Case Purpose Each generated case has a `purpose` in its sweep metadata: @@ -256,7 +293,7 @@ each oracle **models** it. List each gap with its fix. Typical gaps: | Asymmetric parameters tested in one reference only | Conv asymmetric padding | | Types with no reference | fp8 and int8 conv | | Special values | NaN and Inf need a validator that matches infinities. | -| Harness support | Exact-alignment placement, value-domain control, repeat-and-compare (§14.3) | +| Harness support | Exact-alignment placement, value-domain control, repeat-and-compare (§15) | A feature with no oracle stays in the plan and is marked "not run". A golden case can cover it (§12, criterion G-B). @@ -279,7 +316,13 @@ Golden data does four jobs that a live reference cannot: Golden tests are **not** a coverage tool. They make the judges of the sweep trustworthy. -### 12.2 Selection criteria +### 12.2 Policy and selection criteria + +**The policy is a baseline, not an exhaustive set.** Each op gets the baseline +third-party check (G-A, with PyTorch or a similar third party). The other +criteria add a case only where that criterion applies. Exhaustive golden +coverage is not a goal. The separate exhaustive tool (§14) covers the wide +space. A golden case meets at least one criterion. Most ops need 6–20 golden cases. @@ -321,7 +364,7 @@ disagreement. ### 12.5 Placement and lifecycle - Put one or two G-A anchors in quick. Put the rest in PR or nightly. -- A golden case is never the only coverage of a feature in a tier that some +- A golden case is never the only coverage of a feature in a level that some lanes run without `dvc pull`. - Regenerate golden data when semantics, third-party version or strategy change. Review it like code. @@ -334,19 +377,20 @@ disagreement. 2. **Audit** the bundles at the tip of the integration branch: - map uids to roles through the template; - derive the real parameters from the dims; - - count distinct behaviors, skip-only cases and per-tier cover. + - count distinct behaviors, skip-only cases and per-level cover. 3. **Shape catalog** (§7). -4. **Combination and tier rules** (§8). -5. **Estimate** counts, FLOP and oracle cost for each tier. +4. **Combination and level rules** (§8). +5. **Estimate** counts, FLOP and oracle cost for each level. 6. **Trade-off register** (§10). 7. **Oracle prerequisites** (§11). 8. **Golden selection** (§12). -9. **Generate, time, harvest and finalize:** - - generate the bundles and tag each case purpose; - - time a cold-cache run and fix the counts; - - harvest claims; - - update everything that uses case ids (FFM pins, TOML globs); - - update the plan in the same PR. +9. **Exhaustive-mode hook** (§14). +10. **Generate, time, harvest and finalize:** + - generate the bundles and tag each case purpose; + - time a cold-cache run and fix the counts; + - harvest claims; + - update everything that uses case ids (FFM pins, TOML globs); + - update the plan in the same PR. ### 13.2 Template @@ -356,11 +400,14 @@ Scope, bundles, out of scope. ## 1. The operation formula, tensors, shape rules, parameters, sources ## 2. Feature inventory per group: Id | Feature | Values | Oracle ready?; legality; rejection list ## 3. Shape catalog ceilings; S, L, M, H, X -## 4. Building the sweep rules; tier contents; counts by size class; probes and regressions; trade-off register -## 5. Golden reference Id | Case | Criterion | Precision strategy | Tier; generator; confirmation +## 4. Building the sweep rules; level contents; counts by size class; probes and regressions; trade-off register +## 5. Golden reference Id | Case | Criterion | Precision strategy | Level; generator; confirmation ## 6. Pass criteria verification, tolerances, claims; oracle and harness gaps +## 7. Exhaustive mode states that the op has one; axes, legality, shape ranges, value domains for the exhaustive tool ``` +A plan has no engine matrix (§8.4). + ### 13.3 Review checklist - [ ] Each feature traces to an API field or validation rule. No section depends @@ -370,117 +417,58 @@ Scope, bundles, out of scope. - [ ] Each axis has classes, boundaries and hazards. No default is degenerate. - [ ] Each case has a purpose. Quick executes each core cell. Nightly retests each feature at medium size. -- [ ] Each complete product is justified or in the register. Each tier has +- [ ] Each complete product is justified or in the register. Each level has rejection probes. - [ ] Each oracle gap is listed with its fix. -- [ ] Counts, FLOP and oracle cost fit the tier budgets. +- [ ] Counts, FLOP and oracle cost fit the level budgets. - [ ] Golden cases follow §12.2 and are confirmed. +- [ ] The exhaustive-mode section is present. - [ ] Bundle cases and plan items map in both directions. -## 14. Test Generator and Fuzzing Tool - -### 14.1 Purpose - -One tool (working name `testgen`) reads one declaration of axes and legality for -each op. It has two modes: - -| Mode | Output | Where it runs | -|---|---|---| -| **Plan** | Deterministic bundles for the tiers (all-pairs, seeded rotation). We check in the output. | When a plan changes | -| **Fuzz** | Random legal graphs and tensors, run at once against engines and oracles | A developer machine, a scheduled non-gating job, the engine integration gate | - -The modes share the legality rules, so fuzz mode never writes an illegal graph. -A failing fuzz case becomes a plan case with no translation. - -### 14.2 Inputs - -- **Axes and legality** of the op (where they live: §19, question 1). -- **Shape catalog** of the plan (§7). Fuzz mode takes sizes from its ranges. A - size budget limits each case. -- **Seed.** The default is fixed. `RANDOM` picks a new seed and prints it. This - follows the `HIPDNN_GLOBAL_TEST_SEED` convention (`Seeds.hpp`). +## 14. Exhaustive Mode and the Separate Tool -### 14.3 Tensor generation +**This RFC does not design exhaustive testing.** Exhaustive test generation, +fuzzing, invalid-graph generation and engine-knob sweeps belong to a separate +tool. That tool has its own RFC. Work on it starts after the attention and +convolution plans exist (§16). -The test SDK has `Tensor::fillWithRandomValues(min, max, seed)`, `fillWithValue` -and `GraphTensorBundle::randomizeTensor`. The per-op tensor bundles fix the value -range in code. For example, the conv bundles fill x and w with uniform values in -[0, 1], and pass one seed to every tensor. The tool adds these **value domains** -(I3, I4): - -| Value domain | What it finds | -|---|---| -| Uniform, range set for each op and dtype | The default case | -| Zero-excluded | Hidden zero assumptions in division and normalization | -| All-zero, constant | Shortcut paths, bias-only paths | -| Large magnitude, near the dtype limit | Overflow, saturation | -| Subnormal | Flush-to-zero behavior | -| NaN and Inf injection (the case sets rate and position) | Special-value propagation. Needs a validator that matches infinities. | -| Index-unique ramp (each element is distinct) | Wrong indexing that random values can hide | -| Same sign, long accumulation | Worst-case rounding error | - -Rules: - -- The domain is an attribute of each tensor in the case, in the sweep metadata. -- The seed of each tensor comes from the case seed and the tensor uid. A new - tensor does not change the values of the other tensors. -- Placement (base alignment and offset, §6) is part of the case. -- Value fill runs in the C++ test SDK. The generator writes only the domain name - and its parameters. - -### 14.4 Fuzz mode - -Each iteration draws axis values from the legal space, builds the graph and -tensors, runs the graph on the selected engines, and checks the result with the -oracle chain (§11). It also checks these invariants: no crash, no hang, and the -same result when the run repeats or the plan is reused with new pointers. - -Each iteration has one typed outcome: `pass`, `mismatch`, `crash`, `timeout` or -`reject` (with the reason). A `reject` is not a failure. It does not count as -executed (P10). - -| Sampling strategy | Use | -|---|---| -| Uniform over the legal space | Broad search | -| Boundary-biased: draw the boundaries and hazards of P3 more often | Faults at limits | -| Corpus mutation: change one axis of a real model layer (set M) | Three-way faults near real use | +This RFC defines only the **hook**. Each plan has an exhaustive-mode section +(§13.2). It states that the op has an exhaustive mode, and it declares: -A run has a count budget or a time budget, and prints its seed. A fault in a -feature that an oracle does not model (§11) is reported as "unverified". +- the axes and legality rules (the same declarations that the bundle generator + reads); +- the shape ranges and a size budget; +- the value domains that matter for the op. -### 14.5 Failure handling +The tool reads these declarations. It must not need a second copy of the +legality rules. -1. Record the seed and the full case description (JSON). -2. **Minimize.** Reset one axis to its default, or halve one dimension. Keep the - change if the failure remains. Repeat until no change keeps the failure. -3. Write the result as a concrete sweep case with purpose `regression` (§9), in - the lowest tier that can afford it. -4. If two oracles disagree, report an oracle bug. Consider a G-B golden case. +In return, the tool gives the plans two things: -The case description is the permanent record. A seed reproduces a case only for -the same tool version. +- minimized reproducers, as concrete sweep cases with purpose `regression` (§9); +- oracle bugs, as input to golden selection (G-B, §12.2). -### 14.6 Boundaries +Boundaries: -- Fuzz mode is not a CI tier. The tiers run checked-in graphs only (§8). -- The engine integration gate decides campaign policy. This RFC defines the tool. -- The tool does not fuzz the descriptor loader or matcher (RFC 0017, RFC 0018). - It shares only the seed convention. +- Exhaustive runs are not a CI level. The levels run checked-in graphs only + (§8). +- The tool design, its campaign policy and its schedule are in the companion + RFC, not here. +- Fuzzing of the descriptor loader and matcher is in RFC 0017 and RFC 0018. ## 15. Required Infrastructure | # | Capability | Status | Needed for | |---|---|---|---| -| I1 | **Sweep generator** (plan mode, §14). Reads declared axes, legality, shape sets and tier rules. Writes `sweep.json` for each tier, with all-pairs and seeded rotation. | Missing. The conv sweeps came from C++ tests (`generator: capture-bundles`) or were written by hand (`generator: manual`). | §8 | +| I1 | **Bundle generator.** Reads declared axes, legality, shape sets and level rules. Writes `sweep.json` for each level, with all-pairs and seeded rotation. | Missing. The conv sweeps came from C++ tests (`generator: capture-bundles`) or were written by hand (`generator: manual`). | §8 | | I2 | **`purpose` field** in sweep case metadata. Support-matrix generation excludes probes. | Missing | §9 | | I3 | **Exact-alignment and offset placement** in the harness | Missing. Pointers are allocator-aligned. | §6 | -| I4 | **Value-domain control** for each tensor, with seeds derived for each tensor (§14.3) | Missing. The tensor bundles fix ranges in code. | §6, §14 | -| I5 | **Repeat-and-compare** on bundles | Exists only inside providers | §6 | +| I4 | **Value-domain control** for each tensor: zero-excluded, all-zero, constant, large magnitude, subnormal, NaN and Inf injection. A seed derived for each tensor. | Missing. The tensor bundles fix ranges in code. For example, conv x and w are uniform in [0, 1], with one seed for every tensor. | §6 | +| I5 | **Repeat-and-compare** on bundles (determinism, plan reuse) | Exists only inside providers | §6 | | I6 | **Oracle fixes** for each plan (for example conv `conv_mode`) | Per op | §11 | | I7 | **Golden generator for each op** in `reference-data-scripts/`, writing `meta.json` with criterion and strategy | Batchnorm and SDPA only | §12 | | I8 | **Corpus intake** for M shapes | `migration-scripts/import_graph.py` is the start. | §7 | | I9 | **Cold-cache timing** for each unique shape and arch | Ad hoc | §8 | -| I10 | **Fuzz mode** (§14.4, §14.5): sampling, invariant checks, minimizer, reproducer writer | Missing | §14 | Templates can also need new placeholders. Today, conv `conv_mode`, `compute_data_type` and pointwise `operation` are hard-coded. These changes are @@ -488,17 +476,36 @@ per op. The plan tracks them. ## 16. Rollout +The priority order is attention, then convolution, then normalization (layer, +batch and RMS). Other ops follow as bandwidth allows. + | Phase | Content | Exit | |---|---|---| -| 1 | This RFC. The methodology guide and index under `docs/test-plans/`. | RFC merged | -| 2 | I1, I2. The first per-op plan, in its own PR, regenerates its bundles from the plan. Harvest claims. Fix FFM pins and TOML globs. | Tiers green on supported engines. Each case traces to the plan. | -| 3 | I3–I7, as the first plan needs them. Its golden set. | Reference gaps closed or recorded | -| 4 | I10. Fuzz mode for the first op, on a developer machine and in a scheduled non-gating job. | Fuzz finds a seeded fault. The minimizer writes a reproducer that runs as a `regression` case. | -| 5 | Plans for the other op families, in epic order (attention, normalization, then the rest). Each plan regenerates its bundles. | Each op family has a plan | +| 1 | This RFC. The methodology guide and index under `docs/test-plans/`. Documents updated (§16.1). | RFC merged | +| 2 | I1 and I2. | Generator writes bundles with `purpose` | +| 3 | The attention plan, then the convolution plan, each in its own PR. Each plan regenerates its bundles, harvests claims, and fixes FFM pins and TOML globs. I3–I7 land as these plans need them. | Attention and convolution have plans. This is the minimum before exhaustive-tool work starts. | +| 4 | Normalization plans: layer, batch, RMS. | Each has a plan | +| 5 | Other op families, as bandwidth allows. | Each op family has a plan | + +The exhaustive tool and its RFC run as a separate effort. It starts after +phase 3. Each phase lands on its own. Bundles without a plan keep working until the plan of their op replaces them. +### 16.1 Documents and tickets to update + +When this RFC merges, update these documents: + +| Document | Change | +|---|---| +| `projects/hipdnn/docs/testing/TESTING_STRATEGY.md` | Link this RFC and the plan index. Describe the four levels and their questions (§8.2). | +| `projects/hipdnn/docs/testing/KNOWN_TESTING_GAPS.md` | Link each known gap to the plan that closes it, when that plan exists. | +| `dnn-providers/integration-tests/README.md` and `docs/README.md` | Link the plan index and the plan-to-file mapping (§8.3). | +| `dnn-providers/integration-tests/docs/test-plans/README.md` | The index. Each plan and its RFC link from here. | + +The existing per-op stories under epic ALMIOPEN-2892 must reference this RFC. + ## 17. Alternatives Considered | Alternative | Why rejected | @@ -506,51 +513,51 @@ of their op replaces them. | **Keep curating bundles by hand** | It caused the K = G degeneracy, toy sizes and blind spots of §2. It does not scale to a full plan. | | **Plans keyed to engine support** | They stay green easily but hide API features (§2) and need a rewrite when support changes. P2 and claims replace them. | | **Full cross-products** | The conv-forward product is ~14,000 cases. Most repeat one check (§10). All-pairs, targeted products and a register replace them. | -| **Random sampling in the CI tiers** | Random cases make CI results change from run to run. Fuzz mode (§14) runs outside the tiers. Its reproducers become pinned cases. | -| **Golden data for every case** | It costs storage, has a 2 MiB limit for each case, cannot hold non-finite values, and adds nothing where a trusted live reference exists. Criteria (§12.2) replace it. | -| **Separate tools for bundles and fuzzing** | Two copies of the legality rules would drift. A fuzz failure would need translation into a plan case. One tool with two modes (§14) replaces them. | +| **Random sampling in the CI levels** | Random cases make CI results change from run to run. The exhaustive tool (§14) runs outside the levels. Its reproducers become pinned cases. | +| **Golden data for every case** | It costs storage, has a 2 MiB limit for each case, cannot hold non-finite values, and adds nothing where a trusted live reference exists. A baseline policy and criteria (§12.2) replace it. | ## 18. Risks | Risk | Mitigation | |---|---| | Plan and bundles drift | The generator writes the bundles. The same-PR rule. Reviewers check traceability in both directions. | -| Case counts exceed the budgets | Estimate first (§8). A cold-cache timing run fixes the counts. Per-tier ceilings. | +| Case counts exceed the budgets | Estimate first (§8.2). A cold-cache timing run fixes the counts. Per-level ceilings. | | Probes fill the suite with skips | A probe costs ~1 ms. The `purpose` tag keeps probes out of the support matrix. P10 requires executed cover. | | A missed oracle gap gives wrong verdicts | Review step of §11. Golden G-B cases. Three-way confirmation. | -| Pairwise misses a three-way fault | Corpus shapes. Fuzz mode. Pinned regressions. | -| Case-id changes break name-keyed config | Step 9 of §13.1 updates FFM pins and TOML globs in the same PR. | +| Pairwise misses a three-way fault | Corpus shapes. The exhaustive tool (§14). Pinned regressions. | +| Case-id changes break name-keyed config | Step 10 of §13.1 updates FFM pins and TOML globs in the same PR. | | The methodology guide and this RFC diverge | The guide (`docs/test-plans/methodology.md`) is the working form of §5–§13. Principle changes need an RFC amendment. | -| Fuzz mode reports false failures from a weak oracle | Check each minimized reproducer against both references. Report faults in unmodeled features as "unverified" (§14.4). | -| A seed stops reproducing a case after a tool change | The JSON case description is the permanent record (§14.5). | +| The exhaustive tool needs declarations that plans do not provide | Each plan has an exhaustive-mode section (§13.2). The companion RFC can ask for more fields. | ## 19. Open Questions and Future Work 1. **Where axes and legality live.** A per-op Python module in - `reference-data-scripts/`, or a data file beside the plan? + `reference-data-scripts/`, or a data file beside the plan? The bundle + generator and the exhaustive tool both read them. 2. **Precision strategy for fp16 and bf16 golden cases.** We propose matched precision. Confirm it against the accumulation contracts of all claiming engines. 3. **Source of tolerance calibration.** Per-case `meta.json` (RFC 0011 level 2), or dynamic tolerances? -4. **Full tier and unclaimed dtypes.** Should full-tier products run only on +4. **Full level and unclaimed dtypes.** Should full-level products run only on claimed dtypes, to cut skip-only cases? 5. **The engine integration gate.** Its proposal, and the interface it takes from the plans. -6. **Tool language split.** Plan mode can be Python, like - `reference-data-scripts/`. Value fill must run in the C++ harness. Does the - Python side drive a fuzz run through process calls, or do the two sides share - only the case-description format? -7. **Fuzz job placement.** Which machine and schedule run the non-gating job? - Where do its reports go? +6. **TheRock bump lane.** The suite must run a wide set of cases on each TheRock + bump, each PR and each nightly. Which level runs on a TheRock bump? §8.2 does + not yet say. +7. **Companion exhaustive-tool RFC.** Its owner, its timing, and the fields that + it needs in the exhaustive-mode section. ## 20. Glossary - **Plan.** A per-op document under `docs/test-plans/` that defines the test - sweep of the op. + sweep of the op. It does not depend on an engine. +- **Level.** One of quick, PR, nightly or full (§8.2). Also called a tier. Each + level answers one question. - **Feature.** A schema field, enum value, validation rule or harness property that can change behavior. -- **Core axes.** The axes that each tier covers completely. Typically rank × +- **Core axes.** The axes that each level covers completely. Typically rank × layout × dtype. - **All-pairs.** A set of cases in which each value of each axis meets each value of each other axis at least once. @@ -561,5 +568,6 @@ of their op replaces them. - **Oracle.** Anything that produces the expected output: golden data, the GPU reference or the CPU reference. - **G-A … G-F.** Golden selection criteria (§12.2). -- **Test generator tool.** The tool of §14, with plan mode and fuzz mode. -- **Value domain.** The class of values that fills a tensor (§14.3). +- **Bundle generator.** The tool that writes bundles from a plan (I1). +- **Exhaustive mode.** The section of a plan that declares the hook for the + separate exhaustive tool (§14). From 75ea253c92ea537adbf35129c51b7b8025018aa6 Mon Sep 17 00:00:00 2001 From: Adam Dickin Date: Mon, 26 Oct 2026 12:48:05 -0600 Subject: [PATCH 4/5] docs(hipdnn): renumber per-op test plans RFC to 0022, link refs, require plans for new ops - Rename to 0022 (0021 is taken by kernel slots, #13839). - Add the status header and the index row. - Fix the RFC 0011 link (0011_BundleTestFramework.md). - Link every section and RFC reference. - Add 16.2: a new op cannot land without a test plan and its bundles. --- ...rOpTestPlans.md => 0022_PerOpTestPlans.md} | 183 ++++++++++-------- projects/hipdnn/docs/rfcs/README.md | 1 + 2 files changed, 106 insertions(+), 78 deletions(-) rename projects/hipdnn/docs/rfcs/{0021_PerOpTestPlans.md => 0022_PerOpTestPlans.md} (76%) diff --git a/projects/hipdnn/docs/rfcs/0021_PerOpTestPlans.md b/projects/hipdnn/docs/rfcs/0022_PerOpTestPlans.md similarity index 76% rename from projects/hipdnn/docs/rfcs/0021_PerOpTestPlans.md rename to projects/hipdnn/docs/rfcs/0022_PerOpTestPlans.md index 66701a4176f4..2f8d72862db0 100644 --- a/projects/hipdnn/docs/rfcs/0021_PerOpTestPlans.md +++ b/projects/hipdnn/docs/rfcs/0022_PerOpTestPlans.md @@ -1,6 +1,9 @@ -# RFC 0021: Per-Op Test Plans +# RFC 0022: Per-Op Test Plans - Contributors: Adam Dickin +- **Status**: Planned +- **Status notes**: Proposed. No per-op plan, bundle generator or `purpose` field exists yet. +- **Last verified**: 2026-10-26 against `develop` (`3f9cd39c31c`) ## Table of Contents @@ -28,7 +31,7 @@ ## 1. Executive Summary The integration suite (`dnn-providers/integration-tests`) tests each op with -template-sweep bundles ([RFC 0011](0011_GoldenReferenceValidation.md)). It +template-sweep bundles ([RFC 0011](0011_BundleTestFramework.md)). It records engine support as claims ([RFC 0015](0015_EngineSupportClaims.md)). It has no rule for **which cases a bundle holds**. Today, bundles hold cases captured from older C++ tests, then re-tiered. No case has a stated reason. @@ -46,14 +49,15 @@ All plans follow one method: - Give each case a purpose: `claim`, `probe` or `regression`. - Record each product that we do not run in a trade-off register. - Put each case in one of four levels (quick, PR, nightly, full). Each level - answers one question (§8). + answers one question ([§8](#8-sweep-construction-levels-and-lanes)). - Select a small golden set with fixed criteria. An independent third party generates it. - Write the bundles from the plan with a generator, so that bundles and plan cannot drift. +- Require a plan and its bundles for every new op before the op lands ([§16.2](#162-new-ops)). Exhaustive testing is **not** part of this RFC. It is a separate tool with its -own RFC. Each plan only declares the hook that the tool reads (§14). +own RFC. Each plan only declares the hook that the tool reads ([§14](#14-exhaustive-mode-and-the-separate-tool)). ## 2. Problem Statement @@ -82,26 +86,28 @@ each op depend on the tests that the op happened to have. ### 3.1 In scope -- The **method** of every per-op plan (§5–§13). +- The **method** of every per-op plan ([§5](#5-principles)–[§13](#13-authoring-process-and-plan-template)). - The **plan document**: location, template, and the rule that bundles and plan stay in sync. - The **four levels**, the question that each answers, and the **mapping** from - a plan to bundles, claims, TOML, tier YAML and CTest lanes (§8). -- **Golden policy and selection** (§12), building on RFC 0011. -- The **hook** that each plan declares for the exhaustive tool (§14). -- The **infrastructure** that the method needs (§15), its rollout and the - documents to update (§16). + a plan to bundles, claims, TOML, tier YAML and CTest lanes ([§8](#8-sweep-construction-levels-and-lanes)). +- **Golden policy and selection** ([§12](#12-golden-tests)), building on [RFC 0011](0011_BundleTestFramework.md). +- The **hook** that each plan declares for the exhaustive tool ([§14](#14-exhaustive-mode-and-the-separate-tool)). +- The **infrastructure** that the method needs ([§15](#15-required-infrastructure)), its rollout and the + documents to update ([§16](#16-rollout)). +- The **test-plan requirement for new ops** ([§16.2](#162-new-ops)). ### 3.2 Out of scope | Item | Where it belongs | |---|---| -| Exhaustive test generation, fuzzing, invalid-graph generation, engine-knob sweeps | A separate tool, with its own RFC. §14 states only the interface. | +| Exhaustive test generation, fuzzing, invalid-graph generation, engine-knob sweeps | A separate tool, with its own RFC. [§14](#14-exhaustive-mode-and-the-separate-tool) states only the interface. | | Gating a new engine integration: per-case outcome reporting, running every kernel of an engine, the days-long gauntlet tier, review of agent-authored integrations | A separate engine-integration-gate proposal | -| Writing the per-op plans and bundles | One PR for each op (§16) | -| Bundle format, verification modes, DVC layout | RFC 0011 | -| Claim schema, enforcement, harvesting | RFC 0015 | -| Fuzzing of the descriptor loader and matcher | RFC 0017, RFC 0018 | +| Writing the per-op plans and bundles | One PR for each op ([§16](#16-rollout)) | +| Everything else that adding an op needs (schema, references, engine support) | The work that adds the op. [§16.2](#162-new-ops) states only the test requirement. | +| Bundle format, verification modes, DVC layout | [RFC 0011](0011_BundleTestFramework.md) | +| Claim schema, enforcement, harvesting | [RFC 0015](0015_EngineSupportClaims.md) | +| Fuzzing of the descriptor loader and matcher | [RFC 0017](0017_UniversalKernelDescriptor.md), [RFC 0018](0018_UniversalMatchDescriptor.md) | | Kernel-internal testing | The project that owns the kernel | | Release-train testing | Release readiness | @@ -127,7 +133,7 @@ flowchart LR lists the plans and their bundles. - **The plan is engine-independent.** It tests the op. It has no engine matrix. Claims (`support.json`) and per-engine TOML apply the plan to each engine - (§8.4). + ([§8.4](#84-from-plan-to-engine)). - **The plan is the source of truth.** The generator reads the declared axes, legality rules, shape catalog and level rules, and writes the bundles. Nobody edits bundles by hand. @@ -142,14 +148,14 @@ flowchart LR | # | Principle | Practice | |---|---|---| | P1 | **Derive features from the API.** | Each field of the op schema (`flatbuffers_sdk/schemas/*.fbs`) is a feature. So is each rule in `pre_validate_node` and `infer_properties_node` of the frontend node. Tests and engine capabilities do not define the op. | -| P2 | **The plan does not depend on an engine.** | The plan tests the op. Engines join through claims. A feature that no engine supports stays as a probe. When support arrives, the run reports `UNCLAIMED_SUPPORT` (RFC 0015). The plan needs no edit. | +| P2 | **The plan does not depend on an engine.** | The plan tests the op. Engines join through claims. A feature that no engine supports stays as a probe. When support arrives, the run reports `UNCLAIMED_SUPPORT` ([RFC 0015](0015_EngineSupportClaims.md)). The plan needs no edit. | | P3 | **Divide each axis into classes. Then add boundaries and hazards.** | Classes: 1, odd, small, power of two, large, not a multiple of the vector width. Boundaries: output size 1, padding ≥ filter, remainders, values near the dtype limit. Hazards: powers of 2 ± 1, dims not divisible by 4 or 8, int32 index overflow. | -| P4 | **One feature at a time. Then all-pairs. Then targeted products.** | One-feature shapes show the cause of a failure. All-pairs finds most interaction faults. Run a complete product only where an interaction is plausible. The corpus and the exhaustive tool (§14) find three-way faults. Pin them. | +| P4 | **One feature at a time. Then all-pairs. Then targeted products.** | One-feature shapes show the cause of a failure. All-pairs finds most interaction faults. Run a complete product only where an interaction is plausible. The corpus and the exhaustive tool ([§14](#14-exhaustive-mode-and-the-separate-tool)) find three-way faults. Pin them. | | P5 | **Size grows with the level.** | Nightly tests each feature again at medium size. | -| P6 | **Each product that we do not run is a recorded decision.** | Use the trade-off register (§10). Never trim rejection probes. | -| P7 | **A case needs an oracle that models it.** | See §11. A case with no oracle is "not run", never green. | +| P6 | **Each product that we do not run is a recorded decision.** | Use the trade-off register ([§10](#10-trade-off-register)). Never trim rejection probes. | +| P7 | **A case needs an oracle that models it.** | See [§11](#11-oracles-and-reference-prerequisites). A case with no oracle is "not run", never green. | | P8 | **Count distinct behaviors, not cases.** | Report distinct geometries, feature values and skip-only cases. | -| P9 | **Each case has a purpose.** | `claim`, `probe` or `regression` (§9). | +| P9 | **Each case has a purpose.** | `claim`, `probe` or `regression` ([§9](#9-case-purpose)). | | P10 | **Covered means executed.** | A case that every engine skipped proves nothing. Quick needs one executed case for each feature × engine. | | P11 | **Realism and coverage have different sources.** | Model layers and the benchmark corpus show what users run. Generated cases, hazards and probes reach what users have not run. | @@ -170,7 +176,7 @@ A plan lists every feature of the op, in these groups: | Invalid inputs | One frontend **unit test** for each validation rule. These are not sweep cases. | For each feature, the plan records the values to cover, the earliest level, the -oracle readiness (§11) and the purpose of its cases (§9). +oracle readiness ([§11](#11-oracles-and-reference-prerequisites)) and the purpose of its cases ([§9](#9-case-purpose)). **Legality is declared once for each op.** It copies the frontend validation. The generator never writes an illegal graph. @@ -217,7 +223,7 @@ so in the default shape. | Full | Complete products only where an interaction is plausible. | | Probes | Each unsupported feature, awkward combination and hazard has a probe. Each level has rejection probes. | | Regressions | The reproducer of each escaped bug is a concrete case in the lowest level that can afford it. | -| Concrete graphs | Each level runs checked-in graphs only. Random sampling belongs to the exhaustive tool (§14). | +| Concrete graphs | Each level runs checked-in graphs only. Random sampling belongs to the exhaustive tool ([§14](#14-exhaustive-mode-and-the-separate-tool)). | ### 8.2 The four levels @@ -244,8 +250,8 @@ execution. |---|---|---| | Op topology | `{level}/{Op}/{Topology}/graph.template.json` | The bundle generator | | Cases: axes, shapes, level, purpose | `sweep.json` (cases and per-case metadata) in the directory of the level | The bundle generator | -| Golden cases | `golden/{CaseId}/tensors.dvc` (RFC 0011) | The golden generator | -| Claims: which engine supports which case | `support.json`, or `{Name}.support.json` for a single graph (RFC 0015) | Harvested with `--write-support-claims`. Never hand-written. | +| Golden cases | `golden/{CaseId}/tensors.dvc` ([RFC 0011](0011_BundleTestFramework.md)) | The golden generator | +| Claims: which engine supports which case | `support.json`, or `{Name}.support.json` for a single graph ([RFC 0015](0015_EngineSupportClaims.md)) | Harvested with `--write-support-claims`. Never hand-written. | | Tolerances, validator overrides, skips for an engine | `dnn-providers//config/.toml` | The provider, for each engine | | Which suites each lane runs | Level YAML (`test_categories_integration.yaml`) in each provider. Its categories become CTest labels. | The provider | | Cascade of levels | Each YAML category lists the patterns of the lower levels. CTest labels do not inherit. | The provider | @@ -263,7 +269,7 @@ Each generated case has a `purpose` in its sweep metadata: | Purpose | Expected outcome | Claims | |---|---|---| -| `claim` | The claiming engine accepts and is correct. | `--write-support-claims` writes them (RFC 0015). | +| `claim` | The claiming engine accepts and is correct. | `--write-support-claims` writes them ([RFC 0015](0015_EngineSupportClaims.md)). | | `probe` | A typed rejection, or a correct accept. | Never written. The published support matrix excludes probes. | | `regression` | Same as the original bug case. | Same as the original case. | @@ -283,7 +289,7 @@ Rejection probes are never in the register. ## 11. Oracles and Reference Prerequisites -Verification follows RFC 0011. In `auto` mode, the order is golden data, GPU +Verification follows [RFC 0011](0011_BundleTestFramework.md). In `auto` mode, the order is golden data, GPU reference, CPU reference. Before you write cases for a feature, confirm that each oracle **models** it. List each gap with its fix. Typical gaps: @@ -293,10 +299,10 @@ each oracle **models** it. List each gap with its fix. Typical gaps: | Asymmetric parameters tested in one reference only | Conv asymmetric padding | | Types with no reference | fp8 and int8 conv | | Special values | NaN and Inf need a validator that matches infinities. | -| Harness support | Exact-alignment placement, value-domain control, repeat-and-compare (§15) | +| Harness support | Exact-alignment placement, value-domain control, repeat-and-compare ([§15](#15-required-infrastructure)) | A feature with no oracle stays in the plan and is marked "not run". A golden case -can cover it (§12, criterion G-B). +can cover it ([§12](#12-golden-tests), criterion G-B). ## 12. Golden Tests @@ -321,7 +327,7 @@ trustworthy. **The policy is a baseline, not an exhaustive set.** Each op gets the baseline third-party check (G-A, with PyTorch or a similar third party). The other criteria add a case only where that criterion applies. Exhaustive golden -coverage is not a goal. The separate exhaustive tool (§14) covers the wide +coverage is not a goal. The separate exhaustive tool ([§14](#14-exhaustive-mode-and-the-separate-tool)) covers the wide space. A golden case meets at least one criterion. Most ops need 6–20 golden cases. @@ -346,7 +352,7 @@ semantics cannot map to hipDNN semantics. default. A hipDNN reference never generates golden data. - **Precision strategy.** Set it for each op in `meta.json`. Use precision uplift for shallow ops and fp32. Use matched precision (the accumulation - contract of the engine) for deep ops and low-precision types (RFC 0011). + contract of the engine) for deep ops and low-precision types ([RFC 0011](0011_BundleTestFramework.md)). - **Provenance.** Pin the third-party version. Seed the inputs. Record `forward_source` for backward ops in `meta.json`. - **Tolerance.** Calibrate it for each case from the observed error of both @@ -373,18 +379,18 @@ disagreement. ### 13.1 Process -1. **Inventory and legality** from the API (§6). +1. **Inventory and legality** from the API ([§6](#6-feature-inventory)). 2. **Audit** the bundles at the tip of the integration branch: - map uids to roles through the template; - derive the real parameters from the dims; - count distinct behaviors, skip-only cases and per-level cover. -3. **Shape catalog** (§7). -4. **Combination and level rules** (§8). +3. **Shape catalog** ([§7](#7-shape-catalog-and-size-ladder)). +4. **Combination and level rules** ([§8](#8-sweep-construction-levels-and-lanes)). 5. **Estimate** counts, FLOP and oracle cost for each level. -6. **Trade-off register** (§10). -7. **Oracle prerequisites** (§11). -8. **Golden selection** (§12). -9. **Exhaustive-mode hook** (§14). +6. **Trade-off register** ([§10](#10-trade-off-register)). +7. **Oracle prerequisites** ([§11](#11-oracles-and-reference-prerequisites)). +8. **Golden selection** ([§12](#12-golden-tests)). +9. **Exhaustive-mode hook** ([§14](#14-exhaustive-mode-and-the-separate-tool)). 10. **Generate, time, harvest and finalize:** - generate the bundles and tag each case purpose; - time a cold-cache run and fix the counts; @@ -406,7 +412,7 @@ Scope, bundles, out of scope. ## 7. Exhaustive mode states that the op has one; axes, legality, shape ranges, value domains for the exhaustive tool ``` -A plan has no engine matrix (§8.4). +A plan has no engine matrix ([§8.4](#84-from-plan-to-engine)). ### 13.3 Review checklist @@ -421,7 +427,7 @@ A plan has no engine matrix (§8.4). rejection probes. - [ ] Each oracle gap is listed with its fix. - [ ] Counts, FLOP and oracle cost fit the level budgets. -- [ ] Golden cases follow §12.2 and are confirmed. +- [ ] Golden cases follow [§12.2](#122-policy-and-selection-criteria) and are confirmed. - [ ] The exhaustive-mode section is present. - [ ] Bundle cases and plan items map in both directions. @@ -430,10 +436,10 @@ A plan has no engine matrix (§8.4). **This RFC does not design exhaustive testing.** Exhaustive test generation, fuzzing, invalid-graph generation and engine-knob sweeps belong to a separate tool. That tool has its own RFC. Work on it starts after the attention and -convolution plans exist (§16). +convolution plans exist ([§16](#16-rollout)). This RFC defines only the **hook**. Each plan has an exhaustive-mode section -(§13.2). It states that the op has an exhaustive mode, and it declares: +([§13.2](#132-template)). It states that the op has an exhaustive mode, and it declares: - the axes and legality rules (the same declarations that the bundle generator reads); @@ -445,30 +451,30 @@ legality rules. In return, the tool gives the plans two things: -- minimized reproducers, as concrete sweep cases with purpose `regression` (§9); -- oracle bugs, as input to golden selection (G-B, §12.2). +- minimized reproducers, as concrete sweep cases with purpose `regression` ([§9](#9-case-purpose)); +- oracle bugs, as input to golden selection (G-B, [§12.2](#122-policy-and-selection-criteria)). Boundaries: - Exhaustive runs are not a CI level. The levels run checked-in graphs only - (§8). + ([§8](#8-sweep-construction-levels-and-lanes)). - The tool design, its campaign policy and its schedule are in the companion RFC, not here. -- Fuzzing of the descriptor loader and matcher is in RFC 0017 and RFC 0018. +- Fuzzing of the descriptor loader and matcher is in [RFC 0017](0017_UniversalKernelDescriptor.md) and [RFC 0018](0018_UniversalMatchDescriptor.md). ## 15. Required Infrastructure | # | Capability | Status | Needed for | |---|---|---|---| -| I1 | **Bundle generator.** Reads declared axes, legality, shape sets and level rules. Writes `sweep.json` for each level, with all-pairs and seeded rotation. | Missing. The conv sweeps came from C++ tests (`generator: capture-bundles`) or were written by hand (`generator: manual`). | §8 | -| I2 | **`purpose` field** in sweep case metadata. Support-matrix generation excludes probes. | Missing | §9 | -| I3 | **Exact-alignment and offset placement** in the harness | Missing. Pointers are allocator-aligned. | §6 | -| I4 | **Value-domain control** for each tensor: zero-excluded, all-zero, constant, large magnitude, subnormal, NaN and Inf injection. A seed derived for each tensor. | Missing. The tensor bundles fix ranges in code. For example, conv x and w are uniform in [0, 1], with one seed for every tensor. | §6 | -| I5 | **Repeat-and-compare** on bundles (determinism, plan reuse) | Exists only inside providers | §6 | -| I6 | **Oracle fixes** for each plan (for example conv `conv_mode`) | Per op | §11 | -| I7 | **Golden generator for each op** in `reference-data-scripts/`, writing `meta.json` with criterion and strategy | Batchnorm and SDPA only | §12 | -| I8 | **Corpus intake** for M shapes | `migration-scripts/import_graph.py` is the start. | §7 | -| I9 | **Cold-cache timing** for each unique shape and arch | Ad hoc | §8 | +| I1 | **Bundle generator.** Reads declared axes, legality, shape sets and level rules. Writes `sweep.json` for each level, with all-pairs and seeded rotation. | Missing. The conv sweeps came from C++ tests (`generator: capture-bundles`) or were written by hand (`generator: manual`). | [§8](#8-sweep-construction-levels-and-lanes) | +| I2 | **`purpose` field** in sweep case metadata. Support-matrix generation excludes probes. | Missing | [§9](#9-case-purpose) | +| I3 | **Exact-alignment and offset placement** in the harness | Missing. Pointers are allocator-aligned. | [§6](#6-feature-inventory) | +| I4 | **Value-domain control** for each tensor: zero-excluded, all-zero, constant, large magnitude, subnormal, NaN and Inf injection. A seed derived for each tensor. | Missing. The tensor bundles fix ranges in code. For example, conv x and w are uniform in [0, 1], with one seed for every tensor. | [§6](#6-feature-inventory) | +| I5 | **Repeat-and-compare** on bundles (determinism, plan reuse) | Exists only inside providers | [§6](#6-feature-inventory) | +| I6 | **Oracle fixes** for each plan (for example conv `conv_mode`) | Per op | [§11](#11-oracles-and-reference-prerequisites) | +| I7 | **Golden generator for each op** in `reference-data-scripts/`, writing `meta.json` with criterion and strategy | Batchnorm and SDPA only | [§12](#12-golden-tests) | +| I8 | **Corpus intake** for M shapes | `migration-scripts/import_graph.py` is the start. | [§7](#7-shape-catalog-and-size-ladder) | +| I9 | **Cold-cache timing** for each unique shape and arch | Ad hoc | [§8](#8-sweep-construction-levels-and-lanes) | Templates can also need new placeholders. Today, conv `conv_mode`, `compute_data_type` and pointwise `operation` are hard-coded. These changes are @@ -481,7 +487,7 @@ batch and RMS). Other ops follow as bandwidth allows. | Phase | Content | Exit | |---|---|---| -| 1 | This RFC. The methodology guide and index under `docs/test-plans/`. Documents updated (§16.1). | RFC merged | +| 1 | This RFC. The methodology guide and index under `docs/test-plans/`. Documents updated ([§16.1](#161-documents-and-tickets-to-update)). | RFC merged | | 2 | I1 and I2. | Generator writes bundles with `purpose` | | 3 | The attention plan, then the convolution plan, each in its own PR. Each plan regenerates its bundles, harvests claims, and fixes FFM pins and TOML globs. I3–I7 land as these plans need them. | Attention and convolution have plans. This is the minimum before exhaustive-tool work starts. | | 4 | Normalization plans: layer, batch, RMS. | Each has a plan | @@ -491,7 +497,7 @@ The exhaustive tool and its RFC run as a separate effort. It starts after phase 3. Each phase lands on its own. Bundles without a plan keep working until the plan -of their op replaces them. +of their op replaces them. After phase 2, every new op follows [§16.2](#162-new-ops). ### 16.1 Documents and tickets to update @@ -499,35 +505,56 @@ When this RFC merges, update these documents: | Document | Change | |---|---| -| `projects/hipdnn/docs/testing/TESTING_STRATEGY.md` | Link this RFC and the plan index. Describe the four levels and their questions (§8.2). | +| `projects/hipdnn/docs/testing/TESTING_STRATEGY.md` | Link this RFC and the plan index. Describe the four levels and their questions ([§8.2](#82-the-four-levels)). | | `projects/hipdnn/docs/testing/KNOWN_TESTING_GAPS.md` | Link each known gap to the plan that closes it, when that plan exists. | -| `dnn-providers/integration-tests/README.md` and `docs/README.md` | Link the plan index and the plan-to-file mapping (§8.3). | -| `dnn-providers/integration-tests/docs/test-plans/README.md` | The index. Each plan and its RFC link from here. | +| `dnn-providers/integration-tests/README.md` and `docs/README.md` | Link the plan index and the plan-to-file mapping ([§8.3](#83-mapping-a-plan-to-files)). | +| `dnn-providers/integration-tests/docs/test-plans/README.md` | The index. Each plan and its RFC link from here. State the new-op rule ([§16.2](#162-new-ops)). | +| `dnn-providers/integration-tests/docs/adding-tests.md` | State that a new op needs a plan and its bundles before it lands ([§16.2](#162-new-ops)). | The existing per-op stories under epic ALMIOPEN-2892 must reference this RFC. +### 16.2 New ops + +A new op cannot land without a test plan and the bundles built from it. The +change that adds an op to the graph API also adds, before it merges: + +- a reviewed plan, `docs/test-plans/.md`, that follows [§5](#5-principles)–[§13](#13-authoring-process-and-plan-template) and the + checklist of [§13.3](#133-review-checklist); +- the bundles that the generator writes from that plan, for every level, with a + purpose on each case; +- the G-A golden cases ([§12.2](#122-policy-and-selection-criteria)); +- the oracle gaps of the op, each with its fix ([§11](#11-oracles-and-reference-prerequisites)). + +The plan does not depend on an engine (P2). An op can land before any engine +supports it. Its unsupported features stay as probes, and claims arrive with +engine support. + +This RFC states only the test requirement. It does not define the rest of the +work to add an op. The rule applies once the bundle generator exists (phase 2), +because nobody writes bundles by hand ([§4](#4-design-overview)). + ## 17. Alternatives Considered | Alternative | Why rejected | |---|---| -| **Keep curating bundles by hand** | It caused the K = G degeneracy, toy sizes and blind spots of §2. It does not scale to a full plan. | -| **Plans keyed to engine support** | They stay green easily but hide API features (§2) and need a rewrite when support changes. P2 and claims replace them. | -| **Full cross-products** | The conv-forward product is ~14,000 cases. Most repeat one check (§10). All-pairs, targeted products and a register replace them. | -| **Random sampling in the CI levels** | Random cases make CI results change from run to run. The exhaustive tool (§14) runs outside the levels. Its reproducers become pinned cases. | -| **Golden data for every case** | It costs storage, has a 2 MiB limit for each case, cannot hold non-finite values, and adds nothing where a trusted live reference exists. A baseline policy and criteria (§12.2) replace it. | +| **Keep curating bundles by hand** | It caused the K = G degeneracy, toy sizes and blind spots of [§2](#2-problem-statement). It does not scale to a full plan. | +| **Plans keyed to engine support** | They stay green easily but hide API features ([§2](#2-problem-statement)) and need a rewrite when support changes. P2 and claims replace them. | +| **Full cross-products** | The conv-forward product is ~14,000 cases. Most repeat one check ([§10](#10-trade-off-register)). All-pairs, targeted products and a register replace them. | +| **Random sampling in the CI levels** | Random cases make CI results change from run to run. The exhaustive tool ([§14](#14-exhaustive-mode-and-the-separate-tool)) runs outside the levels. Its reproducers become pinned cases. | +| **Golden data for every case** | It costs storage, has a 2 MiB limit for each case, cannot hold non-finite values, and adds nothing where a trusted live reference exists. A baseline policy and criteria ([§12.2](#122-policy-and-selection-criteria)) replace it. | ## 18. Risks | Risk | Mitigation | |---|---| | Plan and bundles drift | The generator writes the bundles. The same-PR rule. Reviewers check traceability in both directions. | -| Case counts exceed the budgets | Estimate first (§8.2). A cold-cache timing run fixes the counts. Per-level ceilings. | +| Case counts exceed the budgets | Estimate first ([§8.2](#82-the-four-levels)). A cold-cache timing run fixes the counts. Per-level ceilings. | | Probes fill the suite with skips | A probe costs ~1 ms. The `purpose` tag keeps probes out of the support matrix. P10 requires executed cover. | -| A missed oracle gap gives wrong verdicts | Review step of §11. Golden G-B cases. Three-way confirmation. | -| Pairwise misses a three-way fault | Corpus shapes. The exhaustive tool (§14). Pinned regressions. | -| Case-id changes break name-keyed config | Step 10 of §13.1 updates FFM pins and TOML globs in the same PR. | -| The methodology guide and this RFC diverge | The guide (`docs/test-plans/methodology.md`) is the working form of §5–§13. Principle changes need an RFC amendment. | -| The exhaustive tool needs declarations that plans do not provide | Each plan has an exhaustive-mode section (§13.2). The companion RFC can ask for more fields. | +| A missed oracle gap gives wrong verdicts | Review step of [§11](#11-oracles-and-reference-prerequisites). Golden G-B cases. Three-way confirmation. | +| Pairwise misses a three-way fault | Corpus shapes. The exhaustive tool ([§14](#14-exhaustive-mode-and-the-separate-tool)). Pinned regressions. | +| Case-id changes break name-keyed config | Step 10 of [§13.1](#131-process) updates FFM pins and TOML globs in the same PR. | +| The methodology guide and this RFC diverge | The guide (`docs/test-plans/methodology.md`) is the working form of [§5](#5-principles)–[§13](#13-authoring-process-and-plan-template). Principle changes need an RFC amendment. | +| The exhaustive tool needs declarations that plans do not provide | Each plan has an exhaustive-mode section ([§13.2](#132-template)). The companion RFC can ask for more fields. | ## 19. Open Questions and Future Work @@ -537,14 +564,14 @@ The existing per-op stories under epic ALMIOPEN-2892 must reference this RFC. 2. **Precision strategy for fp16 and bf16 golden cases.** We propose matched precision. Confirm it against the accumulation contracts of all claiming engines. -3. **Source of tolerance calibration.** Per-case `meta.json` (RFC 0011 level 2), +3. **Source of tolerance calibration.** Per-case `meta.json` ([RFC 0011](0011_BundleTestFramework.md) level 2), or dynamic tolerances? 4. **Full level and unclaimed dtypes.** Should full-level products run only on claimed dtypes, to cut skip-only cases? 5. **The engine integration gate.** Its proposal, and the interface it takes from the plans. 6. **TheRock bump lane.** The suite must run a wide set of cases on each TheRock - bump, each PR and each nightly. Which level runs on a TheRock bump? §8.2 does + bump, each PR and each nightly. Which level runs on a TheRock bump? [§8.2](#82-the-four-levels) does not yet say. 7. **Companion exhaustive-tool RFC.** Its owner, its timing, and the fields that it needs in the exhaustive-mode section. @@ -553,7 +580,7 @@ The existing per-op stories under epic ALMIOPEN-2892 must reference this RFC. - **Plan.** A per-op document under `docs/test-plans/` that defines the test sweep of the op. It does not depend on an engine. -- **Level.** One of quick, PR, nightly or full (§8.2). Also called a tier. Each +- **Level.** One of quick, PR, nightly or full ([§8.2](#82-the-four-levels)). Also called a tier. Each level answers one question. - **Feature.** A schema field, enum value, validation rule or harness property that can change behavior. @@ -561,13 +588,13 @@ The existing per-op stories under epic ALMIOPEN-2892 must reference this RFC. layout × dtype. - **All-pairs.** A set of cases in which each value of each axis meets each value of each other axis at least once. -- **Probe, claim, regression.** The purposes of a case (§9). -- **S, L, M, H, X.** Small, medium, model, hazard and stress shape sets (§7). +- **Probe, claim, regression.** The purposes of a case ([§9](#9-case-purpose)). +- **S, L, M, H, X.** Small, medium, model, hazard and stress shape sets ([§7](#7-shape-catalog-and-size-ladder)). - **Trade-off register.** The table in a plan of products that we do not run on purpose. - **Oracle.** Anything that produces the expected output: golden data, the GPU reference or the CPU reference. -- **G-A … G-F.** Golden selection criteria (§12.2). +- **G-A … G-F.** Golden selection criteria ([§12.2](#122-policy-and-selection-criteria)). - **Bundle generator.** The tool that writes bundles from a plan (I1). - **Exhaustive mode.** The section of a plan that declares the hook for the - separate exhaustive tool (§14). + separate exhaustive tool ([§14](#14-exhaustive-mode-and-the-separate-tool)). diff --git a/projects/hipdnn/docs/rfcs/README.md b/projects/hipdnn/docs/rfcs/README.md index a3859e0d6b12..85f0657bad08 100644 --- a/projects/hipdnn/docs/rfcs/README.md +++ b/projects/hipdnn/docs/rfcs/README.md @@ -53,3 +53,4 @@ Update the status, the notes, and the verification date in the same change that | [0018](./0018_UniversalMatchDescriptor.md) | Universal match descriptor (UMD criteria) | In progress | | [0019](./0019_UniversalHeuristicDescriptor.md) | Universal heuristic descriptor (UHD) | In progress | | [0020](./0020_UniversalEngineDescriptor.md) | Universal engine descriptor (UED) | In progress | +| [0022](./0022_PerOpTestPlans.md) | Per-op test plans | Planned | From abf98876083aa08dbd0411f8030af91b8899eb86 Mon Sep 17 00:00:00 2001 From: Adam Dickin Date: Wed, 28 Oct 2026 13:47:11 -0600 Subject: [PATCH 5/5] docs(hipdnn): address RFC 0022 review feedback (ALMIOPEN-2947) Declarations are the source of truth for bundles. Rejection coverage goes beyond frontend validation via expected_rejection bundles; a rejection gap is an engine applicability bug (fix or quarantine). Probe is tolerant, expected_rejection strict. Optional motivation note for engine-specific cases. Wall-time ceilings and default-ratio carve-out in section 7. FP64 uplift is the default golden precision. SDPA -inf stats exception. Fused-topology reference gap. --- .../hipdnn/docs/rfcs/0022_PerOpTestPlans.md | 77 +++++++++++++++---- 1 file changed, 63 insertions(+), 14 deletions(-) diff --git a/projects/hipdnn/docs/rfcs/0022_PerOpTestPlans.md b/projects/hipdnn/docs/rfcs/0022_PerOpTestPlans.md index 2f8d72862db0..2ac2ad5c2c31 100644 --- a/projects/hipdnn/docs/rfcs/0022_PerOpTestPlans.md +++ b/projects/hipdnn/docs/rfcs/0022_PerOpTestPlans.md @@ -134,9 +134,12 @@ flowchart LR - **The plan is engine-independent.** It tests the op. It has no engine matrix. Claims (`support.json`) and per-engine TOML apply the plan to each engine ([§8.4](#84-from-plan-to-engine)). -- **The plan is the source of truth.** The generator reads the declared axes, - legality rules, shape catalog and level rules, and writes the bundles. Nobody - edits bundles by hand. +- **The declarations are the source of truth for the bundles.** The generator + reads the declared axes, legality rules, shape catalog and level rules, and + writes the bundles. Nobody edits bundles by hand. The plan document records + the approach and the reason for each case. It must trace to those + declarations, but it is not itself the input of the generator. Where the + declarations live is open question 1 ([§19](#19-open-questions-and-future-work)). - **Traceability in both directions.** Each bundle case maps to a plan item. Each plan item maps to at least one case. A PR that adds, moves or deletes cases updates the plan in the same PR. @@ -173,7 +176,16 @@ A plan lists every feature of the op, in these groups: | Awkward combinations | Legal but hazardous graphs: inputs of different dtypes, tensors in different layouts, degenerate ratios | | Values | Uniform, zero-excluded, all-zero, constant and large-magnitude fills. NaN, Inf, subnormals. Accumulation length. Determinism. | | Execution | Workspace limits, plan reuse with new pointers, serialization round trip ([RFC 0009](0009_CompiledPlanSerialization.md)) | -| Invalid inputs | One frontend **unit test** for each validation rule. These are not sweep cases. | +| Invalid inputs | A rule that only the frontend enforces: one frontend **unit test** for each rule. A rejection that an engine must enforce: an `expected_rejection` bundle ([§9](#9-case-purpose)). | + +**Rejection coverage is not limited to what the frontend validates today.** The +plan lists every combination that the API implies must be rejected, including +combinations that the frontend accepts now. Each rejection that is visible to an +engine is an `expected_rejection` bundle: the engine under test must reject it, +and an engine that accepts it fails the lane. A rejection that an engine does +not enforce is an applicability bug in that engine. Fix the engine, or +quarantine the bundle in the provider's tier YAML with a comment and a ticket. +There is no soft "known gap" state. For each feature, the plan records the values to cover, the earliest level, the oracle readiness ([§11](#11-oracles-and-reference-prerequisites)) and the purpose of its cases ([§9](#9-case-purpose)). @@ -199,7 +211,12 @@ pointer *less* aligned than declared and expects `BAD_PARAM`. | **X** stress | Limits. | Huge batch, int32 overflow, very long accumulation. | **Defaults are non-degenerate.** Each dimension or ratio that can exceed 1 does -so in the default shape. +so in the default shape. One exception: a ratio whose non-unit values select a +separate code path (for example grouped heads or unequal head dims in +attention). In the default shape, one defect there would fail every S and L +case, and S and L would stop isolating one feature. Keep that ratio at 1 in the +default. Cover its non-unit values as their own S, H and L cases and as +all-pairs axes. **Per-level ceilings.** Each plan tunes these starting values. @@ -210,6 +227,10 @@ so in the default shape. | nightly | ≤ 8 GFLOP, ≤ 64 MB | S, L, M, H | | full | none | all, including X | +A plan may state a ceiling as measured wall time instead of FLOP and bytes when +the CPU reference dominates the case time. Fit the budget from a cost model +that was measured on one architecture. + ## 8. Sweep Construction, Levels and Lanes ### 8.1 Sweep rules @@ -221,7 +242,7 @@ so in the default shape. | One at a time | S and L shapes isolate one feature. | | All-pairs | By the end of nightly, each value of each axis meets each value of each other axis. A seeded generator makes this from the declared axes and legality. | | Full | Complete products only where an interaction is plausible. | -| Probes | Each unsupported feature, awkward combination and hazard has a probe. Each level has rejection probes. | +| Probes | Each unsupported feature, awkward combination and hazard has a probe. Each level has rejection cases: `expected_rejection` bundles for the rejections that an engine must enforce ([§6](#6-feature-inventory)). | | Regressions | The reproducer of each escaped bug is a concrete case in the lowest level that can afford it. | | Concrete graphs | Each level runs checked-in graphs only. Random sampling belongs to the exhaustive tool ([§14](#14-exhaustive-mode-and-the-separate-tool)). | @@ -270,9 +291,19 @@ Each generated case has a `purpose` in its sweep metadata: | Purpose | Expected outcome | Claims | |---|---|---| | `claim` | The claiming engine accepts and is correct. | `--write-support-claims` writes them ([RFC 0015](0015_EngineSupportClaims.md)). | -| `probe` | A typed rejection, or a correct accept. | Never written. The published support matrix excludes probes. | +| `probe` | Tolerant: a typed rejection, or a correct accept. | Never written. The published support matrix excludes probes. | | `regression` | Same as the original bug case. | Same as the original case. | +A probe whose rejection an engine must enforce also sets the `expected_rejection` +bundle flag. It is strict: it is never executed or compared, and an engine that +accepts it fails the lane. It is outside the claim model, so no claim names it. + +A case that exists for the seam of one engine may carry an optional +`motivation` note that names that engine. It is provenance only. It changes no +behavior and does not appear in the feature inventory, so the plan stays +engine-independent (P2). It lets a reviewer see why the case exists, and +delete it when that engine no longer needs it. + Today, a rejected probe and a probe that never ran both show as SKIP. The engine integration gate will record rejections with a typed reason. This RFC requires only the tag. With it, the support matrix can exclude probes, and reviewers can @@ -287,6 +318,10 @@ Each plan has a register: Rejection probes are never in the register. +A quarantined `expected_rejection` bundle ([§6](#6-feature-inventory)) is listed +in the plan with its ticket. It is a deferral, not a product that we chose not to +run. + ## 11. Oracles and Reference Prerequisites Verification follows [RFC 0011](0011_BundleTestFramework.md). In `auto` mode, the order is golden data, GPU @@ -300,6 +335,7 @@ each oracle **models** it. List each gap with its fix. Typical gaps: | Types with no reference | fp8 and int8 conv | | Special values | NaN and Inf need a validator that matches infinities. | | Harness support | Exact-alignment placement, value-domain control, repeat-and-compare ([§15](#15-required-infrastructure)) | +| Fused topology with no executing reference | A forward-plus-backward graph (for example SDPA) needs a reference that executes the fused graph and a golden generator that writes it. Until then, core cover ([§8.1](#81-sweep-rules)) and a G-A anchor cannot be met for it. | A feature with no oracle stays in the plan and is marked "not run". A golden case can cover it ([§12](#12-golden-tests), criterion G-B). @@ -345,14 +381,23 @@ A case is not golden if it meets no criterion and a live reference can check it. A case is not golden if it exceeds 2 MiB or contains NaN or Inf. `verify_golden_bundles.py` rejects both. A case is not golden if the third-party semantics cannot map to hipDNN semantics. +One exception: in an SDPA stats tensor, −inf on exactly the rows that the +bounds and lengths of the case mask completely is not a non-finite value for +this rule, because such a row has LSE = −inf by definition. The verifier must +apply the same exception. ### 12.3 Generation - **Source.** An architecture-independent third party. PyTorch on CPU is the default. A hipDNN reference never generates golden data. -- **Precision strategy.** Set it for each op in `meta.json`. Use precision - uplift for shallow ops and fp32. Use matched precision (the accumulation - contract of the engine) for deep ops and low-precision types ([RFC 0011](0011_BundleTestFramework.md)). +- **Precision strategy.** Set it for each op in `meta.json`. Default to + precision uplift: compute in FP64 and round once to the output dtype. Matched + precision (the accumulation contract of the engine) is not the default for + deep ops or low-precision types. On SdpaBackward (ALMIOPEN-2894), uplift + golden data for bf16 and fp16 had 0 differing elements between + NumPy/OpenBLAS and torch/MKL over 57 bundles × 5 tensors. Matched FP32 moved + 1–5 ulps between torch builds. The tolerance covers the intermediate rounding + of the engine either way ([RFC 0011](0011_BundleTestFramework.md)). - **Provenance.** Pin the third-party version. Seed the inputs. Record `forward_source` for backward ops in `meta.json`. - **Tolerance.** Calibrate it for each case from the observed error of both @@ -467,7 +512,7 @@ Boundaries: | # | Capability | Status | Needed for | |---|---|---|---| | I1 | **Bundle generator.** Reads declared axes, legality, shape sets and level rules. Writes `sweep.json` for each level, with all-pairs and seeded rotation. | Missing. The conv sweeps came from C++ tests (`generator: capture-bundles`) or were written by hand (`generator: manual`). | [§8](#8-sweep-construction-levels-and-lanes) | -| I2 | **`purpose` field** in sweep case metadata. Support-matrix generation excludes probes. | Missing | [§9](#9-case-purpose) | +| I2 | **`purpose` field** in sweep case metadata, with the optional `motivation` note. Support-matrix generation excludes probes. | Missing | [§9](#9-case-purpose) | | I3 | **Exact-alignment and offset placement** in the harness | Missing. Pointers are allocator-aligned. | [§6](#6-feature-inventory) | | I4 | **Value-domain control** for each tensor: zero-excluded, all-zero, constant, large magnitude, subnormal, NaN and Inf injection. A seed derived for each tensor. | Missing. The tensor bundles fix ranges in code. For example, conv x and w are uniform in [0, 1], with one seed for every tensor. | [§6](#6-feature-inventory) | | I5 | **Repeat-and-compare** on bundles (determinism, plan reuse) | Exists only inside providers | [§6](#6-feature-inventory) | @@ -475,6 +520,7 @@ Boundaries: | I7 | **Golden generator for each op** in `reference-data-scripts/`, writing `meta.json` with criterion and strategy | Batchnorm and SDPA only | [§12](#12-golden-tests) | | I8 | **Corpus intake** for M shapes | `migration-scripts/import_graph.py` is the start. | [§7](#7-shape-catalog-and-size-ladder) | | I9 | **Cold-cache timing** for each unique shape and arch | Ad hoc | [§8](#8-sweep-construction-levels-and-lanes) | +| I10 | **`expected_rejection` bundle flag.** A strict rejection: never executed, an accepting engine fails the lane, no claim names it. | In review (PR #14346) | [§6](#6-feature-inventory), [§9](#9-case-purpose) | Templates can also need new placeholders. Today, conv `conv_mode`, `compute_data_type` and pointwise `operation` are hard-coded. These changes are @@ -561,9 +607,12 @@ because nobody writes bundles by hand ([§4](#4-design-overview)). 1. **Where axes and legality live.** A per-op Python module in `reference-data-scripts/`, or a data file beside the plan? The bundle generator and the exhaustive tool both read them. -2. **Precision strategy for fp16 and bf16 golden cases.** We propose matched - precision. Confirm it against the accumulation contracts of all claiming - engines. + A per-op Python module worked for SdpaBackward: its tests proved all-pairs + coverage, legality and plan-item traceability. +2. **Precision strategy beyond SdpaBackward.** Uplift is the default + ([§12.3](#123-generation)), on SdpaBackward evidence. Confirm it for + convolution and normalization, and against the accumulation contracts of all + claiming engines. 3. **Source of tolerance calibration.** Per-case `meta.json` ([RFC 0011](0011_BundleTestFramework.md) level 2), or dynamic tolerances? 4. **Full level and unclaimed dtypes.** Should full-level products run only on