[hipDNN]: Initial RFC for Golden Reference Validation

[bghimireamd]

Introduces golden reference validation as a third verification mode for the integration test suite.

ALMIOPEN-1925

[AnaghaRaoAMD]

[Note] Generating golden data at higher precision to eliminate variance would work well for shallow operations (batchnorm, pointwise), but is counterproductive for deep, nonlinear operations like SDPA backward.

SDPA backward has 5+ rounding events through softmax (exp, sum, div, matmul, accumulate). An FP64 reference rounds once at the end; a BF16 kernel accumulates rounding at every stage. The delta between them is dominated by accumulated precision loss, not kernel bugs — requiring significantly looser tolerance than simple operations to avoid false failures.

PyTorch's math backend with BF16 inputs auto-upcasts intermediates to FP32, which is a like-for-like match for AITER's a32 accumulator kernels. Using this instead of FP64 produces a reference that follows the same rounding trajectory as the kernel, enabling tighter tolerances.

[BrianHarrisonAMD]

Agreed that we need the golden data to match the kernel’s rounding behavior closely enough to be a useful comparison. For SDPA backward, I don’t think an FP64 reference that rounds once at the end is the right comparison for BF16/a32 kernels, because it can turn expected accumulated precision loss into apparent error.

The distinction I’d make is that internal compute can still be higher precision than the output, but it should match the implementation contract. For example, BF16 inputs with FP32 intermediates/accumulation is a reasonable reference for a32 accumulator kernels, while preserving the relevant casts at semantic boundaries.

I agree we need to be careful about how golden data is generated per operation. The reference path and tolerances should be tied to the operation details, accumulator mode, and expected rounding path, rather than using one FP64 golden strategy for all ops.

[AnaghaRaoAMD]

Non-deterministic engines

The golden validation model assumes: given input X, engine always produces output Y (within tolerance). This breaks for engines using non-deterministic accumulation — e.g., AITER's v3_atomic_fp32 dQ path uses atomic FP32 additions where thread-block scheduling varies run-to-run, producing bit-different results from floating-point non-associativity.

A single golden snapshot captures one realization of the output distribution. Comparing against it produces flaky tests — sometimes passing, sometimes failing depending on GPU scheduling.

Common resolutions: (1) statistical tolerance banding from N runs, (2) envelope min/max bounds

Suggestions:

1. Add a deterministic field to meta.json (boolean or enum) so the test runner can apply different validation strategies
2. Recommend determinism enforcement as the default — generate golden data with the deterministic kernel variant, validate CI against the same variant
3. Add "golden comparison fails intermittently for non-deterministic engines" to the Risk Register with the chosen mitigation

[BrianHarrisonAMD]

I don’t think forcing the deterministic variant is a valid mitigation if the goal is to validate the non-deterministic engine. It could be useful for generating a stable reference, but if the production/selected engine uses atomic accumulation, then CI needs to exercise that path directly. The mitigation should be tolerance/error bands that cover the valid run-to-run variance, not swapping the implementation under test.

For non-deterministic accumulation, single-output comparison is valid only if the error bands account for the output distribution. That could mean derived tolerances from repeated runs, or an envelope/min-max policy, but the important part is that the validation tolerance covers expected scheduling/order variation.

I think tracking determinism on the bundle could still be useful for helping triage test flakiness, though. I just wouldn’t treat that metadata as the mitigation by itself.

[AnaghaRaoAMD]

Good point — I struck out the deterministic variant suggestion since that defeats the purpose of validating the actual engine path. To clarify, my concern is about the golden reference data itself, not the engine under test. If golden snapshots are captured using a non-deterministic GPU reference, the golden data is just one sample from an output distribution, introducing flakiness from the reference side.

Today this isn't an issue — our references are CPU/PyTorch-based and deterministic. But if we ever add GPU-based references to the golden data pipeline, we should be wary of non-deterministic accumulation contaminating the reference. I do not see this happening but it's worth flagging now. At that point we'd either need to constrain golden generation to deterministic paths (CPU or order-enforcing GPU variants), or use statistical tolerance banding (envelope/min-max from N runs).

[AnaghaRaoAMD]

Right now, it seems Forward and backward are independent bundles (ConvFwd, ConvBwd), but backward operations consume saved tensors from the forward pass (O, LSE for SDPA; saved mean/variance for batchnorm; mask for dropout). These inputs must come from a mathematically consistent forward computation — random values produce invalid backward bundles.

For SDPA backward, O and LSE generation has two options: (A) PyTorch forward — architecture-independent but CPU-rounded (inflates tolerance for complex operators), or (B) GPU forward — production-realistic but ties bundles to a specific architecture (SDPAhas platform specific fwd kernels).

Suggestions:

1. Document the forward-backward coupling pattern — backward bundles have a generation constraint that applies to any op with a backward pass
2. Add a forward_source field to meta.json for backward bundles (e.g., {"method": "pytorch_math_backend", "version": "2.3.0"})
3. Recommend Option A (PyTorch forward) as default for architecture-independent bundles; document Option B as an advanced alternative
4. Have the pre-commit verifier check that backward bundles include forward_source in metadata for respective operators

[bghimireamd]

Agreed. Will add a "Forward-Backward Generation Constraint" section to the RFC covering:

1. Backward bundles' saved-tensor inputs (O, LSE for SDPA; saved_mean, inv_variance for batchnorm) are actual outputs of the forward reference, not random values
2. Generation script must produce both bundles in a single pipeline run, regenerating forward alone silently stales the backward bundle (passes CI but validates against a dead reference)
3. forward_source metadata in backward bundles with CI cross-check against the forward bundle's metadata for enforcement

Good call on the forward_source field.

[AnaghaRaoAMD]

Bundle sizes could vary dramatically across operations. Batchnorm bundles (Phase 1) are under 1 MB. SDPA backward bundles contain 9 tensors and scale as O(B×H×S×D):

Shape	Bundle Size
{1, 1, 256, 128}	~513 KB
{2, 8, 1024, 128}	~32 MB
{4, 32, 4096, 128}	~1 GB
{8, 64, 8192, 128}	~8 GB

1. Is there a per-bundle size cap for quick/ tier? (e.g., < 10 MB to avoid DVC dependency in pre-submit CI)
2. What is the total storage budget per tier as operations are onboarded? (6 ops × 2 directions × 3 dtypes × 2 layouts = 72+ bundles)
3. Should the auto-tier classifier enforce size limits or just warn?

[adickin-amd]

there is less of a cap on bundle size and more of a cap on the runtime for each category.

Here is the doc for the test filtering the rock is bringing in. https://github.com/ROCm/TheRock/blob/main/docs/development/test_filtering.md#types-of-filters

[adickin-amd]

Im also making the assumption right now that were going to put all these tests into dvc rather than checking them into the repo since its a ton of binary files. I guess it might be better to check in the graph/meta files normally and just DVC the .bin files.

[adickin-amd]

I also think if we have a ton of graphs that have 8GB bundles that might be kinda nuts so maybe we need some kind of cap so we stay within a reasonable limit. If we need these super large bundles, we probably want to explore hosting these elsewhere since if you have to pull these down for local development you simply wont have the disc space available. If we had a ton of ~8GB tests wed quickly use up 1TB of storage.

[AnaghaRaoAMD]

Agreed this isn't an immediate concern. We've been limiting tensor sizes for SDPA since CPU reference generation is slow, so we can stick with medium-sized tensors for golden data as well. That said, even with moderate shapes, SDPA backward has 9 tensors — bundle sizes in the hundreds of MB to ~1 GB range are something we'll realistically hit as we onboard full test coverage. Worth keeping the storage budget in mind as more operations come online.

[BrianHarrisonAMD]

I’m increasingly thinking tolerance needs to live in the test bundle as graph-specific metadata.

A global tolerance model for all operations seems unlikely to work well. It will either be conservative enough that some checks become too loose, or tight enough that numerically valid cases become flaky. Instead, we should embed tolerances with the graph and have a generation/calibration tool set them by running the relevant reference and engine paths repeatedly to measure expected variance.

That gives us tight tolerances for simple ops and calibrated tolerances for deeper or non-deterministic graphs like SDPA backward.

[adickin-amd]

seems easy enough to add tolerances to the metadata file. This also makes it so the metadata file is actually used and valuable and not just a way to trace where binary data came from.

For tolerances we currently have: Per engine toml tolerance(if set) > hardcoded tolerance in c++ .

This would make it so we have: Per engine toml tolerance(if set) > New Bundle Tolerance(metadata file) > hardcoded tolerance in c++ .