From 150dcfe372fc8e66c1d5e5ab6f166579c9daded9 Mon Sep 17 00:00:00 2001 From: Alex Vasile <48962821+Alex-Vasile@users.noreply.github.com> Date: Fri, 6 Nov 2026 15:56:42 +0000 Subject: [PATCH 1/2] docs(tensilelite): add the timing measurement scripts, results and write-ups One draft-only branch with everything the speed-up work measured and wrote down (78 files, 15,982 lines): the hand-run measurement scripts (16 files, 598 lines; measure_env.sh makes paths, caches, GPU target and cores overridable), the generated analyze_timing / aggregate_timing outputs of six campaigns (43 files, 9,206 lines: first gfx942 runs, gfx950 per-config, gfx950 baseline / after-run / after-build, gfx942 before and after), and the hand-written notes (19 files, 6,178 lines: test-client-plan.md, CI_SPLIT_TIMING.md, the gfx950 investigations, DECISIONS.md, DEFERRED.md, the before/after write-ups, the gfx942 README and FURTHER_SPEEDUPS.md). Nothing in the build, CI or tests reads these files. --- .../hipblaslt/tensilelite/CI_SPLIT_TIMING.md | 78 ++ .../tensilelite/timing_runs/DECISIONS.md | 963 ++++++++++++++++++ .../tensilelite/timing_runs/DEFERRED.md | 153 +++ .../timing_runs/SynchronizerValidator.md | 265 +++++ .../build_fp16_gradient_bias_analysis.txt | 135 +++ .../cgemm/cgemm_1_build_cold_analysis.txt | 135 +++ .../cgemm/cgemm_2_build_warm_analysis.txt | 127 +++ .../cgemm/cgemm_3_run_analysis.txt | 173 ++++ .../tensilelite/timing_runs/cgemm/times.txt | 4 + .../timing_runs/cms/CMS_FINDINGS.md | 110 ++ ...tom_mainloop_scheduling_build_analysis.txt | 135 +++ ...ainloop_scheduling_tf32_build_analysis.txt | 135 +++ .../custom_mainloop_scheduling_tf32_time.txt | 23 + .../cms/custom_mainloop_scheduling_time.txt | 23 + ...ainloop_scheduling_warm_build_analysis.txt | 135 +++ .../custom_mainloop_scheduling_warm_time.txt | 23 + .../cms/f8bf16_tn_cms_build_analysis.txt | 8 + .../timing_runs/cms/f8bf16_tn_cms_time.txt | 24 + .../tensilelite/timing_runs/cms/run_split.sh | 8 + .../timing_runs/cms/run_split_warm.sh | 8 + .../timing_runs/cms/split_25_result.txt | 3 + .../timing_runs/cms/split_25_warm_result.txt | 3 + .../timing_runs/cms/split_5_result.txt | 3 + .../timing_runs/cms/split_experiment.py | 42 + .../tensilelite/timing_runs/cms/times.txt | 4 + .../timing_runs/cold_warm_done.txt | 1 + .../cpu_data_init_investigation.md | 191 ++++ .../cpu_reference_slow_path_investigation.md | 442 ++++++++ .../tensilelite/timing_runs/driver.log | 0 .../dtv/dtv_1_build_cold_analysis.txt | 135 +++ .../dtv/dtv_2_build_warm_analysis.txt | 127 +++ .../timing_runs/dtv/dtv_3_run_analysis.txt | 175 ++++ .../tensilelite/timing_runs/dtv/times.txt | 4 + .../tensilelite/timing_runs/full_runs/run.sh | 31 + .../full_runs_gfx942/FURTHER_SPEEDUPS.md | 483 +++++++++ .../timing_runs/full_runs_gfx942/README.md | 54 + .../full_runs_gfx942/build_comparison.txt | 43 + .../post_build/AGGREGATE_analysis.txt | 172 ++++ .../post_run/AGGREGATE_analysis.txt | 585 +++++++++++ .../pre_build/AGGREGATE_analysis.txt | 167 +++ .../pre_run/AGGREGATE_analysis.txt | 581 +++++++++++ .../full_runs_gfx942/run_comparison.txt | 85 ++ .../full_runs_gfx942/split_post/times.txt | 6 + .../full_runs_gfx942/split_pre/configs.txt | 123 +++ .../full_runs_gfx942/split_pre/times.txt | 6 + .../full_runs_gfx950/after_build.md | 269 +++++ .../after_build/AGGREGATE_analysis.txt | 575 +++++++++++ .../after_build/split_build_outcomes.txt | 572 +++++++++++ .../timing_runs/full_runs_gfx950/after_run.md | 288 ++++++ .../after_run/AGGREGATE_analysis.txt | 735 +++++++++++++ .../after_run/common_test_outcomes.txt | 572 +++++++++++ .../timing_runs/full_runs_gfx950/baseline.md | 268 +++++ .../baseline_build/AGGREGATE_analysis.txt | 595 +++++++++++ .../baseline_run/AGGREGATE_analysis.txt | 753 ++++++++++++++ .../baseline_run/common_test_outcomes.txt | 571 +++++++++++ .../full_runs_gfx950/measure_env.sh | 93 ++ .../per_config/run_per_config.sh | 64 ++ .../per_config/run_per_config_build.sh | 73 ++ .../per_config_build/AGGREGATE_analysis.txt | 598 +++++++++++ .../per_config_rerun/warm_cache_check.sh | 28 + .../per_config_rerun_2/AGGREGATE_analysis.txt | 754 ++++++++++++++ .../full_runs_gfx950/profiling/run_profile.sh | 45 + .../full_runs_gfx950/profiling/run_rerun.sh | 30 + .../timing_runs/full_runs_gfx950/run.sh | 29 + .../full_runs_gfx950/split/run_split.sh | 34 + .../verify_mx/run_verify_mx.sh | 21 + .../input_data_preparation_investigation.md | 233 +++++ .../per_solution_bookkeeping_investigation.md | 721 +++++++++++++ ...python_kernel_bench_setup_investigation.md | 232 +++++ ...ython_kernel_build_src_co_investigation.md | 530 ++++++++++ .../python_kernel_codegen_investigation.md | 508 +++++++++ .../tensilelite/timing_runs/run_cms.sh | 21 + .../tensilelite/timing_runs/run_cold_warm.sh | 45 + .../run_fp16_gradient_bias_analysis.txt | 175 ++++ .../tensilelite/timing_runs/run_timing.sh | 26 + .../tcl_helper_name_mismatch_investigation.md | 335 ++++++ .../tensilelite/timing_runs/times.txt | 3 + projects/hipblaslt/test-client-plan.md | 55 + 78 files changed, 15982 insertions(+) create mode 100644 projects/hipblaslt/tensilelite/CI_SPLIT_TIMING.md create mode 100644 projects/hipblaslt/tensilelite/timing_runs/DECISIONS.md create mode 100644 projects/hipblaslt/tensilelite/timing_runs/DEFERRED.md create mode 100644 projects/hipblaslt/tensilelite/timing_runs/SynchronizerValidator.md create mode 100644 projects/hipblaslt/tensilelite/timing_runs/build_fp16_gradient_bias_analysis.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_1_build_cold_analysis.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_2_build_warm_analysis.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_3_run_analysis.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cgemm/times.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/CMS_FINDINGS.md create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_build_analysis.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_tf32_build_analysis.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_tf32_time.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_time.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_warm_build_analysis.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_warm_time.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/f8bf16_tn_cms_build_analysis.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/f8bf16_tn_cms_time.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/run_split.sh create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/run_split_warm.sh create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/split_25_result.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/split_25_warm_result.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/split_5_result.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/split_experiment.py create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/times.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cold_warm_done.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cpu_data_init_investigation.md create mode 100644 projects/hipblaslt/tensilelite/timing_runs/cpu_reference_slow_path_investigation.md create mode 100644 projects/hipblaslt/tensilelite/timing_runs/driver.log create mode 100644 projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_1_build_cold_analysis.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_2_build_warm_analysis.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_3_run_analysis.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/dtv/times.txt create mode 100755 projects/hipblaslt/tensilelite/timing_runs/full_runs/run.sh create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/FURTHER_SPEEDUPS.md create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/README.md create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/build_comparison.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/post_build/AGGREGATE_analysis.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/post_run/AGGREGATE_analysis.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/pre_build/AGGREGATE_analysis.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/pre_run/AGGREGATE_analysis.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/run_comparison.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_post/times.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_pre/configs.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_pre/times.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_build.md create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_build/AGGREGATE_analysis.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_build/split_build_outcomes.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_run.md create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_run/AGGREGATE_analysis.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_run/common_test_outcomes.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline.md create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline_build/AGGREGATE_analysis.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline_run/AGGREGATE_analysis.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline_run/common_test_outcomes.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/measure_env.sh create mode 100755 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config/run_per_config.sh create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config/run_per_config_build.sh create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_build/AGGREGATE_analysis.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_rerun/warm_cache_check.sh create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_rerun_2/AGGREGATE_analysis.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/profiling/run_profile.sh create mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/profiling/run_rerun.sh create mode 100755 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/run.sh create mode 100755 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/split/run_split.sh create mode 100755 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/verify_mx/run_verify_mx.sh create mode 100644 projects/hipblaslt/tensilelite/timing_runs/input_data_preparation_investigation.md create mode 100644 projects/hipblaslt/tensilelite/timing_runs/per_solution_bookkeeping_investigation.md create mode 100644 projects/hipblaslt/tensilelite/timing_runs/python_kernel_bench_setup_investigation.md create mode 100644 projects/hipblaslt/tensilelite/timing_runs/python_kernel_build_src_co_investigation.md create mode 100644 projects/hipblaslt/tensilelite/timing_runs/python_kernel_codegen_investigation.md create mode 100644 projects/hipblaslt/tensilelite/timing_runs/run_cms.sh create mode 100644 projects/hipblaslt/tensilelite/timing_runs/run_cold_warm.sh create mode 100644 projects/hipblaslt/tensilelite/timing_runs/run_fp16_gradient_bias_analysis.txt create mode 100644 projects/hipblaslt/tensilelite/timing_runs/run_timing.sh create mode 100644 projects/hipblaslt/tensilelite/timing_runs/tcl_helper_name_mismatch_investigation.md create mode 100644 projects/hipblaslt/tensilelite/timing_runs/times.txt create mode 100644 projects/hipblaslt/test-client-plan.md diff --git a/projects/hipblaslt/tensilelite/CI_SPLIT_TIMING.md b/projects/hipblaslt/tensilelite/CI_SPLIT_TIMING.md new file mode 100644 index 000000000000..75a55bf419b6 --- /dev/null +++ b/projects/hipblaslt/tensilelite/CI_SPLIT_TIMING.md @@ -0,0 +1,78 @@ +# TensileLite common tests: CPU build / GPU run split timing + +Measured 2026-10-15 on a gfx942 box, one run per mode, under CI-like conditions. +Uses the `--build-only` / `--use-cache` modes added by PR #6188 (merged). + +## Results + +| Phase | Command mode | Wall time | Exit | Passed / failed / skipped | +|---|---|---|---|---| +| Build | `--build-only --gpu-targets gfx942 --artifact-dir D` | 1,220 s (20m 20s) | 1 | 133 / 1 / 1,508 | +| Run | `--use-cache --artifact-dir D` | 1,592 s (26m 32s) | 0 | 138 / 0 / 1,509 | +| Combined | default (what CI runs today) | 2,692 s (44m 52s) | 0 | 138 / 0 / 1,509 | + +- Split total, back to back: 2,812 s (46m 52s), about 2m longer than combined. +- **GPU-node time: 1,592 s instead of 2,692 s, a saving of 1,100 s (18m 20s, about 41%).** +- The 1,220 s build moves to a CPU-only runner. +- Pipelining: with several arch builds queued on CPU runners, GPU runners only see the run phase. +- The build exit code is 1 because of a single config, see "Findings". +- Not included: artifact upload/download/extract and per-job setup. + The artifacts total 409 MB compressed (123 `.tar.gz`, about 3.76 GB uncompressed, median 0.46 MB, + largest `gemm/dtv` 76.7 MB, then `gemm/fp16_tn` 35.7 MB and `gemm/bufferLoadStore0` 31.1 MB). + Transfer cost must be added when deciding the real saving. + +## Conditions + +Taken from TheRock's `fetch_test_configurations.py` / `pytest_runner.py` and the log of a failing +`tensilelite-common` job on PR #12151 (job 116559736872, gfx94X). No passing CI job was available to +get CI timings, so none are compared here. + +CI (gfx94X): 1 GPU (`ROCR_VISIBLE_DEVICES`), `KUBE_CPU_REQUEST=25` so `OMP_NUM_THREADS` and +`OPENBLAS_NUM_THREADS` are 25, `pytest -v -m "(common) and not skip-gfx94X"`, `--numprocesses=4` +(`parallel_workers: 4` for `hw-common`), `--timeout=5400`, `--prebuilt-client=`, +`--global-parameters=LibraryFormat='msgpack'`, `--tensile-options=--cxx-compiler,/bin/amdclang++`. + +Local reproduction: `taskset -c 0-24` (25 cores), `ROCR_VISIBLE_DEVICES=0`, `OMP_NUM_THREADS=25`, +`OPENBLAS_NUM_THREADS=25`, `-n 4`, `--timeout=5400`, marker `common and not skip-gfx94X`, the same +`--global-parameters` / `--tensile-options`. `~/.tensile` and `~/.tensilelite` were deleted before +each phase. Script: `/tmp/tl_split/run.sh`; logs in `/tmp/tl_split/`. +The machine has 96 cores and 8 GPUs and is shared, so there was contention (load average up to about +100 at times); treat the numbers as one noisy sample. + +To emulate a CPU-only build node, the five GPU-gated tests in `Tests/common/flags/gfx942/` +were deselected in the build phase (see Findings). + +## Findings + +1. **CI marker uses the family name.** The CI marker is `not skip-gfx94X`, but `pytest.ini` registers + exact-arch markers (`skip-gfx942`). It selects 1,647 tests instead of 387. The configs that declare + `skip-gfx942` are still skipped at run time by `configMarks` using the detected arch, so results are + the same, but the marker is not doing the filtering. +2. **Build phase needs a msgpack-capable client.** CI passes `LibraryFormat='msgpack'` with + TheRock's client. The locally built `build_tmp` client was built with `HIPBLASLT_ENABLE_YAML=ON` + and has no msgpack support: 5 tests failed with `Failed to load solution library`. + The runs here use a client built with `-DHIPBLASLT_ENABLE_YAML=OFF`. +3. **`comm/gfx950/fused_a2a.yaml` fails the build phase** with `Your parameters resulted in 0 valid + solutions`. It is a gfx950-only config with no `skip-gfx942` mark. In the combined phase it is skipped + because `comm` tests need 2+ GPUs. With `--build-only --gpu-targets gfx942` it is built anyway and + fails, so a CI build job for gfx942 would go red on it. It needs a `skip-gfx942` mark or a skip + when no multi-GPU is possible. +4. **`Tests/common/flags/gfx942/` is collected in every mode.** `pytest_ignore_collect` filters only + direct children of `common/`. `test_compile` runs on CPU; the other five tests are gated on + `has_arch("gfx942")` and use the GPU, so they should skip on a CPU node and run in the run phase. + `has_arch` uses `rocm_agent_enumerator`/`amdgpu-arch` and ignores `ROCR_VISIBLE_DEVICES`, + so hiding the GPU that way does not skip them. Skipping on a real GPU-less node was not run. +5. **The artifact is a directory of per-config tarballs, not one zip.** CI will need to upload the + directory per target, and gfx120X (gfx1200/gfx1201) needs one per target. +6. **Setup pitfalls.** `~/.tensile` and `~/.tensilelite` were read-only in the agent sandbox, which made + early runs fail; the final runs were done unsandboxed. + +## Caveats + +- Single run per mode; shared, loaded machine. Build phase time scales with cores (25 here); a + CPU runner with more cores will be faster. +- Run and combined phases include the five `flags/` GPU tests; the build phase does not. +- The run phase measures only GPU-side execution on an already-built artifact; it does not include + extracting the 409 MB of artifacts (included in the 1,592 s, since `test_config_run` extracts each). +- CI timings were not available for comparison (the observed CI jobs failed early for an unrelated + `PytestRemovedIn9Warning`). diff --git a/projects/hipblaslt/tensilelite/timing_runs/DECISIONS.md b/projects/hipblaslt/tensilelite/timing_runs/DECISIONS.md new file mode 100644 index 000000000000..36f2dbb3124f --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/DECISIONS.md @@ -0,0 +1,963 @@ +# Timing investigation decisions + +Record of the decisions taken on each investigation in this directory, one topic at a time. +It is the input to the implementation plan at the end of this file. + +Status values: `pending` (not discussed yet), `discussing`, `decided`. + +Deferred and "not for now" work is tracked separately in [DEFERRED.md](DEFERRED.md). + +## Agenda + +Ordered by measured cost, run side first (gfx950 `per_config_rerun_2`), then build side +(gfx950 `per_config_build`). + +| # | Investigation | Cost in aggregate | Status | +| --- | --- | --- | --- | +| 1 | [CPU reference slow path](cpu_reference_slow_path_investigation.md) | 741 s slow path (of 1,080 s Reference GEMM, 42% of run wall) | decided | +| 2 | [Input data preparation](input_data_preparation_investigation.md) | 503 s (19.6%) | decided | +| 3 | [SynchronizerValidator](SynchronizerValidator.md) | 163 s (6.3%) | decided | +| 4 | [`cpu_data_init`](cpu_data_init_investigation.md) | 107 s (4.2%) | decided | +| 5 | [Per-solution bookkeeping](per_solution_bookkeeping_investigation.md) (incl. correctness-only runs) | 68 s (2.7%), plus correctness-mode items | decided | +| 6 | [`python_kernel_build_src_co`](python_kernel_build_src_co_investigation.md) | 1,008 s (39.1% of build wall) | decided | +| 7 | [`python_kernel_codegen`](python_kernel_codegen_investigation.md) | 593 s (23.0%) | decided | +| 8 | [`python_kernel_bench_setup`](python_kernel_bench_setup_investigation.md) | 132 s (5.1%) | decided | + +## Cross-cutting items + +These show up in more than one note. Each is decided once, in the topic listed. + +| Item | Appears in | Decide in | +| --- | --- | --- | +| `KMP_BLOCKTIME=0` / idle OpenMP workers spinning | 1, 5 | 5 (decided: adopt, in the client) | +| GPU wait misattributed to the next blocking call (warmup sync placement, `SyncsPerBenchmark: 0`) | 2, 3, 5 | 3 (decided: syncs only under timing instrumentation) | +| `copyInputs` / `copyInputBuffers` sized by `maxElements` | 2, 4 | 2 (decided: adopt) | +| Redundant MX regeneration and the CPU reference recompute it triggers | 2, 4 | 2 (decided: adopt) | +| `copyInputs` restoring read-only inputs vs only outputs + Synchronizer | 2, 3 | 2 (decided: not now) | +| Pool sizing (`CPUThreadCount`, oversubscription under xdist) | 7 (and every build-side pool) | 7 (decided: 2× CPU budget) | + +## Decisions + +### 1. CPU reference slow path + +Status: decided. + +Adopted: + +- **A.** In the slow path (`ReferenceSolution<…>::SolveCPU`), enter `#pragma omp critical` only + when `problem.outputAmaxD()` is true, and drop the per-element `omp_set_num_threads`. +- **A+.** Replace the AMAX-D critical section with a per-thread maximum combined once + (`reduction(max:…)` or equivalent). Replace the hard-coded `MAX_OMP_THREADS 64` cap with + `omp_get_max_threads()`, so the reference follows `OMP_NUM_THREADS`. +- **B.** Fast-path (`solveCPUFast`) support for: + - Int8 inputs, with outputs I8, I32, Half and BF16. Accumulation must be exact (a float + accumulator is not exact above 2²⁴). + - Float8 outputs (`F8 × F8 → F8`, BF8 variants, and `F8 × Half → F8` with the mixed-input + shadow). + - Write-back must match the slow path's `SaturateCast`. +- **C.** Fast-path support for the smaller rejection reasons: `gradient` (including the bias + reduction), `useE`, MX K not a multiple of the MX block (partial last block), and complex types. + `NumElementsToValidate: 128` stays on the slow path, as intended. + +Constraint: the new fast-path support must follow the established fast-path patterns (as in the +FP8/FP4, f64 and tf32 additions): `isFastPathEligible`/`solveCPUFast` in `Reference.cpp`, +fast/slow `CPUGemm.*` CTest pairs through `cpu-gemm-driver` in `tests/CMakeLists.txt`, and +gtest cases in `tests/ReferenceFastPath_test.cpp` / `ReferenceMXFastPath_test.cpp`. + +Not adopted / open: + +- **D** (making the fast-path rejection reason observable) was not chosen. +- **Int8 accumulation (decided): integer accumulation, then the slow path's epilogue type.** + Accumulate the K loop in `int32_t` (exact, as the GPU's int8 MFMA does), then apply alpha, + beta, bias, activation and the saturating cast in the slow path's accumulator type for that + type id: `int32_t` for `I8_I8_I32` / `I8_I32_I32` (int32 alpha), `float` for `I8_I32_S`, + `I8_I8_S`, `I8_H_S`, `I8_B_S` (float alpha). This needs the fast path to separate its + multiply-accumulate type from its epilogue type (a new template parameter, in the style of + `OperandMathOpT`). Reusing the `double` instantiation was considered and not chosen. +- `KMP_BLOCKTIME` is decided under topic 5. + +### 2. Input data preparation + +Status: decided. + +Adopted: + +- **Fix 1. Skip unchanged per-solution uploads (`copy_valid_to_gpu`).** On the slow reset path, + skip the H2D upload of host `valid` data to GPU `valid` buffers when the host data has not + changed since the last upload (tracked with `m_inputEpoch`). +- **Fix 2. Cache the batched-pointer arrays (`batched_inputs`).** Build and upload them per + problem; re-upload only when the epoch or a buffer base address changes, not per solution. +- **Fix 4. No MX regeneration per solution.** + - Data and canonical scale are generated once at problem level and never regenerated per + solution. + - The GFX950 swizzled scale is produced by permuting the canonical scale + (`preSwizzleScalesGFX950` in mxDataGenerator's `PreSwizzle.hpp`), once per problem, and + cached, ideally in its own GPU buffer so solutions using the canonical and swizzled layouts do + not overwrite each other. + - `m_inputEpoch` is not bumped for it, so the second CPU reference computation goes away. + - Required check: the permuted scale is byte-identical to today's + `generateMXInput(…, GFX950)` output, including padding and the K-fast restride. +- **Shared fix. Size copies by the current problem.** `copyInputs` / `copyInputBuffers` (and the + per-problem `copy_valid_to_gpu`) copy the current problem's allocated extent instead of + `maxElements`. + +Not adopted: + +- **Fix 3 (restore only writable tensors in `copyInputs`): not doing it for now.** Too easy to + get the writable set wrong. + +Measurement: no per-topic measurement; covered by the per-phase aggregate reruns in the +implementation plan. + +### 3. SynchronizerValidator + +Status: decided. + +Adopted: + +- **Item 1. GPU check.** An OR-reduction kernel over the Synchronizer buffer, enqueued on the + client's `stream`, followed by a 4-byte readback. Only when the result is nonzero: a full readback + and the existing scan to build the diagnostic, mark the solution `FAILED`, and clear the buffer. +- **Item 1-alt. Vectorized host check, kept as the FFM-emulator path.** Full readback, then an + OR-reduction first (auto-vectorizes), with the count/first-offset scan only when dirty. Used when + running under the FFM emulator (`HSA_MODEL_MEMFILE`), where the GPU kernel's cost is unknown. To + be tested and tuned later based on measured performance. +- **Diagnostic unchanged.** The dirty-buffer message (nonzero count, first offset) stays as it is, + built from a full readback only on the dirty path. +- **Item 2. Move the check to `postSolution`**, so it sees every launch of the solution (all + warmups and benchmark runs) on both reset paths, including the last solution's. Requires saving + the `ProblemInputs` pointer (from `preSolution` / `validateWarmups`). Ordering correction (from + the plan review, verified in `MetaRunListener.cpp`): `postSolution` runs listeners in reverse, + so `SynchronizerValidator::postSolution` runs *after* `ReferenceValidator::postSolution`, which + is what lets a dirty-buffer `FAILED` override the reference verdict. Keep the registration order + unchanged and do the check inside `SynchronizerValidator::postSolution`. With guards: + 1. **Only check if the solution launched something.** Set a flag in `preWarmup` / + `preEnqueues`; keep the existing `m_mayUseSynchronizer` gate. `postSolution` also runs for + rejected and never-run solutions. + 2. **Exception path.** If the device is usable, check as normal. If the device is lost, catch + the check's own HIP error and only mark the solution `FAILED`; never throw out of + `postSolution`. + 3. **Explicit, separately timed sync before the check**, so queued warmups 1..N-1 + (`SyncsPerBenchmark: 0`) are not absorbed by the check. Only when timing instrumentation is + on (see "Attribution syncs" below); otherwise the check's own readback does the waiting. + - Keep clear-on-dirty, so every checked solution ends with a clean buffer. + +Not adopted: + +- Adding a Synchronizer reset to the fast reset path (`resetOutput`). Not needed with the + `postSolution` check. + +Known limitations (accepted): + +- Solutions classified as unable to use the Synchronizer are not checked (as today). +- **Correction (plan review, verified):** not every solution has one run iteration. + `BoundsCheck: 4` (GuardPageAll, e.g. `bufferLoadStore0`) sets `m_numRunsPerSolution = 2` + (`DataInitialization.cpp`); the second iteration's `prepareGPUInputs` restores the + Synchronizer and wipes the first iteration's residue before `postSolution`. Today's + `validateWarmups` check runs in each iteration. **Decided:** run the check at the end of every + run iteration (before the next iteration's input reset), and report the verdict in + `SynchronizerValidator::postSolution`. Solutions with one iteration behave exactly as planned; + GuardPageAll solutions get both iterations checked. +- Cache effects of the reduction kernel on timed runs: resolved. In `postSolution` it runs after + the solution's timed runs, never between its warmups and timed runs. + +- **Item 3. Wait attribution after warmup 0, only when timing instrumentation is on.** In + `main.cpp`, after launching warmup 0 and before `listeners.validateWarmups`, wait for the stream + (or warmup 0's stop event) under its own timer, **only if `g_timingInstrumentationEnabled`** + (`--timing-instrumentation`). Its only purpose is attribution: without it, + `ReferenceValidator`'s `validate_gpu_readback` absorbs the kernel's runtime once the Synchronizer + check moves to `postSolution`. When instrumentation is off, no extra block is added, so the + launch/host overlap is unchanged. + - Accepted side effect: instrumented runs block where uninstrumented runs do not, so their + schedule differs slightly from production runs. + +- **Attribution syncs, only when timing instrumentation is on.** All explicit syncs added for + attribution follow the same rule as item 3: + - the wait after warmup 0 (item 3); + - guard 3, the sync before the Synchronizer check; + - **an end-of-iteration sync in `main.cpp` for every run iteration that launched** (one per + solution except GuardPageAll, which has two), under its own timer, so warmups 1..N-1 of + `SyncsPerBenchmark: 0` configs are no longer charged to the next solution's + `gpu_input_reset` (the `batched_inputs` inflation from topic 2). For checked solutions this + is the same sync as guard 3. + +Noted, no action: + +- `BenchmarkTimer::validateWarmups` (`--sync-after-warmups`) waits on the last warmup's stop + event, which is not yet recorded when `NumWarmups > 1`, so it returns immediately (code + reading). It does not affect benchmark results: timed runs either record their start event on + the stream after the warmups (GPU timer) or call `hipDeviceSynchronize` first (host timer). + +### 4. `cpu_data_init` + +Status: decided. + +Adopted: + +- **Item 1. Skip the discarded MX reference floats.** Add an opt-out to `generateMXInput` + (`clients/common/src/mxDataGen.cpp`) so it does not build (and free) the `getReferenceFloat()` + vector. Only the TensileLite client opts out; hipBLASLt's own clients (`testing_matmul.hpp`, + `fast_check_gtest.cpp`) use the returned reference, so the default stays unchanged. +- **Item 2a. Cheaper trig init loop, same values.** Hoist the per-element `std::vector` + allocation, and use a flat index for packed strides instead of `CoordNumbered` + + out-of-line `TensorDescriptor::index`. Generated values must stay identical. +- **Item 5. Dynamic OpenMP schedule for the init loops** (`schedule(dynamic, chunk)`). Values do + not depend on the schedule: Trig/Serial values depend only on the element index, and Random init + already uses unseeded per-thread generators. +- `copyInputs` sizing by `maxElements` (finding 3): covered by the shared fix in topic 2. + +Not adopted: + +- **Item 2b. `sinf` / argument reduction in trig init.** Keep double `sin` for now, so the generated + data does not change. +- **Item 3. Removing the dead `m_cpuInit` branch.** Leave it as is. + +Deferred: + +- The MX generator's other costs (single-threaded `resize` zero-fill, 7–11× scaling on 25 + threads) in shared hipBLASLt client code. +- First-call fixed cost per client process (finding 4): the note has no fix. + +### 5. Per-solution bookkeeping and correctness-only runs + +Status: decided. + +#### 5a. Per-solution bookkeeping + +Adopted: + +- **A. Stop idle OpenMP workers from spinning.** Call `kmp_set_blocktime(0)` in the client's + `main.cpp` unless `KMP_BLOCKTIME` is already set in the environment. Covers every client run. + Also expected to speed up the CPU reference (`cpu_reference_gemm` 2–5× faster on dtl with + `KMP_BLOCKTIME=0`). This settles the cross-cutting `KMP_BLOCKTIME` item. +- **C. No predicate-tree dump by default.** Gate `debugEval` in `SolutionIterator::checkSolution` + behind its own client option, default off, driven by a Tensile global parameter. Per-solution + rows are unaffected. + +- **D1. Stop the per-solution stdout CSV row in test runs.** + - `LogReporter::postSolution` checks whether the row's level will be printed *before* copying + the row (`readCurrentRow`), so a suppressed row costs almost nothing. + - Test runs use `ClientLogLevel` Error (0): only FAILED rows and mismatch lines are printed. + Manual benchmarking runs keep the current default and their per-solution table. + - **How a run is identified as a test run:** the `tensile_args` fixture in + `Tests/conftest.py` always appends `--global-parameters ClientLogLevel=0`, placed *before* + any user-supplied `--global-parameters` so a developer can still override it (e.g. + `ClientLogLevel=3`; Tensile applies repeated `--global-parameters` in order, last wins, and + they override the YAML). No Tensile code change. Covers `common/` and `custom/` YAMLs. + A dedicated `TestRun` global parameter was considered and not chosen: nothing else needs a + "this is a test" signal. + - **Test:** move the body of `tensile_args` into a plain helper and unit-test it: + `ClientLogLevel=0` is present, and a user-supplied `--global-parameters ClientLogLevel=3` + comes after it. (A `LogReporter` gtest and an end-to-end check were considered and not + chosen.) +- **D2. Skip the granularity/memory values when nothing consumes them.** + `BenchmarkTimer::preSolution` does not compute or report its 9 granularity/memory values unless + the stdout row will be printed or `CSVExportExtraCols` is set. +- The results-file CSV (`ResultFileReporter`) is handled in 5b (write it only when consumed). + +Not adopted: + +- **B. Cheaper formatting** (`to_chars`, no row copy, `'\n'` + per-problem flush). Dropped in + favour of not producing the output; only the D1 level check is kept. + +Deferred: + +- **D3. Typed reporter keys** in `MetaResultReporter`. + +#### 5b. Correctness-only runs + +Adopted: + +- **E. Infer "nothing is timed" in the client.** Skip `projectedPerformance` and the per-solution + performance calculation; report the NaN values directly, reproducing today's `-nan` text. Rule + (static, from the client's own arguments): nothing is timed when `num-syncs-per-benchmark == 0`, + or `max-enqueues-per-sync == 0`, or (`num-enqueues-per-sync == 0` **and** + `min-flops-per-sync == 0`). Note: prototype `corr`'s `num-enqueues-per-sync == 0` alone is wrong. + Per-solution run-time skips (restore-from-log, skip-slow) are unchanged. D2 falls under this. +- **F. No warmup timing events when nothing is timed.** Do not create the per-kernel start/stop + `hipEvent`s; only `BenchmarkTimer` reads them. The topic 3 warmup-0 attribution sync uses + `hipStreamSynchronize` so it does not depend on events. +- **G. `gemm/gfx950/xfp32.yaml`: drop `NumWarmups: 5` and `EnqueuesPerSync: 10`.** The two + work-stealing configs keep their deliberate extra warmups (now checked by the `postSolution` + Synchronizer check). +- **H. Copy the results CSV only if it exists** (`BenchmarkProblems.main`, same guard as the + granularity CSV). +- **I. Write the results CSV only when consumed** (the YAML has a `LibraryLogic:` section or uses + the Ductile backend). Otherwise Tensile passes "no results file" and the client has an explicit + path that never constructs `ResultFileReporter`. +- **J. Extend the `SyncsPerBenchmark` guard test** (`Tests/unit/test_common_yaml_syncs_per_benchmark.py`) + to `Tests/custom/`, and set `SyncsPerBenchmark: 0` in the 12 custom YAMLs. +- **K. Remove `LibraryLogic:` / `LibraryClient:` from common YAMLs**, with a whitelist test (same + style as the guard test: mandatory reason, stale entries fail). Scope: `common/` and `custom/`. + Whitelist the 6 files that test the logic/library path with real benchmark data + (`gemm/fp8nfp16mix_hfp8ns`, `gemm/xfp32`, `groupedgemm/grouped_gemm`, + `groupedgemm/gfx11/grouped_gemm_gfx11`, `groupedgemm/grouped_gemm_ck_gfx942`, + `sparse/gfx94x/libray_logic`). Remove from the 14 `ductile/*`, 10 `gemm/gfx950/*mxfp8*` and 12 + `custom/*` files. + - **`Prediction` coverage: replace the 10 mxfp8 sections with one new, minimal, non-MX + `Prediction` config** (option c). The 10 gfx950 mxfp8 configs are today the only end-to-end + coverage of: writing a `Prediction`-type logic file (LibraryLogic step) → building a library + from it → the client selecting a solution through Origami (`ProblemPredictionLibrary:: + findTopSolutions` → `origami::rank_configs`). `Prediction` ships in 9 hipBLASLt logic files + (gfx1170 ×6, gfx1201 ×3, all `Origami/`), all non-MX, while the mxfp8 tests pass + `a/b_mx_block_size = 0` to Origami, so they do not exercise it as shipped. + - New YAML: a non-MX type (e.g. BF16 or F16), as small as possible: very small problem sizes + in both the benchmark step and `LibraryClient`, minimal solution set, numerical validation + off (`NumElementsToValidate: 0`); it only has to exercise logic write → library build → + Origami selection. + - It is added to the K whitelist (reason: only end-to-end `Prediction` coverage); the 10 mxfp8 + configs lose their `LibraryLogic:` / `LibraryClient:` sections. + - To verify when implementing: (1) with validation off and `SyncsPerBenchmark: 0`, the + LibraryLogic step still keeps the solutions (`removeInvalidSolutions` on NaN results) and + writes a non-empty library; (2) the LibraryClient step still performs Origami selection and + fails the test if no solution is selected, even when no kernel is launched; (3) which archs + it can run on: selection needs an Origami hardware model + (`pAMDGPU->analyticalHardware`); without one `findTopSolutions` returns nothing. + +### 6. `python_kernel_build_src_co` + +Status: decided. + +Adopted: + +- **A. One xnack-agnostic helper object in benchmark/test builds.** The benchmark path + (`BenchmarkProblems.cmdLineArchsFor` → `gfxToVariants` → `gfxVariantMap`) compiles the helper + `Kernels.cpp` for the bare arch (`gfx950`) instead of `gfx950:xnack+` and `gfx950:xnack-`. + This matches what already ships: shipped libraries build helpers from CMake `GPU_TARGETS` + (bare arch, xnack-agnostic; `xnack+` only for ASAN), the installed library contains only + `Kernels.so-000-gfx950.hsaco`, assembly kernels are already xnack-agnostic, and the runtime + loader tries the bare-arch file first. Measured −47 to −53% wall and CPU; code identical to + today's `xnack+` object. +- **C. Split `Kernels.cpp` into chunks, compile them in parallel, link once, and cache each + compiled chunk.** + - Kernels defined in `Kernels.h` (the 14 `Dijk_*_Reduction` kernels in gradient/bias configs) + move into the `.cpp`, otherwise chunks fail to link with duplicate symbols. + - Compile each chunk to a relocatable device object and link once with `lld -shared`, using the + driver's own `cc1` / `lld` commands (as in the experiment), which kept every kernel identical. + `-fgpu-rdc` is not used: it changed every kernel's code. + - Chunk count: small (the measurement flattens beyond 4) and bounded by the pool budget from + topic 7, since chunks run in parallel with each other and, with F, with codegen. + - **Per-chunk cache:** each compiled chunk object is cached (same pattern and location as the + helper-kernel cache and the PCH), keyed on the chunk's source, the headers it includes + (including the generated `Kernels.h`), the exact compile arguments and the compiler build. + Chunk assignment must be deterministic and content-stable (e.g. grouped by kernel, not by + balancing line counts over the whole file), so that adding or removing one kernel does not + change every chunk's content. + - Evidence to weigh: within the 4 captured configs, a per-kernel cache would have skipped only + 6% of kernel compiles beyond today's whole-file cache; cross-config reuse was not measured + (production builds were cold per config). + - **Test requirement:** measure with a full `--build-only` run starting from an empty cache, to + get the real chunk hit rate and wall time (and compare against a run without chunk caching). + - The PCH (E) cuts the fixed header cost per chunk from ~400 ms to ~35 ms, which removes most of + C's extra CPU. +- **E. Cached precompiled header of the HIP headers** for the helper-kernel compile. CI keeps + `~/.tensile` (or `TENSILE_HELPER_CACHE_DIR`) across configs, so the PCH is built once per + job and reused. + - PCH contents: HIP headers only (implicit `__clang_hip_runtime_wrapper.h`, + ``, ``, ``), built with + `-fpch-instantiate-templates`; used with `-include-pch`. TensileLite static headers stay out + (they are re-copied per build with new mtimes, which clang rejects). + - Stored next to the helper-kernel cache, using `HelperKernelCache`'s pattern (SHA-256 key, + write to `.tmp__` then `rename`, age-based eviction). + - Key: the exact argument list used to build it (every `-D`, target and features, `-O`, `-std`, + sanitizer flags), the full `amdclang++ --version` output (exact compiler build), and the + target. A `-D` baked into a PCH otherwise leaks silently into compiles that do not define it. + - On any PCH error (e.g. headers modified by a ROCm reinstall), rebuild the PCH or compile + without it and retry, so a stale entry never fails a build. + - Measured: −0.3 s per translation unit and target (≈170 s across the aggregate with both + targets, about half with A), code identical. +- **F. Start the helper compile before codegen in the benchmark path** + (`writeSolutionsAndKernels`): `writeHelpers` only needs `kernelHelperObjs`, available on entry, + so the compile runs while kernel codegen, assembly and the asm link run. Same approach as + PR #13671 on the TensileCreateLibrary path; no coordination needed. Upper bound on time hidden: + ~24% of compile time today (estimate; codegen and the compile share cores). + +Deferred: + +- **D. Shrinking the `PostGSU16` reduction kernels.** Changes GSU-path code; needs a runtime + benchmark of the reduction kernels first. + +Not adopted: + +- **B. `-parallel-jobs=2`.** Not needed with A (one target). + +### 7. `python_kernel_codegen` + +Status: decided. + +Adopted: + +- **A. Remove `CpuThreads: 1` from all 65 `common/` test YAMLs that set it** (19 of them ran in + the gfx950 aggregate, 189 s of codegen; the setting was present when each file was added and no + PR explains it). Measured on `subtile_mxfp4`: codegen 133.5 → 22.6 s, `write_assemble` + 62.9 → 11.6 s, build 244 → 63 s, `.co` and libraries identical. + - **Prerequisite: serial and parallel codegen must produce the same metadata.** `LdsNumBytes` + (set in `Components/Subtile/SubtileLdsLayout.py`) and `LoopIters` (set in `KernelWriter.py`) + are written onto the kernel during codegen and are lost in worker processes, because + `passPostKernelInfoToSolution` (`TensileCreateLibrary/Run.py`) only copies back + `CUOccupancy`, `PrefetchGlobalRead`, `MathClocksUnrolledLoop` and `CustomKernel`. Add both to + the returned `KernelCodeGenResult` so `Data/*.yaml` matches in both modes. +- **B. Fail fast on kernels that will be rejected.** Both checks: + 1. after `_initKernel`, stop if the static VGPR allocation (`states.totalVgprs`) exceeds + `MaxVgpr` (the VGPR pool can only grow, so rejection is certain); + 2. stop as soon as any `overflowedResources` code is set (VGPR, SGPR, LDS scheduling, LDS), at + the code sites that set it. + - Respect `forceGenerateKernel` (no early exit when set); keep the same error code (`-2`); print + the reason under `PrintSolutionRejectionReason`, as the end-of-codegen check does today. + - Validated in the prototype on 4,313 kernels: 0 accept/reject mismatches; CPU on rejected + kernels 374.5 → 10.8 s; `f8_gate_r` build 28 → 11 s, artifacts byte-identical. +- **C. Generate all steps of a config in one pool.** Restructure `BenchmarkProblems` into: + solutions for all steps → one codegen pool → per-step assemble/build/write (or keep one pool fed + across steps). Acceptance check: byte-identical artifacts per step against today (step + independence was inferred from the loop, not proven). Simulated: `custom_mainloop_scheduling` + 36 → 2.1 s, `f8_gate_r` 12.4 → 3.3 s; ≤25-kernel steps across the aggregate 142 → ≤75 s. +- **Pool budget: 2× the available CPUs across all pytest-xdist workers** (replaces the body of + `CPUThreadCount()` in `Common/Parallel.py`, as sketched in the note): + - CPUs available = min(affinity mask, loky `cpu_count()` (applies the cgroup quota), + `KUBE_CPU_REQUEST` when set and parseable). + - Concurrent workers = `PYTEST_XDIST_WORKER_COUNT`, else 1. + - Pool size = max(1, min(CPUs, 2 × CPUs // workers)); `CpuThreads` stays a cap (`-1` = budget, + N = min(budget, N)). + - Accepted limitations: `KUBE_CPU_REQUEST` is a request, not a limit (TheRock already treats it + as the pod's budget); only pytest-xdist parallelism is detected. + - Also fix `ParallelMap2` overwriting an explicit `procs` with `CPUThreadCount()` (so + `TensileLogic/Run.py`'s `procs=jobs` takes effect), and have TensileCreateLibrary's + `_cpuCount()` use the same helper. + +Deferred (see [DEFERRED.md](DEFERRED.md)): D (longest-first dispatch), E (cheaper generator), +and moving the VGPR/SGPR checks into solution validation. + +### 8. `python_kernel_bench_setup` + +Status: decided. + +Adopted: + +- **F2. Build helper kernel objects once per distinct input, in-process.** In + `writeBenchmarkFiles` (`BenchmarkProblems.py`), memoize `initHelperKernelObjects` on the + solution fields it reads (`ProblemType`, `GlobalSplitU`, `GlobalSplitUAlgorithm`, + `isStreamK`, `StreamKAtomic`, `_GlobalAccumulation`, `AdaptiveGemmGSUA`, `ActivationFused`, + `WavefrontSize`, `ISA`, plus the global `internalParameters["GlobalSplitUPGR"]`). The key is + computed inside `KernelHelperNaming`, next to the code that reads these fields, so they cannot + drift apart. The memo is scoped to one `writeBenchmarkFiles` call; the existing name-based + dedupe keeps the kept list and its order unchanged. Measured 82–85× on the block, output + identical on all 45 captured steps. +- **F1. `DataType.__deepcopy__` returns `DataType(self.value)`**, so the class-level + `properties` row is shared again instead of copied. Measured 1.4–1.8× on this block; also sped + up `python_benchpost_lib_construction`; artifacts byte-identical. Affects every `deepcopy` that + touches a `DataType` (nothing assigns `.value` / `.properties` outside `__init__`). +- **Investigation (findings only, no fix): TensileCreateLibrary helper name/object mismatch.** + `generateKernelHelperObjects` (`TensileCreateLibrary/Run.py`), the path that builds shipped + libraries, dedupes helper kernels on names predicted by the `*KernelNames` functions and aligns + them with the built objects positionally (`itertools.compress`). The predicted names do not + match the built objects: they use the solution's `GlobalSplitU` instead of `GlobalSplitUPGR` + (`PostGSU1` vs `PostGSU16`), the vector-width lists disagree (`isDouble()` vs `numBytes()`; 3 + names vs 2 objects for `cgemm`, 3 vs 1 for `zgemm`), and `MultipleBufferSingleKernel` ignores + `AdaptiveGemmGSUA`. On every captured benchmark step the final set happened to be right. + - Task: determine whether any real configuration (shipped logic files, all archs) drops a + helper, duplicates one, or pairs a name with the wrong object; write the findings to a + markdown file in this directory. Do not change the code as part of this plan. + +Not adopted: + +- **Persisting the F2 memo to a file** across runs. After F1 + F2 only a few seconds remain + across the aggregate, and a file cache keyed on solution fields would replay stale helper + source after generator edits unless keyed on the generator code itself. Compiled helpers are + already cached across runs by content hash (`HelperKernelCache`). +- **Reusing TensileCreateLibrary's `generateKernelHelperObjects`**: different helper order + (changes `Kernels.cpp` and its cache key), and its name prediction is unreliable (see the + investigation above). + +## Implementation plan + +This plan implements every adopted item above. Deferred work is in [DEFERRED.md](DEFERRED.md) +and is out of scope. + +### Rules for every change + +1. **One adopted item per commit.** Each item below (numbered `Pn.m`) is its own commit. An + item that is split into steps (e.g. test infrastructure, then the change) gets one commit per + step. Unrelated items are never combined, and commits are not squashed later. +2. **Commit messages say exactly why.** Conventional-commit subject (`perf(tensilelite): …`, + `fix(tensilelite): …`, `test(tensilelite): …`, `docs(tensilelite): …`). The body states: + - **What** changed, in concrete terms (files, functions, behaviour). + - **Why**: the problem, with the measured numbers that motivated it, and a reference to the + investigation note and section (e.g. `timing_runs/cpu_reference_slow_path_investigation.md`, + "Why the slow path is slow") and the topic/item in this file. + - **Expected effect**, marked as measured or estimated. + - **How it was verified** (tests run, equivalence checks, measurements). + - **Risks / behaviour changes**, or "none (output identical)". +3. **Every commit builds and passes its tests.** At minimum the tests listed for that item, plus + the existing suites it touches (`CPUGemm.*` CTest for `cpu-gemm-driver`, `tensilelite-tests`, + TensileLite Python unit tests). No commit relies on a later one to be correct. +4. **No behaviour change beyond what was decided.** Output-identical items are checked to be + identical (byte-identical artifacts, identical client stdout/CSV/exit codes as stated per + item). +5. **Timing results are committed.** Every aggregate measurement run (P0, P3, P6) commits its + `AGGREGATE_analysis.txt` (as the existing `per_config_rerun_2` / `per_config_build` results + are) plus a short comparison note against the baseline, in its own `docs(tensilelite): …` + commit that names the measured commit hash, client binary, host, cores and GPU. +6. **Never touch the measured baseline binaries.** `/tmp/tl_client_msgpack` and + `/tmp/tl_split_gfx950/artifacts` stay as they are; new builds go to separate directories. +7. **Supporting changes travel with the item that needs them.** A timer added by an item updates + `scripts/analyze_timing.py`'s `TIMING_HIERARCHY` in the same commit; a disable switch needed for + a measurement (PCH, chunk cache) lands with that feature. Infrastructure that no single item + owns (script parametrization) is its own item (P0.0). +8. **Temporary investigation code is never committed.** Examples: rejection-reason logging used + for the P1.12 replay, the `hipMemcpy` hash/readback shims, the corruption shim used for + P1.27's injected-failure check. If a check is worth keeping, it becomes an explicit opt-in debug + mode, stated in that item. +9. **Numbers in commit messages are labelled.** Numbers from the investigations were measured + with the `4c6426cea6` client; per-commit "expected effect" is an estimate unless measured at + that commit. Say which. + +### Ordering and dependencies + +```text +P0 baseline + build/test setup + │ + ├─ P1 client (C++) run-side changes ── P2 Python/YAML/harness run-side changes ── P3 run-side measurement + │ (P1.1 first: KMP_BLOCKTIME reduces variance for everything after it) + │ │ + │ P4 build-side changes (land after P3) ──┴── P6 build-side measurement + │ (topic 8 → topic 7 → topic 6) + │ + └─ P5 TensileCreateLibrary helper-name investigation (findings only; read-only, any time) +``` + +Hard dependencies inside phases are listed per item ("Needs: …"). + +P4 commits land after P3's measurement, not interleaved with P1–P3 (plan review): the +measurement scripts clear `~/.tensile` (where the PCH and chunk caches live), pin cores 0–24, and +P3 rebuilds artifacts from the branch head, so interleaved build-side changes would contaminate +the run-side measurement. P4 code may be developed earlier on a separate branch, but nothing is +built or measured on the measurement cores while a measurement runs. P2.5/P2.6 also change which +configs have LibraryLogic/LibraryClient steps, so the P6 note reports those configs separately. + +### P0. Baseline and setup + +- **P0.0 Parametrize the measurement scripts.** `split/run_split.sh`, + `per_config/run_per_config.sh` and `per_config/run_per_config_build.sh` hardcode `CLIENT`, + `ART`, `OUT` and clear `/home/alvasile/.tensile*`. Make `CLIENT`, `ART`, `OUT` and the cache + directories (`TENSILE_HELPER_CACHE_DIR` etc.) overridable, with today's values as defaults, so + P0/P3/P6 never overwrite the committed baseline artifacts and never share caches with + development builds. Own commit. +- **P0.1 Build tree with tests.** Configure a separate build directory (not + `/tmp/tl_client_msgpack`) from the starting commit with `TENSILELITE_BUILD_TESTING=ON` and + `TENSILELITE_ENABLE_CLIENT=ON`, Release, `GPU_TARGETS=gfx950`, same compiler as the baseline. + Targets used throughout: `tensilelite-client`, `cpu-gemm-driver`, `tensilelite-tests`. +- **P0.2 Baseline at the starting commit.** The committed baselines (`per_config_rerun_2`, + `per_config_build`) were measured with a client built at `4c6426cea6`, older than the starting + commit. Re-measure at the starting commit with the same scripts so every comparison is + like-for-like: + - Build artifacts: `full_runs_gfx950/split/run_split.sh` build phase into a new artifact dir. + - Run side: `full_runs_gfx950/per_config/run_per_config.sh` (serial, `TimingInstrumentation=True`, + 25 pinned cores, `OMP_NUM_THREADS=25`) with `OUT` pointing at a new + `full_runs_gfx950/baseline_run/` and the new client/artifacts. + - Build side: `full_runs_gfx950/per_config/run_per_config_build.sh` (cold cache per config) with + `OUT=full_runs_gfx950/baseline_build/`. + - Record the full common-test pass/fail set (split run phase, `-n 4`) as the correctness + baseline. + - Also record the **uninstrumented** wall time of that split run phase (`-n 4`): P1.20 adds + syncs under instrumentation only, so instrumented wall time understates production gains. + - Keep the raw per-config logs (`*.log.gz`) so P0 can be re-aggregated with the final + `analyze_timing.py` hierarchy before comparing. + - Noise: record host load (`uptime`, other users' processes) and GPU clocks at the start and + end of each run; repeat the ~10 most expensive configs 3× for a noise band. + - Commit both `AGGREGATE_analysis.txt` files, the pass/fail list and a note with commit, host, + cores, GPU (rule 5). + +### P1. Client (C++) run-side changes + +All in `projects/hipblaslt/tensilelite/client` unless stated. Order within P1 is the listed order. + +**Topic 5a, A — OpenMP workers** + +- **P1.1 `KMP_BLOCKTIME=0` by default.** In `main.cpp`, before the first OpenMP use: if + `KMP_BLOCKTIME` is not set in the environment, make idle workers sleep immediately + (`kmp_set_blocktime(0)`, or `setenv("KMP_BLOCKTIME","0",0)` before the runtime initializes; + pick whichever is verified to apply to the worker pool used by later parallel regions). + The decision names `kmp_set_blocktime(0)`; if `setenv` is chosen, the commit message says why. + `kmp_set_blocktime` is libomp-only, so guard it for other OpenMP runtimes / Windows builds. + `cpu-gemm-driver` does not get this; pin `KMP_BLOCKTIME` explicitly when quoting driver timings. + - Verify: per-thread CPU sampled from `/proc//task/*/stat` during a dtl solution loop shows + no worker above 25% CPU (the note measured 11–24 spinning workers before); the >1 ms share of + the bookkeeping timers drops on a dtl replay; an explicit `KMP_BLOCKTIME` in the environment + still wins. + +**Topic 1 — CPU reference** (`client/src/Reference.cpp`, `client/cpu_gemm_driver.cpp`, +`tests/CMakeLists.txt`, `tests/ReferenceFastPath_test.cpp`, `tests/ReferenceMXFastPath_test.cpp`) + +Every fast-path extension follows the established pattern of the FP8/FP4 (#8106), f64/tf32 +(#9203) additions: extend `isFastPathEligible` only for what `solveCPUFast` implements in the same +commit; add `cpu-gemm-driver` support for the new case; add fast/slow `CPUGemm.*` CTest pairs that +validate both paths against the driver's golden GEMM; add gtest cases for exactness edge cases. + +- **P1.2 Slow path: lock only for AMAX-D (A).** In `ReferenceSolution<…>::SolveCPU`, enter the + critical section only when `problem.outputAmaxD()`; drop the per-element + `omp_set_num_threads`. + - Verify: existing `CPUGemm.slow_*` tests; `cpu-gemm-driver` 1024³/2048³ fp32 slow path + bit-identical to before (max diff 0) and faster (note: 1,727 → 191 ms; 45,237 → 1,507 ms). +- **P1.3 AMAX-D reduction and thread cap (A+).** Replace the AMAX-D critical section with a + per-thread maximum combined once; delete the `omp_set_num_threads(MAX_OMP_THREADS)` calls in + `SolveCPU` and `ReductionCPU` (setting `omp_get_max_threads()` would be a no-op). Behaviour + change to state in the commit: today the first slow-path call permanently sets the main + thread's team size to 64, so every later OpenMP region (fast path, data init) runs 64 threads + on 25 pinned cores; after this, regions follow `OMP_NUM_THREADS`. + - Verify: new driver support for AMAX-D output if absent, with fast/slow-agnostic tests that + AMAX-D equals the max |D| computed by the golden GEMM; existing tests unchanged. +- **P1.4 Driver: output type separate from accumulator; Int8 inputs.** Test infrastructure only: + `cpu-gemm-driver` gains an output (C/D) type option and alpha/beta type, Int8 A/B storage, and + golden-GEMM support for them (integer accumulation for Int8, the slow path's epilogue type, + `SaturateCast` semantics on the output). Slow-path CTest cases for the new combinations land + here and must pass before any fast-path change. + - The slow path accumulates float-alpha Int8 types (`I8_*_S`) in `float` per K element, so it + is inexact once |sum| > 2²⁴. Slow-path CTest sizes must keep |sum| < 2²⁴ (the default sizes + do: ~151·127² ≈ 2.4M). +- **P1.5 Fast path: separate MAC type from epilogue type.** Refactor `solveCPUFast` to take a + multiply-accumulate type in addition to `AccumT` (epilogue type), defaulting to `AccumT`. No new + types enabled. Verify: all existing `CPUGemm.fast_*` and gtests unchanged (bit-identical). +- **P1.6 Fast path: Int8 inputs.** Int8 A/B shadow buffers, `int32_t` accumulation, epilogue in + `int32_t` (int32 alpha: `I8_I8_I32`, `I8_I32_I32`) or `float` (float alpha: `I8_I32_S`, + `I8_I8_S`, `I8_H_S`, `I8_B_S`); Half/BF16 outputs reuse existing write-back. Enable these type + combinations in `isFastPathEligible`. Output casts must reproduce the slow path's + `SaturateCast` exactly (verified in `Reference.cpp`): + - Int8 output: from a float epilogue, round-to-even then clamp to [−128, 127]; from an int32 + epilogue, clamp only. + - Int32 output: the generic overload, i.e. a plain `static_cast` (truncation, no rounding, no + clamp; out-of-range is undefined behaviour). Tests stay in range. + - **Epilogue operation order (decided):** the Int8 fast-path epilogue applies alpha, the + ScaleAB scales, `scaleAlphaVec`, beta·C, bias, activation and the output cast in exactly the + slow path's order and types (`ReferenceSolution<…>::SolveCPU`), so Int8 outputs match the slow + path exactly, including round-to-even ties. Tests compare fast vs slow exactly for integer + outputs. + - Tests: fast/slow CTest pairs for each output type (I8, I32, Half, BF16) × transposes, beta, + bias, activation, ScaleAB; gtests for exact integer results above 2²⁴ for int32-epilogue + types (for float-epilogue types the expected value is `float(exact sum)` then the epilogue), + and Int8 saturation at ±127/−128. + - Commit message notes that for large-K `I8_*_S` problems the reference changes (now exact in + the dot product, matching the GPU's int32 accumulation better than the slow path did). +- **P1.7 Fast path: Float8 outputs.** `storeFrom` for Float8/BFloat8 (and `_fnuz`) outputs + matching `SaturateCast` (including the BFloat16-accumulator path); enable `F8×F8→F8`, the BF8 + and hybrid variants, and `F8×Half→F8` (mixed-input shadow with compute-input quantization). + - Tests: driver support for F8/BF8 outputs; fast/slow pairs per variant; gtest for rounding and + saturation at the F8 range limits. +- **P1.8 Fast path: complex types (C).** `ComplexFloat` and `ComplexDouble` A/B/C/D, including + conjugate ops (`TensorOp::ComplexConjugate`), matching the slow path's `Transform`/`multiply`. + - Tests: driver complex support; fast/slow pairs incl. conjugate A/B; gtest for conjugation. +- **P1.9 Fast path: `useE` (C).** Store the pre-activation value to E in the epilogue + (`SetValue` semantics as the slow path; not with gradient). + - Tests: driver E output; fast/slow pairs with activation. +- **P1.10 Fast path: gradient (C).** Gradient GEMM: E-based activation derivative multiply, and + the bias reduction (`ReductionCPU` for bias from D/A/B, with the D-source workspace). + - Tests: driver gradient + bias-reduction support; fast/slow pairs for each bias source. +- **P1.11 Fast path: MX K not a multiple of the MX block (C).** Handle a partial last MX block + (K = 333/255/63 with block 32 in `mxfp4_mxfp4_bf16_tn_act`). + - Tests: MX gtests and CTest cases with K tails on both sides; existing MX tests unchanged. +- **P1.12 Topic 1 replay.** After P1.2–P1.11: replay the 59 slow-path YAMLs (as in the note's + method, with uncommitted local rejection-reason logging, rule 8) and confirm every config still + passes. Expected remaining slow-path reasons: `NumElementsToValidate: 128` (sparsity + threshold) plus any of amaxD, scaleCD, mirror indices, multi-index, no batch indices, + non-E8 / one-sided MX that occur; anything else is a finding. Results go into the P3 comparison + note, not a separate commit. + +**Topic 4 — `cpu_data_init`** + +- **P1.13 MX generator: opt out of the reference floats.** Add an opt-out to `generateMXInput` + (`clients/common/src/mxDataGen.cpp`, `clients/common/include/mxDataGen.hpp`) that skips + `getReferenceFloat()`; default unchanged. The TensileLite client's two call sites opt out. + - Verify: `tests/MXDataGen_test.cpp`; build the hipBLASLt clients that use the shared code + (`clients/common`) and run their MX tests (`fast_check_gtest` MX cases), since P0.1 builds only + TensileLite targets; generated data/scale bytes identical with and without the opt-out. +- **P1.14 Trig init without per-element overhead.** In `initArrayTrig` only (decision 2a): + no per-element `std::vector`, flat index for packed strides; keep double `sin`/`cos`. Verify: + generated arrays byte-identical to before for packed and strided descriptors (new gtest in + `DataInit_test.cpp`). +- **P1.15 Dynamic schedule for init loops.** `schedule(dynamic, chunk)` on the init loops. + Verify: Trig/Serial/Identity outputs byte-identical; timing under a contended core set. + +**Topic 2 — input data preparation** (`client/include/DataInitialization.hpp`, +`client/src/DataInitialization.cpp`) + +- **P1.16 Copy the current problem's extent, not `maxElements`.** `copyInputs` / + `copyInputBuffers` / per-problem `copyValidToGPUBuffer` copy the current problem's allocated + extent. In NaN / guard-page modes the guard regions already span the whole buffer + (`padding = (maxElements − allocated)/2`), so the change matters in Disable mode. + - Verify: Disable-mode multi-size configs where the size shrinks and then grows; bounds-check + configs (`nan_bounds_check_odd_padding`, `bufferLoadStore0`); swizzled tensors + (`getSwizzledTensorNumAllocatedElements`), padded MX scales, sparse metadata, grouped GEMM + offsets and rotating-buffer mode. +- **P1.17 Skip unchanged per-solution uploads (fix 1).** The host epoch alone is not enough: + GPU `valid` buffers have other writers (verified): the initial copy, the MX swizzled upload, + every branch of `copySwizzledToGPUBuffer`, and the grouped path. Track, per (tensor, dtype), a + tag of what the GPU `valid` buffer actually holds (host epoch plus layout/source); every writer + sets or invalidates it; the initial state is "unknown". Skip the H2D only when the tag matches. + - Verify: an opt-in debug mode (kept, rule 8) that on every skip reads `gpuInput.valid` back + and compares it with the expected bytes, run once over the full common suite with zero + mismatches; MX, sparse, swizzled, Trig and bounds-check configs pass. +- **P1.18 Cache the batched-pointer arrays (fix 2).** The pointers are `current + padding`, and + padding depends on the bounds-check mode, which flips GuardPageFront → GuardPageBack between the + two iterations of one GuardPageAll solution without any epoch or address change. Key the cache + on the computed per-tensor start pointers plus the descriptors (equivalently: rebuild when the + problem, `m_curBoundsCheck`, swizzle size or sparse metadata changes). + - Verify: grouped/batched, GuardPageAll (`bufferLoadStore0`) and sparse configs pass. +- **P1.19 MX swizzled scale from the canonical scale (fix 4).** Two commits: + 1. Test first: gtest that permuting the generator's **compact** canonical scale (before the + K-fast restride that `cpuInput.valid` receives; `generateMXInput(…, GFX950)` permutes the + compact scale) with the existing `applyMXScaleLayoutInPlace` (`mxDataGen.cpp`) is + byte-identical to `generateMXInput(…, GFX950)` output, for FP4/FP8, K-fast and K-slow + layouts, a padded leading dimension, and batched problems. Also check that the data bytes + from the two generator calls are identical (today's code pairs call 2's data with call 1's + scale). Check the batched case carefully: today's swizzled per-batch size can exceed + `scaleBatchStrideBytes`, so "identical to today" may encode overlapping batches; if so, stop + and report rather than reproduce it. + 2. Change: per-solution MX handling never regenerates data or canonical scale; keep the compact + canonical scale per problem (or un-restride), permute it once per problem, cache it in its + own GPU buffer so canonical- and swizzled-layout solutions do not overwrite each other; + `m_inputEpoch` is not bumped. + - Verify: `pre_solution_mx_side_regenerate` and `cpu_reference_gemm_per_solution_inputs_changed` + drop to zero on `subtile_mxfp4` / `subtile_mxfp8`; MX configs pass. + +**Topic 3 — Synchronizer check and attribution syncs** (`client/src/SynchronizerValidator.cpp`, +`client/include/SynchronizerValidator.hpp`, `client/main.cpp`) + +- **P1.20 Attribution syncs under timing instrumentation.** In `main.cpp`, only when + `g_timingInstrumentationEnabled`: `hipStreamSynchronize` after warmup 0 (before + `listeners.validateWarmups`) under its own timer, and at the end of each run iteration under its + own timer, **before** the per-iteration Synchronizer check (P1.22), so that check does not absorb + queued warmups. It goes **inside the `try`, inside the `needMoreRunsInSolution` loop**, never + between the `catch` and `postSolution`: a sticky device error there would otherwise throw out of + the solution loop. Uses the stream, not timing events + (needed by P1.28). Verify: with instrumentation off, no extra HIP calls (count with a shim or + trace); with it on, `validate_synchronizer_readback` / `validate_gpu_readback` no longer contain + kernel time; a forced kernel fault still reaches the existing device-lost handling. +- **P1.21 Host check: OR-reduction first (1-alt).** Rewrite the clean-path scan as a vectorizable + OR over the buffer; run the count/first-offset scan only when dirty. Diagnostic unchanged. + Verify: `SynchronizerResidue_test.cpp` (clean, dirty, first offset, counts) and a speed check. +- **P1.22 Check in `postSolution` with guards.** Save the `ProblemInputs` pointer; do the check + inside `SynchronizerValidator::postSolution` (it already runs after `ReferenceValidator`'s, so + its `FAILED` overrides the reference verdict; keep registration order unchanged); guard 1 (only + if the solution launched: flag from `preWarmup` / `preEnqueues`, plus `m_mayUseSynchronizer`); + guard 2 (exception path: check if the device is usable, otherwise catch and mark `FAILED`, never + throw); keep clear-on-dirty. GuardPageAll (two iterations per solution): the check itself runs + at the end of every run iteration, before the next iteration's input reset (inside the + `needMoreRunsInSolution` loop, after the iteration's launches; guards 1–2 apply per iteration), + and `postSolution` reports the accumulated verdict. + - Also verify: a GuardPageAll config with residue injected in the first iteration is reported. + - Verify: `SynchronizerValidatorReporting_test.cpp` extended for: residue from a later launch + attributed to its own solution; a dirty buffer gives `FAILED` even when the reference check + passed; no check for non-launched solutions; residue after an `INVALID` launch with a usable + device (state which verdict wins); device-lost path; residue reported exactly once. +- **P1.23 GPU OR-reduction check.** OR-reduction kernel on the client's stream + 4-byte readback; + full readback and existing diagnostic only when nonzero; host check (P1.21) used under the FFM + emulator (`HSA_MODEL_MEMFILE`) and as a fallback when the kernel cannot launch on the device + (no code object for the arch: `hipErrorNoBinaryForGpu` / invalid device function). + - Verify: residue injected into the buffer is detected with identical diagnostic; per-check + time; Stream-K/MBSK/amaxD configs (`streamk/*`, work-stealing configs) pass. + +**Topic 5 — bookkeeping and correctness-only runs (client side)** + +- **P1.24 Predicate dump behind its own option (5a C).** New client option, default off, gating + `debugEval` in `SolutionIterator::checkSolution`; wired to a new Tensile global parameter in + `ClientWriter.py` / `Common/GlobalParameters.py` (Python part in the same commit, so the option + is usable end to end). Verify: dump appears only with the option; rows unchanged. +- **P1.25 `LogReporter`: check the row level before copying (5b D1).** Verify: stdout identical at + every log level on a dtl invocation (normalizing the `srand seed` line). +- **P1.26 Skip unused granularity/memory values (5a D2).** In `BenchmarkTimer::preSolution`, skip + `projectedPerformance` and the 9 granularity/memory reports unless a `LogReporter` will print + the row (stdout at its level, or the `--log-file` reporter `main.cpp` adds at + `LogLevel::Normal`) or `CSVExportExtraCols` is set. Applies to every run, timed or not. + - Verify: stdout, log file and results CSV byte-identical at every log level, with and without + `--log-file` and `CSVExportExtraCols`. +- **P1.27 Infer "nothing is timed" (5b E).** Static rule from the client's arguments: + `num-syncs-per-benchmark == 0`, or `max-enqueues-per-sync == 0`, or + (`num-enqueues-per-sync == 0` and `min-flops-per-sync == 0`). When it holds: skip the + performance calculation in `BenchmarkTimer::postSolution` and report the same NaN values (same + `-nan` text). + - Verify: results CSV and stdout byte-identical against the previous commit on dtl and xfp32 + invocations, clean and with injected failures (local corruption shim, not committed), exit + codes identical; the four syncs/enqueues/min-flops combinations from the note behave as + tabled. +- **P1.28 No warmup timing events when nothing reads them (5b F).** Needs: P1.20, P1.27. + `BenchmarkTimer` reads warmup events in `postWarmup` when `skip-slow-solution-ratio` is set and + in `validateWarmups` under `--sync-after-warmups` (verified). Create the events unless nothing is + timed **and** the skip-slow ratio is 0 **and** `--sync-after-warmups` is off; otherwise keep + them. + - Verify: as P1.27; plus a nothing-timed run with skip-slow set; `BenchmarkTimer` unaffected + when timing is on. +- **P1.29 Client "no results file" path (5b I, client side).** An explicit option that never + constructs `ResultFileReporter`. Verify: no file written, exit code and stdout unchanged. + +### P2. Python, YAML and test-harness run-side changes + +- **P2.1 Copy the results CSV only if it exists (5b H).** `BenchmarkProblems.main`, same guard as + the granularity CSV. Unit test. +- **P2.2 Write the results CSV only when consumed (5b I, Tensile side).** Needs: P1.29, P2.1. + In `BenchmarkProblems` only (not in the shared `ClientWriter.writeClientConfig`, which + `TensileRetuneLibrary.py` also uses and which copies `benchmark.csv`): pass "no results file" + unless the config has a `LibraryLogic:` section or uses the Ductile backend. Behaviour change to + state in the commit: the CSV's existence is also the "already benchmarked, skip" marker + (`BenchmarkProblems.py`), so reruns into the same output directory re-run the client. Unit + tests for both branches; a LibraryLogic config and a Ductile config still produce their + outputs. +- **P2.3 Test runs use `ClientLogLevel=0` (5a D1).** `tensile_args` in `Tests/conftest.py` + always appends `--global-parameters ClientLogLevel=0` before any user-supplied + `--global-parameters`. The fixture body moves into a plain helper with a unit test: the value is + present, and a user-supplied `ClientLogLevel=3` comes after it. +- **P2.4 `xfp32.yaml`: drop `NumWarmups: 5` and `EnqueuesPerSync: 10` (5b G).** +- **P2.5 New minimal `Prediction` config (5b K).** One small non-MX YAML (BF16 or F16): very small + sizes in the benchmark step and `LibraryClient`, minimal solutions, `NumElementsToValidate: 0`. + Must first confirm the three verification points in topic 5 (non-empty library with validation + off; LibraryClient performs Origami selection and fails if none is selected; which archs have an + Origami hardware model) and restrict the config to those archs. If LibraryClient turns out + **not** to fail when nothing is selected, stop and ask before changing code (new scope). +- **P2.6 `LibraryLogic:` / `LibraryClient:` whitelist test and removals (5b K).** Needs: P2.5. + Two commits: the whitelist test (initially listing every current file, with reasons); removal + from the 14 `ductile/*`, 10 mxfp8 and 12 `custom/*` YAMLs, shrinking the whitelist to the 6 + kept files + the new `Prediction` config (verification: the stale-entry check fires on the + removed entries). +- **P2.7 `SyncsPerBenchmark` guard covers `custom/` (5b J).** Needs: P2.6. Must come after the + `custom/*` `LibraryLogic:` removal: all 12 custom YAMLs have `LibraryLogic:`, and with + `SyncsPerBenchmark: 0` their logic step would get NaN results that `removeInvalidSolutions` + (drops only `gflops == 0`) does not filter (verified). Two commits: set `SyncsPerBenchmark: 0` + in the 12 custom YAMLs (run all 12 at that commit), then extend + `Tests/unit/test_common_yaml_syncs_per_benchmark.py` to `Tests/custom/`. + +### P3. Run-side measurement + +- Rebuild the client at the end of P2; regenerate artifacts (split build phase) at that commit. +- Full common-test pass (split run phase, `-n 4`): the pass/fail set must equal the P0 baseline, + except for intended changes (listed in the note). +- Per-config run aggregate (`run_per_config.sh`, same settings as P0) into + `full_runs_gfx950/after_run/`. +- Commit `AGGREGATE_analysis.txt` and a comparison note (`docs(tensilelite): …`): total and + per-category deltas vs P0, the P1.12 replay result, and anything that moved unexpectedly + (attribution changes are expected: wait time moves to the new sync timers under instrumentation). + +### P4. Build-side changes + +Order: topic 8 (P4.1–P4.2), pool budget and fail-fast (P4.3–P4.6), parallel codegen metadata and `CpuThreads` (P4.7–P4.8), compile/codegen overlap (P4.9), one pool per config (P4.10), then the helper compile itself (P4.11–P4.15). P4 commits land after P3 (see "Ordering and dependencies"). + +**Topic 8 — `python_kernel_bench_setup`** + +- **P4.1 F1: `DataType.__deepcopy__` returns `DataType(self.value)`.** Verify: TensileLite unit + tests; artifacts of `fp16_tn` and `fp16_HH_BHS_bf16mfma` byte-identical (excluding `.hsaco` + `__hip_cuid_*`). +- **P4.2 F2: memoize `initHelperKernelObjects` per step.** Key computed in `KernelHelperNaming`; + memo scoped to one `writeBenchmarkFiles` call. Verify: ordered helper list (names, header, + source) identical on the captured steps; artifacts byte-identical; a unit test that records the + solution keys the helper writers actually read while running `initHelperKernelObjects` (e.g. + through a recording dict proxy) and asserts that set is contained in the memo key. + +**Topic 7 — `python_kernel_codegen` and pool sizing** + +- **P4.3 Pool budget (2× CPUs across xdist workers).** `CPUThreadCount()` in `Common/Parallel.py` + as decided, with unit tests (mocked affinity / loky count / environment, as in the prototype). +- **P4.4 `ParallelMap2` honours `procs`; TensileCreateLibrary `_cpuCount()` uses the shared + helper.** Separate commits. +- **P4.5 Fail fast: static VGPR check after `_initKernel` (B.1).** +- **P4.6 Fail fast: stop when `overflowedResources` is set (B.2).** + - Verify (both): accept/reject decision identical for every kernel of the probed steps (the + note's `ff.py` method); `PrintSolutionRejectionReason` output still names the reason; + `forceGenerateKernel` still generates; artifacts byte-identical. +- **P4.7 Return `LdsNumBytes` and `LoopIters` from codegen workers (A prerequisite).** Add them + to what `passPostKernelInfoToSolution` copies back. Verify: a generic test comparing the full + post-codegen kernel/solution state between serial and parallel codegen (not only these two + keys), on `subtile_mxfp4` and `custom_mainloop_scheduling`; `Data/*.yaml` identical. +- **P4.8 Remove `CpuThreads: 1` from the 65 `common/` YAMLs that set it (A).** Needs: P4.3, + P4.7. The 6 commented-out `CpuThreads: 1` lines (`common/sparse/*`) are removed in the same + commit. Verify: `--build-only` serial vs parallel for all 65 YAMLs on their own archs (only 19 + run on gfx950), artifacts identical; the gfx950 ones pass. +- **P4.9 Start the helper compile before codegen in the benchmark path (topic 6, F).** Moved + before the codegen restructure (plan review): in today's per-step structure, start + `writeHelpers` + the helper compile on entry to `writeSolutionsAndKernels`, since it only needs + `kernelHelperObjs`. Verify: artifacts byte-identical; total processes stay within the P4.3 + budget. +- **P4.10 One codegen pool per config (topic 7, C).** Restructure `BenchmarkProblems` to generate + all steps' kernels in one pool, then assemble/build/write per step; keep F's overlap by + launching every step's helper compile before the shared codegen pool, with the pool budget + split between helper compiles/chunks and codegen workers. Verify: per-step artifacts + byte-identical to before on configs with many small steps (`custom_mainloop_scheduling`, + `f8_gate_r`, `spmm_i8_sb`) and large ones (`dtl`); per-step `--use-cache` hits still skip + codegen; peak memory with all steps' kernels held at once. + +**Topic 6 — helper-kernel compile** + +(F, the compile/codegen overlap, is P4.9 above.) + +- **P4.11 xnack-agnostic helper compile in benchmark/test builds (A).** `cmdLineArchsFor` gives + the bare arch for the helper compile, except when sanitizer flags are present, where it keeps + `xnack+` (matching shipped ASAN builds). `gfxVariantMap` also covers gfx906/908/90a/942. + State in the commit: `cmdLineArchsFor` also feeds the benchmark cache key (`_Architectures`) + and `HelperKernelCache`'s key, so existing `--use-cache` / helper-cache entries are invalidated + once. Verify: helper kernels identical to today's `xnack+` object (disassembly with addresses + stripped); the runtime loads `Kernels.so-000-gfx950.hsaco`; configs pass; a build-only check on + gfx942 as well. +- **P4.12 Cached HIP-header PCH (E).** As decided. The key uses the full `amdclang++ --version` + output (not `HelperKernelCache`'s `major.minor.patch`) and the resolved ROCm/HIP include path, + plus the exact argument list and target. Includes a switch to disable the PCH. Verify: kernels + identical with and without the PCH; stale-PCH fallback tested (touch a header's mtime); a cold + and a warm compile timed. +- **P4.13 Move header-defined kernels into `Kernels.cpp` (C prerequisite).** The 14 + `Dijk_*_Reduction` kernels. Verify: kernels identical; gradient/bias configs pass. +- **P4.14 Chunked helper compile (C).** Needs: P4.11–P4.13. Deterministic, content-stable chunk + assignment; each chunk includes a generated header with only its own kernels' declarations (plus + the static headers and shared argument structs), so the per-chunk cache key does not depend on + the whole `Kernels.h`; driver `cc1`/`lld` commands with `-include-pch` passed through; fall back + to the unchunked compile on any failure of the chunked path; chunk count bounded by the P4.3 + budget. Verify: every kernel and its metadata identical to the unchunked build; the fallback + path exercised by a test. +- **P4.15 Per-chunk compile cache (C).** Needs: P4.14. Keyed on the chunk source, its generated + header, the static headers, exact compile arguments and the full compiler version. Includes a + switch to disable the chunk cache (needed for P6). Verify: hit/miss behaviour under unit test; + the empty-cache full `--build-only` run in P6. + +### P5. Investigation: TensileCreateLibrary helper name/object mismatch (findings only) + +Independent; any time. Determine whether `generateKernelHelperObjects` can drop, duplicate or +mispair a helper for any real configuration (shipped logic files, all archs). Write the findings +to a markdown file in this directory and commit it. No code change. + +### P6. Build-side measurement + +- Per-config build aggregate, cold cache per config (`run_per_config_build.sh`, same settings as + P0) into `full_runs_gfx950/after_build/`: comparable with the P0 baseline. +- **Full `--build-only` run of all configs from an empty cache, cache kept across configs** (the + split build phase, `-n 4`, CI-like): measures the PCH and per-chunk cache hit rates and wall + time; repeat once with chunk caching disabled for comparison (required by topic 6, C). +- Commit both aggregates and a comparison note (rule 5). + +### Plan review + +The plan was critically reviewed by a separate agent (Claude Opus 5.5) against the decisions and +the code. Corrections incorporated above: + +- Measurement isolation: P0.0 (parametrized scripts, separate caches), P4 lands after P3, + uninstrumented wall time, raw logs kept, noise band, analyzer hierarchy kept current. +- Synchronizer: listener order corrected (verdict override relies on reverse `postSolution` + order); end-of-iteration sync placed inside the `try`; device fallback for the GPU check. +- CPU reference: Int8/Int32 output-cast semantics corrected to match `SaturateCast`; + float-accumulator slow-path limits for Int8 tests; `omp_set_num_threads` calls deleted, with + the process-wide effect stated; expected remaining slow-path reasons listed for P1.12. +- Input preparation: upload skip keyed on what the GPU buffer actually holds (all writers), with + a readback debug mode; batched-pointer cache keyed on computed pointers and descriptors; + extent-sizing verification in Disable mode and for swizzled/MX/sparse/grouped/rotating tensors; + MX swizzle permutes the compact canonical scale via `applyMXScaleLayoutInPlace`. +- Bookkeeping: D2 split from E and conditioned on every `LogReporter` (incl. `--log-file`); + warmup events kept when skip-slow or `--sync-after-warmups` read them; results-CSV switch only + in `BenchmarkProblems`, with the rerun-marker behaviour change stated. +- YAML ordering: `custom/` `LibraryLogic:` removal before `SyncsPerBenchmark: 0`. +- Build side: F moved before the one-pool restructure; xnack+ kept for sanitizer builds and the + cache-key invalidation stated; PCH key uses the full compiler version and include path; chunk + headers per chunk, fallback to the unchunked compile, cache/PCH disable switches; stronger + equivalence tests for F2 and serial-vs-parallel codegen. +- Commit rules 7–9: supporting changes travel with their item, investigation code is never + committed, and numbers in commit messages are labelled measured or estimated. + +Decisions raised by the review (both decided): + +1. **Synchronizer coverage for GuardPageAll:** check at the end of every run iteration, before the + next input reset; report in `postSolution` (P1.22). Accepting the coverage loss was not chosen. +2. **Int8 epilogue matching:** the Int8 fast-path epilogue mirrors the slow path's operation order + and types, for an exact match (P1.6). A shared epilogue function for all types and a ±1 test + tolerance were not chosen. diff --git a/projects/hipblaslt/tensilelite/timing_runs/DEFERRED.md b/projects/hipblaslt/tensilelite/timing_runs/DEFERRED.md new file mode 100644 index 000000000000..b9981a4cc511 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/DEFERRED.md @@ -0,0 +1,153 @@ +# Deferred timing work + +Work that came out of the timing investigations in this directory and was deliberately **not** +included in the current implementation plan. Each entry says where it comes from, what it is, +what is known, and what it would take to pick it up. Decisions themselves are recorded in +[DECISIONS.md](DECISIONS.md). + +Status values: `deferred` (worth doing later), `not for now` (rejected for the current plan, but +could be revisited), `needs data` (blocked on a measurement). + +## Summary + +| # | Item | Source | Status | Size of the prize (as measured or estimated) | +| --- | --- | --- | --- | --- | +| 1 | Restore only writable tensors in `copyInputs` | Input data preparation, fix 3 | not for now | `copy_inputs_d2d` 26.3 s timer, real cost 4.5–5.5× that | +| 2 | Tune the host-side Synchronizer check for the FFM emulator | SynchronizerValidator, item 1-alt | deferred | unknown (emulator cost not measured) | +| 3 | Cheaper trig argument (`sinf` / argument reduction) | `cpu_data_init`, item 2b | not for now | trig init 22.1 → ~3.7 ns/element; changes generated data | +| 4 | MX data generator: single-threaded zero-fill and poor thread scaling | `cpu_data_init`, finding 1 | deferred | ~90 ms per 516M-element side; 7–11× on 25 threads | +| 5 | First-call fixed cost per client process | `cpu_data_init`, finding 4 | needs data | 54 s of the 106.8 s `cpu_data_init` is first calls | +| 6 | Typed reporter keys in `MetaResultReporter` | Per-solution bookkeeping, D3 | deferred | not measured | +| 7 | Shrink the `PostGSU16` reduction kernels | `python_kernel_build_src_co`, D | deferred | 48–59% of each remaining helper compile | +| 8 | Longest-first dispatch within a codegen step | `python_kernel_codegen`, D | deferred | 0–6% on most configs, 24% on `spmm_i8_sb` (upper bound) | +| 9 | Cheaper kernel generator (epilogue, `hasAnyDependency`) | `python_kernel_codegen`, E | deferred | epilogue 40–64% of per-kernel time; `hasAnyDependency` 25–67% on some sparse kernels | +| 10 | Reject over-budget kernels in solution validation, before codegen | `python_kernel_codegen`, B (follow-up) | deferred | removes the remaining rejected-kernel codegen entirely | + +## 1. Restore only writable tensors in `copyInputs` + +- **Source:** [input_data_preparation_investigation.md](input_data_preparation_investigation.md), + finding 3; topic 2 in DECISIONS.md. +- **What:** on the slow reset path, `copyInputs` restores *every* tensor (D2D, GPU valid → GPU + current) before each solution, including read-only A and B. Only tensors kernels can write need + restoring: D, E, AMAX-D, bias in gradient mode, the Synchronizer, and possibly others. +- **Known:** `hipMemcpy` D2D returns in ~2 µs, so `gpu_input_reset_copy_inputs_d2d` (26.3 s) is + deflated; the real serialized cost was 4.5–5.5× the timer on `spmm_f16` and + `bufferLoadStore0`. The split between necessary and avoidable bytes was not measured. +- **Why not now:** too easy to get the writable set wrong, and a full restore also masks a kernel + that corrupts its inputs. +- **To pick up:** enumerate the writable set per problem type from the kernel arguments (not by + hand), add a debug mode that verifies read-only inputs are unchanged after a solution, then + measure the avoidable bytes. + +## 2. Host-side Synchronizer check for the FFM emulator + +- **Source:** [SynchronizerValidator.md](SynchronizerValidator.md), item 1-alt; topic 3. +- **What:** the adopted design keeps a host-side check (full readback, then a vectorizable + OR-reduction, with the count/first-offset scan only when dirty) for runs under the FFM emulator + (`HSA_MODEL_MEMFILE`), where a 409,600-element reduction kernel may be much slower than a copy. +- **Known:** on hardware, OR-reduction first measured 11.3 µs warm / 32.6 µs right after the D2H + copy for 1.6 MB, against 165 µs for today's scalar scan. Nothing was measured under emulation. +- **To pick up:** measure both the GPU check and the host check under the emulator; choose per + mode, and tune (e.g. explicit SIMD, chunked readback) based on the measurement. + +## 3. Cheaper trig argument in `initArrayTrig` + +- **Source:** [cpu_data_init_investigation.md](cpu_data_init_investigation.md), finding 2; + topic 4, item 2b. +- **What:** trig init calls double-precision `sin`/`cos` of the element index. glibc takes its + slow large-argument path above ~1.06e8 (23.6–25 ns vs 6.9 ns). `sinf` + store measured + 3.7 ns/element; reducing the argument first is another option. +- **Why not now:** it changes the generated input data, so runs before and after would not use + identical inputs. The value-preserving part (no per-element allocation, flat index) is adopted. +- **To pick up:** decide whether input-data compatibility with earlier runs matters; if not, + switch to `sinf` or a bounded argument and re-baseline. + +## 4. MX data generator: zero-fill and thread scaling + +- **Source:** [cpu_data_init_investigation.md](cpu_data_init_investigation.md), finding 1; + topic 4. +- **What:** in `generateMXInput` (`clients/common/src/mxDataGen.cpp`, shared hipBLASLt client + code), `generate()` starts with a single-threaded `m_dataBytes.resize(n)` zero-fill (~90 ms for + 516 MB), and the whole call reaches only 7–11× on 25 threads (20,935 → 2,094 ms for 516M + elements). The discarded reference-float vector is handled separately (adopted). +- **To pick up:** profile `generate()` per stage on a large side; parallelize or avoid the + zero-fill; coordinate with hipBLASLt client owners since the code is shared. + +## 5. First-call fixed cost per client process + +- **Source:** [cpu_data_init_investigation.md](cpu_data_init_investigation.md), finding 4. +- **What:** the first `cpu_data_init` call in each client process (820 calls) is 54.0 s of the + 106.8 s. Standalone, the first H2H `hipMemcpy` after HIP init costs ~18 ms even for 1 KB, and + the first OpenMP parallel region ~3 ms; in-client first calls were larger (e.g. 53.3 ms to copy + 1,024 elements in `spmm_i8`) and were not explained without a profiler. Configs split into many + small client processes pay it repeatedly. +- **To pick up:** profile a first call (perf was unavailable on the investigation host); options + include warming HIP/OpenMP at client start-up off the timed path, or fewer client processes per + config. + +## 6. Typed reporter keys + +- **Source:** [per_solution_bookkeeping_investigation.md](per_solution_bookkeeping_investigation.md), + option D; topic 5a, D3. +- **What:** `MetaResultReporter` fans every `report()` out to each reporter with a `std::string` + key (and copies a `shared_ptr` per reporter, `for(auto r : m_reporters)`); each reporter then + compares the key against its list. Replace with typed keys. +- **Known:** not measured. Much of the fan-out goes away with the adopted items (no granularity + reports when unused, no stdout rows in tests, no results CSV unless consumed). +- **To pick up:** re-measure the per-solution bookkeeping section after the adopted items land; + only worth doing if `report()` still shows up. + +## 7. Shrink the `PostGSU16` reduction kernels + +- **Source:** [python_kernel_build_src_co_investigation.md](python_kernel_build_src_co_investigation.md), + option D; topic 6. +- **What:** the "universal" `PostGSU16` helper kernels are 4,260–5,865 lines each (vs 100–145 for + `PostGSU8/4/2/1`). `KernelWriterConversion.kernelBody` emits a software-pipelined reduction with + a fully unrolled 15-case remainder `switch`, each case re-emitting all 16 accumulate steps, with + every inline-asm add duplicated under `#if defined(__gfx950__)`. Size grows roughly with GSU². +- **Known:** removing these kernels entirely cut a one-target compile by 48–59% (bound, not a + proposal). Options: generate the remainder as a loop, lower `GlobalSplitUPGR`, or share one + remainder path. +- **Why deferred:** it changes the code of kernels on the GSU path. +- **To pick up:** write a runtime benchmark of the reduction kernels (all GSU values and vector + widths), then compare the alternative generators against it. + +## 8. Longest-first dispatch within a codegen step + +- **Source:** [python_kernel_codegen_investigation.md](python_kernel_codegen_investigation.md), + option D; topic 7. +- **What:** joblib dispatches kernels in input order; dispatching the most expensive kernels first + shortens the step's tail. +- **Known:** with perfect knowledge of per-kernel cost (simulated): 0–6% on most probed configs, + 24% on `spmm_i8_sb` (17.6 → 13.4 s). Output size is not known before generation, so a cost + predictor is needed. With topic 7's C (one pool per config), the tail effect shrinks further. +- **To pick up:** re-measure after C lands; if tails still matter, build a cheap cost predictor + (e.g. from tile sizes, epilogue features, or a previous run's per-kernel times). + +## 9. Cheaper kernel generator + +- **Source:** [python_kernel_codegen_investigation.md](python_kernel_codegen_investigation.md), + option E; topic 7. +- **What:** per-kernel time is ~65% Python instruction-tree construction, ~21% `rocIsaPass`, ~8% + freeing the tree and ~6% `str()`. Two concrete targets: + - **Epilogue:** `notLocalSplitUGlobalWrite` is 40–64% of profiled time in 22 of 28 sampled + kernels; it is fully unrolled per element for every variant (edge/non-edge, beta, bias type, + activation). The code itself notes it (`KernelWriter.py`: "Kernels with epilog especially with + activation is too long (50000~ lines). Need to refactor global write elements."). + - **`KernelWriter.hasAnyDependency`:** a local-read × instruction scan, quadratic; 25–67% of + profiled time in four `spmm_i8_sb` kernels (903k calls in one), 0% in `dtl`. A contained + target. +- **To pick up:** start with `hasAnyDependency` (index the dependencies instead of rescanning); + the epilogue is a larger refactor that changes generated code and needs performance validation. + +## 10. Reject over-budget kernels before codegen + +- **Source:** [python_kernel_codegen_investigation.md](python_kernel_codegen_investigation.md), + option B; topic 7. +- **What:** the adopted fail-fast stops codegen early for kernels that are certain to be rejected + (static VGPR allocation over `MaxVgpr`, or any `overflowedResources` code set). Moving the + VGPR/SGPR checks into solution validation would keep these kernels out of codegen altogether. +- **Known:** not tested. After the adopted checks, rejected kernels cost ~10.8 s of CPU across + the probed configs (down from 374.5 s). +- **To pick up:** check whether the static allocation can be computed without `_initKernel`, and + whether rejecting at validation changes which solutions are reported or counted. diff --git a/projects/hipblaslt/tensilelite/timing_runs/SynchronizerValidator.md b/projects/hipblaslt/tensilelite/timing_runs/SynchronizerValidator.md new file mode 100644 index 000000000000..fa9d30f8d78b --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/SynchronizerValidator.md @@ -0,0 +1,265 @@ +# `SynchronizerValidator` investigation (gfx950 per-config rerun 2) + +`full_runs_gfx950/per_config_rerun_2/AGGREGATE_analysis.txt` reports +`validate_warmups_listener.SynchronizerValidator` at 162,734.88 ms over 2,106,827 calls (6.3% of +wall). This note explains where that time goes, proposes moving the check to the GPU, and +describes an attribution edge case that any rework has to handle. Timing claims below come from +measurements; reading the code was only used to decide what to measure. Statements that are +inferred rather than measured are marked as such. + +| Timer | Count | Total (ms) | Mean (ms) | +| --- | --- | --- | --- | +| `validate_synchronizer_check` | 2,106,827 | 162,692.31 | 0.08 | +| ├─ `validate_synchronizer_scan` | 608,873 | 106,747.41 | 0.18 | +| ├─ `validate_synchronizer_readback` | 608,873 | 55,816.84 | 0.09 | +| └─ `validate_synchronizer_inactive` | 1,497,978 | 27.90 | 0.00 | + +All of the cost is in the 608,873 *active* checks; solutions that cannot use the Synchronizer cost +nothing. + +## Method + +- **Binary.** The measured client, `/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client` + (Release, `-O3 -DNDEBUG -std=c++20 -fPIC -mf16c -x hip --offload-arch=gfx950`). The linked + `scanSynchronizerResidue` was disassembled with `llvm-objdump`. +- **Microbenchmarks.** A verbatim copy of `scanSynchronizerResidue` compiled with the client's + flags; its assembly is instruction-for-instruction identical to the shipped binary. Timings are + medians over 300–1000 iterations with `std::chrono::steady_clock`. +- **In-client measurement.** The `dtl` and `subtile_mxfp4` configs re-run exactly as + `full_runs_gfx950/profiling/run_rerun.sh` does (cached artifacts, `TimingInstrumentation=True`, + `taskset -c 0-24`, `ROCR_VISIBLE_DEVICES=0`), with `LD_PRELOAD` shims that intercept `hipMemcpy`: + - *readback split:* for calls made from `SynchronizerValidator::readBuffer` (return address in + `[0x41b1f0, 0x41b3e8)`; the binary is not PIE), time a `hipDeviceSynchronize()` (waiting for + queued GPU work) separately from the copy. On an idle GPU that sync costs 0.07 µs, so the + "wait" figure is real waiting and not shim overhead. + - *reset tracking:* remember the device pointers `readBuffer` copies from, and count other + `hipMemcpy` writes into those pointers between checks. +- **Reproduction.** The re-run reproduces the production numbers for `dtl`: scan 8,376.56 ms vs + 8,313 ms, readback 3,214.89 ms vs 3,381 ms, both over 49,995 checks. +- **Environment.** gfx950 host, EPYC 9575F. `perf` was unavailable (`perf_event_paranoid=4`). +- Scratch sources (`bench.cpp`, `scan.cpp`, `shim.cpp`, `shim_reset.cpp`, `run_cfg.sh`) and logs + are in `/tmp/syncval_bench`. + +## What the check does and when it runs + +Stream-K flags, GSU/MBSK reduction counters and the amax counter all live in one shared +Synchronizer buffer of 409,600 alpha-typed elements (`ClientProblemFactory.cpp`), i.e. 1.6 MB for +a float alpha. Kernels must leave it zeroed on exit; a kernel that leaves residue can corrupt or +hang later launches. The check reads the buffer back after a launch, scans it for any nonzero +32-bit word, and if dirty logs the count and first offset, marks the solution `FAILED`, and +clears the buffer. + +Per solution (`main.cpp`): + +```text +prepareGPUInputs (input reset) +launch warmup 0 (async) +listeners.validateWarmups ← SynchronizerValidator, then ReferenceValidator, then BenchmarkTimer +launch warmups 1..N-1 +postWarmup +benchmark runs (syncs × enqueues; BenchmarkTimer::postEnqueues synchronizes) +postSolution +``` + +The check therefore sees exactly one launch per solution. + +## Slowness 1: the host scan (106.7 s) + +`scanSynchronizerResidue` tracks `nonzero` and `first` per element. In the shipped binary this +compiles to a scalar loop, unrolled 2×, with a loop-carried `cmov`/`add` dependency chain and no +SIMD: + +```text +41b610: testq %r8, %r8 + movq %rcx, %r11 + cmoveq %r9, %r11 + xorl %ebx, %ebx + cmpl $0x0, (%rdi,%r9,4) + setne %bl + cmoveq %rcx, %r11 + addq %r8, %rbx + ... (second element, same pattern) + addq $0x2, %r9 + cmpq %r10, %r9 + jne 41b610 +``` + +Measured (median, identical assembly): + +| Variant | 1.6 MB | 3.2 MB | 6.5 MB | +| --- | --- | --- | --- | +| Current scan, warm cache | 162.9 µs | 326.2 µs | 650.5 µs | +| Current scan, right after the D2H copy | 166.0 µs | 327.9 µs | 657.7 µs | +| Current scan, after evicting 1 GB of cache, then the copy | 164.8 µs | 327.3 µs | 655.8 µs | +| OR-reduction first (auto-vectorizes to 256-bit `vorps`), warm | 11.3 µs | 21.4 µs | 41.3 µs | +| OR-reduction first, right after the D2H copy | 32.6 µs | 66.2 µs | 131.1 µs | + +The current loop runs at a flat ~10 GB/s regardless of where the data is cached. It is bound by the +dependency chain (~0.4 ns per int), not by memory. In the client it measured 168 µs per check on +`dtl`, and per-config means in the aggregate are 0.165–0.175 ms for almost every config, consistent +with 1.6 MB buffers. 608,873 × ~165 µs ≈ 100 s, essentially the whole scan bucket. + +The count and first offset are only needed for the error message on a dirty buffer, which should +be rare. The clean path only needs "is anything nonzero". + +## Slowness 2: the readback (55.8 s) + +The readback is a blocking `hipMemcpy` D2H of the whole buffer into a pinned staging buffer. Its +time has three parts: + +1. **The copy itself.** ~37–40 µs per 1.6 MB in steady state, both in the microbenchmark (36.9 µs + pinned, 37.5 µs pageable, GPU idle) and in the client (37.9 µs on `subtile_mxfp4`). +2. **Waiting for the warmup kernel (misattributed).** Warmup 0 is launched asynchronously and + `SynchronizerValidator` is the first listener to block, so its readback absorbs the rest of the + kernel's runtime plus launch-to-completion latency. In the microbenchmark, a copy issued right + after launching a 0 / 50 / 200 µs kernel takes 53–58 / 104–106 / 254–255 µs. +3. **One-time and outlier copies.** The first copy in each client process is slow (up to 80 ms), + and some later copies take over 100 µs. The cause was not isolated. This matters for configs + split into many small client processes. + +Measured in the client with the readback-split shim: + +| | `dtl` (49,995 checks) | `subtile_mxfp4` (4,327 checks, 47 processes) | +| --- | --- | --- | +| Wait for queued GPU work | 1,184 ms (37%), ~20–29 µs/check | 156 ms (16%), ~36 µs/check | +| Copy, total | 2,009 ms (63%) | 800 ms | +| ├─ steady state | ~37–40 µs/copy | 151 ms (37.9 µs/copy) | +| ├─ first copy per process | small | 413 ms | +| └─ other copies > 100 µs | small | 236 ms (302 copies) | + +**Inferred aggregate split:** 608,873 × ~38 µs ≈ 23 s of the 55.8 s is steady-state copying. The +remaining ~33 s is waiting plus first-copy and outlier costs. That split was measured only on these +two configs. + +### The intended sync runs too late + +`--sync-after-warmups` (default `true`) makes `BenchmarkTimer::validateWarmups` call +`hipEventSynchronize` on the warmup stop event, timed as `validate_gpu_sync`. Listeners run +`validateWarmups` in registration order, and `BenchmarkTimer` is registered after both validators +(`main.cpp`). The GPU is already idle when it runs: + +| Run | `validate_gpu_sync` calls | Total | Mean | +| --- | --- | --- | --- | +| Aggregate | 2,106,827 | 6,867.77 ms | 3.3 µs | +| `dtl` re-run | 270,802 | 68.10 ms | 0.25 µs | + +When `SynchronizerValidator` is inactive, the same wait presumably lands in `ReferenceValidator`'s +`validate_gpu_readback` instead (inferred from the ordering, not measured). + +## Proposal: run the check on the GPU + +Replace the full readback and host scan with an OR-reduction kernel over the buffer, followed by a +4-byte readback of the result. Only read the whole buffer back when the result is nonzero, to +produce the existing diagnostic (nonzero count, first offset). + +```text +hipMemsetAsync(flag, 0, 4, stream) +orReduce<<<…, stream>>>(synchronizer, ints, flag) // atomicOr of per-thread ORs +hipMemcpyAsync(hostFlag, flag, 4, D2H, stream) +hipStreamSynchronize(stream) +if(*hostFlag) { full readback + existing scan for the message; mark FAILED; clear } +``` + +Measured on the 1.6 MB buffer (median): + +| Approach | Per check | +| --- | --- | +| Current: full D2H copy + scalar scan | ≈ 37 + 165 ≈ 200 µs | +| Full copy + vectorized OR-reduction on the host | ≈ 37 + 33 ≈ 70 µs | +| GPU OR-reduction + 4-byte readback | 16.8 µs | + +The GPU version removes both slow parts: the scan (≈ 100 s) and the 1.6 MB D2H transfer (≈ 23 s +steady state, plus the per-process first-copy costs). It adds ~608,873 × 16.8 µs ≈ 10 s. The +expected net saving is roughly 110 s of the 162.7 s (inferred from per-check measurements, not +measured end-to-end). The host-vectorized variant is a smaller change that keeps the copy and saves +roughly 80 s. + +**Separately, fix the wait attribution.** Synchronize on the warmup stop event in `main.cpp` right +after the first warmup launch, under its own timer, before `listeners.validateWarmups`. Then +neither validator absorbs kernel time. Reordering the listeners is riskier, because `postSolution` +runs them in reverse and the verdict ordering depends on that. + +### Interaction with the client execution lock + +None (code reading). `ClientExecutionLock` (`filelock.FileLock`, one lock file per physical GPU +under xdist) is held around the whole `tensilelite-client` subprocess in `ClientWriter.py` and +`ParallelExecution.py`. The reduction kernel is one more launch from the process that already +holds the lock. Points to watch: + +- Enqueue it on the client's `stream`, so it stays ordered after the solution's launches and + before anything that follows. +- It reads 1.6 MB through L2/MALL, which the DMA copy may not. At today's position, warmups 1..N-1 + follow it before the timed runs; at the proposed `postSolution` position (below), it runs after + them. Its effect on cache state was not measured. +- FFM emulator runs (`HSA_MODEL_MEMFILE`) use private per-worker locks. A 409,600-element kernel + may be much slower than a copy under emulation; a host fallback for that mode may be worth + keeping (not measured). +- `CpuOnly` skips the client launch entirely and is unaffected. + +## Edge case: residue attribution + +The check runs once per solution, after the **first** warmup launch. Residue left by any later +launch of the same solution is not observed at that solution's check. What happens to it depends +on whether the next solution's input reset re-zeroes the buffer. There is no dedicated "clear the +Synchronizer between solutions" step; kernels are trusted to self-clean. + +`prepareGPUInputs` (`DataInitialization.hpp`) has two paths: + +- **Slow path** (bounds checking enabled, problem-dependent data, or swizzled tensors). It calls + `copyInputs`, which copies *every* tensor from its pristine GPU copy, including the zero-filled + Synchronizer. The buffer is re-zeroed before every solution. **Measured** on `dtl`: all 49,995 + checks were preceded by a D2D write into the Synchronizer buffer (111,153 writes in total). +- **Fast path** (otherwise). It calls only `resetOutput`, which skips non-output tensors. The + Synchronizer is not marked as an output, so it carries over to the next solution. This is from + the code; not measured, because none of the cached configs take the fast path. + +| Path (aggregate reset counters) | Configs with checks | Checks | Residue from a launch after N's first warmup | +| --- | --- | --- | --- | +| Slow only, no fast-path resets (`dtl`, `subtile_mxfp4`, …) | 82 | 361,102 (59%) | wiped before N+1 runs; **never reported** | +| Fast (`fp16_tn`, `fp32_tn`, `bf16_tn`, `subtile_bf16`, … 22 configs) | 22 | 247,771 (41%) | carried over; **reported against the next active solution**, which is marked `FAILED` | + +On the fast path, the next active solution may also be a different problem (if no remaining +solution of the current problem can use the buffer), and its own first launch runs on stale +flags. Residue that appears on the *first* launch is attributed correctly on both paths; a +deterministic self-cleaning bug is therefore caught. Only residue that first appears on later +launches is affected: race- or timing-dependent behavior, a different kernel variant in later +warmups (`kernels[i % kernels.size()]`), or behavior that depends on the buffer's starting state. +Two more gaps: the last checked solution's late residue is never examined, and configs that +launch with zero warmups (`NumWarmups` 0 and validation off) never trigger `validateWarmups`. + +No dirty-buffer report ("Synchronizer left dirty") appears in any log examined (the `dtl` and +`subtile_mxfp4` re-runs, and the compressed logs under `timing_runs`). The edge case is a +correctness gap in the design, not an observed failure. + +### Requirement for the GPU check + +Whatever replaces the current check must observe **every** launch of a solution before the next +solution's launches, and must attribute the result to that solution. Running the GPU reduction +in `postSolution`, instead of after the first warmup, satisfies this: + +- It sees all warmups and benchmark runs of the solution, on both reset paths. That also closes + the slow-path gap, where late residue is currently wiped unseen. +- It is still one check per solution. +- The GPU is already idle there when benchmark runs happened (`BenchmarkTimer::postEnqueues` + synchronizes), so no kernel time is misattributed. Enqueuing on `stream` keeps it ordered either + way. +- `postSolution` receives no inputs, so the `ProblemInputs` pointer must be saved from + `validateWarmups` or `preSolution`. +- It must run before `ReferenceValidator::postSolution` reports the verdict, as the existing + registration order already arranges. +- Keep clearing the buffer after a dirty result: on the fast path nothing else will. + +Trade-off: today a dirty first launch is cleared before the solution's timed runs. With the check +in `postSolution`, a failing solution's timed runs execute on a dirty buffer. That solution is +marked `FAILED` either way. + +## Not measured / open items + +- End-to-end savings of the GPU check (only per-check costs were measured). +- Fast-path carry-over in a real run. A small fast-path config such as `benchmark_runs__fp32_tn` + (120 checks) or `streamk__sk_hgemm_race` (256 checks) would need to be built to show it with the + reset-tracking shim. +- Where the warmup wait lands when `SynchronizerValidator` is inactive. +- The cause of the slow first copy per process and of the >100 µs outlier copies. +- Cache effects of the reduction kernel on subsequent timed runs, and its cost under the FFM + emulator. diff --git a/projects/hipblaslt/tensilelite/timing_runs/build_fp16_gradient_bias_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/build_fp16_gradient_bias_analysis.txt new file mode 100644 index 000000000000..409207aa18c3 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/build_fp16_gradient_bias_analysis.txt @@ -0,0 +1,135 @@ +======================================================================================================================== +TIMING ANALYSIS SUMMARY +======================================================================================================================== + +TIMING BY PHASE +------------------------------------------------------------------------------------------------------------------------ + Category Count Total (ms) Mean (ms) % of Parent % of Wall +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 1 51,138.23 51,138.23 100.0% + -------------------------------------------------------------------------------------------------------------------- + python_kernel_compilation 6 48,227.66 8,037.94 94.3% 94.3% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_src_co 6 42,216.35 7,036.06 87.5% 82.6% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_kernel_build_src_co.compile 3 42,158.66 14,052.89 99.9% 82.4% + python_kernel_build_src_co.unbundle 3 30.17 10.06 0.1% 0.1% + python_kernel_build_src_co.cache_check 6 21.80 3.63 0.1% 0.0% + python_kernel_build_src_co.cache_populate 3 4.18 1.39 0.0% 0.0% + (untracked) 1.07 0.0% 0.0% + python_kernel_build_src_co.setup 6 0.25 0.04 0.0% 0.0% + python_kernel_build_src_co.move 3 0.22 0.07 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_codegen 6 2,695.94 449.32 5.6% 5.3% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_assemble 6 1,193.80 198.97 2.5% 2.3% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_setup 6 795.21 132.53 1.6% 1.6% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_validate 6 590.25 98.37 1.2% 1.2% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_helpers 6 298.19 49.70 0.6% 0.6% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_postprocess 6 209.56 34.93 0.4% 0.4% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_benchpost_lib_construction 6 142.55 23.76 68.0% 0.3% + python_benchpost_library_write 6 38.59 6.43 18.4% 0.1% + python_benchpost_naming 6 27.01 4.50 12.9% 0.1% + python_benchpost_client_config 6 1.11 0.18 0.5% 0.0% + (untracked) 0.30 0.1% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_co 6 178.51 29.75 0.4% 0.3% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 32.93 0.1% 0.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_setup 6 16.94 2.82 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_solution_generation 6 2,796.55 466.09 5.5% 5.5% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_forked_solutions 6 2,796.40 466.07 100.0% 5.5% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 0.11 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_fork_permutations 6 0.04 0.01 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_custom_kernels 6 0.01 0.00 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + (untracked) 43.17 0.1% 0.1% + -------------------------------------------------------------------------------------------------------------------- + python_write_solutions 6 41.53 6.92 0.1% 0.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_dump 6 40.12 6.69 96.6% 0.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_prepare 6 0.80 0.13 1.9% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_wsol_prepare_nocache 6 0.78 0.13 98.1% 0.0% + (untracked) 0.01 1.9% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 0.52 1.3% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_header 6 0.09 0.02 0.2% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_solution_indexing 6 22.83 3.81 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_write_cache 6 6.48 1.08 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_cache_check 6 0.01 0.00 0.0% 0.0% + +------------------------------------------------------------------------------------------------------------------------ + ESTIMATED WALL CLOCK TIME: 51.14s + +HIERARCHY VALIDATION: OK (all child phases fit within their parents) + +TIME BREAKDOWN (visual, % of wall clock) +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 100.0% ################################################## + python_kernel_compilation 94.3% ############################################### + python_kernel_build_src_co 82.6% ######################################### + python_kernel_build_src_co.compile 82.4% ######################################### + python_kernel_build_src_co.unbundle 0.1% + python_kernel_build_src_co.cache_check 0.0% + python_kernel_build_src_co.cache_populate 0.0% + (untracked) 0.0% + python_kernel_build_src_co.setup 0.0% + python_kernel_build_src_co.move 0.0% + python_kernel_codegen 5.3% ## + python_kernel_write_assemble 2.3% # + python_kernel_bench_setup 1.6% + python_kernel_validate 1.2% + python_kernel_write_helpers 0.6% + python_kernel_bench_postprocess 0.4% + python_benchpost_lib_construction 0.3% + python_benchpost_library_write 0.1% + python_benchpost_naming 0.1% + python_benchpost_client_config 0.0% + (untracked) 0.0% + python_kernel_build_co 0.3% + (untracked) 0.1% + python_kernel_setup 0.0% + python_solution_generation 5.5% ## + python_solgen_forked_solutions 5.5% ## + (untracked) 0.0% + python_solgen_fork_permutations 0.0% + python_solgen_custom_kernels 0.0% + (untracked) 0.1% + python_write_solutions 0.1% + python_wsol_dump 0.1% + python_wsol_prepare 0.0% + python_wsol_prepare_nocache 0.0% + (untracked) 0.0% + (untracked) 0.0% + python_wsol_header 0.0% + python_solution_indexing 0.0% + python_write_cache 0.0% + python_cache_check 0.0% + +KEY INSIGHTS (% of wall clock) +------------------------------------------------------------------------------------------------------------------------ + Python phases (code gen, compilation): 51.02s (99.8%) + Python I/O overhead (cache, YAML): 0.05s (0.1%) + C++ client setup (library loading): 0.00s (0.0%) + Data preparation (CPU init, GPU prep): 0.00s (0.0%) + Reference computation (CPU GEMM): 0.00s (0.0%) + Kernel execution (warmup, benchmark): 0.00s (0.0%) + + diff --git a/projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_1_build_cold_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_1_build_cold_analysis.txt new file mode 100644 index 000000000000..34b5d57d2e17 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_1_build_cold_analysis.txt @@ -0,0 +1,135 @@ +======================================================================================================================== +TIMING ANALYSIS SUMMARY +======================================================================================================================== + +TIMING BY PHASE +------------------------------------------------------------------------------------------------------------------------ + Category Count Total (ms) Mean (ms) % of Parent % of Wall +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 1 91,714.59 91,714.59 100.0% + -------------------------------------------------------------------------------------------------------------------- + python_kernel_compilation 9 74,126.10 8,236.23 80.8% 80.8% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_postprocess 9 25,589.12 2,843.24 34.5% 27.9% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_benchpost_lib_construction 9 17,320.21 1,924.47 67.7% 18.9% + python_benchpost_library_write 9 5,937.58 659.73 23.2% 6.5% + python_benchpost_naming 9 2,327.62 258.62 9.1% 2.5% + python_benchpost_client_config 9 2.83 0.31 0.0% 0.0% + (untracked) 0.87 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_setup 9 14,909.77 1,656.64 20.1% 16.3% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_codegen 9 10,359.67 1,151.07 14.0% 11.3% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_assemble 9 9,826.49 1,091.83 13.3% 10.7% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_validate 9 7,739.34 859.93 10.4% 8.4% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_src_co 9 3,590.01 398.89 4.8% 3.9% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_kernel_build_src_co.compile 1 3,558.18 3,558.18 99.1% 3.9% + python_kernel_build_src_co.cache_check 9 19.38 2.15 0.5% 0.0% + python_kernel_build_src_co.unbundle 1 8.88 8.88 0.2% 0.0% + (untracked) 1.89 0.1% 0.0% + python_kernel_build_src_co.cache_populate 1 0.85 0.85 0.0% 0.0% + python_kernel_build_src_co.setup 9 0.76 0.08 0.0% 0.0% + python_kernel_build_src_co.move 1 0.07 0.07 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_co 9 821.31 91.26 1.1% 0.9% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 636.74 0.9% 0.7% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_setup 9 611.45 67.94 0.8% 0.7% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_helpers 9 42.19 4.69 0.1% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_solution_generation 9 8,591.97 954.66 9.4% 9.4% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_forked_solutions 9 8,591.67 954.63 100.0% 9.4% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 0.23 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_fork_permutations 9 0.06 0.01 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_custom_kernels 9 0.02 0.00 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_write_solutions 9 4,141.41 460.16 4.5% 4.5% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_dump 9 4,012.82 445.87 96.9% 4.4% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_prepare 9 113.64 12.63 2.7% 0.1% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_wsol_prepare_nocache 9 113.60 12.62 100.0% 0.1% + (untracked) 0.04 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 14.75 0.4% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_header 9 0.20 0.02 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + (untracked) 2,531.63 2.8% 2.8% + -------------------------------------------------------------------------------------------------------------------- + python_solution_indexing 9 2,310.87 256.76 2.5% 2.5% + -------------------------------------------------------------------------------------------------------------------- + python_write_cache 9 12.60 1.40 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_cache_check 9 0.01 0.00 0.0% 0.0% + +------------------------------------------------------------------------------------------------------------------------ + ESTIMATED WALL CLOCK TIME: 91.71s + +HIERARCHY VALIDATION: OK (all child phases fit within their parents) + +TIME BREAKDOWN (visual, % of wall clock) +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 100.0% ################################################## + python_kernel_compilation 80.8% ######################################## + python_kernel_bench_postprocess 27.9% ############# + python_benchpost_lib_construction 18.9% ######### + python_benchpost_library_write 6.5% ### + python_benchpost_naming 2.5% # + python_benchpost_client_config 0.0% + (untracked) 0.0% + python_kernel_bench_setup 16.3% ######## + python_kernel_codegen 11.3% ##### + python_kernel_write_assemble 10.7% ##### + python_kernel_validate 8.4% #### + python_kernel_build_src_co 3.9% # + python_kernel_build_src_co.compile 3.9% # + python_kernel_build_src_co.cache_check 0.0% + python_kernel_build_src_co.unbundle 0.0% + (untracked) 0.0% + python_kernel_build_src_co.cache_populate 0.0% + python_kernel_build_src_co.setup 0.0% + python_kernel_build_src_co.move 0.0% + python_kernel_build_co 0.9% + (untracked) 0.7% + python_kernel_setup 0.7% + python_kernel_write_helpers 0.0% + python_solution_generation 9.4% #### + python_solgen_forked_solutions 9.4% #### + (untracked) 0.0% + python_solgen_fork_permutations 0.0% + python_solgen_custom_kernels 0.0% + python_write_solutions 4.5% ## + python_wsol_dump 4.4% ## + python_wsol_prepare 0.1% + python_wsol_prepare_nocache 0.1% + (untracked) 0.0% + (untracked) 0.0% + python_wsol_header 0.0% + (untracked) 2.8% # + python_solution_indexing 2.5% # + python_write_cache 0.0% + python_cache_check 0.0% + +KEY INSIGHTS (% of wall clock) +------------------------------------------------------------------------------------------------------------------------ + Python phases (code gen, compilation): 82.72s (90.2%) + Python I/O overhead (cache, YAML): 4.15s (4.5%) + C++ client setup (library loading): 0.00s (0.0%) + Data preparation (CPU init, GPU prep): 0.00s (0.0%) + Reference computation (CPU GEMM): 0.00s (0.0%) + Kernel execution (warmup, benchmark): 0.00s (0.0%) + + diff --git a/projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_2_build_warm_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_2_build_warm_analysis.txt new file mode 100644 index 000000000000..e2c90f700137 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_2_build_warm_analysis.txt @@ -0,0 +1,127 @@ +======================================================================================================================== +TIMING ANALYSIS SUMMARY +======================================================================================================================== + +TIMING BY PHASE +------------------------------------------------------------------------------------------------------------------------ + Category Count Total (ms) Mean (ms) % of Parent % of Wall +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 1 87,520.82 87,520.82 100.0% + -------------------------------------------------------------------------------------------------------------------- + python_kernel_compilation 9 70,010.08 7,778.90 80.0% 80.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_postprocess 9 25,558.91 2,839.88 36.5% 29.2% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_benchpost_lib_construction 9 17,315.62 1,923.96 67.7% 19.8% + python_benchpost_library_write 9 5,915.03 657.23 23.1% 6.8% + python_benchpost_naming 9 2,324.23 258.25 9.1% 2.7% + python_benchpost_client_config 9 3.07 0.34 0.0% 0.0% + (untracked) 0.96 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_setup 9 14,964.03 1,662.67 21.4% 17.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_codegen 9 10,395.52 1,155.06 14.8% 11.9% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_assemble 9 9,300.36 1,033.37 13.3% 10.6% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_validate 9 7,750.75 861.19 11.1% 8.9% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_co 9 804.50 89.39 1.1% 0.9% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_setup 9 611.86 67.98 0.9% 0.7% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 575.11 0.8% 0.7% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_helpers 9 30.33 3.37 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_src_co 9 18.71 2.08 0.0% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_kernel_build_src_co.cache_check 9 16.38 1.82 87.6% 0.0% + (untracked) 1.75 9.4% 0.0% + python_kernel_build_src_co.setup 9 0.57 0.06 3.1% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_solution_generation 9 8,573.30 952.59 9.8% 9.8% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_forked_solutions 9 8,572.97 952.55 100.0% 9.8% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 0.25 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_fork_permutations 9 0.06 0.01 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_custom_kernels 9 0.02 0.00 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_write_solutions 9 4,140.77 460.09 4.7% 4.7% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_dump 9 4,007.89 445.32 96.8% 4.6% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_prepare 9 119.71 13.30 2.9% 0.1% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_wsol_prepare_nocache 9 119.67 13.30 100.0% 0.1% + (untracked) 0.04 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 12.96 0.3% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_header 9 0.21 0.02 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + (untracked) 2,476.17 2.8% 2.8% + -------------------------------------------------------------------------------------------------------------------- + python_solution_indexing 9 2,307.97 256.44 2.6% 2.6% + -------------------------------------------------------------------------------------------------------------------- + python_write_cache 9 12.52 1.39 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_cache_check 9 0.01 0.00 0.0% 0.0% + +------------------------------------------------------------------------------------------------------------------------ + ESTIMATED WALL CLOCK TIME: 87.52s + +HIERARCHY VALIDATION: OK (all child phases fit within their parents) + +TIME BREAKDOWN (visual, % of wall clock) +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 100.0% ################################################## + python_kernel_compilation 80.0% ####################################### + python_kernel_bench_postprocess 29.2% ############## + python_benchpost_lib_construction 19.8% ######### + python_benchpost_library_write 6.8% ### + python_benchpost_naming 2.7% # + python_benchpost_client_config 0.0% + (untracked) 0.0% + python_kernel_bench_setup 17.1% ######## + python_kernel_codegen 11.9% ##### + python_kernel_write_assemble 10.6% ##### + python_kernel_validate 8.9% #### + python_kernel_build_co 0.9% + python_kernel_setup 0.7% + (untracked) 0.7% + python_kernel_write_helpers 0.0% + python_kernel_build_src_co 0.0% + python_kernel_build_src_co.cache_check 0.0% + (untracked) 0.0% + python_kernel_build_src_co.setup 0.0% + python_solution_generation 9.8% #### + python_solgen_forked_solutions 9.8% #### + (untracked) 0.0% + python_solgen_fork_permutations 0.0% + python_solgen_custom_kernels 0.0% + python_write_solutions 4.7% ## + python_wsol_dump 4.6% ## + python_wsol_prepare 0.1% + python_wsol_prepare_nocache 0.1% + (untracked) 0.0% + (untracked) 0.0% + python_wsol_header 0.0% + (untracked) 2.8% # + python_solution_indexing 2.6% # + python_write_cache 0.0% + python_cache_check 0.0% + +KEY INSIGHTS (% of wall clock) +------------------------------------------------------------------------------------------------------------------------ + Python phases (code gen, compilation): 78.58s (89.8%) + Python I/O overhead (cache, YAML): 4.15s (4.7%) + C++ client setup (library loading): 0.00s (0.0%) + Data preparation (CPU init, GPU prep): 0.00s (0.0%) + Reference computation (CPU GEMM): 0.00s (0.0%) + Kernel execution (warmup, benchmark): 0.00s (0.0%) + + diff --git a/projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_3_run_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_3_run_analysis.txt new file mode 100644 index 000000000000..d63f173e8a18 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_3_run_analysis.txt @@ -0,0 +1,173 @@ +======================================================================================================================== +TIMING ANALYSIS SUMMARY +======================================================================================================================== + +TIMING BY PHASE +------------------------------------------------------------------------------------------------------------------------ + Category Count Total (ms) Mean (ms) % of Parent % of Wall +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 1 41,022.07 41,022.07 99.8% + -------------------------------------------------------------------------------------------------------------------- + python_client_execution 9 40,589.17 4,509.91 98.9% 98.7% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + C++ Client Setup 12,560.29 30.9% 30.6% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + listener_setup 9 4,784.49 531.61 38.1% 11.6% + hip_initialization 9 3,647.34 405.26 29.0% 8.9% + library_loading 9 2,901.38 322.38 23.1% 7.1% + code_object_loading 9 694.46 77.16 5.5% 1.7% + data_init_setup 9 501.74 55.75 4.0% 1.2% + lazy_loading_init 9 29.58 3.29 0.2% 0.1% + reporter_setup 9 1.14 0.13 0.0% 0.0% + solution_iterator_setup 9 0.16 0.02 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Data Preparation 10,631.87 26.2% 25.9% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + pre_problem 54 10,407.95 192.74 97.9% 25.3% + cpu_reference_gemm 54 10,378.99 192.20 99.7% 25.2% + solve_cpu_slow 54 10,378.80 192.20 100.0% 25.2% + (untracked) 0.18 0.0% 0.0% + cpu_data_init 54 27.34 0.51 0.3% 0.1% + (untracked) 1.62 0.0% 0.0% + gpu_input_reset 45,792 214.59 0.00 2.0% 0.5% + gpu_input_preparation 54 7.80 0.14 0.1% 0.0% + rotating_buffer_preparation 54 1.53 0.03 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Reference Computation 5,199.40 12.8% 12.6% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + validate_warmups 45,792 5,199.40 0.11 100.0% 12.6% + validate_gpu_readback 45,792 3,537.91 0.08 68.0% 8.6% + validate_element_comparison 45,792 1,560.88 0.03 30.0% 3.8% + (untracked) 72.72 1.4% 0.2% + validate_gpu_sync 45,792 27.88 0.00 0.5% 0.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Kernel Execution 4,731.00 11.7% 11.5% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + post_solution 122,112 1,937.84 0.02 41.0% 4.7% + post_solution_log 122,112 1,604.87 0.01 82.8% 3.9% + post_solution_reporting 122,112 259.89 0.00 13.4% 0.6% + post_solution_perf_calc 122,112 46.18 0.00 2.4% 0.1% + post_solution_validation 122,112 25.99 0.00 1.3% 0.1% + post_solution_result_file 122,112 3.30 0.00 0.2% 0.0% + post_solution_sol_advance 122,112 3.22 0.00 0.2% 0.0% + post_solution_perf_reset 122,112 3.17 0.00 0.2% 0.0% + pre_solution 122,112 1,251.00 0.01 26.4% 3.0% + warmup_runs 91,584 1,090.34 0.01 23.0% 2.7% + kernel_solving 45,792 272.51 0.01 5.8% 0.7% + post_problem 54 102.96 1.91 2.2% 0.3% + solution_selection 122,112 70.89 0.00 1.5% 0.2% + benchmark_runs 45,792 5.45 0.00 0.1% 0.0% + (untracked) 5.45 100.0% 0.0% + finalize_report 9 0.01 0.00 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Timing Overhead 4,006.26 9.9% 9.7% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + flush_timing_buffer 9 3,917.83 435.31 97.8% 9.5% + timing_overhead 9 88.43 9.83 2.2% 0.2% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked C++ overhead) 3,460.37 8.5% 8.4% + -------------------------------------------------------------------------------------------------------------------- + (untracked) 411.37 1.0% 1.0% + -------------------------------------------------------------------------------------------------------------------- + python_cache_check 9 19.90 2.21 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_write_solutions 9 1.62 0.18 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_prepare 9 1.10 0.12 67.6% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_wsol_prepare_cache 9 1.05 0.12 96.3% 0.0% + (untracked) 0.04 3.7% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_header 9 0.42 0.05 25.8% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 0.11 6.6% 0.0% + +------------------------------------------------------------------------------------------------------------------------ + ESTIMATED WALL CLOCK TIME: 41.11s + +HIERARCHY VALIDATION: OK (all child phases fit within their parents) + +TIME BREAKDOWN (visual, % of wall clock) +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 99.8% ################################################# + python_client_execution 98.7% ################################################# + C++ Client Setup 30.6% ############### + listener_setup 11.6% ##### + hip_initialization 8.9% #### + library_loading 7.1% ### + code_object_loading 1.7% + data_init_setup 1.2% + lazy_loading_init 0.1% + reporter_setup 0.0% + solution_iterator_setup 0.0% + Data Preparation 25.9% ############ + pre_problem 25.3% ############ + cpu_reference_gemm 25.2% ############ + solve_cpu_slow 25.2% ############ + (untracked) 0.0% + cpu_data_init 0.1% + (untracked) 0.0% + gpu_input_reset 0.5% + gpu_input_preparation 0.0% + rotating_buffer_preparation 0.0% + Reference Computation 12.6% ###### + validate_warmups 12.6% ###### + validate_gpu_readback 8.6% #### + validate_element_comparison 3.8% # + (untracked) 0.2% + validate_gpu_sync 0.1% + Kernel Execution 11.5% ##### + post_solution 4.7% ## + post_solution_log 3.9% # + post_solution_reporting 0.6% + post_solution_perf_calc 0.1% + post_solution_validation 0.1% + post_solution_result_file 0.0% + post_solution_sol_advance 0.0% + post_solution_perf_reset 0.0% + pre_solution 3.0% # + warmup_runs 2.7% # + kernel_solving 0.7% + post_problem 0.3% + solution_selection 0.2% + benchmark_runs 0.0% + (untracked) 0.0% + finalize_report 0.0% + Timing Overhead 9.7% #### + flush_timing_buffer 9.5% #### + timing_overhead 0.2% + (untracked C++ overhead) 8.4% #### + (untracked) 1.0% + python_cache_check 0.0% + python_write_solutions 0.0% + python_wsol_prepare 0.0% + python_wsol_prepare_cache 0.0% + (untracked) 0.0% + python_wsol_header 0.0% + (untracked) 0.0% + +KEY INSIGHTS (% of wall clock) +------------------------------------------------------------------------------------------------------------------------ + Python phases (code gen, compilation): 0.00s (0.0%) + Python I/O overhead (cache, YAML): 0.02s (0.1%) + C++ client setup (library loading): 12.56s (30.6%) + Data preparation (CPU init, GPU prep): 10.63s (25.9%) + Reference computation (CPU GEMM): 5.21s (12.7%) + Kernel execution (warmup, benchmark): 4.78s (11.6%) + + +TOP 10 SLOWEST PROBLEMS (by CPU reference time) +------------------------------------------------------------------------------------------------------------------------ + M N K Batch TypeA TypeD CPU Ref (ms) GPU (ms) + ---------------------------------------------------------------------------------------------------------------------- + 128 128 63 2 ComplexFloat ComplexFloat 359.12 0.00 + 127 127 63 2 ComplexFloat ComplexFloat 318.96 0.00 + 129 129 63 2 ComplexFloat ComplexFloat 265.68 0.00 + 127 127 64 2 ComplexFloat ComplexFloat 265.15 0.00 + 127 127 64 2 ComplexFloat ComplexFloat 259.98 0.00 + 127 127 63 2 ComplexFloat ComplexFloat 254.31 0.00 + 129 129 63 2 ComplexFloat ComplexFloat 253.06 0.00 + 129 129 63 2 ComplexFloat ComplexFloat 251.09 0.00 + 129 129 64 2 ComplexFloat ComplexFloat 247.91 0.00 + 129 129 63 2 ComplexFloat ComplexFloat 243.50 0.00 + diff --git a/projects/hipblaslt/tensilelite/timing_runs/cgemm/times.txt b/projects/hipblaslt/tensilelite/timing_runs/cgemm/times.txt new file mode 100644 index 000000000000..f75e044e8876 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cgemm/times.txt @@ -0,0 +1,4 @@ +build_cold rc=0 109s +build_warm rc=0 104s +run rc=0 47s +DONE diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/CMS_FINDINGS.md b/projects/hipblaslt/tensilelite/timing_runs/cms/CMS_FINDINGS.md new file mode 100644 index 000000000000..8fad970fc1d9 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cms/CMS_FINDINGS.md @@ -0,0 +1,110 @@ +# CMS (UseCustomMainLoopSchedule) build-only analysis, gfx950 target + +Run 2026-10-16 on a gfx942 host (build-only, `--gpu-targets gfx950`, so no gfx950 GPU needed), +cold caches, `taskset -c 0-24` (25 cores), `TimingInstrumentation=True`, `/usr/bin/time -v`. +`gfx952` does not exist in this tree; treated as gfx950 (the gfx95x family). + +## Results + +| Config | Wall | User+Sys CPU | CPU used (of 25 cores) | Groups | Kernels/group | +|---|---|---|---|---|---| +| `gemm/gfx950/custom_mainloop_scheduling` | 112 s | 166 s | **148%** (about 1.5 cores) | 66 | 1 (53 groups) or 2 (13) | +| `gemm/gfx950/custom_mainloop_scheduling_tf32` | 45 s | 72 s | 160% | 22 | 1 | +| `ductile/gfx950/f8bf16_tn_cms` | 23 s, failed | | 392% | n/a | n/a | + +`f8bf16_tn_cms` fails by design in build-only: the ductile backend warns that it does not support +build-only and forces a benchmark run, which needs a gfx950 GPU +(`Expected results file does not exist`). Its logs are kept; it is not part of the analysis. + +## Is the pathology still there? Yes. + +The 25-core pool is idle almost the whole run: about 1.5 cores busy on average. + +1. **Groups run strictly one after another.** `BenchmarkProblems.main` loops serially over + `for problemType: for sizeGroup: _benchmarkProblemType(...)`. The big file has 9 problem types and + 66 `BenchmarkProblemSizeGroup`s. Each group does its own fork / codegen / write / compile / library + write before the next starts. +2. **Each group holds almost no parallel work.** Each group has `UseCustomMainLoopSchedule` as a + single value (`[1]` in 51 groups) or `[0, 1]` (14 groups), and every other ForkParameter is fixed + (58 unique fork sets across 66 groups, so the groups are different hand-picked tile configs rather + than duplicates). Result: `Actual Solutions: 1 / 1` in 53 groups and `2 / 2` in 13. + The log shows `Writing assembly kernels: Launching 25 threads for 1 tasks` (or 2 tasks) in all + 66 groups, i.e. 25 workers for 1-2 tasks. +3. **A group costs about 1 s no matter what** (median 0.98 s for 1-solution groups, 1.13 s for + 2-solution groups; `python_kernel_codegen` 0.76 s/group average). Total about 66 x 1.4 s plus the + compiles. +4. **The assembler/compiler step is also serial.** `build_src_co.compile` ran 3 times (9.3 + 4.7 + 4.9 s + = 18.8 s, 18% of the build) on unique helper kernels, one after the other, inside three of the + groups. +5. Per-stage totals (66 groups): kernel codegen 50.4 s (49%), compile 18.8 s (18%), solution + generation 13.8 s (13%), write/assemble 13.4 s (13%). + +Not the cause: process-pool creation. A bare `Pool(25)` plus one task costs about 22 ms here, which +is small compared with the roughly 1 s per group. + +## Experiment: does group-level parallelism recover the time? + +`split_experiment.py` splits the same YAML round-robin by size group into N shard configs and +builds them concurrently (25 cores divided evenly, cold caches, same flags). + +| Shards | Cores/shard | Wall | Speedup vs 112 s | +|---|---|---|---| +| 1 (original) | 25 | 112 s (pytest), 103.6 s inside the benchmark step | 1.0x | +| 5 | 5 | 36.8 s | 3.0x | +| 25 | 1 | 21.9 s | 5.1x | + +The speedup is real but is capped by the compile step: each shard compiles its own copy of the +helper kernels because the shared helper-kernel cache starts cold (29 `cache MISS` and 265 s of total +compile CPU at 25 shards, against 3 misses and 18.8 s in the single run). Each shard at 25 ways takes +12-19 s, which is mostly its own helper compile. + +## Where the speedup is + +1. **Run size groups concurrently** (in `BenchmarkProblems.main`, or by building the problem + types / groups with a process pool). This is the main win: 3x to 5x measured on a split of the + same input. It needs per-group outputs to be independent; the experiment splits them without any + issue, so for build-only they appear to be. +2. **Compile helper kernels once, up front**, instead of inside whichever group first misses the + cache. This removes the serial 18.8 s and the duplicated compiles seen in the split runs + (e.g. warm the helper cache, or key the cache so concurrent builders share it). +3. **Batch the YAML** only if a config author can: groups that share the same ProblemType and are + only different tile configs can sit in one `ForkParameters` sweep, which Tensile already + parallelizes. The CMS file cannot do that as written: each group is a specific hand-picked tile + config, not a cross product, so there is no way to express them as one sweep without generating + extra kernels. +4. Lower priority: Python codegen is 49% of CPU time at 0.76 s per kernel; a faster per-kernel codegen + helps every config, not just these. + +## Caveats + +- Build-only for gfx950 on a gfx942 host: no GPU run, so kernel correctness was not checked. +- Single run per measurement, on a shared machine. The split experiments use hand-written shards, + not a production change; they show the potential, not a verified implementation. +- The experiment ran under `Tensile.Tensile([...])` directly (not pytest), so exact overhead differs + slightly from the pytest runs. + +## Files + +- `custom_mainloop_scheduling_build.log`, `..._time.txt` (`/usr/bin/time -v`), `..._analysis.txt` +- the same three for `custom_mainloop_scheduling_tf32` and `f8bf16_tn_cms` +- `split_experiment.py`, `run_split.sh`, `split_5/`, `split_25/`, `split_*_result.txt` +## What the helper-kernel cache saves, single shard and 25 shards + +Same input and conditions (25 cores total), caches kept (warm) versus deleted (cold): + +| Run | Cold | Warm | Saved by cache | Helper-kernel compiles | +|---|---|---|---|---| +| 1 shard (the 112 s case) | 111 s (pytest) | 101 s | about 10 s (9%) | cold 3 compiles = 18.8 s; warm still 1 miss = 9.2 s | +| 25 shards (1 core each) | 21.9 s | 10.1 s | 11.8 s (54%) | cold 29 misses / 265 s compile CPU; warm 0 misses, 66 hits | + +- In the single shard, the warm run still had one helper-cache miss (9.2 s, `build_src_co.compile` + called once), so it saved 9.6 s of the 18.8 s compile, not all of it. The compile is only + 9-17% of the single-shard run, because Python codegen (50 s), solution generation (14 s) and + write/assemble (13 s) are untouched by the cache. +- With 25 shards the cache matters much more: every shard's wall time is dominated by the one + serial helper compile it would otherwise do. Warm, each shard takes 1.6-7.6 s (benchmark step) and + the whole job finishes in 10.1 s, including about 2-3 s of process start/import per shard. +- Net effect of both changes on the CMS file: 111 s cold single shard to 10.1 s warm 25 shards, + about 11x. Parallelism alone gives 5x (cold), the warm cache adds another 2.2x on top at 25 shards. +- Files: `custom_mainloop_scheduling_warm_*` (single warm run), `split_25_warm/`, + `split_25_warm_result.txt`, `run_split_warm.sh`. diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_build_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_build_analysis.txt new file mode 100644 index 000000000000..1c1a6578e972 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_build_analysis.txt @@ -0,0 +1,135 @@ +======================================================================================================================== +TIMING ANALYSIS SUMMARY +======================================================================================================================== + +TIMING BY PHASE +------------------------------------------------------------------------------------------------------------------------ + Category Count Total (ms) Mean (ms) % of Parent % of Wall +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 1 103,629.97 103,629.97 100.0% + -------------------------------------------------------------------------------------------------------------------- + python_kernel_compilation 66 89,512.32 1,356.25 86.4% 86.4% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_codegen 66 50,352.76 762.92 56.3% 48.6% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_src_co 66 19,058.00 288.76 21.3% 18.4% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_kernel_build_src_co.compile 3 18,805.92 6,268.64 98.7% 18.1% + python_kernel_build_src_co.cache_check 66 212.03 3.21 1.1% 0.2% + python_kernel_build_src_co.unbundle 3 24.04 8.01 0.1% 0.0% + (untracked) 10.00 0.1% 0.0% + python_kernel_build_src_co.setup 66 3.17 0.05 0.0% 0.0% + python_kernel_build_src_co.cache_populate 3 2.62 0.88 0.0% 0.0% + python_kernel_build_src_co.move 3 0.21 0.07 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_assemble 66 13,373.40 202.63 14.9% 12.9% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_helpers 66 2,185.94 33.12 2.4% 2.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_validate 66 2,064.63 31.28 2.3% 2.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_co 66 1,695.29 25.69 1.9% 1.6% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_setup 66 366.11 5.55 0.4% 0.4% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_postprocess 66 205.57 3.11 0.2% 0.2% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_benchpost_lib_construction 66 105.83 1.60 51.5% 0.1% + python_benchpost_library_write 66 38.19 0.58 18.6% 0.0% + python_benchpost_naming 66 29.48 0.45 14.3% 0.0% + python_benchpost_client_config 66 28.77 0.44 14.0% 0.0% + (untracked) 3.30 1.6% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 153.55 0.2% 0.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_setup 66 57.09 0.87 0.1% 0.1% + -------------------------------------------------------------------------------------------------------------------- + python_solution_generation 66 13,842.92 209.74 13.4% 13.4% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_forked_solutions 66 13,841.56 209.72 100.0% 13.4% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 0.98 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_fork_permutations 66 0.28 0.00 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_custom_kernels 66 0.11 0.00 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + (untracked) 145.59 0.1% 0.1% + -------------------------------------------------------------------------------------------------------------------- + python_write_cache 66 81.32 1.23 0.1% 0.1% + -------------------------------------------------------------------------------------------------------------------- + python_write_solutions 66 32.50 0.49 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_dump 66 24.75 0.38 76.2% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 5.50 16.9% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_header 66 1.57 0.02 4.8% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_prepare 66 0.68 0.01 2.1% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_wsol_prepare_nocache 66 0.55 0.01 80.4% 0.0% + (untracked) 0.13 19.6% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_solution_indexing 66 15.25 0.23 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_cache_check 66 0.07 0.00 0.0% 0.0% + +------------------------------------------------------------------------------------------------------------------------ + ESTIMATED WALL CLOCK TIME: 103.63s + +HIERARCHY VALIDATION: OK (all child phases fit within their parents) + +TIME BREAKDOWN (visual, % of wall clock) +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 100.0% ################################################## + python_kernel_compilation 86.4% ########################################### + python_kernel_codegen 48.6% ######################## + python_kernel_build_src_co 18.4% ######### + python_kernel_build_src_co.compile 18.1% ######### + python_kernel_build_src_co.cache_check 0.2% + python_kernel_build_src_co.unbundle 0.0% + (untracked) 0.0% + python_kernel_build_src_co.setup 0.0% + python_kernel_build_src_co.cache_populate 0.0% + python_kernel_build_src_co.move 0.0% + python_kernel_write_assemble 12.9% ###### + python_kernel_write_helpers 2.1% # + python_kernel_validate 2.0% + python_kernel_build_co 1.6% + python_kernel_bench_setup 0.4% + python_kernel_bench_postprocess 0.2% + python_benchpost_lib_construction 0.1% + python_benchpost_library_write 0.0% + python_benchpost_naming 0.0% + python_benchpost_client_config 0.0% + (untracked) 0.0% + (untracked) 0.1% + python_kernel_setup 0.1% + python_solution_generation 13.4% ###### + python_solgen_forked_solutions 13.4% ###### + (untracked) 0.0% + python_solgen_fork_permutations 0.0% + python_solgen_custom_kernels 0.0% + (untracked) 0.1% + python_write_cache 0.1% + python_write_solutions 0.0% + python_wsol_dump 0.0% + (untracked) 0.0% + python_wsol_header 0.0% + python_wsol_prepare 0.0% + python_wsol_prepare_nocache 0.0% + (untracked) 0.0% + python_solution_indexing 0.0% + python_cache_check 0.0% + +KEY INSIGHTS (% of wall clock) +------------------------------------------------------------------------------------------------------------------------ + Python phases (code gen, compilation): 103.36s (99.7%) + Python I/O overhead (cache, YAML): 0.11s (0.1%) + C++ client setup (library loading): 0.00s (0.0%) + Data preparation (CPU init, GPU prep): 0.00s (0.0%) + Reference computation (CPU GEMM): 0.00s (0.0%) + Kernel execution (warmup, benchmark): 0.00s (0.0%) + + diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_tf32_build_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_tf32_build_analysis.txt new file mode 100644 index 000000000000..fea1d7359c3c --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_tf32_build_analysis.txt @@ -0,0 +1,135 @@ +======================================================================================================================== +TIMING ANALYSIS SUMMARY +======================================================================================================================== + +TIMING BY PHASE +------------------------------------------------------------------------------------------------------------------------ + Category Count Total (ms) Mean (ms) % of Parent % of Wall +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 1 39,264.72 39,264.72 100.0% + -------------------------------------------------------------------------------------------------------------------- + python_kernel_compilation 22 26,389.24 1,199.51 67.2% 67.2% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_src_co 22 13,821.39 628.25 52.4% 35.2% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_kernel_build_src_co.compile 2 13,727.73 6,863.86 99.3% 35.0% + python_kernel_build_src_co.cache_check 22 65.52 2.98 0.5% 0.2% + python_kernel_build_src_co.unbundle 2 20.93 10.47 0.2% 0.1% + (untracked) 3.72 0.0% 0.0% + python_kernel_build_src_co.cache_populate 2 1.96 0.98 0.0% 0.0% + python_kernel_build_src_co.setup 22 1.37 0.06 0.0% 0.0% + python_kernel_build_src_co.move 2 0.17 0.08 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_codegen 22 6,607.73 300.35 25.0% 16.8% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_assemble 22 3,082.34 140.11 11.7% 7.9% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_validate 22 1,614.68 73.39 6.1% 4.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_co 22 532.14 24.19 2.0% 1.4% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_helpers 22 472.25 21.47 1.8% 1.2% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_setup 22 89.95 4.09 0.3% 0.2% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_postprocess 22 75.63 3.44 0.3% 0.2% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_benchpost_lib_construction 22 37.10 1.69 49.1% 0.1% + python_benchpost_client_config 22 15.64 0.71 20.7% 0.0% + python_benchpost_library_write 22 12.99 0.59 17.2% 0.0% + python_benchpost_naming 22 8.61 0.39 11.4% 0.0% + (untracked) 1.29 1.7% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 66.46 0.3% 0.2% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_setup 22 26.67 1.21 0.1% 0.1% + -------------------------------------------------------------------------------------------------------------------- + python_solution_generation 22 12,758.49 579.93 32.5% 32.5% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_forked_solutions 22 12,757.84 579.90 100.0% 32.5% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 0.50 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_fork_permutations 22 0.10 0.00 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_custom_kernels 22 0.05 0.00 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + (untracked) 66.79 0.2% 0.2% + -------------------------------------------------------------------------------------------------------------------- + python_write_cache 22 32.89 1.49 0.1% 0.1% + -------------------------------------------------------------------------------------------------------------------- + python_write_solutions 22 12.05 0.55 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_dump 22 8.88 0.40 73.7% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 2.24 18.6% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_header 22 0.69 0.03 5.7% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_prepare 22 0.24 0.01 2.0% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_wsol_prepare_nocache 22 0.18 0.01 75.8% 0.0% + (untracked) 0.06 24.2% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_solution_indexing 22 5.23 0.24 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_cache_check 22 0.03 0.00 0.0% 0.0% + +------------------------------------------------------------------------------------------------------------------------ + ESTIMATED WALL CLOCK TIME: 39.26s + +HIERARCHY VALIDATION: OK (all child phases fit within their parents) + +TIME BREAKDOWN (visual, % of wall clock) +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 100.0% ################################################## + python_kernel_compilation 67.2% ################################# + python_kernel_build_src_co 35.2% ################# + python_kernel_build_src_co.compile 35.0% ################# + python_kernel_build_src_co.cache_check 0.2% + python_kernel_build_src_co.unbundle 0.1% + (untracked) 0.0% + python_kernel_build_src_co.cache_populate 0.0% + python_kernel_build_src_co.setup 0.0% + python_kernel_build_src_co.move 0.0% + python_kernel_codegen 16.8% ######## + python_kernel_write_assemble 7.9% ### + python_kernel_validate 4.1% ## + python_kernel_build_co 1.4% + python_kernel_write_helpers 1.2% + python_kernel_bench_setup 0.2% + python_kernel_bench_postprocess 0.2% + python_benchpost_lib_construction 0.1% + python_benchpost_client_config 0.0% + python_benchpost_library_write 0.0% + python_benchpost_naming 0.0% + (untracked) 0.0% + (untracked) 0.2% + python_kernel_setup 0.1% + python_solution_generation 32.5% ################ + python_solgen_forked_solutions 32.5% ################ + (untracked) 0.0% + python_solgen_fork_permutations 0.0% + python_solgen_custom_kernels 0.0% + (untracked) 0.2% + python_write_cache 0.1% + python_write_solutions 0.0% + python_wsol_dump 0.0% + (untracked) 0.0% + python_wsol_header 0.0% + python_wsol_prepare 0.0% + python_wsol_prepare_nocache 0.0% + (untracked) 0.0% + python_solution_indexing 0.0% + python_cache_check 0.0% + +KEY INSIGHTS (% of wall clock) +------------------------------------------------------------------------------------------------------------------------ + Python phases (code gen, compilation): 39.15s (99.7%) + Python I/O overhead (cache, YAML): 0.04s (0.1%) + C++ client setup (library loading): 0.00s (0.0%) + Data preparation (CPU init, GPU prep): 0.00s (0.0%) + Reference computation (CPU GEMM): 0.00s (0.0%) + Kernel execution (warmup, benchmark): 0.00s (0.0%) + + diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_tf32_time.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_tf32_time.txt new file mode 100644 index 000000000000..a32dc3884f7d --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_tf32_time.txt @@ -0,0 +1,23 @@ + Command being timed: "taskset -c 0-24 pytest tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/custom_mainloop_scheduling_tf32.yaml] --build-only --gpu-targets gfx950 --artifact-dir /tmp/tl_cms_art_custom_mainloop_scheduling_tf32 -s -v -n 0 --durations=0 --timeout=14400 --global-parameters=LibraryFormat='msgpack' --tensile-options=--cxx-compiler,/usr/bin/amdclang++,--global-parameters,TimingInstrumentation=True -p no:warnings" + User time (seconds): 64.01 + System time (seconds): 7.99 + Percent of CPU this job got: 160% + Elapsed (wall clock) time (h:mm:ss or m:ss): 0:44.86 + Average shared text size (kbytes): 0 + Average unshared data size (kbytes): 0 + Average stack size (kbytes): 0 + Average total size (kbytes): 0 + Maximum resident set size (kbytes): 278952 + Average resident set size (kbytes): 0 + Major (requiring I/O) page faults: 29 + Minor (reclaiming a frame) page faults: 2582591 + Voluntary context switches: 8293 + Involuntary context switches: 4941 + Swaps: 0 + File system inputs: 0 + File system outputs: 165680 + Socket messages sent: 0 + Socket messages received: 0 + Signals delivered: 0 + Page size (bytes): 4096 + Exit status: 0 diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_time.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_time.txt new file mode 100644 index 000000000000..569cda5b0876 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_time.txt @@ -0,0 +1,23 @@ + Command being timed: "taskset -c 0-24 pytest tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/custom_mainloop_scheduling.yaml] --build-only --gpu-targets gfx950 --artifact-dir /tmp/tl_cms_art_custom_mainloop_scheduling -s -v -n 0 --durations=0 --timeout=14400 --global-parameters=LibraryFormat='msgpack' --tensile-options=--cxx-compiler,/usr/bin/amdclang++,--global-parameters,TimingInstrumentation=True -p no:warnings" + User time (seconds): 149.58 + System time (seconds): 16.64 + Percent of CPU this job got: 148% + Elapsed (wall clock) time (h:mm:ss or m:ss): 1:52.11 + Average shared text size (kbytes): 0 + Average unshared data size (kbytes): 0 + Average stack size (kbytes): 0 + Average total size (kbytes): 0 + Maximum resident set size (kbytes): 622760 + Average resident set size (kbytes): 0 + Major (requiring I/O) page faults: 115 + Minor (reclaiming a frame) page faults: 5556720 + Voluntary context switches: 26154 + Involuntary context switches: 7999 + Swaps: 0 + File system inputs: 8 + File system outputs: 811136 + Socket messages sent: 0 + Socket messages received: 0 + Signals delivered: 0 + Page size (bytes): 4096 + Exit status: 0 diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_warm_build_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_warm_build_analysis.txt new file mode 100644 index 000000000000..d4062f531a40 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_warm_build_analysis.txt @@ -0,0 +1,135 @@ +======================================================================================================================== +TIMING ANALYSIS SUMMARY +======================================================================================================================== + +TIMING BY PHASE +------------------------------------------------------------------------------------------------------------------------ + Category Count Total (ms) Mean (ms) % of Parent % of Wall +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 1 94,008.80 94,008.80 100.0% + -------------------------------------------------------------------------------------------------------------------- + python_kernel_compilation 66 79,940.15 1,211.21 85.0% 85.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_codegen 66 50,332.85 762.62 63.0% 53.5% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_assemble 66 13,371.46 202.60 16.7% 14.2% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_src_co 66 9,473.59 143.54 11.9% 10.1% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_kernel_build_src_co.compile 1 9,195.85 9,195.85 97.1% 9.8% + python_kernel_build_src_co.cache_check 66 249.74 3.78 2.6% 0.3% + (untracked) 13.50 0.1% 0.0% + python_kernel_build_src_co.unbundle 1 9.59 9.59 0.1% 0.0% + python_kernel_build_src_co.setup 66 3.68 0.06 0.0% 0.0% + python_kernel_build_src_co.cache_populate 1 1.16 1.16 0.0% 0.0% + python_kernel_build_src_co.move 1 0.08 0.08 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_helpers 66 2,264.19 34.31 2.8% 2.4% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_validate 66 2,044.68 30.98 2.6% 2.2% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_co 66 1,630.71 24.71 2.0% 1.7% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_setup 66 371.90 5.63 0.5% 0.4% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_postprocess 66 226.08 3.43 0.3% 0.2% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_benchpost_lib_construction 66 118.27 1.79 52.3% 0.1% + python_benchpost_library_write 66 43.41 0.66 19.2% 0.0% + python_benchpost_client_config 66 32.16 0.49 14.2% 0.0% + python_benchpost_naming 66 28.49 0.43 12.6% 0.0% + (untracked) 3.75 1.7% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 165.38 0.2% 0.2% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_setup 66 59.30 0.90 0.1% 0.1% + -------------------------------------------------------------------------------------------------------------------- + python_solution_generation 66 13,756.23 208.43 14.6% 14.6% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_forked_solutions 66 13,754.77 208.41 100.0% 14.6% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 1.01 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_fork_permutations 66 0.34 0.01 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_custom_kernels 66 0.12 0.00 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + (untracked) 164.15 0.2% 0.2% + -------------------------------------------------------------------------------------------------------------------- + python_write_cache 66 93.01 1.41 0.1% 0.1% + -------------------------------------------------------------------------------------------------------------------- + python_write_solutions 66 37.83 0.57 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_dump 66 28.96 0.44 76.6% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 6.33 16.7% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_header 66 1.75 0.03 4.6% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_prepare 66 0.80 0.01 2.1% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_wsol_prepare_nocache 66 0.63 0.01 79.2% 0.0% + (untracked) 0.17 20.8% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_solution_indexing 66 17.36 0.26 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_cache_check 66 0.07 0.00 0.0% 0.0% + +------------------------------------------------------------------------------------------------------------------------ + ESTIMATED WALL CLOCK TIME: 94.01s + +HIERARCHY VALIDATION: OK (all child phases fit within their parents) + +TIME BREAKDOWN (visual, % of wall clock) +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 100.0% ################################################## + python_kernel_compilation 85.0% ########################################## + python_kernel_codegen 53.5% ########################## + python_kernel_write_assemble 14.2% ####### + python_kernel_build_src_co 10.1% ##### + python_kernel_build_src_co.compile 9.8% #### + python_kernel_build_src_co.cache_check 0.3% + (untracked) 0.0% + python_kernel_build_src_co.unbundle 0.0% + python_kernel_build_src_co.setup 0.0% + python_kernel_build_src_co.cache_populate 0.0% + python_kernel_build_src_co.move 0.0% + python_kernel_write_helpers 2.4% # + python_kernel_validate 2.2% # + python_kernel_build_co 1.7% + python_kernel_bench_setup 0.4% + python_kernel_bench_postprocess 0.2% + python_benchpost_lib_construction 0.1% + python_benchpost_library_write 0.0% + python_benchpost_client_config 0.0% + python_benchpost_naming 0.0% + (untracked) 0.0% + (untracked) 0.2% + python_kernel_setup 0.1% + python_solution_generation 14.6% ####### + python_solgen_forked_solutions 14.6% ####### + (untracked) 0.0% + python_solgen_fork_permutations 0.0% + python_solgen_custom_kernels 0.0% + (untracked) 0.2% + python_write_cache 0.1% + python_write_solutions 0.0% + python_wsol_dump 0.0% + (untracked) 0.0% + python_wsol_header 0.0% + python_wsol_prepare 0.0% + python_wsol_prepare_nocache 0.0% + (untracked) 0.0% + python_solution_indexing 0.0% + python_cache_check 0.0% + +KEY INSIGHTS (% of wall clock) +------------------------------------------------------------------------------------------------------------------------ + Python phases (code gen, compilation): 93.70s (99.7%) + Python I/O overhead (cache, YAML): 0.13s (0.1%) + C++ client setup (library loading): 0.00s (0.0%) + Data preparation (CPU init, GPU prep): 0.00s (0.0%) + Reference computation (CPU GEMM): 0.00s (0.0%) + Kernel execution (warmup, benchmark): 0.00s (0.0%) + + diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_warm_time.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_warm_time.txt new file mode 100644 index 000000000000..438e02afcae3 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_warm_time.txt @@ -0,0 +1,23 @@ + Command being timed: "taskset -c 0-24 pytest tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/custom_mainloop_scheduling.yaml] --build-only --gpu-targets gfx950 --artifact-dir /tmp/tl_cms_art_warm -s -v -n 0 --durations=0 --timeout=14400 --global-parameters=LibraryFormat='msgpack' --tensile-options=--cxx-compiler,/usr/bin/amdclang++,--global-parameters,TimingInstrumentation=True -p no:warnings" + User time (seconds): 137.84 + System time (seconds): 16.04 + Percent of CPU this job got: 150% + Elapsed (wall clock) time (h:mm:ss or m:ss): 1:42.57 + Average shared text size (kbytes): 0 + Average unshared data size (kbytes): 0 + Average stack size (kbytes): 0 + Average total size (kbytes): 0 + Maximum resident set size (kbytes): 624096 + Average resident set size (kbytes): 0 + Major (requiring I/O) page faults: 106 + Minor (reclaiming a frame) page faults: 5451311 + Voluntary context switches: 25967 + Involuntary context switches: 13936 + Swaps: 0 + File system inputs: 0 + File system outputs: 805432 + Socket messages sent: 0 + Socket messages received: 0 + Signals delivered: 0 + Page size (bytes): 4096 + Exit status: 0 diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/f8bf16_tn_cms_build_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/f8bf16_tn_cms_build_analysis.txt new file mode 100644 index 000000000000..9689a634e5aa --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cms/f8bf16_tn_cms_build_analysis.txt @@ -0,0 +1,8 @@ +======================================================================================================================== +TIMING ANALYSIS SUMMARY +======================================================================================================================== + +No timing data found! + +Make sure you ran with --global-parameters "TimingInstrumentation=True" +and captured stderr output (2>&1). diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/f8bf16_tn_cms_time.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/f8bf16_tn_cms_time.txt new file mode 100644 index 000000000000..3f3209bb43f2 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cms/f8bf16_tn_cms_time.txt @@ -0,0 +1,24 @@ +Command exited with non-zero status 1 + Command being timed: "taskset -c 0-24 pytest tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx950/f8bf16_tn_cms.yaml] --build-only --gpu-targets gfx950 --artifact-dir /tmp/tl_cms_art_f8bf16_tn_cms -s -v -n 0 --durations=0 --timeout=14400 --global-parameters=LibraryFormat='msgpack' --tensile-options=--cxx-compiler,/usr/bin/amdclang++,--global-parameters,TimingInstrumentation=True -p no:warnings" + User time (seconds): 80.67 + System time (seconds): 10.13 + Percent of CPU this job got: 392% + Elapsed (wall clock) time (h:mm:ss or m:ss): 0:23.12 + Average shared text size (kbytes): 0 + Average unshared data size (kbytes): 0 + Average stack size (kbytes): 0 + Average total size (kbytes): 0 + Maximum resident set size (kbytes): 483220 + Average resident set size (kbytes): 0 + Major (requiring I/O) page faults: 13 + Minor (reclaiming a frame) page faults: 2817252 + Voluntary context switches: 4991 + Involuntary context switches: 7192 + Swaps: 0 + File system inputs: 0 + File system outputs: 84912 + Socket messages sent: 0 + Socket messages received: 0 + Signals delivered: 0 + Page size (bytes): 4096 + Exit status: 1 diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/run_split.sh b/projects/hipblaslt/tensilelite/timing_runs/cms/run_split.sh new file mode 100644 index 000000000000..9728ab608363 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cms/run_split.sh @@ -0,0 +1,8 @@ +# Usage: bash run_split.sh N -- cold caches, then the N-way concurrent build-only experiment. +N=${1:-5} +TL=/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite +cd $TL || exit 1 +export OMP_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=0 PYTHONUNBUFFERED=1 +rm -rf /home/alvasile/.tensile /home/alvasile/.tensilelite /tmp/tl_split_out_${N}_* +python timing_runs/cms/split_experiment.py $N > timing_runs/cms/split_${N}_result.txt 2>&1 +echo finished >> timing_runs/cms/split_${N}_result.txt diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/run_split_warm.sh b/projects/hipblaslt/tensilelite/timing_runs/cms/run_split_warm.sh new file mode 100644 index 000000000000..353376a4a310 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cms/run_split_warm.sh @@ -0,0 +1,8 @@ +# Usage: bash run_split_warm.sh N -- N-way concurrent build-only with the caches KEPT (warm). +# Run after a previous cold run so ~/.tensile and ~/.tensilelite are populated. +N=${1:-25} +TL=/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite +cd $TL || exit 1 +export OMP_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=0 PYTHONUNBUFFERED=1 +python timing_runs/cms/split_experiment.py $N _warm > timing_runs/cms/split_${N}_warm_result.txt 2>&1 +echo finished >> timing_runs/cms/split_${N}_warm_result.txt diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/split_25_result.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/split_25_result.txt new file mode 100644 index 000000000000..10297c996930 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cms/split_25_result.txt @@ -0,0 +1,3 @@ +total groups 66 +rcs [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] wall 21.9 s +finished diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/split_25_warm_result.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/split_25_warm_result.txt new file mode 100644 index 000000000000..5fa0fd22b930 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cms/split_25_warm_result.txt @@ -0,0 +1,3 @@ +total groups 66 +rcs [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] wall 10.1 s +finished diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/split_5_result.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/split_5_result.txt new file mode 100644 index 000000000000..7bd445f5c55f --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cms/split_5_result.txt @@ -0,0 +1,3 @@ +total groups 66 +rcs [0, 0, 0, 0, 0] wall 36.8 s +finished diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/split_experiment.py b/projects/hipblaslt/tensilelite/timing_runs/cms/split_experiment.py new file mode 100644 index 000000000000..0332223b284d --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cms/split_experiment.py @@ -0,0 +1,42 @@ +"""Split custom_mainloop_scheduling.yaml into N shards (round-robin by size group) and +build-only them concurrently, to test whether group-level parallelism recovers the idle cores. +Usage: python split_experiment.py N (run from the tensilelite dir, caches cleared by caller) +""" +import copy, os, subprocess, sys, time +import yaml + +N = int(sys.argv[1]) +TAG = sys.argv[2] if len(sys.argv) > 2 else "" +TL = "/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite" +SRC = f"{TL}/tensilelite/Tests/common/gemm/gfx950/custom_mainloop_scheduling.yaml" +OUT = f"{TL}/timing_runs/cms/split_{N}{TAG}" +os.makedirs(OUT, exist_ok=True) + +doc = yaml.safe_load(open(SRC)) +flat = [(pt[0], g) for pt in doc["BenchmarkProblems"] for g in pt[1:]] +print("total groups", len(flat)) +shards = [[] for _ in range(N)] +for i, item in enumerate(flat): + shards[i % N].append(item) + +procs = [] +cores_per = 25 // N +t0 = time.time() +for k, items in enumerate(shards): + d = copy.deepcopy(doc) + d["BenchmarkProblems"] = [[pt, g] for pt, g in items] + d.pop("TestParameters", None) + cfg = f"{OUT}/shard{k}.yaml" + yaml.safe_dump(d, open(cfg, "w"), sort_keys=False) + cores = f"{k*cores_per}-{(k+1)*cores_per-1}" + cmd = ["taskset", "-c", cores, "python", "-c", + "import sys; from tensilelite import Tensile; Tensile.Tensile(sys.argv[1:])", + cfg, f"/tmp/tl_split_out_{N}{TAG}_{k}", "--build-only", "--gpu-targets", "gfx950", + "--cxx-compiler", "/usr/bin/amdclang++", + "--global-parameters", "TimingInstrumentation=True", "LibraryFormat='msgpack'"] + log = open(f"{OUT}/shard{k}.log", "w") + procs.append((subprocess.Popen(cmd, stdout=log, stderr=subprocess.STDOUT), time.time())) +rcs = [] +for p, st in procs: + rcs.append(p.wait()) +print("rcs", rcs, "wall %.1f s" % (time.time() - t0)) diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/times.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/times.txt new file mode 100644 index 000000000000..04a4d313a9ed --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cms/times.txt @@ -0,0 +1,4 @@ +custom_mainloop_scheduling rc=0 112s +custom_mainloop_scheduling_tf32 rc=0 45s +f8bf16_tn_cms rc=1 23s +DONE diff --git a/projects/hipblaslt/tensilelite/timing_runs/cold_warm_done.txt b/projects/hipblaslt/tensilelite/timing_runs/cold_warm_done.txt new file mode 100644 index 000000000000..2fdae8d75aac --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cold_warm_done.txt @@ -0,0 +1 @@ +ALLDONE diff --git a/projects/hipblaslt/tensilelite/timing_runs/cpu_data_init_investigation.md b/projects/hipblaslt/tensilelite/timing_runs/cpu_data_init_investigation.md new file mode 100644 index 000000000000..afd0bc712723 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cpu_data_init_investigation.md @@ -0,0 +1,191 @@ +# `cpu_data_init` investigation (gfx950 per-config rerun 2) + +`full_runs_gfx950/per_config_rerun_2/AGGREGATE_analysis.txt` reports `cpu_data_init` at +106,829.68 ms over 28,596 calls (mean 3.74 ms, 4.2% of wall). This note explains where that time +goes. Every number below is a timing measurement; reading the code was only used to decide what to +measure. + +## Method + +- **Raw logs.** Re-parsed the 188 per-config logs (`TIMING_CONTEXT` + `TIMING:cpu_data_init`). + They reconcile exactly with the aggregate: 28,596 calls, 106,830 ms. +- **Instrumented client.** Rebuilt `tensilelite-client` from the exact sources the measured binary + was built from (commit `4c6426cea6`, client and include trees), using the original + `compile_commands.json` / `link.txt`, with extra `ScopedTimer`s in `prepareCPUInputs`, + `initializeCPUInputs` (per tensor), `copyInputs` (per tensor) and around each + `generateMXInput` call. It reproduces CI: `sk_bgemm_div` 8.4–15.0 s vs 15.1 s in CI, and the + MX regression problems 0.4–2.4 s vs 0.3–2.2 s. +- **Microbenchmarks.** Built with the client's flags (`-O3 -DNDEBUG -mf16c -x hip -fopenmp=libomp`, + no `-march`), using the real `TensorDescriptor`, `CoordNumbered`, `BFloat16` and + `libhipblaslt-mxdatagen.a`. Their assembly was checked against the shipped binary with + `objdump`. +- **Environment.** gfx950 host, 2× EPYC 9575F; runs pinned to cores 0–24 with + `OMP_NUM_THREADS=25`, as in CI. `perf` was unavailable (`perf_event_paranoid=4`). The host was + shared, with other users' processes using 30–90% of several cores in 0–24. + +## What the timer covers + +`ScopedTimer("cpu_data_init")` in `ReferenceValidator::preProblem` wraps +`DataInitialization::prepareCPUInputs(problem)`: + +```text +cpu_data_init +└─ prepareCPUInputs + ├─ initializeCPUInputs (only if m_problemDependentData) + │ ├─ initializeMXData → generateMXInput (MX configs) + │ └─ initArray per changed tensor (initArrayTrig, Random C, Zero D, ...) + ├─ copyInputs (hipMemcpy H2H of every tensor, maxElements each) + ├─ initializeConstantInputs (~0 ms) + └─ ConvertToProblemInputs (~0 ms) +``` + +The instrumented sub-timers add up to the parent: + +| Run | initializeCPUInputs | copyInputs | cpu_data_init | +| --- | --- | --- | --- | +| 4096×4096×8192 TrigSin, first call | 246.5 ms | 65.3 ms | 311.8 ms | +| `sk_bgemm_div` | 8524.8 ms | 1213.2 ms | 9738.0 ms | +| MX regression, first problem | 570.0 ms | 574.0 ms | 1144.0 ms | + +Not part of `cpu_data_init`: `data_init_setup` (one-time Random fills for configs whose data is +not problem-dependent), `cpu_reference_gemm`, and `pre_solution_mx_side_regenerate` (44.1 s, which +calls the same `generateMXInput` per solution but was not measured here). + +## Where the 106.8 s goes + +The distribution is heavy-tailed: the top 10 calls are 28.8% of the time, the top 200 are 59.6%, +and the top 1000 are 81.3%. + +| Bucket (by A/B/C init mode and type) | Time | Share | Calls | +| --- | --- | --- | --- | +| MX configs (8/6/4-bit types, mxDataGenerator) | 44.6 s | 42% | 8,961 | +| Trig / Serial / Identity init modes | 43.1 s | 40% | 6,674 | +| Sparse configs | 12.5 s | 12% | 2,688 | +| Everything else (copy only) | 6.6 s | 6% | 10,273 | + +Across all buckets, the first call in each client process (820 calls) is 54.0 s (50.5%). The +remaining 27,776 calls average 1.9 ms. + +Largest single contributors: `streamk/sk_bgemm_div` 15.1 s (1 call), +`gfx950/subtile_mxfp8` 10.3 s (39 calls), `gfx950/subtile_mxfp4` 8.8 s (685 calls), and the two +`subtile_mxfp8_correctness_regression_*` configs 5.3 s and 5.2 s (5 calls each). + +## Findings + +### 1. MX data generation: about half the work is discarded + +For MX configs, `initializeCPUInputs → initializeMXData` calls `generateMXInput` +(`clients/common/src/mxDataGen.cpp`) once per MX side. At problem level there is no current +solution, so no swizzled scale is generated (`swizzle=0` in every call observed). + +Example: the 516M-element FP8 E4M3 A side (7168×71936) of the MX regression config. In-client, +`generateMXInput` took 2094 ms. A harness calling the same library reproduces 2130–2251 ms and +splits it: + +| Step | Time (ms) | Share | +| --- | --- | --- | +| `generate()` (resize + RNG + FP8 convert) | 903–935 | ~42% | +| `getDataBytes()` copy | 128–132 | ~6% | +| `memcpy` into the client buffer | 18–19 | ~1% | +| **`getReferenceFloat()`, result discarded** | 938–1006 | **~45%** | +| **free of that 2 GB vector** | 102–116 | **~5%** | + +- `generateMXInput` returns a full `std::vector` reference (4 bytes per element). The + TensileLite client ignores the return value, so that work is wasted. +- `generate()` includes a single-threaded `m_dataBytes.resize(n)` zero-fill: about 90 ms for + 516 MB. +- On 1 thread the call costs about 40 ns per element. It reaches only 7–11× speedup on 25 threads + (20,935 ms → 2,094 ms for 516M elements). + +### 2. Trig init: an expensive per-element loop + +`initArrayTrig(array, tensor)` runs for Trig init modes, which are problem-dependent and so +re-run whenever a tensor's descriptor changes. In the shipped binary's assembly, every element: + +- allocates and frees a `std::vector` (`operator new`, `memset`, `operator delete`) +- executes 2–4 64-bit `div` instructions (`CoordNumbered`) and calls `TensorDescriptor::index` + out of line +- converts the index to `int`, then calls double-precision libm `sin`/`cos` (`sin@plt`), with no + vectorization + +Microbenchmark, 1 thread, 32M BF16 elements: + +| Variant | ns/element | +| --- | --- | +| The client's loop | 22.1 | +| Without the vector allocation | 14.9 | +| Without the `sin` call | 12.1 | +| Just `sin` + store | 7.0 | +| `sinf` + store | 3.7 | + +- **glibc `sin` slow path.** The argument is the element index. Single-thread cost is 6.9 ns + below about 1.0e8 and 23.6–25 ns at 1.06e8 and above, glibc's large-argument reduction path. + Large tensors pay the slow rate for most elements. +- **All changed tensors are re-initialized.** When `m_problemDependentData` is set, + `initializeCPUInputs` re-runs `initArray` for every tensor whose descriptor changed, not only + the Trig ones. +- **Worst case: `sk_bgemm_div`.** One 8192×57344×28672 BF16 problem with TrigSin A, TrigCos B and + TrigSin C. It initializes about 2.35G trig elements and copies 5.6 GB, to validate + `NumElementsToValidate: 128`; the CPU reference takes 13 ms. Instrumented breakdown of one run: + A 843 ms, B 4,105 ms, C 3,551 ms, D 23 ms, `copyInputs` 1,213 ms. + +Steady-state behaviour on a 4096×4096×8192 BF16→F32 problem (TrigSin A/B, 25 threads, 3 runs): + +| Call | Trig init A + B | Init C + D | `copyInputs` | Total | +| --- | --- | --- | --- | --- | +| 1st (fresh process) | 160–263 ms | 15–22 ms | 65–90 ms | 273–352 ms | +| Same problem again | 0 (skipped) | 0 | 9.5–9.7 ms | 9.5–9.8 ms | +| K changes (4096) | 54–67 ms | 0 | 9.5–9.9 ms | 64–77 ms | +| K changes back (8192) | 106–134 ms | 0 | 9.6–9.8 ms | 116–143 ms | +| 256×256×256 | 0.5–0.9 ms | — | 9.6–10.0 ms | 10.2–10.6 ms | + +### 3. `copyInputs` copies the largest size seen, every call + +- `copyInputBuffers` copies `p.maxElements` (the run's largest allocation), not the current + problem's size. In the table above, the 256×256×256 problem pays the same ~9.6 ms as the large + one; the instrumented log shows A copying 33,554,432 elements for a problem whose A has 65,536. +- The cheap branch in `prepareCPUInputs` that would only reset the output never runs: + `m_cpuInit` is initialized to `false` and only ever assigned `false`. +- A process's first copy is 4–9× slower (65–90 ms vs 9.6 ms) because the destination buffers are + touched for the first time. + +### 4. Fixed cost on each process's first call + +- In the sparse `spmm_i8` run, the first call spent 53.3 ms copying a 1,024-element buffer and + 18.0 ms initializing 1,024 elements. Every later call took 0.6–3 ms. +- Standalone, the first `hipMemcpy(H2H)` after HIP init costs about 18 ms even for 1 KB (later + calls ~0 ms), and the first OpenMP parallel region about 3 ms (0.2 ms later). The larger + in-client values were not explained further without a profiler. +- Across CI, the 752 first calls on small problems (<4M elements) total 17.6 s, median 20.6 ms. + This is why sparse configs show 50–70 ms for problems as small as 256×256×128. + +### 5. Thread scaling breaks down on a shared host + +The init loops use OpenMP's default static schedule, so one core shared with another process holds +up the whole loop. + +- Pure `sin` scales linearly to 8 threads (462 → 60 ms), then flattens: 39.8 ms at 16 threads, + 40.2 ms at 25. +- With `schedule(runtime)` on the same cores, 25 threads take 58 ms with `OMP_SCHEDULE=static` + and 21 ms with `dynamic,65536`. +- Identical `sk_bgemm_div` runs took 9.7, 15.0 and 8.4 s; B's init alone ranged from 3,665 to + 11,690 ms. + +## Caveats + +- HEAD's `DataInitialization` differs from the measured binary by about 300 lines (MX data + generator on/off). The trig loop, `copyInputs`/`maxElements`, `m_cpuInit` and + `getReferenceFloat` paths are the same shape in both, but absolute numbers on HEAD may differ. +- The ~50% MX waste figure comes from the standalone harness; `getReferenceFloat` was not timed + inside the client. +- Sparse configs were only checked for first-call overhead; the pruning code was not profiled. +- Machine contention adds large run-to-run variance; ranges above are across repeated runs. + +## Possible follow-ups (not implemented) + +- Stop building the discarded MX reference floats when the caller ignores them. +- Make trig init cheaper: no per-element allocation, flat index for default strides, `sinf` or a + bounded argument. +- Copy only the current problem's size in `copyInputs`, and fix or remove the dead `m_cpuInit` + path. +- Use a dynamic schedule (or larger chunks) for the init loops. diff --git a/projects/hipblaslt/tensilelite/timing_runs/cpu_reference_slow_path_investigation.md b/projects/hipblaslt/tensilelite/timing_runs/cpu_reference_slow_path_investigation.md new file mode 100644 index 000000000000..1c9e5f31ab3e --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/cpu_reference_slow_path_investigation.md @@ -0,0 +1,442 @@ +# Slow CPU reference GEMM path investigation (gfx950 per-config rerun 2) + +`full_runs_gfx950/per_config_rerun_2/AGGREGATE_analysis.txt` reports **Reference GEMM (CPU)** at +1,080,469 ms (42.1% of wall). This is the largest category. The client's reference GEMM +(`SolveGemmCPU` in `client/src/Reference.cpp`) has two implementations: + +- a tiled, vectorised **fast path** (`solveCPUFast`), used when `isFastPathEligible()` accepts the + problem; +- a scalar **slow path** (`ReferenceSolution<…>::SolveCPU`) for everything else. + +The aggregate already splits the two with the `solve_cpu_fast*` and `solve_cpu_slow*` timers: + +| Timer | Calls | Total (ms) | Mean (ms) | +| --- | --- | --- | --- | +| `solve_cpu_slow` | 1,947 | 727,756.12 | 373.78 | +| `solve_cpu_slow_per_solution_inputs_changed` | 19 | 13,166.07 | 692.95 | +| **Slow path, total** | **1,966** | **740,922** | | +| `solve_cpu_fast` | 26,656 | 200,931.33 | 7.54 | +| `solve_cpu_fast_per_solution_inputs_changed` | 1,127 | 138,501.63 | 122.89 | +| **Fast path, total** | **27,783** | **339,433** | | + +The slow path is 6.6% of reference calls but 68.6% of reference time. This note covers: + +- which YAMLs take the slow path; +- why each problem is rejected from the fast path; +- why the slow path is so slow. + +Timing claims come from measurements; code reading was only used to decide what to measure. +Inferences are marked as such. + +## Summary + +1. **Two YAMLs are 81% of the slow path.** `streamk/sk_I8gemm_quick.yaml` (310.7 s) and + `streamk/sk_f8gemm_quick.yaml` (289.4 s) together take 600 s of the 741 s. All of it is their + `4111×4111×4111` problems. The fast path rejects them on type: + - Int8 inputs (all four output types: I8, BF16, Half, I32); + - Float8 output (`F8 × F8 → F8`). + + In the same YAML, the same-size problems with a Float, Half or BF16 output take the fast path in + 0.47–1.21 s. The slow-path calls each took 45–84 s. +2. **The slow path is serialised by a lock taken once per output element.** After computing each + output element, every thread calls `omp_set_num_threads(64)` and then enters a global + `#pragma omp critical`. The critical section exists only for the AMAX-D reduction, but it is + entered even when `outputAmaxD()` is false; the check is inside it. + - **Stack samples:** 75 of the roughly 100 thread samples taken during a slow-path GEMM were + waiting in `__kmp_acquire_queuing_lock`. + - **Disassembly:** the per-element call sequence is + `omp_set_num_threads` → `__kmpc_critical` → `outputAmaxD` test → `__kmpc_end_critical`, on + the path every element takes. + - **Scaling:** with 25 threads on 25 cores, a 1024³ fp32 GEMM is barely faster than with one + thread (1,727–2,364 ms against 2,626–2,660 ms). +3. **Entering the critical section only when AMAX-D is requested gives about a 4.6× speed-up** + (prototype `refpath3`, scratch only). + - **Driver, 2048³ fp32:** 45,237 ms → 1,507 ms. Other unmodified runs of the same problem took + 6.8–14 s; the contended lock makes the baseline erratic. + - **Driver, 1024³ fp32:** 1,727 ms → 191 ms. + - **Replay of all 59 slow-path YAMLs:** slow-path total 546.6 s → 119.5 s. Every config still + passes, and the driver's results are bit-identical (max diff 0). +4. **Even after that fix, the slow path is about 25–30× slower than the fast path on the same + problem.** At 4111³ on 25 cores, the guarded slow path took 16.3 s (fp32) and 17.5 s (f8 in). + The fast path took 0.65 s (f64), 0.76 s (f32) and 0.54 s (f8 in). + + The slow path computes each output element as a separate scalar dot product. In the I8 K loop, + each MAC is 2 sign-extending loads, an `imul` and an `add`, with no vectorisation or blocking. + Adding fast-path support for **Int8 inputs** and **Float8 outputs** would move 79% of the + remaining (post-fix) slow-path time to the fast path. +5. **The other rejection reasons are small.** After the fix, measured in the replay: + - `gradient`: 8.8 s + - `useE`: 7.7 s + - complex types: 4.1 s + - MX K not a multiple of the MX block: 2.6 s + - `NumElementsToValidate: 128` (sparse validation, intended): 1.7 s + +## Method + +- **Production numbers:** the per-config logs in + `/home/alvasile/timing_runs/full_runs_gfx950/per_config_rerun_2//_analysis.txt`. For + each config, the `solve_cpu_slow*` and `solve_cpu_fast*` rows were summed from the first + "TIMING BY PHASE" table (`/tmp/refpath/percfg.py` → `/tmp/refpath/percfg_prod.txt`). + 59 configs have at least one slow-path call. +- **Instrumented client (`refpath`):** + - A copy of the current `client/` sources in `/tmp/refpath/client_src`, built with the + production flags by `/tmp/bookkeep/build.sh`. + - `isFastPathEligible()` already has a `rejectFast(reason)` hook (disabled with `if (false)`). + The copy stores the reason instead, and `SolveGemmCPU` prints one `REFPATH` line per call: the + path, the reason, M/N/K/batch, the A/B/C/D types, `elementsToValidate`, and the elapsed ms. + - Reasons are reported in the order `SolveGemmCPU` checks them: `tryFastPath`, then the 20% + validation-density rule, then `isFastPathEligible()`. +- **Replays:** + - Each of the 59 configs was rerun through pytest with the production artifacts: + `--use-cache --artifact-dir` holding a symlink to `/tmp/tl_split_gfx950/artifacts/.tar.gz`, + and `--prebuilt-client`. + - Same environment as the production rerun: `OMP_NUM_THREADS=25`, 25 pinned idle cores, default + `KMP_BLOCKTIME`, `TimingInstrumentation=True`, msgpack library format. + - **Baseline (`refpath`):** cores `28-39,42-46,49-53,55-57`, GPU 4. Logs and per-call `REFPATH` + lines are in `/tmp/refpath/logs/`. + - **Guarded critical section (`refpath3`):** cores `64-88`, GPU 5. Results are in + `/tmp/refpath/logs3/`. + - The baseline replay has the same 1,966 slow-path calls as production, problem for problem. + Totals differ (546.6 s against 740.9 s) because the contended lock makes run times erratic; + see finding 3. + - Only host-side timers are compared. The GPUs are shared with other users. +- **Same-problem comparisons:** + - The `cpu-gemm-driver` object from the production build was relinked against the instrumented + client library: `/tmp/refpath/cpu-gemm-driver`, `-driver2` and `-driver3`. + - `-driver2` and `-driver3` read the reference's thread cap from `TL_MAXT`. Production hard-codes + `MAX_OMP_THREADS 64`. + - Runs used cores `192-216`, whose SMT siblings were idle (2% busy), with `KMP_BLOCKTIME=0` + unless stated. The timing is the `REFPATH` value for the reference call only; the driver's + validation step is excluded. +- **Stack sampling:** `perf` is blocked and `ptrace_scope=1` (no attaching), so the driver was + launched under `rocgdb -batch`. Four `SIGINT`s, 1.5 s apart, each triggered + `thread apply all bt 4`. +- **Assembly:** `objdump -dr` of `Reference.cpp.o` from the production-flag build, on the + `.omp_outlined` body of + `ReferenceSolution,int,int>::SolveCPU` (the I8 + case from `sk_I8gemm_quick`). + +## Which YAMLs take the slow path, and why + +### By rejection reason + +| Reason | Calls | Replay (s) | Replay, guarded critical (s) | +| --- | --- | --- | --- | +| Type: Float8 × Float8 → **Float8** | 6 | 172.4 | 44.1 | +| Type: **Int8** × Int8 → Int8 | 714 | 92.6 | 15.1 | +| Type: **Int8** × Int8 → BFloat16 | 99 | 79.1 | 12.7 | +| Type: **Int8** × Int8 → Int32 | 39 | 70.2 | 10.2 | +| Type: **Int8** × Int8 → Half | 99 | 51.8 | 11.0 | +| `useE` | 18 | 20.1 | 7.7 | +| `gradient` | 47 | 19.9 | 8.8 | +| `K_not_multiple_of_mxBlockA` (MXFP4, K = 333/255/63, block 32) | 36 | 19.8 | 2.6 | +| Type: ComplexFloat | 251 | 9.2 | 2.5 | +| Type: Float8 × **Half** → **Float8** | 24 | 4.6 | 1.5 | +| Type: ComplexDouble | 64 | 4.4 | 1.6 | +| `NumElementsToValidate: 128` (fewer than 20% of D validated) | 569 | 2.4 | 1.7 | +| **Total** | **1,966** | **546.6** | **119.5** | + +Notes on the reasons: + +- **Type rejections:** the fast path accepts these types: + - inputs: Float, Double, Half, BFloat16, the four FP8/BF8 variants, and Float4; + - outputs (C and D): Float, Double, Half, BFloat16. + + Int8 inputs, Int8/Int32/FP8 outputs and complex types fall through. +- **`NumElementsToValidate: 128`:** this is the intended behaviour. The slow path then computes + only every `NextPrime(total/128)`-th element, so these 569 calls cost 2.4 s in total. + `elementsToValidate` arrives as `size_t`, so the YAML default `-1` counts as "validate + everything" and does not trigger this rule. +- **No other reasons occurred** for this set: no `amaxD`, `scaleCD`, `layout`, `multi_index`, + `mirror_indices`, `no_batch_indices` or `tryFastPath=false` rejections. + +### By YAML (all 59) + +"Prod" is the production per-config log. "Replay" is the baseline instrumented replay. +"Guarded critical" is the `refpath3` replay; see Option A. + +| YAML (`Tests/common/…`) | Slow calls | Reason(s) (calls) | Prod (s) | Replay (s) | Replay, guarded critical (s) | +| --- | --- | --- | --- | --- | --- | +| `streamk/sk_I8gemm_quick.yaml` | 12 | type A=Int8/B=Int8→D=Int8 (3); type A=Int8/B=Int8→D=BFloat16 (3); type A=Int8/B=Int8→D=Half (3); type A=Int8/B=Int8→D=Int32 (3) | 310.66 | 252.50 | 43.20 | +| `streamk/sk_f8gemm_quick.yaml` | 6 | type A=Float8/B=Float8→D=Float8 (6) | 289.45 | 172.42 | 44.06 | +| `gemm/gfx950/mxfp4_mxfp4_bf16_tn_act.yaml` | 36 | K_not_multiple_of_mxBlockA (36) | 25.26 | 19.84 | 2.64 | +| `streamk/gfx950/sk_hgemm_use_e.yaml` | 11 | useE (8); gradient (3) | 22.87 | 17.01 | 7.36 | +| `streamk/gfx950/data_parallel_static_hgemm_use_e.yaml` | 8 | useE (6); gradient (2) | 14.51 | 13.04 | 5.70 | +| `sparse/gfx950/spmm_i8_sb.yaml` | 196 | type A=Int8/B=Int8→D=Int8 (196) | 6.92 | 5.94 | 1.12 | +| `sparse/gfx950/spmm_i8.yaml` | 196 | type A=Int8/B=Int8→D=Int8 (196) | 6.66 | 6.03 | 1.10 | +| `gemm/gfx950/f8f16mix_f8s.yaml` | 24 | type A=Float8/B=Half→D=Float8 (24) | 5.80 | 4.58 | 1.53 | +| `gemm/bufferLoadStore0.yaml` | 189 | type A=ComplexFloat/B=ComplexFloat→D=ComplexFloat (189) | 5.01 | 3.96 | 1.03 | +| `gradient/fp16_gradient_bias.yaml` | 21 | gradient (21) | 4.64 | 5.21 | 2.02 | +| `gemm/cgemm.yaml` | 54 | type A=ComplexFloat/B=ComplexFloat→D=ComplexFloat (54) | 4.47 | 3.94 | 0.62 | +| `gradient/gfx950/bf16_gradient_bias.yaml` | 21 | gradient (21) | 4.39 | 4.69 | 1.35 | +| `gemm/zgemm.yaml` | 54 | type A=ComplexDouble/B=ComplexDouble→D=ComplexDouble (54) | 4.05 | 3.67 | 0.59 | +| `sparse/gfx94x/spmm_i8_mi16_sb.yaml` | 40 | type A=Int8/B=Int8→D=Int8 (40) | 3.99 | 2.22 | 0.19 | +| `sparse/gfx950/spmm_dtl.yaml` | 16 | type A=Int8/B=Int8→D=Int8 (16) | 3.23 | 1.90 | 0.55 | +| `sparse/gfx94x/spmm_i8_mi32_sb.yaml` | 40 | type A=Int8/B=Int8→D=Int8 (40) | 2.96 | 2.22 | 0.24 | +| `sparse/gfx950/spmm_i8hs.yaml` | 64 | type A=Int8/B=Int8→D=Half (64) | 2.75 | 2.70 | 0.28 | +| `sparse/gfx94x/spmm_i8_mi32.yaml` | 40 | type A=Int8/B=Int8→D=Int8 (40) | 2.72 | 2.94 | 0.22 | +| `sparse/gfx94x/spmm_i8_mi16.yaml` | 40 | type A=Int8/B=Int8→D=Int8 (40) | 2.43 | 2.11 | 0.27 | +| `sparse/gfx950/spmm_i8bs.yaml` | 64 | type A=Int8/B=Int8→D=BFloat16 (64) | 2.06 | 2.56 | 0.45 | +| `streamk/sk_cgemm_quick.yaml` | 8 | type A=ComplexFloat/B=ComplexFloat→D=ComplexFloat (8) | 2.03 | 1.31 | 0.87 | +| `sparse/gfx94x/i8_activation.yaml` | 44 | type A=Int8/B=Int8→D=Int8 (44) | 1.96 | 2.01 | 0.16 | +| `gemm/gemm_bias_strided_batched_gfx940.yaml` | 24 | type A=Int8/B=Int8→D=Int8 (24) | 1.50 | 1.53 | 0.13 | +| `gemm/gemm_bias_strided_batched_broadcast_gfx940.yaml` | 24 | type A=Int8/B=Int8→D=Int8 (24) | 1.32 | 1.32 | 0.35 | +| `sparse/gfx94x/spmm_i8bs_sb.yaml` | 16 | type A=Int8/B=Int8→D=BFloat16 (16) | 1.06 | 0.70 | 0.05 | +| `sparse/gfx94x/spmm_i8hs.yaml` | 16 | type A=Int8/B=Int8→D=Half (16) | 0.89 | 1.24 | 0.07 | +| `sparse/gfx94x/spmm_i8is.yaml` | 16 | type A=Int8/B=Int8→D=Int32 (16) | 0.86 | 0.84 | 0.02 | +| `sparse/gfx94x/spmm_i8is_sb.yaml` | 16 | type A=Int8/B=Int8→D=Int32 (16) | 0.72 | 0.75 | 0.07 | +| `sparse/gfx94x/spmm_i8bs.yaml` | 16 | type A=Int8/B=Int8→D=BFloat16 (16) | 0.71 | 0.82 | 0.04 | +| `gemm/gfx950/i8_gsu_gfx950.yaml` | 4 | type A=Int8/B=Int8→D=Int8 (2); type A=Int8/B=Int8→D=Int32 (2) | 0.71 | 0.97 | 0.06 | +| `gemm/i8_gsu_gfx940.yaml` | 4 | type A=Int8/B=Int8→D=Int8 (2); type A=Int8/B=Int8→D=Int32 (2) | 0.70 | 0.79 | 0.13 | +| `sparse/gfx94x/spmm_i8hs_sb.yaml` | 16 | type A=Int8/B=Int8→D=Half (16) | 0.69 | 1.04 | 0.07 | +| `streamk/sk_zgemm_quick.yaml` | 10 | type A=ComplexDouble/B=ComplexDouble→D=ComplexDouble (10) | 0.66 | 0.75 | 1.02 | +| `gemm/fp16_tn.yaml` | 166 | NumElementsToValidate=128 (166) | 0.38 | 0.19 | 0.22 | +| `gemm/gfx950/fp16_use_e_wide_store.yaml` | 4 | useE (4) | 0.32 | 0.12 | 0.02 | +| `gemm/ag.yaml` | 180 | NumElementsToValidate=128 (180) | 0.31 | 1.13 | 0.46 | +| `gemm/gfx950/lds_tr.yaml` | 78 | NumElementsToValidate=128 (78) | 0.29 | 0.38 | 0.43 | +| `gemm/lsu_i8.yaml` | 15 | type A=Int8/B=Int8→D=Int8 (15) | 0.27 | 0.43 | 0.14 | +| `sparse/gfx94x/spmm_vw_lg_one.yaml` | 8 | type A=Int8/B=Int8→D=Int8 (8) | 0.07 | 0.04 | 0.02 | +| `gemm/use_beta_false.yaml` | 6 | NumElementsToValidate=128 (6) | 0.07 | 0.02 | 0.02 | +| `gradient/fp16_gradient_activation.yaml` | 8 | NumElementsToValidate=128 (8) | 0.07 | 0.04 | 0.06 | +| `gemm/fp16fp32mix.yaml` | 8 | NumElementsToValidate=128 (8) | 0.06 | 0.05 | 0.06 | +| `sparse/gfx950/spmm_vw_lg_one.yaml` | 8 | type A=Int8/B=Int8→D=Int8 (8) | 0.06 | 0.05 | 0.06 | +| `gemm/bf16_tn.yaml` | 27 | NumElementsToValidate=128 (27) | 0.06 | 0.02 | 0.07 | +| `gemm/fp16_use_e.yaml` | 18 | NumElementsToValidate=128 (18) | 0.06 | 0.40 | 0.13 | +| `sparse/gfx94x/spmm_vw_lg_one_sb.yaml` | 8 | type A=Int8/B=Int8→D=Int8 (8) | 0.06 | 0.02 | 0.02 | +| `sparse/gfx950/spmm_vw_lg_one_sb.yaml` | 8 | type A=Int8/B=Int8→D=Int8 (8) | 0.06 | 0.03 | 0.02 | +| `gemm/fp16_HH_BHS_bf16mfma.yaml` | 27 | NumElementsToValidate=128 (27) | 0.04 | 0.03 | 0.03 | +| `ductile/gfx950/f8_tn_mx_subtile.yaml` | 2 | NumElementsToValidate=128 (2) | 0.03 | 0.01 | 0.01 | +| `groupedgemm/grouped_gemm.yaml` | 8 | NumElementsToValidate=128 (8) | 0.03 | 0.02 | 0.03 | +| `gemm/fp32_tn.yaml` | 27 | NumElementsToValidate=128 (27) | 0.03 | 0.03 | 0.08 | +| `streamk/sk_bgemm_div.yaml` | 1 | NumElementsToValidate=128 (1) | 0.01 | 0.01 | 0.02 | +| `exception/f16_cf32_nan.yaml` | 3 | NumElementsToValidate=128 (3) | 0.01 | 0.02 | 0.05 | +| `benchmark_runs/fp32_tn.yaml` | 3 | NumElementsToValidate=128 (3) | 0.01 | 0.00 | 0.01 | +| `gemm/gfx950/bf16_cvt.yaml` | 1 | NumElementsToValidate=128 (1) | 0.01 | 0.00 | 0.00 | +| `gemm/large_size.yaml` | 1 | NumElementsToValidate=128 (1) | 0.01 | 0.01 | 0.00 | +| `gemm/fp32_nt.yaml` | 1 | NumElementsToValidate=128 (1) | 0.00 | 0.00 | 0.00 | +| `gemm/nan_bounds_check_odd_padding.yaml` | 3 | NumElementsToValidate=128 (3) | 0.00 | 0.00 | 0.00 | +| `gemm/bf16_tn_predict.yaml` | 1 | NumElementsToValidate=128 (1) | 0.00 | 0.01 | 0.00 | +| **Total** | 1966 | | **740.9** | **546.6** | **119.5** | + +Rows below about 0.1 s are within run-to-run noise. Some small rows read slightly higher with +the guarded critical section for that reason. + +### The two dominant YAMLs, per call (4111³, one call each) + +| YAML | A × B → D | Path | Replay (ms) | Guarded critical (ms) | +| --- | --- | --- | --- | --- | +| `sk_I8gemm_quick` | I8 × I8 → I8 | slow (type) | 60,465 | 9,985 | +| `sk_I8gemm_quick` | I8 × I8 → BF16 | slow (type) | 72,756 | 11,626 | +| `sk_I8gemm_quick` | I8 × I8 → Half | slow (type) | 45,210 | 9,998 | +| `sk_I8gemm_quick` | I8 × I8 → I32 | slow (type) | 66,115 | 9,491 | +| `sk_f8gemm_quick` | F8 × F8 → **F8** (2 calls) | slow (type) | 84,331 / 83,567 | 19,464 / 23,092 | +| `sk_f8gemm_quick` | F8 × F8 → Float (2 calls) | fast | — | 702 / 1,206 | +| `sk_f8gemm_quick` | F8 × F8 → Half | fast | — | 655 | +| `sk_f8gemm_quick` | BF8 × BF8 → Float | fast | — | 636 | +| `sk_f8gemm_quick` | F8 × Half → Float | fast | — | 655 | +| `sk_f8gemm_quick` | BF8 × F8 → BF16 | fast | — | 469 | + +In `sk_f8gemm_quick`, the only problem type that takes the slow path is the one whose output is +Float8. Its 4111³ problem is 28× (guarded) to 180× (unguarded) slower than its fast-path siblings +in the same YAML. + +## Why the slow path is slow + +### Measured: it does not scale with threads + +`cpu-gemm-driver`, fp32, `--transA`, slow path, cores `192-216` (25 cores, SMT siblings idle). +The fast path is shown for scale. + +| Threads (cores) | Slow path, 1024³ (ms) | Fast path, 1024³ (ms) | +| --- | --- | --- | +| 1 (1 core) | 2,642 | 99 | +| 25 (25 cores), `MAX_OMP_THREADS=25` | 2,329 | 21 | +| 64 on 25 cores (production cap) | 2,364 | 25 | +| 64 (64 cores) | 1,492 | 14 | + +Going from 1 to 25 threads speeds the slow path up by only 1.1×. A thread-state trace (every +0.25 s from `/proc//task/*/status`) showed all 25 threads running for the whole reference +call, so the threads are busy, not idle. + +### Measured: the threads are waiting on one lock + +`rocgdb` stack samples were taken during a 2048³ fp32 slow-path call, 25 threads, 4 samples: + +| Top of stack | Thread samples | +| --- | --- | +| `__kmpc_critical_with_hint` → `__kmp_acquire_queuing_lock` → `__kmp_wait_4` | 54 | +| … → `__kmp_wait_4` → `sched_yield` (same lock, yielding) | 21 | +| `__kmp_suspend_64` → `pthread_cond_wait` (idle pool threads) | 24 | +| Main thread in `runGemm` | 1 | + +### The code and its assembly + +`Reference.cpp`, at the end of the per-output-element body of the `#pragma omp parallel` / +`#pragma omp for` loop in `ReferenceSolution<…>::SolveCPU`: + +```cpp + omp_set_num_threads(MAX_OMP_THREADS); +#pragma omp critical + { + if constexpr(notCmplxAmaxD) + { + if(problem.outputAmaxD()) + { /* amaxD = max(amaxD, |resultD|) */ } + } + } +``` + +The I8 instantiation compiles to the following, executed once per output element (`.omp_outlined` +body, `objdump -dr`): + +``` +62c7a: mov $0x40,%edi +62c7f: call omp_set_num_threads +62c94: call __kmpc_critical ; .gomp_critical_user_.var (one global lock) +62ca1: cmpb $0x1,0x134(%rax) ; problem.outputAmaxD() — tested inside the lock +62ca8: jne 62cd2 +... +62ce2: call __kmpc_end_critical +``` + +Just before this, the same body calls `operator delete`. That is the per-element +`std::vector actArgs` built for `Activation()`. Its cost was not measured +separately. + +The K loop itself is cheap per instruction. The I8 version (`61da0`–`61dcf`) is unrolled by 2 and +does two `movsbl` loads, an `imul` and an `add` per MAC, stepping A and B by their K strides. With +the lock removed, single-thread throughput is about 0.41 GMAC/s against the fast path's +10.8 GMAC/s (1024³: 2,625 ms against 99 ms). So even an uncontended slow path is about 25× slower +per core. + +### Memory layout is a secondary effect + +fp32 and f8 inputs, 2048³, slow path, unmodified (64 threads on 25 cores): + +| Layout | fp32 (ms) | f8 in (ms) | +| --- | --- | --- | +| NN | 13,900 | 16,704 | +| TN (A contiguous in K) | 11,273 | 12,453 | +| NT (B strided in K) | 20,152 | 18,856 | + +The spread is about 1.5–1.8×. That is small next to the lock (a 9–30× change), so layout was not +pursued further. + +## Options + +### A. Take the critical section only when AMAX-D is requested (measured, prototype `refpath3`) + +**Change.** Move `#pragma omp critical` inside `if(problem.outputAmaxD())`, and drop the +per-element `omp_set_num_threads`. The whole diff is in `/tmp/refpath/client_src3/src/Reference.cpp`. + +**Further improvement (not measured).** AMAX-D problems would still serialise. They should use a +per-thread maximum combined once, e.g. `reduction(max:amaxD)`. No gfx950 config in this set hit +the slow path with AMAX-D. + +| Measurement | Unmodified | Guarded | +| --- | --- | --- | +| Driver 1024³ fp32, 25 threads | 1,727 ms | 191 ms | +| Driver 2048³ fp32, 25 threads | 45,237 ms (other runs: 6.8–14 s) | 1,507 ms | +| Driver 1024³ fp32, 1 thread | 2,660 ms | 2,625 ms | +| Driver results | PASSED, max diff 0 | PASSED, max diff 0 | +| Replay, slow-path total (59 YAMLs) | 546.6 s | 119.5 s (−78%) | +| Replay, `sk_I8gemm_quick` pytest wall | 258 s | 49 s | +| Replay, `sk_f8gemm_quick` pytest wall | 184 s | 56 s | +| Replay, configs passing | 59 / 59 | 59 / 59 | + +**Estimated effect on the aggregate.** Applying the replay ratio (119.5 / 546.6) to production's +740.9 s gives about 162 s, saving about 580 s, or 22% of the aggregate wall. This is an +estimate; the production baseline itself varies, as shown above. + +**The 64-thread cap.** Once the lock is gone, the hard-coded cap no longer matters on 25 cores. +At 2048³ it measured 1,392 / 1,388 ms with the 64 cap against 1,437 / 1,373 ms with 25 threads. +`omp_set_num_threads` persists for the calling thread, so the cap also changes later OpenMP +regions; using `omp_get_max_threads()` would make it follow `OMP_NUM_THREADS`. + +**`KMP_BLOCKTIME`.** With the default (200 ms), the guarded version measured 2,010–2,165 ms +against 1,388–1,437 ms with `KMP_BLOCKTIME=0`. This matches the +[per-solution bookkeeping](per_solution_bookkeeping_investigation.md) finding that +`KMP_BLOCKTIME=0` cut `cpu_reference_gemm` by 2–5×. + +### B. Fast-path support for Int8 inputs and Float8 outputs (not prototyped) + +These two type gaps are 94.6 s of the 119.5 s left after Option A (79%), and nearly all of the +production slow path. Same-size fast-path calls take 0.47–1.21 s, against 9.5–23 s guarded and +45–84 s unguarded. + +For reference, driver measurements of the fast path at 4111³ on 25 cores: + +| Inputs | Fast path, 4111³ (ms) | +| --- | --- | +| f64 | 651 | +| f32 | 761 | +| f8 | 541 | + +What each extension needs: + +- **Int8 inputs.** The accumulation must be exact. A float accumulator is not: a 4111-long I8 dot + product can reach about 6.7e7, above 2²⁴. Options are the existing `double` instantiation + (exact below 2⁵³; 651 ms at 4111³) or a dedicated int32/int64 inner kernel. + - Int8 and Int32 outputs also need `storeFrom` specialisations that use the slow path's + `SaturateCast` semantics. + - Half and BF16 outputs already have write-back. +- **Float8 output.** A `storeFrom` (and BF8) write-back that matches the slow + path's `SaturateCast`. + - The F8 × Half → F8 case (`f8f16mix_f8s`) also needs the mixed-input shadow; the fast path + already handles mixed F8/Half inputs (`F8 × Half → Float` is fast in `sk_f8gemm_quick`). + +**Estimate.** If the 4111³ calls cost what their fast-path siblings do, these YAMLs' reference +time would drop from about 600 s in production to about 10 s. This is an inference from the +sibling measurements; equivalence with the slow path has to be validated per type, as it was when +FP8/FP4 were added to the fast path. + +### C. Smaller rejection reasons (not prototyped) + +Times are after Option A, from the replay. + +| Reason | Remaining (s) | What it would take | +| --- | --- | --- | +| `gradient` | 8.8 | Gradient GEMM with bias reduction. The slow path also runs a separate reduction region. Larger change. | +| `useE` | 7.7 | Store the pre-activation value to E in the fast-path epilogue. | +| `K_not_multiple_of_mxBlockA` | 2.6 | MXFP4 K tails (K = 333, 255, 63 with block 32). Handle a partial last MX block. | +| Complex types | 4.1 | New fast-path variant. Probably not worth it at this size. | +| `NumElementsToValidate: 128` | 1.7 | Intended (sparse validation). Leave as is. | + +### D. Make the rejection reason observable + +The `rejectFast()` logging is compiled out with `if (false)`, so this audit needed a rebuilt +client. Two cheap ways to keep it reproducible: + +- an environment or log-level switch for the reason line; +- a per-reason timer category, e.g. `solve_cpu_slow.`, so the aggregate shows it directly. + +## Not measured / open items + +- **The fast path's own 339 s.** It is out of scope here; most of it is the + `subtile_mxfp8*` YAMLs (`gemm/gfx950/subtile_mxfp8.yaml` 143 s and the + `*_correctness_regression*` variants). +- **The per-element `actArgs` vector** allocated and freed in the slow path is visible in the + assembly but was not timed separately. +- **AMAX-D problems** would still take the lock per element after Option A; use a reduction. +- **`ductile/gfx950/f8_tn_mx_subtile.yaml`** reaches the reference with `elementsToValidate=128`, + although the YAML sets `NumElementsToValidate: 0`. The cost is negligible (0.03 s); the cause + was not investigated. +- **Run-to-run variance.** Production slow-path time (740.9 s) and the baseline replay (546.6 s) + differ, because lock contention makes the unmodified slow path erratic; the same 2048³ problem + measured 6.8 s to 45 s. The guarded version was stable (1.37–1.51 s). + +## Scratch files + +All in `/tmp/refpath/` (nothing in the repo was changed): + +- **Sources:** + - `client_src/`: `REFPATH` logging. + - `client_src2/`: adds the `TL_MAXT` thread cap. + - `client_src3/`: adds the guarded critical section. +- **Builds:** `/tmp/bookkeep/build_refpath{,2,3}/`, plus the relinked `cpu-gemm-driver{,2,3}`. +- **Replays:** + - Runners: `run.sh` and `run3.sh`; config list `cfgs.txt`. + - Per-config `.refpath` lines and analyses: `logs/`, `logs3/`. + - Summaries: `summ.py` → `summ_base.txt`, `summ_fix.txt`; YAML table `table.md`. +- **Driver experiments:** `layout.out`, `threads.out`, `ab.sh`, and `gdb.out` (stack samples). +- **Assembly:** `i8_outlined_r.s`. diff --git a/projects/hipblaslt/tensilelite/timing_runs/driver.log b/projects/hipblaslt/tensilelite/timing_runs/driver.log new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_1_build_cold_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_1_build_cold_analysis.txt new file mode 100644 index 000000000000..8b297fc49bdc --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_1_build_cold_analysis.txt @@ -0,0 +1,135 @@ +======================================================================================================================== +TIMING ANALYSIS SUMMARY +======================================================================================================================== + +TIMING BY PHASE +------------------------------------------------------------------------------------------------------------------------ + Category Count Total (ms) Mean (ms) % of Parent % of Wall +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 1 223,910.86 223,910.86 100.0% + -------------------------------------------------------------------------------------------------------------------- + python_kernel_compilation 25 190,938.97 7,637.56 85.3% 85.3% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_codegen 25 59,556.26 2,382.25 31.2% 26.6% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_src_co 25 46,690.02 1,867.60 24.5% 20.9% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_kernel_build_src_co.compile 9 46,538.74 5,170.97 99.7% 20.8% + python_kernel_build_src_co.unbundle 9 82.14 9.13 0.2% 0.0% + python_kernel_build_src_co.cache_check 25 53.03 2.12 0.1% 0.0% + python_kernel_build_src_co.cache_populate 9 8.75 0.97 0.0% 0.0% + (untracked) 5.37 0.0% 0.0% + python_kernel_build_src_co.setup 25 1.21 0.05 0.0% 0.0% + python_kernel_build_src_co.move 9 0.79 0.09 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_setup 25 30,494.49 1,219.78 16.0% 13.6% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_assemble 25 20,585.84 823.43 10.8% 9.2% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_postprocess 25 16,598.97 663.96 8.7% 7.4% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_benchpost_lib_construction 25 10,985.80 439.43 66.2% 4.9% + python_benchpost_library_write 25 4,023.58 160.94 24.2% 1.8% + python_benchpost_naming 25 1,582.13 63.29 9.5% 0.7% + python_benchpost_client_config 25 5.83 0.23 0.0% 0.0% + (untracked) 1.63 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_validate 25 12,878.83 515.15 6.7% 5.8% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_co 25 1,691.70 67.67 0.9% 0.8% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_setup 25 1,433.88 57.36 0.8% 0.6% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 680.64 0.4% 0.3% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_helpers 25 328.35 13.13 0.2% 0.1% + -------------------------------------------------------------------------------------------------------------------- + python_solution_generation 25 26,205.76 1,048.23 11.7% 11.7% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_forked_solutions 25 26,204.81 1,048.19 100.0% 11.7% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 0.67 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_fork_permutations 25 0.24 0.01 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_custom_kernels 25 0.04 0.00 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_write_solutions 25 2,811.70 112.47 1.3% 1.3% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_dump 25 2,686.05 107.44 95.5% 1.2% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_prepare 25 70.57 2.82 2.5% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_wsol_prepare_nocache 25 70.50 2.82 99.9% 0.0% + (untracked) 0.07 0.1% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 54.60 1.9% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_header 25 0.49 0.02 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + (untracked) 2,396.58 1.1% 1.1% + -------------------------------------------------------------------------------------------------------------------- + python_solution_indexing 25 1,525.74 61.03 0.7% 0.7% + -------------------------------------------------------------------------------------------------------------------- + python_write_cache 25 32.07 1.28 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_cache_check 25 0.03 0.00 0.0% 0.0% + +------------------------------------------------------------------------------------------------------------------------ + ESTIMATED WALL CLOCK TIME: 223.91s + +HIERARCHY VALIDATION: OK (all child phases fit within their parents) + +TIME BREAKDOWN (visual, % of wall clock) +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 100.0% ################################################## + python_kernel_compilation 85.3% ########################################## + python_kernel_codegen 26.6% ############# + python_kernel_build_src_co 20.9% ########## + python_kernel_build_src_co.compile 20.8% ########## + python_kernel_build_src_co.unbundle 0.0% + python_kernel_build_src_co.cache_check 0.0% + python_kernel_build_src_co.cache_populate 0.0% + (untracked) 0.0% + python_kernel_build_src_co.setup 0.0% + python_kernel_build_src_co.move 0.0% + python_kernel_bench_setup 13.6% ###### + python_kernel_write_assemble 9.2% #### + python_kernel_bench_postprocess 7.4% ### + python_benchpost_lib_construction 4.9% ## + python_benchpost_library_write 1.8% + python_benchpost_naming 0.7% + python_benchpost_client_config 0.0% + (untracked) 0.0% + python_kernel_validate 5.8% ## + python_kernel_build_co 0.8% + python_kernel_setup 0.6% + (untracked) 0.3% + python_kernel_write_helpers 0.1% + python_solution_generation 11.7% ##### + python_solgen_forked_solutions 11.7% ##### + (untracked) 0.0% + python_solgen_fork_permutations 0.0% + python_solgen_custom_kernels 0.0% + python_write_solutions 1.3% + python_wsol_dump 1.2% + python_wsol_prepare 0.0% + python_wsol_prepare_nocache 0.0% + (untracked) 0.0% + (untracked) 0.0% + python_wsol_header 0.0% + (untracked) 1.1% + python_solution_indexing 0.7% + python_write_cache 0.0% + python_cache_check 0.0% + +KEY INSIGHTS (% of wall clock) +------------------------------------------------------------------------------------------------------------------------ + Python phases (code gen, compilation): 217.14s (97.0%) + Python I/O overhead (cache, YAML): 2.84s (1.3%) + C++ client setup (library loading): 0.00s (0.0%) + Data preparation (CPU init, GPU prep): 0.00s (0.0%) + Reference computation (CPU GEMM): 0.00s (0.0%) + Kernel execution (warmup, benchmark): 0.00s (0.0%) + + diff --git a/projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_2_build_warm_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_2_build_warm_analysis.txt new file mode 100644 index 000000000000..36c62fca0436 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_2_build_warm_analysis.txt @@ -0,0 +1,127 @@ +======================================================================================================================== +TIMING ANALYSIS SUMMARY +======================================================================================================================== + +TIMING BY PHASE +------------------------------------------------------------------------------------------------------------------------ + Category Count Total (ms) Mean (ms) % of Parent % of Wall +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 1 178,518.51 178,518.51 100.0% + -------------------------------------------------------------------------------------------------------------------- + python_kernel_compilation 25 145,363.45 5,814.54 81.4% 81.4% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_codegen 25 59,610.54 2,384.42 41.0% 33.4% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_setup 25 30,651.53 1,226.06 21.1% 17.2% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_assemble 25 20,640.99 825.64 14.2% 11.6% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_postprocess 25 16,806.22 672.25 11.6% 9.4% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_benchpost_lib_construction 25 11,229.74 449.19 66.8% 6.3% + python_benchpost_library_write 25 4,026.50 161.06 24.0% 2.3% + python_benchpost_naming 25 1,542.74 61.71 9.2% 0.9% + python_benchpost_client_config 25 5.68 0.23 0.0% 0.0% + (untracked) 1.56 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_validate 25 13,391.89 535.68 9.2% 7.5% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_co 25 1,681.10 67.24 1.2% 0.9% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_setup 25 1,461.13 58.45 1.0% 0.8% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 744.42 0.5% 0.4% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_helpers 25 313.15 12.53 0.2% 0.2% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_src_co 25 62.48 2.50 0.0% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_kernel_build_src_co.cache_check 25 54.79 2.19 87.7% 0.0% + (untracked) 6.38 10.2% 0.0% + python_kernel_build_src_co.setup 25 1.31 0.05 2.1% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_solution_generation 25 26,379.70 1,055.19 14.8% 14.8% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_forked_solutions 25 26,378.74 1,055.15 100.0% 14.8% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 0.66 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_fork_permutations 25 0.25 0.01 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_custom_kernels 25 0.04 0.00 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_write_solutions 25 2,825.55 113.02 1.6% 1.6% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_dump 25 2,681.36 107.25 94.9% 1.5% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_prepare 25 80.71 3.23 2.9% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_wsol_prepare_nocache 25 80.63 3.23 99.9% 0.0% + (untracked) 0.07 0.1% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 62.97 2.2% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_header 25 0.50 0.02 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + (untracked) 2,392.22 1.3% 1.3% + -------------------------------------------------------------------------------------------------------------------- + python_solution_indexing 25 1,525.05 61.00 0.9% 0.9% + -------------------------------------------------------------------------------------------------------------------- + python_write_cache 25 32.52 1.30 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_cache_check 25 0.03 0.00 0.0% 0.0% + +------------------------------------------------------------------------------------------------------------------------ + ESTIMATED WALL CLOCK TIME: 178.52s + +HIERARCHY VALIDATION: OK (all child phases fit within their parents) + +TIME BREAKDOWN (visual, % of wall clock) +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 100.0% ################################################## + python_kernel_compilation 81.4% ######################################## + python_kernel_codegen 33.4% ################ + python_kernel_bench_setup 17.2% ######## + python_kernel_write_assemble 11.6% ##### + python_kernel_bench_postprocess 9.4% #### + python_benchpost_lib_construction 6.3% ### + python_benchpost_library_write 2.3% # + python_benchpost_naming 0.9% + python_benchpost_client_config 0.0% + (untracked) 0.0% + python_kernel_validate 7.5% ### + python_kernel_build_co 0.9% + python_kernel_setup 0.8% + (untracked) 0.4% + python_kernel_write_helpers 0.2% + python_kernel_build_src_co 0.0% + python_kernel_build_src_co.cache_check 0.0% + (untracked) 0.0% + python_kernel_build_src_co.setup 0.0% + python_solution_generation 14.8% ####### + python_solgen_forked_solutions 14.8% ####### + (untracked) 0.0% + python_solgen_fork_permutations 0.0% + python_solgen_custom_kernels 0.0% + python_write_solutions 1.6% + python_wsol_dump 1.5% + python_wsol_prepare 0.0% + python_wsol_prepare_nocache 0.0% + (untracked) 0.0% + (untracked) 0.0% + python_wsol_header 0.0% + (untracked) 1.3% + python_solution_indexing 0.9% + python_write_cache 0.0% + python_cache_check 0.0% + +KEY INSIGHTS (% of wall clock) +------------------------------------------------------------------------------------------------------------------------ + Python phases (code gen, compilation): 171.74s (96.2%) + Python I/O overhead (cache, YAML): 2.86s (1.6%) + C++ client setup (library loading): 0.00s (0.0%) + Data preparation (CPU init, GPU prep): 0.00s (0.0%) + Reference computation (CPU GEMM): 0.00s (0.0%) + Kernel execution (warmup, benchmark): 0.00s (0.0%) + + diff --git a/projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_3_run_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_3_run_analysis.txt new file mode 100644 index 000000000000..c6a6b83236a4 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_3_run_analysis.txt @@ -0,0 +1,175 @@ +======================================================================================================================== +TIMING ANALYSIS SUMMARY +======================================================================================================================== + +TIMING BY PHASE +------------------------------------------------------------------------------------------------------------------------ + Category Count Total (ms) Mean (ms) % of Parent % of Wall +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 1 84,602.83 84,602.83 99.9% + -------------------------------------------------------------------------------------------------------------------- + python_client_execution 25 84,217.36 3,368.69 99.5% 99.4% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Data Preparation 29,254.25 34.7% 34.5% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + gpu_input_reset 86,908 23,807.02 0.27 81.4% 28.1% + pre_problem 224 4,700.56 20.98 16.1% 5.5% + cpu_reference_gemm 224 3,626.66 16.19 77.2% 4.3% + solve_cpu_slow 8 3,299.45 412.43 91.0% 3.9% + solve_cpu_fast 216 326.78 1.51 9.0% 0.4% + (untracked) 0.43 0.0% 0.0% + cpu_data_init 224 1,069.86 4.78 22.8% 1.3% + (untracked) 4.03 0.1% 0.0% + gpu_input_preparation 224 739.19 3.30 2.5% 0.9% + rotating_buffer_preparation 224 7.48 0.03 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + C++ Client Setup 29,210.24 34.7% 34.5% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + listener_setup 25 13,275.86 531.03 45.4% 15.7% + hip_initialization 25 10,099.40 403.98 34.6% 11.9% + code_object_loading 25 3,704.91 148.20 12.7% 4.4% + library_loading 25 2,072.25 82.89 7.1% 2.4% + lazy_loading_init 25 35.48 1.42 0.1% 0.0% + data_init_setup 25 17.47 0.70 0.1% 0.0% + reporter_setup 25 4.61 0.18 0.0% 0.0% + solution_iterator_setup 25 0.27 0.01 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Reference Computation 11,474.78 13.6% 13.5% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + validate_warmups 86,908 11,474.78 0.13 100.0% 13.5% + validate_gpu_readback 86,908 4,655.34 0.05 40.6% 5.5% + (untracked) 3,735.15 32.6% 4.4% + validate_element_comparison 86,908 3,033.46 0.03 26.4% 3.6% + validate_gpu_sync 86,908 50.82 0.00 0.4% 0.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Kernel Execution 5,177.49 6.1% 6.1% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + post_solution 95,512 2,267.92 0.02 43.8% 2.7% + post_solution_log 95,512 1,789.08 0.02 78.9% 2.1% + post_solution_reporting 95,512 358.50 0.00 15.8% 0.4% + post_solution_perf_calc 95,512 58.86 0.00 2.6% 0.1% + post_solution_validation 95,512 53.62 0.00 2.4% 0.1% + post_solution_sol_advance 95,512 2.72 0.00 0.1% 0.0% + post_solution_result_file 95,512 2.66 0.00 0.1% 0.0% + post_solution_perf_reset 95,512 2.62 0.00 0.1% 0.0% + pre_solution 95,512 1,088.56 0.01 21.0% 1.3% + warmup_runs 173,816 1,081.02 0.01 20.9% 1.3% + kernel_solving 86,908 518.41 0.01 10.0% 0.6% + post_problem 224 105.59 0.47 2.0% 0.1% + solution_selection 95,512 104.20 0.00 2.0% 0.1% + benchmark_runs 86,908 11.77 0.00 0.2% 0.0% + (untracked) 11.77 100.0% 0.0% + finalize_report 25 0.01 0.00 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked C++ overhead) 5,016.77 6.0% 5.9% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Timing Overhead 4,083.83 4.8% 4.8% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + flush_timing_buffer 25 3,989.59 159.58 97.7% 4.7% + timing_overhead 25 94.23 3.77 2.3% 0.1% + -------------------------------------------------------------------------------------------------------------------- + (untracked) 317.18 0.4% 0.4% + -------------------------------------------------------------------------------------------------------------------- + python_cache_check 25 60.60 2.42 0.1% 0.1% + -------------------------------------------------------------------------------------------------------------------- + python_write_solutions 25 7.70 0.31 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_prepare 25 4.33 0.17 56.2% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_wsol_prepare_cache 25 4.19 0.17 96.9% 0.0% + (untracked) 0.13 3.1% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_header 25 3.07 0.12 39.9% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 0.30 3.9% 0.0% + +------------------------------------------------------------------------------------------------------------------------ + ESTIMATED WALL CLOCK TIME: 84.70s + +HIERARCHY VALIDATION: OK (all child phases fit within their parents) + +TIME BREAKDOWN (visual, % of wall clock) +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 99.9% ################################################# + python_client_execution 99.4% ################################################# + Data Preparation 34.5% ################# + gpu_input_reset 28.1% ############## + pre_problem 5.5% ## + cpu_reference_gemm 4.3% ## + solve_cpu_slow 3.9% # + solve_cpu_fast 0.4% + (untracked) 0.0% + cpu_data_init 1.3% + (untracked) 0.0% + gpu_input_preparation 0.9% + rotating_buffer_preparation 0.0% + C++ Client Setup 34.5% ################# + listener_setup 15.7% ####### + hip_initialization 11.9% ##### + code_object_loading 4.4% ## + library_loading 2.4% # + lazy_loading_init 0.0% + data_init_setup 0.0% + reporter_setup 0.0% + solution_iterator_setup 0.0% + Reference Computation 13.5% ###### + validate_warmups 13.5% ###### + validate_gpu_readback 5.5% ## + (untracked) 4.4% ## + validate_element_comparison 3.6% # + validate_gpu_sync 0.1% + Kernel Execution 6.1% ### + post_solution 2.7% # + post_solution_log 2.1% # + post_solution_reporting 0.4% + post_solution_perf_calc 0.1% + post_solution_validation 0.1% + post_solution_sol_advance 0.0% + post_solution_result_file 0.0% + post_solution_perf_reset 0.0% + pre_solution 1.3% + warmup_runs 1.3% + kernel_solving 0.6% + post_problem 0.1% + solution_selection 0.1% + benchmark_runs 0.0% + (untracked) 0.0% + finalize_report 0.0% + (untracked C++ overhead) 5.9% ## + Timing Overhead 4.8% ## + flush_timing_buffer 4.7% ## + timing_overhead 0.1% + (untracked) 0.4% + python_cache_check 0.1% + python_write_solutions 0.0% + python_wsol_prepare 0.0% + python_wsol_prepare_cache 0.0% + (untracked) 0.0% + python_wsol_header 0.0% + (untracked) 0.0% + +KEY INSIGHTS (% of wall clock) +------------------------------------------------------------------------------------------------------------------------ + Python phases (code gen, compilation): 0.00s (0.0%) + Python I/O overhead (cache, YAML): 0.07s (0.1%) + C++ client setup (library loading): 29.21s (34.5%) + Data preparation (CPU init, GPU prep): 29.25s (34.5%) + Reference computation (CPU GEMM): 11.49s (13.6%) + Kernel execution (warmup, benchmark): 5.21s (6.2%) + + +TOP 10 SLOWEST PROBLEMS (by CPU reference time) +------------------------------------------------------------------------------------------------------------------------ + M N K Batch TypeA TypeD CPU Ref (ms) GPU (ms) + ---------------------------------------------------------------------------------------------------------------------- + 255 255 256 1 Float8_fnuz Float8_fnuz 593.36 0.00 + 255 255 193 1 Float8_fnuz Float8_fnuz 446.05 0.00 + 255 255 193 1 Float8_fnuz Float8_fnuz 442.14 0.00 + 255 255 191 1 Float8_fnuz Float8_fnuz 392.43 0.00 + 255 255 193 1 Float8_fnuz Float8_fnuz 378.14 0.00 + 255 255 256 1 Float8_fnuz Float8_fnuz 363.47 0.00 + 255 255 256 1 Float8_fnuz Float8_fnuz 348.50 0.00 + 255 255 256 1 Float8_fnuz Float8_fnuz 335.40 0.00 + 1600 1280 192 1 Half Half 35.50 0.00 + 1600 1280 192 1 Half Half 34.14 0.00 + diff --git a/projects/hipblaslt/tensilelite/timing_runs/dtv/times.txt b/projects/hipblaslt/tensilelite/timing_runs/dtv/times.txt new file mode 100644 index 000000000000..c434fad13812 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/dtv/times.txt @@ -0,0 +1,4 @@ +build_cold rc=0 251s +build_warm rc=0 205s +run rc=0 91s +DONE diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs/run.sh b/projects/hipblaslt/tensilelite/timing_runs/full_runs/run.sh new file mode 100755 index 000000000000..a1066fc7691f --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs/run.sh @@ -0,0 +1,31 @@ +#!/bin/bash +# CI-like conditions: 1 GPU, 25 cores, 4 xdist workers, 5400s per-test timeout. +cd /home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite || exit 1 +W=/tmp/tl_split +ART=$W/artifacts +CLIENT=/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client +export OMP_NUM_THREADS=25 OPENBLAS_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=0 PYTHONUNBUFFERED=1 +COMMON=(tensilelite/Tests/common -v --durations=0 --timeout=5400 -n 4 -m "common and not skip-gfx94X" + "--global-parameters=LibraryFormat='msgpack'" "--tensile-options=--cxx-compiler,/usr/bin/amdclang++") +phase() { # name, extra args... + local name=$1; shift + rm -rf /home/alvasile/.tensile /home/alvasile/.tensilelite + echo "START $name $(date -u +%FT%TZ)" >> $W/times.txt + local s=$(date +%s) + taskset -c 0-24 pytest "${COMMON[@]}" "$@" --junit-xml=$W/$name.xml > $W/$name.log 2>&1 + local rc=$? + echo "END $name rc=$rc seconds=$(( $(date +%s)-s )) $(date -u +%FT%TZ)" >> $W/times.txt +} +rm -rf $ART; mkdir -p $ART +# Emulate a CPU-only node: the 5 GPU-gated flags tests would skip there. +F=tensilelite/Tests/common/flags/gfx942 +phase build --build-only --gpu-targets gfx942 --artifact-dir $ART --prebuilt-client=$CLIENT \ + --deselect $F/test_build_only.py::test_use_cache \ + --deselect $F/test_use_cache.py::test_compile_and_benchmark \ + --deselect $F/test_use_cache.py::test_use_cache \ + --deselect $F/test_use_cache.py::test_use_cache_creates_cache_dir \ + --deselect $F/test_use_cache.py::test_use_cache_different_params +du -sh $ART >> $W/times.txt; ls $ART | wc -l >> $W/times.txt +phase run --use-cache --artifact-dir $ART --prebuilt-client=$CLIENT +phase combined --prebuilt-client=$CLIENT +echo ALLDONE >> $W/times.txt diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/FURTHER_SPEEDUPS.md b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/FURTHER_SPEEDUPS.md new file mode 100644 index 000000000000..76175a113b89 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/FURTHER_SPEEDUPS.md @@ -0,0 +1,483 @@ +# TensileLite common tests: further speed-up avenues (gfx942) + +Collected from the gfx942 timing work of 2026-11-03..05 on branch `users/alvasile/new_test_client` +(host `sharkmi300x-1`, MI300X, 25 pinned cores, `OMP_NUM_THREADS=25`). For each avenue: why that part +is slow today, the evidence, the options, an estimated saving in CI terms, and the risk. + +## How to read the numbers + +- **CI-like stages** are what matter: the whole build stage in one pytest session + (`--build-only --gpu-targets gfx942 -n 4 --dist=worksteal`, cache cleared at the start), then the + whole run stage (`--use-cache -n 4`). Current wall times at `9c5e825e`/`97c59828`: build ≈ 441 s, + run ≈ 742 s (run measured at `c1345541`; nothing since changes the run stage). +- **Build stage:** timers are summed over the 4 xdist workers (≈1,640 s of instrumented time). + With `worksteal` the workers are balanced, so a summed saving of X s is roughly X/4 s of wall. +- **Run stage:** every client run on a 1-GPU runner takes the same per-GPU lock + (`Tests/conftest.py:92-163`, `ClientWriter.py:314`), so clients run one at a time and client time + is 88-95% of the run wall. A second saved inside the client is about a second of run wall. +- Per-launch costs below come from the instrumented per-config run pass + (`full_runs_gfx942/post_run/AGGREGATE_analysis.txt`, 441 client launches for 123 configs); CI + makes the same 441 launches. +- "Estimated" means not measured end to end. + +## Already done (for context) + +| Change | Commit | Effect (CI-like) | +|---|---|---| +| Deselect configs that opt out of every target arch (563 of 709 on gfx942) | `e2bfff82` | build 612 → 564 s with `load` | +| `--dist=worksteal` for `hw-common` | `056fa353` | build 564 → 475 s (one run); run stage unchanged | +| Artifact gzip level 9 → 4 | `545d5d23` | packing 137.7 → 25.7 s summed (replay), +11.5% size | +| Invalid-kernel check in the parent | `97c59828` | validate 124 → 28 s summed, but the pool resize moved to assembly; wall unchanged (see B1) | +| Timers for artifact packing/extraction; analyzer fixes | `e04a42eb`, `094b04a4` | measurement only | + +--- + +## A. Run stage + +### A1. Hardware-monitor setup on every client launch + +- **Why it is slow:** `HardwareMonitorListener` (`client/src/HardwareMonitorListener.cpp:38-55`) + creates a `HardwareMonitor` whenever `--hardware-monitor` is true, which calls + `amdsmi_init(AMDSMI_INIT_AMD_GPUS)` (`client/src/HardwareMonitor.cpp:139-158`) and starts a + sampling thread. `HardwareMonitor` defaults to `True` (`Common/GlobalParameters.py:61`); only + gfx950 and gfx1250 force it off (`Tensile.py:837-839`). The bundled rocm_smi lists `/tmp` twice + per GPU to find partition files (16 full scans with 8 GPUs); this host's `/tmp` has ~37k entries. +- **Evidence:** `listener_setup` = 179.3 s over 441 launches (407 ms each), 22% of the 829 s + instrumented run total. Measured with the real client (3 runs each, cores 48-72, GPU 6): + + | Variant | `listener_setup` (ms) | `hip_initialization` (ms) | + |---|---:|---:| + | as in the pass (`ROCR_VISIBLE_DEVICES=6`, monitor on) | 404-411 | 378-409 | + | `hardware-monitor=False` | **0.02-0.03** | 386-430 | + | `HIP_VISIBLE_DEVICES=6` instead of ROCR (monitor on / off) | 406-407 / 0.02-0.03 | 384-391 | + | monitor on, private empty `/tmp` (mount namespace) | **102-103** | 380-407 | + +- **Why `amdsmi_init` costs ~400 ms here:** the bundled rocm_smi looks up + `rocmsmi_boot_{compute,memory}_partition_.*` by listing `/tmp`, twice per GPU (16 full + directory scans with 8 GPUs). One `ls -f /tmp` of the 37,120 entries costs 25-45 ms. The path is + hard-coded: `TMPDIR` does not change it. The remaining ~100 ms is amd-smi's sysfs walk of all 8 + GPUs (KFD `io_links`, PCI, DRM, hwmon), which does not honour `ROCR_VISIBLE_DEVICES`. So on a CI + runner with a small `/tmp` the cost is probably ~100 ms per launch (not measured on a runner). +- **Wasted:** the monitor only samples around timed enqueues (`preEnqueues`/`postEnqueues`), and only + 11 of 123 configs time anything (A2). The other ~410 launches initialize it for nothing. +- **Options:** + 1. Start the monitor only when something is timed: `m_active = hardware-monitor && + num-syncs-per-benchmark > 0` (or create `m_monitor` lazily in `preEnqueues`). + 2. Pass `HardwareMonitor: False` for test runs (as `Tensile.py` already does for gfx950). +- **Estimated saving:** ~165 s of run wall on this host; ~40 s on a runner with a small `/tmp`. +- **Risk:** low; option 1 keeps the clock/temperature reports for the 11 timing configs. + +### A2. Why 11 YAMLs use the hardware monitor at all + +- **Why:** these configs keep the default `SyncsPerBenchmark: 1` (the other ~110 set 0), so the + client runs the timed benchmark loop and the monitor samples it. Each is an explicit opt-out in + `Tests/unit/test_common_yaml_syncs_per_benchmark.py` with a reason: + + | Config | Stated reason for the benchmark loop | + |---|---| + | `benchmark_runs/fp32_tn` | benchmark timing coverage | + | `client/rotate_mode0`, `client/rotate_mode1` | RotatingMode test requires the benchmark loop | + | `gemm/icache_flush` | flush_icache only runs inside the sync loop | + | `gemm/fp8nfp16mix_hfp8ns`, `gemm/xfp32`, `groupedgemm/grouped_gemm`, `groupedgemm/grouped_gemm_ck_gfx942`, `sparse/gfx94x/libray_logic` | LibraryLogic requires benchmark data | + | `streamk/sk_hybrid`, `streamk/sk_queue_selfclean` | back-to-back launches validate output and synchronizer | + +- **Open question:** none of these reasons needs clock, temperature or fan readings; they need the + timed loop. If LibraryLogic and the CSV consumers do not read the monitor's columns + (`ClockRateSys`, `TempEdge`, ...), the monitor can be off for every test run, timed or not. To + check: which `ResultKey`s the LibraryLogic path reads. +- **Saving:** the remaining ~31 launches × 0.1-0.4 s once A1 is done (a few seconds). + +### A3. HIP initialization on every client launch + +- **Where:** `client/main.cpp:883-897`: `GetHardware` (`hipGetDeviceCount`, `hipSetDevice`, + `hip::GetCurrentDevice`, `main.cpp:449-462`) and one `DeviceContext` (stream creation). +- **Why it is slow:** a minimal probe (`hipGetDeviceCount` → `hipSetDevice` → `hipStreamCreate` → + `hipMalloc`) reproduces the 381 ms: `hipGetDeviceCount` 334-361 ms, `hipSetDevice` 0.0 ms, first + `hipStreamCreate` 44-45 ms, `hipMalloc` 0.1 ms. + - `hipGetDeviceCount` (= ROCr `hsa_init`): the thunk reads the topology of every KFD node before + `ROCR_VISIBLE_DEVICES` filtering. `strace` counts 16,266 `openat` and 26,481 `read`, including + 10,016 opens of `/sys/devices/virtual/kfd/kfd/topology/nodes//caches//properties` + (626 cache entries per MI300X node, 8 GPU nodes, each read twice) and ~5,000 per-CPU cache + files (192 CPUs). The ~350 ms between the first KFD ioctl and the first queue ioctl contains + no ioctls: pure sysfs parsing. Reading the 5,008 GPU cache files once with `cat` costs 188 ms, + so two passes explain ~340 ms. + - First stream: each hardware queue gets a 186,019,840-byte anonymous mapping registered with + `AMDKFD_IOC_SVM` (~15 ms each, probably the per-queue CWSR save area) before + `AMDKFD_IOC_CREATE_QUEUE`; three are made at start-up. + - No variable helps: `ROCR_VISIBLE_DEVICES` vs `HIP_VISIBLE_DEVICES` gave 378-430 vs 384-391 ms, + and no `HSA_*` / `HSAKMT_*` variable in `libhsa-runtime64.so.1.18.70204` skips the + cache-topology parsing (checked the full list of environment strings). + - Concurrency: 4 simultaneous probes took 472-491 ms (`amdsmi_init`) and 439-453 ms + (`hipGetDeviceCount`) instead of 397 / 334 ms, so start-up mostly overlaps across processes + (relevant to A6b). +- **Evidence:** `hip_initialization` 167.9 s / 441 launches (381 ms each). gfx950's ~150 ms is + probably a smaller topology (inferred). +- **Options:** fewer launches (A4); `GPU_MAX_HW_QUEUES=1` saves ~15 ms setup + ~15 ms exit per launch. +- **Saving:** `GPU_MAX_HW_QUEUES=1` ≈ 13 s; the rest only via A4. **Risk:** low-medium. + +### A4. Too many client launches (441 for 123 configs) + +- **Why:** `BenchmarkProblems._benchmarkProblemType` runs one client process per benchmark step + (`BenchmarkProblems.py:1175-1176` → `runClient`, `:1116`). A config with several problem types or + problem-size groups launches the client several times, each paying start-up, HIP init, monitor, + library/code-object loading and exit. +- **Evidence:** fixed per-launch costs ≈ 468 s (Client launch group: `listener_setup` 179, + `hip_initialization` 168, `process_exit` 55, `library_loading` 17, `data_init_setup` 15, + `code_object_loading` 13, `main_teardown_locals` 11, `process_startup` 9) = 56% of the run total. + `DEFERRED.md` #5 also records a first-call cost per process in `cpu_data_init`. +- **Can the client be reused today? No.** One client process serves exactly one job: one solution + library (`--library-file`, a single value), its code objects and one problem set. `main()` reads + its arguments once and builds the problem factory, data initialization, listeners and reporters + for that job (`client/main.cpp:812`, `:864`, `:967`, `:997`). Several `--config-file` arguments are + merged into one option set, not run as separate jobs (`main.cpp:643-653`), and Tensile's run script + starts the client once per config file (`ClientWriter.py:441-442`). +- **What reuse would take:** + 1. A batch mode in the client: a list of jobs per process, with `main()`'s body moved into a + per-job function and per-job state reset between jobs (code-object modules, caches, listeners, + results file). + 2. Tensile launching once per config with all of its steps. In the run stage (`--use-cache`) every + step's library already exists, so this works if a config's steps do not depend on earlier + steps' results (believed true in the test flow; not verified). + 3. Or, larger: a persistent client per xdist worker receiving jobs over a pipe. +- **What reuse saves, and what it does not** (per launch / summed over 441 launches): + + | Cost | Per launch | Summed | Removed by reuse? | + |---|---:|---:|---| + | `hip_initialization` | 381 ms | 168 s | yes | + | `process_exit` | 124 ms | 55 s | yes | + | `process_startup` | 19 ms | 9 s | yes | + | `main_teardown_locals` | 26 ms | 11 s | partly | + | `listener_setup` (hardware monitor) | 407 ms | 179 s | yes, but A1 removes it more simply | + | `library_loading` | 39 ms | 17 s | no (per job) | + | `code_object_loading` | 30 ms | 13 s | no (per job) | + | `data_init_setup` | 34 ms | 15 s | no (per job) | + + After A1, reuse removes ~0.5 s per launch, ~230 s summed, mostly HIP initialization. +- **What reuse costs:** isolation. A GPU memory fault or hung kernel in one step would take down + every later step in that process (today each step is its own process), and state left over by one + job (buffers, cached pointer arrays, loaded modules) can affect the next. +- **Interaction with the GPU lock (A6b):** both target the same fixed per-launch cost. + Shrinking the lock keeps the start-up work but lets the 4 workers overlap it with another + client's GPU phase (saves wall, not CPU) and can also overlap the CPU reference and comparison. + Reuse removes the start-up work (saves CPU and wall) but does nothing for the CPU phases inside + the lock. Reuse also needs the narrower lock: a client living across several steps cannot be + wrapped by today's process-level lock without holding the GPU for the whole config, so the lock + has to move into the client first. Once the lock is narrowed, the start-up cost is already + overlapped, and reuse's remaining CI gain is at most ~230 s / 4 ≈ 60 s of run wall; it matters + more for serial (per-config) and multi-GPU runs. +- **Estimated saving (CI):** ~210 s of run wall if done instead of A6b (after A1); ~60 s at most + if done after A6b. **Risk:** medium-high. +- **Recommendation:** do A1, then A6b option 1 (lock inside the client, after start-up), measure; + consider reuse only if the remaining HIP initialization cost still matters. + +### A5. Client process exit + +- **Why:** `process_exit` 124 ms per launch: mostly unmapping the five 186 MB per-queue regions + (~15 ms each) plus kernel-side teardown. An early `_exit` saves only ~14 ms. +- **Saving:** ~6 s via fast exit, ~8 s via `GPU_MAX_HW_QUEUES=1`. Mostly folded into A3/A4. + +### A6. Clients run one at a time under `-n 4` + +- **Why:** with one visible GPU every xdist worker gets device id "0" and the same lock file; the + lock is held around the whole client run. Run wall ≈ sum of client times; the GPU is busy only a + fraction of the time (≈13% estimated on gfx950). +- **Evidence:** `worksteal` balanced the run-stage workers but the wall stayed 747/736 s vs + 751/746 s; summed test time rose (lock waiting). +- **Options:** a shared lock for configs that tolerate a shared GPU, exclusive for Stream-K etc.; + split `OMP_NUM_THREADS` among concurrent clients. +- **Estimated saving:** 30-50% of the run wall (not measured). **Risk:** high: a Stream-K config + (`sk_sgemm_hybrid_pap`) failed validation once while sharing a GPU on gfx950. + +### A6b. Shrink the region the GPU lock covers (investigate) + +- **What the lock covers today:** `runClient` takes `ClientExecutionLock` around the whole client + subprocess (`ClientWriter.py:314-322`: `Popen` → `communicate()`). The lock file is per physical + GPU (`Tests/conftest.py:92-163`, `worker_lock_path`), passed to Tensile as `--client-lock` + (`Tensile.py:261, :295`). So everything the client does is serialized, not only the GPU work. +- **What inside it actually needs the GPU to itself** (per-config run totals, + `post_run/AGGREGATE_analysis.txt`, summed over 441 launches, instrumented): + + | Part of the client run | Time | Needs exclusive GPU? | + |---|---:|---| + | Process start, HIP init, hardware monitor, library/code-object load, teardown, exit (A1-A5) | ~468 s | no (HIP init touches the device but does not compete with another client's kernels) | + | CPU reference GEMM (`solve_cpu_fast`) | ~38 s | no (CPU) | + | Result comparison on the CPU (`validate_element_comparison`) | ~32 s | no (CPU) | + | Input preparation (host init, uploads/resets) | ~54 s | partly (copies) | + | GPU readback, Synchronizer reduce | ~30 s | partly | + | Kernel launches and execution (incl. 51 s of instrumentation-only `warmup_first_sync`) | ~68 s | yes | + + On this split most of the ~720 s of client time does not need the GPU exclusively, yet with one + GPU it all runs one client at a time (A6). +- **Why the lock exists (to confirm):** keep benchmark timings free of interference, avoid device + memory pressure from 4 clients, and avoid the shared-GPU failure seen with a Stream-K config + (`sk_sgemm_hybrid_pap`, gfx950). A narrower lock that still covers all kernel execution keeps the + last two properties; timed configs (A2) could keep holding it for their timed loop. +- **Options, smallest first:** + 1. Take the lock inside the client instead of around the process: pass the lock path to the + client (`--gpu-lock`), acquire it (`flock`) after start-up/HIP init/library load and release it + before teardown and exit. This alone moves the ~1 s per launch of start-up and exit (A1, A3, + A5) out of the serialized region. + 2. Hold it only around the solution loop's GPU phases (uploads, launches, readback), not around + CPU reference and CPU comparison. + 3. Per-problem: compute the CPU reference before taking the lock. +- **Constraints:** 4 clients each run `OMP_NUM_THREADS=25` CPU work on the same 25 cores, so + overlapping CPU phases will contend; split the thread count among clients or keep the CPU-heavy + phases short. Any client crash must release the lock (`flock` is released on process exit). +- **Device memory:** with the lock taken after start-up, up to 4 clients hold their allocated + buffers on the GPU at once. Fine for MI300X; check configs with large buffers. +- **Relation to client reuse (A4):** the narrower lock is the smaller first step and a prerequisite + for reuse (see A4). +- **Estimated saving:** not measured. Option 1 alone removes ~1 s per launch from the serialized + region (~440 s of the ~720 s), which could cut the run wall substantially once overlapped across 4 + workers; the realistic gain depends on CPU contention. **Risk:** medium (client and harness + change; timing-sensitive configs must keep exclusivity). **Before building:** confirm why the + lock was added (benchmark timing interference, device memory, Stream-K sharing) and whether a + config's benchmark steps are independent in the run stage; then prototype option 1 on one + multi-step config (e.g. `gemm/fp16_tn`) with `-n 4` and compare run-stage wall. + +### A7. Result validation + +- **Why:** `validate_element_comparison` (32.4 s) compares on one thread; `validate_gpu_readback` + (24.9 s, 11-17 µs per call, dominated by call count in `fp16_tn`, `bufferLoadStore0`); the pinned + host buffer is reallocated whenever the size changes rather than when it grows + (`ReferenceValidator.cpp:729`, 3.3 s). +- **Options:** multi-threaded comparison; grow-only buffer; later a GPU-side compare. +- **Estimated saving:** 15-20 s (CPU changes), 40-50 s with a GPU compare. **Risk:** low / medium + (must keep tolerance, NaN and bounds-check semantics). + +### A8. Smaller run-side items + +- `gpu_input_reset_batched_inputs` 19.7 s: the GuardPageAll pointer arrays are re-uploaded; noted + in `after_run.md`, not filed anywhere. +- CPU reference `solve_cpu_fast` 37.4 s on gfx942 (the large FP8 MX cases are gfx950-only). +- Artifact extraction: < 1 s per config, outside the lock. Not worth pursuing. +- `warmup_first_sync` exists only under instrumentation; not a CI cost. + +--- + +## B. Build stage + +CI-like instrumented build stage, summed over the 4 workers (mean of 2 runs at `97c59828`): +`python_kernel_codegen` 520.5 s, `python_kernel_write_assemble` 388.9 s, +`python_solgen_forked_solutions` 316.6 s, `python_kernel_bench_postprocess` 132.5 s, +`python_kernel_validate` 27.7 s, `python_artifact_compress` 28.3 s; TOTAL 1,637 s. + +### B1. Process-pool resizes and restarts + +- **Why it is slow:** `ParallelMap2` (`Common/Parallel.py`, ~line 272) asks joblib for + `min(budget, number of tasks)` workers, so every small step asks for a different pool size. + joblib's loky backend passes thread-limit variables computed as `cpu_count() // n_jobs` for any + of them not set in the parent (`MKL_NUM_THREADS`, `NUMEXPR_NUM_THREADS`, ...; + `joblib/_parallel_backends.py:221-245`), so a different size changes the executor's arguments and + loky shuts the whole pool down and starts a new one (`loky/reusable_executor.py:189-215`). A plain + shrink also waits for the extra workers to exit (`_resize`). +- **Evidence:** on `gemm/gsuasb.yaml`, each size change cost ~0.5 s (515 / 500 / 580 / 472 ms) vs + 16 ms when the size did not change. The per-config pass attributed ~190 s summed to restarts, + shrinks and grows, plus ~136 s to cold starts. Moving the invalid-kernel check out of the pool + only moved the resize to assembly (`97c59828`). +- **What shrinking buys:** nothing measurable: idle workers use no CPU, the threads-per-worker + change is irrelevant (workers run Python and launch clang; `OMP_NUM_THREADS` is fixed), and the + workers being shut down already exist. +- **Options:** keep the worker count constant within a build when `procs` is given (no task-count + cap); keep the thread-limit variables constant (`parallel_config(inner_max_num_threads=...)` or set + them once in the parent); keep running one-task calls in the parent. In CI the pool lives in each + xdist worker process and would also stay warm across configs. +- **Estimated saving:** 100-200 s summed → ~25-50 s of build wall. **Risk:** low-medium + (`ParallelMap2` is shared with TensileCreateLibrary; can be limited to calls with explicit `procs`). + +### B2. Candidate solutions that are built and then thrown away + +- **Why it is slow:** forking expands every parameter combination, and each candidate is fully + constructed before validity is known: `deepcopy(problemType.state)` (`BenchmarkProblems.py:286`), + `Solution(...)` (`Solution.py:910-1047`), then rejection inside + `Solution.assignDerivedParameters`. Each pool task also pickles `globalParameters`, + `isaInfoMap`, `rocIsaData` and the assembler (`Parallel.py:291`, `BenchmarkProblems.py:342-353`). +- **Evidence:** across the build, 843,891 candidates → 105,160 valid (87.5% rejected). `dtv`: + 534,736 → 18,620 (96.5% rejected), 28.4 s in the per-config pass; `dtl`: 16,688 → 2,078 (87.5%); + `cgemm` 55,296 → 20,520; `bufferLoadStore0` 70,656 → 12,106. Real work ≈ 0.8 ms CPU per candidate + (~675 CPU-s, ~590 of it on rejected ones); the rest of `python_solgen_forked_solutions` is pool + overhead (B1). +- **Options:** reject before building the full `Solution` (cheap pre-checks on the fork parameters + for the rules that reject most); ship constants once per worker instead of per task; batch forks + across steps. +- **Estimated saving:** 20-40 s of per-config time (mostly `dtv`, `cgemm`, `bufferLoadStore0`, + `zgemm`) → ~10-20 s of build wall. **Risk:** medium: the early check must reject exactly what + `Solution` rejects. **First step:** profile `dtv`'s generation to find which rules reject most. + +### B3. Kernels generated and then rejected (codegen) + +- **Why:** some valid solutions overflow resources only during code generation. +- **Evidence:** 105,160 valid solutions → 97,220 kernels kept (7,940 rejected by KernelWriter). + The branch already stops codegen early once the static VGPR allocation exceeds `MaxVgpr` or + `overflowedResources` is set; `DEFERRED.md` #10 records the rejected kernels' cost falling from + 374.5 to ~10.8 CPU-s on the probed configs (gfx950). +- **Option:** reject over-budget kernels in solution validation, before codegen (#10). +- **Saving:** small (~10 CPU-s). **Status:** deferred. + +### B4. Kernel-name and compile-key work repeated per solution + +- **Why:** `getKernelCompileKey` → `getKernelNameMin` is computed per solution and per kernel three + times: in `removeInvalidSolutionsAndKernels`, in `validKeys` (`Run.py:1253-1257`) and in + `passPostKernelInfoToSolution` (`Run.py:727+`); `python_kernel_bench_postprocess` (132.5 s) also + spends time in naming and library construction (1.26 ms per solution, 74% in six configs). +- **Option:** compute each name/key once and cache it on the solution. +- **Estimated saving:** 30-60 s summed (~29 s of it in the validate path). **Risk:** low. + +### B5. Assembly path + +- **Why:** codegen results go back to the parent and then into a second pool per step; each `.s` is + written, then re-read and regex-scanned (`_retargetAssemblySource`, `Toolchain/Component.py:301`), + and assembled with one clang process per kernel (~56 ms per kernel). +- **Option:** assemble inside the codegen workers and drop the re-read. +- **Estimated saving:** 40-70 s summed. **Risk:** low-medium. + +### B6. Code generation itself + +- **Why:** ~50 ms of worker CPU per kernel (`dtv` 12,406 kernels; large tn configs ~200 ms per + kernel). +- **Options:** `DEFERRED.md` #8 (longest-first dispatch within a codegen step) and #9 (cheaper + generator: `hasAnyDependency`, epilogue). **Risk:** medium-high (changes generated code). + +### B7. Scheduling tail and core sharing + +- **Why:** `dtv` alone takes ~230-270 s of a ~441-475 s build, and with collection order it starts + mid-stage. Each xdist worker's pool is sized for 4 busy workers (2 × 25 / 4 = 12 processes), so + when workers go idle at the end, the last one cannot use the freed cores; when all 4 are busy + there are up to 48 processes on 25 cores (test time rose 8% under `worksteal`). +- **Options:** start the longest configs first (cost-ordered collection; pairs with + `load --maxschedchunk=1/2`, not with `worksteal`); split `dtv`; a shared core budget across xdist + workers (jobserver-style token pipe) so the last worker gets all cores. +- **Estimated saving:** up to ~30-40 s (the stage's lower bound is ~435 s of work / 4 workers). + **Risk:** medium for the shared budget (tokens must be released if a worker crashes). + +### B8. Artifact packing in the combined `hw-common` job + +- **Why:** `hw-common` runs the combined mode (`Tests/common/test_config.py`): each config's build + output is packed into a `.tar.gz` in the test's tmpdir and unpacked immediately. Nothing is + transferred, so packing and unpacking are pure overhead there. +- **Evidence:** level 4 packing ≈ 25.7 s summed for the gfx942 set (replay); level 0 5.4 s. +- **Options:** skip the archive in combined mode (pass the output directory to the run step), or + pack at level 0/1 there. Keep level 4 for a split CI that uploads artifacts. +- **Estimated saving:** ~20 s summed (≈5 s of wall) plus extraction. **Risk:** low, but the + round trip currently also checks the artifact whitelist; keep that check if it matters. + +### B9. CMS configs fork `UseCustomMainLoopSchedule: [0, 1]` + +- **Where:** `gemm/gfx950/custom_mainloop_scheduling.yaml` (14 forks with `[0, 1]`), + `gemm/gfx950/agntab_coverage_gfx950.yaml` (1), `gemm/gfx12/custom_schedule.yaml` (1). All of them + skip gfx942, so this costs time on gfx950 and gfx12, not in the gfx942 numbers above. +- **Why it is slow:** the fork builds every tile configuration twice: with the custom main-loop + schedule (1) and with the regular scheduler (0). The default `-1` already resolves to 1 where CMS + is supported and 0 otherwise (`SolutionStructs/Solution.py:3950-3963`). The `0` kernels exercise the + regular scheduler, which the general configs already cover, so in a CMS test they mostly double the + codegen, assembly and benchmark work of those steps. +- **Evidence (gfx950):** `custom_mainloop_scheduling` has 66 benchmark steps (so 66 client launches, + see A4) and 79 kernels; it was the second-slowest test of the gfx950 split run phase (159 s under + `-n 4`) and 30 s of per-config build after the build changes (`after_run.md`, `after_build.md`, + `python_kernel_codegen_investigation.md`). How much of that is the `0` half was not measured. +- **Option:** fork `[1]` only (or `[-1]`) in the CMS configs, unless a step specifically compares + the two; keep one explicit `0` case if the "CMS off" path for CMS-capable tiles needs coverage. +- **Estimated saving:** up to half of the kernels in the 14 forked steps. **Risk:** low (test + coverage decision for the CMS owners). + +--- + +## C. Per-process start-up (Tensile and pytest) + +CI pays these once per xdist worker per stage (~4 s of wall per stage), not per config; the +per-config timing harness pays them per config (~4 s × 123). + +### C1. Collection loads every YAML + +- **Why:** `configMarks` parses all 744 test YAMLs to compute marks even when one test is selected + (`Tests/conftest.py:82` → `Tests/common/config_helpers.py:125, :155`), ~2.25 s per pytest process + (already `CSafeLoader`). The new arch deselection reuses these marks and adds nothing. +- **Option:** cache the marks per YAML keyed on mtime/size. **Saving:** ~2 s per stage in CI. + +### C2. ISA capability probing + +- **Why:** `makeIsaInfoMap` (`Tensile.py:841` → `Capabilities.py:166`) runs 114 `amdclang++` + probes serially (~12 ms each, `hardware_caps.hpp:33-75`, `:97+`) for the one ISA: 1.4-1.9 s per + process. +- **Options:** cache on disk keyed on assembler identity, ISA and probe-table version; or run the + probes in parallel (measured 1.3 → 0.35 s); or probe with `llvm-mc` (0.42 s serial). +- **Saving:** ~1-1.5 s per stage in CI. **Risk:** low-medium (stale cache if the compiler changes). + +### C3. `rocm_agent_enumerator` on every rocisa import + +- **Why:** with `ROCISA_BACKEND` unset, `import rocisa` runs `rocm_agent_enumerator` (45 ms, + `rocisa/__init__.py:124-171`) in every process, including every new loky worker (32 times for + `gemm/ag` in one build). `--gpu-targets` does not skip it. +- **Option:** pass the parent's backend choice to workers. **Saving:** estimated 10-15 s summed in + CI (more while B1 restarts pools). **Risk:** low. + +### C4. Process exit + +- **Why:** shutting the reusable loky executor down at exit takes 0.2-0.7 s; interpreter teardown + up to 0.45 s (~816k live objects after `Tensile()` returns). Once per worker in CI. +- **Option:** `os._exit` after flushing. **Saving:** ~0 in CI. **Risk:** medium. + +--- + +## D. Measurement environment and CI configuration (not product code) + +- **Editable rocisa rebuilds on every import.** The venv's scikit-build-core editable install runs + `cmake --build/--install` under a machine-wide lock on each `import rocisa` (1-9 s; 0.11 s with + `SKBUILD_EDITABLE_SKIP=`). It inflated a whole first measurement 2-5× and coupled the + two measurement sides. The measurement scripts should set `SKBUILD_EDITABLE_SKIP`. CI installs a + wheel and is unaffected. +- **Shared-host `/tmp`** with ~37k entries adds ~300 ms to every `amdsmi_init` (A1). +- **CI marker `not skip-gfx94X` matches nothing** (TheRock's `fetch_test_configurations.py`): configs + carry exact-arch marks. The in-repo deselection (`e2bfff82`) makes it moot for speed; the dead + expression could still be cleaned up in TheRock. +- **`comm/gfx950/fused_a2a.yaml` fails the gfx942 build-only stage** ("0 valid solutions"): a + gfx950-only config without a `skip-gfx942` mark (`CI_SPLIT_TIMING.md` finding 3). Correctness, not + speed. +- **Per-config timing harness overhead:** the serial scripts start one pytest per config (~5 s each + for collection, probing, start-up); this is harness cost, not CI cost. + +--- + +## E. Test-suite hygiene (not speed) + +### E1. Unit tests insert into `sys.path` at import time + +- **What:** 24 files under `Tests/` change `sys.path`, most at module import (collection) time, e.g. + `Tests/unit/test_subtile_gfx1250_codegen.py:19-21` puts the TensileLite root at index 0 so it can + `import gpu_test_helpers`; `gpu_test_helpers.py:39`, `test_mxs_tilespan.py`, + `test_emitMfmaInstruction.py`, `test_storeD_roundtrip.py`, ... do the same; several insert the + characterization `_codegen` dir; `test_analyze_timing.py` and `test_aggregate_timing.py` insert + `scripts/`, and `test_artifact_timing.py` (added in this work, `e04a42eb`) inserts `Tests/common`. + `Tests/conftest.py:39` already appends the same root at the end of the path. +- **Why it matters:** the change is process-wide and permanent, so every later test in the same + pytest/xdist worker process sees it, and what a test imports depends on which other modules were + collected first. Inserting at index 0 makes the source tree shadow an installed package (CI runs + against the installed `share/hipblaslt/tensilelite` tree), and lets bare module names + (`gpu_test_helpers`, `artifact_helpers`, `analyze_timing`) resolve only by accident of order. The + order-dependence bug fixed in `9c5e825e` is the same class of problem (import-time side effects). +- **Options:** declare the import roots once (`pythonpath =` in `pytest.ini`, or the root + `conftest.py`), import helpers as package modules, and load scripts with `importlib` from an + explicit path in the tests that need them; then remove the per-file insertions, starting with the + ones that insert at index 0. +- **Speed:** negligible (a longer `sys.path` only adds lookups). **Risk:** low, but touches many + files; run the unit suite in random order / per file to confirm nothing relied on the side effect. + +--- + +## Suggested order + +| # | Avenue | Stage | Estimated CI wall saving | Risk | +|---|---|---|---:|---| +| 1 | A1 hardware monitor only when timed | run | ~40 s (CI runner) / ~165 s (this host) | low | +| 2 | B1 constant pool size and thread-limit env | build | ~25-50 s | low-medium | +| 3 | A6b lock inside the client (after start-up), then narrow to GPU phases | run | potentially large; not measured | medium | +| 4 | B2 early candidate rejection (profile `dtv` first) | build | ~10-20 s | medium | +| 5 | B4 / B5 name caching, assemble in codegen workers | build | ~20-30 s | low-medium | +| 6 | A7 validation | run | ~15-50 s | low-medium | +| 7 | B7 longest-first, shared core budget | build | ~30-40 s | medium | +| 8 | B8 skip packing in combined mode | build | ~5 s | low | +| 9 | A6 concurrent clients | run | 30-50% of run (unmeasured) | high | +| 9b | A4 client reuse (batch mode), only if HIP init still matters after A6b | run | ≤ ~60 s after A6b (~210 s without it) | medium-high | +| 10 | C1-C3 start-up caching | both | a few s per stage | low | +| 11 | B9 CMS `[0, 1]` forks (gfx950/gfx12) | build + run | not measured; up to half of those steps | low | +| — | E1 `sys.path` insertions | hygiene | none | low | diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/README.md b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/README.md new file mode 100644 index 000000000000..6606388d8840 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/README.md @@ -0,0 +1,54 @@ +# gfx942 before/after timing of the TensileLite common tests + +| Item | Before | After | +|---|---|---| +| Commit | `aa19af72` (all timing instrumentation, before the first optimization `8ec52f37`) | `c1345541` (branch head + gfx942 script parameters) | +| Client | `tensilelite-client`, Release, `GPU_TARGETS=gfx942`, msgpack only, `/usr/bin/amdclang++` (ROCm 7.2.4) | same, built from its commit | +| Cores / GPU | `taskset -c 0-24`, `ROCR_VISIBLE_DEVICES=2` (NUMA 0) | `taskset -c 48-72`, `ROCR_VISIBLE_DEVICES=6` (NUMA 1) | + +Host `sharkmi300x-1`: 2x EPYC 9454, 8x MI300X, shared (40+ users, load average 20-50, of which ~50 is +these two passes). `OMP_NUM_THREADS=OPENBLAS_NUM_THREADS=25`, `--timeout=5400`, +`-m "common and not skip-gfx94X"`, `LibraryFormat='msgpack'`. The two sides ran concurrently on +different sockets and GPUs. Both analysed and aggregated with the head's `scripts/analyze_timing.py` / +`aggregate_timing.py`. Scripts: `../full_runs_gfx950/{split,per_config}/` (gfx942 parameters from +`c1345541`). + +`SKBUILD_EDITABLE_SKIP` was set to the rocisa build dir: the venv's editable rocisa otherwise re-runs +`cmake --build/--install` under a machine-wide lock on every import (1-9 s per pytest process and per +loky worker). CI installs a wheel. A first attempt without it was discarded. + +## Folders + +Committed: the four `AGGREGATE_analysis.txt`, the comparison tables, the split `times.txt` and the +config list. Not committed (too large, or local working notes): the raw per-config logs (`*.log.gz`, +~430 MB), the per-config `*_analysis.txt` and `times.txt`, the split logs/XML, the driver and the +hotspot analyses. They are kept on the measurement host next to this file. + +- `split_pre/`, `split_post/`: CI-like split (uninstrumented, `-n 4`): `build` (`--build-only + --gpu-targets gfx942`, caches cleared) then `run` (`--use-cache`, caches cleared). Logs, JUnit XML, + `times.txt`, the passed run configs (`split_run_passed.txt`) and the shared list (`configs.txt`, + 123 configs, identical on both sides). +- `pre_build/`, `post_build/`: every config built serially, `TimingInstrumentation=True`; caches cleared + once before the first config, artifacts kept. Per config: raw `*_build.log.gz`, `*_build_analysis.txt`, + `times.txt` (pytest wall). `AGGREGATE_analysis.txt` over all 123. +- `pre_run/`, `post_run/`: every config run serially (`--use-cache`) on that build's artifacts and with + its cache (not cleared). Same files. +- `build_comparison.txt`, `run_comparison.txt`: purpose tables side by side (`cmp_agg.py`). +- `opportunity_inventory.md`, `build_hotspots.md`, `run_hotspots.md`: remaining-opportunity analyses. + +All 123 configs returned 0 in all four per-config passes. Split: build 145 passed / 1 failed +(`comm/gfx950/fused_a2a.yaml`, known, `CI_SPLIT_TIMING.md` finding 3) on both sides; run 0 failed. + +## Results + +| Measurement | Before | After | Change | +|---|---:|---:|---:| +| CI-like build phase wall (`-n 4`) | 1,028 s | 634 s | -38% | +| CI-like run phase wall (`-n 4`) | 1,494 s | 742 s | -50% | +| Per-config build TOTAL (instrumented) | 2,134 s | 1,152 s | -46% | +| Per-config build pytest wall sum | 2,898 s | 1,886 s | -35% | +| Per-config run TOTAL (instrumented) | 1,644 s | 829 s | -50% | +| Per-config run pytest wall sum | 2,320 s | 1,484 s | -36% | + +No config got slower by more than 2 s in either per-config pass. Single samples (no repeats); the +sides ran on different sockets/GPUs. diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/build_comparison.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/build_comparison.txt new file mode 100644 index 000000000000..b7cc05a62665 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/build_comparison.txt @@ -0,0 +1,43 @@ +Category pre s post s delta +TOTAL RUNTIME 2134.4 1152.1 -46.0% +BUILD: solution generation & compile 2086.5 1104.8 -47.1% + Solution generation & build (Python) 2086.5 1104.8 -47.1% + python_kernel_compilation 1667.0 807.2 -51.6% + python_kernel_codegen 455.1 268.2 -41.1% + python_kernel_write_assemble 158.1 173.4 +9.7% + python_kernel_bench_postprocess 123.7 122.7 -0.8% + python_benchpost_lib_construction 83.0 77.8 -6.3% + python_benchpost_library_write 28.5 32.6 +14.5% + python_benchpost_naming 12.1 12.2 +0.6% + python_kernel_bench_setup 123.6 4.4 -96.4% + python_kernel_validate 73.2 125.9 +71.9% + python_kernel_setup 31.8 31.5 -0.8% + python_kernel_build_co 20.3 20.7 +1.6% + python_kernel_write_helpers 8.0 6.8 -15.4% + (untracked) 673.1 5.8 -99.1% + python_solution_generation 419.6 297.6 -29.1% + python_solgen_forked_solutions 418.5 296.4 -29.2% + python_solgen_custom_kernels 1.1 1.2 +7.4% +SHARED: cache & solution files 33.0 32.6 -1.1% + python_write_solutions 20.8 20.5 -1.2% + python_wsol_dump 20.1 20.0 -0.8% + python_solution_indexing 11.7 11.5 -1.5% +UNATTRIBUTED (untracked time & timer overhead) 14.9 14.7 -1.2% + (untracked in python_benchmark_problems) 14.9 14.7 -1.2% + python_kernel_build_src_co_wait 0.0 47.7 new + +pytest wall sum: pre 2898 s, post 1886 s (-34.9%) +largest per-config (pre -> post s): + common__gemm__dtv 251 -> 160 + common__gemm__cgemm 107 -> 96 + common__gemm__bufferLoadStore0 99 -> 81 + common__gemm__mix_cvt_after_ds_fnuz 77 -> 60 + common__gemm__zgemm 76 -> 70 + common__gemm__fp16_tn 74 -> 57 + common__gradient__fp16_gradient_bias 68 -> 18 + common__gemm__gemm_bias_strided_batched_broadcast_gfx940 61 -> 13 + common__gemm__lsu_fnuz 52 -> 35 + common__sparse__gfx94x__f8n_gate_r 49 -> 22 + common__gemm__fp16_HH_BHS_bf16mfma 49 -> 29 + common__gemm__bf16_tn 46 -> 35 +slower by >2 s: [] diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/post_build/AGGREGATE_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/post_build/AGGREGATE_analysis.txt new file mode 100644 index 000000000000..d67c52a7f68f --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/post_build/AGGREGATE_analysis.txt @@ -0,0 +1,172 @@ +======================================================================================================================== +TIMING ANALYSIS SUMMARY +======================================================================================================================== + +TIMING BY PURPOSE +------------------------------------------------------------------------------------------------------------------------ + Category Count Total (ms) Mean (ms) % of Parent % of Wall +------------------------------------------------------------------------------------------------------------------------ + TOTAL RUNTIME 1,152,118.78 100.0% + BUILD: solution generation & compile 1,104,771.86 95.9% 95.9% + Solution generation & build (Python) 1,104,771.86 100.0% 95.9% + python_kernel_compilation 1,005 807,165.44 803.15 73.1% 70.1% + python_kernel_codegen 123 268,182.72 2,180.35 33.2% 23.3% + python_kernel_write_assemble 441 173,421.33 393.25 21.5% 15.1% + python_kernel_validate 441 125,911.83 285.51 15.6% 10.9% + python_kernel_bench_postprocess 441 122,727.30 278.29 15.2% 10.7% + python_benchpost_lib_construction 441 77,770.74 176.35 63.4% 6.8% + python_benchpost_library_write 441 32,602.71 73.93 26.6% 2.8% + python_benchpost_naming 441 12,213.89 27.70 10.0% 1.1% + python_benchpost_client_config 441 112.12 0.25 0.1% 0.0% + (untracked) 27.83 0.0% 0.0% + python_kernel_build_src_co_wait 441 47,743.03 108.26 5.9% 4.1% + python_kernel_setup 441 31,525.89 71.49 3.9% 2.7% + python_kernel_build_co 441 20,667.26 46.86 2.6% 1.8% + python_kernel_write_helpers 441 6,792.76 15.40 0.8% 0.6% + python_kernel_bench_setup 441 4,417.09 10.02 0.5% 0.4% + (untracked) 5,776.25 0.7% 0.5% + python_solution_generation 441 297,606.41 674.84 26.9% 25.8% + python_solgen_forked_solutions 441 296,425.93 672.17 99.6% 25.7% + python_solgen_custom_kernels 441 1,168.99 2.65 0.4% 0.1% + python_solgen_fork_permutations 441 3.18 0.01 0.0% 0.0% + (untracked) 8.32 0.0% 0.0% + SHARED: cache & solution files 32,643.25 2.8% 2.8% + python_write_solutions 441 20,547.22 46.59 62.9% 1.8% + python_wsol_dump 441 19,990.69 45.33 97.3% 1.7% + python_wsol_prepare 441 453.03 1.03 2.2% 0.0% + python_wsol_prepare_nocache 441 451.75 1.02 99.7% 0.0% + (untracked) 1.28 0.3% 0.0% + python_wsol_header 441 9.24 0.02 0.0% 0.0% + (untracked) 94.25 0.5% 0.0% + python_solution_indexing 441 11,484.12 26.04 35.2% 1.0% + python_write_cache 441 611.27 1.39 1.9% 0.1% + python_cache_check 441 0.64 0.00 0.0% 0.0% + UNATTRIBUTED (untracked time & timer overhead) 14,703.68 1.3% 1.3% + (untracked in python_benchmark_problems) 14,703.68 100.0% 1.3% + +TIMING BY TIMER NESTING (legacy view) +------------------------------------------------------------------------------------------------------------------------ + Category Count Total (ms) Mean (ms) % of Parent % of Wall +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 123 1,152,118.78 9,366.82 100.0% + -------------------------------------------------------------------------------------------------------------------- + python_kernel_compilation 1,005 807,165.44 803.15 70.1% 70.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_codegen 123 268,182.72 2,180.35 33.2% 23.3% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_assemble 441 173,421.33 393.25 21.5% 15.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_validate 441 125,911.83 285.51 15.6% 10.9% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_postprocess 441 122,727.30 278.29 15.2% 10.7% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_benchpost_lib_construction 441 77,770.74 176.35 63.4% 6.8% + python_benchpost_library_write 441 32,602.71 73.93 26.6% 2.8% + python_benchpost_naming 441 12,213.89 27.70 10.0% 1.1% + python_benchpost_client_config 441 112.12 0.25 0.1% 0.0% + (untracked) 27.83 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_src_co_wait 441 47,743.03 108.26 5.9% 4.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_setup 441 31,525.89 71.49 3.9% 2.7% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_co 441 20,667.26 46.86 2.6% 1.8% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_helpers 441 6,792.76 15.40 0.8% 0.6% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 5,776.25 0.7% 0.5% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_setup 441 4,417.09 10.02 0.5% 0.4% + -------------------------------------------------------------------------------------------------------------------- + python_solution_generation 441 297,606.41 674.84 25.8% 25.8% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_forked_solutions 441 296,425.93 672.17 99.6% 25.7% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_custom_kernels 441 1,168.99 2.65 0.4% 0.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 8.32 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_fork_permutations 441 3.18 0.01 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_write_solutions 441 20,547.22 46.59 1.8% 1.8% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_dump 441 19,990.69 45.33 97.3% 1.7% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_prepare 441 453.03 1.03 2.2% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_wsol_prepare_nocache 441 451.75 1.02 99.7% 0.0% + (untracked) 1.28 0.3% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 94.25 0.5% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_header 441 9.24 0.02 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + (untracked) 14,703.67 1.3% 1.3% + -------------------------------------------------------------------------------------------------------------------- + python_solution_indexing 441 11,484.12 26.04 1.0% 1.0% + -------------------------------------------------------------------------------------------------------------------- + python_write_cache 441 611.27 1.39 0.1% 0.1% + -------------------------------------------------------------------------------------------------------------------- + python_cache_check 441 0.64 0.00 0.0% 0.0% + + Overlapping Totals (wrap or overlap other timers; not summed into the tree) + ---------------------------------------------------------------------------------------------------------------------- + python_kernel_build_src_co 441 185,511.28 420.66 16.1% + python_kernel_build_src_co.setup 441 224.67 0.51 0.0% + python_kernel_build_src_co.cache_check 441 5,995.91 13.60 0.5% + python_kernel_build_src_co.pch 97 4,780.47 49.28 0.4% + python_kernel_build_src_co.compile 97 166,252.19 1,713.94 14.4% + python_kernel_build_src_co.unbundle 97 878.00 9.05 0.1% + python_kernel_build_src_co.move 97 27.55 0.28 0.0% + python_kernel_build_src_co.cache_populate 97 248.95 2.57 0.0% + +------------------------------------------------------------------------------------------------------------------------ +HIERARCHY VALIDATION: OK (all child phases fit within their parents) + +TIME BREAKDOWN (visual, % of wall clock) +------------------------------------------------------------------------------------------------------------------------ + TOTAL RUNTIME 100.0% ################################################## + BUILD: solution generation & compile 95.9% ############################################### + Solution generation & build (Python) 95.9% ############################################### + python_kernel_compilation 70.1% ################################### + python_kernel_codegen 23.3% ########### + python_kernel_write_assemble 15.1% ####### + python_kernel_validate 10.9% ##### + python_kernel_bench_postprocess 10.7% ##### + python_benchpost_lib_construction 6.8% ### + python_benchpost_library_write 2.8% # + python_benchpost_naming 1.1% + python_benchpost_client_config 0.0% + (untracked) 0.0% + python_kernel_build_src_co_wait 4.1% ## + python_kernel_setup 2.7% # + python_kernel_build_co 1.8% + python_kernel_write_helpers 0.6% + python_kernel_bench_setup 0.4% + (untracked) 0.5% + python_solution_generation 25.8% ############ + python_solgen_forked_solutions 25.7% ############ + python_solgen_custom_kernels 0.1% + python_solgen_fork_permutations 0.0% + (untracked) 0.0% + SHARED: cache & solution files 2.8% # + python_write_solutions 1.8% + python_wsol_dump 1.7% + python_wsol_prepare 0.0% + python_wsol_prepare_nocache 0.0% + (untracked) 0.0% + python_wsol_header 0.0% + (untracked) 0.0% + python_solution_indexing 1.0% + python_write_cache 0.1% + python_cache_check 0.0% + UNATTRIBUTED (untracked time & timer overhead) 1.3% + (untracked in python_benchmark_problems) 1.3% + +KEY INSIGHTS (% of wall clock) +------------------------------------------------------------------------------------------------------------------------ + Solution generation & build (Python) 1104.77s (95.9%) + SHARED: cache & solution files 32.64s (2.8%) + UNATTRIBUTED (untracked time & timer overhead) 14.70s (1.3%) + + diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/post_run/AGGREGATE_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/post_run/AGGREGATE_analysis.txt new file mode 100644 index 000000000000..8841b9e0d152 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/post_run/AGGREGATE_analysis.txt @@ -0,0 +1,585 @@ +==================================================================================================================================== +TIMING ANALYSIS SUMMARY +==================================================================================================================================== + +TIMING BY PURPOSE +------------------------------------------------------------------------------------------------------------------------------------ + Category Count Total (ms) Mean (ms) % of Parent % of Wall +------------------------------------------------------------------------------------------------------------------------------------ + TOTAL RUNTIME 828,655.65 100.0% + RUN: benchmark client 718,919.62 86.8% 86.8% + Client launch (startup, setup & teardown) 468,439.79 65.2% 56.5% + listener_setup 441 179,268.92 406.51 38.3% 21.6% + hip_initialization 441 167,901.69 380.73 35.8% 20.3% + process_exit 441 54,706.34 124.05 11.7% 6.6% + library_loading 441 17,247.98 39.11 3.7% 2.1% + data_init_setup 441 15,082.37 34.20 3.2% 1.8% + code_object_loading 441 13,085.11 29.67 2.8% 1.6% + main_teardown_locals 441 11,335.07 25.70 2.4% 1.4% + process_startup 441 8,596.68 19.49 1.8% 1.0% + lazy_loading_init 441 793.21 1.80 0.2% 0.1% + main_parse_args 441 245.05 0.56 0.1% 0.0% + main_problem_factory 441 92.29 0.21 0.0% 0.0% + (untracked) 92.29 100.0% 0.0% + python_client_write_run_script 441 54.17 0.12 0.0% 0.0% + reporter_setup 441 23.17 0.05 0.0% 0.0% + solution_iterator_setup 441 7.74 0.02 0.0% 0.0% + Kernel launch & execution 67,792.53 9.4% 8.2% + warmup_first_sync 1,229,030 50,716.93 0.04 74.8% 6.1% + warmup_runs 2,458,060 11,484.24 0.00 16.9% 1.4% + kernel_solving 1,229,034 3,365.17 0.00 5.0% 0.4% + benchmark_runs 1,229,034 1,206.77 0.00 1.8% 0.1% + benchmark_validate_enqueues 602 800.92 1.33 66.4% 0.1% + validate_enqueues_listener.HardwareMonitorListener 602 358.72 0.60 44.8% 0.0% + validate_enqueues_listener.ReferenceValidator 602 220.51 0.37 27.5% 0.0% + validate_enqueues_listener.BenchmarkTimer 602 199.42 0.33 24.9% 0.0% + benchmark_sleep 601 163.57 0.27 82.0% 0.0% + benchmark_event_sync 219 35.49 0.16 17.8% 0.0% + benchmark_event_elapsed_time 219 0.20 0.00 0.1% 0.0% + benchmark_event_destroy 219 0.16 0.00 0.1% 0.0% + validate_enqueues_listener.SynchronizerValidator 602 14.01 0.02 1.7% 0.0% + validate_enqueues_listener.ProgressListener 602 7.28 0.01 0.9% 0.0% + validate_enqueues_listener.MetaResultReporter 602 0.03 0.00 0.0% 0.0% + validate_enqueues_listener.DataInitialization 602 0.02 0.00 0.0% 0.0% + validate_enqueues_listener.AllSolutionsIterator 602 0.02 0.00 0.0% 0.0% + (untracked) 0.91 0.1% 0.0% + gpu_kernel_execution 602 305.37 0.51 25.3% 0.0% + benchmark_post_syncs 1,229,034 53.04 0.00 4.4% 0.0% + benchmark_pre_syncs 1,229,034 46.49 0.00 3.9% 0.0% + benchmark_events_setup 602 0.68 0.00 0.1% 0.0% + benchmark_events_teardown 602 0.28 0.00 0.0% 0.0% + warmup_events_setup 1,229,034 611.83 0.00 0.9% 0.1% + run_iteration_sync 1,229,034 372.15 0.00 0.5% 0.0% + validate_warmups_listener.BenchmarkTimer 1,229,030 35.42 0.00 0.1% 0.0% + validate_gpu_sync 598 0.44 0.00 1.2% 0.0% + (untracked) 34.98 98.8% 0.0% + Validation 66,752.37 9.3% 8.1% + validate_warmups_listener.ReferenceValidator 1,229,030 61,018.96 0.05 91.4% 7.4% + validate_element_comparison 1,042,067 32,369.88 0.03 53.0% 3.9% + validate_gpu_readback 1,042,067 24,904.42 0.02 40.8% 3.0% + validate_result_buffer_alloc 1,042,067 3,253.38 0.00 5.3% 0.4% + validate_inputs_copy 1,036,487 134.21 0.00 0.2% 0.0% + validate_setup 1,037,703 30.65 0.00 0.1% 0.0% + validate_report 1,042,067 30.60 0.00 0.1% 0.0% + (untracked) 295.81 0.5% 0.0% + synchronizer_check 1,229,034 5,522.58 0.00 8.3% 0.7% + synchronizer_reduce 253,709 5,354.20 0.02 97.0% 0.6% + synchronizer_inactive 975,349 23.65 0.00 0.4% 0.0% + (untracked) 144.73 2.6% 0.0% + validate_warmups_listener.MetaResultReporter 1,229,030 47.93 0.00 0.1% 0.0% + validate_warmups_listener.HardwareMonitorListener 1,229,030 34.44 0.00 0.1% 0.0% + validate_warmups_listener.ProgressListener 1,229,030 32.62 0.00 0.0% 0.0% + validate_warmups_listener.AllSolutionsIterator 1,229,030 32.38 0.00 0.0% 0.0% + validate_warmups_listener.SynchronizerValidator 1,229,030 32.12 0.00 0.0% 0.0% + (untracked) 32.12 100.0% 0.0% + validate_warmups_listener.DataInitialization 1,229,030 31.34 0.00 0.0% 0.0% + Input data preparation 53,726.75 7.5% 6.5% + gpu_input_reset 1,229,034 31,704.99 0.03 59.0% 3.8% + gpu_input_reset_batched_inputs 658,043 19,665.29 0.03 62.0% 2.4% + gpu_input_reset_copy_inputs_d2d 658,043 10,024.05 0.02 31.6% 1.2% + gpu_input_reset_fast_reset_output 570,863 1,625.23 0.00 5.1% 0.2% + gpu_input_reset_convert_to_inputs 658,043 83.75 0.00 0.3% 0.0% + gpu_input_reset_copy_valid_to_gpu 137,906 59.24 0.00 0.2% 0.0% + gpu_input_reset_slow_path 658,043 16.04 0.00 0.1% 0.0% + gpu_input_reset_copy_swizzled 13,132 2.45 0.00 0.0% 0.0% + gpu_input_reset_rotating_init 2 0.04 0.02 0.0% 0.0% + gpu_input_reset_constant_inputs 5 0.02 0.00 0.0% 0.0% + gpu_input_reset_init_cpu_inputs 5 0.01 0.00 0.0% 0.0% + (untracked) 228.87 0.7% 0.0% + cpu_data_init 4,029 15,135.73 3.76 28.2% 1.8% + gpu_input_preparation 4,035 6,761.73 1.68 12.6% 0.8% + gpu_input_prep_copy_valid_to_gpu 1,797 4,948.50 2.75 73.2% 0.6% + gpu_input_prep_copy_swizzled 98 973.72 9.94 14.4% 0.1% + gpu_input_prep_init_cpu_inputs 3 573.60 191.20 8.5% 0.1% + gpu_input_prep_batched_inputs 2,903 169.01 0.06 2.5% 0.0% + gpu_input_prep_copy_inputs_d2d 2,903 76.10 0.03 1.1% 0.0% + gpu_input_prep_fast_reset_output 1,119 9.77 0.01 0.1% 0.0% + gpu_input_prep_convert_to_inputs 2,903 1.76 0.00 0.0% 0.0% + gpu_input_prep_slow_path 2,903 0.14 0.00 0.0% 0.0% + gpu_input_prep_constant_inputs 4 0.04 0.01 0.0% 0.0% + gpu_input_prep_rotating_init 1 0.01 0.01 0.0% 0.0% + (untracked) 9.07 0.1% 0.0% + rotating_buffer_preparation 4,035 82.06 0.02 0.2% 0.0% + pre_solution_listener.DataInitialization 1,286,411 42.24 0.00 0.1% 0.0% + pre_solution_mx_check 1,286,411 33.65 0.00 79.7% 0.0% + (untracked) 8.58 20.3% 0.0% + Reference GEMM (CPU) 38,352.93 5.3% 4.6% + cpu_reference_gemm 4,029 38,288.38 9.50 99.8% 4.6% + solve_cpu_fast 3,503 37,422.80 10.68 97.7% 4.5% + solve_cpu_slow 585 858.06 1.47 2.2% 0.1% + (untracked) 7.52 0.0% 0.0% + pre_solution_listener.ReferenceValidator 1,286,411 64.55 0.00 0.2% 0.0% + pre_solution_ref_skipped 1,286,230 31.28 0.00 48.5% 0.0% + (untracked) 33.28 51.5% 0.0% + Per-solution bookkeeping & reporting 9,263.67 1.3% 1.1% + run_current_solution_check 1,286,411 1,592.51 0.00 17.2% 0.2% + post_solution_reporting 1,286,411 1,468.29 0.00 15.8% 0.2% + pre_solution_listener.BenchmarkTimer 1,286,411 1,459.57 0.00 15.8% 0.2% + post_solution_log 1,286,411 1,050.71 0.00 11.3% 0.1% + pre_solution_listener.AllSolutionsIterator 1,286,411 981.53 0.00 10.6% 0.1% + pre_solution_listener.MetaResultReporter 1,286,411 607.95 0.00 6.6% 0.1% + solution_selection 1,286,411 586.47 0.00 6.3% 0.1% + pre_solution_listener.ProgressListener 1,286,411 556.61 0.00 6.0% 0.1% + post_solution_validation 1,286,411 307.05 0.00 3.3% 0.0% + pre_solution_listener.SynchronizerValidator 1,286,411 211.26 0.00 2.3% 0.0% + run_iteration_teardown 1,229,034 195.86 0.00 2.1% 0.0% + post_problem 4,035 65.20 0.02 0.7% 0.0% + solution_teardown 1,286,411 43.35 0.00 0.5% 0.0% + pre_solution_listener.HardwareMonitorListener 1,286,411 34.15 0.00 0.4% 0.0% + post_solution_perf_calc 1,286,411 33.63 0.00 0.4% 0.0% + post_solution_sol_advance 1,286,411 32.80 0.00 0.4% 0.0% + post_solution_perf_reset 1,286,411 31.38 0.00 0.3% 0.0% + problem_report_index 4,035 4.79 0.00 0.1% 0.0% + finalize_report 441 0.29 0.00 0.0% 0.0% + problem_teardown 4,035 0.25 0.00 0.0% 0.0% + post_solution_result_file 335 0.01 0.00 0.0% 0.0% + Library logic (Python) 8,379.03 1.2% 1.0% + python_logic_write 5 7,278.92 1,455.78 86.9% 0.9% + python_logic_parse_solutions 5 987.41 197.48 11.8% 0.1% + python_logic_analyze 5 112.70 22.54 1.3% 0.0% + Library client (Python) 6,212.54 0.9% 0.7% + python_library_client_create_library 1 4,131.51 4,131.51 66.5% 0.5% + python_library_client_execution 1 2,070.88 2,070.88 33.3% 0.2% + python_library_client_subprocess 1 2,070.26 2,070.26 100.0% 0.2% + python_library_client_write_run_script 1 0.08 0.08 0.0% 0.0% + (untracked) 0.54 0.0% 0.0% + python_library_client_write_configs 1 8.63 8.63 0.1% 0.0% + python_library_client_setup 1 1.52 1.52 0.0% 0.0% + MEASUREMENT OVERHEAD (instrumentation) 107,970.48 13.0% 13.0% + flush_timing_buffer 443 89,828.91 202.77 83.2% 10.8% + main_timing_calibration 441 10,990.36 24.92 10.2% 1.3% + timing_overhead 441 3,700.16 8.39 3.4% 0.4% + (timer overhead subtracted from parent timers) 3,451.04 3.2% 0.4% + SHARED: cache & solution files 965.91 0.1% 0.1% + python_cache_check 441 847.49 1.92 87.7% 0.1% + python_write_solutions 441 118.42 0.27 12.3% 0.0% + python_wsol_prepare 441 85.38 0.19 72.1% 0.0% + python_wsol_prepare_cache 441 82.85 0.19 97.0% 0.0% + (untracked) 2.54 3.0% 0.0% + python_wsol_header 441 26.45 0.06 22.3% 0.0% + (untracked) 6.58 5.6% 0.0% + UNATTRIBUTED (untracked time & timer overhead) 799.64 0.1% 0.1% + (untracked in python_benchmark_problems) 3,199.25 400.1% 0.4% + (untracked in validate_warmups) 336.05 42.0% 0.0% + (untracked in pre_solution) 262.48 32.8% 0.0% + (untracked in pre_problem) 39.53 4.9% 0.0% + (untracked in python_library_logic) 5.15 0.6% 0.0% + (untracked in python_client_writer) 0.93 0.1% 0.0% + (children exceed benchmark_runs) -16.61 -2.1% -0.0% + (children exceed validate_enqueues_listener.BenchmarkTimer) -35.42 -4.4% -0.0% + (children exceed post_solution) -52.43 -6.6% -0.0% + (children exceed python_client_execution) -2,939.28 -367.6% -0.4% + +TIMING BY TIMER NESTING (legacy view) +------------------------------------------------------------------------------------------------------------------------------------ + Category Count Total (ms) Mean (ms) % of Parent % of Wall +------------------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 123 810,606.95 6,590.30 97.8% + -------------------------------------------------------------------------------------------------------------------------------- + python_client_execution 441 806,441.80 1,828.67 99.5% 97.3% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + C++ Client Setup & Teardown 479,430.15 59.5% 57.9% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + listener_setup 441 179,268.92 406.51 37.4% 21.6% + hip_initialization 441 167,901.69 380.73 35.0% 20.3% + process_exit 441 54,706.34 124.05 11.4% 6.6% + library_loading 441 17,247.98 39.11 3.6% 2.1% + data_init_setup 441 15,082.37 34.20 3.1% 1.8% + code_object_loading 441 13,085.11 29.67 2.7% 1.6% + main_teardown_locals 441 11,335.07 25.70 2.4% 1.4% + main_timing_calibration 441 10,990.36 24.92 2.3% 1.3% + process_startup 441 8,596.68 19.49 1.8% 1.0% + lazy_loading_init 441 793.21 1.80 0.2% 0.1% + main_parse_args 441 245.05 0.56 0.1% 0.0% + main_problem_factory 441 92.29 0.21 0.0% 0.0% + (untracked) 92.29 100.0% 0.0% + python_client_write_run_script 441 54.17 0.12 0.0% 0.0% + reporter_setup 441 23.17 0.05 0.0% 0.0% + solution_iterator_setup 441 7.74 0.02 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Timing Overhead 93,529.07 11.6% 11.3% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + flush_timing_buffer 443 89,828.91 202.77 96.0% 10.8% + timing_overhead 441 3,700.16 8.39 4.0% 0.4% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Data Preparation 92,012.42 11.4% 11.1% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + pre_problem 4,035 53,463.64 13.25 58.1% 6.5% + cpu_reference_gemm 4,029 38,288.38 9.50 71.6% 4.6% + solve_cpu_fast 3,503 37,422.80 10.68 97.7% 4.5% + solve_cpu_slow 585 858.06 1.47 2.2% 0.1% + (untracked) 7.52 0.0% 0.0% + cpu_data_init 4,029 15,135.73 3.76 28.3% 1.8% + (untracked) 39.53 0.1% 0.0% + gpu_input_reset 1,229,034 31,704.99 0.03 34.5% 3.8% + gpu_input_reset_batched_inputs 658,043 19,665.29 0.03 62.0% 2.4% + gpu_input_reset_copy_inputs_d2d 658,043 10,024.05 0.02 31.6% 1.2% + gpu_input_reset_fast_reset_output 570,863 1,625.23 0.00 5.1% 0.2% + (untracked) 228.87 0.7% 0.0% + gpu_input_reset_convert_to_inputs 658,043 83.75 0.00 0.3% 0.0% + gpu_input_reset_copy_valid_to_gpu 137,906 59.24 0.00 0.2% 0.0% + gpu_input_reset_slow_path 658,043 16.04 0.00 0.1% 0.0% + gpu_input_reset_copy_swizzled 13,132 2.45 0.00 0.0% 0.0% + gpu_input_reset_rotating_init 2 0.04 0.02 0.0% 0.0% + gpu_input_reset_constant_inputs 5 0.02 0.00 0.0% 0.0% + gpu_input_reset_init_cpu_inputs 5 0.01 0.00 0.0% 0.0% + gpu_input_preparation 4,035 6,761.73 1.68 7.3% 0.8% + gpu_input_prep_copy_valid_to_gpu 1,797 4,948.50 2.75 73.2% 0.6% + gpu_input_prep_copy_swizzled 98 973.72 9.94 14.4% 0.1% + gpu_input_prep_init_cpu_inputs 3 573.60 191.20 8.5% 0.1% + gpu_input_prep_batched_inputs 2,903 169.01 0.06 2.5% 0.0% + gpu_input_prep_copy_inputs_d2d 2,903 76.10 0.03 1.1% 0.0% + gpu_input_prep_fast_reset_output 1,119 9.77 0.01 0.1% 0.0% + (untracked) 9.07 0.1% 0.0% + gpu_input_prep_convert_to_inputs 2,903 1.76 0.00 0.0% 0.0% + gpu_input_prep_slow_path 2,903 0.14 0.00 0.0% 0.0% + gpu_input_prep_constant_inputs 4 0.04 0.01 0.0% 0.0% + gpu_input_prep_rotating_init 1 0.01 0.01 0.0% 0.0% + rotating_buffer_preparation 4,035 82.06 0.02 0.1% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Kernel Execution 77,285.59 9.6% 9.3% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + warmup_first_sync 1,229,030 50,716.93 0.04 65.6% 6.1% + warmup_runs 2,458,060 11,484.24 0.00 14.9% 1.4% + pre_solution 1,286,411 4,220.34 0.00 5.5% 0.5% + pre_solution_listener.BenchmarkTimer 1,286,411 1,459.57 0.00 34.6% 0.2% + pre_solution_listener.AllSolutionsIterator 1,286,411 981.53 0.00 23.3% 0.1% + pre_solution_listener.MetaResultReporter 1,286,411 607.95 0.00 14.4% 0.1% + pre_solution_listener.ProgressListener 1,286,411 556.61 0.00 13.2% 0.1% + (untracked) 262.48 6.2% 0.0% + pre_solution_listener.SynchronizerValidator 1,286,411 211.26 0.00 5.0% 0.0% + pre_solution_listener.ReferenceValidator 1,286,411 64.55 0.00 1.5% 0.0% + (untracked) 33.28 51.5% 0.0% + pre_solution_ref_skipped 1,286,230 31.28 0.00 48.5% 0.0% + pre_solution_listener.DataInitialization 1,286,411 42.24 0.00 1.0% 0.0% + pre_solution_mx_check 1,286,411 33.65 0.00 79.7% 0.0% + (untracked) 8.58 20.3% 0.0% + pre_solution_listener.HardwareMonitorListener 1,286,411 34.15 0.00 0.8% 0.0% + kernel_solving 1,229,034 3,365.17 0.00 4.4% 0.4% + post_solution 1,286,411 2,871.44 0.00 3.7% 0.3% + post_solution_reporting 1,286,411 1,468.29 0.00 51.1% 0.2% + post_solution_log 1,286,411 1,050.71 0.00 36.6% 0.1% + post_solution_validation 1,286,411 307.05 0.00 10.7% 0.0% + post_solution_perf_calc 1,286,411 33.63 0.00 1.2% 0.0% + post_solution_sol_advance 1,286,411 32.80 0.00 1.1% 0.0% + post_solution_perf_reset 1,286,411 31.38 0.00 1.1% 0.0% + post_solution_result_file 335 0.01 0.00 0.0% 0.0% + run_current_solution_check 1,286,411 1,592.51 0.00 2.1% 0.2% + benchmark_runs 1,229,034 1,154.74 0.00 1.5% 0.1% + benchmark_validate_enqueues 602 765.50 1.27 66.3% 0.1% + validate_enqueues_listener.HardwareMonitorListener 602 358.72 0.60 46.9% 0.0% + validate_enqueues_listener.ReferenceValidator 602 220.51 0.37 28.8% 0.0% + validate_enqueues_listener.BenchmarkTimer 602 164.00 0.27 21.4% 0.0% + benchmark_sleep 601 163.57 0.27 99.7% 0.0% + benchmark_event_sync 219 35.49 0.16 21.6% 0.0% + benchmark_event_elapsed_time 219 0.20 0.00 0.1% 0.0% + benchmark_event_destroy 219 0.16 0.00 0.1% 0.0% + validate_enqueues_listener.SynchronizerValidator 602 14.01 0.02 1.8% 0.0% + validate_enqueues_listener.ProgressListener 602 7.28 0.01 1.0% 0.0% + (untracked) 0.91 0.1% 0.0% + validate_enqueues_listener.MetaResultReporter 602 0.03 0.00 0.0% 0.0% + validate_enqueues_listener.DataInitialization 602 0.02 0.00 0.0% 0.0% + validate_enqueues_listener.AllSolutionsIterator 602 0.02 0.00 0.0% 0.0% + gpu_kernel_execution 602 305.37 0.51 26.4% 0.0% + benchmark_post_syncs 1,229,034 53.04 0.00 4.6% 0.0% + benchmark_pre_syncs 1,229,034 46.49 0.00 4.0% 0.0% + benchmark_events_setup 602 0.68 0.00 0.1% 0.0% + benchmark_events_teardown 602 0.28 0.00 0.0% 0.0% + warmup_events_setup 1,229,034 611.83 0.00 0.8% 0.1% + solution_selection 1,286,411 586.47 0.00 0.8% 0.1% + run_iteration_sync 1,229,034 372.15 0.00 0.5% 0.0% + run_iteration_teardown 1,229,034 195.86 0.00 0.3% 0.0% + post_problem 4,035 65.20 0.02 0.1% 0.0% + solution_teardown 1,286,411 43.35 0.00 0.1% 0.0% + problem_report_index 4,035 4.79 0.00 0.0% 0.0% + finalize_report 441 0.29 0.00 0.0% 0.0% + problem_teardown 4,035 0.25 0.00 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Reference Computation 67,123.84 8.3% 8.1% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + validate_warmups 1,229,030 61,601.26 0.05 91.8% 7.4% + validate_warmups_listener.ReferenceValidator 1,229,030 61,018.96 0.05 99.1% 7.4% + validate_element_comparison 1,042,067 32,369.88 0.03 53.0% 3.9% + validate_gpu_readback 1,042,067 24,904.42 0.02 40.8% 3.0% + validate_result_buffer_alloc 1,042,067 3,253.38 0.00 5.3% 0.4% + (untracked) 295.81 0.5% 0.0% + validate_inputs_copy 1,036,487 134.21 0.00 0.2% 0.0% + validate_setup 1,037,703 30.65 0.00 0.1% 0.0% + validate_report 1,042,067 30.60 0.00 0.1% 0.0% + (untracked) 336.05 0.5% 0.0% + validate_warmups_listener.MetaResultReporter 1,229,030 47.93 0.00 0.1% 0.0% + validate_warmups_listener.BenchmarkTimer 1,229,030 35.42 0.00 0.1% 0.0% + (untracked) 34.98 98.8% 0.0% + validate_gpu_sync 598 0.44 0.00 1.2% 0.0% + validate_warmups_listener.HardwareMonitorListener 1,229,030 34.44 0.00 0.1% 0.0% + validate_warmups_listener.ProgressListener 1,229,030 32.62 0.00 0.1% 0.0% + validate_warmups_listener.AllSolutionsIterator 1,229,030 32.38 0.00 0.1% 0.0% + validate_warmups_listener.SynchronizerValidator 1,229,030 32.12 0.00 0.1% 0.0% + (untracked) 32.12 100.0% 0.0% + validate_warmups_listener.DataInitialization 1,229,030 31.34 0.00 0.1% 0.0% + synchronizer_check 1,229,034 5,522.58 0.00 8.2% 0.7% + synchronizer_reduce 253,709 5,354.20 0.02 97.0% 0.6% + (untracked) 144.73 2.6% 0.0% + synchronizer_inactive 975,349 23.65 0.00 0.4% 0.0% + -------------------------------------------------------------------------------------------------------------------------------- + (untracked) 3,199.25 0.4% 0.4% + -------------------------------------------------------------------------------------------------------------------------------- + python_cache_check 441 847.49 1.92 0.1% 0.1% + -------------------------------------------------------------------------------------------------------------------------------- + python_write_solutions 441 118.42 0.27 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_prepare 441 85.38 0.19 72.1% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_wsol_prepare_cache 441 82.85 0.19 97.0% 0.0% + (untracked) 2.54 3.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_header 441 26.45 0.06 22.3% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 6.58 5.6% 0.0% + ================================================================================================================================== + python_library_logic 5 8,384.18 1,676.84 1.0% + -------------------------------------------------------------------------------------------------------------------------------- + python_logic_write 5 7,278.92 1,455.78 86.8% 0.9% + -------------------------------------------------------------------------------------------------------------------------------- + python_logic_parse_solutions 5 987.41 197.48 11.8% 0.1% + -------------------------------------------------------------------------------------------------------------------------------- + python_logic_analyze 5 112.70 22.54 1.3% 0.0% + -------------------------------------------------------------------------------------------------------------------------------- + (untracked) 5.15 0.1% 0.0% + ================================================================================================================================== + python_client_writer 1 6,213.47 6,213.47 0.7% + -------------------------------------------------------------------------------------------------------------------------------- + python_library_client_create_library 1 4,131.51 4,131.51 66.5% 0.5% + -------------------------------------------------------------------------------------------------------------------------------- + python_library_client_execution 1 2,070.88 2,070.88 33.3% 0.2% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_library_client_subprocess 1 2,070.26 2,070.26 100.0% 0.2% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 0.54 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_library_client_write_run_script 1 0.08 0.08 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------------------- + python_library_client_write_configs 1 8.63 8.63 0.1% 0.0% + -------------------------------------------------------------------------------------------------------------------------------- + python_library_client_setup 1 1.52 1.52 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------------------- + (untracked) 0.93 0.0% 0.0% + + Unknown Categories + ---------------------------------------------------------------------------------------------------------------------------------- + icache_rotate_extra_copies_loading 1 5.86 5.86 0.0% + + Overlapping Totals (wrap or overlap other timers; not summed into the tree) + ---------------------------------------------------------------------------------------------------------------------------------- + python_client_subprocess 441 809,527.67 1,835.66 97.7% + main_total 441 645,030.44 1,462.65 77.8% + problem_total 4,035 240,270.61 59.55 29.0% + solution_total 1,286,411 179,822.86 0.14 21.7% + run_iteration_total 1,229,034 168,774.82 0.14 20.4% + validate_total 1,037,703 61,012.13 0.06 7.4% + validate_check_results_typed 1,042,067 60,795.53 0.06 7.3% + +------------------------------------------------------------------------------------------------------------------------------------ +HIERARCHY VALIDATION WARNINGS +------------------------------------------------------------------------------------------------------------------------------------ + WARNING: Child phases of python_client_execution (811,749.69 ms) exceed parent (809,892.69 ms) by 1,856.99 ms (0.2%) + WARNING: Child phases of validate_enqueues_listener.BenchmarkTimer (199.42 ms) exceed parent (164.07 ms) by 35.36 ms (21.5%) + +TIME BREAKDOWN (visual, % of wall clock) +------------------------------------------------------------------------------------------------------------------------------------ + TOTAL RUNTIME 100.0% ################################################## + RUN: benchmark client 86.8% ########################################### + Client launch (startup, setup & teardown) 56.5% ############################ + listener_setup 21.6% ########## + hip_initialization 20.3% ########## + process_exit 6.6% ### + library_loading 2.1% # + data_init_setup 1.8% + code_object_loading 1.6% + main_teardown_locals 1.4% + process_startup 1.0% + lazy_loading_init 0.1% + main_parse_args 0.0% + main_problem_factory 0.0% + (untracked) 0.0% + python_client_write_run_script 0.0% + reporter_setup 0.0% + solution_iterator_setup 0.0% + Kernel launch & execution 8.2% #### + warmup_first_sync 6.1% ### + warmup_runs 1.4% + kernel_solving 0.4% + benchmark_runs 0.1% + benchmark_validate_enqueues 0.1% + validate_enqueues_listener.HardwareMonitorListener 0.0% + validate_enqueues_listener.ReferenceValidator 0.0% + validate_enqueues_listener.BenchmarkTimer 0.0% + benchmark_sleep 0.0% + benchmark_event_sync 0.0% + benchmark_event_elapsed_time 0.0% + benchmark_event_destroy 0.0% + validate_enqueues_listener.SynchronizerValidator 0.0% + validate_enqueues_listener.ProgressListener 0.0% + validate_enqueues_listener.MetaResultReporter 0.0% + validate_enqueues_listener.DataInitialization 0.0% + validate_enqueues_listener.AllSolutionsIterator 0.0% + (untracked) 0.0% + gpu_kernel_execution 0.0% + benchmark_post_syncs 0.0% + benchmark_pre_syncs 0.0% + benchmark_events_setup 0.0% + benchmark_events_teardown 0.0% + warmup_events_setup 0.1% + run_iteration_sync 0.0% + validate_warmups_listener.BenchmarkTimer 0.0% + validate_gpu_sync 0.0% + (untracked) 0.0% + Validation 8.1% #### + validate_warmups_listener.ReferenceValidator 7.4% ### + validate_element_comparison 3.9% # + validate_gpu_readback 3.0% # + validate_result_buffer_alloc 0.4% + validate_inputs_copy 0.0% + validate_setup 0.0% + validate_report 0.0% + (untracked) 0.0% + synchronizer_check 0.7% + synchronizer_reduce 0.6% + synchronizer_inactive 0.0% + (untracked) 0.0% + validate_warmups_listener.MetaResultReporter 0.0% + validate_warmups_listener.HardwareMonitorListener 0.0% + validate_warmups_listener.ProgressListener 0.0% + validate_warmups_listener.AllSolutionsIterator 0.0% + validate_warmups_listener.SynchronizerValidator 0.0% + (untracked) 0.0% + validate_warmups_listener.DataInitialization 0.0% + Input data preparation 6.5% ### + gpu_input_reset 3.8% # + gpu_input_reset_batched_inputs 2.4% # + gpu_input_reset_copy_inputs_d2d 1.2% + gpu_input_reset_fast_reset_output 0.2% + gpu_input_reset_convert_to_inputs 0.0% + gpu_input_reset_copy_valid_to_gpu 0.0% + gpu_input_reset_slow_path 0.0% + gpu_input_reset_copy_swizzled 0.0% + gpu_input_reset_rotating_init 0.0% + gpu_input_reset_constant_inputs 0.0% + gpu_input_reset_init_cpu_inputs 0.0% + (untracked) 0.0% + cpu_data_init 1.8% + gpu_input_preparation 0.8% + gpu_input_prep_copy_valid_to_gpu 0.6% + gpu_input_prep_copy_swizzled 0.1% + gpu_input_prep_init_cpu_inputs 0.1% + gpu_input_prep_batched_inputs 0.0% + gpu_input_prep_copy_inputs_d2d 0.0% + gpu_input_prep_fast_reset_output 0.0% + gpu_input_prep_convert_to_inputs 0.0% + gpu_input_prep_slow_path 0.0% + gpu_input_prep_constant_inputs 0.0% + gpu_input_prep_rotating_init 0.0% + (untracked) 0.0% + rotating_buffer_preparation 0.0% + pre_solution_listener.DataInitialization 0.0% + pre_solution_mx_check 0.0% + (untracked) 0.0% + Reference GEMM (CPU) 4.6% ## + cpu_reference_gemm 4.6% ## + solve_cpu_fast 4.5% ## + solve_cpu_slow 0.1% + (untracked) 0.0% + pre_solution_listener.ReferenceValidator 0.0% + pre_solution_ref_skipped 0.0% + (untracked) 0.0% + Per-solution bookkeeping & reporting 1.1% + run_current_solution_check 0.2% + post_solution_reporting 0.2% + pre_solution_listener.BenchmarkTimer 0.2% + post_solution_log 0.1% + pre_solution_listener.AllSolutionsIterator 0.1% + pre_solution_listener.MetaResultReporter 0.1% + solution_selection 0.1% + pre_solution_listener.ProgressListener 0.1% + post_solution_validation 0.0% + pre_solution_listener.SynchronizerValidator 0.0% + run_iteration_teardown 0.0% + post_problem 0.0% + solution_teardown 0.0% + pre_solution_listener.HardwareMonitorListener 0.0% + post_solution_perf_calc 0.0% + post_solution_sol_advance 0.0% + post_solution_perf_reset 0.0% + problem_report_index 0.0% + finalize_report 0.0% + problem_teardown 0.0% + post_solution_result_file 0.0% + Library logic (Python) 1.0% + python_logic_write 0.9% + python_logic_parse_solutions 0.1% + python_logic_analyze 0.0% + Library client (Python) 0.7% + python_library_client_create_library 0.5% + python_library_client_execution 0.2% + python_library_client_subprocess 0.2% + python_library_client_write_run_script 0.0% + (untracked) 0.0% + python_library_client_write_configs 0.0% + python_library_client_setup 0.0% + MEASUREMENT OVERHEAD (instrumentation) 13.0% ###### + flush_timing_buffer 10.8% ##### + main_timing_calibration 1.3% + timing_overhead 0.4% + (timer overhead subtracted from parent timers) 0.4% + SHARED: cache & solution files 0.1% + python_cache_check 0.1% + python_write_solutions 0.0% + python_wsol_prepare 0.0% + python_wsol_prepare_cache 0.0% + (untracked) 0.0% + python_wsol_header 0.0% + (untracked) 0.0% + UNATTRIBUTED (untracked time & timer overhead) 0.1% + (untracked in python_benchmark_problems) 0.4% + (untracked in validate_warmups) 0.0% + (untracked in pre_solution) 0.0% + (untracked in pre_problem) 0.0% + (untracked in python_library_logic) 0.0% + (untracked in python_client_writer) 0.0% + (children exceed benchmark_runs) -0.0% + (children exceed validate_enqueues_listener.BenchmarkTimer) -0.0% + (children exceed post_solution) -0.0% + (children exceed python_client_execution) -0.4% + +KEY INSIGHTS (% of wall clock) +------------------------------------------------------------------------------------------------------------------------------------ + Client launch (startup, setup & teardown) 468.44s (56.5%) + MEASUREMENT OVERHEAD (instrumentation) 104.52s (12.6%) + Kernel launch & execution 67.79s (8.2%) + Validation 66.75s (8.1%) + Input data preparation 53.73s (6.5%) + Reference GEMM (CPU) 38.35s (4.6%) + Per-solution bookkeeping & reporting 9.26s (1.1%) + Library logic (Python) 8.38s (1.0%) + Library client (Python) 6.21s (0.7%) + SHARED: cache & solution files 0.97s (0.1%) + UNATTRIBUTED (untracked time & timer overhead) 0.80s (0.1%) + + CPU reference / GPU execution ratio: 125.4x + >>> CPU reference > 10x GPU execution time! + +TOP 10 SLOWEST PROBLEMS (by CPU reference time) +------------------------------------------------------------------------------------------------------------------------------------ + M N K Batch TypeA TypeD CPU Ref (ms) GPU (ms) + ---------------------------------------------------------------------------------------------------------------------------------- + 2160 10240 16384 1 BFloat16 BFloat16 5439.82 0.00 + 10240 2160 16384 1 BFloat16 BFloat16 5283.25 0.00 + 4096 4096 4096 1 Half Half 1025.10 0.00 + 4111 4111 4111 1 Float8_fnuz Float8_fnuz 942.27 0.00 + 4111 4111 4111 1 Float8_fnuz Float8_fnuz 874.63 0.00 + 4111 4111 4111 1 Float8_fnuz Float 771.01 0.00 + 4111 4111 4111 1 Float8_fnuz Half 682.97 0.00 + 4111 4111 4111 1 Float8_fnuz Float 680.12 0.00 + 4111 4111 4111 1 BFloat8_fnuz BFloat16 623.44 0.00 + 4111 4111 4111 1 Float8_fnuz Float 608.52 0.00 + diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/pre_build/AGGREGATE_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/pre_build/AGGREGATE_analysis.txt new file mode 100644 index 000000000000..dbeb98c4f05d --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/pre_build/AGGREGATE_analysis.txt @@ -0,0 +1,167 @@ +======================================================================================================================== +TIMING ANALYSIS SUMMARY +======================================================================================================================== + +TIMING BY PURPOSE +------------------------------------------------------------------------------------------------------------------------ + Category Count Total (ms) Mean (ms) % of Parent % of Wall +------------------------------------------------------------------------------------------------------------------------ + TOTAL RUNTIME 2,134,428.47 100.0% + BUILD: solution generation & compile 2,086,540.33 97.8% 97.8% + Solution generation & build (Python) 2,086,540.33 100.0% 97.8% + python_kernel_compilation 441 1,666,977.28 3,779.99 79.9% 78.1% + python_kernel_codegen 441 455,097.09 1,031.97 27.3% 21.3% + python_kernel_write_assemble 441 158,059.56 358.41 9.5% 7.4% + python_kernel_bench_postprocess 441 123,716.13 280.54 7.4% 5.8% + python_benchpost_lib_construction 441 82,981.66 188.17 67.1% 3.9% + python_benchpost_library_write 441 28,464.85 64.55 23.0% 1.3% + python_benchpost_naming 441 12,141.42 27.53 9.8% 0.6% + python_benchpost_client_config 441 103.14 0.23 0.1% 0.0% + (untracked) 25.05 0.0% 0.0% + python_kernel_bench_setup 441 123,611.11 280.30 7.4% 5.8% + python_kernel_validate 441 73,242.57 166.08 4.4% 3.4% + python_kernel_setup 441 31,789.44 72.08 1.9% 1.5% + python_kernel_build_co 441 20,337.45 46.12 1.2% 1.0% + python_kernel_write_helpers 441 8,028.18 18.20 0.5% 0.4% + (untracked) 673,095.75 40.4% 31.5% + python_solution_generation 441 419,563.05 951.39 20.1% 19.7% + python_solgen_forked_solutions 441 418,462.19 948.89 99.7% 19.6% + python_solgen_custom_kernels 441 1,088.92 2.47 0.3% 0.1% + python_solgen_fork_permutations 441 3.29 0.01 0.0% 0.0% + (untracked) 8.65 0.0% 0.0% + SHARED: cache & solution files 33,006.70 1.5% 1.5% + python_write_solutions 441 20,805.35 47.18 63.0% 1.0% + python_wsol_dump 441 20,147.83 45.69 96.8% 0.9% + python_wsol_prepare 441 544.88 1.24 2.6% 0.0% + python_wsol_prepare_nocache 441 543.71 1.23 99.8% 0.0% + (untracked) 1.18 0.2% 0.0% + python_wsol_header 441 8.86 0.02 0.0% 0.0% + (untracked) 103.78 0.5% 0.0% + python_solution_indexing 441 11,653.85 26.43 35.3% 0.5% + python_write_cache 441 546.90 1.24 1.7% 0.0% + python_cache_check 441 0.61 0.00 0.0% 0.0% + UNATTRIBUTED (untracked time & timer overhead) 14,881.43 0.7% 0.7% + (untracked in python_benchmark_problems) 14,881.43 100.0% 0.7% + +TIMING BY TIMER NESTING (legacy view) +------------------------------------------------------------------------------------------------------------------------ + Category Count Total (ms) Mean (ms) % of Parent % of Wall +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 123 2,134,428.47 17,353.08 100.0% + -------------------------------------------------------------------------------------------------------------------- + python_kernel_compilation 441 1,666,977.28 3,779.99 78.1% 78.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 673,095.75 40.4% 31.5% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_codegen 441 455,097.09 1,031.97 27.3% 21.3% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_assemble 441 158,059.56 358.41 9.5% 7.4% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_postprocess 441 123,716.13 280.54 7.4% 5.8% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_benchpost_lib_construction 441 82,981.66 188.17 67.1% 3.9% + python_benchpost_library_write 441 28,464.85 64.55 23.0% 1.3% + python_benchpost_naming 441 12,141.42 27.53 9.8% 0.6% + python_benchpost_client_config 441 103.14 0.23 0.1% 0.0% + (untracked) 25.05 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_setup 441 123,611.11 280.30 7.4% 5.8% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_validate 441 73,242.57 166.08 4.4% 3.4% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_setup 441 31,789.44 72.08 1.9% 1.5% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_co 441 20,337.45 46.12 1.2% 1.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_helpers 441 8,028.18 18.20 0.5% 0.4% + -------------------------------------------------------------------------------------------------------------------- + python_solution_generation 441 419,563.05 951.39 19.7% 19.7% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_forked_solutions 441 418,462.19 948.89 99.7% 19.6% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_custom_kernels 441 1,088.92 2.47 0.3% 0.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 8.65 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_fork_permutations 441 3.29 0.01 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_write_solutions 441 20,805.35 47.18 1.0% 1.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_dump 441 20,147.83 45.69 96.8% 0.9% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_prepare 441 544.88 1.24 2.6% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_wsol_prepare_nocache 441 543.71 1.23 99.8% 0.0% + (untracked) 1.18 0.2% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 103.78 0.5% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_header 441 8.86 0.02 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + (untracked) 14,881.43 0.7% 0.7% + -------------------------------------------------------------------------------------------------------------------- + python_solution_indexing 441 11,653.85 26.43 0.5% 0.5% + -------------------------------------------------------------------------------------------------------------------- + python_write_cache 441 546.90 1.24 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------- + python_cache_check 441 0.61 0.00 0.0% 0.0% + + Overlapping Totals (wrap or overlap other timers; not summed into the tree) + ---------------------------------------------------------------------------------------------------------------------- + python_kernel_build_src_co 441 667,895.15 1,514.50 31.3% + python_kernel_build_src_co.setup 441 23.62 0.05 0.0% + python_kernel_build_src_co.cache_check 441 1,087.91 2.47 0.1% + python_kernel_build_src_co.compile 97 665,369.87 6,859.48 31.2% + python_kernel_build_src_co.unbundle 97 884.38 9.12 0.0% + python_kernel_build_src_co.move 97 8.86 0.09 0.0% + python_kernel_build_src_co.cache_populate 97 113.60 1.17 0.0% + +------------------------------------------------------------------------------------------------------------------------ +HIERARCHY VALIDATION: OK (all child phases fit within their parents) + +TIME BREAKDOWN (visual, % of wall clock) +------------------------------------------------------------------------------------------------------------------------ + TOTAL RUNTIME 100.0% ################################################## + BUILD: solution generation & compile 97.8% ################################################ + Solution generation & build (Python) 97.8% ################################################ + python_kernel_compilation 78.1% ####################################### + python_kernel_codegen 21.3% ########## + python_kernel_write_assemble 7.4% ### + python_kernel_bench_postprocess 5.8% ## + python_benchpost_lib_construction 3.9% # + python_benchpost_library_write 1.3% + python_benchpost_naming 0.6% + python_benchpost_client_config 0.0% + (untracked) 0.0% + python_kernel_bench_setup 5.8% ## + python_kernel_validate 3.4% # + python_kernel_setup 1.5% + python_kernel_build_co 1.0% + python_kernel_write_helpers 0.4% + (untracked) 31.5% ############### + python_solution_generation 19.7% ######### + python_solgen_forked_solutions 19.6% ######### + python_solgen_custom_kernels 0.1% + python_solgen_fork_permutations 0.0% + (untracked) 0.0% + SHARED: cache & solution files 1.5% + python_write_solutions 1.0% + python_wsol_dump 0.9% + python_wsol_prepare 0.0% + python_wsol_prepare_nocache 0.0% + (untracked) 0.0% + python_wsol_header 0.0% + (untracked) 0.0% + python_solution_indexing 0.5% + python_write_cache 0.0% + python_cache_check 0.0% + UNATTRIBUTED (untracked time & timer overhead) 0.7% + (untracked in python_benchmark_problems) 0.7% + +KEY INSIGHTS (% of wall clock) +------------------------------------------------------------------------------------------------------------------------ + Solution generation & build (Python) 2086.54s (97.8%) + SHARED: cache & solution files 33.01s (1.5%) + UNATTRIBUTED (untracked time & timer overhead) 14.88s (0.7%) + + diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/pre_run/AGGREGATE_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/pre_run/AGGREGATE_analysis.txt new file mode 100644 index 000000000000..2beba206ba45 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/pre_run/AGGREGATE_analysis.txt @@ -0,0 +1,581 @@ +========================================================================================================================================== +TIMING ANALYSIS SUMMARY +========================================================================================================================================== + +TIMING BY PURPOSE +------------------------------------------------------------------------------------------------------------------------------------------ + Category Count Total (ms) Mean (ms) % of Parent % of Wall +------------------------------------------------------------------------------------------------------------------------------------------ + TOTAL RUNTIME 1,644,424.74 100.0% + RUN: benchmark client 1,534,553.50 93.3% 93.3% + Reference GEMM (CPU) 657,031.21 42.8% 40.0% + cpu_reference_gemm 4,029 656,952.69 163.06 100.0% 40.0% + solve_cpu_slow 1,462 624,526.75 427.17 95.1% 38.0% + solve_cpu_fast 2,626 32,418.16 12.35 4.9% 2.0% + (untracked) 7.78 0.0% 0.0% + pre_solution_listener.ReferenceValidator 1,286,411 78.52 0.00 0.0% 0.0% + pre_solution_ref_skipped 1,286,230 31.60 0.00 40.2% 0.0% + (untracked) 46.92 59.8% 0.0% + Client launch (startup, setup & teardown) 472,555.44 30.8% 28.7% + listener_setup 441 178,629.05 405.05 37.8% 10.9% + hip_initialization 441 167,171.50 379.07 35.4% 10.2% + process_exit 441 53,972.70 122.39 11.4% 3.3% + data_init_setup 441 18,751.47 42.52 4.0% 1.1% + library_loading 441 17,364.72 39.38 3.7% 1.1% + code_object_loading 441 14,211.38 32.23 3.0% 0.9% + main_teardown_locals 441 13,070.82 29.64 2.8% 0.8% + process_startup 441 8,165.24 18.52 1.7% 0.5% + lazy_loading_init 441 791.75 1.80 0.2% 0.0% + main_parse_args 441 238.62 0.54 0.1% 0.0% + main_problem_factory 441 90.54 0.21 0.0% 0.0% + (untracked) 90.54 100.0% 0.0% + reporter_setup 441 48.88 0.11 0.0% 0.0% + python_client_write_run_script 441 40.25 0.09 0.0% 0.0% + solution_iterator_setup 441 8.53 0.02 0.0% 0.0% + Validation 180,751.69 11.8% 11.0% + validate_warmups_listener.ReferenceValidator 1,229,030 96,941.00 0.08 53.6% 5.9% + validate_gpu_readback 1,042,067 59,865.63 0.06 61.8% 3.6% + validate_element_comparison 1,042,067 32,234.41 0.03 33.3% 2.0% + validate_result_buffer_alloc 1,042,067 3,897.37 0.00 4.0% 0.2% + validate_inputs_copy 1,036,487 175.42 0.00 0.2% 0.0% + validate_setup 1,037,703 40.51 0.00 0.0% 0.0% + validate_report 1,042,067 36.03 0.00 0.0% 0.0% + (untracked) 691.64 0.7% 0.0% + validate_warmups_listener.SynchronizerValidator 1,229,030 83,620.41 0.07 46.3% 5.1% + validate_synchronizer_check 1,229,052 83,620.41 0.07 100.0% 5.1% + validate_synchronizer_scan 253,775 55,951.86 0.22 66.9% 3.4% + validate_synchronizer_readback 253,775 27,406.94 0.11 32.8% 1.7% + validate_synchronizer_inactive 975,345 23.90 0.00 0.0% 0.0% + (untracked) 237.72 0.3% 0.0% + validate_warmups_listener.MetaResultReporter 1,229,030 52.86 0.00 0.0% 0.0% + validate_warmups_listener.AllSolutionsIterator 1,229,030 36.30 0.00 0.0% 0.0% + validate_warmups_listener.HardwareMonitorListener 1,229,030 36.14 0.00 0.0% 0.0% + validate_warmups_listener.ProgressListener 1,229,030 32.80 0.00 0.0% 0.0% + validate_warmups_listener.DataInitialization 1,229,030 32.17 0.00 0.0% 0.0% + Input data preparation 128,321.13 8.4% 7.8% + gpu_input_reset 1,229,034 93,246.57 0.08 72.7% 5.7% + gpu_input_reset_copy_valid_to_gpu 137,906 39,099.41 0.28 41.9% 2.4% + gpu_input_reset_batched_inputs 658,043 36,356.02 0.06 39.0% 2.2% + gpu_input_reset_copy_inputs_d2d 658,043 11,435.13 0.02 12.3% 0.7% + gpu_input_reset_copy_swizzled 13,132 3,904.59 0.30 4.2% 0.2% + gpu_input_reset_fast_reset_output 570,863 2,009.75 0.00 2.2% 0.1% + gpu_input_reset_convert_to_inputs 658,043 105.63 0.00 0.1% 0.0% + gpu_input_reset_slow_path 658,043 16.27 0.00 0.0% 0.0% + gpu_input_reset_rotating_init 2 0.04 0.02 0.0% 0.0% + gpu_input_reset_constant_inputs 5 0.03 0.01 0.0% 0.0% + gpu_input_reset_init_cpu_inputs 5 0.01 0.00 0.0% 0.0% + (untracked) 319.71 0.3% 0.0% + cpu_data_init 4,029 22,931.94 5.69 17.9% 1.4% + gpu_input_preparation 4,035 11,636.68 2.88 9.1% 0.7% + gpu_input_prep_copy_valid_to_gpu 1,797 8,578.75 4.77 73.7% 0.5% + gpu_input_prep_init_cpu_inputs 3 1,718.87 572.96 14.8% 0.1% + gpu_input_prep_copy_swizzled 98 1,008.54 10.29 8.7% 0.1% + gpu_input_prep_batched_inputs 2,903 223.07 0.08 1.9% 0.0% + gpu_input_prep_copy_inputs_d2d 2,903 81.88 0.03 0.7% 0.0% + gpu_input_prep_fast_reset_output 1,119 12.93 0.01 0.1% 0.0% + gpu_input_prep_convert_to_inputs 2,903 2.05 0.00 0.0% 0.0% + gpu_input_prep_slow_path 2,903 0.13 0.00 0.0% 0.0% + gpu_input_prep_constant_inputs 4 0.05 0.01 0.0% 0.0% + gpu_input_prep_rotating_init 1 0.02 0.02 0.0% 0.0% + (untracked) 10.40 0.1% 0.0% + rotating_buffer_preparation 4,035 438.30 0.11 0.3% 0.0% + pre_solution_listener.DataInitialization 1,286,411 67.63 0.00 0.1% 0.0% + pre_solution_mx_check 1,286,411 40.76 0.00 60.3% 0.0% + (untracked) 26.87 39.7% 0.0% + Per-solution bookkeeping & reporting 44,258.67 2.9% 2.7% + post_solution_log 1,286,411 15,785.47 0.01 35.7% 1.0% + pre_solution_listener.BenchmarkTimer 1,286,411 8,312.12 0.01 18.8% 0.5% + run_current_solution_check 1,286,411 5,748.15 0.00 13.0% 0.3% + post_solution_reporting 1,286,411 3,557.37 0.00 8.0% 0.2% + pre_solution_listener.ProgressListener 1,286,411 2,442.82 0.00 5.5% 0.1% + pre_solution_listener.AllSolutionsIterator 1,286,411 1,995.09 0.00 4.5% 0.1% + post_problem 4,035 1,691.10 0.42 3.8% 0.1% + run_iteration_teardown 1,229,034 1,365.25 0.00 3.1% 0.1% + solution_selection 1,286,411 790.82 0.00 1.8% 0.0% + pre_solution_listener.MetaResultReporter 1,286,411 716.53 0.00 1.6% 0.0% + post_solution_perf_calc 1,286,411 594.29 0.00 1.3% 0.0% + post_solution_validation 1,286,411 440.22 0.00 1.0% 0.0% + finalize_report 441 375.26 0.85 0.8% 0.0% + pre_solution_listener.SynchronizerValidator 1,286,411 225.16 0.00 0.5% 0.0% + post_solution_result_file 1,286,411 50.75 0.00 0.1% 0.0% + solution_teardown 1,286,411 50.70 0.00 0.1% 0.0% + post_solution_sol_advance 1,286,411 38.20 0.00 0.1% 0.0% + pre_solution_listener.HardwareMonitorListener 1,286,411 38.08 0.00 0.1% 0.0% + post_solution_perf_reset 1,286,411 32.98 0.00 0.1% 0.0% + problem_report_index 4,035 8.11 0.00 0.0% 0.0% + problem_teardown 4,035 0.20 0.00 0.0% 0.0% + Kernel launch & execution 35,439.17 2.3% 2.2% + warmup_runs 2,458,060 20,311.11 0.01 57.3% 1.2% + kernel_solving 1,229,034 6,192.62 0.01 17.5% 0.4% + validate_warmups_listener.BenchmarkTimer 1,229,030 5,863.27 0.00 16.5% 0.4% + validate_gpu_sync 1,229,030 5,845.64 0.00 99.7% 0.4% + (untracked) 17.64 0.3% 0.0% + benchmark_runs 1,229,034 1,673.68 0.00 4.7% 0.1% + benchmark_validate_enqueues 605 1,076.67 1.78 64.3% 0.1% + validate_enqueues_listener.BenchmarkTimer 605 600.07 0.99 55.7% 0.0% + benchmark_sleep 604 564.95 0.94 94.1% 0.0% + benchmark_event_sync 219 34.58 0.16 5.8% 0.0% + benchmark_event_elapsed_time 219 0.32 0.00 0.1% 0.0% + benchmark_event_destroy 219 0.22 0.00 0.0% 0.0% + validate_enqueues_listener.ReferenceValidator 605 211.26 0.35 19.6% 0.0% + validate_enqueues_listener.HardwareMonitorListener 605 171.94 0.28 16.0% 0.0% + validate_enqueues_listener.SynchronizerValidator 605 85.09 0.14 7.9% 0.0% + validate_enqueues_listener.ProgressListener 605 7.37 0.01 0.7% 0.0% + validate_enqueues_listener.MetaResultReporter 605 0.04 0.00 0.0% 0.0% + validate_enqueues_listener.DataInitialization 605 0.02 0.00 0.0% 0.0% + validate_enqueues_listener.AllSolutionsIterator 605 0.02 0.00 0.0% 0.0% + (untracked) 0.87 0.1% 0.0% + gpu_kernel_execution 605 440.16 0.73 26.3% 0.0% + benchmark_post_syncs 1,229,034 92.03 0.00 5.5% 0.0% + benchmark_pre_syncs 1,229,034 63.70 0.00 3.8% 0.0% + benchmark_events_setup 605 0.78 0.00 0.0% 0.0% + benchmark_events_teardown 605 0.36 0.00 0.0% 0.0% + warmup_events_setup 1,229,034 1,398.48 0.00 3.9% 0.1% + Library logic (Python) 9,347.35 0.6% 0.6% + python_logic_write 8 8,282.05 1,035.26 88.6% 0.5% + python_logic_parse_solutions 8 938.09 117.26 10.0% 0.1% + python_logic_analyze 8 127.21 15.90 1.4% 0.0% + Library client (Python) 6,848.84 0.4% 0.4% + python_library_client_create_library 1 4,719.06 4,719.06 68.9% 0.3% + python_library_client_execution 1 2,104.68 2,104.68 30.7% 0.1% + python_library_client_subprocess 1 2,103.74 2,103.74 100.0% 0.1% + python_library_client_write_run_script 1 0.16 0.16 0.0% 0.0% + (untracked) 0.78 0.0% 0.0% + python_library_client_write_configs 1 23.56 23.56 0.3% 0.0% + python_library_client_setup 1 1.54 1.54 0.0% 0.0% + MEASUREMENT OVERHEAD (instrumentation) 108,054.22 6.6% 6.6% + flush_timing_buffer 443 89,840.08 202.80 83.1% 5.5% + main_timing_calibration 441 11,030.32 25.01 10.2% 0.7% + timing_overhead 441 3,716.45 8.43 3.4% 0.2% + (timer overhead subtracted from parent timers) 3,467.37 3.2% 0.2% + SHARED: cache & solution files 924.24 0.1% 0.1% + python_cache_check 441 836.07 1.90 90.5% 0.1% + python_write_solutions 441 88.17 0.20 9.5% 0.0% + python_wsol_prepare 441 63.41 0.14 71.9% 0.0% + python_wsol_prepare_cache 441 61.49 0.14 97.0% 0.0% + (untracked) 1.93 3.0% 0.0% + python_wsol_header 441 19.71 0.04 22.4% 0.0% + (untracked) 5.05 5.7% 0.0% + UNATTRIBUTED (untracked time & timer overhead) 892.79 0.1% 0.1% + (untracked in python_benchmark_problems) 2,908.43 325.8% 0.2% + (untracked in validate_warmups) 564.34 63.2% 0.0% + (untracked in pre_solution) 512.63 57.4% 0.0% + (untracked in pre_problem) 67.88 7.6% 0.0% + (untracked in python_library_logic) 6.33 0.7% 0.0% + (untracked in python_client_writer) 1.08 0.1% 0.0% + (children exceed benchmark_runs) -12.92 -1.4% -0.0% + (children exceed post_solution) -26.81 -3.0% -0.0% + (children exceed validate_enqueues_listener.BenchmarkTimer) -34.53 -3.9% -0.0% + (children exceed validate_warmups_listener.SynchronizerValidator) -55.75 -6.2% -0.0% + (children exceed python_client_execution) -3,037.90 -340.3% -0.2% + +TIMING BY TIMER NESTING (legacy view) +------------------------------------------------------------------------------------------------------------------------------------------ + Category Count Total (ms) Mean (ms) % of Parent % of Wall +------------------------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 123 1,624,753.78 13,209.38 98.8% + -------------------------------------------------------------------------------------------------------------------------------------- + python_client_execution 441 1,620,921.10 3,675.56 99.8% 98.6% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Data Preparation 785,274.07 48.4% 47.8% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + pre_problem 4,035 679,952.51 168.51 86.6% 41.3% + cpu_reference_gemm 4,029 656,952.69 163.06 96.6% 40.0% + solve_cpu_slow 1,462 624,526.75 427.17 95.1% 38.0% + solve_cpu_fast 2,626 32,418.16 12.35 4.9% 2.0% + (untracked) 7.78 0.0% 0.0% + cpu_data_init 4,029 22,931.94 5.69 3.4% 1.4% + (untracked) 67.88 0.0% 0.0% + gpu_input_reset 1,229,034 93,246.57 0.08 11.9% 5.7% + gpu_input_reset_copy_valid_to_gpu 137,906 39,099.41 0.28 41.9% 2.4% + gpu_input_reset_batched_inputs 658,043 36,356.02 0.06 39.0% 2.2% + gpu_input_reset_copy_inputs_d2d 658,043 11,435.13 0.02 12.3% 0.7% + gpu_input_reset_copy_swizzled 13,132 3,904.59 0.30 4.2% 0.2% + gpu_input_reset_fast_reset_output 570,863 2,009.75 0.00 2.2% 0.1% + (untracked) 319.71 0.3% 0.0% + gpu_input_reset_convert_to_inputs 658,043 105.63 0.00 0.1% 0.0% + gpu_input_reset_slow_path 658,043 16.27 0.00 0.0% 0.0% + gpu_input_reset_rotating_init 2 0.04 0.02 0.0% 0.0% + gpu_input_reset_constant_inputs 5 0.03 0.01 0.0% 0.0% + gpu_input_reset_init_cpu_inputs 5 0.01 0.00 0.0% 0.0% + gpu_input_preparation 4,035 11,636.68 2.88 1.5% 0.7% + gpu_input_prep_copy_valid_to_gpu 1,797 8,578.75 4.77 73.7% 0.5% + gpu_input_prep_init_cpu_inputs 3 1,718.87 572.96 14.8% 0.1% + gpu_input_prep_copy_swizzled 98 1,008.54 10.29 8.7% 0.1% + gpu_input_prep_batched_inputs 2,903 223.07 0.08 1.9% 0.0% + gpu_input_prep_copy_inputs_d2d 2,903 81.88 0.03 0.7% 0.0% + gpu_input_prep_fast_reset_output 1,119 12.93 0.01 0.1% 0.0% + (untracked) 10.40 0.1% 0.0% + gpu_input_prep_convert_to_inputs 2,903 2.05 0.00 0.0% 0.0% + gpu_input_prep_slow_path 2,903 0.13 0.00 0.0% 0.0% + gpu_input_prep_constant_inputs 4 0.05 0.01 0.0% 0.0% + gpu_input_prep_rotating_init 1 0.02 0.02 0.0% 0.0% + rotating_buffer_preparation 4,035 438.30 0.11 0.1% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + C++ Client Setup & Teardown 483,585.76 29.8% 29.4% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + listener_setup 441 178,629.05 405.05 36.9% 10.9% + hip_initialization 441 167,171.50 379.07 34.6% 10.2% + process_exit 441 53,972.70 122.39 11.2% 3.3% + data_init_setup 441 18,751.47 42.52 3.9% 1.1% + library_loading 441 17,364.72 39.38 3.6% 1.1% + code_object_loading 441 14,211.38 32.23 2.9% 0.9% + main_teardown_locals 441 13,070.82 29.64 2.7% 0.8% + main_timing_calibration 441 11,030.32 25.01 2.3% 0.7% + process_startup 441 8,165.24 18.52 1.7% 0.5% + lazy_loading_init 441 791.75 1.80 0.2% 0.0% + main_parse_args 441 238.62 0.54 0.0% 0.0% + main_problem_factory 441 90.54 0.21 0.0% 0.0% + (untracked) 90.54 100.0% 0.0% + reporter_setup 441 48.88 0.11 0.0% 0.0% + python_client_write_run_script 441 40.25 0.09 0.0% 0.0% + solution_iterator_setup 441 8.53 0.02 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Reference Computation 187,123.55 11.5% 11.4% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + validate_warmups 1,229,030 187,123.55 0.15 100.0% 11.4% + validate_warmups_listener.ReferenceValidator 1,229,030 96,941.00 0.08 51.8% 5.9% + validate_gpu_readback 1,042,067 59,865.63 0.06 61.8% 3.6% + validate_element_comparison 1,042,067 32,234.41 0.03 33.3% 2.0% + validate_result_buffer_alloc 1,042,067 3,897.37 0.00 4.0% 0.2% + (untracked) 691.64 0.7% 0.0% + validate_inputs_copy 1,036,487 175.42 0.00 0.2% 0.0% + validate_setup 1,037,703 40.51 0.00 0.0% 0.0% + validate_report 1,042,067 36.03 0.00 0.0% 0.0% + validate_warmups_listener.SynchronizerValidator 1,229,030 83,564.67 0.07 44.7% 5.1% + validate_synchronizer_check 1,229,052 83,620.41 0.07 100.1% 5.1% + validate_synchronizer_scan 253,775 55,951.86 0.22 66.9% 3.4% + validate_synchronizer_readback 253,775 27,406.94 0.11 32.8% 1.7% + (untracked) 237.72 0.3% 0.0% + validate_synchronizer_inactive 975,345 23.90 0.00 0.0% 0.0% + validate_warmups_listener.BenchmarkTimer 1,229,030 5,863.27 0.00 3.1% 0.4% + validate_gpu_sync 1,229,030 5,845.64 0.00 99.7% 0.4% + (untracked) 17.64 0.3% 0.0% + (untracked) 564.34 0.3% 0.0% + validate_warmups_listener.MetaResultReporter 1,229,030 52.86 0.00 0.0% 0.0% + validate_warmups_listener.AllSolutionsIterator 1,229,030 36.30 0.00 0.0% 0.0% + validate_warmups_listener.HardwareMonitorListener 1,229,030 36.14 0.00 0.0% 0.0% + validate_warmups_listener.ProgressListener 1,229,030 32.80 0.00 0.0% 0.0% + validate_warmups_listener.DataInitialization 1,229,030 32.17 0.00 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Timing Overhead 93,556.53 5.8% 5.7% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + flush_timing_buffer 443 89,840.08 202.80 96.0% 5.5% + timing_overhead 441 3,716.45 8.43 4.0% 0.2% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Kernel Execution 74,419.09 4.6% 4.5% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + post_solution 1,286,411 20,472.48 0.02 27.5% 1.2% + post_solution_log 1,286,411 15,785.47 0.01 77.1% 1.0% + post_solution_reporting 1,286,411 3,557.37 0.00 17.4% 0.2% + post_solution_perf_calc 1,286,411 594.29 0.00 2.9% 0.0% + post_solution_validation 1,286,411 440.22 0.00 2.2% 0.0% + post_solution_result_file 1,286,411 50.75 0.00 0.2% 0.0% + post_solution_sol_advance 1,286,411 38.20 0.00 0.2% 0.0% + post_solution_perf_reset 1,286,411 32.98 0.00 0.2% 0.0% + warmup_runs 2,458,060 20,311.11 0.01 27.3% 1.2% + pre_solution 1,286,411 14,388.58 0.01 19.3% 0.9% + pre_solution_listener.BenchmarkTimer 1,286,411 8,312.12 0.01 57.8% 0.5% + pre_solution_listener.ProgressListener 1,286,411 2,442.82 0.00 17.0% 0.1% + pre_solution_listener.AllSolutionsIterator 1,286,411 1,995.09 0.00 13.9% 0.1% + pre_solution_listener.MetaResultReporter 1,286,411 716.53 0.00 5.0% 0.0% + (untracked) 512.63 3.6% 0.0% + pre_solution_listener.SynchronizerValidator 1,286,411 225.16 0.00 1.6% 0.0% + pre_solution_listener.ReferenceValidator 1,286,411 78.52 0.00 0.5% 0.0% + (untracked) 46.92 59.8% 0.0% + pre_solution_ref_skipped 1,286,230 31.60 0.00 40.2% 0.0% + pre_solution_listener.DataInitialization 1,286,411 67.63 0.00 0.5% 0.0% + pre_solution_mx_check 1,286,411 40.76 0.00 60.3% 0.0% + (untracked) 26.87 39.7% 0.0% + pre_solution_listener.HardwareMonitorListener 1,286,411 38.08 0.00 0.3% 0.0% + kernel_solving 1,229,034 6,192.62 0.01 8.3% 0.4% + run_current_solution_check 1,286,411 5,748.15 0.00 7.7% 0.3% + post_problem 4,035 1,691.10 0.42 2.3% 0.1% + benchmark_runs 1,229,034 1,626.23 0.00 2.2% 0.1% + benchmark_validate_enqueues 605 1,042.13 1.72 64.1% 0.1% + validate_enqueues_listener.BenchmarkTimer 605 565.54 0.93 54.3% 0.0% + benchmark_sleep 604 564.95 0.94 99.9% 0.0% + benchmark_event_sync 219 34.58 0.16 6.1% 0.0% + benchmark_event_elapsed_time 219 0.32 0.00 0.1% 0.0% + benchmark_event_destroy 219 0.22 0.00 0.0% 0.0% + validate_enqueues_listener.ReferenceValidator 605 211.26 0.35 20.3% 0.0% + validate_enqueues_listener.HardwareMonitorListener 605 171.94 0.28 16.5% 0.0% + validate_enqueues_listener.SynchronizerValidator 605 85.09 0.14 8.2% 0.0% + validate_enqueues_listener.ProgressListener 605 7.37 0.01 0.7% 0.0% + (untracked) 0.87 0.1% 0.0% + validate_enqueues_listener.MetaResultReporter 605 0.04 0.00 0.0% 0.0% + validate_enqueues_listener.DataInitialization 605 0.02 0.00 0.0% 0.0% + validate_enqueues_listener.AllSolutionsIterator 605 0.02 0.00 0.0% 0.0% + gpu_kernel_execution 605 440.16 0.73 27.1% 0.0% + benchmark_post_syncs 1,229,034 92.03 0.00 5.7% 0.0% + benchmark_pre_syncs 1,229,034 63.70 0.00 3.9% 0.0% + benchmark_events_setup 605 0.78 0.00 0.0% 0.0% + benchmark_events_teardown 605 0.36 0.00 0.0% 0.0% + warmup_events_setup 1,229,034 1,398.48 0.00 1.9% 0.1% + run_iteration_teardown 1,229,034 1,365.25 0.00 1.8% 0.1% + solution_selection 1,286,411 790.82 0.00 1.1% 0.0% + finalize_report 441 375.26 0.85 0.5% 0.0% + solution_teardown 1,286,411 50.70 0.00 0.1% 0.0% + problem_report_index 4,035 8.11 0.00 0.0% 0.0% + problem_teardown 4,035 0.20 0.00 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------------------------- + (untracked) 2,908.43 0.2% 0.2% + -------------------------------------------------------------------------------------------------------------------------------------- + python_cache_check 441 836.07 1.90 0.1% 0.1% + -------------------------------------------------------------------------------------------------------------------------------------- + python_write_solutions 441 88.17 0.20 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_prepare 441 63.41 0.14 71.9% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_wsol_prepare_cache 441 61.49 0.14 97.0% 0.0% + (untracked) 1.93 3.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_header 441 19.71 0.04 22.4% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 5.05 5.7% 0.0% + ======================================================================================================================================== + python_library_logic 8 9,353.68 1,169.21 0.6% + -------------------------------------------------------------------------------------------------------------------------------------- + python_logic_write 8 8,282.05 1,035.26 88.5% 0.5% + -------------------------------------------------------------------------------------------------------------------------------------- + python_logic_parse_solutions 8 938.09 117.26 10.0% 0.1% + -------------------------------------------------------------------------------------------------------------------------------------- + python_logic_analyze 8 127.21 15.90 1.4% 0.0% + -------------------------------------------------------------------------------------------------------------------------------------- + (untracked) 6.33 0.1% 0.0% + ======================================================================================================================================== + python_client_writer 1 6,849.91 6,849.91 0.4% + -------------------------------------------------------------------------------------------------------------------------------------- + python_library_client_create_library 1 4,719.06 4,719.06 68.9% 0.3% + -------------------------------------------------------------------------------------------------------------------------------------- + python_library_client_execution 1 2,104.68 2,104.68 30.7% 0.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_library_client_subprocess 1 2,103.74 2,103.74 100.0% 0.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 0.78 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_library_client_write_run_script 1 0.16 0.16 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------------------------- + python_library_client_write_configs 1 23.56 23.56 0.3% 0.0% + -------------------------------------------------------------------------------------------------------------------------------------- + python_library_client_setup 1 1.54 1.54 0.0% 0.0% + -------------------------------------------------------------------------------------------------------------------------------------- + (untracked) 1.08 0.0% 0.0% + + Unknown Categories + ---------------------------------------------------------------------------------------------------------------------------------------- + icache_rotate_extra_copies_loading 1 5.72 5.72 0.0% + + Overlapping Totals (wrap or overlap other timers; not summed into the tree) + ---------------------------------------------------------------------------------------------------------------------------------------- + python_client_subprocess 441 1,624,103.51 3,682.77 98.8% + main_total 441 1,459,029.87 3,308.46 88.7% + problem_total 4,035 1,050,297.13 260.30 63.9% + solution_total 1,286,411 356,477.99 0.28 21.7% + run_iteration_total 1,229,034 313,383.40 0.25 19.1% + validate_total 1,037,703 96,753.31 0.09 5.9% + validate_check_results_typed 1,042,067 96,457.33 0.09 5.9% + +------------------------------------------------------------------------------------------------------------------------------------------ +HIERARCHY VALIDATION WARNINGS +------------------------------------------------------------------------------------------------------------------------------------------ + WARNING: Child phases of python_client_execution (1,626,543.75 ms) exceed parent (1,624,388.33 ms) by 2,155.42 ms (0.1%) + WARNING: Child phases of validate_enqueues_listener.BenchmarkTimer (600.07 ms) exceed parent (565.61 ms) by 34.47 ms (6.1%) + +TIME BREAKDOWN (visual, % of wall clock) +------------------------------------------------------------------------------------------------------------------------------------------ + TOTAL RUNTIME 100.0% ################################################## + RUN: benchmark client 93.3% ############################################## + Reference GEMM (CPU) 40.0% ################### + cpu_reference_gemm 40.0% ################### + solve_cpu_slow 38.0% ################## + solve_cpu_fast 2.0% + (untracked) 0.0% + pre_solution_listener.ReferenceValidator 0.0% + pre_solution_ref_skipped 0.0% + (untracked) 0.0% + Client launch (startup, setup & teardown) 28.7% ############## + listener_setup 10.9% ##### + hip_initialization 10.2% ##### + process_exit 3.3% # + data_init_setup 1.1% + library_loading 1.1% + code_object_loading 0.9% + main_teardown_locals 0.8% + process_startup 0.5% + lazy_loading_init 0.0% + main_parse_args 0.0% + main_problem_factory 0.0% + (untracked) 0.0% + reporter_setup 0.0% + python_client_write_run_script 0.0% + solution_iterator_setup 0.0% + Validation 11.0% ##### + validate_warmups_listener.ReferenceValidator 5.9% ## + validate_gpu_readback 3.6% # + validate_element_comparison 2.0% + validate_result_buffer_alloc 0.2% + validate_inputs_copy 0.0% + validate_setup 0.0% + validate_report 0.0% + (untracked) 0.0% + validate_warmups_listener.SynchronizerValidator 5.1% ## + validate_synchronizer_check 5.1% ## + validate_synchronizer_scan 3.4% # + validate_synchronizer_readback 1.7% + validate_synchronizer_inactive 0.0% + (untracked) 0.0% + validate_warmups_listener.MetaResultReporter 0.0% + validate_warmups_listener.AllSolutionsIterator 0.0% + validate_warmups_listener.HardwareMonitorListener 0.0% + validate_warmups_listener.ProgressListener 0.0% + validate_warmups_listener.DataInitialization 0.0% + Input data preparation 7.8% ### + gpu_input_reset 5.7% ## + gpu_input_reset_copy_valid_to_gpu 2.4% # + gpu_input_reset_batched_inputs 2.2% # + gpu_input_reset_copy_inputs_d2d 0.7% + gpu_input_reset_copy_swizzled 0.2% + gpu_input_reset_fast_reset_output 0.1% + gpu_input_reset_convert_to_inputs 0.0% + gpu_input_reset_slow_path 0.0% + gpu_input_reset_rotating_init 0.0% + gpu_input_reset_constant_inputs 0.0% + gpu_input_reset_init_cpu_inputs 0.0% + (untracked) 0.0% + cpu_data_init 1.4% + gpu_input_preparation 0.7% + gpu_input_prep_copy_valid_to_gpu 0.5% + gpu_input_prep_init_cpu_inputs 0.1% + gpu_input_prep_copy_swizzled 0.1% + gpu_input_prep_batched_inputs 0.0% + gpu_input_prep_copy_inputs_d2d 0.0% + gpu_input_prep_fast_reset_output 0.0% + gpu_input_prep_convert_to_inputs 0.0% + gpu_input_prep_slow_path 0.0% + gpu_input_prep_constant_inputs 0.0% + gpu_input_prep_rotating_init 0.0% + (untracked) 0.0% + rotating_buffer_preparation 0.0% + pre_solution_listener.DataInitialization 0.0% + pre_solution_mx_check 0.0% + (untracked) 0.0% + Per-solution bookkeeping & reporting 2.7% # + post_solution_log 1.0% + pre_solution_listener.BenchmarkTimer 0.5% + run_current_solution_check 0.3% + post_solution_reporting 0.2% + pre_solution_listener.ProgressListener 0.1% + pre_solution_listener.AllSolutionsIterator 0.1% + post_problem 0.1% + run_iteration_teardown 0.1% + solution_selection 0.0% + pre_solution_listener.MetaResultReporter 0.0% + post_solution_perf_calc 0.0% + post_solution_validation 0.0% + finalize_report 0.0% + pre_solution_listener.SynchronizerValidator 0.0% + post_solution_result_file 0.0% + solution_teardown 0.0% + post_solution_sol_advance 0.0% + pre_solution_listener.HardwareMonitorListener 0.0% + post_solution_perf_reset 0.0% + problem_report_index 0.0% + problem_teardown 0.0% + Kernel launch & execution 2.2% # + warmup_runs 1.2% + kernel_solving 0.4% + validate_warmups_listener.BenchmarkTimer 0.4% + validate_gpu_sync 0.4% + (untracked) 0.0% + benchmark_runs 0.1% + benchmark_validate_enqueues 0.1% + validate_enqueues_listener.BenchmarkTimer 0.0% + benchmark_sleep 0.0% + benchmark_event_sync 0.0% + benchmark_event_elapsed_time 0.0% + benchmark_event_destroy 0.0% + validate_enqueues_listener.ReferenceValidator 0.0% + validate_enqueues_listener.HardwareMonitorListener 0.0% + validate_enqueues_listener.SynchronizerValidator 0.0% + validate_enqueues_listener.ProgressListener 0.0% + validate_enqueues_listener.MetaResultReporter 0.0% + validate_enqueues_listener.DataInitialization 0.0% + validate_enqueues_listener.AllSolutionsIterator 0.0% + (untracked) 0.0% + gpu_kernel_execution 0.0% + benchmark_post_syncs 0.0% + benchmark_pre_syncs 0.0% + benchmark_events_setup 0.0% + benchmark_events_teardown 0.0% + warmup_events_setup 0.1% + Library logic (Python) 0.6% + python_logic_write 0.5% + python_logic_parse_solutions 0.1% + python_logic_analyze 0.0% + Library client (Python) 0.4% + python_library_client_create_library 0.3% + python_library_client_execution 0.1% + python_library_client_subprocess 0.1% + python_library_client_write_run_script 0.0% + (untracked) 0.0% + python_library_client_write_configs 0.0% + python_library_client_setup 0.0% + MEASUREMENT OVERHEAD (instrumentation) 6.6% ### + flush_timing_buffer 5.5% ## + main_timing_calibration 0.7% + timing_overhead 0.2% + (timer overhead subtracted from parent timers) 0.2% + SHARED: cache & solution files 0.1% + python_cache_check 0.1% + python_write_solutions 0.0% + python_wsol_prepare 0.0% + python_wsol_prepare_cache 0.0% + (untracked) 0.0% + python_wsol_header 0.0% + (untracked) 0.0% + UNATTRIBUTED (untracked time & timer overhead) 0.1% + (untracked in python_benchmark_problems) 0.2% + (untracked in validate_warmups) 0.0% + (untracked in pre_solution) 0.0% + (untracked in pre_problem) 0.0% + (untracked in python_library_logic) 0.0% + (untracked in python_client_writer) 0.0% + (children exceed benchmark_runs) -0.0% + (children exceed post_solution) -0.0% + (children exceed validate_enqueues_listener.BenchmarkTimer) -0.0% + (children exceed validate_warmups_listener.SynchronizerValidator) -0.0% + (children exceed python_client_execution) -0.2% + +KEY INSIGHTS (% of wall clock) +------------------------------------------------------------------------------------------------------------------------------------------ + Reference GEMM (CPU) 657.03s (40.0%) + Client launch (startup, setup & teardown) 472.56s (28.7%) + Validation 180.75s (11.0%) + Input data preparation 128.32s (7.8%) + MEASUREMENT OVERHEAD (instrumentation) 104.59s (6.4%) + Per-solution bookkeeping & reporting 44.26s (2.7%) + Kernel launch & execution 35.44s (2.2%) + Library logic (Python) 9.35s (0.6%) + Library client (Python) 6.85s (0.4%) + SHARED: cache & solution files 0.92s (0.1%) + UNATTRIBUTED (untracked time & timer overhead) 0.89s (0.1%) + + CPU reference / GPU execution ratio: 1492.5x + >>> CPU reference > 10x GPU execution time! + +TOP 10 SLOWEST PROBLEMS (by CPU reference time) +------------------------------------------------------------------------------------------------------------------------------------------ + M N K Batch TypeA TypeD CPU Ref (ms) GPU (ms) + ---------------------------------------------------------------------------------------------------------------------------------------- + 4111 4111 4111 1 Float8_fnuz Float8_fnuz 233400.59 0.00 + 4111 4111 4111 1 Float8_fnuz Float8_fnuz 221845.69 0.00 + 1024 1024 1024 1 Float8_fnuz Float8_fnuz 7128.91 0.00 + 1024 1024 1024 1 Float8_fnuz Float8_fnuz 6008.58 0.00 + 2160 10240 16384 1 BFloat16 BFloat16 4751.19 0.00 + 10240 2160 16384 1 BFloat16 BFloat16 3400.16 0.00 + 4096 4096 4096 1 Half Half 1175.05 0.00 + 512 256 511 1 Float8_fnuz Float 903.25 0.00 + 4111 4111 4111 1 Float8_fnuz Half 884.17 0.00 + 511 256 1024 1 Half Half 840.64 0.00 + diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/run_comparison.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/run_comparison.txt new file mode 100644 index 000000000000..870aaaae98ef --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/run_comparison.txt @@ -0,0 +1,85 @@ +Category pre s post s delta +TOTAL RUNTIME 1644.4 828.7 -49.6% +RUN: benchmark client 1534.6 718.9 -53.2% + Reference GEMM (CPU) 657.0 38.4 -94.2% + cpu_reference_gemm 657.0 38.3 -94.2% + solve_cpu_slow 624.5 0.9 -99.9% + solve_cpu_fast 32.4 37.4 +15.4% + Client launch (startup, setup & teardown) 472.6 468.4 -0.9% + listener_setup 178.6 179.3 +0.4% + hip_initialization 167.2 167.9 +0.4% + process_exit 54.0 54.7 +1.4% + data_init_setup 18.8 15.1 -19.6% + library_loading 17.4 17.2 -0.7% + code_object_loading 14.2 13.1 -7.9% + main_teardown_locals 13.1 11.3 -13.3% + process_startup 8.2 8.6 +5.3% + Validation 180.8 66.8 -63.1% + validate_warmups_listener.ReferenceValidator 96.9 61.0 -37.1% + validate_gpu_readback 59.9 24.9 -58.4% + validate_element_comparison 32.2 32.4 +0.4% + validate_result_buffer_alloc 3.9 3.3 -16.5% + validate_warmups_listener.SynchronizerValidator 83.6 0.0 -100.0% + validate_synchronizer_check 83.6 0.0 -100.0% + validate_synchronizer_scan 56.0 0.0 -100.0% + validate_synchronizer_readback 27.4 0.0 -100.0% + Input data preparation 128.3 53.7 -58.1% + gpu_input_reset 93.2 31.7 -66.0% + gpu_input_reset_copy_valid_to_gpu 39.1 0.1 -99.8% + gpu_input_reset_batched_inputs 36.4 19.7 -45.9% + gpu_input_reset_copy_inputs_d2d 11.4 10.0 -12.3% + gpu_input_reset_copy_swizzled 3.9 0.0 -99.9% + gpu_input_reset_fast_reset_output 2.0 1.6 -19.1% + cpu_data_init 22.9 15.1 -34.0% + gpu_input_preparation 11.6 6.8 -41.9% + gpu_input_prep_copy_valid_to_gpu 8.6 4.9 -42.3% + gpu_input_prep_init_cpu_inputs 1.7 0.6 -66.6% + gpu_input_prep_copy_swizzled 1.0 1.0 -3.5% + Per-solution bookkeeping & reporting 44.3 9.3 -79.1% + post_solution_log 15.8 1.1 -93.3% + pre_solution_listener.BenchmarkTimer 8.3 1.5 -82.4% + run_current_solution_check 5.7 1.6 -72.3% + post_solution_reporting 3.6 1.5 -58.7% + pre_solution_listener.ProgressListener 2.4 0.6 -77.2% + pre_solution_listener.AllSolutionsIterator 2.0 1.0 -50.8% + post_problem 1.7 0.1 -96.1% + run_iteration_teardown 1.4 0.2 -85.7% + Kernel launch & execution 35.4 67.8 +91.3% + warmup_runs 20.3 11.5 -43.5% + kernel_solving 6.2 3.4 -45.7% + validate_warmups_listener.BenchmarkTimer 5.9 0.0 -99.4% + validate_gpu_sync 5.8 0.0 -100.0% + benchmark_runs 1.7 1.2 -27.9% + benchmark_validate_enqueues 1.1 0.8 -25.6% + warmup_events_setup 1.4 0.6 -56.3% + Library logic (Python) 9.3 8.4 -10.4% + python_logic_write 8.3 7.3 -12.1% + Library client (Python) 6.8 6.2 -9.3% + python_library_client_create_library 4.7 4.1 -12.5% + python_library_client_execution 2.1 2.1 -1.6% + python_library_client_subprocess 2.1 2.1 -1.6% +MEASUREMENT OVERHEAD (instrumentation) 108.1 108.0 -0.1% + flush_timing_buffer 89.8 89.8 -0.0% + main_timing_calibration 11.0 11.0 -0.4% + timing_overhead 3.7 3.7 -0.4% + (timer overhead subtracted from parent timers) 3.5 3.5 -0.5% + (untracked in python_benchmark_problems) 2.9 3.2 +10.0% + warmup_first_sync 0.0 50.7 new + synchronizer_check 0.0 5.5 new + synchronizer_reduce 0.0 5.4 new + +pytest wall sum: pre 2320 s, post 1484 s (-36.0%) +largest per-config (pre -> post s): + common__streamk__sk_f8ngemm_quick 490 -> 21 + common__gemm__fp16_tn 127 -> 70 + common__gemm__bufferLoadStore0 119 -> 98 + common__gemm__dtv 88 -> 58 + common__gemm__cgemm 45 -> 32 + common__gemm__zgemm 41 -> 27 + common__gemm__mix_cvt_after_ds_fnuz 36 -> 29 + common__streamk__sk_sgemm_quick 30 -> 23 + common__gemm__bf16_tn 27 -> 18 + common__gemm__fp32_tn 27 -> 17 + common__gemm__fp8nfp16mix_hfp8ns 26 -> 22 + common__streamk__sk_dgemm_quick 24 -> 17 +slower by >2 s: [] diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_post/times.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_post/times.txt new file mode 100644 index 000000000000..6b201d8216f7 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_post/times.txt @@ -0,0 +1,6 @@ +START build 2026-11-04T00:21:39Z +END build rc=1 seconds=634 2026-11-04T00:32:13Z +380M /home/alvasile/tl_gfx942_timing/post/split_art +START run 2026-11-04T00:32:14Z +END run rc=0 seconds=742 2026-11-04T00:44:36Z +ALLDONE diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_pre/configs.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_pre/configs.txt new file mode 100644 index 000000000000..30203ebfcf49 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_pre/configs.txt @@ -0,0 +1,123 @@ +tensilelite/Tests/common/amaxd/8rn_amaxd.yaml +tensilelite/Tests/common/benchmark_runs/fp32_tn.yaml +tensilelite/Tests/common/client/rotate_mode0.yaml +tensilelite/Tests/common/client/rotate_mode1.yaml +tensilelite/Tests/common/exception/f16_cf32_nan.yaml +tensilelite/Tests/common/gemm/8rn_b_s.yaml +tensilelite/Tests/common/gemm/ag.yaml +tensilelite/Tests/common/gemm/agntab_coverage_gfx942.yaml +tensilelite/Tests/common/gemm/bf16_tn.yaml +tensilelite/Tests/common/gemm/bf16_tn_predict.yaml +tensilelite/Tests/common/gemm/bufferLoadStore0.yaml +tensilelite/Tests/common/gemm/cgemm.yaml +tensilelite/Tests/common/gemm/dot2_gfx942.yaml +tensilelite/Tests/common/gemm/dtl.yaml +tensilelite/Tests/common/gemm/dtv.yaml +tensilelite/Tests/common/gemm/f64.yaml +tensilelite/Tests/common/gemm/fp16_HH_BHS_bf16mfma.yaml +tensilelite/Tests/common/gemm/fp16_tn.yaml +tensilelite/Tests/common/gemm/fp16_use_e.yaml +tensilelite/Tests/common/gemm/fp16fp32mix.yaml +tensilelite/Tests/common/gemm/fp32_nt.yaml +tensilelite/Tests/common/gemm/fp32_scalealphavec.yaml +tensilelite/Tests/common/gemm/fp32_tn.yaml +tensilelite/Tests/common/gemm/fp8bf8nss.yaml +tensilelite/Tests/common/gemm/fp8n.yaml +tensilelite/Tests/common/gemm/fp8n_rowwise.yaml +tensilelite/Tests/common/gemm/fp8n_use_e.yaml +tensilelite/Tests/common/gemm/fp8nfp16mix_fp8nss.yaml +tensilelite/Tests/common/gemm/fp8nfp16mix_hfp8ns.yaml +tensilelite/Tests/common/gemm/fp8nfp16mix_hhs.yaml +tensilelite/Tests/common/gemm/fp8nfp16mix_hhs_lsu4.yaml +tensilelite/Tests/common/gemm/gemm_bias_strided_batched_broadcast_gfx940.yaml +tensilelite/Tests/common/gemm/gemm_bias_strided_batched_gfx940.yaml +tensilelite/Tests/common/gemm/gemm_ck_gfx942.yaml +tensilelite/Tests/common/gemm/gsuasb.yaml +tensilelite/Tests/common/gemm/gsuc.yaml +tensilelite/Tests/common/gemm/hh_f8nhs.yaml +tensilelite/Tests/common/gemm/i8_gsu_gfx940.yaml +tensilelite/Tests/common/gemm/icache_flush.yaml +tensilelite/Tests/common/gemm/largeMT.yaml +tensilelite/Tests/common/gemm/large_size.yaml +tensilelite/Tests/common/gemm/lsu_fnuz.yaml +tensilelite/Tests/common/gemm/lsu_i8.yaml +tensilelite/Tests/common/gemm/mbskPrefetchOpt.yaml +tensilelite/Tests/common/gemm/mix_cvt_after_ds_fnuz.yaml +tensilelite/Tests/common/gemm/nan_bounds_check_odd_padding.yaml +tensilelite/Tests/common/gemm/no_shadowinit.yaml +tensilelite/Tests/common/gemm/sia0_pgr.yaml +tensilelite/Tests/common/gemm/swizzleA.yaml +tensilelite/Tests/common/gemm/swizzleB.yaml +tensilelite/Tests/common/gemm/tail_small_size.yaml +tensilelite/Tests/common/gemm/ulsgro1.yaml +tensilelite/Tests/common/gemm/use_beta_false.yaml +tensilelite/Tests/common/gemm/wgm.yaml +tensilelite/Tests/common/gemm/xfp32.yaml +tensilelite/Tests/common/gemm/zgemm.yaml +tensilelite/Tests/common/gradient/fp16_gradient_activation.yaml +tensilelite/Tests/common/gradient/fp16_gradient_bias.yaml +tensilelite/Tests/common/gradient/fp8bf8nss_gradient_bias_b.yaml +tensilelite/Tests/common/groupedgemm/grouped_gemm.yaml +tensilelite/Tests/common/groupedgemm/grouped_gemm_ck_gfx942.yaml +tensilelite/Tests/common/groupedgemm/grouped_gemm_userargs.yaml +tensilelite/Tests/common/gsu/auto_gsu.yaml +tensilelite/Tests/common/gsu/auto_gsu_zero_dim.yaml +tensilelite/Tests/common/gsu/f32_gsu.yaml +tensilelite/Tests/common/gsu/gsu_mbsk.yaml +tensilelite/Tests/common/sgemm_xf32_asm.yaml +tensilelite/Tests/common/sparse/gfx94x/bf16_activation.yaml +tensilelite/Tests/common/sparse/gfx94x/bf16_gate_r.yaml +tensilelite/Tests/common/sparse/gfx94x/f8n_gate_r.yaml +tensilelite/Tests/common/sparse/gfx94x/fp16_activation.yaml +tensilelite/Tests/common/sparse/gfx94x/fp16_gate_r.yaml +tensilelite/Tests/common/sparse/gfx94x/i8_activation.yaml +tensilelite/Tests/common/sparse/gfx94x/libray_logic.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_b8f8n.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_b8f8n_sb.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_bf16.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_bf16_sb.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_bf8n.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_bf8n_sb.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_f8b8n.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_f8b8n_sb.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_f8n.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_f8n_sb.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi16.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi16_sb.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi32.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi32_sb.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_gsuc.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi16.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi16_sb.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi32.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi32_sb.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_i8bs.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_i8bs_sb.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_i8hs.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_i8hs_sb.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_i8is.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_i8is_sb.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_vw_lg_one.yaml +tensilelite/Tests/common/sparse/gfx94x/spmm_vw_lg_one_sb.yaml +tensilelite/Tests/common/sparse/gfx94x/use_sgpr_for_gro.yaml +tensilelite/Tests/common/sparse/gfx94x/use_sgpr_for_gro_sb.yaml +tensilelite/Tests/common/streamk/data_parallel_static_sgemm.yaml +tensilelite/Tests/common/streamk/sk_bgemm_quick.yaml +tensilelite/Tests/common/streamk/sk_cgemm_quick.yaml +tensilelite/Tests/common/streamk/sk_dgemm_quick.yaml +tensilelite/Tests/common/streamk/sk_dynamic.yaml +tensilelite/Tests/common/streamk/sk_dynamic_work_stealing.yaml +tensilelite/Tests/common/streamk/sk_f8ngemm_quick.yaml +tensilelite/Tests/common/streamk/sk_hgemm_epilogues.yaml +tensilelite/Tests/common/streamk/sk_hgemm_lsu.yaml +tensilelite/Tests/common/streamk/sk_hgemm_quick.yaml +tensilelite/Tests/common/streamk/sk_hgemm_race.yaml +tensilelite/Tests/common/streamk/sk_hybrid.yaml +tensilelite/Tests/common/streamk/sk_hybrid_work_stealing.yaml +tensilelite/Tests/common/streamk/sk_queue_selfclean.yaml +tensilelite/Tests/common/streamk/sk_sgemm_alpha0.yaml +tensilelite/Tests/common/streamk/sk_sgemm_quick.yaml +tensilelite/Tests/common/streamk/sk_zgemm_quick.yaml +tensilelite/Tests/custom/custom_ck.yaml +tensilelite/Tests/custom/custom_rr.yaml +tensilelite/Tests/custom/custom_tensile_sk.yaml diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_pre/times.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_pre/times.txt new file mode 100644 index 000000000000..df9e42f563b3 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_pre/times.txt @@ -0,0 +1,6 @@ +START build 2026-11-04T00:21:39Z +END build rc=1 seconds=1028 2026-11-04T00:38:47Z +398M /home/alvasile/tl_gfx942_timing/pre/split_art +START run 2026-11-04T00:38:48Z +END run rc=0 seconds=1494 2026-11-04T01:03:42Z +ALLDONE diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_build.md b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_build.md new file mode 100644 index 000000000000..2229080daa92 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_build.md @@ -0,0 +1,269 @@ +# gfx950 build-side measurement after the build-side changes (P6) + +Build-side measurement at the end of P4 of the implementation plan (`timing_runs/DECISIONS.md`, +P6), compared with the P0.2 baseline (`baseline.md`). It covers the build-side changes +P4.1–P4.15 and the fix of a regression this measurement found (see "Regression found"). + +## What was measured + +| Item | Value | +|---|---| +| Commit | `9f493d7a6c1` (end of P4 plus the custom-kernel fix; detached worktree). The first 25 per-config pairs and one supplementary split triple ran at `38c29661a7d` (end of P4), see "Regression found". | +| Client | P3's `tensilelite-client` (built at `3b4ec8e125d`, sha256 `15f20fa8a2eb…`). No client, library or CMake source changed between `3b4ec8e125d` and `9f493d7a6c1` (P4 and the fix are Python, YAML and `measure_env.sh` only). The client only runs during the build phase in ductile configs. | +| Baseline | P0.2's worktree (`2083e6254bf`) and client (sha256 `4b2769f7f9a2…`), for the same-session control runs | +| Python | venv Python 3.12.3; TensileLite Python sources from each worktree; venv `rocisa` (unchanged) | +| Host | `smci355-1b112-b13-15` (as P0.2): 2× EPYC 9575F, 8× MI355X (gfx950), ROCm 7.2.1, `/usr/bin/amdclang++` | +| Cores / threads | `taskset -c 0-24`, `OMP_NUM_THREADS=25`, `OPENBLAS_NUM_THREADS=25` (as P0.2) | +| GPU | An idle MI355X per pair or split run: `ROCR_VISIBLE_DEVICES=5` (PCI `85:00.0`) or `0` (PCI `75:00.0`, the P0.2 GPU). The build phase uses it only for the ductile configs' client runs. | +| Config list | Per-config pass: the 185 configs of P3 (P0.2's 184 plus `gemm/gfx950/prediction_bf16`, new in P2.5). Split: all of `Tests/common` with the gfx950 markers, as `split/run_split.sh`. | +| Date | 2026-11-01, 14:25–17:49 UTC | + +Committed results: + +- `after_build/AGGREGATE_analysis.txt`: per-config build aggregate (cold cache per config) at + `9f493d7a6c1`/`38c29661a7d` (see above), 185 logs. +- `after_build/split_build_outcomes.txt`: outcome of every test in the split build phase at + `9f493d7a6c1`. + +Raw logs (not committed, as for P0.2/P3) are in `/home/alvasile/timing_runs_raw/P6/`, gzipped: +per-config logs of both sides, every split build log (with captured Tensile output), the +same-session P0.2 control aggregate, monitoring logs and the driver/analysis scripts (`scripts/`). + +## How it was measured + +`baseline.md` asks P6 to re-run the P0.2 build pass and split build phase back to back with its +own, under identical conditions, because both P0.2 build numbers overlapped another user's job. +As in P3, every comparison here is same-session: + +- **Per-config pass (interleaved).** For each config, P0.2 (its worktree and client) and P6 ran + back to back on the same idle GPU, alternating which went first, with + `run_per_config_build.sh`'s settings (serial, `TimingInstrumentation=True`, caches cleared + before every config, 25 pinned cores). The script copy used for both sides differs only in + setting `ROCR_VISIBLE_DEVICES` to the pair's GPU and in skipping the per-call aggregate. A pair + was re-run (up to three tries) if another user's process took more than 1 GB on the GPU or + more than 10 CPU-seconds of pinned work on cores 0–24 during either run. +- **Split build phase** (`--build-only`, `-n 4`, cache emptied at the start and kept across + configs, the CI-like build): the same command as `split/run_split.sh`'s build phase, plus `-rP` + so that the captured Tensile output of passed tests (the cache lines counted below) is in the + log. Three runs per triple, back to back: P0.2, P6, and P6 with + `TENSILE_DISABLE_HELPER_CHUNK_CACHE=1`. At `9f493d7a6c1` there are two triples, in opposite + order (C–B–A at 16:51, A–B–C at 17:20). +- Pytest temp dirs (`TMPDIR`), caches, scratch artifacts and output were on `/dev/shm` for both + sides, because `/` was nearly full. + +Conditions: no CPU from other users' jobs pinned into cores 0–24 in any run (no corpus job +there today; sampled every 30 s). The 1-minute load average was 17–40, which includes the +measurement's own 25 cores. GPUs: the pairs ran on `ROCR` 5 (141 pairs) and 0 (43 pairs). One +pair, `gemm/zgemm`, had another process on the GPU in all three tries and is kept (it builds no +client run, so the GPU is idle in it). In the last four split runs, on `ROCR` 0, idle root +processes held 0.5 GB of VRAM and briefly up to 16 GB, which the monitor flagged; the build does +not use the GPU outside the ductile configs, and those runs match their `ROCR` 5 counterparts. + +## Regression found: custom-kernel configs failed to build at `38c29661a7d` + +The first split build at `38c29661a7d` had **8 new build failures**: +`gemm/gfx950/custom_xfp32` and `custom/{custom_aiter_bf16, custom_aiter_f4, +custom_rocblas_gemv_bf16_tn_m1, _m2, _m4, custom_triton_f4, custom_wave_bf16}`. They all failed +with `KeyError: 'LdsNumBytes'` in `TensileCreateLibrary/Run.py` `processKernelSource`. P4.7 +(`168da253f63`) and P4.8 (`66281bc369e`) read `LdsNumBytes`, `LoopIters` and `LdsOffsetBias` +from every kernel after codegen, and custom kernels (assembly read from a file) never have those +keys. Neither step's verification built a custom-kernel config. The fix, `9f493d7a6c1` +(`fix(tensilelite): do not require LdsNumBytes / LoopIters / LdsOffsetBias on custom kernels`), +reads them with `.get()` and copies only keys that exist. + +At `9f493d7a6c1` every split build phase gives 196 passed, 2 failed, 370 skipped (4 runs, +identical lists). That differs from P0.2's build phase only by the new `prediction_bf16` +(PASSED). The 2 failures are the known `custom_rr_f6`/`custom_rr_mx` ("No valid solutions +found"). + +The worktree was moved to `9f493d7a6c1` between per-config pairs 25 and 26 (15:12:01 UTC). +Pairs 1–25 were measured at `38c29661a7d` and kept: they include none of the 8 configs (those are +pairs 31 and 178–184, all measured at `9f493d7a6c1`). For kernels that have the three keys, which +is every kernel of a config that built at `38c29661a7d`, the fix runs the same code. The split +triple at `38c29661a7d` is reported only as a supplement, since 8 of its configs stopped early. + +## Split build phase (`-n 4`, cache kept across configs, uninstrumented) + +| Run (UTC start, GPU) | Commit | Wall | Δ vs P0.2 (same triple) | Sum of test call times | +|---|---|---:|---:|---:| +| P6, chunk cache off (16:51, `ROCR` 5) | `9f493d7a6c1` | 539 s | −12.8% | 1,922 s | +| P6, chunk cache on (17:00, `ROCR` 5) | `9f493d7a6c1` | 546 s | −11.7% | 1,959 s | +| P0.2 (17:09, `ROCR` 0) | `2083e6254bf` | 618 s | | 2,428 s | +| P0.2 (17:20, `ROCR` 0) | `2083e6254bf` | 615 s | | 2,407 s | +| P6, chunk cache on (17:31, `ROCR` 0) | `9f493d7a6c1` | 525 s | −14.6% | 1,924 s | +| P6, chunk cache off (17:40, `ROCR` 0) | `9f493d7a6c1` | 511 s | −16.9% | 1,862 s | +| *Supplement:* P0.2 / P6 on / P6 off (14:25, `ROCR` 5) | `38c29661a7d` | 620 / 502 / 484 s | | 2,424 / 1,754 / 1,722 s | + +- **Wall time: P0.2 616.5 s (mean of 2), P6 535.5 s with the chunk cache (−13.1%), 525 s + without it (−14.8%).** P0.2's two runs agree within 0.5% and with the 620 s at 14:25, so + this session's P0.2 build phase is 615–620 s. The 634 s in `baseline.md` was measured under + contention, about 3% higher. +- The summed test call times fall more (−20%) than the wall time. With `-n 4` the wall time is + bounded by how the 4 workers' queues end, and P6 sizes each worker's pools to the P4.3 budget + (2 × 25 / 4 = 12 processes: 10 codegen workers plus 2 helper-compile chunks). P0.2 ran up to 25 + processes per worker. In the same-session runs at `9f493d7a6c1`, `dtl` takes 271–301 s under + `-n 4` with P6 against 190–191 s with P0.2. With P6 it still ends at 63–71% of the run's wall + time (P0.2: about 82%), so it is not on P6's critical path. Per-worker busy time ranges from + about 430 to 542 s with P6, against 603–612 s with P0.2 (`split_p0_2` and the P6 runs, from + `build.xml` durations in worker order). +- **Chunk cache on versus off: no measurable benefit.** In both orders the run without the + chunk cache was faster, by 7 s and 14 s (1–3%). That is within the session's drift (the same + configuration differed by 21–28 s between the triples). +- The two known failing configs now get further before failing (P4.10 prepares every step + first), at a cost of 1–2 s of test time. + +### Cache hit rates (CI-like run, empty cache at start) + +Counted from the Tensile output in the split logs (`scripts/split_stats.py`), P6 at +`9f493d7a6c1`, identical in all four runs except where noted: + +| Cache | Lookups | Hits | Rate | Notes | +|---|---:|---:|---:|---| +| Whole-file helper cache (`HelperKernelCache`) | 927 | 820 | 88.5% | P0.2: 753 of 859 (87.7%). P6 does 67 more lookups, all hits, in the two failing `custom_rr_*` configs, which get further now (see above), and 1 (a miss) in the new `prediction_bf16`. That miss is also why there are 107 PCH lookups instead of 106. | +| HIP-header PCH (P4.12) | 107 (one per whole-file miss) | 103–104 | 96–97% | 3–4 PCH builds: up to 3 xdist workers built the shared key at the same time at the start (each stores it once, by rename), and `custom_rr_mx` uses its own key. 2 entries, 26 MB. | +| Helper chunk cache (P4.15) | 182 chunks in 91 chunked compiles | 21 | **11.5%** | 159 entries, 24 MB. 0 key-computation failures, so FOLLOWUPS P4.15's caveat (a failed key computation still prints "0 of n") did not affect the count. 0 fallbacks to the one-piece compile. | + +The chunk cache only gets lookups that the whole-file cache missed, and those are mostly helper +sets no earlier config had. The 21 hits are spread over 18 configs, 1–3 chunks each. +`subtile_bf16` hits 3 of 8. Under `-n 4` each helper compile uses 2 chunks, while a serial +build uses 4. Chunk contents depend on the budget (FOLLOWUPS P4.14), so a cache filled by one +of these builds does not serve the other. + +In the per-config pass (cache cleared before every config), P6 built the PCH once per config +(184 builds, 79 hits by later helper compiles of the same config). 14 of 719 chunks hit (1.9%). + +## Per-config pass (cold cache per config, instrumented) + +Pytest wall seconds summed over the 184 configs common to both sides. "P0.2" is the committed +`baseline_build` pass. "Control" is P0.2 in the interleaved pass. + +| Set | Configs | P0.2 | Control | P6 | Δ vs control | +|---|---:|---:|---:|---:|---:| +| All | 184 | 3,361 | 3,359 | 1,970 | **−41.4%** | +| … `LibraryLogic:`/`LibraryClient:` removed or added (P2.5/P2.6) | 25 | 259 | 257 | 178 | −30.7% | +| … all others | 159 | 3,102 | 3,102 | 1,792 | −42.2% | +| … `CpuThreads: 1` removed (P4.8) | 19 | 526 | 537 | 195 | −63.7% | +| … the 8 configs fixed by `9f493d7a6c1` | 8 | 57 | 57 | 40 | −29.8% | +| … pairs 1–25 (measured at `38c29661a7d`) | 25 | 600 | 595 | 430 | −27.7% | + +`prediction_bf16` (P6 only) adds 5 s and is in the committed aggregate only. + +**The control reproduces the committed P0.2 build pass**: pytest sums 3,359 against 3,361 s +(−0.1%), TOTAL RUNTIME 2,662.4 against 2,654.8 s (+0.3%). The core-9 contention during the +P0.2 pass therefore did not inflate it measurably, and comparisons against the committed +baseline are equally valid. + +Seconds from "TIMING BY PURPOSE" over the same 184 configs. P0.2 is re-aggregated with the +analyzer at `9f493d7a6c1`. This only moves the helper compile, see the note below the table. + +| Category | P0.2 | Control | P6 | Δ vs control | +|---|---:|---:|---:|---:| +| TOTAL RUNTIME | 2,654.8 | 2,662.4 | 1,284.1 | −51.8% | +| BUILD: solution generation & compile | 2,605.6 | 2,613.0 | 1,238.9 | −52.6% | +| … `python_kernel_compilation` | 2,260.4 | 2,273.7 | 903.2 | −60.3% | +| … … helper compile in the build's path (P0.2: serial; P6: `python_kernel_build_src_co_wait`) | 1,013.4 | 1,023.6 | 138.1 | −86.5% | +| … … `python_kernel_codegen` (calls) | 615.5 (855) | 607.4 (855) | 301.2 (213) | −50.4% | +| … … `python_kernel_write_assemble` | 253.8 | 264.1 | 236.0 | −10.6% | +| … … `python_kernel_bench_setup` | 132.2 | 132.6 | 4.5 | −96.6% | +| … … `python_kernel_bench_postprocess` | 105.2 | 103.9 | 106.0 | +2.0% | +| … … `python_kernel_validate` | 82.6 | 80.9 | 58.3 | −27.9% | +| … … `python_kernel_build_co` | 22.1 | 25.9 | 25.7 | −0.8% | +| … `python_solution_generation` | 321.6 | 319.0 | 314.2 | −1.5% | +| … Ductile GA search (Python) | 23.7 | 20.2 | 21.5 | +6.4% | +| SHARED: cache & solution files | 23.2 | 23.2 | 23.4 | +0.9% | +| RUN: benchmark client | 14.3 | 14.9 | 10.0 | −32.9% | +| Helper compile itself (`python_kernel_build_src_co`, overlapping total) | 1,013.4 | 1,023.6 | 393.0 | −61.6% | +| … `.compile` | 1,010.5 | 1,020.9 | 286.9 | −71.9% | +| … `.pch` (new; PCH built once per config here) | – | – | 98.7 | | +| Sum of per-config pytest wall times | 3,361 | 3,359 | 1,970 | −41.4% | + +Note on the helper compile: since P4.9 it runs in a background thread during codegen. The tree +therefore shows only the time the build then waits for it (`python_kernel_build_src_co_wait`), +and the compile itself is listed under "Overlapping Totals". In P0.2's logs it ran serially and +appears as untracked time in `python_kernel_compilation` (1,017.2 s in P0.2, 1,027.1 s in the +control). + +What the numbers show, by plan item: + +- **Helper compile (topic 6, P4.9 and P4.11–P4.15): 1,024 s → 138 s in the build's path.** Its + own time falls to 393 s: the bare-arch compile (P4.11), the PCH (P4.12), and chunks compiled in + parallel (P4.14). Most of what remains overlaps codegen (P4.9/P4.10). In this cold-cache pass, + building the PCH once per config costs 98.7 s of that, in the background. +- **Codegen (topic 7, P4.3–P4.8, P4.10): 607 → 301 s.** One pool per config (213 instead of 855 + `python_kernel_codegen` calls). The static-VGPR and overflow checks stop rejected kernels early + (P4.5/P4.6). The 19 configs that had `CpuThreads: 1` now generate in parallel (P4.8): −64% for + that set. `python_kernel_validate` −28% and `write_assemble` −11% probably follow from the + same changes (not measured separately). +- **Benchmark setup (topic 8, P4.1/P4.2): 133 → 4.5 s.** +- **Unchanged as expected:** solution generation (−1.5%), bench postprocess, solution/cache + files, Ductile GA search. +- **RUN: benchmark client −4.9 s** is the ductile configs' client runs during the build. P1's + client changes act there too (input preparation −3.8 s, as in P3). +- The untimed part per config (pytest start-up, imports, artifact packing) is the same on both + sides (697 against 686 s over 184 configs). So the wall-time gain (−41%) is smaller than the + timed gain (−52%). +- Largest configs (pytest s, control → P6): `subtile_mxfp4` 211 → 45, `dtl` 168 → 137, + `subtile_bf16` 121 → 19, `cgemm` 97 → 79, `bufferLoadStore0` 82 → 68, + `custom_mainloop_scheduling` 76 → 30, `plr_zero` 72 → 60, `spmm_f16_sb` 72 → 43, `zgemm` + 67 → 61, `fp16_tn` 60 → 53. No config is slower than its pair. + +### Configs with `LibraryLogic:`/`LibraryClient:` changes (P2.5/P2.6) + +`--build-only` skips both steps (`Tensile.py`: "Build-only mode: skipping LibraryLogic and +LibraryClient"), so these sections do not change the build-side work. Their YAML changes do +not explain any of the gain. The 25 configs whose sections changed between `2083e6254bf` and +`9f493d7a6c1` are reported separately above. All 25 lost their sections (P2.6): the 8 +`ductile/gfx950/*`, the 2 mxfp8 repro configs, the 8 `subtile_mxfp8_*` configs and the 7 gfx950 +`custom/*` configs. The 8 `subtile_mxfp8_*` configs also lost `CpuThreads: 1` (P4.8). +`prediction_bf16` gained both sections (P2.5); it is P6-only, so it is not in the 25. The set's +gain (−30.7%) is smaller than the rest's (−42.2%) because all 25 are small configs (6–19 s each +in the control), where the untimed per-config overhead (about 3.7 s) dominates. + +## Findings that bear on the build side + +- **P4.7/P4.8 regression** (above), fixed in `9f493d7a6c1`. The verifications of those steps + built no custom-kernel config. A unit-level case for a custom kernel now covers it (in the fix + commit). +- **The chunk cache (P4.15) gives no measurable wall-time gain** in either build: 11.5% chunk + hits in the CI-like build, 1.9% within cold single configs. The run without it was 1–3% faster + in both orders (within drift). Its benefit depends on configs that share many helpers but + miss the whole-file cache, which the common set rarely has. Also, chunk contents depend on + the pool budget (FOLLOWUPS P4.14), so a cache filled by a `-n 4` build (2 chunks per compile) + does not serve a serial build (4 chunks). Keeping it on is harmless. Whether to keep the + feature is a decision for the plan owner. +- **FOLLOWUPS P4.15** (a failed chunk-key computation still prints "0 of n chunks hit"): no key + computation failed in any P6 run, so the hit counts above are exact. +- **FOLLOWUPS P4.12** (`.tmp_*` entries of killed processes are never evicted; per-process temp + name in `HelperKernelCache`): no cache-related build failure occurred in the 4 split runs at + `9f493d7a6c1`, and the last run's cache had no `.tmp_*` left. (The earlier runs' caches were + cleared before they could be checked.) 3 xdist workers built the same PCH key at once, which is + the race the rename guards against. +- **FOLLOWUPS P4.14:** TensileCreateLibrary's (shipped library) helper compile is still in one + piece. P6 measures only the benchmark/test build path. +- **FOLLOWUPS P4.7/P4.9:** `DisableAsmComments` reaches the helper `Kernels.h` inconsistently in + serial codegen. Since P4.8 only pool-size-1 runs are affected, and no P6 run has pool size 1. +- The per-worker budget (P4.3) leaves cores idle at the end of a `-n 4` run once fewer than 4 + tests remain (see the split section). Not a regression: P0.2 oversubscribed instead. + +## Reproducing + +```bash +git worktree add --detach 9f493d7a6c1 +source /home/alvasile/venv/bin/activate +export TL=/projects/hipblaslt/tensilelite +export CLIENT= +export TENSILE_CACHE=/tensile TENSILELITE_CACHE=/tensilelite ART= +# split phases: run_split.sh runs the build phase, then the --use-cache run phase. The P6 number +# is the "END build" line of $OUT/times.txt; the run phase that follows is not part of P6. P6 +# also added -rP to the build command (to log the cache lines); +# TENSILE_DISABLE_HELPER_CHUNK_CACHE=1 for the chunk-cache-off runs +OUT= bash $TL/timing_runs/full_runs_gfx950/split/run_split.sh +# per-config cold build +ROCR_VISIBLE_DEVICES= OUT= SCRATCH= \ + bash $TL/timing_runs/full_runs_gfx950/per_config/run_per_config_build.sh +``` + +The P0.2 side used P0.2's worktree and client (`baseline.md`, "Reproducing"). The interleaved +driver, the per-config script copy and the analysis scripts are in +`/home/alvasile/timing_runs_raw/P6/scripts/` (`driver.sh`, `split_stats.py`, `cmp_pairs.py`, +`cmp_agg.py`). diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_build/AGGREGATE_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_build/AGGREGATE_analysis.txt new file mode 100644 index 000000000000..96a7afcf836f --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_build/AGGREGATE_analysis.txt @@ -0,0 +1,575 @@ +============================================================================================================================= +TIMING ANALYSIS SUMMARY +============================================================================================================================= + +TIMING BY PURPOSE +----------------------------------------------------------------------------------------------------------------------------- + Category Count Total (ms) Mean (ms) % of Parent % of Wall +----------------------------------------------------------------------------------------------------------------------------- + TOTAL RUNTIME 1,286,569.24 100.0% + BUILD: solution generation & compile 1,241,290.68 96.5% 96.5% + Solution generation & build (Python) 1,219,840.35 98.3% 94.8% + python_kernel_compilation 1,862 904,610.66 485.83 74.2% 70.3% + python_kernel_codegen 214 301,368.91 1,408.27 33.3% 23.4% + python_kernel_write_assemble 856 236,194.18 275.93 26.1% 18.4% + python_kernel_build_src_co_wait 856 139,026.68 162.41 15.4% 10.8% + python_kernel_bench_postprocess 856 106,019.05 123.85 11.7% 8.2% + python_benchpost_lib_construction 856 69,091.03 80.71 65.2% 5.4% + python_benchpost_library_write 856 26,755.03 31.26 25.2% 2.1% + python_benchpost_naming 856 9,931.36 11.60 9.4% 0.8% + python_benchpost_client_config 856 205.40 0.24 0.2% 0.0% + (untracked) 36.24 0.0% 0.0% + python_kernel_validate 856 58,429.40 68.26 6.5% 4.5% + python_kernel_build_co 856 25,692.09 30.01 2.8% 2.0% + python_kernel_setup 856 21,607.69 25.24 2.4% 1.7% + python_kernel_write_helpers 856 7,968.35 9.31 0.9% 0.6% + python_kernel_bench_setup 856 4,549.36 5.31 0.5% 0.4% + (untracked) 3,754.95 0.4% 0.3% + python_solution_generation 824 315,229.69 382.56 25.8% 24.5% + python_solgen_forked_solutions 824 315,154.32 382.47 100.0% 24.5% + python_solgen_custom_kernels 824 57.74 0.07 0.0% 0.0% + python_solgen_fork_permutations 824 4.61 0.01 0.0% 0.0% + (untracked) 13.02 0.0% 0.0% + Ductile GA search (Python) 21,450.33 1.7% 1.7% + python_ductile_initial_sampling 8 15,265.87 1,908.23 71.2% 1.2% + python_ductile_generate_solutions 32 4,525.43 141.42 21.1% 0.4% + python_ductile_ga_between_evaluations 8 1,609.22 201.15 7.5% 0.1% + python_ductile_setup 8 35.96 4.50 0.2% 0.0% + python_ductile_clear_artifacts 24 7.81 0.33 0.0% 0.0% + python_ductile_read_scores 32 5.63 0.18 0.0% 0.0% + python_ductile_verification 8 0.40 0.05 0.0% 0.0% + SHARED: cache & solution files 23,439.24 1.8% 1.8% + python_write_solutions 856 14,118.81 16.49 60.2% 1.1% + python_wsol_dump 856 13,565.73 15.85 96.1% 1.1% + python_wsol_prepare 856 452.00 0.53 3.2% 0.0% + python_wsol_prepare_nocache 856 450.45 0.53 99.7% 0.0% + (untracked) 1.54 0.3% 0.0% + python_wsol_header 856 13.40 0.02 0.1% 0.0% + (untracked) 87.68 0.6% 0.0% + python_solution_indexing 856 8,401.41 9.81 35.8% 0.7% + python_write_cache 856 918.42 1.07 3.9% 0.1% + python_cache_check 832 0.61 0.00 0.0% 0.0% + RUN: benchmark client 9,953.37 0.8% 0.8% + Client launch (startup, setup & teardown) 8,872.09 89.1% 0.7% + hip_initialization 32 5,103.79 159.49 57.5% 0.4% + process_exit 32 2,186.50 68.33 24.6% 0.2% + main_problem_factory 32 640.52 20.02 7.2% 0.0% + factory_hip_get_device_properties 4 637.01 159.25 99.5% 0.0% + (untracked) 3.52 0.5% 0.0% + process_startup 32 480.60 15.02 5.4% 0.0% + main_teardown_locals 32 150.57 4.71 1.7% 0.0% + data_init_setup 32 97.89 3.06 1.1% 0.0% + code_object_loading 32 92.63 2.89 1.0% 0.0% + library_loading 32 62.38 1.95 0.7% 0.0% + lazy_loading_init 32 43.61 1.36 0.5% 0.0% + main_parse_args 32 10.34 0.32 0.1% 0.0% + python_client_write_run_script 32 1.41 0.04 0.0% 0.0% + reporter_setup 32 1.14 0.04 0.0% 0.0% + listener_setup 32 0.41 0.01 0.0% 0.0% + solution_iterator_setup 32 0.28 0.01 0.0% 0.0% + Input data preparation 1,058.18 10.6% 0.1% + gpu_input_preparation 40 943.11 23.58 89.1% 0.1% + gpu_input_prep_copy_valid_to_gpu 40 472.41 11.81 50.1% 0.0% + gpu_input_prep_init_cpu_inputs 39 404.45 10.37 42.9% 0.0% + gpu_input_prep_copy_swizzled 4 61.73 15.43 6.5% 0.0% + gpu_input_prep_batched_inputs 40 2.89 0.07 0.3% 0.0% + gpu_input_prep_copy_inputs_d2d 40 1.06 0.03 0.1% 0.0% + gpu_input_prep_constant_inputs 39 0.43 0.01 0.0% 0.0% + gpu_input_prep_convert_to_inputs 40 0.09 0.00 0.0% 0.0% + gpu_input_prep_slow_path 40 0.00 0.00 0.0% 0.0% + (untracked) 0.06 0.0% 0.0% + cpu_data_init 1 74.69 74.69 7.1% 0.0% + pre_solution_listener.DataInitialization 585 33.93 0.06 3.2% 0.0% + pre_solution_mx_init_data 130 19.99 0.15 58.9% 0.0% + pre_solution_mx_side_swizzle 8 19.95 2.49 99.8% 0.0% + pre_solution_mx_side_cache_hit 252 0.02 0.00 0.1% 0.0% + (untracked) 0.03 0.1% 0.0% + pre_solution_mx_cpu_sync_no_regen 130 10.19 0.08 30.0% 0.0% + pre_solution_mx_copy_inputs_d2d_no_regen 130 3.67 0.03 10.8% 0.0% + pre_solution_mx_check 585 0.03 0.00 0.1% 0.0% + pre_solution_mx_copy_valid_to_gpu_no_regen 130 0.02 0.00 0.0% 0.0% + (untracked) 0.03 0.1% 0.0% + rotating_buffer_preparation 40 6.41 0.16 0.6% 0.0% + gpu_input_reset 1 0.03 0.03 0.0% 0.0% + gpu_input_reset_copy_inputs_d2d 1 0.03 0.03 95.3% 0.0% + gpu_input_reset_batched_inputs 1 0.00 0.00 1.7% 0.0% + gpu_input_reset_convert_to_inputs 1 0.00 0.00 1.4% 0.0% + gpu_input_reset_copy_swizzled 1 0.00 0.00 0.9% 0.0% + gpu_input_reset_copy_valid_to_gpu 1 0.00 0.00 0.6% 0.0% + gpu_input_reset_slow_path 1 0.00 0.00 0.1% 0.0% + Per-solution bookkeeping & reporting 12.66 0.1% 0.0% + pre_solution_listener.BenchmarkTimer 585 3.85 0.01 30.4% 0.0% + run_current_solution_check 585 1.66 0.00 13.1% 0.0% + post_problem 40 1.52 0.04 12.0% 0.0% + pre_solution_listener.ProgressListener 585 1.29 0.00 10.2% 0.0% + pre_solution_listener.AllSolutionsIterator 585 1.16 0.00 9.2% 0.0% + post_solution_reporting 585 1.12 0.00 8.8% 0.0% + post_solution_log 585 0.60 0.00 4.8% 0.0% + pre_solution_listener.MetaResultReporter 585 0.49 0.00 3.8% 0.0% + finalize_report 32 0.40 0.01 3.1% 0.0% + solution_selection 585 0.15 0.00 1.2% 0.0% + problem_report_index 40 0.13 0.00 1.1% 0.0% + pre_solution_listener.SynchronizerValidator 585 0.10 0.00 0.8% 0.0% + problem_teardown 40 0.08 0.00 0.6% 0.0% + solution_teardown 585 0.02 0.00 0.2% 0.0% + pre_solution_listener.HardwareMonitorListener 585 0.02 0.00 0.1% 0.0% + run_iteration_teardown 1 0.01 0.01 0.1% 0.0% + post_solution_result_file 585 0.01 0.00 0.1% 0.0% + post_solution_validation 585 0.01 0.00 0.1% 0.0% + post_solution_perf_calc 585 0.01 0.00 0.1% 0.0% + post_solution_sol_advance 585 0.01 0.00 0.1% 0.0% + post_solution_perf_reset 585 0.01 0.00 0.1% 0.0% + Validation 9.84 0.1% 0.0% + validate_warmups_listener.ReferenceValidator 1 9.22 9.22 93.7% 0.0% + validate_gpu_readback 1 7.91 7.91 85.9% 0.0% + validate_result_buffer_alloc 1 1.29 1.29 14.0% 0.0% + validate_element_comparison 1 0.01 0.01 0.1% 0.0% + validate_inputs_copy 1 0.00 0.00 0.0% 0.0% + validate_setup 1 0.00 0.00 0.0% 0.0% + validate_report 1 0.00 0.00 0.0% 0.0% + synchronizer_check 1 0.62 0.62 6.3% 0.0% + synchronizer_reduce 1 0.62 0.62 100.0% 0.0% + validate_warmups_listener.HardwareMonitorListener 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.MetaResultReporter 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.DataInitialization 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.AllSolutionsIterator 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.ProgressListener 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.SynchronizerValidator 1 0.00 0.00 0.0% 0.0% + Kernel launch & execution 0.43 0.0% 0.0% + run_iteration_sync 1 0.18 0.18 41.6% 0.0% + warmup_runs 2 0.12 0.06 28.7% 0.0% + kernel_solving 1 0.06 0.06 15.0% 0.0% + warmup_first_sync 1 0.04 0.04 8.5% 0.0% + warmup_events_setup 1 0.02 0.02 4.8% 0.0% + validate_warmups_listener.BenchmarkTimer 1 0.01 0.01 1.3% 0.0% + validate_gpu_sync 1 0.01 0.01 100.0% 0.0% + benchmark_runs 1 0.00 0.00 0.1% 0.0% + benchmark_pre_syncs 1 0.00 0.00 51.4% 0.0% + benchmark_post_syncs 1 0.00 0.00 48.6% 0.0% + Reference GEMM (CPU) 0.17 0.0% 0.0% + cpu_reference_gemm 1 0.15 0.15 88.6% 0.0% + solve_cpu_slow 1 0.15 0.15 100.0% 0.0% + pre_solution_listener.ReferenceValidator 585 0.02 0.00 11.4% 0.0% + pre_solution_ref_skipped 1 0.00 0.00 0.1% 0.0% + (untracked) 0.02 99.9% 0.0% + MEASUREMENT OVERHEAD (instrumentation) 360.36 0.0% 0.0% + main_timing_calibration 32 354.10 11.07 98.3% 0.0% + flush_timing_buffer 32 5.10 0.16 1.4% 0.0% + timing_overhead 32 0.60 0.02 0.2% 0.0% + (timer overhead subtracted from parent timers) 0.55 0.2% 0.0% + UNATTRIBUTED (untracked time & timer overhead) 11,525.59 0.9% 0.9% + (untracked in python_benchmark_problems) 11,509.17 99.9% 0.9% + (untracked in python_client_execution) 14.71 0.1% 0.0% + (untracked in pre_solution) 1.01 0.0% 0.0% + (untracked in pre_problem) 0.61 0.0% 0.0% + (untracked in post_solution) 0.09 0.0% 0.0% + (untracked in validate_warmups) 0.01 0.0% 0.0% + +TIMING BY TIMER NESTING (legacy view) +----------------------------------------------------------------------------------------------------------------------------- + Category Count Total (ms) Mean (ms) % of Parent % of Wall +----------------------------------------------------------------------------------------------------------------------------- + python_benchmark_problems 185 1,286,568.71 6,954.43 100.0% + ------------------------------------------------------------------------------------------------------------------------- + python_kernel_compilation 1,862 904,610.66 485.83 70.3% 70.3% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_codegen 214 301,368.91 1,408.27 33.3% 23.4% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_assemble 856 236,194.18 275.93 26.1% 18.4% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_src_co_wait 856 139,026.68 162.41 15.4% 10.8% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_postprocess 856 106,019.05 123.85 11.7% 8.2% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_benchpost_lib_construction 856 69,091.03 80.71 65.2% 5.4% + python_benchpost_library_write 856 26,755.03 31.26 25.2% 2.1% + python_benchpost_naming 856 9,931.36 11.60 9.4% 0.8% + python_benchpost_client_config 856 205.40 0.24 0.2% 0.0% + (untracked) 36.24 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_validate 856 58,429.40 68.26 6.5% 4.5% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_co 856 25,692.09 30.01 2.8% 2.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_setup 856 21,607.69 25.24 2.4% 1.7% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_helpers 856 7,968.35 9.31 0.9% 0.6% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_setup 856 4,549.36 5.31 0.5% 0.4% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 3,754.95 0.4% 0.3% + ------------------------------------------------------------------------------------------------------------------------- + python_solution_generation 824 315,229.69 382.56 24.5% 24.5% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_forked_solutions 824 315,154.32 382.47 100.0% 24.5% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_custom_kernels 824 57.74 0.07 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 13.02 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_fork_permutations 824 4.61 0.01 0.0% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + python_ductile_initial_sampling 8 15,265.87 1,908.23 1.2% 1.2% + ------------------------------------------------------------------------------------------------------------------------- + python_write_solutions 856 14,118.81 16.49 1.1% 1.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_dump 856 13,565.73 15.85 96.1% 1.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_prepare 856 452.00 0.53 3.2% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_wsol_prepare_nocache 856 450.45 0.53 99.7% 0.0% + (untracked) 1.54 0.3% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 87.68 0.6% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_header 856 13.40 0.02 0.1% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + (untracked) 11,509.17 0.9% 0.9% + ------------------------------------------------------------------------------------------------------------------------- + python_client_execution 32 10,329.62 322.80 0.8% 0.8% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + C++ Client Setup & Teardown 9,226.19 89.3% 0.7% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + hip_initialization 32 5,103.79 159.49 55.3% 0.4% + process_exit 32 2,186.50 68.33 23.7% 0.2% + main_problem_factory 32 640.52 20.02 6.9% 0.0% + factory_hip_get_device_properties 4 637.01 159.25 99.5% 0.0% + (untracked) 3.52 0.5% 0.0% + process_startup 32 480.60 15.02 5.2% 0.0% + main_timing_calibration 32 354.10 11.07 3.8% 0.0% + main_teardown_locals 32 150.57 4.71 1.6% 0.0% + data_init_setup 32 97.89 3.06 1.1% 0.0% + code_object_loading 32 92.63 2.89 1.0% 0.0% + library_loading 32 62.38 1.95 0.7% 0.0% + lazy_loading_init 32 43.61 1.36 0.5% 0.0% + main_parse_args 32 10.34 0.32 0.1% 0.0% + python_client_write_run_script 32 1.41 0.04 0.0% 0.0% + reporter_setup 32 1.14 0.04 0.0% 0.0% + listener_setup 32 0.41 0.01 0.0% 0.0% + solution_iterator_setup 32 0.28 0.01 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Data Preparation 1,025.02 9.9% 0.1% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + gpu_input_preparation 40 943.11 23.58 92.0% 0.1% + gpu_input_prep_copy_valid_to_gpu 40 472.41 11.81 50.1% 0.0% + gpu_input_prep_init_cpu_inputs 39 404.45 10.37 42.9% 0.0% + gpu_input_prep_copy_swizzled 4 61.73 15.43 6.5% 0.0% + gpu_input_prep_batched_inputs 40 2.89 0.07 0.3% 0.0% + gpu_input_prep_copy_inputs_d2d 40 1.06 0.03 0.1% 0.0% + gpu_input_prep_constant_inputs 39 0.43 0.01 0.0% 0.0% + gpu_input_prep_convert_to_inputs 40 0.09 0.00 0.0% 0.0% + (untracked) 0.06 0.0% 0.0% + gpu_input_prep_slow_path 40 0.00 0.00 0.0% 0.0% + pre_problem 40 75.46 1.89 7.4% 0.0% + cpu_data_init 1 74.69 74.69 99.0% 0.0% + (untracked) 0.61 0.8% 0.0% + cpu_reference_gemm 1 0.16 0.16 0.2% 0.0% + solve_cpu_slow 1 0.15 0.15 95.3% 0.0% + rotating_buffer_preparation 40 6.41 0.16 0.6% 0.0% + gpu_input_reset 1 0.03 0.03 0.0% 0.0% + gpu_input_reset_copy_inputs_d2d 1 0.03 0.03 93.3% 0.0% + gpu_input_reset_batched_inputs 1 0.00 0.00 1.7% 0.0% + gpu_input_reset_convert_to_inputs 1 0.00 0.00 1.4% 0.0% + gpu_input_reset_copy_swizzled 1 0.00 0.00 0.9% 0.0% + gpu_input_reset_copy_valid_to_gpu 1 0.00 0.00 0.6% 0.0% + gpu_input_reset_slow_path 1 0.00 0.00 0.1% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Kernel Execution 48.14 0.5% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + pre_solution 585 41.86 0.07 87.0% 0.0% + pre_solution_listener.DataInitialization 585 33.93 0.06 81.1% 0.0% + pre_solution_mx_init_data 130 19.99 0.15 58.9% 0.0% + pre_solution_mx_side_swizzle 8 19.95 2.49 99.8% 0.0% + (untracked) 0.03 0.1% 0.0% + pre_solution_mx_side_cache_hit 252 0.02 0.00 0.1% 0.0% + pre_solution_mx_cpu_sync_no_regen 130 10.19 0.08 30.0% 0.0% + pre_solution_mx_copy_inputs_d2d_no_regen 130 3.67 0.03 10.8% 0.0% + pre_solution_mx_check 585 0.03 0.00 0.1% 0.0% + (untracked) 0.03 0.1% 0.0% + pre_solution_mx_copy_valid_to_gpu_no_regen 130 0.02 0.00 0.0% 0.0% + pre_solution_listener.BenchmarkTimer 585 3.85 0.01 9.2% 0.0% + pre_solution_listener.ProgressListener 585 1.29 0.00 3.1% 0.0% + pre_solution_listener.AllSolutionsIterator 585 1.16 0.00 2.8% 0.0% + (untracked) 1.01 2.4% 0.0% + pre_solution_listener.MetaResultReporter 585 0.49 0.00 1.2% 0.0% + pre_solution_listener.SynchronizerValidator 585 0.10 0.00 0.2% 0.0% + pre_solution_listener.ReferenceValidator 585 0.02 0.00 0.0% 0.0% + (untracked) 0.02 99.9% 0.0% + pre_solution_ref_skipped 1 0.00 0.00 0.1% 0.0% + pre_solution_listener.HardwareMonitorListener 585 0.02 0.00 0.0% 0.0% + post_solution 585 1.88 0.00 3.9% 0.0% + post_solution_reporting 585 1.12 0.00 59.6% 0.0% + post_solution_log 585 0.60 0.00 32.1% 0.0% + (untracked) 0.09 4.7% 0.0% + post_solution_result_file 585 0.01 0.00 0.8% 0.0% + post_solution_validation 585 0.01 0.00 0.8% 0.0% + post_solution_perf_calc 585 0.01 0.00 0.7% 0.0% + post_solution_sol_advance 585 0.01 0.00 0.7% 0.0% + post_solution_perf_reset 585 0.01 0.00 0.6% 0.0% + run_current_solution_check 585 1.66 0.00 3.4% 0.0% + post_problem 40 1.52 0.04 3.2% 0.0% + finalize_report 32 0.40 0.01 0.8% 0.0% + run_iteration_sync 1 0.18 0.18 0.4% 0.0% + solution_selection 585 0.15 0.00 0.3% 0.0% + problem_report_index 40 0.13 0.00 0.3% 0.0% + warmup_runs 2 0.12 0.06 0.3% 0.0% + problem_teardown 40 0.08 0.00 0.2% 0.0% + kernel_solving 1 0.06 0.06 0.1% 0.0% + warmup_first_sync 1 0.04 0.04 0.1% 0.0% + solution_teardown 585 0.02 0.00 0.0% 0.0% + warmup_events_setup 1 0.02 0.02 0.0% 0.0% + run_iteration_teardown 1 0.01 0.01 0.0% 0.0% + benchmark_runs 1 0.00 0.00 0.0% 0.0% + benchmark_pre_syncs 1 0.00 0.00 36.5% 0.0% + benchmark_post_syncs 1 0.00 0.00 34.5% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked C++ overhead) 14.71 0.1% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Reference Computation 9.86 0.1% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + validate_warmups 1 9.24 9.24 93.7% 0.0% + validate_warmups_listener.ReferenceValidator 1 9.22 9.22 99.8% 0.0% + validate_gpu_readback 1 7.91 7.91 85.8% 0.0% + validate_result_buffer_alloc 1 1.29 1.29 14.0% 0.0% + validate_element_comparison 1 0.01 0.01 0.1% 0.0% + validate_inputs_copy 1 0.00 0.00 0.0% 0.0% + validate_setup 1 0.00 0.00 0.0% 0.0% + validate_report 1 0.00 0.00 0.0% 0.0% + (untracked) 0.01 0.1% 0.0% + validate_warmups_listener.BenchmarkTimer 1 0.01 0.01 0.1% 0.0% + validate_gpu_sync 1 0.01 0.01 87.1% 0.0% + validate_warmups_listener.HardwareMonitorListener 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.MetaResultReporter 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.SynchronizerValidator 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.DataInitialization 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.AllSolutionsIterator 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.ProgressListener 1 0.00 0.00 0.0% 0.0% + synchronizer_check 1 0.62 0.62 6.3% 0.0% + synchronizer_reduce 1 0.62 0.62 99.6% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Timing Overhead 5.71 0.1% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + flush_timing_buffer 32 5.10 0.16 89.4% 0.0% + timing_overhead 32 0.60 0.02 10.6% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + python_solution_indexing 856 8,401.41 9.81 0.7% 0.7% + ------------------------------------------------------------------------------------------------------------------------- + python_ductile_generate_solutions 32 4,525.43 141.42 0.4% 0.4% + ------------------------------------------------------------------------------------------------------------------------- + python_ductile_ga_between_evaluations 8 1,609.22 201.15 0.1% 0.1% + ------------------------------------------------------------------------------------------------------------------------- + python_write_cache 856 918.42 1.07 0.1% 0.1% + ------------------------------------------------------------------------------------------------------------------------- + python_ductile_setup 8 35.96 4.50 0.0% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + python_ductile_clear_artifacts 24 7.81 0.33 0.0% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + python_ductile_read_scores 32 5.63 0.18 0.0% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + python_cache_check 832 0.61 0.00 0.0% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + python_ductile_verification 8 0.40 0.05 0.0% 0.0% + + Overlapping Totals (wrap or overlap other timers; not summed into the tree) + --------------------------------------------------------------------------------------------------------------------------- + python_client_subprocess 32 10,315.58 322.36 0.8% + main_total 32 7,491.91 234.12 0.6% + problem_total 40 1,083.35 27.08 0.1% + solution_total 585 56.18 0.10 0.0% + run_iteration_total 1 10.34 10.34 0.0% + validate_total 1 9.22 9.22 0.0% + validate_check_results_typed 1 9.22 9.22 0.0% + python_kernel_build_src_co 856 394,368.66 460.71 30.7% + python_kernel_build_src_co.setup 856 191.72 0.22 0.0% + python_kernel_build_src_co.cache_check 856 5,359.01 6.26 0.4% + python_kernel_build_src_co.pch 264 99,216.79 375.82 7.7% + python_kernel_build_src_co.compile 264 287,874.71 1,090.43 22.4% + python_kernel_build_src_co.unbundle 264 1,170.44 4.43 0.1% + python_kernel_build_src_co.move 264 21.46 0.08 0.0% + python_kernel_build_src_co.cache_populate 264 135.07 0.51 0.0% + +----------------------------------------------------------------------------------------------------------------------------- +HIERARCHY VALIDATION: OK (all child phases fit within their parents) + +TIME BREAKDOWN (visual, % of wall clock) +----------------------------------------------------------------------------------------------------------------------------- + TOTAL RUNTIME 100.0% ################################################## + BUILD: solution generation & compile 96.5% ################################################ + Solution generation & build (Python) 94.8% ############################################### + python_kernel_compilation 70.3% ################################### + python_kernel_codegen 23.4% ########### + python_kernel_write_assemble 18.4% ######### + python_kernel_build_src_co_wait 10.8% ##### + python_kernel_bench_postprocess 8.2% #### + python_benchpost_lib_construction 5.4% ## + python_benchpost_library_write 2.1% # + python_benchpost_naming 0.8% + python_benchpost_client_config 0.0% + (untracked) 0.0% + python_kernel_validate 4.5% ## + python_kernel_build_co 2.0% + python_kernel_setup 1.7% + python_kernel_write_helpers 0.6% + python_kernel_bench_setup 0.4% + (untracked) 0.3% + python_solution_generation 24.5% ############ + python_solgen_forked_solutions 24.5% ############ + python_solgen_custom_kernels 0.0% + python_solgen_fork_permutations 0.0% + (untracked) 0.0% + Ductile GA search (Python) 1.7% + python_ductile_initial_sampling 1.2% + python_ductile_generate_solutions 0.4% + python_ductile_ga_between_evaluations 0.1% + python_ductile_setup 0.0% + python_ductile_clear_artifacts 0.0% + python_ductile_read_scores 0.0% + python_ductile_verification 0.0% + SHARED: cache & solution files 1.8% + python_write_solutions 1.1% + python_wsol_dump 1.1% + python_wsol_prepare 0.0% + python_wsol_prepare_nocache 0.0% + (untracked) 0.0% + python_wsol_header 0.0% + (untracked) 0.0% + python_solution_indexing 0.7% + python_write_cache 0.1% + python_cache_check 0.0% + RUN: benchmark client 0.8% + Client launch (startup, setup & teardown) 0.7% + hip_initialization 0.4% + process_exit 0.2% + main_problem_factory 0.0% + factory_hip_get_device_properties 0.0% + (untracked) 0.0% + process_startup 0.0% + main_teardown_locals 0.0% + data_init_setup 0.0% + code_object_loading 0.0% + library_loading 0.0% + lazy_loading_init 0.0% + main_parse_args 0.0% + python_client_write_run_script 0.0% + reporter_setup 0.0% + listener_setup 0.0% + solution_iterator_setup 0.0% + Input data preparation 0.1% + gpu_input_preparation 0.1% + gpu_input_prep_copy_valid_to_gpu 0.0% + gpu_input_prep_init_cpu_inputs 0.0% + gpu_input_prep_copy_swizzled 0.0% + gpu_input_prep_batched_inputs 0.0% + gpu_input_prep_copy_inputs_d2d 0.0% + gpu_input_prep_constant_inputs 0.0% + gpu_input_prep_convert_to_inputs 0.0% + gpu_input_prep_slow_path 0.0% + (untracked) 0.0% + cpu_data_init 0.0% + pre_solution_listener.DataInitialization 0.0% + pre_solution_mx_init_data 0.0% + pre_solution_mx_side_swizzle 0.0% + pre_solution_mx_side_cache_hit 0.0% + (untracked) 0.0% + pre_solution_mx_cpu_sync_no_regen 0.0% + pre_solution_mx_copy_inputs_d2d_no_regen 0.0% + pre_solution_mx_check 0.0% + pre_solution_mx_copy_valid_to_gpu_no_regen 0.0% + (untracked) 0.0% + rotating_buffer_preparation 0.0% + gpu_input_reset 0.0% + gpu_input_reset_copy_inputs_d2d 0.0% + gpu_input_reset_batched_inputs 0.0% + gpu_input_reset_convert_to_inputs 0.0% + gpu_input_reset_copy_swizzled 0.0% + gpu_input_reset_copy_valid_to_gpu 0.0% + gpu_input_reset_slow_path 0.0% + Per-solution bookkeeping & reporting 0.0% + pre_solution_listener.BenchmarkTimer 0.0% + run_current_solution_check 0.0% + post_problem 0.0% + pre_solution_listener.ProgressListener 0.0% + pre_solution_listener.AllSolutionsIterator 0.0% + post_solution_reporting 0.0% + post_solution_log 0.0% + pre_solution_listener.MetaResultReporter 0.0% + finalize_report 0.0% + solution_selection 0.0% + problem_report_index 0.0% + pre_solution_listener.SynchronizerValidator 0.0% + problem_teardown 0.0% + solution_teardown 0.0% + pre_solution_listener.HardwareMonitorListener 0.0% + run_iteration_teardown 0.0% + post_solution_result_file 0.0% + post_solution_validation 0.0% + post_solution_perf_calc 0.0% + post_solution_sol_advance 0.0% + post_solution_perf_reset 0.0% + Validation 0.0% + validate_warmups_listener.ReferenceValidator 0.0% + validate_gpu_readback 0.0% + validate_result_buffer_alloc 0.0% + validate_element_comparison 0.0% + validate_inputs_copy 0.0% + validate_setup 0.0% + validate_report 0.0% + synchronizer_check 0.0% + synchronizer_reduce 0.0% + validate_warmups_listener.HardwareMonitorListener 0.0% + validate_warmups_listener.MetaResultReporter 0.0% + validate_warmups_listener.DataInitialization 0.0% + validate_warmups_listener.AllSolutionsIterator 0.0% + validate_warmups_listener.ProgressListener 0.0% + validate_warmups_listener.SynchronizerValidator 0.0% + Kernel launch & execution 0.0% + run_iteration_sync 0.0% + warmup_runs 0.0% + kernel_solving 0.0% + warmup_first_sync 0.0% + warmup_events_setup 0.0% + validate_warmups_listener.BenchmarkTimer 0.0% + validate_gpu_sync 0.0% + benchmark_runs 0.0% + benchmark_pre_syncs 0.0% + benchmark_post_syncs 0.0% + Reference GEMM (CPU) 0.0% + cpu_reference_gemm 0.0% + solve_cpu_slow 0.0% + pre_solution_listener.ReferenceValidator 0.0% + pre_solution_ref_skipped 0.0% + (untracked) 0.0% + MEASUREMENT OVERHEAD (instrumentation) 0.0% + main_timing_calibration 0.0% + flush_timing_buffer 0.0% + timing_overhead 0.0% + (timer overhead subtracted from parent timers) 0.0% + UNATTRIBUTED (untracked time & timer overhead) 0.9% + (untracked in python_benchmark_problems) 0.9% + (untracked in python_client_execution) 0.0% + (untracked in pre_solution) 0.0% + (untracked in pre_problem) 0.0% + (untracked in post_solution) 0.0% + (untracked in validate_warmups) 0.0% + +KEY INSIGHTS (% of wall clock) +----------------------------------------------------------------------------------------------------------------------------- + Solution generation & build (Python) 1219.84s (94.8%) + SHARED: cache & solution files 23.44s (1.8%) + Ductile GA search (Python) 21.45s (1.7%) + Client launch (startup, setup & teardown) 8.87s (0.7%) + Input data preparation 1.06s (0.1%) + MEASUREMENT OVERHEAD (instrumentation) 0.36s (0.0%) + Per-solution bookkeeping & reporting 0.01s (0.0%) + Validation 0.01s (0.0%) + Kernel launch & execution 0.00s (0.0%) + Reference GEMM (CPU) 0.00s (0.0%) + UNATTRIBUTED (untracked time & timer overhead) 11.53s (0.9%) + + +TOP 10 SLOWEST PROBLEMS (by CPU reference time) +----------------------------------------------------------------------------------------------------------------------------- + M N K Batch TypeA TypeD CPU Ref (ms) GPU (ms) + --------------------------------------------------------------------------------------------------------------------------- + 2048 256 128 16 Float8 BFloat16 0.16 0.00 + diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_build/split_build_outcomes.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_build/split_build_outcomes.txt new file mode 100644 index 000000000000..64f1aee192e4 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_build/split_build_outcomes.txt @@ -0,0 +1,572 @@ +# Build-phase correctness after P4 + FIX1 (P6): outcome of every test in the gfx950 split build phase +# (split/run_split.sh build phase: --build-only, -n 4, cores 0-24, empty cache at start) at 9f493d7a6c1. +# One " " per line, sorted. 568 tests: 196 PASSED, 2 FAILED, 370 SKIPPED. Identical in +# all 4 runs at 9f493d7a6c1 (chunk cache on and off); differs from the P0.2 build phase only by prediction_bf16 (new). +tensilelite/Tests/common/flags/gfx942/test_build_only.py::test_compile PASSED +tensilelite/Tests/common/flags/gfx942/test_build_only.py::test_use_cache SKIPPED +tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_compile_and_benchmark SKIPPED +tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_use_cache SKIPPED +tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_use_cache_creates_cache_dir SKIPPED +tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_use_cache_different_params SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/amaxd/8rn_amaxd.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/amaxd/gfx1250/amax_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/benchmark_runs/bf16_tn_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/benchmark_runs/fp16_tn_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/benchmark_runs/fp32_tn.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/client/rotate_mode0.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/client/rotate_mode0_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/client/rotate_mode1.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/client/rotate_mode1_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/comm/gfx950/fused_a2a.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx1250/bbs_tn_dp.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx1250/bbs_tn_multi_size_single_kernel.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx1250/bbs_tn_streamK.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx1250/f32_tt_batched.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx1250/f8_tn_mx.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx1250/hhs_nt_multi_size.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx950/bf16_tn_dp.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx950/bf16_tn_multi_size_single_kernel.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx950/bf16_tn_streamK.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx950/f16_nt_multi_size.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx950/f32_tt_batched.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx950/f8_tn_mx_subtile.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx950/f8bf16_tn_cms.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx950/xf32_nn.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/exception/f16_cf32_nan.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/exception/f16_cf32_nan_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/8rn_b_s.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/ag.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/agntab_coverage_gfx942.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/bf16_tn.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/bf16_tn_predict.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/bufferLoadStore0.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/cgemm.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/dot2_gfx942.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/dtl.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/dtv.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/dtv_gfx90a.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/f32_mac.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/f64.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/f64_gfx90a.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/f64_mac.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp16_HH_BHS_bf16mfma.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp16_tn.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp16_use_e.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp16fp32mix.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp32_nt.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp32_scalealphavec.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp32_tn.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp8bf8nss.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp8n.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp8n_rowwise.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp8n_use_e.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp8nfp16mix_fp8nss.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp8nfp16mix_hfp8ns.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp8nfp16mix_hhs.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp8nfp16mix_hhs_lsu4.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gemm_bias_strided_batched.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gemm_bias_strided_batched_broadcast.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gemm_bias_strided_batched_broadcast_gfx940.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gemm_bias_strided_batched_gfx940.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gemm_ck_gfx942.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx11/bf16_tn_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx11/fp16_HH_BHS_bf16mfma_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx11/fp16_tn_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx11/fp16_use_e_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx11/gemm_bias_strided_batched_broadcast_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx11/gemm_bias_strided_batched_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx11/gsuasb_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx11/i8_gsu_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/1024_vgpr_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/1024_vgpr_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/8r_b_s_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/agntab_coverage_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/agntab_coverage_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/b6f4ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/b6f4ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/b8b8s_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/b8b8s_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/b8f8ss_grvw1_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/b8hs_grvw1_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/bf16_CLS_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/bf16_CLS_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/bf16_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/bf16_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/bf16_tn_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/bf16_tn_gfx12_predict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/bf6_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/bf6_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/bf6_tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/bf6_tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/cgemm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/cgemm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/decouple_pgr_tdm_fuse_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f32_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f32_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f32_grid_limit_size.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f4_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f4_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f4b8ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f4b8ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f4f6ss_tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f4f6ss_tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f64_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f64_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f6_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f6_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f6_tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f6_tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f6b6ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f6b6ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f6b8ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f6b8ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8b6ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8b6ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8b8ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8b8ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8f4ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8f4ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8f8s_cls_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8f8s_cls_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8f8s_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8f8s_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8f8s_grvw1_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8f8s_pk8_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8f8s_pk8_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8f8s_sr_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8f8s_sr_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/fp16_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/fp16_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/fp16_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/fp16_use_e_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/fp16_use_e_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/fp4_tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/fp4_tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/fp8_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/fp8_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/fp8_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/fp8bf8ss_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/gemm_bias_strided_batched_broadcast_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/gemm_bias_strided_batched_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/gltr_bounds_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/gsu_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/gsu_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/gsuaad_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/gsuasb_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/i8_gsu_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/i8i8i_grvw1_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/i8ii_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/i8ii_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/i8ii_grvw1_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/interleave_alpha2_group_store_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/largeLds_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/largeLds_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/mxf4_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/mxf4_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/mxf6_tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/mxf6_tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/mxf6f4ss_tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/mxf6f4ss_tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/mxf8_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/mxf8_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/mxf8f4ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/mxf8f4ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/nt_th_nv_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/nt_th_nv_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/pgl_bound_check.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/rap_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/segment_b_rotate_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/segment_interleave_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/segment_interleave_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/stinky_sia4.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/stinky_sia4_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250-strict_bench.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250-strict_cluster.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250_bench.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250_cluster.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/tdm_general_batched_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/tdm_general_batched_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/tdm_multicast_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/tdm_multicast_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/tdm_split_gfx1250-strict_xfail.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/tdm_split_gfx1250_xfail.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/tdmpluslds_tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/tdmpluslds_tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/xfp32_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/xfp32_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/xfp32_inf_support_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/zgemm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/zgemm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx1250/cluster_entry_handoff.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/agntab_coverage_gfx950.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/bf16_cvt.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/custom_mainloop_scheduling.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/custom_mainloop_scheduling_tf32.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/custom_xfp32.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/dot2_gfx950.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/dtl.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/f16f8mix_ss.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/f16f8mix_ss_stoch.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/f8b8hs.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/f8f16mix_f8s.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/fp16_use_e_wide_store.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/fp8.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_bf16_tn_act.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_bf16_tn_act_groupgemm.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_fp32_tn_act.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_fp32_tn_act_groupgemm.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/general_wgm.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/i8_gsu_gfx950.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/lds160K.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/lds_tr.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mx32f4_tn.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mx32f8_tn.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_bf16_tn_act.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_bf16_tn_act_groupgemm.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_fp32_tn_act.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_fp32_tn_act_groupgemm.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_bf16_tn_act.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_bf16_tn_act_groupgemm.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_fp32_tn_act.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_fp32_tn_act_groupgemm.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mxfp8_residual_fail_classes_repro.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mxfp8_verify_fail_repro.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/plr_zero.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/prediction_bf16.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/preloop_local_read_wait.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/ss_bss.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_bf16.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_fp8.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp4.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_bias_sav.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt192x256_plr1_wgm4.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt256x192_plr1_wgm1.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt256x320_plr0_wgm1.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt320x256_plr1_wgm4.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_multi_du_multi_partition.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_tail_loop.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_tail_loop_smoke.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/xfp32.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gsuasb.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gsuc.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/hh_f8nhs.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/i8_gsu.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/i8_gsu_gfx940.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/icache_flush.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/largeMT.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/large_size.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/lsu_fnuz.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/lsu_i8.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/mbskPrefetchOpt.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/mix_cvt_after_ds_fnuz.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/nan_bounds_check_odd_padding.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/no_shadowinit.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/sia0_pgr.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/swizzleA.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/swizzleB.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/swizzle_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/tail_small_size.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/ulsgro1.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/use_beta_false.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/wgm.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/xfp32.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/zgemm.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/fp16_gradient_activation.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/fp16_gradient_bias.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/fp8bf8nss_gradient_bias_b.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx11/fp16_gradient_activation_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx11/fp16_gradient_bias_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_bgrada_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_bgradb_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_bgradd_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_dgelu_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_bgrada_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_bgradb_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_bgradd_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_dgelu_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250/bbs_bgrada_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250/bbs_bgradb_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250/bbs_bgradd_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250/bbs_dgelu_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250/hhs_bgrada_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250/hhs_bgradb_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250/hhs_bgradd_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250/hhs_dgelu_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx950/bf16_gradient_bias.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/groupedgemm/gfx11/grouped_gemm_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/groupedgemm/gfx11/grouped_gemm_userargs_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/groupedgemm/grouped_gemm.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/groupedgemm/grouped_gemm_ck_gfx942.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/groupedgemm/grouped_gemm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/groupedgemm/grouped_gemm_userargs.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gsu/auto_gsu.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gsu/auto_gsu_zero_dim.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gsu/f32_gsu.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gsu/gsu_mbsk.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sgemm_dual_fmac_asm.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sgemm_streamk_asm.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sgemm_xf32_asm.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/bf16_activation.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/f16_activation.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8bs.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8bs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8f8.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8f8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8hs.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8hs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_bf16.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_bf16_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f16.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f16_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8_ml.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8b8.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8b8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8bs.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8bs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8hs.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8hs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_fp16_ml1.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8bs.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8bs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8hs.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8hs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8is.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8is_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_all.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_f16_transposes.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_gl2prefetch.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_ldspadding.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_multicast.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/bf16_activation.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/f16_activation.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_b8.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_b8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_b8bs.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_b8bs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_b8f8.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_b8f8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_b8hs.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_b8hs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_bf16.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_bf16_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_f16.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_f16_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_f8.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_f8_ml.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_f8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_f8b8.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_f8b8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_f8bs.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_f8bs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_f8hs.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_f8hs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_fp16_ml1.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_gate.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_i8.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_i8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_i8bs.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_i8bs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_i8hs.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_i8hs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_i8is.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_i8is_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_all.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_f16_transposes.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_gl2prefetch.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_iteratemode.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_ldspadding.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_multicast.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_split_xfail.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/bf16_activation.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/bf16_gate_r.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/f8n_gate_r.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/fp16_activation.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/fp16_gate_r.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/i8_activation.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/libray_logic.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_b8f8n.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_b8f8n_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_bf16.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_bf16_sb.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_bf8n.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_bf8n_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_f8b8n.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_f8b8n_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_f8n.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_f8n_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi16.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi16_sb.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi32.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi32_sb.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_gsuc.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi16.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi16_sb.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi32.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi32_sb.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_i8bs.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_i8bs_sb.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_i8hs.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_i8hs_sb.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_i8is.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_i8is_sb.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_vw_lg_one.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_vw_lg_one_sb.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/use_sgpr_for_gro.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/use_sgpr_for_gro_sb.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/bf16_gate_r.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/f8_gate_r.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/fp16_gate_r.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_b8.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_b8_sb.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_bf16.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_bf16_sb.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_dtl.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_f16.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_f16_sb.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_f8.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_f8_sb.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_i8.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_i8_sb.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_i8bs.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_i8hs.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_ldstr.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_vw_lg_one.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_vw_lg_one_sb.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/data_parallel_static_mx32f4.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/data_parallel_static_sgemm.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx11/sk_bgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx11/sk_hgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx12/sk_bgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx12/sk_hgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/bf16_vector_epilogue_persistent.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf4.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf4_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf8.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf8_halfplr_tdm_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf8_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_bf16gemm_tdm_subtile.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_bgemm_tdm_split.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_bgemm_tdm_split_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_dynamic_hgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_dynamic_sgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_f8gemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_halfplr_f8gemm_tdm.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_halfplr_mxf8gemm_tdm.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_hgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_hybrid_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_explicit.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_pap_prefetchgl2.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_tdm.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_tdm_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf6b8gemm_tdm.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf6b8gemm_tdm_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8b6gemm_tdm.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8b6gemm_tdm_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8f4gemm_tdm.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8f4gemm_tdm_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_explicit.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_pap_prefetchgl2.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm_split.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm_split_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_sgemm_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_sgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_tdm_general_batched.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_tdm_subtile_general_batched.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/data_parallel_cluster_multicast_boundaries.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf4_cluster_multicast_ext.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf4_pap_cluster_multicast_ext.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf8_cluster_multicast_ext.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf8_pap_cluster_multicast_ext.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/sk_hgemm_debug_modes.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/sk_mxf4gemm_tdm_cluster_ext.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/sk_mxf4gemm_tdm_ext.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/sk_mxf4gemm_tdm_pap_ext.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/data_parallel_static_hgemm_use_e.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/data_parallel_static_sgemm_pap.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_bgemm_pap.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_hgemm_use_e.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_hybrid_xcc_mapping.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxf4gemm_pap.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_basic.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_batch_combo.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_full_coverage.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_streamk_combo.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_atomic_variants.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_basic_variants.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_batch_mi_combo.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_batch_sizes.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_comprehensive.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_debug_modes.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_full_coverage.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_activation_combo.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_basic.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_batch_combo.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_full_coverage.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_streamk_combo.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_matrix_instructions.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_problem_sizes.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_tree_reduction.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_xcc_mapping.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_debug_modes.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_dynamic_pap.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_hybrid_pap.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_pap.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_I8gemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_bgemm_div.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_bgemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_bgemm_sia1.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_cgemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_dgemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_dynamic.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_dynamic_mxfp4.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_dynamic_work_stealing.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_f8gemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_f8ngemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_hgemm_epilogues.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_hgemm_lsu.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_hgemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_hgemm_race.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_hybrid.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_hybrid_work_stealing.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_mx32f4_quick.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_mx32f8_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_queue_selfclean.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_reduction.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_sgemm_alpha0.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_sgemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_zgemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_aiter_bf16.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_aiter_f4.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_ck.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_rocblas_gemv_bf16_tn_m1.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_rocblas_gemv_bf16_tn_m2.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_rocblas_gemv_bf16_tn_m4.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_rr.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_rr_f6.yaml] FAILED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_rr_mx.yaml] FAILED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_tensile_sk.yaml] SKIPPED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_triton_f4.yaml] PASSED +tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_wave_bf16.yaml] PASSED +tensilelite/Tests/common/test_config_helpers.py::test_base_skip_gfx1250_inert_on_strict_target PASSED +tensilelite/Tests/common/test_config_helpers.py::test_ffm_fail_inert_on_hardware PASSED +tensilelite/Tests/common/test_config_helpers.py::test_ffm_fail_inert_on_other_arch PASSED +tensilelite/Tests/common/test_config_helpers.py::test_ffm_fail_xfails_under_ffm PASSED +tensilelite/Tests/common/test_config_helpers.py::test_find_available_archs_does_not_map_the_retired_v0_name_to_the_base PASSED +tensilelite/Tests/common/test_config_helpers.py::test_find_available_archs_keeps_stepping_name PASSED +tensilelite/Tests/common/test_config_helpers.py::test_skip_gfx1250_strict_fires_on_strict_target PASSED +tensilelite/Tests/common/test_config_helpers.py::test_skip_gfx1250_strict_inert_on_base_target PASSED +tensilelite/Tests/common/test_config_helpers.py::test_unmarked_config_never_xfails_under_ffm PASSED diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_run.md b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_run.md new file mode 100644 index 000000000000..fd759fecee8e --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_run.md @@ -0,0 +1,288 @@ +# gfx950 run-side measurement after the client and harness changes (P3) + +Run-side measurement at the end of P2 of the implementation plan (`timing_runs/DECISIONS.md`, +P3), compared with the P0.2 baseline (`baseline.md`). It covers the client changes P1.1–P1.29 and +the Python/YAML/harness changes P2.1–P2.7. + +## What was measured + +| Item | Value | +|---|---| +| Commit | `3b4ec8e125d` (end of P2; detached worktree) | +| Client | `tensilelite-client` built from that commit, CMake cache identical to the P0.2 client's (paths normalised): Release, `GPU_TARGETS=gfx950`, `/usr/bin/amdclang++` (ROCm 7.2.1), msgpack only, `TENSILELITE_BUILD_TESTING=OFF`. sha256 `15f20fa8a2eb…`. | +| Baseline client | P0.2's (`2083e6254bf`, sha256 `4b2769f7f9a2…`), its worktree and artifacts, for the same-session control runs | +| Artifacts | Split build phase at `3b4ec8e125d` into a new directory, separate caches | +| Python | venv Python 3.12.3; TensileLite Python sources from each worktree; venv `rocisa` (unchanged) | +| Host | `smci355-1b112-b13-15` (as P0.2): 2× EPYC 9575F, 8× MI355X (gfx950), ROCm 7.2.1 | +| Cores / threads | `taskset -c 0-24`, `OMP_NUM_THREADS=25`, `OPENBLAS_NUM_THREADS=25` (as P0.2) | +| GPU | Per run, see below: the P0.2 GPU (`ROCR_VISIBLE_DEVICES=0`, PCI `75:00.0`) when idle, otherwise another idle MI355X (`ROCR` 5 = PCI `85:00.0` or 6 = `e5:00.0`, NUMA node 1) | +| Config list | The 185 configs that passed the split run phase: P0.2's 184 plus the new `gemm/gfx950/prediction_bf16` (P2.5) | + +Committed results: + +- `after_run/AGGREGATE_analysis.txt`: per-config run aggregate, P3 client, 185 logs. +- `after_run/common_test_outcomes.txt`: outcome of every test in the split run phase. + +Raw logs (not committed, as for P0.2) are in `/home/alvasile/timing_runs_raw/P3/`, gzipped: +the per-config logs of both clients, the split logs, the P0.2-client control aggregate and the +monitoring logs. + +## Conditions: why the comparison is same-session + +The host was shared throughout (2026-10-31 18:40 to 2026-11-01 14:30 UTC): + +- Another user's corpus run again pinned jobs to cores 0–8, 10–47 and 9 (the job described in + `baseline.md`), from 19:25 to about 03:00. It was gone during the per-config pass and the two + clean split pairs (no CPU growth of it in any of those runs). +- A root vLLM server held the measurement GPU (and rocm-smi GPUs 0–3, i.e. `ROCR` 0–3) at 100% / + 279 GB for hours at a time; other root jobs used the remaining GPUs in turns. +- `/` filled up twice (0 bytes free), so pytest temp dirs, caches and output were moved to + `/dev/shm` (RAM); this applied to both clients alike. + +An uncontended 1-hour window for a P0.2-style serial pass did not occur: a clean pass started at +03:43 got through 36 of 185 configs before the measurement GPU was taken again. So, as decided +during the run, the per-config pass was measured **interleaved**: for each config, the P0.2 +client (with P0.2's worktree and artifacts) and the P3 client ran back to back on the same idle +GPU, alternating which went first, with the same script and settings as P0.2 +(`run_per_config.sh`, serial, `TimingInstrumentation=True`, 25 pinned cores). A GPU was idle when +it showed < 5% use and < 2 GB VRAM twice; if another user's process took more than 1 GB on it +during a pair, the pair was re-run (up to three tries). So each config's two numbers share their +conditions, and the P0.2-client sum is a same-session control. Absolute numbers are not +comparable with the P0.2 pass on their own (see "Control"). + +- GPUs used per pair: `ROCR` 5 for 130 pairs, 0 for 46, 6 for 8. +- Other users' CPU on 0–24: no CPU growth of pinned corpus jobs during any run; unpinned root + processes (vLLM workers, downloads) were parked on idle cores of 0–24 at times, as in P0.2. +- Two pairs, `gemm/gfx950/subtile_mxfp4` and `subtile_mxfp8`, had another process start on the + GPU during every one of three tries; they are kept (both clients were exposed) and shown + separately below. +- The run script clears the caches at the start of each call. One call per config (to re-run single + pairs) therefore clears the helper-kernel cache before every config, while P0.2 cleared it once. + That costs both clients the same ~21 s of helper-kernel compiles (see "Control"). + +## Correctness: common-test pass/fail set + +Split run phase (`--use-cache`, `-n 4`) with the P3 client and artifacts: 568 tests, **195 +passed, 2 failed, 371 skipped**. Compared with `baseline_run/common_test_outcomes.txt`, the only +difference is the new config: + +- `test_config_run[gemm/gfx950/prediction_bf16.yaml]` PASSED (new in P2.5). + +Three P3 split runs (2026-10-31 23:44 on `ROCR` 0, 2026-11-01 12:20 and 13:59 on `ROCR` 5) gave +identical outcome lists. The split build phase also differs from P0.2's only by that config: +196 passed, 2 failed (the known `custom_rr_f6`/`custom_rr_mx` build failures), 370 skipped. + +Changes that were intended and do not change an outcome: + +- P2.6 removed the `LibraryLogic:`/`LibraryClient:` sections of the 14 `ductile/*`, the 10 gfx950 + mxfp8 configs and the 12 `custom/*` configs. Of those, the ones that run on gfx950 all pass: + the 8 `ductile/gfx950/*`, the 10 mxfp8 configs and 7 `custom/*` (`aiter_bf16`, `aiter_f4`, + `rocblas_gemv_bf16_tn_m1/m2/m4`, `triton_f4`, `wave_bf16`); their run phase no longer writes + logic files or runs the library client. The 6 `ductile/gfx1250/*` and `custom_ck`, + `custom_rr`, `custom_tensile_sk` are skipped on gfx950, and `custom_rr_f6`/`custom_rr_mx` are + the 2 known build-phase failures, all as in P0.2. +- P1.22 checks the Synchronizer after every run iteration. `streamk/gfx950/sk_sgemm_debug_modes` + omits Stream-K reduction steps on purpose, so it sets `CheckSynchronizer: False`; it passes. +- P2.3 runs the client at `ClientLogLevel=0` and P2.4 changed `gemm/gfx950/xfp32`'s warmups: + every gfx950 config passes as before. P2.7 set `SyncsPerBenchmark: 0` in `custom/*`: the same + 7 gfx950 custom configs pass; the other 5 are skipped or fail as in P0.2 (above). + +The P0.2 client's split runs in this session: identical to the baseline on a free GPU +(2026-11-01 12:36 and 13:17). On 2026-10-31 22:31, while vLLM shared the measurement GPU, +`streamk/gfx950/sk_sgemm_hybrid_pap` **failed validation** with the P0.2 client (1.39 M of +8.4 M values wrong, 67 s instead of 2–4 s). It passed in every other run of either client. This +is a pre-existing sensitivity of that Stream-K config to sharing the GPU with another process, +not a P3 change; it is listed under "Findings". + +## Uninstrumented split run phase (`-n 4`) + +Same command as `split/run_split.sh`'s run phase, both clients back to back on one GPU: + +| Pair | GPU | Conditions | P0.2 client | P3 client | Δ | +|---|---|---|---:|---:|---:| +| A–B, 2026-10-31 22:31 / 23:44 | `ROCR` 0 | corpus job on 0–24 throughout; vLLM on the GPU for the last ~28 min of the P0.2 run; during the P3 run one 30-s monitor sample showed another process with 13.8 GB on the GPU | 3,115 s | 966 s | not comparable | +| B–A, 2026-11-01 12:20 / 12:36 | `ROCR` 5 | clean (no corpus CPU, GPU free) | 2,445 s | 941 s | **−61.5%** | +| A–B, 2026-11-01 13:17 / 13:59 | `ROCR` 5 | clean | 2,499 s | 943 s | **−62.3%** | + +The P0.2 split run (2,950 s, contended) in `baseline.md` is an upper bound; the two clean +same-session pairs (ABBA order) are the comparison: P0.2 client 2,445–2,499 s, P3 client +941–943 s, **about −62%**. The run-phase critical path changed (B–A pair): the slowest P0.2 +tests were `subtile_mxfp4` (561 s), `spmm_f16_sb` (377 s), `sk_dynamic_mxfp4` (336 s), `spmm_bf16` +(332 s) and `sk_I8gemm_quick` (320 s); with P3 they are `subtile_mxfp4` (163 s), +`custom_mainloop_scheduling` (159 s), `dtl` (128 s), `plr_zero` (105 s), `bufferLoadStore0` +(101 s). The call-phase per-test sum fell from 8,929 s to 3,389 s (−62%). (With `-n 4`, four +25-thread OpenMP clients share 25 cores, so per-test times are larger than in the serial pass.) + +## Run side (per-config, instrumented) + +Seconds from "TIMING BY PURPOSE", summed over the 184 configs common to both (P3's +`prediction_bf16` adds 3.9 s and is in the committed aggregate only). "P0.2" is the committed +baseline pass, re-aggregated with the analyzer at `3b4ec8e125d` (byte-identical to the committed +`baseline_run/AGGREGATE_analysis.txt`, so analyzer changes contribute nothing). "Control" is the +P0.2 client in the interleaved pass. + +| Category | P0.2 | Control | P3 | Δ vs P0.2 | Δ vs control | +|---|---:|---:|---:|---:|---:| +| TOTAL RUNTIME | 2,401.1 | 2,655.0 | 1,024.5 | −57.3% | −61.4% | +| RUN: benchmark client | 2,284.9 | 2,519.0 | 887.7 | −61.1% | −64.8% | +| Reference GEMM (CPU) | 1,044.4 | 1,138.2 | 227.6 | −78.2% | −80.0% | +| … `solve_cpu_slow` | 689.0 | 740.1 | 0.6 | −99.9% | −99.9% | +| … `solve_cpu_fast` | 198.9 | 229.6 | 226.9 | +14.1% | −1.2% | +| … per-solution reference re-runs | 156.3 | 168.3 | – | −100% | −100% | +| Input data preparation | 500.7 | 582.7 | 122.5 | −75.5% | −79.0% | +| … `gpu_input_reset` | 298.8 | 328.7 | 46.1 | −84.6% | −86.0% | +| … `cpu_data_init` | 105.0 | 151.1 | 52.4 | −50.1% | −65.3% | +| … MX per-solution input (`DataInitialization`) | 54.9 | 61.4 | 3.1 | −94.3% | −94.9% | +| … `gpu_input_preparation` | 41.3 | 40.7 | 20.4 | −50.6% | −49.9% | +| Validation | 329.1 | 341.6 | 112.5 | −65.8% | −67.1% | +| … Synchronizer check | 165.4 | 169.9 | 11.8 | −92.9% | −93.1% | +| … `ReferenceValidator` | 163.3 | 171.3 | 100.3 | −38.5% | −41.4% | +| Client launch (startup, setup & teardown) | 289.8 | 316.3 | 284.4 | −1.8% | −10.1% | +| Per-solution bookkeeping & reporting | 54.8 | 57.4 | 13.7 | −75.0% | −76.2% | +| Kernel launch & execution | 35.7 | 39.3 | 126.0 | +252.7% | +220.2% | +| … `warmup_first_sync` (new) | – | – | 93.9 | | | +| Library client (Python) | 24.6 | 37.3 | – | −100% | −100% | +| Library logic (Python) | 5.8 | 6.3 | 1.0 | −81.9% | −83.4% | +| BUILD: solution generation & compile | 63.1 | 85.4 | 85.6 | +35.8% | +0.3% | +| MEASUREMENT OVERHEAD (instrumentation) | 52.8 | 50.2 | 50.1 | −5.1% | −0.2% | +| Sum of per-config pytest wall times | 2,942 | 3,212 | 1,571 | −46.6% | −51.1% | + +Over the 182 pairs without GPU interference the pytest sums are 2,917 s (control) and 1,394 s +(P3), −52.2%. The two GPU-shared pairs: `subtile_mxfp8` 203 → 109 s, `subtile_mxfp4` 92 → 68 s. + +Per-config pytest seconds (P0.2 / control / P3) of the largest configs: `sk_I8gemm_quick` +311 / 323 / 6, `sk_f8gemm_quick` 262 / 295 / 12, `spmm_f16_sb` 124 / 133 / 44, `plr_zero` +94 / 102 / 52, `dtl` 82 / 90 / 59, `fp16_tn` 87 / 87 / 43, `bufferLoadStore0` 81 / 86 / 60, +`mxfp4_mxfp4_bf16_tn_act` 31 / 38 / 4, `xfp32` 62 / 66 / 24. No config got slower by more +than 2 s against its pair (ten configs are 1–2 s slower, all with 3–16 s runs). + +What the numbers show: + +- **The work is the same.** The main call counts are identical to P0.2 (28,596 + `cpu_reference_gemm`, 2,355,016 solutions, 2,106,835 `gpu_input_reset`, 855 client starts). + Counts changed only where an item changed them on purpose: 10 library-client invocations fewer + (P2.6), 592 instead of 1,947 slow-path reference calls (the rest moved to the fast path, + P1.2–P1.11), and the renamed/new timers of P1.19–P1.23. +- **CPU reference (P1.1–P1.11):** the slow path went from 740 s (control; 689 s in P0.2) to + 0.6 s. The fast path took over those problems at about no net cost: 1,355 more calls (26,656 → + 28,011) in 226.9 s against the control's 229.6 s (−1.2%; +28 s against the P0.2 pass, which ran + faster than the control). The per-solution re-runs of the reference (168 s in the control, + 156 s in P0.2) are gone: they were caused by re-uploaded inputs, which P1.16–P1.19 skip. +- **Input preparation (P1.13–P1.19):** `gpu_input_reset_copy_valid_to_gpu` 175 s → 0.1 s, + batched-pointer uploads 93 → 19 s, MX regeneration 50 → 1.5 s, `cpu_data_init` −50%. +- **Validation (P1.20–P1.23):** the Synchronizer check is a GPU OR-reduction now (165 → 12 s); + `validate_gpu_readback` 94 → 36 s. +- **Bookkeeping (P1.24–P1.29, P2.3):** −75%. +- **Client launch is unchanged** (`hip_initialization` 128 s, `process_exit` 68 s for 855 + client starts): no item of the plan targeted it. It is now the largest category (28% of the + P3 total). + +## Moved as expected, or for a known reason + +- **Kernel launch & execution +87 s vs the control (+90 s vs P0.2)** is attribution, as the + plan expected: P1.20 times the wait for the first warmup's completion in its own timer, + `warmup_first_sync` (93.9 s). Before, + that GPU time was hidden in whichever blocking call came next (`validate_gpu_readback`, the + Synchronizer readback, input uploads), which is part of why those dropped by more than their + own speedups. +- **BUILD +22 s** against P0.2, but equal for both clients (85.4 / 85.6 s): the interleaved pass + clears the helper-kernel cache before every config (see "Conditions"); P0.2 cleared it once. + `python_kernel_compilation` 43.5 → 65 s in both. +- **Library client −24.6 s, library logic −4.8 s** (P2.6), and `flush_timing_buffer` −10 calls. + +## Control: P0.2 client in this session versus the P0.2 pass + +The same client and artifacts took 10.6% longer in total than in the P0.2 pass (2,655 vs +2,401 s; pytest sums 3,212 vs 2,942 s). P0.2's clean-repeat spread was about 1%, so the session +was slower. The difference is: + +- BUILD +22 s: the cache clearing above. +- Library client +12.7 s (`python_library_client_create_library`, Python). +- CPU reference +94 s (+9%), `cpu_data_init` +46 s, `process_exit` +18 s, input reset +30 s. + Most of `cpu_data_init` is one config, `sk_bgemm_div` (one 2.35·10⁹-element tensor): 32.7 s + instead of 6.1 s (P3 client: 3.7 s). These are host-memory-heavy phases. During the pass + `/dev/shm` held 1.2–1.3 TB of other users' data in RAM, and 130 pairs ran on a GPU on the + other NUMA node. Either may explain it, but neither was isolated. + +So the deltas against the control are the like-for-like ones. The deltas against P0.2 are +smaller, because the control ran slower than P0.2 did. + +## P1.12 replay (topic 1, slow-path reasons) + +After P1.2–P1.11 (`44cd1a416fe`), the 59 slow-path YAMLs of the investigation were replayed with +an uncommitted client that logs each reference call's path and reason (rule 8). **All 59 pass.** +The reference calls are the same as in the investigation's replay (2,517). Slow-path calls went +from 1,966 (546.6 s) to 593 (0.53 s), and fast-path calls from 551 to 1,924. The remaining slow +calls are the expected ones: `NumElementsToValidate: 128` (sparse validation, 569 calls in 20 +YAMLs) and `scaleCD` (24 calls, `gemm/gfx950/f8f16mix_f8s`). No other reason occurred. The +per-config pass here is consistent with it: 592 slow-path calls, 0.6 s. + +## Findings that bear on the run side (not changed by the plan) + +Decisions needed: + +- **P2.5, option A (user decision).** With `--best-solution`, the library client silently falls back + to library solution 0 when Origami selection returns nothing, and exits 0. A client option that + fails on an empty selection (written only for the `LibraryClient` step) would let the new + `prediction_bf16` config detect selection regressions. Not implemented; the plan said to stop + and ask. +- **P1.28, `sync-after-warmups` default (user decision).** The client defaults + `sync-after-warmups` to true, and `ClientWriter.py` never writes it. So P1.28's warmup-event + drop never triggers in Tensile-generated runs. Measured on `dtl` with `sync-after-warmups=False`: + −6.7 s (−13%) uninstrumented wall per pass, and 18,872 → 0 `hipEvent*` calls on one ini. Options: + `ClientWriter` writes `sync-after-warmups=False` for correctness-only runs, or the client stops + treating that sync as an event reader when nothing is timed. + +Pre-existing issues (candidates for a deferred list): + +- `sk_sgemm_hybrid_pap` failed validation once with the P0.2 client while another process + used the GPU (above). +- `cpu-gemm-driver --bias` with `--batchCount > 1` fails on both reference paths: the driver + registers the bias with batch stride m but allocates m elements, so batches > 0 read past the + buffer (test infrastructure only; no CTest combines them; P1.9). +- Client launch (`hip_initialization`, `process_exit`, `factory_hip_get_device_properties`) is + now 28% of the instrumented run total and was not in scope. +- GuardPageAll configs (`bufferLoadStore0`) re-upload batched-pointer arrays every run, because + the Front/Back start pointers differ; one cached array per bounds-check mode would recover + most of the remaining `gpu_input_reset_batched_inputs` (P1.18). +- `f8f16mix_f8s` and 3 calls of `use_beta_false` stay on the slow path (`scaleCD`, P1.7/P1.12), + and the slow path applies ScaleAB differently from the fast path when a Half operand is + stored wider than its F8 compute type (P1.7). +- The slow path's MX K tail is wrong for K not a multiple of the MX block; P1.11 moved those + problems to the fast path, and the slow path is unchanged (P1.11). +- Slow-path bias-gradient error handling (missing `throw`, double `free`) for batched source + shapes (P1.10). +- `generateMXInput`: the K tail in the re-arranged orientations is not deterministic, and neither + is `norm_dist` between calls (P1.13). `rocm_random` shares one `std::mt19937` across OpenMP + threads (data race, P1.15). Trig init takes an `int` index, which wraps above 2³¹ elements + (`sk_bgemm_div`, P1.14). +- The MX GFX950 scale swizzle with a padded leading dimension (P1.19). MX configs crash in NaN + and GuardPage modes (P1.19). Grouped GEMM with bounds checking is broken (P1.17). None of these + is used by a common config. +- Client: `selection-only=True` segfaults (P1.25); `max-enqueues-per-sync=0` hangs (P1.27); the + device-lost path exits via SIGABRT (P1.20). +- `ductile/gfx950/f8_tn_mx_subtile` sets `NumElementsToValidate: 0` but still validates 128 + elements (P1.12). + +## Reproducing + +```bash +git worktree add --detach /tmp/tl_plan/P3/wt 3b4ec8e125d +source /home/alvasile/venv/bin/activate +cmake --preset tensilelite -S /tmp/tl_plan/P3/wt/projects/hipblaslt -B /tmp/tl_plan/P3/build_client \ + -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DGPU_TARGETS=gfx950 \ + -DCMAKE_CXX_COMPILER=/usr/bin/amdclang++ -DCMAKE_C_COMPILER=/opt/rocm/bin/amdclang \ + -DHIPBLASLT_ENABLE_YAML=OFF -DTENSILELITE_ENABLE_CLIENT=ON -DTENSILELITE_BUILD_TESTING=OFF \ + -DTENSILELITE_CLIENT_ENABLE_ROCPROFSDK=OFF -DCMAKE_EXPORT_COMPILE_COMMANDS=ON +taskset -c 128-191 cmake --build /tmp/tl_plan/P3/build_client -j64 --target tensilelite-client +export TL=/tmp/tl_plan/P3/wt/projects/hipblaslt/tensilelite +export CLIENT=/tmp/tl_plan/P3/build_client/tensilelite/client/tensilelite-client +export ART=/tmp/tl_plan/P3/artifacts TENSILE_CACHE=/tensile TENSILELITE_CACHE=/tensilelite +OUT= bash $TL/timing_runs/full_runs_gfx950/split/run_split.sh # artifacts + split run +OUT= KEEP_CACHE=1 bash $TL/timing_runs/full_runs_gfx950/per_config/run_per_config.sh +``` + +The interleaved pass ran `run_per_config.sh` once per config and client, with +`ROCR_VISIBLE_DEVICES` set to the pair's GPU and `TMPDIR`, caches and `OUT` under `/dev/shm` +(driver scripts and the per-config script copy are kept with the raw logs in +`/home/alvasile/timing_runs_raw/P3/`, see its `README.txt`). diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_run/AGGREGATE_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_run/AGGREGATE_analysis.txt new file mode 100644 index 000000000000..01e8b4120a52 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_run/AGGREGATE_analysis.txt @@ -0,0 +1,735 @@ +================================================================================================================================= +TIMING ANALYSIS SUMMARY +================================================================================================================================= + +TIMING BY PURPOSE +--------------------------------------------------------------------------------------------------------------------------------- + Category Count Total (ms) Mean (ms) % of Parent % of Wall +--------------------------------------------------------------------------------------------------------------------------------- + TOTAL RUNTIME 1,028,384.20 100.0% + RUN: benchmark client 891,532.00 86.7% 86.7% + Client launch (startup, setup & teardown) 284,728.58 31.9% 27.7% + hip_initialization 856 127,976.07 149.50 44.9% 12.4% + process_exit 856 67,612.92 78.99 23.7% 6.6% + main_problem_factory 856 34,298.15 40.07 12.0% 3.3% + factory_hip_get_device_properties 192 34,095.91 177.58 99.4% 3.3% + (untracked) 202.25 0.6% 0.0% + process_startup 856 13,627.96 15.92 4.8% 1.3% + code_object_loading 856 10,927.55 12.77 3.8% 1.1% + library_loading 856 9,743.29 11.38 3.4% 0.9% + main_teardown_locals 856 9,697.28 11.33 3.4% 0.9% + data_init_setup 856 9,445.57 11.03 3.3% 0.9% + lazy_loading_init 856 953.97 1.11 0.3% 0.1% + main_parse_args 856 359.25 0.42 0.1% 0.0% + python_client_write_run_script 856 41.59 0.05 0.0% 0.0% + reporter_setup 856 22.85 0.03 0.0% 0.0% + listener_setup 856 12.29 0.01 0.0% 0.0% + solution_iterator_setup 856 9.81 0.01 0.0% 0.0% + Reference GEMM (CPU) 227,584.42 25.5% 22.1% + cpu_reference_gemm 28,596 227,526.92 7.96 100.0% 22.1% + solve_cpu_fast 28,011 226,894.88 8.10 99.7% 22.1% + solve_cpu_slow 592 614.05 1.04 0.3% 0.1% + (untracked) 17.99 0.0% 0.0% + pre_solution_listener.ReferenceValidator 2,355,018 57.51 0.00 0.0% 0.0% + pre_solution_ref_skipped 2,354,328 44.95 0.00 78.2% 0.0% + (untracked) 12.56 21.8% 0.0% + Kernel launch & execution 125,980.76 14.1% 12.3% + warmup_first_sync 2,106,827 93,859.44 0.04 74.5% 9.1% + warmup_runs 4,213,654 25,891.30 0.01 20.6% 2.5% + kernel_solving 2,106,835 4,562.94 0.00 3.6% 0.4% + warmup_events_setup 2,106,835 827.64 0.00 0.7% 0.1% + validate_warmups_listener.BenchmarkTimer 2,106,827 351.37 0.00 0.3% 0.0% + validate_gpu_sync 2,106,827 307.63 0.00 87.6% 0.0% + (untracked) 43.74 12.4% 0.0% + benchmark_runs 2,106,835 267.02 0.00 0.2% 0.0% + benchmark_validate_enqueues 288 104.47 0.36 39.1% 0.0% + validate_enqueues_listener.BenchmarkTimer 288 103.54 0.36 99.1% 0.0% + benchmark_sleep 288 103.48 0.36 99.9% 0.0% + benchmark_event_elapsed_time 8 0.03 0.00 0.0% 0.0% + benchmark_event_destroy 8 0.01 0.00 0.0% 0.0% + (untracked) 0.02 0.0% 0.0% + validate_enqueues_listener.ProgressListener 288 0.78 0.00 0.8% 0.0% + validate_enqueues_listener.MetaResultReporter 288 0.02 0.00 0.0% 0.0% + validate_enqueues_listener.DataInitialization 288 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.ReferenceValidator 288 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.HardwareMonitorListener 288 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.AllSolutionsIterator 288 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.SynchronizerValidator 288 0.01 0.00 0.0% 0.0% + (untracked) 0.09 0.1% 0.0% + benchmark_post_syncs 2,106,835 74.22 0.00 27.8% 0.0% + benchmark_pre_syncs 2,106,835 52.38 0.00 19.6% 0.0% + gpu_kernel_execution 288 35.88 0.12 13.4% 0.0% + benchmark_events_setup 288 0.06 0.00 0.0% 0.0% + benchmark_events_teardown 288 0.02 0.00 0.0% 0.0% + run_iteration_sync 2,106,835 221.04 0.00 0.2% 0.0% + Input data preparation 122,498.81 13.7% 11.9% + cpu_data_init 28,596 52,377.99 1.83 42.8% 5.1% + gpu_input_reset 2,106,835 46,059.55 0.02 37.6% 4.5% + gpu_input_reset_copy_inputs_d2d 1,473,329 24,899.36 0.02 54.1% 2.4% + gpu_input_reset_batched_inputs 1,473,329 19,254.70 0.01 41.8% 1.9% + gpu_input_reset_fast_reset_output 633,454 1,658.62 0.00 3.6% 0.2% + gpu_input_reset_convert_to_inputs 1,473,329 101.59 0.00 0.2% 0.0% + gpu_input_reset_copy_valid_to_gpu 719,180 95.95 0.00 0.2% 0.0% + gpu_input_reset_slow_path 1,473,329 28.18 0.00 0.1% 0.0% + gpu_input_reset_copy_swizzled 15,719 2.10 0.00 0.0% 0.0% + gpu_input_reset_rotating_init 2 0.03 0.01 0.0% 0.0% + gpu_input_reset_constant_inputs 4 0.01 0.00 0.0% 0.0% + gpu_input_reset_init_cpu_inputs 4 0.00 0.00 0.0% 0.0% + (untracked) 19.02 0.0% 0.0% + gpu_input_preparation 28,642 20,424.80 0.71 16.7% 2.0% + gpu_input_prep_copy_swizzled 1,151 9,496.79 8.25 46.5% 0.9% + gpu_input_prep_copy_valid_to_gpu 11,446 8,549.51 0.75 41.9% 0.8% + gpu_input_prep_batched_inputs 23,230 1,178.15 0.05 5.8% 0.1% + gpu_input_prep_init_cpu_inputs 41 708.11 17.27 3.5% 0.1% + gpu_input_prep_copy_inputs_d2d 23,230 443.67 0.02 2.2% 0.0% + gpu_input_prep_fast_reset_output 5,403 27.11 0.01 0.1% 0.0% + gpu_input_prep_convert_to_inputs 23,230 10.27 0.00 0.1% 0.0% + gpu_input_prep_slow_path 23,230 2.73 0.00 0.0% 0.0% + gpu_input_prep_constant_inputs 44 0.51 0.01 0.0% 0.0% + gpu_input_prep_rotating_init 1 0.01 0.01 0.0% 0.0% + (untracked) 7.93 0.0% 0.0% + pre_solution_listener.DataInitialization 2,355,018 3,144.42 0.00 2.6% 0.3% + pre_solution_mx_init_data 16,038 1,525.20 0.10 48.5% 0.1% + pre_solution_mx_side_swizzle 2,200 1,521.52 0.69 99.8% 0.1% + pre_solution_mx_side_cache_hit 29,876 2.44 0.00 0.2% 0.0% + (untracked) 1.24 0.1% 0.0% + pre_solution_mx_cpu_sync_no_regen 16,038 1,208.35 0.08 38.4% 0.1% + pre_solution_mx_copy_inputs_d2d_no_regen 16,038 349.86 0.02 11.1% 0.0% + pre_solution_mx_check 2,355,018 49.20 0.00 1.6% 0.0% + pre_solution_mx_copy_valid_to_gpu_no_regen 16,038 1.84 0.00 0.1% 0.0% + (untracked) 9.97 0.3% 0.0% + rotating_buffer_preparation 28,642 492.04 0.02 0.4% 0.0% + Validation 112,494.26 12.6% 10.9% + validate_warmups_listener.ReferenceValidator 2,106,827 100,346.61 0.05 89.2% 9.8% + validate_element_comparison 1,907,339 56,551.97 0.03 56.4% 5.5% + validate_gpu_readback 1,907,339 35,614.28 0.02 35.5% 3.5% + validate_result_buffer_alloc 1,907,339 7,737.11 0.00 7.7% 0.8% + validate_inputs_copy 1,905,787 175.47 0.00 0.2% 0.0% + validate_setup 1,905,883 39.60 0.00 0.0% 0.0% + validate_report 1,907,339 39.04 0.00 0.0% 0.0% + (untracked) 189.15 0.2% 0.0% + synchronizer_check 2,106,835 11,709.85 0.01 10.4% 1.1% + synchronizer_reduce 608,873 11,629.10 0.02 99.3% 1.1% + synchronizer_inactive 1,497,986 28.05 0.00 0.2% 0.0% + (untracked) 52.71 0.5% 0.0% + validate_warmups_listener.MetaResultReporter 2,106,827 116.49 0.00 0.1% 0.0% + validate_warmups_listener.HardwareMonitorListener 2,106,827 67.33 0.00 0.1% 0.0% + validate_warmups_listener.AllSolutionsIterator 2,106,827 64.13 0.00 0.1% 0.0% + validate_warmups_listener.SynchronizerValidator 2,106,827 63.68 0.00 0.1% 0.0% + (untracked) 63.68 100.0% 0.0% + validate_warmups_listener.DataInitialization 2,106,827 63.33 0.00 0.1% 0.0% + validate_warmups_listener.ProgressListener 2,106,827 62.84 0.00 0.1% 0.0% + Per-solution bookkeeping & reporting 13,672.43 1.5% 1.3% + pre_solution_listener.BenchmarkTimer 2,355,018 2,280.84 0.00 16.7% 0.2% + post_solution_log 2,355,018 1,934.47 0.00 14.1% 0.2% + post_solution_reporting 2,355,018 1,812.60 0.00 13.3% 0.2% + run_current_solution_check 2,355,018 1,687.62 0.00 12.3% 0.2% + run_iteration_teardown 2,106,835 1,667.13 0.00 12.2% 0.2% + pre_solution_listener.AllSolutionsIterator 2,355,018 1,225.59 0.00 9.0% 0.1% + pre_solution_listener.MetaResultReporter 2,355,018 836.79 0.00 6.1% 0.1% + pre_solution_listener.ProgressListener 2,355,018 714.19 0.00 5.2% 0.1% + post_solution_validation 2,355,018 530.92 0.00 3.9% 0.1% + solution_selection 2,355,018 401.58 0.00 2.9% 0.0% + pre_solution_listener.SynchronizerValidator 2,355,018 237.01 0.00 1.7% 0.0% + post_problem 28,642 74.45 0.00 0.5% 0.0% + solution_teardown 2,355,018 64.87 0.00 0.5% 0.0% + pre_solution_listener.HardwareMonitorListener 2,355,018 48.70 0.00 0.4% 0.0% + post_solution_sol_advance 2,355,018 46.16 0.00 0.3% 0.0% + post_solution_perf_calc 2,355,018 45.64 0.00 0.3% 0.0% + post_solution_perf_reset 2,355,018 44.36 0.00 0.3% 0.0% + problem_report_index 28,642 13.85 0.00 0.1% 0.0% + problem_teardown 28,642 4.72 0.00 0.0% 0.0% + finalize_report 856 0.95 0.00 0.0% 0.0% + post_solution_result_file 627 0.02 0.00 0.0% 0.0% + Library client (Python) 3,304.79 0.4% 0.3% + python_library_client_create_library 1 3,020.62 3,020.62 91.4% 0.3% + python_library_client_execution 1 277.59 277.59 8.4% 0.0% + python_library_client_subprocess 1 277.11 277.11 99.8% 0.0% + python_library_client_write_run_script 1 0.04 0.04 0.0% 0.0% + (untracked) 0.45 0.2% 0.0% + python_library_client_write_configs 1 5.80 5.80 0.2% 0.0% + python_library_client_setup 1 0.78 0.78 0.0% 0.0% + Library logic (Python) 1,267.93 0.1% 0.1% + python_logic_write 2 1,064.77 532.38 84.0% 0.1% + python_logic_parse_solutions 2 170.41 85.20 13.4% 0.0% + python_logic_analyze 2 32.75 16.38 2.6% 0.0% + BUILD: solution generation & compile 85,606.85 8.3% 8.3% + Solution generation & build (Python) 64,833.54 75.7% 6.3% + python_kernel_compilation 32 64,833.54 2,026.05 100.0% 6.3% + python_kernel_build_src_co 32 41,324.67 1,291.40 63.7% 4.0% + python_kernel_build_src_co.compile 8 41,220.79 5,152.60 99.7% 4.0% + python_kernel_build_src_co.cache_check 32 47.66 1.49 0.1% 0.0% + python_kernel_build_src_co.unbundle 8 45.67 5.71 0.1% 0.0% + python_kernel_build_src_co.cache_populate 8 6.70 0.84 0.0% 0.0% + python_kernel_build_src_co.setup 32 0.82 0.03 0.0% 0.0% + python_kernel_build_src_co.move 8 0.31 0.04 0.0% 0.0% + (untracked) 2.72 0.0% 0.0% + python_kernel_codegen 32 15,972.79 499.15 24.6% 1.6% + python_kernel_write_assemble 32 2,641.41 82.54 4.1% 0.3% + python_kernel_bench_setup 32 1,765.97 55.19 2.7% 0.2% + python_kernel_validate 32 1,316.28 41.13 2.0% 0.1% + python_kernel_build_co 32 688.42 21.51 1.1% 0.1% + python_kernel_write_helpers 32 479.25 14.98 0.7% 0.0% + python_kernel_bench_postprocess 32 470.30 14.70 0.7% 0.0% + python_benchpost_lib_construction 32 335.31 10.48 71.3% 0.0% + python_benchpost_library_write 32 81.61 2.55 17.4% 0.0% + python_benchpost_naming 32 49.05 1.53 10.4% 0.0% + python_benchpost_client_config 32 3.39 0.11 0.7% 0.0% + (untracked) 0.94 0.2% 0.0% + python_kernel_setup 32 131.57 4.11 0.2% 0.0% + (untracked) 42.88 0.1% 0.0% + Ductile GA search (Python) 20,773.31 24.3% 2.0% + python_ductile_initial_sampling 8 14,625.09 1,828.14 70.4% 1.4% + python_ductile_generate_solutions 32 4,612.18 144.13 22.2% 0.4% + python_ductile_ga_between_evaluations 8 1,484.56 185.57 7.1% 0.1% + python_ductile_setup 8 35.59 4.45 0.2% 0.0% + python_ductile_clear_artifacts 32 10.64 0.33 0.1% 0.0% + python_ductile_read_scores 32 4.85 0.15 0.0% 0.0% + python_ductile_verification 8 0.40 0.05 0.0% 0.0% + MEASUREMENT OVERHEAD (instrumentation) 50,077.42 4.9% 4.9% + flush_timing_buffer 857 29,795.24 34.77 59.5% 2.9% + main_timing_calibration 856 10,109.19 11.81 20.2% 1.0% + timing_overhead 856 5,260.68 6.15 10.5% 0.5% + (timer overhead subtracted from parent timers) 4,912.31 9.8% 0.5% + SHARED: cache & solution files 2,590.39 0.3% 0.3% + python_cache_check 832 2,249.35 2.70 86.8% 0.2% + python_write_solutions 856 251.56 0.29 9.7% 0.0% + python_wsol_prepare 856 89.31 0.10 35.5% 0.0% + python_wsol_prepare_cache 824 85.29 0.10 95.5% 0.0% + python_wsol_prepare_nocache 32 0.94 0.03 1.1% 0.0% + (untracked) 3.07 3.4% 0.0% + python_wsol_header 856 85.68 0.10 34.1% 0.0% + python_wsol_dump 32 63.04 1.97 25.1% 0.0% + (untracked) 13.55 5.4% 0.0% + python_write_cache 32 56.19 1.76 2.2% 0.0% + python_solution_indexing 32 33.30 1.04 1.3% 0.0% + UNATTRIBUTED (untracked time & timer overhead) -1,422.47 -0.1% -0.1% + (untracked in python_benchmark_problems) 2,869.74 0.0% 0.3% + (untracked in validate_warmups) 281.75 0.0% 0.0% + (untracked in pre_solution) 249.50 0.0% 0.0% + (untracked in pre_problem) 146.73 0.0% 0.0% + (untracked in python_library_logic) 1.00 0.0% 0.0% + (untracked in python_client_writer) 0.58 0.0% 0.0% + (children exceed benchmark_runs) -39.12 0.0% -0.0% + (children exceed post_solution) -139.33 0.0% -0.0% + (children exceed python_client_execution) -4,793.32 0.0% -0.5% + +TIMING BY TIMER NESTING (legacy view) +--------------------------------------------------------------------------------------------------------------------------------- + Category Count Total (ms) Mean (ms) % of Parent % of Wall +--------------------------------------------------------------------------------------------------------------------------------- + python_benchmark_problems 185 1,018,897.58 5,507.55 99.1% + ----------------------------------------------------------------------------------------------------------------------------- + python_client_execution 856 927,830.59 1,083.91 91.1% 90.2% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Data Preparation 347,028.05 37.4% 33.7% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + pre_problem 28,642 280,051.64 9.78 80.7% 27.2% + cpu_reference_gemm 28,596 227,526.92 7.96 81.2% 22.1% + solve_cpu_fast 28,011 226,894.88 8.10 99.7% 22.1% + solve_cpu_slow 592 614.05 1.04 0.3% 0.1% + (untracked) 17.99 0.0% 0.0% + cpu_data_init 28,596 52,377.99 1.83 18.7% 5.1% + (untracked) 146.73 0.1% 0.0% + gpu_input_reset 2,106,835 46,059.55 0.02 13.3% 4.5% + gpu_input_reset_copy_inputs_d2d 1,473,329 24,899.36 0.02 54.1% 2.4% + gpu_input_reset_batched_inputs 1,473,329 19,254.70 0.01 41.8% 1.9% + gpu_input_reset_fast_reset_output 633,454 1,658.62 0.00 3.6% 0.2% + gpu_input_reset_convert_to_inputs 1,473,329 101.59 0.00 0.2% 0.0% + gpu_input_reset_copy_valid_to_gpu 719,180 95.95 0.00 0.2% 0.0% + gpu_input_reset_slow_path 1,473,329 28.18 0.00 0.1% 0.0% + (untracked) 19.02 0.0% 0.0% + gpu_input_reset_copy_swizzled 15,719 2.10 0.00 0.0% 0.0% + gpu_input_reset_rotating_init 2 0.03 0.01 0.0% 0.0% + gpu_input_reset_constant_inputs 4 0.01 0.00 0.0% 0.0% + gpu_input_reset_init_cpu_inputs 4 0.00 0.00 0.0% 0.0% + gpu_input_preparation 28,642 20,424.80 0.71 5.9% 2.0% + gpu_input_prep_copy_swizzled 1,151 9,496.79 8.25 46.5% 0.9% + gpu_input_prep_copy_valid_to_gpu 11,446 8,549.51 0.75 41.9% 0.8% + gpu_input_prep_batched_inputs 23,230 1,178.15 0.05 5.8% 0.1% + gpu_input_prep_init_cpu_inputs 41 708.11 17.27 3.5% 0.1% + gpu_input_prep_copy_inputs_d2d 23,230 443.67 0.02 2.2% 0.0% + gpu_input_prep_fast_reset_output 5,403 27.11 0.01 0.1% 0.0% + gpu_input_prep_convert_to_inputs 23,230 10.27 0.00 0.1% 0.0% + (untracked) 7.93 0.0% 0.0% + gpu_input_prep_slow_path 23,230 2.73 0.00 0.0% 0.0% + gpu_input_prep_constant_inputs 44 0.51 0.01 0.0% 0.0% + gpu_input_prep_rotating_init 1 0.01 0.01 0.0% 0.0% + rotating_buffer_preparation 28,642 492.04 0.02 0.1% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + C++ Client Setup & Teardown 294,837.77 31.8% 28.7% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + hip_initialization 856 127,976.07 149.50 43.4% 12.4% + process_exit 856 67,612.92 78.99 22.9% 6.6% + main_problem_factory 856 34,298.15 40.07 11.6% 3.3% + factory_hip_get_device_properties 192 34,095.91 177.58 99.4% 3.3% + (untracked) 202.25 0.6% 0.0% + process_startup 856 13,627.96 15.92 4.6% 1.3% + code_object_loading 856 10,927.55 12.77 3.7% 1.1% + main_timing_calibration 856 10,109.19 11.81 3.4% 1.0% + library_loading 856 9,743.29 11.38 3.3% 0.9% + main_teardown_locals 856 9,697.28 11.33 3.3% 0.9% + data_init_setup 856 9,445.57 11.03 3.2% 0.9% + lazy_loading_init 856 953.97 1.11 0.3% 0.1% + main_parse_args 856 359.25 0.42 0.1% 0.0% + python_client_write_run_script 856 41.59 0.05 0.0% 0.0% + reporter_setup 856 22.85 0.03 0.0% 0.0% + listener_setup 856 12.29 0.01 0.0% 0.0% + solution_iterator_setup 856 9.81 0.01 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Kernel Execution 142,574.80 15.4% 13.9% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + warmup_first_sync 2,106,827 93,859.44 0.04 65.8% 9.1% + warmup_runs 4,213,654 25,891.30 0.01 18.2% 2.5% + pre_solution 2,355,018 8,794.55 0.00 6.2% 0.9% + pre_solution_listener.DataInitialization 2,355,018 3,144.42 0.00 35.8% 0.3% + pre_solution_mx_init_data 16,038 1,525.20 0.10 48.5% 0.1% + pre_solution_mx_side_swizzle 2,200 1,521.52 0.69 99.8% 0.1% + pre_solution_mx_side_cache_hit 29,876 2.44 0.00 0.2% 0.0% + (untracked) 1.24 0.1% 0.0% + pre_solution_mx_cpu_sync_no_regen 16,038 1,208.35 0.08 38.4% 0.1% + pre_solution_mx_copy_inputs_d2d_no_regen 16,038 349.86 0.02 11.1% 0.0% + pre_solution_mx_check 2,355,018 49.20 0.00 1.6% 0.0% + (untracked) 9.97 0.3% 0.0% + pre_solution_mx_copy_valid_to_gpu_no_regen 16,038 1.84 0.00 0.1% 0.0% + pre_solution_listener.BenchmarkTimer 2,355,018 2,280.84 0.00 25.9% 0.2% + pre_solution_listener.AllSolutionsIterator 2,355,018 1,225.59 0.00 13.9% 0.1% + pre_solution_listener.MetaResultReporter 2,355,018 836.79 0.00 9.5% 0.1% + pre_solution_listener.ProgressListener 2,355,018 714.19 0.00 8.1% 0.1% + (untracked) 249.50 2.8% 0.0% + pre_solution_listener.SynchronizerValidator 2,355,018 237.01 0.00 2.7% 0.0% + pre_solution_listener.ReferenceValidator 2,355,018 57.51 0.00 0.7% 0.0% + pre_solution_ref_skipped 2,354,328 44.95 0.00 78.2% 0.0% + (untracked) 12.56 21.8% 0.0% + pre_solution_listener.HardwareMonitorListener 2,355,018 48.70 0.00 0.6% 0.0% + kernel_solving 2,106,835 4,562.94 0.00 3.2% 0.4% + post_solution 2,355,018 4,274.82 0.00 3.0% 0.4% + post_solution_log 2,355,018 1,934.47 0.00 45.3% 0.2% + post_solution_reporting 2,355,018 1,812.60 0.00 42.4% 0.2% + post_solution_validation 2,355,018 530.92 0.00 12.4% 0.1% + post_solution_sol_advance 2,355,018 46.16 0.00 1.1% 0.0% + post_solution_perf_calc 2,355,018 45.64 0.00 1.1% 0.0% + post_solution_perf_reset 2,355,018 44.36 0.00 1.0% 0.0% + post_solution_result_file 627 0.02 0.00 0.0% 0.0% + run_current_solution_check 2,355,018 1,687.62 0.00 1.2% 0.2% + run_iteration_teardown 2,106,835 1,667.13 0.00 1.2% 0.2% + warmup_events_setup 2,106,835 827.64 0.00 0.6% 0.1% + solution_selection 2,355,018 401.58 0.00 0.3% 0.0% + benchmark_runs 2,106,835 227.90 0.00 0.2% 0.0% + benchmark_validate_enqueues 288 104.47 0.36 45.8% 0.0% + validate_enqueues_listener.BenchmarkTimer 288 103.54 0.36 99.1% 0.0% + benchmark_sleep 288 103.48 0.36 99.9% 0.0% + benchmark_event_elapsed_time 8 0.03 0.00 0.0% 0.0% + (untracked) 0.02 0.0% 0.0% + benchmark_event_destroy 8 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.ProgressListener 288 0.78 0.00 0.8% 0.0% + (untracked) 0.09 0.1% 0.0% + validate_enqueues_listener.MetaResultReporter 288 0.02 0.00 0.0% 0.0% + validate_enqueues_listener.DataInitialization 288 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.ReferenceValidator 288 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.HardwareMonitorListener 288 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.AllSolutionsIterator 288 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.SynchronizerValidator 288 0.01 0.00 0.0% 0.0% + benchmark_post_syncs 2,106,835 74.22 0.00 32.6% 0.0% + benchmark_pre_syncs 2,106,835 52.38 0.00 23.0% 0.0% + gpu_kernel_execution 288 35.88 0.12 15.7% 0.0% + benchmark_events_setup 288 0.06 0.00 0.0% 0.0% + benchmark_events_teardown 288 0.02 0.00 0.0% 0.0% + run_iteration_sync 2,106,835 221.04 0.00 0.2% 0.0% + post_problem 28,642 74.45 0.00 0.1% 0.0% + solution_teardown 2,355,018 64.87 0.00 0.0% 0.0% + problem_report_index 28,642 13.85 0.00 0.0% 0.0% + problem_teardown 28,642 4.72 0.00 0.0% 0.0% + finalize_report 856 0.95 0.00 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Reference Computation 113,127.38 12.2% 11.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + validate_warmups 2,106,827 101,417.53 0.05 89.6% 9.9% + validate_warmups_listener.ReferenceValidator 2,106,827 100,346.61 0.05 98.9% 9.8% + validate_element_comparison 1,907,339 56,551.97 0.03 56.4% 5.5% + validate_gpu_readback 1,907,339 35,614.28 0.02 35.5% 3.5% + validate_result_buffer_alloc 1,907,339 7,737.11 0.00 7.7% 0.8% + (untracked) 189.15 0.2% 0.0% + validate_inputs_copy 1,905,787 175.47 0.00 0.2% 0.0% + validate_setup 1,905,883 39.60 0.00 0.0% 0.0% + validate_report 1,907,339 39.04 0.00 0.0% 0.0% + validate_warmups_listener.BenchmarkTimer 2,106,827 351.37 0.00 0.3% 0.0% + validate_gpu_sync 2,106,827 307.63 0.00 87.6% 0.0% + (untracked) 43.74 12.4% 0.0% + (untracked) 281.75 0.3% 0.0% + validate_warmups_listener.MetaResultReporter 2,106,827 116.49 0.00 0.1% 0.0% + validate_warmups_listener.HardwareMonitorListener 2,106,827 67.33 0.00 0.1% 0.0% + validate_warmups_listener.AllSolutionsIterator 2,106,827 64.13 0.00 0.1% 0.0% + validate_warmups_listener.SynchronizerValidator 2,106,827 63.68 0.00 0.1% 0.0% + (untracked) 63.68 100.0% 0.0% + validate_warmups_listener.DataInitialization 2,106,827 63.33 0.00 0.1% 0.0% + validate_warmups_listener.ProgressListener 2,106,827 62.84 0.00 0.1% 0.0% + synchronizer_check 2,106,835 11,709.85 0.01 10.4% 1.1% + synchronizer_reduce 608,873 11,629.10 0.02 99.3% 1.1% + (untracked) 52.71 0.5% 0.0% + synchronizer_inactive 1,497,986 28.05 0.00 0.2% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Timing Overhead 35,055.92 3.8% 3.4% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + flush_timing_buffer 857 29,795.24 34.77 85.0% 2.9% + timing_overhead 856 5,260.68 6.15 15.0% 0.5% + ----------------------------------------------------------------------------------------------------------------------------- + python_kernel_compilation 32 64,833.54 2,026.05 6.4% 6.3% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_src_co 32 41,324.67 1,291.40 63.7% 4.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_kernel_build_src_co.compile 8 41,220.79 5,152.60 99.7% 4.0% + python_kernel_build_src_co.cache_check 32 47.66 1.49 0.1% 0.0% + python_kernel_build_src_co.unbundle 8 45.67 5.71 0.1% 0.0% + python_kernel_build_src_co.cache_populate 8 6.70 0.84 0.0% 0.0% + (untracked) 2.72 0.0% 0.0% + python_kernel_build_src_co.setup 32 0.82 0.03 0.0% 0.0% + python_kernel_build_src_co.move 8 0.31 0.04 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_codegen 32 15,972.79 499.15 24.6% 1.6% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_assemble 32 2,641.41 82.54 4.1% 0.3% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_setup 32 1,765.97 55.19 2.7% 0.2% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_validate 32 1,316.28 41.13 2.0% 0.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_co 32 688.42 21.51 1.1% 0.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_helpers 32 479.25 14.98 0.7% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_postprocess 32 470.30 14.70 0.7% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_benchpost_lib_construction 32 335.31 10.48 71.3% 0.0% + python_benchpost_library_write 32 81.61 2.55 17.4% 0.0% + python_benchpost_naming 32 49.05 1.53 10.4% 0.0% + python_benchpost_client_config 32 3.39 0.11 0.7% 0.0% + (untracked) 0.94 0.2% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_setup 32 131.57 4.11 0.2% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 42.88 0.1% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_ductile_initial_sampling 8 14,625.09 1,828.14 1.4% 1.4% + ----------------------------------------------------------------------------------------------------------------------------- + python_ductile_generate_solutions 32 4,612.18 144.13 0.5% 0.4% + ----------------------------------------------------------------------------------------------------------------------------- + (untracked) 2,869.74 0.3% 0.3% + ----------------------------------------------------------------------------------------------------------------------------- + python_cache_check 832 2,249.35 2.70 0.2% 0.2% + ----------------------------------------------------------------------------------------------------------------------------- + python_ductile_ga_between_evaluations 8 1,484.56 185.57 0.1% 0.1% + ----------------------------------------------------------------------------------------------------------------------------- + python_write_solutions 856 251.56 0.29 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_prepare 856 89.31 0.10 35.5% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_wsol_prepare_cache 824 85.29 0.10 95.5% 0.0% + (untracked) 3.07 3.4% 0.0% + python_wsol_prepare_nocache 32 0.94 0.03 1.1% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_header 856 85.68 0.10 34.1% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_dump 32 63.04 1.97 25.1% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 13.55 5.4% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_write_cache 32 56.19 1.76 0.0% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_ductile_setup 8 35.59 4.45 0.0% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_solution_indexing 32 33.30 1.04 0.0% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_ductile_clear_artifacts 32 10.64 0.33 0.0% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_ductile_read_scores 32 4.85 0.15 0.0% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_ductile_verification 8 0.40 0.05 0.0% 0.0% + =============================================================================================================================== + python_client_writer 1 3,305.37 3,305.37 0.3% + ----------------------------------------------------------------------------------------------------------------------------- + python_library_client_create_library 1 3,020.62 3,020.62 91.4% 0.3% + ----------------------------------------------------------------------------------------------------------------------------- + python_library_client_execution 1 277.59 277.59 8.4% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_library_client_subprocess 1 277.11 277.11 99.8% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 0.45 0.2% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_library_client_write_run_script 1 0.04 0.04 0.0% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_library_client_write_configs 1 5.80 5.80 0.2% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_library_client_setup 1 0.78 0.78 0.0% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + (untracked) 0.58 0.0% 0.0% + =============================================================================================================================== + python_library_logic 2 1,268.93 634.46 0.1% + ----------------------------------------------------------------------------------------------------------------------------- + python_logic_write 2 1,064.77 532.38 83.9% 0.1% + ----------------------------------------------------------------------------------------------------------------------------- + python_logic_parse_solutions 2 170.41 85.20 13.4% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_logic_analyze 2 32.75 16.38 2.6% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + (untracked) 1.00 0.1% 0.0% + + Overlapping Totals (wrap other timers; not summed into the tree) + ------------------------------------------------------------------------------------------------------------------------------- + python_client_subprocess 856 932,308.02 1,089.14 90.7% + main_total 856 811,544.39 948.07 78.9% + problem_total 28,642 607,652.46 21.22 59.1% + solution_total 2,355,018 306,431.61 0.13 29.8% + run_iteration_total 2,106,835 289,194.98 0.14 28.1% + validate_total 1,905,883 100,395.19 0.05 9.8% + validate_check_results_typed 1,907,339 100,168.89 0.05 9.7% + +--------------------------------------------------------------------------------------------------------------------------------- +HIERARCHY VALIDATION WARNINGS +--------------------------------------------------------------------------------------------------------------------------------- + WARNING: Child phases of python_client_execution (936,062.04 ms) exceed parent (932,742.66 ms) by 3,319.38 ms (0.4%) + +TIME BREAKDOWN (visual, % of wall clock) +--------------------------------------------------------------------------------------------------------------------------------- + TOTAL RUNTIME 100.0% ################################################## + RUN: benchmark client 86.7% ########################################### + Client launch (startup, setup & teardown) 27.7% ############# + hip_initialization 12.4% ###### + process_exit 6.6% ### + main_problem_factory 3.3% # + factory_hip_get_device_properties 3.3% # + (untracked) 0.0% + process_startup 1.3% + code_object_loading 1.1% + library_loading 0.9% + main_teardown_locals 0.9% + data_init_setup 0.9% + lazy_loading_init 0.1% + main_parse_args 0.0% + python_client_write_run_script 0.0% + reporter_setup 0.0% + listener_setup 0.0% + solution_iterator_setup 0.0% + Reference GEMM (CPU) 22.1% ########### + cpu_reference_gemm 22.1% ########### + solve_cpu_fast 22.1% ########### + solve_cpu_slow 0.1% + (untracked) 0.0% + pre_solution_listener.ReferenceValidator 0.0% + pre_solution_ref_skipped 0.0% + (untracked) 0.0% + Kernel launch & execution 12.3% ###### + warmup_first_sync 9.1% #### + warmup_runs 2.5% # + kernel_solving 0.4% + warmup_events_setup 0.1% + validate_warmups_listener.BenchmarkTimer 0.0% + validate_gpu_sync 0.0% + (untracked) 0.0% + benchmark_runs 0.0% + benchmark_validate_enqueues 0.0% + validate_enqueues_listener.BenchmarkTimer 0.0% + benchmark_sleep 0.0% + benchmark_event_elapsed_time 0.0% + benchmark_event_destroy 0.0% + (untracked) 0.0% + validate_enqueues_listener.ProgressListener 0.0% + validate_enqueues_listener.MetaResultReporter 0.0% + validate_enqueues_listener.DataInitialization 0.0% + validate_enqueues_listener.ReferenceValidator 0.0% + validate_enqueues_listener.HardwareMonitorListener 0.0% + validate_enqueues_listener.AllSolutionsIterator 0.0% + validate_enqueues_listener.SynchronizerValidator 0.0% + (untracked) 0.0% + benchmark_post_syncs 0.0% + benchmark_pre_syncs 0.0% + gpu_kernel_execution 0.0% + benchmark_events_setup 0.0% + benchmark_events_teardown 0.0% + run_iteration_sync 0.0% + Input data preparation 11.9% ##### + cpu_data_init 5.1% ## + gpu_input_reset 4.5% ## + gpu_input_reset_copy_inputs_d2d 2.4% # + gpu_input_reset_batched_inputs 1.9% + gpu_input_reset_fast_reset_output 0.2% + gpu_input_reset_convert_to_inputs 0.0% + gpu_input_reset_copy_valid_to_gpu 0.0% + gpu_input_reset_slow_path 0.0% + gpu_input_reset_copy_swizzled 0.0% + gpu_input_reset_rotating_init 0.0% + gpu_input_reset_constant_inputs 0.0% + gpu_input_reset_init_cpu_inputs 0.0% + (untracked) 0.0% + gpu_input_preparation 2.0% + gpu_input_prep_copy_swizzled 0.9% + gpu_input_prep_copy_valid_to_gpu 0.8% + gpu_input_prep_batched_inputs 0.1% + gpu_input_prep_init_cpu_inputs 0.1% + gpu_input_prep_copy_inputs_d2d 0.0% + gpu_input_prep_fast_reset_output 0.0% + gpu_input_prep_convert_to_inputs 0.0% + gpu_input_prep_slow_path 0.0% + gpu_input_prep_constant_inputs 0.0% + gpu_input_prep_rotating_init 0.0% + (untracked) 0.0% + pre_solution_listener.DataInitialization 0.3% + pre_solution_mx_init_data 0.1% + pre_solution_mx_side_swizzle 0.1% + pre_solution_mx_side_cache_hit 0.0% + (untracked) 0.0% + pre_solution_mx_cpu_sync_no_regen 0.1% + pre_solution_mx_copy_inputs_d2d_no_regen 0.0% + pre_solution_mx_check 0.0% + pre_solution_mx_copy_valid_to_gpu_no_regen 0.0% + (untracked) 0.0% + rotating_buffer_preparation 0.0% + Validation 10.9% ##### + validate_warmups_listener.ReferenceValidator 9.8% #### + validate_element_comparison 5.5% ## + validate_gpu_readback 3.5% # + validate_result_buffer_alloc 0.8% + validate_inputs_copy 0.0% + validate_setup 0.0% + validate_report 0.0% + (untracked) 0.0% + synchronizer_check 1.1% + synchronizer_reduce 1.1% + synchronizer_inactive 0.0% + (untracked) 0.0% + validate_warmups_listener.MetaResultReporter 0.0% + validate_warmups_listener.HardwareMonitorListener 0.0% + validate_warmups_listener.AllSolutionsIterator 0.0% + validate_warmups_listener.SynchronizerValidator 0.0% + (untracked) 0.0% + validate_warmups_listener.DataInitialization 0.0% + validate_warmups_listener.ProgressListener 0.0% + Per-solution bookkeeping & reporting 1.3% + pre_solution_listener.BenchmarkTimer 0.2% + post_solution_log 0.2% + post_solution_reporting 0.2% + run_current_solution_check 0.2% + run_iteration_teardown 0.2% + pre_solution_listener.AllSolutionsIterator 0.1% + pre_solution_listener.MetaResultReporter 0.1% + pre_solution_listener.ProgressListener 0.1% + post_solution_validation 0.1% + solution_selection 0.0% + pre_solution_listener.SynchronizerValidator 0.0% + post_problem 0.0% + solution_teardown 0.0% + pre_solution_listener.HardwareMonitorListener 0.0% + post_solution_sol_advance 0.0% + post_solution_perf_calc 0.0% + post_solution_perf_reset 0.0% + problem_report_index 0.0% + problem_teardown 0.0% + finalize_report 0.0% + post_solution_result_file 0.0% + Library client (Python) 0.3% + python_library_client_create_library 0.3% + python_library_client_execution 0.0% + python_library_client_subprocess 0.0% + python_library_client_write_run_script 0.0% + (untracked) 0.0% + python_library_client_write_configs 0.0% + python_library_client_setup 0.0% + Library logic (Python) 0.1% + python_logic_write 0.1% + python_logic_parse_solutions 0.0% + python_logic_analyze 0.0% + BUILD: solution generation & compile 8.3% #### + Solution generation & build (Python) 6.3% ### + python_kernel_compilation 6.3% ### + python_kernel_build_src_co 4.0% ## + python_kernel_build_src_co.compile 4.0% ## + python_kernel_build_src_co.cache_check 0.0% + python_kernel_build_src_co.unbundle 0.0% + python_kernel_build_src_co.cache_populate 0.0% + python_kernel_build_src_co.setup 0.0% + python_kernel_build_src_co.move 0.0% + (untracked) 0.0% + python_kernel_codegen 1.6% + python_kernel_write_assemble 0.3% + python_kernel_bench_setup 0.2% + python_kernel_validate 0.1% + python_kernel_build_co 0.1% + python_kernel_write_helpers 0.0% + python_kernel_bench_postprocess 0.0% + python_benchpost_lib_construction 0.0% + python_benchpost_library_write 0.0% + python_benchpost_naming 0.0% + python_benchpost_client_config 0.0% + (untracked) 0.0% + python_kernel_setup 0.0% + (untracked) 0.0% + Ductile GA search (Python) 2.0% # + python_ductile_initial_sampling 1.4% + python_ductile_generate_solutions 0.4% + python_ductile_ga_between_evaluations 0.1% + python_ductile_setup 0.0% + python_ductile_clear_artifacts 0.0% + python_ductile_read_scores 0.0% + python_ductile_verification 0.0% + MEASUREMENT OVERHEAD (instrumentation) 4.9% ## + flush_timing_buffer 2.9% # + main_timing_calibration 1.0% + timing_overhead 0.5% + (timer overhead subtracted from parent timers) 0.5% + SHARED: cache & solution files 0.3% + python_cache_check 0.2% + python_write_solutions 0.0% + python_wsol_prepare 0.0% + python_wsol_prepare_cache 0.0% + python_wsol_prepare_nocache 0.0% + (untracked) 0.0% + python_wsol_header 0.0% + python_wsol_dump 0.0% + (untracked) 0.0% + python_write_cache 0.0% + python_solution_indexing 0.0% + UNATTRIBUTED (untracked time & timer overhead) -0.1% + (untracked in python_benchmark_problems) 0.3% + (untracked in validate_warmups) 0.0% + (untracked in pre_solution) 0.0% + (untracked in pre_problem) 0.0% + (untracked in python_library_logic) 0.0% + (untracked in python_client_writer) 0.0% + (children exceed benchmark_runs) -0.0% + (children exceed post_solution) -0.0% + (children exceed python_client_execution) -0.5% + +KEY INSIGHTS (% of wall clock) +--------------------------------------------------------------------------------------------------------------------------------- + Client launch (startup, setup & teardown) 284.73s (27.7%) + Reference GEMM (CPU) 227.58s (22.1%) + Kernel launch & execution 125.98s (12.3%) + Input data preparation 122.50s (11.9%) + Validation 112.49s (10.9%) + Solution generation & build (Python) 64.83s (6.3%) + MEASUREMENT OVERHEAD (instrumentation) 45.17s (4.4%) + Ductile GA search (Python) 20.77s (2.0%) + Per-solution bookkeeping & reporting 13.67s (1.3%) + Library client (Python) 3.30s (0.3%) + SHARED: cache & solution files 2.59s (0.3%) + Library logic (Python) 1.27s (0.1%) + UNATTRIBUTED (untracked time & timer overhead) -1.42s (-0.1%) + + CPU reference / GPU execution ratio: 6341.3x + >>> CPU reference > 10x GPU execution time! + +TOP 10 SLOWEST PROBLEMS (by CPU reference time) +--------------------------------------------------------------------------------------------------------------------------------- + M N K Batch TypeA TypeD CPU Ref (ms) GPU (ms) + ------------------------------------------------------------------------------------------------------------------------------- + 43776 10624 6400 1 Float8 BFloat16 19546.98 0.00 + 13056 7680 21504 1 Float8 BFloat16 17040.82 0.00 + 71936 4096 7168 1 Float8 BFloat16 15624.38 0.00 + 13696 15488 8448 1 Float8 BFloat16 14073.45 0.00 + 14336 10880 9472 1 Float8 BFloat16 10054.99 0.00 + 9728 9600 13824 1 Float8 BFloat16 9428.36 0.00 + 173312 3200 2560 1 Float8 BFloat16 8879.09 0.00 + 14336 10880 9472 1 Float8 BFloat16 7652.50 0.00 + 9728 9600 13824 1 Float8 BFloat16 7205.78 0.00 + 209536 2048 1280 1 Float8 BFloat16 5264.93 0.00 + diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_run/common_test_outcomes.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_run/common_test_outcomes.txt new file mode 100644 index 000000000000..da4c9268c8d2 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_run/common_test_outcomes.txt @@ -0,0 +1,572 @@ +# Correctness after P2 (P3): outcome of every test in the gfx950 split run phase +# (split/run_split.sh run phase: --use-cache, -n 4, cores 0-24) at 3b4ec8e125d, P3 client and artifacts. +# One " " per line, sorted. 568 tests: 195 PASSED, 2 FAILED, 371 SKIPPED. +# Differs from ../baseline_run/common_test_outcomes.txt only by prediction_bf16 (new). See ../after_run.md. +tensilelite/Tests/common/flags/gfx942/test_build_only.py::test_compile PASSED +tensilelite/Tests/common/flags/gfx942/test_build_only.py::test_use_cache SKIPPED +tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_compile_and_benchmark SKIPPED +tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_use_cache SKIPPED +tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_use_cache_creates_cache_dir SKIPPED +tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_use_cache_different_params SKIPPED +tensilelite/Tests/common/test_config_helpers.py::test_base_skip_gfx1250_inert_on_strict_target PASSED +tensilelite/Tests/common/test_config_helpers.py::test_ffm_fail_inert_on_hardware PASSED +tensilelite/Tests/common/test_config_helpers.py::test_ffm_fail_inert_on_other_arch PASSED +tensilelite/Tests/common/test_config_helpers.py::test_ffm_fail_xfails_under_ffm PASSED +tensilelite/Tests/common/test_config_helpers.py::test_find_available_archs_does_not_map_the_retired_v0_name_to_the_base PASSED +tensilelite/Tests/common/test_config_helpers.py::test_find_available_archs_keeps_stepping_name PASSED +tensilelite/Tests/common/test_config_helpers.py::test_skip_gfx1250_strict_fires_on_strict_target PASSED +tensilelite/Tests/common/test_config_helpers.py::test_skip_gfx1250_strict_inert_on_base_target PASSED +tensilelite/Tests/common/test_config_helpers.py::test_unmarked_config_never_xfails_under_ffm PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/amaxd/8rn_amaxd.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/amaxd/gfx1250/amax_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/benchmark_runs/bf16_tn_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/benchmark_runs/fp16_tn_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/benchmark_runs/fp32_tn.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/client/rotate_mode0.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/client/rotate_mode0_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/client/rotate_mode1.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/client/rotate_mode1_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/comm/gfx950/fused_a2a.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/bbs_tn_dp.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/bbs_tn_multi_size_single_kernel.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/bbs_tn_streamK.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/f32_tt_batched.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/f8_tn_mx.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/hhs_nt_multi_size.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/bf16_tn_dp.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/bf16_tn_multi_size_single_kernel.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/bf16_tn_streamK.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/f16_nt_multi_size.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/f32_tt_batched.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/f8_tn_mx_subtile.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/f8bf16_tn_cms.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/xf32_nn.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/exception/f16_cf32_nan.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/exception/f16_cf32_nan_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/8rn_b_s.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/ag.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/agntab_coverage_gfx942.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/bf16_tn.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/bf16_tn_predict.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/bufferLoadStore0.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/cgemm.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/dot2_gfx942.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/dtl.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/dtv.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/dtv_gfx90a.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/f32_mac.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/f64.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/f64_gfx90a.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/f64_mac.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp16_HH_BHS_bf16mfma.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp16_tn.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp16_use_e.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp16fp32mix.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp32_nt.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp32_scalealphavec.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp32_tn.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8bf8nss.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8n.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8n_rowwise.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8n_use_e.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8nfp16mix_fp8nss.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8nfp16mix_hfp8ns.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8nfp16mix_hhs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8nfp16mix_hhs_lsu4.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gemm_bias_strided_batched.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gemm_bias_strided_batched_broadcast.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gemm_bias_strided_batched_broadcast_gfx940.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gemm_bias_strided_batched_gfx940.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gemm_ck_gfx942.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/bf16_tn_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/fp16_HH_BHS_bf16mfma_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/fp16_tn_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/fp16_use_e_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/gemm_bias_strided_batched_broadcast_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/gemm_bias_strided_batched_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/gsuasb_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/i8_gsu_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/1024_vgpr_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/1024_vgpr_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/8r_b_s_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/agntab_coverage_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/agntab_coverage_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b6f4ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b6f4ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b8b8s_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b8b8s_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b8f8ss_grvw1_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b8hs_grvw1_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_CLS_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_CLS_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_tn_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_tn_gfx12_predict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf6_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf6_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf6_tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf6_tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/cgemm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/cgemm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/decouple_pgr_tdm_fuse_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f32_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f32_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f32_grid_limit_size.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4b8ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4b8ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4f6ss_tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4f6ss_tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f64_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f64_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6_tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6_tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6b6ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6b6ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6b8ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6b8ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8b6ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8b6ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8b8ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8b8ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f4ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f4ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_cls_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_cls_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_grvw1_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_pk8_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_pk8_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_sr_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_sr_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp16_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp16_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp16_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp16_use_e_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp16_use_e_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp4_tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp4_tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp8_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp8_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp8_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp8bf8ss_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gemm_bias_strided_batched_broadcast_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gemm_bias_strided_batched_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gltr_bounds_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gsu_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gsu_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gsuaad_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gsuasb_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/i8_gsu_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/i8i8i_grvw1_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/i8ii_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/i8ii_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/i8ii_grvw1_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/interleave_alpha2_group_store_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/largeLds_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/largeLds_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf4_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf4_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf6_tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf6_tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf6f4ss_tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf6f4ss_tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf8_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf8_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf8f4ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf8f4ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/nt_th_nv_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/nt_th_nv_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/pgl_bound_check.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/rap_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/segment_b_rotate_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/segment_interleave_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/segment_interleave_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/stinky_sia4.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/stinky_sia4_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250-strict_bench.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250-strict_cluster.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250_bench.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250_cluster.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_general_batched_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_general_batched_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_multicast_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_multicast_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_split_gfx1250-strict_xfail.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_split_gfx1250_xfail.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdmpluslds_tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdmpluslds_tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/xfp32_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/xfp32_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/xfp32_inf_support_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/zgemm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/zgemm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx1250/cluster_entry_handoff.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/agntab_coverage_gfx950.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/bf16_cvt.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/custom_mainloop_scheduling.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/custom_mainloop_scheduling_tf32.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/custom_xfp32.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/dot2_gfx950.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/dtl.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/f16f8mix_ss.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/f16f8mix_ss_stoch.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/f8b8hs.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/f8f16mix_f8s.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp16_use_e_wide_store.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp8.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_bf16_tn_act.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_bf16_tn_act_groupgemm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_fp32_tn_act.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_fp32_tn_act_groupgemm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/general_wgm.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/i8_gsu_gfx950.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/lds160K.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/lds_tr.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mx32f4_tn.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mx32f8_tn.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_bf16_tn_act.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_bf16_tn_act_groupgemm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_fp32_tn_act.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_fp32_tn_act_groupgemm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_bf16_tn_act.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_bf16_tn_act_groupgemm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_fp32_tn_act.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_fp32_tn_act_groupgemm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp8_residual_fail_classes_repro.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp8_verify_fail_repro.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/plr_zero.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/prediction_bf16.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/preloop_local_read_wait.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/ss_bss.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_bf16.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_fp8.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp4.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_bias_sav.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt192x256_plr1_wgm4.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt256x192_plr1_wgm1.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt256x320_plr0_wgm1.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt320x256_plr1_wgm4.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_multi_du_multi_partition.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_tail_loop.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_tail_loop_smoke.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/xfp32.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gsuasb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gsuc.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/hh_f8nhs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/i8_gsu.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/i8_gsu_gfx940.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/icache_flush.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/largeMT.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/large_size.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/lsu_fnuz.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/lsu_i8.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/mbskPrefetchOpt.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/mix_cvt_after_ds_fnuz.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/nan_bounds_check_odd_padding.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/no_shadowinit.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/sia0_pgr.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/swizzleA.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/swizzleB.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/swizzle_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/tail_small_size.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/ulsgro1.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/use_beta_false.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/wgm.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/xfp32.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/zgemm.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/fp16_gradient_activation.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/fp16_gradient_bias.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/fp8bf8nss_gradient_bias_b.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx11/fp16_gradient_activation_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx11/fp16_gradient_bias_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_bgrada_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_bgradb_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_bgradd_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_dgelu_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_bgrada_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_bgradb_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_bgradd_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_dgelu_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/bbs_bgrada_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/bbs_bgradb_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/bbs_bgradd_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/bbs_dgelu_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/hhs_bgrada_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/hhs_bgradb_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/hhs_bgradd_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/hhs_dgelu_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx950/bf16_gradient_bias.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/gfx11/grouped_gemm_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/gfx11/grouped_gemm_userargs_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/grouped_gemm.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/grouped_gemm_ck_gfx942.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/grouped_gemm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/grouped_gemm_userargs.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gsu/auto_gsu.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gsu/auto_gsu_zero_dim.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gsu/f32_gsu.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gsu/gsu_mbsk.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sgemm_dual_fmac_asm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sgemm_streamk_asm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sgemm_xf32_asm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/bf16_activation.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/f16_activation.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8bs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8bs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8f8.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8f8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8hs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8hs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_bf16.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_bf16_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f16.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f16_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8_ml.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8b8.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8b8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8bs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8bs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8hs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8hs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_fp16_ml1.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8bs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8bs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8hs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8hs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8is.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8is_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_all.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_f16_transposes.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_gl2prefetch.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_ldspadding.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_multicast.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/bf16_activation.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/f16_activation.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8bs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8bs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8f8.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8f8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8hs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8hs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_bf16.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_bf16_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f16.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f16_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8_ml.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8b8.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8b8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8bs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8bs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8hs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8hs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_fp16_ml1.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_gate.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8bs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8bs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8hs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8hs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8is.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8is_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_all.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_f16_transposes.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_gl2prefetch.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_iteratemode.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_ldspadding.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_multicast.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_split_xfail.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/bf16_activation.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/bf16_gate_r.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/f8n_gate_r.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/fp16_activation.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/fp16_gate_r.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/i8_activation.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/libray_logic.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_b8f8n.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_b8f8n_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_bf16.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_bf16_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_bf8n.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_bf8n_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_f8b8n.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_f8b8n_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_f8n.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_f8n_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi16.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi16_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi32.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi32_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_gsuc.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi16.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi16_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi32.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi32_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8bs.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8bs_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8hs.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8hs_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8is.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8is_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_vw_lg_one.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_vw_lg_one_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/use_sgpr_for_gro.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/use_sgpr_for_gro_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/bf16_gate_r.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/f8_gate_r.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/fp16_gate_r.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_b8.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_b8_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_bf16.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_bf16_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_dtl.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_f16.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_f16_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_f8.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_f8_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_i8.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_i8_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_i8bs.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_i8hs.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_ldstr.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_vw_lg_one.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_vw_lg_one_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/data_parallel_static_mx32f4.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/data_parallel_static_sgemm.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx11/sk_bgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx11/sk_hgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx12/sk_bgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx12/sk_hgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/bf16_vector_epilogue_persistent.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf4.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf4_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf8.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf8_halfplr_tdm_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf8_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_bf16gemm_tdm_subtile.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_bgemm_tdm_split.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_bgemm_tdm_split_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_dynamic_hgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_dynamic_sgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_f8gemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_halfplr_f8gemm_tdm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_halfplr_mxf8gemm_tdm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_hgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_hybrid_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_explicit.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_pap_prefetchgl2.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_tdm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_tdm_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf6b8gemm_tdm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf6b8gemm_tdm_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8b6gemm_tdm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8b6gemm_tdm_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8f4gemm_tdm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8f4gemm_tdm_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_explicit.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_pap_prefetchgl2.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm_split.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm_split_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_sgemm_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_sgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_tdm_general_batched.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_tdm_subtile_general_batched.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/data_parallel_cluster_multicast_boundaries.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf4_cluster_multicast_ext.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf4_pap_cluster_multicast_ext.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf8_cluster_multicast_ext.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf8_pap_cluster_multicast_ext.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/sk_hgemm_debug_modes.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/sk_mxf4gemm_tdm_cluster_ext.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/sk_mxf4gemm_tdm_ext.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/sk_mxf4gemm_tdm_pap_ext.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/data_parallel_static_hgemm_use_e.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/data_parallel_static_sgemm_pap.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_bgemm_pap.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_hgemm_use_e.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_hybrid_xcc_mapping.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxf4gemm_pap.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_basic.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_batch_combo.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_full_coverage.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_streamk_combo.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_atomic_variants.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_basic_variants.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_batch_mi_combo.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_batch_sizes.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_comprehensive.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_debug_modes.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_full_coverage.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_activation_combo.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_basic.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_batch_combo.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_full_coverage.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_streamk_combo.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_matrix_instructions.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_problem_sizes.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_tree_reduction.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_xcc_mapping.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_debug_modes.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_dynamic_pap.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_hybrid_pap.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_pap.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_I8gemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_bgemm_div.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_bgemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_bgemm_sia1.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_cgemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_dgemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_dynamic.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_dynamic_mxfp4.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_dynamic_work_stealing.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_f8gemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_f8ngemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hgemm_epilogues.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hgemm_lsu.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hgemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hgemm_race.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hybrid.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hybrid_work_stealing.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_mx32f4_quick.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_mx32f8_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_queue_selfclean.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_reduction.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_sgemm_alpha0.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_sgemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_zgemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_aiter_bf16.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_aiter_f4.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_ck.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rocblas_gemv_bf16_tn_m1.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rocblas_gemv_bf16_tn_m2.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rocblas_gemv_bf16_tn_m4.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rr.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rr_f6.yaml] FAILED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rr_mx.yaml] FAILED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_tensile_sk.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_triton_f4.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_wave_bf16.yaml] PASSED diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline.md b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline.md new file mode 100644 index 000000000000..68ec5d189da1 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline.md @@ -0,0 +1,268 @@ +# gfx950 timing baseline at the starting commit (P0.2) + +Re-measurement of the gfx950 common tests at the starting commit of the implementation plan +(`timing_runs/DECISIONS.md`, P0.2). The earlier committed baselines (`per_config_rerun_2`, +`per_config_build`) were measured with a client built at `4c6426cea6`. Every later measurement +(P3 `after_run/`, P6 `after_build/`) is compared with the numbers here, not with those. + +## What was measured + +| Item | Value | +|---|---| +| Commit | `2083e6254bf` (detached worktree). Code-identical to the starting commit `55e29436e9a`: the only difference is P0.0's measurement-script parametrization. Client, library and Python sources are also identical to `4c6426cea6`; only `scripts/{analyze,aggregate}_timing.py` and their tests changed since then. | +| Client | `tensilelite-client` built from that commit, CMake cache identical to `/tmp/tl_client_msgpack` (the `4c6426cea6` baseline client): Release, `GPU_TARGETS=gfx950`, `/usr/bin/amdclang++` (AMD clang 22.0.0git, ROCm 7.2.1), msgpack only (`HIPBLASLT_ENABLE_YAML=OFF`), `TENSILELITE_BUILD_TESTING=OFF`. sha256 `4b2769f7f9a2…`. | +| Python | venv Python 3.12.3; TensileLite Python sources from the worktree; `rocisa` is the venv's installed build (2026-10-19, unchanged). | +| Host | `smci355-1b112-b13-15`: 2× AMD EPYC 9575F (64 cores each, SMT 2), 8× AMD Instinct MI355X (gfx950), Linux 6.8.0-38, ROCm 7.2.1, amdgpu 6.16.13. Shared by ~45 users. | +| Cores / threads | `taskset -c 0-24` (25 physical cores, NUMA node 0), `OMP_NUM_THREADS=25`, `OPENBLAS_NUM_THREADS=25` | +| GPU | `ROCR_VISIBLE_DEVICES=0` = KFD node 2 = PCI `0000:75:00.0`, NUMA node 0. This is the same physical GPU the earlier baselines used. Note that `rocm-smi`/`amd-smi` call it GPU **3**, because they order by PCI bus and ROCR orders by KFD node. | +| Scripts | `split/run_split.sh`, `per_config/run_per_config.sh` (`KEEP_CACHE=1`, as `per_config_rerun_2`), `per_config/run_per_config_build.sh` (cold cache per config), all with caches and artifacts redirected through `measure_env.sh` | +| Config list | The 184 configs that passed the split run phase. Identical to `per_config_rerun_2`'s list. | + +Committed results: + +- `baseline_run/AGGREGATE_analysis.txt`: per-config run aggregate (184 logs). +- `baseline_build/AGGREGATE_analysis.txt`: per-config build aggregate (184 logs). +- `baseline_run/common_test_outcomes.txt`: correctness baseline (outcome of every test in the split + run phase). + +The raw per-config logs are not committed, as for the earlier baselines. That is +`/.log.gz` and `_build.log.gz`, plus each config's `_analysis.txt` and +`times.txt`. A durable copy (757 MB, checked identical by md5) is in +`/home/alvasile/timing_runs_raw/P0.2/`: + +- `baseline_run/`, `baseline_build/`: the committed passes. +- `noise_run_1/`, `noise_run_2_contended/`: the repeats used for the noise band. +- `split/`: split phase logs, junit XML and outcomes. +- The config lists, plus `snapshots.txt`, `contention.log`, `corpus.log` and `gpu.log`. + +The working copy is in `/tmp/tl_plan/P0.2/`. **P3/P6 must re-aggregate these raw logs with the +`analyze_timing.py` of the commit they compare at**, not compare against the committed +`AGGREGATE_analysis.txt` as is: P1/P4 add timers and change the hierarchy. The command is +`python scripts/aggregate_timing.py /*/*.log.gz`, or `/*/*_build.log.gz` for the build +pass. The earlier baselines' raw logs re-aggregate byte-identically with the analyzer at this +commit, so analyzer changes do not contribute to any difference below. + +`/tmp` is cleaned of files older than 30 days on this host +(`/usr/lib/tmpfiles.d/tmp.conf`: `D /tmp 1777 root root 30d`). The P0.2 worktree +(`/tmp/tl_plan/P0.2/wt`), client (`/tmp/tl_plan/P0.2/build_client`) and artifacts +(`/tmp/tl_plan/P0.2/artifacts`) may therefore disappear before P3/P6. If they are gone, rebuild +them from `2083e6254bf` with the configure and commands under "Reproducing" (client ~1.5 min, +artifacts = split build phase ~11 min). + +## Correctness baseline + +Split run phase (`--use-cache`, `-n 4`): 567 tests, **194 passed, 2 failed, 371 skipped**. The +outcome of every test is identical to the 2026-10-20 split run. The split build phase also +matches it: 195 passed, 2 failed, 370 skipped. + +The two failures are pre-existing. `custom/custom_rr_f6.yaml` and `custom/custom_rr_mx.yaml` fail +to build (`RuntimeError: No valid solutions found`), so their run test finds no artifact tarball. +Every config in the per-config run and build passes returned 0. + +## Run side (per-config, instrumented) + +`run_per_config.sh`, serial, `TimingInstrumentation=True`, `KEEP_CACHE=1` (caches cleared once, +as for `per_config_rerun_2`). Measured 2026-10-31 01:40–02:34 UTC with no other users' work +pinned to cores 0–24. In seconds, from `AGGREGATE_analysis.txt` ("TIMING BY PURPOSE"): + +| Category | Baseline (this) | `per_config_rerun_2` (`4c6426cea6`) | Δ | +|---|---:|---:|---:| +| TOTAL RUNTIME | 2,401.1 | 2,568.3 | −6.5% | +| RUN: benchmark client | 2,284.9 | 2,330.8 | −2.0% | +| Reference GEMM (CPU) | 1,044.4 | 1,080.5 | −3.3% | +| … `solve_cpu_slow` | 689.0 | 727.8 | −5.3% | +| … `solve_cpu_fast` | 198.9 | 200.9 | −1.0% | +| Input data preparation | 500.7 | 502.9 | −0.4% | +| Validation | 329.1 | 323.9 | +1.6% | +| Client launch (startup, setup & teardown) | 289.8 | 284.7 | +1.8% | +| Per-solution bookkeeping & reporting | 54.8 | 68.3 | −19.9% | +| Kernel launch & execution | 35.7 | 40.9 | −12.7% | +| BUILD: solution generation & compile | 63.1 | 64.0 | −1.5% | +| MEASUREMENT OVERHEAD (instrumentation) | 52.8 | 172.8 | −69.5% | +| … `flush_timing_buffer` | 33.6 | 152.9 | −78.1% | +| Sum of per-config pytest wall times | 2,942 | 3,123 | −5.8% | + +- Every timer's call count is identical (for example 28,596 `cpu_reference_gemm`, 2,355,016 + solutions, 2,106,835 `gpu_input_reset`). The work is the same, as expected from identical + sources. +- Most of the −167 s difference is `flush_timing_buffer`: the instrumentation writing its buffer + to the log, −119 s. The rest is CPU reference −36 s, bookkeeping −14 s and kernel launch −5 s. +- The CPU reference difference (−3.3%; `solve_cpu_slow` −5.3%) is **unexplained and larger than + the clean-repeat spread** of that category (0.6%, see "Noise band"). The old pass differs in + client build (same sources), day and host load. Treat it as a difference between passes, not as + a code effect. +- Use this baseline for P3, not `per_config_rerun_2`. + +## Build side (per-config, cold cache per config) + +`run_per_config_build.sh`, measured 2026-10-31 00:26–01:23 UTC: + +| Category | Baseline (this) | `per_config_build` (`4c6426cea6`) | Δ | +|---|---:|---:|---:| +| TOTAL RUNTIME | 2,654.8 | 2,577.8 | +3.0% | +| BUILD: solution generation & compile | 2,605.6 | 2,530.1 | +3.0% | +| … `python_kernel_compilation` | 2,260.4 | 2,203.8 | +2.6% | +| … … `python_kernel_build_src_co` | 1,013.4 | 1,007.8 | +0.6% | +| … … `python_kernel_codegen` | 615.5 | 592.6 | +3.9% | +| … … `python_kernel_write_assemble` | 253.8 | 227.9 | +11.4% | +| … `python_solution_generation` | 321.6 | 306.6 | +4.9% | +| Ductile GA search (Python) | 23.7 | 19.7 | +20.2% | +| SHARED: cache & solution files | 23.2 | 23.2 | +0.1% | +| RUN: benchmark client | 14.3 | 13.0 | +10.3% | +| Sum of per-config pytest wall times | 3,361 | 3,273 | +2.7% | + +No config differs by more than 15 s from `per_config_build`. Over the 10 most expensive build +configs, the two passes differ by 2.5% in total and by at most 9.4% per config (`plr_zero`). + +**The whole build pass (00:26–01:23 UTC) overlapped another user's corpus run.** That run pins +short-lived compile jobs to core 9. `contention.log` shows them on core 9 in 51 of the 55 +minutes; pidstat only catches part of them, so the low average it reports understates the load. +The effect on the build is not quantified. It is probably small, since build workers are +independent processes rather than OpenMP threads. Still, the +3.0% against `per_config_build` +(a uniform drift: +88 s summed over 184 configs, at most 7 s per config) cannot be separated from +it. The baseline may be inflated by a few percent. Re-measuring was not possible: no 1-hour clean +window occurred before this was recorded. **P6 must therefore re-run this P0.2 build pass** +(P0.2 worktree/client, `run_per_config_build.sh`, same settings) **back to back with its +`after_build/` pass, under identical conditions, and compare against that re-run.** It should +also re-run the split build phase, for P6's `--build-only` comparison, since the 634 s below was +contended too. + +## Split phases (`-n 4`, uninstrumented) + +| Phase | Wall time | Exit | Conditions | +|---|---:|---:|---| +| Build (`--build-only`) | 634 s | 1 (the 2 known failures) | core 9 contended throughout | +| Run (`--use-cache`) | 2,950 s | 1 (the 2 known failures) | core 9 contended throughout | + +The 2026-10-20 split, measured at an older tree, took 1,183 s (build) and 2,804 s (run). The +run phase here (22:18–23:07 UTC) overlapped another user's job pinned to core 9 for its whole +duration. That job slows the OpenMP CPU reference about 2.5× (see "Noise band"), so **2,950 s is +an upper bound, not a clean uninstrumented baseline**. A clean repeat could not be obtained +(see "Conditions"). **P3 must re-measure this phase with the P0.2 worktree, client and artifacts +back to back with its own, under identical conditions, and compare against that**, not against +2,950 s. Likewise, the split build phase (634 s, also contended throughout) is not a clean +baseline: P6 re-measures it (see "Build side"). + +## Noise band + +The 10 most expensive run configs (by pytest wall time in this baseline) were repeated +(`noise_run_`, same script and settings, `KEEP_CACHE=1`). Samples whose run overlapped other +users' work pinned inside cores 0–24 were **excluded**: see "Conditions" for how this was +detected. Three repeats were run, but only part of them was uncontended. **Clean samples per +config: 3 for `sk_I8gemm_quick` and `sk_f8gemm_quick`**: the baseline pass, repeat 1, and +repeat 2's first two configs. **2 for the other eight configs**: the baseline pass and repeat 1. +All other repeat samples (repeat 2 configs 3–10, all of repeat 3, a restarted repeat 2) were +excluded. A further 2.7 h of waiting for an uncontended window (06:33–09:16 UTC) found none. + +Per-config pytest wall time (s), clean samples only. Spread = (max − min) / mean. + +| Config | Baseline | Repeat 1 | Repeat 2 | Spread | `per_config_rerun_2` (reference) | +|---|---:|---:|---:|---:|---:| +| streamk/sk_I8gemm_quick | 311 | 313 | 304 | 2.9% | 316 | +| streamk/sk_f8gemm_quick | 262 | 278 | 266 | 6.0% | 305 | +| gemm/gfx950/subtile_mxfp8 | 186 | 180 | – | 3.3% | 182 | +| sparse/gfx950/spmm_f16_sb | 124 | 126 | – | 1.6% | 144 | +| gemm/gfx950/subtile_mxfp4 | 95 | 89 | – | 6.5% | 82 | +| gemm/gfx950/plr_zero | 94 | 96 | – | 2.1% | 107 | +| gemm/fp16_tn | 87 | 87 | – | 0.0% | 103 | +| gemm/gfx950/dtl | 82 | 85 | – | 3.6% | 103 | +| gemm/bufferLoadStore0 | 81 | 83 | – | 2.4% | 99 | +| gemm/gfx950/subtile_mxfp8_correctness_regression_mt320x256_plr1_wgm4 | 71 | 67 | – | 5.8% | 69 | +| Sum | 1,393 | 1,404 | | 0.8% | 1,510 | + +Categories summed over the same 10 configs (s), baseline vs repeat 1: + +| Category | Baseline | Repeat 1 | Spread | +|---|---:|---:|---:| +| TOTAL RUNTIME | 1,359.6 | 1,369.7 | 0.7% | +| Reference GEMM (CPU) | 787.8 | 792.7 | 0.6% | +| Input data preparation | 229.0 | 226.7 | 1.0% | +| Validation | 179.9 | 181.7 | 1.0% | +| Client launch (startup, setup & teardown) | 82.5 | 82.2 | 0.4% | +| Per-solution bookkeeping & reporting | 28.6 | 30.2 | 5.4% | +| Kernel launch & execution | 21.6 | 24.2 | 11.4% | +| MEASUREMENT OVERHEAD (instrumentation) | 29.0 | 29.4 | 1.4% | + +So, on an uncontended host, totals and the large categories repeat to within about 1%. Single +configs repeat to within about 6%. The small categories (bookkeeping, kernel launch) vary by +5–11%. With 2–3 samples this is an indication, not a confidence interval. + +Contended samples, for scale (all excluded): with the core-9 job active, `sk_I8gemm_quick` took +1,169 s in the discarded first per-config run pass (23:07–00:26 UTC) and 749 s in repeat 3, +instead of 304–313 s. Over the 10 configs, Reference GEMM (CPU) in that discarded first pass was +2,008 s instead of 788 s, while GPU-side categories barely moved. CPU-reference changes in P3 +can only be judged on uncontended runs. + +Build side: no repeat could be measured uncontended. The only second sample is +`per_config_build`: same sources, different client build, measured 2026-10-21. It differs by ++3.0% in total, and per config by 0–9.4% over the 10 most expensive build configs. + +## Conditions and interference + +The host is shared, and the measurement cores and GPU are not reserved. Recorded with the +measurements in `/tmp/tl_plan/P0.2/`: + +- `snapshots.txt`: `uptime`, per-core idle on 0–24, and other processes on 0–24 at the start and end + of every pass. Load averages at the clean run pass: 8.8 (start), 7.9 (end). At the build pass: + 8.6 (start), 21.1 (end). +- `contention.log` (from 00:29): every 30 s, other users' processes on cores 0–24 with ≥1% CPU + (pidstat). +- `corpus.log` (from 05:26): cumulative CPU time of other users' pinned corpus runs. +- `gpu.log` (from 06:16): use, sclk, VRAM and power of the measurement GPU. Before 06:16 the + snapshots read `rocm-smi -d 0`, which is a different card (`0000:05:00.0`, ROCR device 1). + So there is no GPU-side record for the committed passes. Indirect evidence that they were not + GPU-contended: GPU-side categories are at or below `per_config_rerun_2`, and the clean + repeat matches the baseline to within 1.4% on every category except bookkeeping (5.4%) and + kernel launch (11.4%). + +Interference seen: + +- Another user's corpus run pins short-lived compile/timing jobs to core 9 (`--timing-cpus 9`) + and to cores 0–8, 10–47 (`--cpus`). It ran about 20:20–01:25, 03:33–05:51 and from 06:26 UTC + (still running at 09:16). + - The first per-config run pass (23:07–00:26) overlapped it and was discarded: TOTAL 3,911 s, + and the Int8 4111³ slow-path problem took 201.7 s instead of 72–75 s. + - The committed run pass was re-measured afterwards in a clean window (01:40–02:34; no jobs + from it on cores 0–24 after 01:30). + - The committed build pass (00:26–01:23) and both split phases (22:07–23:07) overlapped it and + were kept, with the caveats above. +- Unpinned processes (for example a root `python3` at 100% of one core) are placed on idle cores + in 0–24 by the scheduler and move away once the measurement threads load those cores. They are + not counted as contention. +- A root vLLM server (TP=4) occupied rocm-smi GPUs 0–3, including the measurement GPU, at up to + ~279 GB and 100% use. It was there in short bursts from 06:35 UTC and continuously + 07:20–08:05 and 08:17–09:12. This was after all the committed passes. + +Recommendation for P3/P6: monitor the measurement GPU as `rocm-smi -d 3`. Before and during each +pass, check for other users' jobs pinned into 0–24 (by CPU-time accounting, since pidstat misses +short-lived processes). Start passes only in a clean window (as `/tmp/tl_plan/P0.2/driver4.sh` +does), and keep or repeat only uncontended samples. + +## Reproducing + +```bash +git worktree add --detach /tmp/tl_plan/P0.2/wt 2083e6254bf +# client (CMake cache identical to /tmp/tl_client_msgpack); venv provides cmake 4.4.4 +source /home/alvasile/venv/bin/activate +cmake --preset tensilelite -S /tmp/tl_plan/P0.2/wt/projects/hipblaslt -B /tmp/tl_plan/P0.2/build_client \ + -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DGPU_TARGETS=gfx950 \ + -DCMAKE_CXX_COMPILER=/usr/bin/amdclang++ -DCMAKE_C_COMPILER=/opt/rocm/bin/amdclang \ + -DHIPBLASLT_ENABLE_YAML=OFF -DTENSILELITE_ENABLE_CLIENT=ON -DTENSILELITE_BUILD_TESTING=OFF \ + -DTENSILELITE_CLIENT_ENABLE_ROCPROFSDK=OFF -DCMAKE_EXPORT_COMPILE_COMMANDS=ON +taskset -c 128-191 cmake --build /tmp/tl_plan/P0.2/build_client -j64 --target tensilelite-client + +export TL=/tmp/tl_plan/P0.2/wt/projects/hipblaslt/tensilelite +export CLIENT=/tmp/tl_plan/P0.2/build_client/tensilelite/client/tensilelite-client +export ART=/tmp/tl_plan/P0.2/artifacts +export TENSILE_CACHE=/tmp/tl_plan/P0.2/cache/tensile TENSILELITE_CACHE=/tmp/tl_plan/P0.2/cache/tensilelite +S=$TL/timing_runs/full_runs_gfx950 +OUT=/tmp/tl_plan/P0.2/split bash $S/split/run_split.sh +OUT=/tmp/tl_plan/P0.2/baseline_run KEEP_CACHE=1 bash $S/per_config/run_per_config.sh /tmp/tl_plan/P0.2/configs.txt +ROCR_VISIBLE_DEVICES=0 HIP_VISIBLE_DEVICES=0 OUT=/tmp/tl_plan/P0.2/baseline_build \ + SCRATCH=/tmp/tl_plan/P0.2/build_scratch bash $S/per_config/run_per_config_build.sh /tmp/tl_plan/P0.2/configs.txt +``` + +`configs.txt` holds the `test_config_run` IDs that passed the split run phase (184 lines, the +same set as `per_config_rerun_2/configs.txt`). A durable copy is +`/home/alvasile/timing_runs_raw/P0.2/configs.txt`; use it if the `/tmp` copy is gone. The noise +lists are next to it (`noise_run_configs.txt`, `noise_build_configs.txt`). diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline_build/AGGREGATE_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline_build/AGGREGATE_analysis.txt new file mode 100644 index 000000000000..ba79d2b75150 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline_build/AGGREGATE_analysis.txt @@ -0,0 +1,595 @@ +============================================================================================================================= +TIMING ANALYSIS SUMMARY +============================================================================================================================= + +TIMING BY PURPOSE +----------------------------------------------------------------------------------------------------------------------------- + Category Count Total (ms) Mean (ms) % of Parent % of Wall +----------------------------------------------------------------------------------------------------------------------------- + TOTAL RUNTIME 2,654,787.19 100.0% + BUILD: solution generation & compile 2,605,637.61 98.1% 98.1% + Solution generation & build (Python) 2,581,934.64 99.1% 97.3% + python_kernel_compilation 855 2,260,356.50 2,643.69 87.5% 85.1% + python_kernel_build_src_co 855 1,013,356.87 1,185.21 44.8% 38.2% + python_kernel_build_src_co.compile 263 1,010,494.31 3,842.18 99.7% 38.1% + python_kernel_build_src_co.unbundle 263 1,350.69 5.14 0.1% 0.1% + python_kernel_build_src_co.cache_check 855 1,203.64 1.41 0.1% 0.0% + python_kernel_build_src_co.cache_populate 263 175.68 0.67 0.0% 0.0% + python_kernel_build_src_co.setup 855 25.18 0.03 0.0% 0.0% + python_kernel_build_src_co.move 263 14.10 0.05 0.0% 0.0% + (untracked) 93.27 0.0% 0.0% + python_kernel_codegen 855 615,536.08 719.93 27.2% 23.2% + python_kernel_write_assemble 855 253,795.98 296.84 11.2% 9.6% + python_kernel_bench_setup 855 132,219.33 154.64 5.8% 5.0% + python_kernel_bench_postprocess 855 105,243.40 123.09 4.7% 4.0% + python_benchpost_lib_construction 855 72,859.94 85.22 69.2% 2.7% + python_benchpost_library_write 855 22,779.07 26.64 21.6% 0.9% + python_benchpost_naming 855 9,378.78 10.97 8.9% 0.4% + python_benchpost_client_config 855 194.33 0.23 0.2% 0.0% + (untracked) 31.28 0.0% 0.0% + python_kernel_validate 855 82,550.97 96.55 3.7% 3.1% + python_kernel_build_co 855 22,054.87 25.80 1.0% 0.8% + python_kernel_setup 855 20,891.95 24.44 0.9% 0.8% + python_kernel_write_helpers 855 10,823.20 12.66 0.5% 0.4% + (untracked) 3,883.85 0.2% 0.1% + python_solution_generation 823 321,578.14 390.74 12.5% 12.1% + python_solgen_forked_solutions 823 321,501.91 390.65 100.0% 12.1% + python_solgen_custom_kernels 823 59.09 0.07 0.0% 0.0% + python_solgen_fork_permutations 823 4.51 0.01 0.0% 0.0% + (untracked) 12.63 0.0% 0.0% + Ductile GA search (Python) 23,702.97 0.9% 0.9% + python_ductile_initial_sampling 8 17,418.90 2,177.36 73.5% 0.7% + python_ductile_generate_solutions 32 4,866.91 152.09 20.5% 0.2% + python_ductile_ga_between_evaluations 8 1,364.98 170.62 5.8% 0.1% + python_ductile_setup 8 29.94 3.74 0.1% 0.0% + python_ductile_clear_artifacts 24 17.63 0.73 0.1% 0.0% + python_ductile_read_scores 32 4.30 0.13 0.0% 0.0% + python_ductile_verification 8 0.30 0.04 0.0% 0.0% + SHARED: cache & solution files 23,204.09 0.9% 0.9% + python_write_solutions 855 14,089.48 16.48 60.7% 0.5% + python_wsol_dump 855 13,420.73 15.70 95.3% 0.5% + python_wsol_prepare 855 528.55 0.62 3.8% 0.0% + python_wsol_prepare_nocache 855 527.16 0.62 99.7% 0.0% + (untracked) 1.39 0.3% 0.0% + python_wsol_header 855 11.83 0.01 0.1% 0.0% + (untracked) 128.36 0.9% 0.0% + python_solution_indexing 855 8,355.06 9.77 36.0% 0.3% + python_write_cache 855 758.98 0.89 3.3% 0.0% + python_cache_check 831 0.58 0.00 0.0% 0.0% + RUN: benchmark client 14,327.88 0.5% 0.5% + Client launch (startup, setup & teardown) 9,447.60 65.9% 0.4% + hip_initialization 32 5,005.78 156.43 53.0% 0.2% + process_exit 32 2,438.60 76.21 25.8% 0.1% + main_problem_factory 32 629.47 19.67 6.7% 0.0% + factory_hip_get_device_properties 4 625.92 156.48 99.4% 0.0% + (untracked) 3.55 0.6% 0.0% + main_teardown_locals 32 589.74 18.43 6.2% 0.0% + process_startup 32 467.09 14.60 4.9% 0.0% + data_init_setup 32 98.04 3.06 1.0% 0.0% + code_object_loading 32 97.55 3.05 1.0% 0.0% + library_loading 32 64.25 2.01 0.7% 0.0% + lazy_loading_init 32 42.62 1.33 0.5% 0.0% + main_parse_args 32 10.48 0.33 0.1% 0.0% + reporter_setup 32 1.66 0.05 0.0% 0.0% + python_client_write_run_script 32 1.58 0.05 0.0% 0.0% + listener_setup 32 0.46 0.01 0.0% 0.0% + solution_iterator_setup 32 0.27 0.01 0.0% 0.0% + Input data preparation 4,796.72 33.5% 0.2% + gpu_input_preparation 40 3,536.05 88.40 73.7% 0.1% + gpu_input_prep_init_cpu_inputs 39 2,097.58 53.78 59.3% 0.1% + gpu_input_prep_copy_valid_to_gpu 40 1,349.34 33.73 38.2% 0.1% + gpu_input_prep_copy_swizzled 4 85.02 21.25 2.4% 0.0% + gpu_input_prep_batched_inputs 40 2.57 0.06 0.1% 0.0% + gpu_input_prep_copy_inputs_d2d 40 0.98 0.02 0.0% 0.0% + gpu_input_prep_constant_inputs 39 0.43 0.01 0.0% 0.0% + gpu_input_prep_convert_to_inputs 40 0.08 0.00 0.0% 0.0% + gpu_input_prep_slow_path 40 0.00 0.00 0.0% 0.0% + (untracked) 0.05 0.0% 0.0% + pre_solution_listener.DataInitialization 585 1,120.43 1.92 23.4% 0.0% + pre_solution_mx_init_data 130 1,000.38 7.70 89.3% 0.0% + pre_solution_mx_side_regenerate 8 1,000.33 125.04 100.0% 0.0% + pre_solution_mx_side_cache_hit 252 0.02 0.00 0.0% 0.0% + (untracked) 0.03 0.0% 0.0% + pre_solution_mx_copy_valid_to_gpu_no_regen 126 103.32 0.82 9.2% 0.0% + pre_solution_mx_cpu_sync_no_regen 126 8.60 0.07 0.8% 0.0% + pre_solution_mx_copy_valid_to_gpu_after_regen 4 3.38 0.85 0.3% 0.0% + pre_solution_mx_copy_inputs_d2d_no_regen 126 3.22 0.03 0.3% 0.0% + pre_solution_mx_cpu_sync_after_regen 4 1.32 0.33 0.1% 0.0% + pre_solution_mx_copy_inputs_d2d_after_regen 4 0.14 0.03 0.0% 0.0% + pre_solution_mx_check 585 0.03 0.00 0.0% 0.0% + (untracked) 0.03 0.0% 0.0% + cpu_data_init 1 132.63 132.63 2.8% 0.0% + rotating_buffer_preparation 40 5.84 0.15 0.1% 0.0% + gpu_input_reset 1 1.76 1.76 0.0% 0.0% + gpu_input_reset_copy_valid_to_gpu 1 0.84 0.84 47.8% 0.0% + gpu_input_reset_copy_swizzled 1 0.83 0.83 46.9% 0.0% + gpu_input_reset_batched_inputs 1 0.06 0.06 3.6% 0.0% + gpu_input_reset_copy_inputs_d2d 1 0.03 0.03 1.7% 0.0% + gpu_input_reset_convert_to_inputs 1 0.00 0.00 0.0% 0.0% + gpu_input_reset_slow_path 1 0.00 0.00 0.0% 0.0% + Per-solution bookkeeping & reporting 67.12 0.5% 0.0% + pre_solution_listener.MetaResultReporter 585 36.70 0.06 54.7% 0.0% + pre_solution_listener.ProgressListener 585 15.18 0.03 22.6% 0.0% + post_solution_log 585 5.47 0.01 8.2% 0.0% + pre_solution_listener.BenchmarkTimer 585 3.41 0.01 5.1% 0.0% + run_current_solution_check 585 1.61 0.00 2.4% 0.0% + post_problem 40 1.61 0.04 2.4% 0.0% + pre_solution_listener.AllSolutionsIterator 585 1.11 0.00 1.6% 0.0% + post_solution_reporting 585 0.99 0.00 1.5% 0.0% + finalize_report 32 0.34 0.01 0.5% 0.0% + post_solution_perf_calc 585 0.20 0.00 0.3% 0.0% + solution_selection 585 0.14 0.00 0.2% 0.0% + problem_report_index 40 0.13 0.00 0.2% 0.0% + pre_solution_listener.SynchronizerValidator 585 0.10 0.00 0.1% 0.0% + problem_teardown 40 0.04 0.00 0.1% 0.0% + solution_teardown 585 0.02 0.00 0.0% 0.0% + post_solution_result_file 585 0.02 0.00 0.0% 0.0% + pre_solution_listener.HardwareMonitorListener 585 0.01 0.00 0.0% 0.0% + post_solution_validation 585 0.01 0.00 0.0% 0.0% + post_solution_sol_advance 585 0.01 0.00 0.0% 0.0% + post_solution_perf_reset 585 0.01 0.00 0.0% 0.0% + run_iteration_teardown 1 0.01 0.01 0.0% 0.0% + Reference GEMM (CPU) 14.60 0.1% 0.0% + cpu_reference_gemm 1 13.61 13.61 93.2% 0.0% + solve_cpu_slow 1 13.61 13.61 100.0% 0.0% + pre_solution_listener.ReferenceValidator 585 0.99 0.00 6.8% 0.0% + cpu_reference_gemm_per_solution_inputs_changed 1 0.97 0.97 98.1% 0.0% + solve_cpu_slow_per_solution_inputs_changed 1 0.97 0.97 100.0% 0.0% + (untracked) 0.02 1.9% 0.0% + Validation 1.69 0.0% 0.0% + validate_warmups_listener.ReferenceValidator 1 1.18 1.18 69.7% 0.0% + validate_result_buffer_alloc 1 0.87 0.87 73.5% 0.0% + validate_gpu_readback 1 0.30 0.30 25.7% 0.0% + validate_element_comparison 1 0.01 0.01 0.8% 0.0% + validate_inputs_copy 1 0.00 0.00 0.1% 0.0% + validate_report 1 0.00 0.00 0.0% 0.0% + validate_setup 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.SynchronizerValidator 1 0.51 0.51 30.3% 0.0% + validate_synchronizer_check 1 0.51 0.51 100.0% 0.0% + validate_synchronizer_readback 1 0.34 0.34 66.6% 0.0% + validate_synchronizer_scan 1 0.17 0.17 33.4% 0.0% + validate_warmups_listener.MetaResultReporter 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.HardwareMonitorListener 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.AllSolutionsIterator 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.DataInitialization 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.ProgressListener 1 0.00 0.00 0.0% 0.0% + Kernel launch & execution 0.16 0.0% 0.0% + warmup_runs 2 0.07 0.04 46.1% 0.0% + kernel_solving 1 0.06 0.06 39.0% 0.0% + warmup_events_setup 1 0.02 0.02 12.0% 0.0% + validate_warmups_listener.BenchmarkTimer 1 0.00 0.00 2.6% 0.0% + validate_gpu_sync 1 0.00 0.00 100.0% 0.0% + benchmark_runs 1 0.00 0.00 0.2% 0.0% + benchmark_pre_syncs 1 0.00 0.00 50.0% 0.0% + benchmark_post_syncs 1 0.00 0.00 50.0% 0.0% + MEASUREMENT OVERHEAD (instrumentation) 362.43 0.0% 0.0% + main_timing_calibration 32 356.49 11.14 98.4% 0.0% + flush_timing_buffer 32 4.73 0.15 1.3% 0.0% + (timer overhead subtracted from parent timers) 0.61 0.2% 0.0% + timing_overhead 32 0.60 0.02 0.2% 0.0% + UNATTRIBUTED (untracked time & timer overhead) 11,255.17 0.4% 0.4% + (untracked in python_benchmark_problems) 11,241.68 99.9% 0.4% + (untracked in python_client_execution) 11.94 0.1% 0.0% + (untracked in pre_solution) 0.89 0.0% 0.0% + (untracked in pre_problem) 0.59 0.0% 0.0% + (untracked in post_solution) 0.07 0.0% 0.0% + +TIMING BY TIMER NESTING (legacy view) +----------------------------------------------------------------------------------------------------------------------------- + Category Count Total (ms) Mean (ms) % of Parent % of Wall +----------------------------------------------------------------------------------------------------------------------------- + python_benchmark_problems 184 2,654,786.60 14,428.19 100.0% + ------------------------------------------------------------------------------------------------------------------------- + python_kernel_compilation 855 2,260,356.50 2,643.69 85.1% 85.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_src_co 855 1,013,356.87 1,185.21 44.8% 38.2% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_kernel_build_src_co.compile 263 1,010,494.31 3,842.18 99.7% 38.1% + python_kernel_build_src_co.unbundle 263 1,350.69 5.14 0.1% 0.1% + python_kernel_build_src_co.cache_check 855 1,203.64 1.41 0.1% 0.0% + python_kernel_build_src_co.cache_populate 263 175.68 0.67 0.0% 0.0% + (untracked) 93.27 0.0% 0.0% + python_kernel_build_src_co.setup 855 25.18 0.03 0.0% 0.0% + python_kernel_build_src_co.move 263 14.10 0.05 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_codegen 855 615,536.08 719.93 27.2% 23.2% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_assemble 855 253,795.98 296.84 11.2% 9.6% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_setup 855 132,219.33 154.64 5.8% 5.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_postprocess 855 105,243.40 123.09 4.7% 4.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_benchpost_lib_construction 855 72,859.94 85.22 69.2% 2.7% + python_benchpost_library_write 855 22,779.07 26.64 21.6% 0.9% + python_benchpost_naming 855 9,378.78 10.97 8.9% 0.4% + python_benchpost_client_config 855 194.33 0.23 0.2% 0.0% + (untracked) 31.28 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_validate 855 82,550.97 96.55 3.7% 3.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_co 855 22,054.87 25.80 1.0% 0.8% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_setup 855 20,891.95 24.44 0.9% 0.8% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_helpers 855 10,823.20 12.66 0.5% 0.4% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 3,883.85 0.2% 0.1% + ------------------------------------------------------------------------------------------------------------------------- + python_solution_generation 823 321,578.14 390.74 12.1% 12.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_forked_solutions 823 321,501.91 390.65 100.0% 12.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_custom_kernels 823 59.09 0.07 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 12.63 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_fork_permutations 823 4.51 0.01 0.0% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + python_ductile_initial_sampling 8 17,418.90 2,177.36 0.7% 0.7% + ------------------------------------------------------------------------------------------------------------------------- + python_client_execution 32 14,703.21 459.48 0.6% 0.6% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + C++ Client Setup & Teardown 9,804.08 66.7% 0.4% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + hip_initialization 32 5,005.78 156.43 51.1% 0.2% + process_exit 32 2,438.60 76.21 24.9% 0.1% + main_problem_factory 32 629.47 19.67 6.4% 0.0% + factory_hip_get_device_properties 4 625.92 156.48 99.4% 0.0% + (untracked) 3.55 0.6% 0.0% + main_teardown_locals 32 589.74 18.43 6.0% 0.0% + process_startup 32 467.09 14.60 4.8% 0.0% + main_timing_calibration 32 356.49 11.14 3.6% 0.0% + data_init_setup 32 98.04 3.06 1.0% 0.0% + code_object_loading 32 97.55 3.05 1.0% 0.0% + library_loading 32 64.25 2.01 0.7% 0.0% + lazy_loading_init 32 42.62 1.33 0.4% 0.0% + main_parse_args 32 10.48 0.33 0.1% 0.0% + reporter_setup 32 1.66 0.05 0.0% 0.0% + python_client_write_run_script 32 1.58 0.05 0.0% 0.0% + listener_setup 32 0.46 0.01 0.0% 0.0% + solution_iterator_setup 32 0.27 0.01 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Data Preparation 3,690.49 25.1% 0.1% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + gpu_input_preparation 40 3,536.05 88.40 95.8% 0.1% + gpu_input_prep_init_cpu_inputs 39 2,097.58 53.78 59.3% 0.1% + gpu_input_prep_copy_valid_to_gpu 40 1,349.34 33.73 38.2% 0.1% + gpu_input_prep_copy_swizzled 4 85.02 21.25 2.4% 0.0% + gpu_input_prep_batched_inputs 40 2.57 0.06 0.1% 0.0% + gpu_input_prep_copy_inputs_d2d 40 0.98 0.02 0.0% 0.0% + gpu_input_prep_constant_inputs 39 0.43 0.01 0.0% 0.0% + gpu_input_prep_convert_to_inputs 40 0.08 0.00 0.0% 0.0% + (untracked) 0.05 0.0% 0.0% + gpu_input_prep_slow_path 40 0.00 0.00 0.0% 0.0% + pre_problem 40 146.83 3.67 4.0% 0.0% + cpu_data_init 1 132.63 132.63 90.3% 0.0% + cpu_reference_gemm 1 13.62 13.62 9.3% 0.0% + solve_cpu_slow 1 13.61 13.61 99.9% 0.0% + (untracked) 0.59 0.4% 0.0% + rotating_buffer_preparation 40 5.84 0.15 0.2% 0.0% + gpu_input_reset 1 1.76 1.76 0.0% 0.0% + gpu_input_reset_copy_valid_to_gpu 1 0.84 0.84 47.8% 0.0% + gpu_input_reset_copy_swizzled 1 0.83 0.83 46.9% 0.0% + gpu_input_reset_batched_inputs 1 0.06 0.06 3.6% 0.0% + gpu_input_reset_copy_inputs_d2d 1 0.03 0.03 1.7% 0.0% + gpu_input_reset_convert_to_inputs 1 0.00 0.00 0.0% 0.0% + gpu_input_reset_slow_path 1 0.00 0.00 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Kernel Execution 1,189.65 8.1% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + pre_solution 585 1,178.81 2.02 99.1% 0.0% + pre_solution_listener.DataInitialization 585 1,120.43 1.92 95.0% 0.0% + pre_solution_mx_init_data 130 1,000.38 7.70 89.3% 0.0% + pre_solution_mx_side_regenerate 8 1,000.33 125.04 100.0% 0.0% + (untracked) 0.03 0.0% 0.0% + pre_solution_mx_side_cache_hit 252 0.02 0.00 0.0% 0.0% + pre_solution_mx_copy_valid_to_gpu_no_regen 126 103.32 0.82 9.2% 0.0% + pre_solution_mx_cpu_sync_no_regen 126 8.60 0.07 0.8% 0.0% + pre_solution_mx_copy_valid_to_gpu_after_regen 4 3.38 0.85 0.3% 0.0% + pre_solution_mx_copy_inputs_d2d_no_regen 126 3.22 0.03 0.3% 0.0% + pre_solution_mx_cpu_sync_after_regen 4 1.32 0.33 0.1% 0.0% + pre_solution_mx_copy_inputs_d2d_after_regen 4 0.14 0.03 0.0% 0.0% + (untracked) 0.03 0.0% 0.0% + pre_solution_mx_check 585 0.03 0.00 0.0% 0.0% + pre_solution_listener.MetaResultReporter 585 36.70 0.06 3.1% 0.0% + pre_solution_listener.ProgressListener 585 15.18 0.03 1.3% 0.0% + pre_solution_listener.BenchmarkTimer 585 3.41 0.01 0.3% 0.0% + pre_solution_listener.AllSolutionsIterator 585 1.11 0.00 0.1% 0.0% + pre_solution_listener.ReferenceValidator 585 0.99 0.00 0.1% 0.0% + cpu_reference_gemm_per_solution_inputs_changed 1 0.97 0.97 98.1% 0.0% + solve_cpu_slow_per_solution_inputs_changed 1 0.97 0.97 99.9% 0.0% + (untracked) 0.02 1.9% 0.0% + (untracked) 0.89 0.1% 0.0% + pre_solution_listener.SynchronizerValidator 585 0.10 0.00 0.0% 0.0% + pre_solution_listener.HardwareMonitorListener 585 0.01 0.00 0.0% 0.0% + post_solution 585 6.78 0.01 0.6% 0.0% + post_solution_log 585 5.47 0.01 80.7% 0.0% + post_solution_reporting 585 0.99 0.00 14.6% 0.0% + post_solution_perf_calc 585 0.20 0.00 2.9% 0.0% + (untracked) 0.07 1.1% 0.0% + post_solution_result_file 585 0.02 0.00 0.2% 0.0% + post_solution_validation 585 0.01 0.00 0.2% 0.0% + post_solution_sol_advance 585 0.01 0.00 0.2% 0.0% + post_solution_perf_reset 585 0.01 0.00 0.2% 0.0% + run_current_solution_check 585 1.61 0.00 0.1% 0.0% + post_problem 40 1.61 0.04 0.1% 0.0% + finalize_report 32 0.34 0.01 0.0% 0.0% + solution_selection 585 0.14 0.00 0.0% 0.0% + problem_report_index 40 0.13 0.00 0.0% 0.0% + warmup_runs 2 0.07 0.04 0.0% 0.0% + kernel_solving 1 0.06 0.06 0.0% 0.0% + problem_teardown 40 0.04 0.00 0.0% 0.0% + solution_teardown 585 0.02 0.00 0.0% 0.0% + warmup_events_setup 1 0.02 0.02 0.0% 0.0% + run_iteration_teardown 1 0.01 0.01 0.0% 0.0% + benchmark_runs 1 0.00 0.00 0.0% 0.0% + benchmark_pre_syncs 1 0.00 0.00 35.2% 0.0% + benchmark_post_syncs 1 0.00 0.00 35.2% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked C++ overhead) 11.94 0.1% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Timing Overhead 5.33 0.0% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + flush_timing_buffer 32 4.73 0.15 88.7% 0.0% + timing_overhead 32 0.60 0.02 11.3% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Reference Computation 1.71 0.0% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + validate_warmups 1 1.71 1.71 100.0% 0.0% + validate_warmups_listener.ReferenceValidator 1 1.18 1.18 69.3% 0.0% + validate_result_buffer_alloc 1 0.87 0.87 73.2% 0.0% + validate_gpu_readback 1 0.30 0.30 25.6% 0.0% + validate_element_comparison 1 0.01 0.01 0.8% 0.0% + validate_inputs_copy 1 0.00 0.00 0.1% 0.0% + validate_report 1 0.00 0.00 0.0% 0.0% + validate_setup 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.SynchronizerValidator 1 0.51 0.51 30.0% 0.0% + validate_synchronizer_check 1 0.51 0.51 100.0% 0.0% + validate_synchronizer_readback 1 0.34 0.34 66.6% 0.0% + validate_synchronizer_scan 1 0.17 0.17 33.3% 0.0% + validate_warmups_listener.BenchmarkTimer 1 0.00 0.00 0.3% 0.0% + validate_gpu_sync 1 0.00 0.00 87.9% 0.0% + validate_warmups_listener.MetaResultReporter 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.HardwareMonitorListener 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.AllSolutionsIterator 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.DataInitialization 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.ProgressListener 1 0.00 0.00 0.0% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + python_write_solutions 855 14,089.48 16.48 0.5% 0.5% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_dump 855 13,420.73 15.70 95.3% 0.5% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_prepare 855 528.55 0.62 3.8% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_wsol_prepare_nocache 855 527.16 0.62 99.7% 0.0% + (untracked) 1.39 0.3% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 128.36 0.9% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_header 855 11.83 0.01 0.1% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + (untracked) 11,241.68 0.4% 0.4% + ------------------------------------------------------------------------------------------------------------------------- + python_solution_indexing 855 8,355.06 9.77 0.3% 0.3% + ------------------------------------------------------------------------------------------------------------------------- + python_ductile_generate_solutions 32 4,866.91 152.09 0.2% 0.2% + ------------------------------------------------------------------------------------------------------------------------- + python_ductile_ga_between_evaluations 8 1,364.98 170.62 0.1% 0.1% + ------------------------------------------------------------------------------------------------------------------------- + python_write_cache 855 758.98 0.89 0.0% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + python_ductile_setup 8 29.94 3.74 0.0% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + python_ductile_clear_artifacts 24 17.63 0.73 0.0% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + python_ductile_read_scores 32 4.30 0.13 0.0% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + python_cache_check 831 0.58 0.00 0.0% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + python_ductile_verification 8 0.30 0.04 0.0% 0.0% + + Overlapping Totals (wrap other timers; not summed into the tree) + --------------------------------------------------------------------------------------------------------------------------- + python_client_subprocess 32 14,691.68 459.12 0.6% + main_total 32 11,190.67 349.71 0.4% + problem_total 40 4,882.23 122.06 0.2% + solution_total 585 1,191.28 2.04 0.0% + run_iteration_total 1 3.64 3.64 0.0% + validate_total 1 1.18 1.18 0.0% + validate_check_results_typed 1 1.18 1.18 0.0% + +----------------------------------------------------------------------------------------------------------------------------- +HIERARCHY VALIDATION: OK (all child phases fit within their parents) + +TIME BREAKDOWN (visual, % of wall clock) +----------------------------------------------------------------------------------------------------------------------------- + TOTAL RUNTIME 100.0% ################################################## + BUILD: solution generation & compile 98.1% ################################################# + Solution generation & build (Python) 97.3% ################################################ + python_kernel_compilation 85.1% ########################################## + python_kernel_build_src_co 38.2% ################### + python_kernel_build_src_co.compile 38.1% ################### + python_kernel_build_src_co.unbundle 0.1% + python_kernel_build_src_co.cache_check 0.0% + python_kernel_build_src_co.cache_populate 0.0% + python_kernel_build_src_co.setup 0.0% + python_kernel_build_src_co.move 0.0% + (untracked) 0.0% + python_kernel_codegen 23.2% ########### + python_kernel_write_assemble 9.6% #### + python_kernel_bench_setup 5.0% ## + python_kernel_bench_postprocess 4.0% # + python_benchpost_lib_construction 2.7% # + python_benchpost_library_write 0.9% + python_benchpost_naming 0.4% + python_benchpost_client_config 0.0% + (untracked) 0.0% + python_kernel_validate 3.1% # + python_kernel_build_co 0.8% + python_kernel_setup 0.8% + python_kernel_write_helpers 0.4% + (untracked) 0.1% + python_solution_generation 12.1% ###### + python_solgen_forked_solutions 12.1% ###### + python_solgen_custom_kernels 0.0% + python_solgen_fork_permutations 0.0% + (untracked) 0.0% + Ductile GA search (Python) 0.9% + python_ductile_initial_sampling 0.7% + python_ductile_generate_solutions 0.2% + python_ductile_ga_between_evaluations 0.1% + python_ductile_setup 0.0% + python_ductile_clear_artifacts 0.0% + python_ductile_read_scores 0.0% + python_ductile_verification 0.0% + SHARED: cache & solution files 0.9% + python_write_solutions 0.5% + python_wsol_dump 0.5% + python_wsol_prepare 0.0% + python_wsol_prepare_nocache 0.0% + (untracked) 0.0% + python_wsol_header 0.0% + (untracked) 0.0% + python_solution_indexing 0.3% + python_write_cache 0.0% + python_cache_check 0.0% + RUN: benchmark client 0.5% + Client launch (startup, setup & teardown) 0.4% + hip_initialization 0.2% + process_exit 0.1% + main_problem_factory 0.0% + factory_hip_get_device_properties 0.0% + (untracked) 0.0% + main_teardown_locals 0.0% + process_startup 0.0% + data_init_setup 0.0% + code_object_loading 0.0% + library_loading 0.0% + lazy_loading_init 0.0% + main_parse_args 0.0% + reporter_setup 0.0% + python_client_write_run_script 0.0% + listener_setup 0.0% + solution_iterator_setup 0.0% + Input data preparation 0.2% + gpu_input_preparation 0.1% + gpu_input_prep_init_cpu_inputs 0.1% + gpu_input_prep_copy_valid_to_gpu 0.1% + gpu_input_prep_copy_swizzled 0.0% + gpu_input_prep_batched_inputs 0.0% + gpu_input_prep_copy_inputs_d2d 0.0% + gpu_input_prep_constant_inputs 0.0% + gpu_input_prep_convert_to_inputs 0.0% + gpu_input_prep_slow_path 0.0% + (untracked) 0.0% + pre_solution_listener.DataInitialization 0.0% + pre_solution_mx_init_data 0.0% + pre_solution_mx_side_regenerate 0.0% + pre_solution_mx_side_cache_hit 0.0% + (untracked) 0.0% + pre_solution_mx_copy_valid_to_gpu_no_regen 0.0% + pre_solution_mx_cpu_sync_no_regen 0.0% + pre_solution_mx_copy_valid_to_gpu_after_regen 0.0% + pre_solution_mx_copy_inputs_d2d_no_regen 0.0% + pre_solution_mx_cpu_sync_after_regen 0.0% + pre_solution_mx_copy_inputs_d2d_after_regen 0.0% + pre_solution_mx_check 0.0% + (untracked) 0.0% + cpu_data_init 0.0% + rotating_buffer_preparation 0.0% + gpu_input_reset 0.0% + gpu_input_reset_copy_valid_to_gpu 0.0% + gpu_input_reset_copy_swizzled 0.0% + gpu_input_reset_batched_inputs 0.0% + gpu_input_reset_copy_inputs_d2d 0.0% + gpu_input_reset_convert_to_inputs 0.0% + gpu_input_reset_slow_path 0.0% + Per-solution bookkeeping & reporting 0.0% + pre_solution_listener.MetaResultReporter 0.0% + pre_solution_listener.ProgressListener 0.0% + post_solution_log 0.0% + pre_solution_listener.BenchmarkTimer 0.0% + run_current_solution_check 0.0% + post_problem 0.0% + pre_solution_listener.AllSolutionsIterator 0.0% + post_solution_reporting 0.0% + finalize_report 0.0% + post_solution_perf_calc 0.0% + solution_selection 0.0% + problem_report_index 0.0% + pre_solution_listener.SynchronizerValidator 0.0% + problem_teardown 0.0% + solution_teardown 0.0% + post_solution_result_file 0.0% + pre_solution_listener.HardwareMonitorListener 0.0% + post_solution_validation 0.0% + post_solution_sol_advance 0.0% + post_solution_perf_reset 0.0% + run_iteration_teardown 0.0% + Reference GEMM (CPU) 0.0% + cpu_reference_gemm 0.0% + solve_cpu_slow 0.0% + pre_solution_listener.ReferenceValidator 0.0% + cpu_reference_gemm_per_solution_inputs_changed 0.0% + solve_cpu_slow_per_solution_inputs_changed 0.0% + (untracked) 0.0% + Validation 0.0% + validate_warmups_listener.ReferenceValidator 0.0% + validate_result_buffer_alloc 0.0% + validate_gpu_readback 0.0% + validate_element_comparison 0.0% + validate_inputs_copy 0.0% + validate_report 0.0% + validate_setup 0.0% + validate_warmups_listener.SynchronizerValidator 0.0% + validate_synchronizer_check 0.0% + validate_synchronizer_readback 0.0% + validate_synchronizer_scan 0.0% + validate_warmups_listener.MetaResultReporter 0.0% + validate_warmups_listener.HardwareMonitorListener 0.0% + validate_warmups_listener.AllSolutionsIterator 0.0% + validate_warmups_listener.DataInitialization 0.0% + validate_warmups_listener.ProgressListener 0.0% + Kernel launch & execution 0.0% + warmup_runs 0.0% + kernel_solving 0.0% + warmup_events_setup 0.0% + validate_warmups_listener.BenchmarkTimer 0.0% + validate_gpu_sync 0.0% + benchmark_runs 0.0% + benchmark_pre_syncs 0.0% + benchmark_post_syncs 0.0% + MEASUREMENT OVERHEAD (instrumentation) 0.0% + main_timing_calibration 0.0% + flush_timing_buffer 0.0% + (timer overhead subtracted from parent timers) 0.0% + timing_overhead 0.0% + UNATTRIBUTED (untracked time & timer overhead) 0.4% + (untracked in python_benchmark_problems) 0.4% + (untracked in python_client_execution) 0.0% + (untracked in pre_solution) 0.0% + (untracked in pre_problem) 0.0% + (untracked in post_solution) 0.0% + +KEY INSIGHTS (% of wall clock) +----------------------------------------------------------------------------------------------------------------------------- + Solution generation & build (Python) 2581.93s (97.3%) + Ductile GA search (Python) 23.70s (0.9%) + SHARED: cache & solution files 23.20s (0.9%) + Client launch (startup, setup & teardown) 9.45s (0.4%) + Input data preparation 4.80s (0.2%) + MEASUREMENT OVERHEAD (instrumentation) 0.36s (0.0%) + Per-solution bookkeeping & reporting 0.07s (0.0%) + Reference GEMM (CPU) 0.01s (0.0%) + Validation 0.00s (0.0%) + Kernel launch & execution 0.00s (0.0%) + UNATTRIBUTED (untracked time & timer overhead) 11.26s (0.4%) + + +TOP 10 SLOWEST PROBLEMS (by CPU reference time) +----------------------------------------------------------------------------------------------------------------------------- + M N K Batch TypeA TypeD CPU Ref (ms) GPU (ms) + --------------------------------------------------------------------------------------------------------------------------- + 2048 256 128 16 Float8 BFloat16 13.62 0.00 + diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline_run/AGGREGATE_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline_run/AGGREGATE_analysis.txt new file mode 100644 index 000000000000..7039a2cf57fd --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline_run/AGGREGATE_analysis.txt @@ -0,0 +1,753 @@ +================================================================================================================================= +TIMING ANALYSIS SUMMARY +================================================================================================================================= + +TIMING BY PURPOSE +--------------------------------------------------------------------------------------------------------------------------------- + Category Count Total (ms) Mean (ms) % of Parent % of Wall +--------------------------------------------------------------------------------------------------------------------------------- + TOTAL RUNTIME 2,401,081.82 100.0% + RUN: benchmark client 2,284,890.54 95.2% 95.2% + Reference GEMM (CPU) 1,044,373.53 45.7% 43.5% + cpu_reference_gemm 28,596 887,980.04 31.05 85.0% 37.0% + solve_cpu_slow 1,947 689,043.16 353.90 77.6% 28.7% + solve_cpu_fast 26,656 198,918.72 7.46 22.4% 8.3% + (untracked) 18.17 0.0% 0.0% + pre_solution_listener.ReferenceValidator 2,355,016 156,393.49 0.07 15.0% 6.5% + cpu_reference_gemm_per_solution_inputs_changed 1,146 156,323.60 136.41 100.0% 6.5% + solve_cpu_fast_per_solution_inputs_changed 1,127 144,255.07 128.00 92.3% 6.0% + solve_cpu_slow_per_solution_inputs_changed 19 12,067.95 635.16 7.7% 0.5% + (untracked) 0.58 0.0% 0.0% + pre_solution_ref_skipped 2,353,182 45.85 0.00 0.0% 0.0% + (untracked) 24.04 0.0% 0.0% + Input data preparation 500,723.39 21.9% 20.9% + gpu_input_reset 2,106,835 298,818.12 0.14 59.7% 12.4% + gpu_input_reset_copy_valid_to_gpu 719,180 175,364.38 0.24 58.7% 7.3% + gpu_input_reset_batched_inputs 1,473,329 92,844.41 0.06 31.1% 3.9% + gpu_input_reset_copy_inputs_d2d 1,473,329 24,642.09 0.02 8.2% 1.0% + gpu_input_reset_copy_swizzled 15,719 3,967.98 0.25 1.3% 0.2% + gpu_input_reset_fast_reset_output 633,454 1,776.88 0.00 0.6% 0.1% + gpu_input_reset_convert_to_inputs 1,473,329 131.80 0.00 0.0% 0.0% + gpu_input_reset_slow_path 1,473,329 28.33 0.00 0.0% 0.0% + gpu_input_reset_rotating_init 2 0.02 0.01 0.0% 0.0% + gpu_input_reset_constant_inputs 4 0.01 0.00 0.0% 0.0% + gpu_input_reset_init_cpu_inputs 4 0.00 0.00 0.0% 0.0% + (untracked) 62.21 0.0% 0.0% + cpu_data_init 28,596 104,977.26 3.67 21.0% 4.4% + pre_solution_listener.DataInitialization 2,355,016 54,893.95 0.02 11.0% 2.3% + pre_solution_mx_init_data 16,038 49,609.33 3.09 90.4% 2.1% + pre_solution_mx_side_regenerate 2,298 49,605.78 21.59 100.0% 2.1% + pre_solution_mx_side_cache_hit 29,778 2.59 0.00 0.0% 0.0% + (untracked) 0.96 0.0% 0.0% + pre_solution_mx_copy_valid_to_gpu_no_regen 14,889 2,988.73 0.20 5.4% 0.1% + pre_solution_mx_copy_valid_to_gpu_after_regen 1,149 1,142.31 0.99 2.1% 0.0% + pre_solution_mx_cpu_sync_after_regen 1,149 425.75 0.37 0.8% 0.0% + pre_solution_mx_cpu_sync_no_regen 14,889 313.25 0.02 0.6% 0.0% + pre_solution_mx_copy_inputs_d2d_no_regen 14,889 312.41 0.02 0.6% 0.0% + pre_solution_mx_check 2,355,016 66.94 0.00 0.1% 0.0% + pre_solution_mx_copy_inputs_d2d_after_regen 1,149 23.67 0.02 0.0% 0.0% + (untracked) 11.55 0.0% 0.0% + gpu_input_preparation 28,641 41,335.21 1.44 8.3% 1.7% + gpu_input_prep_copy_valid_to_gpu 11,446 25,907.85 2.26 62.7% 1.1% + gpu_input_prep_copy_swizzled 1,151 10,866.81 9.44 26.3% 0.5% + gpu_input_prep_init_cpu_inputs 41 2,774.14 67.66 6.7% 0.1% + gpu_input_prep_batched_inputs 23,229 1,360.84 0.06 3.3% 0.1% + gpu_input_prep_copy_inputs_d2d 23,229 384.36 0.02 0.9% 0.0% + gpu_input_prep_fast_reset_output 5,403 22.99 0.00 0.1% 0.0% + gpu_input_prep_convert_to_inputs 23,229 8.08 0.00 0.0% 0.0% + gpu_input_prep_slow_path 23,229 2.12 0.00 0.0% 0.0% + gpu_input_prep_constant_inputs 43 0.49 0.01 0.0% 0.0% + gpu_input_prep_rotating_init 1 0.01 0.01 0.0% 0.0% + (untracked) 7.49 0.0% 0.0% + rotating_buffer_preparation 28,641 698.85 0.02 0.1% 0.0% + Validation 329,115.09 14.4% 13.7% + validate_warmups_listener.SynchronizerValidator 2,106,827 165,415.38 0.08 50.3% 6.9% + validate_synchronizer_check 2,106,827 165,375.26 0.08 100.0% 6.9% + validate_synchronizer_scan 608,873 106,738.74 0.18 64.5% 4.4% + validate_synchronizer_readback 608,873 58,500.01 0.10 35.4% 2.4% + validate_synchronizer_inactive 1,497,978 31.08 0.00 0.0% 0.0% + (untracked) 105.43 0.1% 0.0% + (untracked) 40.12 0.0% 0.0% + validate_warmups_listener.ReferenceValidator 2,106,827 163,290.06 0.08 49.6% 6.8% + validate_gpu_readback 1,907,339 94,421.18 0.05 57.8% 3.9% + validate_element_comparison 1,907,339 59,441.20 0.03 36.4% 2.5% + validate_result_buffer_alloc 1,907,339 8,739.14 0.00 5.4% 0.4% + validate_inputs_copy 1,905,787 212.65 0.00 0.1% 0.0% + validate_setup 1,905,883 55.78 0.00 0.0% 0.0% + validate_report 1,907,339 52.77 0.00 0.0% 0.0% + (untracked) 367.35 0.2% 0.0% + validate_warmups_listener.MetaResultReporter 2,106,827 136.41 0.00 0.0% 0.0% + validate_warmups_listener.AllSolutionsIterator 2,106,827 72.06 0.00 0.0% 0.0% + validate_warmups_listener.DataInitialization 2,106,827 69.82 0.00 0.0% 0.0% + validate_warmups_listener.ProgressListener 2,106,827 66.13 0.00 0.0% 0.0% + validate_warmups_listener.HardwareMonitorListener 2,106,827 65.23 0.00 0.0% 0.0% + Client launch (startup, setup & teardown) 289,773.74 12.7% 12.1% + hip_initialization 855 125,416.91 146.69 43.3% 5.2% + process_exit 855 67,796.11 79.29 23.4% 2.8% + main_problem_factory 855 30,518.37 35.69 10.5% 1.3% + factory_hip_get_device_properties 192 30,325.24 157.94 99.4% 1.3% + (untracked) 193.13 0.6% 0.0% + data_init_setup 855 20,125.24 23.54 6.9% 0.8% + process_startup 855 12,583.38 14.72 4.3% 0.5% + main_teardown_locals 855 11,812.48 13.82 4.1% 0.5% + code_object_loading 855 10,438.69 12.21 3.6% 0.4% + library_loading 855 9,704.53 11.35 3.3% 0.4% + lazy_loading_init 855 894.36 1.05 0.3% 0.0% + main_parse_args 855 361.93 0.42 0.1% 0.0% + reporter_setup 855 59.03 0.07 0.0% 0.0% + python_client_write_run_script 855 40.28 0.05 0.0% 0.0% + listener_setup 855 13.25 0.02 0.0% 0.0% + solution_iterator_setup 855 9.16 0.01 0.0% 0.0% + Per-solution bookkeeping & reporting 54,750.84 2.4% 2.3% + post_solution_log 2,355,016 17,525.78 0.01 32.0% 0.7% + pre_solution_listener.BenchmarkTimer 2,355,016 11,577.12 0.00 21.1% 0.5% + run_current_solution_check 2,355,016 6,825.34 0.00 12.5% 0.3% + post_solution_reporting 2,355,016 5,414.85 0.00 9.9% 0.2% + pre_solution_listener.ProgressListener 2,355,016 2,976.28 0.00 5.4% 0.1% + pre_solution_listener.AllSolutionsIterator 2,355,016 2,862.81 0.00 5.2% 0.1% + post_problem 28,641 1,717.04 0.06 3.1% 0.1% + run_iteration_teardown 2,106,835 1,469.03 0.00 2.7% 0.1% + pre_solution_listener.MetaResultReporter 2,355,016 1,451.38 0.00 2.7% 0.1% + post_solution_perf_calc 2,355,016 620.73 0.00 1.1% 0.0% + solution_selection 2,355,016 597.01 0.00 1.1% 0.0% + post_solution_validation 2,355,016 589.90 0.00 1.1% 0.0% + finalize_report 855 476.73 0.56 0.9% 0.0% + pre_solution_listener.SynchronizerValidator 2,355,016 320.74 0.00 0.6% 0.0% + solution_teardown 2,355,016 80.13 0.00 0.1% 0.0% + post_solution_perf_reset 2,355,016 63.52 0.00 0.1% 0.0% + post_solution_result_file 2,355,016 54.04 0.00 0.1% 0.0% + pre_solution_listener.HardwareMonitorListener 2,355,016 53.95 0.00 0.1% 0.0% + post_solution_sol_advance 2,355,016 46.72 0.00 0.1% 0.0% + problem_report_index 28,641 23.70 0.00 0.0% 0.0% + problem_teardown 28,641 4.07 0.00 0.0% 0.0% + Kernel launch & execution 35,714.28 1.6% 1.5% + warmup_runs 4,213,654 21,843.85 0.01 61.2% 0.9% + validate_warmups_listener.BenchmarkTimer 2,106,827 7,041.82 0.00 19.7% 0.3% + validate_gpu_sync 2,106,827 6,985.70 0.00 99.2% 0.3% + (untracked) 56.12 0.8% 0.0% + kernel_solving 2,106,835 5,531.00 0.00 15.5% 0.2% + warmup_events_setup 2,106,835 1,009.36 0.00 2.8% 0.0% + benchmark_runs 2,106,835 288.25 0.00 0.8% 0.0% + benchmark_validate_enqueues 319 138.66 0.43 48.1% 0.0% + validate_enqueues_listener.BenchmarkTimer 319 136.50 0.43 98.4% 0.0% + benchmark_sleep 319 136.44 0.43 100.0% 0.0% + benchmark_event_elapsed_time 8 0.02 0.00 0.0% 0.0% + benchmark_event_destroy 8 0.01 0.00 0.0% 0.0% + (untracked) 0.03 0.0% 0.0% + validate_enqueues_listener.ProgressListener 319 1.89 0.01 1.4% 0.0% + validate_enqueues_listener.MetaResultReporter 319 0.02 0.00 0.0% 0.0% + validate_enqueues_listener.HardwareMonitorListener 319 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.SynchronizerValidator 319 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.AllSolutionsIterator 319 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.ReferenceValidator 319 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.DataInitialization 319 0.01 0.00 0.0% 0.0% + (untracked) 0.20 0.1% 0.0% + benchmark_pre_syncs 2,106,835 57.56 0.00 20.0% 0.0% + benchmark_post_syncs 2,106,835 52.30 0.00 18.1% 0.0% + gpu_kernel_execution 319 39.61 0.12 13.7% 0.0% + benchmark_events_setup 319 0.09 0.00 0.0% 0.0% + benchmark_events_teardown 319 0.04 0.00 0.0% 0.0% + Library client (Python) 24,635.19 1.1% 1.0% + python_library_client_create_library 10 21,176.98 2,117.70 86.0% 0.9% + python_library_client_execution 10 3,324.12 332.41 13.5% 0.1% + python_library_client_subprocess 10 3,318.02 331.80 99.8% 0.1% + python_library_client_write_run_script 10 0.84 0.08 0.0% 0.0% + (untracked) 5.26 0.2% 0.0% + python_library_client_write_configs 10 118.05 11.80 0.5% 0.0% + python_library_client_setup 10 16.05 1.60 0.1% 0.0% + Library logic (Python) 5,804.47 0.3% 0.2% + python_logic_write 26 5,475.33 210.59 94.3% 0.2% + python_logic_parse_solutions 26 259.38 9.98 4.5% 0.0% + python_logic_analyze 26 69.77 2.68 1.2% 0.0% + BUILD: solution generation & compile 63,058.11 2.6% 2.6% + Solution generation & build (Python) 43,478.00 68.9% 1.8% + python_kernel_compilation 32 43,478.00 1,358.69 100.0% 1.8% + python_kernel_build_src_co 32 20,373.20 636.66 46.9% 0.8% + python_kernel_build_src_co.compile 4 20,291.46 5,072.87 99.6% 0.8% + python_kernel_build_src_co.cache_check 32 51.79 1.62 0.3% 0.0% + python_kernel_build_src_co.unbundle 4 22.95 5.74 0.1% 0.0% + python_kernel_build_src_co.cache_populate 4 3.07 0.77 0.0% 0.0% + python_kernel_build_src_co.setup 32 0.79 0.02 0.0% 0.0% + python_kernel_build_src_co.move 4 0.24 0.06 0.0% 0.0% + (untracked) 2.90 0.0% 0.0% + python_kernel_codegen 32 16,226.35 507.07 37.3% 0.7% + python_kernel_write_assemble 32 1,887.76 58.99 4.3% 0.1% + python_kernel_bench_setup 32 1,758.49 54.95 4.0% 0.1% + python_kernel_validate 32 1,519.26 47.48 3.5% 0.1% + python_kernel_build_co 32 623.12 19.47 1.4% 0.0% + python_kernel_bench_postprocess 32 454.53 14.20 1.0% 0.0% + python_benchpost_lib_construction 32 320.06 10.00 70.4% 0.0% + python_benchpost_library_write 32 78.93 2.47 17.4% 0.0% + python_benchpost_naming 32 50.83 1.59 11.2% 0.0% + python_benchpost_client_config 32 3.75 0.12 0.8% 0.0% + (untracked) 0.96 0.2% 0.0% + python_kernel_write_helpers 32 449.80 14.06 1.0% 0.0% + python_kernel_setup 32 135.43 4.23 0.3% 0.0% + (untracked) 50.05 0.1% 0.0% + Ductile GA search (Python) 19,580.11 31.1% 0.8% + python_ductile_initial_sampling 8 13,596.32 1,699.54 69.4% 0.6% + python_ductile_generate_solutions 32 4,572.39 142.89 23.4% 0.2% + python_ductile_ga_between_evaluations 8 1,355.61 169.45 6.9% 0.1% + python_ductile_setup 8 29.52 3.69 0.2% 0.0% + python_ductile_clear_artifacts 32 21.63 0.68 0.1% 0.0% + python_ductile_read_scores 32 4.31 0.13 0.0% 0.0% + python_ductile_verification 8 0.33 0.04 0.0% 0.0% + MEASUREMENT OVERHEAD (instrumentation) 52,754.57 2.2% 2.2% + flush_timing_buffer 865 33,553.73 38.79 63.6% 1.4% + main_timing_calibration 855 9,354.43 10.94 17.7% 0.4% + timing_overhead 855 5,093.55 5.96 9.7% 0.2% + (timer overhead subtracted from parent timers) 4,752.86 9.0% 0.2% + SHARED: cache & solution files 1,541.95 0.1% 0.1% + python_cache_check 831 1,249.54 1.50 81.0% 0.1% + python_write_solutions 855 198.78 0.23 12.9% 0.0% + python_wsol_dump 32 64.63 2.02 32.5% 0.0% + python_wsol_prepare 855 61.58 0.07 31.0% 0.0% + python_wsol_prepare_cache 823 58.61 0.07 95.2% 0.0% + python_wsol_prepare_nocache 32 0.96 0.03 1.6% 0.0% + (untracked) 2.00 3.2% 0.0% + python_wsol_header 855 60.44 0.07 30.4% 0.0% + (untracked) 12.13 6.1% 0.0% + python_write_cache 32 59.62 1.86 3.9% 0.0% + python_solution_indexing 32 34.01 1.06 2.2% 0.0% + UNATTRIBUTED (untracked time & timer overhead) -1,163.34 -0.0% -0.0% + (untracked in python_benchmark_problems) 2,280.60 0.0% 0.1% + (untracked in validate_warmups) 507.43 0.0% 0.0% + (untracked in pre_solution) 490.58 0.0% 0.0% + (untracked in pre_problem) 248.49 0.0% 0.0% + (untracked in python_library_logic) 9.08 0.0% 0.0% + (untracked in python_client_writer) 8.69 0.0% 0.0% + (children exceed benchmark_runs) -2.74 0.0% -0.0% + (children exceed post_solution) -148.90 0.0% -0.0% + (children exceed python_client_execution) -4,556.57 0.0% -0.2% + +TIMING BY TIMER NESTING (legacy view) +--------------------------------------------------------------------------------------------------------------------------------- + Category Count Total (ms) Mean (ms) % of Parent % of Wall +--------------------------------------------------------------------------------------------------------------------------------- + python_benchmark_problems 184 2,365,871.53 12,858.00 98.5% + ----------------------------------------------------------------------------------------------------------------------------- + python_client_execution 855 2,298,990.88 2,688.88 97.2% 95.7% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Data Preparation 1,334,057.97 58.0% 55.6% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + pre_problem 28,641 993,205.79 34.68 74.4% 41.4% + cpu_reference_gemm 28,596 887,980.04 31.05 89.4% 37.0% + solve_cpu_slow 1,947 689,043.16 353.90 77.6% 28.7% + solve_cpu_fast 26,656 198,918.72 7.46 22.4% 8.3% + (untracked) 18.17 0.0% 0.0% + cpu_data_init 28,596 104,977.26 3.67 10.6% 4.4% + (untracked) 248.49 0.0% 0.0% + gpu_input_reset 2,106,835 298,818.12 0.14 22.4% 12.4% + gpu_input_reset_copy_valid_to_gpu 719,180 175,364.38 0.24 58.7% 7.3% + gpu_input_reset_batched_inputs 1,473,329 92,844.41 0.06 31.1% 3.9% + gpu_input_reset_copy_inputs_d2d 1,473,329 24,642.09 0.02 8.2% 1.0% + gpu_input_reset_copy_swizzled 15,719 3,967.98 0.25 1.3% 0.2% + gpu_input_reset_fast_reset_output 633,454 1,776.88 0.00 0.6% 0.1% + gpu_input_reset_convert_to_inputs 1,473,329 131.80 0.00 0.0% 0.0% + (untracked) 62.21 0.0% 0.0% + gpu_input_reset_slow_path 1,473,329 28.33 0.00 0.0% 0.0% + gpu_input_reset_rotating_init 2 0.02 0.01 0.0% 0.0% + gpu_input_reset_constant_inputs 4 0.01 0.00 0.0% 0.0% + gpu_input_reset_init_cpu_inputs 4 0.00 0.00 0.0% 0.0% + gpu_input_preparation 28,641 41,335.21 1.44 3.1% 1.7% + gpu_input_prep_copy_valid_to_gpu 11,446 25,907.85 2.26 62.7% 1.1% + gpu_input_prep_copy_swizzled 1,151 10,866.81 9.44 26.3% 0.5% + gpu_input_prep_init_cpu_inputs 41 2,774.14 67.66 6.7% 0.1% + gpu_input_prep_batched_inputs 23,229 1,360.84 0.06 3.3% 0.1% + gpu_input_prep_copy_inputs_d2d 23,229 384.36 0.02 0.9% 0.0% + gpu_input_prep_fast_reset_output 5,403 22.99 0.00 0.1% 0.0% + gpu_input_prep_convert_to_inputs 23,229 8.08 0.00 0.0% 0.0% + (untracked) 7.49 0.0% 0.0% + gpu_input_prep_slow_path 23,229 2.12 0.00 0.0% 0.0% + gpu_input_prep_constant_inputs 43 0.49 0.01 0.0% 0.0% + gpu_input_prep_rotating_init 1 0.01 0.01 0.0% 0.0% + rotating_buffer_preparation 28,641 698.85 0.02 0.1% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Reference Computation 336,664.34 14.6% 14.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + validate_warmups 2,106,827 336,664.34 0.16 100.0% 14.0% + validate_warmups_listener.SynchronizerValidator 2,106,827 165,415.38 0.08 49.1% 6.9% + validate_synchronizer_check 2,106,827 165,375.26 0.08 100.0% 6.9% + validate_synchronizer_scan 608,873 106,738.74 0.18 64.5% 4.4% + validate_synchronizer_readback 608,873 58,500.01 0.10 35.4% 2.4% + (untracked) 105.43 0.1% 0.0% + validate_synchronizer_inactive 1,497,978 31.08 0.00 0.0% 0.0% + (untracked) 40.12 0.0% 0.0% + validate_warmups_listener.ReferenceValidator 2,106,827 163,290.06 0.08 48.5% 6.8% + validate_gpu_readback 1,907,339 94,421.18 0.05 57.8% 3.9% + validate_element_comparison 1,907,339 59,441.20 0.03 36.4% 2.5% + validate_result_buffer_alloc 1,907,339 8,739.14 0.00 5.4% 0.4% + (untracked) 367.35 0.2% 0.0% + validate_inputs_copy 1,905,787 212.65 0.00 0.1% 0.0% + validate_setup 1,905,883 55.78 0.00 0.0% 0.0% + validate_report 1,907,339 52.77 0.00 0.0% 0.0% + validate_warmups_listener.BenchmarkTimer 2,106,827 7,041.82 0.00 2.1% 0.3% + validate_gpu_sync 2,106,827 6,985.70 0.00 99.2% 0.3% + (untracked) 56.12 0.8% 0.0% + (untracked) 507.43 0.2% 0.0% + validate_warmups_listener.MetaResultReporter 2,106,827 136.41 0.00 0.0% 0.0% + validate_warmups_listener.AllSolutionsIterator 2,106,827 72.06 0.00 0.0% 0.0% + validate_warmups_listener.DataInitialization 2,106,827 69.82 0.00 0.0% 0.0% + validate_warmups_listener.ProgressListener 2,106,827 66.13 0.00 0.0% 0.0% + validate_warmups_listener.HardwareMonitorListener 2,106,827 65.23 0.00 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + C++ Client Setup & Teardown 299,128.16 13.0% 12.5% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + hip_initialization 855 125,416.91 146.69 41.9% 5.2% + process_exit 855 67,796.11 79.29 22.7% 2.8% + main_problem_factory 855 30,518.37 35.69 10.2% 1.3% + factory_hip_get_device_properties 192 30,325.24 157.94 99.4% 1.3% + (untracked) 193.13 0.6% 0.0% + data_init_setup 855 20,125.24 23.54 6.7% 0.8% + process_startup 855 12,583.38 14.72 4.2% 0.5% + main_teardown_locals 855 11,812.48 13.82 3.9% 0.5% + code_object_loading 855 10,438.69 12.21 3.5% 0.4% + library_loading 855 9,704.53 11.35 3.2% 0.4% + main_timing_calibration 855 9,354.43 10.94 3.1% 0.4% + lazy_loading_init 855 894.36 1.05 0.3% 0.0% + main_parse_args 855 361.93 0.42 0.1% 0.0% + reporter_setup 855 59.03 0.07 0.0% 0.0% + python_client_write_run_script 855 40.28 0.05 0.0% 0.0% + listener_setup 855 13.25 0.02 0.0% 0.0% + solution_iterator_setup 855 9.16 0.01 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Kernel Execution 295,049.68 12.8% 12.3% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + pre_solution 2,355,016 231,020.29 0.10 78.3% 9.6% + pre_solution_listener.ReferenceValidator 2,355,016 156,393.49 0.07 67.7% 6.5% + cpu_reference_gemm_per_solution_inputs_changed 1,146 156,323.60 136.41 100.0% 6.5% + solve_cpu_fast_per_solution_inputs_changed 1,127 144,255.07 128.00 92.3% 6.0% + solve_cpu_slow_per_solution_inputs_changed 19 12,067.95 635.16 7.7% 0.5% + (untracked) 0.58 0.0% 0.0% + pre_solution_ref_skipped 2,353,182 45.85 0.00 0.0% 0.0% + (untracked) 24.04 0.0% 0.0% + pre_solution_listener.DataInitialization 2,355,016 54,893.95 0.02 23.8% 2.3% + pre_solution_mx_init_data 16,038 49,609.33 3.09 90.4% 2.1% + pre_solution_mx_side_regenerate 2,298 49,605.78 21.59 100.0% 2.1% + pre_solution_mx_side_cache_hit 29,778 2.59 0.00 0.0% 0.0% + (untracked) 0.96 0.0% 0.0% + pre_solution_mx_copy_valid_to_gpu_no_regen 14,889 2,988.73 0.20 5.4% 0.1% + pre_solution_mx_copy_valid_to_gpu_after_regen 1,149 1,142.31 0.99 2.1% 0.0% + pre_solution_mx_cpu_sync_after_regen 1,149 425.75 0.37 0.8% 0.0% + pre_solution_mx_cpu_sync_no_regen 14,889 313.25 0.02 0.6% 0.0% + pre_solution_mx_copy_inputs_d2d_no_regen 14,889 312.41 0.02 0.6% 0.0% + pre_solution_mx_check 2,355,016 66.94 0.00 0.1% 0.0% + pre_solution_mx_copy_inputs_d2d_after_regen 1,149 23.67 0.02 0.0% 0.0% + (untracked) 11.55 0.0% 0.0% + pre_solution_listener.BenchmarkTimer 2,355,016 11,577.12 0.00 5.0% 0.5% + pre_solution_listener.ProgressListener 2,355,016 2,976.28 0.00 1.3% 0.1% + pre_solution_listener.AllSolutionsIterator 2,355,016 2,862.81 0.00 1.2% 0.1% + pre_solution_listener.MetaResultReporter 2,355,016 1,451.38 0.00 0.6% 0.1% + (untracked) 490.58 0.2% 0.0% + pre_solution_listener.SynchronizerValidator 2,355,016 320.74 0.00 0.1% 0.0% + pre_solution_listener.HardwareMonitorListener 2,355,016 53.95 0.00 0.0% 0.0% + post_solution 2,355,016 24,166.63 0.01 8.2% 1.0% + post_solution_log 2,355,016 17,525.78 0.01 72.5% 0.7% + post_solution_reporting 2,355,016 5,414.85 0.00 22.4% 0.2% + post_solution_perf_calc 2,355,016 620.73 0.00 2.6% 0.0% + post_solution_validation 2,355,016 589.90 0.00 2.4% 0.0% + post_solution_perf_reset 2,355,016 63.52 0.00 0.3% 0.0% + post_solution_result_file 2,355,016 54.04 0.00 0.2% 0.0% + post_solution_sol_advance 2,355,016 46.72 0.00 0.2% 0.0% + warmup_runs 4,213,654 21,843.85 0.01 7.4% 0.9% + run_current_solution_check 2,355,016 6,825.34 0.00 2.3% 0.3% + kernel_solving 2,106,835 5,531.00 0.00 1.9% 0.2% + post_problem 28,641 1,717.04 0.06 0.6% 0.1% + run_iteration_teardown 2,106,835 1,469.03 0.00 0.5% 0.1% + warmup_events_setup 2,106,835 1,009.36 0.00 0.3% 0.0% + solution_selection 2,355,016 597.01 0.00 0.2% 0.0% + finalize_report 855 476.73 0.56 0.2% 0.0% + benchmark_runs 2,106,835 285.51 0.00 0.1% 0.0% + benchmark_validate_enqueues 319 138.66 0.43 48.6% 0.0% + validate_enqueues_listener.BenchmarkTimer 319 136.50 0.43 98.4% 0.0% + benchmark_sleep 319 136.44 0.43 100.0% 0.0% + (untracked) 0.03 0.0% 0.0% + benchmark_event_elapsed_time 8 0.02 0.00 0.0% 0.0% + benchmark_event_destroy 8 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.ProgressListener 319 1.89 0.01 1.4% 0.0% + (untracked) 0.20 0.1% 0.0% + validate_enqueues_listener.MetaResultReporter 319 0.02 0.00 0.0% 0.0% + validate_enqueues_listener.HardwareMonitorListener 319 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.SynchronizerValidator 319 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.AllSolutionsIterator 319 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.ReferenceValidator 319 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.DataInitialization 319 0.01 0.00 0.0% 0.0% + benchmark_pre_syncs 2,106,835 57.56 0.00 20.2% 0.0% + benchmark_post_syncs 2,106,835 52.30 0.00 18.3% 0.0% + gpu_kernel_execution 319 39.61 0.12 13.9% 0.0% + benchmark_events_setup 319 0.09 0.00 0.0% 0.0% + benchmark_events_teardown 319 0.04 0.00 0.0% 0.0% + solution_teardown 2,355,016 80.13 0.00 0.0% 0.0% + problem_report_index 28,641 23.70 0.00 0.0% 0.0% + problem_teardown 28,641 4.07 0.00 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Timing Overhead 38,647.28 1.7% 1.6% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + flush_timing_buffer 865 33,553.73 38.79 86.8% 1.4% + timing_overhead 855 5,093.55 5.96 13.2% 0.2% + ----------------------------------------------------------------------------------------------------------------------------- + python_kernel_compilation 32 43,478.00 1,358.69 1.8% 1.8% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_src_co 32 20,373.20 636.66 46.9% 0.8% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_kernel_build_src_co.compile 4 20,291.46 5,072.87 99.6% 0.8% + python_kernel_build_src_co.cache_check 32 51.79 1.62 0.3% 0.0% + python_kernel_build_src_co.unbundle 4 22.95 5.74 0.1% 0.0% + python_kernel_build_src_co.cache_populate 4 3.07 0.77 0.0% 0.0% + (untracked) 2.90 0.0% 0.0% + python_kernel_build_src_co.setup 32 0.79 0.02 0.0% 0.0% + python_kernel_build_src_co.move 4 0.24 0.06 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_codegen 32 16,226.35 507.07 37.3% 0.7% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_assemble 32 1,887.76 58.99 4.3% 0.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_setup 32 1,758.49 54.95 4.0% 0.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_validate 32 1,519.26 47.48 3.5% 0.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_co 32 623.12 19.47 1.4% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_postprocess 32 454.53 14.20 1.0% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_benchpost_lib_construction 32 320.06 10.00 70.4% 0.0% + python_benchpost_library_write 32 78.93 2.47 17.4% 0.0% + python_benchpost_naming 32 50.83 1.59 11.2% 0.0% + python_benchpost_client_config 32 3.75 0.12 0.8% 0.0% + (untracked) 0.96 0.2% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_helpers 32 449.80 14.06 1.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_setup 32 135.43 4.23 0.3% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 50.05 0.1% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_ductile_initial_sampling 8 13,596.32 1,699.54 0.6% 0.6% + ----------------------------------------------------------------------------------------------------------------------------- + python_ductile_generate_solutions 32 4,572.39 142.89 0.2% 0.2% + ----------------------------------------------------------------------------------------------------------------------------- + (untracked) 2,280.60 0.1% 0.1% + ----------------------------------------------------------------------------------------------------------------------------- + python_ductile_ga_between_evaluations 8 1,355.61 169.45 0.1% 0.1% + ----------------------------------------------------------------------------------------------------------------------------- + python_cache_check 831 1,249.54 1.50 0.1% 0.1% + ----------------------------------------------------------------------------------------------------------------------------- + python_write_solutions 855 198.78 0.23 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_dump 32 64.63 2.02 32.5% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_prepare 855 61.58 0.07 31.0% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_wsol_prepare_cache 823 58.61 0.07 95.2% 0.0% + (untracked) 2.00 3.2% 0.0% + python_wsol_prepare_nocache 32 0.96 0.03 1.6% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_header 855 60.44 0.07 30.4% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 12.13 6.1% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_write_cache 32 59.62 1.86 0.0% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_solution_indexing 32 34.01 1.06 0.0% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_ductile_setup 8 29.52 3.69 0.0% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_ductile_clear_artifacts 32 21.63 0.68 0.0% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_ductile_read_scores 32 4.31 0.13 0.0% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_ductile_verification 8 0.33 0.04 0.0% 0.0% + =============================================================================================================================== + python_client_writer 10 24,643.88 2,464.39 1.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_library_client_create_library 10 21,176.98 2,117.70 85.9% 0.9% + ----------------------------------------------------------------------------------------------------------------------------- + python_library_client_execution 10 3,324.12 332.41 13.5% 0.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_library_client_subprocess 10 3,318.02 331.80 99.8% 0.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 5.26 0.2% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_library_client_write_run_script 10 0.84 0.08 0.0% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_library_client_write_configs 10 118.05 11.80 0.5% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_library_client_setup 10 16.05 1.60 0.1% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + (untracked) 8.69 0.0% 0.0% + =============================================================================================================================== + python_library_logic 26 5,813.56 223.60 0.2% + ----------------------------------------------------------------------------------------------------------------------------- + python_logic_write 26 5,475.33 210.59 94.2% 0.2% + ----------------------------------------------------------------------------------------------------------------------------- + python_logic_parse_solutions 26 259.38 9.98 4.5% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_logic_analyze 26 69.77 2.68 1.2% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + (untracked) 9.08 0.2% 0.0% + + Overlapping Totals (wrap other timers; not summed into the tree) + ------------------------------------------------------------------------------------------------------------------------------- + python_client_subprocess 855 2,303,413.47 2,694.05 95.9% + main_total 855 2,177,641.12 2,546.95 90.7% + problem_total 28,641 1,970,235.51 68.79 82.1% + solution_total 2,355,016 933,104.61 0.40 38.9% + run_iteration_total 2,106,835 668,286.95 0.32 27.8% + validate_total 1,905,883 163,188.57 0.09 6.8% + validate_check_results_typed 1,907,339 162,915.98 0.09 6.8% + +--------------------------------------------------------------------------------------------------------------------------------- +HIERARCHY VALIDATION WARNINGS +--------------------------------------------------------------------------------------------------------------------------------- + WARNING: Child phases of python_client_execution (2,307,117.88 ms) exceed parent (2,303,743.49 ms) by 3,374.39 ms (0.1%) + +TIME BREAKDOWN (visual, % of wall clock) +--------------------------------------------------------------------------------------------------------------------------------- + TOTAL RUNTIME 100.0% ################################################## + RUN: benchmark client 95.2% ############################################### + Reference GEMM (CPU) 43.5% ##################### + cpu_reference_gemm 37.0% ################## + solve_cpu_slow 28.7% ############## + solve_cpu_fast 8.3% #### + (untracked) 0.0% + pre_solution_listener.ReferenceValidator 6.5% ### + cpu_reference_gemm_per_solution_inputs_changed 6.5% ### + solve_cpu_fast_per_solution_inputs_changed 6.0% ### + solve_cpu_slow_per_solution_inputs_changed 0.5% + (untracked) 0.0% + pre_solution_ref_skipped 0.0% + (untracked) 0.0% + Input data preparation 20.9% ########## + gpu_input_reset 12.4% ###### + gpu_input_reset_copy_valid_to_gpu 7.3% ### + gpu_input_reset_batched_inputs 3.9% # + gpu_input_reset_copy_inputs_d2d 1.0% + gpu_input_reset_copy_swizzled 0.2% + gpu_input_reset_fast_reset_output 0.1% + gpu_input_reset_convert_to_inputs 0.0% + gpu_input_reset_slow_path 0.0% + gpu_input_reset_rotating_init 0.0% + gpu_input_reset_constant_inputs 0.0% + gpu_input_reset_init_cpu_inputs 0.0% + (untracked) 0.0% + cpu_data_init 4.4% ## + pre_solution_listener.DataInitialization 2.3% # + pre_solution_mx_init_data 2.1% # + pre_solution_mx_side_regenerate 2.1% # + pre_solution_mx_side_cache_hit 0.0% + (untracked) 0.0% + pre_solution_mx_copy_valid_to_gpu_no_regen 0.1% + pre_solution_mx_copy_valid_to_gpu_after_regen 0.0% + pre_solution_mx_cpu_sync_after_regen 0.0% + pre_solution_mx_cpu_sync_no_regen 0.0% + pre_solution_mx_copy_inputs_d2d_no_regen 0.0% + pre_solution_mx_check 0.0% + pre_solution_mx_copy_inputs_d2d_after_regen 0.0% + (untracked) 0.0% + gpu_input_preparation 1.7% + gpu_input_prep_copy_valid_to_gpu 1.1% + gpu_input_prep_copy_swizzled 0.5% + gpu_input_prep_init_cpu_inputs 0.1% + gpu_input_prep_batched_inputs 0.1% + gpu_input_prep_copy_inputs_d2d 0.0% + gpu_input_prep_fast_reset_output 0.0% + gpu_input_prep_convert_to_inputs 0.0% + gpu_input_prep_slow_path 0.0% + gpu_input_prep_constant_inputs 0.0% + gpu_input_prep_rotating_init 0.0% + (untracked) 0.0% + rotating_buffer_preparation 0.0% + Validation 13.7% ###### + validate_warmups_listener.SynchronizerValidator 6.9% ### + validate_synchronizer_check 6.9% ### + validate_synchronizer_scan 4.4% ## + validate_synchronizer_readback 2.4% # + validate_synchronizer_inactive 0.0% + (untracked) 0.0% + (untracked) 0.0% + validate_warmups_listener.ReferenceValidator 6.8% ### + validate_gpu_readback 3.9% # + validate_element_comparison 2.5% # + validate_result_buffer_alloc 0.4% + validate_inputs_copy 0.0% + validate_setup 0.0% + validate_report 0.0% + (untracked) 0.0% + validate_warmups_listener.MetaResultReporter 0.0% + validate_warmups_listener.AllSolutionsIterator 0.0% + validate_warmups_listener.DataInitialization 0.0% + validate_warmups_listener.ProgressListener 0.0% + validate_warmups_listener.HardwareMonitorListener 0.0% + Client launch (startup, setup & teardown) 12.1% ###### + hip_initialization 5.2% ## + process_exit 2.8% # + main_problem_factory 1.3% + factory_hip_get_device_properties 1.3% + (untracked) 0.0% + data_init_setup 0.8% + process_startup 0.5% + main_teardown_locals 0.5% + code_object_loading 0.4% + library_loading 0.4% + lazy_loading_init 0.0% + main_parse_args 0.0% + reporter_setup 0.0% + python_client_write_run_script 0.0% + listener_setup 0.0% + solution_iterator_setup 0.0% + Per-solution bookkeeping & reporting 2.3% # + post_solution_log 0.7% + pre_solution_listener.BenchmarkTimer 0.5% + run_current_solution_check 0.3% + post_solution_reporting 0.2% + pre_solution_listener.ProgressListener 0.1% + pre_solution_listener.AllSolutionsIterator 0.1% + post_problem 0.1% + run_iteration_teardown 0.1% + pre_solution_listener.MetaResultReporter 0.1% + post_solution_perf_calc 0.0% + solution_selection 0.0% + post_solution_validation 0.0% + finalize_report 0.0% + pre_solution_listener.SynchronizerValidator 0.0% + solution_teardown 0.0% + post_solution_perf_reset 0.0% + post_solution_result_file 0.0% + pre_solution_listener.HardwareMonitorListener 0.0% + post_solution_sol_advance 0.0% + problem_report_index 0.0% + problem_teardown 0.0% + Kernel launch & execution 1.5% + warmup_runs 0.9% + validate_warmups_listener.BenchmarkTimer 0.3% + validate_gpu_sync 0.3% + (untracked) 0.0% + kernel_solving 0.2% + warmup_events_setup 0.0% + benchmark_runs 0.0% + benchmark_validate_enqueues 0.0% + validate_enqueues_listener.BenchmarkTimer 0.0% + benchmark_sleep 0.0% + benchmark_event_elapsed_time 0.0% + benchmark_event_destroy 0.0% + (untracked) 0.0% + validate_enqueues_listener.ProgressListener 0.0% + validate_enqueues_listener.MetaResultReporter 0.0% + validate_enqueues_listener.HardwareMonitorListener 0.0% + validate_enqueues_listener.SynchronizerValidator 0.0% + validate_enqueues_listener.AllSolutionsIterator 0.0% + validate_enqueues_listener.ReferenceValidator 0.0% + validate_enqueues_listener.DataInitialization 0.0% + (untracked) 0.0% + benchmark_pre_syncs 0.0% + benchmark_post_syncs 0.0% + gpu_kernel_execution 0.0% + benchmark_events_setup 0.0% + benchmark_events_teardown 0.0% + Library client (Python) 1.0% + python_library_client_create_library 0.9% + python_library_client_execution 0.1% + python_library_client_subprocess 0.1% + python_library_client_write_run_script 0.0% + (untracked) 0.0% + python_library_client_write_configs 0.0% + python_library_client_setup 0.0% + Library logic (Python) 0.2% + python_logic_write 0.2% + python_logic_parse_solutions 0.0% + python_logic_analyze 0.0% + BUILD: solution generation & compile 2.6% # + Solution generation & build (Python) 1.8% + python_kernel_compilation 1.8% + python_kernel_build_src_co 0.8% + python_kernel_build_src_co.compile 0.8% + python_kernel_build_src_co.cache_check 0.0% + python_kernel_build_src_co.unbundle 0.0% + python_kernel_build_src_co.cache_populate 0.0% + python_kernel_build_src_co.setup 0.0% + python_kernel_build_src_co.move 0.0% + (untracked) 0.0% + python_kernel_codegen 0.7% + python_kernel_write_assemble 0.1% + python_kernel_bench_setup 0.1% + python_kernel_validate 0.1% + python_kernel_build_co 0.0% + python_kernel_bench_postprocess 0.0% + python_benchpost_lib_construction 0.0% + python_benchpost_library_write 0.0% + python_benchpost_naming 0.0% + python_benchpost_client_config 0.0% + (untracked) 0.0% + python_kernel_write_helpers 0.0% + python_kernel_setup 0.0% + (untracked) 0.0% + Ductile GA search (Python) 0.8% + python_ductile_initial_sampling 0.6% + python_ductile_generate_solutions 0.2% + python_ductile_ga_between_evaluations 0.1% + python_ductile_setup 0.0% + python_ductile_clear_artifacts 0.0% + python_ductile_read_scores 0.0% + python_ductile_verification 0.0% + MEASUREMENT OVERHEAD (instrumentation) 2.2% # + flush_timing_buffer 1.4% + main_timing_calibration 0.4% + timing_overhead 0.2% + (timer overhead subtracted from parent timers) 0.2% + SHARED: cache & solution files 0.1% + python_cache_check 0.1% + python_write_solutions 0.0% + python_wsol_dump 0.0% + python_wsol_prepare 0.0% + python_wsol_prepare_cache 0.0% + python_wsol_prepare_nocache 0.0% + (untracked) 0.0% + python_wsol_header 0.0% + (untracked) 0.0% + python_write_cache 0.0% + python_solution_indexing 0.0% + UNATTRIBUTED (untracked time & timer overhead) -0.0% + (untracked in python_benchmark_problems) 0.1% + (untracked in validate_warmups) 0.0% + (untracked in pre_solution) 0.0% + (untracked in pre_problem) 0.0% + (untracked in python_library_logic) 0.0% + (untracked in python_client_writer) 0.0% + (children exceed benchmark_runs) -0.0% + (children exceed post_solution) -0.0% + (children exceed python_client_execution) -0.2% + +KEY INSIGHTS (% of wall clock) +--------------------------------------------------------------------------------------------------------------------------------- + Reference GEMM (CPU) 1044.37s (43.5%) + Input data preparation 500.72s (20.9%) + Validation 329.12s (13.7%) + Client launch (startup, setup & teardown) 289.77s (12.1%) + Per-solution bookkeeping & reporting 54.75s (2.3%) + MEASUREMENT OVERHEAD (instrumentation) 48.00s (2.0%) + Solution generation & build (Python) 43.48s (1.8%) + Kernel launch & execution 35.71s (1.5%) + Library client (Python) 24.64s (1.0%) + Ductile GA search (Python) 19.58s (0.8%) + Library logic (Python) 5.80s (0.2%) + SHARED: cache & solution files 1.54s (0.1%) + UNATTRIBUTED (untracked time & timer overhead) -1.16s (-0.0%) + + CPU reference / GPU execution ratio: 22420.9x + >>> CPU reference > 10x GPU execution time! + +TOP 10 SLOWEST PROBLEMS (by CPU reference time) +--------------------------------------------------------------------------------------------------------------------------------- + M N K Batch TypeA TypeD CPU Ref (ms) GPU (ms) + ------------------------------------------------------------------------------------------------------------------------------- + 4111 4111 4111 1 Float8 Float8 132701.21 0.00 + 4111 4111 4111 1 Float8 Float8 111415.30 0.00 + 4111 4111 4111 1 Int8 Int8 84486.58 0.00 + 4111 4111 4111 1 Int8 BFloat16 79404.49 0.00 + 4111 4111 4111 1 Int8 Int32 77682.79 0.00 + 4111 4111 4111 1 Int8 Half 55478.46 0.00 + 43776 10624 6400 1 Float8 BFloat16 15800.60 0.00 + 71936 4096 7168 1 Float8 BFloat16 13588.45 0.00 + 13056 7680 21504 1 Float8 BFloat16 13101.60 0.00 + 13696 15488 8448 1 Float8 BFloat16 9323.37 0.00 + diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline_run/common_test_outcomes.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline_run/common_test_outcomes.txt new file mode 100644 index 000000000000..f4b047051e2c --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline_run/common_test_outcomes.txt @@ -0,0 +1,571 @@ +# Correctness baseline (P0.2): outcome of every test in the gfx950 split run phase +# (split/run_split.sh run phase: --use-cache, -n 4, cores 0-24, GPU 0) at 2083e6254bf +# (code-identical to 55e29436e9a). One " " per line, sorted. +# 567 tests: 194 PASSED, 2 FAILED, 371 SKIPPED. See ../baseline.md. +tensilelite/Tests/common/flags/gfx942/test_build_only.py::test_compile PASSED +tensilelite/Tests/common/flags/gfx942/test_build_only.py::test_use_cache SKIPPED +tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_compile_and_benchmark SKIPPED +tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_use_cache SKIPPED +tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_use_cache_creates_cache_dir SKIPPED +tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_use_cache_different_params SKIPPED +tensilelite/Tests/common/test_config_helpers.py::test_base_skip_gfx1250_inert_on_strict_target PASSED +tensilelite/Tests/common/test_config_helpers.py::test_ffm_fail_inert_on_hardware PASSED +tensilelite/Tests/common/test_config_helpers.py::test_ffm_fail_inert_on_other_arch PASSED +tensilelite/Tests/common/test_config_helpers.py::test_ffm_fail_xfails_under_ffm PASSED +tensilelite/Tests/common/test_config_helpers.py::test_find_available_archs_does_not_map_the_retired_v0_name_to_the_base PASSED +tensilelite/Tests/common/test_config_helpers.py::test_find_available_archs_keeps_stepping_name PASSED +tensilelite/Tests/common/test_config_helpers.py::test_skip_gfx1250_strict_fires_on_strict_target PASSED +tensilelite/Tests/common/test_config_helpers.py::test_skip_gfx1250_strict_inert_on_base_target PASSED +tensilelite/Tests/common/test_config_helpers.py::test_unmarked_config_never_xfails_under_ffm PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/amaxd/8rn_amaxd.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/amaxd/gfx1250/amax_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/benchmark_runs/bf16_tn_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/benchmark_runs/fp16_tn_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/benchmark_runs/fp32_tn.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/client/rotate_mode0.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/client/rotate_mode0_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/client/rotate_mode1.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/client/rotate_mode1_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/comm/gfx950/fused_a2a.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/bbs_tn_dp.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/bbs_tn_multi_size_single_kernel.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/bbs_tn_streamK.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/f32_tt_batched.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/f8_tn_mx.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/hhs_nt_multi_size.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/bf16_tn_dp.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/bf16_tn_multi_size_single_kernel.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/bf16_tn_streamK.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/f16_nt_multi_size.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/f32_tt_batched.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/f8_tn_mx_subtile.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/f8bf16_tn_cms.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/xf32_nn.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/exception/f16_cf32_nan.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/exception/f16_cf32_nan_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/8rn_b_s.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/ag.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/agntab_coverage_gfx942.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/bf16_tn.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/bf16_tn_predict.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/bufferLoadStore0.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/cgemm.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/dot2_gfx942.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/dtl.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/dtv.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/dtv_gfx90a.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/f32_mac.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/f64.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/f64_gfx90a.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/f64_mac.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp16_HH_BHS_bf16mfma.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp16_tn.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp16_use_e.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp16fp32mix.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp32_nt.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp32_scalealphavec.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp32_tn.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8bf8nss.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8n.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8n_rowwise.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8n_use_e.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8nfp16mix_fp8nss.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8nfp16mix_hfp8ns.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8nfp16mix_hhs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8nfp16mix_hhs_lsu4.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gemm_bias_strided_batched.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gemm_bias_strided_batched_broadcast.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gemm_bias_strided_batched_broadcast_gfx940.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gemm_bias_strided_batched_gfx940.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gemm_ck_gfx942.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/bf16_tn_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/fp16_HH_BHS_bf16mfma_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/fp16_tn_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/fp16_use_e_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/gemm_bias_strided_batched_broadcast_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/gemm_bias_strided_batched_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/gsuasb_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/i8_gsu_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/1024_vgpr_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/1024_vgpr_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/8r_b_s_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/agntab_coverage_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/agntab_coverage_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b6f4ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b6f4ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b8b8s_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b8b8s_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b8f8ss_grvw1_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b8hs_grvw1_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_CLS_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_CLS_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_tn_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_tn_gfx12_predict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf6_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf6_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf6_tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf6_tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/cgemm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/cgemm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/decouple_pgr_tdm_fuse_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f32_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f32_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f32_grid_limit_size.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4b8ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4b8ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4f6ss_tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4f6ss_tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f64_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f64_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6_tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6_tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6b6ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6b6ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6b8ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6b8ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8b6ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8b6ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8b8ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8b8ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f4ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f4ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_cls_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_cls_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_grvw1_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_pk8_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_pk8_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_sr_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_sr_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp16_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp16_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp16_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp16_use_e_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp16_use_e_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp4_tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp4_tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp8_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp8_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp8_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp8bf8ss_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gemm_bias_strided_batched_broadcast_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gemm_bias_strided_batched_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gltr_bounds_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gsu_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gsu_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gsuaad_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gsuasb_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/i8_gsu_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/i8i8i_grvw1_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/i8ii_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/i8ii_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/i8ii_grvw1_gfx12.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/interleave_alpha2_group_store_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/largeLds_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/largeLds_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf4_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf4_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf6_tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf6_tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf6f4ss_tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf6f4ss_tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf8_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf8_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf8f4ss_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf8f4ss_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/nt_th_nv_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/nt_th_nv_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/pgl_bound_check.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/rap_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/segment_b_rotate_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/segment_interleave_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/segment_interleave_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/stinky_sia4.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/stinky_sia4_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250-strict_bench.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250-strict_cluster.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250_bench.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250_cluster.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_general_batched_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_general_batched_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_multicast_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_multicast_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_split_gfx1250-strict_xfail.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_split_gfx1250_xfail.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdmpluslds_tdm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdmpluslds_tdm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/xfp32_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/xfp32_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/xfp32_inf_support_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/zgemm_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/zgemm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx1250/cluster_entry_handoff.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/agntab_coverage_gfx950.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/bf16_cvt.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/custom_mainloop_scheduling.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/custom_mainloop_scheduling_tf32.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/custom_xfp32.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/dot2_gfx950.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/dtl.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/f16f8mix_ss.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/f16f8mix_ss_stoch.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/f8b8hs.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/f8f16mix_f8s.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp16_use_e_wide_store.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp8.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_bf16_tn_act.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_bf16_tn_act_groupgemm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_fp32_tn_act.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_fp32_tn_act_groupgemm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/general_wgm.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/i8_gsu_gfx950.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/lds160K.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/lds_tr.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mx32f4_tn.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mx32f8_tn.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_bf16_tn_act.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_bf16_tn_act_groupgemm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_fp32_tn_act.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_fp32_tn_act_groupgemm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_bf16_tn_act.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_bf16_tn_act_groupgemm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_fp32_tn_act.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_fp32_tn_act_groupgemm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp8_residual_fail_classes_repro.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp8_verify_fail_repro.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/plr_zero.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/preloop_local_read_wait.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/ss_bss.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_bf16.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_fp8.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp4.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_bias_sav.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt192x256_plr1_wgm4.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt256x192_plr1_wgm1.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt256x320_plr0_wgm1.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt320x256_plr1_wgm4.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_multi_du_multi_partition.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_tail_loop.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_tail_loop_smoke.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/xfp32.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gsuasb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gsuc.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/hh_f8nhs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/i8_gsu.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/i8_gsu_gfx940.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/icache_flush.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/largeMT.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/large_size.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/lsu_fnuz.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/lsu_i8.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/mbskPrefetchOpt.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/mix_cvt_after_ds_fnuz.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/nan_bounds_check_odd_padding.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/no_shadowinit.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/sia0_pgr.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/swizzleA.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/swizzleB.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/swizzle_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/tail_small_size.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/ulsgro1.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/use_beta_false.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/wgm.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/xfp32.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/zgemm.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/fp16_gradient_activation.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/fp16_gradient_bias.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/fp8bf8nss_gradient_bias_b.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx11/fp16_gradient_activation_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx11/fp16_gradient_bias_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_bgrada_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_bgradb_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_bgradd_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_dgelu_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_bgrada_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_bgradb_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_bgradd_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_dgelu_gfx1250-strict.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/bbs_bgrada_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/bbs_bgradb_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/bbs_bgradd_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/bbs_dgelu_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/hhs_bgrada_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/hhs_bgradb_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/hhs_bgradd_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/hhs_dgelu_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx950/bf16_gradient_bias.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/gfx11/grouped_gemm_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/gfx11/grouped_gemm_userargs_gfx11.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/grouped_gemm.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/grouped_gemm_ck_gfx942.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/grouped_gemm_gfx1250.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/grouped_gemm_userargs.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gsu/auto_gsu.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gsu/auto_gsu_zero_dim.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gsu/f32_gsu.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gsu/gsu_mbsk.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sgemm_dual_fmac_asm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sgemm_streamk_asm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sgemm_xf32_asm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/bf16_activation.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/f16_activation.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8bs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8bs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8f8.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8f8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8hs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8hs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_bf16.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_bf16_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f16.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f16_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8_ml.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8b8.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8b8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8bs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8bs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8hs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8hs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_fp16_ml1.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8bs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8bs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8hs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8hs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8is.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8is_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_all.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_f16_transposes.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_gl2prefetch.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_ldspadding.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_multicast.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/bf16_activation.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/f16_activation.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8bs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8bs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8f8.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8f8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8hs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8hs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_bf16.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_bf16_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f16.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f16_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8_ml.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8b8.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8b8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8bs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8bs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8hs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8hs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_fp16_ml1.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_gate.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8bs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8bs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8hs.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8hs_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8is.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8is_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_all.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_f16_transposes.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_gl2prefetch.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_iteratemode.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_ldspadding.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_multicast.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_split_xfail.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/bf16_activation.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/bf16_gate_r.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/f8n_gate_r.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/fp16_activation.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/fp16_gate_r.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/i8_activation.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/libray_logic.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_b8f8n.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_b8f8n_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_bf16.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_bf16_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_bf8n.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_bf8n_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_f8b8n.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_f8b8n_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_f8n.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_f8n_sb.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi16.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi16_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi32.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi32_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_gsuc.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi16.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi16_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi32.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi32_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8bs.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8bs_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8hs.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8hs_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8is.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8is_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_vw_lg_one.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_vw_lg_one_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/use_sgpr_for_gro.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/use_sgpr_for_gro_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/bf16_gate_r.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/f8_gate_r.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/fp16_gate_r.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_b8.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_b8_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_bf16.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_bf16_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_dtl.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_f16.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_f16_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_f8.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_f8_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_i8.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_i8_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_i8bs.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_i8hs.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_ldstr.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_vw_lg_one.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_vw_lg_one_sb.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/data_parallel_static_mx32f4.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/data_parallel_static_sgemm.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx11/sk_bgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx11/sk_hgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx12/sk_bgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx12/sk_hgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/bf16_vector_epilogue_persistent.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf4.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf4_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf8.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf8_halfplr_tdm_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf8_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_bf16gemm_tdm_subtile.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_bgemm_tdm_split.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_bgemm_tdm_split_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_dynamic_hgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_dynamic_sgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_f8gemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_halfplr_f8gemm_tdm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_halfplr_mxf8gemm_tdm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_hgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_hybrid_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_explicit.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_pap_prefetchgl2.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_tdm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_tdm_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf6b8gemm_tdm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf6b8gemm_tdm_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8b6gemm_tdm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8b6gemm_tdm_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8f4gemm_tdm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8f4gemm_tdm_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_explicit.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_pap_prefetchgl2.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm_split.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm_split_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_sgemm_pap.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_sgemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_tdm_general_batched.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_tdm_subtile_general_batched.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/data_parallel_cluster_multicast_boundaries.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf4_cluster_multicast_ext.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf4_pap_cluster_multicast_ext.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf8_cluster_multicast_ext.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf8_pap_cluster_multicast_ext.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/sk_hgemm_debug_modes.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/sk_mxf4gemm_tdm_cluster_ext.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/sk_mxf4gemm_tdm_ext.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/sk_mxf4gemm_tdm_pap_ext.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/data_parallel_static_hgemm_use_e.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/data_parallel_static_sgemm_pap.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_bgemm_pap.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_hgemm_use_e.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_hybrid_xcc_mapping.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxf4gemm_pap.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_basic.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_batch_combo.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_full_coverage.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_streamk_combo.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_atomic_variants.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_basic_variants.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_batch_mi_combo.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_batch_sizes.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_comprehensive.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_debug_modes.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_full_coverage.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_activation_combo.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_basic.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_batch_combo.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_full_coverage.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_streamk_combo.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_matrix_instructions.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_problem_sizes.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_tree_reduction.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_xcc_mapping.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_debug_modes.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_dynamic_pap.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_hybrid_pap.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_pap.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_I8gemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_bgemm_div.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_bgemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_bgemm_sia1.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_cgemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_dgemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_dynamic.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_dynamic_mxfp4.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_dynamic_work_stealing.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_f8gemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_f8ngemm_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hgemm_epilogues.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hgemm_lsu.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hgemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hgemm_race.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hybrid.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hybrid_work_stealing.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_mx32f4_quick.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_mx32f8_quick.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_queue_selfclean.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_reduction.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_sgemm_alpha0.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_sgemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_zgemm_quick.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_aiter_bf16.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_aiter_f4.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_ck.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rocblas_gemv_bf16_tn_m1.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rocblas_gemv_bf16_tn_m2.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rocblas_gemv_bf16_tn_m4.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rr.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rr_f6.yaml] FAILED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rr_mx.yaml] FAILED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_tensile_sk.yaml] SKIPPED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_triton_f4.yaml] PASSED +tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_wave_bf16.yaml] PASSED diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/measure_env.sh b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/measure_env.sh new file mode 100644 index 000000000000..3d2134cf5933 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/measure_env.sh @@ -0,0 +1,93 @@ +# Copyright Advanced Micro Devices, Inc., or its affiliates. +# SPDX-License-Identifier: MIT +# +# Sourced by split/run_split.sh, per_config/run_per_config.sh and per_config/run_per_config_build.sh. +# Every location the measurement scripts read, write or delete can be overridden from the +# environment; the defaults are the paths the committed baselines were measured with. +# TL TensileLite source tree pytest runs from (e.g. a git worktree of another commit) +# CLIENT prebuilt tensilelite-client +# ART build artifact dir (split build phase writes it, the run phases read it). The +# default is the measured baseline, which check_rm_target refuses to delete, so +# split/run_split.sh needs ART set to a new dir. +# TENSILE_CACHE Tensile's cache root, cleared by clear_caches. TENSILE_HELPER_CACHE_DIR, +# TENSILE_HELPER_PCH_DIR and TENSILE_HELPER_CHUNK_CACHE_DIR are always +# exported as $TENSILE_CACHE/helper_cache, $TENSILE_CACHE/helper_pch and +# $TENSILE_CACHE/helper_chunk_cache (inherited values are ignored), so clearing +# TENSILE_CACHE clears the helper-kernel cache, HIP-header PCH and helper chunk +# cache Tensile really uses. TENSILE_DISABLE_HELPER_PCH=1 (compiles without a +# PCH) and TENSILE_DISABLE_HELPER_CHUNK_CACHE=1 (compiles every helper chunk) +# are passed through. +# TENSILELITE_CACHE ~/.tensilelite, cleared by clear_caches as before (current TensileLite +# sources neither read nor write it). +# GPU_TARGET --gpu-targets of the build phases (default gfx950) +# CORES taskset core list (default 0-24; OMP_NUM_THREADS stays 25, so give 25 cores) +# GPU ROCR_VISIBLE_DEVICES of the runs (default 0) +# ANALYZE_DIR directory holding analyze_timing.py / aggregate_timing.py (default $TL/scripts), +# so logs of an older TL can be analysed with a newer analyzer +# CLEAR_CACHE per-config | once | never: when the per-config scripts clear the caches (default +# per-config; KEEP_CACHE=1 means once). once clears before the first config of the +# pass; never leaves the caches as the previous pass left them. +# OUT, LIST and SCRATCH are set by the scripts themselves (their defaults depend on the script); +# each script then calls check_rm_target on every directory it deletes. +TL=${TL:-/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite} +CLIENT=${CLIENT:-/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client} +ART=${ART:-/tmp/tl_split_gfx950/artifacts} +TENSILE_CACHE=${TENSILE_CACHE:-/home/alvasile/.tensile} +TENSILELITE_CACHE=${TENSILELITE_CACHE:-/home/alvasile/.tensilelite} +GPU_TARGET=${GPU_TARGET:-gfx950} +CORES=${CORES:-0-24} +GPU=${GPU:-0} +ANALYZE_DIR=${ANALYZE_DIR:-$TL/scripts} +if [ -z "$CLEAR_CACHE" ]; then + if [ "${KEEP_CACHE:-0}" = "1" ]; then CLEAR_CACHE=once; else CLEAR_CACHE=per-config; fi +fi +case $CLEAR_CACHE in + per-config|once|never) ;; + *) echo "CLEAR_CACHE='$CLEAR_CACHE' must be per-config, once or never" >&2; exit 1 ;; +esac +export TENSILE_HELPER_CACHE_DIR=$TENSILE_CACHE/helper_cache +export TENSILE_HELPER_PCH_DIR=$TENSILE_CACHE/helper_pch +export TENSILE_HELPER_CHUNK_CACHE_DIR=$TENSILE_CACHE/helper_chunk_cache +MEASURE_DIR=$(dirname "$(realpath -- "${BASH_SOURCE[0]}")") +# Measured baseline client build and artifacts; never deleted. +BASELINE_DIRS=(/tmp/tl_client_msgpack /tmp/tl_split_gfx950/artifacts) + +# check_rm_target VAR...: exit unless every $VAR is safe to `rm -rf`: an absolute path at least two +# levels deep that is not $HOME or a directory containing it, and that neither is, contains, nor is +# inside $TL, $OUT, this script's directory, a baseline dir, $CLIENT's directory or $ART (unless +# VAR is ART). +check_rm_target() { + local var val real keep keeps + for var in "$@"; do + val=${!var} + case $val in + /*) ;; + *) echo "$var='$val' must be an absolute path" >&2; exit 1 ;; + esac + real=$(realpath -m -- "$val") + case $real in + /*/*) ;; + *) echo "$var='$val' is too broad to delete" >&2; exit 1 ;; + esac + if [ -n "$HOME" ]; then + keep=$(realpath -m -- "$HOME") + case $keep/ in + "$real"/*) echo "$var='$val' is or contains $keep; refusing to delete it" >&2; exit 1 ;; + esac + fi + keeps=("$TL" "$OUT" "$MEASURE_DIR" "${BASELINE_DIRS[@]}" "$(dirname -- "$CLIENT")") + [ "$var" = ART ] || keeps+=("$ART") + for keep in "${keeps[@]}"; do + [ -n "$keep" ] || continue + keep=$(realpath -m -- "$keep") + case $keep/ in + "$real"/*) echo "$var='$val' is or contains $keep; refusing to delete it" >&2; exit 1 ;; + esac + case $real/ in + "$keep"/*) echo "$var='$val' is inside $keep; refusing to delete it" >&2; exit 1 ;; + esac + done + done +} + +clear_caches() { rm -rf -- "$TENSILE_CACHE" "$TENSILELITE_CACHE"; } diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config/run_per_config.sh b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config/run_per_config.sh new file mode 100755 index 000000000000..b983a9e38e6a --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config/run_per_config.sh @@ -0,0 +1,64 @@ +#!/bin/bash +# Copyright Advanced Micro Devices, Inc., or its affiliates. +# SPDX-License-Identifier: MIT +# +# Run step of every gfx950 common config that passed in the split run phase, one at a time +# (serial, no xdist), with TimingInstrumentation on, and save per config: +# /.log.gz raw instrumented pytest/client log +# /_analysis.txt scripts/analyze_timing.py output +# /times.txt pytest wall seconds + return code +# Build artifacts come from the split build phase (../split) or from +# run_per_config_build.sh with KEEP_ARTIFACTS=1. Caches (~/.tensile, ~/.tensilelite) are cleared +# before each config, as in run_cold_warm.sh / the split phases, unless CLEAR_CACHE=once +# (or KEEP_CACHE=1): then they are cleared once before the first config and kept between configs +# (a normal run, where e.g. the helper-kernel cache is shared by all configs). CLEAR_CACHE=never +# keeps the cache a preceding build pass left (a CI-like build-then-run on one node). +# When every config is DONE, scripts/aggregate_timing.py sums the logs of ALL configs (failed ones +# included) into $OUT/AGGREGATE_analysis.txt. Re-running the script after a finished pass skips the +# runs and only regenerates the aggregate. +# Usage: bash run_per_config.sh [configs.txt] (one tensilelite/Tests/... yaml path per line) +# Env: TL, CLIENT, ART, TENSILE_CACHE, TENSILELITE_CACHE (see ../measure_env.sh), +# CORES, GPU, ANALYZE_DIR, CLEAR_CACHE (see ../measure_env.sh), +# OUT (default $TL/timing_runs/full_runs_gfx950/per_config), KEEP_CACHE. +source "$(dirname "${BASH_SOURCE[0]}")/../measure_env.sh" || exit 1 +OUT=${OUT:-$TL/timing_runs/full_runs_gfx950/per_config} +LIST=${1:-$OUT/configs.txt} +check_rm_target TENSILE_CACHE TENSILELITE_CACHE +cd "$TL" || exit 1 +mkdir -p "$OUT" +export OMP_NUM_THREADS=25 OPENBLAS_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=$GPU PYTHONUNBUFFERED=1 +TO="--cxx-compiler,/usr/bin/amdclang++,--global-parameters,TimingInstrumentation=True" +COMMON=(-s -v -n 0 --timeout=5400 -m "common and not skip-gfx94X" + "--global-parameters=LibraryFormat='msgpack'" "--tensile-options=$TO" -p no:warnings -p no:cacheprovider) + +n=0; total=$(wc -l < "$LIST") +if [ "$CLEAR_CACHE" = once ]; then + clear_caches + echo "CLEAR_CACHE=once: caches cleared once at $(date -u +%FT%TZ)" >> "$OUT/progress.txt" +elif [ "$CLEAR_CACHE" = never ]; then + echo "CLEAR_CACHE=never: caches kept at $(date -u +%FT%TZ)" >> "$OUT/progress.txt" +fi +while read -r cfg; do + [ -z "$cfg" ] && continue + n=$((n+1)) + name=$(echo "${cfg#tensilelite/Tests/}" | sed -e 's/\.yaml$//' -e 's#/#__#g') + D=$OUT/$name + if [ -f "$D/times.txt" ] && grep -q DONE "$D/times.txt"; then continue; fi # resumable + mkdir -p "$D" + RID="tensilelite/Tests/common/test_config_run.py::test_config_run[$cfg]" + if [ "$CLEAR_CACHE" = per-config ]; then + clear_caches + fi + s=$(date +%s) + taskset -c "$CORES" pytest "$RID" --use-cache --artifact-dir "$ART" --prebuilt-client="$CLIENT" \ + "${COMMON[@]}" > "$D/$name.log" 2>&1 + rc=$? + echo "run rc=$rc seconds=$(( $(date +%s)-s )) ($n/$total)" > "$D/times.txt" + python "$ANALYZE_DIR/analyze_timing.py" "$D/$name.log" > "$D/${name}_analysis.txt" 2>&1 + gzip -f "$D/$name.log" + echo DONE >> "$D/times.txt" + echo "$(date -u +%T) $n/$total $cfg rc=$rc" >> "$OUT/progress.txt" +done < "$LIST" +python "$ANALYZE_DIR/aggregate_timing.py" "$OUT"/*/*.log.gz > "$OUT/AGGREGATE_analysis.txt" 2>&1 +echo "aggregate rc=$? logs=$(ls "$OUT"/*/*.log.gz | wc -l)" >> "$OUT/progress.txt" +echo ALLDONE > "$OUT/done.txt" diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config/run_per_config_build.sh b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config/run_per_config_build.sh new file mode 100644 index 000000000000..93f200f1ec96 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config/run_per_config_build.sh @@ -0,0 +1,73 @@ +#!/bin/bash +# Copyright Advanced Micro Devices, Inc., or its affiliates. +# SPDX-License-Identifier: MIT +# +# Build step of every common config, one at a time (serial, no xdist), with +# TimingInstrumentation on, COLD: the caches (~/.tensile, ~/.tensilelite) are cleared before every +# config, so each build pays the full helper-kernel / assembler / library cost. With +# CLEAR_CACHE=once they are cleared before the first config only (a CI-like build that starts +# with an empty cache). Counterpart of run_per_config.sh (run step). Saves per config: +# /_build.log.gz raw instrumented pytest log +# /_build_analysis.txt scripts/analyze_timing.py output +# /times.txt pytest wall seconds + return code +# Artifacts go to a per-config scratch dir that is deleted after the config's log is saved (they +# are not used by anything else; the run phase uses $ART). With KEEP_ARTIFACTS=1 they go to $ART +# instead (emptied once before the first config) and are kept, for run_per_config.sh. +# When every config is DONE, scripts/aggregate_timing.py sums the logs of ALL configs (failed ones +# included) into $OUT/AGGREGATE_analysis.txt. Re-running the script after a finished pass skips the +# builds and only regenerates the aggregate. +# Usage: bash run_per_config_build.sh [configs.txt] (one tensilelite/Tests/... yaml path per line) +# Env: TL, CLIENT, TENSILE_CACHE, TENSILELITE_CACHE (see ../measure_env.sh), +# GPU_TARGET, CORES, GPU, ANALYZE_DIR, CLEAR_CACHE (see ../measure_env.sh), KEEP_ARTIFACTS, ART, +# OUT (default .../per_config_build), SCRATCH (default /tmp/tl_build_scratch_gfx950) +source "$(dirname "${BASH_SOURCE[0]}")/../measure_env.sh" || exit 1 +OUT=${OUT:-$TL/timing_runs/full_runs_gfx950/per_config_build} +LIST=${1:-$TL/timing_runs/full_runs_gfx950/per_config/configs.txt} +SCRATCH=${SCRATCH:-/tmp/tl_build_scratch_gfx950} +check_rm_target TENSILE_CACHE TENSILELITE_CACHE SCRATCH +if [ "${KEEP_ARTIFACTS:-0}" = "1" ]; then check_rm_target ART; fi +if [ "$CLEAR_CACHE" = never ]; then + echo "CLEAR_CACHE=never is not supported by the build pass" >&2; exit 1 +fi +cd "$TL" || exit 1 +export OMP_NUM_THREADS=25 OPENBLAS_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=$GPU PYTHONUNBUFFERED=1 +TO="--cxx-compiler,/usr/bin/amdclang++,--global-parameters,TimingInstrumentation=True" +COMMON=(-s -v -n 0 --timeout=5400 -m "common and not skip-gfx94X" + "--global-parameters=LibraryFormat='msgpack'" "--tensile-options=$TO" -p no:warnings -p no:cacheprovider) + +mkdir -p "$OUT" "$SCRATCH" +n=0; total=$(wc -l < "$LIST") +if [ ! -f "$OUT/started.txt" ]; then # first start of the pass, not a resume + if [ "$CLEAR_CACHE" = once ]; then + clear_caches + echo "CLEAR_CACHE=once: caches cleared once at $(date -u +%FT%TZ)" >> "$OUT/progress.txt" + fi + if [ "${KEEP_ARTIFACTS:-0}" = "1" ]; then rm -rf -- "$ART"; fi + date -u +%FT%TZ > "$OUT/started.txt" +fi +if [ "${KEEP_ARTIFACTS:-0}" = "1" ]; then mkdir -p "$ART"; fi +while read -r cfg; do + [ -z "$cfg" ] && continue + n=$((n+1)) + name=$(echo "${cfg#tensilelite/Tests/}" | sed -e 's/\.yaml$//' -e 's#/#__#g') + D=$OUT/$name + if [ -f "$D/times.txt" ] && grep -q DONE "$D/times.txt"; then continue; fi # resumable + mkdir -p "$D" + A=$SCRATCH/$name + if [ "${KEEP_ARTIFACTS:-0}" = "1" ]; then A=$ART; else rm -rf -- "$A"; fi + BID="tensilelite/Tests/common/test_config_build.py::test_config_build[$cfg]" + if [ "$CLEAR_CACHE" = per-config ]; then clear_caches; fi # cold cache for every build + s=$(date +%s) + taskset -c "$CORES" pytest "$BID" --build-only --gpu-targets "$GPU_TARGET" --artifact-dir "$A" \ + --prebuilt-client="$CLIENT" "${COMMON[@]}" > "$D/${name}_build.log" 2>&1 # ductile builds run the client + rc=$? + echo "build_cold rc=$rc seconds=$(( $(date +%s)-s )) ($n/$total)" > "$D/times.txt" + python "$ANALYZE_DIR/analyze_timing.py" "$D/${name}_build.log" > "$D/${name}_build_analysis.txt" 2>&1 + gzip -f "$D/${name}_build.log" + if [ "${KEEP_ARTIFACTS:-0}" != "1" ]; then rm -rf -- "$A"; fi + echo DONE >> "$D/times.txt" + echo "$(date -u +%T) $n/$total $cfg rc=$rc" >> "$OUT/progress.txt" +done < "$LIST" +python "$ANALYZE_DIR/aggregate_timing.py" "$OUT"/*/*_build.log.gz > "$OUT/AGGREGATE_analysis.txt" 2>&1 +echo "aggregate rc=$? logs=$(ls "$OUT"/*/*_build.log.gz | wc -l)" >> "$OUT/progress.txt" +echo ALLDONE > "$OUT/done.txt" diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_build/AGGREGATE_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_build/AGGREGATE_analysis.txt new file mode 100644 index 000000000000..87cc51111044 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_build/AGGREGATE_analysis.txt @@ -0,0 +1,598 @@ +============================================================================================================================= +TIMING ANALYSIS SUMMARY +============================================================================================================================= + +TIMING BY PURPOSE +----------------------------------------------------------------------------------------------------------------------------- + Category Count Total (ms) Mean (ms) % of Parent % of Wall +----------------------------------------------------------------------------------------------------------------------------- + TOTAL RUNTIME 2,577,849.77 100.0% + BUILD: solution generation & compile 2,530,063.12 98.1% 98.1% + Solution generation & build (Python) 2,510,348.29 99.2% 97.4% + python_kernel_compilation 855 2,203,797.86 2,577.54 87.8% 85.5% + python_kernel_build_src_co 855 1,007,817.04 1,178.73 45.7% 39.1% + python_kernel_build_src_co.compile 263 1,004,724.33 3,820.24 99.7% 39.0% + python_kernel_build_src_co.unbundle 263 1,566.60 5.96 0.2% 0.1% + python_kernel_build_src_co.cache_check 855 1,172.50 1.37 0.1% 0.0% + python_kernel_build_src_co.cache_populate 263 214.63 0.82 0.0% 0.0% + python_kernel_build_src_co.setup 855 22.87 0.03 0.0% 0.0% + python_kernel_build_src_co.move 263 17.46 0.07 0.0% 0.0% + (untracked) 98.65 0.0% 0.0% + python_kernel_codegen 855 592,584.11 693.08 26.9% 23.0% + python_kernel_write_assemble 855 227,868.45 266.51 10.3% 8.8% + python_kernel_bench_setup 855 131,638.45 153.96 6.0% 5.1% + python_kernel_bench_postprocess 855 106,816.89 124.93 4.8% 4.1% + python_benchpost_lib_construction 855 74,773.69 87.45 70.0% 2.9% + python_benchpost_library_write 855 22,324.17 26.11 20.9% 0.9% + python_benchpost_naming 855 9,436.66 11.04 8.8% 0.4% + python_benchpost_client_config 855 249.07 0.29 0.2% 0.0% + (untracked) 33.30 0.0% 0.0% + python_kernel_validate 855 81,431.47 95.24 3.7% 3.2% + python_kernel_setup 855 20,886.91 24.43 0.9% 0.8% + python_kernel_build_co 855 18,912.53 22.12 0.9% 0.7% + python_kernel_write_helpers 855 11,912.59 13.93 0.5% 0.5% + (untracked) 3,929.43 0.2% 0.2% + python_solution_generation 823 306,550.42 372.48 12.2% 11.9% + python_solgen_forked_solutions 823 306,464.54 372.37 100.0% 11.9% + python_solgen_custom_kernels 823 67.99 0.08 0.0% 0.0% + python_solgen_fork_permutations 823 4.83 0.01 0.0% 0.0% + (untracked) 13.06 0.0% 0.0% + Ductile GA search (Python) 19,714.83 0.8% 0.8% + python_ductile_initial_sampling 8 13,760.99 1,720.12 69.8% 0.5% + python_ductile_generate_solutions 32 4,536.07 141.75 23.0% 0.2% + python_ductile_ga_between_evaluations 8 1,366.26 170.78 6.9% 0.1% + python_ductile_setup 8 30.08 3.76 0.2% 0.0% + python_ductile_clear_artifacts 24 16.81 0.70 0.1% 0.0% + python_ductile_read_scores 32 4.29 0.13 0.0% 0.0% + python_ductile_verification 8 0.33 0.04 0.0% 0.0% + SHARED: cache & solution files 23,180.53 0.9% 0.9% + python_write_solutions 855 14,069.93 16.46 60.7% 0.5% + python_wsol_dump 855 13,390.16 15.66 95.2% 0.5% + python_wsol_prepare 855 531.38 0.62 3.8% 0.0% + python_wsol_prepare_nocache 855 529.92 0.62 99.7% 0.0% + (untracked) 1.47 0.3% 0.0% + python_wsol_header 855 12.63 0.01 0.1% 0.0% + (untracked) 135.75 1.0% 0.0% + python_solution_indexing 855 8,279.91 9.68 35.7% 0.3% + python_write_cache 855 830.02 0.97 3.6% 0.0% + python_cache_check 831 0.67 0.00 0.0% 0.0% + RUN: benchmark client 12,995.06 0.5% 0.5% + Client launch (startup, setup & teardown) 9,950.79 76.6% 0.4% + hip_initialization 32 4,712.80 147.27 47.4% 0.2% + process_exit 32 3,467.40 108.36 34.8% 0.1% + main_problem_factory 32 595.29 18.60 6.0% 0.0% + factory_hip_get_device_properties 4 590.74 147.68 99.2% 0.0% + (untracked) 4.56 0.8% 0.0% + process_startup 32 468.60 14.64 4.7% 0.0% + main_teardown_locals 32 386.04 12.06 3.9% 0.0% + data_init_setup 32 97.08 3.03 1.0% 0.0% + code_object_loading 32 96.45 3.01 1.0% 0.0% + library_loading 32 65.62 2.05 0.7% 0.0% + lazy_loading_init 32 45.39 1.42 0.5% 0.0% + main_parse_args 32 11.80 0.37 0.1% 0.0% + reporter_setup 32 1.84 0.06 0.0% 0.0% + python_client_write_run_script 32 1.63 0.05 0.0% 0.0% + listener_setup 32 0.52 0.02 0.0% 0.0% + solution_iterator_setup 32 0.33 0.01 0.0% 0.0% + Input data preparation 3,004.17 23.1% 0.1% + gpu_input_preparation 40 2,387.72 59.69 79.5% 0.1% + gpu_input_prep_init_cpu_inputs 39 1,309.51 33.58 54.8% 0.1% + gpu_input_prep_copy_valid_to_gpu 40 937.87 23.45 39.3% 0.0% + gpu_input_prep_copy_swizzled 4 135.91 33.98 5.7% 0.0% + gpu_input_prep_batched_inputs 40 2.63 0.07 0.1% 0.0% + gpu_input_prep_copy_inputs_d2d 40 0.98 0.02 0.0% 0.0% + gpu_input_prep_constant_inputs 39 0.64 0.02 0.0% 0.0% + gpu_input_prep_convert_to_inputs 40 0.11 0.00 0.0% 0.0% + gpu_input_prep_slow_path 40 0.00 0.00 0.0% 0.0% + (untracked) 0.07 0.0% 0.0% + pre_solution_listener.DataInitialization 585 488.60 0.84 16.3% 0.0% + pre_solution_mx_init_data 130 369.08 2.84 75.5% 0.0% + pre_solution_mx_side_regenerate 8 369.03 46.13 100.0% 0.0% + pre_solution_mx_side_cache_hit 252 0.02 0.00 0.0% 0.0% + (untracked) 0.02 0.0% 0.0% + pre_solution_mx_copy_valid_to_gpu_no_regen 126 103.07 0.82 21.1% 0.0% + pre_solution_mx_cpu_sync_no_regen 126 8.60 0.07 1.8% 0.0% + pre_solution_mx_copy_valid_to_gpu_after_regen 4 3.29 0.82 0.7% 0.0% + pre_solution_mx_copy_inputs_d2d_no_regen 126 3.18 0.03 0.7% 0.0% + pre_solution_mx_cpu_sync_after_regen 4 1.19 0.30 0.2% 0.0% + pre_solution_mx_copy_inputs_d2d_after_regen 4 0.13 0.03 0.0% 0.0% + pre_solution_mx_check 585 0.03 0.00 0.0% 0.0% + (untracked) 0.03 0.0% 0.0% + cpu_data_init 1 119.66 119.66 4.0% 0.0% + rotating_buffer_preparation 40 6.46 0.16 0.2% 0.0% + gpu_input_reset 1 1.73 1.73 0.1% 0.0% + gpu_input_reset_copy_valid_to_gpu 1 0.82 0.82 47.6% 0.0% + gpu_input_reset_copy_swizzled 1 0.81 0.81 46.9% 0.0% + gpu_input_reset_batched_inputs 1 0.06 0.06 3.7% 0.0% + gpu_input_reset_copy_inputs_d2d 1 0.03 0.03 1.7% 0.0% + gpu_input_reset_convert_to_inputs 1 0.00 0.00 0.0% 0.0% + gpu_input_reset_slow_path 1 0.00 0.00 0.0% 0.0% + Per-solution bookkeeping & reporting 30.61 0.2% 0.0% + pre_solution_listener.AllSolutionsIterator 585 10.23 0.02 33.4% 0.0% + post_solution_log 585 5.20 0.01 17.0% 0.0% + pre_solution_listener.BenchmarkTimer 585 3.99 0.01 13.0% 0.0% + pre_solution_listener.ProgressListener 585 2.60 0.00 8.5% 0.0% + run_current_solution_check 585 2.03 0.00 6.6% 0.0% + pre_solution_listener.MetaResultReporter 585 2.00 0.00 6.5% 0.0% + post_solution_reporting 585 1.75 0.00 5.7% 0.0% + post_problem 40 1.66 0.04 5.4% 0.0% + finalize_report 32 0.35 0.01 1.1% 0.0% + post_solution_perf_calc 585 0.25 0.00 0.8% 0.0% + problem_report_index 40 0.15 0.00 0.5% 0.0% + solution_selection 585 0.14 0.00 0.5% 0.0% + pre_solution_listener.SynchronizerValidator 585 0.10 0.00 0.3% 0.0% + problem_teardown 40 0.06 0.00 0.2% 0.0% + solution_teardown 585 0.02 0.00 0.1% 0.0% + post_solution_result_file 585 0.01 0.00 0.0% 0.0% + post_solution_validation 585 0.01 0.00 0.0% 0.0% + pre_solution_listener.HardwareMonitorListener 585 0.01 0.00 0.0% 0.0% + post_solution_sol_advance 585 0.01 0.00 0.0% 0.0% + post_solution_perf_reset 585 0.01 0.00 0.0% 0.0% + run_iteration_teardown 1 0.01 0.01 0.0% 0.0% + Reference GEMM (CPU) 7.11 0.1% 0.0% + pre_solution_listener.ReferenceValidator 585 4.64 0.01 65.2% 0.0% + cpu_reference_gemm_per_solution_inputs_changed 1 4.62 4.62 99.5% 0.0% + solve_cpu_slow_per_solution_inputs_changed 1 4.62 4.62 100.0% 0.0% + (untracked) 0.02 0.5% 0.0% + cpu_reference_gemm 1 2.47 2.47 34.8% 0.0% + solve_cpu_slow 1 2.46 2.46 99.5% 0.0% + (untracked) 0.01 0.5% 0.0% + Validation 2.20 0.0% 0.0% + validate_warmups_listener.ReferenceValidator 1 1.25 1.25 57.0% 0.0% + validate_result_buffer_alloc 1 0.94 0.94 74.9% 0.0% + validate_gpu_readback 1 0.30 0.30 24.2% 0.0% + validate_element_comparison 1 0.01 0.01 0.7% 0.0% + validate_inputs_copy 1 0.00 0.00 0.1% 0.0% + validate_report 1 0.00 0.00 0.0% 0.0% + validate_setup 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.SynchronizerValidator 1 0.94 0.94 42.9% 0.0% + validate_synchronizer_check 1 0.94 0.94 100.0% 0.0% + validate_synchronizer_readback 1 0.78 0.78 82.5% 0.0% + validate_synchronizer_scan 1 0.16 0.16 17.5% 0.0% + validate_warmups_listener.MetaResultReporter 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.HardwareMonitorListener 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.AllSolutionsIterator 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.ProgressListener 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.DataInitialization 1 0.00 0.00 0.0% 0.0% + Kernel launch & execution 0.18 0.0% 0.0% + kernel_solving 1 0.08 0.08 45.7% 0.0% + warmup_runs 2 0.08 0.04 43.5% 0.0% + warmup_events_setup 1 0.02 0.02 8.2% 0.0% + validate_warmups_listener.BenchmarkTimer 1 0.00 0.00 2.4% 0.0% + validate_gpu_sync 1 0.00 0.00 100.0% 0.0% + benchmark_runs 1 0.00 0.00 0.2% 0.0% + benchmark_pre_syncs 1 0.00 0.00 50.0% 0.0% + benchmark_post_syncs 1 0.00 0.00 50.0% 0.0% + MEASUREMENT OVERHEAD (instrumentation) 362.86 0.0% 0.0% + main_timing_calibration 32 346.48 10.83 95.5% 0.0% + flush_timing_buffer 32 15.18 0.47 4.2% 0.0% + (timer overhead subtracted from parent timers) 0.60 0.2% 0.0% + timing_overhead 32 0.60 0.02 0.2% 0.0% + UNATTRIBUTED (untracked time & timer overhead) 11,248.20 0.4% 0.4% + (untracked in python_benchmark_problems) 11,227.86 99.8% 0.4% + (untracked in python_client_execution) 13.36 0.1% 0.0% + (untracked in pre_solution) 6.21 0.1% 0.0% + (untracked in pre_problem) 0.69 0.0% 0.0% + (untracked in post_solution) 0.08 0.0% 0.0% + +TIMING BY TIMER NESTING (legacy view) +----------------------------------------------------------------------------------------------------------------------------- + Category Count Total (ms) Mean (ms) % of Parent % of Wall +----------------------------------------------------------------------------------------------------------------------------- + python_benchmark_problems 184 2,577,849.18 14,010.05 100.0% + ------------------------------------------------------------------------------------------------------------------------- + python_kernel_compilation 855 2,203,797.86 2,577.54 85.5% 85.5% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_src_co 855 1,007,817.04 1,178.73 45.7% 39.1% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_kernel_build_src_co.compile 263 1,004,724.33 3,820.24 99.7% 39.0% + python_kernel_build_src_co.unbundle 263 1,566.60 5.96 0.2% 0.1% + python_kernel_build_src_co.cache_check 855 1,172.50 1.37 0.1% 0.0% + python_kernel_build_src_co.cache_populate 263 214.63 0.82 0.0% 0.0% + (untracked) 98.65 0.0% 0.0% + python_kernel_build_src_co.setup 855 22.87 0.03 0.0% 0.0% + python_kernel_build_src_co.move 263 17.46 0.07 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_codegen 855 592,584.11 693.08 26.9% 23.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_assemble 855 227,868.45 266.51 10.3% 8.8% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_setup 855 131,638.45 153.96 6.0% 5.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_postprocess 855 106,816.89 124.93 4.8% 4.1% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_benchpost_lib_construction 855 74,773.69 87.45 70.0% 2.9% + python_benchpost_library_write 855 22,324.17 26.11 20.9% 0.9% + python_benchpost_naming 855 9,436.66 11.04 8.8% 0.4% + python_benchpost_client_config 855 249.07 0.29 0.2% 0.0% + (untracked) 33.30 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_validate 855 81,431.47 95.24 3.7% 3.2% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_setup 855 20,886.91 24.43 0.9% 0.8% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_co 855 18,912.53 22.12 0.9% 0.7% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_helpers 855 11,912.59 13.93 0.5% 0.5% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 3,929.43 0.2% 0.2% + ------------------------------------------------------------------------------------------------------------------------- + python_solution_generation 823 306,550.42 372.48 11.9% 11.9% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_forked_solutions 823 306,464.54 372.37 100.0% 11.9% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_custom_kernels 823 67.99 0.08 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 13.06 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_solgen_fork_permutations 823 4.83 0.01 0.0% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + python_write_solutions 855 14,069.93 16.46 0.5% 0.5% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_dump 855 13,390.16 15.66 95.2% 0.5% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_prepare 855 531.38 0.62 3.8% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_wsol_prepare_nocache 855 529.92 0.62 99.7% 0.0% + (untracked) 1.47 0.3% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 135.75 1.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_header 855 12.63 0.01 0.1% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + python_ductile_initial_sampling 8 13,760.99 1,720.12 0.5% 0.5% + ------------------------------------------------------------------------------------------------------------------------- + python_client_execution 32 13,377.67 418.05 0.5% 0.5% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + C++ Client Setup & Teardown 10,297.27 77.0% 0.4% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + hip_initialization 32 4,712.80 147.27 45.8% 0.2% + process_exit 32 3,467.40 108.36 33.7% 0.1% + main_problem_factory 32 595.29 18.60 5.8% 0.0% + factory_hip_get_device_properties 4 590.74 147.68 99.2% 0.0% + (untracked) 4.56 0.8% 0.0% + process_startup 32 468.60 14.64 4.6% 0.0% + main_teardown_locals 32 386.04 12.06 3.7% 0.0% + main_timing_calibration 32 346.48 10.83 3.4% 0.0% + data_init_setup 32 97.08 3.03 0.9% 0.0% + code_object_loading 32 96.45 3.01 0.9% 0.0% + library_loading 32 65.62 2.05 0.6% 0.0% + lazy_loading_init 32 45.39 1.42 0.4% 0.0% + main_parse_args 32 11.80 0.37 0.1% 0.0% + reporter_setup 32 1.84 0.06 0.0% 0.0% + python_client_write_run_script 32 1.63 0.05 0.0% 0.0% + listener_setup 32 0.52 0.02 0.0% 0.0% + solution_iterator_setup 32 0.33 0.01 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Data Preparation 2,518.74 18.8% 0.1% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + gpu_input_preparation 40 2,387.72 59.69 94.8% 0.1% + gpu_input_prep_init_cpu_inputs 39 1,309.51 33.58 54.8% 0.1% + gpu_input_prep_copy_valid_to_gpu 40 937.87 23.45 39.3% 0.0% + gpu_input_prep_copy_swizzled 4 135.91 33.98 5.7% 0.0% + gpu_input_prep_batched_inputs 40 2.63 0.07 0.1% 0.0% + gpu_input_prep_copy_inputs_d2d 40 0.98 0.02 0.0% 0.0% + gpu_input_prep_constant_inputs 39 0.64 0.02 0.0% 0.0% + gpu_input_prep_convert_to_inputs 40 0.11 0.00 0.0% 0.0% + (untracked) 0.07 0.0% 0.0% + gpu_input_prep_slow_path 40 0.00 0.00 0.0% 0.0% + pre_problem 40 122.83 3.07 4.9% 0.0% + cpu_data_init 1 119.66 119.66 97.4% 0.0% + cpu_reference_gemm 1 2.47 2.47 2.0% 0.0% + solve_cpu_slow 1 2.46 2.46 99.5% 0.0% + (untracked) 0.01 0.5% 0.0% + (untracked) 0.69 0.6% 0.0% + rotating_buffer_preparation 40 6.46 0.16 0.3% 0.0% + gpu_input_reset 1 1.73 1.73 0.1% 0.0% + gpu_input_reset_copy_valid_to_gpu 1 0.82 0.82 47.6% 0.0% + gpu_input_reset_copy_swizzled 1 0.81 0.81 46.9% 0.0% + gpu_input_reset_batched_inputs 1 0.06 0.06 3.7% 0.0% + gpu_input_reset_copy_inputs_d2d 1 0.03 0.03 1.7% 0.0% + gpu_input_reset_convert_to_inputs 1 0.00 0.00 0.0% 0.0% + gpu_input_reset_slow_path 1 0.00 0.00 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Kernel Execution 530.31 4.0% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + pre_solution 585 518.39 0.89 97.8% 0.0% + pre_solution_listener.DataInitialization 585 488.60 0.84 94.3% 0.0% + pre_solution_mx_init_data 130 369.08 2.84 75.5% 0.0% + pre_solution_mx_side_regenerate 8 369.03 46.13 100.0% 0.0% + (untracked) 0.02 0.0% 0.0% + pre_solution_mx_side_cache_hit 252 0.02 0.00 0.0% 0.0% + pre_solution_mx_copy_valid_to_gpu_no_regen 126 103.07 0.82 21.1% 0.0% + pre_solution_mx_cpu_sync_no_regen 126 8.60 0.07 1.8% 0.0% + pre_solution_mx_copy_valid_to_gpu_after_regen 4 3.29 0.82 0.7% 0.0% + pre_solution_mx_copy_inputs_d2d_no_regen 126 3.18 0.03 0.7% 0.0% + pre_solution_mx_cpu_sync_after_regen 4 1.19 0.30 0.2% 0.0% + pre_solution_mx_copy_inputs_d2d_after_regen 4 0.13 0.03 0.0% 0.0% + (untracked) 0.03 0.0% 0.0% + pre_solution_mx_check 585 0.03 0.00 0.0% 0.0% + pre_solution_listener.AllSolutionsIterator 585 10.23 0.02 2.0% 0.0% + (untracked) 6.21 1.2% 0.0% + pre_solution_listener.ReferenceValidator 585 4.64 0.01 0.9% 0.0% + cpu_reference_gemm_per_solution_inputs_changed 1 4.62 4.62 99.5% 0.0% + solve_cpu_slow_per_solution_inputs_changed 1 4.62 4.62 100.0% 0.0% + (untracked) 0.02 0.5% 0.0% + pre_solution_listener.BenchmarkTimer 585 3.99 0.01 0.8% 0.0% + pre_solution_listener.ProgressListener 585 2.60 0.00 0.5% 0.0% + pre_solution_listener.MetaResultReporter 585 2.00 0.00 0.4% 0.0% + pre_solution_listener.SynchronizerValidator 585 0.10 0.00 0.0% 0.0% + pre_solution_listener.HardwareMonitorListener 585 0.01 0.00 0.0% 0.0% + post_solution 585 7.33 0.01 1.4% 0.0% + post_solution_log 585 5.20 0.01 70.9% 0.0% + post_solution_reporting 585 1.75 0.00 23.9% 0.0% + post_solution_perf_calc 585 0.25 0.00 3.5% 0.0% + (untracked) 0.08 1.1% 0.0% + post_solution_result_file 585 0.01 0.00 0.2% 0.0% + post_solution_validation 585 0.01 0.00 0.2% 0.0% + post_solution_sol_advance 585 0.01 0.00 0.2% 0.0% + post_solution_perf_reset 585 0.01 0.00 0.2% 0.0% + run_current_solution_check 585 2.03 0.00 0.4% 0.0% + post_problem 40 1.66 0.04 0.3% 0.0% + finalize_report 32 0.35 0.01 0.1% 0.0% + problem_report_index 40 0.15 0.00 0.0% 0.0% + solution_selection 585 0.14 0.00 0.0% 0.0% + kernel_solving 1 0.08 0.08 0.0% 0.0% + warmup_runs 2 0.08 0.04 0.0% 0.0% + problem_teardown 40 0.06 0.00 0.0% 0.0% + solution_teardown 585 0.02 0.00 0.0% 0.0% + warmup_events_setup 1 0.02 0.02 0.0% 0.0% + run_iteration_teardown 1 0.01 0.01 0.0% 0.0% + benchmark_runs 1 0.00 0.00 0.0% 0.0% + benchmark_pre_syncs 1 0.00 0.00 31.1% 0.0% + benchmark_post_syncs 1 0.00 0.00 31.1% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Timing Overhead 15.78 0.1% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + flush_timing_buffer 32 15.18 0.47 96.2% 0.0% + timing_overhead 32 0.60 0.02 3.8% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked C++ overhead) 13.36 0.1% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Reference Computation 2.22 0.0% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + validate_warmups 1 2.22 2.22 100.0% 0.0% + validate_warmups_listener.ReferenceValidator 1 1.26 1.26 56.8% 0.0% + validate_result_buffer_alloc 1 0.94 0.94 74.6% 0.0% + validate_gpu_readback 1 0.30 0.30 24.1% 0.0% + validate_element_comparison 1 0.01 0.01 0.7% 0.0% + validate_inputs_copy 1 0.00 0.00 0.1% 0.0% + validate_report 1 0.00 0.00 0.0% 0.0% + validate_setup 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.SynchronizerValidator 1 0.94 0.94 42.6% 0.0% + validate_synchronizer_check 1 0.94 0.94 100.0% 0.0% + validate_synchronizer_readback 1 0.78 0.78 82.5% 0.0% + validate_synchronizer_scan 1 0.16 0.16 17.4% 0.0% + validate_warmups_listener.BenchmarkTimer 1 0.01 0.01 0.2% 0.0% + validate_gpu_sync 1 0.00 0.00 85.9% 0.0% + validate_warmups_listener.MetaResultReporter 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.HardwareMonitorListener 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.AllSolutionsIterator 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.ProgressListener 1 0.00 0.00 0.0% 0.0% + validate_warmups_listener.DataInitialization 1 0.00 0.00 0.0% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + (untracked) 11,227.86 0.4% 0.4% + ------------------------------------------------------------------------------------------------------------------------- + python_solution_indexing 855 8,279.91 9.68 0.3% 0.3% + ------------------------------------------------------------------------------------------------------------------------- + python_ductile_generate_solutions 32 4,536.07 141.75 0.2% 0.2% + ------------------------------------------------------------------------------------------------------------------------- + python_ductile_ga_between_evaluations 8 1,366.26 170.78 0.1% 0.1% + ------------------------------------------------------------------------------------------------------------------------- + python_write_cache 855 830.02 0.97 0.0% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + python_ductile_setup 8 30.08 3.76 0.0% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + python_ductile_clear_artifacts 24 16.81 0.70 0.0% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + python_ductile_read_scores 32 4.29 0.13 0.0% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + python_cache_check 831 0.67 0.00 0.0% 0.0% + ------------------------------------------------------------------------------------------------------------------------- + python_ductile_verification 8 0.33 0.04 0.0% 0.0% + + Overlapping Totals (wrap other timers; not summed into the tree) + --------------------------------------------------------------------------------------------------------------------------- + python_client_subprocess 32 13,365.64 417.68 0.5% + main_total 32 9,027.25 282.10 0.4% + problem_total 40 3,051.85 76.30 0.1% + solution_total 585 532.32 0.91 0.0% + run_iteration_total 1 4.14 4.14 0.0% + validate_total 1 1.25 1.25 0.0% + validate_check_results_typed 1 1.25 1.25 0.0% + +----------------------------------------------------------------------------------------------------------------------------- +HIERARCHY VALIDATION: OK (all child phases fit within their parents) + +TIME BREAKDOWN (visual, % of wall clock) +----------------------------------------------------------------------------------------------------------------------------- + TOTAL RUNTIME 100.0% ################################################## + BUILD: solution generation & compile 98.1% ################################################# + Solution generation & build (Python) 97.4% ################################################ + python_kernel_compilation 85.5% ########################################## + python_kernel_build_src_co 39.1% ################### + python_kernel_build_src_co.compile 39.0% ################### + python_kernel_build_src_co.unbundle 0.1% + python_kernel_build_src_co.cache_check 0.0% + python_kernel_build_src_co.cache_populate 0.0% + python_kernel_build_src_co.setup 0.0% + python_kernel_build_src_co.move 0.0% + (untracked) 0.0% + python_kernel_codegen 23.0% ########### + python_kernel_write_assemble 8.8% #### + python_kernel_bench_setup 5.1% ## + python_kernel_bench_postprocess 4.1% ## + python_benchpost_lib_construction 2.9% # + python_benchpost_library_write 0.9% + python_benchpost_naming 0.4% + python_benchpost_client_config 0.0% + (untracked) 0.0% + python_kernel_validate 3.2% # + python_kernel_setup 0.8% + python_kernel_build_co 0.7% + python_kernel_write_helpers 0.5% + (untracked) 0.2% + python_solution_generation 11.9% ##### + python_solgen_forked_solutions 11.9% ##### + python_solgen_custom_kernels 0.0% + python_solgen_fork_permutations 0.0% + (untracked) 0.0% + Ductile GA search (Python) 0.8% + python_ductile_initial_sampling 0.5% + python_ductile_generate_solutions 0.2% + python_ductile_ga_between_evaluations 0.1% + python_ductile_setup 0.0% + python_ductile_clear_artifacts 0.0% + python_ductile_read_scores 0.0% + python_ductile_verification 0.0% + SHARED: cache & solution files 0.9% + python_write_solutions 0.5% + python_wsol_dump 0.5% + python_wsol_prepare 0.0% + python_wsol_prepare_nocache 0.0% + (untracked) 0.0% + python_wsol_header 0.0% + (untracked) 0.0% + python_solution_indexing 0.3% + python_write_cache 0.0% + python_cache_check 0.0% + RUN: benchmark client 0.5% + Client launch (startup, setup & teardown) 0.4% + hip_initialization 0.2% + process_exit 0.1% + main_problem_factory 0.0% + factory_hip_get_device_properties 0.0% + (untracked) 0.0% + process_startup 0.0% + main_teardown_locals 0.0% + data_init_setup 0.0% + code_object_loading 0.0% + library_loading 0.0% + lazy_loading_init 0.0% + main_parse_args 0.0% + reporter_setup 0.0% + python_client_write_run_script 0.0% + listener_setup 0.0% + solution_iterator_setup 0.0% + Input data preparation 0.1% + gpu_input_preparation 0.1% + gpu_input_prep_init_cpu_inputs 0.1% + gpu_input_prep_copy_valid_to_gpu 0.0% + gpu_input_prep_copy_swizzled 0.0% + gpu_input_prep_batched_inputs 0.0% + gpu_input_prep_copy_inputs_d2d 0.0% + gpu_input_prep_constant_inputs 0.0% + gpu_input_prep_convert_to_inputs 0.0% + gpu_input_prep_slow_path 0.0% + (untracked) 0.0% + pre_solution_listener.DataInitialization 0.0% + pre_solution_mx_init_data 0.0% + pre_solution_mx_side_regenerate 0.0% + pre_solution_mx_side_cache_hit 0.0% + (untracked) 0.0% + pre_solution_mx_copy_valid_to_gpu_no_regen 0.0% + pre_solution_mx_cpu_sync_no_regen 0.0% + pre_solution_mx_copy_valid_to_gpu_after_regen 0.0% + pre_solution_mx_copy_inputs_d2d_no_regen 0.0% + pre_solution_mx_cpu_sync_after_regen 0.0% + pre_solution_mx_copy_inputs_d2d_after_regen 0.0% + pre_solution_mx_check 0.0% + (untracked) 0.0% + cpu_data_init 0.0% + rotating_buffer_preparation 0.0% + gpu_input_reset 0.0% + gpu_input_reset_copy_valid_to_gpu 0.0% + gpu_input_reset_copy_swizzled 0.0% + gpu_input_reset_batched_inputs 0.0% + gpu_input_reset_copy_inputs_d2d 0.0% + gpu_input_reset_convert_to_inputs 0.0% + gpu_input_reset_slow_path 0.0% + Per-solution bookkeeping & reporting 0.0% + pre_solution_listener.AllSolutionsIterator 0.0% + post_solution_log 0.0% + pre_solution_listener.BenchmarkTimer 0.0% + pre_solution_listener.ProgressListener 0.0% + run_current_solution_check 0.0% + pre_solution_listener.MetaResultReporter 0.0% + post_solution_reporting 0.0% + post_problem 0.0% + finalize_report 0.0% + post_solution_perf_calc 0.0% + problem_report_index 0.0% + solution_selection 0.0% + pre_solution_listener.SynchronizerValidator 0.0% + problem_teardown 0.0% + solution_teardown 0.0% + post_solution_result_file 0.0% + post_solution_validation 0.0% + pre_solution_listener.HardwareMonitorListener 0.0% + post_solution_sol_advance 0.0% + post_solution_perf_reset 0.0% + run_iteration_teardown 0.0% + Reference GEMM (CPU) 0.0% + pre_solution_listener.ReferenceValidator 0.0% + cpu_reference_gemm_per_solution_inputs_changed 0.0% + solve_cpu_slow_per_solution_inputs_changed 0.0% + (untracked) 0.0% + cpu_reference_gemm 0.0% + solve_cpu_slow 0.0% + (untracked) 0.0% + Validation 0.0% + validate_warmups_listener.ReferenceValidator 0.0% + validate_result_buffer_alloc 0.0% + validate_gpu_readback 0.0% + validate_element_comparison 0.0% + validate_inputs_copy 0.0% + validate_report 0.0% + validate_setup 0.0% + validate_warmups_listener.SynchronizerValidator 0.0% + validate_synchronizer_check 0.0% + validate_synchronizer_readback 0.0% + validate_synchronizer_scan 0.0% + validate_warmups_listener.MetaResultReporter 0.0% + validate_warmups_listener.HardwareMonitorListener 0.0% + validate_warmups_listener.AllSolutionsIterator 0.0% + validate_warmups_listener.ProgressListener 0.0% + validate_warmups_listener.DataInitialization 0.0% + Kernel launch & execution 0.0% + kernel_solving 0.0% + warmup_runs 0.0% + warmup_events_setup 0.0% + validate_warmups_listener.BenchmarkTimer 0.0% + validate_gpu_sync 0.0% + benchmark_runs 0.0% + benchmark_pre_syncs 0.0% + benchmark_post_syncs 0.0% + MEASUREMENT OVERHEAD (instrumentation) 0.0% + main_timing_calibration 0.0% + flush_timing_buffer 0.0% + (timer overhead subtracted from parent timers) 0.0% + timing_overhead 0.0% + UNATTRIBUTED (untracked time & timer overhead) 0.4% + (untracked in python_benchmark_problems) 0.4% + (untracked in python_client_execution) 0.0% + (untracked in pre_solution) 0.0% + (untracked in pre_problem) 0.0% + (untracked in post_solution) 0.0% + +KEY INSIGHTS (% of wall clock) +----------------------------------------------------------------------------------------------------------------------------- + Solution generation & build (Python) 2510.35s (97.4%) + SHARED: cache & solution files 23.18s (0.9%) + Ductile GA search (Python) 19.71s (0.8%) + Client launch (startup, setup & teardown) 9.95s (0.4%) + Input data preparation 3.00s (0.1%) + MEASUREMENT OVERHEAD (instrumentation) 0.36s (0.0%) + Per-solution bookkeeping & reporting 0.03s (0.0%) + Reference GEMM (CPU) 0.01s (0.0%) + Validation 0.00s (0.0%) + Kernel launch & execution 0.00s (0.0%) + UNATTRIBUTED (untracked time & timer overhead) 11.25s (0.4%) + + +TOP 10 SLOWEST PROBLEMS (by CPU reference time) +----------------------------------------------------------------------------------------------------------------------------- + M N K Batch TypeA TypeD CPU Ref (ms) GPU (ms) + --------------------------------------------------------------------------------------------------------------------------- + 2048 256 128 16 Float8 BFloat16 2.47 0.00 + diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_rerun/warm_cache_check.sh b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_rerun/warm_cache_check.sh new file mode 100644 index 000000000000..2111c60f97bc --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_rerun/warm_cache_check.sh @@ -0,0 +1,28 @@ +#!/bin/bash +# Copyright Advanced Micro Devices, Inc., or its affiliates. +# SPDX-License-Identifier: MIT +# +# Does the LibraryClient phase hit the helper-kernel cache when ~/.tensile is NOT cleared between runs? +# Clears ~/.tensile once, then runs the same config twice (cache cold, then warm), saving the analysis +# of each. Usage: bash warm_cache_check.sh gemm/gfx950/subtile_mxfp8_tail_loop_smoke +TL=/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite +CFG=$1 +NAME=$(basename $CFG) +OUT=$TL/timing_runs/full_runs_gfx950/per_config_rerun/warm_cache_check/$NAME +ART=/tmp/tl_split_gfx950/artifacts +CLIENT=/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client +mkdir -p $OUT +cd $TL || exit 1 +export OMP_NUM_THREADS=25 OPENBLAS_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=0 PYTHONUNBUFFERED=1 +TO="--cxx-compiler,/usr/bin/amdclang++,--global-parameters,TimingInstrumentation=True" +RID="tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/$CFG.yaml]" +rm -rf /home/alvasile/.tensile /home/alvasile/.tensilelite +for pass in cold warm; do + taskset -c 0-24 pytest "$RID" --use-cache --artifact-dir $ART --prebuilt-client=$CLIENT \ + -s -v -n 0 --timeout=5400 -m "common and not skip-gfx94X" \ + "--global-parameters=LibraryFormat='msgpack'" "--tensile-options=$TO" -p no:warnings -p no:cacheprovider \ + > $OUT/$pass.log 2>&1 + echo "$pass rc=$?" >> $OUT/times.txt + python scripts/analyze_timing.py $OUT/$pass.log > $OUT/${pass}_analysis.txt 2>&1 + ls /home/alvasile/.tensile/helper_cache >> $OUT/times.txt 2>&1 +done diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_rerun_2/AGGREGATE_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_rerun_2/AGGREGATE_analysis.txt new file mode 100644 index 000000000000..fba65d8aa12a --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_rerun_2/AGGREGATE_analysis.txt @@ -0,0 +1,754 @@ +================================================================================================================================= +TIMING ANALYSIS SUMMARY +================================================================================================================================= + +TIMING BY PURPOSE +--------------------------------------------------------------------------------------------------------------------------------- + Category Count Total (ms) Mean (ms) % of Parent % of Wall +--------------------------------------------------------------------------------------------------------------------------------- + TOTAL RUNTIME 2,568,261.07 100.0% + RUN: benchmark client 2,330,795.57 90.8% 90.8% + Reference GEMM (CPU) 1,080,469.12 46.4% 42.1% + cpu_reference_gemm 28,596 928,712.97 32.48 86.0% 36.2% + solve_cpu_slow 1,947 727,756.12 373.78 78.4% 28.3% + solve_cpu_fast 26,656 200,931.33 7.54 21.6% 7.8% + (untracked) 25.51 0.0% 0.0% + pre_solution_listener.ReferenceValidator 2,355,016 151,756.15 0.06 14.0% 5.9% + cpu_reference_gemm_per_solution_inputs_changed 1,146 151,668.42 132.35 99.9% 5.9% + solve_cpu_fast_per_solution_inputs_changed 1,127 138,501.63 122.89 91.3% 5.4% + solve_cpu_slow_per_solution_inputs_changed 19 13,166.07 692.95 8.7% 0.5% + (untracked) 0.72 0.0% 0.0% + pre_solution_ref_skipped 2,353,182 49.50 0.00 0.0% 0.0% + (untracked) 38.23 0.0% 0.0% + Input data preparation 502,878.60 21.6% 19.6% + gpu_input_reset 2,106,835 306,247.35 0.15 60.9% 11.9% + gpu_input_reset_copy_valid_to_gpu 719,180 177,839.34 0.25 58.1% 6.9% + gpu_input_reset_batched_inputs 1,473,329 95,916.03 0.07 31.3% 3.7% + gpu_input_reset_copy_inputs_d2d 1,473,329 26,309.71 0.02 8.6% 1.0% + gpu_input_reset_copy_swizzled 15,719 4,006.86 0.25 1.3% 0.2% + gpu_input_reset_fast_reset_output 633,454 1,917.65 0.00 0.6% 0.1% + gpu_input_reset_convert_to_inputs 1,473,329 139.58 0.00 0.0% 0.0% + gpu_input_reset_slow_path 1,473,329 34.85 0.00 0.0% 0.0% + gpu_input_reset_rotating_init 2 0.02 0.01 0.0% 0.0% + gpu_input_reset_constant_inputs 4 0.01 0.00 0.0% 0.0% + gpu_input_reset_init_cpu_inputs 4 0.00 0.00 0.0% 0.0% + (untracked) 83.31 0.0% 0.0% + cpu_data_init 28,596 106,829.68 3.74 21.2% 4.2% + pre_solution_listener.DataInitialization 2,355,016 49,481.67 0.02 9.8% 1.9% + pre_solution_mx_init_data 16,038 44,073.15 2.75 89.1% 1.7% + pre_solution_mx_side_regenerate 2,298 44,069.34 19.18 100.0% 1.7% + pre_solution_mx_side_cache_hit 29,778 2.60 0.00 0.0% 0.0% + (untracked) 1.21 0.0% 0.0% + pre_solution_mx_copy_valid_to_gpu_no_regen 14,889 3,124.70 0.21 6.3% 0.1% + pre_solution_mx_copy_valid_to_gpu_after_regen 1,149 1,143.05 0.99 2.3% 0.0% + pre_solution_mx_cpu_sync_after_regen 1,149 438.52 0.38 0.9% 0.0% + pre_solution_mx_copy_inputs_d2d_no_regen 14,889 307.63 0.02 0.6% 0.0% + pre_solution_mx_cpu_sync_no_regen 14,889 307.33 0.02 0.6% 0.0% + pre_solution_mx_check 2,355,016 49.85 0.00 0.1% 0.0% + pre_solution_mx_copy_inputs_d2d_after_regen 1,149 22.97 0.02 0.0% 0.0% + (untracked) 14.47 0.0% 0.0% + gpu_input_preparation 28,641 39,596.08 1.38 7.9% 1.5% + gpu_input_prep_copy_valid_to_gpu 11,446 22,935.66 2.00 57.9% 0.9% + gpu_input_prep_copy_swizzled 1,151 10,766.76 9.35 27.2% 0.4% + gpu_input_prep_init_cpu_inputs 41 4,029.28 98.28 10.2% 0.2% + gpu_input_prep_batched_inputs 23,229 1,410.04 0.06 3.6% 0.1% + gpu_input_prep_copy_inputs_d2d 23,229 411.51 0.02 1.0% 0.0% + gpu_input_prep_fast_reset_output 5,403 22.52 0.00 0.1% 0.0% + gpu_input_prep_convert_to_inputs 23,229 9.26 0.00 0.0% 0.0% + gpu_input_prep_slow_path 23,229 1.96 0.00 0.0% 0.0% + gpu_input_prep_constant_inputs 43 0.73 0.02 0.0% 0.0% + gpu_input_prep_rotating_init 1 0.01 0.01 0.0% 0.0% + (untracked) 8.37 0.0% 0.0% + rotating_buffer_preparation 28,641 723.82 0.03 0.1% 0.0% + Validation 323,888.88 13.9% 12.6% + validate_warmups_listener.SynchronizerValidator 2,106,827 162,734.88 0.08 50.2% 6.3% + validate_synchronizer_check 2,106,827 162,692.31 0.08 100.0% 6.3% + validate_synchronizer_scan 608,873 106,747.41 0.18 65.6% 4.2% + validate_synchronizer_readback 608,873 55,816.84 0.09 34.3% 2.2% + validate_synchronizer_inactive 1,497,978 27.90 0.00 0.0% 0.0% + (untracked) 100.16 0.1% 0.0% + (untracked) 42.58 0.0% 0.0% + validate_warmups_listener.ReferenceValidator 2,106,827 160,728.73 0.08 49.6% 6.3% + validate_gpu_readback 1,907,339 91,856.16 0.05 57.1% 3.6% + validate_element_comparison 1,907,339 59,265.75 0.03 36.9% 2.3% + validate_result_buffer_alloc 1,907,339 9,020.90 0.00 5.6% 0.4% + validate_inputs_copy 1,905,787 194.93 0.00 0.1% 0.0% + validate_report 1,907,339 43.63 0.00 0.0% 0.0% + validate_setup 1,905,883 43.37 0.00 0.0% 0.0% + (untracked) 303.99 0.2% 0.0% + validate_warmups_listener.MetaResultReporter 2,106,827 149.34 0.00 0.0% 0.0% + validate_warmups_listener.HardwareMonitorListener 2,106,827 74.67 0.00 0.0% 0.0% + validate_warmups_listener.ProgressListener 2,106,827 70.80 0.00 0.0% 0.0% + validate_warmups_listener.AllSolutionsIterator 2,106,827 66.34 0.00 0.0% 0.0% + validate_warmups_listener.DataInitialization 2,106,827 64.11 0.00 0.0% 0.0% + Client launch (startup, setup & teardown) 284,745.47 12.2% 11.1% + hip_initialization 855 124,448.44 145.55 43.7% 4.8% + process_exit 855 64,634.38 75.60 22.7% 2.5% + main_problem_factory 855 29,154.81 34.10 10.2% 1.1% + factory_hip_get_device_properties 192 28,922.86 150.64 99.2% 1.1% + (untracked) 231.95 0.8% 0.0% + data_init_setup 855 18,185.96 21.27 6.4% 0.7% + main_teardown_locals 855 13,182.66 15.42 4.6% 0.5% + process_startup 855 12,710.53 14.87 4.5% 0.5% + code_object_loading 855 11,001.27 12.87 3.9% 0.4% + library_loading 855 9,940.49 11.63 3.5% 0.4% + lazy_loading_init 855 924.59 1.08 0.3% 0.0% + main_parse_args 855 410.00 0.48 0.1% 0.0% + reporter_setup 855 68.52 0.08 0.0% 0.0% + python_client_write_run_script 855 57.11 0.07 0.0% 0.0% + listener_setup 855 15.77 0.02 0.0% 0.0% + solution_iterator_setup 855 10.94 0.01 0.0% 0.0% + Per-solution bookkeeping & reporting 68,333.60 2.9% 2.7% + post_solution_log 2,355,016 28,760.07 0.01 42.1% 1.1% + pre_solution_listener.BenchmarkTimer 2,355,016 11,859.71 0.01 17.4% 0.5% + run_current_solution_check 2,355,016 8,420.00 0.00 12.3% 0.3% + post_solution_reporting 2,355,016 5,582.67 0.00 8.2% 0.2% + pre_solution_listener.ProgressListener 2,355,016 3,134.22 0.00 4.6% 0.1% + pre_solution_listener.AllSolutionsIterator 2,355,016 2,647.18 0.00 3.9% 0.1% + post_problem 28,641 1,945.83 0.07 2.8% 0.1% + run_iteration_teardown 2,106,835 1,677.91 0.00 2.5% 0.1% + pre_solution_listener.MetaResultReporter 2,355,016 1,159.39 0.00 1.7% 0.0% + post_solution_validation 2,355,016 740.26 0.00 1.1% 0.0% + post_solution_perf_calc 2,355,016 652.16 0.00 1.0% 0.0% + solution_selection 2,355,016 571.95 0.00 0.8% 0.0% + finalize_report 855 507.49 0.59 0.7% 0.0% + pre_solution_listener.SynchronizerValidator 2,355,016 358.33 0.00 0.5% 0.0% + solution_teardown 2,355,016 77.56 0.00 0.1% 0.0% + post_solution_result_file 2,355,016 57.99 0.00 0.1% 0.0% + post_solution_perf_reset 2,355,016 52.77 0.00 0.1% 0.0% + post_solution_sol_advance 2,355,016 50.05 0.00 0.1% 0.0% + pre_solution_listener.HardwareMonitorListener 2,355,016 47.08 0.00 0.1% 0.0% + problem_report_index 28,641 26.02 0.00 0.0% 0.0% + problem_teardown 28,641 4.94 0.00 0.0% 0.0% + Kernel launch & execution 40,886.54 1.8% 1.6% + warmup_runs 4,213,654 25,917.98 0.01 63.4% 1.0% + validate_warmups_listener.BenchmarkTimer 2,106,827 6,922.29 0.00 16.9% 0.3% + validate_gpu_sync 2,106,827 6,867.77 0.00 99.2% 0.3% + (untracked) 54.52 0.8% 0.0% + kernel_solving 2,106,835 6,730.10 0.00 16.5% 0.3% + warmup_events_setup 2,106,835 970.97 0.00 2.4% 0.0% + benchmark_runs 2,106,835 345.20 0.00 0.8% 0.0% + benchmark_validate_enqueues 319 181.62 0.57 52.6% 0.0% + validate_enqueues_listener.BenchmarkTimer 319 179.53 0.56 98.8% 0.0% + benchmark_sleep 319 179.48 0.56 100.0% 0.0% + benchmark_event_elapsed_time 8 0.01 0.00 0.0% 0.0% + benchmark_event_destroy 8 0.01 0.00 0.0% 0.0% + (untracked) 0.03 0.0% 0.0% + validate_enqueues_listener.ProgressListener 319 1.81 0.01 1.0% 0.0% + validate_enqueues_listener.MetaResultReporter 319 0.02 0.00 0.0% 0.0% + validate_enqueues_listener.HardwareMonitorListener 319 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.AllSolutionsIterator 319 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.DataInitialization 319 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.SynchronizerValidator 319 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.ReferenceValidator 319 0.01 0.00 0.0% 0.0% + (untracked) 0.21 0.1% 0.0% + benchmark_pre_syncs 2,106,835 56.09 0.00 16.2% 0.0% + benchmark_post_syncs 2,106,835 54.72 0.00 15.9% 0.0% + gpu_kernel_execution 319 39.52 0.12 11.4% 0.0% + benchmark_events_setup 319 0.06 0.00 0.0% 0.0% + benchmark_events_teardown 319 0.03 0.00 0.0% 0.0% + (untracked) 13.15 3.8% 0.0% + Library client (Python) 23,840.65 1.0% 0.9% + python_library_client_create_library 10 20,540.37 2,054.04 86.2% 0.8% + python_library_client_execution 10 3,161.37 316.14 13.3% 0.1% + python_library_client_subprocess 10 3,155.15 315.52 99.8% 0.1% + python_library_client_write_run_script 10 0.81 0.08 0.0% 0.0% + (untracked) 5.41 0.2% 0.0% + python_library_client_write_configs 10 120.68 12.07 0.5% 0.0% + python_library_client_setup 10 18.23 1.82 0.1% 0.0% + Library logic (Python) 5,752.70 0.2% 0.2% + python_logic_write 26 5,424.03 208.62 94.3% 0.2% + python_logic_parse_solutions 26 259.52 9.98 4.5% 0.0% + python_logic_analyze 26 69.15 2.66 1.2% 0.0% + MEASUREMENT OVERHEAD (instrumentation) 172,815.65 6.7% 6.7% + flush_timing_buffer 865 152,892.88 176.75 88.5% 6.0% + main_timing_calibration 855 10,096.95 11.81 5.8% 0.4% + timing_overhead 855 5,082.90 5.94 2.9% 0.2% + (timer overhead subtracted from parent timers) 4,742.92 2.7% 0.2% + BUILD: solution generation & compile 63,999.91 2.5% 2.5% + Solution generation & build (Python) 44,036.26 68.8% 1.7% + python_kernel_compilation 32 44,036.26 1,376.13 100.0% 1.7% + python_kernel_build_src_co 32 20,333.49 635.42 46.2% 0.8% + python_kernel_build_src_co.compile 4 20,243.96 5,060.99 99.6% 0.8% + python_kernel_build_src_co.cache_check 32 53.80 1.68 0.3% 0.0% + python_kernel_build_src_co.unbundle 4 27.97 6.99 0.1% 0.0% + python_kernel_build_src_co.cache_populate 4 3.20 0.80 0.0% 0.0% + python_kernel_build_src_co.setup 32 0.84 0.03 0.0% 0.0% + python_kernel_build_src_co.move 4 0.26 0.07 0.0% 0.0% + (untracked) 3.45 0.0% 0.0% + python_kernel_codegen 32 16,254.60 507.96 36.9% 0.6% + python_kernel_write_assemble 32 2,214.46 69.20 5.0% 0.1% + python_kernel_bench_setup 32 1,786.93 55.84 4.1% 0.1% + python_kernel_validate 32 1,686.90 52.72 3.8% 0.1% + python_kernel_write_helpers 32 545.76 17.05 1.2% 0.0% + python_kernel_build_co 32 526.58 16.46 1.2% 0.0% + python_kernel_bench_postprocess 32 495.87 15.50 1.1% 0.0% + python_benchpost_lib_construction 32 355.66 11.11 71.7% 0.0% + python_benchpost_library_write 32 84.30 2.63 17.0% 0.0% + python_benchpost_naming 32 50.98 1.59 10.3% 0.0% + python_benchpost_client_config 32 3.91 0.12 0.8% 0.0% + (untracked) 1.03 0.2% 0.0% + python_kernel_setup 32 137.71 4.30 0.3% 0.0% + (untracked) 53.96 0.1% 0.0% + Ductile GA search (Python) 19,963.65 31.2% 0.8% + python_ductile_initial_sampling 8 13,753.39 1,719.17 68.9% 0.5% + python_ductile_generate_solutions 32 4,760.25 148.76 23.8% 0.2% + python_ductile_ga_between_evaluations 8 1,387.06 173.38 6.9% 0.1% + python_ductile_setup 8 32.30 4.04 0.2% 0.0% + python_ductile_clear_artifacts 32 26.20 0.82 0.1% 0.0% + python_ductile_read_scores 32 4.15 0.13 0.0% 0.0% + python_ductile_verification 8 0.30 0.04 0.0% 0.0% + SHARED: cache & solution files 1,580.00 0.1% 0.1% + python_cache_check 831 1,280.29 1.54 81.0% 0.0% + python_write_solutions 855 205.93 0.24 13.0% 0.0% + python_wsol_dump 32 66.32 2.07 32.2% 0.0% + python_wsol_header 855 65.76 0.08 31.9% 0.0% + python_wsol_prepare 855 60.96 0.07 29.6% 0.0% + python_wsol_prepare_cache 823 58.04 0.07 95.2% 0.0% + python_wsol_prepare_nocache 32 0.90 0.03 1.5% 0.0% + (untracked) 2.02 3.3% 0.0% + (untracked) 12.89 6.3% 0.0% + python_write_cache 32 58.71 1.83 3.7% 0.0% + python_solution_indexing 32 35.07 1.10 2.2% 0.0% + UNATTRIBUTED (untracked time & timer overhead) -930.06 -0.0% -0.0% + (untracked in python_benchmark_problems) 2,407.19 0.0% 0.1% + (untracked in validate_warmups) 560.19 0.0% 0.0% + (untracked in pre_solution) 472.96 0.0% 0.0% + (untracked in pre_problem) 255.77 0.0% 0.0% + (untracked in python_library_logic) 7.63 0.0% 0.0% + (untracked in python_client_writer) 7.58 0.0% 0.0% + (children exceed post_solution) -168.12 0.0% -0.0% + (children exceed python_client_execution) -4,473.27 0.0% -0.2% + +TIMING BY TIMER NESTING (legacy view) +--------------------------------------------------------------------------------------------------------------------------------- + Category Count Total (ms) Mean (ms) % of Parent % of Wall +--------------------------------------------------------------------------------------------------------------------------------- + python_benchmark_problems 184 2,533,909.58 13,771.25 98.7% + ----------------------------------------------------------------------------------------------------------------------------- + python_client_execution 855 2,465,922.48 2,884.12 97.3% 96.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Data Preparation 1,382,365.67 56.1% 53.8% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + pre_problem 28,641 1,035,798.42 36.16 74.9% 40.3% + cpu_reference_gemm 28,596 928,712.97 32.48 89.7% 36.2% + solve_cpu_slow 1,947 727,756.12 373.78 78.4% 28.3% + solve_cpu_fast 26,656 200,931.33 7.54 21.6% 7.8% + (untracked) 25.51 0.0% 0.0% + cpu_data_init 28,596 106,829.68 3.74 10.3% 4.2% + (untracked) 255.77 0.0% 0.0% + gpu_input_reset 2,106,835 306,247.35 0.15 22.2% 11.9% + gpu_input_reset_copy_valid_to_gpu 719,180 177,839.34 0.25 58.1% 6.9% + gpu_input_reset_batched_inputs 1,473,329 95,916.03 0.07 31.3% 3.7% + gpu_input_reset_copy_inputs_d2d 1,473,329 26,309.71 0.02 8.6% 1.0% + gpu_input_reset_copy_swizzled 15,719 4,006.86 0.25 1.3% 0.2% + gpu_input_reset_fast_reset_output 633,454 1,917.65 0.00 0.6% 0.1% + gpu_input_reset_convert_to_inputs 1,473,329 139.58 0.00 0.0% 0.0% + (untracked) 83.31 0.0% 0.0% + gpu_input_reset_slow_path 1,473,329 34.85 0.00 0.0% 0.0% + gpu_input_reset_rotating_init 2 0.02 0.01 0.0% 0.0% + gpu_input_reset_constant_inputs 4 0.01 0.00 0.0% 0.0% + gpu_input_reset_init_cpu_inputs 4 0.00 0.00 0.0% 0.0% + gpu_input_preparation 28,641 39,596.08 1.38 2.9% 1.5% + gpu_input_prep_copy_valid_to_gpu 11,446 22,935.66 2.00 57.9% 0.9% + gpu_input_prep_copy_swizzled 1,151 10,766.76 9.35 27.2% 0.4% + gpu_input_prep_init_cpu_inputs 41 4,029.28 98.28 10.2% 0.2% + gpu_input_prep_batched_inputs 23,229 1,410.04 0.06 3.6% 0.1% + gpu_input_prep_copy_inputs_d2d 23,229 411.51 0.02 1.0% 0.0% + gpu_input_prep_fast_reset_output 5,403 22.52 0.00 0.1% 0.0% + gpu_input_prep_convert_to_inputs 23,229 9.26 0.00 0.0% 0.0% + (untracked) 8.37 0.0% 0.0% + gpu_input_prep_slow_path 23,229 1.96 0.00 0.0% 0.0% + gpu_input_prep_constant_inputs 43 0.73 0.02 0.0% 0.0% + gpu_input_prep_rotating_init 1 0.01 0.01 0.0% 0.0% + rotating_buffer_preparation 28,641 723.82 0.03 0.1% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Reference Computation 331,371.37 13.4% 12.9% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + validate_warmups 2,106,827 331,371.37 0.16 100.0% 12.9% + validate_warmups_listener.SynchronizerValidator 2,106,827 162,734.88 0.08 49.1% 6.3% + validate_synchronizer_check 2,106,827 162,692.31 0.08 100.0% 6.3% + validate_synchronizer_scan 608,873 106,747.41 0.18 65.6% 4.2% + validate_synchronizer_readback 608,873 55,816.84 0.09 34.3% 2.2% + (untracked) 100.16 0.1% 0.0% + validate_synchronizer_inactive 1,497,978 27.90 0.00 0.0% 0.0% + (untracked) 42.58 0.0% 0.0% + validate_warmups_listener.ReferenceValidator 2,106,827 160,728.73 0.08 48.5% 6.3% + validate_gpu_readback 1,907,339 91,856.16 0.05 57.1% 3.6% + validate_element_comparison 1,907,339 59,265.75 0.03 36.9% 2.3% + validate_result_buffer_alloc 1,907,339 9,020.90 0.00 5.6% 0.4% + (untracked) 303.99 0.2% 0.0% + validate_inputs_copy 1,905,787 194.93 0.00 0.1% 0.0% + validate_report 1,907,339 43.63 0.00 0.0% 0.0% + validate_setup 1,905,883 43.37 0.00 0.0% 0.0% + validate_warmups_listener.BenchmarkTimer 2,106,827 6,922.29 0.00 2.1% 0.3% + validate_gpu_sync 2,106,827 6,867.77 0.00 99.2% 0.3% + (untracked) 54.52 0.8% 0.0% + (untracked) 560.19 0.2% 0.0% + validate_warmups_listener.MetaResultReporter 2,106,827 149.34 0.00 0.0% 0.0% + validate_warmups_listener.HardwareMonitorListener 2,106,827 74.67 0.00 0.0% 0.0% + validate_warmups_listener.ProgressListener 2,106,827 70.80 0.00 0.0% 0.0% + validate_warmups_listener.AllSolutionsIterator 2,106,827 66.34 0.00 0.0% 0.0% + validate_warmups_listener.DataInitialization 2,106,827 64.11 0.00 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Kernel Execution 303,840.52 12.3% 11.8% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + pre_solution 2,355,016 220,916.68 0.09 72.7% 8.6% + pre_solution_listener.ReferenceValidator 2,355,016 151,756.15 0.06 68.7% 5.9% + cpu_reference_gemm_per_solution_inputs_changed 1,146 151,668.42 132.35 99.9% 5.9% + solve_cpu_fast_per_solution_inputs_changed 1,127 138,501.63 122.89 91.3% 5.4% + solve_cpu_slow_per_solution_inputs_changed 19 13,166.07 692.95 8.7% 0.5% + (untracked) 0.72 0.0% 0.0% + pre_solution_ref_skipped 2,353,182 49.50 0.00 0.0% 0.0% + (untracked) 38.23 0.0% 0.0% + pre_solution_listener.DataInitialization 2,355,016 49,481.67 0.02 22.4% 1.9% + pre_solution_mx_init_data 16,038 44,073.15 2.75 89.1% 1.7% + pre_solution_mx_side_regenerate 2,298 44,069.34 19.18 100.0% 1.7% + pre_solution_mx_side_cache_hit 29,778 2.60 0.00 0.0% 0.0% + (untracked) 1.21 0.0% 0.0% + pre_solution_mx_copy_valid_to_gpu_no_regen 14,889 3,124.70 0.21 6.3% 0.1% + pre_solution_mx_copy_valid_to_gpu_after_regen 1,149 1,143.05 0.99 2.3% 0.0% + pre_solution_mx_cpu_sync_after_regen 1,149 438.52 0.38 0.9% 0.0% + pre_solution_mx_copy_inputs_d2d_no_regen 14,889 307.63 0.02 0.6% 0.0% + pre_solution_mx_cpu_sync_no_regen 14,889 307.33 0.02 0.6% 0.0% + pre_solution_mx_check 2,355,016 49.85 0.00 0.1% 0.0% + pre_solution_mx_copy_inputs_d2d_after_regen 1,149 22.97 0.02 0.0% 0.0% + (untracked) 14.47 0.0% 0.0% + pre_solution_listener.BenchmarkTimer 2,355,016 11,859.71 0.01 5.4% 0.5% + pre_solution_listener.ProgressListener 2,355,016 3,134.22 0.00 1.4% 0.1% + pre_solution_listener.AllSolutionsIterator 2,355,016 2,647.18 0.00 1.2% 0.1% + pre_solution_listener.MetaResultReporter 2,355,016 1,159.39 0.00 0.5% 0.0% + (untracked) 472.96 0.2% 0.0% + pre_solution_listener.SynchronizerValidator 2,355,016 358.33 0.00 0.2% 0.0% + pre_solution_listener.HardwareMonitorListener 2,355,016 47.08 0.00 0.0% 0.0% + post_solution 2,355,016 35,727.86 0.02 11.8% 1.4% + post_solution_log 2,355,016 28,760.07 0.01 80.5% 1.1% + post_solution_reporting 2,355,016 5,582.67 0.00 15.6% 0.2% + post_solution_validation 2,355,016 740.26 0.00 2.1% 0.0% + post_solution_perf_calc 2,355,016 652.16 0.00 1.8% 0.0% + post_solution_result_file 2,355,016 57.99 0.00 0.2% 0.0% + post_solution_perf_reset 2,355,016 52.77 0.00 0.1% 0.0% + post_solution_sol_advance 2,355,016 50.05 0.00 0.1% 0.0% + warmup_runs 4,213,654 25,917.98 0.01 8.5% 1.0% + run_current_solution_check 2,355,016 8,420.00 0.00 2.8% 0.3% + kernel_solving 2,106,835 6,730.10 0.00 2.2% 0.3% + post_problem 28,641 1,945.83 0.07 0.6% 0.1% + run_iteration_teardown 2,106,835 1,677.91 0.00 0.6% 0.1% + warmup_events_setup 2,106,835 970.97 0.00 0.3% 0.0% + solution_selection 2,355,016 571.95 0.00 0.2% 0.0% + finalize_report 855 507.49 0.59 0.2% 0.0% + benchmark_runs 2,106,835 345.20 0.00 0.1% 0.0% + benchmark_validate_enqueues 319 181.62 0.57 52.6% 0.0% + validate_enqueues_listener.BenchmarkTimer 319 179.53 0.56 98.8% 0.0% + benchmark_sleep 319 179.48 0.56 100.0% 0.0% + (untracked) 0.03 0.0% 0.0% + benchmark_event_elapsed_time 8 0.01 0.00 0.0% 0.0% + benchmark_event_destroy 8 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.ProgressListener 319 1.81 0.01 1.0% 0.0% + (untracked) 0.21 0.1% 0.0% + validate_enqueues_listener.MetaResultReporter 319 0.02 0.00 0.0% 0.0% + validate_enqueues_listener.HardwareMonitorListener 319 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.AllSolutionsIterator 319 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.DataInitialization 319 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.SynchronizerValidator 319 0.01 0.00 0.0% 0.0% + validate_enqueues_listener.ReferenceValidator 319 0.01 0.00 0.0% 0.0% + benchmark_pre_syncs 2,106,835 56.09 0.00 16.2% 0.0% + benchmark_post_syncs 2,106,835 54.72 0.00 15.9% 0.0% + gpu_kernel_execution 319 39.52 0.12 11.4% 0.0% + (untracked) 13.15 3.8% 0.0% + benchmark_events_setup 319 0.06 0.00 0.0% 0.0% + benchmark_events_teardown 319 0.03 0.00 0.0% 0.0% + solution_teardown 2,355,016 77.56 0.00 0.0% 0.0% + problem_report_index 28,641 26.02 0.00 0.0% 0.0% + problem_teardown 28,641 4.94 0.00 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + C++ Client Setup & Teardown 294,842.41 12.0% 11.5% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + hip_initialization 855 124,448.44 145.55 42.2% 4.8% + process_exit 855 64,634.38 75.60 21.9% 2.5% + main_problem_factory 855 29,154.81 34.10 9.9% 1.1% + factory_hip_get_device_properties 192 28,922.86 150.64 99.2% 1.1% + (untracked) 231.95 0.8% 0.0% + data_init_setup 855 18,185.96 21.27 6.2% 0.7% + main_teardown_locals 855 13,182.66 15.42 4.5% 0.5% + process_startup 855 12,710.53 14.87 4.3% 0.5% + code_object_loading 855 11,001.27 12.87 3.7% 0.4% + main_timing_calibration 855 10,096.95 11.81 3.4% 0.4% + library_loading 855 9,940.49 11.63 3.4% 0.4% + lazy_loading_init 855 924.59 1.08 0.3% 0.0% + main_parse_args 855 410.00 0.48 0.1% 0.0% + reporter_setup 855 68.52 0.08 0.0% 0.0% + python_client_write_run_script 855 57.11 0.07 0.0% 0.0% + listener_setup 855 15.77 0.02 0.0% 0.0% + solution_iterator_setup 855 10.94 0.01 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Timing Overhead 157,975.78 6.4% 6.2% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + flush_timing_buffer 865 152,892.88 176.75 96.8% 6.0% + timing_overhead 855 5,082.90 5.94 3.2% 0.2% + ----------------------------------------------------------------------------------------------------------------------------- + python_kernel_compilation 32 44,036.26 1,376.13 1.7% 1.7% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_src_co 32 20,333.49 635.42 46.2% 0.8% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_kernel_build_src_co.compile 4 20,243.96 5,060.99 99.6% 0.8% + python_kernel_build_src_co.cache_check 32 53.80 1.68 0.3% 0.0% + python_kernel_build_src_co.unbundle 4 27.97 6.99 0.1% 0.0% + (untracked) 3.45 0.0% 0.0% + python_kernel_build_src_co.cache_populate 4 3.20 0.80 0.0% 0.0% + python_kernel_build_src_co.setup 32 0.84 0.03 0.0% 0.0% + python_kernel_build_src_co.move 4 0.26 0.07 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_codegen 32 16,254.60 507.96 36.9% 0.6% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_assemble 32 2,214.46 69.20 5.0% 0.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_setup 32 1,786.93 55.84 4.1% 0.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_validate 32 1,686.90 52.72 3.8% 0.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_write_helpers 32 545.76 17.05 1.2% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_build_co 32 526.58 16.46 1.2% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_bench_postprocess 32 495.87 15.50 1.1% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_benchpost_lib_construction 32 355.66 11.11 71.7% 0.0% + python_benchpost_library_write 32 84.30 2.63 17.0% 0.0% + python_benchpost_naming 32 50.98 1.59 10.3% 0.0% + python_benchpost_client_config 32 3.91 0.12 0.8% 0.0% + (untracked) 1.03 0.2% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_kernel_setup 32 137.71 4.30 0.3% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 53.96 0.1% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_ductile_initial_sampling 8 13,753.39 1,719.17 0.5% 0.5% + ----------------------------------------------------------------------------------------------------------------------------- + python_ductile_generate_solutions 32 4,760.25 148.76 0.2% 0.2% + ----------------------------------------------------------------------------------------------------------------------------- + (untracked) 2,407.19 0.1% 0.1% + ----------------------------------------------------------------------------------------------------------------------------- + python_ductile_ga_between_evaluations 8 1,387.06 173.38 0.1% 0.1% + ----------------------------------------------------------------------------------------------------------------------------- + python_cache_check 831 1,280.29 1.54 0.1% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_write_solutions 855 205.93 0.24 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_dump 32 66.32 2.07 32.2% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_header 855 65.76 0.08 31.9% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_prepare 855 60.96 0.07 29.6% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_wsol_prepare_cache 823 58.04 0.07 95.2% 0.0% + (untracked) 2.02 3.3% 0.0% + python_wsol_prepare_nocache 32 0.90 0.03 1.5% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 12.89 6.3% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_write_cache 32 58.71 1.83 0.0% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_solution_indexing 32 35.07 1.10 0.0% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_ductile_setup 8 32.30 4.04 0.0% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_ductile_clear_artifacts 32 26.20 0.82 0.0% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_ductile_read_scores 32 4.15 0.13 0.0% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_ductile_verification 8 0.30 0.04 0.0% 0.0% + =============================================================================================================================== + python_client_writer 10 23,848.23 2,384.82 0.9% + ----------------------------------------------------------------------------------------------------------------------------- + python_library_client_create_library 10 20,540.37 2,054.04 86.1% 0.8% + ----------------------------------------------------------------------------------------------------------------------------- + python_library_client_execution 10 3,161.37 316.14 13.3% 0.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_library_client_subprocess 10 3,155.15 315.52 99.8% 0.1% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 5.41 0.2% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_library_client_write_run_script 10 0.81 0.08 0.0% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_library_client_write_configs 10 120.68 12.07 0.5% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_library_client_setup 10 18.23 1.82 0.1% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + (untracked) 7.58 0.0% 0.0% + =============================================================================================================================== + python_library_logic 26 5,760.33 221.55 0.2% + ----------------------------------------------------------------------------------------------------------------------------- + python_logic_write 26 5,424.03 208.62 94.2% 0.2% + ----------------------------------------------------------------------------------------------------------------------------- + python_logic_parse_solutions 26 259.52 9.98 4.5% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + python_logic_analyze 26 69.15 2.66 1.2% 0.0% + ----------------------------------------------------------------------------------------------------------------------------- + (untracked) 7.63 0.1% 0.0% + + Overlapping Totals (wrap other timers; not summed into the tree) + ------------------------------------------------------------------------------------------------------------------------------- + python_client_subprocess 855 2,470,323.45 2,889.27 96.2% + main_total 855 2,226,863.00 2,604.52 86.7% + problem_total 28,641 2,022,050.86 70.60 78.7% + solution_total 2,355,016 943,803.95 0.40 36.7% + run_iteration_total 2,106,835 675,915.95 0.32 26.3% + validate_total 1,905,883 160,686.27 0.08 6.3% + validate_check_results_typed 1,907,339 160,445.08 0.08 6.2% + +--------------------------------------------------------------------------------------------------------------------------------- +HIERARCHY VALIDATION WARNINGS +--------------------------------------------------------------------------------------------------------------------------------- + WARNING: Child phases of python_client_execution (2,473,958.72 ms) exceed parent (2,470,665.16 ms) by 3,293.56 ms (0.1%) + +TIME BREAKDOWN (visual, % of wall clock) +--------------------------------------------------------------------------------------------------------------------------------- + TOTAL RUNTIME 100.0% ################################################## + RUN: benchmark client 90.8% ############################################# + Reference GEMM (CPU) 42.1% ##################### + cpu_reference_gemm 36.2% ################## + solve_cpu_slow 28.3% ############## + solve_cpu_fast 7.8% ### + (untracked) 0.0% + pre_solution_listener.ReferenceValidator 5.9% ## + cpu_reference_gemm_per_solution_inputs_changed 5.9% ## + solve_cpu_fast_per_solution_inputs_changed 5.4% ## + solve_cpu_slow_per_solution_inputs_changed 0.5% + (untracked) 0.0% + pre_solution_ref_skipped 0.0% + (untracked) 0.0% + Input data preparation 19.6% ######### + gpu_input_reset 11.9% ##### + gpu_input_reset_copy_valid_to_gpu 6.9% ### + gpu_input_reset_batched_inputs 3.7% # + gpu_input_reset_copy_inputs_d2d 1.0% + gpu_input_reset_copy_swizzled 0.2% + gpu_input_reset_fast_reset_output 0.1% + gpu_input_reset_convert_to_inputs 0.0% + gpu_input_reset_slow_path 0.0% + gpu_input_reset_rotating_init 0.0% + gpu_input_reset_constant_inputs 0.0% + gpu_input_reset_init_cpu_inputs 0.0% + (untracked) 0.0% + cpu_data_init 4.2% ## + pre_solution_listener.DataInitialization 1.9% + pre_solution_mx_init_data 1.7% + pre_solution_mx_side_regenerate 1.7% + pre_solution_mx_side_cache_hit 0.0% + (untracked) 0.0% + pre_solution_mx_copy_valid_to_gpu_no_regen 0.1% + pre_solution_mx_copy_valid_to_gpu_after_regen 0.0% + pre_solution_mx_cpu_sync_after_regen 0.0% + pre_solution_mx_copy_inputs_d2d_no_regen 0.0% + pre_solution_mx_cpu_sync_no_regen 0.0% + pre_solution_mx_check 0.0% + pre_solution_mx_copy_inputs_d2d_after_regen 0.0% + (untracked) 0.0% + gpu_input_preparation 1.5% + gpu_input_prep_copy_valid_to_gpu 0.9% + gpu_input_prep_copy_swizzled 0.4% + gpu_input_prep_init_cpu_inputs 0.2% + gpu_input_prep_batched_inputs 0.1% + gpu_input_prep_copy_inputs_d2d 0.0% + gpu_input_prep_fast_reset_output 0.0% + gpu_input_prep_convert_to_inputs 0.0% + gpu_input_prep_slow_path 0.0% + gpu_input_prep_constant_inputs 0.0% + gpu_input_prep_rotating_init 0.0% + (untracked) 0.0% + rotating_buffer_preparation 0.0% + Validation 12.6% ###### + validate_warmups_listener.SynchronizerValidator 6.3% ### + validate_synchronizer_check 6.3% ### + validate_synchronizer_scan 4.2% ## + validate_synchronizer_readback 2.2% # + validate_synchronizer_inactive 0.0% + (untracked) 0.0% + (untracked) 0.0% + validate_warmups_listener.ReferenceValidator 6.3% ### + validate_gpu_readback 3.6% # + validate_element_comparison 2.3% # + validate_result_buffer_alloc 0.4% + validate_inputs_copy 0.0% + validate_report 0.0% + validate_setup 0.0% + (untracked) 0.0% + validate_warmups_listener.MetaResultReporter 0.0% + validate_warmups_listener.HardwareMonitorListener 0.0% + validate_warmups_listener.ProgressListener 0.0% + validate_warmups_listener.AllSolutionsIterator 0.0% + validate_warmups_listener.DataInitialization 0.0% + Client launch (startup, setup & teardown) 11.1% ##### + hip_initialization 4.8% ## + process_exit 2.5% # + main_problem_factory 1.1% + factory_hip_get_device_properties 1.1% + (untracked) 0.0% + data_init_setup 0.7% + main_teardown_locals 0.5% + process_startup 0.5% + code_object_loading 0.4% + library_loading 0.4% + lazy_loading_init 0.0% + main_parse_args 0.0% + reporter_setup 0.0% + python_client_write_run_script 0.0% + listener_setup 0.0% + solution_iterator_setup 0.0% + Per-solution bookkeeping & reporting 2.7% # + post_solution_log 1.1% + pre_solution_listener.BenchmarkTimer 0.5% + run_current_solution_check 0.3% + post_solution_reporting 0.2% + pre_solution_listener.ProgressListener 0.1% + pre_solution_listener.AllSolutionsIterator 0.1% + post_problem 0.1% + run_iteration_teardown 0.1% + pre_solution_listener.MetaResultReporter 0.0% + post_solution_validation 0.0% + post_solution_perf_calc 0.0% + solution_selection 0.0% + finalize_report 0.0% + pre_solution_listener.SynchronizerValidator 0.0% + solution_teardown 0.0% + post_solution_result_file 0.0% + post_solution_perf_reset 0.0% + post_solution_sol_advance 0.0% + pre_solution_listener.HardwareMonitorListener 0.0% + problem_report_index 0.0% + problem_teardown 0.0% + Kernel launch & execution 1.6% + warmup_runs 1.0% + validate_warmups_listener.BenchmarkTimer 0.3% + validate_gpu_sync 0.3% + (untracked) 0.0% + kernel_solving 0.3% + warmup_events_setup 0.0% + benchmark_runs 0.0% + benchmark_validate_enqueues 0.0% + validate_enqueues_listener.BenchmarkTimer 0.0% + benchmark_sleep 0.0% + benchmark_event_elapsed_time 0.0% + benchmark_event_destroy 0.0% + (untracked) 0.0% + validate_enqueues_listener.ProgressListener 0.0% + validate_enqueues_listener.MetaResultReporter 0.0% + validate_enqueues_listener.HardwareMonitorListener 0.0% + validate_enqueues_listener.AllSolutionsIterator 0.0% + validate_enqueues_listener.DataInitialization 0.0% + validate_enqueues_listener.SynchronizerValidator 0.0% + validate_enqueues_listener.ReferenceValidator 0.0% + (untracked) 0.0% + benchmark_pre_syncs 0.0% + benchmark_post_syncs 0.0% + gpu_kernel_execution 0.0% + benchmark_events_setup 0.0% + benchmark_events_teardown 0.0% + (untracked) 0.0% + Library client (Python) 0.9% + python_library_client_create_library 0.8% + python_library_client_execution 0.1% + python_library_client_subprocess 0.1% + python_library_client_write_run_script 0.0% + (untracked) 0.0% + python_library_client_write_configs 0.0% + python_library_client_setup 0.0% + Library logic (Python) 0.2% + python_logic_write 0.2% + python_logic_parse_solutions 0.0% + python_logic_analyze 0.0% + MEASUREMENT OVERHEAD (instrumentation) 6.7% ### + flush_timing_buffer 6.0% ## + main_timing_calibration 0.4% + timing_overhead 0.2% + (timer overhead subtracted from parent timers) 0.2% + BUILD: solution generation & compile 2.5% # + Solution generation & build (Python) 1.7% + python_kernel_compilation 1.7% + python_kernel_build_src_co 0.8% + python_kernel_build_src_co.compile 0.8% + python_kernel_build_src_co.cache_check 0.0% + python_kernel_build_src_co.unbundle 0.0% + python_kernel_build_src_co.cache_populate 0.0% + python_kernel_build_src_co.setup 0.0% + python_kernel_build_src_co.move 0.0% + (untracked) 0.0% + python_kernel_codegen 0.6% + python_kernel_write_assemble 0.1% + python_kernel_bench_setup 0.1% + python_kernel_validate 0.1% + python_kernel_write_helpers 0.0% + python_kernel_build_co 0.0% + python_kernel_bench_postprocess 0.0% + python_benchpost_lib_construction 0.0% + python_benchpost_library_write 0.0% + python_benchpost_naming 0.0% + python_benchpost_client_config 0.0% + (untracked) 0.0% + python_kernel_setup 0.0% + (untracked) 0.0% + Ductile GA search (Python) 0.8% + python_ductile_initial_sampling 0.5% + python_ductile_generate_solutions 0.2% + python_ductile_ga_between_evaluations 0.1% + python_ductile_setup 0.0% + python_ductile_clear_artifacts 0.0% + python_ductile_read_scores 0.0% + python_ductile_verification 0.0% + SHARED: cache & solution files 0.1% + python_cache_check 0.0% + python_write_solutions 0.0% + python_wsol_dump 0.0% + python_wsol_header 0.0% + python_wsol_prepare 0.0% + python_wsol_prepare_cache 0.0% + python_wsol_prepare_nocache 0.0% + (untracked) 0.0% + (untracked) 0.0% + python_write_cache 0.0% + python_solution_indexing 0.0% + UNATTRIBUTED (untracked time & timer overhead) -0.0% + (untracked in python_benchmark_problems) 0.1% + (untracked in validate_warmups) 0.0% + (untracked in pre_solution) 0.0% + (untracked in pre_problem) 0.0% + (untracked in python_library_logic) 0.0% + (untracked in python_client_writer) 0.0% + (children exceed post_solution) -0.0% + (children exceed python_client_execution) -0.2% + +KEY INSIGHTS (% of wall clock) +--------------------------------------------------------------------------------------------------------------------------------- + Reference GEMM (CPU) 1080.47s (42.1%) + Input data preparation 502.88s (19.6%) + Validation 323.89s (12.6%) + Client launch (startup, setup & teardown) 284.75s (11.1%) + MEASUREMENT OVERHEAD (instrumentation) 168.07s (6.5%) + Per-solution bookkeeping & reporting 68.33s (2.7%) + Solution generation & build (Python) 44.04s (1.7%) + Kernel launch & execution 40.89s (1.6%) + Library client (Python) 23.84s (0.9%) + Ductile GA search (Python) 19.96s (0.8%) + Library logic (Python) 5.75s (0.2%) + SHARED: cache & solution files 1.58s (0.1%) + UNATTRIBUTED (untracked time & timer overhead) -0.93s (-0.0%) + + CPU reference / GPU execution ratio: 23498.2x + >>> CPU reference > 10x GPU execution time! + +TOP 10 SLOWEST PROBLEMS (by CPU reference time) +--------------------------------------------------------------------------------------------------------------------------------- + M N K Batch TypeA TypeD CPU Ref (ms) GPU (ms) + ------------------------------------------------------------------------------------------------------------------------------- + 4111 4111 4111 1 Float8 Float8 153530.08 0.00 + 4111 4111 4111 1 Float8 Float8 127609.73 0.00 + 4111 4111 4111 1 Int8 Int32 87409.00 0.00 + 4111 4111 4111 1 Int8 BFloat16 80626.39 0.00 + 4111 4111 4111 1 Int8 Int8 74748.05 0.00 + 4111 4111 4111 1 Int8 Half 59459.98 0.00 + 43776 10624 6400 1 Float8 BFloat16 17730.69 0.00 + 13056 7680 21504 1 Float8 BFloat16 15203.52 0.00 + 71936 4096 7168 1 Float8 BFloat16 13391.12 0.00 + 173312 3200 2560 1 Float8 BFloat16 11993.15 0.00 + diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/profiling/run_profile.sh b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/profiling/run_profile.sh new file mode 100644 index 000000000000..05976bbb424e --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/profiling/run_profile.sh @@ -0,0 +1,45 @@ +#!/bin/bash +# Copyright Advanced Micro Devices, Inc., or its affiliates. +# SPDX-License-Identifier: MIT +# +# Profiled cold build + run for the slowest gfx950 common configs. +# Adapted from ../../run_cold_warm.sh (gfx942) with gfx950 as the target and without the warm-build stage. +# Caches (~/.tensile, ~/.tensilelite) are deleted before the build and before the run, so both are cold. +# Usage: bash run_profile.sh gemm/gfx950/subtile_mxfp4 gemm/gfx950/dtl gemm/bufferLoadStore0 +TL=/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite +CL=/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client +cd $TL || exit 1 +export OMP_NUM_THREADS=25 OPENBLAS_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=0 PYTHONUNBUFFERED=1 +TO="--cxx-compiler,/usr/bin/amdclang++,--global-parameters,TimingInstrumentation=True" +COMMON=(-s -v -n 0 --durations=0 --timeout=5400 -m "common and not skip-gfx94X" + "--global-parameters=LibraryFormat='msgpack'" "--tensile-options=$TO" -p no:warnings) +clear_caches() { rm -rf /home/alvasile/.tensile /home/alvasile/.tensilelite; } + +for cfg in "$@"; do + name=$(basename $cfg) + D=$TL/timing_runs/full_runs_gfx950/profiling/$name + A=/tmp/tl_timing_art_gfx950_$name + mkdir -p $D + rm -rf $A + BID="tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/$cfg.yaml]" + RID="tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/$cfg.yaml]" + : > $D/times.txt + + clear_caches + s=$(date +%s) + taskset -c 0-24 pytest "$BID" --build-only --gpu-targets gfx950 --artifact-dir $A \ + "${COMMON[@]}" > $D/${name}_1_build_cold.log 2>&1 + echo "build_cold rc=$? $(( $(date +%s)-s ))s" >> $D/times.txt + + clear_caches + s=$(date +%s) + taskset -c 0-24 pytest "$RID" --use-cache --artifact-dir $A --prebuilt-client=$CL \ + "${COMMON[@]}" > $D/${name}_2_run.log 2>&1 + echo "run rc=$? $(( $(date +%s)-s ))s" >> $D/times.txt + + for st in 1_build_cold 2_run; do + python scripts/analyze_timing.py $D/${name}_$st.log > $D/${name}_${st}_analysis.txt 2>&1 + done + echo DONE >> $D/times.txt +done +echo ALLDONE > $TL/timing_runs/full_runs_gfx950/profiling/done.txt diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/profiling/run_rerun.sh b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/profiling/run_rerun.sh new file mode 100644 index 000000000000..4eae21d59725 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/profiling/run_rerun.sh @@ -0,0 +1,30 @@ +#!/bin/bash +# Copyright Advanced Micro Devices, Inc., or its affiliates. +# SPDX-License-Identifier: MIT +# +# Repeat of the run stage from run_profile.sh, to measure run-to-run variation. +# Reuses the build artifacts from run_profile.sh; caches are cleared before the run, as before. +# Usage: bash run_rerun.sh gemm/gfx950/subtile_mxfp4 gemm/gfx950/dtl gemm/bufferLoadStore0 +TL=/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite +CL=/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client +cd $TL || exit 1 +export OMP_NUM_THREADS=25 OPENBLAS_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=0 PYTHONUNBUFFERED=1 +TO="--cxx-compiler,/usr/bin/amdclang++,--global-parameters,TimingInstrumentation=True" +COMMON=(-s -v -n 0 --durations=0 --timeout=5400 -m "common and not skip-gfx94X" + "--global-parameters=LibraryFormat='msgpack'" "--tensile-options=$TO" -p no:warnings) +TAG=$1; shift + +for cfg in "$@"; do + name=$(basename $cfg) + D=$TL/timing_runs/full_runs_gfx950/profiling/$name + A=/tmp/tl_timing_art_gfx950_$name + RID="tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/$cfg.yaml]" + + rm -rf /home/alvasile/.tensile /home/alvasile/.tensilelite + s=$(date +%s) + taskset -c 0-24 pytest "$RID" --use-cache --artifact-dir $A --prebuilt-client=$CL \ + "${COMMON[@]}" > $D/${name}_3_run_$TAG.log 2>&1 + echo "run_$TAG rc=$? $(( $(date +%s)-s ))s" >> $D/times.txt + python scripts/analyze_timing.py $D/${name}_3_run_$TAG.log > $D/${name}_3_run_${TAG}_analysis.txt 2>&1 +done +echo ALLDONE > $TL/timing_runs/full_runs_gfx950/profiling/done_$TAG.txt diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/run.sh b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/run.sh new file mode 100755 index 000000000000..cc1b8a11aaf3 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/run.sh @@ -0,0 +1,29 @@ +#!/bin/bash +# Copyright Advanced Micro Devices, Inc., or its affiliates. +# SPDX-License-Identifier: MIT +# +# gfx950 counterpart of ../full_runs/run.sh (gfx942). Same CI-like conditions: +# 1 GPU, 25 cores, 4 xdist workers, 5400s per-test timeout, msgpack client. +# Single combined pass (what CI runs): build + run in one pytest invocation. +# Per-test timings come from pytest's --durations=0. +TL=/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite +OUT=$TL/timing_runs/full_runs_gfx950 +CLIENT=/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client +cd $TL || exit 1 +export OMP_NUM_THREADS=25 OPENBLAS_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=0 PYTHONUNBUFFERED=1 +COMMON=(tensilelite/Tests/common -v --durations=0 --timeout=5400 -n 4 -m "common and not skip-gfx94X" + "--global-parameters=LibraryFormat='msgpack'" "--tensile-options=--cxx-compiler,/usr/bin/amdclang++") + +rm -rf /home/alvasile/.tensile /home/alvasile/.tensilelite +{ + echo "START combined $(date -u +%FT%TZ)" + rocminfo | grep -m1 -o 'gfx950' + git rev-parse HEAD +} >> $OUT/times.txt +s=$(date +%s) +taskset -c 0-24 pytest "${COMMON[@]}" --prebuilt-client=$CLIENT --junit-xml=$OUT/combined.xml > $OUT/combined.log 2>&1 +rc=$? +echo "END combined rc=$rc seconds=$(( $(date +%s)-s )) $(date -u +%FT%TZ)" >> $OUT/times.txt +# Extract the per-test duration table +sed -n '/slowest durations/,/^=* .* in [0-9.]*s/p' $OUT/combined.log > $OUT/durations.txt +echo ALLDONE >> $OUT/times.txt diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/split/run_split.sh b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/split/run_split.sh new file mode 100755 index 000000000000..95d12a5e8723 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/split/run_split.sh @@ -0,0 +1,34 @@ +#!/bin/bash +# Copyright Advanced Micro Devices, Inc., or its affiliates. +# SPDX-License-Identifier: MIT +# +# gfx950 build / run split of the full common set (build on --build-only, then --use-cache run), +# same conditions as ../../full_runs/run.sh: 25 cores, 4 xdist workers, 5400s per-test timeout, +# msgpack client. Gives per-test RUN-phase durations, which is what a client-launch-batching change +# can affect (the combined pass is dominated by kernel builds). +# Env: TL, CLIENT, ART, TENSILE_CACHE, TENSILELITE_CACHE (see ../measure_env.sh), +# GPU_TARGET, CORES, GPU (see ../measure_env.sh), +# OUT (default $TL/timing_runs/full_runs_gfx950/split). ART is deleted and rebuilt, so it must +# be set: the default ART is the measured baseline, which the script refuses to delete. +source "$(dirname "${BASH_SOURCE[0]}")/../measure_env.sh" || exit 1 +OUT=${OUT:-$TL/timing_runs/full_runs_gfx950/split} +check_rm_target TENSILE_CACHE TENSILELITE_CACHE ART +cd "$TL" || exit 1 +mkdir -p "$OUT" +export OMP_NUM_THREADS=25 OPENBLAS_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=$GPU PYTHONUNBUFFERED=1 +COMMON=(tensilelite/Tests/common -v --durations=0 --timeout=5400 -n 4 -m "common and not skip-gfx94X" + "--global-parameters=LibraryFormat='msgpack'" "--tensile-options=--cxx-compiler,/usr/bin/amdclang++") +phase() { # name, extra args... + local name=$1; shift + clear_caches + echo "START $name $(date -u +%FT%TZ)" >> "$OUT/times.txt" + local s=$(date +%s) + taskset -c "$CORES" pytest "${COMMON[@]}" "$@" --junit-xml="$OUT/$name.xml" > "$OUT/$name.log" 2>&1 + local rc=$? + echo "END $name rc=$rc seconds=$(( $(date +%s)-s )) $(date -u +%FT%TZ)" >> "$OUT/times.txt" +} +rm -rf -- "$ART"; mkdir -p "$ART" +phase build --build-only --gpu-targets "$GPU_TARGET" --artifact-dir "$ART" --prebuilt-client="$CLIENT" +du -sh "$ART" >> "$OUT/times.txt" +phase run --use-cache --artifact-dir "$ART" --prebuilt-client="$CLIENT" +echo ALLDONE >> "$OUT/times.txt" diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/verify_mx/run_verify_mx.sh b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/verify_mx/run_verify_mx.sh new file mode 100755 index 000000000000..8d15881925fb --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/verify_mx/run_verify_mx.sh @@ -0,0 +1,21 @@ +#!/bin/bash +# Copyright Advanced Micro Devices, Inc., or its affiliates. +# SPDX-License-Identifier: MIT +# +# Runs the gfx950 MX common configs with TENSILELITE_VERIFY_REFERENCE_REUSE=1, i.e. every time the +# client skips a CPU-reference recompute because DataInitialization's input epoch did not move, +# it recomputes anyway and fails the run if the result is not bitwise identical. +# CI-like conditions as in ../full_runs/run.sh (combined build+run pass). +TL=/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite +OUT=$TL/timing_runs/full_runs_gfx950/verify_mx +CLIENT=/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client +cd $TL || exit 1 +export OMP_NUM_THREADS=25 OPENBLAS_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=0 PYTHONUNBUFFERED=1 +export TENSILELITE_VERIFY_REFERENCE_REUSE=1 +rm -rf /home/alvasile/.tensile /home/alvasile/.tensilelite +s=$(date +%s) +taskset -c 0-24 pytest tensilelite/Tests/common/test_config.py -v --durations=0 --timeout=5400 -n 4 \ + -m "common and not skip-gfx94X" -k "mx and gfx950" \ + "--global-parameters=LibraryFormat='msgpack'" "--tensile-options=--cxx-compiler,/usr/bin/amdclang++" \ + --prebuilt-client=$CLIENT --junit-xml=$OUT/verify_mx.xml > $OUT/verify_mx.log 2>&1 +echo "rc=$? seconds=$(( $(date +%s)-s ))" > $OUT/result.txt diff --git a/projects/hipblaslt/tensilelite/timing_runs/input_data_preparation_investigation.md b/projects/hipblaslt/tensilelite/timing_runs/input_data_preparation_investigation.md new file mode 100644 index 000000000000..f08aa704f047 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/input_data_preparation_investigation.md @@ -0,0 +1,233 @@ +# "Input data preparation" investigation (gfx950 per-config rerun 2) + +`full_runs_gfx950/per_config_rerun_2/AGGREGATE_analysis.txt` reports **Input data preparation** +at 502,878.60 ms (19.6% of wall). This note explains why it is slow and which of its timers are +inflated or deflated by work that belongs elsewhere. Timing claims come from measurements; code +reading was only used to decide what to measure. Inferences are marked as such. + +`cpu_data_init` (106.8 s) is covered separately in `cpu_data_init_investigation.md` and is not +repeated here. + +| Timer | Count | Total (ms) | Mean (ms) | +| --- | --- | --- | --- | +| `gpu_input_reset` | 2,106,835 | 306,247.35 | 0.15 | +| ├─ `gpu_input_reset_copy_valid_to_gpu` | 719,180 | 177,839.34 | 0.25 | +| ├─ `gpu_input_reset_batched_inputs` | 1,473,329 | 95,916.03 | 0.07 | +| ├─ `gpu_input_reset_copy_inputs_d2d` | 1,473,329 | 26,309.71 | 0.02 | +| ├─ `gpu_input_reset_copy_swizzled` | 15,719 | 4,006.86 | 0.25 | +| └─ `gpu_input_reset_fast_reset_output` | 633,454 | 1,917.65 | 0.00 | +| `cpu_data_init` | 28,596 | 106,829.68 | 3.74 | +| `pre_solution_listener.DataInitialization` | 2,355,016 | 49,481.67 | 0.02 | +| └─ `pre_solution_mx_side_regenerate` | 2,298 | 44,069.34 | 19.18 | +| `gpu_input_preparation` | 28,641 | 39,596.08 | 1.38 | +| ├─ `gpu_input_prep_copy_valid_to_gpu` | 11,446 | 22,935.66 | 2.00 | +| └─ `gpu_input_prep_copy_swizzled` | 1,151 | 10,766.76 | 9.35 | +| `rotating_buffer_preparation` | 28,641 | 723.82 | 0.03 | + +## Summary + +1. **Per-solution re-upload of unchanged data (`copy_valid_to_gpu`, 177.8 s).** On the slow + reset path, every solution re-uploads every input tensor from host memory. 99.45–99.92% of + those uploads are byte-identical to the previous upload into the same buffer. +2. **`batched_inputs` (95.9 s) is inflated.** Its own work is 4–7 tiny (8–24 byte) uploads of batch + pointer arrays per solution; 99.99% are byte-identical to the previous ones. Each costs + ~12–15 µs of latency, and because they are the first blocking call after the D2D copies they + also absorb those copies' GPU time and, on configs with `SyncsPerBenchmark: 0`, the previous + solution's still-running warmup kernels. +3. **`copy_inputs_d2d` (26.3 s) is deflated.** `hipMemcpy` D2D returns in ~2 µs regardless of + size; the copy itself is charged to whatever blocks next. Its real serialized cost is + 4.5–5.5× the timer. +4. **MX regeneration (`pre_solution_mx_side_regenerate`, 44.1 s) is redundant** and, through + `m_inputEpoch`, triggers a redundant CPU reference recompute + (`cpu_reference_gemm_per_solution_inputs_changed`, 151.7 s, booked under "Reference GEMM"). + +## Method + +- **Binary.** The measured client, `/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client` + (Release, `-O3`). Configs re-run with the cached split artifacts + (`/tmp/tl_split_gfx950/artifacts`) exactly as `full_runs_gfx950/profiling/run_rerun.sh` does + (`TimingInstrumentation=True`, `OMP_NUM_THREADS=25`, 25 pinned cores). +- **Environment.** The host was shared: during these runs cores 0–47 were ~99% busy with other + users' work, so runs used idle NUMA-1 cores (64–88, 89–113) with NUMA-1 GPUs 4 and 5, two runs + at a time. `perf` was unavailable (`perf_event_paranoid=4`). +- **`hipMemcpy` profiler.** An `LD_PRELOAD` shim (`memprof.cpp`) records per call site (return + address, mapped to functions with `nm`; the client has no debug info) and kind: count, bytes and + time inside `hipMemcpy`. Optional modes: + - *hash*: hash each H2D source and count uploads byte-identical to the previous upload into the + same device address; + - *presync*: `hipDeviceSynchronize()` before each `hipMemcpy`/`hipMemset`, charging the wait to + the **previous** GPU operation (memcpy call site, `hipExtModuleLaunchKernel`/`hipLaunchKernel`, + memset, async copy). This serializes everything, so it gives each operation's full cost and + shows where in-flight work is absorbed. An idle-GPU `hipDeviceSynchronize` costs 0.07 µs. +- **Shim overhead.** In passive mode the timers match production: `spmm_f16` + `copy_valid_to_gpu` 21,126 ms vs 20,468 ms, `batched_inputs` 5,993 vs 5,622 ms, + `copy_inputs_d2d` 1,366 vs 1,489 ms. +- **Microbenchmarks.** `copybench.cpp` (copy behavior) and `mxdeterm.cpp` (MX generator + determinism, linked against the client's `libhipblaslt-mxdatagen.a`). +- Scratch sources, scripts and per-run outputs are in `/tmp/syncval_bench` (`memprof.cpp`, + `memprof_sum.py`, `run_art.sh`, `copybench.cpp`, `mxdeterm.cpp`, `inprep/_/`). + +## Background: the two reset paths + +Before each solution, `main.cpp` calls `DataInitialization::prepareGPUInputs` +(`DataInitialization.hpp`), timed as `gpu_input_reset`: + +- **Fast path** (`fast_reset_output`, 633,454 calls, 1.9 s): GPU-generated data, bounds checking + off, no problem-dependent data, no swizzle. Only output tensors are reset. +- **Slow path** (1,473,329 calls): taken when bounds checking is on, data is problem-dependent + (`m_problemDependentData`: sparse, MX, Trig/Serial/Identity init modes, multiple bias types), or + tensors are swizzled. Per solution it runs: + +```text +copyValidToGPUBuffer if problem-dependent: H2D every tensor, host valid → GPU valid (copy_valid_to_gpu) +copySwizzledToGPUBuffer if swizzled (copy_swizzled) +copyInputs D2D every tensor, GPU valid → GPU current (copy_inputs_d2d) +initializeGPUBatchedInputs H2D a batch-pointer array per A/B/C/D (+ more) (batched_inputs) +``` + +The host data only changes per problem (`cpu_data_init`), or when MX data is regenerated. + +## Finding 1: per-solution re-upload of unchanged data (`copy_valid_to_gpu`, 177.8 s) + +The biggest contributors are `spmm_f16_sb` (53.9 s), `xfp32` (27.7 s), `spmm_f16` (20.5 s), +`plr_zero` (15.2 s) and `spmm_bf16_sb` (11.3 s). + +Hash mode, `copyInputBuffers` H2D (the `copy_valid_to_gpu` uploads): + +| Config | Uploads | Byte-identical to previous upload | MB/call (min–max) | Upload time | +| --- | --- | --- | --- | --- | +| `spmm_f16` | 567,420 | 566,162 (99.78%) | 1.26 (0.001–3.94) | 21.3 s | +| `spmm_f16_sb` | 1,508,910 | 1,507,646 (99.92%) | 1.25 (0.001–3.94) | 55.7 s | +| `xfp32` | 462,000 | 459,436 (99.45%) | 2.22 (2.10–16.8) | 30.2 s | +| `subtile_mxfp4` | 167,071 | 163,921 (98.1%) | 0.59 (0–16.8) | 7.1 s | + +- Nearly all of the bytes uploaded on every solution are already on the GPU. `spmm_f16` alone + re-uploads ~715 GB at ~34 GB/s. +- The copies are sized by `p.maxElements` (the largest allocation in the run), not the current + problem: in `xfp32`, every upload over 2,052 problems of varying size is at least 2.10 MB. The + `cpu_data_init` investigation measured the same `maxElements` sizing directly. +- A pinned source would help only for small tensors (`copybench`, pageable vs pinned: 1 MB 41.8 + vs 26.7 µs; 4 MB 84.6 vs 81.7 µs; 16 MB and up ~56 GB/s both). Not uploading is the real fix. +- `DataInitialization` already tracks host rewrites with `m_inputEpoch` (used by + `ReferenceValidator` to skip recomputing the CPU reference). Skipping the upload when the + epoch is unchanged would remove ≥99% of this bucket, ~176 s (inferred from the identical-upload + rates, not measured end-to-end). + +The per-problem `gpu_input_prep_copy_valid_to_gpu` (22.9 s) is needed once per problem, but is +subject to the same `maxElements` sizing. + +## Finding 2: `batched_inputs` is inflated (95.9 s) + +`initializeGPUBatchedInputs` builds a pointer array per tensor on the host and uploads it with a +pageable `hipMemcpy`. In every profiled config each upload is 8–24 bytes (8 bytes per batch). + +| Config | Uploads | Per solution | Byte-identical | Mean call | +| --- | --- | --- | --- | --- | +| `spmm_f16` | 449,568 | 7 | 449,511 (99.99%) | 13.3 µs | +| `spmm_f16_sb` | 1,190,378 | 7 | 1,190,321 (99.995%) | 12.4 µs | +| `xfp32` | 369,600 | 4 | 369,560 (99.99%) | 15.1 µs | +| `bufferLoadStore0` | 1,609,364 | 4 | not hashed | 14.9 µs | + +The arrays only change when a problem or buffer address changes, yet they are rebuilt and +re-uploaded for every solution. The timer then holds three things: + +1. **Latency of the tiny uploads.** An 8-byte pageable H2D costs 11.6 µs on an idle GPU + (`copybench`). In presync mode each also leaves ~7–8 µs of work in flight after returning. +2. **The preceding D2D copies.** They return immediately (Finding 3), so the first blocking + upload waits for them: in `copybench`, an 8-byte upload issued right after a 64 MB D2D takes + 28.5 µs instead of 11.6 µs. +3. **The previous solution's warmup kernels.** With `SyncsPerBenchmark: 0` (524 occurrences in + the test YAMLs, including `bufferLoadStore0` and `spmm_f16`), warmups 1..N-1 are launched and + nothing waits for them: `BenchmarkTimer::postWarmup` returns early unless + `skip-slow-solution-ratio` is set. The next solution's reset absorbs them. Measured in presync + mode on `bufferLoadStore0`: 2,345 ms of still-running-kernel time absorbed inside the reset, + over 200,635 of 401,984 resets (11.7 µs each). + +Passive-mode excess over the 11.6 µs floor: `bufferLoadStore0` 24.0 s measured vs 18.7 s floor +(5.3 s absorbed); `spmm_f16` 5.96 s vs 5.21 s (0.75 s absorbed). Caching the arrays per problem +(or re-uploading only when the epoch or a base address changes) would remove nearly all of the +bucket; the absorbed work would then surface in the next blocking call (inferred). + +## Finding 3: `copy_inputs_d2d` is deflated (26.3 s) + +`copybench`, GPU otherwise idle: + +| D2D size | `hipMemcpy` returns after | Copy complete after | +| --- | --- | --- | +| 64 KB | 2.2 µs | 14.5 µs | +| 4 MB | 2.1 µs | 21.4 µs | +| 64 MB | 2.2 µs | 38.2 µs | +| 256 MB | 2.3 µs | 113.5 µs | + +In the client the D2D calls average 2–3 µs. Presync mode gives their real serialized cost (time +in the call plus the wait charged to them): + +| Config | `copy_inputs_d2d` timer (passive) | D2D call time + wait charged to D2D (presync) | +| --- | --- | --- | +| `spmm_f16` | 1,366 ms | 1,443 + 6,015 = 7,458 ms | +| `bufferLoadStore0` (NaN bounds check: `copyNaNInputBuffers` + `copyInputBuffers`) | 6,144 ms | 2,646 + 10,932 + 2,887 + 11,271 = 27,736 ms | + +The remainder lands in `batched_inputs`, in the warmup kernel's start, and in the validators' +first blocking readback. `copyInputs` copies every tensor, including read-only inputs (A, B) that +kernels do not modify; only outputs (and the Synchronizer, see `SynchronizerValidator.md`) need +restoring between solutions. The split between necessary and avoidable D2D bytes was not +measured. + +## Finding 4: redundant MX regeneration and the CPU reference recompute it triggers + +On gfx950, `initializeMXData` caches MX data per side, keyed by the tensor descriptor plus a tag +saying `gpuInput.valid` holds a scale swizzled for the current solution's MI shape. The tag is +erased whenever a side is initialized without a swizzle, which includes the problem-level init. +So the first solution of each problem that needs the GFX950 swizzle misses the cache and +**regenerates** the side: one `generateMXInput` call for data and canonical scale into +`cpuInput.valid`, and a second one (with the swizzle layout) that rewrites the data buffer again +to produce the swizzled scale. It also increments `m_inputEpoch`. + +- **The regenerated data is identical.** `generateMXInput` uses a fixed seed (`1713573849`, + `mxDataGen.cpp`). Measured with `mxdeterm.cpp` (4096×4096, FP4 E2M1 and FP8 E4M3): two calls + with identical arguments give byte-identical data and scales, and the data written by a + `None`-layout call is byte-identical to that written by a `GFX950`-layout call. Each call costs + 77–160 ms at this size. The descriptor is unchanged at the point of regeneration, so the + arguments are identical. +- **It happens once per problem.** In `subtile_mxfp4`: 1,370 regenerated sides = 685 problems × 2 + sides, and 685 `pre_solution_mx_copy_valid_to_gpu_after_regen` uploads. The hash run finds + 3,150 non-identical uploads in total, ~4.6 per problem. If the regenerated data differed from + the problem-level data there would be at least 4 more per problem (A, B and both scales), so + the uploads after regeneration are byte-identical too (inferred from the counts). +- **It triggers a second CPU reference.** `ReferenceValidator` recomputes the CPU reference when + `inputEpoch(solution)` changes. Across the aggregate, in 35 of the 36 configs involved, the + number of `cpu_reference_gemm_per_solution_inputs_changed` recomputes equals the number of + after-regeneration uploads. In `subtile_mxfp4`, each of the 685 problems computes the reference + twice (685 problem-level + 685 "inputs changed", 5,916 ms). Aggregate: 1,146 recomputes, + 151.7 s, booked under "Reference GEMM (CPU)". + +| Config | Regenerate (ms) | Recompute (ms) | Count | +| --- | --- | --- | --- | +| `subtile_mxfp8` | 14,806 | 70,729 | 39 | +| `subtile_mxfp8_correctness_regression_mt256x320_plr0_wgm1` | 7,104 | 27,987 | 5 | +| `subtile_mxfp8_correctness_regression_mt320x256_plr1_wgm4` | 6,883 | 23,140 | 5 | +| `mxfp4_mxfp4_bf16_tn_act` | 1,241 | 13,744 | 78 | +| `subtile_mxfp4` | 5,908 | 4,126 | 685 | + +Producing only the swizzled scale from the canonical one, without touching the data or the +epoch, would remove the 44.1 s here and the 151.7 s recompute (inferred). + +## Inflated and deflated timers + +| Timer | Effect | Why | +| --- | --- | --- | +| `gpu_input_reset_batched_inputs` | **inflated** | absorbs the preceding D2D copies and, with `SyncsPerBenchmark: 0`, the previous solution's warmup kernels | +| `gpu_input_reset_copy_inputs_d2d` | **deflated** | `hipMemcpy` D2D returns in ~2 µs; real cost 4.5–5.5× the timer, charged downstream | +| `gpu_input_reset` (whole) | **inflated** on `SyncsPerBenchmark: 0` configs | previous solution's kernel time (measured 2.3 s on `bufferLoadStore0`) | +| `gpu_input_reset_copy_valid_to_gpu` | accurate, but ≥99% redundant | re-uploads unchanged, allocation-sized data every solution | +| `pre_solution_mx_side_regenerate` | accurate, but redundant | regenerates byte-identical data; inflates "Reference GEMM" by 151.7 s via the epoch | + +## Not measured / open items + +- End-to-end effect of any fix (only per-call costs and redundancy were measured). +- How much of `copy_inputs_d2d` is necessary (outputs) vs avoidable (read-only inputs). +- Where the D2D and kernel time absorbed by `batched_inputs` would land once that bucket is + removed; an explicit, separately timed sync after the warmups would make it visible. +- `gpu_input_prep_copy_swizzled` / `gpu_input_reset_copy_swizzled` (14.8 s combined) and the + per-problem paths were not profiled beyond the counts above. +- Run-to-run variance on the shared host. diff --git a/projects/hipblaslt/tensilelite/timing_runs/per_solution_bookkeeping_investigation.md b/projects/hipblaslt/tensilelite/timing_runs/per_solution_bookkeeping_investigation.md new file mode 100644 index 000000000000..66a560d7ab5e --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/per_solution_bookkeeping_investigation.md @@ -0,0 +1,721 @@ +# "Per-solution bookkeeping & reporting" investigation (gfx950 per-config rerun 2) + +`full_runs_gfx950/per_config_rerun_2/AGGREGATE_analysis.txt` reports **Per-solution bookkeeping & +reporting** at 68,333.60 ms (2.7% of wall). This note explains where that time goes and what can +be done about it. Timing claims come from measurements; code reading was only used to decide what +to measure. Inferences are marked as such. + +The section [Correctness-only runs](#correctness-only-runs-what-can-be-turned-off) covers which of +this work, and of the per-launch timing work, can be switched off when Tensile is run only to check +correctness (the common tests), rather than to benchmark. + +| Timer | Count | Total (ms) | Mean (µs) | +| --- | --- | --- | --- | +| `post_solution_log` | 2,355,016 | 28,760.07 | 12.2 | +| `pre_solution_listener.BenchmarkTimer` | 2,355,016 | 11,859.71 | 5.0 | +| `run_current_solution_check` | 2,355,016 | 8,420.00 | 3.6 | +| `post_solution_reporting` | 2,355,016 | 5,582.67 | 2.4 | +| `pre_solution_listener.ProgressListener` | 2,355,016 | 3,134.22 | 1.3 | +| `pre_solution_listener.AllSolutionsIterator` | 2,355,016 | 2,647.18 | 1.1 | +| `post_problem` | 28,641 | 1,945.83 | 67.9 | +| `run_iteration_teardown` | 2,106,835 | 1,677.91 | 0.8 | +| `pre_solution_listener.MetaResultReporter` | 2,355,016 | 1,159.39 | 0.5 | +| 12 smaller timers | | 3,146.62 | | + +## Summary + +There is no single slow function. The section is a fixed host-side cost of about 29 µs per +solution, paid 2,355,016 times. Weighted by solution count, the per-config cost is 24.1 µs (p10), +25.6 µs (p50) and 36.6 µs (p90). The 13 configs with at least 50k solutions hold 85.8% of the +solutions and 80.7% of the time. Four things make up that per-solution cost (shares measured on +dtl unless stated): + +1. **Preemption by spinning OpenMP workers, about 15% of the section.** In the production logs, + 9.9 s of these timers is in individual calls longer than 1 ms, including timers that only do + CPU work. After each OpenMP parallel region (e.g. the CPU reference), the client's 24 idle + libomp workers keep spinning (`KMP_BLOCKTIME` defaults to 200 ms) on the same 25 cores as the + main thread: 11–24 workers were measured busy during the solution loop. In the dtl replay, + `KMP_BLOCKTIME=0` cut the >1 ms share from 1.8–4.1 s to 0.08–0.16 s. + It also cut `cpu_reference_gemm` from 5.8–14.8 s to 2.4–2.7 s, which is outside this section. +2. **The `report()` fan-out, about 38%.** Each solution makes about 19 `report()` calls. Each is + forwarded to 3 reporters, which format numbers with a fresh `std::ostringstream` (about 150 ns + each), up to three times per value. +3. **One stdout CSV row per solution, about 28%.** `LogReporter` copies the 30-column row into a + new `unordered_map`, formats a row of about 1.1 KB (mostly the solution name), and flushes + stdout with `std::endl`. Over the aggregate, that is 2,335,808 rows and 2.64 GB of text that + nothing parses: Tensile reads only the `results-file` CSV. +4. **Predicate debug dumps for rejected solutions, 16% in dtl and about 6% of the aggregate + (estimate).** The default `ClientLogLevel` is 3 (Debug). For every solution that fails its + predicates (443,391 in the aggregate, 18.8%), `checkSolution` calls `debugEval` on the whole + predicate tree, which costs 8.9 µs per rejected solution. + +A prototype that fixes 2 and 3 produced output identical to the shipped client and cut the dtl +bookkeeping section by 33%. Adding 4 (Terse log level) brought the cut to 58%, and fixing 1 removes +the tail on top of that (see [Options](#options)). + +## Method + +- **Binary.** The measured client, `/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client` + (Release, `-O3 -DNDEBUG -std=c++20 -x hip --offload-arch=gfx950 -fopenmp=libomp`). +- **Rebuilds.** Copies of `tensilelite/client` were rebuilt outside the repo with the exact + compile and link lines from that build's `compile_commands.json` and `link.txt`, linked against + the same `libtensilelite-host.so`. There are three copies: + - `ref`: unmodified; its timings match the shipped binary within run-to-run noise. + - `instr`: `ref` plus finer `ScopedTimer`s (`bk_*`) inside the timers above, and per-reporter + timers in `MetaResultReporter`. + - `opt`: the prototype described under [Options](#options). +- **Workload.** `common__gemm__gfx950__dtl` (443,808 solutions; the largest config in this section, + at 10.9 s). It was first run through pytest exactly as `full_runs_gfx950/profiling/run_rerun.sh` + does (cached artifact, `TimingInstrumentation=True`, `--prebuilt-client`). That run gave 9.8 s + for the section, against 10.9 s in production. Its 26 client invocations were then replayed + directly, in the same order with the same `ClientParameters.ini`. The replay matches the pytest + run (e.g. `post_solution_log` 2,780 ms vs 2,700 ms). +- **Variants.** Each variant was run 2–3 times. They use the same binary and inputs, and change only + `KMP_BLOCKTIME` (environment) or `log-level` (ini edit). +- **Microbenchmarks.** The formatting primitives were compiled with the same compiler and `-O3`. + Timings are medians of 9 repetitions. +- **Production logs.** All 184 `per_config_rerun_2` logs were scanned for per-call distributions + and for stdout content (CSV rows, validation outcomes, predicate dumps). +- **Environment.** gfx950 host, 25 idle cores via `taskset`, `OMP_NUM_THREADS=25`. The host is + shared: every GPU was 69–97% busy with other users' work. All conclusions are drawn from + host-only timers. Whole-client wall times moved by ±20% between repeats, because of GPU-bound + timers such as `gpu_input_reset`, and are not used. +- Scratch scripts, summaries and logs are in `/tmp/bookkeep`: `build.sh`, `direct.sh`, `summ.sh`, + `cmp.sh`, `tail.sh`, `rej.sh`, `client_src/` (instr), `client_opt/` (opt) and `micro/fmt.cpp`. + +## Where the time goes (one solution, dtl, instrumented build) + +Means per call in µs. Each nested timer adds about 42 ns (calibrated `timing_overhead`), so the +instrumented parents read higher than in the `ref` build; both are shown. + +| Timer / sub-step | `ref` mean | `instr` mean | Notes | +| --- | --- | --- | --- | +| `post_solution_log` | 6.62 | 6.12 | | +| ├─ `readCurrentRow` (copy 30 columns into a new map) | | 2.40 | done for every row, printed or not | +| ├─ `writeCurrentRow`: format the row | | 1.67 | `escape()` copies each value again | +| ├─ `writeCurrentRow`: `std::endl` flush | | 1.14 | one `write()` of about 1.1 KB per solution | +| └─ `pop` (restore the row from the stack) | | 0.39 | | +| `pre_solution_listener.BenchmarkTimer` | 4.91 | 6.62 | | +| ├─ `projectedPerformance` | | 0.27 | | +| └─ 9 `report()` calls | | 6.28 | about 0.5 µs per report after removing timer overhead | +| `run_current_solution_check` (average) | 5.05 | 5.18 | | +| ├─ predicate evaluation (all solutions) | | 0.60 | | +| ├─ hardware predicate (all solutions) | | 0.25 | | +| └─ reject path (173,006 rejected solutions) | | 10.44 | per rejected solution | +| ├─ `debugEval` of problem and task predicates | | 8.86 | | +| └─ `log()` of the dump | | 0.57 | | +| `post_solution_reporting` (4 double `report()` calls) | 1.73 | 2.50 | | +| `pre_solution_listener.ProgressListener` | 1.20 | 1.62 | `solution->name()` 0.21; 2 reports | +| `pre_solution_listener.AllSolutionsIterator` | 1.12 | 1.64 | 3 reports, one built with `concatenate` | +| `pre_solution_listener.MetaResultReporter` | 0.48 | 0.36 | `CSVStackFile::push` copies the row map | + +### The `report()` fan-out + +Per solution, the listeners make 19.2 `report()` calls: 10.0 double, 4.0 int, 3.1 string and 2.1 +uint. `MetaResultReporter` forwards each call to 3 reporters; `LibraryUpdateReporter` is null +here. Including timer overhead, the per-call cost by reporter is: + +| Reporter | Mean per call (µs) | What it does with a double | +| --- | --- | --- | +| `ResultFileReporter` | 0.273 | `ostringstream << value` to get a string, compares the key against about 8 keys, then `setValueForKey(key, value)`, which formats the double again with a second `ostringstream` | +| `LogReporter` | 0.168 | `setValueForKey(key, double)`: `ostringstream << setprecision(6)` | +| `PerformanceReporter` | 0.023 | key comparisons only | + +Microbenchmark of the primitives, using the client's compiler and `-O3`: + +| Operation | ns | +| --- | --- | +| `ostringstream` + `setprecision(6)` + double → string | 153.6 | +| `ostringstream` + double → string | 148.7 | +| `ostringstream` + int64 → string | 50.5 | +| `std::to_chars(double, general, 6)` → string (same text) | 36.9 | +| `std::to_chars(int64)` → string | 4.7 | +| copy 30–33 columns into a new `unordered_map` (the `readCurrentRow` pattern) | 1,016 | + +The fan-out comes to 3.7 s of the dtl replay's 9.8 s (about 8.4 µs per solution) after removing +the instrumentation's own overhead. That overhead was 34M nested timers at 42 ns, or 1.4 s. + +### The stdout CSV row + +In the dtl baseline log, the client wrote 1.19 M non-TIMING lines (520 MB), counted by line type: + +| Line type | Lines | Bytes | +| --- | --- | --- | +| CSV row per solution (30 columns; about 1,000 of 1,125 bytes is the solution name) | 443,808 | 499.5 MB | +| predicate debug dump lines (`[!!] ...`) | 382,108 | 20.1 MB | +| blank, `####` and `++++` separators | 365,548 | 0.9 MB | + +Across all 184 production logs: 2,335,808 CSV rows (2.64 GB) and 923,357 dump lines (48.8 MB). +Tensile does not read any of it: `ClientWriter.py` starts the client with `subprocess.Popen` and no +`stdout=` argument, and `BenchmarkProblems.py` reads results only from `results-file`. In +`run_rerun.sh`, the client's stdout and stderr were a regular file (`/proc//fd/1` → the log), +not a pipe. + +### Predicate debug dumps for rejected solutions + +`SolutionIterator::checkSolution` evaluates the problem and task predicates. If either fails and +`logAtLevel(Verbose)` is true, it calls `debugEval` on both trees. `debugEval` re-evaluates every +term (the `And` nodes have 35–39 terms) and formats a row per term into an `ostringstream`; the +result is then logged. `ClientLogLevel` defaults to 3 (Debug) in `Common/GlobalParameters.py`, and +no common test YAML overrides it, so every rejected solution pays for this. + +| | dtl | aggregate (184 logs) | +| --- | --- | --- | +| solutions rejected by predicates (`DID_NOT_SATISFY_ASSERTS`) | 173,006 (39.0%) | 443,391 (18.8%) | +| `debugEval` + log per rejected solution | 9.4 µs | | +| `run_current_solution_check`, Debug → Terse (dtl, `KMP_BLOCKTIME=0`) | 2,109 → 473 ms | | + +The Python enum has no `Normal`, so Tensile can only select Error, Terse, Verbose or Debug. At +Terse, rows of PASSED solutions (row level Normal) are not printed. Rows of rejected solutions +(row level Terse) are still printed, but without the dump. + +## The >1 ms tail: spinning OpenMP workers + +Share of each timer spent in calls longer than 1 ms, across all 184 production logs: + +| Timer | Total (ms) | in calls > 1 ms | +| --- | --- | --- | +| `post_solution_log` | 28,760.1 | 4,364.0 (15.2%) | +| `pre_solution_listener.BenchmarkTimer` | 11,859.7 | 1,022.7 (8.6%) | +| `run_current_solution_check` | 8,420.0 | 1,110.1 (13.2%) | +| `post_solution_reporting` | 5,582.7 | 1,175.2 (21.1%) | +| `pre_solution_listener.ProgressListener` | 3,134.2 | 321.2 (10.2%) | +| `pre_solution_listener.AllSolutionsIterator` | 2,647.2 | 312.6 (11.8%) | +| `post_problem` | 1,945.8 | 970.9 (49.9%) | +| `run_iteration_teardown` | 1,677.9 | 305.7 (18.2%) | +| `pre_solution_listener.MetaResultReporter` | 1,159.4 | 353.7 (30.5%) | +| **all nine** | 65,187.0 | 9,936.0 (15.2%) | + +Medians are far below the means. For example, `post_solution_log` in `spmm_i8hs` has a median of +14.9 µs, a p99 of 5.0 ms and a maximum of 42.7 ms. Pure-CPU timers such as +`pre_solution_listener.ProgressListener` show the same spikes, so the cause is not I/O. + +The client links libomp, and its process has 27 threads: main, 24 OpenMP workers and 2 runtime +threads. All of them share the 25 cores of the `taskset` mask. Per-thread CPU time was sampled +from `/proc//task/*/stat` during the solution loop: + +| `KMP_BLOCKTIME` | Windows sampled | Workers above 25% CPU | Whole process, cores busy | +| --- | --- | --- | --- | +| default (200 ms) | 6 | 11–24 | 12–20 | +| 0 | 3 | 0 | 1.2 (main 1.07) | + +Effect on the dtl replay, in ms (each row is one run): + +| Variant | Section total | ≤ 1 ms calls | > 1 ms calls | `cpu_reference_gemm` | +| --- | --- | --- | --- | --- | +| default | 10,246 | 8,466 | 1,780 | 5,819 | +| default | 14,419 | 10,770 | 3,649 | 14,770 | +| default | 14,523 | 10,407 | 4,116 | 14,142 | +| `KMP_BLOCKTIME=0` | 9,362 | 9,240 | 122 | 2,420 | +| `KMP_BLOCKTIME=0` | 9,813 | 9,730 | 83 | 2,737 | +| `KMP_BLOCKTIME=0` | 10,585 | 10,423 | 162 | 2,670 | + +With `KMP_BLOCKTIME=0`, the tail disappears and run-to-run variance drops. The ≤1 ms part does not +shrink; it is the real per-solution work covered by the other findings. + +Inference, not measured: with spinning workers, 26–27 threads compete for 25 cores, and the main +thread is regularly descheduled for a full timeslice. That would explain both the millisecond +spikes and their spread across every timer on the main thread. The same mechanism probably inflates +other main-thread sections. The `cpu_reference_gemm` drop (2–5×) is large and belongs to the +Reference GEMM section, so it is listed under open items. + +## Options + +All measurements below come from the dtl replay with `KMP_BLOCKTIME=0`, so that the tail does not +hide the differences. Totals are for the whole section, in ms. + +| Variant | Runs | Section total | `post_solution_log` | `BenchmarkTimer` pre | `run_current_solution_check` | `post_solution_reporting` | Progress + AllSolutions | +| --- | --- | --- | --- | --- | --- | --- | --- | +| `ref` | 3 | 9,362 / 9,813 / 10,585 | 2,890 | 2,136 | 2,109 | 757 | 1,010 | +| `opt` (B) | 3 | 6,507 / 6,588 / 6,708 | 1,367 | 838 | 2,225 | 335 | 793 | +| `ref` + Terse (C) | 2 | 7,285 / 7,724 | 2,090 | 2,170 | 473 | 755 | 1,006 | +| `opt` + Terse (B + C) | 3 | 4,099 / 4,151 / 4,115 | 738 | 822 | 421 | 327 | 788 | + +Per-timer columns are means over the runs. The section drops from 9.9 s to 6.6 s with B (−33%), +and to 4.1 s with B + C (−58%), or 9.3 µs per solution. + +### A. Stop idle OpenMP workers from spinning + +Set `KMP_BLOCKTIME=0` for the client, either in the environment `ClientWriter.py` passes to the +client, or with `kmp_set_blocktime(0)` or a smaller worker pool in `main.cpp`. `OMP_WAIT_POLICY=passive` +should have a similar effect, but was not measured. Measured effect: the >1 ms tail falls from +1.8–4.1 s to 0.08–0.16 s in dtl. In the aggregate, that tail is 9.9 s of this section. It costs +nothing in the measured runs: `cpu_reference_gemm`, the main OpenMP user, got faster, not slower. + +### B. Cheaper formatting and row handling (prototype `opt`, output-identical) + +The changes, all in the client: + +1. `CSVStackFile::setValueForKey(double)` and the integral `setValueForKey` use `std::to_chars` + instead of `std::ostringstream`. For doubles, `std::chars_format::general` with precision 6 + produces the same text as `<< setprecision(6)`. +2. `ResultFileReporter::reportValue` builds its value string with `std::to_chars` (once) instead + of an `ostringstream`. +3. `LogReporter::postSolution` looks up the 3–7 keys it uses (`findCurrentValue`) instead of + copying all 30 columns with `readCurrentRow`. The header is still written on the first row. +4. `CSVStackFile::writeRow` ends rows with `'\n'` instead of `std::endl`. `LogReporter::postProblem` + flushes once per problem. + +Correctness check: `ref` and `opt` were run on three dtl invocations (3,508, 320 and 60,940 stdout +lines). The results CSVs were byte-identical, and stdout differed only in the time-based +`srand seed` line. dtl runs no timed iterations, so its rows are deterministic. + +Measured: −33% on the section (9.9 → 6.6 s). `post_solution_log` fell by 53%, `BenchmarkTimer` +pre by 61% and `post_solution_reporting` by 56%. The remaining Progress/AllSolutions cost is mostly +string reports and `concatenate`, which the prototype does not touch. + +Trade-off of item 4: if the client crashes, up to one problem's worth of stdout rows can be lost. +The client already prints a `##########################` / `++++` pair per problem, so the flush +point matches an existing boundary. + +Robustness note: in two consecutive runs, about 1.5 minutes in total, 37 of 52 `opt` invocations +crashed with SIGSEGV. The crashed invocations never printed `loading config file`, which is printed +with `std::endl` during argument parsing, before any reporter code runs. That points either to +crashes before the prototype's code executed, or to log writes failing. The root filesystem was at +100% use (27 GB free) before that batch and at 96% after it. The crash did not reproduce in 4 +isolated reruns of the same invocation, or in the other 208 `opt` invocations (26 before, 182 +after). It was not seen in any `ref` run. Root cause unknown. + +### C. Don't dump predicate trees by default + +Either lower the default `ClientLogLevel`, or gate the `debugEval` call in `checkSolution` behind +its own option and keep the per-solution rows. Measured with Terse: `run_current_solution_check` +fell from 2,109 to 473 ms in dtl (−9.4 µs per rejected solution). Scaled to the aggregate's +443,391 rejected solutions, that is about 4.2 s (estimate). The dump remains useful when debugging +why a kernel is rejected, so it should stay available behind an option. Terse also removes the +stdout rows of passing solutions, which costs log readability in CI when a test fails. + +### D. Larger changes (not measured) + +- Skip the stdout row entirely unless asked for. It is 2.64 GB of text in the aggregate that no + tool reads; `readCurrentRow`/`writeCurrentRow` account for about 6 µs per solution, before B. +- Report the 9 granularity and memory values in `BenchmarkTimer::preSolution` only when a consumer + needs them (CSV extra columns or the log). They are 9 of the 19 `report()` calls. +- Replace `MetaResultReporter`'s per-call virtual fan-out over `std::string` keys with typed keys. + Each call also copies a `shared_ptr` per reporter (`for(auto r : m_reporters)`). + +### Estimated effect on the aggregate + +These figures are estimates: they scale the dtl measurements to the aggregate's counts. Other +configs have a different mix, for example 18.8% rejected solutions overall against 39% in dtl. + +| Option | Basis | Estimated saving | +| --- | --- | --- | +| A | measured >1 ms share in the production logs | about 9.9 s | +| B | 7.5 µs per solution (dtl) × 2,355,016 | about 17.6 s | +| C | 9.4 µs per rejected solution × 443,391 | about 4.2 s | +| A + B + C | | about 32 s of 68.3 s | + +## Correctness-only runs: what can be turned off + +`Tensile input.yaml out` serves two purposes: benchmarking, and the correctness common tests. A +correctness run needs only three things: + +- whether each solution produced a correct result; +- where it differed; +- the client's exit code (`main` returns the number of failed validations, clamped to 255; the test + fails on a non-zero code); + +Everything else the client does per solution is reporting or timing. That includes the +`results-file` CSV: it has to exist, but its values carry no information in a correctness run (see +[The results CSV in correctness runs](#the-results-csv-in-correctness-runs)). + +### The results CSV in correctness runs + +The client writes one row per problem size. Each row holds the sizes and one GFlops cell per +solution: `-nan` for a passed solution when nothing is timed, and `-1` for a failed or rejected +one. Python then uses it like this: + +1. `BenchmarkProblems` copies it to `2_BenchmarkData/` (`shutil.copy`, which fails if the file is + missing). Pass/fail does not come from the CSV: it is the client's exit code + (`benchmarkTestFails`), followed by `sys.exit(1)` under `ExitOnFails`. No common-test harness + file reads the CSV, `2_BenchmarkData` or `3_LibraryLogic`. +2. LibraryLogic reads it (`addFromCSV`) only for YAMLs with a `LibraryLogic:` section. In the + production logs, 26 of the 184 gfx950 configs did: + - **10 `gemm/gfx950/*mxfp8*` correctness configs, `SyncsPerBenchmark: 0`, `LibraryType: + Prediction`, followed by LibraryClient.** Prediction skips winner, range and exact logic, and + `removeInvalidSolutions()` drops only solutions with GFlops exactly 0, which a correctness run + never writes. Test: in a saved `subtile_mxfp8_correctness_regression` run, every `-nan` was + replaced with `123.5`, and LibraryLogic and LibraryClient were re-run (with + `ForceRedoBenchmarkProblems=False`, so the benchmark step was skipped). LibraryLogic then + found exact winners for all 5 sizes instead of `{}`. Yet the written logic YAML was + byte-identical, and LibraryClient passed. + - **8 `ductile/gfx950/*` configs, `SyncsPerBenchmark: 0`.** Ductile's GA reads the GFlops cells + as its fitness scores (`_read_and_validate_scores_csv`), and every one is NaN. Every Ductile + log shows `# ExactWinners: {}`. The GA and its LibraryLogic step run without any performance + signal; only the plumbing and the per-candidate correctness are tested. No LibraryClient + follows. + - **`groupedgemm/grouped_gemm` and the 7 `custom/*` configs.** These benchmark: `grouped_gemm` + sets `SyncsPerBenchmark: 1`, and the custom configs leave it at the default of 1. They get + real winners. +3. In the remaining 158 configs, the CSV is written, copied, and never read. + +So in correctness mode the CSV needs only to exist, with the right shape (sizes plus one column per +solution), for the `shutil.copy` and `addFromCSV` calls. `BenchmarkProblems._writeSyntheticResultsCSV` +already writes such a file for `--cpu-only`. Two caveats: + +- That synthetic file uses a fixed positive GFlops. Non-Prediction LibraryLogic would then pick + winners where today it picks none. This was not tested for the Ductile configs. +- The per-solution `-1` for failures is not consumed in correctness runs, since failures already + fail the test through the exit code. + +### YAMLs with `LibraryLogic:` / `LibraryClient:` sections + +**What the sections do.** `Tensile.py` runs these steps after BenchmarkProblems: + +- `LibraryLogic:` reads `2_BenchmarkData/*.csv` and the solutions YAML, picks winners per size, + and writes the logic YAML (the format hipBLASLt ships) to `3_LibraryLogic/`. It costs about + 0.2 s per config. +- `LibraryClient:` reads `3_LibraryLogic/`, so it depends on `LibraryLogic:`. It builds a real + library from the logic with `TensileCreateLibrary` (1.9–3.3 s), then runs the client against that + library: solution selection through the library, with validation on (0.26–0.43 s). + +With an empty `LibraryClient:` section and no exact logic, `ClientWriter` falls back to +`ProblemSizesMockDummy`, which is a single size, `[128, 128, 1, 512]`. So the library client never +runs the YAML's own benchmark sizes. + +**Who has them.** 30 common YAMLs and all 12 `custom/` YAMLs: + +| Group | Added by | `SyncsPerBenchmark` | `LibraryClient` | What the section achieves | +| --- | --- | --- | --- | --- | +| 10 `gemm/gfx950/*mxfp8*` correctness/repro configs | #7781 (multi-DU support), all at once; the first common YAMLs to use `Prediction` | 0 | yes, empty section | `Prediction` logic lists every solution; the CSV values do not change it (tested above). The library client validates only the dummy 128×128×1×512 size, about 24 s for all 10. No reason is given in the commit message or the YAMLs. | +| 14 `ductile/*` (8 on gfx950) | #8831 (Ductile backend) | 0 | commented out | Mirrors a tuning flow, whose product is a logic file. With NaN scores, the logs show `Winners set()`, `Exact Logic: {}` and a warning, and nothing reads the file. Nothing in Ductile reads `config["LibraryLogic"]`. | +| `gemm/fp8nfp16mix_hfp8ns`, `gemm/xfp32` (+ `LibraryClient`), `groupedgemm/{grouped_gemm, gfx11/grouped_gemm_gfx11, grouped_gemm_ck_gfx942}`, `sparse/gfx94x/libray_logic` | older | 1 (opted out in the unit test: "LibraryLogic requires benchmark data to build the library table") | `gemm/xfp32` only | Exercise winner selection with real timing data. Only `grouped_gemm` runs on gfx950. | +| 12 `custom/*` (see below) | #9304 / #11155 | unset, so the default 1 | no | `GridBased` logic with real winners, because they benchmark. Nothing reads it. | + +The opt-out reason "LibraryLogic requires benchmark data" holds for `GridBased` logic: with NaN +data it finds no winners. It does not hold for `Prediction` logic, whose output was byte-identical +with and without real values. + +### The `custom/` folder + +`Tests/custom/` is a sibling of `common/`. It holds 12 YAMLs: `custom_aiter_bf16`, +`custom_aiter_f4`, `custom_ck`, `custom_rocblas_gemv_bf16_tn_m1/_m2/_m4`, `custom_rr`, `custom_rr_f6`, +`custom_rr_mx`, `custom_tensile_sk`, `custom_triton_f4` and `custom_wave_bf16`. They come from the +"Gemm-From-Anywhere" custom-kernel framework (#9304, reverted in #11104, re-landed in #11155; +later additions such as #12965). Each feeds a prebuilt external kernel (aiter, CK, rocBLAS GEMV, +rr, Triton, wave, or Tensile StreamK) through the `CustomKernel` fork parameter, with +`ValidateMetadata: True`. The PR's test plan states their purpose as "build and dispatch each demo +kernel end-to-end through the real `CustomKernels:` fork-parameter path". Producing logic files is +not mentioned. + +They run as part of the common tests, even though they are outside `common/`: + +- `config_helpers.findConfigs()` walks all of `Tests/` and parametrizes every YAML into + `common/test_config*.py`. +- `conftest.pytest_collection_modifyitems` marks each test with the top directory of its *test + file*. So a `custom/` YAML gets the `common` marker, plus `custom` from `configMarks`. +- With `-m "common and not skip-gfx94X"` and `--gpu-targets gfx950`, `--collect-only` lists 552 + tests, including all 12 custom YAMLs. + +Results on gfx950: + +- **CI-like combined run (`full_runs_gfx950/run.sh`):** 7 PASSED (`aiter_bf16`, `aiter_f4`, the + three `rocblas_gemv`, `triton_f4`, `wave_bf16`); 3 SKIPPED (`ck`, `rr`, `tensile_sk`); 2 FAILED + (`rr_f6`, and `rr_mx`, which failed to assemble). That run was on an older commit; the current + status of the two failures was not rechecked. +- **Per-config reruns:** the 7 passing configs took 5–7 s each. + +None of the 12 sets `SyncsPerBenchmark`, so all of them time their kernels. All 12 have a +`GridBased` `LibraryLogic:` section and no `LibraryClient:`. + +### Existing parameters already express "no benchmarking" + +524 of the 540 common YAMLs set `SyncsPerBenchmark: 0`, so no timed enqueues run. Only 16 set it +to 1. Of those 16, 5 ran on gfx950, with 288 solutions out of 2,355,016 in total: the +`rotate_mode0/1`, `sk_sgemm_debug_modes`, `grouped_gemm` and `benchmark_runs/fp32_tn` configs. +For today's YAMLs, "`SyncsPerBenchmark == 0`" therefore matches "nothing is timed". It is not the +client's full rule, though, and "nothing is timed" is not the same as "this is a test". See +[Deciding when a run is a test](#deciding-when-a-run-is-a-test). + +With `SyncsPerBenchmark: 0`, the following parameters change nothing, measured or read in code: + +- `EnqueuesPerSync`, `MaxEnqueuesPerSync`, `MinFlopsPerSync`, `SleepPercent` and `KernelTime`: only + read inside the timed-enqueue loop, or by `BenchmarkTimer` when it times that loop. +- `HardwareMonitor`: samples only around timed enqueues, and `Tensile.py` already forces it off on + gfx950 and gfx1250. +- `SkipSlowSolutionRatio`: 0.0 in all 19 YAMLs that set it. + +### What each piece of output is for + +| Output | Needed for correctness? | Controlled by | +| --- | --- | --- | +| exit code = number of failed validations | yes | always | +| `results-file` CSV (one GFlops column per solution) | must exist with the right shape; values are NaN/−1 and do not change any output (see above) | always | +| FAILED row on stdout (row level Error) | yes | printed at every log level | +| mismatch lines (`[i] elem=… idx=…: ref!=gpu`), `Check failed in output tensor` | yes | printed with `std::cout` at every level, up to `ValidationMaxToPrint` | +| Synchronizer residue messages | yes | `LogLevel::Error` | +| CSV row for every PASSED solution | no | `ClientLogLevel` ≥ Verbose | +| CSV row and predicate `debugEval` dump for every rejected solution | no | `ClientLogLevel` ≥ Terse (row) and ≥ Verbose (dump) | +| granularity, tiles-per-CU and memory-byte columns (`BenchmarkTimer::preSolution`) | no | only shown in the stdout row, or with `CSVExportExtraCols` | +| time-us, gflops, gbps (`BenchmarkTimer::postSolution`) | no; always NaN when nothing is timed | always computed | +| warmup start/stop `hipEvent`s (2 per launched kernel) | no; only `BenchmarkTimer` reads them | always created | +| warmups beyond the first launch | no, unless the YAML deliberately relaunches (see below) | `NumWarmups` | + +### Measurements + +**Setup.** All runs replay the dtl client invocations (443,808 solutions) or the +`gemm/gfx950/xfp32` ones (94,668 solutions, `NumWarmups: 5`), with `KMP_BLOCKTIME=0`, 2 runs per +variant. The per-solution host cost is measured as `pre_solution` + `run_current_solution_check` + +`post_solution`. These are the enclosing timers, so work moved out of an inner timer is still +counted. + +**Output equivalence.** This was checked for every variant against `ref`, on 3 dtl and 2 xfp32 +invocations, at Debug and Error, both with clean runs and with injected failures. Failures were +injected by an `LD_PRELOAD` shim that flips one byte of every 7th device-to-host `hipMemcpy` of at +least 4 KiB (`/tmp/bookkeep/micro/corrupt.cpp`). In all cases: + +- exit codes were identical (0, 20, 216 and 255); +- results CSVs were byte-identical; +- the FAILED rows (61, 216, 2,128 and 3,934 per invocation) and the mismatch locations were + identical, after normalising the run-specific random data values (`srand seed` is time-based). + +| Variant (dtl) | How | Per-solution host cost (ms) | dtl stdout | +| --- | --- | --- | --- | +| today (Debug) | – | 8,941 / 8,780 | 1,191,874 lines, 520 MB | +| `ClientLogLevel: 0` | existing parameter | 6,517 / 6,279 (−28%) | 10,350 lines, 0.31 MB | +| `ClientLogLevel: 0` + correctness mode (prototype `corr`) | client change | 3,121 / 3,167 (−65%) | 10,324 lines, 0.30 MB | +| `ClientLogLevel: 0` + `corr` + no timing events (prototype `corr2`) | client change | 3,166 / 3,103 | same | + +With `ClientLogLevel: 0`, a clean invocation that printed 60,940 lines prints 87. One with 3,934 +failures prints 14,259: exactly the FAILED rows and the mismatch lines. + +`ClientLogLevel` is already settable from the test command line. `--tensile-options=…, +--global-parameters,ClientLogLevel=0` (the same mechanism as `TimingInstrumentation=True`) gave +`log-level=Error` in every `ClientParameters.ini` of `streamk/sk_dynamic_work_stealing`, and the +test passed. + +Prototype `corr`, in `/tmp/bookkeep/client_corr`, applies when `num-syncs-per-benchmark == 0` (or +`num-enqueues-per-sync == 0`) and `csv-export-extra-cols` is false. + +**Correction:** the `num-enqueues-per-sync == 0` half of that condition is wrong. With +`num-syncs-per-benchmark=1`, `num-enqueues-per-sync=0` and the default `min-flops-per-sync=1`, the +client still times every solution: 288/288 timed loops, all with real `time-us`; see +[Deciding when a run is a test](#deciding-when-a-run-is-a-test). No current YAML has that +combination (all six `EnqueuesPerSync: 0` files also set `SyncsPerBenchmark: 0`), so the +measurements above are unaffected. A real implementation must use the full rule below. + +When the prototype's condition holds: + +- `BenchmarkTimer::preSolution` skips `projectedPerformance` and its 9 reports. +- `BenchmarkTimer::postSolution` skips the performance calculation and reports the same NaN values + directly. The original's 0/0 gives a negative quiet NaN, which prints as `-nan`; the prototype + reproduces it. +- `LogReporter::postSolution` checks the row level before copying the row. The copy is only used for + printed rows. One side effect: the CSV header line is printed just before the first printed row, + instead of on the first solution. + +Prototype `corr2` (`/tmp/bookkeep/client_corr2`) additionally creates the warmup `TimingEvents` +with null events, and skips `BenchmarkTimer`'s `hipEventSynchronize` after warmup 0. That wait is +taken over by the validators' blocking readback, so removing it only moves attribution. Measured +on dtl: `warmup_runs` (host side of the launches) fell from 1,981–2,046 to 1,291–1,295 ms, +`warmup_events_setup` from 72 to 30 ms, and `run_iteration_teardown` from 122–129 to 46–47 ms. That +is about 0.85 s, or 3.1 µs per launched solution. On xfp32 it saved about 2.4 µs per launched +solution. + +### Warmups that nothing consumes + +`numWarmupRuns()` is the maximum over listeners: the validators ask for 1 launch, and +`BenchmarkTimer` asks for `NumWarmups`. So with `SyncsPerBenchmark: 0`, every warmup beyond the +first only matters if the YAML relaunches deliberately. On gfx950, ten configs set +`SyncsPerBenchmark: 0` with `NumWarmups > 1`: + +| Config | `NumWarmups` | Solutions | Assessment | +| --- | --- | --- | --- | +| `gemm/gfx950/xfp32` | 5 | 94,668 | no stated purpose; also sets `EnqueuesPerSync: 10`, which is unused with 0 syncs | +| 7 `ductile/gfx950/*` | 5–10 | 37–130 each | negligible | +| `streamk/sk_dynamic_work_stealing`, `sk_hybrid_work_stealing` | 2 | 32, 64 | deliberate: "Back-to-back enqueues reuse the same Synchronizer workspace, exercising the per-queue auto-reset across launches." | + +`cluster_entry_handoff` (288 warmups) and the gfx12 configs are not run on gfx950. + +xfp32 with `num-warmups=0` instead of 5 (one launch per solution; each row is one run, times in s): + +| Variant | `main_total` | `warmup_runs` (host) | `gpu_input_reset` | +| --- | --- | --- | --- | +| as configured (5 warmups) | 76.4 | 2.91 | 46.4 | +| as configured (5 warmups) | 65.9 | 2.29 | 39.9 | +| `num-warmups=0` | 53.1 | 0.58 | 30.1 | +| `num-warmups=0` | 59.6 | 0.60 | 33.5 | +| `num-warmups=0` + `ClientLogLevel: 0` | 59.3 | 0.57 | 30.6 | +| `num-warmups=0` + `ClientLogLevel: 0` | 54.5 | 0.59 | 32.0 | + +About 15 s (20%) of xfp32's client time goes. Most of the extra warmups' GPU time showed up in the +*next* solution's `gpu_input_reset`, whose blocking copy waits for the queued launches. That is the +same attribution effect described in `SynchronizerValidator.md`. Because the GPU is shared, the +GPU-bound timers vary by ±15%, but all four `num-warmups=0` runs are below both baseline runs. The +warmup count cannot be inferred away globally, because some configs relaunch on purpose. It is a +per-YAML fix. + +### Options for correctness runs + +| Option | Change type | Measured | Aggregate estimate (scaled from dtl/xfp32 counts) | +| --- | --- | --- | --- | +| 1. pass `ClientLogLevel=0` for the common tests (test harness / CI command line) | none: existing parameter | −28% per-solution host cost on dtl; stdout 520 MB → 0.31 MB | ≈ 9 s (2.1 µs per solution for the rows, plus 9.4 µs per rejected solution) and 2.6 GB of stdout | +| 2. drop `NumWarmups: 5` / `EnqueuesPerSync: 10` from `gemm/gfx950/xfp32.yaml` | YAML | xfp32 client time −20% (≈ 15 s) | ≈ 15 s; no other gfx950 config is affected | +| 3. infer "nothing is timed" in the client (prototype `corr`; the real rule is in [Deciding when a run is a test](#deciding-when-a-run-is-a-test)) | client code | a further −51% on top of option 1 (6.4 → 3.1 s on dtl) | ≈ 17 s (7.3 µs per solution) | +| 4. no timing events in correctness mode (prototype `corr2`) | client code | 3.1 µs per launched solution (dtl) | ≈ 5–6.5 s | +| 5. `KMP_BLOCKTIME=0` (option A above) | environment | removes the >1 ms tail | ≈ 9.9 s, plus the `cpu_reference_gemm` effect | + +Option 1 could also be inferred rather than passed. `ClientWriter` could write `log-level=Error` +when `SyncsPerBenchmark == 0` and the YAML does not set `ClientLogLevel`. That needs a way to tell +"explicitly set" from "default" for that parameter; not implemented. + +Remaining per-solution cost in correctness mode (dtl, `corr`): about 7 µs per solution. It is +`AllSolutionsIterator`/`ProgressListener` progress strings and solution name (1.0 s), predicate +evaluation (0.47 s), and `ResultFileReporter` building the results CSV. The CSV must exist, and its +formatting could use option B. Because its values carry no information when nothing is timed, +the client could also write it once per problem with constant cells, instead of fanning every +value out to `ResultFileReporter`. This is not prototyped. + +Not affected: data initialisation, CPU reference, input reset, the kernel launch, and both +validators. These are the correctness work itself and are covered in the other docs. + +### Planned follow-ups (recorded, not implemented) + +**1. Extend the `SyncsPerBenchmark` guard to `custom/`.** + +`Tests/unit/test_common_yaml_syncs_per_benchmark.py` today globs only `common/` +(`_COMMON_DIR.rglob`). It requires `GlobalParameters.SyncsPerBenchmark == 0`, as an int, for every +file not in `_SYNCSPERBENCHMARK_OPT_OUTS`. Each opt-out must give a reason, and stale opt-outs fail +the test. `custom/` YAMLs run as common tests (see above), so the test should also walk +`Tests/custom/`. + +All 12 custom YAMLs would fail it today, since the key is unset. Each needs either +`SyncsPerBenchmark: 0` or an opt-out with a reason. Given their stated purpose (build and dispatch +end to end), `SyncsPerBenchmark: 0` looks right. Their only consumer of timing data is the +`LibraryLogic:` step, which nothing reads (see item 2). + +**2. Remove `LibraryLogic:` / `LibraryClient:` from common YAMLs, and pin it with a test.** + +Add a unit test, in the same style as the `SyncsPerBenchmark` guard: a dataclass whitelist with a +mandatory reason, plus a stale-entry check. It fails if any YAML under `common/` has a top-level +`LibraryLogic` or `LibraryClient` key, unless the file is whitelisted. Applying it to `custom/` as +well would match item 1, but that scope is still to be decided. + +Proposed whitelist: the 6 files whose purpose is the logic/library path with real benchmark data, +which are already the `SyncsPerBenchmark` opt-outs with the LibraryLogic reason. + +| File | Sections | +| --- | --- | +| `common/gemm/fp8nfp16mix_hfp8ns.yaml` | `LibraryLogic` | +| `common/gemm/xfp32.yaml` | `LibraryLogic` + `LibraryClient` | +| `common/groupedgemm/grouped_gemm.yaml` | `LibraryLogic` | +| `common/groupedgemm/gfx11/grouped_gemm_gfx11.yaml` | `LibraryLogic` | +| `common/groupedgemm/grouped_gemm_ck_gfx942.yaml` | `LibraryLogic` | +| `common/sparse/gfx94x/libray_logic.yaml` | `LibraryLogic` | + +To remove: + +- the 14 `ductile/*` sections: the logic is written from NaN data and never read; +- the 10 `gemm/gfx950/*mxfp8*` sections: the library client validates only the dummy + 128×128×1×512 size, about 24 s per gfx950 run; +- the 12 `custom/*` sections, if the scope is extended. + +Decision needed before removal: the 10 mxfp8 configs are the only common coverage of `Prediction` +logic, library build from that logic, and `Prediction` selection in the client. Either accept +losing that coverage, or whitelist one designated config (for example +`subtile_mxfp8_tail_loop_smoke`), ideally with explicit `LibraryClient` sizes so that it validates +more than the dummy size. + +**3. Copy the results CSV only if it exists.** + +After the client runs, `BenchmarkProblems.main` does `shutil.copy(resultsFileName, +newResultsFileName)` unconditionally. The granularity CSV is already copied only `if +os.path.isfile(...)`. Apply the same guard to the results CSV, so that a run that did not write one +does not fail. + +The remaining readers fail hard on a missing file, so they must only run when the CSV is written: + +- `LibraryLogic.addFromCSV` calls `printExit`; +- Ductile's `_read_and_validate_scores_csv` checks `os.path.isfile` and calls `printExit`. + +**4. Don't write the results CSV for tests.** + +Have Tensile pass "no results file" to the client, and have the client skip `ResultFileReporter` +entirely in that case. Today `results-file` defaults to `results.csv`, and +`ResultFileReporter::Default` always constructs the reporter. An empty name would only produce a +silently failing `ofstream` while still doing all the per-solution formatting, so the client needs +an explicit "no reporter" path. The CSV must still be written whenever something reads it: a +`LibraryLogic:` section, or the Ductile backend. With item 2, that leaves only the whitelisted +files and Ductile's own benchmark runs. + +### Deciding when a run is a test + +**What the client actually checks.** Whether timed runs happen depends on several inputs, combined +across all listeners: + +| Main-loop query (`main.cpp`) | Combined as | Who returns non-zero | +| --- | --- | --- | +| `needMoreBenchmarkRuns()` (outer loop) | any listener | `BenchmarkTimer` (`num-benchmarks`), `ReferenceValidator` (once, if enabled), `ProgressListener` (once) | +| `needMoreRunsInSolution()` (per-solution loop) | any listener | `ReferenceValidator` (until validated); `BenchmarkTimer` while `enqueuesDone < num-enqueues-per-sync × num-syncs-per-benchmark` and the solution is not skipped | +| `numWarmupRuns()` | max | `ReferenceValidator` 1 (until validated); `BenchmarkTimer` `num-warmups` | +| `numSyncs()` | max | only `BenchmarkTimer`: `num-syncs-per-benchmark` (the profiler returns 0 and is compiled out) | +| `numEnqueuesPerSync()` | max | only `BenchmarkTimer`: 0 if the solution is skipped (restore-from-log map, or skip-slow); otherwise `min(max(num-enqueues-per-sync, ceil(min-flops-per-sync / flops)), (size_t)max-enqueues-per-sync)` | + +The timed loop is `if(enq) for(i < syncs)`. So nothing is timed for a solution exactly when one of +these holds: + +- `num-syncs-per-benchmark == 0`; +- `max-enqueues-per-sync == 0`; +- `num-enqueues-per-sync == 0` **and** `min-flops-per-sync == 0`; +- the solution is skipped (`prob-sol-map` restore, or `skip-slow-solution-ratio` > 0 after a slow + warmup). + +A negative `max-enqueues-per-sync` (the default −1) casts to a huge `size_t`, so it means no cap. + +Measured on one dtl invocation (288 run solutions; `ref` build): + +| syncs | enqueues | min-flops | timed loops | rows with real `time-us` | +| --- | --- | --- | --- | --- | +| 0 | 1 | 1 | 0 | 0 (all NaN) | +| 1 | 0 | 1 | 288 | 288 | +| 1 | 0 | 0 | 0 | 0 (all NaN) | +| 1 | 1 | 1 | 288 | 288 | + +`SyncsPerBenchmark` is therefore the only parameter that turns timing off on its own. `EnqueuesPerSync: +0` does not, unless `MinFlopsPerSync` is also 0 (no YAML sets it; its default is 1). +`MaxEnqueuesPerSync` is set in 4 YAMLs (`agntab_coverage_*`), always to 1. + +**Three different questions.** "Is this a test?" mixes three separate decisions: + +1. **Is anything timed?** This decides the timing-only work: granularity and performance values, + timing events, the `validate_gpu_sync` wait, and warmups beyond what the validators need. Use + the client's own rule above, evaluated in the client from its own arguments (prototype `corr` + did this, with the wrong enqueue term). Inferring this is safe: when nothing is timed, the + skipped values are NaN by construction. +2. **Is the results CSV consumed?** This decides whether to write and copy the CSV. It is a + property of the YAML and backend, not of timing: write it if the config has `LibraryLogic:` or + uses the Ductile backend. Python knows both before it launches the client. +3. **Is this run part of the test suite?** This decides verbosity (`ClientLogLevel`) and any + test-only shortcuts. It cannot be inferred from timing parameters: 16 common YAMLs time on + purpose and are still tests (`benchmark_runs/*`, `rotate_mode*`, `icache_flush`, + `*_debug_modes`, the LibraryLogic opt-outs), and a user benchmarking with `SyncsPerBenchmark: + 0` is not running a test. The reliable signal is the harness saying so: `tensile_args` in + `Tests/conftest.py` builds the argument list for every Tensile invocation in the common tests, + so it can append a dedicated global parameter, which `ClientWriter` turns into client options. + One constraint: the conftest `--global-parameters` option forwards a single value, which `tox` + already uses for `CheckASMCodeSize=True`. So the flag should be appended by the fixture itself, + or the option made to accept several values; `--tensile-options=…,--global-parameters,X=Y` + also works, as tested for `ClientLogLevel=0`. + +Recommendation (not implemented): use rule 1 inside the client for timing-only work, rule 2 in +Python for the CSV, and an explicit harness-set parameter for rule 3. Do not key everything on +`SyncsPerBenchmark`. + +## Not measured / open items + +- **`cpu_reference_gemm` and `KMP_BLOCKTIME`.** In the dtl replay, `cpu_reference_gemm` took + 5.8–14.8 s with the default blocktime and 2.4–2.7 s with `KMP_BLOCKTIME=0`, over 3 runs each. The + Reference GEMM section is 1,080 s (42%) of the aggregate, so this deserves its own investigation. + Mechanism not established. +- **CI thread counts.** Not checked: whether CI sets `OMP_NUM_THREADS` for the client. If it does + not, libomp sizes the pool from the affinity mask, not from the pod's CPU quota, while 4 xdist + workers run clients at the same time. Spin contention would then be worse than measured here. + Inference. +- **`post_problem` (1.9 s, 68 µs mean, 50% in calls > 1 ms).** `ResultFileReporter` adds a column + per solution name, so each problem's results row grows with the solution count. Not broken down. +- **Other configs.** Only dtl was replayed. The narrow per-config spread (24–37 µs per solution, + p10–p90 weighted by solutions) suggests a similar mix elsewhere. Configs with timed iterations + (non-NaN `time-us`) exercise a few extra branches in `ResultFileReporter`. +- **The `opt` SIGSEGV window** described under option B. +- **Whole-run effect.** Not measured; GPU sharing made whole-client wall times too noisy. diff --git a/projects/hipblaslt/tensilelite/timing_runs/python_kernel_bench_setup_investigation.md b/projects/hipblaslt/tensilelite/timing_runs/python_kernel_bench_setup_investigation.md new file mode 100644 index 000000000000..9081ef042494 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/python_kernel_bench_setup_investigation.md @@ -0,0 +1,232 @@ +# `python_kernel_bench_setup` investigation (gfx950 per-config build) + +`full_runs_gfx950/per_config_build/AGGREGATE_analysis.txt` reports `python_kernel_bench_setup` at +131,638.45 ms (5.1% of build wall time) over 855 calls, mean 154 ms. This note explains what that +time is and how to remove almost all of it. Timing claims come from measurements (in-run timers, +offline replays of the exact inputs, and cProfile used only for attribution); code reading was +used to decide what to measure. Inferences are marked as such. + +## Summary + +The timer wraps one loop in `writeBenchmarkFiles` (`tensilelite/BenchmarkProblems.py:504`) that +builds the list of unique *helper kernels* (the HIP source kernels in `Kernels.cpp`: `PostGSU` +conversion, `BetaOnly`, activation, reduction). For every solution it calls +`initHelperKernelObjects(...)`, which constructs every helper object that solution would need, +then keeps only the ones whose name has not been seen yet. + +- **Almost all constructed helpers are thrown away.** A benchmark step has one `ProblemType`, and + the helpers depend only on it and on a handful of solution fields, so thousands of solutions + produce the same 2–14 helpers. `fp16_tn` step 0: 4,096 solutions build **24,576** helper objects + to keep **9**. +- **Each helper object costs two full `deepcopy(ProblemType)`s**: once in the `init*KernelObjects` + function, and again in the `KernelWriter*` constructor. That is 3.4–37 deep copies per solution + depending on the problem type (bias types × vector widths × helper kinds). +- **One `deepcopy(ProblemType)` costs ~150 µs in the live process**, ~90% of which is the 13–15 + `DataType` members each copying their private `properties` dict. `deepcopy` is ~90% of the block + under cProfile. + +| Fix | `fp16_tn` (in-run) | `fp16_HH_BHS_bf16mfma` (in-run) | Output | +| --- | --- | --- | --- | +| baseline | 8,821 ms | 11,485 ms | — | +| **F1** `DataType.__deepcopy__` returns `DataType(self.value)` | 4,996 ms (1.8×) | 8,049 ms (1.4×) | identical | +| **F2** call `initHelperKernelObjects` once per distinct input key | 107 ms (82×) | 135 ms (85×) | identical | +| **F1 + F2** | **96 ms (92×)** | **79 ms (146×)** | identical | + +"Identical" means: the ordered list of helper objects passed to `writeSolutionsAndKernels` has the +same names, header text and source text (SHA-256 over all of them, every step), and every +non-`.hsaco` file in the build artifact (`TensileLibrary.dat` decompressed, `cache.yaml`, `.co`, +`Data/*.yaml`, and the cache directory names, which are content hashes) is byte-identical to the +baseline run. `.hsaco` helper objects differ between any two builds (random `__hip_cuid_*`); their +input source is covered by the helper-source hash. + +**Projected effect on the aggregate (estimate, not measured end to end):** F2 leaves ~25–65 µs per +solution in-run (measured on the two configs above, including building the unique helpers); +117,270 solutions × that ≈ 3–8 s, plus the 314 steps with ≤4 solutions (1.9 s today, which F2 +cannot shrink and F1 roughly halves). So ~131.6 s → roughly 5–10 s. + +## Method + +1. **Distribution.** The per-config build logs + (`/home/alvasile/timing_runs/full_runs_gfx950/per_config_build//_build.log.gz`) print + one `TIMING:python_kernel_bench_setup` line per benchmark step at the end of each run, in step + order, and one `# Actual Solutions: N / M after SolutionStructs` line per step; the *n*-th of + each were paired (`N` is the list passed to `writeBenchmarkFiles`). All 855 timers paired; the + 8 `ductile` configs print no solution count and were left with an empty count (2.6 s total). + Output: `/tmp/benchsetup/steps.txt`. +2. **Capture.** A pytest plugin (`/tmp/benchsetup/bsdump.py`, loaded with `-p bsdump` by + `/tmp/benchsetup/run_dump.sh`, the same build-only `test_config_build` invocation as the + aggregate runs) wraps `BenchmarkProblems.writeBenchmarkFiles` and `timing_context`. Per step it + pickles the exact `solutions`, `isaInfoMap`, assembler path and `debugConfig`; optionally + cProfiles the block; and records wall time, main-thread CPU time (`time.thread_time`), GC pause + time (`gc.callbacks`), live threads and in-process `deepcopy(ProblemType)` cost. It also hashes + every helper object's name + header + source as passed to `writeSolutionsAndKernels`. + Configs captured: `fp16_tn`, `cgemm`, `zgemm`, `gfx950/dtl`, `fp16_HH_BHS_bf16mfma`, + `gfx950/bf16_gradient_bias`, `gfx950/spmm_f16_sb` (45 steps). +3. **Replay.** `/tmp/benchsetup/replay.py` re-runs the exact loop on a pickled step, counts objects + and `deepcopy` calls, and times candidate replacements with an exact output check (ordered list + of name, header text, source text). +4. **In-run A/B.** The plugin can apply F1 and/or F2 by monkeypatching (no repo edits); + `BSDUMP_NOPROF=1` disables the profiler. Each config ran baseline / F1 / F2 / F1+F2 back to back + on the same idle cores (`fp16_tn` on 87–104, `fp16_HH_BHS_bf16mfma` on 160–177); + `/tmp/benchsetup/compare.py` diffs timers, helper hashes and artifacts. + +## Distribution in the aggregate + +Cost scales with the number of solutions in a step: 117,270 solutions in 855 steps, **1.1 ms per +solution** overall. + +| Solutions in step | Steps | Solutions | Time | +| --- | ---: | ---: | ---: | +| 1–4 | 314 | 571 | 1.9 s | +| 5–50 | 340 | 6,315 | 12.0 s | +| 51–500 | 148 | 24,494 | 30.1 s | +| >500 | 53 | 85,889 | 85.8 s | + +Top configs: `cgemm` 17.4 s (22,880 solutions), `gfx950/dtl` 15.4 s, `bufferLoadStore0` 9.8 s, +`zgemm` 9.3 s, `fp16_HH_BHS_bf16mfma` 8.9 s (2,048 solutions, 4.3 ms each), `spmm_f16_sb` 8.0 s, +`fp16_tn` 7.8 s. Per-solution cost ranges from ~0.5 ms (`zgemm`) to ~4.3 ms +(`fp16_HH_BHS_bf16mfma`); steps with 1–2 solutions in the gradient/bias configs still cost +~9–33 ms. + +## Where the time goes + +### What the loop builds versus what it keeps (replay, exact counts) + +| Step | Solutions | Helper objects built | Unique kept | Top-level `deepcopy`/solution | Distinct `ProblemType`s | Distinct F2 keys | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| `fp16_tn` step 0 | 4,096 | 24,576 | 9 | 16.4 | 1 | 4 | +| `fp16_HH_BHS_bf16mfma` step 0 | 2,048 | 25,600 | 12 | 35.5 | 1 | 2 | +| `spmm_f16_sb` step 0 | 2,608 | 15,648 | 9 | 24.2 | 1 | 4 | +| `cgemm` step 0 | 2,480 | 6,820 | 3 | 5.4 | 1 | 4 | +| `zgemm` step 0 | 2,112 | 3,696 | 2 | 3.4 | 1 | 4 | +| `gfx950/dtl` step 3 | 1,984 | 6,944 | 4 | 6.0 | 1 | 2 | +| `bf16_gradient_bias` step 0 | 2 | 28 | 14 | 37.0 | 1 | 2 | + +Every one of the 45 captured steps had exactly one distinct `ProblemType` and 1–4 distinct F2 keys. + +### The profile + +cProfile of `fp16_tn` step 0 in-run (26.6 s profiled versus 8.6 s unprofiled; used only for +attribution): `initHelperKernelObjects` 26.4 s cumulative, of which `copy.deepcopy` 24.3 s +(65,536 top-level calls recursing into 17.2 M). Split by helper kind: conversion (`PostGSU`) +12.7 s, `BetaOnly` 4.2 s, activation enum header 4.0 s, activation function 4.0 s. The per-call +`sorted(result, key=lambda x: ("Enum" in x.getKernelName(), result.index(x)))` is 1.5 s. +`KernelWriterAssembly(...)` and `cmdLineArchsFor` do not appear (negligible). + +### Why one `deepcopy(ProblemType)` is expensive + +`ProblemType.state` has 90 keys; 13–15 values are `DataType` objects. A `DataType` is an int plus +`properties`, a pointer to a row of the class-level table `DataType.properties` +(`Common/DataType.py:63-77`). `deepcopy` cannot know that row is shared and immutable, so every +copy duplicates the dict; after the first copy each `DataType` owns a private dict, and later +copies duplicate those too. + +| Measurement | Value | +| --- | ---: | +| `deepcopy(ProblemType)` in the live build process (`fp16_tn`, two runs) | 151.5–155.5 µs | +| same object after a pickle round trip (replay) | 72–80 µs (fp16/zgemm), 112–130 µs (cgemm/dtl) | +| `deepcopy(DataType)` (replay) | 7.3 µs, of which the `properties` dict 2.45 µs | +| replay `deepcopy(ProblemType)` with `DataType.__deepcopy__ = DataType(value)` | 41.6 µs (from 77.5) | +| replay, `__deepcopy__` returning `self` | 36.3 µs | + +The replay understates the in-run cost about 2×. In the live process `dt.properties is +DataType.properties[dt.value]` is `False` for the `ProblemType`'s members (verified in-run): they +were deep-copied earlier, so 13 `DataType`s carry 13 private dicts. Pickling a solution list +re-shares them (3 distinct dicts for 13 `DataType`s in the replayed object), and `deepcopy`'s memo +then copies each shared dict once. All speedups in the summary table are therefore from in-run +measurements; replay numbers are only used for counts and equivalence. + +### Ruled out + +- **GC:** in-run, GC pauses inside the block are 147–167 ms of 8.6–8.7 s (1,026–1,029 collections) + for `fp16_tn`, 309 ms of 11.5 s for `fp16_HH_BHS_bf16mfma`. +- **GIL / descheduling:** main-thread CPU time equals wall time to within 0.1 ms on every large + step. Live threads during the block are only `MainThread` plus the + `concurrent.futures` `ExecutorManagerThread`/`QueueFeederThread`. +- **`KernelWriterAssembly` construction, `getKernels()`:** not visible in the profile; F2 keeps + both and the remaining in-run time is 25–66 µs per solution. + +## Fixes + +### F2. Build helpers once per distinct input (the main fix) + +`initHelperKernelObjects` and the functions it calls read only these solution fields: +`ProblemType`, `GlobalSplitU`, `GlobalSplitUAlgorithm`, `isStreamK(solution)`, `StreamKAtomic`, +`_GlobalAccumulation`, `AdaptiveGemmGSUA`, `ActivationFused`, `WavefrontSize`, `ISA` (plus the +global `internalParameters["GlobalSplitUPGR"]`). The prototype memoizes the result on + +```python +key = (kind, index of an equal ProblemType in a small per-step list, + s["GlobalSplitU"], s["GlobalSplitUAlgorithm"], isStreamK(s), s.get("StreamKAtomic"), + s["_GlobalAccumulation"], s.get("AdaptiveGemmGSUA"), s["ActivationFused"], + s["WavefrontSize"], tuple(s["ISA"])) +``` + +and returns the cached list for a repeated key; the existing name-based dedupe then skips them, so +the kept list and its order are unchanged. `ProblemType` equality (`__eq__` compares the state +dicts) is checked against a list that in practice holds one entry. + +Measured: in-run 82× (`fp16_tn`) and 85× (`fp16_HH_BHS_bf16mfma`); replay 1–180× across all 45 +captured steps (smaller ratios only on steps with very few solutions), identical output on all 45. + +Implementation notes (inference): the key must be kept in sync with the fields the helper +writers read; a safer variant is to compute the key inside `KernelHelperNaming` next to the code +that reads the fields. The memo should be scoped to one `writeBenchmarkFiles` call. + +### F1. Copy `DataType` by value + +```python +def __deepcopy__(self, memo): + return DataType(self.value) +``` + +A freshly constructed `DataType` is what the code makes everywhere else, and its `properties` +points at the shared class row again. Grep found no code that assigns `.value`, `.properties` or +`properties[...]` on a `DataType` instance outside `__init__`. Measured: in-run 1.4–1.8× on this +block alone, and it is the only fix that helps the 1–4-solution steps (where every helper built is +unique). Because it changes every `deepcopy` that touches a `DataType`, it also affects other +stages; in one run each, `python_benchpost_lib_construction` dropped 3,416 → 2,938 ms (`fp16_tn`) +and 2,647 → 2,327 ms (`fp16_HH_BHS_bf16mfma`), with byte-identical artifacts. Other timers moved +within run-to-run noise; those side effects were not measured with repeats. + +### Smaller items (not needed once F2 is in) + +- Each helper is deep-copied twice (`init*KernelObjects` copies, mutates, then the + `KernelWriter*` constructor copies again). Removing one copy would halve the cost; not measured + separately because F2 makes it irrelevant here. +- The per-solution `sorted(..., key=... result.index(x))` calls `getKernelName()` and an O(n) + `index` for every element (1.5 s profiled in `fp16_tn` step 0); with F2 it runs once per key. + +### Not recommended: reuse `TensileCreateLibrary.generateKernelHelperObjects` + +TensileCreateLibrary already avoids most of this cost: `generateKernelHelperObjects` +(`TensileCreateLibrary/Run.py:897`) computes names with cheap `*KernelNames` functions and only +builds objects when a name is new. On the replayed steps it was 3–69× faster and produced the same +*set* of helpers, but in a **different order** (it sorts the enum header first globally), which +changes `Kernels.cpp` text and therefore the helper-cache key. More importantly, its name functions +do not compute the same names as the objects (verified on the replayed solutions): + +| Solution | `conversionKernelNames` | names of built objects | +| --- | --- | --- | +| `fp16_tn` step 0 | `..._PostGSU1_VW1/2/4` | `..._PostGSU16_VW1/2/4` | +| `cgemm` step 0 (`ComplexFloat`) | `Cijk_SC_PostGSU1_VW1/2/4` (3) | `Cijk_SC_PostGSU16_VW1/2` (2) | +| `zgemm` step 0 (`ComplexDouble`) | 3 names | 1 object | + +The names use the solution's `GlobalSplitU` rather than `GlobalSplitUPGR`, and the vector-width +lists disagree (`isDouble()` versus `numBytes()`); `MultipleBufferSingleKernel` also ignores +`AdaptiveGemmGSUA`. The function dedupes on these proxy names and aligns them with the objects +positionally (`itertools.compress`), which happened to give the right set on every captured step +but is not guaranteed to. This is also worth a separate look for the TensileCreateLibrary path +itself (not investigated further here). + +## Not measured / open items + +- End-to-end aggregate after the fixes (the projection above is an estimate from two configs). +- F1's effect on `python_kernel_codegen`, `python_solution_generation` and the other stages that + deep-copy solutions, with repeats; single runs showed no change beyond noise except + `python_benchpost_lib_construction`. +- Whether the TensileCreateLibrary name/object mismatch above can drop or misselect a helper for + some configuration. +- Baseline in-run times in these A/B runs (8.8 s, 11.5 s) are higher than the same steps in the + aggregate (7.7 s, 8.9 s): different cores and host load. Comparisons are only made within one + core set and back-to-back runs. diff --git a/projects/hipblaslt/tensilelite/timing_runs/python_kernel_build_src_co_investigation.md b/projects/hipblaslt/tensilelite/timing_runs/python_kernel_build_src_co_investigation.md new file mode 100644 index 000000000000..6781487c0a27 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/python_kernel_build_src_co_investigation.md @@ -0,0 +1,530 @@ +# `python_kernel_build_src_co` investigation (gfx950 per-config build) + +`full_runs_gfx950/per_config_build/AGGREGATE_analysis.txt` reports `python_kernel_build_src_co` at +1,007,817.04 ms (39.1% of build wall time) over 855 calls. Essentially all of it is +`python_kernel_build_src_co.compile`: 1,004,724.33 ms over the 263 calls that missed the helper +kernel cache (mean 3.8 s). The cache already removes duplicate compiles, and PR #13671 overlaps +the compile with other work. This note looks at the work that is left: why a single compile takes +3–10 s, and what would make it substantially cheaper. Timing claims come from measurements; code +reading was only used to decide what to measure. Inferences are marked as such. + +## Summary + +| Option | Wall time | CPU time | Code produced | Notes | +| --- | --- | --- | --- | --- | +| **A. Build one xnack-agnostic object instead of `xnack+` and `xnack-`** | **−47 to −53%** | **−47 to −53%** | identical to today's `xnack+` | loader already tries the bare name first; asm kernels already ship this way | +| B. `-parallel-jobs=2` (if both variants are kept) | −45 to −50% | unchanged | identical | one flag | +| C. Split `Kernels.cpp` into ~4 chunks, compile in parallel, link once | −71 to −82% alone; −72 to −83% with A | +34 to +93% alone; −7 to −38% with A | identical | needs header-defined kernels moved into one TU | +| D. Make the `PostGSU16` reduction kernels smaller | up to −48 to −59% of the remaining work | same | **different** | generator change; runtime perf must be validated | +| E. Cached precompiled header of the HIP headers | −0.3 s per target per compile (−7 to −19%) | same | identical | removes most of the per-file fixed cost; works with today's build layout; makes C cheap in CPU | + +### All results + +Wall / CPU time of one helper-kernel compile, cores 0–15 idle. "Chunks" means `Kernels.cpp` +split into that many translation units, compiled in parallel (2 jobs per chunk for two targets, +1 per chunk for one target, capped at 16), then linked once per target. + +| Option | Targets | `gsuasb` | `bias_sb` | `spmm_bf16_sb` | `grad_bias` | Code | +| --- | --- | --- | --- | --- | --- | --- | +| Today | `xnack+`, `xnack-` | 3.07 / 3.11 s | 6.09 / 6.13 s | 6.96 / 7.01 s | 9.35 / 9.40 s | — | +| B: `-parallel-jobs=2` | `xnack+`, `xnack-` | 1.57 / 3.12 s | 3.33 / 6.69 s | 3.47 / 6.97 s | 4.77 / 9.47 s | identical | +| C: 4 chunks | `xnack+`, `xnack-` | 0.88 / 6.00 s | 1.27 / 9.60 s | 1.35 / 10.25 s | 1.70 / 12.59 s | identical | +| C: 8 chunks | `xnack+`, `xnack-` | 0.82 / 10.26 s | 1.30 / 14.09 s | 1.15 / 15.07 s | 1.64 / 17.44 s | identical | +| A: one xnack-agnostic object | `gfx950` | 1.62 / 1.65 s | 3.07 / 3.09 s | 3.26 / 3.29 s | 4.58 / 4.60 s | identical to `xnack+` | +| A + C: 2 chunks | `gfx950` | 1.06 / 2.05 s | 1.74 / 3.44 s | 1.90 / 3.70 s | 2.61 / 4.91 s | identical | +| **A + C: 4 chunks** | `gfx950` | **0.85 / 2.87 s** | **1.10 / 4.36 s** | **1.17 / 4.52 s** | **1.60 / 5.86 s** | identical | +| D bound: one target, `PostGSU16` kernels removed | one | 0.90 s | 1.40 s | 1.60 s | 2.02 s | n/a (kernels missing) | + +The D row is not a proposal: it removes the `PostGSU16` kernels entirely, to bound what shrinking +them could save (compare one target with them: 1.72, 3.18, 3.43 and 4.98 s). It was measured +separately from the other rows (median of 3). + +Not worth pursuing (measured): lowering `-O` (slower), `-nogpulib` (link fails, no gain), and a +per-kernel cache (would skip 6% of kernel compiles beyond today's whole-file cache), and keeping a +compiler process alive (process start-up is only ~7–15 ms; see E). + +## Relation to PR #13671 + +Checked against `origin/users/talumbau/tcl-speed-1-helper-overlap` as fetched on 2026-10-29 +(`27a83902aa7` "compile helper kernels while generating asm kernels", `d6301b0de2e` tests). + +- **What it changes.** In `writeSolutionsAndKernelsTCL` (the TensileCreateLibrary path), it + submits `buildSourceCodeObjectFiles` to a one-thread `ThreadPoolExecutor` before assembly + codegen starts and joins it after `buildAssemblyCodeObjectFiles`. The compile itself is + unchanged: still both xnack targets, one sequential clang job. The commit message cites + ~175 s for one gfx942 `Kernels.cpp`. +- **It does not touch the timer measured here.** `python_kernel_build_src_co` is recorded in + `writeSolutionsAndKernels`, the benchmark path called from `BenchmarkProblems.py`. On the PR + branch that function still runs codegen, assembly, `writeHelpers`, the assembly link and only + then the helper compile, serially. +- **The same overlap in the benchmark path would hide at most 24%.** `writeHelpers` there only + needs `kernelHelperObjs`, which is available on entry, so the compile could also start before + codegen. Per `python_kernel_compilation` call in the 186 production logs, the hidden time is at + most min(helper compile, codegen + validate + write_assemble + build_co of the same call): + + | Compile cost | Total compile | Upper bound hidden by overlap | Still exposed | + | --- | --- | --- | --- | + | Today | 1,004.7 s | 237.7 s (24%) | 767.0 s | + | With A (× 0.5) | 502.4 s | 194.6 s | 307.8 s | + | With A + C, 4 chunks (× 0.22) | 221.0 s | 143.6 s | 77.4 s | + + Helper compiles happen in only 263 of the 855 calls (the rest hit the cache), while the 920.8 s + of assembly work is spread over all 855; only 17 of the 263 compiles are shorter than their + call's assembly work. These are upper bounds (inferred): codegen runs on a worker pool, so a + concurrent compile competes with it for cores, especially a chunked one. +- **The options here apply to the PR's path too.** The TCL path compiles the same kind of + `Kernels.cpp` with the same `Compiler`, so A and B halve that ~175 s job and C splits it + across cores, independently of the overlap (inferred; the gfx942 library build was not + measured here). + +## Method + +- **Captured compiles.** Four configs re-built with `--build-only` exactly as + `run_per_config_build.sh` does, with `TENSILE_DISABLE_HELPER_CACHE=1` and + `Tensile_CXX_COMPILER_LAUNCHER` pointing at a wrapper that saves `Kernels.cpp`, the headers in + its include directory, and the argv, then runs the command unchanged. The captured times match + production: + + | Config | Kernels in `Kernels.cpp` | Lines | Captured compile | Production mean | + | --- | --- | --- | --- | --- | + | `gemm/gsuasb` | 32 | 28,377 | 2.95–3.12 s | ~3.0 s | + | `gemm/gemm_bias_strided_batched_gfx940` | 50 | 55,990 | 5.98–6.47 s | 5.25 s | + | `sparse/gfx950/spmm_bf16_sb` | 50 | 56,806 | 6.19–6.48 s | 6.23 s | + | `gradient/gfx950/bf16_gradient_bias` | 74 (+14 in `Kernels.h`) | 84,062 | 9.20–9.83 s | 9.33 s | + +- **Experiments** re-run the captured command (`amdclang++ -D__HIP_HCC_COMPAT_MODE__=1 + --offload-device-only -x hip -O3 -Xoffload-linker --build-id=sha1 -std=c++17 + --offload-arch=gfx950:xnack+ --offload-arch=gfx950:xnack-`, AMD clang 22.0.0git, ROCm 7.2.1) + with one change at a time, on cores 0–15, which were idle during these runs (the host is + shared). +- **Code equivalence** is checked per kernel: disassembly with addresses and comments stripped + and each function cut after its last `s_endpgm` (inter-function alignment padding moves with + function order), plus register, LDS, scratch and spill counts from the code-object metadata. + Whole files are never byte-identical, even between two identical compiles: each compile gets a + random `__hip_cuid_*` symbol and its own build ID. +- Scratch scripts and captures: `/tmp/syncval_bench/kcap` (`launcher.sh`, `run_build.sh`, + `cc.sh`, `chunkbuild.py`, `perkernel.py`, captures in `/NNN/`). + +## Distribution in the aggregate + +From the 186 per-config build logs (184 configs compile at least once): 263 compiles, median +3.1 s, p10 2.8 s, p90 6.1 s, max 9.5 s. 183 of them take 2–4 s; a second group of 33+29 takes +5–7 s (bias, grouped, sparse), and 7 take 8–10 s (gradient + bias). No single config dominates: +the largest, `subtile_mxfp4`, is 29.9 s over 9 compiles. + +The number of distinct (`Kernels.cpp`, `Kernels.h`) pairs in each captured config equals its +production compile count (2, 3, 2, 3), so the existing cache already removes every exact +duplicate. + +## What `Kernels.cpp` contains + +Generated HIP C++ helper kernels: GSU reduction/conversion kernels (`*_PostGSU{16,8,4,2,1}_VW{4,2,1}`, +each with a grouped `_GG_` twin), global-accumulation kernels (`*_GA`), and for bias gradients, +reduction kernels. `KernelHelperNaming.initConversionKernelObjects` creates one conversion object +per load vector width; each expands into the whole `PostGSU` family, which the runtime chooses +from by the actual GSU value. All kernels are `extern "C"`; macros are `#define`d and `#undef`d per +kernel. `Kernels.h` holds the argument structs and declarations, and for bias gradients also +defines `Dijk_*_Reduction` kernels. + +## Where the time goes + +### 1. Every kernel is compiled twice (50%) + +`--gpu-targets gfx950` expands to `gfx950:xnack+` and `gfx950:xnack-` +(`Common/Architectures.py`), and the driver compiles the device code once per target, +sequentially, front end included: + +| Config | Both variants | One variant | +| --- | --- | --- | +| `gsuasb` | 3.16 s | 1.56 s | +| `grad_bias` | 9.27 s | 4.64 s | + +### 2. A fixed ~0.4–0.7 s per target, almost all HIP headers + +Single target, `gsuasb` include directory: + +| Translation unit | Time | +| --- | --- | +| empty | 0.23–0.24 s | +| `#include ` | 0.33 s | +| + `` | 0.39 s | +| `#include ` | 0.24 s | +| `#include "Kernels.h"` | 0.41–0.42 s | +| full `Kernels.cpp`, `-fsyntax-only` | 0.44 s | + +The headers cost ~0.42 s (`gsuasb`) to ~0.67 s (`grad_bias`, larger `Kernels.h`); parsing the +kernels themselves is cheap. `-nogpulib` does not change the empty-TU time and makes the link +fail (the kernels need `ocml`/`ockl`). + +The "empty" row is not start-up: it includes `__clang_hip_runtime_wrapper.h`, which the driver +adds to every HIP file. Process start-up, driver, compiler start-up, device libraries, link and +bundle together are ~21 ms. Of the ~400 ms, ~350 ms is HIP headers and ~25 ms TensileLite static +headers; the per-header breakdown and how to remove it are in option E. + +### 3. The rest is optimization and machine code generation, proportional to code size + +`-ftime-report`, `gsuasb`, one target (1.54 s): machine code generation 0.69 s (45%), front end +0.46 s (30%), optimizer 0.35 s (22%), IR generation 0.05 s (3%). The largest passes are AMDGPU +instruction selection (0.19 s), `InstCombine` (0.13 s) and register coalescing (0.07 s); no single +pass is pathological. + +Lower optimization makes it slower, because unoptimized code is more expensive to lower: + +| Config, one target | -O0 | -O1 | -O2 | -O3 | +| --- | --- | --- | --- | --- | +| `gsuasb` | 2.86 s | 2.51 s | 1.55 s | 1.56 s | +| `grad_bias` | 8.19 s | 7.39 s | 4.53 s | 4.65 s | + +### 4. The `PostGSU16` kernels are about half of it + +The "universal" `PostGSU16` kernels (non-`UnrollOnly`, `GlobalSplitUPGR` = 16) are 4,260–5,865 +lines each, against 100–145 lines for `PostGSU8/4/2/1`. `KernelWriterConversion.kernelBody` +generates a software-pipelined reduction: 16 prefetched loads, a `while` loop over full groups of +16, then a `switch(gsuRemain)` with 15 fully unrolled cases, each re-emitting all 16 accumulate +steps plus its interleaved loads, with every inline-asm add duplicated under `#if +defined(__gfx950__)`/`#else`. One kernel contains 392 inline-asm blocks; code size grows roughly +with GSU². Compiled alone, a `PostGSU16_VW4` kernel costs 210–226 ms, `VW2` 94–117 ms, `VW1` +80–92 ms, while a `PostGSU8` kernel costs 3–37 ms. + +Removing just these kernels from each file (one target, median of 3): + +| Config | Full | Without `PostGSU16` | `PostGSU16` share | `PostGSU16` kernels | +| --- | --- | --- | --- | --- | +| `gsuasb` | 1.72 s | 0.90 s | 48% | 6 of 32 | +| `bias_sb` | 3.18 s | 1.40 s | 56% | 12 of 50 | +| `spmm_bf16_sb` | 3.43 s | 1.60 s | 53% | 12 of 50 | +| `grad_bias` | 4.98 s | 2.02 s | 59% | 18 of 74 | + +## Options + +### A. Build one xnack-agnostic helper object + +Compiling with `--offload-arch=gfx950` (no xnack feature) produces one code object that loads in +either xnack mode. + +- **Time.** `gsuasb` 1.62–1.65 s against 3.07–3.16 s today; `bias_sb` 3.07 s against 6.09 s; + `spmm_bf16_sb` 3.26 s against 6.96 s; `grad_bias` 4.58 s against 9.35 s. CPU time halves too. +- **Code.** All 32 `gsuasb` kernels are identical to today's `xnack+` object. Against `xnack-`, + 17 kernels differ by `s_nop` padding: 720 against 714 `s_nop 0` in 26,771 instructions. +- **Loading.** `SolutionAdapter::initializeLazyLoading` already tries + `Kernels.so-000-.hsaco` before the `-xnack-` and `-xnack+` names. +- **Consistency.** The main assembly kernels already ship this way: `TensileLibrary_gfx950.co` + has ELF flags `gfx950, xnack, sramecc` (agnostic), while the helpers are + `Kernels.so-000-gfx950-xnack+.hsaco` (`xnack+`) and `-xnack-.hsaco` (`xnack-`). +- **Risk.** On `xnack-` systems the helpers would run the `xnack+`-style code (the extra + `s_nop`s). The runtime effect was not measured. If shipped libraries must keep both variants, + this could apply to benchmark/test builds only. + +### B. Compile the two variants in parallel + +`-parallel-jobs=2` makes the driver run the two device compiles concurrently. `--offload-jobs=2` +is ignored by this driver path ("argument unused"). + +| Config | Today | `-parallel-jobs=2` | +| --- | --- | --- | +| `gsuasb` | 3.07 s wall, 3.1 s CPU | 1.57 s wall, 3.1 s CPU | +| `bias_sb` | 6.09 s, 6.1 s | 3.33 s, 6.7 s | +| `spmm_bf16_sb` | 6.96 s, 7.0 s | 3.47 s, 7.0 s | +| `grad_bias` | 9.35 s, 9.4 s | 4.77 s, 9.5 s | + +The disassembly is identical to the sequential build. Not needed if A is adopted. + +### C. Split the file and compile the pieces in parallel + +The kernels are independent, so `Kernels.cpp` can be split into N translation units that each +include `Kernels.h`. Each (chunk, target) is compiled to a relocatable object, and one `lld +-shared` link per target produces the single `Kernels.so-000-*.hsaco` the runtime loads (the +driver already does exactly this compile-then-link for one TU). The experiment ran the driver's +own `cc1` and `lld` commands (from `-###`) in a process pool, with chunks balanced by line count. + +- **Code.** Every kernel identical, with identical metadata, in every case tested: 32, 50, 50 and + 88 kernels, both targets. +- **Header-defined kernels.** `grad_bias`'s `Kernels.h` defines 14 `Dijk_*_Reduction` kernels; + with more than one chunk, the link fails with duplicate symbols. The experiment guarded them + so only chunk 0 defines them; a real change should move them into the `.cpp`. + +Both targets (wall / CPU): + +| Config | Today | 1 chunk, 2 jobs | 4 chunks, 8 jobs | 8 chunks, 16 jobs | +| --- | --- | --- | --- | --- | +| `gsuasb` | 3.07 / 3.1 s | 1.61 / 3.2 s | 0.88 / 6.0 s | 0.82 / 10.3 s | +| `bias_sb` | 6.09 / 6.1 s | 3.35 / 6.7 s | 1.27 / 9.6 s | 1.30 / 14.1 s | +| `spmm_bf16_sb` | 6.96 / 7.0 s | 3.27 s / — | 1.35 / 10.3 s | 1.15 / 15.1 s | +| `grad_bias` | 9.35 / 9.4 s | 4.75 / 9.4 s | 1.70 / 12.6 s | 1.64 / 17.4 s | + +Combined with A (one xnack-agnostic target): + +| Config | Today | A alone | A + 2 chunks | A + 4 chunks | +| --- | --- | --- | --- | --- | +| `gsuasb` | 3.07 / 3.1 s | 1.62 / 1.65 s | 1.06 / 2.05 s | 0.85 / 2.87 s | +| `bias_sb` | 6.09 / 6.1 s | 3.07 / 3.09 s | 1.74 / 3.44 s | 1.10 / 4.36 s | +| `spmm_bf16_sb` | 6.96 / 7.01 s | 3.26 / 3.29 s | 1.90 / 3.70 s | 1.17 / 4.52 s | +| `grad_bias` | 9.35 / 9.4 s | 4.58 / 4.60 s | 2.61 / 4.91 s | 1.60 / 5.86 s | + +Each extra translation unit repays the fixed ~0.4 s, so CPU time grows with the chunk count and +gains flatten beyond 4 chunks. In CI (4 xdist workers on 25 cores) the compile competes with the +other workers, so a small chunk count (or one chosen from free cores) is the safer default. A +cached precompiled header (option E) cuts the fixed cost per chunk from ~400 ms to ~35 ms. + +The driver has no supported switch for "compile to a relocatable device object without +linking": `--no-gpu-bundle-output` still links. The supported alternative is `-fgpu-rdc` +(bitcode per TU, machine codegen in `lld` LTO, which the driver already runs with +`--lto-partitions=8`). On `gsuasb`, one target: bitcode compile 0.98 s, then `lld` with 8 LTO +partitions 0.30 s wall (0.97 s with 1), so ~1.28 s against 1.62 s. **But none of the 32 kernels +came out identical** (26,761 against 26,771 instructions), so this route needs performance +validation; invoking `clang -cc1`/`lld` directly, or `-S` plus assembling, keeps today's code. + +### D. Shrink the `PostGSU16` kernels + +The `PostGSU16` kernels are 48–59% of each compile (table above). Their size comes from the +unrolled remainder `switch` in `KernelWriterConversion.kernelBody` (non-`UnrollOnly` path). +Generating the remainder as a loop, lowering `GlobalSplitUPGR`, or sharing one remainder path +would cut most of that, but it changes the code of a kernel on the GSU path. It is only worth doing +with a runtime benchmark of the reduction kernels showing no regression (not measured here). + +### E. A cached precompiled header for the fixed cost + +With hundreds of compiles (or chunks), the fixed per-file cost matters. It is not process +start-up: clang has no persistent server mode, and it would not help much anyway. Breakdown of a +compile of `#include "Kernels.h"` with no kernels (`gsuasb`), one target, median of 15, cores 0–7. +Headers were added one at a time in include order, so each row is that header's increment; the +rows sum to 400.3 ms against 405.1 ms measured for the whole driver run: + +| Step | Owner | Time | +| --- | --- | --- | +| process start (`amdclang++ --version`) | driver | 3.4 ms | +| driver planning the jobs (`-###`) | driver | 0.9 ms | +| compiler (`cc1`) start-up: empty file, no headers, no device libraries | compiler | 5.9 ms | +| device-library bitcode (`ocml`, `ockl`, `oclc_*`) | ROCm | 7.3 ms | +| `__clang_hip_runtime_wrapper.h` (implicit, added by the driver) | HIP | 192.7 ms | +| `` | HIP | 100.0 ms | +| `` | HIP | 0.7 ms | +| `` | HIP | 57.2 ms | +| C++ standard headers (``, ``, ``, ``, ``, ``, ``) | C++ std | 1.5 ms | +| `tensile_bfloat16.h` | TensileLite | 3.4 ms | +| `tensile_float8_bfloat8.h` | TensileLite | 8.3 ms | +| `TensileTypes.h` | TensileLite | 10.5 ms | +| `ReductionTemplate.h` | TensileLite | 1.6 ms | +| `memory_gfx.h` | TensileLite | 0.0 ms | +| `KernelHeader.h` | TensileLite | 1.1 ms | +| `Kernels.h` (generated: argument structs and declarations) | TensileLite, generated | 2.2 ms | +| `lld` (link) | linker | 3.0 ms | +| `clang-offload-bundler` | bundler | 0.8 ms | + +Subtotals: HIP 350.6 ms (88%), TensileLite static headers 24.9 ms, generated `Kernels.h` +2.2 ms, C++ standard headers 1.5 ms, everything else (process, driver, compiler start-up, device +libraries, link, bundle) 21.3 ms. + +The same breakdown with a PCH of the HIP and TensileLite static headers (`shared.pch`: implicit +wrapper, `hip_runtime.h`, `hip_ext.h`, `hip_fp8.h`, `KernelHeader.h` and everything it includes), +measured side by side in a second run (median of 15). With the PCH, its load cost shows up in the +first header row; increments within ±5 ms (including the small negative ones) are noise: + +| Step | Owner | No PCH | HIP + TensileLite PCH | +| --- | --- | --- | --- | +| process start (`amdclang++ --version`) | driver | 3.3 ms | 3.3 ms | +| driver planning the jobs (`-###`) | driver | 0.9 ms | 0.9 ms | +| compiler (`cc1`) start-up: empty file, no headers, no device libraries | compiler | 5.7 ms | 5.7 ms | +| device-library bitcode (`ocml`, `ockl`, `oclc_*`) | ROCm | 7.3 ms | 7.3 ms | +| `__clang_hip_runtime_wrapper.h` (implicit) / **loading the PCH** | HIP / PCH | 197.0 ms | **65.6 ms** | +| `` | HIP | 105.3 ms | −0.4 ms | +| `` | HIP | −3.2 ms | −0.1 ms | +| `` | HIP | 60.9 ms | 0.1 ms | +| C++ standard headers | C++ std | −0.4 ms | −0.5 ms | +| `tensile_bfloat16.h` | TensileLite | 9.9 ms | 0.7 ms | +| `tensile_float8_bfloat8.h` | TensileLite | 0.6 ms | −0.6 ms | +| `TensileTypes.h` | TensileLite | 4.8 ms | 0.5 ms | +| `ReductionTemplate.h` | TensileLite | 4.5 ms | 0.2 ms | +| `memory_gfx.h` | TensileLite | −1.3 ms | −0.4 ms | +| `KernelHeader.h` | TensileLite | 1.6 ms | 0.7 ms | +| `Kernels.h` (generated; not in the PCH) | TensileLite, generated | −1.7 ms | 2.3 ms | +| `lld` (link) | linker | 3.7 ms | 3.7 ms | +| `clang-offload-bundler` | bundler | 0.8 ms | 0.8 ms | +| **Sum** | | **399.5 ms** | **89.7 ms** | + +Measured whole driver runs: 412.9 ms without the PCH, 96.7 ms with it. Subtotals: HIP +360.0 → 65.1 ms (the PCH load), TensileLite static headers 20.0 → 1.1 ms, generated `Kernels.h` +about 2 ms either way. The per-header TensileLite numbers vary by ~5 ms between runs (compare +with the table above); their subtotal (20–25 ms) is stable. + +**Why "loading the PCH" still costs ~66 ms, and how to remove most of it.** It is not reading the +file: in a syntax-only run of an empty file with the PCH (56 ms), `-ftime-trace` shows `ReadAST` +0.5 ms, `-fno-validate-pch` changes nothing, the PCH records no input files to re-check, and +system calls are negligible (`strace -c`). The empty file contributes no functions or globals to +the module. 46.8 ms is `PerformPendingInstantiations`: the headers' inline code uses function +templates whose implicit instantiation is deferred to the end of the translation unit, and a PCH +stores that list, so every file that uses the PCH redoes them (483 `InstantiateFunction` events, +mostly C++ standard library: `std::basic_string` 24 ms inclusive, `std::vector`, `__stoa` from +`std::to_string`, `uninitialized_copy`, and the `` special functions such as +`__cyl_bessel_j`). `-fpch-instantiate-templates` performs them once while building the PCH: + +| Translation unit, one target | No PCH | PCH | PCH built with `-fpch-instantiate-templates` | +| --- | --- | --- | --- | +| empty file, `-fsyntax-only` | 192.9 ms | 56.5 ms | 14.0 ms | +| empty file, full compile | 210.2 ms | 79.0 ms | 33.6 ms | +| `#include "Kernels.h"`, full compile | 397.5 ms | 81.3 ms | 34.8 ms | + +Building that PCH costs 497 ms instead of 465 ms (13.5 MB instead of 12.9 MB), and afterwards +`PerformPendingInstantiations` is 0.0 ms per file. The HIP-only version through the normal driver +(`amdclang++ -x hip -fpch-instantiate-templates -Xclang -emit-pch -fsyntax-only +hip.h > hip.pch`), used on today's build directories, one target, median of 5: + +| Config | No PCH | HIP PCH | HIP PCH, `-fpch-instantiate-templates` | Code | +| --- | --- | --- | --- | --- | +| `gsuasb` | 1.57 s | 1.28 s | 1.24 s | 32/32 identical | +| `bias_sb` | 3.05 s | 2.76 s | 2.72 s | 50/50 identical | +| `spmm_bf16_sb` | 3.26 s | 2.96 s | 2.94 s | 50/50 identical | +| `grad_bias` | 4.61 s | 4.31 s | 4.28 s | 88/88 identical | + +`-fpch-instantiate-templates` saves another ~30–40 ms per translation unit compiled. With today's +single `Kernels.cpp` that is one translation unit per compile and target; with C (chunks) it is +saved once per chunk. + +**How the PCH is used, and when it pays off.** In all of these measurements the PCH was built +once and reused unchanged by every later compile (all four configs, every repetition); the +per-compile times above do not include building it. In practice it would be built once per +(compiler/ROCm version, flags, target) and cached, like `HelperKernelCache`, or built with the +TensileLite installation. One-time cost: ~0.5 s per target. Saving: ~0.33–0.36 s per translation +unit per target (no PCH vs instantiated HIP PCH above). Arithmetic, not measured end to end: + +- **Cache kept across configs (e.g. a CI job reusing `~/.tensile`):** built once per target for + the whole run; 263 compiles × 2 targets × ~0.33 s ≈ 170 s saved for ~1 s of PCH builds. +- **Cold cache per config, as in `run_per_config_build.sh`:** the PCH is rebuilt for every config. + The 184 configs average 263 / 184 = 1.43 compiles, i.e. ~2.9 uses with both targets + (~0.9 s saved for ~1.0 s of builds) or ~1.4 uses with option A (~0.47 s saved for ~0.5 s): + roughly break-even. With A + C at 4 chunks it is ~5.7 uses per config (~1.9 s saved for 0.5 s). + +**Caching it like the helper kernels.** The intended design: build the HIP PCH once per +(compiler build, ROCm install, flags, target), store it next to the helper-kernel cache, and have +every later TensileLite run read it. `HelperKernelCache` already has the pattern +(`~/.tensile/helper_cache` or `TENSILE_HELPER_CACHE_DIR`; SHA-256 key; write to +`.tmp__`, then `rename`, which is safe with concurrent xdist workers; age-based +eviction). What clang does with a cached PCH that does not match (measured, one target): + +| Cached PCH used with... | Result | +| --- | --- | +| the same flags (control) | works | +| a different xnack feature (`gfx950:xnack-` vs `gfx950`) | error: target feature mismatch | +| a different arch (`gfx942`) | error: target CPU mismatch | +| `-O2` instead of `-O3` | error: `OptimizationLevel differs` | +| `-std=c++20` instead of `-std=c++17` | error: C++20 disabled in the PCH | +| a header's mtime changed (content unchanged), or its content changed | fatal error: file modified since the PCH was built | +| `-fsanitize=address` added | accepted, with a warning that ASan is ignored for this target | +| a `-D` macro defined at PCH build time but not in the compile | **accepted silently; the macro stays defined in the file** | +| a `-D` macro with a different value | error: definition of macro differs | +| a `-D` macro defined only in the compile | works (normal behavior) | + +So: + +- **Key on the full compile flags**, including every `-D`, the target and its features, `-O`, + `-std`, and ASan, because a macro baked into the PCH would otherwise leak silently into files + that do not define it. Simplest: hash the exact argument list used to build the PCH (the same + `Compiler.default_args` plus the per-target flags) and use that same list for the compiles. +- **Key on the exact compiler build**, not only `major.minor.patch`: `HelperKernelCache` uses + `compiler.version` (`22.0.0` for this `22.0.0git` build), while clang rejects a PCH from a + different compiler build. Hash the full `amdclang++ --version` output, which includes the commit. +- **Keep the PCH to HIP headers.** Clang validates every header in a PCH by mtime and size, and an + mtime-only change is fatal (`-fvalidate-ast-input-files-content` did not relax it here). The + TensileLite static headers are re-copied into each build directory (`shutil.copy`, new mtime), + so a PCH containing them would fail on every build unless built and used from one fixed path. + The HIP headers sit in the ROCm install, unchanged until ROCm is reinstalled; this is also where + ~350 of the ~375 ms of header cost is. +- **Fall back on errors.** A reinstall of the same ROCm version changes header mtimes; the + compile then fails with "has been modified since the precompiled header was built". On any PCH + error, rebuild the PCH (or compile without it) and retry, so a stale cache entry costs one + rebuild, not a failed build. + +That is header parsing, which a precompiled header (PCH) removes. Splitting it with two PCHs +(cc1 jobs, one target, median of 9; `hip.pch` = implicit wrapper + `hip_runtime.h` + +`hip_ext.h` + `hip_fp8.h`, 434 ms to build, 12.5 MB; `shared.pch` = that + `"KernelHeader.h"`): + +| Translation unit | No PCH | `hip.pch` | `shared.pch` | +| --- | --- | --- | --- | +| empty file (only the implicit `__clang_hip_runtime_wrapper.h`) | 205 ms | 69 ms | 76 ms | +| `hip_runtime.h`, `hip_ext.h`, `hip_fp8.h` | 364 ms | 70 ms | 77 ms | +| `#include "Kernels.h"` (`gsuasb`) | 389 ms | 100 ms | 78 ms | + +(Empty file with the implicit wrapper removed: 13 ms.) So the fixed cost is almost all HIP +headers: ~192 ms for the wrapper the driver adds to every HIP file and ~160 ms for the +`hip_runtime.h`/`hip_ext.h`/`hip_fp8.h` that Tensile's headers include, ~350 ms in total. The +Tensile static headers themselves add only ~25 ms. (`grad_bias`'s `Kernels.h` costs more, 626 ms, +because it contains kernel code.) With a PCH, ~70 ms remains: loading the PCH plus an empty +compile. + +**The HIP-only PCH is the one to use.** It gets almost all of the saving, depends only on the +ROCm/clang version, flags and target (the headers live at fixed `/opt/rocm` paths), and works with +today's build directories unchanged. Built with the normal driver and used on the captured build +directories as they are (static headers copied next to `Kernels.h`), one target: + +| Config | No PCH | HIP-only PCH | Code | +| --- | --- | --- | --- | +| `gsuasb` | 1.54 s | 1.25 s | 32/32 identical | +| `bias_sb` | 2.99 s | 2.71 s | 50/50 identical | +| `spmm_bf16_sb` | 3.18 s | 2.91 s | 50/50 identical | +| `grad_bias` | 4.53 s | 4.21 s | 88/88 identical | + +Adding the Tensile static headers to the PCH saves another ~10–50 ms per compile (table below), +but requires changing where those headers are found (see "Header location"). + +Only `Kernels.h` is generated (`writeHelpers`; 10 distinct versions across the 27 captured +compiles). The other six headers (`TensileTypes.h`, `tensile_bfloat16.h`, +`tensile_float8_bfloat8.h`, `KernelHeader.h`, `ReductionTemplate.h`, `memory_gfx.h`) are static +files shipped in the package at `tensilelite/Source/`. `copyStaticFiles` → +`resources.copy_static_headers` copies them verbatim (`shutil.copy`) next to `Kernels.cpp` for +every build, so they are identical by construction; they only change when the TensileLite +checkout changes. The HIP headers come from the ROCm install. So one PCH of +``, `` and `"KernelHeader.h"` per target can serve every config +built with the same TensileLite and ROCm. Built once (463 ms, 12.9 MB) from a shared directory of +the static headers, then used for each config's whole `Kernels.cpp`, one target: + +| Config | `#include "Kernels.h"` only | Whole `Kernels.cpp` | Code | +| --- | --- | --- | --- | +| `gsuasb` | 389 → 78 ms | 1.53 → 1.22 s | 32/32 identical | +| `bias_sb` | 392 → 82 ms | 2.97 → 2.66 s | 50/50 identical | +| `spmm_bf16_sb` | 392 → 81 ms | 3.18 → 2.89 s | 50/50 identical | +| `grad_bias` | 626 → 318 ms | 4.53 → 4.20 s | 88/88 identical | + +- **With chunks (C).** Each chunk pays ~80 ms instead of ~390 ms of headers. Measured with a + per-config PCH and 4 chunks, one target: total compile CPU 2.72 → 1.50 s (`gsuasb`) and + 4.26 → 2.98 s (`bias_sb`), all kernels identical. This removes most of C's extra CPU cost. +- **Through the normal driver.** Build: `amdclang++ -x hip -Xclang -emit-pch + -fsyntax-only hip.h > hip.pch`, where `hip.h` includes ``, + `` and `` (`-x hip-header` is rejected, and with + `-fsyntax-only` the PCH is written to stdout, not `-o`). Use: add `-include-pch hip.pch`. +- **Caching.** A PCH is only valid for the same compiler, flags (`-O3`, `-std`, `-D`), target and + xnack feature, and header contents, so it would be cached per target keyed on those (e.g. the + ROCm/clang version plus a hash of `tensilelite/Source/*.h`), like `HelperKernelCache`. +- **Header location (only for the Tensile part).** Today the copies sit next to `Kernels.h`, and `Kernels.h` includes + `"KernelHeader.h"` with quotes, so the compile resolves to the per-build copy. For the PCH to be + used, the compile must resolve the static headers to the same files the PCH was built from: + build the PCH against one fixed directory (e.g. the package's `Source/`) and have the compile + find the static headers there, not in per-build copies. `KernelHeader.h`, `TensileTypes.h` and + `memory_gfx.h` use `#ifndef` guards, which would hide a duplicate copy, but + `ReductionTemplate.h`, `tensile_bfloat16.h` and `tensile_float8_bfloat8.h` use `#pragma once`, + which is per file, so a second copy at another path would be parsed again. In the experiment + `Kernels.h` was in its own directory and the static headers only in the shared one. +- **Aggregate (inferred).** ~0.3 s per target per compile × 263 compiles × 2 targets ≈ 160 s of + the 1,004.7 s, before any chunking; more with chunks, since every chunk saves it. + +## Measured and not worth pursuing + +- **Lower `-O`:** -O0/-O1 are 1.6–1.8× slower than -O3; -O2 is the same as -O3. +- **`-nogpulib`:** no change in the fixed cost, and the link fails (undefined `ocml`/`ockl` + symbols). +- **Per-kernel cache:** across the distinct files compiled for each captured config, 474 of 506 + kernel blocks are unique; a per-kernel cache would skip 32 (6%) beyond today's whole-file cache. + Cross-config reuse was not measured: these 4 configs share almost nothing, and production builds + were cold per config. + +## Not measured / open items + +- Runtime performance of the xnack-agnostic helpers on an `xnack-` system (option A), and of + `-fgpu-rdc` code (option C variant) or a smaller `PostGSU16` (option D). +- End-to-end build-time effect, and the real (contended) overlap if PR #13671's approach is + extended to the benchmark path, under CI's xdist load. +- The gfx942 TensileCreateLibrary `Kernels.cpp` (~175 s per the PR) was not captured. +- A precompiled header combined with chunking end to end (measured: per-config PCH with 4 + chunks, and the instantiated HIP PCH on whole files, separately); the PCH cache itself is a + design, not implemented. +- Only 4 of the 184 configs were captured; the per-config distribution above suggests the rest + are the same kind of files (most compiles 2–4 s), but they were not inspected individually. diff --git a/projects/hipblaslt/tensilelite/timing_runs/python_kernel_codegen_investigation.md b/projects/hipblaslt/tensilelite/timing_runs/python_kernel_codegen_investigation.md new file mode 100644 index 000000000000..475a5127cb7a --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/python_kernel_codegen_investigation.md @@ -0,0 +1,508 @@ +# `python_kernel_codegen` investigation (gfx950 per-config build) + +`full_runs_gfx950/per_config_build/AGGREGATE_analysis.txt` reports `python_kernel_codegen` at +592,584.11 ms (23.0% of build wall time) over 855 calls, mean 693 ms. This note explains where +that time goes and what can be saved. Timing claims come from measurements: in-run probes of the +real builds, offline replays of the exact kernels, py-spy with native frames, cProfile (ranking +only) and in-run A/B runs with artifact comparison. Code reading was used to decide what to +measure. Inferences and estimates are marked as such. + +## Summary + +The timer wraps one call (`TensileCreateLibrary/Run.py:666`): `ParallelMap2(processKernelSource, +...)`, which generates the assembly text of every kernel of one benchmark step in a joblib/loky +pool (25 processes in these runs) and returns the text to the parent. Pool start-up, dispatch and +result return are **not** the cost: 2–12 ms before the first kernel starts and 2–85 ms after the +last one ends, per step (135 ms on a config's first step, when the pool starts). The time is the +generation of each kernel, and how much of it can run in parallel. + +1. **Generating one kernel costs 0.04–3 s of single-threaded Python + rocisa C++, proportional to + the assembly it produces (~90–280 ms per MB; kernels are 0.4–7 MB of text).** Of that, about 65% + is Python building the instruction tree (the store/epilogue code alone is 40–64%), ~21% is the + C++ `rocIsaPass`, ~8% freeing the tree and ~6% `str()` of the tree. +2. **19 configs set `CpuThreads: 1`, so codegen runs in one process.** They are 189.3 s of the + 592.6 s (32%), including all 13 `subtile_*` configs (169.4 s). `subtile_mxfp4`: 133.5 s → 22.6 s + in-run with 25 processes; `.co` files and libraries identical, but see the `LdsNumBytes` caveat. +3. **Kernels that will be rejected are generated in full first.** A kernel whose static VGPR + allocation already exceeds 256 is guaranteed to be rejected, but this is only acted on after + the whole kernel is generated. These are often the slowest kernels: 8–89% of codegen CPU in four + of the probed configs. Failing at `_initKernel` was exact on 798 kernels and cut `f8_gate_r` + codegen 12.6 s → 3.5 s in-run (whole build 28 s → 11 s) with identical artifacts. +4. **Most steps are too small to use the pool.** 479 steps write ≤25 kernels (159.5 s); a step's + wall time is then its slowest kernel while the other workers idle (`custom_mainloop_scheduling`: + 36 s of wall for 40.7 s of CPU on 25 cores). Steps are independent, but are generated one after + another. + +| Option | Measured / estimated effect | Output | Effort | +| --- | --- | --- | --- | +| **A. Remove `CpuThreads: 1` from the 19 test configs** (plus return `LdsNumBytes` from workers) | `subtile_mxfp4` codegen 133.5 → 22.6 s (5.9×), `write_assemble` 62.9 → 11.6 s, build 244 → 63 s, measured. Aggregate estimate: ~189 s → ~35 s codegen | identical `.co`/library; `Data/*.yaml` `LdsNumBytes` differs unless passed back | config + small code fix | +| **B. Fail fast on rejected kernels** (initial VGPR allocation > `MaxVgpr`, plus abort as soon as any rejection code is set) | VGPR check alone: `f8_gate_r` codegen −72% (12.6 → 3.5 s) in-run, `plr_zero` worker CPU −20%; both checks: CPU on rejected kernels 374.5 → 10.8 s (−97%) over 1,242 rejected kernels | identical (VGPR check: 798/798 kernels, artifacts byte-identical; both: 4,313/4,313 kernels same accept/reject) | ~10 lines | +| **C. Generate all steps of a config in one pool** | small-step configs: `custom_mainloop_scheduling` 36.0 → ≤2.1 s, `f8_gate_r` 12.4 → 3.3 s, `spmm_i8_sb` 17.6 → 9.1 s (simulated from measured per-kernel times); ≤25-kernel steps aggregate 142.4 s → ≤75 s (conservative bound) | identical (same kernels) | restructure `BenchmarkProblems` loop | +| D. Longest-first dispatch within a step | 0–6% on most configs, 24% on `spmm_i8_sb` (simulated, perfect cost knowledge) | identical | needs a cost predictor | +| E. Make the generator cheaper | epilogue 40–64%, `rocIsaPass` ~21%, free ~8%, `str` ~6% of per-kernel time; quadratic `hasAnyDependency` up to 67% on some sparse kernels | depends | larger refactors | + +A, B and C are independent and additive (inference): A applies to the 19 single-process configs, +B to rejected kernels everywhere, C to the many-small-steps configs. None of these change the +generated code. + +## Method + +1. **Distribution.** Each step's `TIMING:python_kernel_codegen` line in the build logs + (`/home/alvasile/timing_runs/full_runs_gfx950/per_config_build//_build.log.gz`) was + paired with the step's `Writing assembly kernels: Launching N threads for K tasks` line (K = + kernels that passed validation; generated kernels are more, e.g. 4,093 generated vs 3,420 written + for `plr_zero`). The 169 steps of the `CpuThreads: 1` `subtile_*` configs print no such line + (single-process path) and are kept separately. Output: `/tmp/codegen/steps.txt`, + `/tmp/codegen/cfg_totals.txt`. +2. **In-run probe.** pytest plugin `/tmp/codegen/cgprobe.py` (`-p cgprobe`, run by + `/tmp/codegen/run_cg.sh`, the same build-only `test_config_build` invocation as the aggregate, + 25 cores via `taskset`). It replaces `Run.processKernelSource` with a module-level wrapper that the + loky workers import by name, so every worker logs per-kernel start/end wall time, thread CPU + time, output size and name (`w_.txt`). The parent logs the block and `ParallelMap2` window + per step, the pickled task size, and (optionally) pickles the step's kernels, kernel writer and + rocisa data for replay. Switches: `CG_FORCE_THREADS=N` (override `CpuThreads`), + `CG_FAILFAST=1` (option B in the workers). `/tmp/codegen/analyze.py` splits each step into + pool start → first kernel, kernel span, last kernel → return, worker busy/CPU, slowest kernel. +3. **Replay.** `/tmp/codegen/replay_cg.py` (time / cProfile one kernel), `phases.py` (wraps + `rocIsaPass`, `kernelBody`, `checkResources`, `_initKernel`; times `str(module)` and freeing the + module separately; `RIPO=` toggles `rocIsaPass` options, `NOCOMMENT=1` sets + `outputNoComment`), `shares.py` (cProfile section shares), `ff.py` (baseline vs fail-fast over + every kernel of a step, 16 processes), `sched.py` (actual vs longest-first vs lower bound). + Replays reproduce in-run per-kernel times (1,931 vs 2,037 ms, 1,577 vs 1,646 ms, 122 vs 125 ms). +4. **Native profile.** `perf` is blocked (`perf_event_paranoid` = 4); py-spy 0.4.2 (installed to + `/tmp/codegen/pyspy_inst`) with `--native` works, but `_rocisa.so` is stripped, so C++ frames are + addresses only. C++ costs were therefore measured by timing the native calls directly. +5. Probed configs: `custom_mainloop_scheduling`, `fp16_tn`, `f8_gate_r`, `spmm_i8_sb`, + `gfx950/dtl`, `plr_zero`, `subtile_mxfp4`. + +## Distribution in the aggregate + +| Step (kernels written) | Steps | Kernels | Codegen wall | +| --- | ---: | ---: | ---: | +| ≤25 | 479 | 2,974 | 159.5 s | +| 26–200 | 143 | 10,051 | 127.8 s | +| >200 | 64 | 35,101 | 135.9 s | +| `subtile_*` (`CpuThreads: 1`, no count logged) | 169 | — | 169.4 s | + +Top configs: `subtile_mxfp4` 97.4 s (130 steps), `subtile_bf16` 61.8 s, `gfx950/dtl` 47.5 s, +`custom_mainloop_scheduling` 34.2 s (66 steps, 79 kernels), `plr_zero` 29.0 s, `spmm_f16_sb` +19.1 s, `spmm_{i8,bf16,f8,b8}_sb` 12–16 s each, `fp16_tn` 11.7 s, `f8_gate_r` 11.4 s (8 kernels +written). + +## Where the time goes + +### Pool overhead is negligible + +| Config | Codegen block | Before first kernel | After last kernel | Worker CPU | Cores busy (of 25) | +| --- | ---: | ---: | ---: | ---: | ---: | +| `custom_mainloop_scheduling` (66 steps) | 36.8 s | 0.6 s total (135 ms on step 0, 2–12 ms after) | 0.4 s | 40.5 s | 1.1 | +| `f8_gate_r` (4 steps) | 12.6 s | 0.14 s | 0.11 s | 46.7 s | 3.7 | +| `spmm_i8_sb` (8 steps) | 17.8 s | 0.03 s | 0.12 s | 219.5 s | 12.4 | +| `fp16_tn` (3 steps) | 14.6 s | 0.01 s | 0.02 s | 330.7 s | 22.7 | +| `gfx950/dtl` (26 steps) | 61.1 s | 0.10 s | 0.19 s | 1,312 s | 21.5 | +| `plr_zero` (17 steps) | 36.6 s | 0.06 s | 0.10 s | 789 s | 21.6 | + +A task pickles to ~29 KB (kernel 13 KB, `KernelWriterAssembly` 10 KB, `globalParameters` 3 KB) in +0.2–0.7 ms. The large configs keep the cores ~86–96% busy; the small-step configs barely use them. + +### One kernel: cost scales with the assembly text + +| Config | Kernels | Median CPU / kernel | Median output | CPU per MB of output | +| --- | ---: | ---: | ---: | ---: | +| `custom_mainloop_scheduling` | 79 | 294 ms | 1.24 MB | 281 ms | +| `fp16_tn` | 2,060 | 124 ms | 0.93 MB | 139 ms | +| `gfx950/dtl` | 8,650 | 110 ms | 0.72 MB | 160 ms | +| `plr_zero` | 3,420 | 136 ms | 0.85 MB | 164 ms | +| `spmm_i8_sb` | 546 | 167 ms | 1.28 MB | 127 ms | +| `subtile_mxfp4` | 1,414 | 67 ms | 0.68 MB | 91 ms | + +(Accepted kernels only; `dtl` alone generated 7.5 GB of assembly text.) The largest kernel seen, +`custom_mainloop_scheduling` step 40, is 7.0 MB / 161,079 lines and takes 1.9–2.0 s. + +**Phases** (`phases.py`, GC disabled, median of 3; freeing measured separately by holding a +reference — in the real run it happens inside the call when `kernelBody` returns): + +| Kernel | Total | `rocIsaPass` | `str(module)` | free module | Python tree construction (rest) | +| --- | ---: | ---: | ---: | ---: | ---: | +| cms step 40 (7.0 MB) | 1,708 ms | 617 ms | 123 ms | +208 ms | ~966 ms | +| cms step 30 (0.57 MB) | 104 ms | 30 ms | 7 ms | +10 ms | ~67 ms | +| `fp16_tn` k700 (1.56 MB) | 182 ms | 36 ms | 17 ms | +25 ms | ~129 ms | +| `f8_gate_r` k0 (2.98 MB) | 359 ms | 57 ms | 26 ms | +41 ms | ~276 ms | +| 24 kernels sampled evenly from `dtl` s11, `plr_zero` s12, `spmm_i8_sb` s0 | 5,023 ms | 1,160 ms | 315 ms | +434 ms | ~3,548 ms | + +Over the 24-kernel sample (free included): Python ~65%, `rocIsaPass` ~21%, free ~8%, `str` ~6%. +GC is not a factor (1,852 ms GC off vs 1,783 ms GC on for the largest kernel; noise). + +**Inside the Python part** (cProfile, ranking only): the store/epilogue +`notLocalSplitUGlobalWrite` is 40–64% of profiled time in 22 of 28 sampled kernels (17–39% in the rest); the main loop +(`PersistentLoop.compute`) 4–38%. The epilogue is large because it is fully unrolled per element +for every variant (edge/non-edge, beta, bias type, activation): in `fp16_tn` k700, 56% of the +21,419 lines follow `label_ActivationSetPCAddrEnd_*`. On some sparse kernels +`KernelWriter.hasAnyDependency` (a local-read × instruction scan, `KernelWriter.py:1887`) dominates: +25–67% of profiled time in four `spmm_i8_sb` kernels (903 k calls in one), 0% in `dtl`. + +**Inside `rocIsaPass`** (`rocisa/src/pass/pass.cpp`; toggling options on the 7.0 MB kernel, which +changes output so only for attribution): mandatory macro/composite/text-variable passes ~231 ms; +register graph + `removeDuplicateAssignment` ~330 ms (also yields `maxVgpr` for occupancy); +`getCycles` ~94 ms (`MathClocksUnrolledLoop`, serialized into the library). None of these can +simply be skipped. + +**Comments** (`--disable-asm-comments` / `outputNoComment`): output 1.56 → 0.86 MB but time only +−5% (156 → 149 ms), because comments are still built in Python; a minor lever for codegen +(it would also shrink the text written and assembled in `python_kernel_write_assemble`, not +measured). + +### Repeat generation is not the issue + +Within each probed config almost every generated kernel is unique (`dtl`: 9,107 generations, 9,056 +unique, 8 s of 1,312 s duplicated), and no kernel name appeared in more than one probed config. + +## Options + +### A. Run the `CpuThreads: 1` configs in parallel + +`CpuThreads: 1` is set in 71 test yamls; 19 ran in this aggregate (all 13 `subtile_*` gfx950 +configs, `spmm_vw_lg_one{,_sb}` gfx94x/gfx950, `use_sgpr_for_gro{,_sb}`), totalling 189.3 s of +codegen and 92.0 s of `python_kernel_write_assemble`. `ParallelMap2` then takes its serial path +(no "Launching" line). In the subtile configs the setting was present from the moment each file +was added, in three PRs, none of which explains it: + +| Commit | PR | Configs | +| --- | --- | --- | +| `1fb7e7dcc69` (bnemanich, 2026-05-03) | #7017 "Add gfx950 MXFP4 and BF16 Subtile-based kernel implementation" | `subtile_mxfp4`, `subtile_bf16` | +| `42e06b9f3f2` (Prabhjot Sandhu, 2026-06-03) | #7656 "[SubtileImpl] Add support for MXFP8 MT128x128 with DU=256" | `subtile_fp8`, `subtile_mxfp8` | +| `e91ff88833c` (Matthew Emmett, 2026-06-24) | #7781 "TensileLite: Add multi-DU support." | the nine `subtile_mxfp8_*` configs | + +In #7017 it sits next to debug-style settings (`KeepBuildTmp: True`, commented +`PrintSolutionRejectionReason` / `GenerateSourcesAndExit`); the later files appear to have +copied it. (The files moved to `tensilelite/` in `11d0433bd6d`, #12150.) The non-subtile +`CpuThreads: 1` configs were not traced. + +In-run, `subtile_mxfp4` with `CpuThreads` overridden to 25 (same cores, back to back): + +| | Serial | 25 processes | +| --- | ---: | ---: | +| `python_kernel_codegen` | 133.5 s | 22.6 s | +| `python_kernel_write_assemble` | 62.9 s | 11.6 s | +| whole build | 244 s | 63 s | + +All 130 `.co` files and 130 libraries (`TensileLibrary.dat`, decompressed) are byte-identical. +**25 of 155 `Data/00_Final.yaml` files differ in `LdsNumBytes`** (e.g. 82,432 serial vs 69,632 +parallel): the subtile generator sets `kernel["LdsNumBytes"]` during codegen +(`Components/Subtile/SubtileLdsLayout.py:183`). Serially the kernel object *is* the solution, so +the value lands in the YAML; in a worker it is lost, because `passPostKernelInfoToSolution` +(`Run.py:457`) only copies `CUOccupancy`, `PrefetchGlobalRead`, `MathClocksUnrolledLoop` and +`CustomKernel` back. This is not subtile-specific: forcing `custom_mainloop_scheduling` serial +changed one solution's `LoopIters` (4 vs 1; set at `KernelWriter.py:7691`), again with identical +`.co` and libraries. So the solution metadata already depends on `CpuThreads`; enabling +parallelism for subtile should add `LdsNumBytes` (and probably `LoopIters`) to the returned +`KernelCodeGenResult` so both modes agree. + +Aggregate estimate (not measured end to end): applying the measured 5.9× to the 189.3 s gives +~32 s, saving ~157 s of codegen and ~75 s of `write_assemble`. The non-subtile ones have few +kernels per step, so their gain will be smaller. + +### B. Fail fast on a guaranteed VGPR overflow + +`_initKernel` computes the static VGPR allocation and only prints +`warning: total VGPRS (370) overflowed max VGPRS (256).` (`KernelWriter.py:9616`). The VGPR pool +is created with that size (`KernelWriter.py:10270`) and can only grow (`RegisterPool::remove` +marks registers unavailable, `resize` only extends), so the final check +`vgprPool.size() > MaxVgpr` (`KernelWriterAssembly.py:2343`) always rejects the kernel — after the +whole kernel, including the expensive epilogue, has been generated. The rejection surfaces as +`err=-2` and is not printed unless `PrintSolutionRejectionReason` is set. + +Prototype: after `_initKernel`, raise if `states.totalVgprs > regCaps["MaxVgpr"]` and +`forceGenerateKernel` is off. + +| Step (every kernel, 16 processes) | Kernels | Rejected | Accept/reject mismatches | Codegen CPU | +| --- | ---: | ---: | ---: | ---: | +| `f8_gate_r` step 0 | 8 | 6 | 0 | 11.9 → 1.4 s (−88%) | +| `f8_gate_r` step 1 | 8 | 6 | 0 | 13.2 → 1.3 s (−90%) | +| `spmm_i8_sb` step 0 | 80 | 16 | 0 | 29.5 → 14.1 s (−52%) | +| `plr_zero` step 12 | 546 | 14 | 0 | 86.3 → 85.3 s (−1%; its rejections are mostly other reasons) | +| `plr_zero` step 13 | 156 | 92 | 0 | 81.6 → 26.8 s (−67%) | + +Accepted kernels' output size was also unchanged. In-run with the check applied in the workers: +`f8_gate_r` codegen 12.6 → 3.5 s, whole build 28 → 11 s, all 16 artifacts byte-identical; +`plr_zero` worker CPU 789 → 630 s (−20%), codegen 36.6 → 33.6 s (baseline ran on other cores; +wall is noisy), all 68 artifacts byte-identical. Share of codegen CPU spent on rejected kernels in +the probes: `f8_gate_r` 89%, `plr_zero` 29%, `spmm_i8_sb` 24%, `dtl` 8%, `fp16_tn` 1%, +`subtile_mxfp4` 0.4%, `custom_mainloop_scheduling` 0%. Across the aggregate 8,457 of 116,264 +solutions were rejected by the KernelWriter; their CPU share outside the probes was not measured. + +#### All rejection reasons, and when each is known + +Every kernel the probes recorded as rejected (1,242 kernels from `f8_gate_r`, `spmm_i8_sb`, +`plr_zero`, `gfx950/dtl`, `fp16_tn`, `subtile_mxfp4`) was replayed with a hook on +`StateValues.__setattr__` recording the first non-zero `overflowedResources` code, the time it was +set and the setting function (`/tmp/codegen/rejreasons.py`, output `/tmp/codegen/rejreasons.tsv`): + +| Code | Reason | Kernels | Share of rejected-kernel CPU | Known at (median fraction of codegen time) | Set by | Caught by init VGPR check | +| --- | --- | ---: | ---: | ---: | --- | ---: | +| 1 | too many VGPRs | 679 | 83.6% | 0.50 | `checkResources` (`KernelWriterAssembly.py:2343`) | 598 | +| 2 | too many SGPRs | 427 | 15.0% | 0.03 | `overflowListener` in `allocTmpSgpr` (`KernelWriterAssembly.py:789`) | 5 | +| 5 | local read/write LDS scheduling | 132 | 1.3% | 0.21 | `_makeSubIterSchedule` (`KernelWriter.py:2011`) | 0 | +| 8 | not enough LDS (subtile) | 4 | 0.1% | 0.01 | `applyLdsLayout` (`SubtileLdsLayout.py:185`) | 0 | + +No rejection came from any other exception. `overflowedResources` is never reset and is read only at +the end (to build the error and print the reason: `KernelWriter.py:5664`, `:7265`, +`KernelWriterAssembly.py:2368`), so once it is non-zero the kernel is guaranteed to be rejected +(unless `forceGenerateKernel`), yet generation continues: SGPR overflow is known 3% of the way in; +the VGPR flag is set at the end of `kernelBody`, after which `rocIsaPass`, `str()` and freeing the +tree (~50% of the kernel's time) still run. The 81 VGPR rejections the init check misses overflow +through registers allocated during generation. + +**Second check: abort as soon as `overflowedResources` becomes non-zero.** Estimated CPU on rejected +kernels, from the recorded set times: + +| Config | Rejected | Today | Init VGPR check | Abort on set | Both | +| --- | ---: | ---: | ---: | ---: | ---: | +| `f8_gate_r` | 24 | 41.4 s | 0.0 s | 25.5 s | 0.0 s | +| `spmm_i8_sb` | 54 | 48.1 s | 0.0 s | 31.4 s | 0.0 s | +| `plr_zero` | 673 | 191.0 s | 57.1 s | 73.9 s | 2.7 s | +| `gfx950/dtl` | 457 | 90.2 s | 19.2 s | 44.0 s | 8.1 s | +| `fp16_tn` | 30 | 3.4 s | 0.0 s | 2.1 s | 0.0 s | +| `subtile_mxfp4` | 4 | 0.4 s | 0.4 s | 0.0 s | 0.0 s | +| **Total** | **1,242** | **374.5 s** | **76.7 s** | **177.0 s** | **10.8 s (−97%)** | + +Validated with both checks (`ff.py`, `FF_ABORT_ON_SET=1`) on 9 complete steps covering every reason, +4,313 kernels: 0 accept/reject mismatches, accepted kernels' output size unchanged, CPU on rejected +kernels 160.1 → 4.3 s. + +| Step | Kernels | Rejected | Step codegen CPU | +| --- | ---: | --- | ---: | +| `plr_zero` step 0 | 874 | 218 (SGPR) | 104.2 → 74.4 s (−29%) | +| `gfx950/dtl` step 4 | 336 | 136 (VGPR) | 75.4 → 32.3 s (−57%) | +| `plr_zero` step 13 | 156 | 92 (VGPR + SGPR) | 69.5 → 22.1 s (−68%) | +| `gfx950/dtl` step 11 | 2,424 | 64 (LDS scheduling) | 338.5 → 354.7 s (noise; 1.6 s of rejected work) | +| `gfx950/dtl` step 24 | 307 | 43 (LDS scheduling + VGPR) | 14.4 → 11.9 s (−17%) | +| `fp16_tn` step 1 | 38 | 30 | 3.6 → 0.6 s (−85%) | +| `spmm_i8_sb` step 1 | 80 | 24 | 27.2 → 6.4 s (−77%) | +| `f8_gate_r` step 0 | 8 | 6 | 10.4 → 1.0 s (−90%) | +| `subtile_mxfp4` step 0 | 90 | 0 | unchanged | + +For a real implementation: the prototype raised from the `overflowedResources` setter and ignored +`forceGenerateKernel`; the real change should check that flag and raise at the setting sites. The +reason printed under `PrintSolutionRejectionReason` comes from `checkResources` at the end, so the +early exit must print it too; the error code (`-2`) is already the same. Moving the VGPR/SGPR checks +into solution validation, so these kernels never reach codegen, is a possible further step (not +tested). + +### C. Generate all steps of a config together + +Each `(problem type, size group)` step is generated, compiled and post-processed before the next +starts (`BenchmarkProblems.py`, loop around `_benchmarkProblemType`); no step reads another's +results (inference from the loop; build-only skips benchmarking). Simulating one 25-worker pool +per config with the measured per-kernel times (longest-first): + +| Config | Sum of per-step spans (measured) | One pool (simulated) | +| --- | ---: | ---: | +| `custom_mainloop_scheduling` | 36.0 s | 2.1 s | +| `f8_gate_r` | 12.4 s | 3.3 s | +| `spmm_i8_sb` | 17.6 s | 9.1 s | +| `fp16_tn` | 14.4 s | 13.4 s | +| `gfx950/dtl` | 60.3 s | 54.1 s | +| `plr_zero` | 36.2 s | 32.5 s | + +Aggregate bound (estimate): for the 447 ≤25-kernel steps outside the `CpuThreads: 1` configs, CPU ≤ +kernels × step wall, so per config the pooled time is ≤ CPU/25 + longest step: 142.4 s → ≤75.2 s. +The probes suggest the real gain is larger. This needs the step loop split into "generate +solutions for all steps → one codegen pool → per-step assemble/build/write" (or keeping one pool +fed across steps); not prototyped. + +### D. Longest-first dispatch + +Within a step, joblib dispatches in input order. With perfect knowledge of per-kernel cost, +longest-first would save 0–6% on most probed configs and 24% on `spmm_i8_sb` (17.6 → 13.4 s, +which is also the lower bound). Needs a cost predictor (output size is not known before +generation); upper bound only. + +### E. A cheaper generator (larger work) + +Per-kernel shares above: Python tree construction ~65% (epilogue 40–64% of profiled time), +`rocIsaPass` ~21%, freeing ~8%, `str` ~6%. The code itself notes the epilogue size problem +(`KernelWriter.py:7102`: "Kernels with epilog especially with activation is too long (50000~ +lines). Need to refactor global write elements."). The quadratic `hasAnyDependency` scan is a +contained target for the sparse configs. Not prototyped. + +## Removing `CpuThreads: 1` in CI: cores, oversubscription and memory + +### How CI runs the common tests + +| Job | Command | pytest workers | CPUs | +| --- | --- | ---: | --- | +| TheRock `tensilelite-common` (`TEST_CATEGORY=hw-common`; gfx942 on PRs, gfx950/gfx90a postsubmit, gfx120X nightly) | `pytest_runner.py --numprocesses=4` (`test_categories.yaml`: `parallel_workers: 4`) | 4 | Kubernetes pod on a 1-GPU slice of an 8-GPU node (`linux-gfx942-1gpu-ccs-*`, `linux-gfx950-1gpu-ccs-ossci-rocm`); `KUBE_CPU_REQUEST=25` (from a real job log, see `CI_SPLIT_TIMING.md`) | +| Math CI `preliminary` | `tox -e py3 -- tensilelite/Tests -m common` | `TENSILE_NUM_PYTEST_WORKERS`, default 4 | not in this repo (internal Jenkins) | +| `.github/workflows/gfx1250_hipblaslt_tensilelite_test.yml` | `tox ... -n 4` in a container on `blas-rocm-mi455-sandbox` | 4 | not stated | +| FFM/emulation categories (`ffm-quick`, `emu-fast`, `emu-full`) | `pytest_runner.py` | 16 | not stated | + +TheRock's `test_component.yml` turns `KUBE_CPU_REQUEST` into `OMP_NUM_THREADS` / +`OPENBLAS_NUM_THREADS` only. Tensile sizes its pools from `len(os.sched_getaffinity(0))`, which +ignores the Kubernetes request and the cgroup CPU quota, so each pytest worker starts either 25 +processes (if the pod's CPUs are pinned) or one per host core (if not). Which one applies could not +be checked: no CI log was reachable (no GitHub credentials, API rate-limited). Any CI log answers it: +the `Generating assembly kernels: Launching N threads` line gives N per pytest worker. + +### CI is already oversubscribed, regardless of `CpuThreads: 1` + +Of the 540 YAMLs under `Tests/common`, 406 use the default `CpuThreads: -1` (all visible cores), 67 +set 32, 2 set 8 and 65 set 1. Nothing in the test harness divides cores between pytest workers, so +with 4 workers and 25 visible cores up to 100 codegen processes compete for 25 CPUs whenever two +tests generate code at once. Removing `CpuThreads: 1` makes those 65 configs join that existing +pattern; it does not raise the worst case, which the default configs can already reach. + +Cost of oversubscription, measured by replaying the same 1,000 `dtl` kernels on 25 pinned cores +(`/tmp/codegen/oversub.py`): 25 / 50 / 100 worker processes took 7.2 / 8.2 / 7.6 s (worker CPU +174.9 / 197.3 / 186.4 s). Throughput is unchanged within noise on this shared host; CPU rises +7–13%. Memory is the real cost. + +### Per-worker memory + +The pool is joblib's loky backend. Every worker is a fresh interpreter +(`python3 -m joblib.externals.loky.backend.popen_loky_posix`), not a fork of the parent, so each one +imports everything itself (537 modules); only the file-backed shared libraries (~38 MB) are shared. +Loky reuses the same workers across `ParallelMap2` calls in a build, so start-up is paid once per +build, not per step. + +Single fresh process (`/tmp/codegen/memsteps.py`), resident memory: + +| Stage | RSS | +| --- | ---: | +| bare Python | 10 MB | +| + joblib / numpy | 36 MB | +| + rocisa | 64 MB | +| + Tensile modules | 80 MB (38 MB shared libraries, ~42 MB private) | +| after one typical kernel (1.6–2.1 MB of assembly) | +1–2 MB | +| after the 7.0 MB `custom_mainloop_scheduling` kernel | peak 495 MB; still 355 MB after the result is dropped | + +Peak memory scales with kernel size, about 60 MB of instruction tree per MB of assembly, and the +allocator does not return it to the OS, so a worker stays near the peak of its biggest kernel. +(An earlier 245 MB figure came from a harness that loaded the whole pickled step into every worker, +which real workers never do; it is superseded by the numbers below.) + +Real loky workers, sampled once per second during real builds (`/tmp/codegen/memsample.py`): + +| Build | Workers | Peak RSS median / max | Peak PSS median / max | Shared libs | +| --- | ---: | ---: | ---: | ---: | +| `fp16_tn` | 25 | 288 / 315 MB | 251 / 276 MB | 38 MB | +| `custom_mainloop_scheduling` | 29 | 308 / 569 MB | 259 / 531 MB | 38–40 MB | + +One `fp16_tn` worker over the build: 122 → 196 → 252 → 281 → 295 MB. So 4 pytest workers × 25 +loky workers ≈ 100 × ~250 MB ≈ 25 GB, more if workers hit very large kernels; if a pod exposed a +whole 192-core node, 4 × 192 workers would need ~190 GB. That exposure exists today for the +default configs. + +Loky also has memory-leak protection (`joblib/externals/loky/process_executor.py`, +`_MAX_MEMORY_LEAK_SIZE = int(3e8)`): a worker whose memory grew more than 300 MB above its starting +size exits and is replaced. Observed in a `pytest -n 4` run: `f8_gate_r`'s workers exited at +513–596 MB after 10–30 s and were respawned (20 distinct pids for a 12-worker pool), each +replacement briefly overlapping the worker it replaced. + +### Proposal: a total pool budget of 2× the available CPUs + +Size every pool so that all pytest workers together start at most 2× the CPUs available, and a +single process never more than 1×: + +1. CPUs available = min(affinity mask, loky `cpu_count()` (also applies the cgroup quota), + `KUBE_CPU_REQUEST` when set). +2. Parallel workers = `PYTEST_XDIST_WORKER_COUNT` (set by pytest-xdist in every worker), else 1. +3. Pool size = max(1, min(CPUs, 2 × CPUs // workers)). +4. `CpuThreads` stays a cap: `-1` = the budget, N = min(budget, N), `0` = no threading. + +| Setup | Per process | Total | +| --- | ---: | ---: | +| standalone, 96 CPUs | 96 | 96 | +| `-n 4`, 25 CPUs (TheRock / Math CI) | 12 | 48 | +| `-n 4`, pod sees 192 cores, `KUBE_CPU_REQUEST=25` | 12 | 48 | +| `-n 2`, 25 CPUs | 25 | 50 | +| `-n 16`, 128 CPUs (FFM) | 16 | 256 | + +Sketch, replacing the body of `CPUThreadCount()` in `Common/Parallel.py`: + +```python +OVERSUBSCRIPTION_FACTOR = 2 + +def _kubeCpuRequest(): + value = os.environ.get("KUBE_CPU_REQUEST") # may be fractional, e.g. "8.0" + try: + return max(1, int(float(value))) if value else None + except ValueError: + return None + +def availableCPUCount(): + if os.name == "nt": + count = min(os.cpu_count() or 1, 61) + else: + count = len(os.sched_getaffinity(0)) + try: + from joblib.externals.loky import cpu_count as lokyCpuCount # affinity + cgroup quota + count = min(count, lokyCpuCount()) + except Exception: + pass + request = _kubeCpuRequest() + return max(1, min(count, request) if request else count) + +def concurrentWorkerCount(): + try: + return max(1, int(os.environ.get("PYTEST_XDIST_WORKER_COUNT", "1"))) + except ValueError: + return 1 + +def poolSizeBudget(): + cpus = availableCPUCount() + return max(1, min(cpus, (OVERSUBSCRIPTION_FACTOR * cpus) // concurrentWorkerCount())) + +def CPUThreadCount(enable=True): + if not enable: + return 1 + budget = poolSizeBudget() + cpuThreads = globalParameters["CpuThreads"] + return budget if cpuThreads == -1 else min(budget, cpuThreads) +``` + +This was prototyped and validated, then reverted (not part of this change): + +- 15 unit tests (mocked affinity / loky count / environment) covering the table above, cgroup + limits, unparsable environment values, `CpuThreads` caps and `0`, and that per-process × workers + ≤ 2 × CPUs for six CPU/worker combinations; together with the existing + `Tests/unit/characterization/Parallel/test_parallel_char.py`, 29 passed. +- Outside pytest with `PYTEST_XDIST_WORKER_COUNT=4` on 25 cores: `ParallelMap2` printed + `Launching 12 threads` and used exactly 12 worker pids. +- Real `pytest -n 4` build of `fp16_tn`, `cgemm`, `zgemm`, `f8_gate_r` on 25 pinned cores, process + tree sampled every 0.2 s (`/tmp/codegen/poolturnover.py`): 12 loky workers per pytest worker (13 + for a moment during a loky memory-leak restart, see above), at most 50 in total; all 4 passed. + (Short-lived "loky" children of loky workers are forks for `amdclang++` before `exec`, which keep + the parent's command line; they are not pools.) + +Open decisions for this proposal: `KUBE_CPU_REQUEST` is a Kubernetes *request*, not a limit, but +TheRock already treats it as the pod's budget for OpenMP threads, and without it a pod that sees the +whole node would size pools from the whole node. Only pytest-xdist parallelism is detected; other +ways of running several Tensile processes on one machine are not. + +### `CPUThreadCount()` / `ParallelMap2` issues + +- `CPUThreadCount()` sizes pools from `len(os.sched_getaffinity(0))` only. It ignores cgroup CPU + quotas, `KUBE_CPU_REQUEST`, and how many pytest-xdist workers run concurrently, so under + `pytest -n N` each worker starts a full-size pool (the oversubscription above). +- `ParallelMap2` (`Common/Parallel.py`) computes + `threadCount = procs if procs else CPUThreadCount(enable)` and on the next line overwrites it with + `threadCount = CPUThreadCount(enable)`, so an explicit `procs` is ignored. The only caller passing + it is `TensileLogic/Run.py:321` (`procs=jobs`), whose job count therefore has no effect. +- TensileCreateLibrary's `_cpuCount()` (`TensileCreateLibrary/Run.py:1226`) duplicates the + affinity-only count to split jobs between architecture groups, with the same blind spots. + +## Not measured / open items + +- End-to-end aggregate with A/B/C applied; the aggregate numbers above are estimates from the probes. +- Whether `LdsNumBytes`/`LoopIters` in `Data/*.yaml` are consumed downstream (libraries and code + objects are identical either way); which value is intended. +- CPU share of rejected kernels across all configs, and early-exit for the other overflow reasons. +- Whether any `CpuThreads: 1` config needs it for a reason not visible in the commit. +- How many cores Tensile sees inside CI pods (pinned 25 vs the whole node); read the + `Launching N threads` line from any `tensilelite-common` / Math CI log. +- Step independence for option C was inferred from the loop, not proven for every config feature. +- Native (C++) hotspots inside `rocIsaPass`/`str`/free: `_rocisa.so` is stripped and `perf` is + unavailable; only whole-call times were measured. +- Probe A/B timings ran on a shared host; comparisons are back to back on the same 25 cores except + the `plr_zero` fail-fast run. diff --git a/projects/hipblaslt/tensilelite/timing_runs/run_cms.sh b/projects/hipblaslt/tensilelite/timing_runs/run_cms.sh new file mode 100644 index 000000000000..2191e9656a81 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/run_cms.sh @@ -0,0 +1,21 @@ +# Build-only of the CMS (UseCustomMainLoopSchedule) configs, targeting gfx950, cold caches. +TL=/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite +D=$TL/timing_runs/cms +cd $TL || exit 1 +mkdir -p $D +export OMP_NUM_THREADS=25 OPENBLAS_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=0 PYTHONUNBUFFERED=1 +TO="--cxx-compiler,/usr/bin/amdclang++,--global-parameters,TimingInstrumentation=True" +COMMON=(-s -v -n 0 --durations=0 --timeout=14400 "--global-parameters=LibraryFormat='msgpack'" + "--tensile-options=$TO" -p no:warnings) +: > $D/times.txt +for cfg in gemm/gfx950/custom_mainloop_scheduling gemm/gfx950/custom_mainloop_scheduling_tf32 ductile/gfx950/f8bf16_tn_cms; do + name=$(basename $cfg) + rm -rf /home/alvasile/.tensile /home/alvasile/.tensilelite /tmp/tl_cms_art_$name + BID="tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/$cfg.yaml]" + s=$(date +%s) + /usr/bin/time -v -o $D/${name}_time.txt taskset -c 0-24 pytest "$BID" --build-only --gpu-targets gfx950 \ + --artifact-dir /tmp/tl_cms_art_$name "${COMMON[@]}" > $D/${name}_build.log 2>&1 + echo "$name rc=$? $(( $(date +%s)-s ))s" >> $D/times.txt + python scripts/analyze_timing.py $D/${name}_build.log > $D/${name}_build_analysis.txt 2>&1 +done +echo DONE >> $D/times.txt diff --git a/projects/hipblaslt/tensilelite/timing_runs/run_cold_warm.sh b/projects/hipblaslt/tensilelite/timing_runs/run_cold_warm.sh new file mode 100644 index 000000000000..44ed391b0d68 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/run_cold_warm.sh @@ -0,0 +1,45 @@ +# For each config: cold build (caches deleted), warm build (caches kept), run (caches deleted +# first, as on a fresh GPU node). Raw logs + analyze_timing.py output per stage. +# Usage: bash run_cold_warm.sh gemm/dtv gemm/cgemm +TL=/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite +CL=/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client +cd $TL || exit 1 +export OMP_NUM_THREADS=25 OPENBLAS_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=0 PYTHONUNBUFFERED=1 +TO="--cxx-compiler,/usr/bin/amdclang++,--global-parameters,TimingInstrumentation=True" +COMMON=(-s -v -n 0 --durations=0 --timeout=5400 -m "common and not skip-gfx94X" + "--global-parameters=LibraryFormat='msgpack'" "--tensile-options=$TO" -p no:warnings) +clear_caches() { rm -rf /home/alvasile/.tensile /home/alvasile/.tensilelite; } + +for cfg in "$@"; do + name=$(basename $cfg) + D=$TL/timing_runs/$name + A=/tmp/tl_timing_art_$name + mkdir -p $D + rm -rf $A + BID="tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/$cfg.yaml]" + RID="tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/$cfg.yaml]" + : > $D/times.txt + + clear_caches + s=$(date +%s) + taskset -c 0-24 pytest "$BID" --build-only --gpu-targets gfx942 --artifact-dir $A/cold \ + "${COMMON[@]}" > $D/${name}_1_build_cold.log 2>&1 + echo "build_cold rc=$? $(( $(date +%s)-s ))s" >> $D/times.txt + + s=$(date +%s) # caches kept from the cold build + taskset -c 0-24 pytest "$BID" --build-only --gpu-targets gfx942 --artifact-dir $A/warm \ + "${COMMON[@]}" > $D/${name}_2_build_warm.log 2>&1 + echo "build_warm rc=$? $(( $(date +%s)-s ))s" >> $D/times.txt + + clear_caches + s=$(date +%s) + taskset -c 0-24 pytest "$RID" --use-cache --artifact-dir $A/cold --prebuilt-client=$CL \ + "${COMMON[@]}" > $D/${name}_3_run.log 2>&1 + echo "run rc=$? $(( $(date +%s)-s ))s" >> $D/times.txt + + for st in 1_build_cold 2_build_warm 3_run; do + python scripts/analyze_timing.py $D/${name}_$st.log > $D/${name}_${st}_analysis.txt 2>&1 + done + echo DONE >> $D/times.txt +done +echo ALLDONE > $TL/timing_runs/cold_warm_done.txt diff --git a/projects/hipblaslt/tensilelite/timing_runs/run_fp16_gradient_bias_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/run_fp16_gradient_bias_analysis.txt new file mode 100644 index 000000000000..19344e21b825 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/run_fp16_gradient_bias_analysis.txt @@ -0,0 +1,175 @@ +======================================================================================================================== +TIMING ANALYSIS SUMMARY +======================================================================================================================== + +TIMING BY PHASE +------------------------------------------------------------------------------------------------------------------------ + Category Count Total (ms) Mean (ms) % of Parent % of Wall +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 1 19,225.06 19,225.06 100.0% + -------------------------------------------------------------------------------------------------------------------- + python_client_execution 6 19,192.08 3,198.68 99.8% 99.8% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Data Preparation 11,010.50 57.4% 57.3% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + pre_problem 21 11,004.54 524.03 99.9% 57.2% + cpu_reference_gemm 21 10,976.40 522.69 99.7% 57.1% + solve_cpu_slow 21 10,976.33 522.68 100.0% 57.1% + (untracked) 0.07 0.0% 0.0% + cpu_data_init 21 27.66 1.32 0.3% 0.1% + (untracked) 0.48 0.0% 0.0% + gpu_input_reset 562 3.86 0.01 0.0% 0.0% + gpu_input_preparation 21 1.61 0.08 0.0% 0.0% + rotating_buffer_preparation 21 0.48 0.02 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + C++ Client Setup 6,327.17 33.0% 32.9% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + listener_setup 6 3,195.49 532.58 50.5% 16.6% + hip_initialization 6 2,494.86 415.81 39.4% 13.0% + data_init_setup 6 525.33 87.56 8.3% 2.7% + code_object_loading 6 57.12 9.52 0.9% 0.3% + library_loading 6 30.78 5.13 0.5% 0.2% + lazy_loading_init 6 22.71 3.79 0.4% 0.1% + reporter_setup 6 0.76 0.13 0.0% 0.0% + solution_iterator_setup 6 0.11 0.02 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked C++ overhead) 1,105.31 5.8% 5.7% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Reference Computation 561.45 2.9% 2.9% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + validate_warmups 562 561.45 1.00 100.0% 2.9% + (untracked) 314.55 56.0% 1.6% + validate_gpu_readback 1,124 204.75 0.18 36.5% 1.1% + validate_element_comparison 1,124 41.78 0.04 7.4% 0.2% + validate_gpu_sync 562 0.38 0.00 0.1% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Kernel Execution 158.20 0.8% 0.8% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + warmup_runs 1,124 117.87 0.10 74.5% 0.6% + post_solution 726 21.02 0.03 13.3% 0.1% + post_solution_log 726 15.78 0.02 75.1% 0.1% + post_solution_reporting 726 3.81 0.01 18.1% 0.0% + post_solution_perf_calc 726 0.94 0.00 4.5% 0.0% + post_solution_validation 726 0.33 0.00 1.6% 0.0% + (untracked) 0.09 0.4% 0.0% + post_solution_result_file 726 0.03 0.00 0.1% 0.0% + post_solution_sol_advance 726 0.02 0.00 0.1% 0.0% + post_solution_perf_reset 726 0.02 0.00 0.1% 0.0% + pre_solution 726 10.96 0.02 6.9% 0.1% + kernel_solving 562 5.23 0.01 3.3% 0.0% + post_problem 21 1.72 0.08 1.1% 0.0% + finalize_report 6 0.62 0.10 0.4% 0.0% + solution_selection 726 0.60 0.00 0.4% 0.0% + benchmark_runs 562 0.17 0.00 0.1% 0.0% + (untracked) 0.17 100.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Timing Overhead 29.44 0.2% 0.2% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + flush_timing_buffer 6 28.70 4.78 97.5% 0.1% + timing_overhead 6 0.74 0.12 2.5% 0.0% + -------------------------------------------------------------------------------------------------------------------- + (untracked) 20.05 0.1% 0.1% + -------------------------------------------------------------------------------------------------------------------- + python_cache_check 6 11.73 1.96 0.1% 0.1% + -------------------------------------------------------------------------------------------------------------------- + python_write_solutions 6 1.20 0.20 0.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_prepare 6 0.84 0.14 69.9% 0.0% + . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + python_wsol_prepare_cache 6 0.81 0.14 96.7% 0.0% + (untracked) 0.03 3.3% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + python_wsol_header 6 0.29 0.05 24.0% 0.0% + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + (untracked) 0.07 6.1% 0.0% + +------------------------------------------------------------------------------------------------------------------------ + ESTIMATED WALL CLOCK TIME: 19.23s + +HIERARCHY VALIDATION: OK (all child phases fit within their parents) + +TIME BREAKDOWN (visual, % of wall clock) +------------------------------------------------------------------------------------------------------------------------ + python_benchmark_problems 100.0% ################################################# + python_client_execution 99.8% ################################################# + Data Preparation 57.3% ############################ + pre_problem 57.2% ############################ + cpu_reference_gemm 57.1% ############################ + solve_cpu_slow 57.1% ############################ + (untracked) 0.0% + cpu_data_init 0.1% + (untracked) 0.0% + gpu_input_reset 0.0% + gpu_input_preparation 0.0% + rotating_buffer_preparation 0.0% + C++ Client Setup 32.9% ################ + listener_setup 16.6% ######## + hip_initialization 13.0% ###### + data_init_setup 2.7% # + code_object_loading 0.3% + library_loading 0.2% + lazy_loading_init 0.1% + reporter_setup 0.0% + solution_iterator_setup 0.0% + (untracked C++ overhead) 5.7% ## + Reference Computation 2.9% # + validate_warmups 2.9% # + (untracked) 1.6% + validate_gpu_readback 1.1% + validate_element_comparison 0.2% + validate_gpu_sync 0.0% + Kernel Execution 0.8% + warmup_runs 0.6% + post_solution 0.1% + post_solution_log 0.1% + post_solution_reporting 0.0% + post_solution_perf_calc 0.0% + post_solution_validation 0.0% + (untracked) 0.0% + post_solution_result_file 0.0% + post_solution_sol_advance 0.0% + post_solution_perf_reset 0.0% + pre_solution 0.1% + kernel_solving 0.0% + post_problem 0.0% + finalize_report 0.0% + solution_selection 0.0% + benchmark_runs 0.0% + (untracked) 0.0% + Timing Overhead 0.2% + flush_timing_buffer 0.1% + timing_overhead 0.0% + (untracked) 0.1% + python_cache_check 0.1% + python_write_solutions 0.0% + python_wsol_prepare 0.0% + python_wsol_prepare_cache 0.0% + (untracked) 0.0% + python_wsol_header 0.0% + (untracked) 0.0% + +KEY INSIGHTS (% of wall clock) +------------------------------------------------------------------------------------------------------------------------ + Python phases (code gen, compilation): 0.00s (0.0%) + Python I/O overhead (cache, YAML): 0.01s (0.1%) + C++ client setup (library loading): 6.33s (32.9%) + Data preparation (CPU init, GPU prep): 11.01s (57.3%) + Reference computation (CPU GEMM): 0.56s (2.9%) + Kernel execution (warmup, benchmark): 0.16s (0.8%) + + +TOP 10 SLOWEST PROBLEMS (by CPU reference time) +------------------------------------------------------------------------------------------------------------------------ + M N K Batch TypeA TypeD CPU Ref (ms) GPU (ms) + ---------------------------------------------------------------------------------------------------------------------- + 512 256 511 1 Half Half 927.99 0.00 + 512 256 511 1 Half Half 742.78 0.00 + 512 256 1023 1 Half Half 738.13 0.00 + 511 256 1024 1 Half Half 686.00 0.00 + 511 256 1024 1 Half Half 663.89 0.00 + 512 256 511 1 Half Half 659.53 0.00 + 511 256 1024 1 Half Half 654.96 0.00 + 512 256 1023 1 Half Half 632.12 0.00 + 512 256 1023 1 Half Half 612.11 0.00 + 512 256 1023 1 Half Half 571.57 0.00 + diff --git a/projects/hipblaslt/tensilelite/timing_runs/run_timing.sh b/projects/hipblaslt/tensilelite/timing_runs/run_timing.sh new file mode 100644 index 000000000000..72abc6d9b824 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/run_timing.sh @@ -0,0 +1,26 @@ +# Timing-instrumented single-config run: build phase then run phase, CI-like conditions. +# Usage: bash run_timing.sh [config-name-substring] +T=${1:-fp16_gradient_bias} +TL=/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite +D=$TL/timing_runs +CL=/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client +A=/tmp/tl_timing_art +cd $TL || exit 1 +export OMP_NUM_THREADS=25 OPENBLAS_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=0 PYTHONUNBUFFERED=1 +TO="--cxx-compiler,/usr/bin/amdclang++,--global-parameters,TimingInstrumentation=True" +COMMON=(-s -v -n 0 --timeout=5400 -m "common and not skip-gfx94X" + "--global-parameters=LibraryFormat='msgpack'" "--tensile-options=$TO" -p no:warnings -k $T) +rm -rf /home/alvasile/.tensile /home/alvasile/.tensilelite $A +: > $D/times.txt +s=$(date +%s) +taskset -c 0-24 pytest tensilelite/Tests/common/test_config_build.py --build-only --gpu-targets gfx942 \ + --artifact-dir $A "${COMMON[@]}" > $D/build_$T.log 2>&1 +echo "build rc=$? $(( $(date +%s)-s ))s" >> $D/times.txt +rm -rf /home/alvasile/.tensile /home/alvasile/.tensilelite +s=$(date +%s) +taskset -c 0-24 pytest tensilelite/Tests/common/test_config_run.py --use-cache --artifact-dir $A \ + --prebuilt-client=$CL "${COMMON[@]}" > $D/run_$T.log 2>&1 +echo "run rc=$? $(( $(date +%s)-s ))s" >> $D/times.txt +python scripts/analyze_timing.py $D/build_$T.log > $D/build_${T}_analysis.txt 2>&1 +python scripts/analyze_timing.py $D/run_$T.log > $D/run_${T}_analysis.txt 2>&1 +echo DONE >> $D/times.txt diff --git a/projects/hipblaslt/tensilelite/timing_runs/tcl_helper_name_mismatch_investigation.md b/projects/hipblaslt/tensilelite/timing_runs/tcl_helper_name_mismatch_investigation.md new file mode 100644 index 000000000000..70dc22326365 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/tcl_helper_name_mismatch_investigation.md @@ -0,0 +1,335 @@ +# TensileCreateLibrary helper name/object mismatch investigation (all shipped logic) + +Findings only, no code change ([DECISIONS.md](DECISIONS.md), topic 8, "Investigation (findings +only, no fix)", plan step P5). The question comes from +[python_kernel_bench_setup_investigation.md](python_kernel_bench_setup_investigation.md), +"Not recommended: reuse `TensileCreateLibrary.generateKernelHelperObjects`": the function dedupes +helper kernels on names predicted by the `*KernelNames` functions, and those names do not match the +objects it then builds. Can that drop a helper, duplicate one, or pair a name with the wrong object +for any real configuration? + +Code state: commit `2083e6254bf` (the Python under `tensilelite/` is identical to `55e29436e9a`). +Logic: every `.yaml` under `library/src/amd_detail/rocblaslt/src/Tensile/Logic/asm_full` (3,313 +files, all architectures). Every count below comes from running the real TensileCreateLibrary code +on these files; code reading was used to explain the counts. Inferences are marked as such. + +## Summary + +- **One real configuration drops helpers: the `gfx1250-strict` build.** It is missing + `Cijk_SB_PostGSU1_VW1`, `_VW2` and `_VW4` (the `SingleBuffer` output-conversion kernel for + `F8BS`/`BBS`-style problems: float workspace, bf16 output, no bias). The one `SingleBuffer` + solution in `gfx1250-strict/gfx1250-strict/Equality/gfx1250-strict_Cijk_Alik_Bljk_F8BS_BH_UserArgs.yaml` + calls for them (the benchmark path would emit them); the library's helper list does not declare + them (checked on the real output, see "Confirming the drop"). `gfx1250-strict` is always built as + its own TensileCreateLibrary run (it is a stepping of `gfx1250`), so this applies to the default + `all` build too. Cause: the predicted name uses the solution's `GlobalSplitU`, so a + `MultipleBuffer` solution with `GlobalSplitU: 0` and the `SingleBuffer` solution with + `GlobalSplitU: 0` predict the same `..._PostGSU0_VW*` names while needing different objects + (`PostGSU16` and `PostGSU1`). The `MultipleBuffer` one comes first, so the `SingleBuffer` objects + are never built. + **At run time the missing kernels cannot be requested** (from the host code, not run): this + solution is persistent data-parallel (`StreamKForceDPOnly: 1`), so it has no accumulation buffer, + GSU is fixed at 0 and the output-conversion kernel is never launched for it (see "Runtime reach"). + The shipped library is not broken today; the defect is the order-dependent dedupe, which would + break a library as soon as a `SingleBuffer` solution that does use the conversion kernel sits + behind a same-name `MultipleBuffer` one. +- **No other configuration drops or adds a helper.** All 18 single-arch builds except + `gfx1250-strict`, `gfx942 --experimental`, `gfx950 --enable-gemm-a2a-fusion`, the default + multi-arch `all` build (16 archs in one run) and every logic file built on its own (3,305 files + that reach the helper step): the helper set equals the set the objects call for. +- **No duplicates reach `Kernels.cpp`, but most helper objects are built more than once.** The final + `dict.fromkeys` dedupes on kernel name, so the output never repeats a name. Before it, 1,807 + objects are built for 265 kept on `gfx942` (2,189 for 427 in the `all` build), because the proxy + names differ between solutions that need the same objects. +- **Positional mispairing happens but never selects the wrong set.** `itertools.compress` pairs + 3 predicted names with 2 objects (`ComplexFloat`) or 1 object (`ComplexDouble`) in 34 calls on + `gfx942`, 4 on `gfx950`, 2 each on `gfx1250`/`gfx1250-strict` (692 summed over per-file builds). + In every one of them the mask is all `True`, so all objects are kept whatever the pairing. A + partially-visited mask together with a length mismatch, the case that would keep the wrong object, + occurs 0 times. +- **`MultipleBufferSingleKernel` with `AdaptiveGemmGSUA` is not reachable today**: no shipped + solution has `AdaptiveGemmGSUA != 0` (0 of the 682,765 solutions of the `all` build, the 2,868 of + `gfx1250-strict`, and the gfx1102 / experimental / GEMM+A2A captures). +- **Related, not caused by the name prediction: two different BetaOnly kernels share one name.** + `Cijk_I_GA` (gfx950: `I8IS` with float compute and `I8II` with int32 compute) and, in the `all` + build, also `Cijk_I8_GA` (gfx950 `I8I8S` float compute versus gfx1170 int32 compute). The name + omits the compute type, so the name dedupe keeps whichever comes first and the other problem type + is left with a beta-only kernel of the wrong `D`/`beta` types. Which one is kept depends on the + build: the gfx950-only build keeps the float version, the `all` build keeps the int32 one (from + gfx1101). **Latent at run time**: every solution that produces these names is `MultipleBuffer` or + `MultipleBufferSingleKernel` with `AdaptiveGemmGSUA: 0`, and the host never launches a beta-only + kernel for those, so the wrong kernel is not reached today. It would be if a `SingleBuffer` (or + adaptive) solution of one of these types were added. The benchmark path dedupes by name the same + way, but a benchmark step has one `ProblemType`, so it cannot hit this. + +## Method + +Scripts are throwaway, in `/tmp/tl_plan/P5/` (not in the repo). + +1. **Capture** (`capture.py [TCL flags]`). Runs the real + `TensileCreateLibrary.Run.run()` with the same arguments as the CMake build + (`--architecture=`, `--cxx-compiler=/opt/rocm/llvm/bin/clang++`, lazy loading on, + experimental off unless asked), so logic-file selection, parsing, merging and solution order are + the real ones. Two populations appear below. TCL's **unique solutions** are the solution list + after the identity dedupe at `Run.py:1203-1204` ("Number of unique solutions" in the log). + `run()` then reduces it to the **kernel list** with `generateKernelObjectsFromSolutions` + (dedupe on `getKeyNoInternalArgs`, `Run.py:884-895`) and passes that to + `generateKernelHelperObjects` (`Run.py:1515-1516`). Two wrappers are installed by monkeypatching + (no repo edit): + - `generateKernelObjectsFromSolutions`: records the unique-solution list (before kernel dedupe), + for the per-file analysis and a second replay check. + - `generateKernelHelperObjects`: calls the real function and records its output (kernel name + + SHA-1 of header and source of every object, in order). Then, for every kernel and every helper + type, records the raw predicted names (`kernelObjectNameCallables()`) and the names and source + hashes of the objects `initHelperKernelObjects(kernel, type, ...)` returns. Then it stops the + run (no code objects are built). + + Per-kernel data is memoized on the fields the helper code reads (canonical `ProblemType` state, + `GlobalSplitU`, `GlobalSplitUAlgorithm`, `isStreamK`, `StreamKAtomic`, `_GlobalAccumulation`, + `AdaptiveGemmGSUA`, `ActivationFused`, `WavefrontSize`, `ISA`, `GlobalSplitUPGR`). The memo was + checked against computing every kernel without it on gfx1103 (1,019 kernels, 0 differences), and + every replay below reproduces the real output exactly. +2. **Replay** (`analyze.py`, `report.py`). Re-runs the `generateKernelHelperObjects` loop on the + recorded strings (same callable order, same `"Enum"` sort with `reverse=True`, same `visited` set, + same `itertools.compress`, same `dict.fromkeys` and final sort) and checks it against the real + output: **identical name, source hash and order for every capture below**, both on the kernel list + and on the unique-solution list. All per-configuration statistics (required, kept, built, length + mismatch, partial mask, kernel indices) are computed on the kernel list, which is what the real + function sees. The replay then compares with the **required set**: the union, by + name, of the objects `initHelperKernelObjects` returns for every kernel, i.e. what the benchmark + path (`writeBenchmarkFiles`, which builds every object and dedupes by name) would emit for the + same solutions. It counts missing and extra names, names whose objects have more than one source, + objects built before dedupe, calls where the name and object lists differ in length, calls with a + partially-visited mask, and selected positions whose object does not match the predicted name. + "Needed but skipped" counts object names a solution needed that were not built at that point + (whether or not a later solution built them). +3. **Per logic file**: the same replay on each file's unique solutions in their real order, which is what + a TensileCreateLibrary run with `--logic-filter` naming that one file does (kernel dedupe within + a file never changes the result: a repeated kernel has the same names and they are all visited). +4. **Mechanism statistics** (`stats.py`), and the drill-down scripts `dig*.py` for the examples. +5. **Confirming the drop** (`check_symbols.py`): runs the real `gfx1250-strict` build up to + `generateKernelHelperObjects` and greps the header text of the real helper list (what + `writeHelpers` writes to `Kernels.h`/`Kernels.cpp`) for `__global__` declarations. + +Configurations captured: one run per architecture directory (gfx908, gfx90a, gfx942, gfx950, +gfx1100, gfx1101, gfx1102, gfx1103, gfx1150, gfx1151, gfx1152, gfx1153, gfx1170, gfx1171, +gfx1200, gfx1201, gfx1250, gfx1250-strict), `gfx942 --experimental` (adds the 16 +`aquavanjaram/gfx942/Experimental` files), `gfx950 --enable-gemm-a2a-fusion` (adds the one +GEMM+A2A file filtered by default), and the default `all` group: the base architectures from +`cmake/tensilelite_supported_architectures.cmake` in one run, minus `gfx1250-strict`, which +TensileCreateLibrary always builds as a separate run because it shares `gfx1250`'s ISA +(`isaCollisionFreeGroups`). Together the runs select all 3,313 logic files, and 3,305 of them reach +the helper step with at least one solution: 3,288 in the default builds, plus the 16 experimental +files and the GEMM+A2A file. The other 8 are the one-solution files +`*_Cijk_Alik_Bljk_B8BS_BH_BiasS_HAS_SAV_UserArgs.yaml` and +`*_Cijk_Alik_Bljk_B8F8BS_BH_BiasS_HAS_SAV_UserArgs.yaml` of gfx1170, gfx1171, gfx1200 and gfx1201: +their solution is parsed, and then removed by the identity dedupe at `Run.py:1203-1204` because an +equal solution comes from another file of the same arch (`count_yaml.py`). + +**Limitation, gfx1170/gfx1171.** The installed ROCm 7.2 compiler does not accept these targets, so +`makeIsaInfoMap` reports `SupportedISA` false and solution construction asserts. For these two (and +for the `all` run, which includes them) `capture_capsub.py` substitutes gfx1200's probed +capabilities. The helper inputs come from the logic files. Every solution entry in the 154 logic +files of each arch is parsed into the library (raw YAML entries 33,972 / 38,153 = "Number of +solutions parsed" in the log; `count_yaml.py`); the identity dedupe then leaves 33,748 / 37,925 +unique solutions, and 152 files per arch reach the helper step, the same two-file pattern as gfx1200 +and gfx1201 with the real compiler (see above). Fields the generator derives from the capabilities +(and the `Valid` flag, which is not checked before the helper step) could still differ with a +compiler that supports these targets. + +## Results + +### Per configuration + +"Solutions" is TCL's unique solutions (before kernel dedupe); "Kernels" is the list +`generateKernelHelperObjects` receives, and every column to its right is computed on it. +"Required" is the number of distinct helper names the kernels call for; "Kept" is what +`generateKernelHelperObjects` returns. "Built" is objects built before the final name dedupe. +"Len mismatch" and "Partial mask" count `itertools.compress` calls (calls where at least one name +was new). + +| Configuration | Logic files | Solutions | Kernels | Required | Kept | Missing | Built | Len mismatch | Partial mask | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| gfx908 | 55 | 58 | 58 | 69 | 69 | 0 | 84 | 0 | 1 | +| gfx90a | 108 | 24,597 | 9,670 | 88 | 88 | 0 | 236 | 0 | 3 | +| gfx942 | 844 | 351,748 | 232,710 | 265 | 265 | 0 | 1,807 | 34 | 9 | +| gfx942 `--experimental` | 860 | 355,260 | 236,222 | 265 | 265 | 0 | 1,807 | 34 | 9 | +| gfx950 | 720 | 84,262 | 84,159 | 186 | 186 | 0 (1 name, 2 sources) | 277 | 4 | 5 | +| gfx950 `--enable-gemm-a2a-fusion` | 721 | 84,263 | 84,160 | 186 | 186 | 0 (1 name, 2 sources) | 277 | 4 | 5 | +| gfx1100 | 46 | 1,836 | 1,722 | 57 | 57 | 0 | 96 | 0 | 4 | +| gfx1101 | 60 | 1,554 | 1,358 | 66 | 66 | 0 | 117 | 0 | 4 | +| gfx1102 | 46 | 1,223 | 1,019 | 56 | 56 | 0 | 80 | 0 | 4 | +| gfx1103 | 46 | 1,223 | 1,019 | 56 | 56 | 0 | 80 | 0 | 4 | +| gfx1150 | 47 | 1,472 | 1,290 | 59 | 59 | 0 | 83 | 0 | 4 | +| gfx1151 | 50 | 2,968 | 2,537 | 59 | 59 | 0 | 86 | 0 | 4 | +| gfx1152 | 46 | 1,223 | 1,019 | 56 | 56 | 0 | 80 | 0 | 4 | +| gfx1153 | 47 | 1,269 | 1,061 | 56 | 56 | 0 | 86 | 0 | 4 | +| gfx1170 (caps substituted) | 152 | 33,748 | 32,373 | 114 | 114 | 0 | 243 | 0 | 13 | +| gfx1171 (caps substituted) | 152 | 37,925 | 36,504 | 114 | 114 | 0 | 195 | 0 | 12 | +| gfx1200 | 155 | 84,019 | 77,816 | 108 | 108 | 0 | 192 | 0 | 10 | +| gfx1201 | 152 | 51,930 | 48,098 | 108 | 108 | 0 | 207 | 0 | 9 | +| gfx1250 | 284 | 2,933 | 2,933 | 173 | 173 | 0 | 203 | 2 | 4 | +| **gfx1250-strict** | 278 | 2,868 | 2,868 | 171 | **168** | **3** | 195 | 2 | 4 | +| `all` (16 base archs, one run) | 2,964 | 682,765 | 534,327 | 427 | 427 | 0 (2 names, 2 sources each) | 2,189 | 34 | 27 | + +Logic files are those with at least one solution reaching the helper step. In every row the replay +equals the real output, no extra name is produced, and every kept object has the source the +required set calls for (where a name has one source). + +**Per logic file** (each file built on its own: 3,288 default files, the 16 experimental ones and +the GEMM+A2A one, 3,305 in total): 0 files with a missing, extra or wrong-source helper; 0 names +with two sources; 0 calls with both a length mismatch and a partial mask; 0 "needed but skipped" +names. Summed over files: 9,036 redundant object builds, 692 length-mismatch calls, 659 +partial-mask calls. + +### Confirming the drop + +On the real `gfx1250-strict` helper list (168 objects, 1,287 declared `__global__` kernels): +`Cijk_SB_PostGSU1_VW1`, `Cijk_SB_PostGSU1_VW2`, `Cijk_SB_PostGSU1_VW4` and `Cijk_SB_GG_PostGSU1_VW1` +are absent; `Cijk_SB_PostGSU16_VW1` and `Cijk_SB_PostGSU2_VW1` are declared. No other object emits +a `PostGSU1` kernel for this name: `PostGSU16` objects emit GSU 16, 8, 4, 2 only +(`KernelWriterConversion.__init__`, `gsuKernels`, `KernelWriterConversion.py:73`). + +The sequence in the real kernel-list order ("Kernel index" is the position in the list +`generateKernelHelperObjects` receives): + +| Kernel index | Logic file (under `gfx1250-strict/`) | `GlobalSplitU` | Algorithm | Predicted names | Objects | +| ---: | --- | ---: | --- | --- | --- | +| 1426 | `gfx1250-strict/Equality/gfx1250-strict_Cijk_Alik_Bljk_BBS_BH_UserArgs.yaml` | 0 | MultipleBuffer | `Cijk_SB_PostGSU0_VW{1,2,4}` (new) | `Cijk_SB_PostGSU16_VW{4,2,1}` (built) | +| 1811–1823, 2486 | five MX `F4BS`/`F8BS` files | 0 | MultipleBuffer | same, visited | not built (already present) | +| 2492 | `gfx1250-strict/Equality/gfx1250-strict_Cijk_Alik_Bljk_F8BS_BH_UserArgs.yaml` | 0 | **SingleBuffer** | same, visited | `Cijk_SB_PostGSU1_VW{4,2,1}`: **never built** | + +The `Alik_Bljk_F8BS_BH_UserArgs` logic file exists only under `gfx1250-strict/` (three copies: +`gfx1250-strict`, `_96cu`, `_192cu`), and only the `gfx1250-strict/gfx1250-strict` copy has the +`SingleBuffer` solution; no gfx1250 (non-strict) logic file has one, so the gfx1250 build has +nothing that calls for `PostGSU1`. Built on its own, the `F8BS_BH_UserArgs` file is fine (its +`MultipleBuffer` solutions have `GlobalSplitU` 1 and -1, which predict different names). + +#### Runtime reach (from the code, not run: no gfx1250 hardware here) + +The `SingleBuffer` solution is solution 1 of that file: `GlobalSplitU: 0`, `StreamK: 3`, +`StreamKForceDPOnly: 1` (the only `StreamKForceDPOnly: 1` in the gfx1250 and gfx1250-strict +logic). The generator makes it persistent data-parallel: `Solution.py:2325-2328` sets +`GlobalSplitU = 0`, `SupportUserGSU = False` and `GlobalSplitUAlgorithm = "SingleBuffer"` +(non-StreamK), and the `_GlobalAccumulation` assignment (`Solution.py:2170-2181`) leaves it `None` +for persistent data-parallel solutions, so the library gets `globalAccumulation = 0` +(`Contractions.py:811`; the capture confirms `_GlobalAccumulation` `None` for this solution). On the +host: + +- GSU is 0: `calculateAutoGSU` returns `sizeMapping.globalSplitU` unless it is -1 + (`ContractionSolution.cpp:2170`), and a user GSU is rejected because `SupportUserGSU` is off + (`checkInternalArgumentsSupport`, `ContractionSolution.cpp:5620`). The persistent data-parallel + branch also throws for GSU > 1 (`ContractionSolution.cpp:5141`). +- The output-conversion launch needs `(gsu > 1 && globalAccumulation && globalAccumulation != 3)` + or a parallel reduction (`ContractionSolution.cpp:5279`; `getAccumulation` returns the solution's + `globalAccumulation` when `AdaptiveGemmGSUA` is 0). Both GSU and `globalAccumulation` are 0, + and `resolvePersistentSettings` leaves the reduction at `none` for a non-StreamK persistent + solution (`ContractionSolution.cpp:5014-5033`; default in `ContractionSolution.hpp:521`). + +So no problem can make this solution ask for `Cijk_SB_PostGSU1_VW*`. The conversion objects +`initConversionKernelObjects` builds for it are themselves unused: it keys on +`GlobalSplitUAlgorithm` only, not on `_GlobalAccumulation`, so the benchmark path emits kernels +that nothing launches. The drop is real in the build output but harmless for this logic. It is +not harmless in general: a `SingleBuffer` solution with `GlobalSplitU` > 1 or -1 (`_GlobalAccumulation: +SingleBuffer`, `globalAccumulation = 1`) launches `..._PostGSU1_VW` whenever the chosen GSU is +above 1 (`gsu = 1` for `globalAccumulation == 1` before `outputConversionKernelName`), and the same +ordering would leave it without its kernel. + +### Mechanisms + +The name functions and object builders disagree in four ways +(`KernelHelperNaming.py`; `generateKernelHelperObjects` is `TensileCreateLibrary/Run.py:897`): + +| Difference | Names | Objects | Effect in shipped logic | +| --- | --- | --- | --- | +| `PostGSU` number | `KernelWriterConversion.kernelName` with the solution's `GlobalSplitU` (`PostGSU0`, `PostGSU1`, `PostGSU-1`, `PostGSU7`, ...) | `GlobalSplitUPGR` (16), or 1 for `SingleBuffer` (`KernelHelperNaming.py:171`, `:185`) | Solutions that need the same objects predict different names (most of the redundant builds). Solutions that need different objects can predict the same name: the gfx1250-strict drop. | +| `_GG` tag | solution's `GroupedGemm` | `GroupedGemm = False` (each object emits both variants) | More redundant builds only. | +| Vector widths | `[1, 2]` if `isDouble()` else `[1, 2, 4]` (`:51`) | by `numBytes()`: `[1]` if > 8, `[1, 2]` if > 4, else `[1, 2, 4]` (`:162`) | `ComplexFloat`: 3 names / 2 objects; `ComplexDouble`: 3 names / 1 object. After the sort the names are `VW4, VW2, VW1` against objects `VW2, VW1` (or `VW1`), so every position is paired with the wrong object, and `compress` ignores the surplus mask entries. Harmless only because the mask is all-`True` in every such call. | +| `MultipleBufferSingleKernel` | returns `None` (no names) | objects when `AdaptiveGemmGSUA != 0` (`:172`) | Such objects would never be built; no shipped solution has `AdaptiveGemmGSUA != 0`. | + +`BetaOnly`, `Reduction`, `ActivationEnumHeader` and `ActivationFunction` names match their objects +apart from `_GG`. + +Why the lucky outcomes hold, from the data: + +- The length mismatch only occurs for complex types, whose problem types have no bias list, so all + names of one call are new together or visited together. The partial masks (27 calls in the `all` + build: 19 conversion, 7 beta-only, 1 reduction) come from problem types sharing some bias-typed + names (e.g. an `HSS` problem type with bias types `[H, S]` whose `BiasS` names an earlier problem + type already claimed: mask `BiasH` new, `BiasS` visited) and from beta-only lists + that repeat one name per bias type when `BetaOnlyUseBias` is off (`Cijk_B_GA`, `Cijk_B_GA`). In + all of them the name and object lists have the same length and the same order (vector width + outer, bias type inner), so `compress` keeps the right objects. +- Predicted names that map to more than one object list are common (e.g. 41 of 2,089 on gfx942, 59 + of 192 on gfx1200): bias-typed names shared between problem types with different bias lists, + `Cijk_*_GA` beta-only names, and `PostGSU` names shared by solutions with different objects. + They only drop something when the first solution to claim the name does not build the object a + later solution needs, and no other solution builds it later. "Needed but skipped" is 0 in every + configuration except gfx1250-strict (3, never built). + +### Same name, different source (`BetaOnly`) + +`KernelWriterBetaOnly.kernelName` (`KernelWriterBetaOnly.py:301`) encodes `DestDataType`, grouped / +batched, bias type and `_GA`, but not `ComputeDataType`, while the kernel's `D` and `beta` types come +from `ComputeDataType` (`KernelWriterBetaOnly.py:42`). The host name (`betaOnlyKernelName`, +`ContractionSolution.cpp:4010`) omits it too. Found: + +| Build | Name | Sources (first solution) | Kept | +| --- | --- | --- | --- | +| gfx950 | `Cijk_I_GA` | `gfx950_Cijk_Ailk_Bjlk_I8IS_BH_HAS_Bias_HAS_SAV_UserArgs` (`float * D`, `float beta`); `gfx950_Cijk_Ailk_Bljk_I8II_BH_AI_SAV_UserArgs` (`int32_t * D`, `int32_t beta`) | float version | +| `all` | `Cijk_I_GA` | `navi32_Cijk_Ailk_Bljk_I8II_BH_UserArgs` (int32) first; the gfx950 `I8IS` file (float) | int32 version | +| `all` | `Cijk_I8_GA` | `gfx1170_Cijk_Ailk_Bjlk_I8BH_SHB_HA_S_SCD_SAV_UserArgs` (int32 compute) first; `gfx950_Cijk_Ailk_Bjlk_I8I8S_BH_HAS_Bias_HAS_SAV_UserArgs` (float compute) | int32 version | + +The diff of the two `Cijk_I_GA` sources is exactly the `D` pointer type, the `beta` type and the +`SCALAR_ZERO`/multiply casts (`float` versus `int32_t`). Both the TensileCreateLibrary path and the +benchmark path keep the first object per name, so this is a helper naming collision, independent of +the name prediction. The gfx1170 side of the `Cijk_I8_GA` collision comes from the +capability-substituted capture. + +Runtime reach (from the code, not run): the only beta-only launch is in `solve()` under +`gsu > 1 && globalAccumulation != 2 && globalAccumulation != 3` (`ContractionSolution.cpp:5127`; the +grouped-GEMM call at `:5416` is commented out), so `MultipleBuffer` (2) and +`MultipleBufferSingleKernel` (3) solutions never launch it. Every solution that produces a +colliding name, in every capture, is one of those with `AdaptiveGemmGSUA: 0` and not StreamK +(counts are unique solutions, kernels in parentheses): gfx950 `Cijk_I_GA` 11 (11) `MultipleBuffer` +(GSU 2); in the `all` build `Cijk_I_GA` 3,705 (2,824) `MultipleBuffer` (GSU -1 or 2–30) and +1,566 (1,444) `MultipleBufferSingleKernel`, `Cijk_I8_GA` 2,827 (2,396) `MultipleBuffer` and +4,909 (4,000) `MultipleBufferSingleKernel`. So the wrong-typed kernel is never launched with shipped logic; +it would be if a `SingleBuffer` solution with GSU > 1 or -1 (or an adaptive one) of one of these +problem types were added. + +## Conclusion + +- **Drop: yes, in one shipped configuration, harmless at run time today.** The `gfx1250-strict` + library lacks `Cijk_SB_PostGSU1_VW{1,2,4}` (and their `_GG` variants), which one `SingleBuffer` + solution calls for in the build. That solution is persistent data-parallel and never launches the + conversion kernel, so nothing requests the missing kernels. The cause is the `PostGSU` number in + the predicted name; it depends on solution order, so any logic change can add or remove such drops + without warning, including ones a launching `SingleBuffer` solution would hit. +- **Duplicate: no duplicate names reach the output** (name dedupe at the end). Objects are built + redundantly (up to ~7× on gfx942), a cost, not a correctness issue. +- **Mispair: the positional pairing is wrong for every complex-type call** (3 names, 1–2 objects), + but no shipped configuration hits the combination (length mismatch with a partially-visited mask) + that would select the wrong objects. `MultipleBufferSingleKernel` + `AdaptiveGemmGSUA` would drop + helpers but is not used by shipped logic. +- **Separately**, the `BetaOnly` name omits the compute type, so `Cijk_I_GA` (gfx950 and `all`) and + `Cijk_I8_GA` (`all`) each stand for two different kernels and one problem type gets the other's. + Latent: all solutions involved are `MultipleBuffer`/`MultipleBufferSingleKernel`, for which the + host never launches a beta-only kernel. + +Not fixed as part of this plan (DECISIONS.md topic 8). A fix direction, for whoever picks this up: +dedupe on the names of the built objects (or on the F2 input key from the benchmark path) instead +of predicted names, and add the compute type to the beta-only kernel name on both the Python and the +host side. + +## Not measured / open items + +- Runtime reach of the dropped `gfx1250-strict` kernels and of the beta-only collisions is derived + from the host code, not run (no gfx1250 hardware here; the gfx950 beta-only case was not run + either). +- gfx1170/gfx1171 with a compiler that supports them (capabilities were substituted, see Method). +- Builds with an explicit `--logic-filter` other than one file, or arch subsets other than the ones + listed. The per-file and per-arch results bracket them, but order-dependent drops like the + gfx1250-strict one can appear in any combination that puts a `MultipleBuffer` solution with the + same `GlobalSplitU` value before a `SingleBuffer` one of a problem type with the same conversion + name. diff --git a/projects/hipblaslt/tensilelite/timing_runs/times.txt b/projects/hipblaslt/tensilelite/timing_runs/times.txt new file mode 100644 index 000000000000..19cb2f9be521 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/times.txt @@ -0,0 +1,3 @@ +build rc=0 60s +run rc=0 26s +DONE diff --git a/projects/hipblaslt/test-client-plan.md b/projects/hipblaslt/test-client-plan.md new file mode 100644 index 000000000000..36bdfdc5b3f7 --- /dev/null +++ b/projects/hipblaslt/test-client-plan.md @@ -0,0 +1,55 @@ +# Goal +Improve the speed of the tensilelite common tests, and of the standalone test done through that path. + +# Problem +The current testing is done through using code written to check both correctness AND performance through benchmarking. The benchmark requirements places additional contrainst on the numerical tesing which make it harder to improve performance. E.g. it include hard barriers and forces gpu->host drains before it continues to the next problem. + +I'm attaching data for the two slowest tensilelite common tests on gfx942: dtv.yaml and cgemm.yaml. I'm only including the run side of things since this is what needs to live on the GPU-node. + +Here is the breakdown of the top times, with the gpu work also included: + +dtv.yaml: 84s total + - 23.8s waiting for gpu buffers to be reset between solutions + - 29.0s for client setup + - 11.5s to do the gpu vs reference comparison + - 3.6s for reference gemm calc + - 2.3s for logging and perf calc + - 0.5s for actual gpu work + +cgemm: 41s total + - 12.5s for client setup + - 10.6s for the slow reference gemm + - 5.2s for the gpu vs reference comparison + - 2.0s for logging and perf calc + - 0.3s for actual gpu work. + +So for the run step, both of these yaml files spend less that 1% of their total wall clock time on the GPU. We need to seriously improve that. + + +# Some more info + +The yamls are set up as: +``` +BenchmarkProblems: + - case 1 + - ... + - case N +``` +The subentries of each run in parallel. with N sequential runs; N=9 for cgemm and N=25 for dtv). + +The CMS yamls take much longer per kernel in their build step because each case contains a single kernel, and Tensile is set up to parallelize only *within* a case, not across them. + +None of these times actually measure GPU time, we don't know exactly how long the kernel takes to solve from these traces as they're CPU only, kernel_solving only captures dispatch, dtv/cgemm's 10us per dispatch it too low. It's likely that this the GPU execution time is being misattributed to the validate_gpu_readback entry for validate_warmup ' +# Initial Thoughts +- client setup seems to be repeated N times, if that can be shared it's an immediate saving of (N-1)/N (22.9 s for dtv and 12 s for cgem, 27-30% speedup). +- gpu buffer readbacks should not block the way they do now, we should use a ring buffer on the gpu(s) to pipeline the {reseting, solving, transfer, validation} loop. +- The reference gemm calc should be made faster, or moved to GPU, and results should either be cached or runs internally re-ordered to not duplicate reference gemm calcs. +- Logging and perf calc can be removed entirely. Numerical correctness should done on an all-or-nothing basis. We don't need to know which entries are wrong, we just need to know that they are. A separate path can be made available for letting the user figure out which values are wrong as useful debug info, but for CI we only need a pass-fail. +- New approach takes the pre-compiled kernels (as first pass) and pipelines everything (loading, data generation, reference solution, data transfers, gpu work, cleanup, validation). +- Looking at the times from the dtv run, the + +# Outstanding questions +- How much of the GPU can we claim for ourselves? How much of its RAM +- Can we get rid of ClientExecutionLockPath? +- Do we want to still run them as individual yaml files, or can we group them together into a single stream of tests to do? +- How to handle the A2A fusions, and any other kernels that requires multiple-gpus \ No newline at end of file From b718d58d2ca6ebcd491f687a3bcbe56175e02640 Mon Sep 17 00:00:00 2001 From: Alex Vasile <48962821+Alex-Vasile@users.noreply.github.com> Date: Fri, 6 Nov 2026 20:36:55 +0000 Subject: [PATCH 2/2] docs(tensilelite): keep only the before/after timing aggregates and their scripts Keep the 8 per-config timing aggregates of the before and after points (gfx950 baseline, after the run-side changes and after the build-side changes; gfx942 before and after) and the scripts that reproduce them. Remove the exploratory runs, the early gfx950 passes, the investigation and decision notes, the write-ups, the test-outcome lists, the comparison tables, the split times and the other scripts (66 files). They stay in the parent commit's history. Add before_after/run_side.sh, which runs one side of a before/after timing (CI-like split, shared config list, per-config build and run), and before_after/compare_aggregates.py, which prints two per-config aggregates side by side; it reproduces the removed gfx942 build_comparison.txt from the same data. run_split.sh gains DIST (pytest-xdist scheduler, default worksteal as in the hw-common CI category) and CLEAR_RUN_CACHE (default 0: the run phase keeps the cache the build phase left); the removed split numbers were measured with DIST=load and CLEAR_RUN_CACHE=1. --- .../hipblaslt/tensilelite/CI_SPLIT_TIMING.md | 78 -- .../tensilelite/timing_runs/DECISIONS.md | 963 ------------------ .../tensilelite/timing_runs/DEFERRED.md | 153 --- .../timing_runs/SynchronizerValidator.md | 265 ----- .../build_fp16_gradient_bias_analysis.txt | 135 --- .../cgemm/cgemm_1_build_cold_analysis.txt | 135 --- .../cgemm/cgemm_2_build_warm_analysis.txt | 127 --- .../cgemm/cgemm_3_run_analysis.txt | 173 ---- .../tensilelite/timing_runs/cgemm/times.txt | 4 - .../timing_runs/cms/CMS_FINDINGS.md | 110 -- ...tom_mainloop_scheduling_build_analysis.txt | 135 --- ...ainloop_scheduling_tf32_build_analysis.txt | 135 --- .../custom_mainloop_scheduling_tf32_time.txt | 23 - .../cms/custom_mainloop_scheduling_time.txt | 23 - ...ainloop_scheduling_warm_build_analysis.txt | 135 --- .../custom_mainloop_scheduling_warm_time.txt | 23 - .../cms/f8bf16_tn_cms_build_analysis.txt | 8 - .../timing_runs/cms/f8bf16_tn_cms_time.txt | 24 - .../tensilelite/timing_runs/cms/run_split.sh | 8 - .../timing_runs/cms/run_split_warm.sh | 8 - .../timing_runs/cms/split_25_result.txt | 3 - .../timing_runs/cms/split_25_warm_result.txt | 3 - .../timing_runs/cms/split_5_result.txt | 3 - .../timing_runs/cms/split_experiment.py | 42 - .../tensilelite/timing_runs/cms/times.txt | 4 - .../timing_runs/cold_warm_done.txt | 1 - .../cpu_data_init_investigation.md | 191 ---- .../cpu_reference_slow_path_investigation.md | 442 -------- .../tensilelite/timing_runs/driver.log | 0 .../dtv/dtv_1_build_cold_analysis.txt | 135 --- .../dtv/dtv_2_build_warm_analysis.txt | 127 --- .../timing_runs/dtv/dtv_3_run_analysis.txt | 175 ---- .../tensilelite/timing_runs/dtv/times.txt | 4 - .../tensilelite/timing_runs/full_runs/run.sh | 31 - .../full_runs_gfx942/FURTHER_SPEEDUPS.md | 483 --------- .../timing_runs/full_runs_gfx942/README.md | 54 - .../full_runs_gfx942/build_comparison.txt | 43 - .../full_runs_gfx942/run_comparison.txt | 85 -- .../full_runs_gfx942/split_post/times.txt | 6 - .../full_runs_gfx942/split_pre/configs.txt | 123 --- .../full_runs_gfx942/split_pre/times.txt | 6 - .../full_runs_gfx950/after_build.md | 269 ----- .../after_build/split_build_outcomes.txt | 572 ----------- .../timing_runs/full_runs_gfx950/after_run.md | 288 ------ .../after_run/common_test_outcomes.txt | 572 ----------- .../timing_runs/full_runs_gfx950/baseline.md | 268 ----- .../baseline_run/common_test_outcomes.txt | 571 ----------- .../before_after/compare_aggregates.py | 97 ++ .../full_runs_gfx950/before_after/run_side.sh | 83 ++ .../per_config_build/AGGREGATE_analysis.txt | 598 ----------- .../per_config_rerun/warm_cache_check.sh | 28 - .../per_config_rerun_2/AGGREGATE_analysis.txt | 754 -------------- .../full_runs_gfx950/profiling/run_profile.sh | 45 - .../full_runs_gfx950/profiling/run_rerun.sh | 30 - .../timing_runs/full_runs_gfx950/run.sh | 29 - .../full_runs_gfx950/split/run_split.sh | 25 +- .../verify_mx/run_verify_mx.sh | 21 - .../input_data_preparation_investigation.md | 233 ----- .../per_solution_bookkeeping_investigation.md | 721 ------------- ...python_kernel_bench_setup_investigation.md | 232 ----- ...ython_kernel_build_src_co_investigation.md | 530 ---------- .../python_kernel_codegen_investigation.md | 508 --------- .../tensilelite/timing_runs/run_cms.sh | 21 - .../tensilelite/timing_runs/run_cold_warm.sh | 45 - .../run_fp16_gradient_bias_analysis.txt | 175 ---- .../tensilelite/timing_runs/run_timing.sh | 26 - .../tcl_helper_name_mismatch_investigation.md | 335 ------ .../tensilelite/timing_runs/times.txt | 3 - projects/hipblaslt/test-client-plan.md | 55 - 69 files changed, 198 insertions(+), 11562 deletions(-) delete mode 100644 projects/hipblaslt/tensilelite/CI_SPLIT_TIMING.md delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/DECISIONS.md delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/DEFERRED.md delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/SynchronizerValidator.md delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/build_fp16_gradient_bias_analysis.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_1_build_cold_analysis.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_2_build_warm_analysis.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_3_run_analysis.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cgemm/times.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/CMS_FINDINGS.md delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_build_analysis.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_tf32_build_analysis.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_tf32_time.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_time.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_warm_build_analysis.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_warm_time.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/f8bf16_tn_cms_build_analysis.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/f8bf16_tn_cms_time.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/run_split.sh delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/run_split_warm.sh delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/split_25_result.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/split_25_warm_result.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/split_5_result.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/split_experiment.py delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cms/times.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cold_warm_done.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cpu_data_init_investigation.md delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/cpu_reference_slow_path_investigation.md delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/driver.log delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_1_build_cold_analysis.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_2_build_warm_analysis.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_3_run_analysis.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/dtv/times.txt delete mode 100755 projects/hipblaslt/tensilelite/timing_runs/full_runs/run.sh delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/FURTHER_SPEEDUPS.md delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/README.md delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/build_comparison.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/run_comparison.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_post/times.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_pre/configs.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_pre/times.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_build.md delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_build/split_build_outcomes.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_run.md delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_run/common_test_outcomes.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline.md delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline_run/common_test_outcomes.txt create mode 100755 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/before_after/compare_aggregates.py create mode 100755 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/before_after/run_side.sh delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_build/AGGREGATE_analysis.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_rerun/warm_cache_check.sh delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_rerun_2/AGGREGATE_analysis.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/profiling/run_profile.sh delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/profiling/run_rerun.sh delete mode 100755 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/run.sh delete mode 100755 projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/verify_mx/run_verify_mx.sh delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/input_data_preparation_investigation.md delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/per_solution_bookkeeping_investigation.md delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/python_kernel_bench_setup_investigation.md delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/python_kernel_build_src_co_investigation.md delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/python_kernel_codegen_investigation.md delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/run_cms.sh delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/run_cold_warm.sh delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/run_fp16_gradient_bias_analysis.txt delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/run_timing.sh delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/tcl_helper_name_mismatch_investigation.md delete mode 100644 projects/hipblaslt/tensilelite/timing_runs/times.txt delete mode 100644 projects/hipblaslt/test-client-plan.md diff --git a/projects/hipblaslt/tensilelite/CI_SPLIT_TIMING.md b/projects/hipblaslt/tensilelite/CI_SPLIT_TIMING.md deleted file mode 100644 index 75a55bf419b6..000000000000 --- a/projects/hipblaslt/tensilelite/CI_SPLIT_TIMING.md +++ /dev/null @@ -1,78 +0,0 @@ -# TensileLite common tests: CPU build / GPU run split timing - -Measured 2026-10-15 on a gfx942 box, one run per mode, under CI-like conditions. -Uses the `--build-only` / `--use-cache` modes added by PR #6188 (merged). - -## Results - -| Phase | Command mode | Wall time | Exit | Passed / failed / skipped | -|---|---|---|---|---| -| Build | `--build-only --gpu-targets gfx942 --artifact-dir D` | 1,220 s (20m 20s) | 1 | 133 / 1 / 1,508 | -| Run | `--use-cache --artifact-dir D` | 1,592 s (26m 32s) | 0 | 138 / 0 / 1,509 | -| Combined | default (what CI runs today) | 2,692 s (44m 52s) | 0 | 138 / 0 / 1,509 | - -- Split total, back to back: 2,812 s (46m 52s), about 2m longer than combined. -- **GPU-node time: 1,592 s instead of 2,692 s, a saving of 1,100 s (18m 20s, about 41%).** -- The 1,220 s build moves to a CPU-only runner. -- Pipelining: with several arch builds queued on CPU runners, GPU runners only see the run phase. -- The build exit code is 1 because of a single config, see "Findings". -- Not included: artifact upload/download/extract and per-job setup. - The artifacts total 409 MB compressed (123 `.tar.gz`, about 3.76 GB uncompressed, median 0.46 MB, - largest `gemm/dtv` 76.7 MB, then `gemm/fp16_tn` 35.7 MB and `gemm/bufferLoadStore0` 31.1 MB). - Transfer cost must be added when deciding the real saving. - -## Conditions - -Taken from TheRock's `fetch_test_configurations.py` / `pytest_runner.py` and the log of a failing -`tensilelite-common` job on PR #12151 (job 116559736872, gfx94X). No passing CI job was available to -get CI timings, so none are compared here. - -CI (gfx94X): 1 GPU (`ROCR_VISIBLE_DEVICES`), `KUBE_CPU_REQUEST=25` so `OMP_NUM_THREADS` and -`OPENBLAS_NUM_THREADS` are 25, `pytest -v -m "(common) and not skip-gfx94X"`, `--numprocesses=4` -(`parallel_workers: 4` for `hw-common`), `--timeout=5400`, `--prebuilt-client=`, -`--global-parameters=LibraryFormat='msgpack'`, `--tensile-options=--cxx-compiler,/bin/amdclang++`. - -Local reproduction: `taskset -c 0-24` (25 cores), `ROCR_VISIBLE_DEVICES=0`, `OMP_NUM_THREADS=25`, -`OPENBLAS_NUM_THREADS=25`, `-n 4`, `--timeout=5400`, marker `common and not skip-gfx94X`, the same -`--global-parameters` / `--tensile-options`. `~/.tensile` and `~/.tensilelite` were deleted before -each phase. Script: `/tmp/tl_split/run.sh`; logs in `/tmp/tl_split/`. -The machine has 96 cores and 8 GPUs and is shared, so there was contention (load average up to about -100 at times); treat the numbers as one noisy sample. - -To emulate a CPU-only build node, the five GPU-gated tests in `Tests/common/flags/gfx942/` -were deselected in the build phase (see Findings). - -## Findings - -1. **CI marker uses the family name.** The CI marker is `not skip-gfx94X`, but `pytest.ini` registers - exact-arch markers (`skip-gfx942`). It selects 1,647 tests instead of 387. The configs that declare - `skip-gfx942` are still skipped at run time by `configMarks` using the detected arch, so results are - the same, but the marker is not doing the filtering. -2. **Build phase needs a msgpack-capable client.** CI passes `LibraryFormat='msgpack'` with - TheRock's client. The locally built `build_tmp` client was built with `HIPBLASLT_ENABLE_YAML=ON` - and has no msgpack support: 5 tests failed with `Failed to load solution library`. - The runs here use a client built with `-DHIPBLASLT_ENABLE_YAML=OFF`. -3. **`comm/gfx950/fused_a2a.yaml` fails the build phase** with `Your parameters resulted in 0 valid - solutions`. It is a gfx950-only config with no `skip-gfx942` mark. In the combined phase it is skipped - because `comm` tests need 2+ GPUs. With `--build-only --gpu-targets gfx942` it is built anyway and - fails, so a CI build job for gfx942 would go red on it. It needs a `skip-gfx942` mark or a skip - when no multi-GPU is possible. -4. **`Tests/common/flags/gfx942/` is collected in every mode.** `pytest_ignore_collect` filters only - direct children of `common/`. `test_compile` runs on CPU; the other five tests are gated on - `has_arch("gfx942")` and use the GPU, so they should skip on a CPU node and run in the run phase. - `has_arch` uses `rocm_agent_enumerator`/`amdgpu-arch` and ignores `ROCR_VISIBLE_DEVICES`, - so hiding the GPU that way does not skip them. Skipping on a real GPU-less node was not run. -5. **The artifact is a directory of per-config tarballs, not one zip.** CI will need to upload the - directory per target, and gfx120X (gfx1200/gfx1201) needs one per target. -6. **Setup pitfalls.** `~/.tensile` and `~/.tensilelite` were read-only in the agent sandbox, which made - early runs fail; the final runs were done unsandboxed. - -## Caveats - -- Single run per mode; shared, loaded machine. Build phase time scales with cores (25 here); a - CPU runner with more cores will be faster. -- Run and combined phases include the five `flags/` GPU tests; the build phase does not. -- The run phase measures only GPU-side execution on an already-built artifact; it does not include - extracting the 409 MB of artifacts (included in the 1,592 s, since `test_config_run` extracts each). -- CI timings were not available for comparison (the observed CI jobs failed early for an unrelated - `PytestRemovedIn9Warning`). diff --git a/projects/hipblaslt/tensilelite/timing_runs/DECISIONS.md b/projects/hipblaslt/tensilelite/timing_runs/DECISIONS.md deleted file mode 100644 index 36f2dbb3124f..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/DECISIONS.md +++ /dev/null @@ -1,963 +0,0 @@ -# Timing investigation decisions - -Record of the decisions taken on each investigation in this directory, one topic at a time. -It is the input to the implementation plan at the end of this file. - -Status values: `pending` (not discussed yet), `discussing`, `decided`. - -Deferred and "not for now" work is tracked separately in [DEFERRED.md](DEFERRED.md). - -## Agenda - -Ordered by measured cost, run side first (gfx950 `per_config_rerun_2`), then build side -(gfx950 `per_config_build`). - -| # | Investigation | Cost in aggregate | Status | -| --- | --- | --- | --- | -| 1 | [CPU reference slow path](cpu_reference_slow_path_investigation.md) | 741 s slow path (of 1,080 s Reference GEMM, 42% of run wall) | decided | -| 2 | [Input data preparation](input_data_preparation_investigation.md) | 503 s (19.6%) | decided | -| 3 | [SynchronizerValidator](SynchronizerValidator.md) | 163 s (6.3%) | decided | -| 4 | [`cpu_data_init`](cpu_data_init_investigation.md) | 107 s (4.2%) | decided | -| 5 | [Per-solution bookkeeping](per_solution_bookkeeping_investigation.md) (incl. correctness-only runs) | 68 s (2.7%), plus correctness-mode items | decided | -| 6 | [`python_kernel_build_src_co`](python_kernel_build_src_co_investigation.md) | 1,008 s (39.1% of build wall) | decided | -| 7 | [`python_kernel_codegen`](python_kernel_codegen_investigation.md) | 593 s (23.0%) | decided | -| 8 | [`python_kernel_bench_setup`](python_kernel_bench_setup_investigation.md) | 132 s (5.1%) | decided | - -## Cross-cutting items - -These show up in more than one note. Each is decided once, in the topic listed. - -| Item | Appears in | Decide in | -| --- | --- | --- | -| `KMP_BLOCKTIME=0` / idle OpenMP workers spinning | 1, 5 | 5 (decided: adopt, in the client) | -| GPU wait misattributed to the next blocking call (warmup sync placement, `SyncsPerBenchmark: 0`) | 2, 3, 5 | 3 (decided: syncs only under timing instrumentation) | -| `copyInputs` / `copyInputBuffers` sized by `maxElements` | 2, 4 | 2 (decided: adopt) | -| Redundant MX regeneration and the CPU reference recompute it triggers | 2, 4 | 2 (decided: adopt) | -| `copyInputs` restoring read-only inputs vs only outputs + Synchronizer | 2, 3 | 2 (decided: not now) | -| Pool sizing (`CPUThreadCount`, oversubscription under xdist) | 7 (and every build-side pool) | 7 (decided: 2× CPU budget) | - -## Decisions - -### 1. CPU reference slow path - -Status: decided. - -Adopted: - -- **A.** In the slow path (`ReferenceSolution<…>::SolveCPU`), enter `#pragma omp critical` only - when `problem.outputAmaxD()` is true, and drop the per-element `omp_set_num_threads`. -- **A+.** Replace the AMAX-D critical section with a per-thread maximum combined once - (`reduction(max:…)` or equivalent). Replace the hard-coded `MAX_OMP_THREADS 64` cap with - `omp_get_max_threads()`, so the reference follows `OMP_NUM_THREADS`. -- **B.** Fast-path (`solveCPUFast`) support for: - - Int8 inputs, with outputs I8, I32, Half and BF16. Accumulation must be exact (a float - accumulator is not exact above 2²⁴). - - Float8 outputs (`F8 × F8 → F8`, BF8 variants, and `F8 × Half → F8` with the mixed-input - shadow). - - Write-back must match the slow path's `SaturateCast`. -- **C.** Fast-path support for the smaller rejection reasons: `gradient` (including the bias - reduction), `useE`, MX K not a multiple of the MX block (partial last block), and complex types. - `NumElementsToValidate: 128` stays on the slow path, as intended. - -Constraint: the new fast-path support must follow the established fast-path patterns (as in the -FP8/FP4, f64 and tf32 additions): `isFastPathEligible`/`solveCPUFast` in `Reference.cpp`, -fast/slow `CPUGemm.*` CTest pairs through `cpu-gemm-driver` in `tests/CMakeLists.txt`, and -gtest cases in `tests/ReferenceFastPath_test.cpp` / `ReferenceMXFastPath_test.cpp`. - -Not adopted / open: - -- **D** (making the fast-path rejection reason observable) was not chosen. -- **Int8 accumulation (decided): integer accumulation, then the slow path's epilogue type.** - Accumulate the K loop in `int32_t` (exact, as the GPU's int8 MFMA does), then apply alpha, - beta, bias, activation and the saturating cast in the slow path's accumulator type for that - type id: `int32_t` for `I8_I8_I32` / `I8_I32_I32` (int32 alpha), `float` for `I8_I32_S`, - `I8_I8_S`, `I8_H_S`, `I8_B_S` (float alpha). This needs the fast path to separate its - multiply-accumulate type from its epilogue type (a new template parameter, in the style of - `OperandMathOpT`). Reusing the `double` instantiation was considered and not chosen. -- `KMP_BLOCKTIME` is decided under topic 5. - -### 2. Input data preparation - -Status: decided. - -Adopted: - -- **Fix 1. Skip unchanged per-solution uploads (`copy_valid_to_gpu`).** On the slow reset path, - skip the H2D upload of host `valid` data to GPU `valid` buffers when the host data has not - changed since the last upload (tracked with `m_inputEpoch`). -- **Fix 2. Cache the batched-pointer arrays (`batched_inputs`).** Build and upload them per - problem; re-upload only when the epoch or a buffer base address changes, not per solution. -- **Fix 4. No MX regeneration per solution.** - - Data and canonical scale are generated once at problem level and never regenerated per - solution. - - The GFX950 swizzled scale is produced by permuting the canonical scale - (`preSwizzleScalesGFX950` in mxDataGenerator's `PreSwizzle.hpp`), once per problem, and - cached, ideally in its own GPU buffer so solutions using the canonical and swizzled layouts do - not overwrite each other. - - `m_inputEpoch` is not bumped for it, so the second CPU reference computation goes away. - - Required check: the permuted scale is byte-identical to today's - `generateMXInput(…, GFX950)` output, including padding and the K-fast restride. -- **Shared fix. Size copies by the current problem.** `copyInputs` / `copyInputBuffers` (and the - per-problem `copy_valid_to_gpu`) copy the current problem's allocated extent instead of - `maxElements`. - -Not adopted: - -- **Fix 3 (restore only writable tensors in `copyInputs`): not doing it for now.** Too easy to - get the writable set wrong. - -Measurement: no per-topic measurement; covered by the per-phase aggregate reruns in the -implementation plan. - -### 3. SynchronizerValidator - -Status: decided. - -Adopted: - -- **Item 1. GPU check.** An OR-reduction kernel over the Synchronizer buffer, enqueued on the - client's `stream`, followed by a 4-byte readback. Only when the result is nonzero: a full readback - and the existing scan to build the diagnostic, mark the solution `FAILED`, and clear the buffer. -- **Item 1-alt. Vectorized host check, kept as the FFM-emulator path.** Full readback, then an - OR-reduction first (auto-vectorizes), with the count/first-offset scan only when dirty. Used when - running under the FFM emulator (`HSA_MODEL_MEMFILE`), where the GPU kernel's cost is unknown. To - be tested and tuned later based on measured performance. -- **Diagnostic unchanged.** The dirty-buffer message (nonzero count, first offset) stays as it is, - built from a full readback only on the dirty path. -- **Item 2. Move the check to `postSolution`**, so it sees every launch of the solution (all - warmups and benchmark runs) on both reset paths, including the last solution's. Requires saving - the `ProblemInputs` pointer (from `preSolution` / `validateWarmups`). Ordering correction (from - the plan review, verified in `MetaRunListener.cpp`): `postSolution` runs listeners in reverse, - so `SynchronizerValidator::postSolution` runs *after* `ReferenceValidator::postSolution`, which - is what lets a dirty-buffer `FAILED` override the reference verdict. Keep the registration order - unchanged and do the check inside `SynchronizerValidator::postSolution`. With guards: - 1. **Only check if the solution launched something.** Set a flag in `preWarmup` / - `preEnqueues`; keep the existing `m_mayUseSynchronizer` gate. `postSolution` also runs for - rejected and never-run solutions. - 2. **Exception path.** If the device is usable, check as normal. If the device is lost, catch - the check's own HIP error and only mark the solution `FAILED`; never throw out of - `postSolution`. - 3. **Explicit, separately timed sync before the check**, so queued warmups 1..N-1 - (`SyncsPerBenchmark: 0`) are not absorbed by the check. Only when timing instrumentation is - on (see "Attribution syncs" below); otherwise the check's own readback does the waiting. - - Keep clear-on-dirty, so every checked solution ends with a clean buffer. - -Not adopted: - -- Adding a Synchronizer reset to the fast reset path (`resetOutput`). Not needed with the - `postSolution` check. - -Known limitations (accepted): - -- Solutions classified as unable to use the Synchronizer are not checked (as today). -- **Correction (plan review, verified):** not every solution has one run iteration. - `BoundsCheck: 4` (GuardPageAll, e.g. `bufferLoadStore0`) sets `m_numRunsPerSolution = 2` - (`DataInitialization.cpp`); the second iteration's `prepareGPUInputs` restores the - Synchronizer and wipes the first iteration's residue before `postSolution`. Today's - `validateWarmups` check runs in each iteration. **Decided:** run the check at the end of every - run iteration (before the next iteration's input reset), and report the verdict in - `SynchronizerValidator::postSolution`. Solutions with one iteration behave exactly as planned; - GuardPageAll solutions get both iterations checked. -- Cache effects of the reduction kernel on timed runs: resolved. In `postSolution` it runs after - the solution's timed runs, never between its warmups and timed runs. - -- **Item 3. Wait attribution after warmup 0, only when timing instrumentation is on.** In - `main.cpp`, after launching warmup 0 and before `listeners.validateWarmups`, wait for the stream - (or warmup 0's stop event) under its own timer, **only if `g_timingInstrumentationEnabled`** - (`--timing-instrumentation`). Its only purpose is attribution: without it, - `ReferenceValidator`'s `validate_gpu_readback` absorbs the kernel's runtime once the Synchronizer - check moves to `postSolution`. When instrumentation is off, no extra block is added, so the - launch/host overlap is unchanged. - - Accepted side effect: instrumented runs block where uninstrumented runs do not, so their - schedule differs slightly from production runs. - -- **Attribution syncs, only when timing instrumentation is on.** All explicit syncs added for - attribution follow the same rule as item 3: - - the wait after warmup 0 (item 3); - - guard 3, the sync before the Synchronizer check; - - **an end-of-iteration sync in `main.cpp` for every run iteration that launched** (one per - solution except GuardPageAll, which has two), under its own timer, so warmups 1..N-1 of - `SyncsPerBenchmark: 0` configs are no longer charged to the next solution's - `gpu_input_reset` (the `batched_inputs` inflation from topic 2). For checked solutions this - is the same sync as guard 3. - -Noted, no action: - -- `BenchmarkTimer::validateWarmups` (`--sync-after-warmups`) waits on the last warmup's stop - event, which is not yet recorded when `NumWarmups > 1`, so it returns immediately (code - reading). It does not affect benchmark results: timed runs either record their start event on - the stream after the warmups (GPU timer) or call `hipDeviceSynchronize` first (host timer). - -### 4. `cpu_data_init` - -Status: decided. - -Adopted: - -- **Item 1. Skip the discarded MX reference floats.** Add an opt-out to `generateMXInput` - (`clients/common/src/mxDataGen.cpp`) so it does not build (and free) the `getReferenceFloat()` - vector. Only the TensileLite client opts out; hipBLASLt's own clients (`testing_matmul.hpp`, - `fast_check_gtest.cpp`) use the returned reference, so the default stays unchanged. -- **Item 2a. Cheaper trig init loop, same values.** Hoist the per-element `std::vector` - allocation, and use a flat index for packed strides instead of `CoordNumbered` + - out-of-line `TensorDescriptor::index`. Generated values must stay identical. -- **Item 5. Dynamic OpenMP schedule for the init loops** (`schedule(dynamic, chunk)`). Values do - not depend on the schedule: Trig/Serial values depend only on the element index, and Random init - already uses unseeded per-thread generators. -- `copyInputs` sizing by `maxElements` (finding 3): covered by the shared fix in topic 2. - -Not adopted: - -- **Item 2b. `sinf` / argument reduction in trig init.** Keep double `sin` for now, so the generated - data does not change. -- **Item 3. Removing the dead `m_cpuInit` branch.** Leave it as is. - -Deferred: - -- The MX generator's other costs (single-threaded `resize` zero-fill, 7–11× scaling on 25 - threads) in shared hipBLASLt client code. -- First-call fixed cost per client process (finding 4): the note has no fix. - -### 5. Per-solution bookkeeping and correctness-only runs - -Status: decided. - -#### 5a. Per-solution bookkeeping - -Adopted: - -- **A. Stop idle OpenMP workers from spinning.** Call `kmp_set_blocktime(0)` in the client's - `main.cpp` unless `KMP_BLOCKTIME` is already set in the environment. Covers every client run. - Also expected to speed up the CPU reference (`cpu_reference_gemm` 2–5× faster on dtl with - `KMP_BLOCKTIME=0`). This settles the cross-cutting `KMP_BLOCKTIME` item. -- **C. No predicate-tree dump by default.** Gate `debugEval` in `SolutionIterator::checkSolution` - behind its own client option, default off, driven by a Tensile global parameter. Per-solution - rows are unaffected. - -- **D1. Stop the per-solution stdout CSV row in test runs.** - - `LogReporter::postSolution` checks whether the row's level will be printed *before* copying - the row (`readCurrentRow`), so a suppressed row costs almost nothing. - - Test runs use `ClientLogLevel` Error (0): only FAILED rows and mismatch lines are printed. - Manual benchmarking runs keep the current default and their per-solution table. - - **How a run is identified as a test run:** the `tensile_args` fixture in - `Tests/conftest.py` always appends `--global-parameters ClientLogLevel=0`, placed *before* - any user-supplied `--global-parameters` so a developer can still override it (e.g. - `ClientLogLevel=3`; Tensile applies repeated `--global-parameters` in order, last wins, and - they override the YAML). No Tensile code change. Covers `common/` and `custom/` YAMLs. - A dedicated `TestRun` global parameter was considered and not chosen: nothing else needs a - "this is a test" signal. - - **Test:** move the body of `tensile_args` into a plain helper and unit-test it: - `ClientLogLevel=0` is present, and a user-supplied `--global-parameters ClientLogLevel=3` - comes after it. (A `LogReporter` gtest and an end-to-end check were considered and not - chosen.) -- **D2. Skip the granularity/memory values when nothing consumes them.** - `BenchmarkTimer::preSolution` does not compute or report its 9 granularity/memory values unless - the stdout row will be printed or `CSVExportExtraCols` is set. -- The results-file CSV (`ResultFileReporter`) is handled in 5b (write it only when consumed). - -Not adopted: - -- **B. Cheaper formatting** (`to_chars`, no row copy, `'\n'` + per-problem flush). Dropped in - favour of not producing the output; only the D1 level check is kept. - -Deferred: - -- **D3. Typed reporter keys** in `MetaResultReporter`. - -#### 5b. Correctness-only runs - -Adopted: - -- **E. Infer "nothing is timed" in the client.** Skip `projectedPerformance` and the per-solution - performance calculation; report the NaN values directly, reproducing today's `-nan` text. Rule - (static, from the client's own arguments): nothing is timed when `num-syncs-per-benchmark == 0`, - or `max-enqueues-per-sync == 0`, or (`num-enqueues-per-sync == 0` **and** - `min-flops-per-sync == 0`). Note: prototype `corr`'s `num-enqueues-per-sync == 0` alone is wrong. - Per-solution run-time skips (restore-from-log, skip-slow) are unchanged. D2 falls under this. -- **F. No warmup timing events when nothing is timed.** Do not create the per-kernel start/stop - `hipEvent`s; only `BenchmarkTimer` reads them. The topic 3 warmup-0 attribution sync uses - `hipStreamSynchronize` so it does not depend on events. -- **G. `gemm/gfx950/xfp32.yaml`: drop `NumWarmups: 5` and `EnqueuesPerSync: 10`.** The two - work-stealing configs keep their deliberate extra warmups (now checked by the `postSolution` - Synchronizer check). -- **H. Copy the results CSV only if it exists** (`BenchmarkProblems.main`, same guard as the - granularity CSV). -- **I. Write the results CSV only when consumed** (the YAML has a `LibraryLogic:` section or uses - the Ductile backend). Otherwise Tensile passes "no results file" and the client has an explicit - path that never constructs `ResultFileReporter`. -- **J. Extend the `SyncsPerBenchmark` guard test** (`Tests/unit/test_common_yaml_syncs_per_benchmark.py`) - to `Tests/custom/`, and set `SyncsPerBenchmark: 0` in the 12 custom YAMLs. -- **K. Remove `LibraryLogic:` / `LibraryClient:` from common YAMLs**, with a whitelist test (same - style as the guard test: mandatory reason, stale entries fail). Scope: `common/` and `custom/`. - Whitelist the 6 files that test the logic/library path with real benchmark data - (`gemm/fp8nfp16mix_hfp8ns`, `gemm/xfp32`, `groupedgemm/grouped_gemm`, - `groupedgemm/gfx11/grouped_gemm_gfx11`, `groupedgemm/grouped_gemm_ck_gfx942`, - `sparse/gfx94x/libray_logic`). Remove from the 14 `ductile/*`, 10 `gemm/gfx950/*mxfp8*` and 12 - `custom/*` files. - - **`Prediction` coverage: replace the 10 mxfp8 sections with one new, minimal, non-MX - `Prediction` config** (option c). The 10 gfx950 mxfp8 configs are today the only end-to-end - coverage of: writing a `Prediction`-type logic file (LibraryLogic step) → building a library - from it → the client selecting a solution through Origami (`ProblemPredictionLibrary:: - findTopSolutions` → `origami::rank_configs`). `Prediction` ships in 9 hipBLASLt logic files - (gfx1170 ×6, gfx1201 ×3, all `Origami/`), all non-MX, while the mxfp8 tests pass - `a/b_mx_block_size = 0` to Origami, so they do not exercise it as shipped. - - New YAML: a non-MX type (e.g. BF16 or F16), as small as possible: very small problem sizes - in both the benchmark step and `LibraryClient`, minimal solution set, numerical validation - off (`NumElementsToValidate: 0`); it only has to exercise logic write → library build → - Origami selection. - - It is added to the K whitelist (reason: only end-to-end `Prediction` coverage); the 10 mxfp8 - configs lose their `LibraryLogic:` / `LibraryClient:` sections. - - To verify when implementing: (1) with validation off and `SyncsPerBenchmark: 0`, the - LibraryLogic step still keeps the solutions (`removeInvalidSolutions` on NaN results) and - writes a non-empty library; (2) the LibraryClient step still performs Origami selection and - fails the test if no solution is selected, even when no kernel is launched; (3) which archs - it can run on: selection needs an Origami hardware model - (`pAMDGPU->analyticalHardware`); without one `findTopSolutions` returns nothing. - -### 6. `python_kernel_build_src_co` - -Status: decided. - -Adopted: - -- **A. One xnack-agnostic helper object in benchmark/test builds.** The benchmark path - (`BenchmarkProblems.cmdLineArchsFor` → `gfxToVariants` → `gfxVariantMap`) compiles the helper - `Kernels.cpp` for the bare arch (`gfx950`) instead of `gfx950:xnack+` and `gfx950:xnack-`. - This matches what already ships: shipped libraries build helpers from CMake `GPU_TARGETS` - (bare arch, xnack-agnostic; `xnack+` only for ASAN), the installed library contains only - `Kernels.so-000-gfx950.hsaco`, assembly kernels are already xnack-agnostic, and the runtime - loader tries the bare-arch file first. Measured −47 to −53% wall and CPU; code identical to - today's `xnack+` object. -- **C. Split `Kernels.cpp` into chunks, compile them in parallel, link once, and cache each - compiled chunk.** - - Kernels defined in `Kernels.h` (the 14 `Dijk_*_Reduction` kernels in gradient/bias configs) - move into the `.cpp`, otherwise chunks fail to link with duplicate symbols. - - Compile each chunk to a relocatable device object and link once with `lld -shared`, using the - driver's own `cc1` / `lld` commands (as in the experiment), which kept every kernel identical. - `-fgpu-rdc` is not used: it changed every kernel's code. - - Chunk count: small (the measurement flattens beyond 4) and bounded by the pool budget from - topic 7, since chunks run in parallel with each other and, with F, with codegen. - - **Per-chunk cache:** each compiled chunk object is cached (same pattern and location as the - helper-kernel cache and the PCH), keyed on the chunk's source, the headers it includes - (including the generated `Kernels.h`), the exact compile arguments and the compiler build. - Chunk assignment must be deterministic and content-stable (e.g. grouped by kernel, not by - balancing line counts over the whole file), so that adding or removing one kernel does not - change every chunk's content. - - Evidence to weigh: within the 4 captured configs, a per-kernel cache would have skipped only - 6% of kernel compiles beyond today's whole-file cache; cross-config reuse was not measured - (production builds were cold per config). - - **Test requirement:** measure with a full `--build-only` run starting from an empty cache, to - get the real chunk hit rate and wall time (and compare against a run without chunk caching). - - The PCH (E) cuts the fixed header cost per chunk from ~400 ms to ~35 ms, which removes most of - C's extra CPU. -- **E. Cached precompiled header of the HIP headers** for the helper-kernel compile. CI keeps - `~/.tensile` (or `TENSILE_HELPER_CACHE_DIR`) across configs, so the PCH is built once per - job and reused. - - PCH contents: HIP headers only (implicit `__clang_hip_runtime_wrapper.h`, - ``, ``, ``), built with - `-fpch-instantiate-templates`; used with `-include-pch`. TensileLite static headers stay out - (they are re-copied per build with new mtimes, which clang rejects). - - Stored next to the helper-kernel cache, using `HelperKernelCache`'s pattern (SHA-256 key, - write to `.tmp__` then `rename`, age-based eviction). - - Key: the exact argument list used to build it (every `-D`, target and features, `-O`, `-std`, - sanitizer flags), the full `amdclang++ --version` output (exact compiler build), and the - target. A `-D` baked into a PCH otherwise leaks silently into compiles that do not define it. - - On any PCH error (e.g. headers modified by a ROCm reinstall), rebuild the PCH or compile - without it and retry, so a stale entry never fails a build. - - Measured: −0.3 s per translation unit and target (≈170 s across the aggregate with both - targets, about half with A), code identical. -- **F. Start the helper compile before codegen in the benchmark path** - (`writeSolutionsAndKernels`): `writeHelpers` only needs `kernelHelperObjs`, available on entry, - so the compile runs while kernel codegen, assembly and the asm link run. Same approach as - PR #13671 on the TensileCreateLibrary path; no coordination needed. Upper bound on time hidden: - ~24% of compile time today (estimate; codegen and the compile share cores). - -Deferred: - -- **D. Shrinking the `PostGSU16` reduction kernels.** Changes GSU-path code; needs a runtime - benchmark of the reduction kernels first. - -Not adopted: - -- **B. `-parallel-jobs=2`.** Not needed with A (one target). - -### 7. `python_kernel_codegen` - -Status: decided. - -Adopted: - -- **A. Remove `CpuThreads: 1` from all 65 `common/` test YAMLs that set it** (19 of them ran in - the gfx950 aggregate, 189 s of codegen; the setting was present when each file was added and no - PR explains it). Measured on `subtile_mxfp4`: codegen 133.5 → 22.6 s, `write_assemble` - 62.9 → 11.6 s, build 244 → 63 s, `.co` and libraries identical. - - **Prerequisite: serial and parallel codegen must produce the same metadata.** `LdsNumBytes` - (set in `Components/Subtile/SubtileLdsLayout.py`) and `LoopIters` (set in `KernelWriter.py`) - are written onto the kernel during codegen and are lost in worker processes, because - `passPostKernelInfoToSolution` (`TensileCreateLibrary/Run.py`) only copies back - `CUOccupancy`, `PrefetchGlobalRead`, `MathClocksUnrolledLoop` and `CustomKernel`. Add both to - the returned `KernelCodeGenResult` so `Data/*.yaml` matches in both modes. -- **B. Fail fast on kernels that will be rejected.** Both checks: - 1. after `_initKernel`, stop if the static VGPR allocation (`states.totalVgprs`) exceeds - `MaxVgpr` (the VGPR pool can only grow, so rejection is certain); - 2. stop as soon as any `overflowedResources` code is set (VGPR, SGPR, LDS scheduling, LDS), at - the code sites that set it. - - Respect `forceGenerateKernel` (no early exit when set); keep the same error code (`-2`); print - the reason under `PrintSolutionRejectionReason`, as the end-of-codegen check does today. - - Validated in the prototype on 4,313 kernels: 0 accept/reject mismatches; CPU on rejected - kernels 374.5 → 10.8 s; `f8_gate_r` build 28 → 11 s, artifacts byte-identical. -- **C. Generate all steps of a config in one pool.** Restructure `BenchmarkProblems` into: - solutions for all steps → one codegen pool → per-step assemble/build/write (or keep one pool fed - across steps). Acceptance check: byte-identical artifacts per step against today (step - independence was inferred from the loop, not proven). Simulated: `custom_mainloop_scheduling` - 36 → 2.1 s, `f8_gate_r` 12.4 → 3.3 s; ≤25-kernel steps across the aggregate 142 → ≤75 s. -- **Pool budget: 2× the available CPUs across all pytest-xdist workers** (replaces the body of - `CPUThreadCount()` in `Common/Parallel.py`, as sketched in the note): - - CPUs available = min(affinity mask, loky `cpu_count()` (applies the cgroup quota), - `KUBE_CPU_REQUEST` when set and parseable). - - Concurrent workers = `PYTEST_XDIST_WORKER_COUNT`, else 1. - - Pool size = max(1, min(CPUs, 2 × CPUs // workers)); `CpuThreads` stays a cap (`-1` = budget, - N = min(budget, N)). - - Accepted limitations: `KUBE_CPU_REQUEST` is a request, not a limit (TheRock already treats it - as the pod's budget); only pytest-xdist parallelism is detected. - - Also fix `ParallelMap2` overwriting an explicit `procs` with `CPUThreadCount()` (so - `TensileLogic/Run.py`'s `procs=jobs` takes effect), and have TensileCreateLibrary's - `_cpuCount()` use the same helper. - -Deferred (see [DEFERRED.md](DEFERRED.md)): D (longest-first dispatch), E (cheaper generator), -and moving the VGPR/SGPR checks into solution validation. - -### 8. `python_kernel_bench_setup` - -Status: decided. - -Adopted: - -- **F2. Build helper kernel objects once per distinct input, in-process.** In - `writeBenchmarkFiles` (`BenchmarkProblems.py`), memoize `initHelperKernelObjects` on the - solution fields it reads (`ProblemType`, `GlobalSplitU`, `GlobalSplitUAlgorithm`, - `isStreamK`, `StreamKAtomic`, `_GlobalAccumulation`, `AdaptiveGemmGSUA`, `ActivationFused`, - `WavefrontSize`, `ISA`, plus the global `internalParameters["GlobalSplitUPGR"]`). The key is - computed inside `KernelHelperNaming`, next to the code that reads these fields, so they cannot - drift apart. The memo is scoped to one `writeBenchmarkFiles` call; the existing name-based - dedupe keeps the kept list and its order unchanged. Measured 82–85× on the block, output - identical on all 45 captured steps. -- **F1. `DataType.__deepcopy__` returns `DataType(self.value)`**, so the class-level - `properties` row is shared again instead of copied. Measured 1.4–1.8× on this block; also sped - up `python_benchpost_lib_construction`; artifacts byte-identical. Affects every `deepcopy` that - touches a `DataType` (nothing assigns `.value` / `.properties` outside `__init__`). -- **Investigation (findings only, no fix): TensileCreateLibrary helper name/object mismatch.** - `generateKernelHelperObjects` (`TensileCreateLibrary/Run.py`), the path that builds shipped - libraries, dedupes helper kernels on names predicted by the `*KernelNames` functions and aligns - them with the built objects positionally (`itertools.compress`). The predicted names do not - match the built objects: they use the solution's `GlobalSplitU` instead of `GlobalSplitUPGR` - (`PostGSU1` vs `PostGSU16`), the vector-width lists disagree (`isDouble()` vs `numBytes()`; 3 - names vs 2 objects for `cgemm`, 3 vs 1 for `zgemm`), and `MultipleBufferSingleKernel` ignores - `AdaptiveGemmGSUA`. On every captured benchmark step the final set happened to be right. - - Task: determine whether any real configuration (shipped logic files, all archs) drops a - helper, duplicates one, or pairs a name with the wrong object; write the findings to a - markdown file in this directory. Do not change the code as part of this plan. - -Not adopted: - -- **Persisting the F2 memo to a file** across runs. After F1 + F2 only a few seconds remain - across the aggregate, and a file cache keyed on solution fields would replay stale helper - source after generator edits unless keyed on the generator code itself. Compiled helpers are - already cached across runs by content hash (`HelperKernelCache`). -- **Reusing TensileCreateLibrary's `generateKernelHelperObjects`**: different helper order - (changes `Kernels.cpp` and its cache key), and its name prediction is unreliable (see the - investigation above). - -## Implementation plan - -This plan implements every adopted item above. Deferred work is in [DEFERRED.md](DEFERRED.md) -and is out of scope. - -### Rules for every change - -1. **One adopted item per commit.** Each item below (numbered `Pn.m`) is its own commit. An - item that is split into steps (e.g. test infrastructure, then the change) gets one commit per - step. Unrelated items are never combined, and commits are not squashed later. -2. **Commit messages say exactly why.** Conventional-commit subject (`perf(tensilelite): …`, - `fix(tensilelite): …`, `test(tensilelite): …`, `docs(tensilelite): …`). The body states: - - **What** changed, in concrete terms (files, functions, behaviour). - - **Why**: the problem, with the measured numbers that motivated it, and a reference to the - investigation note and section (e.g. `timing_runs/cpu_reference_slow_path_investigation.md`, - "Why the slow path is slow") and the topic/item in this file. - - **Expected effect**, marked as measured or estimated. - - **How it was verified** (tests run, equivalence checks, measurements). - - **Risks / behaviour changes**, or "none (output identical)". -3. **Every commit builds and passes its tests.** At minimum the tests listed for that item, plus - the existing suites it touches (`CPUGemm.*` CTest for `cpu-gemm-driver`, `tensilelite-tests`, - TensileLite Python unit tests). No commit relies on a later one to be correct. -4. **No behaviour change beyond what was decided.** Output-identical items are checked to be - identical (byte-identical artifacts, identical client stdout/CSV/exit codes as stated per - item). -5. **Timing results are committed.** Every aggregate measurement run (P0, P3, P6) commits its - `AGGREGATE_analysis.txt` (as the existing `per_config_rerun_2` / `per_config_build` results - are) plus a short comparison note against the baseline, in its own `docs(tensilelite): …` - commit that names the measured commit hash, client binary, host, cores and GPU. -6. **Never touch the measured baseline binaries.** `/tmp/tl_client_msgpack` and - `/tmp/tl_split_gfx950/artifacts` stay as they are; new builds go to separate directories. -7. **Supporting changes travel with the item that needs them.** A timer added by an item updates - `scripts/analyze_timing.py`'s `TIMING_HIERARCHY` in the same commit; a disable switch needed for - a measurement (PCH, chunk cache) lands with that feature. Infrastructure that no single item - owns (script parametrization) is its own item (P0.0). -8. **Temporary investigation code is never committed.** Examples: rejection-reason logging used - for the P1.12 replay, the `hipMemcpy` hash/readback shims, the corruption shim used for - P1.27's injected-failure check. If a check is worth keeping, it becomes an explicit opt-in debug - mode, stated in that item. -9. **Numbers in commit messages are labelled.** Numbers from the investigations were measured - with the `4c6426cea6` client; per-commit "expected effect" is an estimate unless measured at - that commit. Say which. - -### Ordering and dependencies - -```text -P0 baseline + build/test setup - │ - ├─ P1 client (C++) run-side changes ── P2 Python/YAML/harness run-side changes ── P3 run-side measurement - │ (P1.1 first: KMP_BLOCKTIME reduces variance for everything after it) - │ │ - │ P4 build-side changes (land after P3) ──┴── P6 build-side measurement - │ (topic 8 → topic 7 → topic 6) - │ - └─ P5 TensileCreateLibrary helper-name investigation (findings only; read-only, any time) -``` - -Hard dependencies inside phases are listed per item ("Needs: …"). - -P4 commits land after P3's measurement, not interleaved with P1–P3 (plan review): the -measurement scripts clear `~/.tensile` (where the PCH and chunk caches live), pin cores 0–24, and -P3 rebuilds artifacts from the branch head, so interleaved build-side changes would contaminate -the run-side measurement. P4 code may be developed earlier on a separate branch, but nothing is -built or measured on the measurement cores while a measurement runs. P2.5/P2.6 also change which -configs have LibraryLogic/LibraryClient steps, so the P6 note reports those configs separately. - -### P0. Baseline and setup - -- **P0.0 Parametrize the measurement scripts.** `split/run_split.sh`, - `per_config/run_per_config.sh` and `per_config/run_per_config_build.sh` hardcode `CLIENT`, - `ART`, `OUT` and clear `/home/alvasile/.tensile*`. Make `CLIENT`, `ART`, `OUT` and the cache - directories (`TENSILE_HELPER_CACHE_DIR` etc.) overridable, with today's values as defaults, so - P0/P3/P6 never overwrite the committed baseline artifacts and never share caches with - development builds. Own commit. -- **P0.1 Build tree with tests.** Configure a separate build directory (not - `/tmp/tl_client_msgpack`) from the starting commit with `TENSILELITE_BUILD_TESTING=ON` and - `TENSILELITE_ENABLE_CLIENT=ON`, Release, `GPU_TARGETS=gfx950`, same compiler as the baseline. - Targets used throughout: `tensilelite-client`, `cpu-gemm-driver`, `tensilelite-tests`. -- **P0.2 Baseline at the starting commit.** The committed baselines (`per_config_rerun_2`, - `per_config_build`) were measured with a client built at `4c6426cea6`, older than the starting - commit. Re-measure at the starting commit with the same scripts so every comparison is - like-for-like: - - Build artifacts: `full_runs_gfx950/split/run_split.sh` build phase into a new artifact dir. - - Run side: `full_runs_gfx950/per_config/run_per_config.sh` (serial, `TimingInstrumentation=True`, - 25 pinned cores, `OMP_NUM_THREADS=25`) with `OUT` pointing at a new - `full_runs_gfx950/baseline_run/` and the new client/artifacts. - - Build side: `full_runs_gfx950/per_config/run_per_config_build.sh` (cold cache per config) with - `OUT=full_runs_gfx950/baseline_build/`. - - Record the full common-test pass/fail set (split run phase, `-n 4`) as the correctness - baseline. - - Also record the **uninstrumented** wall time of that split run phase (`-n 4`): P1.20 adds - syncs under instrumentation only, so instrumented wall time understates production gains. - - Keep the raw per-config logs (`*.log.gz`) so P0 can be re-aggregated with the final - `analyze_timing.py` hierarchy before comparing. - - Noise: record host load (`uptime`, other users' processes) and GPU clocks at the start and - end of each run; repeat the ~10 most expensive configs 3× for a noise band. - - Commit both `AGGREGATE_analysis.txt` files, the pass/fail list and a note with commit, host, - cores, GPU (rule 5). - -### P1. Client (C++) run-side changes - -All in `projects/hipblaslt/tensilelite/client` unless stated. Order within P1 is the listed order. - -**Topic 5a, A — OpenMP workers** - -- **P1.1 `KMP_BLOCKTIME=0` by default.** In `main.cpp`, before the first OpenMP use: if - `KMP_BLOCKTIME` is not set in the environment, make idle workers sleep immediately - (`kmp_set_blocktime(0)`, or `setenv("KMP_BLOCKTIME","0",0)` before the runtime initializes; - pick whichever is verified to apply to the worker pool used by later parallel regions). - The decision names `kmp_set_blocktime(0)`; if `setenv` is chosen, the commit message says why. - `kmp_set_blocktime` is libomp-only, so guard it for other OpenMP runtimes / Windows builds. - `cpu-gemm-driver` does not get this; pin `KMP_BLOCKTIME` explicitly when quoting driver timings. - - Verify: per-thread CPU sampled from `/proc//task/*/stat` during a dtl solution loop shows - no worker above 25% CPU (the note measured 11–24 spinning workers before); the >1 ms share of - the bookkeeping timers drops on a dtl replay; an explicit `KMP_BLOCKTIME` in the environment - still wins. - -**Topic 1 — CPU reference** (`client/src/Reference.cpp`, `client/cpu_gemm_driver.cpp`, -`tests/CMakeLists.txt`, `tests/ReferenceFastPath_test.cpp`, `tests/ReferenceMXFastPath_test.cpp`) - -Every fast-path extension follows the established pattern of the FP8/FP4 (#8106), f64/tf32 -(#9203) additions: extend `isFastPathEligible` only for what `solveCPUFast` implements in the same -commit; add `cpu-gemm-driver` support for the new case; add fast/slow `CPUGemm.*` CTest pairs that -validate both paths against the driver's golden GEMM; add gtest cases for exactness edge cases. - -- **P1.2 Slow path: lock only for AMAX-D (A).** In `ReferenceSolution<…>::SolveCPU`, enter the - critical section only when `problem.outputAmaxD()`; drop the per-element - `omp_set_num_threads`. - - Verify: existing `CPUGemm.slow_*` tests; `cpu-gemm-driver` 1024³/2048³ fp32 slow path - bit-identical to before (max diff 0) and faster (note: 1,727 → 191 ms; 45,237 → 1,507 ms). -- **P1.3 AMAX-D reduction and thread cap (A+).** Replace the AMAX-D critical section with a - per-thread maximum combined once; delete the `omp_set_num_threads(MAX_OMP_THREADS)` calls in - `SolveCPU` and `ReductionCPU` (setting `omp_get_max_threads()` would be a no-op). Behaviour - change to state in the commit: today the first slow-path call permanently sets the main - thread's team size to 64, so every later OpenMP region (fast path, data init) runs 64 threads - on 25 pinned cores; after this, regions follow `OMP_NUM_THREADS`. - - Verify: new driver support for AMAX-D output if absent, with fast/slow-agnostic tests that - AMAX-D equals the max |D| computed by the golden GEMM; existing tests unchanged. -- **P1.4 Driver: output type separate from accumulator; Int8 inputs.** Test infrastructure only: - `cpu-gemm-driver` gains an output (C/D) type option and alpha/beta type, Int8 A/B storage, and - golden-GEMM support for them (integer accumulation for Int8, the slow path's epilogue type, - `SaturateCast` semantics on the output). Slow-path CTest cases for the new combinations land - here and must pass before any fast-path change. - - The slow path accumulates float-alpha Int8 types (`I8_*_S`) in `float` per K element, so it - is inexact once |sum| > 2²⁴. Slow-path CTest sizes must keep |sum| < 2²⁴ (the default sizes - do: ~151·127² ≈ 2.4M). -- **P1.5 Fast path: separate MAC type from epilogue type.** Refactor `solveCPUFast` to take a - multiply-accumulate type in addition to `AccumT` (epilogue type), defaulting to `AccumT`. No new - types enabled. Verify: all existing `CPUGemm.fast_*` and gtests unchanged (bit-identical). -- **P1.6 Fast path: Int8 inputs.** Int8 A/B shadow buffers, `int32_t` accumulation, epilogue in - `int32_t` (int32 alpha: `I8_I8_I32`, `I8_I32_I32`) or `float` (float alpha: `I8_I32_S`, - `I8_I8_S`, `I8_H_S`, `I8_B_S`); Half/BF16 outputs reuse existing write-back. Enable these type - combinations in `isFastPathEligible`. Output casts must reproduce the slow path's - `SaturateCast` exactly (verified in `Reference.cpp`): - - Int8 output: from a float epilogue, round-to-even then clamp to [−128, 127]; from an int32 - epilogue, clamp only. - - Int32 output: the generic overload, i.e. a plain `static_cast` (truncation, no rounding, no - clamp; out-of-range is undefined behaviour). Tests stay in range. - - **Epilogue operation order (decided):** the Int8 fast-path epilogue applies alpha, the - ScaleAB scales, `scaleAlphaVec`, beta·C, bias, activation and the output cast in exactly the - slow path's order and types (`ReferenceSolution<…>::SolveCPU`), so Int8 outputs match the slow - path exactly, including round-to-even ties. Tests compare fast vs slow exactly for integer - outputs. - - Tests: fast/slow CTest pairs for each output type (I8, I32, Half, BF16) × transposes, beta, - bias, activation, ScaleAB; gtests for exact integer results above 2²⁴ for int32-epilogue - types (for float-epilogue types the expected value is `float(exact sum)` then the epilogue), - and Int8 saturation at ±127/−128. - - Commit message notes that for large-K `I8_*_S` problems the reference changes (now exact in - the dot product, matching the GPU's int32 accumulation better than the slow path did). -- **P1.7 Fast path: Float8 outputs.** `storeFrom` for Float8/BFloat8 (and `_fnuz`) outputs - matching `SaturateCast` (including the BFloat16-accumulator path); enable `F8×F8→F8`, the BF8 - and hybrid variants, and `F8×Half→F8` (mixed-input shadow with compute-input quantization). - - Tests: driver support for F8/BF8 outputs; fast/slow pairs per variant; gtest for rounding and - saturation at the F8 range limits. -- **P1.8 Fast path: complex types (C).** `ComplexFloat` and `ComplexDouble` A/B/C/D, including - conjugate ops (`TensorOp::ComplexConjugate`), matching the slow path's `Transform`/`multiply`. - - Tests: driver complex support; fast/slow pairs incl. conjugate A/B; gtest for conjugation. -- **P1.9 Fast path: `useE` (C).** Store the pre-activation value to E in the epilogue - (`SetValue` semantics as the slow path; not with gradient). - - Tests: driver E output; fast/slow pairs with activation. -- **P1.10 Fast path: gradient (C).** Gradient GEMM: E-based activation derivative multiply, and - the bias reduction (`ReductionCPU` for bias from D/A/B, with the D-source workspace). - - Tests: driver gradient + bias-reduction support; fast/slow pairs for each bias source. -- **P1.11 Fast path: MX K not a multiple of the MX block (C).** Handle a partial last MX block - (K = 333/255/63 with block 32 in `mxfp4_mxfp4_bf16_tn_act`). - - Tests: MX gtests and CTest cases with K tails on both sides; existing MX tests unchanged. -- **P1.12 Topic 1 replay.** After P1.2–P1.11: replay the 59 slow-path YAMLs (as in the note's - method, with uncommitted local rejection-reason logging, rule 8) and confirm every config still - passes. Expected remaining slow-path reasons: `NumElementsToValidate: 128` (sparsity - threshold) plus any of amaxD, scaleCD, mirror indices, multi-index, no batch indices, - non-E8 / one-sided MX that occur; anything else is a finding. Results go into the P3 comparison - note, not a separate commit. - -**Topic 4 — `cpu_data_init`** - -- **P1.13 MX generator: opt out of the reference floats.** Add an opt-out to `generateMXInput` - (`clients/common/src/mxDataGen.cpp`, `clients/common/include/mxDataGen.hpp`) that skips - `getReferenceFloat()`; default unchanged. The TensileLite client's two call sites opt out. - - Verify: `tests/MXDataGen_test.cpp`; build the hipBLASLt clients that use the shared code - (`clients/common`) and run their MX tests (`fast_check_gtest` MX cases), since P0.1 builds only - TensileLite targets; generated data/scale bytes identical with and without the opt-out. -- **P1.14 Trig init without per-element overhead.** In `initArrayTrig` only (decision 2a): - no per-element `std::vector`, flat index for packed strides; keep double `sin`/`cos`. Verify: - generated arrays byte-identical to before for packed and strided descriptors (new gtest in - `DataInit_test.cpp`). -- **P1.15 Dynamic schedule for init loops.** `schedule(dynamic, chunk)` on the init loops. - Verify: Trig/Serial/Identity outputs byte-identical; timing under a contended core set. - -**Topic 2 — input data preparation** (`client/include/DataInitialization.hpp`, -`client/src/DataInitialization.cpp`) - -- **P1.16 Copy the current problem's extent, not `maxElements`.** `copyInputs` / - `copyInputBuffers` / per-problem `copyValidToGPUBuffer` copy the current problem's allocated - extent. In NaN / guard-page modes the guard regions already span the whole buffer - (`padding = (maxElements − allocated)/2`), so the change matters in Disable mode. - - Verify: Disable-mode multi-size configs where the size shrinks and then grows; bounds-check - configs (`nan_bounds_check_odd_padding`, `bufferLoadStore0`); swizzled tensors - (`getSwizzledTensorNumAllocatedElements`), padded MX scales, sparse metadata, grouped GEMM - offsets and rotating-buffer mode. -- **P1.17 Skip unchanged per-solution uploads (fix 1).** The host epoch alone is not enough: - GPU `valid` buffers have other writers (verified): the initial copy, the MX swizzled upload, - every branch of `copySwizzledToGPUBuffer`, and the grouped path. Track, per (tensor, dtype), a - tag of what the GPU `valid` buffer actually holds (host epoch plus layout/source); every writer - sets or invalidates it; the initial state is "unknown". Skip the H2D only when the tag matches. - - Verify: an opt-in debug mode (kept, rule 8) that on every skip reads `gpuInput.valid` back - and compares it with the expected bytes, run once over the full common suite with zero - mismatches; MX, sparse, swizzled, Trig and bounds-check configs pass. -- **P1.18 Cache the batched-pointer arrays (fix 2).** The pointers are `current + padding`, and - padding depends on the bounds-check mode, which flips GuardPageFront → GuardPageBack between the - two iterations of one GuardPageAll solution without any epoch or address change. Key the cache - on the computed per-tensor start pointers plus the descriptors (equivalently: rebuild when the - problem, `m_curBoundsCheck`, swizzle size or sparse metadata changes). - - Verify: grouped/batched, GuardPageAll (`bufferLoadStore0`) and sparse configs pass. -- **P1.19 MX swizzled scale from the canonical scale (fix 4).** Two commits: - 1. Test first: gtest that permuting the generator's **compact** canonical scale (before the - K-fast restride that `cpuInput.valid` receives; `generateMXInput(…, GFX950)` permutes the - compact scale) with the existing `applyMXScaleLayoutInPlace` (`mxDataGen.cpp`) is - byte-identical to `generateMXInput(…, GFX950)` output, for FP4/FP8, K-fast and K-slow - layouts, a padded leading dimension, and batched problems. Also check that the data bytes - from the two generator calls are identical (today's code pairs call 2's data with call 1's - scale). Check the batched case carefully: today's swizzled per-batch size can exceed - `scaleBatchStrideBytes`, so "identical to today" may encode overlapping batches; if so, stop - and report rather than reproduce it. - 2. Change: per-solution MX handling never regenerates data or canonical scale; keep the compact - canonical scale per problem (or un-restride), permute it once per problem, cache it in its - own GPU buffer so canonical- and swizzled-layout solutions do not overwrite each other; - `m_inputEpoch` is not bumped. - - Verify: `pre_solution_mx_side_regenerate` and `cpu_reference_gemm_per_solution_inputs_changed` - drop to zero on `subtile_mxfp4` / `subtile_mxfp8`; MX configs pass. - -**Topic 3 — Synchronizer check and attribution syncs** (`client/src/SynchronizerValidator.cpp`, -`client/include/SynchronizerValidator.hpp`, `client/main.cpp`) - -- **P1.20 Attribution syncs under timing instrumentation.** In `main.cpp`, only when - `g_timingInstrumentationEnabled`: `hipStreamSynchronize` after warmup 0 (before - `listeners.validateWarmups`) under its own timer, and at the end of each run iteration under its - own timer, **before** the per-iteration Synchronizer check (P1.22), so that check does not absorb - queued warmups. It goes **inside the `try`, inside the `needMoreRunsInSolution` loop**, never - between the `catch` and `postSolution`: a sticky device error there would otherwise throw out of - the solution loop. Uses the stream, not timing events - (needed by P1.28). Verify: with instrumentation off, no extra HIP calls (count with a shim or - trace); with it on, `validate_synchronizer_readback` / `validate_gpu_readback` no longer contain - kernel time; a forced kernel fault still reaches the existing device-lost handling. -- **P1.21 Host check: OR-reduction first (1-alt).** Rewrite the clean-path scan as a vectorizable - OR over the buffer; run the count/first-offset scan only when dirty. Diagnostic unchanged. - Verify: `SynchronizerResidue_test.cpp` (clean, dirty, first offset, counts) and a speed check. -- **P1.22 Check in `postSolution` with guards.** Save the `ProblemInputs` pointer; do the check - inside `SynchronizerValidator::postSolution` (it already runs after `ReferenceValidator`'s, so - its `FAILED` overrides the reference verdict; keep registration order unchanged); guard 1 (only - if the solution launched: flag from `preWarmup` / `preEnqueues`, plus `m_mayUseSynchronizer`); - guard 2 (exception path: check if the device is usable, otherwise catch and mark `FAILED`, never - throw); keep clear-on-dirty. GuardPageAll (two iterations per solution): the check itself runs - at the end of every run iteration, before the next iteration's input reset (inside the - `needMoreRunsInSolution` loop, after the iteration's launches; guards 1–2 apply per iteration), - and `postSolution` reports the accumulated verdict. - - Also verify: a GuardPageAll config with residue injected in the first iteration is reported. - - Verify: `SynchronizerValidatorReporting_test.cpp` extended for: residue from a later launch - attributed to its own solution; a dirty buffer gives `FAILED` even when the reference check - passed; no check for non-launched solutions; residue after an `INVALID` launch with a usable - device (state which verdict wins); device-lost path; residue reported exactly once. -- **P1.23 GPU OR-reduction check.** OR-reduction kernel on the client's stream + 4-byte readback; - full readback and existing diagnostic only when nonzero; host check (P1.21) used under the FFM - emulator (`HSA_MODEL_MEMFILE`) and as a fallback when the kernel cannot launch on the device - (no code object for the arch: `hipErrorNoBinaryForGpu` / invalid device function). - - Verify: residue injected into the buffer is detected with identical diagnostic; per-check - time; Stream-K/MBSK/amaxD configs (`streamk/*`, work-stealing configs) pass. - -**Topic 5 — bookkeeping and correctness-only runs (client side)** - -- **P1.24 Predicate dump behind its own option (5a C).** New client option, default off, gating - `debugEval` in `SolutionIterator::checkSolution`; wired to a new Tensile global parameter in - `ClientWriter.py` / `Common/GlobalParameters.py` (Python part in the same commit, so the option - is usable end to end). Verify: dump appears only with the option; rows unchanged. -- **P1.25 `LogReporter`: check the row level before copying (5b D1).** Verify: stdout identical at - every log level on a dtl invocation (normalizing the `srand seed` line). -- **P1.26 Skip unused granularity/memory values (5a D2).** In `BenchmarkTimer::preSolution`, skip - `projectedPerformance` and the 9 granularity/memory reports unless a `LogReporter` will print - the row (stdout at its level, or the `--log-file` reporter `main.cpp` adds at - `LogLevel::Normal`) or `CSVExportExtraCols` is set. Applies to every run, timed or not. - - Verify: stdout, log file and results CSV byte-identical at every log level, with and without - `--log-file` and `CSVExportExtraCols`. -- **P1.27 Infer "nothing is timed" (5b E).** Static rule from the client's arguments: - `num-syncs-per-benchmark == 0`, or `max-enqueues-per-sync == 0`, or - (`num-enqueues-per-sync == 0` and `min-flops-per-sync == 0`). When it holds: skip the - performance calculation in `BenchmarkTimer::postSolution` and report the same NaN values (same - `-nan` text). - - Verify: results CSV and stdout byte-identical against the previous commit on dtl and xfp32 - invocations, clean and with injected failures (local corruption shim, not committed), exit - codes identical; the four syncs/enqueues/min-flops combinations from the note behave as - tabled. -- **P1.28 No warmup timing events when nothing reads them (5b F).** Needs: P1.20, P1.27. - `BenchmarkTimer` reads warmup events in `postWarmup` when `skip-slow-solution-ratio` is set and - in `validateWarmups` under `--sync-after-warmups` (verified). Create the events unless nothing is - timed **and** the skip-slow ratio is 0 **and** `--sync-after-warmups` is off; otherwise keep - them. - - Verify: as P1.27; plus a nothing-timed run with skip-slow set; `BenchmarkTimer` unaffected - when timing is on. -- **P1.29 Client "no results file" path (5b I, client side).** An explicit option that never - constructs `ResultFileReporter`. Verify: no file written, exit code and stdout unchanged. - -### P2. Python, YAML and test-harness run-side changes - -- **P2.1 Copy the results CSV only if it exists (5b H).** `BenchmarkProblems.main`, same guard as - the granularity CSV. Unit test. -- **P2.2 Write the results CSV only when consumed (5b I, Tensile side).** Needs: P1.29, P2.1. - In `BenchmarkProblems` only (not in the shared `ClientWriter.writeClientConfig`, which - `TensileRetuneLibrary.py` also uses and which copies `benchmark.csv`): pass "no results file" - unless the config has a `LibraryLogic:` section or uses the Ductile backend. Behaviour change to - state in the commit: the CSV's existence is also the "already benchmarked, skip" marker - (`BenchmarkProblems.py`), so reruns into the same output directory re-run the client. Unit - tests for both branches; a LibraryLogic config and a Ductile config still produce their - outputs. -- **P2.3 Test runs use `ClientLogLevel=0` (5a D1).** `tensile_args` in `Tests/conftest.py` - always appends `--global-parameters ClientLogLevel=0` before any user-supplied - `--global-parameters`. The fixture body moves into a plain helper with a unit test: the value is - present, and a user-supplied `ClientLogLevel=3` comes after it. -- **P2.4 `xfp32.yaml`: drop `NumWarmups: 5` and `EnqueuesPerSync: 10` (5b G).** -- **P2.5 New minimal `Prediction` config (5b K).** One small non-MX YAML (BF16 or F16): very small - sizes in the benchmark step and `LibraryClient`, minimal solutions, `NumElementsToValidate: 0`. - Must first confirm the three verification points in topic 5 (non-empty library with validation - off; LibraryClient performs Origami selection and fails if none is selected; which archs have an - Origami hardware model) and restrict the config to those archs. If LibraryClient turns out - **not** to fail when nothing is selected, stop and ask before changing code (new scope). -- **P2.6 `LibraryLogic:` / `LibraryClient:` whitelist test and removals (5b K).** Needs: P2.5. - Two commits: the whitelist test (initially listing every current file, with reasons); removal - from the 14 `ductile/*`, 10 mxfp8 and 12 `custom/*` YAMLs, shrinking the whitelist to the 6 - kept files + the new `Prediction` config (verification: the stale-entry check fires on the - removed entries). -- **P2.7 `SyncsPerBenchmark` guard covers `custom/` (5b J).** Needs: P2.6. Must come after the - `custom/*` `LibraryLogic:` removal: all 12 custom YAMLs have `LibraryLogic:`, and with - `SyncsPerBenchmark: 0` their logic step would get NaN results that `removeInvalidSolutions` - (drops only `gflops == 0`) does not filter (verified). Two commits: set `SyncsPerBenchmark: 0` - in the 12 custom YAMLs (run all 12 at that commit), then extend - `Tests/unit/test_common_yaml_syncs_per_benchmark.py` to `Tests/custom/`. - -### P3. Run-side measurement - -- Rebuild the client at the end of P2; regenerate artifacts (split build phase) at that commit. -- Full common-test pass (split run phase, `-n 4`): the pass/fail set must equal the P0 baseline, - except for intended changes (listed in the note). -- Per-config run aggregate (`run_per_config.sh`, same settings as P0) into - `full_runs_gfx950/after_run/`. -- Commit `AGGREGATE_analysis.txt` and a comparison note (`docs(tensilelite): …`): total and - per-category deltas vs P0, the P1.12 replay result, and anything that moved unexpectedly - (attribution changes are expected: wait time moves to the new sync timers under instrumentation). - -### P4. Build-side changes - -Order: topic 8 (P4.1–P4.2), pool budget and fail-fast (P4.3–P4.6), parallel codegen metadata and `CpuThreads` (P4.7–P4.8), compile/codegen overlap (P4.9), one pool per config (P4.10), then the helper compile itself (P4.11–P4.15). P4 commits land after P3 (see "Ordering and dependencies"). - -**Topic 8 — `python_kernel_bench_setup`** - -- **P4.1 F1: `DataType.__deepcopy__` returns `DataType(self.value)`.** Verify: TensileLite unit - tests; artifacts of `fp16_tn` and `fp16_HH_BHS_bf16mfma` byte-identical (excluding `.hsaco` - `__hip_cuid_*`). -- **P4.2 F2: memoize `initHelperKernelObjects` per step.** Key computed in `KernelHelperNaming`; - memo scoped to one `writeBenchmarkFiles` call. Verify: ordered helper list (names, header, - source) identical on the captured steps; artifacts byte-identical; a unit test that records the - solution keys the helper writers actually read while running `initHelperKernelObjects` (e.g. - through a recording dict proxy) and asserts that set is contained in the memo key. - -**Topic 7 — `python_kernel_codegen` and pool sizing** - -- **P4.3 Pool budget (2× CPUs across xdist workers).** `CPUThreadCount()` in `Common/Parallel.py` - as decided, with unit tests (mocked affinity / loky count / environment, as in the prototype). -- **P4.4 `ParallelMap2` honours `procs`; TensileCreateLibrary `_cpuCount()` uses the shared - helper.** Separate commits. -- **P4.5 Fail fast: static VGPR check after `_initKernel` (B.1).** -- **P4.6 Fail fast: stop when `overflowedResources` is set (B.2).** - - Verify (both): accept/reject decision identical for every kernel of the probed steps (the - note's `ff.py` method); `PrintSolutionRejectionReason` output still names the reason; - `forceGenerateKernel` still generates; artifacts byte-identical. -- **P4.7 Return `LdsNumBytes` and `LoopIters` from codegen workers (A prerequisite).** Add them - to what `passPostKernelInfoToSolution` copies back. Verify: a generic test comparing the full - post-codegen kernel/solution state between serial and parallel codegen (not only these two - keys), on `subtile_mxfp4` and `custom_mainloop_scheduling`; `Data/*.yaml` identical. -- **P4.8 Remove `CpuThreads: 1` from the 65 `common/` YAMLs that set it (A).** Needs: P4.3, - P4.7. The 6 commented-out `CpuThreads: 1` lines (`common/sparse/*`) are removed in the same - commit. Verify: `--build-only` serial vs parallel for all 65 YAMLs on their own archs (only 19 - run on gfx950), artifacts identical; the gfx950 ones pass. -- **P4.9 Start the helper compile before codegen in the benchmark path (topic 6, F).** Moved - before the codegen restructure (plan review): in today's per-step structure, start - `writeHelpers` + the helper compile on entry to `writeSolutionsAndKernels`, since it only needs - `kernelHelperObjs`. Verify: artifacts byte-identical; total processes stay within the P4.3 - budget. -- **P4.10 One codegen pool per config (topic 7, C).** Restructure `BenchmarkProblems` to generate - all steps' kernels in one pool, then assemble/build/write per step; keep F's overlap by - launching every step's helper compile before the shared codegen pool, with the pool budget - split between helper compiles/chunks and codegen workers. Verify: per-step artifacts - byte-identical to before on configs with many small steps (`custom_mainloop_scheduling`, - `f8_gate_r`, `spmm_i8_sb`) and large ones (`dtl`); per-step `--use-cache` hits still skip - codegen; peak memory with all steps' kernels held at once. - -**Topic 6 — helper-kernel compile** - -(F, the compile/codegen overlap, is P4.9 above.) - -- **P4.11 xnack-agnostic helper compile in benchmark/test builds (A).** `cmdLineArchsFor` gives - the bare arch for the helper compile, except when sanitizer flags are present, where it keeps - `xnack+` (matching shipped ASAN builds). `gfxVariantMap` also covers gfx906/908/90a/942. - State in the commit: `cmdLineArchsFor` also feeds the benchmark cache key (`_Architectures`) - and `HelperKernelCache`'s key, so existing `--use-cache` / helper-cache entries are invalidated - once. Verify: helper kernels identical to today's `xnack+` object (disassembly with addresses - stripped); the runtime loads `Kernels.so-000-gfx950.hsaco`; configs pass; a build-only check on - gfx942 as well. -- **P4.12 Cached HIP-header PCH (E).** As decided. The key uses the full `amdclang++ --version` - output (not `HelperKernelCache`'s `major.minor.patch`) and the resolved ROCm/HIP include path, - plus the exact argument list and target. Includes a switch to disable the PCH. Verify: kernels - identical with and without the PCH; stale-PCH fallback tested (touch a header's mtime); a cold - and a warm compile timed. -- **P4.13 Move header-defined kernels into `Kernels.cpp` (C prerequisite).** The 14 - `Dijk_*_Reduction` kernels. Verify: kernels identical; gradient/bias configs pass. -- **P4.14 Chunked helper compile (C).** Needs: P4.11–P4.13. Deterministic, content-stable chunk - assignment; each chunk includes a generated header with only its own kernels' declarations (plus - the static headers and shared argument structs), so the per-chunk cache key does not depend on - the whole `Kernels.h`; driver `cc1`/`lld` commands with `-include-pch` passed through; fall back - to the unchunked compile on any failure of the chunked path; chunk count bounded by the P4.3 - budget. Verify: every kernel and its metadata identical to the unchunked build; the fallback - path exercised by a test. -- **P4.15 Per-chunk compile cache (C).** Needs: P4.14. Keyed on the chunk source, its generated - header, the static headers, exact compile arguments and the full compiler version. Includes a - switch to disable the chunk cache (needed for P6). Verify: hit/miss behaviour under unit test; - the empty-cache full `--build-only` run in P6. - -### P5. Investigation: TensileCreateLibrary helper name/object mismatch (findings only) - -Independent; any time. Determine whether `generateKernelHelperObjects` can drop, duplicate or -mispair a helper for any real configuration (shipped logic files, all archs). Write the findings -to a markdown file in this directory and commit it. No code change. - -### P6. Build-side measurement - -- Per-config build aggregate, cold cache per config (`run_per_config_build.sh`, same settings as - P0) into `full_runs_gfx950/after_build/`: comparable with the P0 baseline. -- **Full `--build-only` run of all configs from an empty cache, cache kept across configs** (the - split build phase, `-n 4`, CI-like): measures the PCH and per-chunk cache hit rates and wall - time; repeat once with chunk caching disabled for comparison (required by topic 6, C). -- Commit both aggregates and a comparison note (rule 5). - -### Plan review - -The plan was critically reviewed by a separate agent (Claude Opus 5.5) against the decisions and -the code. Corrections incorporated above: - -- Measurement isolation: P0.0 (parametrized scripts, separate caches), P4 lands after P3, - uninstrumented wall time, raw logs kept, noise band, analyzer hierarchy kept current. -- Synchronizer: listener order corrected (verdict override relies on reverse `postSolution` - order); end-of-iteration sync placed inside the `try`; device fallback for the GPU check. -- CPU reference: Int8/Int32 output-cast semantics corrected to match `SaturateCast`; - float-accumulator slow-path limits for Int8 tests; `omp_set_num_threads` calls deleted, with - the process-wide effect stated; expected remaining slow-path reasons listed for P1.12. -- Input preparation: upload skip keyed on what the GPU buffer actually holds (all writers), with - a readback debug mode; batched-pointer cache keyed on computed pointers and descriptors; - extent-sizing verification in Disable mode and for swizzled/MX/sparse/grouped/rotating tensors; - MX swizzle permutes the compact canonical scale via `applyMXScaleLayoutInPlace`. -- Bookkeeping: D2 split from E and conditioned on every `LogReporter` (incl. `--log-file`); - warmup events kept when skip-slow or `--sync-after-warmups` read them; results-CSV switch only - in `BenchmarkProblems`, with the rerun-marker behaviour change stated. -- YAML ordering: `custom/` `LibraryLogic:` removal before `SyncsPerBenchmark: 0`. -- Build side: F moved before the one-pool restructure; xnack+ kept for sanitizer builds and the - cache-key invalidation stated; PCH key uses the full compiler version and include path; chunk - headers per chunk, fallback to the unchunked compile, cache/PCH disable switches; stronger - equivalence tests for F2 and serial-vs-parallel codegen. -- Commit rules 7–9: supporting changes travel with their item, investigation code is never - committed, and numbers in commit messages are labelled measured or estimated. - -Decisions raised by the review (both decided): - -1. **Synchronizer coverage for GuardPageAll:** check at the end of every run iteration, before the - next input reset; report in `postSolution` (P1.22). Accepting the coverage loss was not chosen. -2. **Int8 epilogue matching:** the Int8 fast-path epilogue mirrors the slow path's operation order - and types, for an exact match (P1.6). A shared epilogue function for all types and a ±1 test - tolerance were not chosen. diff --git a/projects/hipblaslt/tensilelite/timing_runs/DEFERRED.md b/projects/hipblaslt/tensilelite/timing_runs/DEFERRED.md deleted file mode 100644 index b9981a4cc511..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/DEFERRED.md +++ /dev/null @@ -1,153 +0,0 @@ -# Deferred timing work - -Work that came out of the timing investigations in this directory and was deliberately **not** -included in the current implementation plan. Each entry says where it comes from, what it is, -what is known, and what it would take to pick it up. Decisions themselves are recorded in -[DECISIONS.md](DECISIONS.md). - -Status values: `deferred` (worth doing later), `not for now` (rejected for the current plan, but -could be revisited), `needs data` (blocked on a measurement). - -## Summary - -| # | Item | Source | Status | Size of the prize (as measured or estimated) | -| --- | --- | --- | --- | --- | -| 1 | Restore only writable tensors in `copyInputs` | Input data preparation, fix 3 | not for now | `copy_inputs_d2d` 26.3 s timer, real cost 4.5–5.5× that | -| 2 | Tune the host-side Synchronizer check for the FFM emulator | SynchronizerValidator, item 1-alt | deferred | unknown (emulator cost not measured) | -| 3 | Cheaper trig argument (`sinf` / argument reduction) | `cpu_data_init`, item 2b | not for now | trig init 22.1 → ~3.7 ns/element; changes generated data | -| 4 | MX data generator: single-threaded zero-fill and poor thread scaling | `cpu_data_init`, finding 1 | deferred | ~90 ms per 516M-element side; 7–11× on 25 threads | -| 5 | First-call fixed cost per client process | `cpu_data_init`, finding 4 | needs data | 54 s of the 106.8 s `cpu_data_init` is first calls | -| 6 | Typed reporter keys in `MetaResultReporter` | Per-solution bookkeeping, D3 | deferred | not measured | -| 7 | Shrink the `PostGSU16` reduction kernels | `python_kernel_build_src_co`, D | deferred | 48–59% of each remaining helper compile | -| 8 | Longest-first dispatch within a codegen step | `python_kernel_codegen`, D | deferred | 0–6% on most configs, 24% on `spmm_i8_sb` (upper bound) | -| 9 | Cheaper kernel generator (epilogue, `hasAnyDependency`) | `python_kernel_codegen`, E | deferred | epilogue 40–64% of per-kernel time; `hasAnyDependency` 25–67% on some sparse kernels | -| 10 | Reject over-budget kernels in solution validation, before codegen | `python_kernel_codegen`, B (follow-up) | deferred | removes the remaining rejected-kernel codegen entirely | - -## 1. Restore only writable tensors in `copyInputs` - -- **Source:** [input_data_preparation_investigation.md](input_data_preparation_investigation.md), - finding 3; topic 2 in DECISIONS.md. -- **What:** on the slow reset path, `copyInputs` restores *every* tensor (D2D, GPU valid → GPU - current) before each solution, including read-only A and B. Only tensors kernels can write need - restoring: D, E, AMAX-D, bias in gradient mode, the Synchronizer, and possibly others. -- **Known:** `hipMemcpy` D2D returns in ~2 µs, so `gpu_input_reset_copy_inputs_d2d` (26.3 s) is - deflated; the real serialized cost was 4.5–5.5× the timer on `spmm_f16` and - `bufferLoadStore0`. The split between necessary and avoidable bytes was not measured. -- **Why not now:** too easy to get the writable set wrong, and a full restore also masks a kernel - that corrupts its inputs. -- **To pick up:** enumerate the writable set per problem type from the kernel arguments (not by - hand), add a debug mode that verifies read-only inputs are unchanged after a solution, then - measure the avoidable bytes. - -## 2. Host-side Synchronizer check for the FFM emulator - -- **Source:** [SynchronizerValidator.md](SynchronizerValidator.md), item 1-alt; topic 3. -- **What:** the adopted design keeps a host-side check (full readback, then a vectorizable - OR-reduction, with the count/first-offset scan only when dirty) for runs under the FFM emulator - (`HSA_MODEL_MEMFILE`), where a 409,600-element reduction kernel may be much slower than a copy. -- **Known:** on hardware, OR-reduction first measured 11.3 µs warm / 32.6 µs right after the D2H - copy for 1.6 MB, against 165 µs for today's scalar scan. Nothing was measured under emulation. -- **To pick up:** measure both the GPU check and the host check under the emulator; choose per - mode, and tune (e.g. explicit SIMD, chunked readback) based on the measurement. - -## 3. Cheaper trig argument in `initArrayTrig` - -- **Source:** [cpu_data_init_investigation.md](cpu_data_init_investigation.md), finding 2; - topic 4, item 2b. -- **What:** trig init calls double-precision `sin`/`cos` of the element index. glibc takes its - slow large-argument path above ~1.06e8 (23.6–25 ns vs 6.9 ns). `sinf` + store measured - 3.7 ns/element; reducing the argument first is another option. -- **Why not now:** it changes the generated input data, so runs before and after would not use - identical inputs. The value-preserving part (no per-element allocation, flat index) is adopted. -- **To pick up:** decide whether input-data compatibility with earlier runs matters; if not, - switch to `sinf` or a bounded argument and re-baseline. - -## 4. MX data generator: zero-fill and thread scaling - -- **Source:** [cpu_data_init_investigation.md](cpu_data_init_investigation.md), finding 1; - topic 4. -- **What:** in `generateMXInput` (`clients/common/src/mxDataGen.cpp`, shared hipBLASLt client - code), `generate()` starts with a single-threaded `m_dataBytes.resize(n)` zero-fill (~90 ms for - 516 MB), and the whole call reaches only 7–11× on 25 threads (20,935 → 2,094 ms for 516M - elements). The discarded reference-float vector is handled separately (adopted). -- **To pick up:** profile `generate()` per stage on a large side; parallelize or avoid the - zero-fill; coordinate with hipBLASLt client owners since the code is shared. - -## 5. First-call fixed cost per client process - -- **Source:** [cpu_data_init_investigation.md](cpu_data_init_investigation.md), finding 4. -- **What:** the first `cpu_data_init` call in each client process (820 calls) is 54.0 s of the - 106.8 s. Standalone, the first H2H `hipMemcpy` after HIP init costs ~18 ms even for 1 KB, and - the first OpenMP parallel region ~3 ms; in-client first calls were larger (e.g. 53.3 ms to copy - 1,024 elements in `spmm_i8`) and were not explained without a profiler. Configs split into many - small client processes pay it repeatedly. -- **To pick up:** profile a first call (perf was unavailable on the investigation host); options - include warming HIP/OpenMP at client start-up off the timed path, or fewer client processes per - config. - -## 6. Typed reporter keys - -- **Source:** [per_solution_bookkeeping_investigation.md](per_solution_bookkeeping_investigation.md), - option D; topic 5a, D3. -- **What:** `MetaResultReporter` fans every `report()` out to each reporter with a `std::string` - key (and copies a `shared_ptr` per reporter, `for(auto r : m_reporters)`); each reporter then - compares the key against its list. Replace with typed keys. -- **Known:** not measured. Much of the fan-out goes away with the adopted items (no granularity - reports when unused, no stdout rows in tests, no results CSV unless consumed). -- **To pick up:** re-measure the per-solution bookkeeping section after the adopted items land; - only worth doing if `report()` still shows up. - -## 7. Shrink the `PostGSU16` reduction kernels - -- **Source:** [python_kernel_build_src_co_investigation.md](python_kernel_build_src_co_investigation.md), - option D; topic 6. -- **What:** the "universal" `PostGSU16` helper kernels are 4,260–5,865 lines each (vs 100–145 for - `PostGSU8/4/2/1`). `KernelWriterConversion.kernelBody` emits a software-pipelined reduction with - a fully unrolled 15-case remainder `switch`, each case re-emitting all 16 accumulate steps, with - every inline-asm add duplicated under `#if defined(__gfx950__)`. Size grows roughly with GSU². -- **Known:** removing these kernels entirely cut a one-target compile by 48–59% (bound, not a - proposal). Options: generate the remainder as a loop, lower `GlobalSplitUPGR`, or share one - remainder path. -- **Why deferred:** it changes the code of kernels on the GSU path. -- **To pick up:** write a runtime benchmark of the reduction kernels (all GSU values and vector - widths), then compare the alternative generators against it. - -## 8. Longest-first dispatch within a codegen step - -- **Source:** [python_kernel_codegen_investigation.md](python_kernel_codegen_investigation.md), - option D; topic 7. -- **What:** joblib dispatches kernels in input order; dispatching the most expensive kernels first - shortens the step's tail. -- **Known:** with perfect knowledge of per-kernel cost (simulated): 0–6% on most probed configs, - 24% on `spmm_i8_sb` (17.6 → 13.4 s). Output size is not known before generation, so a cost - predictor is needed. With topic 7's C (one pool per config), the tail effect shrinks further. -- **To pick up:** re-measure after C lands; if tails still matter, build a cheap cost predictor - (e.g. from tile sizes, epilogue features, or a previous run's per-kernel times). - -## 9. Cheaper kernel generator - -- **Source:** [python_kernel_codegen_investigation.md](python_kernel_codegen_investigation.md), - option E; topic 7. -- **What:** per-kernel time is ~65% Python instruction-tree construction, ~21% `rocIsaPass`, ~8% - freeing the tree and ~6% `str()`. Two concrete targets: - - **Epilogue:** `notLocalSplitUGlobalWrite` is 40–64% of profiled time in 22 of 28 sampled - kernels; it is fully unrolled per element for every variant (edge/non-edge, beta, bias type, - activation). The code itself notes it (`KernelWriter.py`: "Kernels with epilog especially with - activation is too long (50000~ lines). Need to refactor global write elements."). - - **`KernelWriter.hasAnyDependency`:** a local-read × instruction scan, quadratic; 25–67% of - profiled time in four `spmm_i8_sb` kernels (903k calls in one), 0% in `dtl`. A contained - target. -- **To pick up:** start with `hasAnyDependency` (index the dependencies instead of rescanning); - the epilogue is a larger refactor that changes generated code and needs performance validation. - -## 10. Reject over-budget kernels before codegen - -- **Source:** [python_kernel_codegen_investigation.md](python_kernel_codegen_investigation.md), - option B; topic 7. -- **What:** the adopted fail-fast stops codegen early for kernels that are certain to be rejected - (static VGPR allocation over `MaxVgpr`, or any `overflowedResources` code set). Moving the - VGPR/SGPR checks into solution validation would keep these kernels out of codegen altogether. -- **Known:** not tested. After the adopted checks, rejected kernels cost ~10.8 s of CPU across - the probed configs (down from 374.5 s). -- **To pick up:** check whether the static allocation can be computed without `_initKernel`, and - whether rejecting at validation changes which solutions are reported or counted. diff --git a/projects/hipblaslt/tensilelite/timing_runs/SynchronizerValidator.md b/projects/hipblaslt/tensilelite/timing_runs/SynchronizerValidator.md deleted file mode 100644 index fa9d30f8d78b..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/SynchronizerValidator.md +++ /dev/null @@ -1,265 +0,0 @@ -# `SynchronizerValidator` investigation (gfx950 per-config rerun 2) - -`full_runs_gfx950/per_config_rerun_2/AGGREGATE_analysis.txt` reports -`validate_warmups_listener.SynchronizerValidator` at 162,734.88 ms over 2,106,827 calls (6.3% of -wall). This note explains where that time goes, proposes moving the check to the GPU, and -describes an attribution edge case that any rework has to handle. Timing claims below come from -measurements; reading the code was only used to decide what to measure. Statements that are -inferred rather than measured are marked as such. - -| Timer | Count | Total (ms) | Mean (ms) | -| --- | --- | --- | --- | -| `validate_synchronizer_check` | 2,106,827 | 162,692.31 | 0.08 | -| ├─ `validate_synchronizer_scan` | 608,873 | 106,747.41 | 0.18 | -| ├─ `validate_synchronizer_readback` | 608,873 | 55,816.84 | 0.09 | -| └─ `validate_synchronizer_inactive` | 1,497,978 | 27.90 | 0.00 | - -All of the cost is in the 608,873 *active* checks; solutions that cannot use the Synchronizer cost -nothing. - -## Method - -- **Binary.** The measured client, `/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client` - (Release, `-O3 -DNDEBUG -std=c++20 -fPIC -mf16c -x hip --offload-arch=gfx950`). The linked - `scanSynchronizerResidue` was disassembled with `llvm-objdump`. -- **Microbenchmarks.** A verbatim copy of `scanSynchronizerResidue` compiled with the client's - flags; its assembly is instruction-for-instruction identical to the shipped binary. Timings are - medians over 300–1000 iterations with `std::chrono::steady_clock`. -- **In-client measurement.** The `dtl` and `subtile_mxfp4` configs re-run exactly as - `full_runs_gfx950/profiling/run_rerun.sh` does (cached artifacts, `TimingInstrumentation=True`, - `taskset -c 0-24`, `ROCR_VISIBLE_DEVICES=0`), with `LD_PRELOAD` shims that intercept `hipMemcpy`: - - *readback split:* for calls made from `SynchronizerValidator::readBuffer` (return address in - `[0x41b1f0, 0x41b3e8)`; the binary is not PIE), time a `hipDeviceSynchronize()` (waiting for - queued GPU work) separately from the copy. On an idle GPU that sync costs 0.07 µs, so the - "wait" figure is real waiting and not shim overhead. - - *reset tracking:* remember the device pointers `readBuffer` copies from, and count other - `hipMemcpy` writes into those pointers between checks. -- **Reproduction.** The re-run reproduces the production numbers for `dtl`: scan 8,376.56 ms vs - 8,313 ms, readback 3,214.89 ms vs 3,381 ms, both over 49,995 checks. -- **Environment.** gfx950 host, EPYC 9575F. `perf` was unavailable (`perf_event_paranoid=4`). -- Scratch sources (`bench.cpp`, `scan.cpp`, `shim.cpp`, `shim_reset.cpp`, `run_cfg.sh`) and logs - are in `/tmp/syncval_bench`. - -## What the check does and when it runs - -Stream-K flags, GSU/MBSK reduction counters and the amax counter all live in one shared -Synchronizer buffer of 409,600 alpha-typed elements (`ClientProblemFactory.cpp`), i.e. 1.6 MB for -a float alpha. Kernels must leave it zeroed on exit; a kernel that leaves residue can corrupt or -hang later launches. The check reads the buffer back after a launch, scans it for any nonzero -32-bit word, and if dirty logs the count and first offset, marks the solution `FAILED`, and -clears the buffer. - -Per solution (`main.cpp`): - -```text -prepareGPUInputs (input reset) -launch warmup 0 (async) -listeners.validateWarmups ← SynchronizerValidator, then ReferenceValidator, then BenchmarkTimer -launch warmups 1..N-1 -postWarmup -benchmark runs (syncs × enqueues; BenchmarkTimer::postEnqueues synchronizes) -postSolution -``` - -The check therefore sees exactly one launch per solution. - -## Slowness 1: the host scan (106.7 s) - -`scanSynchronizerResidue` tracks `nonzero` and `first` per element. In the shipped binary this -compiles to a scalar loop, unrolled 2×, with a loop-carried `cmov`/`add` dependency chain and no -SIMD: - -```text -41b610: testq %r8, %r8 - movq %rcx, %r11 - cmoveq %r9, %r11 - xorl %ebx, %ebx - cmpl $0x0, (%rdi,%r9,4) - setne %bl - cmoveq %rcx, %r11 - addq %r8, %rbx - ... (second element, same pattern) - addq $0x2, %r9 - cmpq %r10, %r9 - jne 41b610 -``` - -Measured (median, identical assembly): - -| Variant | 1.6 MB | 3.2 MB | 6.5 MB | -| --- | --- | --- | --- | -| Current scan, warm cache | 162.9 µs | 326.2 µs | 650.5 µs | -| Current scan, right after the D2H copy | 166.0 µs | 327.9 µs | 657.7 µs | -| Current scan, after evicting 1 GB of cache, then the copy | 164.8 µs | 327.3 µs | 655.8 µs | -| OR-reduction first (auto-vectorizes to 256-bit `vorps`), warm | 11.3 µs | 21.4 µs | 41.3 µs | -| OR-reduction first, right after the D2H copy | 32.6 µs | 66.2 µs | 131.1 µs | - -The current loop runs at a flat ~10 GB/s regardless of where the data is cached. It is bound by the -dependency chain (~0.4 ns per int), not by memory. In the client it measured 168 µs per check on -`dtl`, and per-config means in the aggregate are 0.165–0.175 ms for almost every config, consistent -with 1.6 MB buffers. 608,873 × ~165 µs ≈ 100 s, essentially the whole scan bucket. - -The count and first offset are only needed for the error message on a dirty buffer, which should -be rare. The clean path only needs "is anything nonzero". - -## Slowness 2: the readback (55.8 s) - -The readback is a blocking `hipMemcpy` D2H of the whole buffer into a pinned staging buffer. Its -time has three parts: - -1. **The copy itself.** ~37–40 µs per 1.6 MB in steady state, both in the microbenchmark (36.9 µs - pinned, 37.5 µs pageable, GPU idle) and in the client (37.9 µs on `subtile_mxfp4`). -2. **Waiting for the warmup kernel (misattributed).** Warmup 0 is launched asynchronously and - `SynchronizerValidator` is the first listener to block, so its readback absorbs the rest of the - kernel's runtime plus launch-to-completion latency. In the microbenchmark, a copy issued right - after launching a 0 / 50 / 200 µs kernel takes 53–58 / 104–106 / 254–255 µs. -3. **One-time and outlier copies.** The first copy in each client process is slow (up to 80 ms), - and some later copies take over 100 µs. The cause was not isolated. This matters for configs - split into many small client processes. - -Measured in the client with the readback-split shim: - -| | `dtl` (49,995 checks) | `subtile_mxfp4` (4,327 checks, 47 processes) | -| --- | --- | --- | -| Wait for queued GPU work | 1,184 ms (37%), ~20–29 µs/check | 156 ms (16%), ~36 µs/check | -| Copy, total | 2,009 ms (63%) | 800 ms | -| ├─ steady state | ~37–40 µs/copy | 151 ms (37.9 µs/copy) | -| ├─ first copy per process | small | 413 ms | -| └─ other copies > 100 µs | small | 236 ms (302 copies) | - -**Inferred aggregate split:** 608,873 × ~38 µs ≈ 23 s of the 55.8 s is steady-state copying. The -remaining ~33 s is waiting plus first-copy and outlier costs. That split was measured only on these -two configs. - -### The intended sync runs too late - -`--sync-after-warmups` (default `true`) makes `BenchmarkTimer::validateWarmups` call -`hipEventSynchronize` on the warmup stop event, timed as `validate_gpu_sync`. Listeners run -`validateWarmups` in registration order, and `BenchmarkTimer` is registered after both validators -(`main.cpp`). The GPU is already idle when it runs: - -| Run | `validate_gpu_sync` calls | Total | Mean | -| --- | --- | --- | --- | -| Aggregate | 2,106,827 | 6,867.77 ms | 3.3 µs | -| `dtl` re-run | 270,802 | 68.10 ms | 0.25 µs | - -When `SynchronizerValidator` is inactive, the same wait presumably lands in `ReferenceValidator`'s -`validate_gpu_readback` instead (inferred from the ordering, not measured). - -## Proposal: run the check on the GPU - -Replace the full readback and host scan with an OR-reduction kernel over the buffer, followed by a -4-byte readback of the result. Only read the whole buffer back when the result is nonzero, to -produce the existing diagnostic (nonzero count, first offset). - -```text -hipMemsetAsync(flag, 0, 4, stream) -orReduce<<<…, stream>>>(synchronizer, ints, flag) // atomicOr of per-thread ORs -hipMemcpyAsync(hostFlag, flag, 4, D2H, stream) -hipStreamSynchronize(stream) -if(*hostFlag) { full readback + existing scan for the message; mark FAILED; clear } -``` - -Measured on the 1.6 MB buffer (median): - -| Approach | Per check | -| --- | --- | -| Current: full D2H copy + scalar scan | ≈ 37 + 165 ≈ 200 µs | -| Full copy + vectorized OR-reduction on the host | ≈ 37 + 33 ≈ 70 µs | -| GPU OR-reduction + 4-byte readback | 16.8 µs | - -The GPU version removes both slow parts: the scan (≈ 100 s) and the 1.6 MB D2H transfer (≈ 23 s -steady state, plus the per-process first-copy costs). It adds ~608,873 × 16.8 µs ≈ 10 s. The -expected net saving is roughly 110 s of the 162.7 s (inferred from per-check measurements, not -measured end-to-end). The host-vectorized variant is a smaller change that keeps the copy and saves -roughly 80 s. - -**Separately, fix the wait attribution.** Synchronize on the warmup stop event in `main.cpp` right -after the first warmup launch, under its own timer, before `listeners.validateWarmups`. Then -neither validator absorbs kernel time. Reordering the listeners is riskier, because `postSolution` -runs them in reverse and the verdict ordering depends on that. - -### Interaction with the client execution lock - -None (code reading). `ClientExecutionLock` (`filelock.FileLock`, one lock file per physical GPU -under xdist) is held around the whole `tensilelite-client` subprocess in `ClientWriter.py` and -`ParallelExecution.py`. The reduction kernel is one more launch from the process that already -holds the lock. Points to watch: - -- Enqueue it on the client's `stream`, so it stays ordered after the solution's launches and - before anything that follows. -- It reads 1.6 MB through L2/MALL, which the DMA copy may not. At today's position, warmups 1..N-1 - follow it before the timed runs; at the proposed `postSolution` position (below), it runs after - them. Its effect on cache state was not measured. -- FFM emulator runs (`HSA_MODEL_MEMFILE`) use private per-worker locks. A 409,600-element kernel - may be much slower than a copy under emulation; a host fallback for that mode may be worth - keeping (not measured). -- `CpuOnly` skips the client launch entirely and is unaffected. - -## Edge case: residue attribution - -The check runs once per solution, after the **first** warmup launch. Residue left by any later -launch of the same solution is not observed at that solution's check. What happens to it depends -on whether the next solution's input reset re-zeroes the buffer. There is no dedicated "clear the -Synchronizer between solutions" step; kernels are trusted to self-clean. - -`prepareGPUInputs` (`DataInitialization.hpp`) has two paths: - -- **Slow path** (bounds checking enabled, problem-dependent data, or swizzled tensors). It calls - `copyInputs`, which copies *every* tensor from its pristine GPU copy, including the zero-filled - Synchronizer. The buffer is re-zeroed before every solution. **Measured** on `dtl`: all 49,995 - checks were preceded by a D2D write into the Synchronizer buffer (111,153 writes in total). -- **Fast path** (otherwise). It calls only `resetOutput`, which skips non-output tensors. The - Synchronizer is not marked as an output, so it carries over to the next solution. This is from - the code; not measured, because none of the cached configs take the fast path. - -| Path (aggregate reset counters) | Configs with checks | Checks | Residue from a launch after N's first warmup | -| --- | --- | --- | --- | -| Slow only, no fast-path resets (`dtl`, `subtile_mxfp4`, …) | 82 | 361,102 (59%) | wiped before N+1 runs; **never reported** | -| Fast (`fp16_tn`, `fp32_tn`, `bf16_tn`, `subtile_bf16`, … 22 configs) | 22 | 247,771 (41%) | carried over; **reported against the next active solution**, which is marked `FAILED` | - -On the fast path, the next active solution may also be a different problem (if no remaining -solution of the current problem can use the buffer), and its own first launch runs on stale -flags. Residue that appears on the *first* launch is attributed correctly on both paths; a -deterministic self-cleaning bug is therefore caught. Only residue that first appears on later -launches is affected: race- or timing-dependent behavior, a different kernel variant in later -warmups (`kernels[i % kernels.size()]`), or behavior that depends on the buffer's starting state. -Two more gaps: the last checked solution's late residue is never examined, and configs that -launch with zero warmups (`NumWarmups` 0 and validation off) never trigger `validateWarmups`. - -No dirty-buffer report ("Synchronizer left dirty") appears in any log examined (the `dtl` and -`subtile_mxfp4` re-runs, and the compressed logs under `timing_runs`). The edge case is a -correctness gap in the design, not an observed failure. - -### Requirement for the GPU check - -Whatever replaces the current check must observe **every** launch of a solution before the next -solution's launches, and must attribute the result to that solution. Running the GPU reduction -in `postSolution`, instead of after the first warmup, satisfies this: - -- It sees all warmups and benchmark runs of the solution, on both reset paths. That also closes - the slow-path gap, where late residue is currently wiped unseen. -- It is still one check per solution. -- The GPU is already idle there when benchmark runs happened (`BenchmarkTimer::postEnqueues` - synchronizes), so no kernel time is misattributed. Enqueuing on `stream` keeps it ordered either - way. -- `postSolution` receives no inputs, so the `ProblemInputs` pointer must be saved from - `validateWarmups` or `preSolution`. -- It must run before `ReferenceValidator::postSolution` reports the verdict, as the existing - registration order already arranges. -- Keep clearing the buffer after a dirty result: on the fast path nothing else will. - -Trade-off: today a dirty first launch is cleared before the solution's timed runs. With the check -in `postSolution`, a failing solution's timed runs execute on a dirty buffer. That solution is -marked `FAILED` either way. - -## Not measured / open items - -- End-to-end savings of the GPU check (only per-check costs were measured). -- Fast-path carry-over in a real run. A small fast-path config such as `benchmark_runs__fp32_tn` - (120 checks) or `streamk__sk_hgemm_race` (256 checks) would need to be built to show it with the - reset-tracking shim. -- Where the warmup wait lands when `SynchronizerValidator` is inactive. -- The cause of the slow first copy per process and of the >100 µs outlier copies. -- Cache effects of the reduction kernel on subsequent timed runs, and its cost under the FFM - emulator. diff --git a/projects/hipblaslt/tensilelite/timing_runs/build_fp16_gradient_bias_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/build_fp16_gradient_bias_analysis.txt deleted file mode 100644 index 409207aa18c3..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/build_fp16_gradient_bias_analysis.txt +++ /dev/null @@ -1,135 +0,0 @@ -======================================================================================================================== -TIMING ANALYSIS SUMMARY -======================================================================================================================== - -TIMING BY PHASE ------------------------------------------------------------------------------------------------------------------------- - Category Count Total (ms) Mean (ms) % of Parent % of Wall ------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 1 51,138.23 51,138.23 100.0% - -------------------------------------------------------------------------------------------------------------------- - python_kernel_compilation 6 48,227.66 8,037.94 94.3% 94.3% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_build_src_co 6 42,216.35 7,036.06 87.5% 82.6% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_kernel_build_src_co.compile 3 42,158.66 14,052.89 99.9% 82.4% - python_kernel_build_src_co.unbundle 3 30.17 10.06 0.1% 0.1% - python_kernel_build_src_co.cache_check 6 21.80 3.63 0.1% 0.0% - python_kernel_build_src_co.cache_populate 3 4.18 1.39 0.0% 0.0% - (untracked) 1.07 0.0% 0.0% - python_kernel_build_src_co.setup 6 0.25 0.04 0.0% 0.0% - python_kernel_build_src_co.move 3 0.22 0.07 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_codegen 6 2,695.94 449.32 5.6% 5.3% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_write_assemble 6 1,193.80 198.97 2.5% 2.3% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_bench_setup 6 795.21 132.53 1.6% 1.6% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_validate 6 590.25 98.37 1.2% 1.2% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_write_helpers 6 298.19 49.70 0.6% 0.6% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_bench_postprocess 6 209.56 34.93 0.4% 0.4% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_benchpost_lib_construction 6 142.55 23.76 68.0% 0.3% - python_benchpost_library_write 6 38.59 6.43 18.4% 0.1% - python_benchpost_naming 6 27.01 4.50 12.9% 0.1% - python_benchpost_client_config 6 1.11 0.18 0.5% 0.0% - (untracked) 0.30 0.1% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_build_co 6 178.51 29.75 0.4% 0.3% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 32.93 0.1% 0.1% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_setup 6 16.94 2.82 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - python_solution_generation 6 2,796.55 466.09 5.5% 5.5% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_forked_solutions 6 2,796.40 466.07 100.0% 5.5% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 0.11 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_fork_permutations 6 0.04 0.01 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_custom_kernels 6 0.01 0.00 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - (untracked) 43.17 0.1% 0.1% - -------------------------------------------------------------------------------------------------------------------- - python_write_solutions 6 41.53 6.92 0.1% 0.1% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_dump 6 40.12 6.69 96.6% 0.1% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_prepare 6 0.80 0.13 1.9% 0.0% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_wsol_prepare_nocache 6 0.78 0.13 98.1% 0.0% - (untracked) 0.01 1.9% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 0.52 1.3% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_header 6 0.09 0.02 0.2% 0.0% - -------------------------------------------------------------------------------------------------------------------- - python_solution_indexing 6 22.83 3.81 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - python_write_cache 6 6.48 1.08 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - python_cache_check 6 0.01 0.00 0.0% 0.0% - ------------------------------------------------------------------------------------------------------------------------- - ESTIMATED WALL CLOCK TIME: 51.14s - -HIERARCHY VALIDATION: OK (all child phases fit within their parents) - -TIME BREAKDOWN (visual, % of wall clock) ------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 100.0% ################################################## - python_kernel_compilation 94.3% ############################################### - python_kernel_build_src_co 82.6% ######################################### - python_kernel_build_src_co.compile 82.4% ######################################### - python_kernel_build_src_co.unbundle 0.1% - python_kernel_build_src_co.cache_check 0.0% - python_kernel_build_src_co.cache_populate 0.0% - (untracked) 0.0% - python_kernel_build_src_co.setup 0.0% - python_kernel_build_src_co.move 0.0% - python_kernel_codegen 5.3% ## - python_kernel_write_assemble 2.3% # - python_kernel_bench_setup 1.6% - python_kernel_validate 1.2% - python_kernel_write_helpers 0.6% - python_kernel_bench_postprocess 0.4% - python_benchpost_lib_construction 0.3% - python_benchpost_library_write 0.1% - python_benchpost_naming 0.1% - python_benchpost_client_config 0.0% - (untracked) 0.0% - python_kernel_build_co 0.3% - (untracked) 0.1% - python_kernel_setup 0.0% - python_solution_generation 5.5% ## - python_solgen_forked_solutions 5.5% ## - (untracked) 0.0% - python_solgen_fork_permutations 0.0% - python_solgen_custom_kernels 0.0% - (untracked) 0.1% - python_write_solutions 0.1% - python_wsol_dump 0.1% - python_wsol_prepare 0.0% - python_wsol_prepare_nocache 0.0% - (untracked) 0.0% - (untracked) 0.0% - python_wsol_header 0.0% - python_solution_indexing 0.0% - python_write_cache 0.0% - python_cache_check 0.0% - -KEY INSIGHTS (% of wall clock) ------------------------------------------------------------------------------------------------------------------------- - Python phases (code gen, compilation): 51.02s (99.8%) - Python I/O overhead (cache, YAML): 0.05s (0.1%) - C++ client setup (library loading): 0.00s (0.0%) - Data preparation (CPU init, GPU prep): 0.00s (0.0%) - Reference computation (CPU GEMM): 0.00s (0.0%) - Kernel execution (warmup, benchmark): 0.00s (0.0%) - - diff --git a/projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_1_build_cold_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_1_build_cold_analysis.txt deleted file mode 100644 index 34b5d57d2e17..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_1_build_cold_analysis.txt +++ /dev/null @@ -1,135 +0,0 @@ -======================================================================================================================== -TIMING ANALYSIS SUMMARY -======================================================================================================================== - -TIMING BY PHASE ------------------------------------------------------------------------------------------------------------------------- - Category Count Total (ms) Mean (ms) % of Parent % of Wall ------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 1 91,714.59 91,714.59 100.0% - -------------------------------------------------------------------------------------------------------------------- - python_kernel_compilation 9 74,126.10 8,236.23 80.8% 80.8% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_bench_postprocess 9 25,589.12 2,843.24 34.5% 27.9% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_benchpost_lib_construction 9 17,320.21 1,924.47 67.7% 18.9% - python_benchpost_library_write 9 5,937.58 659.73 23.2% 6.5% - python_benchpost_naming 9 2,327.62 258.62 9.1% 2.5% - python_benchpost_client_config 9 2.83 0.31 0.0% 0.0% - (untracked) 0.87 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_bench_setup 9 14,909.77 1,656.64 20.1% 16.3% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_codegen 9 10,359.67 1,151.07 14.0% 11.3% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_write_assemble 9 9,826.49 1,091.83 13.3% 10.7% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_validate 9 7,739.34 859.93 10.4% 8.4% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_build_src_co 9 3,590.01 398.89 4.8% 3.9% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_kernel_build_src_co.compile 1 3,558.18 3,558.18 99.1% 3.9% - python_kernel_build_src_co.cache_check 9 19.38 2.15 0.5% 0.0% - python_kernel_build_src_co.unbundle 1 8.88 8.88 0.2% 0.0% - (untracked) 1.89 0.1% 0.0% - python_kernel_build_src_co.cache_populate 1 0.85 0.85 0.0% 0.0% - python_kernel_build_src_co.setup 9 0.76 0.08 0.0% 0.0% - python_kernel_build_src_co.move 1 0.07 0.07 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_build_co 9 821.31 91.26 1.1% 0.9% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 636.74 0.9% 0.7% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_setup 9 611.45 67.94 0.8% 0.7% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_write_helpers 9 42.19 4.69 0.1% 0.0% - -------------------------------------------------------------------------------------------------------------------- - python_solution_generation 9 8,591.97 954.66 9.4% 9.4% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_forked_solutions 9 8,591.67 954.63 100.0% 9.4% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 0.23 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_fork_permutations 9 0.06 0.01 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_custom_kernels 9 0.02 0.00 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - python_write_solutions 9 4,141.41 460.16 4.5% 4.5% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_dump 9 4,012.82 445.87 96.9% 4.4% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_prepare 9 113.64 12.63 2.7% 0.1% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_wsol_prepare_nocache 9 113.60 12.62 100.0% 0.1% - (untracked) 0.04 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 14.75 0.4% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_header 9 0.20 0.02 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - (untracked) 2,531.63 2.8% 2.8% - -------------------------------------------------------------------------------------------------------------------- - python_solution_indexing 9 2,310.87 256.76 2.5% 2.5% - -------------------------------------------------------------------------------------------------------------------- - python_write_cache 9 12.60 1.40 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - python_cache_check 9 0.01 0.00 0.0% 0.0% - ------------------------------------------------------------------------------------------------------------------------- - ESTIMATED WALL CLOCK TIME: 91.71s - -HIERARCHY VALIDATION: OK (all child phases fit within their parents) - -TIME BREAKDOWN (visual, % of wall clock) ------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 100.0% ################################################## - python_kernel_compilation 80.8% ######################################## - python_kernel_bench_postprocess 27.9% ############# - python_benchpost_lib_construction 18.9% ######### - python_benchpost_library_write 6.5% ### - python_benchpost_naming 2.5% # - python_benchpost_client_config 0.0% - (untracked) 0.0% - python_kernel_bench_setup 16.3% ######## - python_kernel_codegen 11.3% ##### - python_kernel_write_assemble 10.7% ##### - python_kernel_validate 8.4% #### - python_kernel_build_src_co 3.9% # - python_kernel_build_src_co.compile 3.9% # - python_kernel_build_src_co.cache_check 0.0% - python_kernel_build_src_co.unbundle 0.0% - (untracked) 0.0% - python_kernel_build_src_co.cache_populate 0.0% - python_kernel_build_src_co.setup 0.0% - python_kernel_build_src_co.move 0.0% - python_kernel_build_co 0.9% - (untracked) 0.7% - python_kernel_setup 0.7% - python_kernel_write_helpers 0.0% - python_solution_generation 9.4% #### - python_solgen_forked_solutions 9.4% #### - (untracked) 0.0% - python_solgen_fork_permutations 0.0% - python_solgen_custom_kernels 0.0% - python_write_solutions 4.5% ## - python_wsol_dump 4.4% ## - python_wsol_prepare 0.1% - python_wsol_prepare_nocache 0.1% - (untracked) 0.0% - (untracked) 0.0% - python_wsol_header 0.0% - (untracked) 2.8% # - python_solution_indexing 2.5% # - python_write_cache 0.0% - python_cache_check 0.0% - -KEY INSIGHTS (% of wall clock) ------------------------------------------------------------------------------------------------------------------------- - Python phases (code gen, compilation): 82.72s (90.2%) - Python I/O overhead (cache, YAML): 4.15s (4.5%) - C++ client setup (library loading): 0.00s (0.0%) - Data preparation (CPU init, GPU prep): 0.00s (0.0%) - Reference computation (CPU GEMM): 0.00s (0.0%) - Kernel execution (warmup, benchmark): 0.00s (0.0%) - - diff --git a/projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_2_build_warm_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_2_build_warm_analysis.txt deleted file mode 100644 index e2c90f700137..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_2_build_warm_analysis.txt +++ /dev/null @@ -1,127 +0,0 @@ -======================================================================================================================== -TIMING ANALYSIS SUMMARY -======================================================================================================================== - -TIMING BY PHASE ------------------------------------------------------------------------------------------------------------------------- - Category Count Total (ms) Mean (ms) % of Parent % of Wall ------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 1 87,520.82 87,520.82 100.0% - -------------------------------------------------------------------------------------------------------------------- - python_kernel_compilation 9 70,010.08 7,778.90 80.0% 80.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_bench_postprocess 9 25,558.91 2,839.88 36.5% 29.2% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_benchpost_lib_construction 9 17,315.62 1,923.96 67.7% 19.8% - python_benchpost_library_write 9 5,915.03 657.23 23.1% 6.8% - python_benchpost_naming 9 2,324.23 258.25 9.1% 2.7% - python_benchpost_client_config 9 3.07 0.34 0.0% 0.0% - (untracked) 0.96 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_bench_setup 9 14,964.03 1,662.67 21.4% 17.1% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_codegen 9 10,395.52 1,155.06 14.8% 11.9% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_write_assemble 9 9,300.36 1,033.37 13.3% 10.6% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_validate 9 7,750.75 861.19 11.1% 8.9% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_build_co 9 804.50 89.39 1.1% 0.9% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_setup 9 611.86 67.98 0.9% 0.7% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 575.11 0.8% 0.7% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_write_helpers 9 30.33 3.37 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_build_src_co 9 18.71 2.08 0.0% 0.0% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_kernel_build_src_co.cache_check 9 16.38 1.82 87.6% 0.0% - (untracked) 1.75 9.4% 0.0% - python_kernel_build_src_co.setup 9 0.57 0.06 3.1% 0.0% - -------------------------------------------------------------------------------------------------------------------- - python_solution_generation 9 8,573.30 952.59 9.8% 9.8% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_forked_solutions 9 8,572.97 952.55 100.0% 9.8% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 0.25 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_fork_permutations 9 0.06 0.01 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_custom_kernels 9 0.02 0.00 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - python_write_solutions 9 4,140.77 460.09 4.7% 4.7% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_dump 9 4,007.89 445.32 96.8% 4.6% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_prepare 9 119.71 13.30 2.9% 0.1% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_wsol_prepare_nocache 9 119.67 13.30 100.0% 0.1% - (untracked) 0.04 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 12.96 0.3% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_header 9 0.21 0.02 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - (untracked) 2,476.17 2.8% 2.8% - -------------------------------------------------------------------------------------------------------------------- - python_solution_indexing 9 2,307.97 256.44 2.6% 2.6% - -------------------------------------------------------------------------------------------------------------------- - python_write_cache 9 12.52 1.39 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - python_cache_check 9 0.01 0.00 0.0% 0.0% - ------------------------------------------------------------------------------------------------------------------------- - ESTIMATED WALL CLOCK TIME: 87.52s - -HIERARCHY VALIDATION: OK (all child phases fit within their parents) - -TIME BREAKDOWN (visual, % of wall clock) ------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 100.0% ################################################## - python_kernel_compilation 80.0% ####################################### - python_kernel_bench_postprocess 29.2% ############## - python_benchpost_lib_construction 19.8% ######### - python_benchpost_library_write 6.8% ### - python_benchpost_naming 2.7% # - python_benchpost_client_config 0.0% - (untracked) 0.0% - python_kernel_bench_setup 17.1% ######## - python_kernel_codegen 11.9% ##### - python_kernel_write_assemble 10.6% ##### - python_kernel_validate 8.9% #### - python_kernel_build_co 0.9% - python_kernel_setup 0.7% - (untracked) 0.7% - python_kernel_write_helpers 0.0% - python_kernel_build_src_co 0.0% - python_kernel_build_src_co.cache_check 0.0% - (untracked) 0.0% - python_kernel_build_src_co.setup 0.0% - python_solution_generation 9.8% #### - python_solgen_forked_solutions 9.8% #### - (untracked) 0.0% - python_solgen_fork_permutations 0.0% - python_solgen_custom_kernels 0.0% - python_write_solutions 4.7% ## - python_wsol_dump 4.6% ## - python_wsol_prepare 0.1% - python_wsol_prepare_nocache 0.1% - (untracked) 0.0% - (untracked) 0.0% - python_wsol_header 0.0% - (untracked) 2.8% # - python_solution_indexing 2.6% # - python_write_cache 0.0% - python_cache_check 0.0% - -KEY INSIGHTS (% of wall clock) ------------------------------------------------------------------------------------------------------------------------- - Python phases (code gen, compilation): 78.58s (89.8%) - Python I/O overhead (cache, YAML): 4.15s (4.7%) - C++ client setup (library loading): 0.00s (0.0%) - Data preparation (CPU init, GPU prep): 0.00s (0.0%) - Reference computation (CPU GEMM): 0.00s (0.0%) - Kernel execution (warmup, benchmark): 0.00s (0.0%) - - diff --git a/projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_3_run_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_3_run_analysis.txt deleted file mode 100644 index d63f173e8a18..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cgemm/cgemm_3_run_analysis.txt +++ /dev/null @@ -1,173 +0,0 @@ -======================================================================================================================== -TIMING ANALYSIS SUMMARY -======================================================================================================================== - -TIMING BY PHASE ------------------------------------------------------------------------------------------------------------------------- - Category Count Total (ms) Mean (ms) % of Parent % of Wall ------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 1 41,022.07 41,022.07 99.8% - -------------------------------------------------------------------------------------------------------------------- - python_client_execution 9 40,589.17 4,509.91 98.9% 98.7% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - C++ Client Setup 12,560.29 30.9% 30.6% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - listener_setup 9 4,784.49 531.61 38.1% 11.6% - hip_initialization 9 3,647.34 405.26 29.0% 8.9% - library_loading 9 2,901.38 322.38 23.1% 7.1% - code_object_loading 9 694.46 77.16 5.5% 1.7% - data_init_setup 9 501.74 55.75 4.0% 1.2% - lazy_loading_init 9 29.58 3.29 0.2% 0.1% - reporter_setup 9 1.14 0.13 0.0% 0.0% - solution_iterator_setup 9 0.16 0.02 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Data Preparation 10,631.87 26.2% 25.9% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - pre_problem 54 10,407.95 192.74 97.9% 25.3% - cpu_reference_gemm 54 10,378.99 192.20 99.7% 25.2% - solve_cpu_slow 54 10,378.80 192.20 100.0% 25.2% - (untracked) 0.18 0.0% 0.0% - cpu_data_init 54 27.34 0.51 0.3% 0.1% - (untracked) 1.62 0.0% 0.0% - gpu_input_reset 45,792 214.59 0.00 2.0% 0.5% - gpu_input_preparation 54 7.80 0.14 0.1% 0.0% - rotating_buffer_preparation 54 1.53 0.03 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Reference Computation 5,199.40 12.8% 12.6% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - validate_warmups 45,792 5,199.40 0.11 100.0% 12.6% - validate_gpu_readback 45,792 3,537.91 0.08 68.0% 8.6% - validate_element_comparison 45,792 1,560.88 0.03 30.0% 3.8% - (untracked) 72.72 1.4% 0.2% - validate_gpu_sync 45,792 27.88 0.00 0.5% 0.1% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Kernel Execution 4,731.00 11.7% 11.5% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - post_solution 122,112 1,937.84 0.02 41.0% 4.7% - post_solution_log 122,112 1,604.87 0.01 82.8% 3.9% - post_solution_reporting 122,112 259.89 0.00 13.4% 0.6% - post_solution_perf_calc 122,112 46.18 0.00 2.4% 0.1% - post_solution_validation 122,112 25.99 0.00 1.3% 0.1% - post_solution_result_file 122,112 3.30 0.00 0.2% 0.0% - post_solution_sol_advance 122,112 3.22 0.00 0.2% 0.0% - post_solution_perf_reset 122,112 3.17 0.00 0.2% 0.0% - pre_solution 122,112 1,251.00 0.01 26.4% 3.0% - warmup_runs 91,584 1,090.34 0.01 23.0% 2.7% - kernel_solving 45,792 272.51 0.01 5.8% 0.7% - post_problem 54 102.96 1.91 2.2% 0.3% - solution_selection 122,112 70.89 0.00 1.5% 0.2% - benchmark_runs 45,792 5.45 0.00 0.1% 0.0% - (untracked) 5.45 100.0% 0.0% - finalize_report 9 0.01 0.00 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Timing Overhead 4,006.26 9.9% 9.7% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - flush_timing_buffer 9 3,917.83 435.31 97.8% 9.5% - timing_overhead 9 88.43 9.83 2.2% 0.2% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked C++ overhead) 3,460.37 8.5% 8.4% - -------------------------------------------------------------------------------------------------------------------- - (untracked) 411.37 1.0% 1.0% - -------------------------------------------------------------------------------------------------------------------- - python_cache_check 9 19.90 2.21 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - python_write_solutions 9 1.62 0.18 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_prepare 9 1.10 0.12 67.6% 0.0% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_wsol_prepare_cache 9 1.05 0.12 96.3% 0.0% - (untracked) 0.04 3.7% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_header 9 0.42 0.05 25.8% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 0.11 6.6% 0.0% - ------------------------------------------------------------------------------------------------------------------------- - ESTIMATED WALL CLOCK TIME: 41.11s - -HIERARCHY VALIDATION: OK (all child phases fit within their parents) - -TIME BREAKDOWN (visual, % of wall clock) ------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 99.8% ################################################# - python_client_execution 98.7% ################################################# - C++ Client Setup 30.6% ############### - listener_setup 11.6% ##### - hip_initialization 8.9% #### - library_loading 7.1% ### - code_object_loading 1.7% - data_init_setup 1.2% - lazy_loading_init 0.1% - reporter_setup 0.0% - solution_iterator_setup 0.0% - Data Preparation 25.9% ############ - pre_problem 25.3% ############ - cpu_reference_gemm 25.2% ############ - solve_cpu_slow 25.2% ############ - (untracked) 0.0% - cpu_data_init 0.1% - (untracked) 0.0% - gpu_input_reset 0.5% - gpu_input_preparation 0.0% - rotating_buffer_preparation 0.0% - Reference Computation 12.6% ###### - validate_warmups 12.6% ###### - validate_gpu_readback 8.6% #### - validate_element_comparison 3.8% # - (untracked) 0.2% - validate_gpu_sync 0.1% - Kernel Execution 11.5% ##### - post_solution 4.7% ## - post_solution_log 3.9% # - post_solution_reporting 0.6% - post_solution_perf_calc 0.1% - post_solution_validation 0.1% - post_solution_result_file 0.0% - post_solution_sol_advance 0.0% - post_solution_perf_reset 0.0% - pre_solution 3.0% # - warmup_runs 2.7% # - kernel_solving 0.7% - post_problem 0.3% - solution_selection 0.2% - benchmark_runs 0.0% - (untracked) 0.0% - finalize_report 0.0% - Timing Overhead 9.7% #### - flush_timing_buffer 9.5% #### - timing_overhead 0.2% - (untracked C++ overhead) 8.4% #### - (untracked) 1.0% - python_cache_check 0.0% - python_write_solutions 0.0% - python_wsol_prepare 0.0% - python_wsol_prepare_cache 0.0% - (untracked) 0.0% - python_wsol_header 0.0% - (untracked) 0.0% - -KEY INSIGHTS (% of wall clock) ------------------------------------------------------------------------------------------------------------------------- - Python phases (code gen, compilation): 0.00s (0.0%) - Python I/O overhead (cache, YAML): 0.02s (0.1%) - C++ client setup (library loading): 12.56s (30.6%) - Data preparation (CPU init, GPU prep): 10.63s (25.9%) - Reference computation (CPU GEMM): 5.21s (12.7%) - Kernel execution (warmup, benchmark): 4.78s (11.6%) - - -TOP 10 SLOWEST PROBLEMS (by CPU reference time) ------------------------------------------------------------------------------------------------------------------------- - M N K Batch TypeA TypeD CPU Ref (ms) GPU (ms) - ---------------------------------------------------------------------------------------------------------------------- - 128 128 63 2 ComplexFloat ComplexFloat 359.12 0.00 - 127 127 63 2 ComplexFloat ComplexFloat 318.96 0.00 - 129 129 63 2 ComplexFloat ComplexFloat 265.68 0.00 - 127 127 64 2 ComplexFloat ComplexFloat 265.15 0.00 - 127 127 64 2 ComplexFloat ComplexFloat 259.98 0.00 - 127 127 63 2 ComplexFloat ComplexFloat 254.31 0.00 - 129 129 63 2 ComplexFloat ComplexFloat 253.06 0.00 - 129 129 63 2 ComplexFloat ComplexFloat 251.09 0.00 - 129 129 64 2 ComplexFloat ComplexFloat 247.91 0.00 - 129 129 63 2 ComplexFloat ComplexFloat 243.50 0.00 - diff --git a/projects/hipblaslt/tensilelite/timing_runs/cgemm/times.txt b/projects/hipblaslt/tensilelite/timing_runs/cgemm/times.txt deleted file mode 100644 index f75e044e8876..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cgemm/times.txt +++ /dev/null @@ -1,4 +0,0 @@ -build_cold rc=0 109s -build_warm rc=0 104s -run rc=0 47s -DONE diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/CMS_FINDINGS.md b/projects/hipblaslt/tensilelite/timing_runs/cms/CMS_FINDINGS.md deleted file mode 100644 index 8fad970fc1d9..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cms/CMS_FINDINGS.md +++ /dev/null @@ -1,110 +0,0 @@ -# CMS (UseCustomMainLoopSchedule) build-only analysis, gfx950 target - -Run 2026-10-16 on a gfx942 host (build-only, `--gpu-targets gfx950`, so no gfx950 GPU needed), -cold caches, `taskset -c 0-24` (25 cores), `TimingInstrumentation=True`, `/usr/bin/time -v`. -`gfx952` does not exist in this tree; treated as gfx950 (the gfx95x family). - -## Results - -| Config | Wall | User+Sys CPU | CPU used (of 25 cores) | Groups | Kernels/group | -|---|---|---|---|---|---| -| `gemm/gfx950/custom_mainloop_scheduling` | 112 s | 166 s | **148%** (about 1.5 cores) | 66 | 1 (53 groups) or 2 (13) | -| `gemm/gfx950/custom_mainloop_scheduling_tf32` | 45 s | 72 s | 160% | 22 | 1 | -| `ductile/gfx950/f8bf16_tn_cms` | 23 s, failed | | 392% | n/a | n/a | - -`f8bf16_tn_cms` fails by design in build-only: the ductile backend warns that it does not support -build-only and forces a benchmark run, which needs a gfx950 GPU -(`Expected results file does not exist`). Its logs are kept; it is not part of the analysis. - -## Is the pathology still there? Yes. - -The 25-core pool is idle almost the whole run: about 1.5 cores busy on average. - -1. **Groups run strictly one after another.** `BenchmarkProblems.main` loops serially over - `for problemType: for sizeGroup: _benchmarkProblemType(...)`. The big file has 9 problem types and - 66 `BenchmarkProblemSizeGroup`s. Each group does its own fork / codegen / write / compile / library - write before the next starts. -2. **Each group holds almost no parallel work.** Each group has `UseCustomMainLoopSchedule` as a - single value (`[1]` in 51 groups) or `[0, 1]` (14 groups), and every other ForkParameter is fixed - (58 unique fork sets across 66 groups, so the groups are different hand-picked tile configs rather - than duplicates). Result: `Actual Solutions: 1 / 1` in 53 groups and `2 / 2` in 13. - The log shows `Writing assembly kernels: Launching 25 threads for 1 tasks` (or 2 tasks) in all - 66 groups, i.e. 25 workers for 1-2 tasks. -3. **A group costs about 1 s no matter what** (median 0.98 s for 1-solution groups, 1.13 s for - 2-solution groups; `python_kernel_codegen` 0.76 s/group average). Total about 66 x 1.4 s plus the - compiles. -4. **The assembler/compiler step is also serial.** `build_src_co.compile` ran 3 times (9.3 + 4.7 + 4.9 s - = 18.8 s, 18% of the build) on unique helper kernels, one after the other, inside three of the - groups. -5. Per-stage totals (66 groups): kernel codegen 50.4 s (49%), compile 18.8 s (18%), solution - generation 13.8 s (13%), write/assemble 13.4 s (13%). - -Not the cause: process-pool creation. A bare `Pool(25)` plus one task costs about 22 ms here, which -is small compared with the roughly 1 s per group. - -## Experiment: does group-level parallelism recover the time? - -`split_experiment.py` splits the same YAML round-robin by size group into N shard configs and -builds them concurrently (25 cores divided evenly, cold caches, same flags). - -| Shards | Cores/shard | Wall | Speedup vs 112 s | -|---|---|---|---| -| 1 (original) | 25 | 112 s (pytest), 103.6 s inside the benchmark step | 1.0x | -| 5 | 5 | 36.8 s | 3.0x | -| 25 | 1 | 21.9 s | 5.1x | - -The speedup is real but is capped by the compile step: each shard compiles its own copy of the -helper kernels because the shared helper-kernel cache starts cold (29 `cache MISS` and 265 s of total -compile CPU at 25 shards, against 3 misses and 18.8 s in the single run). Each shard at 25 ways takes -12-19 s, which is mostly its own helper compile. - -## Where the speedup is - -1. **Run size groups concurrently** (in `BenchmarkProblems.main`, or by building the problem - types / groups with a process pool). This is the main win: 3x to 5x measured on a split of the - same input. It needs per-group outputs to be independent; the experiment splits them without any - issue, so for build-only they appear to be. -2. **Compile helper kernels once, up front**, instead of inside whichever group first misses the - cache. This removes the serial 18.8 s and the duplicated compiles seen in the split runs - (e.g. warm the helper cache, or key the cache so concurrent builders share it). -3. **Batch the YAML** only if a config author can: groups that share the same ProblemType and are - only different tile configs can sit in one `ForkParameters` sweep, which Tensile already - parallelizes. The CMS file cannot do that as written: each group is a specific hand-picked tile - config, not a cross product, so there is no way to express them as one sweep without generating - extra kernels. -4. Lower priority: Python codegen is 49% of CPU time at 0.76 s per kernel; a faster per-kernel codegen - helps every config, not just these. - -## Caveats - -- Build-only for gfx950 on a gfx942 host: no GPU run, so kernel correctness was not checked. -- Single run per measurement, on a shared machine. The split experiments use hand-written shards, - not a production change; they show the potential, not a verified implementation. -- The experiment ran under `Tensile.Tensile([...])` directly (not pytest), so exact overhead differs - slightly from the pytest runs. - -## Files - -- `custom_mainloop_scheduling_build.log`, `..._time.txt` (`/usr/bin/time -v`), `..._analysis.txt` -- the same three for `custom_mainloop_scheduling_tf32` and `f8bf16_tn_cms` -- `split_experiment.py`, `run_split.sh`, `split_5/`, `split_25/`, `split_*_result.txt` -## What the helper-kernel cache saves, single shard and 25 shards - -Same input and conditions (25 cores total), caches kept (warm) versus deleted (cold): - -| Run | Cold | Warm | Saved by cache | Helper-kernel compiles | -|---|---|---|---|---| -| 1 shard (the 112 s case) | 111 s (pytest) | 101 s | about 10 s (9%) | cold 3 compiles = 18.8 s; warm still 1 miss = 9.2 s | -| 25 shards (1 core each) | 21.9 s | 10.1 s | 11.8 s (54%) | cold 29 misses / 265 s compile CPU; warm 0 misses, 66 hits | - -- In the single shard, the warm run still had one helper-cache miss (9.2 s, `build_src_co.compile` - called once), so it saved 9.6 s of the 18.8 s compile, not all of it. The compile is only - 9-17% of the single-shard run, because Python codegen (50 s), solution generation (14 s) and - write/assemble (13 s) are untouched by the cache. -- With 25 shards the cache matters much more: every shard's wall time is dominated by the one - serial helper compile it would otherwise do. Warm, each shard takes 1.6-7.6 s (benchmark step) and - the whole job finishes in 10.1 s, including about 2-3 s of process start/import per shard. -- Net effect of both changes on the CMS file: 111 s cold single shard to 10.1 s warm 25 shards, - about 11x. Parallelism alone gives 5x (cold), the warm cache adds another 2.2x on top at 25 shards. -- Files: `custom_mainloop_scheduling_warm_*` (single warm run), `split_25_warm/`, - `split_25_warm_result.txt`, `run_split_warm.sh`. diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_build_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_build_analysis.txt deleted file mode 100644 index 1c1a6578e972..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_build_analysis.txt +++ /dev/null @@ -1,135 +0,0 @@ -======================================================================================================================== -TIMING ANALYSIS SUMMARY -======================================================================================================================== - -TIMING BY PHASE ------------------------------------------------------------------------------------------------------------------------- - Category Count Total (ms) Mean (ms) % of Parent % of Wall ------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 1 103,629.97 103,629.97 100.0% - -------------------------------------------------------------------------------------------------------------------- - python_kernel_compilation 66 89,512.32 1,356.25 86.4% 86.4% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_codegen 66 50,352.76 762.92 56.3% 48.6% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_build_src_co 66 19,058.00 288.76 21.3% 18.4% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_kernel_build_src_co.compile 3 18,805.92 6,268.64 98.7% 18.1% - python_kernel_build_src_co.cache_check 66 212.03 3.21 1.1% 0.2% - python_kernel_build_src_co.unbundle 3 24.04 8.01 0.1% 0.0% - (untracked) 10.00 0.1% 0.0% - python_kernel_build_src_co.setup 66 3.17 0.05 0.0% 0.0% - python_kernel_build_src_co.cache_populate 3 2.62 0.88 0.0% 0.0% - python_kernel_build_src_co.move 3 0.21 0.07 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_write_assemble 66 13,373.40 202.63 14.9% 12.9% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_write_helpers 66 2,185.94 33.12 2.4% 2.1% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_validate 66 2,064.63 31.28 2.3% 2.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_build_co 66 1,695.29 25.69 1.9% 1.6% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_bench_setup 66 366.11 5.55 0.4% 0.4% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_bench_postprocess 66 205.57 3.11 0.2% 0.2% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_benchpost_lib_construction 66 105.83 1.60 51.5% 0.1% - python_benchpost_library_write 66 38.19 0.58 18.6% 0.0% - python_benchpost_naming 66 29.48 0.45 14.3% 0.0% - python_benchpost_client_config 66 28.77 0.44 14.0% 0.0% - (untracked) 3.30 1.6% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 153.55 0.2% 0.1% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_setup 66 57.09 0.87 0.1% 0.1% - -------------------------------------------------------------------------------------------------------------------- - python_solution_generation 66 13,842.92 209.74 13.4% 13.4% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_forked_solutions 66 13,841.56 209.72 100.0% 13.4% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 0.98 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_fork_permutations 66 0.28 0.00 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_custom_kernels 66 0.11 0.00 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - (untracked) 145.59 0.1% 0.1% - -------------------------------------------------------------------------------------------------------------------- - python_write_cache 66 81.32 1.23 0.1% 0.1% - -------------------------------------------------------------------------------------------------------------------- - python_write_solutions 66 32.50 0.49 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_dump 66 24.75 0.38 76.2% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 5.50 16.9% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_header 66 1.57 0.02 4.8% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_prepare 66 0.68 0.01 2.1% 0.0% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_wsol_prepare_nocache 66 0.55 0.01 80.4% 0.0% - (untracked) 0.13 19.6% 0.0% - -------------------------------------------------------------------------------------------------------------------- - python_solution_indexing 66 15.25 0.23 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - python_cache_check 66 0.07 0.00 0.0% 0.0% - ------------------------------------------------------------------------------------------------------------------------- - ESTIMATED WALL CLOCK TIME: 103.63s - -HIERARCHY VALIDATION: OK (all child phases fit within their parents) - -TIME BREAKDOWN (visual, % of wall clock) ------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 100.0% ################################################## - python_kernel_compilation 86.4% ########################################### - python_kernel_codegen 48.6% ######################## - python_kernel_build_src_co 18.4% ######### - python_kernel_build_src_co.compile 18.1% ######### - python_kernel_build_src_co.cache_check 0.2% - python_kernel_build_src_co.unbundle 0.0% - (untracked) 0.0% - python_kernel_build_src_co.setup 0.0% - python_kernel_build_src_co.cache_populate 0.0% - python_kernel_build_src_co.move 0.0% - python_kernel_write_assemble 12.9% ###### - python_kernel_write_helpers 2.1% # - python_kernel_validate 2.0% - python_kernel_build_co 1.6% - python_kernel_bench_setup 0.4% - python_kernel_bench_postprocess 0.2% - python_benchpost_lib_construction 0.1% - python_benchpost_library_write 0.0% - python_benchpost_naming 0.0% - python_benchpost_client_config 0.0% - (untracked) 0.0% - (untracked) 0.1% - python_kernel_setup 0.1% - python_solution_generation 13.4% ###### - python_solgen_forked_solutions 13.4% ###### - (untracked) 0.0% - python_solgen_fork_permutations 0.0% - python_solgen_custom_kernels 0.0% - (untracked) 0.1% - python_write_cache 0.1% - python_write_solutions 0.0% - python_wsol_dump 0.0% - (untracked) 0.0% - python_wsol_header 0.0% - python_wsol_prepare 0.0% - python_wsol_prepare_nocache 0.0% - (untracked) 0.0% - python_solution_indexing 0.0% - python_cache_check 0.0% - -KEY INSIGHTS (% of wall clock) ------------------------------------------------------------------------------------------------------------------------- - Python phases (code gen, compilation): 103.36s (99.7%) - Python I/O overhead (cache, YAML): 0.11s (0.1%) - C++ client setup (library loading): 0.00s (0.0%) - Data preparation (CPU init, GPU prep): 0.00s (0.0%) - Reference computation (CPU GEMM): 0.00s (0.0%) - Kernel execution (warmup, benchmark): 0.00s (0.0%) - - diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_tf32_build_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_tf32_build_analysis.txt deleted file mode 100644 index fea1d7359c3c..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_tf32_build_analysis.txt +++ /dev/null @@ -1,135 +0,0 @@ -======================================================================================================================== -TIMING ANALYSIS SUMMARY -======================================================================================================================== - -TIMING BY PHASE ------------------------------------------------------------------------------------------------------------------------- - Category Count Total (ms) Mean (ms) % of Parent % of Wall ------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 1 39,264.72 39,264.72 100.0% - -------------------------------------------------------------------------------------------------------------------- - python_kernel_compilation 22 26,389.24 1,199.51 67.2% 67.2% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_build_src_co 22 13,821.39 628.25 52.4% 35.2% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_kernel_build_src_co.compile 2 13,727.73 6,863.86 99.3% 35.0% - python_kernel_build_src_co.cache_check 22 65.52 2.98 0.5% 0.2% - python_kernel_build_src_co.unbundle 2 20.93 10.47 0.2% 0.1% - (untracked) 3.72 0.0% 0.0% - python_kernel_build_src_co.cache_populate 2 1.96 0.98 0.0% 0.0% - python_kernel_build_src_co.setup 22 1.37 0.06 0.0% 0.0% - python_kernel_build_src_co.move 2 0.17 0.08 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_codegen 22 6,607.73 300.35 25.0% 16.8% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_write_assemble 22 3,082.34 140.11 11.7% 7.9% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_validate 22 1,614.68 73.39 6.1% 4.1% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_build_co 22 532.14 24.19 2.0% 1.4% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_write_helpers 22 472.25 21.47 1.8% 1.2% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_bench_setup 22 89.95 4.09 0.3% 0.2% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_bench_postprocess 22 75.63 3.44 0.3% 0.2% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_benchpost_lib_construction 22 37.10 1.69 49.1% 0.1% - python_benchpost_client_config 22 15.64 0.71 20.7% 0.0% - python_benchpost_library_write 22 12.99 0.59 17.2% 0.0% - python_benchpost_naming 22 8.61 0.39 11.4% 0.0% - (untracked) 1.29 1.7% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 66.46 0.3% 0.2% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_setup 22 26.67 1.21 0.1% 0.1% - -------------------------------------------------------------------------------------------------------------------- - python_solution_generation 22 12,758.49 579.93 32.5% 32.5% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_forked_solutions 22 12,757.84 579.90 100.0% 32.5% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 0.50 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_fork_permutations 22 0.10 0.00 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_custom_kernels 22 0.05 0.00 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - (untracked) 66.79 0.2% 0.2% - -------------------------------------------------------------------------------------------------------------------- - python_write_cache 22 32.89 1.49 0.1% 0.1% - -------------------------------------------------------------------------------------------------------------------- - python_write_solutions 22 12.05 0.55 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_dump 22 8.88 0.40 73.7% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 2.24 18.6% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_header 22 0.69 0.03 5.7% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_prepare 22 0.24 0.01 2.0% 0.0% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_wsol_prepare_nocache 22 0.18 0.01 75.8% 0.0% - (untracked) 0.06 24.2% 0.0% - -------------------------------------------------------------------------------------------------------------------- - python_solution_indexing 22 5.23 0.24 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - python_cache_check 22 0.03 0.00 0.0% 0.0% - ------------------------------------------------------------------------------------------------------------------------- - ESTIMATED WALL CLOCK TIME: 39.26s - -HIERARCHY VALIDATION: OK (all child phases fit within their parents) - -TIME BREAKDOWN (visual, % of wall clock) ------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 100.0% ################################################## - python_kernel_compilation 67.2% ################################# - python_kernel_build_src_co 35.2% ################# - python_kernel_build_src_co.compile 35.0% ################# - python_kernel_build_src_co.cache_check 0.2% - python_kernel_build_src_co.unbundle 0.1% - (untracked) 0.0% - python_kernel_build_src_co.cache_populate 0.0% - python_kernel_build_src_co.setup 0.0% - python_kernel_build_src_co.move 0.0% - python_kernel_codegen 16.8% ######## - python_kernel_write_assemble 7.9% ### - python_kernel_validate 4.1% ## - python_kernel_build_co 1.4% - python_kernel_write_helpers 1.2% - python_kernel_bench_setup 0.2% - python_kernel_bench_postprocess 0.2% - python_benchpost_lib_construction 0.1% - python_benchpost_client_config 0.0% - python_benchpost_library_write 0.0% - python_benchpost_naming 0.0% - (untracked) 0.0% - (untracked) 0.2% - python_kernel_setup 0.1% - python_solution_generation 32.5% ################ - python_solgen_forked_solutions 32.5% ################ - (untracked) 0.0% - python_solgen_fork_permutations 0.0% - python_solgen_custom_kernels 0.0% - (untracked) 0.2% - python_write_cache 0.1% - python_write_solutions 0.0% - python_wsol_dump 0.0% - (untracked) 0.0% - python_wsol_header 0.0% - python_wsol_prepare 0.0% - python_wsol_prepare_nocache 0.0% - (untracked) 0.0% - python_solution_indexing 0.0% - python_cache_check 0.0% - -KEY INSIGHTS (% of wall clock) ------------------------------------------------------------------------------------------------------------------------- - Python phases (code gen, compilation): 39.15s (99.7%) - Python I/O overhead (cache, YAML): 0.04s (0.1%) - C++ client setup (library loading): 0.00s (0.0%) - Data preparation (CPU init, GPU prep): 0.00s (0.0%) - Reference computation (CPU GEMM): 0.00s (0.0%) - Kernel execution (warmup, benchmark): 0.00s (0.0%) - - diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_tf32_time.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_tf32_time.txt deleted file mode 100644 index a32dc3884f7d..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_tf32_time.txt +++ /dev/null @@ -1,23 +0,0 @@ - Command being timed: "taskset -c 0-24 pytest tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/custom_mainloop_scheduling_tf32.yaml] --build-only --gpu-targets gfx950 --artifact-dir /tmp/tl_cms_art_custom_mainloop_scheduling_tf32 -s -v -n 0 --durations=0 --timeout=14400 --global-parameters=LibraryFormat='msgpack' --tensile-options=--cxx-compiler,/usr/bin/amdclang++,--global-parameters,TimingInstrumentation=True -p no:warnings" - User time (seconds): 64.01 - System time (seconds): 7.99 - Percent of CPU this job got: 160% - Elapsed (wall clock) time (h:mm:ss or m:ss): 0:44.86 - Average shared text size (kbytes): 0 - Average unshared data size (kbytes): 0 - Average stack size (kbytes): 0 - Average total size (kbytes): 0 - Maximum resident set size (kbytes): 278952 - Average resident set size (kbytes): 0 - Major (requiring I/O) page faults: 29 - Minor (reclaiming a frame) page faults: 2582591 - Voluntary context switches: 8293 - Involuntary context switches: 4941 - Swaps: 0 - File system inputs: 0 - File system outputs: 165680 - Socket messages sent: 0 - Socket messages received: 0 - Signals delivered: 0 - Page size (bytes): 4096 - Exit status: 0 diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_time.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_time.txt deleted file mode 100644 index 569cda5b0876..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_time.txt +++ /dev/null @@ -1,23 +0,0 @@ - Command being timed: "taskset -c 0-24 pytest tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/custom_mainloop_scheduling.yaml] --build-only --gpu-targets gfx950 --artifact-dir /tmp/tl_cms_art_custom_mainloop_scheduling -s -v -n 0 --durations=0 --timeout=14400 --global-parameters=LibraryFormat='msgpack' --tensile-options=--cxx-compiler,/usr/bin/amdclang++,--global-parameters,TimingInstrumentation=True -p no:warnings" - User time (seconds): 149.58 - System time (seconds): 16.64 - Percent of CPU this job got: 148% - Elapsed (wall clock) time (h:mm:ss or m:ss): 1:52.11 - Average shared text size (kbytes): 0 - Average unshared data size (kbytes): 0 - Average stack size (kbytes): 0 - Average total size (kbytes): 0 - Maximum resident set size (kbytes): 622760 - Average resident set size (kbytes): 0 - Major (requiring I/O) page faults: 115 - Minor (reclaiming a frame) page faults: 5556720 - Voluntary context switches: 26154 - Involuntary context switches: 7999 - Swaps: 0 - File system inputs: 8 - File system outputs: 811136 - Socket messages sent: 0 - Socket messages received: 0 - Signals delivered: 0 - Page size (bytes): 4096 - Exit status: 0 diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_warm_build_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_warm_build_analysis.txt deleted file mode 100644 index d4062f531a40..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_warm_build_analysis.txt +++ /dev/null @@ -1,135 +0,0 @@ -======================================================================================================================== -TIMING ANALYSIS SUMMARY -======================================================================================================================== - -TIMING BY PHASE ------------------------------------------------------------------------------------------------------------------------- - Category Count Total (ms) Mean (ms) % of Parent % of Wall ------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 1 94,008.80 94,008.80 100.0% - -------------------------------------------------------------------------------------------------------------------- - python_kernel_compilation 66 79,940.15 1,211.21 85.0% 85.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_codegen 66 50,332.85 762.62 63.0% 53.5% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_write_assemble 66 13,371.46 202.60 16.7% 14.2% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_build_src_co 66 9,473.59 143.54 11.9% 10.1% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_kernel_build_src_co.compile 1 9,195.85 9,195.85 97.1% 9.8% - python_kernel_build_src_co.cache_check 66 249.74 3.78 2.6% 0.3% - (untracked) 13.50 0.1% 0.0% - python_kernel_build_src_co.unbundle 1 9.59 9.59 0.1% 0.0% - python_kernel_build_src_co.setup 66 3.68 0.06 0.0% 0.0% - python_kernel_build_src_co.cache_populate 1 1.16 1.16 0.0% 0.0% - python_kernel_build_src_co.move 1 0.08 0.08 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_write_helpers 66 2,264.19 34.31 2.8% 2.4% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_validate 66 2,044.68 30.98 2.6% 2.2% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_build_co 66 1,630.71 24.71 2.0% 1.7% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_bench_setup 66 371.90 5.63 0.5% 0.4% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_bench_postprocess 66 226.08 3.43 0.3% 0.2% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_benchpost_lib_construction 66 118.27 1.79 52.3% 0.1% - python_benchpost_library_write 66 43.41 0.66 19.2% 0.0% - python_benchpost_client_config 66 32.16 0.49 14.2% 0.0% - python_benchpost_naming 66 28.49 0.43 12.6% 0.0% - (untracked) 3.75 1.7% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 165.38 0.2% 0.2% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_setup 66 59.30 0.90 0.1% 0.1% - -------------------------------------------------------------------------------------------------------------------- - python_solution_generation 66 13,756.23 208.43 14.6% 14.6% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_forked_solutions 66 13,754.77 208.41 100.0% 14.6% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 1.01 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_fork_permutations 66 0.34 0.01 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_custom_kernels 66 0.12 0.00 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - (untracked) 164.15 0.2% 0.2% - -------------------------------------------------------------------------------------------------------------------- - python_write_cache 66 93.01 1.41 0.1% 0.1% - -------------------------------------------------------------------------------------------------------------------- - python_write_solutions 66 37.83 0.57 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_dump 66 28.96 0.44 76.6% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 6.33 16.7% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_header 66 1.75 0.03 4.6% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_prepare 66 0.80 0.01 2.1% 0.0% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_wsol_prepare_nocache 66 0.63 0.01 79.2% 0.0% - (untracked) 0.17 20.8% 0.0% - -------------------------------------------------------------------------------------------------------------------- - python_solution_indexing 66 17.36 0.26 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - python_cache_check 66 0.07 0.00 0.0% 0.0% - ------------------------------------------------------------------------------------------------------------------------- - ESTIMATED WALL CLOCK TIME: 94.01s - -HIERARCHY VALIDATION: OK (all child phases fit within their parents) - -TIME BREAKDOWN (visual, % of wall clock) ------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 100.0% ################################################## - python_kernel_compilation 85.0% ########################################## - python_kernel_codegen 53.5% ########################## - python_kernel_write_assemble 14.2% ####### - python_kernel_build_src_co 10.1% ##### - python_kernel_build_src_co.compile 9.8% #### - python_kernel_build_src_co.cache_check 0.3% - (untracked) 0.0% - python_kernel_build_src_co.unbundle 0.0% - python_kernel_build_src_co.setup 0.0% - python_kernel_build_src_co.cache_populate 0.0% - python_kernel_build_src_co.move 0.0% - python_kernel_write_helpers 2.4% # - python_kernel_validate 2.2% # - python_kernel_build_co 1.7% - python_kernel_bench_setup 0.4% - python_kernel_bench_postprocess 0.2% - python_benchpost_lib_construction 0.1% - python_benchpost_library_write 0.0% - python_benchpost_client_config 0.0% - python_benchpost_naming 0.0% - (untracked) 0.0% - (untracked) 0.2% - python_kernel_setup 0.1% - python_solution_generation 14.6% ####### - python_solgen_forked_solutions 14.6% ####### - (untracked) 0.0% - python_solgen_fork_permutations 0.0% - python_solgen_custom_kernels 0.0% - (untracked) 0.2% - python_write_cache 0.1% - python_write_solutions 0.0% - python_wsol_dump 0.0% - (untracked) 0.0% - python_wsol_header 0.0% - python_wsol_prepare 0.0% - python_wsol_prepare_nocache 0.0% - (untracked) 0.0% - python_solution_indexing 0.0% - python_cache_check 0.0% - -KEY INSIGHTS (% of wall clock) ------------------------------------------------------------------------------------------------------------------------- - Python phases (code gen, compilation): 93.70s (99.7%) - Python I/O overhead (cache, YAML): 0.13s (0.1%) - C++ client setup (library loading): 0.00s (0.0%) - Data preparation (CPU init, GPU prep): 0.00s (0.0%) - Reference computation (CPU GEMM): 0.00s (0.0%) - Kernel execution (warmup, benchmark): 0.00s (0.0%) - - diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_warm_time.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_warm_time.txt deleted file mode 100644 index 438e02afcae3..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cms/custom_mainloop_scheduling_warm_time.txt +++ /dev/null @@ -1,23 +0,0 @@ - Command being timed: "taskset -c 0-24 pytest tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/custom_mainloop_scheduling.yaml] --build-only --gpu-targets gfx950 --artifact-dir /tmp/tl_cms_art_warm -s -v -n 0 --durations=0 --timeout=14400 --global-parameters=LibraryFormat='msgpack' --tensile-options=--cxx-compiler,/usr/bin/amdclang++,--global-parameters,TimingInstrumentation=True -p no:warnings" - User time (seconds): 137.84 - System time (seconds): 16.04 - Percent of CPU this job got: 150% - Elapsed (wall clock) time (h:mm:ss or m:ss): 1:42.57 - Average shared text size (kbytes): 0 - Average unshared data size (kbytes): 0 - Average stack size (kbytes): 0 - Average total size (kbytes): 0 - Maximum resident set size (kbytes): 624096 - Average resident set size (kbytes): 0 - Major (requiring I/O) page faults: 106 - Minor (reclaiming a frame) page faults: 5451311 - Voluntary context switches: 25967 - Involuntary context switches: 13936 - Swaps: 0 - File system inputs: 0 - File system outputs: 805432 - Socket messages sent: 0 - Socket messages received: 0 - Signals delivered: 0 - Page size (bytes): 4096 - Exit status: 0 diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/f8bf16_tn_cms_build_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/f8bf16_tn_cms_build_analysis.txt deleted file mode 100644 index 9689a634e5aa..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cms/f8bf16_tn_cms_build_analysis.txt +++ /dev/null @@ -1,8 +0,0 @@ -======================================================================================================================== -TIMING ANALYSIS SUMMARY -======================================================================================================================== - -No timing data found! - -Make sure you ran with --global-parameters "TimingInstrumentation=True" -and captured stderr output (2>&1). diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/f8bf16_tn_cms_time.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/f8bf16_tn_cms_time.txt deleted file mode 100644 index 3f3209bb43f2..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cms/f8bf16_tn_cms_time.txt +++ /dev/null @@ -1,24 +0,0 @@ -Command exited with non-zero status 1 - Command being timed: "taskset -c 0-24 pytest tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx950/f8bf16_tn_cms.yaml] --build-only --gpu-targets gfx950 --artifact-dir /tmp/tl_cms_art_f8bf16_tn_cms -s -v -n 0 --durations=0 --timeout=14400 --global-parameters=LibraryFormat='msgpack' --tensile-options=--cxx-compiler,/usr/bin/amdclang++,--global-parameters,TimingInstrumentation=True -p no:warnings" - User time (seconds): 80.67 - System time (seconds): 10.13 - Percent of CPU this job got: 392% - Elapsed (wall clock) time (h:mm:ss or m:ss): 0:23.12 - Average shared text size (kbytes): 0 - Average unshared data size (kbytes): 0 - Average stack size (kbytes): 0 - Average total size (kbytes): 0 - Maximum resident set size (kbytes): 483220 - Average resident set size (kbytes): 0 - Major (requiring I/O) page faults: 13 - Minor (reclaiming a frame) page faults: 2817252 - Voluntary context switches: 4991 - Involuntary context switches: 7192 - Swaps: 0 - File system inputs: 0 - File system outputs: 84912 - Socket messages sent: 0 - Socket messages received: 0 - Signals delivered: 0 - Page size (bytes): 4096 - Exit status: 1 diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/run_split.sh b/projects/hipblaslt/tensilelite/timing_runs/cms/run_split.sh deleted file mode 100644 index 9728ab608363..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cms/run_split.sh +++ /dev/null @@ -1,8 +0,0 @@ -# Usage: bash run_split.sh N -- cold caches, then the N-way concurrent build-only experiment. -N=${1:-5} -TL=/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite -cd $TL || exit 1 -export OMP_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=0 PYTHONUNBUFFERED=1 -rm -rf /home/alvasile/.tensile /home/alvasile/.tensilelite /tmp/tl_split_out_${N}_* -python timing_runs/cms/split_experiment.py $N > timing_runs/cms/split_${N}_result.txt 2>&1 -echo finished >> timing_runs/cms/split_${N}_result.txt diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/run_split_warm.sh b/projects/hipblaslt/tensilelite/timing_runs/cms/run_split_warm.sh deleted file mode 100644 index 353376a4a310..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cms/run_split_warm.sh +++ /dev/null @@ -1,8 +0,0 @@ -# Usage: bash run_split_warm.sh N -- N-way concurrent build-only with the caches KEPT (warm). -# Run after a previous cold run so ~/.tensile and ~/.tensilelite are populated. -N=${1:-25} -TL=/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite -cd $TL || exit 1 -export OMP_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=0 PYTHONUNBUFFERED=1 -python timing_runs/cms/split_experiment.py $N _warm > timing_runs/cms/split_${N}_warm_result.txt 2>&1 -echo finished >> timing_runs/cms/split_${N}_warm_result.txt diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/split_25_result.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/split_25_result.txt deleted file mode 100644 index 10297c996930..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cms/split_25_result.txt +++ /dev/null @@ -1,3 +0,0 @@ -total groups 66 -rcs [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] wall 21.9 s -finished diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/split_25_warm_result.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/split_25_warm_result.txt deleted file mode 100644 index 5fa0fd22b930..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cms/split_25_warm_result.txt +++ /dev/null @@ -1,3 +0,0 @@ -total groups 66 -rcs [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] wall 10.1 s -finished diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/split_5_result.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/split_5_result.txt deleted file mode 100644 index 7bd445f5c55f..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cms/split_5_result.txt +++ /dev/null @@ -1,3 +0,0 @@ -total groups 66 -rcs [0, 0, 0, 0, 0] wall 36.8 s -finished diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/split_experiment.py b/projects/hipblaslt/tensilelite/timing_runs/cms/split_experiment.py deleted file mode 100644 index 0332223b284d..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cms/split_experiment.py +++ /dev/null @@ -1,42 +0,0 @@ -"""Split custom_mainloop_scheduling.yaml into N shards (round-robin by size group) and -build-only them concurrently, to test whether group-level parallelism recovers the idle cores. -Usage: python split_experiment.py N (run from the tensilelite dir, caches cleared by caller) -""" -import copy, os, subprocess, sys, time -import yaml - -N = int(sys.argv[1]) -TAG = sys.argv[2] if len(sys.argv) > 2 else "" -TL = "/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite" -SRC = f"{TL}/tensilelite/Tests/common/gemm/gfx950/custom_mainloop_scheduling.yaml" -OUT = f"{TL}/timing_runs/cms/split_{N}{TAG}" -os.makedirs(OUT, exist_ok=True) - -doc = yaml.safe_load(open(SRC)) -flat = [(pt[0], g) for pt in doc["BenchmarkProblems"] for g in pt[1:]] -print("total groups", len(flat)) -shards = [[] for _ in range(N)] -for i, item in enumerate(flat): - shards[i % N].append(item) - -procs = [] -cores_per = 25 // N -t0 = time.time() -for k, items in enumerate(shards): - d = copy.deepcopy(doc) - d["BenchmarkProblems"] = [[pt, g] for pt, g in items] - d.pop("TestParameters", None) - cfg = f"{OUT}/shard{k}.yaml" - yaml.safe_dump(d, open(cfg, "w"), sort_keys=False) - cores = f"{k*cores_per}-{(k+1)*cores_per-1}" - cmd = ["taskset", "-c", cores, "python", "-c", - "import sys; from tensilelite import Tensile; Tensile.Tensile(sys.argv[1:])", - cfg, f"/tmp/tl_split_out_{N}{TAG}_{k}", "--build-only", "--gpu-targets", "gfx950", - "--cxx-compiler", "/usr/bin/amdclang++", - "--global-parameters", "TimingInstrumentation=True", "LibraryFormat='msgpack'"] - log = open(f"{OUT}/shard{k}.log", "w") - procs.append((subprocess.Popen(cmd, stdout=log, stderr=subprocess.STDOUT), time.time())) -rcs = [] -for p, st in procs: - rcs.append(p.wait()) -print("rcs", rcs, "wall %.1f s" % (time.time() - t0)) diff --git a/projects/hipblaslt/tensilelite/timing_runs/cms/times.txt b/projects/hipblaslt/tensilelite/timing_runs/cms/times.txt deleted file mode 100644 index 04a4d313a9ed..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cms/times.txt +++ /dev/null @@ -1,4 +0,0 @@ -custom_mainloop_scheduling rc=0 112s -custom_mainloop_scheduling_tf32 rc=0 45s -f8bf16_tn_cms rc=1 23s -DONE diff --git a/projects/hipblaslt/tensilelite/timing_runs/cold_warm_done.txt b/projects/hipblaslt/tensilelite/timing_runs/cold_warm_done.txt deleted file mode 100644 index 2fdae8d75aac..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cold_warm_done.txt +++ /dev/null @@ -1 +0,0 @@ -ALLDONE diff --git a/projects/hipblaslt/tensilelite/timing_runs/cpu_data_init_investigation.md b/projects/hipblaslt/tensilelite/timing_runs/cpu_data_init_investigation.md deleted file mode 100644 index afd0bc712723..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cpu_data_init_investigation.md +++ /dev/null @@ -1,191 +0,0 @@ -# `cpu_data_init` investigation (gfx950 per-config rerun 2) - -`full_runs_gfx950/per_config_rerun_2/AGGREGATE_analysis.txt` reports `cpu_data_init` at -106,829.68 ms over 28,596 calls (mean 3.74 ms, 4.2% of wall). This note explains where that time -goes. Every number below is a timing measurement; reading the code was only used to decide what to -measure. - -## Method - -- **Raw logs.** Re-parsed the 188 per-config logs (`TIMING_CONTEXT` + `TIMING:cpu_data_init`). - They reconcile exactly with the aggregate: 28,596 calls, 106,830 ms. -- **Instrumented client.** Rebuilt `tensilelite-client` from the exact sources the measured binary - was built from (commit `4c6426cea6`, client and include trees), using the original - `compile_commands.json` / `link.txt`, with extra `ScopedTimer`s in `prepareCPUInputs`, - `initializeCPUInputs` (per tensor), `copyInputs` (per tensor) and around each - `generateMXInput` call. It reproduces CI: `sk_bgemm_div` 8.4–15.0 s vs 15.1 s in CI, and the - MX regression problems 0.4–2.4 s vs 0.3–2.2 s. -- **Microbenchmarks.** Built with the client's flags (`-O3 -DNDEBUG -mf16c -x hip -fopenmp=libomp`, - no `-march`), using the real `TensorDescriptor`, `CoordNumbered`, `BFloat16` and - `libhipblaslt-mxdatagen.a`. Their assembly was checked against the shipped binary with - `objdump`. -- **Environment.** gfx950 host, 2× EPYC 9575F; runs pinned to cores 0–24 with - `OMP_NUM_THREADS=25`, as in CI. `perf` was unavailable (`perf_event_paranoid=4`). The host was - shared, with other users' processes using 30–90% of several cores in 0–24. - -## What the timer covers - -`ScopedTimer("cpu_data_init")` in `ReferenceValidator::preProblem` wraps -`DataInitialization::prepareCPUInputs(problem)`: - -```text -cpu_data_init -└─ prepareCPUInputs - ├─ initializeCPUInputs (only if m_problemDependentData) - │ ├─ initializeMXData → generateMXInput (MX configs) - │ └─ initArray per changed tensor (initArrayTrig, Random C, Zero D, ...) - ├─ copyInputs (hipMemcpy H2H of every tensor, maxElements each) - ├─ initializeConstantInputs (~0 ms) - └─ ConvertToProblemInputs (~0 ms) -``` - -The instrumented sub-timers add up to the parent: - -| Run | initializeCPUInputs | copyInputs | cpu_data_init | -| --- | --- | --- | --- | -| 4096×4096×8192 TrigSin, first call | 246.5 ms | 65.3 ms | 311.8 ms | -| `sk_bgemm_div` | 8524.8 ms | 1213.2 ms | 9738.0 ms | -| MX regression, first problem | 570.0 ms | 574.0 ms | 1144.0 ms | - -Not part of `cpu_data_init`: `data_init_setup` (one-time Random fills for configs whose data is -not problem-dependent), `cpu_reference_gemm`, and `pre_solution_mx_side_regenerate` (44.1 s, which -calls the same `generateMXInput` per solution but was not measured here). - -## Where the 106.8 s goes - -The distribution is heavy-tailed: the top 10 calls are 28.8% of the time, the top 200 are 59.6%, -and the top 1000 are 81.3%. - -| Bucket (by A/B/C init mode and type) | Time | Share | Calls | -| --- | --- | --- | --- | -| MX configs (8/6/4-bit types, mxDataGenerator) | 44.6 s | 42% | 8,961 | -| Trig / Serial / Identity init modes | 43.1 s | 40% | 6,674 | -| Sparse configs | 12.5 s | 12% | 2,688 | -| Everything else (copy only) | 6.6 s | 6% | 10,273 | - -Across all buckets, the first call in each client process (820 calls) is 54.0 s (50.5%). The -remaining 27,776 calls average 1.9 ms. - -Largest single contributors: `streamk/sk_bgemm_div` 15.1 s (1 call), -`gfx950/subtile_mxfp8` 10.3 s (39 calls), `gfx950/subtile_mxfp4` 8.8 s (685 calls), and the two -`subtile_mxfp8_correctness_regression_*` configs 5.3 s and 5.2 s (5 calls each). - -## Findings - -### 1. MX data generation: about half the work is discarded - -For MX configs, `initializeCPUInputs → initializeMXData` calls `generateMXInput` -(`clients/common/src/mxDataGen.cpp`) once per MX side. At problem level there is no current -solution, so no swizzled scale is generated (`swizzle=0` in every call observed). - -Example: the 516M-element FP8 E4M3 A side (7168×71936) of the MX regression config. In-client, -`generateMXInput` took 2094 ms. A harness calling the same library reproduces 2130–2251 ms and -splits it: - -| Step | Time (ms) | Share | -| --- | --- | --- | -| `generate()` (resize + RNG + FP8 convert) | 903–935 | ~42% | -| `getDataBytes()` copy | 128–132 | ~6% | -| `memcpy` into the client buffer | 18–19 | ~1% | -| **`getReferenceFloat()`, result discarded** | 938–1006 | **~45%** | -| **free of that 2 GB vector** | 102–116 | **~5%** | - -- `generateMXInput` returns a full `std::vector` reference (4 bytes per element). The - TensileLite client ignores the return value, so that work is wasted. -- `generate()` includes a single-threaded `m_dataBytes.resize(n)` zero-fill: about 90 ms for - 516 MB. -- On 1 thread the call costs about 40 ns per element. It reaches only 7–11× speedup on 25 threads - (20,935 ms → 2,094 ms for 516M elements). - -### 2. Trig init: an expensive per-element loop - -`initArrayTrig(array, tensor)` runs for Trig init modes, which are problem-dependent and so -re-run whenever a tensor's descriptor changes. In the shipped binary's assembly, every element: - -- allocates and frees a `std::vector` (`operator new`, `memset`, `operator delete`) -- executes 2–4 64-bit `div` instructions (`CoordNumbered`) and calls `TensorDescriptor::index` - out of line -- converts the index to `int`, then calls double-precision libm `sin`/`cos` (`sin@plt`), with no - vectorization - -Microbenchmark, 1 thread, 32M BF16 elements: - -| Variant | ns/element | -| --- | --- | -| The client's loop | 22.1 | -| Without the vector allocation | 14.9 | -| Without the `sin` call | 12.1 | -| Just `sin` + store | 7.0 | -| `sinf` + store | 3.7 | - -- **glibc `sin` slow path.** The argument is the element index. Single-thread cost is 6.9 ns - below about 1.0e8 and 23.6–25 ns at 1.06e8 and above, glibc's large-argument reduction path. - Large tensors pay the slow rate for most elements. -- **All changed tensors are re-initialized.** When `m_problemDependentData` is set, - `initializeCPUInputs` re-runs `initArray` for every tensor whose descriptor changed, not only - the Trig ones. -- **Worst case: `sk_bgemm_div`.** One 8192×57344×28672 BF16 problem with TrigSin A, TrigCos B and - TrigSin C. It initializes about 2.35G trig elements and copies 5.6 GB, to validate - `NumElementsToValidate: 128`; the CPU reference takes 13 ms. Instrumented breakdown of one run: - A 843 ms, B 4,105 ms, C 3,551 ms, D 23 ms, `copyInputs` 1,213 ms. - -Steady-state behaviour on a 4096×4096×8192 BF16→F32 problem (TrigSin A/B, 25 threads, 3 runs): - -| Call | Trig init A + B | Init C + D | `copyInputs` | Total | -| --- | --- | --- | --- | --- | -| 1st (fresh process) | 160–263 ms | 15–22 ms | 65–90 ms | 273–352 ms | -| Same problem again | 0 (skipped) | 0 | 9.5–9.7 ms | 9.5–9.8 ms | -| K changes (4096) | 54–67 ms | 0 | 9.5–9.9 ms | 64–77 ms | -| K changes back (8192) | 106–134 ms | 0 | 9.6–9.8 ms | 116–143 ms | -| 256×256×256 | 0.5–0.9 ms | — | 9.6–10.0 ms | 10.2–10.6 ms | - -### 3. `copyInputs` copies the largest size seen, every call - -- `copyInputBuffers` copies `p.maxElements` (the run's largest allocation), not the current - problem's size. In the table above, the 256×256×256 problem pays the same ~9.6 ms as the large - one; the instrumented log shows A copying 33,554,432 elements for a problem whose A has 65,536. -- The cheap branch in `prepareCPUInputs` that would only reset the output never runs: - `m_cpuInit` is initialized to `false` and only ever assigned `false`. -- A process's first copy is 4–9× slower (65–90 ms vs 9.6 ms) because the destination buffers are - touched for the first time. - -### 4. Fixed cost on each process's first call - -- In the sparse `spmm_i8` run, the first call spent 53.3 ms copying a 1,024-element buffer and - 18.0 ms initializing 1,024 elements. Every later call took 0.6–3 ms. -- Standalone, the first `hipMemcpy(H2H)` after HIP init costs about 18 ms even for 1 KB (later - calls ~0 ms), and the first OpenMP parallel region about 3 ms (0.2 ms later). The larger - in-client values were not explained further without a profiler. -- Across CI, the 752 first calls on small problems (<4M elements) total 17.6 s, median 20.6 ms. - This is why sparse configs show 50–70 ms for problems as small as 256×256×128. - -### 5. Thread scaling breaks down on a shared host - -The init loops use OpenMP's default static schedule, so one core shared with another process holds -up the whole loop. - -- Pure `sin` scales linearly to 8 threads (462 → 60 ms), then flattens: 39.8 ms at 16 threads, - 40.2 ms at 25. -- With `schedule(runtime)` on the same cores, 25 threads take 58 ms with `OMP_SCHEDULE=static` - and 21 ms with `dynamic,65536`. -- Identical `sk_bgemm_div` runs took 9.7, 15.0 and 8.4 s; B's init alone ranged from 3,665 to - 11,690 ms. - -## Caveats - -- HEAD's `DataInitialization` differs from the measured binary by about 300 lines (MX data - generator on/off). The trig loop, `copyInputs`/`maxElements`, `m_cpuInit` and - `getReferenceFloat` paths are the same shape in both, but absolute numbers on HEAD may differ. -- The ~50% MX waste figure comes from the standalone harness; `getReferenceFloat` was not timed - inside the client. -- Sparse configs were only checked for first-call overhead; the pruning code was not profiled. -- Machine contention adds large run-to-run variance; ranges above are across repeated runs. - -## Possible follow-ups (not implemented) - -- Stop building the discarded MX reference floats when the caller ignores them. -- Make trig init cheaper: no per-element allocation, flat index for default strides, `sinf` or a - bounded argument. -- Copy only the current problem's size in `copyInputs`, and fix or remove the dead `m_cpuInit` - path. -- Use a dynamic schedule (or larger chunks) for the init loops. diff --git a/projects/hipblaslt/tensilelite/timing_runs/cpu_reference_slow_path_investigation.md b/projects/hipblaslt/tensilelite/timing_runs/cpu_reference_slow_path_investigation.md deleted file mode 100644 index 1c9e5f31ab3e..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/cpu_reference_slow_path_investigation.md +++ /dev/null @@ -1,442 +0,0 @@ -# Slow CPU reference GEMM path investigation (gfx950 per-config rerun 2) - -`full_runs_gfx950/per_config_rerun_2/AGGREGATE_analysis.txt` reports **Reference GEMM (CPU)** at -1,080,469 ms (42.1% of wall). This is the largest category. The client's reference GEMM -(`SolveGemmCPU` in `client/src/Reference.cpp`) has two implementations: - -- a tiled, vectorised **fast path** (`solveCPUFast`), used when `isFastPathEligible()` accepts the - problem; -- a scalar **slow path** (`ReferenceSolution<…>::SolveCPU`) for everything else. - -The aggregate already splits the two with the `solve_cpu_fast*` and `solve_cpu_slow*` timers: - -| Timer | Calls | Total (ms) | Mean (ms) | -| --- | --- | --- | --- | -| `solve_cpu_slow` | 1,947 | 727,756.12 | 373.78 | -| `solve_cpu_slow_per_solution_inputs_changed` | 19 | 13,166.07 | 692.95 | -| **Slow path, total** | **1,966** | **740,922** | | -| `solve_cpu_fast` | 26,656 | 200,931.33 | 7.54 | -| `solve_cpu_fast_per_solution_inputs_changed` | 1,127 | 138,501.63 | 122.89 | -| **Fast path, total** | **27,783** | **339,433** | | - -The slow path is 6.6% of reference calls but 68.6% of reference time. This note covers: - -- which YAMLs take the slow path; -- why each problem is rejected from the fast path; -- why the slow path is so slow. - -Timing claims come from measurements; code reading was only used to decide what to measure. -Inferences are marked as such. - -## Summary - -1. **Two YAMLs are 81% of the slow path.** `streamk/sk_I8gemm_quick.yaml` (310.7 s) and - `streamk/sk_f8gemm_quick.yaml` (289.4 s) together take 600 s of the 741 s. All of it is their - `4111×4111×4111` problems. The fast path rejects them on type: - - Int8 inputs (all four output types: I8, BF16, Half, I32); - - Float8 output (`F8 × F8 → F8`). - - In the same YAML, the same-size problems with a Float, Half or BF16 output take the fast path in - 0.47–1.21 s. The slow-path calls each took 45–84 s. -2. **The slow path is serialised by a lock taken once per output element.** After computing each - output element, every thread calls `omp_set_num_threads(64)` and then enters a global - `#pragma omp critical`. The critical section exists only for the AMAX-D reduction, but it is - entered even when `outputAmaxD()` is false; the check is inside it. - - **Stack samples:** 75 of the roughly 100 thread samples taken during a slow-path GEMM were - waiting in `__kmp_acquire_queuing_lock`. - - **Disassembly:** the per-element call sequence is - `omp_set_num_threads` → `__kmpc_critical` → `outputAmaxD` test → `__kmpc_end_critical`, on - the path every element takes. - - **Scaling:** with 25 threads on 25 cores, a 1024³ fp32 GEMM is barely faster than with one - thread (1,727–2,364 ms against 2,626–2,660 ms). -3. **Entering the critical section only when AMAX-D is requested gives about a 4.6× speed-up** - (prototype `refpath3`, scratch only). - - **Driver, 2048³ fp32:** 45,237 ms → 1,507 ms. Other unmodified runs of the same problem took - 6.8–14 s; the contended lock makes the baseline erratic. - - **Driver, 1024³ fp32:** 1,727 ms → 191 ms. - - **Replay of all 59 slow-path YAMLs:** slow-path total 546.6 s → 119.5 s. Every config still - passes, and the driver's results are bit-identical (max diff 0). -4. **Even after that fix, the slow path is about 25–30× slower than the fast path on the same - problem.** At 4111³ on 25 cores, the guarded slow path took 16.3 s (fp32) and 17.5 s (f8 in). - The fast path took 0.65 s (f64), 0.76 s (f32) and 0.54 s (f8 in). - - The slow path computes each output element as a separate scalar dot product. In the I8 K loop, - each MAC is 2 sign-extending loads, an `imul` and an `add`, with no vectorisation or blocking. - Adding fast-path support for **Int8 inputs** and **Float8 outputs** would move 79% of the - remaining (post-fix) slow-path time to the fast path. -5. **The other rejection reasons are small.** After the fix, measured in the replay: - - `gradient`: 8.8 s - - `useE`: 7.7 s - - complex types: 4.1 s - - MX K not a multiple of the MX block: 2.6 s - - `NumElementsToValidate: 128` (sparse validation, intended): 1.7 s - -## Method - -- **Production numbers:** the per-config logs in - `/home/alvasile/timing_runs/full_runs_gfx950/per_config_rerun_2//_analysis.txt`. For - each config, the `solve_cpu_slow*` and `solve_cpu_fast*` rows were summed from the first - "TIMING BY PHASE" table (`/tmp/refpath/percfg.py` → `/tmp/refpath/percfg_prod.txt`). - 59 configs have at least one slow-path call. -- **Instrumented client (`refpath`):** - - A copy of the current `client/` sources in `/tmp/refpath/client_src`, built with the - production flags by `/tmp/bookkeep/build.sh`. - - `isFastPathEligible()` already has a `rejectFast(reason)` hook (disabled with `if (false)`). - The copy stores the reason instead, and `SolveGemmCPU` prints one `REFPATH` line per call: the - path, the reason, M/N/K/batch, the A/B/C/D types, `elementsToValidate`, and the elapsed ms. - - Reasons are reported in the order `SolveGemmCPU` checks them: `tryFastPath`, then the 20% - validation-density rule, then `isFastPathEligible()`. -- **Replays:** - - Each of the 59 configs was rerun through pytest with the production artifacts: - `--use-cache --artifact-dir` holding a symlink to `/tmp/tl_split_gfx950/artifacts/.tar.gz`, - and `--prebuilt-client`. - - Same environment as the production rerun: `OMP_NUM_THREADS=25`, 25 pinned idle cores, default - `KMP_BLOCKTIME`, `TimingInstrumentation=True`, msgpack library format. - - **Baseline (`refpath`):** cores `28-39,42-46,49-53,55-57`, GPU 4. Logs and per-call `REFPATH` - lines are in `/tmp/refpath/logs/`. - - **Guarded critical section (`refpath3`):** cores `64-88`, GPU 5. Results are in - `/tmp/refpath/logs3/`. - - The baseline replay has the same 1,966 slow-path calls as production, problem for problem. - Totals differ (546.6 s against 740.9 s) because the contended lock makes run times erratic; - see finding 3. - - Only host-side timers are compared. The GPUs are shared with other users. -- **Same-problem comparisons:** - - The `cpu-gemm-driver` object from the production build was relinked against the instrumented - client library: `/tmp/refpath/cpu-gemm-driver`, `-driver2` and `-driver3`. - - `-driver2` and `-driver3` read the reference's thread cap from `TL_MAXT`. Production hard-codes - `MAX_OMP_THREADS 64`. - - Runs used cores `192-216`, whose SMT siblings were idle (2% busy), with `KMP_BLOCKTIME=0` - unless stated. The timing is the `REFPATH` value for the reference call only; the driver's - validation step is excluded. -- **Stack sampling:** `perf` is blocked and `ptrace_scope=1` (no attaching), so the driver was - launched under `rocgdb -batch`. Four `SIGINT`s, 1.5 s apart, each triggered - `thread apply all bt 4`. -- **Assembly:** `objdump -dr` of `Reference.cpp.o` from the production-flag build, on the - `.omp_outlined` body of - `ReferenceSolution,int,int>::SolveCPU` (the I8 - case from `sk_I8gemm_quick`). - -## Which YAMLs take the slow path, and why - -### By rejection reason - -| Reason | Calls | Replay (s) | Replay, guarded critical (s) | -| --- | --- | --- | --- | -| Type: Float8 × Float8 → **Float8** | 6 | 172.4 | 44.1 | -| Type: **Int8** × Int8 → Int8 | 714 | 92.6 | 15.1 | -| Type: **Int8** × Int8 → BFloat16 | 99 | 79.1 | 12.7 | -| Type: **Int8** × Int8 → Int32 | 39 | 70.2 | 10.2 | -| Type: **Int8** × Int8 → Half | 99 | 51.8 | 11.0 | -| `useE` | 18 | 20.1 | 7.7 | -| `gradient` | 47 | 19.9 | 8.8 | -| `K_not_multiple_of_mxBlockA` (MXFP4, K = 333/255/63, block 32) | 36 | 19.8 | 2.6 | -| Type: ComplexFloat | 251 | 9.2 | 2.5 | -| Type: Float8 × **Half** → **Float8** | 24 | 4.6 | 1.5 | -| Type: ComplexDouble | 64 | 4.4 | 1.6 | -| `NumElementsToValidate: 128` (fewer than 20% of D validated) | 569 | 2.4 | 1.7 | -| **Total** | **1,966** | **546.6** | **119.5** | - -Notes on the reasons: - -- **Type rejections:** the fast path accepts these types: - - inputs: Float, Double, Half, BFloat16, the four FP8/BF8 variants, and Float4; - - outputs (C and D): Float, Double, Half, BFloat16. - - Int8 inputs, Int8/Int32/FP8 outputs and complex types fall through. -- **`NumElementsToValidate: 128`:** this is the intended behaviour. The slow path then computes - only every `NextPrime(total/128)`-th element, so these 569 calls cost 2.4 s in total. - `elementsToValidate` arrives as `size_t`, so the YAML default `-1` counts as "validate - everything" and does not trigger this rule. -- **No other reasons occurred** for this set: no `amaxD`, `scaleCD`, `layout`, `multi_index`, - `mirror_indices`, `no_batch_indices` or `tryFastPath=false` rejections. - -### By YAML (all 59) - -"Prod" is the production per-config log. "Replay" is the baseline instrumented replay. -"Guarded critical" is the `refpath3` replay; see Option A. - -| YAML (`Tests/common/…`) | Slow calls | Reason(s) (calls) | Prod (s) | Replay (s) | Replay, guarded critical (s) | -| --- | --- | --- | --- | --- | --- | -| `streamk/sk_I8gemm_quick.yaml` | 12 | type A=Int8/B=Int8→D=Int8 (3); type A=Int8/B=Int8→D=BFloat16 (3); type A=Int8/B=Int8→D=Half (3); type A=Int8/B=Int8→D=Int32 (3) | 310.66 | 252.50 | 43.20 | -| `streamk/sk_f8gemm_quick.yaml` | 6 | type A=Float8/B=Float8→D=Float8 (6) | 289.45 | 172.42 | 44.06 | -| `gemm/gfx950/mxfp4_mxfp4_bf16_tn_act.yaml` | 36 | K_not_multiple_of_mxBlockA (36) | 25.26 | 19.84 | 2.64 | -| `streamk/gfx950/sk_hgemm_use_e.yaml` | 11 | useE (8); gradient (3) | 22.87 | 17.01 | 7.36 | -| `streamk/gfx950/data_parallel_static_hgemm_use_e.yaml` | 8 | useE (6); gradient (2) | 14.51 | 13.04 | 5.70 | -| `sparse/gfx950/spmm_i8_sb.yaml` | 196 | type A=Int8/B=Int8→D=Int8 (196) | 6.92 | 5.94 | 1.12 | -| `sparse/gfx950/spmm_i8.yaml` | 196 | type A=Int8/B=Int8→D=Int8 (196) | 6.66 | 6.03 | 1.10 | -| `gemm/gfx950/f8f16mix_f8s.yaml` | 24 | type A=Float8/B=Half→D=Float8 (24) | 5.80 | 4.58 | 1.53 | -| `gemm/bufferLoadStore0.yaml` | 189 | type A=ComplexFloat/B=ComplexFloat→D=ComplexFloat (189) | 5.01 | 3.96 | 1.03 | -| `gradient/fp16_gradient_bias.yaml` | 21 | gradient (21) | 4.64 | 5.21 | 2.02 | -| `gemm/cgemm.yaml` | 54 | type A=ComplexFloat/B=ComplexFloat→D=ComplexFloat (54) | 4.47 | 3.94 | 0.62 | -| `gradient/gfx950/bf16_gradient_bias.yaml` | 21 | gradient (21) | 4.39 | 4.69 | 1.35 | -| `gemm/zgemm.yaml` | 54 | type A=ComplexDouble/B=ComplexDouble→D=ComplexDouble (54) | 4.05 | 3.67 | 0.59 | -| `sparse/gfx94x/spmm_i8_mi16_sb.yaml` | 40 | type A=Int8/B=Int8→D=Int8 (40) | 3.99 | 2.22 | 0.19 | -| `sparse/gfx950/spmm_dtl.yaml` | 16 | type A=Int8/B=Int8→D=Int8 (16) | 3.23 | 1.90 | 0.55 | -| `sparse/gfx94x/spmm_i8_mi32_sb.yaml` | 40 | type A=Int8/B=Int8→D=Int8 (40) | 2.96 | 2.22 | 0.24 | -| `sparse/gfx950/spmm_i8hs.yaml` | 64 | type A=Int8/B=Int8→D=Half (64) | 2.75 | 2.70 | 0.28 | -| `sparse/gfx94x/spmm_i8_mi32.yaml` | 40 | type A=Int8/B=Int8→D=Int8 (40) | 2.72 | 2.94 | 0.22 | -| `sparse/gfx94x/spmm_i8_mi16.yaml` | 40 | type A=Int8/B=Int8→D=Int8 (40) | 2.43 | 2.11 | 0.27 | -| `sparse/gfx950/spmm_i8bs.yaml` | 64 | type A=Int8/B=Int8→D=BFloat16 (64) | 2.06 | 2.56 | 0.45 | -| `streamk/sk_cgemm_quick.yaml` | 8 | type A=ComplexFloat/B=ComplexFloat→D=ComplexFloat (8) | 2.03 | 1.31 | 0.87 | -| `sparse/gfx94x/i8_activation.yaml` | 44 | type A=Int8/B=Int8→D=Int8 (44) | 1.96 | 2.01 | 0.16 | -| `gemm/gemm_bias_strided_batched_gfx940.yaml` | 24 | type A=Int8/B=Int8→D=Int8 (24) | 1.50 | 1.53 | 0.13 | -| `gemm/gemm_bias_strided_batched_broadcast_gfx940.yaml` | 24 | type A=Int8/B=Int8→D=Int8 (24) | 1.32 | 1.32 | 0.35 | -| `sparse/gfx94x/spmm_i8bs_sb.yaml` | 16 | type A=Int8/B=Int8→D=BFloat16 (16) | 1.06 | 0.70 | 0.05 | -| `sparse/gfx94x/spmm_i8hs.yaml` | 16 | type A=Int8/B=Int8→D=Half (16) | 0.89 | 1.24 | 0.07 | -| `sparse/gfx94x/spmm_i8is.yaml` | 16 | type A=Int8/B=Int8→D=Int32 (16) | 0.86 | 0.84 | 0.02 | -| `sparse/gfx94x/spmm_i8is_sb.yaml` | 16 | type A=Int8/B=Int8→D=Int32 (16) | 0.72 | 0.75 | 0.07 | -| `sparse/gfx94x/spmm_i8bs.yaml` | 16 | type A=Int8/B=Int8→D=BFloat16 (16) | 0.71 | 0.82 | 0.04 | -| `gemm/gfx950/i8_gsu_gfx950.yaml` | 4 | type A=Int8/B=Int8→D=Int8 (2); type A=Int8/B=Int8→D=Int32 (2) | 0.71 | 0.97 | 0.06 | -| `gemm/i8_gsu_gfx940.yaml` | 4 | type A=Int8/B=Int8→D=Int8 (2); type A=Int8/B=Int8→D=Int32 (2) | 0.70 | 0.79 | 0.13 | -| `sparse/gfx94x/spmm_i8hs_sb.yaml` | 16 | type A=Int8/B=Int8→D=Half (16) | 0.69 | 1.04 | 0.07 | -| `streamk/sk_zgemm_quick.yaml` | 10 | type A=ComplexDouble/B=ComplexDouble→D=ComplexDouble (10) | 0.66 | 0.75 | 1.02 | -| `gemm/fp16_tn.yaml` | 166 | NumElementsToValidate=128 (166) | 0.38 | 0.19 | 0.22 | -| `gemm/gfx950/fp16_use_e_wide_store.yaml` | 4 | useE (4) | 0.32 | 0.12 | 0.02 | -| `gemm/ag.yaml` | 180 | NumElementsToValidate=128 (180) | 0.31 | 1.13 | 0.46 | -| `gemm/gfx950/lds_tr.yaml` | 78 | NumElementsToValidate=128 (78) | 0.29 | 0.38 | 0.43 | -| `gemm/lsu_i8.yaml` | 15 | type A=Int8/B=Int8→D=Int8 (15) | 0.27 | 0.43 | 0.14 | -| `sparse/gfx94x/spmm_vw_lg_one.yaml` | 8 | type A=Int8/B=Int8→D=Int8 (8) | 0.07 | 0.04 | 0.02 | -| `gemm/use_beta_false.yaml` | 6 | NumElementsToValidate=128 (6) | 0.07 | 0.02 | 0.02 | -| `gradient/fp16_gradient_activation.yaml` | 8 | NumElementsToValidate=128 (8) | 0.07 | 0.04 | 0.06 | -| `gemm/fp16fp32mix.yaml` | 8 | NumElementsToValidate=128 (8) | 0.06 | 0.05 | 0.06 | -| `sparse/gfx950/spmm_vw_lg_one.yaml` | 8 | type A=Int8/B=Int8→D=Int8 (8) | 0.06 | 0.05 | 0.06 | -| `gemm/bf16_tn.yaml` | 27 | NumElementsToValidate=128 (27) | 0.06 | 0.02 | 0.07 | -| `gemm/fp16_use_e.yaml` | 18 | NumElementsToValidate=128 (18) | 0.06 | 0.40 | 0.13 | -| `sparse/gfx94x/spmm_vw_lg_one_sb.yaml` | 8 | type A=Int8/B=Int8→D=Int8 (8) | 0.06 | 0.02 | 0.02 | -| `sparse/gfx950/spmm_vw_lg_one_sb.yaml` | 8 | type A=Int8/B=Int8→D=Int8 (8) | 0.06 | 0.03 | 0.02 | -| `gemm/fp16_HH_BHS_bf16mfma.yaml` | 27 | NumElementsToValidate=128 (27) | 0.04 | 0.03 | 0.03 | -| `ductile/gfx950/f8_tn_mx_subtile.yaml` | 2 | NumElementsToValidate=128 (2) | 0.03 | 0.01 | 0.01 | -| `groupedgemm/grouped_gemm.yaml` | 8 | NumElementsToValidate=128 (8) | 0.03 | 0.02 | 0.03 | -| `gemm/fp32_tn.yaml` | 27 | NumElementsToValidate=128 (27) | 0.03 | 0.03 | 0.08 | -| `streamk/sk_bgemm_div.yaml` | 1 | NumElementsToValidate=128 (1) | 0.01 | 0.01 | 0.02 | -| `exception/f16_cf32_nan.yaml` | 3 | NumElementsToValidate=128 (3) | 0.01 | 0.02 | 0.05 | -| `benchmark_runs/fp32_tn.yaml` | 3 | NumElementsToValidate=128 (3) | 0.01 | 0.00 | 0.01 | -| `gemm/gfx950/bf16_cvt.yaml` | 1 | NumElementsToValidate=128 (1) | 0.01 | 0.00 | 0.00 | -| `gemm/large_size.yaml` | 1 | NumElementsToValidate=128 (1) | 0.01 | 0.01 | 0.00 | -| `gemm/fp32_nt.yaml` | 1 | NumElementsToValidate=128 (1) | 0.00 | 0.00 | 0.00 | -| `gemm/nan_bounds_check_odd_padding.yaml` | 3 | NumElementsToValidate=128 (3) | 0.00 | 0.00 | 0.00 | -| `gemm/bf16_tn_predict.yaml` | 1 | NumElementsToValidate=128 (1) | 0.00 | 0.01 | 0.00 | -| **Total** | 1966 | | **740.9** | **546.6** | **119.5** | - -Rows below about 0.1 s are within run-to-run noise. Some small rows read slightly higher with -the guarded critical section for that reason. - -### The two dominant YAMLs, per call (4111³, one call each) - -| YAML | A × B → D | Path | Replay (ms) | Guarded critical (ms) | -| --- | --- | --- | --- | --- | -| `sk_I8gemm_quick` | I8 × I8 → I8 | slow (type) | 60,465 | 9,985 | -| `sk_I8gemm_quick` | I8 × I8 → BF16 | slow (type) | 72,756 | 11,626 | -| `sk_I8gemm_quick` | I8 × I8 → Half | slow (type) | 45,210 | 9,998 | -| `sk_I8gemm_quick` | I8 × I8 → I32 | slow (type) | 66,115 | 9,491 | -| `sk_f8gemm_quick` | F8 × F8 → **F8** (2 calls) | slow (type) | 84,331 / 83,567 | 19,464 / 23,092 | -| `sk_f8gemm_quick` | F8 × F8 → Float (2 calls) | fast | — | 702 / 1,206 | -| `sk_f8gemm_quick` | F8 × F8 → Half | fast | — | 655 | -| `sk_f8gemm_quick` | BF8 × BF8 → Float | fast | — | 636 | -| `sk_f8gemm_quick` | F8 × Half → Float | fast | — | 655 | -| `sk_f8gemm_quick` | BF8 × F8 → BF16 | fast | — | 469 | - -In `sk_f8gemm_quick`, the only problem type that takes the slow path is the one whose output is -Float8. Its 4111³ problem is 28× (guarded) to 180× (unguarded) slower than its fast-path siblings -in the same YAML. - -## Why the slow path is slow - -### Measured: it does not scale with threads - -`cpu-gemm-driver`, fp32, `--transA`, slow path, cores `192-216` (25 cores, SMT siblings idle). -The fast path is shown for scale. - -| Threads (cores) | Slow path, 1024³ (ms) | Fast path, 1024³ (ms) | -| --- | --- | --- | -| 1 (1 core) | 2,642 | 99 | -| 25 (25 cores), `MAX_OMP_THREADS=25` | 2,329 | 21 | -| 64 on 25 cores (production cap) | 2,364 | 25 | -| 64 (64 cores) | 1,492 | 14 | - -Going from 1 to 25 threads speeds the slow path up by only 1.1×. A thread-state trace (every -0.25 s from `/proc//task/*/status`) showed all 25 threads running for the whole reference -call, so the threads are busy, not idle. - -### Measured: the threads are waiting on one lock - -`rocgdb` stack samples were taken during a 2048³ fp32 slow-path call, 25 threads, 4 samples: - -| Top of stack | Thread samples | -| --- | --- | -| `__kmpc_critical_with_hint` → `__kmp_acquire_queuing_lock` → `__kmp_wait_4` | 54 | -| … → `__kmp_wait_4` → `sched_yield` (same lock, yielding) | 21 | -| `__kmp_suspend_64` → `pthread_cond_wait` (idle pool threads) | 24 | -| Main thread in `runGemm` | 1 | - -### The code and its assembly - -`Reference.cpp`, at the end of the per-output-element body of the `#pragma omp parallel` / -`#pragma omp for` loop in `ReferenceSolution<…>::SolveCPU`: - -```cpp - omp_set_num_threads(MAX_OMP_THREADS); -#pragma omp critical - { - if constexpr(notCmplxAmaxD) - { - if(problem.outputAmaxD()) - { /* amaxD = max(amaxD, |resultD|) */ } - } - } -``` - -The I8 instantiation compiles to the following, executed once per output element (`.omp_outlined` -body, `objdump -dr`): - -``` -62c7a: mov $0x40,%edi -62c7f: call omp_set_num_threads -62c94: call __kmpc_critical ; .gomp_critical_user_.var (one global lock) -62ca1: cmpb $0x1,0x134(%rax) ; problem.outputAmaxD() — tested inside the lock -62ca8: jne 62cd2 -... -62ce2: call __kmpc_end_critical -``` - -Just before this, the same body calls `operator delete`. That is the per-element -`std::vector actArgs` built for `Activation()`. Its cost was not measured -separately. - -The K loop itself is cheap per instruction. The I8 version (`61da0`–`61dcf`) is unrolled by 2 and -does two `movsbl` loads, an `imul` and an `add` per MAC, stepping A and B by their K strides. With -the lock removed, single-thread throughput is about 0.41 GMAC/s against the fast path's -10.8 GMAC/s (1024³: 2,625 ms against 99 ms). So even an uncontended slow path is about 25× slower -per core. - -### Memory layout is a secondary effect - -fp32 and f8 inputs, 2048³, slow path, unmodified (64 threads on 25 cores): - -| Layout | fp32 (ms) | f8 in (ms) | -| --- | --- | --- | -| NN | 13,900 | 16,704 | -| TN (A contiguous in K) | 11,273 | 12,453 | -| NT (B strided in K) | 20,152 | 18,856 | - -The spread is about 1.5–1.8×. That is small next to the lock (a 9–30× change), so layout was not -pursued further. - -## Options - -### A. Take the critical section only when AMAX-D is requested (measured, prototype `refpath3`) - -**Change.** Move `#pragma omp critical` inside `if(problem.outputAmaxD())`, and drop the -per-element `omp_set_num_threads`. The whole diff is in `/tmp/refpath/client_src3/src/Reference.cpp`. - -**Further improvement (not measured).** AMAX-D problems would still serialise. They should use a -per-thread maximum combined once, e.g. `reduction(max:amaxD)`. No gfx950 config in this set hit -the slow path with AMAX-D. - -| Measurement | Unmodified | Guarded | -| --- | --- | --- | -| Driver 1024³ fp32, 25 threads | 1,727 ms | 191 ms | -| Driver 2048³ fp32, 25 threads | 45,237 ms (other runs: 6.8–14 s) | 1,507 ms | -| Driver 1024³ fp32, 1 thread | 2,660 ms | 2,625 ms | -| Driver results | PASSED, max diff 0 | PASSED, max diff 0 | -| Replay, slow-path total (59 YAMLs) | 546.6 s | 119.5 s (−78%) | -| Replay, `sk_I8gemm_quick` pytest wall | 258 s | 49 s | -| Replay, `sk_f8gemm_quick` pytest wall | 184 s | 56 s | -| Replay, configs passing | 59 / 59 | 59 / 59 | - -**Estimated effect on the aggregate.** Applying the replay ratio (119.5 / 546.6) to production's -740.9 s gives about 162 s, saving about 580 s, or 22% of the aggregate wall. This is an -estimate; the production baseline itself varies, as shown above. - -**The 64-thread cap.** Once the lock is gone, the hard-coded cap no longer matters on 25 cores. -At 2048³ it measured 1,392 / 1,388 ms with the 64 cap against 1,437 / 1,373 ms with 25 threads. -`omp_set_num_threads` persists for the calling thread, so the cap also changes later OpenMP -regions; using `omp_get_max_threads()` would make it follow `OMP_NUM_THREADS`. - -**`KMP_BLOCKTIME`.** With the default (200 ms), the guarded version measured 2,010–2,165 ms -against 1,388–1,437 ms with `KMP_BLOCKTIME=0`. This matches the -[per-solution bookkeeping](per_solution_bookkeeping_investigation.md) finding that -`KMP_BLOCKTIME=0` cut `cpu_reference_gemm` by 2–5×. - -### B. Fast-path support for Int8 inputs and Float8 outputs (not prototyped) - -These two type gaps are 94.6 s of the 119.5 s left after Option A (79%), and nearly all of the -production slow path. Same-size fast-path calls take 0.47–1.21 s, against 9.5–23 s guarded and -45–84 s unguarded. - -For reference, driver measurements of the fast path at 4111³ on 25 cores: - -| Inputs | Fast path, 4111³ (ms) | -| --- | --- | -| f64 | 651 | -| f32 | 761 | -| f8 | 541 | - -What each extension needs: - -- **Int8 inputs.** The accumulation must be exact. A float accumulator is not: a 4111-long I8 dot - product can reach about 6.7e7, above 2²⁴. Options are the existing `double` instantiation - (exact below 2⁵³; 651 ms at 4111³) or a dedicated int32/int64 inner kernel. - - Int8 and Int32 outputs also need `storeFrom` specialisations that use the slow path's - `SaturateCast` semantics. - - Half and BF16 outputs already have write-back. -- **Float8 output.** A `storeFrom` (and BF8) write-back that matches the slow - path's `SaturateCast`. - - The F8 × Half → F8 case (`f8f16mix_f8s`) also needs the mixed-input shadow; the fast path - already handles mixed F8/Half inputs (`F8 × Half → Float` is fast in `sk_f8gemm_quick`). - -**Estimate.** If the 4111³ calls cost what their fast-path siblings do, these YAMLs' reference -time would drop from about 600 s in production to about 10 s. This is an inference from the -sibling measurements; equivalence with the slow path has to be validated per type, as it was when -FP8/FP4 were added to the fast path. - -### C. Smaller rejection reasons (not prototyped) - -Times are after Option A, from the replay. - -| Reason | Remaining (s) | What it would take | -| --- | --- | --- | -| `gradient` | 8.8 | Gradient GEMM with bias reduction. The slow path also runs a separate reduction region. Larger change. | -| `useE` | 7.7 | Store the pre-activation value to E in the fast-path epilogue. | -| `K_not_multiple_of_mxBlockA` | 2.6 | MXFP4 K tails (K = 333, 255, 63 with block 32). Handle a partial last MX block. | -| Complex types | 4.1 | New fast-path variant. Probably not worth it at this size. | -| `NumElementsToValidate: 128` | 1.7 | Intended (sparse validation). Leave as is. | - -### D. Make the rejection reason observable - -The `rejectFast()` logging is compiled out with `if (false)`, so this audit needed a rebuilt -client. Two cheap ways to keep it reproducible: - -- an environment or log-level switch for the reason line; -- a per-reason timer category, e.g. `solve_cpu_slow.`, so the aggregate shows it directly. - -## Not measured / open items - -- **The fast path's own 339 s.** It is out of scope here; most of it is the - `subtile_mxfp8*` YAMLs (`gemm/gfx950/subtile_mxfp8.yaml` 143 s and the - `*_correctness_regression*` variants). -- **The per-element `actArgs` vector** allocated and freed in the slow path is visible in the - assembly but was not timed separately. -- **AMAX-D problems** would still take the lock per element after Option A; use a reduction. -- **`ductile/gfx950/f8_tn_mx_subtile.yaml`** reaches the reference with `elementsToValidate=128`, - although the YAML sets `NumElementsToValidate: 0`. The cost is negligible (0.03 s); the cause - was not investigated. -- **Run-to-run variance.** Production slow-path time (740.9 s) and the baseline replay (546.6 s) - differ, because lock contention makes the unmodified slow path erratic; the same 2048³ problem - measured 6.8 s to 45 s. The guarded version was stable (1.37–1.51 s). - -## Scratch files - -All in `/tmp/refpath/` (nothing in the repo was changed): - -- **Sources:** - - `client_src/`: `REFPATH` logging. - - `client_src2/`: adds the `TL_MAXT` thread cap. - - `client_src3/`: adds the guarded critical section. -- **Builds:** `/tmp/bookkeep/build_refpath{,2,3}/`, plus the relinked `cpu-gemm-driver{,2,3}`. -- **Replays:** - - Runners: `run.sh` and `run3.sh`; config list `cfgs.txt`. - - Per-config `.refpath` lines and analyses: `logs/`, `logs3/`. - - Summaries: `summ.py` → `summ_base.txt`, `summ_fix.txt`; YAML table `table.md`. -- **Driver experiments:** `layout.out`, `threads.out`, `ab.sh`, and `gdb.out` (stack samples). -- **Assembly:** `i8_outlined_r.s`. diff --git a/projects/hipblaslt/tensilelite/timing_runs/driver.log b/projects/hipblaslt/tensilelite/timing_runs/driver.log deleted file mode 100644 index e69de29bb2d1..000000000000 diff --git a/projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_1_build_cold_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_1_build_cold_analysis.txt deleted file mode 100644 index 8b297fc49bdc..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_1_build_cold_analysis.txt +++ /dev/null @@ -1,135 +0,0 @@ -======================================================================================================================== -TIMING ANALYSIS SUMMARY -======================================================================================================================== - -TIMING BY PHASE ------------------------------------------------------------------------------------------------------------------------- - Category Count Total (ms) Mean (ms) % of Parent % of Wall ------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 1 223,910.86 223,910.86 100.0% - -------------------------------------------------------------------------------------------------------------------- - python_kernel_compilation 25 190,938.97 7,637.56 85.3% 85.3% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_codegen 25 59,556.26 2,382.25 31.2% 26.6% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_build_src_co 25 46,690.02 1,867.60 24.5% 20.9% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_kernel_build_src_co.compile 9 46,538.74 5,170.97 99.7% 20.8% - python_kernel_build_src_co.unbundle 9 82.14 9.13 0.2% 0.0% - python_kernel_build_src_co.cache_check 25 53.03 2.12 0.1% 0.0% - python_kernel_build_src_co.cache_populate 9 8.75 0.97 0.0% 0.0% - (untracked) 5.37 0.0% 0.0% - python_kernel_build_src_co.setup 25 1.21 0.05 0.0% 0.0% - python_kernel_build_src_co.move 9 0.79 0.09 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_bench_setup 25 30,494.49 1,219.78 16.0% 13.6% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_write_assemble 25 20,585.84 823.43 10.8% 9.2% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_bench_postprocess 25 16,598.97 663.96 8.7% 7.4% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_benchpost_lib_construction 25 10,985.80 439.43 66.2% 4.9% - python_benchpost_library_write 25 4,023.58 160.94 24.2% 1.8% - python_benchpost_naming 25 1,582.13 63.29 9.5% 0.7% - python_benchpost_client_config 25 5.83 0.23 0.0% 0.0% - (untracked) 1.63 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_validate 25 12,878.83 515.15 6.7% 5.8% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_build_co 25 1,691.70 67.67 0.9% 0.8% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_setup 25 1,433.88 57.36 0.8% 0.6% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 680.64 0.4% 0.3% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_write_helpers 25 328.35 13.13 0.2% 0.1% - -------------------------------------------------------------------------------------------------------------------- - python_solution_generation 25 26,205.76 1,048.23 11.7% 11.7% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_forked_solutions 25 26,204.81 1,048.19 100.0% 11.7% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 0.67 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_fork_permutations 25 0.24 0.01 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_custom_kernels 25 0.04 0.00 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - python_write_solutions 25 2,811.70 112.47 1.3% 1.3% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_dump 25 2,686.05 107.44 95.5% 1.2% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_prepare 25 70.57 2.82 2.5% 0.0% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_wsol_prepare_nocache 25 70.50 2.82 99.9% 0.0% - (untracked) 0.07 0.1% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 54.60 1.9% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_header 25 0.49 0.02 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - (untracked) 2,396.58 1.1% 1.1% - -------------------------------------------------------------------------------------------------------------------- - python_solution_indexing 25 1,525.74 61.03 0.7% 0.7% - -------------------------------------------------------------------------------------------------------------------- - python_write_cache 25 32.07 1.28 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - python_cache_check 25 0.03 0.00 0.0% 0.0% - ------------------------------------------------------------------------------------------------------------------------- - ESTIMATED WALL CLOCK TIME: 223.91s - -HIERARCHY VALIDATION: OK (all child phases fit within their parents) - -TIME BREAKDOWN (visual, % of wall clock) ------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 100.0% ################################################## - python_kernel_compilation 85.3% ########################################## - python_kernel_codegen 26.6% ############# - python_kernel_build_src_co 20.9% ########## - python_kernel_build_src_co.compile 20.8% ########## - python_kernel_build_src_co.unbundle 0.0% - python_kernel_build_src_co.cache_check 0.0% - python_kernel_build_src_co.cache_populate 0.0% - (untracked) 0.0% - python_kernel_build_src_co.setup 0.0% - python_kernel_build_src_co.move 0.0% - python_kernel_bench_setup 13.6% ###### - python_kernel_write_assemble 9.2% #### - python_kernel_bench_postprocess 7.4% ### - python_benchpost_lib_construction 4.9% ## - python_benchpost_library_write 1.8% - python_benchpost_naming 0.7% - python_benchpost_client_config 0.0% - (untracked) 0.0% - python_kernel_validate 5.8% ## - python_kernel_build_co 0.8% - python_kernel_setup 0.6% - (untracked) 0.3% - python_kernel_write_helpers 0.1% - python_solution_generation 11.7% ##### - python_solgen_forked_solutions 11.7% ##### - (untracked) 0.0% - python_solgen_fork_permutations 0.0% - python_solgen_custom_kernels 0.0% - python_write_solutions 1.3% - python_wsol_dump 1.2% - python_wsol_prepare 0.0% - python_wsol_prepare_nocache 0.0% - (untracked) 0.0% - (untracked) 0.0% - python_wsol_header 0.0% - (untracked) 1.1% - python_solution_indexing 0.7% - python_write_cache 0.0% - python_cache_check 0.0% - -KEY INSIGHTS (% of wall clock) ------------------------------------------------------------------------------------------------------------------------- - Python phases (code gen, compilation): 217.14s (97.0%) - Python I/O overhead (cache, YAML): 2.84s (1.3%) - C++ client setup (library loading): 0.00s (0.0%) - Data preparation (CPU init, GPU prep): 0.00s (0.0%) - Reference computation (CPU GEMM): 0.00s (0.0%) - Kernel execution (warmup, benchmark): 0.00s (0.0%) - - diff --git a/projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_2_build_warm_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_2_build_warm_analysis.txt deleted file mode 100644 index 36c62fca0436..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_2_build_warm_analysis.txt +++ /dev/null @@ -1,127 +0,0 @@ -======================================================================================================================== -TIMING ANALYSIS SUMMARY -======================================================================================================================== - -TIMING BY PHASE ------------------------------------------------------------------------------------------------------------------------- - Category Count Total (ms) Mean (ms) % of Parent % of Wall ------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 1 178,518.51 178,518.51 100.0% - -------------------------------------------------------------------------------------------------------------------- - python_kernel_compilation 25 145,363.45 5,814.54 81.4% 81.4% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_codegen 25 59,610.54 2,384.42 41.0% 33.4% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_bench_setup 25 30,651.53 1,226.06 21.1% 17.2% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_write_assemble 25 20,640.99 825.64 14.2% 11.6% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_bench_postprocess 25 16,806.22 672.25 11.6% 9.4% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_benchpost_lib_construction 25 11,229.74 449.19 66.8% 6.3% - python_benchpost_library_write 25 4,026.50 161.06 24.0% 2.3% - python_benchpost_naming 25 1,542.74 61.71 9.2% 0.9% - python_benchpost_client_config 25 5.68 0.23 0.0% 0.0% - (untracked) 1.56 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_validate 25 13,391.89 535.68 9.2% 7.5% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_build_co 25 1,681.10 67.24 1.2% 0.9% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_setup 25 1,461.13 58.45 1.0% 0.8% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 744.42 0.5% 0.4% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_write_helpers 25 313.15 12.53 0.2% 0.2% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_build_src_co 25 62.48 2.50 0.0% 0.0% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_kernel_build_src_co.cache_check 25 54.79 2.19 87.7% 0.0% - (untracked) 6.38 10.2% 0.0% - python_kernel_build_src_co.setup 25 1.31 0.05 2.1% 0.0% - -------------------------------------------------------------------------------------------------------------------- - python_solution_generation 25 26,379.70 1,055.19 14.8% 14.8% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_forked_solutions 25 26,378.74 1,055.15 100.0% 14.8% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 0.66 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_fork_permutations 25 0.25 0.01 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_custom_kernels 25 0.04 0.00 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - python_write_solutions 25 2,825.55 113.02 1.6% 1.6% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_dump 25 2,681.36 107.25 94.9% 1.5% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_prepare 25 80.71 3.23 2.9% 0.0% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_wsol_prepare_nocache 25 80.63 3.23 99.9% 0.0% - (untracked) 0.07 0.1% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 62.97 2.2% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_header 25 0.50 0.02 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - (untracked) 2,392.22 1.3% 1.3% - -------------------------------------------------------------------------------------------------------------------- - python_solution_indexing 25 1,525.05 61.00 0.9% 0.9% - -------------------------------------------------------------------------------------------------------------------- - python_write_cache 25 32.52 1.30 0.0% 0.0% - -------------------------------------------------------------------------------------------------------------------- - python_cache_check 25 0.03 0.00 0.0% 0.0% - ------------------------------------------------------------------------------------------------------------------------- - ESTIMATED WALL CLOCK TIME: 178.52s - -HIERARCHY VALIDATION: OK (all child phases fit within their parents) - -TIME BREAKDOWN (visual, % of wall clock) ------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 100.0% ################################################## - python_kernel_compilation 81.4% ######################################## - python_kernel_codegen 33.4% ################ - python_kernel_bench_setup 17.2% ######## - python_kernel_write_assemble 11.6% ##### - python_kernel_bench_postprocess 9.4% #### - python_benchpost_lib_construction 6.3% ### - python_benchpost_library_write 2.3% # - python_benchpost_naming 0.9% - python_benchpost_client_config 0.0% - (untracked) 0.0% - python_kernel_validate 7.5% ### - python_kernel_build_co 0.9% - python_kernel_setup 0.8% - (untracked) 0.4% - python_kernel_write_helpers 0.2% - python_kernel_build_src_co 0.0% - python_kernel_build_src_co.cache_check 0.0% - (untracked) 0.0% - python_kernel_build_src_co.setup 0.0% - python_solution_generation 14.8% ####### - python_solgen_forked_solutions 14.8% ####### - (untracked) 0.0% - python_solgen_fork_permutations 0.0% - python_solgen_custom_kernels 0.0% - python_write_solutions 1.6% - python_wsol_dump 1.5% - python_wsol_prepare 0.0% - python_wsol_prepare_nocache 0.0% - (untracked) 0.0% - (untracked) 0.0% - python_wsol_header 0.0% - (untracked) 1.3% - python_solution_indexing 0.9% - python_write_cache 0.0% - python_cache_check 0.0% - -KEY INSIGHTS (% of wall clock) ------------------------------------------------------------------------------------------------------------------------- - Python phases (code gen, compilation): 171.74s (96.2%) - Python I/O overhead (cache, YAML): 2.86s (1.6%) - C++ client setup (library loading): 0.00s (0.0%) - Data preparation (CPU init, GPU prep): 0.00s (0.0%) - Reference computation (CPU GEMM): 0.00s (0.0%) - Kernel execution (warmup, benchmark): 0.00s (0.0%) - - diff --git a/projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_3_run_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_3_run_analysis.txt deleted file mode 100644 index c6a6b83236a4..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/dtv/dtv_3_run_analysis.txt +++ /dev/null @@ -1,175 +0,0 @@ -======================================================================================================================== -TIMING ANALYSIS SUMMARY -======================================================================================================================== - -TIMING BY PHASE ------------------------------------------------------------------------------------------------------------------------- - Category Count Total (ms) Mean (ms) % of Parent % of Wall ------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 1 84,602.83 84,602.83 99.9% - -------------------------------------------------------------------------------------------------------------------- - python_client_execution 25 84,217.36 3,368.69 99.5% 99.4% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Data Preparation 29,254.25 34.7% 34.5% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - gpu_input_reset 86,908 23,807.02 0.27 81.4% 28.1% - pre_problem 224 4,700.56 20.98 16.1% 5.5% - cpu_reference_gemm 224 3,626.66 16.19 77.2% 4.3% - solve_cpu_slow 8 3,299.45 412.43 91.0% 3.9% - solve_cpu_fast 216 326.78 1.51 9.0% 0.4% - (untracked) 0.43 0.0% 0.0% - cpu_data_init 224 1,069.86 4.78 22.8% 1.3% - (untracked) 4.03 0.1% 0.0% - gpu_input_preparation 224 739.19 3.30 2.5% 0.9% - rotating_buffer_preparation 224 7.48 0.03 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - C++ Client Setup 29,210.24 34.7% 34.5% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - listener_setup 25 13,275.86 531.03 45.4% 15.7% - hip_initialization 25 10,099.40 403.98 34.6% 11.9% - code_object_loading 25 3,704.91 148.20 12.7% 4.4% - library_loading 25 2,072.25 82.89 7.1% 2.4% - lazy_loading_init 25 35.48 1.42 0.1% 0.0% - data_init_setup 25 17.47 0.70 0.1% 0.0% - reporter_setup 25 4.61 0.18 0.0% 0.0% - solution_iterator_setup 25 0.27 0.01 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Reference Computation 11,474.78 13.6% 13.5% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - validate_warmups 86,908 11,474.78 0.13 100.0% 13.5% - validate_gpu_readback 86,908 4,655.34 0.05 40.6% 5.5% - (untracked) 3,735.15 32.6% 4.4% - validate_element_comparison 86,908 3,033.46 0.03 26.4% 3.6% - validate_gpu_sync 86,908 50.82 0.00 0.4% 0.1% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Kernel Execution 5,177.49 6.1% 6.1% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - post_solution 95,512 2,267.92 0.02 43.8% 2.7% - post_solution_log 95,512 1,789.08 0.02 78.9% 2.1% - post_solution_reporting 95,512 358.50 0.00 15.8% 0.4% - post_solution_perf_calc 95,512 58.86 0.00 2.6% 0.1% - post_solution_validation 95,512 53.62 0.00 2.4% 0.1% - post_solution_sol_advance 95,512 2.72 0.00 0.1% 0.0% - post_solution_result_file 95,512 2.66 0.00 0.1% 0.0% - post_solution_perf_reset 95,512 2.62 0.00 0.1% 0.0% - pre_solution 95,512 1,088.56 0.01 21.0% 1.3% - warmup_runs 173,816 1,081.02 0.01 20.9% 1.3% - kernel_solving 86,908 518.41 0.01 10.0% 0.6% - post_problem 224 105.59 0.47 2.0% 0.1% - solution_selection 95,512 104.20 0.00 2.0% 0.1% - benchmark_runs 86,908 11.77 0.00 0.2% 0.0% - (untracked) 11.77 100.0% 0.0% - finalize_report 25 0.01 0.00 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked C++ overhead) 5,016.77 6.0% 5.9% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Timing Overhead 4,083.83 4.8% 4.8% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - flush_timing_buffer 25 3,989.59 159.58 97.7% 4.7% - timing_overhead 25 94.23 3.77 2.3% 0.1% - -------------------------------------------------------------------------------------------------------------------- - (untracked) 317.18 0.4% 0.4% - -------------------------------------------------------------------------------------------------------------------- - python_cache_check 25 60.60 2.42 0.1% 0.1% - -------------------------------------------------------------------------------------------------------------------- - python_write_solutions 25 7.70 0.31 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_prepare 25 4.33 0.17 56.2% 0.0% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_wsol_prepare_cache 25 4.19 0.17 96.9% 0.0% - (untracked) 0.13 3.1% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_header 25 3.07 0.12 39.9% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 0.30 3.9% 0.0% - ------------------------------------------------------------------------------------------------------------------------- - ESTIMATED WALL CLOCK TIME: 84.70s - -HIERARCHY VALIDATION: OK (all child phases fit within their parents) - -TIME BREAKDOWN (visual, % of wall clock) ------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 99.9% ################################################# - python_client_execution 99.4% ################################################# - Data Preparation 34.5% ################# - gpu_input_reset 28.1% ############## - pre_problem 5.5% ## - cpu_reference_gemm 4.3% ## - solve_cpu_slow 3.9% # - solve_cpu_fast 0.4% - (untracked) 0.0% - cpu_data_init 1.3% - (untracked) 0.0% - gpu_input_preparation 0.9% - rotating_buffer_preparation 0.0% - C++ Client Setup 34.5% ################# - listener_setup 15.7% ####### - hip_initialization 11.9% ##### - code_object_loading 4.4% ## - library_loading 2.4% # - lazy_loading_init 0.0% - data_init_setup 0.0% - reporter_setup 0.0% - solution_iterator_setup 0.0% - Reference Computation 13.5% ###### - validate_warmups 13.5% ###### - validate_gpu_readback 5.5% ## - (untracked) 4.4% ## - validate_element_comparison 3.6% # - validate_gpu_sync 0.1% - Kernel Execution 6.1% ### - post_solution 2.7% # - post_solution_log 2.1% # - post_solution_reporting 0.4% - post_solution_perf_calc 0.1% - post_solution_validation 0.1% - post_solution_sol_advance 0.0% - post_solution_result_file 0.0% - post_solution_perf_reset 0.0% - pre_solution 1.3% - warmup_runs 1.3% - kernel_solving 0.6% - post_problem 0.1% - solution_selection 0.1% - benchmark_runs 0.0% - (untracked) 0.0% - finalize_report 0.0% - (untracked C++ overhead) 5.9% ## - Timing Overhead 4.8% ## - flush_timing_buffer 4.7% ## - timing_overhead 0.1% - (untracked) 0.4% - python_cache_check 0.1% - python_write_solutions 0.0% - python_wsol_prepare 0.0% - python_wsol_prepare_cache 0.0% - (untracked) 0.0% - python_wsol_header 0.0% - (untracked) 0.0% - -KEY INSIGHTS (% of wall clock) ------------------------------------------------------------------------------------------------------------------------- - Python phases (code gen, compilation): 0.00s (0.0%) - Python I/O overhead (cache, YAML): 0.07s (0.1%) - C++ client setup (library loading): 29.21s (34.5%) - Data preparation (CPU init, GPU prep): 29.25s (34.5%) - Reference computation (CPU GEMM): 11.49s (13.6%) - Kernel execution (warmup, benchmark): 5.21s (6.2%) - - -TOP 10 SLOWEST PROBLEMS (by CPU reference time) ------------------------------------------------------------------------------------------------------------------------- - M N K Batch TypeA TypeD CPU Ref (ms) GPU (ms) - ---------------------------------------------------------------------------------------------------------------------- - 255 255 256 1 Float8_fnuz Float8_fnuz 593.36 0.00 - 255 255 193 1 Float8_fnuz Float8_fnuz 446.05 0.00 - 255 255 193 1 Float8_fnuz Float8_fnuz 442.14 0.00 - 255 255 191 1 Float8_fnuz Float8_fnuz 392.43 0.00 - 255 255 193 1 Float8_fnuz Float8_fnuz 378.14 0.00 - 255 255 256 1 Float8_fnuz Float8_fnuz 363.47 0.00 - 255 255 256 1 Float8_fnuz Float8_fnuz 348.50 0.00 - 255 255 256 1 Float8_fnuz Float8_fnuz 335.40 0.00 - 1600 1280 192 1 Half Half 35.50 0.00 - 1600 1280 192 1 Half Half 34.14 0.00 - diff --git a/projects/hipblaslt/tensilelite/timing_runs/dtv/times.txt b/projects/hipblaslt/tensilelite/timing_runs/dtv/times.txt deleted file mode 100644 index c434fad13812..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/dtv/times.txt +++ /dev/null @@ -1,4 +0,0 @@ -build_cold rc=0 251s -build_warm rc=0 205s -run rc=0 91s -DONE diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs/run.sh b/projects/hipblaslt/tensilelite/timing_runs/full_runs/run.sh deleted file mode 100755 index a1066fc7691f..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/full_runs/run.sh +++ /dev/null @@ -1,31 +0,0 @@ -#!/bin/bash -# CI-like conditions: 1 GPU, 25 cores, 4 xdist workers, 5400s per-test timeout. -cd /home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite || exit 1 -W=/tmp/tl_split -ART=$W/artifacts -CLIENT=/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client -export OMP_NUM_THREADS=25 OPENBLAS_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=0 PYTHONUNBUFFERED=1 -COMMON=(tensilelite/Tests/common -v --durations=0 --timeout=5400 -n 4 -m "common and not skip-gfx94X" - "--global-parameters=LibraryFormat='msgpack'" "--tensile-options=--cxx-compiler,/usr/bin/amdclang++") -phase() { # name, extra args... - local name=$1; shift - rm -rf /home/alvasile/.tensile /home/alvasile/.tensilelite - echo "START $name $(date -u +%FT%TZ)" >> $W/times.txt - local s=$(date +%s) - taskset -c 0-24 pytest "${COMMON[@]}" "$@" --junit-xml=$W/$name.xml > $W/$name.log 2>&1 - local rc=$? - echo "END $name rc=$rc seconds=$(( $(date +%s)-s )) $(date -u +%FT%TZ)" >> $W/times.txt -} -rm -rf $ART; mkdir -p $ART -# Emulate a CPU-only node: the 5 GPU-gated flags tests would skip there. -F=tensilelite/Tests/common/flags/gfx942 -phase build --build-only --gpu-targets gfx942 --artifact-dir $ART --prebuilt-client=$CLIENT \ - --deselect $F/test_build_only.py::test_use_cache \ - --deselect $F/test_use_cache.py::test_compile_and_benchmark \ - --deselect $F/test_use_cache.py::test_use_cache \ - --deselect $F/test_use_cache.py::test_use_cache_creates_cache_dir \ - --deselect $F/test_use_cache.py::test_use_cache_different_params -du -sh $ART >> $W/times.txt; ls $ART | wc -l >> $W/times.txt -phase run --use-cache --artifact-dir $ART --prebuilt-client=$CLIENT -phase combined --prebuilt-client=$CLIENT -echo ALLDONE >> $W/times.txt diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/FURTHER_SPEEDUPS.md b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/FURTHER_SPEEDUPS.md deleted file mode 100644 index 76175a113b89..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/FURTHER_SPEEDUPS.md +++ /dev/null @@ -1,483 +0,0 @@ -# TensileLite common tests: further speed-up avenues (gfx942) - -Collected from the gfx942 timing work of 2026-11-03..05 on branch `users/alvasile/new_test_client` -(host `sharkmi300x-1`, MI300X, 25 pinned cores, `OMP_NUM_THREADS=25`). For each avenue: why that part -is slow today, the evidence, the options, an estimated saving in CI terms, and the risk. - -## How to read the numbers - -- **CI-like stages** are what matter: the whole build stage in one pytest session - (`--build-only --gpu-targets gfx942 -n 4 --dist=worksteal`, cache cleared at the start), then the - whole run stage (`--use-cache -n 4`). Current wall times at `9c5e825e`/`97c59828`: build ≈ 441 s, - run ≈ 742 s (run measured at `c1345541`; nothing since changes the run stage). -- **Build stage:** timers are summed over the 4 xdist workers (≈1,640 s of instrumented time). - With `worksteal` the workers are balanced, so a summed saving of X s is roughly X/4 s of wall. -- **Run stage:** every client run on a 1-GPU runner takes the same per-GPU lock - (`Tests/conftest.py:92-163`, `ClientWriter.py:314`), so clients run one at a time and client time - is 88-95% of the run wall. A second saved inside the client is about a second of run wall. -- Per-launch costs below come from the instrumented per-config run pass - (`full_runs_gfx942/post_run/AGGREGATE_analysis.txt`, 441 client launches for 123 configs); CI - makes the same 441 launches. -- "Estimated" means not measured end to end. - -## Already done (for context) - -| Change | Commit | Effect (CI-like) | -|---|---|---| -| Deselect configs that opt out of every target arch (563 of 709 on gfx942) | `e2bfff82` | build 612 → 564 s with `load` | -| `--dist=worksteal` for `hw-common` | `056fa353` | build 564 → 475 s (one run); run stage unchanged | -| Artifact gzip level 9 → 4 | `545d5d23` | packing 137.7 → 25.7 s summed (replay), +11.5% size | -| Invalid-kernel check in the parent | `97c59828` | validate 124 → 28 s summed, but the pool resize moved to assembly; wall unchanged (see B1) | -| Timers for artifact packing/extraction; analyzer fixes | `e04a42eb`, `094b04a4` | measurement only | - ---- - -## A. Run stage - -### A1. Hardware-monitor setup on every client launch - -- **Why it is slow:** `HardwareMonitorListener` (`client/src/HardwareMonitorListener.cpp:38-55`) - creates a `HardwareMonitor` whenever `--hardware-monitor` is true, which calls - `amdsmi_init(AMDSMI_INIT_AMD_GPUS)` (`client/src/HardwareMonitor.cpp:139-158`) and starts a - sampling thread. `HardwareMonitor` defaults to `True` (`Common/GlobalParameters.py:61`); only - gfx950 and gfx1250 force it off (`Tensile.py:837-839`). The bundled rocm_smi lists `/tmp` twice - per GPU to find partition files (16 full scans with 8 GPUs); this host's `/tmp` has ~37k entries. -- **Evidence:** `listener_setup` = 179.3 s over 441 launches (407 ms each), 22% of the 829 s - instrumented run total. Measured with the real client (3 runs each, cores 48-72, GPU 6): - - | Variant | `listener_setup` (ms) | `hip_initialization` (ms) | - |---|---:|---:| - | as in the pass (`ROCR_VISIBLE_DEVICES=6`, monitor on) | 404-411 | 378-409 | - | `hardware-monitor=False` | **0.02-0.03** | 386-430 | - | `HIP_VISIBLE_DEVICES=6` instead of ROCR (monitor on / off) | 406-407 / 0.02-0.03 | 384-391 | - | monitor on, private empty `/tmp` (mount namespace) | **102-103** | 380-407 | - -- **Why `amdsmi_init` costs ~400 ms here:** the bundled rocm_smi looks up - `rocmsmi_boot_{compute,memory}_partition_.*` by listing `/tmp`, twice per GPU (16 full - directory scans with 8 GPUs). One `ls -f /tmp` of the 37,120 entries costs 25-45 ms. The path is - hard-coded: `TMPDIR` does not change it. The remaining ~100 ms is amd-smi's sysfs walk of all 8 - GPUs (KFD `io_links`, PCI, DRM, hwmon), which does not honour `ROCR_VISIBLE_DEVICES`. So on a CI - runner with a small `/tmp` the cost is probably ~100 ms per launch (not measured on a runner). -- **Wasted:** the monitor only samples around timed enqueues (`preEnqueues`/`postEnqueues`), and only - 11 of 123 configs time anything (A2). The other ~410 launches initialize it for nothing. -- **Options:** - 1. Start the monitor only when something is timed: `m_active = hardware-monitor && - num-syncs-per-benchmark > 0` (or create `m_monitor` lazily in `preEnqueues`). - 2. Pass `HardwareMonitor: False` for test runs (as `Tensile.py` already does for gfx950). -- **Estimated saving:** ~165 s of run wall on this host; ~40 s on a runner with a small `/tmp`. -- **Risk:** low; option 1 keeps the clock/temperature reports for the 11 timing configs. - -### A2. Why 11 YAMLs use the hardware monitor at all - -- **Why:** these configs keep the default `SyncsPerBenchmark: 1` (the other ~110 set 0), so the - client runs the timed benchmark loop and the monitor samples it. Each is an explicit opt-out in - `Tests/unit/test_common_yaml_syncs_per_benchmark.py` with a reason: - - | Config | Stated reason for the benchmark loop | - |---|---| - | `benchmark_runs/fp32_tn` | benchmark timing coverage | - | `client/rotate_mode0`, `client/rotate_mode1` | RotatingMode test requires the benchmark loop | - | `gemm/icache_flush` | flush_icache only runs inside the sync loop | - | `gemm/fp8nfp16mix_hfp8ns`, `gemm/xfp32`, `groupedgemm/grouped_gemm`, `groupedgemm/grouped_gemm_ck_gfx942`, `sparse/gfx94x/libray_logic` | LibraryLogic requires benchmark data | - | `streamk/sk_hybrid`, `streamk/sk_queue_selfclean` | back-to-back launches validate output and synchronizer | - -- **Open question:** none of these reasons needs clock, temperature or fan readings; they need the - timed loop. If LibraryLogic and the CSV consumers do not read the monitor's columns - (`ClockRateSys`, `TempEdge`, ...), the monitor can be off for every test run, timed or not. To - check: which `ResultKey`s the LibraryLogic path reads. -- **Saving:** the remaining ~31 launches × 0.1-0.4 s once A1 is done (a few seconds). - -### A3. HIP initialization on every client launch - -- **Where:** `client/main.cpp:883-897`: `GetHardware` (`hipGetDeviceCount`, `hipSetDevice`, - `hip::GetCurrentDevice`, `main.cpp:449-462`) and one `DeviceContext` (stream creation). -- **Why it is slow:** a minimal probe (`hipGetDeviceCount` → `hipSetDevice` → `hipStreamCreate` → - `hipMalloc`) reproduces the 381 ms: `hipGetDeviceCount` 334-361 ms, `hipSetDevice` 0.0 ms, first - `hipStreamCreate` 44-45 ms, `hipMalloc` 0.1 ms. - - `hipGetDeviceCount` (= ROCr `hsa_init`): the thunk reads the topology of every KFD node before - `ROCR_VISIBLE_DEVICES` filtering. `strace` counts 16,266 `openat` and 26,481 `read`, including - 10,016 opens of `/sys/devices/virtual/kfd/kfd/topology/nodes//caches//properties` - (626 cache entries per MI300X node, 8 GPU nodes, each read twice) and ~5,000 per-CPU cache - files (192 CPUs). The ~350 ms between the first KFD ioctl and the first queue ioctl contains - no ioctls: pure sysfs parsing. Reading the 5,008 GPU cache files once with `cat` costs 188 ms, - so two passes explain ~340 ms. - - First stream: each hardware queue gets a 186,019,840-byte anonymous mapping registered with - `AMDKFD_IOC_SVM` (~15 ms each, probably the per-queue CWSR save area) before - `AMDKFD_IOC_CREATE_QUEUE`; three are made at start-up. - - No variable helps: `ROCR_VISIBLE_DEVICES` vs `HIP_VISIBLE_DEVICES` gave 378-430 vs 384-391 ms, - and no `HSA_*` / `HSAKMT_*` variable in `libhsa-runtime64.so.1.18.70204` skips the - cache-topology parsing (checked the full list of environment strings). - - Concurrency: 4 simultaneous probes took 472-491 ms (`amdsmi_init`) and 439-453 ms - (`hipGetDeviceCount`) instead of 397 / 334 ms, so start-up mostly overlaps across processes - (relevant to A6b). -- **Evidence:** `hip_initialization` 167.9 s / 441 launches (381 ms each). gfx950's ~150 ms is - probably a smaller topology (inferred). -- **Options:** fewer launches (A4); `GPU_MAX_HW_QUEUES=1` saves ~15 ms setup + ~15 ms exit per launch. -- **Saving:** `GPU_MAX_HW_QUEUES=1` ≈ 13 s; the rest only via A4. **Risk:** low-medium. - -### A4. Too many client launches (441 for 123 configs) - -- **Why:** `BenchmarkProblems._benchmarkProblemType` runs one client process per benchmark step - (`BenchmarkProblems.py:1175-1176` → `runClient`, `:1116`). A config with several problem types or - problem-size groups launches the client several times, each paying start-up, HIP init, monitor, - library/code-object loading and exit. -- **Evidence:** fixed per-launch costs ≈ 468 s (Client launch group: `listener_setup` 179, - `hip_initialization` 168, `process_exit` 55, `library_loading` 17, `data_init_setup` 15, - `code_object_loading` 13, `main_teardown_locals` 11, `process_startup` 9) = 56% of the run total. - `DEFERRED.md` #5 also records a first-call cost per process in `cpu_data_init`. -- **Can the client be reused today? No.** One client process serves exactly one job: one solution - library (`--library-file`, a single value), its code objects and one problem set. `main()` reads - its arguments once and builds the problem factory, data initialization, listeners and reporters - for that job (`client/main.cpp:812`, `:864`, `:967`, `:997`). Several `--config-file` arguments are - merged into one option set, not run as separate jobs (`main.cpp:643-653`), and Tensile's run script - starts the client once per config file (`ClientWriter.py:441-442`). -- **What reuse would take:** - 1. A batch mode in the client: a list of jobs per process, with `main()`'s body moved into a - per-job function and per-job state reset between jobs (code-object modules, caches, listeners, - results file). - 2. Tensile launching once per config with all of its steps. In the run stage (`--use-cache`) every - step's library already exists, so this works if a config's steps do not depend on earlier - steps' results (believed true in the test flow; not verified). - 3. Or, larger: a persistent client per xdist worker receiving jobs over a pipe. -- **What reuse saves, and what it does not** (per launch / summed over 441 launches): - - | Cost | Per launch | Summed | Removed by reuse? | - |---|---:|---:|---| - | `hip_initialization` | 381 ms | 168 s | yes | - | `process_exit` | 124 ms | 55 s | yes | - | `process_startup` | 19 ms | 9 s | yes | - | `main_teardown_locals` | 26 ms | 11 s | partly | - | `listener_setup` (hardware monitor) | 407 ms | 179 s | yes, but A1 removes it more simply | - | `library_loading` | 39 ms | 17 s | no (per job) | - | `code_object_loading` | 30 ms | 13 s | no (per job) | - | `data_init_setup` | 34 ms | 15 s | no (per job) | - - After A1, reuse removes ~0.5 s per launch, ~230 s summed, mostly HIP initialization. -- **What reuse costs:** isolation. A GPU memory fault or hung kernel in one step would take down - every later step in that process (today each step is its own process), and state left over by one - job (buffers, cached pointer arrays, loaded modules) can affect the next. -- **Interaction with the GPU lock (A6b):** both target the same fixed per-launch cost. - Shrinking the lock keeps the start-up work but lets the 4 workers overlap it with another - client's GPU phase (saves wall, not CPU) and can also overlap the CPU reference and comparison. - Reuse removes the start-up work (saves CPU and wall) but does nothing for the CPU phases inside - the lock. Reuse also needs the narrower lock: a client living across several steps cannot be - wrapped by today's process-level lock without holding the GPU for the whole config, so the lock - has to move into the client first. Once the lock is narrowed, the start-up cost is already - overlapped, and reuse's remaining CI gain is at most ~230 s / 4 ≈ 60 s of run wall; it matters - more for serial (per-config) and multi-GPU runs. -- **Estimated saving (CI):** ~210 s of run wall if done instead of A6b (after A1); ~60 s at most - if done after A6b. **Risk:** medium-high. -- **Recommendation:** do A1, then A6b option 1 (lock inside the client, after start-up), measure; - consider reuse only if the remaining HIP initialization cost still matters. - -### A5. Client process exit - -- **Why:** `process_exit` 124 ms per launch: mostly unmapping the five 186 MB per-queue regions - (~15 ms each) plus kernel-side teardown. An early `_exit` saves only ~14 ms. -- **Saving:** ~6 s via fast exit, ~8 s via `GPU_MAX_HW_QUEUES=1`. Mostly folded into A3/A4. - -### A6. Clients run one at a time under `-n 4` - -- **Why:** with one visible GPU every xdist worker gets device id "0" and the same lock file; the - lock is held around the whole client run. Run wall ≈ sum of client times; the GPU is busy only a - fraction of the time (≈13% estimated on gfx950). -- **Evidence:** `worksteal` balanced the run-stage workers but the wall stayed 747/736 s vs - 751/746 s; summed test time rose (lock waiting). -- **Options:** a shared lock for configs that tolerate a shared GPU, exclusive for Stream-K etc.; - split `OMP_NUM_THREADS` among concurrent clients. -- **Estimated saving:** 30-50% of the run wall (not measured). **Risk:** high: a Stream-K config - (`sk_sgemm_hybrid_pap`) failed validation once while sharing a GPU on gfx950. - -### A6b. Shrink the region the GPU lock covers (investigate) - -- **What the lock covers today:** `runClient` takes `ClientExecutionLock` around the whole client - subprocess (`ClientWriter.py:314-322`: `Popen` → `communicate()`). The lock file is per physical - GPU (`Tests/conftest.py:92-163`, `worker_lock_path`), passed to Tensile as `--client-lock` - (`Tensile.py:261, :295`). So everything the client does is serialized, not only the GPU work. -- **What inside it actually needs the GPU to itself** (per-config run totals, - `post_run/AGGREGATE_analysis.txt`, summed over 441 launches, instrumented): - - | Part of the client run | Time | Needs exclusive GPU? | - |---|---:|---| - | Process start, HIP init, hardware monitor, library/code-object load, teardown, exit (A1-A5) | ~468 s | no (HIP init touches the device but does not compete with another client's kernels) | - | CPU reference GEMM (`solve_cpu_fast`) | ~38 s | no (CPU) | - | Result comparison on the CPU (`validate_element_comparison`) | ~32 s | no (CPU) | - | Input preparation (host init, uploads/resets) | ~54 s | partly (copies) | - | GPU readback, Synchronizer reduce | ~30 s | partly | - | Kernel launches and execution (incl. 51 s of instrumentation-only `warmup_first_sync`) | ~68 s | yes | - - On this split most of the ~720 s of client time does not need the GPU exclusively, yet with one - GPU it all runs one client at a time (A6). -- **Why the lock exists (to confirm):** keep benchmark timings free of interference, avoid device - memory pressure from 4 clients, and avoid the shared-GPU failure seen with a Stream-K config - (`sk_sgemm_hybrid_pap`, gfx950). A narrower lock that still covers all kernel execution keeps the - last two properties; timed configs (A2) could keep holding it for their timed loop. -- **Options, smallest first:** - 1. Take the lock inside the client instead of around the process: pass the lock path to the - client (`--gpu-lock`), acquire it (`flock`) after start-up/HIP init/library load and release it - before teardown and exit. This alone moves the ~1 s per launch of start-up and exit (A1, A3, - A5) out of the serialized region. - 2. Hold it only around the solution loop's GPU phases (uploads, launches, readback), not around - CPU reference and CPU comparison. - 3. Per-problem: compute the CPU reference before taking the lock. -- **Constraints:** 4 clients each run `OMP_NUM_THREADS=25` CPU work on the same 25 cores, so - overlapping CPU phases will contend; split the thread count among clients or keep the CPU-heavy - phases short. Any client crash must release the lock (`flock` is released on process exit). -- **Device memory:** with the lock taken after start-up, up to 4 clients hold their allocated - buffers on the GPU at once. Fine for MI300X; check configs with large buffers. -- **Relation to client reuse (A4):** the narrower lock is the smaller first step and a prerequisite - for reuse (see A4). -- **Estimated saving:** not measured. Option 1 alone removes ~1 s per launch from the serialized - region (~440 s of the ~720 s), which could cut the run wall substantially once overlapped across 4 - workers; the realistic gain depends on CPU contention. **Risk:** medium (client and harness - change; timing-sensitive configs must keep exclusivity). **Before building:** confirm why the - lock was added (benchmark timing interference, device memory, Stream-K sharing) and whether a - config's benchmark steps are independent in the run stage; then prototype option 1 on one - multi-step config (e.g. `gemm/fp16_tn`) with `-n 4` and compare run-stage wall. - -### A7. Result validation - -- **Why:** `validate_element_comparison` (32.4 s) compares on one thread; `validate_gpu_readback` - (24.9 s, 11-17 µs per call, dominated by call count in `fp16_tn`, `bufferLoadStore0`); the pinned - host buffer is reallocated whenever the size changes rather than when it grows - (`ReferenceValidator.cpp:729`, 3.3 s). -- **Options:** multi-threaded comparison; grow-only buffer; later a GPU-side compare. -- **Estimated saving:** 15-20 s (CPU changes), 40-50 s with a GPU compare. **Risk:** low / medium - (must keep tolerance, NaN and bounds-check semantics). - -### A8. Smaller run-side items - -- `gpu_input_reset_batched_inputs` 19.7 s: the GuardPageAll pointer arrays are re-uploaded; noted - in `after_run.md`, not filed anywhere. -- CPU reference `solve_cpu_fast` 37.4 s on gfx942 (the large FP8 MX cases are gfx950-only). -- Artifact extraction: < 1 s per config, outside the lock. Not worth pursuing. -- `warmup_first_sync` exists only under instrumentation; not a CI cost. - ---- - -## B. Build stage - -CI-like instrumented build stage, summed over the 4 workers (mean of 2 runs at `97c59828`): -`python_kernel_codegen` 520.5 s, `python_kernel_write_assemble` 388.9 s, -`python_solgen_forked_solutions` 316.6 s, `python_kernel_bench_postprocess` 132.5 s, -`python_kernel_validate` 27.7 s, `python_artifact_compress` 28.3 s; TOTAL 1,637 s. - -### B1. Process-pool resizes and restarts - -- **Why it is slow:** `ParallelMap2` (`Common/Parallel.py`, ~line 272) asks joblib for - `min(budget, number of tasks)` workers, so every small step asks for a different pool size. - joblib's loky backend passes thread-limit variables computed as `cpu_count() // n_jobs` for any - of them not set in the parent (`MKL_NUM_THREADS`, `NUMEXPR_NUM_THREADS`, ...; - `joblib/_parallel_backends.py:221-245`), so a different size changes the executor's arguments and - loky shuts the whole pool down and starts a new one (`loky/reusable_executor.py:189-215`). A plain - shrink also waits for the extra workers to exit (`_resize`). -- **Evidence:** on `gemm/gsuasb.yaml`, each size change cost ~0.5 s (515 / 500 / 580 / 472 ms) vs - 16 ms when the size did not change. The per-config pass attributed ~190 s summed to restarts, - shrinks and grows, plus ~136 s to cold starts. Moving the invalid-kernel check out of the pool - only moved the resize to assembly (`97c59828`). -- **What shrinking buys:** nothing measurable: idle workers use no CPU, the threads-per-worker - change is irrelevant (workers run Python and launch clang; `OMP_NUM_THREADS` is fixed), and the - workers being shut down already exist. -- **Options:** keep the worker count constant within a build when `procs` is given (no task-count - cap); keep the thread-limit variables constant (`parallel_config(inner_max_num_threads=...)` or set - them once in the parent); keep running one-task calls in the parent. In CI the pool lives in each - xdist worker process and would also stay warm across configs. -- **Estimated saving:** 100-200 s summed → ~25-50 s of build wall. **Risk:** low-medium - (`ParallelMap2` is shared with TensileCreateLibrary; can be limited to calls with explicit `procs`). - -### B2. Candidate solutions that are built and then thrown away - -- **Why it is slow:** forking expands every parameter combination, and each candidate is fully - constructed before validity is known: `deepcopy(problemType.state)` (`BenchmarkProblems.py:286`), - `Solution(...)` (`Solution.py:910-1047`), then rejection inside - `Solution.assignDerivedParameters`. Each pool task also pickles `globalParameters`, - `isaInfoMap`, `rocIsaData` and the assembler (`Parallel.py:291`, `BenchmarkProblems.py:342-353`). -- **Evidence:** across the build, 843,891 candidates → 105,160 valid (87.5% rejected). `dtv`: - 534,736 → 18,620 (96.5% rejected), 28.4 s in the per-config pass; `dtl`: 16,688 → 2,078 (87.5%); - `cgemm` 55,296 → 20,520; `bufferLoadStore0` 70,656 → 12,106. Real work ≈ 0.8 ms CPU per candidate - (~675 CPU-s, ~590 of it on rejected ones); the rest of `python_solgen_forked_solutions` is pool - overhead (B1). -- **Options:** reject before building the full `Solution` (cheap pre-checks on the fork parameters - for the rules that reject most); ship constants once per worker instead of per task; batch forks - across steps. -- **Estimated saving:** 20-40 s of per-config time (mostly `dtv`, `cgemm`, `bufferLoadStore0`, - `zgemm`) → ~10-20 s of build wall. **Risk:** medium: the early check must reject exactly what - `Solution` rejects. **First step:** profile `dtv`'s generation to find which rules reject most. - -### B3. Kernels generated and then rejected (codegen) - -- **Why:** some valid solutions overflow resources only during code generation. -- **Evidence:** 105,160 valid solutions → 97,220 kernels kept (7,940 rejected by KernelWriter). - The branch already stops codegen early once the static VGPR allocation exceeds `MaxVgpr` or - `overflowedResources` is set; `DEFERRED.md` #10 records the rejected kernels' cost falling from - 374.5 to ~10.8 CPU-s on the probed configs (gfx950). -- **Option:** reject over-budget kernels in solution validation, before codegen (#10). -- **Saving:** small (~10 CPU-s). **Status:** deferred. - -### B4. Kernel-name and compile-key work repeated per solution - -- **Why:** `getKernelCompileKey` → `getKernelNameMin` is computed per solution and per kernel three - times: in `removeInvalidSolutionsAndKernels`, in `validKeys` (`Run.py:1253-1257`) and in - `passPostKernelInfoToSolution` (`Run.py:727+`); `python_kernel_bench_postprocess` (132.5 s) also - spends time in naming and library construction (1.26 ms per solution, 74% in six configs). -- **Option:** compute each name/key once and cache it on the solution. -- **Estimated saving:** 30-60 s summed (~29 s of it in the validate path). **Risk:** low. - -### B5. Assembly path - -- **Why:** codegen results go back to the parent and then into a second pool per step; each `.s` is - written, then re-read and regex-scanned (`_retargetAssemblySource`, `Toolchain/Component.py:301`), - and assembled with one clang process per kernel (~56 ms per kernel). -- **Option:** assemble inside the codegen workers and drop the re-read. -- **Estimated saving:** 40-70 s summed. **Risk:** low-medium. - -### B6. Code generation itself - -- **Why:** ~50 ms of worker CPU per kernel (`dtv` 12,406 kernels; large tn configs ~200 ms per - kernel). -- **Options:** `DEFERRED.md` #8 (longest-first dispatch within a codegen step) and #9 (cheaper - generator: `hasAnyDependency`, epilogue). **Risk:** medium-high (changes generated code). - -### B7. Scheduling tail and core sharing - -- **Why:** `dtv` alone takes ~230-270 s of a ~441-475 s build, and with collection order it starts - mid-stage. Each xdist worker's pool is sized for 4 busy workers (2 × 25 / 4 = 12 processes), so - when workers go idle at the end, the last one cannot use the freed cores; when all 4 are busy - there are up to 48 processes on 25 cores (test time rose 8% under `worksteal`). -- **Options:** start the longest configs first (cost-ordered collection; pairs with - `load --maxschedchunk=1/2`, not with `worksteal`); split `dtv`; a shared core budget across xdist - workers (jobserver-style token pipe) so the last worker gets all cores. -- **Estimated saving:** up to ~30-40 s (the stage's lower bound is ~435 s of work / 4 workers). - **Risk:** medium for the shared budget (tokens must be released if a worker crashes). - -### B8. Artifact packing in the combined `hw-common` job - -- **Why:** `hw-common` runs the combined mode (`Tests/common/test_config.py`): each config's build - output is packed into a `.tar.gz` in the test's tmpdir and unpacked immediately. Nothing is - transferred, so packing and unpacking are pure overhead there. -- **Evidence:** level 4 packing ≈ 25.7 s summed for the gfx942 set (replay); level 0 5.4 s. -- **Options:** skip the archive in combined mode (pass the output directory to the run step), or - pack at level 0/1 there. Keep level 4 for a split CI that uploads artifacts. -- **Estimated saving:** ~20 s summed (≈5 s of wall) plus extraction. **Risk:** low, but the - round trip currently also checks the artifact whitelist; keep that check if it matters. - -### B9. CMS configs fork `UseCustomMainLoopSchedule: [0, 1]` - -- **Where:** `gemm/gfx950/custom_mainloop_scheduling.yaml` (14 forks with `[0, 1]`), - `gemm/gfx950/agntab_coverage_gfx950.yaml` (1), `gemm/gfx12/custom_schedule.yaml` (1). All of them - skip gfx942, so this costs time on gfx950 and gfx12, not in the gfx942 numbers above. -- **Why it is slow:** the fork builds every tile configuration twice: with the custom main-loop - schedule (1) and with the regular scheduler (0). The default `-1` already resolves to 1 where CMS - is supported and 0 otherwise (`SolutionStructs/Solution.py:3950-3963`). The `0` kernels exercise the - regular scheduler, which the general configs already cover, so in a CMS test they mostly double the - codegen, assembly and benchmark work of those steps. -- **Evidence (gfx950):** `custom_mainloop_scheduling` has 66 benchmark steps (so 66 client launches, - see A4) and 79 kernels; it was the second-slowest test of the gfx950 split run phase (159 s under - `-n 4`) and 30 s of per-config build after the build changes (`after_run.md`, `after_build.md`, - `python_kernel_codegen_investigation.md`). How much of that is the `0` half was not measured. -- **Option:** fork `[1]` only (or `[-1]`) in the CMS configs, unless a step specifically compares - the two; keep one explicit `0` case if the "CMS off" path for CMS-capable tiles needs coverage. -- **Estimated saving:** up to half of the kernels in the 14 forked steps. **Risk:** low (test - coverage decision for the CMS owners). - ---- - -## C. Per-process start-up (Tensile and pytest) - -CI pays these once per xdist worker per stage (~4 s of wall per stage), not per config; the -per-config timing harness pays them per config (~4 s × 123). - -### C1. Collection loads every YAML - -- **Why:** `configMarks` parses all 744 test YAMLs to compute marks even when one test is selected - (`Tests/conftest.py:82` → `Tests/common/config_helpers.py:125, :155`), ~2.25 s per pytest process - (already `CSafeLoader`). The new arch deselection reuses these marks and adds nothing. -- **Option:** cache the marks per YAML keyed on mtime/size. **Saving:** ~2 s per stage in CI. - -### C2. ISA capability probing - -- **Why:** `makeIsaInfoMap` (`Tensile.py:841` → `Capabilities.py:166`) runs 114 `amdclang++` - probes serially (~12 ms each, `hardware_caps.hpp:33-75`, `:97+`) for the one ISA: 1.4-1.9 s per - process. -- **Options:** cache on disk keyed on assembler identity, ISA and probe-table version; or run the - probes in parallel (measured 1.3 → 0.35 s); or probe with `llvm-mc` (0.42 s serial). -- **Saving:** ~1-1.5 s per stage in CI. **Risk:** low-medium (stale cache if the compiler changes). - -### C3. `rocm_agent_enumerator` on every rocisa import - -- **Why:** with `ROCISA_BACKEND` unset, `import rocisa` runs `rocm_agent_enumerator` (45 ms, - `rocisa/__init__.py:124-171`) in every process, including every new loky worker (32 times for - `gemm/ag` in one build). `--gpu-targets` does not skip it. -- **Option:** pass the parent's backend choice to workers. **Saving:** estimated 10-15 s summed in - CI (more while B1 restarts pools). **Risk:** low. - -### C4. Process exit - -- **Why:** shutting the reusable loky executor down at exit takes 0.2-0.7 s; interpreter teardown - up to 0.45 s (~816k live objects after `Tensile()` returns). Once per worker in CI. -- **Option:** `os._exit` after flushing. **Saving:** ~0 in CI. **Risk:** medium. - ---- - -## D. Measurement environment and CI configuration (not product code) - -- **Editable rocisa rebuilds on every import.** The venv's scikit-build-core editable install runs - `cmake --build/--install` under a machine-wide lock on each `import rocisa` (1-9 s; 0.11 s with - `SKBUILD_EDITABLE_SKIP=`). It inflated a whole first measurement 2-5× and coupled the - two measurement sides. The measurement scripts should set `SKBUILD_EDITABLE_SKIP`. CI installs a - wheel and is unaffected. -- **Shared-host `/tmp`** with ~37k entries adds ~300 ms to every `amdsmi_init` (A1). -- **CI marker `not skip-gfx94X` matches nothing** (TheRock's `fetch_test_configurations.py`): configs - carry exact-arch marks. The in-repo deselection (`e2bfff82`) makes it moot for speed; the dead - expression could still be cleaned up in TheRock. -- **`comm/gfx950/fused_a2a.yaml` fails the gfx942 build-only stage** ("0 valid solutions"): a - gfx950-only config without a `skip-gfx942` mark (`CI_SPLIT_TIMING.md` finding 3). Correctness, not - speed. -- **Per-config timing harness overhead:** the serial scripts start one pytest per config (~5 s each - for collection, probing, start-up); this is harness cost, not CI cost. - ---- - -## E. Test-suite hygiene (not speed) - -### E1. Unit tests insert into `sys.path` at import time - -- **What:** 24 files under `Tests/` change `sys.path`, most at module import (collection) time, e.g. - `Tests/unit/test_subtile_gfx1250_codegen.py:19-21` puts the TensileLite root at index 0 so it can - `import gpu_test_helpers`; `gpu_test_helpers.py:39`, `test_mxs_tilespan.py`, - `test_emitMfmaInstruction.py`, `test_storeD_roundtrip.py`, ... do the same; several insert the - characterization `_codegen` dir; `test_analyze_timing.py` and `test_aggregate_timing.py` insert - `scripts/`, and `test_artifact_timing.py` (added in this work, `e04a42eb`) inserts `Tests/common`. - `Tests/conftest.py:39` already appends the same root at the end of the path. -- **Why it matters:** the change is process-wide and permanent, so every later test in the same - pytest/xdist worker process sees it, and what a test imports depends on which other modules were - collected first. Inserting at index 0 makes the source tree shadow an installed package (CI runs - against the installed `share/hipblaslt/tensilelite` tree), and lets bare module names - (`gpu_test_helpers`, `artifact_helpers`, `analyze_timing`) resolve only by accident of order. The - order-dependence bug fixed in `9c5e825e` is the same class of problem (import-time side effects). -- **Options:** declare the import roots once (`pythonpath =` in `pytest.ini`, or the root - `conftest.py`), import helpers as package modules, and load scripts with `importlib` from an - explicit path in the tests that need them; then remove the per-file insertions, starting with the - ones that insert at index 0. -- **Speed:** negligible (a longer `sys.path` only adds lookups). **Risk:** low, but touches many - files; run the unit suite in random order / per file to confirm nothing relied on the side effect. - ---- - -## Suggested order - -| # | Avenue | Stage | Estimated CI wall saving | Risk | -|---|---|---|---:|---| -| 1 | A1 hardware monitor only when timed | run | ~40 s (CI runner) / ~165 s (this host) | low | -| 2 | B1 constant pool size and thread-limit env | build | ~25-50 s | low-medium | -| 3 | A6b lock inside the client (after start-up), then narrow to GPU phases | run | potentially large; not measured | medium | -| 4 | B2 early candidate rejection (profile `dtv` first) | build | ~10-20 s | medium | -| 5 | B4 / B5 name caching, assemble in codegen workers | build | ~20-30 s | low-medium | -| 6 | A7 validation | run | ~15-50 s | low-medium | -| 7 | B7 longest-first, shared core budget | build | ~30-40 s | medium | -| 8 | B8 skip packing in combined mode | build | ~5 s | low | -| 9 | A6 concurrent clients | run | 30-50% of run (unmeasured) | high | -| 9b | A4 client reuse (batch mode), only if HIP init still matters after A6b | run | ≤ ~60 s after A6b (~210 s without it) | medium-high | -| 10 | C1-C3 start-up caching | both | a few s per stage | low | -| 11 | B9 CMS `[0, 1]` forks (gfx950/gfx12) | build + run | not measured; up to half of those steps | low | -| — | E1 `sys.path` insertions | hygiene | none | low | diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/README.md b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/README.md deleted file mode 100644 index 6606388d8840..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/README.md +++ /dev/null @@ -1,54 +0,0 @@ -# gfx942 before/after timing of the TensileLite common tests - -| Item | Before | After | -|---|---|---| -| Commit | `aa19af72` (all timing instrumentation, before the first optimization `8ec52f37`) | `c1345541` (branch head + gfx942 script parameters) | -| Client | `tensilelite-client`, Release, `GPU_TARGETS=gfx942`, msgpack only, `/usr/bin/amdclang++` (ROCm 7.2.4) | same, built from its commit | -| Cores / GPU | `taskset -c 0-24`, `ROCR_VISIBLE_DEVICES=2` (NUMA 0) | `taskset -c 48-72`, `ROCR_VISIBLE_DEVICES=6` (NUMA 1) | - -Host `sharkmi300x-1`: 2x EPYC 9454, 8x MI300X, shared (40+ users, load average 20-50, of which ~50 is -these two passes). `OMP_NUM_THREADS=OPENBLAS_NUM_THREADS=25`, `--timeout=5400`, -`-m "common and not skip-gfx94X"`, `LibraryFormat='msgpack'`. The two sides ran concurrently on -different sockets and GPUs. Both analysed and aggregated with the head's `scripts/analyze_timing.py` / -`aggregate_timing.py`. Scripts: `../full_runs_gfx950/{split,per_config}/` (gfx942 parameters from -`c1345541`). - -`SKBUILD_EDITABLE_SKIP` was set to the rocisa build dir: the venv's editable rocisa otherwise re-runs -`cmake --build/--install` under a machine-wide lock on every import (1-9 s per pytest process and per -loky worker). CI installs a wheel. A first attempt without it was discarded. - -## Folders - -Committed: the four `AGGREGATE_analysis.txt`, the comparison tables, the split `times.txt` and the -config list. Not committed (too large, or local working notes): the raw per-config logs (`*.log.gz`, -~430 MB), the per-config `*_analysis.txt` and `times.txt`, the split logs/XML, the driver and the -hotspot analyses. They are kept on the measurement host next to this file. - -- `split_pre/`, `split_post/`: CI-like split (uninstrumented, `-n 4`): `build` (`--build-only - --gpu-targets gfx942`, caches cleared) then `run` (`--use-cache`, caches cleared). Logs, JUnit XML, - `times.txt`, the passed run configs (`split_run_passed.txt`) and the shared list (`configs.txt`, - 123 configs, identical on both sides). -- `pre_build/`, `post_build/`: every config built serially, `TimingInstrumentation=True`; caches cleared - once before the first config, artifacts kept. Per config: raw `*_build.log.gz`, `*_build_analysis.txt`, - `times.txt` (pytest wall). `AGGREGATE_analysis.txt` over all 123. -- `pre_run/`, `post_run/`: every config run serially (`--use-cache`) on that build's artifacts and with - its cache (not cleared). Same files. -- `build_comparison.txt`, `run_comparison.txt`: purpose tables side by side (`cmp_agg.py`). -- `opportunity_inventory.md`, `build_hotspots.md`, `run_hotspots.md`: remaining-opportunity analyses. - -All 123 configs returned 0 in all four per-config passes. Split: build 145 passed / 1 failed -(`comm/gfx950/fused_a2a.yaml`, known, `CI_SPLIT_TIMING.md` finding 3) on both sides; run 0 failed. - -## Results - -| Measurement | Before | After | Change | -|---|---:|---:|---:| -| CI-like build phase wall (`-n 4`) | 1,028 s | 634 s | -38% | -| CI-like run phase wall (`-n 4`) | 1,494 s | 742 s | -50% | -| Per-config build TOTAL (instrumented) | 2,134 s | 1,152 s | -46% | -| Per-config build pytest wall sum | 2,898 s | 1,886 s | -35% | -| Per-config run TOTAL (instrumented) | 1,644 s | 829 s | -50% | -| Per-config run pytest wall sum | 2,320 s | 1,484 s | -36% | - -No config got slower by more than 2 s in either per-config pass. Single samples (no repeats); the -sides ran on different sockets/GPUs. diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/build_comparison.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/build_comparison.txt deleted file mode 100644 index b7cc05a62665..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/build_comparison.txt +++ /dev/null @@ -1,43 +0,0 @@ -Category pre s post s delta -TOTAL RUNTIME 2134.4 1152.1 -46.0% -BUILD: solution generation & compile 2086.5 1104.8 -47.1% - Solution generation & build (Python) 2086.5 1104.8 -47.1% - python_kernel_compilation 1667.0 807.2 -51.6% - python_kernel_codegen 455.1 268.2 -41.1% - python_kernel_write_assemble 158.1 173.4 +9.7% - python_kernel_bench_postprocess 123.7 122.7 -0.8% - python_benchpost_lib_construction 83.0 77.8 -6.3% - python_benchpost_library_write 28.5 32.6 +14.5% - python_benchpost_naming 12.1 12.2 +0.6% - python_kernel_bench_setup 123.6 4.4 -96.4% - python_kernel_validate 73.2 125.9 +71.9% - python_kernel_setup 31.8 31.5 -0.8% - python_kernel_build_co 20.3 20.7 +1.6% - python_kernel_write_helpers 8.0 6.8 -15.4% - (untracked) 673.1 5.8 -99.1% - python_solution_generation 419.6 297.6 -29.1% - python_solgen_forked_solutions 418.5 296.4 -29.2% - python_solgen_custom_kernels 1.1 1.2 +7.4% -SHARED: cache & solution files 33.0 32.6 -1.1% - python_write_solutions 20.8 20.5 -1.2% - python_wsol_dump 20.1 20.0 -0.8% - python_solution_indexing 11.7 11.5 -1.5% -UNATTRIBUTED (untracked time & timer overhead) 14.9 14.7 -1.2% - (untracked in python_benchmark_problems) 14.9 14.7 -1.2% - python_kernel_build_src_co_wait 0.0 47.7 new - -pytest wall sum: pre 2898 s, post 1886 s (-34.9%) -largest per-config (pre -> post s): - common__gemm__dtv 251 -> 160 - common__gemm__cgemm 107 -> 96 - common__gemm__bufferLoadStore0 99 -> 81 - common__gemm__mix_cvt_after_ds_fnuz 77 -> 60 - common__gemm__zgemm 76 -> 70 - common__gemm__fp16_tn 74 -> 57 - common__gradient__fp16_gradient_bias 68 -> 18 - common__gemm__gemm_bias_strided_batched_broadcast_gfx940 61 -> 13 - common__gemm__lsu_fnuz 52 -> 35 - common__sparse__gfx94x__f8n_gate_r 49 -> 22 - common__gemm__fp16_HH_BHS_bf16mfma 49 -> 29 - common__gemm__bf16_tn 46 -> 35 -slower by >2 s: [] diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/run_comparison.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/run_comparison.txt deleted file mode 100644 index 870aaaae98ef..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/run_comparison.txt +++ /dev/null @@ -1,85 +0,0 @@ -Category pre s post s delta -TOTAL RUNTIME 1644.4 828.7 -49.6% -RUN: benchmark client 1534.6 718.9 -53.2% - Reference GEMM (CPU) 657.0 38.4 -94.2% - cpu_reference_gemm 657.0 38.3 -94.2% - solve_cpu_slow 624.5 0.9 -99.9% - solve_cpu_fast 32.4 37.4 +15.4% - Client launch (startup, setup & teardown) 472.6 468.4 -0.9% - listener_setup 178.6 179.3 +0.4% - hip_initialization 167.2 167.9 +0.4% - process_exit 54.0 54.7 +1.4% - data_init_setup 18.8 15.1 -19.6% - library_loading 17.4 17.2 -0.7% - code_object_loading 14.2 13.1 -7.9% - main_teardown_locals 13.1 11.3 -13.3% - process_startup 8.2 8.6 +5.3% - Validation 180.8 66.8 -63.1% - validate_warmups_listener.ReferenceValidator 96.9 61.0 -37.1% - validate_gpu_readback 59.9 24.9 -58.4% - validate_element_comparison 32.2 32.4 +0.4% - validate_result_buffer_alloc 3.9 3.3 -16.5% - validate_warmups_listener.SynchronizerValidator 83.6 0.0 -100.0% - validate_synchronizer_check 83.6 0.0 -100.0% - validate_synchronizer_scan 56.0 0.0 -100.0% - validate_synchronizer_readback 27.4 0.0 -100.0% - Input data preparation 128.3 53.7 -58.1% - gpu_input_reset 93.2 31.7 -66.0% - gpu_input_reset_copy_valid_to_gpu 39.1 0.1 -99.8% - gpu_input_reset_batched_inputs 36.4 19.7 -45.9% - gpu_input_reset_copy_inputs_d2d 11.4 10.0 -12.3% - gpu_input_reset_copy_swizzled 3.9 0.0 -99.9% - gpu_input_reset_fast_reset_output 2.0 1.6 -19.1% - cpu_data_init 22.9 15.1 -34.0% - gpu_input_preparation 11.6 6.8 -41.9% - gpu_input_prep_copy_valid_to_gpu 8.6 4.9 -42.3% - gpu_input_prep_init_cpu_inputs 1.7 0.6 -66.6% - gpu_input_prep_copy_swizzled 1.0 1.0 -3.5% - Per-solution bookkeeping & reporting 44.3 9.3 -79.1% - post_solution_log 15.8 1.1 -93.3% - pre_solution_listener.BenchmarkTimer 8.3 1.5 -82.4% - run_current_solution_check 5.7 1.6 -72.3% - post_solution_reporting 3.6 1.5 -58.7% - pre_solution_listener.ProgressListener 2.4 0.6 -77.2% - pre_solution_listener.AllSolutionsIterator 2.0 1.0 -50.8% - post_problem 1.7 0.1 -96.1% - run_iteration_teardown 1.4 0.2 -85.7% - Kernel launch & execution 35.4 67.8 +91.3% - warmup_runs 20.3 11.5 -43.5% - kernel_solving 6.2 3.4 -45.7% - validate_warmups_listener.BenchmarkTimer 5.9 0.0 -99.4% - validate_gpu_sync 5.8 0.0 -100.0% - benchmark_runs 1.7 1.2 -27.9% - benchmark_validate_enqueues 1.1 0.8 -25.6% - warmup_events_setup 1.4 0.6 -56.3% - Library logic (Python) 9.3 8.4 -10.4% - python_logic_write 8.3 7.3 -12.1% - Library client (Python) 6.8 6.2 -9.3% - python_library_client_create_library 4.7 4.1 -12.5% - python_library_client_execution 2.1 2.1 -1.6% - python_library_client_subprocess 2.1 2.1 -1.6% -MEASUREMENT OVERHEAD (instrumentation) 108.1 108.0 -0.1% - flush_timing_buffer 89.8 89.8 -0.0% - main_timing_calibration 11.0 11.0 -0.4% - timing_overhead 3.7 3.7 -0.4% - (timer overhead subtracted from parent timers) 3.5 3.5 -0.5% - (untracked in python_benchmark_problems) 2.9 3.2 +10.0% - warmup_first_sync 0.0 50.7 new - synchronizer_check 0.0 5.5 new - synchronizer_reduce 0.0 5.4 new - -pytest wall sum: pre 2320 s, post 1484 s (-36.0%) -largest per-config (pre -> post s): - common__streamk__sk_f8ngemm_quick 490 -> 21 - common__gemm__fp16_tn 127 -> 70 - common__gemm__bufferLoadStore0 119 -> 98 - common__gemm__dtv 88 -> 58 - common__gemm__cgemm 45 -> 32 - common__gemm__zgemm 41 -> 27 - common__gemm__mix_cvt_after_ds_fnuz 36 -> 29 - common__streamk__sk_sgemm_quick 30 -> 23 - common__gemm__bf16_tn 27 -> 18 - common__gemm__fp32_tn 27 -> 17 - common__gemm__fp8nfp16mix_hfp8ns 26 -> 22 - common__streamk__sk_dgemm_quick 24 -> 17 -slower by >2 s: [] diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_post/times.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_post/times.txt deleted file mode 100644 index 6b201d8216f7..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_post/times.txt +++ /dev/null @@ -1,6 +0,0 @@ -START build 2026-11-04T00:21:39Z -END build rc=1 seconds=634 2026-11-04T00:32:13Z -380M /home/alvasile/tl_gfx942_timing/post/split_art -START run 2026-11-04T00:32:14Z -END run rc=0 seconds=742 2026-11-04T00:44:36Z -ALLDONE diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_pre/configs.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_pre/configs.txt deleted file mode 100644 index 30203ebfcf49..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_pre/configs.txt +++ /dev/null @@ -1,123 +0,0 @@ -tensilelite/Tests/common/amaxd/8rn_amaxd.yaml -tensilelite/Tests/common/benchmark_runs/fp32_tn.yaml -tensilelite/Tests/common/client/rotate_mode0.yaml -tensilelite/Tests/common/client/rotate_mode1.yaml -tensilelite/Tests/common/exception/f16_cf32_nan.yaml -tensilelite/Tests/common/gemm/8rn_b_s.yaml -tensilelite/Tests/common/gemm/ag.yaml -tensilelite/Tests/common/gemm/agntab_coverage_gfx942.yaml -tensilelite/Tests/common/gemm/bf16_tn.yaml -tensilelite/Tests/common/gemm/bf16_tn_predict.yaml -tensilelite/Tests/common/gemm/bufferLoadStore0.yaml -tensilelite/Tests/common/gemm/cgemm.yaml -tensilelite/Tests/common/gemm/dot2_gfx942.yaml -tensilelite/Tests/common/gemm/dtl.yaml -tensilelite/Tests/common/gemm/dtv.yaml -tensilelite/Tests/common/gemm/f64.yaml -tensilelite/Tests/common/gemm/fp16_HH_BHS_bf16mfma.yaml -tensilelite/Tests/common/gemm/fp16_tn.yaml -tensilelite/Tests/common/gemm/fp16_use_e.yaml -tensilelite/Tests/common/gemm/fp16fp32mix.yaml -tensilelite/Tests/common/gemm/fp32_nt.yaml -tensilelite/Tests/common/gemm/fp32_scalealphavec.yaml -tensilelite/Tests/common/gemm/fp32_tn.yaml -tensilelite/Tests/common/gemm/fp8bf8nss.yaml -tensilelite/Tests/common/gemm/fp8n.yaml -tensilelite/Tests/common/gemm/fp8n_rowwise.yaml -tensilelite/Tests/common/gemm/fp8n_use_e.yaml -tensilelite/Tests/common/gemm/fp8nfp16mix_fp8nss.yaml -tensilelite/Tests/common/gemm/fp8nfp16mix_hfp8ns.yaml -tensilelite/Tests/common/gemm/fp8nfp16mix_hhs.yaml -tensilelite/Tests/common/gemm/fp8nfp16mix_hhs_lsu4.yaml -tensilelite/Tests/common/gemm/gemm_bias_strided_batched_broadcast_gfx940.yaml -tensilelite/Tests/common/gemm/gemm_bias_strided_batched_gfx940.yaml -tensilelite/Tests/common/gemm/gemm_ck_gfx942.yaml -tensilelite/Tests/common/gemm/gsuasb.yaml -tensilelite/Tests/common/gemm/gsuc.yaml -tensilelite/Tests/common/gemm/hh_f8nhs.yaml -tensilelite/Tests/common/gemm/i8_gsu_gfx940.yaml -tensilelite/Tests/common/gemm/icache_flush.yaml -tensilelite/Tests/common/gemm/largeMT.yaml -tensilelite/Tests/common/gemm/large_size.yaml -tensilelite/Tests/common/gemm/lsu_fnuz.yaml -tensilelite/Tests/common/gemm/lsu_i8.yaml -tensilelite/Tests/common/gemm/mbskPrefetchOpt.yaml -tensilelite/Tests/common/gemm/mix_cvt_after_ds_fnuz.yaml -tensilelite/Tests/common/gemm/nan_bounds_check_odd_padding.yaml -tensilelite/Tests/common/gemm/no_shadowinit.yaml -tensilelite/Tests/common/gemm/sia0_pgr.yaml -tensilelite/Tests/common/gemm/swizzleA.yaml -tensilelite/Tests/common/gemm/swizzleB.yaml -tensilelite/Tests/common/gemm/tail_small_size.yaml -tensilelite/Tests/common/gemm/ulsgro1.yaml -tensilelite/Tests/common/gemm/use_beta_false.yaml -tensilelite/Tests/common/gemm/wgm.yaml -tensilelite/Tests/common/gemm/xfp32.yaml -tensilelite/Tests/common/gemm/zgemm.yaml -tensilelite/Tests/common/gradient/fp16_gradient_activation.yaml -tensilelite/Tests/common/gradient/fp16_gradient_bias.yaml -tensilelite/Tests/common/gradient/fp8bf8nss_gradient_bias_b.yaml -tensilelite/Tests/common/groupedgemm/grouped_gemm.yaml -tensilelite/Tests/common/groupedgemm/grouped_gemm_ck_gfx942.yaml -tensilelite/Tests/common/groupedgemm/grouped_gemm_userargs.yaml -tensilelite/Tests/common/gsu/auto_gsu.yaml -tensilelite/Tests/common/gsu/auto_gsu_zero_dim.yaml -tensilelite/Tests/common/gsu/f32_gsu.yaml -tensilelite/Tests/common/gsu/gsu_mbsk.yaml -tensilelite/Tests/common/sgemm_xf32_asm.yaml -tensilelite/Tests/common/sparse/gfx94x/bf16_activation.yaml -tensilelite/Tests/common/sparse/gfx94x/bf16_gate_r.yaml -tensilelite/Tests/common/sparse/gfx94x/f8n_gate_r.yaml -tensilelite/Tests/common/sparse/gfx94x/fp16_activation.yaml -tensilelite/Tests/common/sparse/gfx94x/fp16_gate_r.yaml -tensilelite/Tests/common/sparse/gfx94x/i8_activation.yaml -tensilelite/Tests/common/sparse/gfx94x/libray_logic.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_b8f8n.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_b8f8n_sb.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_bf16.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_bf16_sb.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_bf8n.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_bf8n_sb.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_f8b8n.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_f8b8n_sb.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_f8n.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_f8n_sb.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi16.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi16_sb.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi32.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi32_sb.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_gsuc.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi16.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi16_sb.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi32.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi32_sb.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_i8bs.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_i8bs_sb.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_i8hs.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_i8hs_sb.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_i8is.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_i8is_sb.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_vw_lg_one.yaml -tensilelite/Tests/common/sparse/gfx94x/spmm_vw_lg_one_sb.yaml -tensilelite/Tests/common/sparse/gfx94x/use_sgpr_for_gro.yaml -tensilelite/Tests/common/sparse/gfx94x/use_sgpr_for_gro_sb.yaml -tensilelite/Tests/common/streamk/data_parallel_static_sgemm.yaml -tensilelite/Tests/common/streamk/sk_bgemm_quick.yaml -tensilelite/Tests/common/streamk/sk_cgemm_quick.yaml -tensilelite/Tests/common/streamk/sk_dgemm_quick.yaml -tensilelite/Tests/common/streamk/sk_dynamic.yaml -tensilelite/Tests/common/streamk/sk_dynamic_work_stealing.yaml -tensilelite/Tests/common/streamk/sk_f8ngemm_quick.yaml -tensilelite/Tests/common/streamk/sk_hgemm_epilogues.yaml -tensilelite/Tests/common/streamk/sk_hgemm_lsu.yaml -tensilelite/Tests/common/streamk/sk_hgemm_quick.yaml -tensilelite/Tests/common/streamk/sk_hgemm_race.yaml -tensilelite/Tests/common/streamk/sk_hybrid.yaml -tensilelite/Tests/common/streamk/sk_hybrid_work_stealing.yaml -tensilelite/Tests/common/streamk/sk_queue_selfclean.yaml -tensilelite/Tests/common/streamk/sk_sgemm_alpha0.yaml -tensilelite/Tests/common/streamk/sk_sgemm_quick.yaml -tensilelite/Tests/common/streamk/sk_zgemm_quick.yaml -tensilelite/Tests/custom/custom_ck.yaml -tensilelite/Tests/custom/custom_rr.yaml -tensilelite/Tests/custom/custom_tensile_sk.yaml diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_pre/times.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_pre/times.txt deleted file mode 100644 index df9e42f563b3..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx942/split_pre/times.txt +++ /dev/null @@ -1,6 +0,0 @@ -START build 2026-11-04T00:21:39Z -END build rc=1 seconds=1028 2026-11-04T00:38:47Z -398M /home/alvasile/tl_gfx942_timing/pre/split_art -START run 2026-11-04T00:38:48Z -END run rc=0 seconds=1494 2026-11-04T01:03:42Z -ALLDONE diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_build.md b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_build.md deleted file mode 100644 index 2229080daa92..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_build.md +++ /dev/null @@ -1,269 +0,0 @@ -# gfx950 build-side measurement after the build-side changes (P6) - -Build-side measurement at the end of P4 of the implementation plan (`timing_runs/DECISIONS.md`, -P6), compared with the P0.2 baseline (`baseline.md`). It covers the build-side changes -P4.1–P4.15 and the fix of a regression this measurement found (see "Regression found"). - -## What was measured - -| Item | Value | -|---|---| -| Commit | `9f493d7a6c1` (end of P4 plus the custom-kernel fix; detached worktree). The first 25 per-config pairs and one supplementary split triple ran at `38c29661a7d` (end of P4), see "Regression found". | -| Client | P3's `tensilelite-client` (built at `3b4ec8e125d`, sha256 `15f20fa8a2eb…`). No client, library or CMake source changed between `3b4ec8e125d` and `9f493d7a6c1` (P4 and the fix are Python, YAML and `measure_env.sh` only). The client only runs during the build phase in ductile configs. | -| Baseline | P0.2's worktree (`2083e6254bf`) and client (sha256 `4b2769f7f9a2…`), for the same-session control runs | -| Python | venv Python 3.12.3; TensileLite Python sources from each worktree; venv `rocisa` (unchanged) | -| Host | `smci355-1b112-b13-15` (as P0.2): 2× EPYC 9575F, 8× MI355X (gfx950), ROCm 7.2.1, `/usr/bin/amdclang++` | -| Cores / threads | `taskset -c 0-24`, `OMP_NUM_THREADS=25`, `OPENBLAS_NUM_THREADS=25` (as P0.2) | -| GPU | An idle MI355X per pair or split run: `ROCR_VISIBLE_DEVICES=5` (PCI `85:00.0`) or `0` (PCI `75:00.0`, the P0.2 GPU). The build phase uses it only for the ductile configs' client runs. | -| Config list | Per-config pass: the 185 configs of P3 (P0.2's 184 plus `gemm/gfx950/prediction_bf16`, new in P2.5). Split: all of `Tests/common` with the gfx950 markers, as `split/run_split.sh`. | -| Date | 2026-11-01, 14:25–17:49 UTC | - -Committed results: - -- `after_build/AGGREGATE_analysis.txt`: per-config build aggregate (cold cache per config) at - `9f493d7a6c1`/`38c29661a7d` (see above), 185 logs. -- `after_build/split_build_outcomes.txt`: outcome of every test in the split build phase at - `9f493d7a6c1`. - -Raw logs (not committed, as for P0.2/P3) are in `/home/alvasile/timing_runs_raw/P6/`, gzipped: -per-config logs of both sides, every split build log (with captured Tensile output), the -same-session P0.2 control aggregate, monitoring logs and the driver/analysis scripts (`scripts/`). - -## How it was measured - -`baseline.md` asks P6 to re-run the P0.2 build pass and split build phase back to back with its -own, under identical conditions, because both P0.2 build numbers overlapped another user's job. -As in P3, every comparison here is same-session: - -- **Per-config pass (interleaved).** For each config, P0.2 (its worktree and client) and P6 ran - back to back on the same idle GPU, alternating which went first, with - `run_per_config_build.sh`'s settings (serial, `TimingInstrumentation=True`, caches cleared - before every config, 25 pinned cores). The script copy used for both sides differs only in - setting `ROCR_VISIBLE_DEVICES` to the pair's GPU and in skipping the per-call aggregate. A pair - was re-run (up to three tries) if another user's process took more than 1 GB on the GPU or - more than 10 CPU-seconds of pinned work on cores 0–24 during either run. -- **Split build phase** (`--build-only`, `-n 4`, cache emptied at the start and kept across - configs, the CI-like build): the same command as `split/run_split.sh`'s build phase, plus `-rP` - so that the captured Tensile output of passed tests (the cache lines counted below) is in the - log. Three runs per triple, back to back: P0.2, P6, and P6 with - `TENSILE_DISABLE_HELPER_CHUNK_CACHE=1`. At `9f493d7a6c1` there are two triples, in opposite - order (C–B–A at 16:51, A–B–C at 17:20). -- Pytest temp dirs (`TMPDIR`), caches, scratch artifacts and output were on `/dev/shm` for both - sides, because `/` was nearly full. - -Conditions: no CPU from other users' jobs pinned into cores 0–24 in any run (no corpus job -there today; sampled every 30 s). The 1-minute load average was 17–40, which includes the -measurement's own 25 cores. GPUs: the pairs ran on `ROCR` 5 (141 pairs) and 0 (43 pairs). One -pair, `gemm/zgemm`, had another process on the GPU in all three tries and is kept (it builds no -client run, so the GPU is idle in it). In the last four split runs, on `ROCR` 0, idle root -processes held 0.5 GB of VRAM and briefly up to 16 GB, which the monitor flagged; the build does -not use the GPU outside the ductile configs, and those runs match their `ROCR` 5 counterparts. - -## Regression found: custom-kernel configs failed to build at `38c29661a7d` - -The first split build at `38c29661a7d` had **8 new build failures**: -`gemm/gfx950/custom_xfp32` and `custom/{custom_aiter_bf16, custom_aiter_f4, -custom_rocblas_gemv_bf16_tn_m1, _m2, _m4, custom_triton_f4, custom_wave_bf16}`. They all failed -with `KeyError: 'LdsNumBytes'` in `TensileCreateLibrary/Run.py` `processKernelSource`. P4.7 -(`168da253f63`) and P4.8 (`66281bc369e`) read `LdsNumBytes`, `LoopIters` and `LdsOffsetBias` -from every kernel after codegen, and custom kernels (assembly read from a file) never have those -keys. Neither step's verification built a custom-kernel config. The fix, `9f493d7a6c1` -(`fix(tensilelite): do not require LdsNumBytes / LoopIters / LdsOffsetBias on custom kernels`), -reads them with `.get()` and copies only keys that exist. - -At `9f493d7a6c1` every split build phase gives 196 passed, 2 failed, 370 skipped (4 runs, -identical lists). That differs from P0.2's build phase only by the new `prediction_bf16` -(PASSED). The 2 failures are the known `custom_rr_f6`/`custom_rr_mx` ("No valid solutions -found"). - -The worktree was moved to `9f493d7a6c1` between per-config pairs 25 and 26 (15:12:01 UTC). -Pairs 1–25 were measured at `38c29661a7d` and kept: they include none of the 8 configs (those are -pairs 31 and 178–184, all measured at `9f493d7a6c1`). For kernels that have the three keys, which -is every kernel of a config that built at `38c29661a7d`, the fix runs the same code. The split -triple at `38c29661a7d` is reported only as a supplement, since 8 of its configs stopped early. - -## Split build phase (`-n 4`, cache kept across configs, uninstrumented) - -| Run (UTC start, GPU) | Commit | Wall | Δ vs P0.2 (same triple) | Sum of test call times | -|---|---|---:|---:|---:| -| P6, chunk cache off (16:51, `ROCR` 5) | `9f493d7a6c1` | 539 s | −12.8% | 1,922 s | -| P6, chunk cache on (17:00, `ROCR` 5) | `9f493d7a6c1` | 546 s | −11.7% | 1,959 s | -| P0.2 (17:09, `ROCR` 0) | `2083e6254bf` | 618 s | | 2,428 s | -| P0.2 (17:20, `ROCR` 0) | `2083e6254bf` | 615 s | | 2,407 s | -| P6, chunk cache on (17:31, `ROCR` 0) | `9f493d7a6c1` | 525 s | −14.6% | 1,924 s | -| P6, chunk cache off (17:40, `ROCR` 0) | `9f493d7a6c1` | 511 s | −16.9% | 1,862 s | -| *Supplement:* P0.2 / P6 on / P6 off (14:25, `ROCR` 5) | `38c29661a7d` | 620 / 502 / 484 s | | 2,424 / 1,754 / 1,722 s | - -- **Wall time: P0.2 616.5 s (mean of 2), P6 535.5 s with the chunk cache (−13.1%), 525 s - without it (−14.8%).** P0.2's two runs agree within 0.5% and with the 620 s at 14:25, so - this session's P0.2 build phase is 615–620 s. The 634 s in `baseline.md` was measured under - contention, about 3% higher. -- The summed test call times fall more (−20%) than the wall time. With `-n 4` the wall time is - bounded by how the 4 workers' queues end, and P6 sizes each worker's pools to the P4.3 budget - (2 × 25 / 4 = 12 processes: 10 codegen workers plus 2 helper-compile chunks). P0.2 ran up to 25 - processes per worker. In the same-session runs at `9f493d7a6c1`, `dtl` takes 271–301 s under - `-n 4` with P6 against 190–191 s with P0.2. With P6 it still ends at 63–71% of the run's wall - time (P0.2: about 82%), so it is not on P6's critical path. Per-worker busy time ranges from - about 430 to 542 s with P6, against 603–612 s with P0.2 (`split_p0_2` and the P6 runs, from - `build.xml` durations in worker order). -- **Chunk cache on versus off: no measurable benefit.** In both orders the run without the - chunk cache was faster, by 7 s and 14 s (1–3%). That is within the session's drift (the same - configuration differed by 21–28 s between the triples). -- The two known failing configs now get further before failing (P4.10 prepares every step - first), at a cost of 1–2 s of test time. - -### Cache hit rates (CI-like run, empty cache at start) - -Counted from the Tensile output in the split logs (`scripts/split_stats.py`), P6 at -`9f493d7a6c1`, identical in all four runs except where noted: - -| Cache | Lookups | Hits | Rate | Notes | -|---|---:|---:|---:|---| -| Whole-file helper cache (`HelperKernelCache`) | 927 | 820 | 88.5% | P0.2: 753 of 859 (87.7%). P6 does 67 more lookups, all hits, in the two failing `custom_rr_*` configs, which get further now (see above), and 1 (a miss) in the new `prediction_bf16`. That miss is also why there are 107 PCH lookups instead of 106. | -| HIP-header PCH (P4.12) | 107 (one per whole-file miss) | 103–104 | 96–97% | 3–4 PCH builds: up to 3 xdist workers built the shared key at the same time at the start (each stores it once, by rename), and `custom_rr_mx` uses its own key. 2 entries, 26 MB. | -| Helper chunk cache (P4.15) | 182 chunks in 91 chunked compiles | 21 | **11.5%** | 159 entries, 24 MB. 0 key-computation failures, so FOLLOWUPS P4.15's caveat (a failed key computation still prints "0 of n") did not affect the count. 0 fallbacks to the one-piece compile. | - -The chunk cache only gets lookups that the whole-file cache missed, and those are mostly helper -sets no earlier config had. The 21 hits are spread over 18 configs, 1–3 chunks each. -`subtile_bf16` hits 3 of 8. Under `-n 4` each helper compile uses 2 chunks, while a serial -build uses 4. Chunk contents depend on the budget (FOLLOWUPS P4.14), so a cache filled by one -of these builds does not serve the other. - -In the per-config pass (cache cleared before every config), P6 built the PCH once per config -(184 builds, 79 hits by later helper compiles of the same config). 14 of 719 chunks hit (1.9%). - -## Per-config pass (cold cache per config, instrumented) - -Pytest wall seconds summed over the 184 configs common to both sides. "P0.2" is the committed -`baseline_build` pass. "Control" is P0.2 in the interleaved pass. - -| Set | Configs | P0.2 | Control | P6 | Δ vs control | -|---|---:|---:|---:|---:|---:| -| All | 184 | 3,361 | 3,359 | 1,970 | **−41.4%** | -| … `LibraryLogic:`/`LibraryClient:` removed or added (P2.5/P2.6) | 25 | 259 | 257 | 178 | −30.7% | -| … all others | 159 | 3,102 | 3,102 | 1,792 | −42.2% | -| … `CpuThreads: 1` removed (P4.8) | 19 | 526 | 537 | 195 | −63.7% | -| … the 8 configs fixed by `9f493d7a6c1` | 8 | 57 | 57 | 40 | −29.8% | -| … pairs 1–25 (measured at `38c29661a7d`) | 25 | 600 | 595 | 430 | −27.7% | - -`prediction_bf16` (P6 only) adds 5 s and is in the committed aggregate only. - -**The control reproduces the committed P0.2 build pass**: pytest sums 3,359 against 3,361 s -(−0.1%), TOTAL RUNTIME 2,662.4 against 2,654.8 s (+0.3%). The core-9 contention during the -P0.2 pass therefore did not inflate it measurably, and comparisons against the committed -baseline are equally valid. - -Seconds from "TIMING BY PURPOSE" over the same 184 configs. P0.2 is re-aggregated with the -analyzer at `9f493d7a6c1`. This only moves the helper compile, see the note below the table. - -| Category | P0.2 | Control | P6 | Δ vs control | -|---|---:|---:|---:|---:| -| TOTAL RUNTIME | 2,654.8 | 2,662.4 | 1,284.1 | −51.8% | -| BUILD: solution generation & compile | 2,605.6 | 2,613.0 | 1,238.9 | −52.6% | -| … `python_kernel_compilation` | 2,260.4 | 2,273.7 | 903.2 | −60.3% | -| … … helper compile in the build's path (P0.2: serial; P6: `python_kernel_build_src_co_wait`) | 1,013.4 | 1,023.6 | 138.1 | −86.5% | -| … … `python_kernel_codegen` (calls) | 615.5 (855) | 607.4 (855) | 301.2 (213) | −50.4% | -| … … `python_kernel_write_assemble` | 253.8 | 264.1 | 236.0 | −10.6% | -| … … `python_kernel_bench_setup` | 132.2 | 132.6 | 4.5 | −96.6% | -| … … `python_kernel_bench_postprocess` | 105.2 | 103.9 | 106.0 | +2.0% | -| … … `python_kernel_validate` | 82.6 | 80.9 | 58.3 | −27.9% | -| … … `python_kernel_build_co` | 22.1 | 25.9 | 25.7 | −0.8% | -| … `python_solution_generation` | 321.6 | 319.0 | 314.2 | −1.5% | -| … Ductile GA search (Python) | 23.7 | 20.2 | 21.5 | +6.4% | -| SHARED: cache & solution files | 23.2 | 23.2 | 23.4 | +0.9% | -| RUN: benchmark client | 14.3 | 14.9 | 10.0 | −32.9% | -| Helper compile itself (`python_kernel_build_src_co`, overlapping total) | 1,013.4 | 1,023.6 | 393.0 | −61.6% | -| … `.compile` | 1,010.5 | 1,020.9 | 286.9 | −71.9% | -| … `.pch` (new; PCH built once per config here) | – | – | 98.7 | | -| Sum of per-config pytest wall times | 3,361 | 3,359 | 1,970 | −41.4% | - -Note on the helper compile: since P4.9 it runs in a background thread during codegen. The tree -therefore shows only the time the build then waits for it (`python_kernel_build_src_co_wait`), -and the compile itself is listed under "Overlapping Totals". In P0.2's logs it ran serially and -appears as untracked time in `python_kernel_compilation` (1,017.2 s in P0.2, 1,027.1 s in the -control). - -What the numbers show, by plan item: - -- **Helper compile (topic 6, P4.9 and P4.11–P4.15): 1,024 s → 138 s in the build's path.** Its - own time falls to 393 s: the bare-arch compile (P4.11), the PCH (P4.12), and chunks compiled in - parallel (P4.14). Most of what remains overlaps codegen (P4.9/P4.10). In this cold-cache pass, - building the PCH once per config costs 98.7 s of that, in the background. -- **Codegen (topic 7, P4.3–P4.8, P4.10): 607 → 301 s.** One pool per config (213 instead of 855 - `python_kernel_codegen` calls). The static-VGPR and overflow checks stop rejected kernels early - (P4.5/P4.6). The 19 configs that had `CpuThreads: 1` now generate in parallel (P4.8): −64% for - that set. `python_kernel_validate` −28% and `write_assemble` −11% probably follow from the - same changes (not measured separately). -- **Benchmark setup (topic 8, P4.1/P4.2): 133 → 4.5 s.** -- **Unchanged as expected:** solution generation (−1.5%), bench postprocess, solution/cache - files, Ductile GA search. -- **RUN: benchmark client −4.9 s** is the ductile configs' client runs during the build. P1's - client changes act there too (input preparation −3.8 s, as in P3). -- The untimed part per config (pytest start-up, imports, artifact packing) is the same on both - sides (697 against 686 s over 184 configs). So the wall-time gain (−41%) is smaller than the - timed gain (−52%). -- Largest configs (pytest s, control → P6): `subtile_mxfp4` 211 → 45, `dtl` 168 → 137, - `subtile_bf16` 121 → 19, `cgemm` 97 → 79, `bufferLoadStore0` 82 → 68, - `custom_mainloop_scheduling` 76 → 30, `plr_zero` 72 → 60, `spmm_f16_sb` 72 → 43, `zgemm` - 67 → 61, `fp16_tn` 60 → 53. No config is slower than its pair. - -### Configs with `LibraryLogic:`/`LibraryClient:` changes (P2.5/P2.6) - -`--build-only` skips both steps (`Tensile.py`: "Build-only mode: skipping LibraryLogic and -LibraryClient"), so these sections do not change the build-side work. Their YAML changes do -not explain any of the gain. The 25 configs whose sections changed between `2083e6254bf` and -`9f493d7a6c1` are reported separately above. All 25 lost their sections (P2.6): the 8 -`ductile/gfx950/*`, the 2 mxfp8 repro configs, the 8 `subtile_mxfp8_*` configs and the 7 gfx950 -`custom/*` configs. The 8 `subtile_mxfp8_*` configs also lost `CpuThreads: 1` (P4.8). -`prediction_bf16` gained both sections (P2.5); it is P6-only, so it is not in the 25. The set's -gain (−30.7%) is smaller than the rest's (−42.2%) because all 25 are small configs (6–19 s each -in the control), where the untimed per-config overhead (about 3.7 s) dominates. - -## Findings that bear on the build side - -- **P4.7/P4.8 regression** (above), fixed in `9f493d7a6c1`. The verifications of those steps - built no custom-kernel config. A unit-level case for a custom kernel now covers it (in the fix - commit). -- **The chunk cache (P4.15) gives no measurable wall-time gain** in either build: 11.5% chunk - hits in the CI-like build, 1.9% within cold single configs. The run without it was 1–3% faster - in both orders (within drift). Its benefit depends on configs that share many helpers but - miss the whole-file cache, which the common set rarely has. Also, chunk contents depend on - the pool budget (FOLLOWUPS P4.14), so a cache filled by a `-n 4` build (2 chunks per compile) - does not serve a serial build (4 chunks). Keeping it on is harmless. Whether to keep the - feature is a decision for the plan owner. -- **FOLLOWUPS P4.15** (a failed chunk-key computation still prints "0 of n chunks hit"): no key - computation failed in any P6 run, so the hit counts above are exact. -- **FOLLOWUPS P4.12** (`.tmp_*` entries of killed processes are never evicted; per-process temp - name in `HelperKernelCache`): no cache-related build failure occurred in the 4 split runs at - `9f493d7a6c1`, and the last run's cache had no `.tmp_*` left. (The earlier runs' caches were - cleared before they could be checked.) 3 xdist workers built the same PCH key at once, which is - the race the rename guards against. -- **FOLLOWUPS P4.14:** TensileCreateLibrary's (shipped library) helper compile is still in one - piece. P6 measures only the benchmark/test build path. -- **FOLLOWUPS P4.7/P4.9:** `DisableAsmComments` reaches the helper `Kernels.h` inconsistently in - serial codegen. Since P4.8 only pool-size-1 runs are affected, and no P6 run has pool size 1. -- The per-worker budget (P4.3) leaves cores idle at the end of a `-n 4` run once fewer than 4 - tests remain (see the split section). Not a regression: P0.2 oversubscribed instead. - -## Reproducing - -```bash -git worktree add --detach 9f493d7a6c1 -source /home/alvasile/venv/bin/activate -export TL=/projects/hipblaslt/tensilelite -export CLIENT= -export TENSILE_CACHE=/tensile TENSILELITE_CACHE=/tensilelite ART= -# split phases: run_split.sh runs the build phase, then the --use-cache run phase. The P6 number -# is the "END build" line of $OUT/times.txt; the run phase that follows is not part of P6. P6 -# also added -rP to the build command (to log the cache lines); -# TENSILE_DISABLE_HELPER_CHUNK_CACHE=1 for the chunk-cache-off runs -OUT= bash $TL/timing_runs/full_runs_gfx950/split/run_split.sh -# per-config cold build -ROCR_VISIBLE_DEVICES= OUT= SCRATCH= \ - bash $TL/timing_runs/full_runs_gfx950/per_config/run_per_config_build.sh -``` - -The P0.2 side used P0.2's worktree and client (`baseline.md`, "Reproducing"). The interleaved -driver, the per-config script copy and the analysis scripts are in -`/home/alvasile/timing_runs_raw/P6/scripts/` (`driver.sh`, `split_stats.py`, `cmp_pairs.py`, -`cmp_agg.py`). diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_build/split_build_outcomes.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_build/split_build_outcomes.txt deleted file mode 100644 index 64f1aee192e4..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_build/split_build_outcomes.txt +++ /dev/null @@ -1,572 +0,0 @@ -# Build-phase correctness after P4 + FIX1 (P6): outcome of every test in the gfx950 split build phase -# (split/run_split.sh build phase: --build-only, -n 4, cores 0-24, empty cache at start) at 9f493d7a6c1. -# One " " per line, sorted. 568 tests: 196 PASSED, 2 FAILED, 370 SKIPPED. Identical in -# all 4 runs at 9f493d7a6c1 (chunk cache on and off); differs from the P0.2 build phase only by prediction_bf16 (new). -tensilelite/Tests/common/flags/gfx942/test_build_only.py::test_compile PASSED -tensilelite/Tests/common/flags/gfx942/test_build_only.py::test_use_cache SKIPPED -tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_compile_and_benchmark SKIPPED -tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_use_cache SKIPPED -tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_use_cache_creates_cache_dir SKIPPED -tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_use_cache_different_params SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/amaxd/8rn_amaxd.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/amaxd/gfx1250/amax_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/benchmark_runs/bf16_tn_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/benchmark_runs/fp16_tn_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/benchmark_runs/fp32_tn.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/client/rotate_mode0.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/client/rotate_mode0_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/client/rotate_mode1.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/client/rotate_mode1_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/comm/gfx950/fused_a2a.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx1250/bbs_tn_dp.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx1250/bbs_tn_multi_size_single_kernel.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx1250/bbs_tn_streamK.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx1250/f32_tt_batched.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx1250/f8_tn_mx.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx1250/hhs_nt_multi_size.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx950/bf16_tn_dp.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx950/bf16_tn_multi_size_single_kernel.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx950/bf16_tn_streamK.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx950/f16_nt_multi_size.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx950/f32_tt_batched.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx950/f8_tn_mx_subtile.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx950/f8bf16_tn_cms.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/ductile/gfx950/xf32_nn.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/exception/f16_cf32_nan.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/exception/f16_cf32_nan_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/8rn_b_s.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/ag.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/agntab_coverage_gfx942.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/bf16_tn.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/bf16_tn_predict.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/bufferLoadStore0.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/cgemm.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/dot2_gfx942.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/dtl.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/dtv.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/dtv_gfx90a.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/f32_mac.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/f64.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/f64_gfx90a.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/f64_mac.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp16_HH_BHS_bf16mfma.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp16_tn.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp16_use_e.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp16fp32mix.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp32_nt.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp32_scalealphavec.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp32_tn.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp8bf8nss.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp8n.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp8n_rowwise.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp8n_use_e.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp8nfp16mix_fp8nss.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp8nfp16mix_hfp8ns.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp8nfp16mix_hhs.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/fp8nfp16mix_hhs_lsu4.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gemm_bias_strided_batched.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gemm_bias_strided_batched_broadcast.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gemm_bias_strided_batched_broadcast_gfx940.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gemm_bias_strided_batched_gfx940.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gemm_ck_gfx942.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx11/bf16_tn_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx11/fp16_HH_BHS_bf16mfma_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx11/fp16_tn_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx11/fp16_use_e_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx11/gemm_bias_strided_batched_broadcast_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx11/gemm_bias_strided_batched_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx11/gsuasb_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx11/i8_gsu_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/1024_vgpr_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/1024_vgpr_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/8r_b_s_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/agntab_coverage_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/agntab_coverage_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/b6f4ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/b6f4ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/b8b8s_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/b8b8s_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/b8f8ss_grvw1_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/b8hs_grvw1_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/bf16_CLS_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/bf16_CLS_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/bf16_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/bf16_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/bf16_tn_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/bf16_tn_gfx12_predict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/bf6_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/bf6_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/bf6_tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/bf6_tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/cgemm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/cgemm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/decouple_pgr_tdm_fuse_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f32_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f32_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f32_grid_limit_size.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f4_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f4_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f4b8ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f4b8ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f4f6ss_tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f4f6ss_tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f64_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f64_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f6_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f6_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f6_tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f6_tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f6b6ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f6b6ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f6b8ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f6b8ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8b6ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8b6ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8b8ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8b8ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8f4ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8f4ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8f8s_cls_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8f8s_cls_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8f8s_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8f8s_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8f8s_grvw1_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8f8s_pk8_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8f8s_pk8_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8f8s_sr_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/f8f8s_sr_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/fp16_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/fp16_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/fp16_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/fp16_use_e_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/fp16_use_e_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/fp4_tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/fp4_tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/fp8_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/fp8_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/fp8_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/fp8bf8ss_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/gemm_bias_strided_batched_broadcast_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/gemm_bias_strided_batched_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/gltr_bounds_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/gsu_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/gsu_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/gsuaad_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/gsuasb_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/i8_gsu_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/i8i8i_grvw1_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/i8ii_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/i8ii_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/i8ii_grvw1_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/interleave_alpha2_group_store_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/largeLds_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/largeLds_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/mxf4_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/mxf4_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/mxf6_tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/mxf6_tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/mxf6f4ss_tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/mxf6f4ss_tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/mxf8_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/mxf8_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/mxf8f4ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/mxf8f4ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/nt_th_nv_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/nt_th_nv_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/pgl_bound_check.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/rap_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/segment_b_rotate_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/segment_interleave_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/segment_interleave_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/stinky_sia4.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/stinky_sia4_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250-strict_bench.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250-strict_cluster.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250_bench.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250_cluster.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/tdm_general_batched_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/tdm_general_batched_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/tdm_multicast_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/tdm_multicast_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/tdm_split_gfx1250-strict_xfail.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/tdm_split_gfx1250_xfail.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/tdmpluslds_tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/tdmpluslds_tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/xfp32_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/xfp32_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/xfp32_inf_support_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/zgemm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx12/zgemm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx1250/cluster_entry_handoff.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/agntab_coverage_gfx950.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/bf16_cvt.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/custom_mainloop_scheduling.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/custom_mainloop_scheduling_tf32.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/custom_xfp32.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/dot2_gfx950.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/dtl.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/f16f8mix_ss.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/f16f8mix_ss_stoch.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/f8b8hs.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/f8f16mix_f8s.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/fp16_use_e_wide_store.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/fp8.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_bf16_tn_act.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_bf16_tn_act_groupgemm.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_fp32_tn_act.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_fp32_tn_act_groupgemm.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/general_wgm.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/i8_gsu_gfx950.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/lds160K.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/lds_tr.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mx32f4_tn.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mx32f8_tn.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_bf16_tn_act.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_bf16_tn_act_groupgemm.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_fp32_tn_act.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_fp32_tn_act_groupgemm.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_bf16_tn_act.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_bf16_tn_act_groupgemm.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_fp32_tn_act.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_fp32_tn_act_groupgemm.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mxfp8_residual_fail_classes_repro.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/mxfp8_verify_fail_repro.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/plr_zero.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/prediction_bf16.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/preloop_local_read_wait.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/ss_bss.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_bf16.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_fp8.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp4.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_bias_sav.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt192x256_plr1_wgm4.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt256x192_plr1_wgm1.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt256x320_plr0_wgm1.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt320x256_plr1_wgm4.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_multi_du_multi_partition.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_tail_loop.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_tail_loop_smoke.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gfx950/xfp32.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gsuasb.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/gsuc.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/hh_f8nhs.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/i8_gsu.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/i8_gsu_gfx940.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/icache_flush.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/largeMT.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/large_size.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/lsu_fnuz.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/lsu_i8.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/mbskPrefetchOpt.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/mix_cvt_after_ds_fnuz.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/nan_bounds_check_odd_padding.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/no_shadowinit.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/sia0_pgr.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/swizzleA.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/swizzleB.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/swizzle_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/tail_small_size.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/ulsgro1.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/use_beta_false.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/wgm.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/xfp32.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gemm/zgemm.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/fp16_gradient_activation.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/fp16_gradient_bias.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/fp8bf8nss_gradient_bias_b.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx11/fp16_gradient_activation_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx11/fp16_gradient_bias_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_bgrada_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_bgradb_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_bgradd_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_dgelu_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_bgrada_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_bgradb_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_bgradd_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_dgelu_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250/bbs_bgrada_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250/bbs_bgradb_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250/bbs_bgradd_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250/bbs_dgelu_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250/hhs_bgrada_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250/hhs_bgradb_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250/hhs_bgradd_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx1250/hhs_dgelu_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gradient/gfx950/bf16_gradient_bias.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/groupedgemm/gfx11/grouped_gemm_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/groupedgemm/gfx11/grouped_gemm_userargs_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/groupedgemm/grouped_gemm.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/groupedgemm/grouped_gemm_ck_gfx942.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/groupedgemm/grouped_gemm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/groupedgemm/grouped_gemm_userargs.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gsu/auto_gsu.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gsu/auto_gsu_zero_dim.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gsu/f32_gsu.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/gsu/gsu_mbsk.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sgemm_dual_fmac_asm.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sgemm_streamk_asm.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sgemm_xf32_asm.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/bf16_activation.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/f16_activation.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8bs.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8bs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8f8.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8f8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8hs.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8hs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_bf16.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_bf16_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f16.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f16_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8_ml.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8b8.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8b8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8bs.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8bs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8hs.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8hs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_fp16_ml1.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8bs.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8bs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8hs.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8hs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8is.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8is_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_all.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_f16_transposes.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_gl2prefetch.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_ldspadding.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_multicast.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/bf16_activation.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/f16_activation.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_b8.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_b8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_b8bs.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_b8bs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_b8f8.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_b8f8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_b8hs.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_b8hs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_bf16.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_bf16_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_f16.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_f16_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_f8.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_f8_ml.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_f8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_f8b8.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_f8b8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_f8bs.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_f8bs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_f8hs.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_f8hs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_fp16_ml1.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_gate.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_i8.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_i8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_i8bs.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_i8bs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_i8hs.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_i8hs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_i8is.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_i8is_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_all.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_f16_transposes.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_gl2prefetch.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_iteratemode.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_ldspadding.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_multicast.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_split_xfail.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/bf16_activation.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/bf16_gate_r.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/f8n_gate_r.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/fp16_activation.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/fp16_gate_r.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/i8_activation.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/libray_logic.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_b8f8n.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_b8f8n_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_bf16.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_bf16_sb.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_bf8n.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_bf8n_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_f8b8n.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_f8b8n_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_f8n.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_f8n_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi16.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi16_sb.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi32.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi32_sb.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_gsuc.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi16.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi16_sb.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi32.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi32_sb.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_i8bs.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_i8bs_sb.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_i8hs.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_i8hs_sb.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_i8is.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_i8is_sb.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_vw_lg_one.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/spmm_vw_lg_one_sb.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/use_sgpr_for_gro.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx94x/use_sgpr_for_gro_sb.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/bf16_gate_r.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/f8_gate_r.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/fp16_gate_r.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_b8.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_b8_sb.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_bf16.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_bf16_sb.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_dtl.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_f16.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_f16_sb.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_f8.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_f8_sb.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_i8.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_i8_sb.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_i8bs.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_i8hs.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_ldstr.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_vw_lg_one.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/sparse/gfx950/spmm_vw_lg_one_sb.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/data_parallel_static_mx32f4.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/data_parallel_static_sgemm.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx11/sk_bgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx11/sk_hgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx12/sk_bgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx12/sk_hgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/bf16_vector_epilogue_persistent.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf4.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf4_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf8.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf8_halfplr_tdm_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf8_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_bf16gemm_tdm_subtile.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_bgemm_tdm_split.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_bgemm_tdm_split_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_dynamic_hgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_dynamic_sgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_f8gemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_halfplr_f8gemm_tdm.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_halfplr_mxf8gemm_tdm.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_hgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_hybrid_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_explicit.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_pap_prefetchgl2.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_tdm.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_tdm_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf6b8gemm_tdm.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf6b8gemm_tdm_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8b6gemm_tdm.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8b6gemm_tdm_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8f4gemm_tdm.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8f4gemm_tdm_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_explicit.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_pap_prefetchgl2.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm_split.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm_split_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_sgemm_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_sgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_tdm_general_batched.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/core/sk_tdm_subtile_general_batched.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/data_parallel_cluster_multicast_boundaries.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf4_cluster_multicast_ext.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf4_pap_cluster_multicast_ext.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf8_cluster_multicast_ext.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf8_pap_cluster_multicast_ext.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/sk_hgemm_debug_modes.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/sk_mxf4gemm_tdm_cluster_ext.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/sk_mxf4gemm_tdm_ext.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx1250/sk_mxf4gemm_tdm_pap_ext.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/data_parallel_static_hgemm_use_e.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/data_parallel_static_sgemm_pap.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_bgemm_pap.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_hgemm_use_e.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_hybrid_xcc_mapping.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxf4gemm_pap.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_basic.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_batch_combo.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_full_coverage.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_streamk_combo.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_atomic_variants.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_basic_variants.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_batch_mi_combo.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_batch_sizes.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_comprehensive.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_debug_modes.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_full_coverage.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_activation_combo.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_basic.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_batch_combo.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_full_coverage.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_streamk_combo.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_matrix_instructions.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_problem_sizes.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_tree_reduction.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_xcc_mapping.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_debug_modes.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_dynamic_pap.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_hybrid_pap.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_pap.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_I8gemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_bgemm_div.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_bgemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_bgemm_sia1.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_cgemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_dgemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_dynamic.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_dynamic_mxfp4.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_dynamic_work_stealing.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_f8gemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_f8ngemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_hgemm_epilogues.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_hgemm_lsu.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_hgemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_hgemm_race.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_hybrid.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_hybrid_work_stealing.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_mx32f4_quick.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_mx32f8_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_queue_selfclean.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_reduction.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_sgemm_alpha0.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_sgemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/streamk/sk_zgemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_aiter_bf16.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_aiter_f4.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_ck.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_rocblas_gemv_bf16_tn_m1.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_rocblas_gemv_bf16_tn_m2.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_rocblas_gemv_bf16_tn_m4.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_rr.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_rr_f6.yaml] FAILED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_rr_mx.yaml] FAILED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_tensile_sk.yaml] SKIPPED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_triton_f4.yaml] PASSED -tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/custom/custom_wave_bf16.yaml] PASSED -tensilelite/Tests/common/test_config_helpers.py::test_base_skip_gfx1250_inert_on_strict_target PASSED -tensilelite/Tests/common/test_config_helpers.py::test_ffm_fail_inert_on_hardware PASSED -tensilelite/Tests/common/test_config_helpers.py::test_ffm_fail_inert_on_other_arch PASSED -tensilelite/Tests/common/test_config_helpers.py::test_ffm_fail_xfails_under_ffm PASSED -tensilelite/Tests/common/test_config_helpers.py::test_find_available_archs_does_not_map_the_retired_v0_name_to_the_base PASSED -tensilelite/Tests/common/test_config_helpers.py::test_find_available_archs_keeps_stepping_name PASSED -tensilelite/Tests/common/test_config_helpers.py::test_skip_gfx1250_strict_fires_on_strict_target PASSED -tensilelite/Tests/common/test_config_helpers.py::test_skip_gfx1250_strict_inert_on_base_target PASSED -tensilelite/Tests/common/test_config_helpers.py::test_unmarked_config_never_xfails_under_ffm PASSED diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_run.md b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_run.md deleted file mode 100644 index fd759fecee8e..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_run.md +++ /dev/null @@ -1,288 +0,0 @@ -# gfx950 run-side measurement after the client and harness changes (P3) - -Run-side measurement at the end of P2 of the implementation plan (`timing_runs/DECISIONS.md`, -P3), compared with the P0.2 baseline (`baseline.md`). It covers the client changes P1.1–P1.29 and -the Python/YAML/harness changes P2.1–P2.7. - -## What was measured - -| Item | Value | -|---|---| -| Commit | `3b4ec8e125d` (end of P2; detached worktree) | -| Client | `tensilelite-client` built from that commit, CMake cache identical to the P0.2 client's (paths normalised): Release, `GPU_TARGETS=gfx950`, `/usr/bin/amdclang++` (ROCm 7.2.1), msgpack only, `TENSILELITE_BUILD_TESTING=OFF`. sha256 `15f20fa8a2eb…`. | -| Baseline client | P0.2's (`2083e6254bf`, sha256 `4b2769f7f9a2…`), its worktree and artifacts, for the same-session control runs | -| Artifacts | Split build phase at `3b4ec8e125d` into a new directory, separate caches | -| Python | venv Python 3.12.3; TensileLite Python sources from each worktree; venv `rocisa` (unchanged) | -| Host | `smci355-1b112-b13-15` (as P0.2): 2× EPYC 9575F, 8× MI355X (gfx950), ROCm 7.2.1 | -| Cores / threads | `taskset -c 0-24`, `OMP_NUM_THREADS=25`, `OPENBLAS_NUM_THREADS=25` (as P0.2) | -| GPU | Per run, see below: the P0.2 GPU (`ROCR_VISIBLE_DEVICES=0`, PCI `75:00.0`) when idle, otherwise another idle MI355X (`ROCR` 5 = PCI `85:00.0` or 6 = `e5:00.0`, NUMA node 1) | -| Config list | The 185 configs that passed the split run phase: P0.2's 184 plus the new `gemm/gfx950/prediction_bf16` (P2.5) | - -Committed results: - -- `after_run/AGGREGATE_analysis.txt`: per-config run aggregate, P3 client, 185 logs. -- `after_run/common_test_outcomes.txt`: outcome of every test in the split run phase. - -Raw logs (not committed, as for P0.2) are in `/home/alvasile/timing_runs_raw/P3/`, gzipped: -the per-config logs of both clients, the split logs, the P0.2-client control aggregate and the -monitoring logs. - -## Conditions: why the comparison is same-session - -The host was shared throughout (2026-10-31 18:40 to 2026-11-01 14:30 UTC): - -- Another user's corpus run again pinned jobs to cores 0–8, 10–47 and 9 (the job described in - `baseline.md`), from 19:25 to about 03:00. It was gone during the per-config pass and the two - clean split pairs (no CPU growth of it in any of those runs). -- A root vLLM server held the measurement GPU (and rocm-smi GPUs 0–3, i.e. `ROCR` 0–3) at 100% / - 279 GB for hours at a time; other root jobs used the remaining GPUs in turns. -- `/` filled up twice (0 bytes free), so pytest temp dirs, caches and output were moved to - `/dev/shm` (RAM); this applied to both clients alike. - -An uncontended 1-hour window for a P0.2-style serial pass did not occur: a clean pass started at -03:43 got through 36 of 185 configs before the measurement GPU was taken again. So, as decided -during the run, the per-config pass was measured **interleaved**: for each config, the P0.2 -client (with P0.2's worktree and artifacts) and the P3 client ran back to back on the same idle -GPU, alternating which went first, with the same script and settings as P0.2 -(`run_per_config.sh`, serial, `TimingInstrumentation=True`, 25 pinned cores). A GPU was idle when -it showed < 5% use and < 2 GB VRAM twice; if another user's process took more than 1 GB on it -during a pair, the pair was re-run (up to three tries). So each config's two numbers share their -conditions, and the P0.2-client sum is a same-session control. Absolute numbers are not -comparable with the P0.2 pass on their own (see "Control"). - -- GPUs used per pair: `ROCR` 5 for 130 pairs, 0 for 46, 6 for 8. -- Other users' CPU on 0–24: no CPU growth of pinned corpus jobs during any run; unpinned root - processes (vLLM workers, downloads) were parked on idle cores of 0–24 at times, as in P0.2. -- Two pairs, `gemm/gfx950/subtile_mxfp4` and `subtile_mxfp8`, had another process start on the - GPU during every one of three tries; they are kept (both clients were exposed) and shown - separately below. -- The run script clears the caches at the start of each call. One call per config (to re-run single - pairs) therefore clears the helper-kernel cache before every config, while P0.2 cleared it once. - That costs both clients the same ~21 s of helper-kernel compiles (see "Control"). - -## Correctness: common-test pass/fail set - -Split run phase (`--use-cache`, `-n 4`) with the P3 client and artifacts: 568 tests, **195 -passed, 2 failed, 371 skipped**. Compared with `baseline_run/common_test_outcomes.txt`, the only -difference is the new config: - -- `test_config_run[gemm/gfx950/prediction_bf16.yaml]` PASSED (new in P2.5). - -Three P3 split runs (2026-10-31 23:44 on `ROCR` 0, 2026-11-01 12:20 and 13:59 on `ROCR` 5) gave -identical outcome lists. The split build phase also differs from P0.2's only by that config: -196 passed, 2 failed (the known `custom_rr_f6`/`custom_rr_mx` build failures), 370 skipped. - -Changes that were intended and do not change an outcome: - -- P2.6 removed the `LibraryLogic:`/`LibraryClient:` sections of the 14 `ductile/*`, the 10 gfx950 - mxfp8 configs and the 12 `custom/*` configs. Of those, the ones that run on gfx950 all pass: - the 8 `ductile/gfx950/*`, the 10 mxfp8 configs and 7 `custom/*` (`aiter_bf16`, `aiter_f4`, - `rocblas_gemv_bf16_tn_m1/m2/m4`, `triton_f4`, `wave_bf16`); their run phase no longer writes - logic files or runs the library client. The 6 `ductile/gfx1250/*` and `custom_ck`, - `custom_rr`, `custom_tensile_sk` are skipped on gfx950, and `custom_rr_f6`/`custom_rr_mx` are - the 2 known build-phase failures, all as in P0.2. -- P1.22 checks the Synchronizer after every run iteration. `streamk/gfx950/sk_sgemm_debug_modes` - omits Stream-K reduction steps on purpose, so it sets `CheckSynchronizer: False`; it passes. -- P2.3 runs the client at `ClientLogLevel=0` and P2.4 changed `gemm/gfx950/xfp32`'s warmups: - every gfx950 config passes as before. P2.7 set `SyncsPerBenchmark: 0` in `custom/*`: the same - 7 gfx950 custom configs pass; the other 5 are skipped or fail as in P0.2 (above). - -The P0.2 client's split runs in this session: identical to the baseline on a free GPU -(2026-11-01 12:36 and 13:17). On 2026-10-31 22:31, while vLLM shared the measurement GPU, -`streamk/gfx950/sk_sgemm_hybrid_pap` **failed validation** with the P0.2 client (1.39 M of -8.4 M values wrong, 67 s instead of 2–4 s). It passed in every other run of either client. This -is a pre-existing sensitivity of that Stream-K config to sharing the GPU with another process, -not a P3 change; it is listed under "Findings". - -## Uninstrumented split run phase (`-n 4`) - -Same command as `split/run_split.sh`'s run phase, both clients back to back on one GPU: - -| Pair | GPU | Conditions | P0.2 client | P3 client | Δ | -|---|---|---|---:|---:|---:| -| A–B, 2026-10-31 22:31 / 23:44 | `ROCR` 0 | corpus job on 0–24 throughout; vLLM on the GPU for the last ~28 min of the P0.2 run; during the P3 run one 30-s monitor sample showed another process with 13.8 GB on the GPU | 3,115 s | 966 s | not comparable | -| B–A, 2026-11-01 12:20 / 12:36 | `ROCR` 5 | clean (no corpus CPU, GPU free) | 2,445 s | 941 s | **−61.5%** | -| A–B, 2026-11-01 13:17 / 13:59 | `ROCR` 5 | clean | 2,499 s | 943 s | **−62.3%** | - -The P0.2 split run (2,950 s, contended) in `baseline.md` is an upper bound; the two clean -same-session pairs (ABBA order) are the comparison: P0.2 client 2,445–2,499 s, P3 client -941–943 s, **about −62%**. The run-phase critical path changed (B–A pair): the slowest P0.2 -tests were `subtile_mxfp4` (561 s), `spmm_f16_sb` (377 s), `sk_dynamic_mxfp4` (336 s), `spmm_bf16` -(332 s) and `sk_I8gemm_quick` (320 s); with P3 they are `subtile_mxfp4` (163 s), -`custom_mainloop_scheduling` (159 s), `dtl` (128 s), `plr_zero` (105 s), `bufferLoadStore0` -(101 s). The call-phase per-test sum fell from 8,929 s to 3,389 s (−62%). (With `-n 4`, four -25-thread OpenMP clients share 25 cores, so per-test times are larger than in the serial pass.) - -## Run side (per-config, instrumented) - -Seconds from "TIMING BY PURPOSE", summed over the 184 configs common to both (P3's -`prediction_bf16` adds 3.9 s and is in the committed aggregate only). "P0.2" is the committed -baseline pass, re-aggregated with the analyzer at `3b4ec8e125d` (byte-identical to the committed -`baseline_run/AGGREGATE_analysis.txt`, so analyzer changes contribute nothing). "Control" is the -P0.2 client in the interleaved pass. - -| Category | P0.2 | Control | P3 | Δ vs P0.2 | Δ vs control | -|---|---:|---:|---:|---:|---:| -| TOTAL RUNTIME | 2,401.1 | 2,655.0 | 1,024.5 | −57.3% | −61.4% | -| RUN: benchmark client | 2,284.9 | 2,519.0 | 887.7 | −61.1% | −64.8% | -| Reference GEMM (CPU) | 1,044.4 | 1,138.2 | 227.6 | −78.2% | −80.0% | -| … `solve_cpu_slow` | 689.0 | 740.1 | 0.6 | −99.9% | −99.9% | -| … `solve_cpu_fast` | 198.9 | 229.6 | 226.9 | +14.1% | −1.2% | -| … per-solution reference re-runs | 156.3 | 168.3 | – | −100% | −100% | -| Input data preparation | 500.7 | 582.7 | 122.5 | −75.5% | −79.0% | -| … `gpu_input_reset` | 298.8 | 328.7 | 46.1 | −84.6% | −86.0% | -| … `cpu_data_init` | 105.0 | 151.1 | 52.4 | −50.1% | −65.3% | -| … MX per-solution input (`DataInitialization`) | 54.9 | 61.4 | 3.1 | −94.3% | −94.9% | -| … `gpu_input_preparation` | 41.3 | 40.7 | 20.4 | −50.6% | −49.9% | -| Validation | 329.1 | 341.6 | 112.5 | −65.8% | −67.1% | -| … Synchronizer check | 165.4 | 169.9 | 11.8 | −92.9% | −93.1% | -| … `ReferenceValidator` | 163.3 | 171.3 | 100.3 | −38.5% | −41.4% | -| Client launch (startup, setup & teardown) | 289.8 | 316.3 | 284.4 | −1.8% | −10.1% | -| Per-solution bookkeeping & reporting | 54.8 | 57.4 | 13.7 | −75.0% | −76.2% | -| Kernel launch & execution | 35.7 | 39.3 | 126.0 | +252.7% | +220.2% | -| … `warmup_first_sync` (new) | – | – | 93.9 | | | -| Library client (Python) | 24.6 | 37.3 | – | −100% | −100% | -| Library logic (Python) | 5.8 | 6.3 | 1.0 | −81.9% | −83.4% | -| BUILD: solution generation & compile | 63.1 | 85.4 | 85.6 | +35.8% | +0.3% | -| MEASUREMENT OVERHEAD (instrumentation) | 52.8 | 50.2 | 50.1 | −5.1% | −0.2% | -| Sum of per-config pytest wall times | 2,942 | 3,212 | 1,571 | −46.6% | −51.1% | - -Over the 182 pairs without GPU interference the pytest sums are 2,917 s (control) and 1,394 s -(P3), −52.2%. The two GPU-shared pairs: `subtile_mxfp8` 203 → 109 s, `subtile_mxfp4` 92 → 68 s. - -Per-config pytest seconds (P0.2 / control / P3) of the largest configs: `sk_I8gemm_quick` -311 / 323 / 6, `sk_f8gemm_quick` 262 / 295 / 12, `spmm_f16_sb` 124 / 133 / 44, `plr_zero` -94 / 102 / 52, `dtl` 82 / 90 / 59, `fp16_tn` 87 / 87 / 43, `bufferLoadStore0` 81 / 86 / 60, -`mxfp4_mxfp4_bf16_tn_act` 31 / 38 / 4, `xfp32` 62 / 66 / 24. No config got slower by more -than 2 s against its pair (ten configs are 1–2 s slower, all with 3–16 s runs). - -What the numbers show: - -- **The work is the same.** The main call counts are identical to P0.2 (28,596 - `cpu_reference_gemm`, 2,355,016 solutions, 2,106,835 `gpu_input_reset`, 855 client starts). - Counts changed only where an item changed them on purpose: 10 library-client invocations fewer - (P2.6), 592 instead of 1,947 slow-path reference calls (the rest moved to the fast path, - P1.2–P1.11), and the renamed/new timers of P1.19–P1.23. -- **CPU reference (P1.1–P1.11):** the slow path went from 740 s (control; 689 s in P0.2) to - 0.6 s. The fast path took over those problems at about no net cost: 1,355 more calls (26,656 → - 28,011) in 226.9 s against the control's 229.6 s (−1.2%; +28 s against the P0.2 pass, which ran - faster than the control). The per-solution re-runs of the reference (168 s in the control, - 156 s in P0.2) are gone: they were caused by re-uploaded inputs, which P1.16–P1.19 skip. -- **Input preparation (P1.13–P1.19):** `gpu_input_reset_copy_valid_to_gpu` 175 s → 0.1 s, - batched-pointer uploads 93 → 19 s, MX regeneration 50 → 1.5 s, `cpu_data_init` −50%. -- **Validation (P1.20–P1.23):** the Synchronizer check is a GPU OR-reduction now (165 → 12 s); - `validate_gpu_readback` 94 → 36 s. -- **Bookkeeping (P1.24–P1.29, P2.3):** −75%. -- **Client launch is unchanged** (`hip_initialization` 128 s, `process_exit` 68 s for 855 - client starts): no item of the plan targeted it. It is now the largest category (28% of the - P3 total). - -## Moved as expected, or for a known reason - -- **Kernel launch & execution +87 s vs the control (+90 s vs P0.2)** is attribution, as the - plan expected: P1.20 times the wait for the first warmup's completion in its own timer, - `warmup_first_sync` (93.9 s). Before, - that GPU time was hidden in whichever blocking call came next (`validate_gpu_readback`, the - Synchronizer readback, input uploads), which is part of why those dropped by more than their - own speedups. -- **BUILD +22 s** against P0.2, but equal for both clients (85.4 / 85.6 s): the interleaved pass - clears the helper-kernel cache before every config (see "Conditions"); P0.2 cleared it once. - `python_kernel_compilation` 43.5 → 65 s in both. -- **Library client −24.6 s, library logic −4.8 s** (P2.6), and `flush_timing_buffer` −10 calls. - -## Control: P0.2 client in this session versus the P0.2 pass - -The same client and artifacts took 10.6% longer in total than in the P0.2 pass (2,655 vs -2,401 s; pytest sums 3,212 vs 2,942 s). P0.2's clean-repeat spread was about 1%, so the session -was slower. The difference is: - -- BUILD +22 s: the cache clearing above. -- Library client +12.7 s (`python_library_client_create_library`, Python). -- CPU reference +94 s (+9%), `cpu_data_init` +46 s, `process_exit` +18 s, input reset +30 s. - Most of `cpu_data_init` is one config, `sk_bgemm_div` (one 2.35·10⁹-element tensor): 32.7 s - instead of 6.1 s (P3 client: 3.7 s). These are host-memory-heavy phases. During the pass - `/dev/shm` held 1.2–1.3 TB of other users' data in RAM, and 130 pairs ran on a GPU on the - other NUMA node. Either may explain it, but neither was isolated. - -So the deltas against the control are the like-for-like ones. The deltas against P0.2 are -smaller, because the control ran slower than P0.2 did. - -## P1.12 replay (topic 1, slow-path reasons) - -After P1.2–P1.11 (`44cd1a416fe`), the 59 slow-path YAMLs of the investigation were replayed with -an uncommitted client that logs each reference call's path and reason (rule 8). **All 59 pass.** -The reference calls are the same as in the investigation's replay (2,517). Slow-path calls went -from 1,966 (546.6 s) to 593 (0.53 s), and fast-path calls from 551 to 1,924. The remaining slow -calls are the expected ones: `NumElementsToValidate: 128` (sparse validation, 569 calls in 20 -YAMLs) and `scaleCD` (24 calls, `gemm/gfx950/f8f16mix_f8s`). No other reason occurred. The -per-config pass here is consistent with it: 592 slow-path calls, 0.6 s. - -## Findings that bear on the run side (not changed by the plan) - -Decisions needed: - -- **P2.5, option A (user decision).** With `--best-solution`, the library client silently falls back - to library solution 0 when Origami selection returns nothing, and exits 0. A client option that - fails on an empty selection (written only for the `LibraryClient` step) would let the new - `prediction_bf16` config detect selection regressions. Not implemented; the plan said to stop - and ask. -- **P1.28, `sync-after-warmups` default (user decision).** The client defaults - `sync-after-warmups` to true, and `ClientWriter.py` never writes it. So P1.28's warmup-event - drop never triggers in Tensile-generated runs. Measured on `dtl` with `sync-after-warmups=False`: - −6.7 s (−13%) uninstrumented wall per pass, and 18,872 → 0 `hipEvent*` calls on one ini. Options: - `ClientWriter` writes `sync-after-warmups=False` for correctness-only runs, or the client stops - treating that sync as an event reader when nothing is timed. - -Pre-existing issues (candidates for a deferred list): - -- `sk_sgemm_hybrid_pap` failed validation once with the P0.2 client while another process - used the GPU (above). -- `cpu-gemm-driver --bias` with `--batchCount > 1` fails on both reference paths: the driver - registers the bias with batch stride m but allocates m elements, so batches > 0 read past the - buffer (test infrastructure only; no CTest combines them; P1.9). -- Client launch (`hip_initialization`, `process_exit`, `factory_hip_get_device_properties`) is - now 28% of the instrumented run total and was not in scope. -- GuardPageAll configs (`bufferLoadStore0`) re-upload batched-pointer arrays every run, because - the Front/Back start pointers differ; one cached array per bounds-check mode would recover - most of the remaining `gpu_input_reset_batched_inputs` (P1.18). -- `f8f16mix_f8s` and 3 calls of `use_beta_false` stay on the slow path (`scaleCD`, P1.7/P1.12), - and the slow path applies ScaleAB differently from the fast path when a Half operand is - stored wider than its F8 compute type (P1.7). -- The slow path's MX K tail is wrong for K not a multiple of the MX block; P1.11 moved those - problems to the fast path, and the slow path is unchanged (P1.11). -- Slow-path bias-gradient error handling (missing `throw`, double `free`) for batched source - shapes (P1.10). -- `generateMXInput`: the K tail in the re-arranged orientations is not deterministic, and neither - is `norm_dist` between calls (P1.13). `rocm_random` shares one `std::mt19937` across OpenMP - threads (data race, P1.15). Trig init takes an `int` index, which wraps above 2³¹ elements - (`sk_bgemm_div`, P1.14). -- The MX GFX950 scale swizzle with a padded leading dimension (P1.19). MX configs crash in NaN - and GuardPage modes (P1.19). Grouped GEMM with bounds checking is broken (P1.17). None of these - is used by a common config. -- Client: `selection-only=True` segfaults (P1.25); `max-enqueues-per-sync=0` hangs (P1.27); the - device-lost path exits via SIGABRT (P1.20). -- `ductile/gfx950/f8_tn_mx_subtile` sets `NumElementsToValidate: 0` but still validates 128 - elements (P1.12). - -## Reproducing - -```bash -git worktree add --detach /tmp/tl_plan/P3/wt 3b4ec8e125d -source /home/alvasile/venv/bin/activate -cmake --preset tensilelite -S /tmp/tl_plan/P3/wt/projects/hipblaslt -B /tmp/tl_plan/P3/build_client \ - -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DGPU_TARGETS=gfx950 \ - -DCMAKE_CXX_COMPILER=/usr/bin/amdclang++ -DCMAKE_C_COMPILER=/opt/rocm/bin/amdclang \ - -DHIPBLASLT_ENABLE_YAML=OFF -DTENSILELITE_ENABLE_CLIENT=ON -DTENSILELITE_BUILD_TESTING=OFF \ - -DTENSILELITE_CLIENT_ENABLE_ROCPROFSDK=OFF -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -taskset -c 128-191 cmake --build /tmp/tl_plan/P3/build_client -j64 --target tensilelite-client -export TL=/tmp/tl_plan/P3/wt/projects/hipblaslt/tensilelite -export CLIENT=/tmp/tl_plan/P3/build_client/tensilelite/client/tensilelite-client -export ART=/tmp/tl_plan/P3/artifacts TENSILE_CACHE=/tensile TENSILELITE_CACHE=/tensilelite -OUT= bash $TL/timing_runs/full_runs_gfx950/split/run_split.sh # artifacts + split run -OUT= KEEP_CACHE=1 bash $TL/timing_runs/full_runs_gfx950/per_config/run_per_config.sh -``` - -The interleaved pass ran `run_per_config.sh` once per config and client, with -`ROCR_VISIBLE_DEVICES` set to the pair's GPU and `TMPDIR`, caches and `OUT` under `/dev/shm` -(driver scripts and the per-config script copy are kept with the raw logs in -`/home/alvasile/timing_runs_raw/P3/`, see its `README.txt`). diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_run/common_test_outcomes.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_run/common_test_outcomes.txt deleted file mode 100644 index da4c9268c8d2..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/after_run/common_test_outcomes.txt +++ /dev/null @@ -1,572 +0,0 @@ -# Correctness after P2 (P3): outcome of every test in the gfx950 split run phase -# (split/run_split.sh run phase: --use-cache, -n 4, cores 0-24) at 3b4ec8e125d, P3 client and artifacts. -# One " " per line, sorted. 568 tests: 195 PASSED, 2 FAILED, 371 SKIPPED. -# Differs from ../baseline_run/common_test_outcomes.txt only by prediction_bf16 (new). See ../after_run.md. -tensilelite/Tests/common/flags/gfx942/test_build_only.py::test_compile PASSED -tensilelite/Tests/common/flags/gfx942/test_build_only.py::test_use_cache SKIPPED -tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_compile_and_benchmark SKIPPED -tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_use_cache SKIPPED -tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_use_cache_creates_cache_dir SKIPPED -tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_use_cache_different_params SKIPPED -tensilelite/Tests/common/test_config_helpers.py::test_base_skip_gfx1250_inert_on_strict_target PASSED -tensilelite/Tests/common/test_config_helpers.py::test_ffm_fail_inert_on_hardware PASSED -tensilelite/Tests/common/test_config_helpers.py::test_ffm_fail_inert_on_other_arch PASSED -tensilelite/Tests/common/test_config_helpers.py::test_ffm_fail_xfails_under_ffm PASSED -tensilelite/Tests/common/test_config_helpers.py::test_find_available_archs_does_not_map_the_retired_v0_name_to_the_base PASSED -tensilelite/Tests/common/test_config_helpers.py::test_find_available_archs_keeps_stepping_name PASSED -tensilelite/Tests/common/test_config_helpers.py::test_skip_gfx1250_strict_fires_on_strict_target PASSED -tensilelite/Tests/common/test_config_helpers.py::test_skip_gfx1250_strict_inert_on_base_target PASSED -tensilelite/Tests/common/test_config_helpers.py::test_unmarked_config_never_xfails_under_ffm PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/amaxd/8rn_amaxd.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/amaxd/gfx1250/amax_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/benchmark_runs/bf16_tn_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/benchmark_runs/fp16_tn_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/benchmark_runs/fp32_tn.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/client/rotate_mode0.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/client/rotate_mode0_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/client/rotate_mode1.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/client/rotate_mode1_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/comm/gfx950/fused_a2a.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/bbs_tn_dp.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/bbs_tn_multi_size_single_kernel.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/bbs_tn_streamK.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/f32_tt_batched.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/f8_tn_mx.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/hhs_nt_multi_size.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/bf16_tn_dp.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/bf16_tn_multi_size_single_kernel.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/bf16_tn_streamK.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/f16_nt_multi_size.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/f32_tt_batched.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/f8_tn_mx_subtile.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/f8bf16_tn_cms.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/xf32_nn.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/exception/f16_cf32_nan.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/exception/f16_cf32_nan_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/8rn_b_s.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/ag.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/agntab_coverage_gfx942.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/bf16_tn.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/bf16_tn_predict.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/bufferLoadStore0.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/cgemm.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/dot2_gfx942.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/dtl.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/dtv.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/dtv_gfx90a.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/f32_mac.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/f64.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/f64_gfx90a.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/f64_mac.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp16_HH_BHS_bf16mfma.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp16_tn.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp16_use_e.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp16fp32mix.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp32_nt.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp32_scalealphavec.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp32_tn.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8bf8nss.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8n.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8n_rowwise.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8n_use_e.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8nfp16mix_fp8nss.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8nfp16mix_hfp8ns.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8nfp16mix_hhs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8nfp16mix_hhs_lsu4.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gemm_bias_strided_batched.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gemm_bias_strided_batched_broadcast.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gemm_bias_strided_batched_broadcast_gfx940.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gemm_bias_strided_batched_gfx940.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gemm_ck_gfx942.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/bf16_tn_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/fp16_HH_BHS_bf16mfma_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/fp16_tn_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/fp16_use_e_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/gemm_bias_strided_batched_broadcast_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/gemm_bias_strided_batched_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/gsuasb_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/i8_gsu_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/1024_vgpr_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/1024_vgpr_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/8r_b_s_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/agntab_coverage_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/agntab_coverage_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b6f4ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b6f4ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b8b8s_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b8b8s_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b8f8ss_grvw1_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b8hs_grvw1_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_CLS_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_CLS_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_tn_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_tn_gfx12_predict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf6_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf6_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf6_tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf6_tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/cgemm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/cgemm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/decouple_pgr_tdm_fuse_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f32_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f32_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f32_grid_limit_size.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4b8ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4b8ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4f6ss_tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4f6ss_tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f64_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f64_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6_tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6_tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6b6ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6b6ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6b8ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6b8ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8b6ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8b6ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8b8ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8b8ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f4ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f4ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_cls_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_cls_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_grvw1_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_pk8_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_pk8_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_sr_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_sr_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp16_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp16_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp16_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp16_use_e_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp16_use_e_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp4_tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp4_tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp8_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp8_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp8_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp8bf8ss_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gemm_bias_strided_batched_broadcast_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gemm_bias_strided_batched_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gltr_bounds_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gsu_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gsu_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gsuaad_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gsuasb_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/i8_gsu_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/i8i8i_grvw1_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/i8ii_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/i8ii_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/i8ii_grvw1_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/interleave_alpha2_group_store_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/largeLds_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/largeLds_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf4_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf4_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf6_tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf6_tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf6f4ss_tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf6f4ss_tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf8_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf8_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf8f4ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf8f4ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/nt_th_nv_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/nt_th_nv_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/pgl_bound_check.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/rap_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/segment_b_rotate_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/segment_interleave_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/segment_interleave_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/stinky_sia4.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/stinky_sia4_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250-strict_bench.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250-strict_cluster.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250_bench.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250_cluster.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_general_batched_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_general_batched_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_multicast_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_multicast_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_split_gfx1250-strict_xfail.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_split_gfx1250_xfail.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdmpluslds_tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdmpluslds_tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/xfp32_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/xfp32_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/xfp32_inf_support_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/zgemm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/zgemm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx1250/cluster_entry_handoff.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/agntab_coverage_gfx950.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/bf16_cvt.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/custom_mainloop_scheduling.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/custom_mainloop_scheduling_tf32.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/custom_xfp32.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/dot2_gfx950.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/dtl.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/f16f8mix_ss.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/f16f8mix_ss_stoch.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/f8b8hs.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/f8f16mix_f8s.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp16_use_e_wide_store.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp8.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_bf16_tn_act.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_bf16_tn_act_groupgemm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_fp32_tn_act.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_fp32_tn_act_groupgemm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/general_wgm.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/i8_gsu_gfx950.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/lds160K.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/lds_tr.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mx32f4_tn.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mx32f8_tn.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_bf16_tn_act.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_bf16_tn_act_groupgemm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_fp32_tn_act.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_fp32_tn_act_groupgemm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_bf16_tn_act.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_bf16_tn_act_groupgemm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_fp32_tn_act.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_fp32_tn_act_groupgemm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp8_residual_fail_classes_repro.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp8_verify_fail_repro.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/plr_zero.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/prediction_bf16.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/preloop_local_read_wait.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/ss_bss.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_bf16.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_fp8.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp4.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_bias_sav.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt192x256_plr1_wgm4.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt256x192_plr1_wgm1.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt256x320_plr0_wgm1.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt320x256_plr1_wgm4.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_multi_du_multi_partition.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_tail_loop.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_tail_loop_smoke.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/xfp32.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gsuasb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gsuc.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/hh_f8nhs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/i8_gsu.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/i8_gsu_gfx940.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/icache_flush.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/largeMT.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/large_size.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/lsu_fnuz.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/lsu_i8.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/mbskPrefetchOpt.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/mix_cvt_after_ds_fnuz.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/nan_bounds_check_odd_padding.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/no_shadowinit.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/sia0_pgr.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/swizzleA.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/swizzleB.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/swizzle_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/tail_small_size.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/ulsgro1.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/use_beta_false.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/wgm.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/xfp32.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/zgemm.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/fp16_gradient_activation.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/fp16_gradient_bias.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/fp8bf8nss_gradient_bias_b.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx11/fp16_gradient_activation_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx11/fp16_gradient_bias_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_bgrada_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_bgradb_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_bgradd_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_dgelu_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_bgrada_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_bgradb_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_bgradd_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_dgelu_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/bbs_bgrada_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/bbs_bgradb_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/bbs_bgradd_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/bbs_dgelu_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/hhs_bgrada_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/hhs_bgradb_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/hhs_bgradd_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/hhs_dgelu_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx950/bf16_gradient_bias.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/gfx11/grouped_gemm_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/gfx11/grouped_gemm_userargs_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/grouped_gemm.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/grouped_gemm_ck_gfx942.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/grouped_gemm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/grouped_gemm_userargs.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gsu/auto_gsu.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gsu/auto_gsu_zero_dim.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gsu/f32_gsu.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gsu/gsu_mbsk.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sgemm_dual_fmac_asm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sgemm_streamk_asm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sgemm_xf32_asm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/bf16_activation.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/f16_activation.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8bs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8bs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8f8.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8f8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8hs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8hs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_bf16.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_bf16_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f16.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f16_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8_ml.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8b8.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8b8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8bs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8bs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8hs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8hs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_fp16_ml1.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8bs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8bs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8hs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8hs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8is.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8is_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_all.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_f16_transposes.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_gl2prefetch.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_ldspadding.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_multicast.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/bf16_activation.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/f16_activation.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8bs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8bs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8f8.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8f8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8hs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8hs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_bf16.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_bf16_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f16.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f16_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8_ml.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8b8.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8b8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8bs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8bs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8hs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8hs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_fp16_ml1.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_gate.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8bs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8bs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8hs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8hs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8is.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8is_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_all.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_f16_transposes.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_gl2prefetch.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_iteratemode.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_ldspadding.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_multicast.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_split_xfail.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/bf16_activation.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/bf16_gate_r.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/f8n_gate_r.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/fp16_activation.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/fp16_gate_r.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/i8_activation.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/libray_logic.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_b8f8n.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_b8f8n_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_bf16.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_bf16_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_bf8n.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_bf8n_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_f8b8n.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_f8b8n_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_f8n.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_f8n_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi16.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi16_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi32.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi32_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_gsuc.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi16.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi16_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi32.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi32_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8bs.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8bs_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8hs.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8hs_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8is.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8is_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_vw_lg_one.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_vw_lg_one_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/use_sgpr_for_gro.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/use_sgpr_for_gro_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/bf16_gate_r.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/f8_gate_r.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/fp16_gate_r.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_b8.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_b8_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_bf16.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_bf16_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_dtl.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_f16.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_f16_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_f8.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_f8_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_i8.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_i8_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_i8bs.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_i8hs.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_ldstr.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_vw_lg_one.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_vw_lg_one_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/data_parallel_static_mx32f4.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/data_parallel_static_sgemm.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx11/sk_bgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx11/sk_hgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx12/sk_bgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx12/sk_hgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/bf16_vector_epilogue_persistent.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf4.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf4_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf8.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf8_halfplr_tdm_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf8_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_bf16gemm_tdm_subtile.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_bgemm_tdm_split.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_bgemm_tdm_split_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_dynamic_hgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_dynamic_sgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_f8gemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_halfplr_f8gemm_tdm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_halfplr_mxf8gemm_tdm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_hgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_hybrid_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_explicit.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_pap_prefetchgl2.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_tdm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_tdm_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf6b8gemm_tdm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf6b8gemm_tdm_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8b6gemm_tdm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8b6gemm_tdm_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8f4gemm_tdm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8f4gemm_tdm_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_explicit.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_pap_prefetchgl2.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm_split.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm_split_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_sgemm_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_sgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_tdm_general_batched.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_tdm_subtile_general_batched.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/data_parallel_cluster_multicast_boundaries.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf4_cluster_multicast_ext.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf4_pap_cluster_multicast_ext.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf8_cluster_multicast_ext.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf8_pap_cluster_multicast_ext.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/sk_hgemm_debug_modes.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/sk_mxf4gemm_tdm_cluster_ext.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/sk_mxf4gemm_tdm_ext.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/sk_mxf4gemm_tdm_pap_ext.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/data_parallel_static_hgemm_use_e.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/data_parallel_static_sgemm_pap.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_bgemm_pap.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_hgemm_use_e.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_hybrid_xcc_mapping.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxf4gemm_pap.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_basic.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_batch_combo.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_full_coverage.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_streamk_combo.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_atomic_variants.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_basic_variants.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_batch_mi_combo.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_batch_sizes.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_comprehensive.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_debug_modes.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_full_coverage.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_activation_combo.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_basic.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_batch_combo.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_full_coverage.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_streamk_combo.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_matrix_instructions.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_problem_sizes.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_tree_reduction.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_xcc_mapping.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_debug_modes.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_dynamic_pap.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_hybrid_pap.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_pap.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_I8gemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_bgemm_div.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_bgemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_bgemm_sia1.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_cgemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_dgemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_dynamic.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_dynamic_mxfp4.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_dynamic_work_stealing.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_f8gemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_f8ngemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hgemm_epilogues.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hgemm_lsu.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hgemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hgemm_race.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hybrid.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hybrid_work_stealing.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_mx32f4_quick.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_mx32f8_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_queue_selfclean.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_reduction.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_sgemm_alpha0.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_sgemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_zgemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_aiter_bf16.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_aiter_f4.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_ck.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rocblas_gemv_bf16_tn_m1.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rocblas_gemv_bf16_tn_m2.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rocblas_gemv_bf16_tn_m4.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rr.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rr_f6.yaml] FAILED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rr_mx.yaml] FAILED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_tensile_sk.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_triton_f4.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_wave_bf16.yaml] PASSED diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline.md b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline.md deleted file mode 100644 index 68ec5d189da1..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline.md +++ /dev/null @@ -1,268 +0,0 @@ -# gfx950 timing baseline at the starting commit (P0.2) - -Re-measurement of the gfx950 common tests at the starting commit of the implementation plan -(`timing_runs/DECISIONS.md`, P0.2). The earlier committed baselines (`per_config_rerun_2`, -`per_config_build`) were measured with a client built at `4c6426cea6`. Every later measurement -(P3 `after_run/`, P6 `after_build/`) is compared with the numbers here, not with those. - -## What was measured - -| Item | Value | -|---|---| -| Commit | `2083e6254bf` (detached worktree). Code-identical to the starting commit `55e29436e9a`: the only difference is P0.0's measurement-script parametrization. Client, library and Python sources are also identical to `4c6426cea6`; only `scripts/{analyze,aggregate}_timing.py` and their tests changed since then. | -| Client | `tensilelite-client` built from that commit, CMake cache identical to `/tmp/tl_client_msgpack` (the `4c6426cea6` baseline client): Release, `GPU_TARGETS=gfx950`, `/usr/bin/amdclang++` (AMD clang 22.0.0git, ROCm 7.2.1), msgpack only (`HIPBLASLT_ENABLE_YAML=OFF`), `TENSILELITE_BUILD_TESTING=OFF`. sha256 `4b2769f7f9a2…`. | -| Python | venv Python 3.12.3; TensileLite Python sources from the worktree; `rocisa` is the venv's installed build (2026-10-19, unchanged). | -| Host | `smci355-1b112-b13-15`: 2× AMD EPYC 9575F (64 cores each, SMT 2), 8× AMD Instinct MI355X (gfx950), Linux 6.8.0-38, ROCm 7.2.1, amdgpu 6.16.13. Shared by ~45 users. | -| Cores / threads | `taskset -c 0-24` (25 physical cores, NUMA node 0), `OMP_NUM_THREADS=25`, `OPENBLAS_NUM_THREADS=25` | -| GPU | `ROCR_VISIBLE_DEVICES=0` = KFD node 2 = PCI `0000:75:00.0`, NUMA node 0. This is the same physical GPU the earlier baselines used. Note that `rocm-smi`/`amd-smi` call it GPU **3**, because they order by PCI bus and ROCR orders by KFD node. | -| Scripts | `split/run_split.sh`, `per_config/run_per_config.sh` (`KEEP_CACHE=1`, as `per_config_rerun_2`), `per_config/run_per_config_build.sh` (cold cache per config), all with caches and artifacts redirected through `measure_env.sh` | -| Config list | The 184 configs that passed the split run phase. Identical to `per_config_rerun_2`'s list. | - -Committed results: - -- `baseline_run/AGGREGATE_analysis.txt`: per-config run aggregate (184 logs). -- `baseline_build/AGGREGATE_analysis.txt`: per-config build aggregate (184 logs). -- `baseline_run/common_test_outcomes.txt`: correctness baseline (outcome of every test in the split - run phase). - -The raw per-config logs are not committed, as for the earlier baselines. That is -`/.log.gz` and `_build.log.gz`, plus each config's `_analysis.txt` and -`times.txt`. A durable copy (757 MB, checked identical by md5) is in -`/home/alvasile/timing_runs_raw/P0.2/`: - -- `baseline_run/`, `baseline_build/`: the committed passes. -- `noise_run_1/`, `noise_run_2_contended/`: the repeats used for the noise band. -- `split/`: split phase logs, junit XML and outcomes. -- The config lists, plus `snapshots.txt`, `contention.log`, `corpus.log` and `gpu.log`. - -The working copy is in `/tmp/tl_plan/P0.2/`. **P3/P6 must re-aggregate these raw logs with the -`analyze_timing.py` of the commit they compare at**, not compare against the committed -`AGGREGATE_analysis.txt` as is: P1/P4 add timers and change the hierarchy. The command is -`python scripts/aggregate_timing.py /*/*.log.gz`, or `/*/*_build.log.gz` for the build -pass. The earlier baselines' raw logs re-aggregate byte-identically with the analyzer at this -commit, so analyzer changes do not contribute to any difference below. - -`/tmp` is cleaned of files older than 30 days on this host -(`/usr/lib/tmpfiles.d/tmp.conf`: `D /tmp 1777 root root 30d`). The P0.2 worktree -(`/tmp/tl_plan/P0.2/wt`), client (`/tmp/tl_plan/P0.2/build_client`) and artifacts -(`/tmp/tl_plan/P0.2/artifacts`) may therefore disappear before P3/P6. If they are gone, rebuild -them from `2083e6254bf` with the configure and commands under "Reproducing" (client ~1.5 min, -artifacts = split build phase ~11 min). - -## Correctness baseline - -Split run phase (`--use-cache`, `-n 4`): 567 tests, **194 passed, 2 failed, 371 skipped**. The -outcome of every test is identical to the 2026-10-20 split run. The split build phase also -matches it: 195 passed, 2 failed, 370 skipped. - -The two failures are pre-existing. `custom/custom_rr_f6.yaml` and `custom/custom_rr_mx.yaml` fail -to build (`RuntimeError: No valid solutions found`), so their run test finds no artifact tarball. -Every config in the per-config run and build passes returned 0. - -## Run side (per-config, instrumented) - -`run_per_config.sh`, serial, `TimingInstrumentation=True`, `KEEP_CACHE=1` (caches cleared once, -as for `per_config_rerun_2`). Measured 2026-10-31 01:40–02:34 UTC with no other users' work -pinned to cores 0–24. In seconds, from `AGGREGATE_analysis.txt` ("TIMING BY PURPOSE"): - -| Category | Baseline (this) | `per_config_rerun_2` (`4c6426cea6`) | Δ | -|---|---:|---:|---:| -| TOTAL RUNTIME | 2,401.1 | 2,568.3 | −6.5% | -| RUN: benchmark client | 2,284.9 | 2,330.8 | −2.0% | -| Reference GEMM (CPU) | 1,044.4 | 1,080.5 | −3.3% | -| … `solve_cpu_slow` | 689.0 | 727.8 | −5.3% | -| … `solve_cpu_fast` | 198.9 | 200.9 | −1.0% | -| Input data preparation | 500.7 | 502.9 | −0.4% | -| Validation | 329.1 | 323.9 | +1.6% | -| Client launch (startup, setup & teardown) | 289.8 | 284.7 | +1.8% | -| Per-solution bookkeeping & reporting | 54.8 | 68.3 | −19.9% | -| Kernel launch & execution | 35.7 | 40.9 | −12.7% | -| BUILD: solution generation & compile | 63.1 | 64.0 | −1.5% | -| MEASUREMENT OVERHEAD (instrumentation) | 52.8 | 172.8 | −69.5% | -| … `flush_timing_buffer` | 33.6 | 152.9 | −78.1% | -| Sum of per-config pytest wall times | 2,942 | 3,123 | −5.8% | - -- Every timer's call count is identical (for example 28,596 `cpu_reference_gemm`, 2,355,016 - solutions, 2,106,835 `gpu_input_reset`). The work is the same, as expected from identical - sources. -- Most of the −167 s difference is `flush_timing_buffer`: the instrumentation writing its buffer - to the log, −119 s. The rest is CPU reference −36 s, bookkeeping −14 s and kernel launch −5 s. -- The CPU reference difference (−3.3%; `solve_cpu_slow` −5.3%) is **unexplained and larger than - the clean-repeat spread** of that category (0.6%, see "Noise band"). The old pass differs in - client build (same sources), day and host load. Treat it as a difference between passes, not as - a code effect. -- Use this baseline for P3, not `per_config_rerun_2`. - -## Build side (per-config, cold cache per config) - -`run_per_config_build.sh`, measured 2026-10-31 00:26–01:23 UTC: - -| Category | Baseline (this) | `per_config_build` (`4c6426cea6`) | Δ | -|---|---:|---:|---:| -| TOTAL RUNTIME | 2,654.8 | 2,577.8 | +3.0% | -| BUILD: solution generation & compile | 2,605.6 | 2,530.1 | +3.0% | -| … `python_kernel_compilation` | 2,260.4 | 2,203.8 | +2.6% | -| … … `python_kernel_build_src_co` | 1,013.4 | 1,007.8 | +0.6% | -| … … `python_kernel_codegen` | 615.5 | 592.6 | +3.9% | -| … … `python_kernel_write_assemble` | 253.8 | 227.9 | +11.4% | -| … `python_solution_generation` | 321.6 | 306.6 | +4.9% | -| Ductile GA search (Python) | 23.7 | 19.7 | +20.2% | -| SHARED: cache & solution files | 23.2 | 23.2 | +0.1% | -| RUN: benchmark client | 14.3 | 13.0 | +10.3% | -| Sum of per-config pytest wall times | 3,361 | 3,273 | +2.7% | - -No config differs by more than 15 s from `per_config_build`. Over the 10 most expensive build -configs, the two passes differ by 2.5% in total and by at most 9.4% per config (`plr_zero`). - -**The whole build pass (00:26–01:23 UTC) overlapped another user's corpus run.** That run pins -short-lived compile jobs to core 9. `contention.log` shows them on core 9 in 51 of the 55 -minutes; pidstat only catches part of them, so the low average it reports understates the load. -The effect on the build is not quantified. It is probably small, since build workers are -independent processes rather than OpenMP threads. Still, the +3.0% against `per_config_build` -(a uniform drift: +88 s summed over 184 configs, at most 7 s per config) cannot be separated from -it. The baseline may be inflated by a few percent. Re-measuring was not possible: no 1-hour clean -window occurred before this was recorded. **P6 must therefore re-run this P0.2 build pass** -(P0.2 worktree/client, `run_per_config_build.sh`, same settings) **back to back with its -`after_build/` pass, under identical conditions, and compare against that re-run.** It should -also re-run the split build phase, for P6's `--build-only` comparison, since the 634 s below was -contended too. - -## Split phases (`-n 4`, uninstrumented) - -| Phase | Wall time | Exit | Conditions | -|---|---:|---:|---| -| Build (`--build-only`) | 634 s | 1 (the 2 known failures) | core 9 contended throughout | -| Run (`--use-cache`) | 2,950 s | 1 (the 2 known failures) | core 9 contended throughout | - -The 2026-10-20 split, measured at an older tree, took 1,183 s (build) and 2,804 s (run). The -run phase here (22:18–23:07 UTC) overlapped another user's job pinned to core 9 for its whole -duration. That job slows the OpenMP CPU reference about 2.5× (see "Noise band"), so **2,950 s is -an upper bound, not a clean uninstrumented baseline**. A clean repeat could not be obtained -(see "Conditions"). **P3 must re-measure this phase with the P0.2 worktree, client and artifacts -back to back with its own, under identical conditions, and compare against that**, not against -2,950 s. Likewise, the split build phase (634 s, also contended throughout) is not a clean -baseline: P6 re-measures it (see "Build side"). - -## Noise band - -The 10 most expensive run configs (by pytest wall time in this baseline) were repeated -(`noise_run_`, same script and settings, `KEEP_CACHE=1`). Samples whose run overlapped other -users' work pinned inside cores 0–24 were **excluded**: see "Conditions" for how this was -detected. Three repeats were run, but only part of them was uncontended. **Clean samples per -config: 3 for `sk_I8gemm_quick` and `sk_f8gemm_quick`**: the baseline pass, repeat 1, and -repeat 2's first two configs. **2 for the other eight configs**: the baseline pass and repeat 1. -All other repeat samples (repeat 2 configs 3–10, all of repeat 3, a restarted repeat 2) were -excluded. A further 2.7 h of waiting for an uncontended window (06:33–09:16 UTC) found none. - -Per-config pytest wall time (s), clean samples only. Spread = (max − min) / mean. - -| Config | Baseline | Repeat 1 | Repeat 2 | Spread | `per_config_rerun_2` (reference) | -|---|---:|---:|---:|---:|---:| -| streamk/sk_I8gemm_quick | 311 | 313 | 304 | 2.9% | 316 | -| streamk/sk_f8gemm_quick | 262 | 278 | 266 | 6.0% | 305 | -| gemm/gfx950/subtile_mxfp8 | 186 | 180 | – | 3.3% | 182 | -| sparse/gfx950/spmm_f16_sb | 124 | 126 | – | 1.6% | 144 | -| gemm/gfx950/subtile_mxfp4 | 95 | 89 | – | 6.5% | 82 | -| gemm/gfx950/plr_zero | 94 | 96 | – | 2.1% | 107 | -| gemm/fp16_tn | 87 | 87 | – | 0.0% | 103 | -| gemm/gfx950/dtl | 82 | 85 | – | 3.6% | 103 | -| gemm/bufferLoadStore0 | 81 | 83 | – | 2.4% | 99 | -| gemm/gfx950/subtile_mxfp8_correctness_regression_mt320x256_plr1_wgm4 | 71 | 67 | – | 5.8% | 69 | -| Sum | 1,393 | 1,404 | | 0.8% | 1,510 | - -Categories summed over the same 10 configs (s), baseline vs repeat 1: - -| Category | Baseline | Repeat 1 | Spread | -|---|---:|---:|---:| -| TOTAL RUNTIME | 1,359.6 | 1,369.7 | 0.7% | -| Reference GEMM (CPU) | 787.8 | 792.7 | 0.6% | -| Input data preparation | 229.0 | 226.7 | 1.0% | -| Validation | 179.9 | 181.7 | 1.0% | -| Client launch (startup, setup & teardown) | 82.5 | 82.2 | 0.4% | -| Per-solution bookkeeping & reporting | 28.6 | 30.2 | 5.4% | -| Kernel launch & execution | 21.6 | 24.2 | 11.4% | -| MEASUREMENT OVERHEAD (instrumentation) | 29.0 | 29.4 | 1.4% | - -So, on an uncontended host, totals and the large categories repeat to within about 1%. Single -configs repeat to within about 6%. The small categories (bookkeeping, kernel launch) vary by -5–11%. With 2–3 samples this is an indication, not a confidence interval. - -Contended samples, for scale (all excluded): with the core-9 job active, `sk_I8gemm_quick` took -1,169 s in the discarded first per-config run pass (23:07–00:26 UTC) and 749 s in repeat 3, -instead of 304–313 s. Over the 10 configs, Reference GEMM (CPU) in that discarded first pass was -2,008 s instead of 788 s, while GPU-side categories barely moved. CPU-reference changes in P3 -can only be judged on uncontended runs. - -Build side: no repeat could be measured uncontended. The only second sample is -`per_config_build`: same sources, different client build, measured 2026-10-21. It differs by -+3.0% in total, and per config by 0–9.4% over the 10 most expensive build configs. - -## Conditions and interference - -The host is shared, and the measurement cores and GPU are not reserved. Recorded with the -measurements in `/tmp/tl_plan/P0.2/`: - -- `snapshots.txt`: `uptime`, per-core idle on 0–24, and other processes on 0–24 at the start and end - of every pass. Load averages at the clean run pass: 8.8 (start), 7.9 (end). At the build pass: - 8.6 (start), 21.1 (end). -- `contention.log` (from 00:29): every 30 s, other users' processes on cores 0–24 with ≥1% CPU - (pidstat). -- `corpus.log` (from 05:26): cumulative CPU time of other users' pinned corpus runs. -- `gpu.log` (from 06:16): use, sclk, VRAM and power of the measurement GPU. Before 06:16 the - snapshots read `rocm-smi -d 0`, which is a different card (`0000:05:00.0`, ROCR device 1). - So there is no GPU-side record for the committed passes. Indirect evidence that they were not - GPU-contended: GPU-side categories are at or below `per_config_rerun_2`, and the clean - repeat matches the baseline to within 1.4% on every category except bookkeeping (5.4%) and - kernel launch (11.4%). - -Interference seen: - -- Another user's corpus run pins short-lived compile/timing jobs to core 9 (`--timing-cpus 9`) - and to cores 0–8, 10–47 (`--cpus`). It ran about 20:20–01:25, 03:33–05:51 and from 06:26 UTC - (still running at 09:16). - - The first per-config run pass (23:07–00:26) overlapped it and was discarded: TOTAL 3,911 s, - and the Int8 4111³ slow-path problem took 201.7 s instead of 72–75 s. - - The committed run pass was re-measured afterwards in a clean window (01:40–02:34; no jobs - from it on cores 0–24 after 01:30). - - The committed build pass (00:26–01:23) and both split phases (22:07–23:07) overlapped it and - were kept, with the caveats above. -- Unpinned processes (for example a root `python3` at 100% of one core) are placed on idle cores - in 0–24 by the scheduler and move away once the measurement threads load those cores. They are - not counted as contention. -- A root vLLM server (TP=4) occupied rocm-smi GPUs 0–3, including the measurement GPU, at up to - ~279 GB and 100% use. It was there in short bursts from 06:35 UTC and continuously - 07:20–08:05 and 08:17–09:12. This was after all the committed passes. - -Recommendation for P3/P6: monitor the measurement GPU as `rocm-smi -d 3`. Before and during each -pass, check for other users' jobs pinned into 0–24 (by CPU-time accounting, since pidstat misses -short-lived processes). Start passes only in a clean window (as `/tmp/tl_plan/P0.2/driver4.sh` -does), and keep or repeat only uncontended samples. - -## Reproducing - -```bash -git worktree add --detach /tmp/tl_plan/P0.2/wt 2083e6254bf -# client (CMake cache identical to /tmp/tl_client_msgpack); venv provides cmake 4.4.4 -source /home/alvasile/venv/bin/activate -cmake --preset tensilelite -S /tmp/tl_plan/P0.2/wt/projects/hipblaslt -B /tmp/tl_plan/P0.2/build_client \ - -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DGPU_TARGETS=gfx950 \ - -DCMAKE_CXX_COMPILER=/usr/bin/amdclang++ -DCMAKE_C_COMPILER=/opt/rocm/bin/amdclang \ - -DHIPBLASLT_ENABLE_YAML=OFF -DTENSILELITE_ENABLE_CLIENT=ON -DTENSILELITE_BUILD_TESTING=OFF \ - -DTENSILELITE_CLIENT_ENABLE_ROCPROFSDK=OFF -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -taskset -c 128-191 cmake --build /tmp/tl_plan/P0.2/build_client -j64 --target tensilelite-client - -export TL=/tmp/tl_plan/P0.2/wt/projects/hipblaslt/tensilelite -export CLIENT=/tmp/tl_plan/P0.2/build_client/tensilelite/client/tensilelite-client -export ART=/tmp/tl_plan/P0.2/artifacts -export TENSILE_CACHE=/tmp/tl_plan/P0.2/cache/tensile TENSILELITE_CACHE=/tmp/tl_plan/P0.2/cache/tensilelite -S=$TL/timing_runs/full_runs_gfx950 -OUT=/tmp/tl_plan/P0.2/split bash $S/split/run_split.sh -OUT=/tmp/tl_plan/P0.2/baseline_run KEEP_CACHE=1 bash $S/per_config/run_per_config.sh /tmp/tl_plan/P0.2/configs.txt -ROCR_VISIBLE_DEVICES=0 HIP_VISIBLE_DEVICES=0 OUT=/tmp/tl_plan/P0.2/baseline_build \ - SCRATCH=/tmp/tl_plan/P0.2/build_scratch bash $S/per_config/run_per_config_build.sh /tmp/tl_plan/P0.2/configs.txt -``` - -`configs.txt` holds the `test_config_run` IDs that passed the split run phase (184 lines, the -same set as `per_config_rerun_2/configs.txt`). A durable copy is -`/home/alvasile/timing_runs_raw/P0.2/configs.txt`; use it if the `/tmp` copy is gone. The noise -lists are next to it (`noise_run_configs.txt`, `noise_build_configs.txt`). diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline_run/common_test_outcomes.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline_run/common_test_outcomes.txt deleted file mode 100644 index f4b047051e2c..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/baseline_run/common_test_outcomes.txt +++ /dev/null @@ -1,571 +0,0 @@ -# Correctness baseline (P0.2): outcome of every test in the gfx950 split run phase -# (split/run_split.sh run phase: --use-cache, -n 4, cores 0-24, GPU 0) at 2083e6254bf -# (code-identical to 55e29436e9a). One " " per line, sorted. -# 567 tests: 194 PASSED, 2 FAILED, 371 SKIPPED. See ../baseline.md. -tensilelite/Tests/common/flags/gfx942/test_build_only.py::test_compile PASSED -tensilelite/Tests/common/flags/gfx942/test_build_only.py::test_use_cache SKIPPED -tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_compile_and_benchmark SKIPPED -tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_use_cache SKIPPED -tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_use_cache_creates_cache_dir SKIPPED -tensilelite/Tests/common/flags/gfx942/test_use_cache.py::test_use_cache_different_params SKIPPED -tensilelite/Tests/common/test_config_helpers.py::test_base_skip_gfx1250_inert_on_strict_target PASSED -tensilelite/Tests/common/test_config_helpers.py::test_ffm_fail_inert_on_hardware PASSED -tensilelite/Tests/common/test_config_helpers.py::test_ffm_fail_inert_on_other_arch PASSED -tensilelite/Tests/common/test_config_helpers.py::test_ffm_fail_xfails_under_ffm PASSED -tensilelite/Tests/common/test_config_helpers.py::test_find_available_archs_does_not_map_the_retired_v0_name_to_the_base PASSED -tensilelite/Tests/common/test_config_helpers.py::test_find_available_archs_keeps_stepping_name PASSED -tensilelite/Tests/common/test_config_helpers.py::test_skip_gfx1250_strict_fires_on_strict_target PASSED -tensilelite/Tests/common/test_config_helpers.py::test_skip_gfx1250_strict_inert_on_base_target PASSED -tensilelite/Tests/common/test_config_helpers.py::test_unmarked_config_never_xfails_under_ffm PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/amaxd/8rn_amaxd.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/amaxd/gfx1250/amax_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/benchmark_runs/bf16_tn_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/benchmark_runs/fp16_tn_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/benchmark_runs/fp32_tn.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/client/rotate_mode0.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/client/rotate_mode0_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/client/rotate_mode1.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/client/rotate_mode1_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/comm/gfx950/fused_a2a.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/bbs_tn_dp.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/bbs_tn_multi_size_single_kernel.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/bbs_tn_streamK.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/f32_tt_batched.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/f8_tn_mx.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx1250/hhs_nt_multi_size.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/bf16_tn_dp.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/bf16_tn_multi_size_single_kernel.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/bf16_tn_streamK.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/f16_nt_multi_size.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/f32_tt_batched.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/f8_tn_mx_subtile.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/f8bf16_tn_cms.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/ductile/gfx950/xf32_nn.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/exception/f16_cf32_nan.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/exception/f16_cf32_nan_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/8rn_b_s.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/ag.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/agntab_coverage_gfx942.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/bf16_tn.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/bf16_tn_predict.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/bufferLoadStore0.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/cgemm.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/dot2_gfx942.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/dtl.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/dtv.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/dtv_gfx90a.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/f32_mac.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/f64.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/f64_gfx90a.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/f64_mac.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp16_HH_BHS_bf16mfma.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp16_tn.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp16_use_e.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp16fp32mix.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp32_nt.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp32_scalealphavec.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp32_tn.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8bf8nss.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8n.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8n_rowwise.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8n_use_e.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8nfp16mix_fp8nss.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8nfp16mix_hfp8ns.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8nfp16mix_hhs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/fp8nfp16mix_hhs_lsu4.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gemm_bias_strided_batched.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gemm_bias_strided_batched_broadcast.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gemm_bias_strided_batched_broadcast_gfx940.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gemm_bias_strided_batched_gfx940.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gemm_ck_gfx942.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/bf16_tn_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/fp16_HH_BHS_bf16mfma_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/fp16_tn_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/fp16_use_e_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/gemm_bias_strided_batched_broadcast_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/gemm_bias_strided_batched_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/gsuasb_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx11/i8_gsu_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/1024_vgpr_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/1024_vgpr_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/8r_b_s_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/agntab_coverage_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/agntab_coverage_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b6f4ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b6f4ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b8b8s_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b8b8s_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b8f8ss_grvw1_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/b8hs_grvw1_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_CLS_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_CLS_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_tn_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf16_tn_gfx12_predict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf6_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf6_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf6_tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/bf6_tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/cgemm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/cgemm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/decouple_pgr_tdm_fuse_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f32_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f32_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f32_grid_limit_size.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4b8ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4b8ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4f6ss_tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f4f6ss_tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f64_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f64_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6_tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6_tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6b6ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6b6ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6b8ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f6b8ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8b6ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8b6ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8b8ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8b8ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f4ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f4ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_cls_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_cls_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_grvw1_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_pk8_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_pk8_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_sr_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/f8f8s_sr_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp16_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp16_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp16_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp16_use_e_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp16_use_e_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp4_tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp4_tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp8_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp8_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp8_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/fp8bf8ss_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gemm_bias_strided_batched_broadcast_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gemm_bias_strided_batched_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gltr_bounds_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gsu_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gsu_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gsuaad_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/gsuasb_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/i8_gsu_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/i8i8i_grvw1_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/i8ii_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/i8ii_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/i8ii_grvw1_gfx12.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/interleave_alpha2_group_store_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/largeLds_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/largeLds_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf4_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf4_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf6_tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf6_tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf6f4ss_tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf6f4ss_tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf8_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf8_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf8f4ss_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/mxf8f4ss_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/nt_th_nv_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/nt_th_nv_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/pgl_bound_check.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/rap_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/segment_b_rotate_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/segment_interleave_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/segment_interleave_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/stinky_sia4.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/stinky_sia4_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250-strict_bench.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250-strict_cluster.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250_bench.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/subtile_bf16_gfx1250_cluster.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_general_batched_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_general_batched_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_multicast_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_multicast_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_split_gfx1250-strict_xfail.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdm_split_gfx1250_xfail.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdmpluslds_tdm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/tdmpluslds_tdm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/xfp32_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/xfp32_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/xfp32_inf_support_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/zgemm_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx12/zgemm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx1250/cluster_entry_handoff.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/agntab_coverage_gfx950.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/bf16_cvt.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/custom_mainloop_scheduling.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/custom_mainloop_scheduling_tf32.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/custom_xfp32.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/dot2_gfx950.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/dtl.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/f16f8mix_ss.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/f16f8mix_ss_stoch.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/f8b8hs.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/f8f16mix_f8s.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp16_use_e_wide_store.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp8.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_bf16_tn_act.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_bf16_tn_act_groupgemm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_fp32_tn_act.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/fp8_mxfp4_fp32_tn_act_groupgemm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/general_wgm.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/i8_gsu_gfx950.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/lds160K.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/lds_tr.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mx32f4_tn.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mx32f8_tn.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_bf16_tn_act.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_bf16_tn_act_groupgemm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_fp32_tn_act.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_fp8_fp32_tn_act_groupgemm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_bf16_tn_act.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_bf16_tn_act_groupgemm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_fp32_tn_act.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp4_mxfp4_fp32_tn_act_groupgemm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp8_residual_fail_classes_repro.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/mxfp8_verify_fail_repro.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/plr_zero.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/preloop_local_read_wait.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/ss_bss.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_bf16.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_fp8.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp4.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_bias_sav.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt192x256_plr1_wgm4.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt256x192_plr1_wgm1.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt256x320_plr0_wgm1.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_correctness_regression_mt320x256_plr1_wgm4.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_multi_du_multi_partition.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_tail_loop.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/subtile_mxfp8_tail_loop_smoke.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gfx950/xfp32.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gsuasb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/gsuc.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/hh_f8nhs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/i8_gsu.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/i8_gsu_gfx940.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/icache_flush.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/largeMT.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/large_size.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/lsu_fnuz.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/lsu_i8.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/mbskPrefetchOpt.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/mix_cvt_after_ds_fnuz.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/nan_bounds_check_odd_padding.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/no_shadowinit.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/sia0_pgr.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/swizzleA.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/swizzleB.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/swizzle_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/tail_small_size.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/ulsgro1.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/use_beta_false.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/wgm.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/xfp32.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gemm/zgemm.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/fp16_gradient_activation.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/fp16_gradient_bias.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/fp8bf8nss_gradient_bias_b.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx11/fp16_gradient_activation_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx11/fp16_gradient_bias_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_bgrada_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_bgradb_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_bgradd_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/bbs_dgelu_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_bgrada_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_bgradb_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_bgradd_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250-strict/hhs_dgelu_gfx1250-strict.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/bbs_bgrada_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/bbs_bgradb_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/bbs_bgradd_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/bbs_dgelu_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/hhs_bgrada_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/hhs_bgradb_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/hhs_bgradd_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx1250/hhs_dgelu_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gradient/gfx950/bf16_gradient_bias.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/gfx11/grouped_gemm_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/gfx11/grouped_gemm_userargs_gfx11.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/grouped_gemm.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/grouped_gemm_ck_gfx942.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/grouped_gemm_gfx1250.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/groupedgemm/grouped_gemm_userargs.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gsu/auto_gsu.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gsu/auto_gsu_zero_dim.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gsu/f32_gsu.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/gsu/gsu_mbsk.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sgemm_dual_fmac_asm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sgemm_streamk_asm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sgemm_xf32_asm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/bf16_activation.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/f16_activation.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8bs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8bs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8f8.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8f8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8hs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_b8hs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_bf16.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_bf16_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f16.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f16_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8_ml.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8b8.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8b8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8bs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8bs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8hs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_f8hs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_fp16_ml1.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8bs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8bs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8hs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8hs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8is.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_i8is_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_all.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_f16_transposes.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_gl2prefetch.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_ldspadding.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250-strict/spmm_tdm_multicast.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/bf16_activation.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/f16_activation.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8bs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8bs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8f8.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8f8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8hs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_b8hs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_bf16.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_bf16_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f16.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f16_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8_ml.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8b8.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8b8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8bs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8bs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8hs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_f8hs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_fp16_ml1.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_gate.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8bs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8bs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8hs.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8hs_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8is.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_i8is_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_all.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_f16_transposes.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_gl2prefetch.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_iteratemode.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_ldspadding.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_multicast.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx1250/spmm_tdm_split_xfail.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/bf16_activation.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/bf16_gate_r.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/f8n_gate_r.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/fp16_activation.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/fp16_gate_r.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/i8_activation.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/libray_logic.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_b8f8n.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_b8f8n_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_bf16.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_bf16_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_bf8n.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_bf8n_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_f8b8n.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_f8b8n_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_f8n.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_f8n_sb.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi16.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi16_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi32.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_fp16_mi32_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_gsuc.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi16.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi16_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi32.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8_mi32_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8bs.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8bs_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8hs.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8hs_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8is.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_i8is_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_vw_lg_one.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/spmm_vw_lg_one_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/use_sgpr_for_gro.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx94x/use_sgpr_for_gro_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/bf16_gate_r.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/f8_gate_r.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/fp16_gate_r.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_b8.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_b8_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_bf16.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_bf16_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_dtl.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_f16.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_f16_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_f8.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_f8_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_i8.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_i8_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_i8bs.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_i8hs.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_ldstr.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_vw_lg_one.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/sparse/gfx950/spmm_vw_lg_one_sb.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/data_parallel_static_mx32f4.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/data_parallel_static_sgemm.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx11/sk_bgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx11/sk_hgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx12/sk_bgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx12/sk_hgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/bf16_vector_epilogue_persistent.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf4.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf4_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf8.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf8_halfplr_tdm_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/data_parallel_static_mxf8_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_bf16gemm_tdm_subtile.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_bgemm_tdm_split.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_bgemm_tdm_split_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_dynamic_hgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_dynamic_sgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_f8gemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_halfplr_f8gemm_tdm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_halfplr_mxf8gemm_tdm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_hgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_hybrid_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_explicit.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_pap_prefetchgl2.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_tdm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf4gemm_tdm_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf6b8gemm_tdm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf6b8gemm_tdm_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8b6gemm_tdm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8b6gemm_tdm_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8f4gemm_tdm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8f4gemm_tdm_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_explicit.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_pap_prefetchgl2.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm_split.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_mxf8gemm_tdm_split_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_sgemm_pap.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_sgemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_tdm_general_batched.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/core/sk_tdm_subtile_general_batched.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/data_parallel_cluster_multicast_boundaries.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf4_cluster_multicast_ext.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf4_pap_cluster_multicast_ext.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf8_cluster_multicast_ext.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/data_parallel_static_mxf8_pap_cluster_multicast_ext.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/sk_hgemm_debug_modes.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/sk_mxf4gemm_tdm_cluster_ext.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/sk_mxf4gemm_tdm_ext.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx1250/sk_mxf4gemm_tdm_pap_ext.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/data_parallel_static_hgemm_use_e.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/data_parallel_static_sgemm_pap.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_bgemm_pap.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_hgemm_use_e.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_hybrid_xcc_mapping.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxf4gemm_pap.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_basic.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_batch_combo.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_full_coverage.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_activation_streamk_combo.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_atomic_variants.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_basic_variants.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_batch_mi_combo.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_batch_sizes.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_comprehensive.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_debug_modes.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_full_coverage.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_activation_combo.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_basic.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_batch_combo.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_full_coverage.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_groupedgemm_streamk_combo.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_matrix_instructions.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_problem_sizes.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_tree_reduction.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_mxfp4_xcc_mapping.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_debug_modes.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_dynamic_pap.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_hybrid_pap.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/gfx950/sk_sgemm_pap.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_I8gemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_bgemm_div.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_bgemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_bgemm_sia1.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_cgemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_dgemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_dynamic.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_dynamic_mxfp4.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_dynamic_work_stealing.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_f8gemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_f8ngemm_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hgemm_epilogues.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hgemm_lsu.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hgemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hgemm_race.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hybrid.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_hybrid_work_stealing.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_mx32f4_quick.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_mx32f8_quick.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_queue_selfclean.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_reduction.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_sgemm_alpha0.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_sgemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/streamk/sk_zgemm_quick.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_aiter_bf16.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_aiter_f4.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_ck.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rocblas_gemv_bf16_tn_m1.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rocblas_gemv_bf16_tn_m2.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rocblas_gemv_bf16_tn_m4.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rr.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rr_f6.yaml] FAILED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_rr_mx.yaml] FAILED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_tensile_sk.yaml] SKIPPED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_triton_f4.yaml] PASSED -tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/custom/custom_wave_bf16.yaml] PASSED diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/before_after/compare_aggregates.py b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/before_after/compare_aggregates.py new file mode 100755 index 000000000000..fb9c1a86826e --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/before_after/compare_aggregates.py @@ -0,0 +1,97 @@ +#!/usr/bin/env python3 +# Copyright Advanced Micro Devices, Inc., or its affiliates. +# SPDX-License-Identifier: MIT +"""Compare two per-config timing passes side by side. + +Each argument is the OUT directory of a run_per_config.sh or run_per_config_build.sh pass: it holds +AGGREGATE_analysis.txt (written by scripts/aggregate_timing.py) and one /times.txt per +config. The script prints the "TIMING BY PURPOSE" table of both aggregates side by side with the +change in percent, then the summed pytest wall time, the slowest configs and every config that +got more than 2 s slower. + +Usage: compare_aggregates.py BEFORE_DIR AFTER_DIR [--max-depth N] +""" + +import argparse +import glob +import os +import re + + +def purpose_rows(path): + """(indent, category, milliseconds) for each row of the TIMING BY PURPOSE table.""" + rows = [] + inside = False + with open(path) as f: + for line in f: + if line.startswith("TIMING BY PURPOSE"): + inside = True + continue + if inside and line.startswith("TIMING BY TIMER NESTING"): + break + m = re.match(r"^(\s+)(\S.*?)\s{2,}(?:([\d,]+)\s+)?([\d,]+\.\d\d)\s", line) + if inside and m: + rows.append((len(m.group(1)), m.group(2).strip(), float(m.group(4).replace(",", "")))) + return rows + + +def keyed(rows): + """Rows keyed by their path in the category tree, so equal names at different places differ.""" + out = {} + stack = [] + for indent, name, ms in rows: + while stack and stack[-1][0] >= indent: + stack.pop() + stack.append((indent, name)) + out["/".join(n for _, n in stack)] = (indent, ms) + return out + + +def pytest_seconds(directory): + """Pytest wall seconds per config, from /times.txt.""" + seconds = {} + for path in glob.glob(os.path.join(directory, "*", "times.txt")): + with open(path) as f: + m = re.search(r"seconds=(\d+)", f.read()) + if m: + seconds[os.path.basename(os.path.dirname(path))] = int(m.group(1)) + return seconds + + +def main(): + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("before", help="OUT directory of the 'before' pass") + parser.add_argument("after", help="OUT directory of the 'after' pass") + parser.add_argument("--max-depth", type=int, default=8, help="deepest category level printed") + args = parser.parse_args() + + a = keyed(purpose_rows(os.path.join(args.before, "AGGREGATE_analysis.txt"))) + b = keyed(purpose_rows(os.path.join(args.after, "AGGREGATE_analysis.txt"))) + print(f"{'Category':70s} {'before s':>10s} {'after s':>10s} {'delta':>8s}") + for key in list(a) + [k for k in b if k not in a]: + indent = (a.get(key) or b.get(key))[0] + if (indent - 4) // 2 > args.max_depth: + continue + x = a.get(key, (0, 0))[1] / 1000 + y = b.get(key, (0, 0))[1] / 1000 + if max(x, y) < 1: + continue + delta = f"{(y - x) / x * 100:+.1f}%" if x else "new" + width = 70 - indent + 4 + print(f"{' ' * (indent - 4)}{key.split('/')[-1][:width]:{width}s} {x:10.1f} {y:10.1f} {delta:>8s}") + + tb = pytest_seconds(args.before) + ta = pytest_seconds(args.after) + total_b = sum(tb.values()) + total_a = sum(ta.values()) + change = f" ({(total_a - total_b) / total_b * 100:+.1f}%)" if total_b else "" + print(f"\npytest wall sum: before {total_b} s, after {total_a} s{change}") + print("largest per-config (before -> after s):") + for k in sorted(tb, key=lambda k: -tb[k])[:12]: + print(f" {k:60s} {tb[k]:5d} -> {ta.get(k, 0):5d}") + slower = sorted(((k, tb[k], ta[k]) for k in tb if k in ta and ta[k] > tb[k] + 2), key=lambda t: t[1] - t[2]) + print("slower by more than 2 s:", slower if slower else "none") + + +if __name__ == "__main__": + main() diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/before_after/run_side.sh b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/before_after/run_side.sh new file mode 100755 index 000000000000..d0b2ab32b314 --- /dev/null +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/before_after/run_side.sh @@ -0,0 +1,83 @@ +#!/bin/bash +# Copyright Advanced Micro Devices, Inc., or its affiliates. +# SPDX-License-Identifier: MIT +# +# One side ("before" or "after") of a before/after timing of the common tests. Start one instance +# per side, at the same time, each with its own source tree, client, cores and GPU, so that both +# sides see the same host load. Each side runs: +# 1. split/run_split.sh: the CI-like build phase then run phase (uninstrumented, -n 4), and +# records the configs whose run passed (/split_run_passed.txt); +# 2. waits until every side in SIDES has finished step 1, then writes the shared config list +# (/configs.txt): the union of the configs that passed in any side's split run phase; +# 3. per_config/run_per_config_build.sh over that list, instrumented, caches cleared once before +# the first config, artifacts kept; +# 4. per_config/run_per_config.sh on those artifacts, instrumented, caches not cleared. +# Compare the two sides with before_after/compare_aggregates.py. +# Usage: bash run_side.sh ( is one of SIDES) +# Env: ROOT work directory shared by the sides (required); 's output goes to +# $ROOT// +# TL, CLIENT this side's TensileLite tree and prebuilt tensilelite-client (required) +# SIDES side names, default "before after" +# GPU_TARGET, CORES, GPU, ANALYZE_DIR, DIST, CLEAR_RUN_CACHE: see ../measure_env.sh and +# ../split/run_split.sh. Give each side its own CORES and GPU. +set -u +S=${1:?usage: run_side.sh } +: "${ROOT:?set ROOT}" "${TL:?set TL}" "${CLIENT:?set CLIENT}" +SIDES=${SIDES:-before after} +case " $SIDES " in + *" $S "*) ;; + *) echo "side '$S' is not in SIDES='$SIDES'" >&2; exit 1 ;; +esac +SCRIPTS=$(dirname "$(dirname "$(realpath -- "${BASH_SOURCE[0]}")")") +export TL CLIENT +R=$ROOT/$S +mkdir -p "$R" +export TMPDIR=$R/tmp +mkdir -p "$TMPDIR" +log() { echo "$(date -u +%FT%TZ) $*" >> "$R/driver.log"; } +# Host conditions at each step, for judging how comparable the two sides are on a shared host. +snap() { + { + echo "== $1 $(date -u +%FT%TZ)" + uptime + rocm-smi -d "${GPU:-0}" --showuse --showmemuse --showpids 2>/dev/null | grep -Ev '^=+$|^$' + } >> "$R/conditions.log" +} + +# 1. CI-like split: wall time of each phase and the configs that pass +if ! grep -qs ALLDONE "$R/split/times.txt"; then + snap split_start; log split start + OUT=$R/split ART=$R/split_art TENSILE_CACHE=$R/cache_split/tensile \ + TENSILELITE_CACHE=$R/cache_split/tensilelite bash "$SCRIPTS/split/run_split.sh" + snap split_end; log split done +fi +python - "$R/split/run.xml" > "$R/split_run_passed.txt" <<'EOF' +import re +import sys +import xml.etree.ElementTree as ET + +for tc in ET.parse(sys.argv[1]).iter("testcase"): + m = re.search(r"test_config_run\[(.*)\]", tc.get("name", "")) + if m and not any(c.tag in ("failure", "error", "skipped") for c in tc): + print(m.group(1)) +EOF +touch "$R/split.done" + +# 2. shared config list: union of the configs whose run passed in any side's split run phase +for s in $SIDES; do + while [ ! -f "$ROOT/$s/split.done" ]; do sleep 30; done +done +for s in $SIDES; do cat "$ROOT/$s/split_run_passed.txt"; done | sort -u > "$R/configs.txt" +log "configs: $(wc -l < "$R/configs.txt")" + +# 3. per-config build, instrumented; caches cleared once at the start, artifacts kept +export ART=$R/art TENSILE_CACHE=$R/cache/tensile TENSILELITE_CACHE=$R/cache/tensilelite +snap build_start; log build start +OUT=$R/build SCRATCH=$R/scratch CLEAR_CACHE=once KEEP_ARTIFACTS=1 \ + bash "$SCRIPTS/per_config/run_per_config_build.sh" "$R/configs.txt" +snap build_end; log build done + +# 4. per-config run on those artifacts, caches not cleared +snap run_start; log run start +OUT=$R/run CLEAR_CACHE=never bash "$SCRIPTS/per_config/run_per_config.sh" "$R/configs.txt" +snap run_end; log run done diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_build/AGGREGATE_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_build/AGGREGATE_analysis.txt deleted file mode 100644 index 87cc51111044..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_build/AGGREGATE_analysis.txt +++ /dev/null @@ -1,598 +0,0 @@ -============================================================================================================================= -TIMING ANALYSIS SUMMARY -============================================================================================================================= - -TIMING BY PURPOSE ------------------------------------------------------------------------------------------------------------------------------ - Category Count Total (ms) Mean (ms) % of Parent % of Wall ------------------------------------------------------------------------------------------------------------------------------ - TOTAL RUNTIME 2,577,849.77 100.0% - BUILD: solution generation & compile 2,530,063.12 98.1% 98.1% - Solution generation & build (Python) 2,510,348.29 99.2% 97.4% - python_kernel_compilation 855 2,203,797.86 2,577.54 87.8% 85.5% - python_kernel_build_src_co 855 1,007,817.04 1,178.73 45.7% 39.1% - python_kernel_build_src_co.compile 263 1,004,724.33 3,820.24 99.7% 39.0% - python_kernel_build_src_co.unbundle 263 1,566.60 5.96 0.2% 0.1% - python_kernel_build_src_co.cache_check 855 1,172.50 1.37 0.1% 0.0% - python_kernel_build_src_co.cache_populate 263 214.63 0.82 0.0% 0.0% - python_kernel_build_src_co.setup 855 22.87 0.03 0.0% 0.0% - python_kernel_build_src_co.move 263 17.46 0.07 0.0% 0.0% - (untracked) 98.65 0.0% 0.0% - python_kernel_codegen 855 592,584.11 693.08 26.9% 23.0% - python_kernel_write_assemble 855 227,868.45 266.51 10.3% 8.8% - python_kernel_bench_setup 855 131,638.45 153.96 6.0% 5.1% - python_kernel_bench_postprocess 855 106,816.89 124.93 4.8% 4.1% - python_benchpost_lib_construction 855 74,773.69 87.45 70.0% 2.9% - python_benchpost_library_write 855 22,324.17 26.11 20.9% 0.9% - python_benchpost_naming 855 9,436.66 11.04 8.8% 0.4% - python_benchpost_client_config 855 249.07 0.29 0.2% 0.0% - (untracked) 33.30 0.0% 0.0% - python_kernel_validate 855 81,431.47 95.24 3.7% 3.2% - python_kernel_setup 855 20,886.91 24.43 0.9% 0.8% - python_kernel_build_co 855 18,912.53 22.12 0.9% 0.7% - python_kernel_write_helpers 855 11,912.59 13.93 0.5% 0.5% - (untracked) 3,929.43 0.2% 0.2% - python_solution_generation 823 306,550.42 372.48 12.2% 11.9% - python_solgen_forked_solutions 823 306,464.54 372.37 100.0% 11.9% - python_solgen_custom_kernels 823 67.99 0.08 0.0% 0.0% - python_solgen_fork_permutations 823 4.83 0.01 0.0% 0.0% - (untracked) 13.06 0.0% 0.0% - Ductile GA search (Python) 19,714.83 0.8% 0.8% - python_ductile_initial_sampling 8 13,760.99 1,720.12 69.8% 0.5% - python_ductile_generate_solutions 32 4,536.07 141.75 23.0% 0.2% - python_ductile_ga_between_evaluations 8 1,366.26 170.78 6.9% 0.1% - python_ductile_setup 8 30.08 3.76 0.2% 0.0% - python_ductile_clear_artifacts 24 16.81 0.70 0.1% 0.0% - python_ductile_read_scores 32 4.29 0.13 0.0% 0.0% - python_ductile_verification 8 0.33 0.04 0.0% 0.0% - SHARED: cache & solution files 23,180.53 0.9% 0.9% - python_write_solutions 855 14,069.93 16.46 60.7% 0.5% - python_wsol_dump 855 13,390.16 15.66 95.2% 0.5% - python_wsol_prepare 855 531.38 0.62 3.8% 0.0% - python_wsol_prepare_nocache 855 529.92 0.62 99.7% 0.0% - (untracked) 1.47 0.3% 0.0% - python_wsol_header 855 12.63 0.01 0.1% 0.0% - (untracked) 135.75 1.0% 0.0% - python_solution_indexing 855 8,279.91 9.68 35.7% 0.3% - python_write_cache 855 830.02 0.97 3.6% 0.0% - python_cache_check 831 0.67 0.00 0.0% 0.0% - RUN: benchmark client 12,995.06 0.5% 0.5% - Client launch (startup, setup & teardown) 9,950.79 76.6% 0.4% - hip_initialization 32 4,712.80 147.27 47.4% 0.2% - process_exit 32 3,467.40 108.36 34.8% 0.1% - main_problem_factory 32 595.29 18.60 6.0% 0.0% - factory_hip_get_device_properties 4 590.74 147.68 99.2% 0.0% - (untracked) 4.56 0.8% 0.0% - process_startup 32 468.60 14.64 4.7% 0.0% - main_teardown_locals 32 386.04 12.06 3.9% 0.0% - data_init_setup 32 97.08 3.03 1.0% 0.0% - code_object_loading 32 96.45 3.01 1.0% 0.0% - library_loading 32 65.62 2.05 0.7% 0.0% - lazy_loading_init 32 45.39 1.42 0.5% 0.0% - main_parse_args 32 11.80 0.37 0.1% 0.0% - reporter_setup 32 1.84 0.06 0.0% 0.0% - python_client_write_run_script 32 1.63 0.05 0.0% 0.0% - listener_setup 32 0.52 0.02 0.0% 0.0% - solution_iterator_setup 32 0.33 0.01 0.0% 0.0% - Input data preparation 3,004.17 23.1% 0.1% - gpu_input_preparation 40 2,387.72 59.69 79.5% 0.1% - gpu_input_prep_init_cpu_inputs 39 1,309.51 33.58 54.8% 0.1% - gpu_input_prep_copy_valid_to_gpu 40 937.87 23.45 39.3% 0.0% - gpu_input_prep_copy_swizzled 4 135.91 33.98 5.7% 0.0% - gpu_input_prep_batched_inputs 40 2.63 0.07 0.1% 0.0% - gpu_input_prep_copy_inputs_d2d 40 0.98 0.02 0.0% 0.0% - gpu_input_prep_constant_inputs 39 0.64 0.02 0.0% 0.0% - gpu_input_prep_convert_to_inputs 40 0.11 0.00 0.0% 0.0% - gpu_input_prep_slow_path 40 0.00 0.00 0.0% 0.0% - (untracked) 0.07 0.0% 0.0% - pre_solution_listener.DataInitialization 585 488.60 0.84 16.3% 0.0% - pre_solution_mx_init_data 130 369.08 2.84 75.5% 0.0% - pre_solution_mx_side_regenerate 8 369.03 46.13 100.0% 0.0% - pre_solution_mx_side_cache_hit 252 0.02 0.00 0.0% 0.0% - (untracked) 0.02 0.0% 0.0% - pre_solution_mx_copy_valid_to_gpu_no_regen 126 103.07 0.82 21.1% 0.0% - pre_solution_mx_cpu_sync_no_regen 126 8.60 0.07 1.8% 0.0% - pre_solution_mx_copy_valid_to_gpu_after_regen 4 3.29 0.82 0.7% 0.0% - pre_solution_mx_copy_inputs_d2d_no_regen 126 3.18 0.03 0.7% 0.0% - pre_solution_mx_cpu_sync_after_regen 4 1.19 0.30 0.2% 0.0% - pre_solution_mx_copy_inputs_d2d_after_regen 4 0.13 0.03 0.0% 0.0% - pre_solution_mx_check 585 0.03 0.00 0.0% 0.0% - (untracked) 0.03 0.0% 0.0% - cpu_data_init 1 119.66 119.66 4.0% 0.0% - rotating_buffer_preparation 40 6.46 0.16 0.2% 0.0% - gpu_input_reset 1 1.73 1.73 0.1% 0.0% - gpu_input_reset_copy_valid_to_gpu 1 0.82 0.82 47.6% 0.0% - gpu_input_reset_copy_swizzled 1 0.81 0.81 46.9% 0.0% - gpu_input_reset_batched_inputs 1 0.06 0.06 3.7% 0.0% - gpu_input_reset_copy_inputs_d2d 1 0.03 0.03 1.7% 0.0% - gpu_input_reset_convert_to_inputs 1 0.00 0.00 0.0% 0.0% - gpu_input_reset_slow_path 1 0.00 0.00 0.0% 0.0% - Per-solution bookkeeping & reporting 30.61 0.2% 0.0% - pre_solution_listener.AllSolutionsIterator 585 10.23 0.02 33.4% 0.0% - post_solution_log 585 5.20 0.01 17.0% 0.0% - pre_solution_listener.BenchmarkTimer 585 3.99 0.01 13.0% 0.0% - pre_solution_listener.ProgressListener 585 2.60 0.00 8.5% 0.0% - run_current_solution_check 585 2.03 0.00 6.6% 0.0% - pre_solution_listener.MetaResultReporter 585 2.00 0.00 6.5% 0.0% - post_solution_reporting 585 1.75 0.00 5.7% 0.0% - post_problem 40 1.66 0.04 5.4% 0.0% - finalize_report 32 0.35 0.01 1.1% 0.0% - post_solution_perf_calc 585 0.25 0.00 0.8% 0.0% - problem_report_index 40 0.15 0.00 0.5% 0.0% - solution_selection 585 0.14 0.00 0.5% 0.0% - pre_solution_listener.SynchronizerValidator 585 0.10 0.00 0.3% 0.0% - problem_teardown 40 0.06 0.00 0.2% 0.0% - solution_teardown 585 0.02 0.00 0.1% 0.0% - post_solution_result_file 585 0.01 0.00 0.0% 0.0% - post_solution_validation 585 0.01 0.00 0.0% 0.0% - pre_solution_listener.HardwareMonitorListener 585 0.01 0.00 0.0% 0.0% - post_solution_sol_advance 585 0.01 0.00 0.0% 0.0% - post_solution_perf_reset 585 0.01 0.00 0.0% 0.0% - run_iteration_teardown 1 0.01 0.01 0.0% 0.0% - Reference GEMM (CPU) 7.11 0.1% 0.0% - pre_solution_listener.ReferenceValidator 585 4.64 0.01 65.2% 0.0% - cpu_reference_gemm_per_solution_inputs_changed 1 4.62 4.62 99.5% 0.0% - solve_cpu_slow_per_solution_inputs_changed 1 4.62 4.62 100.0% 0.0% - (untracked) 0.02 0.5% 0.0% - cpu_reference_gemm 1 2.47 2.47 34.8% 0.0% - solve_cpu_slow 1 2.46 2.46 99.5% 0.0% - (untracked) 0.01 0.5% 0.0% - Validation 2.20 0.0% 0.0% - validate_warmups_listener.ReferenceValidator 1 1.25 1.25 57.0% 0.0% - validate_result_buffer_alloc 1 0.94 0.94 74.9% 0.0% - validate_gpu_readback 1 0.30 0.30 24.2% 0.0% - validate_element_comparison 1 0.01 0.01 0.7% 0.0% - validate_inputs_copy 1 0.00 0.00 0.1% 0.0% - validate_report 1 0.00 0.00 0.0% 0.0% - validate_setup 1 0.00 0.00 0.0% 0.0% - validate_warmups_listener.SynchronizerValidator 1 0.94 0.94 42.9% 0.0% - validate_synchronizer_check 1 0.94 0.94 100.0% 0.0% - validate_synchronizer_readback 1 0.78 0.78 82.5% 0.0% - validate_synchronizer_scan 1 0.16 0.16 17.5% 0.0% - validate_warmups_listener.MetaResultReporter 1 0.00 0.00 0.0% 0.0% - validate_warmups_listener.HardwareMonitorListener 1 0.00 0.00 0.0% 0.0% - validate_warmups_listener.AllSolutionsIterator 1 0.00 0.00 0.0% 0.0% - validate_warmups_listener.ProgressListener 1 0.00 0.00 0.0% 0.0% - validate_warmups_listener.DataInitialization 1 0.00 0.00 0.0% 0.0% - Kernel launch & execution 0.18 0.0% 0.0% - kernel_solving 1 0.08 0.08 45.7% 0.0% - warmup_runs 2 0.08 0.04 43.5% 0.0% - warmup_events_setup 1 0.02 0.02 8.2% 0.0% - validate_warmups_listener.BenchmarkTimer 1 0.00 0.00 2.4% 0.0% - validate_gpu_sync 1 0.00 0.00 100.0% 0.0% - benchmark_runs 1 0.00 0.00 0.2% 0.0% - benchmark_pre_syncs 1 0.00 0.00 50.0% 0.0% - benchmark_post_syncs 1 0.00 0.00 50.0% 0.0% - MEASUREMENT OVERHEAD (instrumentation) 362.86 0.0% 0.0% - main_timing_calibration 32 346.48 10.83 95.5% 0.0% - flush_timing_buffer 32 15.18 0.47 4.2% 0.0% - (timer overhead subtracted from parent timers) 0.60 0.2% 0.0% - timing_overhead 32 0.60 0.02 0.2% 0.0% - UNATTRIBUTED (untracked time & timer overhead) 11,248.20 0.4% 0.4% - (untracked in python_benchmark_problems) 11,227.86 99.8% 0.4% - (untracked in python_client_execution) 13.36 0.1% 0.0% - (untracked in pre_solution) 6.21 0.1% 0.0% - (untracked in pre_problem) 0.69 0.0% 0.0% - (untracked in post_solution) 0.08 0.0% 0.0% - -TIMING BY TIMER NESTING (legacy view) ------------------------------------------------------------------------------------------------------------------------------ - Category Count Total (ms) Mean (ms) % of Parent % of Wall ------------------------------------------------------------------------------------------------------------------------------ - python_benchmark_problems 184 2,577,849.18 14,010.05 100.0% - ------------------------------------------------------------------------------------------------------------------------- - python_kernel_compilation 855 2,203,797.86 2,577.54 85.5% 85.5% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_build_src_co 855 1,007,817.04 1,178.73 45.7% 39.1% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_kernel_build_src_co.compile 263 1,004,724.33 3,820.24 99.7% 39.0% - python_kernel_build_src_co.unbundle 263 1,566.60 5.96 0.2% 0.1% - python_kernel_build_src_co.cache_check 855 1,172.50 1.37 0.1% 0.0% - python_kernel_build_src_co.cache_populate 263 214.63 0.82 0.0% 0.0% - (untracked) 98.65 0.0% 0.0% - python_kernel_build_src_co.setup 855 22.87 0.03 0.0% 0.0% - python_kernel_build_src_co.move 263 17.46 0.07 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_codegen 855 592,584.11 693.08 26.9% 23.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_write_assemble 855 227,868.45 266.51 10.3% 8.8% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_bench_setup 855 131,638.45 153.96 6.0% 5.1% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_bench_postprocess 855 106,816.89 124.93 4.8% 4.1% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_benchpost_lib_construction 855 74,773.69 87.45 70.0% 2.9% - python_benchpost_library_write 855 22,324.17 26.11 20.9% 0.9% - python_benchpost_naming 855 9,436.66 11.04 8.8% 0.4% - python_benchpost_client_config 855 249.07 0.29 0.2% 0.0% - (untracked) 33.30 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_validate 855 81,431.47 95.24 3.7% 3.2% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_setup 855 20,886.91 24.43 0.9% 0.8% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_build_co 855 18,912.53 22.12 0.9% 0.7% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_write_helpers 855 11,912.59 13.93 0.5% 0.5% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 3,929.43 0.2% 0.2% - ------------------------------------------------------------------------------------------------------------------------- - python_solution_generation 823 306,550.42 372.48 11.9% 11.9% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_forked_solutions 823 306,464.54 372.37 100.0% 11.9% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_custom_kernels 823 67.99 0.08 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 13.06 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_solgen_fork_permutations 823 4.83 0.01 0.0% 0.0% - ------------------------------------------------------------------------------------------------------------------------- - python_write_solutions 855 14,069.93 16.46 0.5% 0.5% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_dump 855 13,390.16 15.66 95.2% 0.5% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_prepare 855 531.38 0.62 3.8% 0.0% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_wsol_prepare_nocache 855 529.92 0.62 99.7% 0.0% - (untracked) 1.47 0.3% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 135.75 1.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_header 855 12.63 0.01 0.1% 0.0% - ------------------------------------------------------------------------------------------------------------------------- - python_ductile_initial_sampling 8 13,760.99 1,720.12 0.5% 0.5% - ------------------------------------------------------------------------------------------------------------------------- - python_client_execution 32 13,377.67 418.05 0.5% 0.5% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - C++ Client Setup & Teardown 10,297.27 77.0% 0.4% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - hip_initialization 32 4,712.80 147.27 45.8% 0.2% - process_exit 32 3,467.40 108.36 33.7% 0.1% - main_problem_factory 32 595.29 18.60 5.8% 0.0% - factory_hip_get_device_properties 4 590.74 147.68 99.2% 0.0% - (untracked) 4.56 0.8% 0.0% - process_startup 32 468.60 14.64 4.6% 0.0% - main_teardown_locals 32 386.04 12.06 3.7% 0.0% - main_timing_calibration 32 346.48 10.83 3.4% 0.0% - data_init_setup 32 97.08 3.03 0.9% 0.0% - code_object_loading 32 96.45 3.01 0.9% 0.0% - library_loading 32 65.62 2.05 0.6% 0.0% - lazy_loading_init 32 45.39 1.42 0.4% 0.0% - main_parse_args 32 11.80 0.37 0.1% 0.0% - reporter_setup 32 1.84 0.06 0.0% 0.0% - python_client_write_run_script 32 1.63 0.05 0.0% 0.0% - listener_setup 32 0.52 0.02 0.0% 0.0% - solution_iterator_setup 32 0.33 0.01 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Data Preparation 2,518.74 18.8% 0.1% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - gpu_input_preparation 40 2,387.72 59.69 94.8% 0.1% - gpu_input_prep_init_cpu_inputs 39 1,309.51 33.58 54.8% 0.1% - gpu_input_prep_copy_valid_to_gpu 40 937.87 23.45 39.3% 0.0% - gpu_input_prep_copy_swizzled 4 135.91 33.98 5.7% 0.0% - gpu_input_prep_batched_inputs 40 2.63 0.07 0.1% 0.0% - gpu_input_prep_copy_inputs_d2d 40 0.98 0.02 0.0% 0.0% - gpu_input_prep_constant_inputs 39 0.64 0.02 0.0% 0.0% - gpu_input_prep_convert_to_inputs 40 0.11 0.00 0.0% 0.0% - (untracked) 0.07 0.0% 0.0% - gpu_input_prep_slow_path 40 0.00 0.00 0.0% 0.0% - pre_problem 40 122.83 3.07 4.9% 0.0% - cpu_data_init 1 119.66 119.66 97.4% 0.0% - cpu_reference_gemm 1 2.47 2.47 2.0% 0.0% - solve_cpu_slow 1 2.46 2.46 99.5% 0.0% - (untracked) 0.01 0.5% 0.0% - (untracked) 0.69 0.6% 0.0% - rotating_buffer_preparation 40 6.46 0.16 0.3% 0.0% - gpu_input_reset 1 1.73 1.73 0.1% 0.0% - gpu_input_reset_copy_valid_to_gpu 1 0.82 0.82 47.6% 0.0% - gpu_input_reset_copy_swizzled 1 0.81 0.81 46.9% 0.0% - gpu_input_reset_batched_inputs 1 0.06 0.06 3.7% 0.0% - gpu_input_reset_copy_inputs_d2d 1 0.03 0.03 1.7% 0.0% - gpu_input_reset_convert_to_inputs 1 0.00 0.00 0.0% 0.0% - gpu_input_reset_slow_path 1 0.00 0.00 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Kernel Execution 530.31 4.0% 0.0% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - pre_solution 585 518.39 0.89 97.8% 0.0% - pre_solution_listener.DataInitialization 585 488.60 0.84 94.3% 0.0% - pre_solution_mx_init_data 130 369.08 2.84 75.5% 0.0% - pre_solution_mx_side_regenerate 8 369.03 46.13 100.0% 0.0% - (untracked) 0.02 0.0% 0.0% - pre_solution_mx_side_cache_hit 252 0.02 0.00 0.0% 0.0% - pre_solution_mx_copy_valid_to_gpu_no_regen 126 103.07 0.82 21.1% 0.0% - pre_solution_mx_cpu_sync_no_regen 126 8.60 0.07 1.8% 0.0% - pre_solution_mx_copy_valid_to_gpu_after_regen 4 3.29 0.82 0.7% 0.0% - pre_solution_mx_copy_inputs_d2d_no_regen 126 3.18 0.03 0.7% 0.0% - pre_solution_mx_cpu_sync_after_regen 4 1.19 0.30 0.2% 0.0% - pre_solution_mx_copy_inputs_d2d_after_regen 4 0.13 0.03 0.0% 0.0% - (untracked) 0.03 0.0% 0.0% - pre_solution_mx_check 585 0.03 0.00 0.0% 0.0% - pre_solution_listener.AllSolutionsIterator 585 10.23 0.02 2.0% 0.0% - (untracked) 6.21 1.2% 0.0% - pre_solution_listener.ReferenceValidator 585 4.64 0.01 0.9% 0.0% - cpu_reference_gemm_per_solution_inputs_changed 1 4.62 4.62 99.5% 0.0% - solve_cpu_slow_per_solution_inputs_changed 1 4.62 4.62 100.0% 0.0% - (untracked) 0.02 0.5% 0.0% - pre_solution_listener.BenchmarkTimer 585 3.99 0.01 0.8% 0.0% - pre_solution_listener.ProgressListener 585 2.60 0.00 0.5% 0.0% - pre_solution_listener.MetaResultReporter 585 2.00 0.00 0.4% 0.0% - pre_solution_listener.SynchronizerValidator 585 0.10 0.00 0.0% 0.0% - pre_solution_listener.HardwareMonitorListener 585 0.01 0.00 0.0% 0.0% - post_solution 585 7.33 0.01 1.4% 0.0% - post_solution_log 585 5.20 0.01 70.9% 0.0% - post_solution_reporting 585 1.75 0.00 23.9% 0.0% - post_solution_perf_calc 585 0.25 0.00 3.5% 0.0% - (untracked) 0.08 1.1% 0.0% - post_solution_result_file 585 0.01 0.00 0.2% 0.0% - post_solution_validation 585 0.01 0.00 0.2% 0.0% - post_solution_sol_advance 585 0.01 0.00 0.2% 0.0% - post_solution_perf_reset 585 0.01 0.00 0.2% 0.0% - run_current_solution_check 585 2.03 0.00 0.4% 0.0% - post_problem 40 1.66 0.04 0.3% 0.0% - finalize_report 32 0.35 0.01 0.1% 0.0% - problem_report_index 40 0.15 0.00 0.0% 0.0% - solution_selection 585 0.14 0.00 0.0% 0.0% - kernel_solving 1 0.08 0.08 0.0% 0.0% - warmup_runs 2 0.08 0.04 0.0% 0.0% - problem_teardown 40 0.06 0.00 0.0% 0.0% - solution_teardown 585 0.02 0.00 0.0% 0.0% - warmup_events_setup 1 0.02 0.02 0.0% 0.0% - run_iteration_teardown 1 0.01 0.01 0.0% 0.0% - benchmark_runs 1 0.00 0.00 0.0% 0.0% - benchmark_pre_syncs 1 0.00 0.00 31.1% 0.0% - benchmark_post_syncs 1 0.00 0.00 31.1% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Timing Overhead 15.78 0.1% 0.0% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - flush_timing_buffer 32 15.18 0.47 96.2% 0.0% - timing_overhead 32 0.60 0.02 3.8% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked C++ overhead) 13.36 0.1% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Reference Computation 2.22 0.0% 0.0% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - validate_warmups 1 2.22 2.22 100.0% 0.0% - validate_warmups_listener.ReferenceValidator 1 1.26 1.26 56.8% 0.0% - validate_result_buffer_alloc 1 0.94 0.94 74.6% 0.0% - validate_gpu_readback 1 0.30 0.30 24.1% 0.0% - validate_element_comparison 1 0.01 0.01 0.7% 0.0% - validate_inputs_copy 1 0.00 0.00 0.1% 0.0% - validate_report 1 0.00 0.00 0.0% 0.0% - validate_setup 1 0.00 0.00 0.0% 0.0% - validate_warmups_listener.SynchronizerValidator 1 0.94 0.94 42.6% 0.0% - validate_synchronizer_check 1 0.94 0.94 100.0% 0.0% - validate_synchronizer_readback 1 0.78 0.78 82.5% 0.0% - validate_synchronizer_scan 1 0.16 0.16 17.4% 0.0% - validate_warmups_listener.BenchmarkTimer 1 0.01 0.01 0.2% 0.0% - validate_gpu_sync 1 0.00 0.00 85.9% 0.0% - validate_warmups_listener.MetaResultReporter 1 0.00 0.00 0.0% 0.0% - validate_warmups_listener.HardwareMonitorListener 1 0.00 0.00 0.0% 0.0% - validate_warmups_listener.AllSolutionsIterator 1 0.00 0.00 0.0% 0.0% - validate_warmups_listener.ProgressListener 1 0.00 0.00 0.0% 0.0% - validate_warmups_listener.DataInitialization 1 0.00 0.00 0.0% 0.0% - ------------------------------------------------------------------------------------------------------------------------- - (untracked) 11,227.86 0.4% 0.4% - ------------------------------------------------------------------------------------------------------------------------- - python_solution_indexing 855 8,279.91 9.68 0.3% 0.3% - ------------------------------------------------------------------------------------------------------------------------- - python_ductile_generate_solutions 32 4,536.07 141.75 0.2% 0.2% - ------------------------------------------------------------------------------------------------------------------------- - python_ductile_ga_between_evaluations 8 1,366.26 170.78 0.1% 0.1% - ------------------------------------------------------------------------------------------------------------------------- - python_write_cache 855 830.02 0.97 0.0% 0.0% - ------------------------------------------------------------------------------------------------------------------------- - python_ductile_setup 8 30.08 3.76 0.0% 0.0% - ------------------------------------------------------------------------------------------------------------------------- - python_ductile_clear_artifacts 24 16.81 0.70 0.0% 0.0% - ------------------------------------------------------------------------------------------------------------------------- - python_ductile_read_scores 32 4.29 0.13 0.0% 0.0% - ------------------------------------------------------------------------------------------------------------------------- - python_cache_check 831 0.67 0.00 0.0% 0.0% - ------------------------------------------------------------------------------------------------------------------------- - python_ductile_verification 8 0.33 0.04 0.0% 0.0% - - Overlapping Totals (wrap other timers; not summed into the tree) - --------------------------------------------------------------------------------------------------------------------------- - python_client_subprocess 32 13,365.64 417.68 0.5% - main_total 32 9,027.25 282.10 0.4% - problem_total 40 3,051.85 76.30 0.1% - solution_total 585 532.32 0.91 0.0% - run_iteration_total 1 4.14 4.14 0.0% - validate_total 1 1.25 1.25 0.0% - validate_check_results_typed 1 1.25 1.25 0.0% - ------------------------------------------------------------------------------------------------------------------------------ -HIERARCHY VALIDATION: OK (all child phases fit within their parents) - -TIME BREAKDOWN (visual, % of wall clock) ------------------------------------------------------------------------------------------------------------------------------ - TOTAL RUNTIME 100.0% ################################################## - BUILD: solution generation & compile 98.1% ################################################# - Solution generation & build (Python) 97.4% ################################################ - python_kernel_compilation 85.5% ########################################## - python_kernel_build_src_co 39.1% ################### - python_kernel_build_src_co.compile 39.0% ################### - python_kernel_build_src_co.unbundle 0.1% - python_kernel_build_src_co.cache_check 0.0% - python_kernel_build_src_co.cache_populate 0.0% - python_kernel_build_src_co.setup 0.0% - python_kernel_build_src_co.move 0.0% - (untracked) 0.0% - python_kernel_codegen 23.0% ########### - python_kernel_write_assemble 8.8% #### - python_kernel_bench_setup 5.1% ## - python_kernel_bench_postprocess 4.1% ## - python_benchpost_lib_construction 2.9% # - python_benchpost_library_write 0.9% - python_benchpost_naming 0.4% - python_benchpost_client_config 0.0% - (untracked) 0.0% - python_kernel_validate 3.2% # - python_kernel_setup 0.8% - python_kernel_build_co 0.7% - python_kernel_write_helpers 0.5% - (untracked) 0.2% - python_solution_generation 11.9% ##### - python_solgen_forked_solutions 11.9% ##### - python_solgen_custom_kernels 0.0% - python_solgen_fork_permutations 0.0% - (untracked) 0.0% - Ductile GA search (Python) 0.8% - python_ductile_initial_sampling 0.5% - python_ductile_generate_solutions 0.2% - python_ductile_ga_between_evaluations 0.1% - python_ductile_setup 0.0% - python_ductile_clear_artifacts 0.0% - python_ductile_read_scores 0.0% - python_ductile_verification 0.0% - SHARED: cache & solution files 0.9% - python_write_solutions 0.5% - python_wsol_dump 0.5% - python_wsol_prepare 0.0% - python_wsol_prepare_nocache 0.0% - (untracked) 0.0% - python_wsol_header 0.0% - (untracked) 0.0% - python_solution_indexing 0.3% - python_write_cache 0.0% - python_cache_check 0.0% - RUN: benchmark client 0.5% - Client launch (startup, setup & teardown) 0.4% - hip_initialization 0.2% - process_exit 0.1% - main_problem_factory 0.0% - factory_hip_get_device_properties 0.0% - (untracked) 0.0% - process_startup 0.0% - main_teardown_locals 0.0% - data_init_setup 0.0% - code_object_loading 0.0% - library_loading 0.0% - lazy_loading_init 0.0% - main_parse_args 0.0% - reporter_setup 0.0% - python_client_write_run_script 0.0% - listener_setup 0.0% - solution_iterator_setup 0.0% - Input data preparation 0.1% - gpu_input_preparation 0.1% - gpu_input_prep_init_cpu_inputs 0.1% - gpu_input_prep_copy_valid_to_gpu 0.0% - gpu_input_prep_copy_swizzled 0.0% - gpu_input_prep_batched_inputs 0.0% - gpu_input_prep_copy_inputs_d2d 0.0% - gpu_input_prep_constant_inputs 0.0% - gpu_input_prep_convert_to_inputs 0.0% - gpu_input_prep_slow_path 0.0% - (untracked) 0.0% - pre_solution_listener.DataInitialization 0.0% - pre_solution_mx_init_data 0.0% - pre_solution_mx_side_regenerate 0.0% - pre_solution_mx_side_cache_hit 0.0% - (untracked) 0.0% - pre_solution_mx_copy_valid_to_gpu_no_regen 0.0% - pre_solution_mx_cpu_sync_no_regen 0.0% - pre_solution_mx_copy_valid_to_gpu_after_regen 0.0% - pre_solution_mx_copy_inputs_d2d_no_regen 0.0% - pre_solution_mx_cpu_sync_after_regen 0.0% - pre_solution_mx_copy_inputs_d2d_after_regen 0.0% - pre_solution_mx_check 0.0% - (untracked) 0.0% - cpu_data_init 0.0% - rotating_buffer_preparation 0.0% - gpu_input_reset 0.0% - gpu_input_reset_copy_valid_to_gpu 0.0% - gpu_input_reset_copy_swizzled 0.0% - gpu_input_reset_batched_inputs 0.0% - gpu_input_reset_copy_inputs_d2d 0.0% - gpu_input_reset_convert_to_inputs 0.0% - gpu_input_reset_slow_path 0.0% - Per-solution bookkeeping & reporting 0.0% - pre_solution_listener.AllSolutionsIterator 0.0% - post_solution_log 0.0% - pre_solution_listener.BenchmarkTimer 0.0% - pre_solution_listener.ProgressListener 0.0% - run_current_solution_check 0.0% - pre_solution_listener.MetaResultReporter 0.0% - post_solution_reporting 0.0% - post_problem 0.0% - finalize_report 0.0% - post_solution_perf_calc 0.0% - problem_report_index 0.0% - solution_selection 0.0% - pre_solution_listener.SynchronizerValidator 0.0% - problem_teardown 0.0% - solution_teardown 0.0% - post_solution_result_file 0.0% - post_solution_validation 0.0% - pre_solution_listener.HardwareMonitorListener 0.0% - post_solution_sol_advance 0.0% - post_solution_perf_reset 0.0% - run_iteration_teardown 0.0% - Reference GEMM (CPU) 0.0% - pre_solution_listener.ReferenceValidator 0.0% - cpu_reference_gemm_per_solution_inputs_changed 0.0% - solve_cpu_slow_per_solution_inputs_changed 0.0% - (untracked) 0.0% - cpu_reference_gemm 0.0% - solve_cpu_slow 0.0% - (untracked) 0.0% - Validation 0.0% - validate_warmups_listener.ReferenceValidator 0.0% - validate_result_buffer_alloc 0.0% - validate_gpu_readback 0.0% - validate_element_comparison 0.0% - validate_inputs_copy 0.0% - validate_report 0.0% - validate_setup 0.0% - validate_warmups_listener.SynchronizerValidator 0.0% - validate_synchronizer_check 0.0% - validate_synchronizer_readback 0.0% - validate_synchronizer_scan 0.0% - validate_warmups_listener.MetaResultReporter 0.0% - validate_warmups_listener.HardwareMonitorListener 0.0% - validate_warmups_listener.AllSolutionsIterator 0.0% - validate_warmups_listener.ProgressListener 0.0% - validate_warmups_listener.DataInitialization 0.0% - Kernel launch & execution 0.0% - kernel_solving 0.0% - warmup_runs 0.0% - warmup_events_setup 0.0% - validate_warmups_listener.BenchmarkTimer 0.0% - validate_gpu_sync 0.0% - benchmark_runs 0.0% - benchmark_pre_syncs 0.0% - benchmark_post_syncs 0.0% - MEASUREMENT OVERHEAD (instrumentation) 0.0% - main_timing_calibration 0.0% - flush_timing_buffer 0.0% - (timer overhead subtracted from parent timers) 0.0% - timing_overhead 0.0% - UNATTRIBUTED (untracked time & timer overhead) 0.4% - (untracked in python_benchmark_problems) 0.4% - (untracked in python_client_execution) 0.0% - (untracked in pre_solution) 0.0% - (untracked in pre_problem) 0.0% - (untracked in post_solution) 0.0% - -KEY INSIGHTS (% of wall clock) ------------------------------------------------------------------------------------------------------------------------------ - Solution generation & build (Python) 2510.35s (97.4%) - SHARED: cache & solution files 23.18s (0.9%) - Ductile GA search (Python) 19.71s (0.8%) - Client launch (startup, setup & teardown) 9.95s (0.4%) - Input data preparation 3.00s (0.1%) - MEASUREMENT OVERHEAD (instrumentation) 0.36s (0.0%) - Per-solution bookkeeping & reporting 0.03s (0.0%) - Reference GEMM (CPU) 0.01s (0.0%) - Validation 0.00s (0.0%) - Kernel launch & execution 0.00s (0.0%) - UNATTRIBUTED (untracked time & timer overhead) 11.25s (0.4%) - - -TOP 10 SLOWEST PROBLEMS (by CPU reference time) ------------------------------------------------------------------------------------------------------------------------------ - M N K Batch TypeA TypeD CPU Ref (ms) GPU (ms) - --------------------------------------------------------------------------------------------------------------------------- - 2048 256 128 16 Float8 BFloat16 2.47 0.00 - diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_rerun/warm_cache_check.sh b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_rerun/warm_cache_check.sh deleted file mode 100644 index 2111c60f97bc..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_rerun/warm_cache_check.sh +++ /dev/null @@ -1,28 +0,0 @@ -#!/bin/bash -# Copyright Advanced Micro Devices, Inc., or its affiliates. -# SPDX-License-Identifier: MIT -# -# Does the LibraryClient phase hit the helper-kernel cache when ~/.tensile is NOT cleared between runs? -# Clears ~/.tensile once, then runs the same config twice (cache cold, then warm), saving the analysis -# of each. Usage: bash warm_cache_check.sh gemm/gfx950/subtile_mxfp8_tail_loop_smoke -TL=/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite -CFG=$1 -NAME=$(basename $CFG) -OUT=$TL/timing_runs/full_runs_gfx950/per_config_rerun/warm_cache_check/$NAME -ART=/tmp/tl_split_gfx950/artifacts -CLIENT=/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client -mkdir -p $OUT -cd $TL || exit 1 -export OMP_NUM_THREADS=25 OPENBLAS_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=0 PYTHONUNBUFFERED=1 -TO="--cxx-compiler,/usr/bin/amdclang++,--global-parameters,TimingInstrumentation=True" -RID="tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/$CFG.yaml]" -rm -rf /home/alvasile/.tensile /home/alvasile/.tensilelite -for pass in cold warm; do - taskset -c 0-24 pytest "$RID" --use-cache --artifact-dir $ART --prebuilt-client=$CLIENT \ - -s -v -n 0 --timeout=5400 -m "common and not skip-gfx94X" \ - "--global-parameters=LibraryFormat='msgpack'" "--tensile-options=$TO" -p no:warnings -p no:cacheprovider \ - > $OUT/$pass.log 2>&1 - echo "$pass rc=$?" >> $OUT/times.txt - python scripts/analyze_timing.py $OUT/$pass.log > $OUT/${pass}_analysis.txt 2>&1 - ls /home/alvasile/.tensile/helper_cache >> $OUT/times.txt 2>&1 -done diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_rerun_2/AGGREGATE_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_rerun_2/AGGREGATE_analysis.txt deleted file mode 100644 index fba65d8aa12a..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/per_config_rerun_2/AGGREGATE_analysis.txt +++ /dev/null @@ -1,754 +0,0 @@ -================================================================================================================================= -TIMING ANALYSIS SUMMARY -================================================================================================================================= - -TIMING BY PURPOSE ---------------------------------------------------------------------------------------------------------------------------------- - Category Count Total (ms) Mean (ms) % of Parent % of Wall ---------------------------------------------------------------------------------------------------------------------------------- - TOTAL RUNTIME 2,568,261.07 100.0% - RUN: benchmark client 2,330,795.57 90.8% 90.8% - Reference GEMM (CPU) 1,080,469.12 46.4% 42.1% - cpu_reference_gemm 28,596 928,712.97 32.48 86.0% 36.2% - solve_cpu_slow 1,947 727,756.12 373.78 78.4% 28.3% - solve_cpu_fast 26,656 200,931.33 7.54 21.6% 7.8% - (untracked) 25.51 0.0% 0.0% - pre_solution_listener.ReferenceValidator 2,355,016 151,756.15 0.06 14.0% 5.9% - cpu_reference_gemm_per_solution_inputs_changed 1,146 151,668.42 132.35 99.9% 5.9% - solve_cpu_fast_per_solution_inputs_changed 1,127 138,501.63 122.89 91.3% 5.4% - solve_cpu_slow_per_solution_inputs_changed 19 13,166.07 692.95 8.7% 0.5% - (untracked) 0.72 0.0% 0.0% - pre_solution_ref_skipped 2,353,182 49.50 0.00 0.0% 0.0% - (untracked) 38.23 0.0% 0.0% - Input data preparation 502,878.60 21.6% 19.6% - gpu_input_reset 2,106,835 306,247.35 0.15 60.9% 11.9% - gpu_input_reset_copy_valid_to_gpu 719,180 177,839.34 0.25 58.1% 6.9% - gpu_input_reset_batched_inputs 1,473,329 95,916.03 0.07 31.3% 3.7% - gpu_input_reset_copy_inputs_d2d 1,473,329 26,309.71 0.02 8.6% 1.0% - gpu_input_reset_copy_swizzled 15,719 4,006.86 0.25 1.3% 0.2% - gpu_input_reset_fast_reset_output 633,454 1,917.65 0.00 0.6% 0.1% - gpu_input_reset_convert_to_inputs 1,473,329 139.58 0.00 0.0% 0.0% - gpu_input_reset_slow_path 1,473,329 34.85 0.00 0.0% 0.0% - gpu_input_reset_rotating_init 2 0.02 0.01 0.0% 0.0% - gpu_input_reset_constant_inputs 4 0.01 0.00 0.0% 0.0% - gpu_input_reset_init_cpu_inputs 4 0.00 0.00 0.0% 0.0% - (untracked) 83.31 0.0% 0.0% - cpu_data_init 28,596 106,829.68 3.74 21.2% 4.2% - pre_solution_listener.DataInitialization 2,355,016 49,481.67 0.02 9.8% 1.9% - pre_solution_mx_init_data 16,038 44,073.15 2.75 89.1% 1.7% - pre_solution_mx_side_regenerate 2,298 44,069.34 19.18 100.0% 1.7% - pre_solution_mx_side_cache_hit 29,778 2.60 0.00 0.0% 0.0% - (untracked) 1.21 0.0% 0.0% - pre_solution_mx_copy_valid_to_gpu_no_regen 14,889 3,124.70 0.21 6.3% 0.1% - pre_solution_mx_copy_valid_to_gpu_after_regen 1,149 1,143.05 0.99 2.3% 0.0% - pre_solution_mx_cpu_sync_after_regen 1,149 438.52 0.38 0.9% 0.0% - pre_solution_mx_copy_inputs_d2d_no_regen 14,889 307.63 0.02 0.6% 0.0% - pre_solution_mx_cpu_sync_no_regen 14,889 307.33 0.02 0.6% 0.0% - pre_solution_mx_check 2,355,016 49.85 0.00 0.1% 0.0% - pre_solution_mx_copy_inputs_d2d_after_regen 1,149 22.97 0.02 0.0% 0.0% - (untracked) 14.47 0.0% 0.0% - gpu_input_preparation 28,641 39,596.08 1.38 7.9% 1.5% - gpu_input_prep_copy_valid_to_gpu 11,446 22,935.66 2.00 57.9% 0.9% - gpu_input_prep_copy_swizzled 1,151 10,766.76 9.35 27.2% 0.4% - gpu_input_prep_init_cpu_inputs 41 4,029.28 98.28 10.2% 0.2% - gpu_input_prep_batched_inputs 23,229 1,410.04 0.06 3.6% 0.1% - gpu_input_prep_copy_inputs_d2d 23,229 411.51 0.02 1.0% 0.0% - gpu_input_prep_fast_reset_output 5,403 22.52 0.00 0.1% 0.0% - gpu_input_prep_convert_to_inputs 23,229 9.26 0.00 0.0% 0.0% - gpu_input_prep_slow_path 23,229 1.96 0.00 0.0% 0.0% - gpu_input_prep_constant_inputs 43 0.73 0.02 0.0% 0.0% - gpu_input_prep_rotating_init 1 0.01 0.01 0.0% 0.0% - (untracked) 8.37 0.0% 0.0% - rotating_buffer_preparation 28,641 723.82 0.03 0.1% 0.0% - Validation 323,888.88 13.9% 12.6% - validate_warmups_listener.SynchronizerValidator 2,106,827 162,734.88 0.08 50.2% 6.3% - validate_synchronizer_check 2,106,827 162,692.31 0.08 100.0% 6.3% - validate_synchronizer_scan 608,873 106,747.41 0.18 65.6% 4.2% - validate_synchronizer_readback 608,873 55,816.84 0.09 34.3% 2.2% - validate_synchronizer_inactive 1,497,978 27.90 0.00 0.0% 0.0% - (untracked) 100.16 0.1% 0.0% - (untracked) 42.58 0.0% 0.0% - validate_warmups_listener.ReferenceValidator 2,106,827 160,728.73 0.08 49.6% 6.3% - validate_gpu_readback 1,907,339 91,856.16 0.05 57.1% 3.6% - validate_element_comparison 1,907,339 59,265.75 0.03 36.9% 2.3% - validate_result_buffer_alloc 1,907,339 9,020.90 0.00 5.6% 0.4% - validate_inputs_copy 1,905,787 194.93 0.00 0.1% 0.0% - validate_report 1,907,339 43.63 0.00 0.0% 0.0% - validate_setup 1,905,883 43.37 0.00 0.0% 0.0% - (untracked) 303.99 0.2% 0.0% - validate_warmups_listener.MetaResultReporter 2,106,827 149.34 0.00 0.0% 0.0% - validate_warmups_listener.HardwareMonitorListener 2,106,827 74.67 0.00 0.0% 0.0% - validate_warmups_listener.ProgressListener 2,106,827 70.80 0.00 0.0% 0.0% - validate_warmups_listener.AllSolutionsIterator 2,106,827 66.34 0.00 0.0% 0.0% - validate_warmups_listener.DataInitialization 2,106,827 64.11 0.00 0.0% 0.0% - Client launch (startup, setup & teardown) 284,745.47 12.2% 11.1% - hip_initialization 855 124,448.44 145.55 43.7% 4.8% - process_exit 855 64,634.38 75.60 22.7% 2.5% - main_problem_factory 855 29,154.81 34.10 10.2% 1.1% - factory_hip_get_device_properties 192 28,922.86 150.64 99.2% 1.1% - (untracked) 231.95 0.8% 0.0% - data_init_setup 855 18,185.96 21.27 6.4% 0.7% - main_teardown_locals 855 13,182.66 15.42 4.6% 0.5% - process_startup 855 12,710.53 14.87 4.5% 0.5% - code_object_loading 855 11,001.27 12.87 3.9% 0.4% - library_loading 855 9,940.49 11.63 3.5% 0.4% - lazy_loading_init 855 924.59 1.08 0.3% 0.0% - main_parse_args 855 410.00 0.48 0.1% 0.0% - reporter_setup 855 68.52 0.08 0.0% 0.0% - python_client_write_run_script 855 57.11 0.07 0.0% 0.0% - listener_setup 855 15.77 0.02 0.0% 0.0% - solution_iterator_setup 855 10.94 0.01 0.0% 0.0% - Per-solution bookkeeping & reporting 68,333.60 2.9% 2.7% - post_solution_log 2,355,016 28,760.07 0.01 42.1% 1.1% - pre_solution_listener.BenchmarkTimer 2,355,016 11,859.71 0.01 17.4% 0.5% - run_current_solution_check 2,355,016 8,420.00 0.00 12.3% 0.3% - post_solution_reporting 2,355,016 5,582.67 0.00 8.2% 0.2% - pre_solution_listener.ProgressListener 2,355,016 3,134.22 0.00 4.6% 0.1% - pre_solution_listener.AllSolutionsIterator 2,355,016 2,647.18 0.00 3.9% 0.1% - post_problem 28,641 1,945.83 0.07 2.8% 0.1% - run_iteration_teardown 2,106,835 1,677.91 0.00 2.5% 0.1% - pre_solution_listener.MetaResultReporter 2,355,016 1,159.39 0.00 1.7% 0.0% - post_solution_validation 2,355,016 740.26 0.00 1.1% 0.0% - post_solution_perf_calc 2,355,016 652.16 0.00 1.0% 0.0% - solution_selection 2,355,016 571.95 0.00 0.8% 0.0% - finalize_report 855 507.49 0.59 0.7% 0.0% - pre_solution_listener.SynchronizerValidator 2,355,016 358.33 0.00 0.5% 0.0% - solution_teardown 2,355,016 77.56 0.00 0.1% 0.0% - post_solution_result_file 2,355,016 57.99 0.00 0.1% 0.0% - post_solution_perf_reset 2,355,016 52.77 0.00 0.1% 0.0% - post_solution_sol_advance 2,355,016 50.05 0.00 0.1% 0.0% - pre_solution_listener.HardwareMonitorListener 2,355,016 47.08 0.00 0.1% 0.0% - problem_report_index 28,641 26.02 0.00 0.0% 0.0% - problem_teardown 28,641 4.94 0.00 0.0% 0.0% - Kernel launch & execution 40,886.54 1.8% 1.6% - warmup_runs 4,213,654 25,917.98 0.01 63.4% 1.0% - validate_warmups_listener.BenchmarkTimer 2,106,827 6,922.29 0.00 16.9% 0.3% - validate_gpu_sync 2,106,827 6,867.77 0.00 99.2% 0.3% - (untracked) 54.52 0.8% 0.0% - kernel_solving 2,106,835 6,730.10 0.00 16.5% 0.3% - warmup_events_setup 2,106,835 970.97 0.00 2.4% 0.0% - benchmark_runs 2,106,835 345.20 0.00 0.8% 0.0% - benchmark_validate_enqueues 319 181.62 0.57 52.6% 0.0% - validate_enqueues_listener.BenchmarkTimer 319 179.53 0.56 98.8% 0.0% - benchmark_sleep 319 179.48 0.56 100.0% 0.0% - benchmark_event_elapsed_time 8 0.01 0.00 0.0% 0.0% - benchmark_event_destroy 8 0.01 0.00 0.0% 0.0% - (untracked) 0.03 0.0% 0.0% - validate_enqueues_listener.ProgressListener 319 1.81 0.01 1.0% 0.0% - validate_enqueues_listener.MetaResultReporter 319 0.02 0.00 0.0% 0.0% - validate_enqueues_listener.HardwareMonitorListener 319 0.01 0.00 0.0% 0.0% - validate_enqueues_listener.AllSolutionsIterator 319 0.01 0.00 0.0% 0.0% - validate_enqueues_listener.DataInitialization 319 0.01 0.00 0.0% 0.0% - validate_enqueues_listener.SynchronizerValidator 319 0.01 0.00 0.0% 0.0% - validate_enqueues_listener.ReferenceValidator 319 0.01 0.00 0.0% 0.0% - (untracked) 0.21 0.1% 0.0% - benchmark_pre_syncs 2,106,835 56.09 0.00 16.2% 0.0% - benchmark_post_syncs 2,106,835 54.72 0.00 15.9% 0.0% - gpu_kernel_execution 319 39.52 0.12 11.4% 0.0% - benchmark_events_setup 319 0.06 0.00 0.0% 0.0% - benchmark_events_teardown 319 0.03 0.00 0.0% 0.0% - (untracked) 13.15 3.8% 0.0% - Library client (Python) 23,840.65 1.0% 0.9% - python_library_client_create_library 10 20,540.37 2,054.04 86.2% 0.8% - python_library_client_execution 10 3,161.37 316.14 13.3% 0.1% - python_library_client_subprocess 10 3,155.15 315.52 99.8% 0.1% - python_library_client_write_run_script 10 0.81 0.08 0.0% 0.0% - (untracked) 5.41 0.2% 0.0% - python_library_client_write_configs 10 120.68 12.07 0.5% 0.0% - python_library_client_setup 10 18.23 1.82 0.1% 0.0% - Library logic (Python) 5,752.70 0.2% 0.2% - python_logic_write 26 5,424.03 208.62 94.3% 0.2% - python_logic_parse_solutions 26 259.52 9.98 4.5% 0.0% - python_logic_analyze 26 69.15 2.66 1.2% 0.0% - MEASUREMENT OVERHEAD (instrumentation) 172,815.65 6.7% 6.7% - flush_timing_buffer 865 152,892.88 176.75 88.5% 6.0% - main_timing_calibration 855 10,096.95 11.81 5.8% 0.4% - timing_overhead 855 5,082.90 5.94 2.9% 0.2% - (timer overhead subtracted from parent timers) 4,742.92 2.7% 0.2% - BUILD: solution generation & compile 63,999.91 2.5% 2.5% - Solution generation & build (Python) 44,036.26 68.8% 1.7% - python_kernel_compilation 32 44,036.26 1,376.13 100.0% 1.7% - python_kernel_build_src_co 32 20,333.49 635.42 46.2% 0.8% - python_kernel_build_src_co.compile 4 20,243.96 5,060.99 99.6% 0.8% - python_kernel_build_src_co.cache_check 32 53.80 1.68 0.3% 0.0% - python_kernel_build_src_co.unbundle 4 27.97 6.99 0.1% 0.0% - python_kernel_build_src_co.cache_populate 4 3.20 0.80 0.0% 0.0% - python_kernel_build_src_co.setup 32 0.84 0.03 0.0% 0.0% - python_kernel_build_src_co.move 4 0.26 0.07 0.0% 0.0% - (untracked) 3.45 0.0% 0.0% - python_kernel_codegen 32 16,254.60 507.96 36.9% 0.6% - python_kernel_write_assemble 32 2,214.46 69.20 5.0% 0.1% - python_kernel_bench_setup 32 1,786.93 55.84 4.1% 0.1% - python_kernel_validate 32 1,686.90 52.72 3.8% 0.1% - python_kernel_write_helpers 32 545.76 17.05 1.2% 0.0% - python_kernel_build_co 32 526.58 16.46 1.2% 0.0% - python_kernel_bench_postprocess 32 495.87 15.50 1.1% 0.0% - python_benchpost_lib_construction 32 355.66 11.11 71.7% 0.0% - python_benchpost_library_write 32 84.30 2.63 17.0% 0.0% - python_benchpost_naming 32 50.98 1.59 10.3% 0.0% - python_benchpost_client_config 32 3.91 0.12 0.8% 0.0% - (untracked) 1.03 0.2% 0.0% - python_kernel_setup 32 137.71 4.30 0.3% 0.0% - (untracked) 53.96 0.1% 0.0% - Ductile GA search (Python) 19,963.65 31.2% 0.8% - python_ductile_initial_sampling 8 13,753.39 1,719.17 68.9% 0.5% - python_ductile_generate_solutions 32 4,760.25 148.76 23.8% 0.2% - python_ductile_ga_between_evaluations 8 1,387.06 173.38 6.9% 0.1% - python_ductile_setup 8 32.30 4.04 0.2% 0.0% - python_ductile_clear_artifacts 32 26.20 0.82 0.1% 0.0% - python_ductile_read_scores 32 4.15 0.13 0.0% 0.0% - python_ductile_verification 8 0.30 0.04 0.0% 0.0% - SHARED: cache & solution files 1,580.00 0.1% 0.1% - python_cache_check 831 1,280.29 1.54 81.0% 0.0% - python_write_solutions 855 205.93 0.24 13.0% 0.0% - python_wsol_dump 32 66.32 2.07 32.2% 0.0% - python_wsol_header 855 65.76 0.08 31.9% 0.0% - python_wsol_prepare 855 60.96 0.07 29.6% 0.0% - python_wsol_prepare_cache 823 58.04 0.07 95.2% 0.0% - python_wsol_prepare_nocache 32 0.90 0.03 1.5% 0.0% - (untracked) 2.02 3.3% 0.0% - (untracked) 12.89 6.3% 0.0% - python_write_cache 32 58.71 1.83 3.7% 0.0% - python_solution_indexing 32 35.07 1.10 2.2% 0.0% - UNATTRIBUTED (untracked time & timer overhead) -930.06 -0.0% -0.0% - (untracked in python_benchmark_problems) 2,407.19 0.0% 0.1% - (untracked in validate_warmups) 560.19 0.0% 0.0% - (untracked in pre_solution) 472.96 0.0% 0.0% - (untracked in pre_problem) 255.77 0.0% 0.0% - (untracked in python_library_logic) 7.63 0.0% 0.0% - (untracked in python_client_writer) 7.58 0.0% 0.0% - (children exceed post_solution) -168.12 0.0% -0.0% - (children exceed python_client_execution) -4,473.27 0.0% -0.2% - -TIMING BY TIMER NESTING (legacy view) ---------------------------------------------------------------------------------------------------------------------------------- - Category Count Total (ms) Mean (ms) % of Parent % of Wall ---------------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 184 2,533,909.58 13,771.25 98.7% - ----------------------------------------------------------------------------------------------------------------------------- - python_client_execution 855 2,465,922.48 2,884.12 97.3% 96.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Data Preparation 1,382,365.67 56.1% 53.8% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - pre_problem 28,641 1,035,798.42 36.16 74.9% 40.3% - cpu_reference_gemm 28,596 928,712.97 32.48 89.7% 36.2% - solve_cpu_slow 1,947 727,756.12 373.78 78.4% 28.3% - solve_cpu_fast 26,656 200,931.33 7.54 21.6% 7.8% - (untracked) 25.51 0.0% 0.0% - cpu_data_init 28,596 106,829.68 3.74 10.3% 4.2% - (untracked) 255.77 0.0% 0.0% - gpu_input_reset 2,106,835 306,247.35 0.15 22.2% 11.9% - gpu_input_reset_copy_valid_to_gpu 719,180 177,839.34 0.25 58.1% 6.9% - gpu_input_reset_batched_inputs 1,473,329 95,916.03 0.07 31.3% 3.7% - gpu_input_reset_copy_inputs_d2d 1,473,329 26,309.71 0.02 8.6% 1.0% - gpu_input_reset_copy_swizzled 15,719 4,006.86 0.25 1.3% 0.2% - gpu_input_reset_fast_reset_output 633,454 1,917.65 0.00 0.6% 0.1% - gpu_input_reset_convert_to_inputs 1,473,329 139.58 0.00 0.0% 0.0% - (untracked) 83.31 0.0% 0.0% - gpu_input_reset_slow_path 1,473,329 34.85 0.00 0.0% 0.0% - gpu_input_reset_rotating_init 2 0.02 0.01 0.0% 0.0% - gpu_input_reset_constant_inputs 4 0.01 0.00 0.0% 0.0% - gpu_input_reset_init_cpu_inputs 4 0.00 0.00 0.0% 0.0% - gpu_input_preparation 28,641 39,596.08 1.38 2.9% 1.5% - gpu_input_prep_copy_valid_to_gpu 11,446 22,935.66 2.00 57.9% 0.9% - gpu_input_prep_copy_swizzled 1,151 10,766.76 9.35 27.2% 0.4% - gpu_input_prep_init_cpu_inputs 41 4,029.28 98.28 10.2% 0.2% - gpu_input_prep_batched_inputs 23,229 1,410.04 0.06 3.6% 0.1% - gpu_input_prep_copy_inputs_d2d 23,229 411.51 0.02 1.0% 0.0% - gpu_input_prep_fast_reset_output 5,403 22.52 0.00 0.1% 0.0% - gpu_input_prep_convert_to_inputs 23,229 9.26 0.00 0.0% 0.0% - (untracked) 8.37 0.0% 0.0% - gpu_input_prep_slow_path 23,229 1.96 0.00 0.0% 0.0% - gpu_input_prep_constant_inputs 43 0.73 0.02 0.0% 0.0% - gpu_input_prep_rotating_init 1 0.01 0.01 0.0% 0.0% - rotating_buffer_preparation 28,641 723.82 0.03 0.1% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Reference Computation 331,371.37 13.4% 12.9% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - validate_warmups 2,106,827 331,371.37 0.16 100.0% 12.9% - validate_warmups_listener.SynchronizerValidator 2,106,827 162,734.88 0.08 49.1% 6.3% - validate_synchronizer_check 2,106,827 162,692.31 0.08 100.0% 6.3% - validate_synchronizer_scan 608,873 106,747.41 0.18 65.6% 4.2% - validate_synchronizer_readback 608,873 55,816.84 0.09 34.3% 2.2% - (untracked) 100.16 0.1% 0.0% - validate_synchronizer_inactive 1,497,978 27.90 0.00 0.0% 0.0% - (untracked) 42.58 0.0% 0.0% - validate_warmups_listener.ReferenceValidator 2,106,827 160,728.73 0.08 48.5% 6.3% - validate_gpu_readback 1,907,339 91,856.16 0.05 57.1% 3.6% - validate_element_comparison 1,907,339 59,265.75 0.03 36.9% 2.3% - validate_result_buffer_alloc 1,907,339 9,020.90 0.00 5.6% 0.4% - (untracked) 303.99 0.2% 0.0% - validate_inputs_copy 1,905,787 194.93 0.00 0.1% 0.0% - validate_report 1,907,339 43.63 0.00 0.0% 0.0% - validate_setup 1,905,883 43.37 0.00 0.0% 0.0% - validate_warmups_listener.BenchmarkTimer 2,106,827 6,922.29 0.00 2.1% 0.3% - validate_gpu_sync 2,106,827 6,867.77 0.00 99.2% 0.3% - (untracked) 54.52 0.8% 0.0% - (untracked) 560.19 0.2% 0.0% - validate_warmups_listener.MetaResultReporter 2,106,827 149.34 0.00 0.0% 0.0% - validate_warmups_listener.HardwareMonitorListener 2,106,827 74.67 0.00 0.0% 0.0% - validate_warmups_listener.ProgressListener 2,106,827 70.80 0.00 0.0% 0.0% - validate_warmups_listener.AllSolutionsIterator 2,106,827 66.34 0.00 0.0% 0.0% - validate_warmups_listener.DataInitialization 2,106,827 64.11 0.00 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Kernel Execution 303,840.52 12.3% 11.8% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - pre_solution 2,355,016 220,916.68 0.09 72.7% 8.6% - pre_solution_listener.ReferenceValidator 2,355,016 151,756.15 0.06 68.7% 5.9% - cpu_reference_gemm_per_solution_inputs_changed 1,146 151,668.42 132.35 99.9% 5.9% - solve_cpu_fast_per_solution_inputs_changed 1,127 138,501.63 122.89 91.3% 5.4% - solve_cpu_slow_per_solution_inputs_changed 19 13,166.07 692.95 8.7% 0.5% - (untracked) 0.72 0.0% 0.0% - pre_solution_ref_skipped 2,353,182 49.50 0.00 0.0% 0.0% - (untracked) 38.23 0.0% 0.0% - pre_solution_listener.DataInitialization 2,355,016 49,481.67 0.02 22.4% 1.9% - pre_solution_mx_init_data 16,038 44,073.15 2.75 89.1% 1.7% - pre_solution_mx_side_regenerate 2,298 44,069.34 19.18 100.0% 1.7% - pre_solution_mx_side_cache_hit 29,778 2.60 0.00 0.0% 0.0% - (untracked) 1.21 0.0% 0.0% - pre_solution_mx_copy_valid_to_gpu_no_regen 14,889 3,124.70 0.21 6.3% 0.1% - pre_solution_mx_copy_valid_to_gpu_after_regen 1,149 1,143.05 0.99 2.3% 0.0% - pre_solution_mx_cpu_sync_after_regen 1,149 438.52 0.38 0.9% 0.0% - pre_solution_mx_copy_inputs_d2d_no_regen 14,889 307.63 0.02 0.6% 0.0% - pre_solution_mx_cpu_sync_no_regen 14,889 307.33 0.02 0.6% 0.0% - pre_solution_mx_check 2,355,016 49.85 0.00 0.1% 0.0% - pre_solution_mx_copy_inputs_d2d_after_regen 1,149 22.97 0.02 0.0% 0.0% - (untracked) 14.47 0.0% 0.0% - pre_solution_listener.BenchmarkTimer 2,355,016 11,859.71 0.01 5.4% 0.5% - pre_solution_listener.ProgressListener 2,355,016 3,134.22 0.00 1.4% 0.1% - pre_solution_listener.AllSolutionsIterator 2,355,016 2,647.18 0.00 1.2% 0.1% - pre_solution_listener.MetaResultReporter 2,355,016 1,159.39 0.00 0.5% 0.0% - (untracked) 472.96 0.2% 0.0% - pre_solution_listener.SynchronizerValidator 2,355,016 358.33 0.00 0.2% 0.0% - pre_solution_listener.HardwareMonitorListener 2,355,016 47.08 0.00 0.0% 0.0% - post_solution 2,355,016 35,727.86 0.02 11.8% 1.4% - post_solution_log 2,355,016 28,760.07 0.01 80.5% 1.1% - post_solution_reporting 2,355,016 5,582.67 0.00 15.6% 0.2% - post_solution_validation 2,355,016 740.26 0.00 2.1% 0.0% - post_solution_perf_calc 2,355,016 652.16 0.00 1.8% 0.0% - post_solution_result_file 2,355,016 57.99 0.00 0.2% 0.0% - post_solution_perf_reset 2,355,016 52.77 0.00 0.1% 0.0% - post_solution_sol_advance 2,355,016 50.05 0.00 0.1% 0.0% - warmup_runs 4,213,654 25,917.98 0.01 8.5% 1.0% - run_current_solution_check 2,355,016 8,420.00 0.00 2.8% 0.3% - kernel_solving 2,106,835 6,730.10 0.00 2.2% 0.3% - post_problem 28,641 1,945.83 0.07 0.6% 0.1% - run_iteration_teardown 2,106,835 1,677.91 0.00 0.6% 0.1% - warmup_events_setup 2,106,835 970.97 0.00 0.3% 0.0% - solution_selection 2,355,016 571.95 0.00 0.2% 0.0% - finalize_report 855 507.49 0.59 0.2% 0.0% - benchmark_runs 2,106,835 345.20 0.00 0.1% 0.0% - benchmark_validate_enqueues 319 181.62 0.57 52.6% 0.0% - validate_enqueues_listener.BenchmarkTimer 319 179.53 0.56 98.8% 0.0% - benchmark_sleep 319 179.48 0.56 100.0% 0.0% - (untracked) 0.03 0.0% 0.0% - benchmark_event_elapsed_time 8 0.01 0.00 0.0% 0.0% - benchmark_event_destroy 8 0.01 0.00 0.0% 0.0% - validate_enqueues_listener.ProgressListener 319 1.81 0.01 1.0% 0.0% - (untracked) 0.21 0.1% 0.0% - validate_enqueues_listener.MetaResultReporter 319 0.02 0.00 0.0% 0.0% - validate_enqueues_listener.HardwareMonitorListener 319 0.01 0.00 0.0% 0.0% - validate_enqueues_listener.AllSolutionsIterator 319 0.01 0.00 0.0% 0.0% - validate_enqueues_listener.DataInitialization 319 0.01 0.00 0.0% 0.0% - validate_enqueues_listener.SynchronizerValidator 319 0.01 0.00 0.0% 0.0% - validate_enqueues_listener.ReferenceValidator 319 0.01 0.00 0.0% 0.0% - benchmark_pre_syncs 2,106,835 56.09 0.00 16.2% 0.0% - benchmark_post_syncs 2,106,835 54.72 0.00 15.9% 0.0% - gpu_kernel_execution 319 39.52 0.12 11.4% 0.0% - (untracked) 13.15 3.8% 0.0% - benchmark_events_setup 319 0.06 0.00 0.0% 0.0% - benchmark_events_teardown 319 0.03 0.00 0.0% 0.0% - solution_teardown 2,355,016 77.56 0.00 0.0% 0.0% - problem_report_index 28,641 26.02 0.00 0.0% 0.0% - problem_teardown 28,641 4.94 0.00 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - C++ Client Setup & Teardown 294,842.41 12.0% 11.5% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - hip_initialization 855 124,448.44 145.55 42.2% 4.8% - process_exit 855 64,634.38 75.60 21.9% 2.5% - main_problem_factory 855 29,154.81 34.10 9.9% 1.1% - factory_hip_get_device_properties 192 28,922.86 150.64 99.2% 1.1% - (untracked) 231.95 0.8% 0.0% - data_init_setup 855 18,185.96 21.27 6.2% 0.7% - main_teardown_locals 855 13,182.66 15.42 4.5% 0.5% - process_startup 855 12,710.53 14.87 4.3% 0.5% - code_object_loading 855 11,001.27 12.87 3.7% 0.4% - main_timing_calibration 855 10,096.95 11.81 3.4% 0.4% - library_loading 855 9,940.49 11.63 3.4% 0.4% - lazy_loading_init 855 924.59 1.08 0.3% 0.0% - main_parse_args 855 410.00 0.48 0.1% 0.0% - reporter_setup 855 68.52 0.08 0.0% 0.0% - python_client_write_run_script 855 57.11 0.07 0.0% 0.0% - listener_setup 855 15.77 0.02 0.0% 0.0% - solution_iterator_setup 855 10.94 0.01 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Timing Overhead 157,975.78 6.4% 6.2% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - flush_timing_buffer 865 152,892.88 176.75 96.8% 6.0% - timing_overhead 855 5,082.90 5.94 3.2% 0.2% - ----------------------------------------------------------------------------------------------------------------------------- - python_kernel_compilation 32 44,036.26 1,376.13 1.7% 1.7% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_build_src_co 32 20,333.49 635.42 46.2% 0.8% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_kernel_build_src_co.compile 4 20,243.96 5,060.99 99.6% 0.8% - python_kernel_build_src_co.cache_check 32 53.80 1.68 0.3% 0.0% - python_kernel_build_src_co.unbundle 4 27.97 6.99 0.1% 0.0% - (untracked) 3.45 0.0% 0.0% - python_kernel_build_src_co.cache_populate 4 3.20 0.80 0.0% 0.0% - python_kernel_build_src_co.setup 32 0.84 0.03 0.0% 0.0% - python_kernel_build_src_co.move 4 0.26 0.07 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_codegen 32 16,254.60 507.96 36.9% 0.6% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_write_assemble 32 2,214.46 69.20 5.0% 0.1% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_bench_setup 32 1,786.93 55.84 4.1% 0.1% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_validate 32 1,686.90 52.72 3.8% 0.1% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_write_helpers 32 545.76 17.05 1.2% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_build_co 32 526.58 16.46 1.2% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_bench_postprocess 32 495.87 15.50 1.1% 0.0% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_benchpost_lib_construction 32 355.66 11.11 71.7% 0.0% - python_benchpost_library_write 32 84.30 2.63 17.0% 0.0% - python_benchpost_naming 32 50.98 1.59 10.3% 0.0% - python_benchpost_client_config 32 3.91 0.12 0.8% 0.0% - (untracked) 1.03 0.2% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_kernel_setup 32 137.71 4.30 0.3% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 53.96 0.1% 0.0% - ----------------------------------------------------------------------------------------------------------------------------- - python_ductile_initial_sampling 8 13,753.39 1,719.17 0.5% 0.5% - ----------------------------------------------------------------------------------------------------------------------------- - python_ductile_generate_solutions 32 4,760.25 148.76 0.2% 0.2% - ----------------------------------------------------------------------------------------------------------------------------- - (untracked) 2,407.19 0.1% 0.1% - ----------------------------------------------------------------------------------------------------------------------------- - python_ductile_ga_between_evaluations 8 1,387.06 173.38 0.1% 0.1% - ----------------------------------------------------------------------------------------------------------------------------- - python_cache_check 831 1,280.29 1.54 0.1% 0.0% - ----------------------------------------------------------------------------------------------------------------------------- - python_write_solutions 855 205.93 0.24 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_dump 32 66.32 2.07 32.2% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_header 855 65.76 0.08 31.9% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_prepare 855 60.96 0.07 29.6% 0.0% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_wsol_prepare_cache 823 58.04 0.07 95.2% 0.0% - (untracked) 2.02 3.3% 0.0% - python_wsol_prepare_nocache 32 0.90 0.03 1.5% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 12.89 6.3% 0.0% - ----------------------------------------------------------------------------------------------------------------------------- - python_write_cache 32 58.71 1.83 0.0% 0.0% - ----------------------------------------------------------------------------------------------------------------------------- - python_solution_indexing 32 35.07 1.10 0.0% 0.0% - ----------------------------------------------------------------------------------------------------------------------------- - python_ductile_setup 8 32.30 4.04 0.0% 0.0% - ----------------------------------------------------------------------------------------------------------------------------- - python_ductile_clear_artifacts 32 26.20 0.82 0.0% 0.0% - ----------------------------------------------------------------------------------------------------------------------------- - python_ductile_read_scores 32 4.15 0.13 0.0% 0.0% - ----------------------------------------------------------------------------------------------------------------------------- - python_ductile_verification 8 0.30 0.04 0.0% 0.0% - =============================================================================================================================== - python_client_writer 10 23,848.23 2,384.82 0.9% - ----------------------------------------------------------------------------------------------------------------------------- - python_library_client_create_library 10 20,540.37 2,054.04 86.1% 0.8% - ----------------------------------------------------------------------------------------------------------------------------- - python_library_client_execution 10 3,161.37 316.14 13.3% 0.1% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_library_client_subprocess 10 3,155.15 315.52 99.8% 0.1% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 5.41 0.2% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_library_client_write_run_script 10 0.81 0.08 0.0% 0.0% - ----------------------------------------------------------------------------------------------------------------------------- - python_library_client_write_configs 10 120.68 12.07 0.5% 0.0% - ----------------------------------------------------------------------------------------------------------------------------- - python_library_client_setup 10 18.23 1.82 0.1% 0.0% - ----------------------------------------------------------------------------------------------------------------------------- - (untracked) 7.58 0.0% 0.0% - =============================================================================================================================== - python_library_logic 26 5,760.33 221.55 0.2% - ----------------------------------------------------------------------------------------------------------------------------- - python_logic_write 26 5,424.03 208.62 94.2% 0.2% - ----------------------------------------------------------------------------------------------------------------------------- - python_logic_parse_solutions 26 259.52 9.98 4.5% 0.0% - ----------------------------------------------------------------------------------------------------------------------------- - python_logic_analyze 26 69.15 2.66 1.2% 0.0% - ----------------------------------------------------------------------------------------------------------------------------- - (untracked) 7.63 0.1% 0.0% - - Overlapping Totals (wrap other timers; not summed into the tree) - ------------------------------------------------------------------------------------------------------------------------------- - python_client_subprocess 855 2,470,323.45 2,889.27 96.2% - main_total 855 2,226,863.00 2,604.52 86.7% - problem_total 28,641 2,022,050.86 70.60 78.7% - solution_total 2,355,016 943,803.95 0.40 36.7% - run_iteration_total 2,106,835 675,915.95 0.32 26.3% - validate_total 1,905,883 160,686.27 0.08 6.3% - validate_check_results_typed 1,907,339 160,445.08 0.08 6.2% - ---------------------------------------------------------------------------------------------------------------------------------- -HIERARCHY VALIDATION WARNINGS ---------------------------------------------------------------------------------------------------------------------------------- - WARNING: Child phases of python_client_execution (2,473,958.72 ms) exceed parent (2,470,665.16 ms) by 3,293.56 ms (0.1%) - -TIME BREAKDOWN (visual, % of wall clock) ---------------------------------------------------------------------------------------------------------------------------------- - TOTAL RUNTIME 100.0% ################################################## - RUN: benchmark client 90.8% ############################################# - Reference GEMM (CPU) 42.1% ##################### - cpu_reference_gemm 36.2% ################## - solve_cpu_slow 28.3% ############## - solve_cpu_fast 7.8% ### - (untracked) 0.0% - pre_solution_listener.ReferenceValidator 5.9% ## - cpu_reference_gemm_per_solution_inputs_changed 5.9% ## - solve_cpu_fast_per_solution_inputs_changed 5.4% ## - solve_cpu_slow_per_solution_inputs_changed 0.5% - (untracked) 0.0% - pre_solution_ref_skipped 0.0% - (untracked) 0.0% - Input data preparation 19.6% ######### - gpu_input_reset 11.9% ##### - gpu_input_reset_copy_valid_to_gpu 6.9% ### - gpu_input_reset_batched_inputs 3.7% # - gpu_input_reset_copy_inputs_d2d 1.0% - gpu_input_reset_copy_swizzled 0.2% - gpu_input_reset_fast_reset_output 0.1% - gpu_input_reset_convert_to_inputs 0.0% - gpu_input_reset_slow_path 0.0% - gpu_input_reset_rotating_init 0.0% - gpu_input_reset_constant_inputs 0.0% - gpu_input_reset_init_cpu_inputs 0.0% - (untracked) 0.0% - cpu_data_init 4.2% ## - pre_solution_listener.DataInitialization 1.9% - pre_solution_mx_init_data 1.7% - pre_solution_mx_side_regenerate 1.7% - pre_solution_mx_side_cache_hit 0.0% - (untracked) 0.0% - pre_solution_mx_copy_valid_to_gpu_no_regen 0.1% - pre_solution_mx_copy_valid_to_gpu_after_regen 0.0% - pre_solution_mx_cpu_sync_after_regen 0.0% - pre_solution_mx_copy_inputs_d2d_no_regen 0.0% - pre_solution_mx_cpu_sync_no_regen 0.0% - pre_solution_mx_check 0.0% - pre_solution_mx_copy_inputs_d2d_after_regen 0.0% - (untracked) 0.0% - gpu_input_preparation 1.5% - gpu_input_prep_copy_valid_to_gpu 0.9% - gpu_input_prep_copy_swizzled 0.4% - gpu_input_prep_init_cpu_inputs 0.2% - gpu_input_prep_batched_inputs 0.1% - gpu_input_prep_copy_inputs_d2d 0.0% - gpu_input_prep_fast_reset_output 0.0% - gpu_input_prep_convert_to_inputs 0.0% - gpu_input_prep_slow_path 0.0% - gpu_input_prep_constant_inputs 0.0% - gpu_input_prep_rotating_init 0.0% - (untracked) 0.0% - rotating_buffer_preparation 0.0% - Validation 12.6% ###### - validate_warmups_listener.SynchronizerValidator 6.3% ### - validate_synchronizer_check 6.3% ### - validate_synchronizer_scan 4.2% ## - validate_synchronizer_readback 2.2% # - validate_synchronizer_inactive 0.0% - (untracked) 0.0% - (untracked) 0.0% - validate_warmups_listener.ReferenceValidator 6.3% ### - validate_gpu_readback 3.6% # - validate_element_comparison 2.3% # - validate_result_buffer_alloc 0.4% - validate_inputs_copy 0.0% - validate_report 0.0% - validate_setup 0.0% - (untracked) 0.0% - validate_warmups_listener.MetaResultReporter 0.0% - validate_warmups_listener.HardwareMonitorListener 0.0% - validate_warmups_listener.ProgressListener 0.0% - validate_warmups_listener.AllSolutionsIterator 0.0% - validate_warmups_listener.DataInitialization 0.0% - Client launch (startup, setup & teardown) 11.1% ##### - hip_initialization 4.8% ## - process_exit 2.5% # - main_problem_factory 1.1% - factory_hip_get_device_properties 1.1% - (untracked) 0.0% - data_init_setup 0.7% - main_teardown_locals 0.5% - process_startup 0.5% - code_object_loading 0.4% - library_loading 0.4% - lazy_loading_init 0.0% - main_parse_args 0.0% - reporter_setup 0.0% - python_client_write_run_script 0.0% - listener_setup 0.0% - solution_iterator_setup 0.0% - Per-solution bookkeeping & reporting 2.7% # - post_solution_log 1.1% - pre_solution_listener.BenchmarkTimer 0.5% - run_current_solution_check 0.3% - post_solution_reporting 0.2% - pre_solution_listener.ProgressListener 0.1% - pre_solution_listener.AllSolutionsIterator 0.1% - post_problem 0.1% - run_iteration_teardown 0.1% - pre_solution_listener.MetaResultReporter 0.0% - post_solution_validation 0.0% - post_solution_perf_calc 0.0% - solution_selection 0.0% - finalize_report 0.0% - pre_solution_listener.SynchronizerValidator 0.0% - solution_teardown 0.0% - post_solution_result_file 0.0% - post_solution_perf_reset 0.0% - post_solution_sol_advance 0.0% - pre_solution_listener.HardwareMonitorListener 0.0% - problem_report_index 0.0% - problem_teardown 0.0% - Kernel launch & execution 1.6% - warmup_runs 1.0% - validate_warmups_listener.BenchmarkTimer 0.3% - validate_gpu_sync 0.3% - (untracked) 0.0% - kernel_solving 0.3% - warmup_events_setup 0.0% - benchmark_runs 0.0% - benchmark_validate_enqueues 0.0% - validate_enqueues_listener.BenchmarkTimer 0.0% - benchmark_sleep 0.0% - benchmark_event_elapsed_time 0.0% - benchmark_event_destroy 0.0% - (untracked) 0.0% - validate_enqueues_listener.ProgressListener 0.0% - validate_enqueues_listener.MetaResultReporter 0.0% - validate_enqueues_listener.HardwareMonitorListener 0.0% - validate_enqueues_listener.AllSolutionsIterator 0.0% - validate_enqueues_listener.DataInitialization 0.0% - validate_enqueues_listener.SynchronizerValidator 0.0% - validate_enqueues_listener.ReferenceValidator 0.0% - (untracked) 0.0% - benchmark_pre_syncs 0.0% - benchmark_post_syncs 0.0% - gpu_kernel_execution 0.0% - benchmark_events_setup 0.0% - benchmark_events_teardown 0.0% - (untracked) 0.0% - Library client (Python) 0.9% - python_library_client_create_library 0.8% - python_library_client_execution 0.1% - python_library_client_subprocess 0.1% - python_library_client_write_run_script 0.0% - (untracked) 0.0% - python_library_client_write_configs 0.0% - python_library_client_setup 0.0% - Library logic (Python) 0.2% - python_logic_write 0.2% - python_logic_parse_solutions 0.0% - python_logic_analyze 0.0% - MEASUREMENT OVERHEAD (instrumentation) 6.7% ### - flush_timing_buffer 6.0% ## - main_timing_calibration 0.4% - timing_overhead 0.2% - (timer overhead subtracted from parent timers) 0.2% - BUILD: solution generation & compile 2.5% # - Solution generation & build (Python) 1.7% - python_kernel_compilation 1.7% - python_kernel_build_src_co 0.8% - python_kernel_build_src_co.compile 0.8% - python_kernel_build_src_co.cache_check 0.0% - python_kernel_build_src_co.unbundle 0.0% - python_kernel_build_src_co.cache_populate 0.0% - python_kernel_build_src_co.setup 0.0% - python_kernel_build_src_co.move 0.0% - (untracked) 0.0% - python_kernel_codegen 0.6% - python_kernel_write_assemble 0.1% - python_kernel_bench_setup 0.1% - python_kernel_validate 0.1% - python_kernel_write_helpers 0.0% - python_kernel_build_co 0.0% - python_kernel_bench_postprocess 0.0% - python_benchpost_lib_construction 0.0% - python_benchpost_library_write 0.0% - python_benchpost_naming 0.0% - python_benchpost_client_config 0.0% - (untracked) 0.0% - python_kernel_setup 0.0% - (untracked) 0.0% - Ductile GA search (Python) 0.8% - python_ductile_initial_sampling 0.5% - python_ductile_generate_solutions 0.2% - python_ductile_ga_between_evaluations 0.1% - python_ductile_setup 0.0% - python_ductile_clear_artifacts 0.0% - python_ductile_read_scores 0.0% - python_ductile_verification 0.0% - SHARED: cache & solution files 0.1% - python_cache_check 0.0% - python_write_solutions 0.0% - python_wsol_dump 0.0% - python_wsol_header 0.0% - python_wsol_prepare 0.0% - python_wsol_prepare_cache 0.0% - python_wsol_prepare_nocache 0.0% - (untracked) 0.0% - (untracked) 0.0% - python_write_cache 0.0% - python_solution_indexing 0.0% - UNATTRIBUTED (untracked time & timer overhead) -0.0% - (untracked in python_benchmark_problems) 0.1% - (untracked in validate_warmups) 0.0% - (untracked in pre_solution) 0.0% - (untracked in pre_problem) 0.0% - (untracked in python_library_logic) 0.0% - (untracked in python_client_writer) 0.0% - (children exceed post_solution) -0.0% - (children exceed python_client_execution) -0.2% - -KEY INSIGHTS (% of wall clock) ---------------------------------------------------------------------------------------------------------------------------------- - Reference GEMM (CPU) 1080.47s (42.1%) - Input data preparation 502.88s (19.6%) - Validation 323.89s (12.6%) - Client launch (startup, setup & teardown) 284.75s (11.1%) - MEASUREMENT OVERHEAD (instrumentation) 168.07s (6.5%) - Per-solution bookkeeping & reporting 68.33s (2.7%) - Solution generation & build (Python) 44.04s (1.7%) - Kernel launch & execution 40.89s (1.6%) - Library client (Python) 23.84s (0.9%) - Ductile GA search (Python) 19.96s (0.8%) - Library logic (Python) 5.75s (0.2%) - SHARED: cache & solution files 1.58s (0.1%) - UNATTRIBUTED (untracked time & timer overhead) -0.93s (-0.0%) - - CPU reference / GPU execution ratio: 23498.2x - >>> CPU reference > 10x GPU execution time! - -TOP 10 SLOWEST PROBLEMS (by CPU reference time) ---------------------------------------------------------------------------------------------------------------------------------- - M N K Batch TypeA TypeD CPU Ref (ms) GPU (ms) - ------------------------------------------------------------------------------------------------------------------------------- - 4111 4111 4111 1 Float8 Float8 153530.08 0.00 - 4111 4111 4111 1 Float8 Float8 127609.73 0.00 - 4111 4111 4111 1 Int8 Int32 87409.00 0.00 - 4111 4111 4111 1 Int8 BFloat16 80626.39 0.00 - 4111 4111 4111 1 Int8 Int8 74748.05 0.00 - 4111 4111 4111 1 Int8 Half 59459.98 0.00 - 43776 10624 6400 1 Float8 BFloat16 17730.69 0.00 - 13056 7680 21504 1 Float8 BFloat16 15203.52 0.00 - 71936 4096 7168 1 Float8 BFloat16 13391.12 0.00 - 173312 3200 2560 1 Float8 BFloat16 11993.15 0.00 - diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/profiling/run_profile.sh b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/profiling/run_profile.sh deleted file mode 100644 index 05976bbb424e..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/profiling/run_profile.sh +++ /dev/null @@ -1,45 +0,0 @@ -#!/bin/bash -# Copyright Advanced Micro Devices, Inc., or its affiliates. -# SPDX-License-Identifier: MIT -# -# Profiled cold build + run for the slowest gfx950 common configs. -# Adapted from ../../run_cold_warm.sh (gfx942) with gfx950 as the target and without the warm-build stage. -# Caches (~/.tensile, ~/.tensilelite) are deleted before the build and before the run, so both are cold. -# Usage: bash run_profile.sh gemm/gfx950/subtile_mxfp4 gemm/gfx950/dtl gemm/bufferLoadStore0 -TL=/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite -CL=/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client -cd $TL || exit 1 -export OMP_NUM_THREADS=25 OPENBLAS_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=0 PYTHONUNBUFFERED=1 -TO="--cxx-compiler,/usr/bin/amdclang++,--global-parameters,TimingInstrumentation=True" -COMMON=(-s -v -n 0 --durations=0 --timeout=5400 -m "common and not skip-gfx94X" - "--global-parameters=LibraryFormat='msgpack'" "--tensile-options=$TO" -p no:warnings) -clear_caches() { rm -rf /home/alvasile/.tensile /home/alvasile/.tensilelite; } - -for cfg in "$@"; do - name=$(basename $cfg) - D=$TL/timing_runs/full_runs_gfx950/profiling/$name - A=/tmp/tl_timing_art_gfx950_$name - mkdir -p $D - rm -rf $A - BID="tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/$cfg.yaml]" - RID="tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/$cfg.yaml]" - : > $D/times.txt - - clear_caches - s=$(date +%s) - taskset -c 0-24 pytest "$BID" --build-only --gpu-targets gfx950 --artifact-dir $A \ - "${COMMON[@]}" > $D/${name}_1_build_cold.log 2>&1 - echo "build_cold rc=$? $(( $(date +%s)-s ))s" >> $D/times.txt - - clear_caches - s=$(date +%s) - taskset -c 0-24 pytest "$RID" --use-cache --artifact-dir $A --prebuilt-client=$CL \ - "${COMMON[@]}" > $D/${name}_2_run.log 2>&1 - echo "run rc=$? $(( $(date +%s)-s ))s" >> $D/times.txt - - for st in 1_build_cold 2_run; do - python scripts/analyze_timing.py $D/${name}_$st.log > $D/${name}_${st}_analysis.txt 2>&1 - done - echo DONE >> $D/times.txt -done -echo ALLDONE > $TL/timing_runs/full_runs_gfx950/profiling/done.txt diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/profiling/run_rerun.sh b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/profiling/run_rerun.sh deleted file mode 100644 index 4eae21d59725..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/profiling/run_rerun.sh +++ /dev/null @@ -1,30 +0,0 @@ -#!/bin/bash -# Copyright Advanced Micro Devices, Inc., or its affiliates. -# SPDX-License-Identifier: MIT -# -# Repeat of the run stage from run_profile.sh, to measure run-to-run variation. -# Reuses the build artifacts from run_profile.sh; caches are cleared before the run, as before. -# Usage: bash run_rerun.sh gemm/gfx950/subtile_mxfp4 gemm/gfx950/dtl gemm/bufferLoadStore0 -TL=/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite -CL=/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client -cd $TL || exit 1 -export OMP_NUM_THREADS=25 OPENBLAS_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=0 PYTHONUNBUFFERED=1 -TO="--cxx-compiler,/usr/bin/amdclang++,--global-parameters,TimingInstrumentation=True" -COMMON=(-s -v -n 0 --durations=0 --timeout=5400 -m "common and not skip-gfx94X" - "--global-parameters=LibraryFormat='msgpack'" "--tensile-options=$TO" -p no:warnings) -TAG=$1; shift - -for cfg in "$@"; do - name=$(basename $cfg) - D=$TL/timing_runs/full_runs_gfx950/profiling/$name - A=/tmp/tl_timing_art_gfx950_$name - RID="tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/$cfg.yaml]" - - rm -rf /home/alvasile/.tensile /home/alvasile/.tensilelite - s=$(date +%s) - taskset -c 0-24 pytest "$RID" --use-cache --artifact-dir $A --prebuilt-client=$CL \ - "${COMMON[@]}" > $D/${name}_3_run_$TAG.log 2>&1 - echo "run_$TAG rc=$? $(( $(date +%s)-s ))s" >> $D/times.txt - python scripts/analyze_timing.py $D/${name}_3_run_$TAG.log > $D/${name}_3_run_${TAG}_analysis.txt 2>&1 -done -echo ALLDONE > $TL/timing_runs/full_runs_gfx950/profiling/done_$TAG.txt diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/run.sh b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/run.sh deleted file mode 100755 index cc1b8a11aaf3..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/run.sh +++ /dev/null @@ -1,29 +0,0 @@ -#!/bin/bash -# Copyright Advanced Micro Devices, Inc., or its affiliates. -# SPDX-License-Identifier: MIT -# -# gfx950 counterpart of ../full_runs/run.sh (gfx942). Same CI-like conditions: -# 1 GPU, 25 cores, 4 xdist workers, 5400s per-test timeout, msgpack client. -# Single combined pass (what CI runs): build + run in one pytest invocation. -# Per-test timings come from pytest's --durations=0. -TL=/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite -OUT=$TL/timing_runs/full_runs_gfx950 -CLIENT=/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client -cd $TL || exit 1 -export OMP_NUM_THREADS=25 OPENBLAS_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=0 PYTHONUNBUFFERED=1 -COMMON=(tensilelite/Tests/common -v --durations=0 --timeout=5400 -n 4 -m "common and not skip-gfx94X" - "--global-parameters=LibraryFormat='msgpack'" "--tensile-options=--cxx-compiler,/usr/bin/amdclang++") - -rm -rf /home/alvasile/.tensile /home/alvasile/.tensilelite -{ - echo "START combined $(date -u +%FT%TZ)" - rocminfo | grep -m1 -o 'gfx950' - git rev-parse HEAD -} >> $OUT/times.txt -s=$(date +%s) -taskset -c 0-24 pytest "${COMMON[@]}" --prebuilt-client=$CLIENT --junit-xml=$OUT/combined.xml > $OUT/combined.log 2>&1 -rc=$? -echo "END combined rc=$rc seconds=$(( $(date +%s)-s )) $(date -u +%FT%TZ)" >> $OUT/times.txt -# Extract the per-test duration table -sed -n '/slowest durations/,/^=* .* in [0-9.]*s/p' $OUT/combined.log > $OUT/durations.txt -echo ALLDONE >> $OUT/times.txt diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/split/run_split.sh b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/split/run_split.sh index 95d12a5e8723..dda6095f0a5b 100755 --- a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/split/run_split.sh +++ b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/split/run_split.sh @@ -10,25 +10,36 @@ # GPU_TARGET, CORES, GPU (see ../measure_env.sh), # OUT (default $TL/timing_runs/full_runs_gfx950/split). ART is deleted and rebuilt, so it must # be set: the default ART is the measured baseline, which the script refuses to delete. +# DIST pytest-xdist scheduler (default worksteal, as the hw-common CI category; the +# committed split numbers were measured with load, xdist's default) +# CLEAR_RUN_CACHE 1 also clears the caches before the run phase (default 0: the run phase +# starts from the cache the build phase left; the committed split numbers were +# measured with 1). The build phase always starts from empty caches. source "$(dirname "${BASH_SOURCE[0]}")/../measure_env.sh" || exit 1 OUT=${OUT:-$TL/timing_runs/full_runs_gfx950/split} +DIST=${DIST:-worksteal} +CLEAR_RUN_CACHE=${CLEAR_RUN_CACHE:-0} +case $CLEAR_RUN_CACHE in + 0|1) ;; + *) echo "CLEAR_RUN_CACHE='$CLEAR_RUN_CACHE' must be 0 or 1" >&2; exit 1 ;; +esac check_rm_target TENSILE_CACHE TENSILELITE_CACHE ART cd "$TL" || exit 1 mkdir -p "$OUT" export OMP_NUM_THREADS=25 OPENBLAS_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=$GPU PYTHONUNBUFFERED=1 -COMMON=(tensilelite/Tests/common -v --durations=0 --timeout=5400 -n 4 -m "common and not skip-gfx94X" +COMMON=(tensilelite/Tests/common -v --durations=0 --timeout=5400 -n 4 --dist="$DIST" -m "common and not skip-gfx94X" "--global-parameters=LibraryFormat='msgpack'" "--tensile-options=--cxx-compiler,/usr/bin/amdclang++") -phase() { # name, extra args... - local name=$1; shift - clear_caches - echo "START $name $(date -u +%FT%TZ)" >> "$OUT/times.txt" +phase() { # name, clear caches first (0|1), extra args... + local name=$1 clear=$2; shift 2 + if [ "$clear" = 1 ]; then clear_caches; fi + echo "START $name dist=$DIST caches_cleared=$clear $(date -u +%FT%TZ)" >> "$OUT/times.txt" local s=$(date +%s) taskset -c "$CORES" pytest "${COMMON[@]}" "$@" --junit-xml="$OUT/$name.xml" > "$OUT/$name.log" 2>&1 local rc=$? echo "END $name rc=$rc seconds=$(( $(date +%s)-s )) $(date -u +%FT%TZ)" >> "$OUT/times.txt" } rm -rf -- "$ART"; mkdir -p "$ART" -phase build --build-only --gpu-targets "$GPU_TARGET" --artifact-dir "$ART" --prebuilt-client="$CLIENT" +phase build 1 --build-only --gpu-targets "$GPU_TARGET" --artifact-dir "$ART" --prebuilt-client="$CLIENT" du -sh "$ART" >> "$OUT/times.txt" -phase run --use-cache --artifact-dir "$ART" --prebuilt-client="$CLIENT" +phase run "$CLEAR_RUN_CACHE" --use-cache --artifact-dir "$ART" --prebuilt-client="$CLIENT" echo ALLDONE >> "$OUT/times.txt" diff --git a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/verify_mx/run_verify_mx.sh b/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/verify_mx/run_verify_mx.sh deleted file mode 100755 index 8d15881925fb..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/full_runs_gfx950/verify_mx/run_verify_mx.sh +++ /dev/null @@ -1,21 +0,0 @@ -#!/bin/bash -# Copyright Advanced Micro Devices, Inc., or its affiliates. -# SPDX-License-Identifier: MIT -# -# Runs the gfx950 MX common configs with TENSILELITE_VERIFY_REFERENCE_REUSE=1, i.e. every time the -# client skips a CPU-reference recompute because DataInitialization's input epoch did not move, -# it recomputes anyway and fails the run if the result is not bitwise identical. -# CI-like conditions as in ../full_runs/run.sh (combined build+run pass). -TL=/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite -OUT=$TL/timing_runs/full_runs_gfx950/verify_mx -CLIENT=/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client -cd $TL || exit 1 -export OMP_NUM_THREADS=25 OPENBLAS_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=0 PYTHONUNBUFFERED=1 -export TENSILELITE_VERIFY_REFERENCE_REUSE=1 -rm -rf /home/alvasile/.tensile /home/alvasile/.tensilelite -s=$(date +%s) -taskset -c 0-24 pytest tensilelite/Tests/common/test_config.py -v --durations=0 --timeout=5400 -n 4 \ - -m "common and not skip-gfx94X" -k "mx and gfx950" \ - "--global-parameters=LibraryFormat='msgpack'" "--tensile-options=--cxx-compiler,/usr/bin/amdclang++" \ - --prebuilt-client=$CLIENT --junit-xml=$OUT/verify_mx.xml > $OUT/verify_mx.log 2>&1 -echo "rc=$? seconds=$(( $(date +%s)-s ))" > $OUT/result.txt diff --git a/projects/hipblaslt/tensilelite/timing_runs/input_data_preparation_investigation.md b/projects/hipblaslt/tensilelite/timing_runs/input_data_preparation_investigation.md deleted file mode 100644 index f08aa704f047..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/input_data_preparation_investigation.md +++ /dev/null @@ -1,233 +0,0 @@ -# "Input data preparation" investigation (gfx950 per-config rerun 2) - -`full_runs_gfx950/per_config_rerun_2/AGGREGATE_analysis.txt` reports **Input data preparation** -at 502,878.60 ms (19.6% of wall). This note explains why it is slow and which of its timers are -inflated or deflated by work that belongs elsewhere. Timing claims come from measurements; code -reading was only used to decide what to measure. Inferences are marked as such. - -`cpu_data_init` (106.8 s) is covered separately in `cpu_data_init_investigation.md` and is not -repeated here. - -| Timer | Count | Total (ms) | Mean (ms) | -| --- | --- | --- | --- | -| `gpu_input_reset` | 2,106,835 | 306,247.35 | 0.15 | -| ├─ `gpu_input_reset_copy_valid_to_gpu` | 719,180 | 177,839.34 | 0.25 | -| ├─ `gpu_input_reset_batched_inputs` | 1,473,329 | 95,916.03 | 0.07 | -| ├─ `gpu_input_reset_copy_inputs_d2d` | 1,473,329 | 26,309.71 | 0.02 | -| ├─ `gpu_input_reset_copy_swizzled` | 15,719 | 4,006.86 | 0.25 | -| └─ `gpu_input_reset_fast_reset_output` | 633,454 | 1,917.65 | 0.00 | -| `cpu_data_init` | 28,596 | 106,829.68 | 3.74 | -| `pre_solution_listener.DataInitialization` | 2,355,016 | 49,481.67 | 0.02 | -| └─ `pre_solution_mx_side_regenerate` | 2,298 | 44,069.34 | 19.18 | -| `gpu_input_preparation` | 28,641 | 39,596.08 | 1.38 | -| ├─ `gpu_input_prep_copy_valid_to_gpu` | 11,446 | 22,935.66 | 2.00 | -| └─ `gpu_input_prep_copy_swizzled` | 1,151 | 10,766.76 | 9.35 | -| `rotating_buffer_preparation` | 28,641 | 723.82 | 0.03 | - -## Summary - -1. **Per-solution re-upload of unchanged data (`copy_valid_to_gpu`, 177.8 s).** On the slow - reset path, every solution re-uploads every input tensor from host memory. 99.45–99.92% of - those uploads are byte-identical to the previous upload into the same buffer. -2. **`batched_inputs` (95.9 s) is inflated.** Its own work is 4–7 tiny (8–24 byte) uploads of batch - pointer arrays per solution; 99.99% are byte-identical to the previous ones. Each costs - ~12–15 µs of latency, and because they are the first blocking call after the D2D copies they - also absorb those copies' GPU time and, on configs with `SyncsPerBenchmark: 0`, the previous - solution's still-running warmup kernels. -3. **`copy_inputs_d2d` (26.3 s) is deflated.** `hipMemcpy` D2D returns in ~2 µs regardless of - size; the copy itself is charged to whatever blocks next. Its real serialized cost is - 4.5–5.5× the timer. -4. **MX regeneration (`pre_solution_mx_side_regenerate`, 44.1 s) is redundant** and, through - `m_inputEpoch`, triggers a redundant CPU reference recompute - (`cpu_reference_gemm_per_solution_inputs_changed`, 151.7 s, booked under "Reference GEMM"). - -## Method - -- **Binary.** The measured client, `/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client` - (Release, `-O3`). Configs re-run with the cached split artifacts - (`/tmp/tl_split_gfx950/artifacts`) exactly as `full_runs_gfx950/profiling/run_rerun.sh` does - (`TimingInstrumentation=True`, `OMP_NUM_THREADS=25`, 25 pinned cores). -- **Environment.** The host was shared: during these runs cores 0–47 were ~99% busy with other - users' work, so runs used idle NUMA-1 cores (64–88, 89–113) with NUMA-1 GPUs 4 and 5, two runs - at a time. `perf` was unavailable (`perf_event_paranoid=4`). -- **`hipMemcpy` profiler.** An `LD_PRELOAD` shim (`memprof.cpp`) records per call site (return - address, mapped to functions with `nm`; the client has no debug info) and kind: count, bytes and - time inside `hipMemcpy`. Optional modes: - - *hash*: hash each H2D source and count uploads byte-identical to the previous upload into the - same device address; - - *presync*: `hipDeviceSynchronize()` before each `hipMemcpy`/`hipMemset`, charging the wait to - the **previous** GPU operation (memcpy call site, `hipExtModuleLaunchKernel`/`hipLaunchKernel`, - memset, async copy). This serializes everything, so it gives each operation's full cost and - shows where in-flight work is absorbed. An idle-GPU `hipDeviceSynchronize` costs 0.07 µs. -- **Shim overhead.** In passive mode the timers match production: `spmm_f16` - `copy_valid_to_gpu` 21,126 ms vs 20,468 ms, `batched_inputs` 5,993 vs 5,622 ms, - `copy_inputs_d2d` 1,366 vs 1,489 ms. -- **Microbenchmarks.** `copybench.cpp` (copy behavior) and `mxdeterm.cpp` (MX generator - determinism, linked against the client's `libhipblaslt-mxdatagen.a`). -- Scratch sources, scripts and per-run outputs are in `/tmp/syncval_bench` (`memprof.cpp`, - `memprof_sum.py`, `run_art.sh`, `copybench.cpp`, `mxdeterm.cpp`, `inprep/_/`). - -## Background: the two reset paths - -Before each solution, `main.cpp` calls `DataInitialization::prepareGPUInputs` -(`DataInitialization.hpp`), timed as `gpu_input_reset`: - -- **Fast path** (`fast_reset_output`, 633,454 calls, 1.9 s): GPU-generated data, bounds checking - off, no problem-dependent data, no swizzle. Only output tensors are reset. -- **Slow path** (1,473,329 calls): taken when bounds checking is on, data is problem-dependent - (`m_problemDependentData`: sparse, MX, Trig/Serial/Identity init modes, multiple bias types), or - tensors are swizzled. Per solution it runs: - -```text -copyValidToGPUBuffer if problem-dependent: H2D every tensor, host valid → GPU valid (copy_valid_to_gpu) -copySwizzledToGPUBuffer if swizzled (copy_swizzled) -copyInputs D2D every tensor, GPU valid → GPU current (copy_inputs_d2d) -initializeGPUBatchedInputs H2D a batch-pointer array per A/B/C/D (+ more) (batched_inputs) -``` - -The host data only changes per problem (`cpu_data_init`), or when MX data is regenerated. - -## Finding 1: per-solution re-upload of unchanged data (`copy_valid_to_gpu`, 177.8 s) - -The biggest contributors are `spmm_f16_sb` (53.9 s), `xfp32` (27.7 s), `spmm_f16` (20.5 s), -`plr_zero` (15.2 s) and `spmm_bf16_sb` (11.3 s). - -Hash mode, `copyInputBuffers` H2D (the `copy_valid_to_gpu` uploads): - -| Config | Uploads | Byte-identical to previous upload | MB/call (min–max) | Upload time | -| --- | --- | --- | --- | --- | -| `spmm_f16` | 567,420 | 566,162 (99.78%) | 1.26 (0.001–3.94) | 21.3 s | -| `spmm_f16_sb` | 1,508,910 | 1,507,646 (99.92%) | 1.25 (0.001–3.94) | 55.7 s | -| `xfp32` | 462,000 | 459,436 (99.45%) | 2.22 (2.10–16.8) | 30.2 s | -| `subtile_mxfp4` | 167,071 | 163,921 (98.1%) | 0.59 (0–16.8) | 7.1 s | - -- Nearly all of the bytes uploaded on every solution are already on the GPU. `spmm_f16` alone - re-uploads ~715 GB at ~34 GB/s. -- The copies are sized by `p.maxElements` (the largest allocation in the run), not the current - problem: in `xfp32`, every upload over 2,052 problems of varying size is at least 2.10 MB. The - `cpu_data_init` investigation measured the same `maxElements` sizing directly. -- A pinned source would help only for small tensors (`copybench`, pageable vs pinned: 1 MB 41.8 - vs 26.7 µs; 4 MB 84.6 vs 81.7 µs; 16 MB and up ~56 GB/s both). Not uploading is the real fix. -- `DataInitialization` already tracks host rewrites with `m_inputEpoch` (used by - `ReferenceValidator` to skip recomputing the CPU reference). Skipping the upload when the - epoch is unchanged would remove ≥99% of this bucket, ~176 s (inferred from the identical-upload - rates, not measured end-to-end). - -The per-problem `gpu_input_prep_copy_valid_to_gpu` (22.9 s) is needed once per problem, but is -subject to the same `maxElements` sizing. - -## Finding 2: `batched_inputs` is inflated (95.9 s) - -`initializeGPUBatchedInputs` builds a pointer array per tensor on the host and uploads it with a -pageable `hipMemcpy`. In every profiled config each upload is 8–24 bytes (8 bytes per batch). - -| Config | Uploads | Per solution | Byte-identical | Mean call | -| --- | --- | --- | --- | --- | -| `spmm_f16` | 449,568 | 7 | 449,511 (99.99%) | 13.3 µs | -| `spmm_f16_sb` | 1,190,378 | 7 | 1,190,321 (99.995%) | 12.4 µs | -| `xfp32` | 369,600 | 4 | 369,560 (99.99%) | 15.1 µs | -| `bufferLoadStore0` | 1,609,364 | 4 | not hashed | 14.9 µs | - -The arrays only change when a problem or buffer address changes, yet they are rebuilt and -re-uploaded for every solution. The timer then holds three things: - -1. **Latency of the tiny uploads.** An 8-byte pageable H2D costs 11.6 µs on an idle GPU - (`copybench`). In presync mode each also leaves ~7–8 µs of work in flight after returning. -2. **The preceding D2D copies.** They return immediately (Finding 3), so the first blocking - upload waits for them: in `copybench`, an 8-byte upload issued right after a 64 MB D2D takes - 28.5 µs instead of 11.6 µs. -3. **The previous solution's warmup kernels.** With `SyncsPerBenchmark: 0` (524 occurrences in - the test YAMLs, including `bufferLoadStore0` and `spmm_f16`), warmups 1..N-1 are launched and - nothing waits for them: `BenchmarkTimer::postWarmup` returns early unless - `skip-slow-solution-ratio` is set. The next solution's reset absorbs them. Measured in presync - mode on `bufferLoadStore0`: 2,345 ms of still-running-kernel time absorbed inside the reset, - over 200,635 of 401,984 resets (11.7 µs each). - -Passive-mode excess over the 11.6 µs floor: `bufferLoadStore0` 24.0 s measured vs 18.7 s floor -(5.3 s absorbed); `spmm_f16` 5.96 s vs 5.21 s (0.75 s absorbed). Caching the arrays per problem -(or re-uploading only when the epoch or a base address changes) would remove nearly all of the -bucket; the absorbed work would then surface in the next blocking call (inferred). - -## Finding 3: `copy_inputs_d2d` is deflated (26.3 s) - -`copybench`, GPU otherwise idle: - -| D2D size | `hipMemcpy` returns after | Copy complete after | -| --- | --- | --- | -| 64 KB | 2.2 µs | 14.5 µs | -| 4 MB | 2.1 µs | 21.4 µs | -| 64 MB | 2.2 µs | 38.2 µs | -| 256 MB | 2.3 µs | 113.5 µs | - -In the client the D2D calls average 2–3 µs. Presync mode gives their real serialized cost (time -in the call plus the wait charged to them): - -| Config | `copy_inputs_d2d` timer (passive) | D2D call time + wait charged to D2D (presync) | -| --- | --- | --- | -| `spmm_f16` | 1,366 ms | 1,443 + 6,015 = 7,458 ms | -| `bufferLoadStore0` (NaN bounds check: `copyNaNInputBuffers` + `copyInputBuffers`) | 6,144 ms | 2,646 + 10,932 + 2,887 + 11,271 = 27,736 ms | - -The remainder lands in `batched_inputs`, in the warmup kernel's start, and in the validators' -first blocking readback. `copyInputs` copies every tensor, including read-only inputs (A, B) that -kernels do not modify; only outputs (and the Synchronizer, see `SynchronizerValidator.md`) need -restoring between solutions. The split between necessary and avoidable D2D bytes was not -measured. - -## Finding 4: redundant MX regeneration and the CPU reference recompute it triggers - -On gfx950, `initializeMXData` caches MX data per side, keyed by the tensor descriptor plus a tag -saying `gpuInput.valid` holds a scale swizzled for the current solution's MI shape. The tag is -erased whenever a side is initialized without a swizzle, which includes the problem-level init. -So the first solution of each problem that needs the GFX950 swizzle misses the cache and -**regenerates** the side: one `generateMXInput` call for data and canonical scale into -`cpuInput.valid`, and a second one (with the swizzle layout) that rewrites the data buffer again -to produce the swizzled scale. It also increments `m_inputEpoch`. - -- **The regenerated data is identical.** `generateMXInput` uses a fixed seed (`1713573849`, - `mxDataGen.cpp`). Measured with `mxdeterm.cpp` (4096×4096, FP4 E2M1 and FP8 E4M3): two calls - with identical arguments give byte-identical data and scales, and the data written by a - `None`-layout call is byte-identical to that written by a `GFX950`-layout call. Each call costs - 77–160 ms at this size. The descriptor is unchanged at the point of regeneration, so the - arguments are identical. -- **It happens once per problem.** In `subtile_mxfp4`: 1,370 regenerated sides = 685 problems × 2 - sides, and 685 `pre_solution_mx_copy_valid_to_gpu_after_regen` uploads. The hash run finds - 3,150 non-identical uploads in total, ~4.6 per problem. If the regenerated data differed from - the problem-level data there would be at least 4 more per problem (A, B and both scales), so - the uploads after regeneration are byte-identical too (inferred from the counts). -- **It triggers a second CPU reference.** `ReferenceValidator` recomputes the CPU reference when - `inputEpoch(solution)` changes. Across the aggregate, in 35 of the 36 configs involved, the - number of `cpu_reference_gemm_per_solution_inputs_changed` recomputes equals the number of - after-regeneration uploads. In `subtile_mxfp4`, each of the 685 problems computes the reference - twice (685 problem-level + 685 "inputs changed", 5,916 ms). Aggregate: 1,146 recomputes, - 151.7 s, booked under "Reference GEMM (CPU)". - -| Config | Regenerate (ms) | Recompute (ms) | Count | -| --- | --- | --- | --- | -| `subtile_mxfp8` | 14,806 | 70,729 | 39 | -| `subtile_mxfp8_correctness_regression_mt256x320_plr0_wgm1` | 7,104 | 27,987 | 5 | -| `subtile_mxfp8_correctness_regression_mt320x256_plr1_wgm4` | 6,883 | 23,140 | 5 | -| `mxfp4_mxfp4_bf16_tn_act` | 1,241 | 13,744 | 78 | -| `subtile_mxfp4` | 5,908 | 4,126 | 685 | - -Producing only the swizzled scale from the canonical one, without touching the data or the -epoch, would remove the 44.1 s here and the 151.7 s recompute (inferred). - -## Inflated and deflated timers - -| Timer | Effect | Why | -| --- | --- | --- | -| `gpu_input_reset_batched_inputs` | **inflated** | absorbs the preceding D2D copies and, with `SyncsPerBenchmark: 0`, the previous solution's warmup kernels | -| `gpu_input_reset_copy_inputs_d2d` | **deflated** | `hipMemcpy` D2D returns in ~2 µs; real cost 4.5–5.5× the timer, charged downstream | -| `gpu_input_reset` (whole) | **inflated** on `SyncsPerBenchmark: 0` configs | previous solution's kernel time (measured 2.3 s on `bufferLoadStore0`) | -| `gpu_input_reset_copy_valid_to_gpu` | accurate, but ≥99% redundant | re-uploads unchanged, allocation-sized data every solution | -| `pre_solution_mx_side_regenerate` | accurate, but redundant | regenerates byte-identical data; inflates "Reference GEMM" by 151.7 s via the epoch | - -## Not measured / open items - -- End-to-end effect of any fix (only per-call costs and redundancy were measured). -- How much of `copy_inputs_d2d` is necessary (outputs) vs avoidable (read-only inputs). -- Where the D2D and kernel time absorbed by `batched_inputs` would land once that bucket is - removed; an explicit, separately timed sync after the warmups would make it visible. -- `gpu_input_prep_copy_swizzled` / `gpu_input_reset_copy_swizzled` (14.8 s combined) and the - per-problem paths were not profiled beyond the counts above. -- Run-to-run variance on the shared host. diff --git a/projects/hipblaslt/tensilelite/timing_runs/per_solution_bookkeeping_investigation.md b/projects/hipblaslt/tensilelite/timing_runs/per_solution_bookkeeping_investigation.md deleted file mode 100644 index 66a560d7ab5e..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/per_solution_bookkeeping_investigation.md +++ /dev/null @@ -1,721 +0,0 @@ -# "Per-solution bookkeeping & reporting" investigation (gfx950 per-config rerun 2) - -`full_runs_gfx950/per_config_rerun_2/AGGREGATE_analysis.txt` reports **Per-solution bookkeeping & -reporting** at 68,333.60 ms (2.7% of wall). This note explains where that time goes and what can -be done about it. Timing claims come from measurements; code reading was only used to decide what -to measure. Inferences are marked as such. - -The section [Correctness-only runs](#correctness-only-runs-what-can-be-turned-off) covers which of -this work, and of the per-launch timing work, can be switched off when Tensile is run only to check -correctness (the common tests), rather than to benchmark. - -| Timer | Count | Total (ms) | Mean (µs) | -| --- | --- | --- | --- | -| `post_solution_log` | 2,355,016 | 28,760.07 | 12.2 | -| `pre_solution_listener.BenchmarkTimer` | 2,355,016 | 11,859.71 | 5.0 | -| `run_current_solution_check` | 2,355,016 | 8,420.00 | 3.6 | -| `post_solution_reporting` | 2,355,016 | 5,582.67 | 2.4 | -| `pre_solution_listener.ProgressListener` | 2,355,016 | 3,134.22 | 1.3 | -| `pre_solution_listener.AllSolutionsIterator` | 2,355,016 | 2,647.18 | 1.1 | -| `post_problem` | 28,641 | 1,945.83 | 67.9 | -| `run_iteration_teardown` | 2,106,835 | 1,677.91 | 0.8 | -| `pre_solution_listener.MetaResultReporter` | 2,355,016 | 1,159.39 | 0.5 | -| 12 smaller timers | | 3,146.62 | | - -## Summary - -There is no single slow function. The section is a fixed host-side cost of about 29 µs per -solution, paid 2,355,016 times. Weighted by solution count, the per-config cost is 24.1 µs (p10), -25.6 µs (p50) and 36.6 µs (p90). The 13 configs with at least 50k solutions hold 85.8% of the -solutions and 80.7% of the time. Four things make up that per-solution cost (shares measured on -dtl unless stated): - -1. **Preemption by spinning OpenMP workers, about 15% of the section.** In the production logs, - 9.9 s of these timers is in individual calls longer than 1 ms, including timers that only do - CPU work. After each OpenMP parallel region (e.g. the CPU reference), the client's 24 idle - libomp workers keep spinning (`KMP_BLOCKTIME` defaults to 200 ms) on the same 25 cores as the - main thread: 11–24 workers were measured busy during the solution loop. In the dtl replay, - `KMP_BLOCKTIME=0` cut the >1 ms share from 1.8–4.1 s to 0.08–0.16 s. - It also cut `cpu_reference_gemm` from 5.8–14.8 s to 2.4–2.7 s, which is outside this section. -2. **The `report()` fan-out, about 38%.** Each solution makes about 19 `report()` calls. Each is - forwarded to 3 reporters, which format numbers with a fresh `std::ostringstream` (about 150 ns - each), up to three times per value. -3. **One stdout CSV row per solution, about 28%.** `LogReporter` copies the 30-column row into a - new `unordered_map`, formats a row of about 1.1 KB (mostly the solution name), and flushes - stdout with `std::endl`. Over the aggregate, that is 2,335,808 rows and 2.64 GB of text that - nothing parses: Tensile reads only the `results-file` CSV. -4. **Predicate debug dumps for rejected solutions, 16% in dtl and about 6% of the aggregate - (estimate).** The default `ClientLogLevel` is 3 (Debug). For every solution that fails its - predicates (443,391 in the aggregate, 18.8%), `checkSolution` calls `debugEval` on the whole - predicate tree, which costs 8.9 µs per rejected solution. - -A prototype that fixes 2 and 3 produced output identical to the shipped client and cut the dtl -bookkeeping section by 33%. Adding 4 (Terse log level) brought the cut to 58%, and fixing 1 removes -the tail on top of that (see [Options](#options)). - -## Method - -- **Binary.** The measured client, `/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client` - (Release, `-O3 -DNDEBUG -std=c++20 -x hip --offload-arch=gfx950 -fopenmp=libomp`). -- **Rebuilds.** Copies of `tensilelite/client` were rebuilt outside the repo with the exact - compile and link lines from that build's `compile_commands.json` and `link.txt`, linked against - the same `libtensilelite-host.so`. There are three copies: - - `ref`: unmodified; its timings match the shipped binary within run-to-run noise. - - `instr`: `ref` plus finer `ScopedTimer`s (`bk_*`) inside the timers above, and per-reporter - timers in `MetaResultReporter`. - - `opt`: the prototype described under [Options](#options). -- **Workload.** `common__gemm__gfx950__dtl` (443,808 solutions; the largest config in this section, - at 10.9 s). It was first run through pytest exactly as `full_runs_gfx950/profiling/run_rerun.sh` - does (cached artifact, `TimingInstrumentation=True`, `--prebuilt-client`). That run gave 9.8 s - for the section, against 10.9 s in production. Its 26 client invocations were then replayed - directly, in the same order with the same `ClientParameters.ini`. The replay matches the pytest - run (e.g. `post_solution_log` 2,780 ms vs 2,700 ms). -- **Variants.** Each variant was run 2–3 times. They use the same binary and inputs, and change only - `KMP_BLOCKTIME` (environment) or `log-level` (ini edit). -- **Microbenchmarks.** The formatting primitives were compiled with the same compiler and `-O3`. - Timings are medians of 9 repetitions. -- **Production logs.** All 184 `per_config_rerun_2` logs were scanned for per-call distributions - and for stdout content (CSV rows, validation outcomes, predicate dumps). -- **Environment.** gfx950 host, 25 idle cores via `taskset`, `OMP_NUM_THREADS=25`. The host is - shared: every GPU was 69–97% busy with other users' work. All conclusions are drawn from - host-only timers. Whole-client wall times moved by ±20% between repeats, because of GPU-bound - timers such as `gpu_input_reset`, and are not used. -- Scratch scripts, summaries and logs are in `/tmp/bookkeep`: `build.sh`, `direct.sh`, `summ.sh`, - `cmp.sh`, `tail.sh`, `rej.sh`, `client_src/` (instr), `client_opt/` (opt) and `micro/fmt.cpp`. - -## Where the time goes (one solution, dtl, instrumented build) - -Means per call in µs. Each nested timer adds about 42 ns (calibrated `timing_overhead`), so the -instrumented parents read higher than in the `ref` build; both are shown. - -| Timer / sub-step | `ref` mean | `instr` mean | Notes | -| --- | --- | --- | --- | -| `post_solution_log` | 6.62 | 6.12 | | -| ├─ `readCurrentRow` (copy 30 columns into a new map) | | 2.40 | done for every row, printed or not | -| ├─ `writeCurrentRow`: format the row | | 1.67 | `escape()` copies each value again | -| ├─ `writeCurrentRow`: `std::endl` flush | | 1.14 | one `write()` of about 1.1 KB per solution | -| └─ `pop` (restore the row from the stack) | | 0.39 | | -| `pre_solution_listener.BenchmarkTimer` | 4.91 | 6.62 | | -| ├─ `projectedPerformance` | | 0.27 | | -| └─ 9 `report()` calls | | 6.28 | about 0.5 µs per report after removing timer overhead | -| `run_current_solution_check` (average) | 5.05 | 5.18 | | -| ├─ predicate evaluation (all solutions) | | 0.60 | | -| ├─ hardware predicate (all solutions) | | 0.25 | | -| └─ reject path (173,006 rejected solutions) | | 10.44 | per rejected solution | -| ├─ `debugEval` of problem and task predicates | | 8.86 | | -| └─ `log()` of the dump | | 0.57 | | -| `post_solution_reporting` (4 double `report()` calls) | 1.73 | 2.50 | | -| `pre_solution_listener.ProgressListener` | 1.20 | 1.62 | `solution->name()` 0.21; 2 reports | -| `pre_solution_listener.AllSolutionsIterator` | 1.12 | 1.64 | 3 reports, one built with `concatenate` | -| `pre_solution_listener.MetaResultReporter` | 0.48 | 0.36 | `CSVStackFile::push` copies the row map | - -### The `report()` fan-out - -Per solution, the listeners make 19.2 `report()` calls: 10.0 double, 4.0 int, 3.1 string and 2.1 -uint. `MetaResultReporter` forwards each call to 3 reporters; `LibraryUpdateReporter` is null -here. Including timer overhead, the per-call cost by reporter is: - -| Reporter | Mean per call (µs) | What it does with a double | -| --- | --- | --- | -| `ResultFileReporter` | 0.273 | `ostringstream << value` to get a string, compares the key against about 8 keys, then `setValueForKey(key, value)`, which formats the double again with a second `ostringstream` | -| `LogReporter` | 0.168 | `setValueForKey(key, double)`: `ostringstream << setprecision(6)` | -| `PerformanceReporter` | 0.023 | key comparisons only | - -Microbenchmark of the primitives, using the client's compiler and `-O3`: - -| Operation | ns | -| --- | --- | -| `ostringstream` + `setprecision(6)` + double → string | 153.6 | -| `ostringstream` + double → string | 148.7 | -| `ostringstream` + int64 → string | 50.5 | -| `std::to_chars(double, general, 6)` → string (same text) | 36.9 | -| `std::to_chars(int64)` → string | 4.7 | -| copy 30–33 columns into a new `unordered_map` (the `readCurrentRow` pattern) | 1,016 | - -The fan-out comes to 3.7 s of the dtl replay's 9.8 s (about 8.4 µs per solution) after removing -the instrumentation's own overhead. That overhead was 34M nested timers at 42 ns, or 1.4 s. - -### The stdout CSV row - -In the dtl baseline log, the client wrote 1.19 M non-TIMING lines (520 MB), counted by line type: - -| Line type | Lines | Bytes | -| --- | --- | --- | -| CSV row per solution (30 columns; about 1,000 of 1,125 bytes is the solution name) | 443,808 | 499.5 MB | -| predicate debug dump lines (`[!!] ...`) | 382,108 | 20.1 MB | -| blank, `####` and `++++` separators | 365,548 | 0.9 MB | - -Across all 184 production logs: 2,335,808 CSV rows (2.64 GB) and 923,357 dump lines (48.8 MB). -Tensile does not read any of it: `ClientWriter.py` starts the client with `subprocess.Popen` and no -`stdout=` argument, and `BenchmarkProblems.py` reads results only from `results-file`. In -`run_rerun.sh`, the client's stdout and stderr were a regular file (`/proc//fd/1` → the log), -not a pipe. - -### Predicate debug dumps for rejected solutions - -`SolutionIterator::checkSolution` evaluates the problem and task predicates. If either fails and -`logAtLevel(Verbose)` is true, it calls `debugEval` on both trees. `debugEval` re-evaluates every -term (the `And` nodes have 35–39 terms) and formats a row per term into an `ostringstream`; the -result is then logged. `ClientLogLevel` defaults to 3 (Debug) in `Common/GlobalParameters.py`, and -no common test YAML overrides it, so every rejected solution pays for this. - -| | dtl | aggregate (184 logs) | -| --- | --- | --- | -| solutions rejected by predicates (`DID_NOT_SATISFY_ASSERTS`) | 173,006 (39.0%) | 443,391 (18.8%) | -| `debugEval` + log per rejected solution | 9.4 µs | | -| `run_current_solution_check`, Debug → Terse (dtl, `KMP_BLOCKTIME=0`) | 2,109 → 473 ms | | - -The Python enum has no `Normal`, so Tensile can only select Error, Terse, Verbose or Debug. At -Terse, rows of PASSED solutions (row level Normal) are not printed. Rows of rejected solutions -(row level Terse) are still printed, but without the dump. - -## The >1 ms tail: spinning OpenMP workers - -Share of each timer spent in calls longer than 1 ms, across all 184 production logs: - -| Timer | Total (ms) | in calls > 1 ms | -| --- | --- | --- | -| `post_solution_log` | 28,760.1 | 4,364.0 (15.2%) | -| `pre_solution_listener.BenchmarkTimer` | 11,859.7 | 1,022.7 (8.6%) | -| `run_current_solution_check` | 8,420.0 | 1,110.1 (13.2%) | -| `post_solution_reporting` | 5,582.7 | 1,175.2 (21.1%) | -| `pre_solution_listener.ProgressListener` | 3,134.2 | 321.2 (10.2%) | -| `pre_solution_listener.AllSolutionsIterator` | 2,647.2 | 312.6 (11.8%) | -| `post_problem` | 1,945.8 | 970.9 (49.9%) | -| `run_iteration_teardown` | 1,677.9 | 305.7 (18.2%) | -| `pre_solution_listener.MetaResultReporter` | 1,159.4 | 353.7 (30.5%) | -| **all nine** | 65,187.0 | 9,936.0 (15.2%) | - -Medians are far below the means. For example, `post_solution_log` in `spmm_i8hs` has a median of -14.9 µs, a p99 of 5.0 ms and a maximum of 42.7 ms. Pure-CPU timers such as -`pre_solution_listener.ProgressListener` show the same spikes, so the cause is not I/O. - -The client links libomp, and its process has 27 threads: main, 24 OpenMP workers and 2 runtime -threads. All of them share the 25 cores of the `taskset` mask. Per-thread CPU time was sampled -from `/proc//task/*/stat` during the solution loop: - -| `KMP_BLOCKTIME` | Windows sampled | Workers above 25% CPU | Whole process, cores busy | -| --- | --- | --- | --- | -| default (200 ms) | 6 | 11–24 | 12–20 | -| 0 | 3 | 0 | 1.2 (main 1.07) | - -Effect on the dtl replay, in ms (each row is one run): - -| Variant | Section total | ≤ 1 ms calls | > 1 ms calls | `cpu_reference_gemm` | -| --- | --- | --- | --- | --- | -| default | 10,246 | 8,466 | 1,780 | 5,819 | -| default | 14,419 | 10,770 | 3,649 | 14,770 | -| default | 14,523 | 10,407 | 4,116 | 14,142 | -| `KMP_BLOCKTIME=0` | 9,362 | 9,240 | 122 | 2,420 | -| `KMP_BLOCKTIME=0` | 9,813 | 9,730 | 83 | 2,737 | -| `KMP_BLOCKTIME=0` | 10,585 | 10,423 | 162 | 2,670 | - -With `KMP_BLOCKTIME=0`, the tail disappears and run-to-run variance drops. The ≤1 ms part does not -shrink; it is the real per-solution work covered by the other findings. - -Inference, not measured: with spinning workers, 26–27 threads compete for 25 cores, and the main -thread is regularly descheduled for a full timeslice. That would explain both the millisecond -spikes and their spread across every timer on the main thread. The same mechanism probably inflates -other main-thread sections. The `cpu_reference_gemm` drop (2–5×) is large and belongs to the -Reference GEMM section, so it is listed under open items. - -## Options - -All measurements below come from the dtl replay with `KMP_BLOCKTIME=0`, so that the tail does not -hide the differences. Totals are for the whole section, in ms. - -| Variant | Runs | Section total | `post_solution_log` | `BenchmarkTimer` pre | `run_current_solution_check` | `post_solution_reporting` | Progress + AllSolutions | -| --- | --- | --- | --- | --- | --- | --- | --- | -| `ref` | 3 | 9,362 / 9,813 / 10,585 | 2,890 | 2,136 | 2,109 | 757 | 1,010 | -| `opt` (B) | 3 | 6,507 / 6,588 / 6,708 | 1,367 | 838 | 2,225 | 335 | 793 | -| `ref` + Terse (C) | 2 | 7,285 / 7,724 | 2,090 | 2,170 | 473 | 755 | 1,006 | -| `opt` + Terse (B + C) | 3 | 4,099 / 4,151 / 4,115 | 738 | 822 | 421 | 327 | 788 | - -Per-timer columns are means over the runs. The section drops from 9.9 s to 6.6 s with B (−33%), -and to 4.1 s with B + C (−58%), or 9.3 µs per solution. - -### A. Stop idle OpenMP workers from spinning - -Set `KMP_BLOCKTIME=0` for the client, either in the environment `ClientWriter.py` passes to the -client, or with `kmp_set_blocktime(0)` or a smaller worker pool in `main.cpp`. `OMP_WAIT_POLICY=passive` -should have a similar effect, but was not measured. Measured effect: the >1 ms tail falls from -1.8–4.1 s to 0.08–0.16 s in dtl. In the aggregate, that tail is 9.9 s of this section. It costs -nothing in the measured runs: `cpu_reference_gemm`, the main OpenMP user, got faster, not slower. - -### B. Cheaper formatting and row handling (prototype `opt`, output-identical) - -The changes, all in the client: - -1. `CSVStackFile::setValueForKey(double)` and the integral `setValueForKey` use `std::to_chars` - instead of `std::ostringstream`. For doubles, `std::chars_format::general` with precision 6 - produces the same text as `<< setprecision(6)`. -2. `ResultFileReporter::reportValue` builds its value string with `std::to_chars` (once) instead - of an `ostringstream`. -3. `LogReporter::postSolution` looks up the 3–7 keys it uses (`findCurrentValue`) instead of - copying all 30 columns with `readCurrentRow`. The header is still written on the first row. -4. `CSVStackFile::writeRow` ends rows with `'\n'` instead of `std::endl`. `LogReporter::postProblem` - flushes once per problem. - -Correctness check: `ref` and `opt` were run on three dtl invocations (3,508, 320 and 60,940 stdout -lines). The results CSVs were byte-identical, and stdout differed only in the time-based -`srand seed` line. dtl runs no timed iterations, so its rows are deterministic. - -Measured: −33% on the section (9.9 → 6.6 s). `post_solution_log` fell by 53%, `BenchmarkTimer` -pre by 61% and `post_solution_reporting` by 56%. The remaining Progress/AllSolutions cost is mostly -string reports and `concatenate`, which the prototype does not touch. - -Trade-off of item 4: if the client crashes, up to one problem's worth of stdout rows can be lost. -The client already prints a `##########################` / `++++` pair per problem, so the flush -point matches an existing boundary. - -Robustness note: in two consecutive runs, about 1.5 minutes in total, 37 of 52 `opt` invocations -crashed with SIGSEGV. The crashed invocations never printed `loading config file`, which is printed -with `std::endl` during argument parsing, before any reporter code runs. That points either to -crashes before the prototype's code executed, or to log writes failing. The root filesystem was at -100% use (27 GB free) before that batch and at 96% after it. The crash did not reproduce in 4 -isolated reruns of the same invocation, or in the other 208 `opt` invocations (26 before, 182 -after). It was not seen in any `ref` run. Root cause unknown. - -### C. Don't dump predicate trees by default - -Either lower the default `ClientLogLevel`, or gate the `debugEval` call in `checkSolution` behind -its own option and keep the per-solution rows. Measured with Terse: `run_current_solution_check` -fell from 2,109 to 473 ms in dtl (−9.4 µs per rejected solution). Scaled to the aggregate's -443,391 rejected solutions, that is about 4.2 s (estimate). The dump remains useful when debugging -why a kernel is rejected, so it should stay available behind an option. Terse also removes the -stdout rows of passing solutions, which costs log readability in CI when a test fails. - -### D. Larger changes (not measured) - -- Skip the stdout row entirely unless asked for. It is 2.64 GB of text in the aggregate that no - tool reads; `readCurrentRow`/`writeCurrentRow` account for about 6 µs per solution, before B. -- Report the 9 granularity and memory values in `BenchmarkTimer::preSolution` only when a consumer - needs them (CSV extra columns or the log). They are 9 of the 19 `report()` calls. -- Replace `MetaResultReporter`'s per-call virtual fan-out over `std::string` keys with typed keys. - Each call also copies a `shared_ptr` per reporter (`for(auto r : m_reporters)`). - -### Estimated effect on the aggregate - -These figures are estimates: they scale the dtl measurements to the aggregate's counts. Other -configs have a different mix, for example 18.8% rejected solutions overall against 39% in dtl. - -| Option | Basis | Estimated saving | -| --- | --- | --- | -| A | measured >1 ms share in the production logs | about 9.9 s | -| B | 7.5 µs per solution (dtl) × 2,355,016 | about 17.6 s | -| C | 9.4 µs per rejected solution × 443,391 | about 4.2 s | -| A + B + C | | about 32 s of 68.3 s | - -## Correctness-only runs: what can be turned off - -`Tensile input.yaml out` serves two purposes: benchmarking, and the correctness common tests. A -correctness run needs only three things: - -- whether each solution produced a correct result; -- where it differed; -- the client's exit code (`main` returns the number of failed validations, clamped to 255; the test - fails on a non-zero code); - -Everything else the client does per solution is reporting or timing. That includes the -`results-file` CSV: it has to exist, but its values carry no information in a correctness run (see -[The results CSV in correctness runs](#the-results-csv-in-correctness-runs)). - -### The results CSV in correctness runs - -The client writes one row per problem size. Each row holds the sizes and one GFlops cell per -solution: `-nan` for a passed solution when nothing is timed, and `-1` for a failed or rejected -one. Python then uses it like this: - -1. `BenchmarkProblems` copies it to `2_BenchmarkData/` (`shutil.copy`, which fails if the file is - missing). Pass/fail does not come from the CSV: it is the client's exit code - (`benchmarkTestFails`), followed by `sys.exit(1)` under `ExitOnFails`. No common-test harness - file reads the CSV, `2_BenchmarkData` or `3_LibraryLogic`. -2. LibraryLogic reads it (`addFromCSV`) only for YAMLs with a `LibraryLogic:` section. In the - production logs, 26 of the 184 gfx950 configs did: - - **10 `gemm/gfx950/*mxfp8*` correctness configs, `SyncsPerBenchmark: 0`, `LibraryType: - Prediction`, followed by LibraryClient.** Prediction skips winner, range and exact logic, and - `removeInvalidSolutions()` drops only solutions with GFlops exactly 0, which a correctness run - never writes. Test: in a saved `subtile_mxfp8_correctness_regression` run, every `-nan` was - replaced with `123.5`, and LibraryLogic and LibraryClient were re-run (with - `ForceRedoBenchmarkProblems=False`, so the benchmark step was skipped). LibraryLogic then - found exact winners for all 5 sizes instead of `{}`. Yet the written logic YAML was - byte-identical, and LibraryClient passed. - - **8 `ductile/gfx950/*` configs, `SyncsPerBenchmark: 0`.** Ductile's GA reads the GFlops cells - as its fitness scores (`_read_and_validate_scores_csv`), and every one is NaN. Every Ductile - log shows `# ExactWinners: {}`. The GA and its LibraryLogic step run without any performance - signal; only the plumbing and the per-candidate correctness are tested. No LibraryClient - follows. - - **`groupedgemm/grouped_gemm` and the 7 `custom/*` configs.** These benchmark: `grouped_gemm` - sets `SyncsPerBenchmark: 1`, and the custom configs leave it at the default of 1. They get - real winners. -3. In the remaining 158 configs, the CSV is written, copied, and never read. - -So in correctness mode the CSV needs only to exist, with the right shape (sizes plus one column per -solution), for the `shutil.copy` and `addFromCSV` calls. `BenchmarkProblems._writeSyntheticResultsCSV` -already writes such a file for `--cpu-only`. Two caveats: - -- That synthetic file uses a fixed positive GFlops. Non-Prediction LibraryLogic would then pick - winners where today it picks none. This was not tested for the Ductile configs. -- The per-solution `-1` for failures is not consumed in correctness runs, since failures already - fail the test through the exit code. - -### YAMLs with `LibraryLogic:` / `LibraryClient:` sections - -**What the sections do.** `Tensile.py` runs these steps after BenchmarkProblems: - -- `LibraryLogic:` reads `2_BenchmarkData/*.csv` and the solutions YAML, picks winners per size, - and writes the logic YAML (the format hipBLASLt ships) to `3_LibraryLogic/`. It costs about - 0.2 s per config. -- `LibraryClient:` reads `3_LibraryLogic/`, so it depends on `LibraryLogic:`. It builds a real - library from the logic with `TensileCreateLibrary` (1.9–3.3 s), then runs the client against that - library: solution selection through the library, with validation on (0.26–0.43 s). - -With an empty `LibraryClient:` section and no exact logic, `ClientWriter` falls back to -`ProblemSizesMockDummy`, which is a single size, `[128, 128, 1, 512]`. So the library client never -runs the YAML's own benchmark sizes. - -**Who has them.** 30 common YAMLs and all 12 `custom/` YAMLs: - -| Group | Added by | `SyncsPerBenchmark` | `LibraryClient` | What the section achieves | -| --- | --- | --- | --- | --- | -| 10 `gemm/gfx950/*mxfp8*` correctness/repro configs | #7781 (multi-DU support), all at once; the first common YAMLs to use `Prediction` | 0 | yes, empty section | `Prediction` logic lists every solution; the CSV values do not change it (tested above). The library client validates only the dummy 128×128×1×512 size, about 24 s for all 10. No reason is given in the commit message or the YAMLs. | -| 14 `ductile/*` (8 on gfx950) | #8831 (Ductile backend) | 0 | commented out | Mirrors a tuning flow, whose product is a logic file. With NaN scores, the logs show `Winners set()`, `Exact Logic: {}` and a warning, and nothing reads the file. Nothing in Ductile reads `config["LibraryLogic"]`. | -| `gemm/fp8nfp16mix_hfp8ns`, `gemm/xfp32` (+ `LibraryClient`), `groupedgemm/{grouped_gemm, gfx11/grouped_gemm_gfx11, grouped_gemm_ck_gfx942}`, `sparse/gfx94x/libray_logic` | older | 1 (opted out in the unit test: "LibraryLogic requires benchmark data to build the library table") | `gemm/xfp32` only | Exercise winner selection with real timing data. Only `grouped_gemm` runs on gfx950. | -| 12 `custom/*` (see below) | #9304 / #11155 | unset, so the default 1 | no | `GridBased` logic with real winners, because they benchmark. Nothing reads it. | - -The opt-out reason "LibraryLogic requires benchmark data" holds for `GridBased` logic: with NaN -data it finds no winners. It does not hold for `Prediction` logic, whose output was byte-identical -with and without real values. - -### The `custom/` folder - -`Tests/custom/` is a sibling of `common/`. It holds 12 YAMLs: `custom_aiter_bf16`, -`custom_aiter_f4`, `custom_ck`, `custom_rocblas_gemv_bf16_tn_m1/_m2/_m4`, `custom_rr`, `custom_rr_f6`, -`custom_rr_mx`, `custom_tensile_sk`, `custom_triton_f4` and `custom_wave_bf16`. They come from the -"Gemm-From-Anywhere" custom-kernel framework (#9304, reverted in #11104, re-landed in #11155; -later additions such as #12965). Each feeds a prebuilt external kernel (aiter, CK, rocBLAS GEMV, -rr, Triton, wave, or Tensile StreamK) through the `CustomKernel` fork parameter, with -`ValidateMetadata: True`. The PR's test plan states their purpose as "build and dispatch each demo -kernel end-to-end through the real `CustomKernels:` fork-parameter path". Producing logic files is -not mentioned. - -They run as part of the common tests, even though they are outside `common/`: - -- `config_helpers.findConfigs()` walks all of `Tests/` and parametrizes every YAML into - `common/test_config*.py`. -- `conftest.pytest_collection_modifyitems` marks each test with the top directory of its *test - file*. So a `custom/` YAML gets the `common` marker, plus `custom` from `configMarks`. -- With `-m "common and not skip-gfx94X"` and `--gpu-targets gfx950`, `--collect-only` lists 552 - tests, including all 12 custom YAMLs. - -Results on gfx950: - -- **CI-like combined run (`full_runs_gfx950/run.sh`):** 7 PASSED (`aiter_bf16`, `aiter_f4`, the - three `rocblas_gemv`, `triton_f4`, `wave_bf16`); 3 SKIPPED (`ck`, `rr`, `tensile_sk`); 2 FAILED - (`rr_f6`, and `rr_mx`, which failed to assemble). That run was on an older commit; the current - status of the two failures was not rechecked. -- **Per-config reruns:** the 7 passing configs took 5–7 s each. - -None of the 12 sets `SyncsPerBenchmark`, so all of them time their kernels. All 12 have a -`GridBased` `LibraryLogic:` section and no `LibraryClient:`. - -### Existing parameters already express "no benchmarking" - -524 of the 540 common YAMLs set `SyncsPerBenchmark: 0`, so no timed enqueues run. Only 16 set it -to 1. Of those 16, 5 ran on gfx950, with 288 solutions out of 2,355,016 in total: the -`rotate_mode0/1`, `sk_sgemm_debug_modes`, `grouped_gemm` and `benchmark_runs/fp32_tn` configs. -For today's YAMLs, "`SyncsPerBenchmark == 0`" therefore matches "nothing is timed". It is not the -client's full rule, though, and "nothing is timed" is not the same as "this is a test". See -[Deciding when a run is a test](#deciding-when-a-run-is-a-test). - -With `SyncsPerBenchmark: 0`, the following parameters change nothing, measured or read in code: - -- `EnqueuesPerSync`, `MaxEnqueuesPerSync`, `MinFlopsPerSync`, `SleepPercent` and `KernelTime`: only - read inside the timed-enqueue loop, or by `BenchmarkTimer` when it times that loop. -- `HardwareMonitor`: samples only around timed enqueues, and `Tensile.py` already forces it off on - gfx950 and gfx1250. -- `SkipSlowSolutionRatio`: 0.0 in all 19 YAMLs that set it. - -### What each piece of output is for - -| Output | Needed for correctness? | Controlled by | -| --- | --- | --- | -| exit code = number of failed validations | yes | always | -| `results-file` CSV (one GFlops column per solution) | must exist with the right shape; values are NaN/−1 and do not change any output (see above) | always | -| FAILED row on stdout (row level Error) | yes | printed at every log level | -| mismatch lines (`[i] elem=… idx=…: ref!=gpu`), `Check failed in output tensor` | yes | printed with `std::cout` at every level, up to `ValidationMaxToPrint` | -| Synchronizer residue messages | yes | `LogLevel::Error` | -| CSV row for every PASSED solution | no | `ClientLogLevel` ≥ Verbose | -| CSV row and predicate `debugEval` dump for every rejected solution | no | `ClientLogLevel` ≥ Terse (row) and ≥ Verbose (dump) | -| granularity, tiles-per-CU and memory-byte columns (`BenchmarkTimer::preSolution`) | no | only shown in the stdout row, or with `CSVExportExtraCols` | -| time-us, gflops, gbps (`BenchmarkTimer::postSolution`) | no; always NaN when nothing is timed | always computed | -| warmup start/stop `hipEvent`s (2 per launched kernel) | no; only `BenchmarkTimer` reads them | always created | -| warmups beyond the first launch | no, unless the YAML deliberately relaunches (see below) | `NumWarmups` | - -### Measurements - -**Setup.** All runs replay the dtl client invocations (443,808 solutions) or the -`gemm/gfx950/xfp32` ones (94,668 solutions, `NumWarmups: 5`), with `KMP_BLOCKTIME=0`, 2 runs per -variant. The per-solution host cost is measured as `pre_solution` + `run_current_solution_check` + -`post_solution`. These are the enclosing timers, so work moved out of an inner timer is still -counted. - -**Output equivalence.** This was checked for every variant against `ref`, on 3 dtl and 2 xfp32 -invocations, at Debug and Error, both with clean runs and with injected failures. Failures were -injected by an `LD_PRELOAD` shim that flips one byte of every 7th device-to-host `hipMemcpy` of at -least 4 KiB (`/tmp/bookkeep/micro/corrupt.cpp`). In all cases: - -- exit codes were identical (0, 20, 216 and 255); -- results CSVs were byte-identical; -- the FAILED rows (61, 216, 2,128 and 3,934 per invocation) and the mismatch locations were - identical, after normalising the run-specific random data values (`srand seed` is time-based). - -| Variant (dtl) | How | Per-solution host cost (ms) | dtl stdout | -| --- | --- | --- | --- | -| today (Debug) | – | 8,941 / 8,780 | 1,191,874 lines, 520 MB | -| `ClientLogLevel: 0` | existing parameter | 6,517 / 6,279 (−28%) | 10,350 lines, 0.31 MB | -| `ClientLogLevel: 0` + correctness mode (prototype `corr`) | client change | 3,121 / 3,167 (−65%) | 10,324 lines, 0.30 MB | -| `ClientLogLevel: 0` + `corr` + no timing events (prototype `corr2`) | client change | 3,166 / 3,103 | same | - -With `ClientLogLevel: 0`, a clean invocation that printed 60,940 lines prints 87. One with 3,934 -failures prints 14,259: exactly the FAILED rows and the mismatch lines. - -`ClientLogLevel` is already settable from the test command line. `--tensile-options=…, ---global-parameters,ClientLogLevel=0` (the same mechanism as `TimingInstrumentation=True`) gave -`log-level=Error` in every `ClientParameters.ini` of `streamk/sk_dynamic_work_stealing`, and the -test passed. - -Prototype `corr`, in `/tmp/bookkeep/client_corr`, applies when `num-syncs-per-benchmark == 0` (or -`num-enqueues-per-sync == 0`) and `csv-export-extra-cols` is false. - -**Correction:** the `num-enqueues-per-sync == 0` half of that condition is wrong. With -`num-syncs-per-benchmark=1`, `num-enqueues-per-sync=0` and the default `min-flops-per-sync=1`, the -client still times every solution: 288/288 timed loops, all with real `time-us`; see -[Deciding when a run is a test](#deciding-when-a-run-is-a-test). No current YAML has that -combination (all six `EnqueuesPerSync: 0` files also set `SyncsPerBenchmark: 0`), so the -measurements above are unaffected. A real implementation must use the full rule below. - -When the prototype's condition holds: - -- `BenchmarkTimer::preSolution` skips `projectedPerformance` and its 9 reports. -- `BenchmarkTimer::postSolution` skips the performance calculation and reports the same NaN values - directly. The original's 0/0 gives a negative quiet NaN, which prints as `-nan`; the prototype - reproduces it. -- `LogReporter::postSolution` checks the row level before copying the row. The copy is only used for - printed rows. One side effect: the CSV header line is printed just before the first printed row, - instead of on the first solution. - -Prototype `corr2` (`/tmp/bookkeep/client_corr2`) additionally creates the warmup `TimingEvents` -with null events, and skips `BenchmarkTimer`'s `hipEventSynchronize` after warmup 0. That wait is -taken over by the validators' blocking readback, so removing it only moves attribution. Measured -on dtl: `warmup_runs` (host side of the launches) fell from 1,981–2,046 to 1,291–1,295 ms, -`warmup_events_setup` from 72 to 30 ms, and `run_iteration_teardown` from 122–129 to 46–47 ms. That -is about 0.85 s, or 3.1 µs per launched solution. On xfp32 it saved about 2.4 µs per launched -solution. - -### Warmups that nothing consumes - -`numWarmupRuns()` is the maximum over listeners: the validators ask for 1 launch, and -`BenchmarkTimer` asks for `NumWarmups`. So with `SyncsPerBenchmark: 0`, every warmup beyond the -first only matters if the YAML relaunches deliberately. On gfx950, ten configs set -`SyncsPerBenchmark: 0` with `NumWarmups > 1`: - -| Config | `NumWarmups` | Solutions | Assessment | -| --- | --- | --- | --- | -| `gemm/gfx950/xfp32` | 5 | 94,668 | no stated purpose; also sets `EnqueuesPerSync: 10`, which is unused with 0 syncs | -| 7 `ductile/gfx950/*` | 5–10 | 37–130 each | negligible | -| `streamk/sk_dynamic_work_stealing`, `sk_hybrid_work_stealing` | 2 | 32, 64 | deliberate: "Back-to-back enqueues reuse the same Synchronizer workspace, exercising the per-queue auto-reset across launches." | - -`cluster_entry_handoff` (288 warmups) and the gfx12 configs are not run on gfx950. - -xfp32 with `num-warmups=0` instead of 5 (one launch per solution; each row is one run, times in s): - -| Variant | `main_total` | `warmup_runs` (host) | `gpu_input_reset` | -| --- | --- | --- | --- | -| as configured (5 warmups) | 76.4 | 2.91 | 46.4 | -| as configured (5 warmups) | 65.9 | 2.29 | 39.9 | -| `num-warmups=0` | 53.1 | 0.58 | 30.1 | -| `num-warmups=0` | 59.6 | 0.60 | 33.5 | -| `num-warmups=0` + `ClientLogLevel: 0` | 59.3 | 0.57 | 30.6 | -| `num-warmups=0` + `ClientLogLevel: 0` | 54.5 | 0.59 | 32.0 | - -About 15 s (20%) of xfp32's client time goes. Most of the extra warmups' GPU time showed up in the -*next* solution's `gpu_input_reset`, whose blocking copy waits for the queued launches. That is the -same attribution effect described in `SynchronizerValidator.md`. Because the GPU is shared, the -GPU-bound timers vary by ±15%, but all four `num-warmups=0` runs are below both baseline runs. The -warmup count cannot be inferred away globally, because some configs relaunch on purpose. It is a -per-YAML fix. - -### Options for correctness runs - -| Option | Change type | Measured | Aggregate estimate (scaled from dtl/xfp32 counts) | -| --- | --- | --- | --- | -| 1. pass `ClientLogLevel=0` for the common tests (test harness / CI command line) | none: existing parameter | −28% per-solution host cost on dtl; stdout 520 MB → 0.31 MB | ≈ 9 s (2.1 µs per solution for the rows, plus 9.4 µs per rejected solution) and 2.6 GB of stdout | -| 2. drop `NumWarmups: 5` / `EnqueuesPerSync: 10` from `gemm/gfx950/xfp32.yaml` | YAML | xfp32 client time −20% (≈ 15 s) | ≈ 15 s; no other gfx950 config is affected | -| 3. infer "nothing is timed" in the client (prototype `corr`; the real rule is in [Deciding when a run is a test](#deciding-when-a-run-is-a-test)) | client code | a further −51% on top of option 1 (6.4 → 3.1 s on dtl) | ≈ 17 s (7.3 µs per solution) | -| 4. no timing events in correctness mode (prototype `corr2`) | client code | 3.1 µs per launched solution (dtl) | ≈ 5–6.5 s | -| 5. `KMP_BLOCKTIME=0` (option A above) | environment | removes the >1 ms tail | ≈ 9.9 s, plus the `cpu_reference_gemm` effect | - -Option 1 could also be inferred rather than passed. `ClientWriter` could write `log-level=Error` -when `SyncsPerBenchmark == 0` and the YAML does not set `ClientLogLevel`. That needs a way to tell -"explicitly set" from "default" for that parameter; not implemented. - -Remaining per-solution cost in correctness mode (dtl, `corr`): about 7 µs per solution. It is -`AllSolutionsIterator`/`ProgressListener` progress strings and solution name (1.0 s), predicate -evaluation (0.47 s), and `ResultFileReporter` building the results CSV. The CSV must exist, and its -formatting could use option B. Because its values carry no information when nothing is timed, -the client could also write it once per problem with constant cells, instead of fanning every -value out to `ResultFileReporter`. This is not prototyped. - -Not affected: data initialisation, CPU reference, input reset, the kernel launch, and both -validators. These are the correctness work itself and are covered in the other docs. - -### Planned follow-ups (recorded, not implemented) - -**1. Extend the `SyncsPerBenchmark` guard to `custom/`.** - -`Tests/unit/test_common_yaml_syncs_per_benchmark.py` today globs only `common/` -(`_COMMON_DIR.rglob`). It requires `GlobalParameters.SyncsPerBenchmark == 0`, as an int, for every -file not in `_SYNCSPERBENCHMARK_OPT_OUTS`. Each opt-out must give a reason, and stale opt-outs fail -the test. `custom/` YAMLs run as common tests (see above), so the test should also walk -`Tests/custom/`. - -All 12 custom YAMLs would fail it today, since the key is unset. Each needs either -`SyncsPerBenchmark: 0` or an opt-out with a reason. Given their stated purpose (build and dispatch -end to end), `SyncsPerBenchmark: 0` looks right. Their only consumer of timing data is the -`LibraryLogic:` step, which nothing reads (see item 2). - -**2. Remove `LibraryLogic:` / `LibraryClient:` from common YAMLs, and pin it with a test.** - -Add a unit test, in the same style as the `SyncsPerBenchmark` guard: a dataclass whitelist with a -mandatory reason, plus a stale-entry check. It fails if any YAML under `common/` has a top-level -`LibraryLogic` or `LibraryClient` key, unless the file is whitelisted. Applying it to `custom/` as -well would match item 1, but that scope is still to be decided. - -Proposed whitelist: the 6 files whose purpose is the logic/library path with real benchmark data, -which are already the `SyncsPerBenchmark` opt-outs with the LibraryLogic reason. - -| File | Sections | -| --- | --- | -| `common/gemm/fp8nfp16mix_hfp8ns.yaml` | `LibraryLogic` | -| `common/gemm/xfp32.yaml` | `LibraryLogic` + `LibraryClient` | -| `common/groupedgemm/grouped_gemm.yaml` | `LibraryLogic` | -| `common/groupedgemm/gfx11/grouped_gemm_gfx11.yaml` | `LibraryLogic` | -| `common/groupedgemm/grouped_gemm_ck_gfx942.yaml` | `LibraryLogic` | -| `common/sparse/gfx94x/libray_logic.yaml` | `LibraryLogic` | - -To remove: - -- the 14 `ductile/*` sections: the logic is written from NaN data and never read; -- the 10 `gemm/gfx950/*mxfp8*` sections: the library client validates only the dummy - 128×128×1×512 size, about 24 s per gfx950 run; -- the 12 `custom/*` sections, if the scope is extended. - -Decision needed before removal: the 10 mxfp8 configs are the only common coverage of `Prediction` -logic, library build from that logic, and `Prediction` selection in the client. Either accept -losing that coverage, or whitelist one designated config (for example -`subtile_mxfp8_tail_loop_smoke`), ideally with explicit `LibraryClient` sizes so that it validates -more than the dummy size. - -**3. Copy the results CSV only if it exists.** - -After the client runs, `BenchmarkProblems.main` does `shutil.copy(resultsFileName, -newResultsFileName)` unconditionally. The granularity CSV is already copied only `if -os.path.isfile(...)`. Apply the same guard to the results CSV, so that a run that did not write one -does not fail. - -The remaining readers fail hard on a missing file, so they must only run when the CSV is written: - -- `LibraryLogic.addFromCSV` calls `printExit`; -- Ductile's `_read_and_validate_scores_csv` checks `os.path.isfile` and calls `printExit`. - -**4. Don't write the results CSV for tests.** - -Have Tensile pass "no results file" to the client, and have the client skip `ResultFileReporter` -entirely in that case. Today `results-file` defaults to `results.csv`, and -`ResultFileReporter::Default` always constructs the reporter. An empty name would only produce a -silently failing `ofstream` while still doing all the per-solution formatting, so the client needs -an explicit "no reporter" path. The CSV must still be written whenever something reads it: a -`LibraryLogic:` section, or the Ductile backend. With item 2, that leaves only the whitelisted -files and Ductile's own benchmark runs. - -### Deciding when a run is a test - -**What the client actually checks.** Whether timed runs happen depends on several inputs, combined -across all listeners: - -| Main-loop query (`main.cpp`) | Combined as | Who returns non-zero | -| --- | --- | --- | -| `needMoreBenchmarkRuns()` (outer loop) | any listener | `BenchmarkTimer` (`num-benchmarks`), `ReferenceValidator` (once, if enabled), `ProgressListener` (once) | -| `needMoreRunsInSolution()` (per-solution loop) | any listener | `ReferenceValidator` (until validated); `BenchmarkTimer` while `enqueuesDone < num-enqueues-per-sync × num-syncs-per-benchmark` and the solution is not skipped | -| `numWarmupRuns()` | max | `ReferenceValidator` 1 (until validated); `BenchmarkTimer` `num-warmups` | -| `numSyncs()` | max | only `BenchmarkTimer`: `num-syncs-per-benchmark` (the profiler returns 0 and is compiled out) | -| `numEnqueuesPerSync()` | max | only `BenchmarkTimer`: 0 if the solution is skipped (restore-from-log map, or skip-slow); otherwise `min(max(num-enqueues-per-sync, ceil(min-flops-per-sync / flops)), (size_t)max-enqueues-per-sync)` | - -The timed loop is `if(enq) for(i < syncs)`. So nothing is timed for a solution exactly when one of -these holds: - -- `num-syncs-per-benchmark == 0`; -- `max-enqueues-per-sync == 0`; -- `num-enqueues-per-sync == 0` **and** `min-flops-per-sync == 0`; -- the solution is skipped (`prob-sol-map` restore, or `skip-slow-solution-ratio` > 0 after a slow - warmup). - -A negative `max-enqueues-per-sync` (the default −1) casts to a huge `size_t`, so it means no cap. - -Measured on one dtl invocation (288 run solutions; `ref` build): - -| syncs | enqueues | min-flops | timed loops | rows with real `time-us` | -| --- | --- | --- | --- | --- | -| 0 | 1 | 1 | 0 | 0 (all NaN) | -| 1 | 0 | 1 | 288 | 288 | -| 1 | 0 | 0 | 0 | 0 (all NaN) | -| 1 | 1 | 1 | 288 | 288 | - -`SyncsPerBenchmark` is therefore the only parameter that turns timing off on its own. `EnqueuesPerSync: -0` does not, unless `MinFlopsPerSync` is also 0 (no YAML sets it; its default is 1). -`MaxEnqueuesPerSync` is set in 4 YAMLs (`agntab_coverage_*`), always to 1. - -**Three different questions.** "Is this a test?" mixes three separate decisions: - -1. **Is anything timed?** This decides the timing-only work: granularity and performance values, - timing events, the `validate_gpu_sync` wait, and warmups beyond what the validators need. Use - the client's own rule above, evaluated in the client from its own arguments (prototype `corr` - did this, with the wrong enqueue term). Inferring this is safe: when nothing is timed, the - skipped values are NaN by construction. -2. **Is the results CSV consumed?** This decides whether to write and copy the CSV. It is a - property of the YAML and backend, not of timing: write it if the config has `LibraryLogic:` or - uses the Ductile backend. Python knows both before it launches the client. -3. **Is this run part of the test suite?** This decides verbosity (`ClientLogLevel`) and any - test-only shortcuts. It cannot be inferred from timing parameters: 16 common YAMLs time on - purpose and are still tests (`benchmark_runs/*`, `rotate_mode*`, `icache_flush`, - `*_debug_modes`, the LibraryLogic opt-outs), and a user benchmarking with `SyncsPerBenchmark: - 0` is not running a test. The reliable signal is the harness saying so: `tensile_args` in - `Tests/conftest.py` builds the argument list for every Tensile invocation in the common tests, - so it can append a dedicated global parameter, which `ClientWriter` turns into client options. - One constraint: the conftest `--global-parameters` option forwards a single value, which `tox` - already uses for `CheckASMCodeSize=True`. So the flag should be appended by the fixture itself, - or the option made to accept several values; `--tensile-options=…,--global-parameters,X=Y` - also works, as tested for `ClientLogLevel=0`. - -Recommendation (not implemented): use rule 1 inside the client for timing-only work, rule 2 in -Python for the CSV, and an explicit harness-set parameter for rule 3. Do not key everything on -`SyncsPerBenchmark`. - -## Not measured / open items - -- **`cpu_reference_gemm` and `KMP_BLOCKTIME`.** In the dtl replay, `cpu_reference_gemm` took - 5.8–14.8 s with the default blocktime and 2.4–2.7 s with `KMP_BLOCKTIME=0`, over 3 runs each. The - Reference GEMM section is 1,080 s (42%) of the aggregate, so this deserves its own investigation. - Mechanism not established. -- **CI thread counts.** Not checked: whether CI sets `OMP_NUM_THREADS` for the client. If it does - not, libomp sizes the pool from the affinity mask, not from the pod's CPU quota, while 4 xdist - workers run clients at the same time. Spin contention would then be worse than measured here. - Inference. -- **`post_problem` (1.9 s, 68 µs mean, 50% in calls > 1 ms).** `ResultFileReporter` adds a column - per solution name, so each problem's results row grows with the solution count. Not broken down. -- **Other configs.** Only dtl was replayed. The narrow per-config spread (24–37 µs per solution, - p10–p90 weighted by solutions) suggests a similar mix elsewhere. Configs with timed iterations - (non-NaN `time-us`) exercise a few extra branches in `ResultFileReporter`. -- **The `opt` SIGSEGV window** described under option B. -- **Whole-run effect.** Not measured; GPU sharing made whole-client wall times too noisy. diff --git a/projects/hipblaslt/tensilelite/timing_runs/python_kernel_bench_setup_investigation.md b/projects/hipblaslt/tensilelite/timing_runs/python_kernel_bench_setup_investigation.md deleted file mode 100644 index 9081ef042494..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/python_kernel_bench_setup_investigation.md +++ /dev/null @@ -1,232 +0,0 @@ -# `python_kernel_bench_setup` investigation (gfx950 per-config build) - -`full_runs_gfx950/per_config_build/AGGREGATE_analysis.txt` reports `python_kernel_bench_setup` at -131,638.45 ms (5.1% of build wall time) over 855 calls, mean 154 ms. This note explains what that -time is and how to remove almost all of it. Timing claims come from measurements (in-run timers, -offline replays of the exact inputs, and cProfile used only for attribution); code reading was -used to decide what to measure. Inferences are marked as such. - -## Summary - -The timer wraps one loop in `writeBenchmarkFiles` (`tensilelite/BenchmarkProblems.py:504`) that -builds the list of unique *helper kernels* (the HIP source kernels in `Kernels.cpp`: `PostGSU` -conversion, `BetaOnly`, activation, reduction). For every solution it calls -`initHelperKernelObjects(...)`, which constructs every helper object that solution would need, -then keeps only the ones whose name has not been seen yet. - -- **Almost all constructed helpers are thrown away.** A benchmark step has one `ProblemType`, and - the helpers depend only on it and on a handful of solution fields, so thousands of solutions - produce the same 2–14 helpers. `fp16_tn` step 0: 4,096 solutions build **24,576** helper objects - to keep **9**. -- **Each helper object costs two full `deepcopy(ProblemType)`s**: once in the `init*KernelObjects` - function, and again in the `KernelWriter*` constructor. That is 3.4–37 deep copies per solution - depending on the problem type (bias types × vector widths × helper kinds). -- **One `deepcopy(ProblemType)` costs ~150 µs in the live process**, ~90% of which is the 13–15 - `DataType` members each copying their private `properties` dict. `deepcopy` is ~90% of the block - under cProfile. - -| Fix | `fp16_tn` (in-run) | `fp16_HH_BHS_bf16mfma` (in-run) | Output | -| --- | --- | --- | --- | -| baseline | 8,821 ms | 11,485 ms | — | -| **F1** `DataType.__deepcopy__` returns `DataType(self.value)` | 4,996 ms (1.8×) | 8,049 ms (1.4×) | identical | -| **F2** call `initHelperKernelObjects` once per distinct input key | 107 ms (82×) | 135 ms (85×) | identical | -| **F1 + F2** | **96 ms (92×)** | **79 ms (146×)** | identical | - -"Identical" means: the ordered list of helper objects passed to `writeSolutionsAndKernels` has the -same names, header text and source text (SHA-256 over all of them, every step), and every -non-`.hsaco` file in the build artifact (`TensileLibrary.dat` decompressed, `cache.yaml`, `.co`, -`Data/*.yaml`, and the cache directory names, which are content hashes) is byte-identical to the -baseline run. `.hsaco` helper objects differ between any two builds (random `__hip_cuid_*`); their -input source is covered by the helper-source hash. - -**Projected effect on the aggregate (estimate, not measured end to end):** F2 leaves ~25–65 µs per -solution in-run (measured on the two configs above, including building the unique helpers); -117,270 solutions × that ≈ 3–8 s, plus the 314 steps with ≤4 solutions (1.9 s today, which F2 -cannot shrink and F1 roughly halves). So ~131.6 s → roughly 5–10 s. - -## Method - -1. **Distribution.** The per-config build logs - (`/home/alvasile/timing_runs/full_runs_gfx950/per_config_build//_build.log.gz`) print - one `TIMING:python_kernel_bench_setup` line per benchmark step at the end of each run, in step - order, and one `# Actual Solutions: N / M after SolutionStructs` line per step; the *n*-th of - each were paired (`N` is the list passed to `writeBenchmarkFiles`). All 855 timers paired; the - 8 `ductile` configs print no solution count and were left with an empty count (2.6 s total). - Output: `/tmp/benchsetup/steps.txt`. -2. **Capture.** A pytest plugin (`/tmp/benchsetup/bsdump.py`, loaded with `-p bsdump` by - `/tmp/benchsetup/run_dump.sh`, the same build-only `test_config_build` invocation as the - aggregate runs) wraps `BenchmarkProblems.writeBenchmarkFiles` and `timing_context`. Per step it - pickles the exact `solutions`, `isaInfoMap`, assembler path and `debugConfig`; optionally - cProfiles the block; and records wall time, main-thread CPU time (`time.thread_time`), GC pause - time (`gc.callbacks`), live threads and in-process `deepcopy(ProblemType)` cost. It also hashes - every helper object's name + header + source as passed to `writeSolutionsAndKernels`. - Configs captured: `fp16_tn`, `cgemm`, `zgemm`, `gfx950/dtl`, `fp16_HH_BHS_bf16mfma`, - `gfx950/bf16_gradient_bias`, `gfx950/spmm_f16_sb` (45 steps). -3. **Replay.** `/tmp/benchsetup/replay.py` re-runs the exact loop on a pickled step, counts objects - and `deepcopy` calls, and times candidate replacements with an exact output check (ordered list - of name, header text, source text). -4. **In-run A/B.** The plugin can apply F1 and/or F2 by monkeypatching (no repo edits); - `BSDUMP_NOPROF=1` disables the profiler. Each config ran baseline / F1 / F2 / F1+F2 back to back - on the same idle cores (`fp16_tn` on 87–104, `fp16_HH_BHS_bf16mfma` on 160–177); - `/tmp/benchsetup/compare.py` diffs timers, helper hashes and artifacts. - -## Distribution in the aggregate - -Cost scales with the number of solutions in a step: 117,270 solutions in 855 steps, **1.1 ms per -solution** overall. - -| Solutions in step | Steps | Solutions | Time | -| --- | ---: | ---: | ---: | -| 1–4 | 314 | 571 | 1.9 s | -| 5–50 | 340 | 6,315 | 12.0 s | -| 51–500 | 148 | 24,494 | 30.1 s | -| >500 | 53 | 85,889 | 85.8 s | - -Top configs: `cgemm` 17.4 s (22,880 solutions), `gfx950/dtl` 15.4 s, `bufferLoadStore0` 9.8 s, -`zgemm` 9.3 s, `fp16_HH_BHS_bf16mfma` 8.9 s (2,048 solutions, 4.3 ms each), `spmm_f16_sb` 8.0 s, -`fp16_tn` 7.8 s. Per-solution cost ranges from ~0.5 ms (`zgemm`) to ~4.3 ms -(`fp16_HH_BHS_bf16mfma`); steps with 1–2 solutions in the gradient/bias configs still cost -~9–33 ms. - -## Where the time goes - -### What the loop builds versus what it keeps (replay, exact counts) - -| Step | Solutions | Helper objects built | Unique kept | Top-level `deepcopy`/solution | Distinct `ProblemType`s | Distinct F2 keys | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | -| `fp16_tn` step 0 | 4,096 | 24,576 | 9 | 16.4 | 1 | 4 | -| `fp16_HH_BHS_bf16mfma` step 0 | 2,048 | 25,600 | 12 | 35.5 | 1 | 2 | -| `spmm_f16_sb` step 0 | 2,608 | 15,648 | 9 | 24.2 | 1 | 4 | -| `cgemm` step 0 | 2,480 | 6,820 | 3 | 5.4 | 1 | 4 | -| `zgemm` step 0 | 2,112 | 3,696 | 2 | 3.4 | 1 | 4 | -| `gfx950/dtl` step 3 | 1,984 | 6,944 | 4 | 6.0 | 1 | 2 | -| `bf16_gradient_bias` step 0 | 2 | 28 | 14 | 37.0 | 1 | 2 | - -Every one of the 45 captured steps had exactly one distinct `ProblemType` and 1–4 distinct F2 keys. - -### The profile - -cProfile of `fp16_tn` step 0 in-run (26.6 s profiled versus 8.6 s unprofiled; used only for -attribution): `initHelperKernelObjects` 26.4 s cumulative, of which `copy.deepcopy` 24.3 s -(65,536 top-level calls recursing into 17.2 M). Split by helper kind: conversion (`PostGSU`) -12.7 s, `BetaOnly` 4.2 s, activation enum header 4.0 s, activation function 4.0 s. The per-call -`sorted(result, key=lambda x: ("Enum" in x.getKernelName(), result.index(x)))` is 1.5 s. -`KernelWriterAssembly(...)` and `cmdLineArchsFor` do not appear (negligible). - -### Why one `deepcopy(ProblemType)` is expensive - -`ProblemType.state` has 90 keys; 13–15 values are `DataType` objects. A `DataType` is an int plus -`properties`, a pointer to a row of the class-level table `DataType.properties` -(`Common/DataType.py:63-77`). `deepcopy` cannot know that row is shared and immutable, so every -copy duplicates the dict; after the first copy each `DataType` owns a private dict, and later -copies duplicate those too. - -| Measurement | Value | -| --- | ---: | -| `deepcopy(ProblemType)` in the live build process (`fp16_tn`, two runs) | 151.5–155.5 µs | -| same object after a pickle round trip (replay) | 72–80 µs (fp16/zgemm), 112–130 µs (cgemm/dtl) | -| `deepcopy(DataType)` (replay) | 7.3 µs, of which the `properties` dict 2.45 µs | -| replay `deepcopy(ProblemType)` with `DataType.__deepcopy__ = DataType(value)` | 41.6 µs (from 77.5) | -| replay, `__deepcopy__` returning `self` | 36.3 µs | - -The replay understates the in-run cost about 2×. In the live process `dt.properties is -DataType.properties[dt.value]` is `False` for the `ProblemType`'s members (verified in-run): they -were deep-copied earlier, so 13 `DataType`s carry 13 private dicts. Pickling a solution list -re-shares them (3 distinct dicts for 13 `DataType`s in the replayed object), and `deepcopy`'s memo -then copies each shared dict once. All speedups in the summary table are therefore from in-run -measurements; replay numbers are only used for counts and equivalence. - -### Ruled out - -- **GC:** in-run, GC pauses inside the block are 147–167 ms of 8.6–8.7 s (1,026–1,029 collections) - for `fp16_tn`, 309 ms of 11.5 s for `fp16_HH_BHS_bf16mfma`. -- **GIL / descheduling:** main-thread CPU time equals wall time to within 0.1 ms on every large - step. Live threads during the block are only `MainThread` plus the - `concurrent.futures` `ExecutorManagerThread`/`QueueFeederThread`. -- **`KernelWriterAssembly` construction, `getKernels()`:** not visible in the profile; F2 keeps - both and the remaining in-run time is 25–66 µs per solution. - -## Fixes - -### F2. Build helpers once per distinct input (the main fix) - -`initHelperKernelObjects` and the functions it calls read only these solution fields: -`ProblemType`, `GlobalSplitU`, `GlobalSplitUAlgorithm`, `isStreamK(solution)`, `StreamKAtomic`, -`_GlobalAccumulation`, `AdaptiveGemmGSUA`, `ActivationFused`, `WavefrontSize`, `ISA` (plus the -global `internalParameters["GlobalSplitUPGR"]`). The prototype memoizes the result on - -```python -key = (kind, index of an equal ProblemType in a small per-step list, - s["GlobalSplitU"], s["GlobalSplitUAlgorithm"], isStreamK(s), s.get("StreamKAtomic"), - s["_GlobalAccumulation"], s.get("AdaptiveGemmGSUA"), s["ActivationFused"], - s["WavefrontSize"], tuple(s["ISA"])) -``` - -and returns the cached list for a repeated key; the existing name-based dedupe then skips them, so -the kept list and its order are unchanged. `ProblemType` equality (`__eq__` compares the state -dicts) is checked against a list that in practice holds one entry. - -Measured: in-run 82× (`fp16_tn`) and 85× (`fp16_HH_BHS_bf16mfma`); replay 1–180× across all 45 -captured steps (smaller ratios only on steps with very few solutions), identical output on all 45. - -Implementation notes (inference): the key must be kept in sync with the fields the helper -writers read; a safer variant is to compute the key inside `KernelHelperNaming` next to the code -that reads the fields. The memo should be scoped to one `writeBenchmarkFiles` call. - -### F1. Copy `DataType` by value - -```python -def __deepcopy__(self, memo): - return DataType(self.value) -``` - -A freshly constructed `DataType` is what the code makes everywhere else, and its `properties` -points at the shared class row again. Grep found no code that assigns `.value`, `.properties` or -`properties[...]` on a `DataType` instance outside `__init__`. Measured: in-run 1.4–1.8× on this -block alone, and it is the only fix that helps the 1–4-solution steps (where every helper built is -unique). Because it changes every `deepcopy` that touches a `DataType`, it also affects other -stages; in one run each, `python_benchpost_lib_construction` dropped 3,416 → 2,938 ms (`fp16_tn`) -and 2,647 → 2,327 ms (`fp16_HH_BHS_bf16mfma`), with byte-identical artifacts. Other timers moved -within run-to-run noise; those side effects were not measured with repeats. - -### Smaller items (not needed once F2 is in) - -- Each helper is deep-copied twice (`init*KernelObjects` copies, mutates, then the - `KernelWriter*` constructor copies again). Removing one copy would halve the cost; not measured - separately because F2 makes it irrelevant here. -- The per-solution `sorted(..., key=... result.index(x))` calls `getKernelName()` and an O(n) - `index` for every element (1.5 s profiled in `fp16_tn` step 0); with F2 it runs once per key. - -### Not recommended: reuse `TensileCreateLibrary.generateKernelHelperObjects` - -TensileCreateLibrary already avoids most of this cost: `generateKernelHelperObjects` -(`TensileCreateLibrary/Run.py:897`) computes names with cheap `*KernelNames` functions and only -builds objects when a name is new. On the replayed steps it was 3–69× faster and produced the same -*set* of helpers, but in a **different order** (it sorts the enum header first globally), which -changes `Kernels.cpp` text and therefore the helper-cache key. More importantly, its name functions -do not compute the same names as the objects (verified on the replayed solutions): - -| Solution | `conversionKernelNames` | names of built objects | -| --- | --- | --- | -| `fp16_tn` step 0 | `..._PostGSU1_VW1/2/4` | `..._PostGSU16_VW1/2/4` | -| `cgemm` step 0 (`ComplexFloat`) | `Cijk_SC_PostGSU1_VW1/2/4` (3) | `Cijk_SC_PostGSU16_VW1/2` (2) | -| `zgemm` step 0 (`ComplexDouble`) | 3 names | 1 object | - -The names use the solution's `GlobalSplitU` rather than `GlobalSplitUPGR`, and the vector-width -lists disagree (`isDouble()` versus `numBytes()`); `MultipleBufferSingleKernel` also ignores -`AdaptiveGemmGSUA`. The function dedupes on these proxy names and aligns them with the objects -positionally (`itertools.compress`), which happened to give the right set on every captured step -but is not guaranteed to. This is also worth a separate look for the TensileCreateLibrary path -itself (not investigated further here). - -## Not measured / open items - -- End-to-end aggregate after the fixes (the projection above is an estimate from two configs). -- F1's effect on `python_kernel_codegen`, `python_solution_generation` and the other stages that - deep-copy solutions, with repeats; single runs showed no change beyond noise except - `python_benchpost_lib_construction`. -- Whether the TensileCreateLibrary name/object mismatch above can drop or misselect a helper for - some configuration. -- Baseline in-run times in these A/B runs (8.8 s, 11.5 s) are higher than the same steps in the - aggregate (7.7 s, 8.9 s): different cores and host load. Comparisons are only made within one - core set and back-to-back runs. diff --git a/projects/hipblaslt/tensilelite/timing_runs/python_kernel_build_src_co_investigation.md b/projects/hipblaslt/tensilelite/timing_runs/python_kernel_build_src_co_investigation.md deleted file mode 100644 index 6781487c0a27..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/python_kernel_build_src_co_investigation.md +++ /dev/null @@ -1,530 +0,0 @@ -# `python_kernel_build_src_co` investigation (gfx950 per-config build) - -`full_runs_gfx950/per_config_build/AGGREGATE_analysis.txt` reports `python_kernel_build_src_co` at -1,007,817.04 ms (39.1% of build wall time) over 855 calls. Essentially all of it is -`python_kernel_build_src_co.compile`: 1,004,724.33 ms over the 263 calls that missed the helper -kernel cache (mean 3.8 s). The cache already removes duplicate compiles, and PR #13671 overlaps -the compile with other work. This note looks at the work that is left: why a single compile takes -3–10 s, and what would make it substantially cheaper. Timing claims come from measurements; code -reading was only used to decide what to measure. Inferences are marked as such. - -## Summary - -| Option | Wall time | CPU time | Code produced | Notes | -| --- | --- | --- | --- | --- | -| **A. Build one xnack-agnostic object instead of `xnack+` and `xnack-`** | **−47 to −53%** | **−47 to −53%** | identical to today's `xnack+` | loader already tries the bare name first; asm kernels already ship this way | -| B. `-parallel-jobs=2` (if both variants are kept) | −45 to −50% | unchanged | identical | one flag | -| C. Split `Kernels.cpp` into ~4 chunks, compile in parallel, link once | −71 to −82% alone; −72 to −83% with A | +34 to +93% alone; −7 to −38% with A | identical | needs header-defined kernels moved into one TU | -| D. Make the `PostGSU16` reduction kernels smaller | up to −48 to −59% of the remaining work | same | **different** | generator change; runtime perf must be validated | -| E. Cached precompiled header of the HIP headers | −0.3 s per target per compile (−7 to −19%) | same | identical | removes most of the per-file fixed cost; works with today's build layout; makes C cheap in CPU | - -### All results - -Wall / CPU time of one helper-kernel compile, cores 0–15 idle. "Chunks" means `Kernels.cpp` -split into that many translation units, compiled in parallel (2 jobs per chunk for two targets, -1 per chunk for one target, capped at 16), then linked once per target. - -| Option | Targets | `gsuasb` | `bias_sb` | `spmm_bf16_sb` | `grad_bias` | Code | -| --- | --- | --- | --- | --- | --- | --- | -| Today | `xnack+`, `xnack-` | 3.07 / 3.11 s | 6.09 / 6.13 s | 6.96 / 7.01 s | 9.35 / 9.40 s | — | -| B: `-parallel-jobs=2` | `xnack+`, `xnack-` | 1.57 / 3.12 s | 3.33 / 6.69 s | 3.47 / 6.97 s | 4.77 / 9.47 s | identical | -| C: 4 chunks | `xnack+`, `xnack-` | 0.88 / 6.00 s | 1.27 / 9.60 s | 1.35 / 10.25 s | 1.70 / 12.59 s | identical | -| C: 8 chunks | `xnack+`, `xnack-` | 0.82 / 10.26 s | 1.30 / 14.09 s | 1.15 / 15.07 s | 1.64 / 17.44 s | identical | -| A: one xnack-agnostic object | `gfx950` | 1.62 / 1.65 s | 3.07 / 3.09 s | 3.26 / 3.29 s | 4.58 / 4.60 s | identical to `xnack+` | -| A + C: 2 chunks | `gfx950` | 1.06 / 2.05 s | 1.74 / 3.44 s | 1.90 / 3.70 s | 2.61 / 4.91 s | identical | -| **A + C: 4 chunks** | `gfx950` | **0.85 / 2.87 s** | **1.10 / 4.36 s** | **1.17 / 4.52 s** | **1.60 / 5.86 s** | identical | -| D bound: one target, `PostGSU16` kernels removed | one | 0.90 s | 1.40 s | 1.60 s | 2.02 s | n/a (kernels missing) | - -The D row is not a proposal: it removes the `PostGSU16` kernels entirely, to bound what shrinking -them could save (compare one target with them: 1.72, 3.18, 3.43 and 4.98 s). It was measured -separately from the other rows (median of 3). - -Not worth pursuing (measured): lowering `-O` (slower), `-nogpulib` (link fails, no gain), and a -per-kernel cache (would skip 6% of kernel compiles beyond today's whole-file cache), and keeping a -compiler process alive (process start-up is only ~7–15 ms; see E). - -## Relation to PR #13671 - -Checked against `origin/users/talumbau/tcl-speed-1-helper-overlap` as fetched on 2026-10-29 -(`27a83902aa7` "compile helper kernels while generating asm kernels", `d6301b0de2e` tests). - -- **What it changes.** In `writeSolutionsAndKernelsTCL` (the TensileCreateLibrary path), it - submits `buildSourceCodeObjectFiles` to a one-thread `ThreadPoolExecutor` before assembly - codegen starts and joins it after `buildAssemblyCodeObjectFiles`. The compile itself is - unchanged: still both xnack targets, one sequential clang job. The commit message cites - ~175 s for one gfx942 `Kernels.cpp`. -- **It does not touch the timer measured here.** `python_kernel_build_src_co` is recorded in - `writeSolutionsAndKernels`, the benchmark path called from `BenchmarkProblems.py`. On the PR - branch that function still runs codegen, assembly, `writeHelpers`, the assembly link and only - then the helper compile, serially. -- **The same overlap in the benchmark path would hide at most 24%.** `writeHelpers` there only - needs `kernelHelperObjs`, which is available on entry, so the compile could also start before - codegen. Per `python_kernel_compilation` call in the 186 production logs, the hidden time is at - most min(helper compile, codegen + validate + write_assemble + build_co of the same call): - - | Compile cost | Total compile | Upper bound hidden by overlap | Still exposed | - | --- | --- | --- | --- | - | Today | 1,004.7 s | 237.7 s (24%) | 767.0 s | - | With A (× 0.5) | 502.4 s | 194.6 s | 307.8 s | - | With A + C, 4 chunks (× 0.22) | 221.0 s | 143.6 s | 77.4 s | - - Helper compiles happen in only 263 of the 855 calls (the rest hit the cache), while the 920.8 s - of assembly work is spread over all 855; only 17 of the 263 compiles are shorter than their - call's assembly work. These are upper bounds (inferred): codegen runs on a worker pool, so a - concurrent compile competes with it for cores, especially a chunked one. -- **The options here apply to the PR's path too.** The TCL path compiles the same kind of - `Kernels.cpp` with the same `Compiler`, so A and B halve that ~175 s job and C splits it - across cores, independently of the overlap (inferred; the gfx942 library build was not - measured here). - -## Method - -- **Captured compiles.** Four configs re-built with `--build-only` exactly as - `run_per_config_build.sh` does, with `TENSILE_DISABLE_HELPER_CACHE=1` and - `Tensile_CXX_COMPILER_LAUNCHER` pointing at a wrapper that saves `Kernels.cpp`, the headers in - its include directory, and the argv, then runs the command unchanged. The captured times match - production: - - | Config | Kernels in `Kernels.cpp` | Lines | Captured compile | Production mean | - | --- | --- | --- | --- | --- | - | `gemm/gsuasb` | 32 | 28,377 | 2.95–3.12 s | ~3.0 s | - | `gemm/gemm_bias_strided_batched_gfx940` | 50 | 55,990 | 5.98–6.47 s | 5.25 s | - | `sparse/gfx950/spmm_bf16_sb` | 50 | 56,806 | 6.19–6.48 s | 6.23 s | - | `gradient/gfx950/bf16_gradient_bias` | 74 (+14 in `Kernels.h`) | 84,062 | 9.20–9.83 s | 9.33 s | - -- **Experiments** re-run the captured command (`amdclang++ -D__HIP_HCC_COMPAT_MODE__=1 - --offload-device-only -x hip -O3 -Xoffload-linker --build-id=sha1 -std=c++17 - --offload-arch=gfx950:xnack+ --offload-arch=gfx950:xnack-`, AMD clang 22.0.0git, ROCm 7.2.1) - with one change at a time, on cores 0–15, which were idle during these runs (the host is - shared). -- **Code equivalence** is checked per kernel: disassembly with addresses and comments stripped - and each function cut after its last `s_endpgm` (inter-function alignment padding moves with - function order), plus register, LDS, scratch and spill counts from the code-object metadata. - Whole files are never byte-identical, even between two identical compiles: each compile gets a - random `__hip_cuid_*` symbol and its own build ID. -- Scratch scripts and captures: `/tmp/syncval_bench/kcap` (`launcher.sh`, `run_build.sh`, - `cc.sh`, `chunkbuild.py`, `perkernel.py`, captures in `/NNN/`). - -## Distribution in the aggregate - -From the 186 per-config build logs (184 configs compile at least once): 263 compiles, median -3.1 s, p10 2.8 s, p90 6.1 s, max 9.5 s. 183 of them take 2–4 s; a second group of 33+29 takes -5–7 s (bias, grouped, sparse), and 7 take 8–10 s (gradient + bias). No single config dominates: -the largest, `subtile_mxfp4`, is 29.9 s over 9 compiles. - -The number of distinct (`Kernels.cpp`, `Kernels.h`) pairs in each captured config equals its -production compile count (2, 3, 2, 3), so the existing cache already removes every exact -duplicate. - -## What `Kernels.cpp` contains - -Generated HIP C++ helper kernels: GSU reduction/conversion kernels (`*_PostGSU{16,8,4,2,1}_VW{4,2,1}`, -each with a grouped `_GG_` twin), global-accumulation kernels (`*_GA`), and for bias gradients, -reduction kernels. `KernelHelperNaming.initConversionKernelObjects` creates one conversion object -per load vector width; each expands into the whole `PostGSU` family, which the runtime chooses -from by the actual GSU value. All kernels are `extern "C"`; macros are `#define`d and `#undef`d per -kernel. `Kernels.h` holds the argument structs and declarations, and for bias gradients also -defines `Dijk_*_Reduction` kernels. - -## Where the time goes - -### 1. Every kernel is compiled twice (50%) - -`--gpu-targets gfx950` expands to `gfx950:xnack+` and `gfx950:xnack-` -(`Common/Architectures.py`), and the driver compiles the device code once per target, -sequentially, front end included: - -| Config | Both variants | One variant | -| --- | --- | --- | -| `gsuasb` | 3.16 s | 1.56 s | -| `grad_bias` | 9.27 s | 4.64 s | - -### 2. A fixed ~0.4–0.7 s per target, almost all HIP headers - -Single target, `gsuasb` include directory: - -| Translation unit | Time | -| --- | --- | -| empty | 0.23–0.24 s | -| `#include ` | 0.33 s | -| + `` | 0.39 s | -| `#include ` | 0.24 s | -| `#include "Kernels.h"` | 0.41–0.42 s | -| full `Kernels.cpp`, `-fsyntax-only` | 0.44 s | - -The headers cost ~0.42 s (`gsuasb`) to ~0.67 s (`grad_bias`, larger `Kernels.h`); parsing the -kernels themselves is cheap. `-nogpulib` does not change the empty-TU time and makes the link -fail (the kernels need `ocml`/`ockl`). - -The "empty" row is not start-up: it includes `__clang_hip_runtime_wrapper.h`, which the driver -adds to every HIP file. Process start-up, driver, compiler start-up, device libraries, link and -bundle together are ~21 ms. Of the ~400 ms, ~350 ms is HIP headers and ~25 ms TensileLite static -headers; the per-header breakdown and how to remove it are in option E. - -### 3. The rest is optimization and machine code generation, proportional to code size - -`-ftime-report`, `gsuasb`, one target (1.54 s): machine code generation 0.69 s (45%), front end -0.46 s (30%), optimizer 0.35 s (22%), IR generation 0.05 s (3%). The largest passes are AMDGPU -instruction selection (0.19 s), `InstCombine` (0.13 s) and register coalescing (0.07 s); no single -pass is pathological. - -Lower optimization makes it slower, because unoptimized code is more expensive to lower: - -| Config, one target | -O0 | -O1 | -O2 | -O3 | -| --- | --- | --- | --- | --- | -| `gsuasb` | 2.86 s | 2.51 s | 1.55 s | 1.56 s | -| `grad_bias` | 8.19 s | 7.39 s | 4.53 s | 4.65 s | - -### 4. The `PostGSU16` kernels are about half of it - -The "universal" `PostGSU16` kernels (non-`UnrollOnly`, `GlobalSplitUPGR` = 16) are 4,260–5,865 -lines each, against 100–145 lines for `PostGSU8/4/2/1`. `KernelWriterConversion.kernelBody` -generates a software-pipelined reduction: 16 prefetched loads, a `while` loop over full groups of -16, then a `switch(gsuRemain)` with 15 fully unrolled cases, each re-emitting all 16 accumulate -steps plus its interleaved loads, with every inline-asm add duplicated under `#if -defined(__gfx950__)`/`#else`. One kernel contains 392 inline-asm blocks; code size grows roughly -with GSU². Compiled alone, a `PostGSU16_VW4` kernel costs 210–226 ms, `VW2` 94–117 ms, `VW1` -80–92 ms, while a `PostGSU8` kernel costs 3–37 ms. - -Removing just these kernels from each file (one target, median of 3): - -| Config | Full | Without `PostGSU16` | `PostGSU16` share | `PostGSU16` kernels | -| --- | --- | --- | --- | --- | -| `gsuasb` | 1.72 s | 0.90 s | 48% | 6 of 32 | -| `bias_sb` | 3.18 s | 1.40 s | 56% | 12 of 50 | -| `spmm_bf16_sb` | 3.43 s | 1.60 s | 53% | 12 of 50 | -| `grad_bias` | 4.98 s | 2.02 s | 59% | 18 of 74 | - -## Options - -### A. Build one xnack-agnostic helper object - -Compiling with `--offload-arch=gfx950` (no xnack feature) produces one code object that loads in -either xnack mode. - -- **Time.** `gsuasb` 1.62–1.65 s against 3.07–3.16 s today; `bias_sb` 3.07 s against 6.09 s; - `spmm_bf16_sb` 3.26 s against 6.96 s; `grad_bias` 4.58 s against 9.35 s. CPU time halves too. -- **Code.** All 32 `gsuasb` kernels are identical to today's `xnack+` object. Against `xnack-`, - 17 kernels differ by `s_nop` padding: 720 against 714 `s_nop 0` in 26,771 instructions. -- **Loading.** `SolutionAdapter::initializeLazyLoading` already tries - `Kernels.so-000-.hsaco` before the `-xnack-` and `-xnack+` names. -- **Consistency.** The main assembly kernels already ship this way: `TensileLibrary_gfx950.co` - has ELF flags `gfx950, xnack, sramecc` (agnostic), while the helpers are - `Kernels.so-000-gfx950-xnack+.hsaco` (`xnack+`) and `-xnack-.hsaco` (`xnack-`). -- **Risk.** On `xnack-` systems the helpers would run the `xnack+`-style code (the extra - `s_nop`s). The runtime effect was not measured. If shipped libraries must keep both variants, - this could apply to benchmark/test builds only. - -### B. Compile the two variants in parallel - -`-parallel-jobs=2` makes the driver run the two device compiles concurrently. `--offload-jobs=2` -is ignored by this driver path ("argument unused"). - -| Config | Today | `-parallel-jobs=2` | -| --- | --- | --- | -| `gsuasb` | 3.07 s wall, 3.1 s CPU | 1.57 s wall, 3.1 s CPU | -| `bias_sb` | 6.09 s, 6.1 s | 3.33 s, 6.7 s | -| `spmm_bf16_sb` | 6.96 s, 7.0 s | 3.47 s, 7.0 s | -| `grad_bias` | 9.35 s, 9.4 s | 4.77 s, 9.5 s | - -The disassembly is identical to the sequential build. Not needed if A is adopted. - -### C. Split the file and compile the pieces in parallel - -The kernels are independent, so `Kernels.cpp` can be split into N translation units that each -include `Kernels.h`. Each (chunk, target) is compiled to a relocatable object, and one `lld --shared` link per target produces the single `Kernels.so-000-*.hsaco` the runtime loads (the -driver already does exactly this compile-then-link for one TU). The experiment ran the driver's -own `cc1` and `lld` commands (from `-###`) in a process pool, with chunks balanced by line count. - -- **Code.** Every kernel identical, with identical metadata, in every case tested: 32, 50, 50 and - 88 kernels, both targets. -- **Header-defined kernels.** `grad_bias`'s `Kernels.h` defines 14 `Dijk_*_Reduction` kernels; - with more than one chunk, the link fails with duplicate symbols. The experiment guarded them - so only chunk 0 defines them; a real change should move them into the `.cpp`. - -Both targets (wall / CPU): - -| Config | Today | 1 chunk, 2 jobs | 4 chunks, 8 jobs | 8 chunks, 16 jobs | -| --- | --- | --- | --- | --- | -| `gsuasb` | 3.07 / 3.1 s | 1.61 / 3.2 s | 0.88 / 6.0 s | 0.82 / 10.3 s | -| `bias_sb` | 6.09 / 6.1 s | 3.35 / 6.7 s | 1.27 / 9.6 s | 1.30 / 14.1 s | -| `spmm_bf16_sb` | 6.96 / 7.0 s | 3.27 s / — | 1.35 / 10.3 s | 1.15 / 15.1 s | -| `grad_bias` | 9.35 / 9.4 s | 4.75 / 9.4 s | 1.70 / 12.6 s | 1.64 / 17.4 s | - -Combined with A (one xnack-agnostic target): - -| Config | Today | A alone | A + 2 chunks | A + 4 chunks | -| --- | --- | --- | --- | --- | -| `gsuasb` | 3.07 / 3.1 s | 1.62 / 1.65 s | 1.06 / 2.05 s | 0.85 / 2.87 s | -| `bias_sb` | 6.09 / 6.1 s | 3.07 / 3.09 s | 1.74 / 3.44 s | 1.10 / 4.36 s | -| `spmm_bf16_sb` | 6.96 / 7.01 s | 3.26 / 3.29 s | 1.90 / 3.70 s | 1.17 / 4.52 s | -| `grad_bias` | 9.35 / 9.4 s | 4.58 / 4.60 s | 2.61 / 4.91 s | 1.60 / 5.86 s | - -Each extra translation unit repays the fixed ~0.4 s, so CPU time grows with the chunk count and -gains flatten beyond 4 chunks. In CI (4 xdist workers on 25 cores) the compile competes with the -other workers, so a small chunk count (or one chosen from free cores) is the safer default. A -cached precompiled header (option E) cuts the fixed cost per chunk from ~400 ms to ~35 ms. - -The driver has no supported switch for "compile to a relocatable device object without -linking": `--no-gpu-bundle-output` still links. The supported alternative is `-fgpu-rdc` -(bitcode per TU, machine codegen in `lld` LTO, which the driver already runs with -`--lto-partitions=8`). On `gsuasb`, one target: bitcode compile 0.98 s, then `lld` with 8 LTO -partitions 0.30 s wall (0.97 s with 1), so ~1.28 s against 1.62 s. **But none of the 32 kernels -came out identical** (26,761 against 26,771 instructions), so this route needs performance -validation; invoking `clang -cc1`/`lld` directly, or `-S` plus assembling, keeps today's code. - -### D. Shrink the `PostGSU16` kernels - -The `PostGSU16` kernels are 48–59% of each compile (table above). Their size comes from the -unrolled remainder `switch` in `KernelWriterConversion.kernelBody` (non-`UnrollOnly` path). -Generating the remainder as a loop, lowering `GlobalSplitUPGR`, or sharing one remainder path -would cut most of that, but it changes the code of a kernel on the GSU path. It is only worth doing -with a runtime benchmark of the reduction kernels showing no regression (not measured here). - -### E. A cached precompiled header for the fixed cost - -With hundreds of compiles (or chunks), the fixed per-file cost matters. It is not process -start-up: clang has no persistent server mode, and it would not help much anyway. Breakdown of a -compile of `#include "Kernels.h"` with no kernels (`gsuasb`), one target, median of 15, cores 0–7. -Headers were added one at a time in include order, so each row is that header's increment; the -rows sum to 400.3 ms against 405.1 ms measured for the whole driver run: - -| Step | Owner | Time | -| --- | --- | --- | -| process start (`amdclang++ --version`) | driver | 3.4 ms | -| driver planning the jobs (`-###`) | driver | 0.9 ms | -| compiler (`cc1`) start-up: empty file, no headers, no device libraries | compiler | 5.9 ms | -| device-library bitcode (`ocml`, `ockl`, `oclc_*`) | ROCm | 7.3 ms | -| `__clang_hip_runtime_wrapper.h` (implicit, added by the driver) | HIP | 192.7 ms | -| `` | HIP | 100.0 ms | -| `` | HIP | 0.7 ms | -| `` | HIP | 57.2 ms | -| C++ standard headers (``, ``, ``, ``, ``, ``, ``) | C++ std | 1.5 ms | -| `tensile_bfloat16.h` | TensileLite | 3.4 ms | -| `tensile_float8_bfloat8.h` | TensileLite | 8.3 ms | -| `TensileTypes.h` | TensileLite | 10.5 ms | -| `ReductionTemplate.h` | TensileLite | 1.6 ms | -| `memory_gfx.h` | TensileLite | 0.0 ms | -| `KernelHeader.h` | TensileLite | 1.1 ms | -| `Kernels.h` (generated: argument structs and declarations) | TensileLite, generated | 2.2 ms | -| `lld` (link) | linker | 3.0 ms | -| `clang-offload-bundler` | bundler | 0.8 ms | - -Subtotals: HIP 350.6 ms (88%), TensileLite static headers 24.9 ms, generated `Kernels.h` -2.2 ms, C++ standard headers 1.5 ms, everything else (process, driver, compiler start-up, device -libraries, link, bundle) 21.3 ms. - -The same breakdown with a PCH of the HIP and TensileLite static headers (`shared.pch`: implicit -wrapper, `hip_runtime.h`, `hip_ext.h`, `hip_fp8.h`, `KernelHeader.h` and everything it includes), -measured side by side in a second run (median of 15). With the PCH, its load cost shows up in the -first header row; increments within ±5 ms (including the small negative ones) are noise: - -| Step | Owner | No PCH | HIP + TensileLite PCH | -| --- | --- | --- | --- | -| process start (`amdclang++ --version`) | driver | 3.3 ms | 3.3 ms | -| driver planning the jobs (`-###`) | driver | 0.9 ms | 0.9 ms | -| compiler (`cc1`) start-up: empty file, no headers, no device libraries | compiler | 5.7 ms | 5.7 ms | -| device-library bitcode (`ocml`, `ockl`, `oclc_*`) | ROCm | 7.3 ms | 7.3 ms | -| `__clang_hip_runtime_wrapper.h` (implicit) / **loading the PCH** | HIP / PCH | 197.0 ms | **65.6 ms** | -| `` | HIP | 105.3 ms | −0.4 ms | -| `` | HIP | −3.2 ms | −0.1 ms | -| `` | HIP | 60.9 ms | 0.1 ms | -| C++ standard headers | C++ std | −0.4 ms | −0.5 ms | -| `tensile_bfloat16.h` | TensileLite | 9.9 ms | 0.7 ms | -| `tensile_float8_bfloat8.h` | TensileLite | 0.6 ms | −0.6 ms | -| `TensileTypes.h` | TensileLite | 4.8 ms | 0.5 ms | -| `ReductionTemplate.h` | TensileLite | 4.5 ms | 0.2 ms | -| `memory_gfx.h` | TensileLite | −1.3 ms | −0.4 ms | -| `KernelHeader.h` | TensileLite | 1.6 ms | 0.7 ms | -| `Kernels.h` (generated; not in the PCH) | TensileLite, generated | −1.7 ms | 2.3 ms | -| `lld` (link) | linker | 3.7 ms | 3.7 ms | -| `clang-offload-bundler` | bundler | 0.8 ms | 0.8 ms | -| **Sum** | | **399.5 ms** | **89.7 ms** | - -Measured whole driver runs: 412.9 ms without the PCH, 96.7 ms with it. Subtotals: HIP -360.0 → 65.1 ms (the PCH load), TensileLite static headers 20.0 → 1.1 ms, generated `Kernels.h` -about 2 ms either way. The per-header TensileLite numbers vary by ~5 ms between runs (compare -with the table above); their subtotal (20–25 ms) is stable. - -**Why "loading the PCH" still costs ~66 ms, and how to remove most of it.** It is not reading the -file: in a syntax-only run of an empty file with the PCH (56 ms), `-ftime-trace` shows `ReadAST` -0.5 ms, `-fno-validate-pch` changes nothing, the PCH records no input files to re-check, and -system calls are negligible (`strace -c`). The empty file contributes no functions or globals to -the module. 46.8 ms is `PerformPendingInstantiations`: the headers' inline code uses function -templates whose implicit instantiation is deferred to the end of the translation unit, and a PCH -stores that list, so every file that uses the PCH redoes them (483 `InstantiateFunction` events, -mostly C++ standard library: `std::basic_string` 24 ms inclusive, `std::vector`, `__stoa` from -`std::to_string`, `uninitialized_copy`, and the `` special functions such as -`__cyl_bessel_j`). `-fpch-instantiate-templates` performs them once while building the PCH: - -| Translation unit, one target | No PCH | PCH | PCH built with `-fpch-instantiate-templates` | -| --- | --- | --- | --- | -| empty file, `-fsyntax-only` | 192.9 ms | 56.5 ms | 14.0 ms | -| empty file, full compile | 210.2 ms | 79.0 ms | 33.6 ms | -| `#include "Kernels.h"`, full compile | 397.5 ms | 81.3 ms | 34.8 ms | - -Building that PCH costs 497 ms instead of 465 ms (13.5 MB instead of 12.9 MB), and afterwards -`PerformPendingInstantiations` is 0.0 ms per file. The HIP-only version through the normal driver -(`amdclang++ -x hip -fpch-instantiate-templates -Xclang -emit-pch -fsyntax-only -hip.h > hip.pch`), used on today's build directories, one target, median of 5: - -| Config | No PCH | HIP PCH | HIP PCH, `-fpch-instantiate-templates` | Code | -| --- | --- | --- | --- | --- | -| `gsuasb` | 1.57 s | 1.28 s | 1.24 s | 32/32 identical | -| `bias_sb` | 3.05 s | 2.76 s | 2.72 s | 50/50 identical | -| `spmm_bf16_sb` | 3.26 s | 2.96 s | 2.94 s | 50/50 identical | -| `grad_bias` | 4.61 s | 4.31 s | 4.28 s | 88/88 identical | - -`-fpch-instantiate-templates` saves another ~30–40 ms per translation unit compiled. With today's -single `Kernels.cpp` that is one translation unit per compile and target; with C (chunks) it is -saved once per chunk. - -**How the PCH is used, and when it pays off.** In all of these measurements the PCH was built -once and reused unchanged by every later compile (all four configs, every repetition); the -per-compile times above do not include building it. In practice it would be built once per -(compiler/ROCm version, flags, target) and cached, like `HelperKernelCache`, or built with the -TensileLite installation. One-time cost: ~0.5 s per target. Saving: ~0.33–0.36 s per translation -unit per target (no PCH vs instantiated HIP PCH above). Arithmetic, not measured end to end: - -- **Cache kept across configs (e.g. a CI job reusing `~/.tensile`):** built once per target for - the whole run; 263 compiles × 2 targets × ~0.33 s ≈ 170 s saved for ~1 s of PCH builds. -- **Cold cache per config, as in `run_per_config_build.sh`:** the PCH is rebuilt for every config. - The 184 configs average 263 / 184 = 1.43 compiles, i.e. ~2.9 uses with both targets - (~0.9 s saved for ~1.0 s of builds) or ~1.4 uses with option A (~0.47 s saved for ~0.5 s): - roughly break-even. With A + C at 4 chunks it is ~5.7 uses per config (~1.9 s saved for 0.5 s). - -**Caching it like the helper kernels.** The intended design: build the HIP PCH once per -(compiler build, ROCm install, flags, target), store it next to the helper-kernel cache, and have -every later TensileLite run read it. `HelperKernelCache` already has the pattern -(`~/.tensile/helper_cache` or `TENSILE_HELPER_CACHE_DIR`; SHA-256 key; write to -`.tmp__`, then `rename`, which is safe with concurrent xdist workers; age-based -eviction). What clang does with a cached PCH that does not match (measured, one target): - -| Cached PCH used with... | Result | -| --- | --- | -| the same flags (control) | works | -| a different xnack feature (`gfx950:xnack-` vs `gfx950`) | error: target feature mismatch | -| a different arch (`gfx942`) | error: target CPU mismatch | -| `-O2` instead of `-O3` | error: `OptimizationLevel differs` | -| `-std=c++20` instead of `-std=c++17` | error: C++20 disabled in the PCH | -| a header's mtime changed (content unchanged), or its content changed | fatal error: file modified since the PCH was built | -| `-fsanitize=address` added | accepted, with a warning that ASan is ignored for this target | -| a `-D` macro defined at PCH build time but not in the compile | **accepted silently; the macro stays defined in the file** | -| a `-D` macro with a different value | error: definition of macro differs | -| a `-D` macro defined only in the compile | works (normal behavior) | - -So: - -- **Key on the full compile flags**, including every `-D`, the target and its features, `-O`, - `-std`, and ASan, because a macro baked into the PCH would otherwise leak silently into files - that do not define it. Simplest: hash the exact argument list used to build the PCH (the same - `Compiler.default_args` plus the per-target flags) and use that same list for the compiles. -- **Key on the exact compiler build**, not only `major.minor.patch`: `HelperKernelCache` uses - `compiler.version` (`22.0.0` for this `22.0.0git` build), while clang rejects a PCH from a - different compiler build. Hash the full `amdclang++ --version` output, which includes the commit. -- **Keep the PCH to HIP headers.** Clang validates every header in a PCH by mtime and size, and an - mtime-only change is fatal (`-fvalidate-ast-input-files-content` did not relax it here). The - TensileLite static headers are re-copied into each build directory (`shutil.copy`, new mtime), - so a PCH containing them would fail on every build unless built and used from one fixed path. - The HIP headers sit in the ROCm install, unchanged until ROCm is reinstalled; this is also where - ~350 of the ~375 ms of header cost is. -- **Fall back on errors.** A reinstall of the same ROCm version changes header mtimes; the - compile then fails with "has been modified since the precompiled header was built". On any PCH - error, rebuild the PCH (or compile without it) and retry, so a stale cache entry costs one - rebuild, not a failed build. - -That is header parsing, which a precompiled header (PCH) removes. Splitting it with two PCHs -(cc1 jobs, one target, median of 9; `hip.pch` = implicit wrapper + `hip_runtime.h` + -`hip_ext.h` + `hip_fp8.h`, 434 ms to build, 12.5 MB; `shared.pch` = that + `"KernelHeader.h"`): - -| Translation unit | No PCH | `hip.pch` | `shared.pch` | -| --- | --- | --- | --- | -| empty file (only the implicit `__clang_hip_runtime_wrapper.h`) | 205 ms | 69 ms | 76 ms | -| `hip_runtime.h`, `hip_ext.h`, `hip_fp8.h` | 364 ms | 70 ms | 77 ms | -| `#include "Kernels.h"` (`gsuasb`) | 389 ms | 100 ms | 78 ms | - -(Empty file with the implicit wrapper removed: 13 ms.) So the fixed cost is almost all HIP -headers: ~192 ms for the wrapper the driver adds to every HIP file and ~160 ms for the -`hip_runtime.h`/`hip_ext.h`/`hip_fp8.h` that Tensile's headers include, ~350 ms in total. The -Tensile static headers themselves add only ~25 ms. (`grad_bias`'s `Kernels.h` costs more, 626 ms, -because it contains kernel code.) With a PCH, ~70 ms remains: loading the PCH plus an empty -compile. - -**The HIP-only PCH is the one to use.** It gets almost all of the saving, depends only on the -ROCm/clang version, flags and target (the headers live at fixed `/opt/rocm` paths), and works with -today's build directories unchanged. Built with the normal driver and used on the captured build -directories as they are (static headers copied next to `Kernels.h`), one target: - -| Config | No PCH | HIP-only PCH | Code | -| --- | --- | --- | --- | -| `gsuasb` | 1.54 s | 1.25 s | 32/32 identical | -| `bias_sb` | 2.99 s | 2.71 s | 50/50 identical | -| `spmm_bf16_sb` | 3.18 s | 2.91 s | 50/50 identical | -| `grad_bias` | 4.53 s | 4.21 s | 88/88 identical | - -Adding the Tensile static headers to the PCH saves another ~10–50 ms per compile (table below), -but requires changing where those headers are found (see "Header location"). - -Only `Kernels.h` is generated (`writeHelpers`; 10 distinct versions across the 27 captured -compiles). The other six headers (`TensileTypes.h`, `tensile_bfloat16.h`, -`tensile_float8_bfloat8.h`, `KernelHeader.h`, `ReductionTemplate.h`, `memory_gfx.h`) are static -files shipped in the package at `tensilelite/Source/`. `copyStaticFiles` → -`resources.copy_static_headers` copies them verbatim (`shutil.copy`) next to `Kernels.cpp` for -every build, so they are identical by construction; they only change when the TensileLite -checkout changes. The HIP headers come from the ROCm install. So one PCH of -``, `` and `"KernelHeader.h"` per target can serve every config -built with the same TensileLite and ROCm. Built once (463 ms, 12.9 MB) from a shared directory of -the static headers, then used for each config's whole `Kernels.cpp`, one target: - -| Config | `#include "Kernels.h"` only | Whole `Kernels.cpp` | Code | -| --- | --- | --- | --- | -| `gsuasb` | 389 → 78 ms | 1.53 → 1.22 s | 32/32 identical | -| `bias_sb` | 392 → 82 ms | 2.97 → 2.66 s | 50/50 identical | -| `spmm_bf16_sb` | 392 → 81 ms | 3.18 → 2.89 s | 50/50 identical | -| `grad_bias` | 626 → 318 ms | 4.53 → 4.20 s | 88/88 identical | - -- **With chunks (C).** Each chunk pays ~80 ms instead of ~390 ms of headers. Measured with a - per-config PCH and 4 chunks, one target: total compile CPU 2.72 → 1.50 s (`gsuasb`) and - 4.26 → 2.98 s (`bias_sb`), all kernels identical. This removes most of C's extra CPU cost. -- **Through the normal driver.** Build: `amdclang++ -x hip -Xclang -emit-pch - -fsyntax-only hip.h > hip.pch`, where `hip.h` includes ``, - `` and `` (`-x hip-header` is rejected, and with - `-fsyntax-only` the PCH is written to stdout, not `-o`). Use: add `-include-pch hip.pch`. -- **Caching.** A PCH is only valid for the same compiler, flags (`-O3`, `-std`, `-D`), target and - xnack feature, and header contents, so it would be cached per target keyed on those (e.g. the - ROCm/clang version plus a hash of `tensilelite/Source/*.h`), like `HelperKernelCache`. -- **Header location (only for the Tensile part).** Today the copies sit next to `Kernels.h`, and `Kernels.h` includes - `"KernelHeader.h"` with quotes, so the compile resolves to the per-build copy. For the PCH to be - used, the compile must resolve the static headers to the same files the PCH was built from: - build the PCH against one fixed directory (e.g. the package's `Source/`) and have the compile - find the static headers there, not in per-build copies. `KernelHeader.h`, `TensileTypes.h` and - `memory_gfx.h` use `#ifndef` guards, which would hide a duplicate copy, but - `ReductionTemplate.h`, `tensile_bfloat16.h` and `tensile_float8_bfloat8.h` use `#pragma once`, - which is per file, so a second copy at another path would be parsed again. In the experiment - `Kernels.h` was in its own directory and the static headers only in the shared one. -- **Aggregate (inferred).** ~0.3 s per target per compile × 263 compiles × 2 targets ≈ 160 s of - the 1,004.7 s, before any chunking; more with chunks, since every chunk saves it. - -## Measured and not worth pursuing - -- **Lower `-O`:** -O0/-O1 are 1.6–1.8× slower than -O3; -O2 is the same as -O3. -- **`-nogpulib`:** no change in the fixed cost, and the link fails (undefined `ocml`/`ockl` - symbols). -- **Per-kernel cache:** across the distinct files compiled for each captured config, 474 of 506 - kernel blocks are unique; a per-kernel cache would skip 32 (6%) beyond today's whole-file cache. - Cross-config reuse was not measured: these 4 configs share almost nothing, and production builds - were cold per config. - -## Not measured / open items - -- Runtime performance of the xnack-agnostic helpers on an `xnack-` system (option A), and of - `-fgpu-rdc` code (option C variant) or a smaller `PostGSU16` (option D). -- End-to-end build-time effect, and the real (contended) overlap if PR #13671's approach is - extended to the benchmark path, under CI's xdist load. -- The gfx942 TensileCreateLibrary `Kernels.cpp` (~175 s per the PR) was not captured. -- A precompiled header combined with chunking end to end (measured: per-config PCH with 4 - chunks, and the instantiated HIP PCH on whole files, separately); the PCH cache itself is a - design, not implemented. -- Only 4 of the 184 configs were captured; the per-config distribution above suggests the rest - are the same kind of files (most compiles 2–4 s), but they were not inspected individually. diff --git a/projects/hipblaslt/tensilelite/timing_runs/python_kernel_codegen_investigation.md b/projects/hipblaslt/tensilelite/timing_runs/python_kernel_codegen_investigation.md deleted file mode 100644 index 475a5127cb7a..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/python_kernel_codegen_investigation.md +++ /dev/null @@ -1,508 +0,0 @@ -# `python_kernel_codegen` investigation (gfx950 per-config build) - -`full_runs_gfx950/per_config_build/AGGREGATE_analysis.txt` reports `python_kernel_codegen` at -592,584.11 ms (23.0% of build wall time) over 855 calls, mean 693 ms. This note explains where -that time goes and what can be saved. Timing claims come from measurements: in-run probes of the -real builds, offline replays of the exact kernels, py-spy with native frames, cProfile (ranking -only) and in-run A/B runs with artifact comparison. Code reading was used to decide what to -measure. Inferences and estimates are marked as such. - -## Summary - -The timer wraps one call (`TensileCreateLibrary/Run.py:666`): `ParallelMap2(processKernelSource, -...)`, which generates the assembly text of every kernel of one benchmark step in a joblib/loky -pool (25 processes in these runs) and returns the text to the parent. Pool start-up, dispatch and -result return are **not** the cost: 2–12 ms before the first kernel starts and 2–85 ms after the -last one ends, per step (135 ms on a config's first step, when the pool starts). The time is the -generation of each kernel, and how much of it can run in parallel. - -1. **Generating one kernel costs 0.04–3 s of single-threaded Python + rocisa C++, proportional to - the assembly it produces (~90–280 ms per MB; kernels are 0.4–7 MB of text).** Of that, about 65% - is Python building the instruction tree (the store/epilogue code alone is 40–64%), ~21% is the - C++ `rocIsaPass`, ~8% freeing the tree and ~6% `str()` of the tree. -2. **19 configs set `CpuThreads: 1`, so codegen runs in one process.** They are 189.3 s of the - 592.6 s (32%), including all 13 `subtile_*` configs (169.4 s). `subtile_mxfp4`: 133.5 s → 22.6 s - in-run with 25 processes; `.co` files and libraries identical, but see the `LdsNumBytes` caveat. -3. **Kernels that will be rejected are generated in full first.** A kernel whose static VGPR - allocation already exceeds 256 is guaranteed to be rejected, but this is only acted on after - the whole kernel is generated. These are often the slowest kernels: 8–89% of codegen CPU in four - of the probed configs. Failing at `_initKernel` was exact on 798 kernels and cut `f8_gate_r` - codegen 12.6 s → 3.5 s in-run (whole build 28 s → 11 s) with identical artifacts. -4. **Most steps are too small to use the pool.** 479 steps write ≤25 kernels (159.5 s); a step's - wall time is then its slowest kernel while the other workers idle (`custom_mainloop_scheduling`: - 36 s of wall for 40.7 s of CPU on 25 cores). Steps are independent, but are generated one after - another. - -| Option | Measured / estimated effect | Output | Effort | -| --- | --- | --- | --- | -| **A. Remove `CpuThreads: 1` from the 19 test configs** (plus return `LdsNumBytes` from workers) | `subtile_mxfp4` codegen 133.5 → 22.6 s (5.9×), `write_assemble` 62.9 → 11.6 s, build 244 → 63 s, measured. Aggregate estimate: ~189 s → ~35 s codegen | identical `.co`/library; `Data/*.yaml` `LdsNumBytes` differs unless passed back | config + small code fix | -| **B. Fail fast on rejected kernels** (initial VGPR allocation > `MaxVgpr`, plus abort as soon as any rejection code is set) | VGPR check alone: `f8_gate_r` codegen −72% (12.6 → 3.5 s) in-run, `plr_zero` worker CPU −20%; both checks: CPU on rejected kernels 374.5 → 10.8 s (−97%) over 1,242 rejected kernels | identical (VGPR check: 798/798 kernels, artifacts byte-identical; both: 4,313/4,313 kernels same accept/reject) | ~10 lines | -| **C. Generate all steps of a config in one pool** | small-step configs: `custom_mainloop_scheduling` 36.0 → ≤2.1 s, `f8_gate_r` 12.4 → 3.3 s, `spmm_i8_sb` 17.6 → 9.1 s (simulated from measured per-kernel times); ≤25-kernel steps aggregate 142.4 s → ≤75 s (conservative bound) | identical (same kernels) | restructure `BenchmarkProblems` loop | -| D. Longest-first dispatch within a step | 0–6% on most configs, 24% on `spmm_i8_sb` (simulated, perfect cost knowledge) | identical | needs a cost predictor | -| E. Make the generator cheaper | epilogue 40–64%, `rocIsaPass` ~21%, free ~8%, `str` ~6% of per-kernel time; quadratic `hasAnyDependency` up to 67% on some sparse kernels | depends | larger refactors | - -A, B and C are independent and additive (inference): A applies to the 19 single-process configs, -B to rejected kernels everywhere, C to the many-small-steps configs. None of these change the -generated code. - -## Method - -1. **Distribution.** Each step's `TIMING:python_kernel_codegen` line in the build logs - (`/home/alvasile/timing_runs/full_runs_gfx950/per_config_build//_build.log.gz`) was - paired with the step's `Writing assembly kernels: Launching N threads for K tasks` line (K = - kernels that passed validation; generated kernels are more, e.g. 4,093 generated vs 3,420 written - for `plr_zero`). The 169 steps of the `CpuThreads: 1` `subtile_*` configs print no such line - (single-process path) and are kept separately. Output: `/tmp/codegen/steps.txt`, - `/tmp/codegen/cfg_totals.txt`. -2. **In-run probe.** pytest plugin `/tmp/codegen/cgprobe.py` (`-p cgprobe`, run by - `/tmp/codegen/run_cg.sh`, the same build-only `test_config_build` invocation as the aggregate, - 25 cores via `taskset`). It replaces `Run.processKernelSource` with a module-level wrapper that the - loky workers import by name, so every worker logs per-kernel start/end wall time, thread CPU - time, output size and name (`w_.txt`). The parent logs the block and `ParallelMap2` window - per step, the pickled task size, and (optionally) pickles the step's kernels, kernel writer and - rocisa data for replay. Switches: `CG_FORCE_THREADS=N` (override `CpuThreads`), - `CG_FAILFAST=1` (option B in the workers). `/tmp/codegen/analyze.py` splits each step into - pool start → first kernel, kernel span, last kernel → return, worker busy/CPU, slowest kernel. -3. **Replay.** `/tmp/codegen/replay_cg.py` (time / cProfile one kernel), `phases.py` (wraps - `rocIsaPass`, `kernelBody`, `checkResources`, `_initKernel`; times `str(module)` and freeing the - module separately; `RIPO=` toggles `rocIsaPass` options, `NOCOMMENT=1` sets - `outputNoComment`), `shares.py` (cProfile section shares), `ff.py` (baseline vs fail-fast over - every kernel of a step, 16 processes), `sched.py` (actual vs longest-first vs lower bound). - Replays reproduce in-run per-kernel times (1,931 vs 2,037 ms, 1,577 vs 1,646 ms, 122 vs 125 ms). -4. **Native profile.** `perf` is blocked (`perf_event_paranoid` = 4); py-spy 0.4.2 (installed to - `/tmp/codegen/pyspy_inst`) with `--native` works, but `_rocisa.so` is stripped, so C++ frames are - addresses only. C++ costs were therefore measured by timing the native calls directly. -5. Probed configs: `custom_mainloop_scheduling`, `fp16_tn`, `f8_gate_r`, `spmm_i8_sb`, - `gfx950/dtl`, `plr_zero`, `subtile_mxfp4`. - -## Distribution in the aggregate - -| Step (kernels written) | Steps | Kernels | Codegen wall | -| --- | ---: | ---: | ---: | -| ≤25 | 479 | 2,974 | 159.5 s | -| 26–200 | 143 | 10,051 | 127.8 s | -| >200 | 64 | 35,101 | 135.9 s | -| `subtile_*` (`CpuThreads: 1`, no count logged) | 169 | — | 169.4 s | - -Top configs: `subtile_mxfp4` 97.4 s (130 steps), `subtile_bf16` 61.8 s, `gfx950/dtl` 47.5 s, -`custom_mainloop_scheduling` 34.2 s (66 steps, 79 kernels), `plr_zero` 29.0 s, `spmm_f16_sb` -19.1 s, `spmm_{i8,bf16,f8,b8}_sb` 12–16 s each, `fp16_tn` 11.7 s, `f8_gate_r` 11.4 s (8 kernels -written). - -## Where the time goes - -### Pool overhead is negligible - -| Config | Codegen block | Before first kernel | After last kernel | Worker CPU | Cores busy (of 25) | -| --- | ---: | ---: | ---: | ---: | ---: | -| `custom_mainloop_scheduling` (66 steps) | 36.8 s | 0.6 s total (135 ms on step 0, 2–12 ms after) | 0.4 s | 40.5 s | 1.1 | -| `f8_gate_r` (4 steps) | 12.6 s | 0.14 s | 0.11 s | 46.7 s | 3.7 | -| `spmm_i8_sb` (8 steps) | 17.8 s | 0.03 s | 0.12 s | 219.5 s | 12.4 | -| `fp16_tn` (3 steps) | 14.6 s | 0.01 s | 0.02 s | 330.7 s | 22.7 | -| `gfx950/dtl` (26 steps) | 61.1 s | 0.10 s | 0.19 s | 1,312 s | 21.5 | -| `plr_zero` (17 steps) | 36.6 s | 0.06 s | 0.10 s | 789 s | 21.6 | - -A task pickles to ~29 KB (kernel 13 KB, `KernelWriterAssembly` 10 KB, `globalParameters` 3 KB) in -0.2–0.7 ms. The large configs keep the cores ~86–96% busy; the small-step configs barely use them. - -### One kernel: cost scales with the assembly text - -| Config | Kernels | Median CPU / kernel | Median output | CPU per MB of output | -| --- | ---: | ---: | ---: | ---: | -| `custom_mainloop_scheduling` | 79 | 294 ms | 1.24 MB | 281 ms | -| `fp16_tn` | 2,060 | 124 ms | 0.93 MB | 139 ms | -| `gfx950/dtl` | 8,650 | 110 ms | 0.72 MB | 160 ms | -| `plr_zero` | 3,420 | 136 ms | 0.85 MB | 164 ms | -| `spmm_i8_sb` | 546 | 167 ms | 1.28 MB | 127 ms | -| `subtile_mxfp4` | 1,414 | 67 ms | 0.68 MB | 91 ms | - -(Accepted kernels only; `dtl` alone generated 7.5 GB of assembly text.) The largest kernel seen, -`custom_mainloop_scheduling` step 40, is 7.0 MB / 161,079 lines and takes 1.9–2.0 s. - -**Phases** (`phases.py`, GC disabled, median of 3; freeing measured separately by holding a -reference — in the real run it happens inside the call when `kernelBody` returns): - -| Kernel | Total | `rocIsaPass` | `str(module)` | free module | Python tree construction (rest) | -| --- | ---: | ---: | ---: | ---: | ---: | -| cms step 40 (7.0 MB) | 1,708 ms | 617 ms | 123 ms | +208 ms | ~966 ms | -| cms step 30 (0.57 MB) | 104 ms | 30 ms | 7 ms | +10 ms | ~67 ms | -| `fp16_tn` k700 (1.56 MB) | 182 ms | 36 ms | 17 ms | +25 ms | ~129 ms | -| `f8_gate_r` k0 (2.98 MB) | 359 ms | 57 ms | 26 ms | +41 ms | ~276 ms | -| 24 kernels sampled evenly from `dtl` s11, `plr_zero` s12, `spmm_i8_sb` s0 | 5,023 ms | 1,160 ms | 315 ms | +434 ms | ~3,548 ms | - -Over the 24-kernel sample (free included): Python ~65%, `rocIsaPass` ~21%, free ~8%, `str` ~6%. -GC is not a factor (1,852 ms GC off vs 1,783 ms GC on for the largest kernel; noise). - -**Inside the Python part** (cProfile, ranking only): the store/epilogue -`notLocalSplitUGlobalWrite` is 40–64% of profiled time in 22 of 28 sampled kernels (17–39% in the rest); the main loop -(`PersistentLoop.compute`) 4–38%. The epilogue is large because it is fully unrolled per element -for every variant (edge/non-edge, beta, bias type, activation): in `fp16_tn` k700, 56% of the -21,419 lines follow `label_ActivationSetPCAddrEnd_*`. On some sparse kernels -`KernelWriter.hasAnyDependency` (a local-read × instruction scan, `KernelWriter.py:1887`) dominates: -25–67% of profiled time in four `spmm_i8_sb` kernels (903 k calls in one), 0% in `dtl`. - -**Inside `rocIsaPass`** (`rocisa/src/pass/pass.cpp`; toggling options on the 7.0 MB kernel, which -changes output so only for attribution): mandatory macro/composite/text-variable passes ~231 ms; -register graph + `removeDuplicateAssignment` ~330 ms (also yields `maxVgpr` for occupancy); -`getCycles` ~94 ms (`MathClocksUnrolledLoop`, serialized into the library). None of these can -simply be skipped. - -**Comments** (`--disable-asm-comments` / `outputNoComment`): output 1.56 → 0.86 MB but time only -−5% (156 → 149 ms), because comments are still built in Python; a minor lever for codegen -(it would also shrink the text written and assembled in `python_kernel_write_assemble`, not -measured). - -### Repeat generation is not the issue - -Within each probed config almost every generated kernel is unique (`dtl`: 9,107 generations, 9,056 -unique, 8 s of 1,312 s duplicated), and no kernel name appeared in more than one probed config. - -## Options - -### A. Run the `CpuThreads: 1` configs in parallel - -`CpuThreads: 1` is set in 71 test yamls; 19 ran in this aggregate (all 13 `subtile_*` gfx950 -configs, `spmm_vw_lg_one{,_sb}` gfx94x/gfx950, `use_sgpr_for_gro{,_sb}`), totalling 189.3 s of -codegen and 92.0 s of `python_kernel_write_assemble`. `ParallelMap2` then takes its serial path -(no "Launching" line). In the subtile configs the setting was present from the moment each file -was added, in three PRs, none of which explains it: - -| Commit | PR | Configs | -| --- | --- | --- | -| `1fb7e7dcc69` (bnemanich, 2026-05-03) | #7017 "Add gfx950 MXFP4 and BF16 Subtile-based kernel implementation" | `subtile_mxfp4`, `subtile_bf16` | -| `42e06b9f3f2` (Prabhjot Sandhu, 2026-06-03) | #7656 "[SubtileImpl] Add support for MXFP8 MT128x128 with DU=256" | `subtile_fp8`, `subtile_mxfp8` | -| `e91ff88833c` (Matthew Emmett, 2026-06-24) | #7781 "TensileLite: Add multi-DU support." | the nine `subtile_mxfp8_*` configs | - -In #7017 it sits next to debug-style settings (`KeepBuildTmp: True`, commented -`PrintSolutionRejectionReason` / `GenerateSourcesAndExit`); the later files appear to have -copied it. (The files moved to `tensilelite/` in `11d0433bd6d`, #12150.) The non-subtile -`CpuThreads: 1` configs were not traced. - -In-run, `subtile_mxfp4` with `CpuThreads` overridden to 25 (same cores, back to back): - -| | Serial | 25 processes | -| --- | ---: | ---: | -| `python_kernel_codegen` | 133.5 s | 22.6 s | -| `python_kernel_write_assemble` | 62.9 s | 11.6 s | -| whole build | 244 s | 63 s | - -All 130 `.co` files and 130 libraries (`TensileLibrary.dat`, decompressed) are byte-identical. -**25 of 155 `Data/00_Final.yaml` files differ in `LdsNumBytes`** (e.g. 82,432 serial vs 69,632 -parallel): the subtile generator sets `kernel["LdsNumBytes"]` during codegen -(`Components/Subtile/SubtileLdsLayout.py:183`). Serially the kernel object *is* the solution, so -the value lands in the YAML; in a worker it is lost, because `passPostKernelInfoToSolution` -(`Run.py:457`) only copies `CUOccupancy`, `PrefetchGlobalRead`, `MathClocksUnrolledLoop` and -`CustomKernel` back. This is not subtile-specific: forcing `custom_mainloop_scheduling` serial -changed one solution's `LoopIters` (4 vs 1; set at `KernelWriter.py:7691`), again with identical -`.co` and libraries. So the solution metadata already depends on `CpuThreads`; enabling -parallelism for subtile should add `LdsNumBytes` (and probably `LoopIters`) to the returned -`KernelCodeGenResult` so both modes agree. - -Aggregate estimate (not measured end to end): applying the measured 5.9× to the 189.3 s gives -~32 s, saving ~157 s of codegen and ~75 s of `write_assemble`. The non-subtile ones have few -kernels per step, so their gain will be smaller. - -### B. Fail fast on a guaranteed VGPR overflow - -`_initKernel` computes the static VGPR allocation and only prints -`warning: total VGPRS (370) overflowed max VGPRS (256).` (`KernelWriter.py:9616`). The VGPR pool -is created with that size (`KernelWriter.py:10270`) and can only grow (`RegisterPool::remove` -marks registers unavailable, `resize` only extends), so the final check -`vgprPool.size() > MaxVgpr` (`KernelWriterAssembly.py:2343`) always rejects the kernel — after the -whole kernel, including the expensive epilogue, has been generated. The rejection surfaces as -`err=-2` and is not printed unless `PrintSolutionRejectionReason` is set. - -Prototype: after `_initKernel`, raise if `states.totalVgprs > regCaps["MaxVgpr"]` and -`forceGenerateKernel` is off. - -| Step (every kernel, 16 processes) | Kernels | Rejected | Accept/reject mismatches | Codegen CPU | -| --- | ---: | ---: | ---: | ---: | -| `f8_gate_r` step 0 | 8 | 6 | 0 | 11.9 → 1.4 s (−88%) | -| `f8_gate_r` step 1 | 8 | 6 | 0 | 13.2 → 1.3 s (−90%) | -| `spmm_i8_sb` step 0 | 80 | 16 | 0 | 29.5 → 14.1 s (−52%) | -| `plr_zero` step 12 | 546 | 14 | 0 | 86.3 → 85.3 s (−1%; its rejections are mostly other reasons) | -| `plr_zero` step 13 | 156 | 92 | 0 | 81.6 → 26.8 s (−67%) | - -Accepted kernels' output size was also unchanged. In-run with the check applied in the workers: -`f8_gate_r` codegen 12.6 → 3.5 s, whole build 28 → 11 s, all 16 artifacts byte-identical; -`plr_zero` worker CPU 789 → 630 s (−20%), codegen 36.6 → 33.6 s (baseline ran on other cores; -wall is noisy), all 68 artifacts byte-identical. Share of codegen CPU spent on rejected kernels in -the probes: `f8_gate_r` 89%, `plr_zero` 29%, `spmm_i8_sb` 24%, `dtl` 8%, `fp16_tn` 1%, -`subtile_mxfp4` 0.4%, `custom_mainloop_scheduling` 0%. Across the aggregate 8,457 of 116,264 -solutions were rejected by the KernelWriter; their CPU share outside the probes was not measured. - -#### All rejection reasons, and when each is known - -Every kernel the probes recorded as rejected (1,242 kernels from `f8_gate_r`, `spmm_i8_sb`, -`plr_zero`, `gfx950/dtl`, `fp16_tn`, `subtile_mxfp4`) was replayed with a hook on -`StateValues.__setattr__` recording the first non-zero `overflowedResources` code, the time it was -set and the setting function (`/tmp/codegen/rejreasons.py`, output `/tmp/codegen/rejreasons.tsv`): - -| Code | Reason | Kernels | Share of rejected-kernel CPU | Known at (median fraction of codegen time) | Set by | Caught by init VGPR check | -| --- | --- | ---: | ---: | ---: | --- | ---: | -| 1 | too many VGPRs | 679 | 83.6% | 0.50 | `checkResources` (`KernelWriterAssembly.py:2343`) | 598 | -| 2 | too many SGPRs | 427 | 15.0% | 0.03 | `overflowListener` in `allocTmpSgpr` (`KernelWriterAssembly.py:789`) | 5 | -| 5 | local read/write LDS scheduling | 132 | 1.3% | 0.21 | `_makeSubIterSchedule` (`KernelWriter.py:2011`) | 0 | -| 8 | not enough LDS (subtile) | 4 | 0.1% | 0.01 | `applyLdsLayout` (`SubtileLdsLayout.py:185`) | 0 | - -No rejection came from any other exception. `overflowedResources` is never reset and is read only at -the end (to build the error and print the reason: `KernelWriter.py:5664`, `:7265`, -`KernelWriterAssembly.py:2368`), so once it is non-zero the kernel is guaranteed to be rejected -(unless `forceGenerateKernel`), yet generation continues: SGPR overflow is known 3% of the way in; -the VGPR flag is set at the end of `kernelBody`, after which `rocIsaPass`, `str()` and freeing the -tree (~50% of the kernel's time) still run. The 81 VGPR rejections the init check misses overflow -through registers allocated during generation. - -**Second check: abort as soon as `overflowedResources` becomes non-zero.** Estimated CPU on rejected -kernels, from the recorded set times: - -| Config | Rejected | Today | Init VGPR check | Abort on set | Both | -| --- | ---: | ---: | ---: | ---: | ---: | -| `f8_gate_r` | 24 | 41.4 s | 0.0 s | 25.5 s | 0.0 s | -| `spmm_i8_sb` | 54 | 48.1 s | 0.0 s | 31.4 s | 0.0 s | -| `plr_zero` | 673 | 191.0 s | 57.1 s | 73.9 s | 2.7 s | -| `gfx950/dtl` | 457 | 90.2 s | 19.2 s | 44.0 s | 8.1 s | -| `fp16_tn` | 30 | 3.4 s | 0.0 s | 2.1 s | 0.0 s | -| `subtile_mxfp4` | 4 | 0.4 s | 0.4 s | 0.0 s | 0.0 s | -| **Total** | **1,242** | **374.5 s** | **76.7 s** | **177.0 s** | **10.8 s (−97%)** | - -Validated with both checks (`ff.py`, `FF_ABORT_ON_SET=1`) on 9 complete steps covering every reason, -4,313 kernels: 0 accept/reject mismatches, accepted kernels' output size unchanged, CPU on rejected -kernels 160.1 → 4.3 s. - -| Step | Kernels | Rejected | Step codegen CPU | -| --- | ---: | --- | ---: | -| `plr_zero` step 0 | 874 | 218 (SGPR) | 104.2 → 74.4 s (−29%) | -| `gfx950/dtl` step 4 | 336 | 136 (VGPR) | 75.4 → 32.3 s (−57%) | -| `plr_zero` step 13 | 156 | 92 (VGPR + SGPR) | 69.5 → 22.1 s (−68%) | -| `gfx950/dtl` step 11 | 2,424 | 64 (LDS scheduling) | 338.5 → 354.7 s (noise; 1.6 s of rejected work) | -| `gfx950/dtl` step 24 | 307 | 43 (LDS scheduling + VGPR) | 14.4 → 11.9 s (−17%) | -| `fp16_tn` step 1 | 38 | 30 | 3.6 → 0.6 s (−85%) | -| `spmm_i8_sb` step 1 | 80 | 24 | 27.2 → 6.4 s (−77%) | -| `f8_gate_r` step 0 | 8 | 6 | 10.4 → 1.0 s (−90%) | -| `subtile_mxfp4` step 0 | 90 | 0 | unchanged | - -For a real implementation: the prototype raised from the `overflowedResources` setter and ignored -`forceGenerateKernel`; the real change should check that flag and raise at the setting sites. The -reason printed under `PrintSolutionRejectionReason` comes from `checkResources` at the end, so the -early exit must print it too; the error code (`-2`) is already the same. Moving the VGPR/SGPR checks -into solution validation, so these kernels never reach codegen, is a possible further step (not -tested). - -### C. Generate all steps of a config together - -Each `(problem type, size group)` step is generated, compiled and post-processed before the next -starts (`BenchmarkProblems.py`, loop around `_benchmarkProblemType`); no step reads another's -results (inference from the loop; build-only skips benchmarking). Simulating one 25-worker pool -per config with the measured per-kernel times (longest-first): - -| Config | Sum of per-step spans (measured) | One pool (simulated) | -| --- | ---: | ---: | -| `custom_mainloop_scheduling` | 36.0 s | 2.1 s | -| `f8_gate_r` | 12.4 s | 3.3 s | -| `spmm_i8_sb` | 17.6 s | 9.1 s | -| `fp16_tn` | 14.4 s | 13.4 s | -| `gfx950/dtl` | 60.3 s | 54.1 s | -| `plr_zero` | 36.2 s | 32.5 s | - -Aggregate bound (estimate): for the 447 ≤25-kernel steps outside the `CpuThreads: 1` configs, CPU ≤ -kernels × step wall, so per config the pooled time is ≤ CPU/25 + longest step: 142.4 s → ≤75.2 s. -The probes suggest the real gain is larger. This needs the step loop split into "generate -solutions for all steps → one codegen pool → per-step assemble/build/write" (or keeping one pool -fed across steps); not prototyped. - -### D. Longest-first dispatch - -Within a step, joblib dispatches in input order. With perfect knowledge of per-kernel cost, -longest-first would save 0–6% on most probed configs and 24% on `spmm_i8_sb` (17.6 → 13.4 s, -which is also the lower bound). Needs a cost predictor (output size is not known before -generation); upper bound only. - -### E. A cheaper generator (larger work) - -Per-kernel shares above: Python tree construction ~65% (epilogue 40–64% of profiled time), -`rocIsaPass` ~21%, freeing ~8%, `str` ~6%. The code itself notes the epilogue size problem -(`KernelWriter.py:7102`: "Kernels with epilog especially with activation is too long (50000~ -lines). Need to refactor global write elements."). The quadratic `hasAnyDependency` scan is a -contained target for the sparse configs. Not prototyped. - -## Removing `CpuThreads: 1` in CI: cores, oversubscription and memory - -### How CI runs the common tests - -| Job | Command | pytest workers | CPUs | -| --- | --- | ---: | --- | -| TheRock `tensilelite-common` (`TEST_CATEGORY=hw-common`; gfx942 on PRs, gfx950/gfx90a postsubmit, gfx120X nightly) | `pytest_runner.py --numprocesses=4` (`test_categories.yaml`: `parallel_workers: 4`) | 4 | Kubernetes pod on a 1-GPU slice of an 8-GPU node (`linux-gfx942-1gpu-ccs-*`, `linux-gfx950-1gpu-ccs-ossci-rocm`); `KUBE_CPU_REQUEST=25` (from a real job log, see `CI_SPLIT_TIMING.md`) | -| Math CI `preliminary` | `tox -e py3 -- tensilelite/Tests -m common` | `TENSILE_NUM_PYTEST_WORKERS`, default 4 | not in this repo (internal Jenkins) | -| `.github/workflows/gfx1250_hipblaslt_tensilelite_test.yml` | `tox ... -n 4` in a container on `blas-rocm-mi455-sandbox` | 4 | not stated | -| FFM/emulation categories (`ffm-quick`, `emu-fast`, `emu-full`) | `pytest_runner.py` | 16 | not stated | - -TheRock's `test_component.yml` turns `KUBE_CPU_REQUEST` into `OMP_NUM_THREADS` / -`OPENBLAS_NUM_THREADS` only. Tensile sizes its pools from `len(os.sched_getaffinity(0))`, which -ignores the Kubernetes request and the cgroup CPU quota, so each pytest worker starts either 25 -processes (if the pod's CPUs are pinned) or one per host core (if not). Which one applies could not -be checked: no CI log was reachable (no GitHub credentials, API rate-limited). Any CI log answers it: -the `Generating assembly kernels: Launching N threads` line gives N per pytest worker. - -### CI is already oversubscribed, regardless of `CpuThreads: 1` - -Of the 540 YAMLs under `Tests/common`, 406 use the default `CpuThreads: -1` (all visible cores), 67 -set 32, 2 set 8 and 65 set 1. Nothing in the test harness divides cores between pytest workers, so -with 4 workers and 25 visible cores up to 100 codegen processes compete for 25 CPUs whenever two -tests generate code at once. Removing `CpuThreads: 1` makes those 65 configs join that existing -pattern; it does not raise the worst case, which the default configs can already reach. - -Cost of oversubscription, measured by replaying the same 1,000 `dtl` kernels on 25 pinned cores -(`/tmp/codegen/oversub.py`): 25 / 50 / 100 worker processes took 7.2 / 8.2 / 7.6 s (worker CPU -174.9 / 197.3 / 186.4 s). Throughput is unchanged within noise on this shared host; CPU rises -7–13%. Memory is the real cost. - -### Per-worker memory - -The pool is joblib's loky backend. Every worker is a fresh interpreter -(`python3 -m joblib.externals.loky.backend.popen_loky_posix`), not a fork of the parent, so each one -imports everything itself (537 modules); only the file-backed shared libraries (~38 MB) are shared. -Loky reuses the same workers across `ParallelMap2` calls in a build, so start-up is paid once per -build, not per step. - -Single fresh process (`/tmp/codegen/memsteps.py`), resident memory: - -| Stage | RSS | -| --- | ---: | -| bare Python | 10 MB | -| + joblib / numpy | 36 MB | -| + rocisa | 64 MB | -| + Tensile modules | 80 MB (38 MB shared libraries, ~42 MB private) | -| after one typical kernel (1.6–2.1 MB of assembly) | +1–2 MB | -| after the 7.0 MB `custom_mainloop_scheduling` kernel | peak 495 MB; still 355 MB after the result is dropped | - -Peak memory scales with kernel size, about 60 MB of instruction tree per MB of assembly, and the -allocator does not return it to the OS, so a worker stays near the peak of its biggest kernel. -(An earlier 245 MB figure came from a harness that loaded the whole pickled step into every worker, -which real workers never do; it is superseded by the numbers below.) - -Real loky workers, sampled once per second during real builds (`/tmp/codegen/memsample.py`): - -| Build | Workers | Peak RSS median / max | Peak PSS median / max | Shared libs | -| --- | ---: | ---: | ---: | ---: | -| `fp16_tn` | 25 | 288 / 315 MB | 251 / 276 MB | 38 MB | -| `custom_mainloop_scheduling` | 29 | 308 / 569 MB | 259 / 531 MB | 38–40 MB | - -One `fp16_tn` worker over the build: 122 → 196 → 252 → 281 → 295 MB. So 4 pytest workers × 25 -loky workers ≈ 100 × ~250 MB ≈ 25 GB, more if workers hit very large kernels; if a pod exposed a -whole 192-core node, 4 × 192 workers would need ~190 GB. That exposure exists today for the -default configs. - -Loky also has memory-leak protection (`joblib/externals/loky/process_executor.py`, -`_MAX_MEMORY_LEAK_SIZE = int(3e8)`): a worker whose memory grew more than 300 MB above its starting -size exits and is replaced. Observed in a `pytest -n 4` run: `f8_gate_r`'s workers exited at -513–596 MB after 10–30 s and were respawned (20 distinct pids for a 12-worker pool), each -replacement briefly overlapping the worker it replaced. - -### Proposal: a total pool budget of 2× the available CPUs - -Size every pool so that all pytest workers together start at most 2× the CPUs available, and a -single process never more than 1×: - -1. CPUs available = min(affinity mask, loky `cpu_count()` (also applies the cgroup quota), - `KUBE_CPU_REQUEST` when set). -2. Parallel workers = `PYTEST_XDIST_WORKER_COUNT` (set by pytest-xdist in every worker), else 1. -3. Pool size = max(1, min(CPUs, 2 × CPUs // workers)). -4. `CpuThreads` stays a cap: `-1` = the budget, N = min(budget, N), `0` = no threading. - -| Setup | Per process | Total | -| --- | ---: | ---: | -| standalone, 96 CPUs | 96 | 96 | -| `-n 4`, 25 CPUs (TheRock / Math CI) | 12 | 48 | -| `-n 4`, pod sees 192 cores, `KUBE_CPU_REQUEST=25` | 12 | 48 | -| `-n 2`, 25 CPUs | 25 | 50 | -| `-n 16`, 128 CPUs (FFM) | 16 | 256 | - -Sketch, replacing the body of `CPUThreadCount()` in `Common/Parallel.py`: - -```python -OVERSUBSCRIPTION_FACTOR = 2 - -def _kubeCpuRequest(): - value = os.environ.get("KUBE_CPU_REQUEST") # may be fractional, e.g. "8.0" - try: - return max(1, int(float(value))) if value else None - except ValueError: - return None - -def availableCPUCount(): - if os.name == "nt": - count = min(os.cpu_count() or 1, 61) - else: - count = len(os.sched_getaffinity(0)) - try: - from joblib.externals.loky import cpu_count as lokyCpuCount # affinity + cgroup quota - count = min(count, lokyCpuCount()) - except Exception: - pass - request = _kubeCpuRequest() - return max(1, min(count, request) if request else count) - -def concurrentWorkerCount(): - try: - return max(1, int(os.environ.get("PYTEST_XDIST_WORKER_COUNT", "1"))) - except ValueError: - return 1 - -def poolSizeBudget(): - cpus = availableCPUCount() - return max(1, min(cpus, (OVERSUBSCRIPTION_FACTOR * cpus) // concurrentWorkerCount())) - -def CPUThreadCount(enable=True): - if not enable: - return 1 - budget = poolSizeBudget() - cpuThreads = globalParameters["CpuThreads"] - return budget if cpuThreads == -1 else min(budget, cpuThreads) -``` - -This was prototyped and validated, then reverted (not part of this change): - -- 15 unit tests (mocked affinity / loky count / environment) covering the table above, cgroup - limits, unparsable environment values, `CpuThreads` caps and `0`, and that per-process × workers - ≤ 2 × CPUs for six CPU/worker combinations; together with the existing - `Tests/unit/characterization/Parallel/test_parallel_char.py`, 29 passed. -- Outside pytest with `PYTEST_XDIST_WORKER_COUNT=4` on 25 cores: `ParallelMap2` printed - `Launching 12 threads` and used exactly 12 worker pids. -- Real `pytest -n 4` build of `fp16_tn`, `cgemm`, `zgemm`, `f8_gate_r` on 25 pinned cores, process - tree sampled every 0.2 s (`/tmp/codegen/poolturnover.py`): 12 loky workers per pytest worker (13 - for a moment during a loky memory-leak restart, see above), at most 50 in total; all 4 passed. - (Short-lived "loky" children of loky workers are forks for `amdclang++` before `exec`, which keep - the parent's command line; they are not pools.) - -Open decisions for this proposal: `KUBE_CPU_REQUEST` is a Kubernetes *request*, not a limit, but -TheRock already treats it as the pod's budget for OpenMP threads, and without it a pod that sees the -whole node would size pools from the whole node. Only pytest-xdist parallelism is detected; other -ways of running several Tensile processes on one machine are not. - -### `CPUThreadCount()` / `ParallelMap2` issues - -- `CPUThreadCount()` sizes pools from `len(os.sched_getaffinity(0))` only. It ignores cgroup CPU - quotas, `KUBE_CPU_REQUEST`, and how many pytest-xdist workers run concurrently, so under - `pytest -n N` each worker starts a full-size pool (the oversubscription above). -- `ParallelMap2` (`Common/Parallel.py`) computes - `threadCount = procs if procs else CPUThreadCount(enable)` and on the next line overwrites it with - `threadCount = CPUThreadCount(enable)`, so an explicit `procs` is ignored. The only caller passing - it is `TensileLogic/Run.py:321` (`procs=jobs`), whose job count therefore has no effect. -- TensileCreateLibrary's `_cpuCount()` (`TensileCreateLibrary/Run.py:1226`) duplicates the - affinity-only count to split jobs between architecture groups, with the same blind spots. - -## Not measured / open items - -- End-to-end aggregate with A/B/C applied; the aggregate numbers above are estimates from the probes. -- Whether `LdsNumBytes`/`LoopIters` in `Data/*.yaml` are consumed downstream (libraries and code - objects are identical either way); which value is intended. -- CPU share of rejected kernels across all configs, and early-exit for the other overflow reasons. -- Whether any `CpuThreads: 1` config needs it for a reason not visible in the commit. -- How many cores Tensile sees inside CI pods (pinned 25 vs the whole node); read the - `Launching N threads` line from any `tensilelite-common` / Math CI log. -- Step independence for option C was inferred from the loop, not proven for every config feature. -- Native (C++) hotspots inside `rocIsaPass`/`str`/free: `_rocisa.so` is stripped and `perf` is - unavailable; only whole-call times were measured. -- Probe A/B timings ran on a shared host; comparisons are back to back on the same 25 cores except - the `plr_zero` fail-fast run. diff --git a/projects/hipblaslt/tensilelite/timing_runs/run_cms.sh b/projects/hipblaslt/tensilelite/timing_runs/run_cms.sh deleted file mode 100644 index 2191e9656a81..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/run_cms.sh +++ /dev/null @@ -1,21 +0,0 @@ -# Build-only of the CMS (UseCustomMainLoopSchedule) configs, targeting gfx950, cold caches. -TL=/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite -D=$TL/timing_runs/cms -cd $TL || exit 1 -mkdir -p $D -export OMP_NUM_THREADS=25 OPENBLAS_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=0 PYTHONUNBUFFERED=1 -TO="--cxx-compiler,/usr/bin/amdclang++,--global-parameters,TimingInstrumentation=True" -COMMON=(-s -v -n 0 --durations=0 --timeout=14400 "--global-parameters=LibraryFormat='msgpack'" - "--tensile-options=$TO" -p no:warnings) -: > $D/times.txt -for cfg in gemm/gfx950/custom_mainloop_scheduling gemm/gfx950/custom_mainloop_scheduling_tf32 ductile/gfx950/f8bf16_tn_cms; do - name=$(basename $cfg) - rm -rf /home/alvasile/.tensile /home/alvasile/.tensilelite /tmp/tl_cms_art_$name - BID="tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/$cfg.yaml]" - s=$(date +%s) - /usr/bin/time -v -o $D/${name}_time.txt taskset -c 0-24 pytest "$BID" --build-only --gpu-targets gfx950 \ - --artifact-dir /tmp/tl_cms_art_$name "${COMMON[@]}" > $D/${name}_build.log 2>&1 - echo "$name rc=$? $(( $(date +%s)-s ))s" >> $D/times.txt - python scripts/analyze_timing.py $D/${name}_build.log > $D/${name}_build_analysis.txt 2>&1 -done -echo DONE >> $D/times.txt diff --git a/projects/hipblaslt/tensilelite/timing_runs/run_cold_warm.sh b/projects/hipblaslt/tensilelite/timing_runs/run_cold_warm.sh deleted file mode 100644 index 44ed391b0d68..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/run_cold_warm.sh +++ /dev/null @@ -1,45 +0,0 @@ -# For each config: cold build (caches deleted), warm build (caches kept), run (caches deleted -# first, as on a fresh GPU node). Raw logs + analyze_timing.py output per stage. -# Usage: bash run_cold_warm.sh gemm/dtv gemm/cgemm -TL=/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite -CL=/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client -cd $TL || exit 1 -export OMP_NUM_THREADS=25 OPENBLAS_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=0 PYTHONUNBUFFERED=1 -TO="--cxx-compiler,/usr/bin/amdclang++,--global-parameters,TimingInstrumentation=True" -COMMON=(-s -v -n 0 --durations=0 --timeout=5400 -m "common and not skip-gfx94X" - "--global-parameters=LibraryFormat='msgpack'" "--tensile-options=$TO" -p no:warnings) -clear_caches() { rm -rf /home/alvasile/.tensile /home/alvasile/.tensilelite; } - -for cfg in "$@"; do - name=$(basename $cfg) - D=$TL/timing_runs/$name - A=/tmp/tl_timing_art_$name - mkdir -p $D - rm -rf $A - BID="tensilelite/Tests/common/test_config_build.py::test_config_build[tensilelite/Tests/common/$cfg.yaml]" - RID="tensilelite/Tests/common/test_config_run.py::test_config_run[tensilelite/Tests/common/$cfg.yaml]" - : > $D/times.txt - - clear_caches - s=$(date +%s) - taskset -c 0-24 pytest "$BID" --build-only --gpu-targets gfx942 --artifact-dir $A/cold \ - "${COMMON[@]}" > $D/${name}_1_build_cold.log 2>&1 - echo "build_cold rc=$? $(( $(date +%s)-s ))s" >> $D/times.txt - - s=$(date +%s) # caches kept from the cold build - taskset -c 0-24 pytest "$BID" --build-only --gpu-targets gfx942 --artifact-dir $A/warm \ - "${COMMON[@]}" > $D/${name}_2_build_warm.log 2>&1 - echo "build_warm rc=$? $(( $(date +%s)-s ))s" >> $D/times.txt - - clear_caches - s=$(date +%s) - taskset -c 0-24 pytest "$RID" --use-cache --artifact-dir $A/cold --prebuilt-client=$CL \ - "${COMMON[@]}" > $D/${name}_3_run.log 2>&1 - echo "run rc=$? $(( $(date +%s)-s ))s" >> $D/times.txt - - for st in 1_build_cold 2_build_warm 3_run; do - python scripts/analyze_timing.py $D/${name}_$st.log > $D/${name}_${st}_analysis.txt 2>&1 - done - echo DONE >> $D/times.txt -done -echo ALLDONE > $TL/timing_runs/cold_warm_done.txt diff --git a/projects/hipblaslt/tensilelite/timing_runs/run_fp16_gradient_bias_analysis.txt b/projects/hipblaslt/tensilelite/timing_runs/run_fp16_gradient_bias_analysis.txt deleted file mode 100644 index 19344e21b825..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/run_fp16_gradient_bias_analysis.txt +++ /dev/null @@ -1,175 +0,0 @@ -======================================================================================================================== -TIMING ANALYSIS SUMMARY -======================================================================================================================== - -TIMING BY PHASE ------------------------------------------------------------------------------------------------------------------------- - Category Count Total (ms) Mean (ms) % of Parent % of Wall ------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 1 19,225.06 19,225.06 100.0% - -------------------------------------------------------------------------------------------------------------------- - python_client_execution 6 19,192.08 3,198.68 99.8% 99.8% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Data Preparation 11,010.50 57.4% 57.3% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - pre_problem 21 11,004.54 524.03 99.9% 57.2% - cpu_reference_gemm 21 10,976.40 522.69 99.7% 57.1% - solve_cpu_slow 21 10,976.33 522.68 100.0% 57.1% - (untracked) 0.07 0.0% 0.0% - cpu_data_init 21 27.66 1.32 0.3% 0.1% - (untracked) 0.48 0.0% 0.0% - gpu_input_reset 562 3.86 0.01 0.0% 0.0% - gpu_input_preparation 21 1.61 0.08 0.0% 0.0% - rotating_buffer_preparation 21 0.48 0.02 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - C++ Client Setup 6,327.17 33.0% 32.9% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - listener_setup 6 3,195.49 532.58 50.5% 16.6% - hip_initialization 6 2,494.86 415.81 39.4% 13.0% - data_init_setup 6 525.33 87.56 8.3% 2.7% - code_object_loading 6 57.12 9.52 0.9% 0.3% - library_loading 6 30.78 5.13 0.5% 0.2% - lazy_loading_init 6 22.71 3.79 0.4% 0.1% - reporter_setup 6 0.76 0.13 0.0% 0.0% - solution_iterator_setup 6 0.11 0.02 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked C++ overhead) 1,105.31 5.8% 5.7% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Reference Computation 561.45 2.9% 2.9% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - validate_warmups 562 561.45 1.00 100.0% 2.9% - (untracked) 314.55 56.0% 1.6% - validate_gpu_readback 1,124 204.75 0.18 36.5% 1.1% - validate_element_comparison 1,124 41.78 0.04 7.4% 0.2% - validate_gpu_sync 562 0.38 0.00 0.1% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Kernel Execution 158.20 0.8% 0.8% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - warmup_runs 1,124 117.87 0.10 74.5% 0.6% - post_solution 726 21.02 0.03 13.3% 0.1% - post_solution_log 726 15.78 0.02 75.1% 0.1% - post_solution_reporting 726 3.81 0.01 18.1% 0.0% - post_solution_perf_calc 726 0.94 0.00 4.5% 0.0% - post_solution_validation 726 0.33 0.00 1.6% 0.0% - (untracked) 0.09 0.4% 0.0% - post_solution_result_file 726 0.03 0.00 0.1% 0.0% - post_solution_sol_advance 726 0.02 0.00 0.1% 0.0% - post_solution_perf_reset 726 0.02 0.00 0.1% 0.0% - pre_solution 726 10.96 0.02 6.9% 0.1% - kernel_solving 562 5.23 0.01 3.3% 0.0% - post_problem 21 1.72 0.08 1.1% 0.0% - finalize_report 6 0.62 0.10 0.4% 0.0% - solution_selection 726 0.60 0.00 0.4% 0.0% - benchmark_runs 562 0.17 0.00 0.1% 0.0% - (untracked) 0.17 100.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Timing Overhead 29.44 0.2% 0.2% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - flush_timing_buffer 6 28.70 4.78 97.5% 0.1% - timing_overhead 6 0.74 0.12 2.5% 0.0% - -------------------------------------------------------------------------------------------------------------------- - (untracked) 20.05 0.1% 0.1% - -------------------------------------------------------------------------------------------------------------------- - python_cache_check 6 11.73 1.96 0.1% 0.1% - -------------------------------------------------------------------------------------------------------------------- - python_write_solutions 6 1.20 0.20 0.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_prepare 6 0.84 0.14 69.9% 0.0% - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - python_wsol_prepare_cache 6 0.81 0.14 96.7% 0.0% - (untracked) 0.03 3.3% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - python_wsol_header 6 0.29 0.05 24.0% 0.0% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (untracked) 0.07 6.1% 0.0% - ------------------------------------------------------------------------------------------------------------------------- - ESTIMATED WALL CLOCK TIME: 19.23s - -HIERARCHY VALIDATION: OK (all child phases fit within their parents) - -TIME BREAKDOWN (visual, % of wall clock) ------------------------------------------------------------------------------------------------------------------------- - python_benchmark_problems 100.0% ################################################# - python_client_execution 99.8% ################################################# - Data Preparation 57.3% ############################ - pre_problem 57.2% ############################ - cpu_reference_gemm 57.1% ############################ - solve_cpu_slow 57.1% ############################ - (untracked) 0.0% - cpu_data_init 0.1% - (untracked) 0.0% - gpu_input_reset 0.0% - gpu_input_preparation 0.0% - rotating_buffer_preparation 0.0% - C++ Client Setup 32.9% ################ - listener_setup 16.6% ######## - hip_initialization 13.0% ###### - data_init_setup 2.7% # - code_object_loading 0.3% - library_loading 0.2% - lazy_loading_init 0.1% - reporter_setup 0.0% - solution_iterator_setup 0.0% - (untracked C++ overhead) 5.7% ## - Reference Computation 2.9% # - validate_warmups 2.9% # - (untracked) 1.6% - validate_gpu_readback 1.1% - validate_element_comparison 0.2% - validate_gpu_sync 0.0% - Kernel Execution 0.8% - warmup_runs 0.6% - post_solution 0.1% - post_solution_log 0.1% - post_solution_reporting 0.0% - post_solution_perf_calc 0.0% - post_solution_validation 0.0% - (untracked) 0.0% - post_solution_result_file 0.0% - post_solution_sol_advance 0.0% - post_solution_perf_reset 0.0% - pre_solution 0.1% - kernel_solving 0.0% - post_problem 0.0% - finalize_report 0.0% - solution_selection 0.0% - benchmark_runs 0.0% - (untracked) 0.0% - Timing Overhead 0.2% - flush_timing_buffer 0.1% - timing_overhead 0.0% - (untracked) 0.1% - python_cache_check 0.1% - python_write_solutions 0.0% - python_wsol_prepare 0.0% - python_wsol_prepare_cache 0.0% - (untracked) 0.0% - python_wsol_header 0.0% - (untracked) 0.0% - -KEY INSIGHTS (% of wall clock) ------------------------------------------------------------------------------------------------------------------------- - Python phases (code gen, compilation): 0.00s (0.0%) - Python I/O overhead (cache, YAML): 0.01s (0.1%) - C++ client setup (library loading): 6.33s (32.9%) - Data preparation (CPU init, GPU prep): 11.01s (57.3%) - Reference computation (CPU GEMM): 0.56s (2.9%) - Kernel execution (warmup, benchmark): 0.16s (0.8%) - - -TOP 10 SLOWEST PROBLEMS (by CPU reference time) ------------------------------------------------------------------------------------------------------------------------- - M N K Batch TypeA TypeD CPU Ref (ms) GPU (ms) - ---------------------------------------------------------------------------------------------------------------------- - 512 256 511 1 Half Half 927.99 0.00 - 512 256 511 1 Half Half 742.78 0.00 - 512 256 1023 1 Half Half 738.13 0.00 - 511 256 1024 1 Half Half 686.00 0.00 - 511 256 1024 1 Half Half 663.89 0.00 - 512 256 511 1 Half Half 659.53 0.00 - 511 256 1024 1 Half Half 654.96 0.00 - 512 256 1023 1 Half Half 632.12 0.00 - 512 256 1023 1 Half Half 612.11 0.00 - 512 256 1023 1 Half Half 571.57 0.00 - diff --git a/projects/hipblaslt/tensilelite/timing_runs/run_timing.sh b/projects/hipblaslt/tensilelite/timing_runs/run_timing.sh deleted file mode 100644 index 72abc6d9b824..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/run_timing.sh +++ /dev/null @@ -1,26 +0,0 @@ -# Timing-instrumented single-config run: build phase then run phase, CI-like conditions. -# Usage: bash run_timing.sh [config-name-substring] -T=${1:-fp16_gradient_bias} -TL=/home/alvasile/rocm-libraries/projects/hipblaslt/tensilelite -D=$TL/timing_runs -CL=/tmp/tl_client_msgpack/tensilelite/client/tensilelite-client -A=/tmp/tl_timing_art -cd $TL || exit 1 -export OMP_NUM_THREADS=25 OPENBLAS_NUM_THREADS=25 ROCR_VISIBLE_DEVICES=0 PYTHONUNBUFFERED=1 -TO="--cxx-compiler,/usr/bin/amdclang++,--global-parameters,TimingInstrumentation=True" -COMMON=(-s -v -n 0 --timeout=5400 -m "common and not skip-gfx94X" - "--global-parameters=LibraryFormat='msgpack'" "--tensile-options=$TO" -p no:warnings -k $T) -rm -rf /home/alvasile/.tensile /home/alvasile/.tensilelite $A -: > $D/times.txt -s=$(date +%s) -taskset -c 0-24 pytest tensilelite/Tests/common/test_config_build.py --build-only --gpu-targets gfx942 \ - --artifact-dir $A "${COMMON[@]}" > $D/build_$T.log 2>&1 -echo "build rc=$? $(( $(date +%s)-s ))s" >> $D/times.txt -rm -rf /home/alvasile/.tensile /home/alvasile/.tensilelite -s=$(date +%s) -taskset -c 0-24 pytest tensilelite/Tests/common/test_config_run.py --use-cache --artifact-dir $A \ - --prebuilt-client=$CL "${COMMON[@]}" > $D/run_$T.log 2>&1 -echo "run rc=$? $(( $(date +%s)-s ))s" >> $D/times.txt -python scripts/analyze_timing.py $D/build_$T.log > $D/build_${T}_analysis.txt 2>&1 -python scripts/analyze_timing.py $D/run_$T.log > $D/run_${T}_analysis.txt 2>&1 -echo DONE >> $D/times.txt diff --git a/projects/hipblaslt/tensilelite/timing_runs/tcl_helper_name_mismatch_investigation.md b/projects/hipblaslt/tensilelite/timing_runs/tcl_helper_name_mismatch_investigation.md deleted file mode 100644 index 70dc22326365..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/tcl_helper_name_mismatch_investigation.md +++ /dev/null @@ -1,335 +0,0 @@ -# TensileCreateLibrary helper name/object mismatch investigation (all shipped logic) - -Findings only, no code change ([DECISIONS.md](DECISIONS.md), topic 8, "Investigation (findings -only, no fix)", plan step P5). The question comes from -[python_kernel_bench_setup_investigation.md](python_kernel_bench_setup_investigation.md), -"Not recommended: reuse `TensileCreateLibrary.generateKernelHelperObjects`": the function dedupes -helper kernels on names predicted by the `*KernelNames` functions, and those names do not match the -objects it then builds. Can that drop a helper, duplicate one, or pair a name with the wrong object -for any real configuration? - -Code state: commit `2083e6254bf` (the Python under `tensilelite/` is identical to `55e29436e9a`). -Logic: every `.yaml` under `library/src/amd_detail/rocblaslt/src/Tensile/Logic/asm_full` (3,313 -files, all architectures). Every count below comes from running the real TensileCreateLibrary code -on these files; code reading was used to explain the counts. Inferences are marked as such. - -## Summary - -- **One real configuration drops helpers: the `gfx1250-strict` build.** It is missing - `Cijk_SB_PostGSU1_VW1`, `_VW2` and `_VW4` (the `SingleBuffer` output-conversion kernel for - `F8BS`/`BBS`-style problems: float workspace, bf16 output, no bias). The one `SingleBuffer` - solution in `gfx1250-strict/gfx1250-strict/Equality/gfx1250-strict_Cijk_Alik_Bljk_F8BS_BH_UserArgs.yaml` - calls for them (the benchmark path would emit them); the library's helper list does not declare - them (checked on the real output, see "Confirming the drop"). `gfx1250-strict` is always built as - its own TensileCreateLibrary run (it is a stepping of `gfx1250`), so this applies to the default - `all` build too. Cause: the predicted name uses the solution's `GlobalSplitU`, so a - `MultipleBuffer` solution with `GlobalSplitU: 0` and the `SingleBuffer` solution with - `GlobalSplitU: 0` predict the same `..._PostGSU0_VW*` names while needing different objects - (`PostGSU16` and `PostGSU1`). The `MultipleBuffer` one comes first, so the `SingleBuffer` objects - are never built. - **At run time the missing kernels cannot be requested** (from the host code, not run): this - solution is persistent data-parallel (`StreamKForceDPOnly: 1`), so it has no accumulation buffer, - GSU is fixed at 0 and the output-conversion kernel is never launched for it (see "Runtime reach"). - The shipped library is not broken today; the defect is the order-dependent dedupe, which would - break a library as soon as a `SingleBuffer` solution that does use the conversion kernel sits - behind a same-name `MultipleBuffer` one. -- **No other configuration drops or adds a helper.** All 18 single-arch builds except - `gfx1250-strict`, `gfx942 --experimental`, `gfx950 --enable-gemm-a2a-fusion`, the default - multi-arch `all` build (16 archs in one run) and every logic file built on its own (3,305 files - that reach the helper step): the helper set equals the set the objects call for. -- **No duplicates reach `Kernels.cpp`, but most helper objects are built more than once.** The final - `dict.fromkeys` dedupes on kernel name, so the output never repeats a name. Before it, 1,807 - objects are built for 265 kept on `gfx942` (2,189 for 427 in the `all` build), because the proxy - names differ between solutions that need the same objects. -- **Positional mispairing happens but never selects the wrong set.** `itertools.compress` pairs - 3 predicted names with 2 objects (`ComplexFloat`) or 1 object (`ComplexDouble`) in 34 calls on - `gfx942`, 4 on `gfx950`, 2 each on `gfx1250`/`gfx1250-strict` (692 summed over per-file builds). - In every one of them the mask is all `True`, so all objects are kept whatever the pairing. A - partially-visited mask together with a length mismatch, the case that would keep the wrong object, - occurs 0 times. -- **`MultipleBufferSingleKernel` with `AdaptiveGemmGSUA` is not reachable today**: no shipped - solution has `AdaptiveGemmGSUA != 0` (0 of the 682,765 solutions of the `all` build, the 2,868 of - `gfx1250-strict`, and the gfx1102 / experimental / GEMM+A2A captures). -- **Related, not caused by the name prediction: two different BetaOnly kernels share one name.** - `Cijk_I_GA` (gfx950: `I8IS` with float compute and `I8II` with int32 compute) and, in the `all` - build, also `Cijk_I8_GA` (gfx950 `I8I8S` float compute versus gfx1170 int32 compute). The name - omits the compute type, so the name dedupe keeps whichever comes first and the other problem type - is left with a beta-only kernel of the wrong `D`/`beta` types. Which one is kept depends on the - build: the gfx950-only build keeps the float version, the `all` build keeps the int32 one (from - gfx1101). **Latent at run time**: every solution that produces these names is `MultipleBuffer` or - `MultipleBufferSingleKernel` with `AdaptiveGemmGSUA: 0`, and the host never launches a beta-only - kernel for those, so the wrong kernel is not reached today. It would be if a `SingleBuffer` (or - adaptive) solution of one of these types were added. The benchmark path dedupes by name the same - way, but a benchmark step has one `ProblemType`, so it cannot hit this. - -## Method - -Scripts are throwaway, in `/tmp/tl_plan/P5/` (not in the repo). - -1. **Capture** (`capture.py [TCL flags]`). Runs the real - `TensileCreateLibrary.Run.run()` with the same arguments as the CMake build - (`--architecture=`, `--cxx-compiler=/opt/rocm/llvm/bin/clang++`, lazy loading on, - experimental off unless asked), so logic-file selection, parsing, merging and solution order are - the real ones. Two populations appear below. TCL's **unique solutions** are the solution list - after the identity dedupe at `Run.py:1203-1204` ("Number of unique solutions" in the log). - `run()` then reduces it to the **kernel list** with `generateKernelObjectsFromSolutions` - (dedupe on `getKeyNoInternalArgs`, `Run.py:884-895`) and passes that to - `generateKernelHelperObjects` (`Run.py:1515-1516`). Two wrappers are installed by monkeypatching - (no repo edit): - - `generateKernelObjectsFromSolutions`: records the unique-solution list (before kernel dedupe), - for the per-file analysis and a second replay check. - - `generateKernelHelperObjects`: calls the real function and records its output (kernel name + - SHA-1 of header and source of every object, in order). Then, for every kernel and every helper - type, records the raw predicted names (`kernelObjectNameCallables()`) and the names and source - hashes of the objects `initHelperKernelObjects(kernel, type, ...)` returns. Then it stops the - run (no code objects are built). - - Per-kernel data is memoized on the fields the helper code reads (canonical `ProblemType` state, - `GlobalSplitU`, `GlobalSplitUAlgorithm`, `isStreamK`, `StreamKAtomic`, `_GlobalAccumulation`, - `AdaptiveGemmGSUA`, `ActivationFused`, `WavefrontSize`, `ISA`, `GlobalSplitUPGR`). The memo was - checked against computing every kernel without it on gfx1103 (1,019 kernels, 0 differences), and - every replay below reproduces the real output exactly. -2. **Replay** (`analyze.py`, `report.py`). Re-runs the `generateKernelHelperObjects` loop on the - recorded strings (same callable order, same `"Enum"` sort with `reverse=True`, same `visited` set, - same `itertools.compress`, same `dict.fromkeys` and final sort) and checks it against the real - output: **identical name, source hash and order for every capture below**, both on the kernel list - and on the unique-solution list. All per-configuration statistics (required, kept, built, length - mismatch, partial mask, kernel indices) are computed on the kernel list, which is what the real - function sees. The replay then compares with the **required set**: the union, by - name, of the objects `initHelperKernelObjects` returns for every kernel, i.e. what the benchmark - path (`writeBenchmarkFiles`, which builds every object and dedupes by name) would emit for the - same solutions. It counts missing and extra names, names whose objects have more than one source, - objects built before dedupe, calls where the name and object lists differ in length, calls with a - partially-visited mask, and selected positions whose object does not match the predicted name. - "Needed but skipped" counts object names a solution needed that were not built at that point - (whether or not a later solution built them). -3. **Per logic file**: the same replay on each file's unique solutions in their real order, which is what - a TensileCreateLibrary run with `--logic-filter` naming that one file does (kernel dedupe within - a file never changes the result: a repeated kernel has the same names and they are all visited). -4. **Mechanism statistics** (`stats.py`), and the drill-down scripts `dig*.py` for the examples. -5. **Confirming the drop** (`check_symbols.py`): runs the real `gfx1250-strict` build up to - `generateKernelHelperObjects` and greps the header text of the real helper list (what - `writeHelpers` writes to `Kernels.h`/`Kernels.cpp`) for `__global__` declarations. - -Configurations captured: one run per architecture directory (gfx908, gfx90a, gfx942, gfx950, -gfx1100, gfx1101, gfx1102, gfx1103, gfx1150, gfx1151, gfx1152, gfx1153, gfx1170, gfx1171, -gfx1200, gfx1201, gfx1250, gfx1250-strict), `gfx942 --experimental` (adds the 16 -`aquavanjaram/gfx942/Experimental` files), `gfx950 --enable-gemm-a2a-fusion` (adds the one -GEMM+A2A file filtered by default), and the default `all` group: the base architectures from -`cmake/tensilelite_supported_architectures.cmake` in one run, minus `gfx1250-strict`, which -TensileCreateLibrary always builds as a separate run because it shares `gfx1250`'s ISA -(`isaCollisionFreeGroups`). Together the runs select all 3,313 logic files, and 3,305 of them reach -the helper step with at least one solution: 3,288 in the default builds, plus the 16 experimental -files and the GEMM+A2A file. The other 8 are the one-solution files -`*_Cijk_Alik_Bljk_B8BS_BH_BiasS_HAS_SAV_UserArgs.yaml` and -`*_Cijk_Alik_Bljk_B8F8BS_BH_BiasS_HAS_SAV_UserArgs.yaml` of gfx1170, gfx1171, gfx1200 and gfx1201: -their solution is parsed, and then removed by the identity dedupe at `Run.py:1203-1204` because an -equal solution comes from another file of the same arch (`count_yaml.py`). - -**Limitation, gfx1170/gfx1171.** The installed ROCm 7.2 compiler does not accept these targets, so -`makeIsaInfoMap` reports `SupportedISA` false and solution construction asserts. For these two (and -for the `all` run, which includes them) `capture_capsub.py` substitutes gfx1200's probed -capabilities. The helper inputs come from the logic files. Every solution entry in the 154 logic -files of each arch is parsed into the library (raw YAML entries 33,972 / 38,153 = "Number of -solutions parsed" in the log; `count_yaml.py`); the identity dedupe then leaves 33,748 / 37,925 -unique solutions, and 152 files per arch reach the helper step, the same two-file pattern as gfx1200 -and gfx1201 with the real compiler (see above). Fields the generator derives from the capabilities -(and the `Valid` flag, which is not checked before the helper step) could still differ with a -compiler that supports these targets. - -## Results - -### Per configuration - -"Solutions" is TCL's unique solutions (before kernel dedupe); "Kernels" is the list -`generateKernelHelperObjects` receives, and every column to its right is computed on it. -"Required" is the number of distinct helper names the kernels call for; "Kept" is what -`generateKernelHelperObjects` returns. "Built" is objects built before the final name dedupe. -"Len mismatch" and "Partial mask" count `itertools.compress` calls (calls where at least one name -was new). - -| Configuration | Logic files | Solutions | Kernels | Required | Kept | Missing | Built | Len mismatch | Partial mask | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| gfx908 | 55 | 58 | 58 | 69 | 69 | 0 | 84 | 0 | 1 | -| gfx90a | 108 | 24,597 | 9,670 | 88 | 88 | 0 | 236 | 0 | 3 | -| gfx942 | 844 | 351,748 | 232,710 | 265 | 265 | 0 | 1,807 | 34 | 9 | -| gfx942 `--experimental` | 860 | 355,260 | 236,222 | 265 | 265 | 0 | 1,807 | 34 | 9 | -| gfx950 | 720 | 84,262 | 84,159 | 186 | 186 | 0 (1 name, 2 sources) | 277 | 4 | 5 | -| gfx950 `--enable-gemm-a2a-fusion` | 721 | 84,263 | 84,160 | 186 | 186 | 0 (1 name, 2 sources) | 277 | 4 | 5 | -| gfx1100 | 46 | 1,836 | 1,722 | 57 | 57 | 0 | 96 | 0 | 4 | -| gfx1101 | 60 | 1,554 | 1,358 | 66 | 66 | 0 | 117 | 0 | 4 | -| gfx1102 | 46 | 1,223 | 1,019 | 56 | 56 | 0 | 80 | 0 | 4 | -| gfx1103 | 46 | 1,223 | 1,019 | 56 | 56 | 0 | 80 | 0 | 4 | -| gfx1150 | 47 | 1,472 | 1,290 | 59 | 59 | 0 | 83 | 0 | 4 | -| gfx1151 | 50 | 2,968 | 2,537 | 59 | 59 | 0 | 86 | 0 | 4 | -| gfx1152 | 46 | 1,223 | 1,019 | 56 | 56 | 0 | 80 | 0 | 4 | -| gfx1153 | 47 | 1,269 | 1,061 | 56 | 56 | 0 | 86 | 0 | 4 | -| gfx1170 (caps substituted) | 152 | 33,748 | 32,373 | 114 | 114 | 0 | 243 | 0 | 13 | -| gfx1171 (caps substituted) | 152 | 37,925 | 36,504 | 114 | 114 | 0 | 195 | 0 | 12 | -| gfx1200 | 155 | 84,019 | 77,816 | 108 | 108 | 0 | 192 | 0 | 10 | -| gfx1201 | 152 | 51,930 | 48,098 | 108 | 108 | 0 | 207 | 0 | 9 | -| gfx1250 | 284 | 2,933 | 2,933 | 173 | 173 | 0 | 203 | 2 | 4 | -| **gfx1250-strict** | 278 | 2,868 | 2,868 | 171 | **168** | **3** | 195 | 2 | 4 | -| `all` (16 base archs, one run) | 2,964 | 682,765 | 534,327 | 427 | 427 | 0 (2 names, 2 sources each) | 2,189 | 34 | 27 | - -Logic files are those with at least one solution reaching the helper step. In every row the replay -equals the real output, no extra name is produced, and every kept object has the source the -required set calls for (where a name has one source). - -**Per logic file** (each file built on its own: 3,288 default files, the 16 experimental ones and -the GEMM+A2A one, 3,305 in total): 0 files with a missing, extra or wrong-source helper; 0 names -with two sources; 0 calls with both a length mismatch and a partial mask; 0 "needed but skipped" -names. Summed over files: 9,036 redundant object builds, 692 length-mismatch calls, 659 -partial-mask calls. - -### Confirming the drop - -On the real `gfx1250-strict` helper list (168 objects, 1,287 declared `__global__` kernels): -`Cijk_SB_PostGSU1_VW1`, `Cijk_SB_PostGSU1_VW2`, `Cijk_SB_PostGSU1_VW4` and `Cijk_SB_GG_PostGSU1_VW1` -are absent; `Cijk_SB_PostGSU16_VW1` and `Cijk_SB_PostGSU2_VW1` are declared. No other object emits -a `PostGSU1` kernel for this name: `PostGSU16` objects emit GSU 16, 8, 4, 2 only -(`KernelWriterConversion.__init__`, `gsuKernels`, `KernelWriterConversion.py:73`). - -The sequence in the real kernel-list order ("Kernel index" is the position in the list -`generateKernelHelperObjects` receives): - -| Kernel index | Logic file (under `gfx1250-strict/`) | `GlobalSplitU` | Algorithm | Predicted names | Objects | -| ---: | --- | ---: | --- | --- | --- | -| 1426 | `gfx1250-strict/Equality/gfx1250-strict_Cijk_Alik_Bljk_BBS_BH_UserArgs.yaml` | 0 | MultipleBuffer | `Cijk_SB_PostGSU0_VW{1,2,4}` (new) | `Cijk_SB_PostGSU16_VW{4,2,1}` (built) | -| 1811–1823, 2486 | five MX `F4BS`/`F8BS` files | 0 | MultipleBuffer | same, visited | not built (already present) | -| 2492 | `gfx1250-strict/Equality/gfx1250-strict_Cijk_Alik_Bljk_F8BS_BH_UserArgs.yaml` | 0 | **SingleBuffer** | same, visited | `Cijk_SB_PostGSU1_VW{4,2,1}`: **never built** | - -The `Alik_Bljk_F8BS_BH_UserArgs` logic file exists only under `gfx1250-strict/` (three copies: -`gfx1250-strict`, `_96cu`, `_192cu`), and only the `gfx1250-strict/gfx1250-strict` copy has the -`SingleBuffer` solution; no gfx1250 (non-strict) logic file has one, so the gfx1250 build has -nothing that calls for `PostGSU1`. Built on its own, the `F8BS_BH_UserArgs` file is fine (its -`MultipleBuffer` solutions have `GlobalSplitU` 1 and -1, which predict different names). - -#### Runtime reach (from the code, not run: no gfx1250 hardware here) - -The `SingleBuffer` solution is solution 1 of that file: `GlobalSplitU: 0`, `StreamK: 3`, -`StreamKForceDPOnly: 1` (the only `StreamKForceDPOnly: 1` in the gfx1250 and gfx1250-strict -logic). The generator makes it persistent data-parallel: `Solution.py:2325-2328` sets -`GlobalSplitU = 0`, `SupportUserGSU = False` and `GlobalSplitUAlgorithm = "SingleBuffer"` -(non-StreamK), and the `_GlobalAccumulation` assignment (`Solution.py:2170-2181`) leaves it `None` -for persistent data-parallel solutions, so the library gets `globalAccumulation = 0` -(`Contractions.py:811`; the capture confirms `_GlobalAccumulation` `None` for this solution). On the -host: - -- GSU is 0: `calculateAutoGSU` returns `sizeMapping.globalSplitU` unless it is -1 - (`ContractionSolution.cpp:2170`), and a user GSU is rejected because `SupportUserGSU` is off - (`checkInternalArgumentsSupport`, `ContractionSolution.cpp:5620`). The persistent data-parallel - branch also throws for GSU > 1 (`ContractionSolution.cpp:5141`). -- The output-conversion launch needs `(gsu > 1 && globalAccumulation && globalAccumulation != 3)` - or a parallel reduction (`ContractionSolution.cpp:5279`; `getAccumulation` returns the solution's - `globalAccumulation` when `AdaptiveGemmGSUA` is 0). Both GSU and `globalAccumulation` are 0, - and `resolvePersistentSettings` leaves the reduction at `none` for a non-StreamK persistent - solution (`ContractionSolution.cpp:5014-5033`; default in `ContractionSolution.hpp:521`). - -So no problem can make this solution ask for `Cijk_SB_PostGSU1_VW*`. The conversion objects -`initConversionKernelObjects` builds for it are themselves unused: it keys on -`GlobalSplitUAlgorithm` only, not on `_GlobalAccumulation`, so the benchmark path emits kernels -that nothing launches. The drop is real in the build output but harmless for this logic. It is -not harmless in general: a `SingleBuffer` solution with `GlobalSplitU` > 1 or -1 (`_GlobalAccumulation: -SingleBuffer`, `globalAccumulation = 1`) launches `..._PostGSU1_VW` whenever the chosen GSU is -above 1 (`gsu = 1` for `globalAccumulation == 1` before `outputConversionKernelName`), and the same -ordering would leave it without its kernel. - -### Mechanisms - -The name functions and object builders disagree in four ways -(`KernelHelperNaming.py`; `generateKernelHelperObjects` is `TensileCreateLibrary/Run.py:897`): - -| Difference | Names | Objects | Effect in shipped logic | -| --- | --- | --- | --- | -| `PostGSU` number | `KernelWriterConversion.kernelName` with the solution's `GlobalSplitU` (`PostGSU0`, `PostGSU1`, `PostGSU-1`, `PostGSU7`, ...) | `GlobalSplitUPGR` (16), or 1 for `SingleBuffer` (`KernelHelperNaming.py:171`, `:185`) | Solutions that need the same objects predict different names (most of the redundant builds). Solutions that need different objects can predict the same name: the gfx1250-strict drop. | -| `_GG` tag | solution's `GroupedGemm` | `GroupedGemm = False` (each object emits both variants) | More redundant builds only. | -| Vector widths | `[1, 2]` if `isDouble()` else `[1, 2, 4]` (`:51`) | by `numBytes()`: `[1]` if > 8, `[1, 2]` if > 4, else `[1, 2, 4]` (`:162`) | `ComplexFloat`: 3 names / 2 objects; `ComplexDouble`: 3 names / 1 object. After the sort the names are `VW4, VW2, VW1` against objects `VW2, VW1` (or `VW1`), so every position is paired with the wrong object, and `compress` ignores the surplus mask entries. Harmless only because the mask is all-`True` in every such call. | -| `MultipleBufferSingleKernel` | returns `None` (no names) | objects when `AdaptiveGemmGSUA != 0` (`:172`) | Such objects would never be built; no shipped solution has `AdaptiveGemmGSUA != 0`. | - -`BetaOnly`, `Reduction`, `ActivationEnumHeader` and `ActivationFunction` names match their objects -apart from `_GG`. - -Why the lucky outcomes hold, from the data: - -- The length mismatch only occurs for complex types, whose problem types have no bias list, so all - names of one call are new together or visited together. The partial masks (27 calls in the `all` - build: 19 conversion, 7 beta-only, 1 reduction) come from problem types sharing some bias-typed - names (e.g. an `HSS` problem type with bias types `[H, S]` whose `BiasS` names an earlier problem - type already claimed: mask `BiasH` new, `BiasS` visited) and from beta-only lists - that repeat one name per bias type when `BetaOnlyUseBias` is off (`Cijk_B_GA`, `Cijk_B_GA`). In - all of them the name and object lists have the same length and the same order (vector width - outer, bias type inner), so `compress` keeps the right objects. -- Predicted names that map to more than one object list are common (e.g. 41 of 2,089 on gfx942, 59 - of 192 on gfx1200): bias-typed names shared between problem types with different bias lists, - `Cijk_*_GA` beta-only names, and `PostGSU` names shared by solutions with different objects. - They only drop something when the first solution to claim the name does not build the object a - later solution needs, and no other solution builds it later. "Needed but skipped" is 0 in every - configuration except gfx1250-strict (3, never built). - -### Same name, different source (`BetaOnly`) - -`KernelWriterBetaOnly.kernelName` (`KernelWriterBetaOnly.py:301`) encodes `DestDataType`, grouped / -batched, bias type and `_GA`, but not `ComputeDataType`, while the kernel's `D` and `beta` types come -from `ComputeDataType` (`KernelWriterBetaOnly.py:42`). The host name (`betaOnlyKernelName`, -`ContractionSolution.cpp:4010`) omits it too. Found: - -| Build | Name | Sources (first solution) | Kept | -| --- | --- | --- | --- | -| gfx950 | `Cijk_I_GA` | `gfx950_Cijk_Ailk_Bjlk_I8IS_BH_HAS_Bias_HAS_SAV_UserArgs` (`float * D`, `float beta`); `gfx950_Cijk_Ailk_Bljk_I8II_BH_AI_SAV_UserArgs` (`int32_t * D`, `int32_t beta`) | float version | -| `all` | `Cijk_I_GA` | `navi32_Cijk_Ailk_Bljk_I8II_BH_UserArgs` (int32) first; the gfx950 `I8IS` file (float) | int32 version | -| `all` | `Cijk_I8_GA` | `gfx1170_Cijk_Ailk_Bjlk_I8BH_SHB_HA_S_SCD_SAV_UserArgs` (int32 compute) first; `gfx950_Cijk_Ailk_Bjlk_I8I8S_BH_HAS_Bias_HAS_SAV_UserArgs` (float compute) | int32 version | - -The diff of the two `Cijk_I_GA` sources is exactly the `D` pointer type, the `beta` type and the -`SCALAR_ZERO`/multiply casts (`float` versus `int32_t`). Both the TensileCreateLibrary path and the -benchmark path keep the first object per name, so this is a helper naming collision, independent of -the name prediction. The gfx1170 side of the `Cijk_I8_GA` collision comes from the -capability-substituted capture. - -Runtime reach (from the code, not run): the only beta-only launch is in `solve()` under -`gsu > 1 && globalAccumulation != 2 && globalAccumulation != 3` (`ContractionSolution.cpp:5127`; the -grouped-GEMM call at `:5416` is commented out), so `MultipleBuffer` (2) and -`MultipleBufferSingleKernel` (3) solutions never launch it. Every solution that produces a -colliding name, in every capture, is one of those with `AdaptiveGemmGSUA: 0` and not StreamK -(counts are unique solutions, kernels in parentheses): gfx950 `Cijk_I_GA` 11 (11) `MultipleBuffer` -(GSU 2); in the `all` build `Cijk_I_GA` 3,705 (2,824) `MultipleBuffer` (GSU -1 or 2–30) and -1,566 (1,444) `MultipleBufferSingleKernel`, `Cijk_I8_GA` 2,827 (2,396) `MultipleBuffer` and -4,909 (4,000) `MultipleBufferSingleKernel`. So the wrong-typed kernel is never launched with shipped logic; -it would be if a `SingleBuffer` solution with GSU > 1 or -1 (or an adaptive one) of one of these -problem types were added. - -## Conclusion - -- **Drop: yes, in one shipped configuration, harmless at run time today.** The `gfx1250-strict` - library lacks `Cijk_SB_PostGSU1_VW{1,2,4}` (and their `_GG` variants), which one `SingleBuffer` - solution calls for in the build. That solution is persistent data-parallel and never launches the - conversion kernel, so nothing requests the missing kernels. The cause is the `PostGSU` number in - the predicted name; it depends on solution order, so any logic change can add or remove such drops - without warning, including ones a launching `SingleBuffer` solution would hit. -- **Duplicate: no duplicate names reach the output** (name dedupe at the end). Objects are built - redundantly (up to ~7× on gfx942), a cost, not a correctness issue. -- **Mispair: the positional pairing is wrong for every complex-type call** (3 names, 1–2 objects), - but no shipped configuration hits the combination (length mismatch with a partially-visited mask) - that would select the wrong objects. `MultipleBufferSingleKernel` + `AdaptiveGemmGSUA` would drop - helpers but is not used by shipped logic. -- **Separately**, the `BetaOnly` name omits the compute type, so `Cijk_I_GA` (gfx950 and `all`) and - `Cijk_I8_GA` (`all`) each stand for two different kernels and one problem type gets the other's. - Latent: all solutions involved are `MultipleBuffer`/`MultipleBufferSingleKernel`, for which the - host never launches a beta-only kernel. - -Not fixed as part of this plan (DECISIONS.md topic 8). A fix direction, for whoever picks this up: -dedupe on the names of the built objects (or on the F2 input key from the benchmark path) instead -of predicted names, and add the compute type to the beta-only kernel name on both the Python and the -host side. - -## Not measured / open items - -- Runtime reach of the dropped `gfx1250-strict` kernels and of the beta-only collisions is derived - from the host code, not run (no gfx1250 hardware here; the gfx950 beta-only case was not run - either). -- gfx1170/gfx1171 with a compiler that supports them (capabilities were substituted, see Method). -- Builds with an explicit `--logic-filter` other than one file, or arch subsets other than the ones - listed. The per-file and per-arch results bracket them, but order-dependent drops like the - gfx1250-strict one can appear in any combination that puts a `MultipleBuffer` solution with the - same `GlobalSplitU` value before a `SingleBuffer` one of a problem type with the same conversion - name. diff --git a/projects/hipblaslt/tensilelite/timing_runs/times.txt b/projects/hipblaslt/tensilelite/timing_runs/times.txt deleted file mode 100644 index 19cb2f9be521..000000000000 --- a/projects/hipblaslt/tensilelite/timing_runs/times.txt +++ /dev/null @@ -1,3 +0,0 @@ -build rc=0 60s -run rc=0 26s -DONE diff --git a/projects/hipblaslt/test-client-plan.md b/projects/hipblaslt/test-client-plan.md deleted file mode 100644 index 36bdfdc5b3f7..000000000000 --- a/projects/hipblaslt/test-client-plan.md +++ /dev/null @@ -1,55 +0,0 @@ -# Goal -Improve the speed of the tensilelite common tests, and of the standalone test done through that path. - -# Problem -The current testing is done through using code written to check both correctness AND performance through benchmarking. The benchmark requirements places additional contrainst on the numerical tesing which make it harder to improve performance. E.g. it include hard barriers and forces gpu->host drains before it continues to the next problem. - -I'm attaching data for the two slowest tensilelite common tests on gfx942: dtv.yaml and cgemm.yaml. I'm only including the run side of things since this is what needs to live on the GPU-node. - -Here is the breakdown of the top times, with the gpu work also included: - -dtv.yaml: 84s total - - 23.8s waiting for gpu buffers to be reset between solutions - - 29.0s for client setup - - 11.5s to do the gpu vs reference comparison - - 3.6s for reference gemm calc - - 2.3s for logging and perf calc - - 0.5s for actual gpu work - -cgemm: 41s total - - 12.5s for client setup - - 10.6s for the slow reference gemm - - 5.2s for the gpu vs reference comparison - - 2.0s for logging and perf calc - - 0.3s for actual gpu work. - -So for the run step, both of these yaml files spend less that 1% of their total wall clock time on the GPU. We need to seriously improve that. - - -# Some more info - -The yamls are set up as: -``` -BenchmarkProblems: - - case 1 - - ... - - case N -``` -The subentries of each run in parallel. with N sequential runs; N=9 for cgemm and N=25 for dtv). - -The CMS yamls take much longer per kernel in their build step because each case contains a single kernel, and Tensile is set up to parallelize only *within* a case, not across them. - -None of these times actually measure GPU time, we don't know exactly how long the kernel takes to solve from these traces as they're CPU only, kernel_solving only captures dispatch, dtv/cgemm's 10us per dispatch it too low. It's likely that this the GPU execution time is being misattributed to the validate_gpu_readback entry for validate_warmup ' -# Initial Thoughts -- client setup seems to be repeated N times, if that can be shared it's an immediate saving of (N-1)/N (22.9 s for dtv and 12 s for cgem, 27-30% speedup). -- gpu buffer readbacks should not block the way they do now, we should use a ring buffer on the gpu(s) to pipeline the {reseting, solving, transfer, validation} loop. -- The reference gemm calc should be made faster, or moved to GPU, and results should either be cached or runs internally re-ordered to not duplicate reference gemm calcs. -- Logging and perf calc can be removed entirely. Numerical correctness should done on an all-or-nothing basis. We don't need to know which entries are wrong, we just need to know that they are. A separate path can be made available for letting the user figure out which values are wrong as useful debug info, but for CI we only need a pass-fail. -- New approach takes the pre-compiled kernels (as first pass) and pipelines everything (loading, data generation, reference solution, data transfers, gpu work, cleanup, validation). -- Looking at the times from the dtv run, the - -# Outstanding questions -- How much of the GPU can we claim for ourselves? How much of its RAM -- Can we get rid of ClientExecutionLockPath? -- Do we want to still run them as individual yaml files, or can we group them together into a single stream of tests to do? -- How to handle the A2A fusions, and any other kernels that requires multiple-gpus \ No newline at end of file