perf(apple): the low-precision MoE path jumped the arbiter and was 12-60x slower

docs/audit/backend/apple/todo.md

@@ -7142,6 +7142,142 @@ documents the one, asserts zero session fallbacks, and checks the survivor is
 still the telemetry read rather than a fallback that crept back under the same
 spelling.
 
+## APPLE-MLPKG-RACE-1: the packaged ML dispatch raced its own signal *(fixed 2026-09-04, M1 Max)*
+
+**How it was found is the useful part.** It was not reported as a bug. It
+surfaced while attributing a pre-existing flake during unrelated work: the same
+command failed **2, then 21, then 62, then 28** tests across four runs, and
+every failure carried one signature — `dispatch returned False;
+last_error_kind=0`. A varying failure count with a constant signature is a race,
+not twenty-one bugs.
+
+**Two defects, and the first hid behind the second.**
+
+*The race.* `mtl4_shared_queue` is shared by every packaged dispatch, and this
+lane deliberately does not take `mtl4_dispatch_mu` (fresh allocator + command
+buffer per call). But `commit:` and `signalEvent:value:` are **two separate
+queue operations**. Unlocked, thread B interleaves between A's commit and A's
+signal, and A's signal no longer denotes A's buffer.
+
+*Why nobody saw it.* The context-wide `mlpkg_event` made any thread's signal
+satisfy any thread's wait. A waiter returned on **another dispatch's**
+completion and read its outputs while its own command buffer was still
+running — a silent wrong answer whose only visible trace was an occasional
+False. Switching to a private event turned that into a wait that never
+completes, which is the *truthful* symptom and the evidence the signal was
+genuinely not arriving. Both halves are needed: a per-dispatch event, and
+commit+signal under one lock. The wait stays outside the lock, so dispatches
+still overlap on the GPU.
+
+This is the correction `7079e95a` already applied to `commit_and_wait_with_timeout`
+and its MPSGraph sibling, with the reasoning recorded there. **The packaged lane
+was simply missed** — worth remembering when a fix is applied "to the waits":
+enumerate them.
+
+*The silence.* The function returned 0 from **ten** places and set the
+last-error channel from **none**. Every failure reached the caller as
+`last_error_kind=0` — true, useless, and indistinguishable between "no ML
+encoder on this SDK" and "the device stopped answering". Each path now names
+itself: kind 2 for an ordinary per-op failure, kind 1 for the timeout — the
+kind that feeds the Python dispatch breaker, so a wedged device now stops being
+asked instead of paying this timeout once per packaged dispatch. The timeout
+path also quarantines pooled buffers, as the sibling waits do.
+
+**Evidence** (M1 Max, unsandboxed, fresh process per trial, hang counted as a
+failure via an external deadline). The test is the one written for this race —
+`test_apple_mlpkg_concurrency`, 4 threads x 8 dispatches:
+
+| runtime | trials | pass | fail |
+|---|---:|---:|---:|
+| unmodified | 20 | 18 | 2 |
+| fixed | 60 | 60 | 0 |
+
+At the baseline's 10% rate, 60 clean trials has probability ~0.0018. The eight
+files that failed in the original sweep run **101 passed / 0 skipped**, and the
+reproducer's failure count stopped varying between runs.
+
+**Two process traps hit while verifying this, both worth avoiding.**
+
+1. **A `build` symlink into another checkout's build tree silently compiles
+   THAT checkout's source.** CMake stores absolute paths, so
+   `ninja -C build TesseraAppleRuntimeShared` from a worktree rebuilt the main
+   tree's `.mm` and produced a dylib without the fix. The tell is the compiler
+   warning paths naming a directory you are not editing. Configure a build in
+   the worktree instead.
+2. **A minimal build turns the tests you are trying to verify into skips.** The
+   first "all green" family run had 21 skips reading *"integration requires the
+   Apple GPU runtime and libtessera_jit ABI"* — and those skips were exactly
+   five of the failures under investigation. Build `libtessera_jit` and
+   `tessera-opt`, then re-run: `-rs` and a skip count are the check that a green
+   result evaluated anything ([[hollow_green_signal_pattern]]).
+
+**Cross-backend:** not applicable to NVIDIA, ROCm or x86 under
+`DISPATCH-BREAKER-RESIDENT-2026-09-03`'s reasoning — this is a Metal 4 queue
+API pairing with no sibling. The *general* lesson does transfer and is worth
+stating: any backend that submits work and signals completion as two separate
+calls on a shared queue has this bug shape, and a shared completion counter
+will hide it as an occasional wrong answer rather than a failure.
+
+## APPLE-MOE-ROUTE-1: the low-precision MoE path jumped the arbiter *(fixed 2026-09-04, M1 Max)*
+
+**The MoE SwiGLU composite had three implementations and no single place that
+chose between them.** `single_fused`, `lowp` and `composed` were three
+fall-through blocks, each ending in `except Exception: pass`, so which one ran
+was not predictable from the inputs and a failure between them was invisible.
+
+**The defect is narrower and worse than "untidy".** `lowp` sat **ahead of the
+arbiter** and preempted it unconditionally for any uniform f16/bf16 operand, so
+`production_route_for` never got to decide for the common low-precision
+inference shape. Measured here (best of 5 after warm-up, ms):
+
+| (T,K,H,N,E) | dtype | single_fused | lowp | composed |
+|---|---|---:|---:|---:|
+| 64,128,256,128,4 | f32 | 13.23 | — | **1.27** |
+| 64,128,256,128,4 | f16 | — | 15.04 | **1.26** |
+| 256,256,256,256,8 | f32 | 32.00 | — | **1.89** |
+| 256,256,256,256,8 | f16 | — | 35.66 | **1.84** |
+| 1024,512,512,512,8 | f16 | — | 1571.06 | **26.31** |
+
+`composed` wins every case and the gap widens with size (10x → 17x → 60x). It
+is also the more accurate: **6.3e-8** relative error against an fp32 reference
+where `lowp` is **2.6e-4**, because `composed` accumulates in fp32. So the
+low-precision default was ~12-60x slower *and* ~4000x less accurate than the
+route it displaced — and `lowp` **has no ledger row at all**: it was preferred
+by default while never having been measured into the arbiter.
+
+**The fix restores the arbiter rather than removing it.** The first attempt
+deleted the `production_route_for` call, which the suite caught
+(`test_moe_dispatch_consumes_the_strict_exact_row`) and was right to: that
+would have made the ledger row a declaration nothing reads (Decision #29) and
+"fixed" a slow route by deleting the mechanism whose job is to choose between
+routes (Decision #28). Checking the ledger directly settled it — on the
+committed ledger it answers `composed`, correctly. It was never the problem.
+
+`_apple_moe_select_route` is now the one selection point, returning
+`(route, reason)`. The arbiter decides between the routes it has evidence
+about; `lowp` is opt-in (`TESSERA_APPLE_MOE_LOWP=1`) until it earns a ledger
+row; `quant` outranks everything, because per-GEMM quantization is the one
+thing the single-kernel paths cannot express. Choosing a route measured slower
+than the default lands in the dispatch fallback log under the op's name
+(Decision #21), so a machine running a slow lane can be found rather than
+guessed at. `tests/unit/test_moe_route_selection.py` (10) pins the matrix,
+including an unreadable ledger and a ledger choice the shape cannot honour.
+
+**Do we need all three? On this evidence, no — but keep them reachable.**
+Neither alternative earns a default. Both are waiting on rewrites that could
+change that: the fused kernel is one-thread-per-token and needs a
+threadgroup-cooperative version, and `lowp`'s single command buffer may win on
+a resident streaming lane where the composed path's three dispatches dominate.
+The rule this entry sets is that such a path must be **measured into the
+arbiter**, not wired ahead of it.
+
+**Cross-backend:** not applicable to NVIDIA, ROCm or x86 — this is Apple
+dispatch-selection code with no sibling, and it changes no dtype, Graph IR
+spelling, ABI or numeric contract. The *general* lesson transfers and is worth
+stating: an implementation that selects itself ahead of the arbiter is outside
+the measured-arbiter model of Decision #28 no matter how fast it is, and this
+one was neither fast nor accurate.
+
 ## Cross-backend sync `MATRIX-LANE-RAGGED-SHAPES-2026-09-01`
 
 **Owning item:** the matrix-core lanes decline ragged shapes ·

docs/audit/evidence/e2e_spine/apple_gpu/apple7/manifest.json

@@ -2,17 +2,17 @@
   "architecture": "apple7",
   "files": {
     "report.json": {
-      "bytes": 4918,
-      "sha256": "e5c6f84030b754895f7c40bf122403c1c6f3a5a02229f90c60451cba1d58c6e8"
+      "bytes": 4910,
+      "sha256": "e710f6406c863a6486e79736881863f26a89c492803ccdd912ba23e224fa1a43"
     },
     "resources.json": {
       "bytes": 4346,
-      "sha256": "9ab93b1bd0307406afe59dc453b5b9f5e0009da026fce98c6ed211dff53d0f48"
+      "sha256": "cb02c4ef5edab4a92a0e792ea310ac5fe3c06540de353f994d3d3045bbb2f704"
     }
   },
   "schema": "tessera.e2e-release-packet.v1",
   "target": "apple_gpu",
-  "tested_commit": "1e4f05cd1756a39ca771f04238746b3ad0af97ab",
+  "tested_commit": "cd12ca2626ee80e1afd039733cb731ee5162efc6",
   "validation": {
     "architecture": "apple7",
     "benchmark_rows": 4,
@@ -21,7 +21,7 @@
     "fixtures": 2,
     "level_c_fixtures": 2,
     "maximum_absolute_error": 0.0,
-    "source_commit": "1e4f05cd1756a39ca771f04238746b3ad0af97ab",
+    "source_commit": "cd12ca2626ee80e1afd039733cb731ee5162efc6",
     "target": "apple_gpu"
   }
 }

docs/audit/evidence/e2e_spine/apple_gpu/apple7/report.json

@@ -4,13 +4,13 @@
     {
       "discard_first": true,
       "family": "matmul",
-      "median_ns": 1039853.75,
+      "median_ns": 1116434.58,
       "repetitions": 750,
-      "resource_fingerprint": "74b60f63ca214b9e242022b500911ee076e0132bcb5360bc6a898fc478a9088b",
+      "resource_fingerprint": "e1ae90e6a9d76f0c83ff120df784846e021955285df126a843814de5bd965b44",
       "route": "apple_gpu_bmm_f32_batch1",
       "run_medians_ns": [
-        1037255.0,
-        1042452.5
+        1120960.0,
+        1111909.16
       ],
       "selected": true,
       "stability_limit_pct": 4.0,
@@ -21,13 +21,13 @@
     {
       "discard_first": true,
       "family": "matmul",
-      "median_ns": 497308.82,
+      "median_ns": 488604.25,
       "repetitions": 750,
-      "resource_fingerprint": "74b60f63ca214b9e242022b500911ee076e0132bcb5360bc6a898fc478a9088b",
+      "resource_fingerprint": "e1ae90e6a9d76f0c83ff120df784846e021955285df126a843814de5bd965b44",
       "route": "apple_gpu_bmm_f32_batch1",
       "run_medians_ns": [
-        496821.34,
-        497796.3
+        491298.82,
+        485909.68
       ],
       "selected": true,
       "stability_limit_pct": 4.0,
@@ -38,13 +38,13 @@
     {
       "discard_first": true,
       "family": "softmax",
-      "median_ns": 400775.42000000004,
+      "median_ns": 440010.0,
       "repetitions": 750,
-      "resource_fingerprint": "d1f5c16643311e04f5b0955b9d46008653a93977c08ea9ac7a8ad3037bb74b73",
+      "resource_fingerprint": "580a8891e929c11af20cfc0a6760f17e02de2b1858326dc297d951b21b1f2aa8",
       "route": "apple_gpu_msl_softmax_f32",
       "run_medians_ns": [
-        398733.34,
-        402817.5
+        441975.0,
+        438045.0
       ],
       "selected": true,
       "stability_limit_pct": 4.0,
@@ -55,13 +55,13 @@
     {
       "discard_first": true,
       "family": "softmax",
-      "median_ns": 106912.73,
+      "median_ns": 104664.08,
       "repetitions": 750,
-      "resource_fingerprint": "d1f5c16643311e04f5b0955b9d46008653a93977c08ea9ac7a8ad3037bb74b73",
+      "resource_fingerprint": "580a8891e929c11af20cfc0a6760f17e02de2b1858326dc297d951b21b1f2aa8",
       "route": "apple_gpu_msl_softmax_f32",
       "run_medians_ns": [
-        106647.7,
-        107177.76
+        104741.22,
+        104586.94
       ],
       "selected": true,
       "stability_limit_pct": 4.0,
@@ -73,32 +73,32 @@
   "cache_proofs": [
     {
       "cold": {
-        "cache_key": "8121b37136181a4c951b2eb27cacb018b5cfd3f81bd355c0bf4138f2fc6c7176",
+        "cache_key": "001fa0bd660e0ca738e8ec3ff80b8185ebe7b43b6785b51c42502bbc7c814d47",
         "compile_state": "prepackaged",
-        "descriptor_digest": "bd5cbadab67ac28df8dd80b5f1587362c95b4b4e203b3fad8c2091f0973bd7c4",
-        "image_digest": "347037c38f1fc46c88fbf64c020022be80091d1c45ee9d8bed48d05a63a51b9c"
+        "descriptor_digest": "bda25129fd1c69d6509678f2b48de6c63b4007c74984c20066d1b2f76c495569",
+        "image_digest": "a76ba792a62b47c6eadb407e9d8a7fe3857caa6ea627f48f443e7b178450856e"
       },
       "fixture_id": "matmul-f32-2x3x2-v1",
       "warm": {
-        "cache_key": "8121b37136181a4c951b2eb27cacb018b5cfd3f81bd355c0bf4138f2fc6c7176",
+        "cache_key": "001fa0bd660e0ca738e8ec3ff80b8185ebe7b43b6785b51c42502bbc7c814d47",
         "compile_state": "prepackaged",
-        "descriptor_digest": "bd5cbadab67ac28df8dd80b5f1587362c95b4b4e203b3fad8c2091f0973bd7c4",
-        "image_digest": "347037c38f1fc46c88fbf64c020022be80091d1c45ee9d8bed48d05a63a51b9c"
+        "descriptor_digest": "bda25129fd1c69d6509678f2b48de6c63b4007c74984c20066d1b2f76c495569",
+        "image_digest": "a76ba792a62b47c6eadb407e9d8a7fe3857caa6ea627f48f443e7b178450856e"
       }
     },
     {
       "cold": {
-        "cache_key": "45101d896a1df9fa333c3b77116909146e03fd15fa355560b98212d3fecb4a68",
+        "cache_key": "595ff485de6160d969dc2590c5227b66283f59e68efa64b20115004f2e27216d",
         "compile_state": "prepackaged",
-        "descriptor_digest": "03ae5f80267f7cf8d4f25e9469b8d07909f34431ad5376b30f6e2ae6df9c4821",
-        "image_digest": "9d6a0d6ba6ab3feb18dbcbdef3505495dc2b79c9c024b9c58b15d3967f12266c"
+        "descriptor_digest": "6cb992c666778ada9b107253e22a90e87721e13a7d6da640bb5160169cef2cf5",
+        "image_digest": "11e7027d86f6c09fca3657db5fd46378144c8b19ff917f08dce82db3f2375391"
       },
       "fixture_id": "softmax-f32-2x2-extreme-v1",
       "warm": {
-        "cache_key": "45101d896a1df9fa333c3b77116909146e03fd15fa355560b98212d3fecb4a68",
+        "cache_key": "595ff485de6160d969dc2590c5227b66283f59e68efa64b20115004f2e27216d",
         "compile_state": "prepackaged",
-        "descriptor_digest": "03ae5f80267f7cf8d4f25e9469b8d07909f34431ad5376b30f6e2ae6df9c4821",
-        "image_digest": "9d6a0d6ba6ab3feb18dbcbdef3505495dc2b79c9c024b9c58b15d3967f12266c"
+        "descriptor_digest": "6cb992c666778ada9b107253e22a90e87721e13a7d6da640bb5160169cef2cf5",
+        "image_digest": "11e7027d86f6c09fca3657db5fd46378144c8b19ff917f08dce82db3f2375391"
       }
     }
   ],
@@ -120,9 +120,9 @@
           ]
         ]
       ],
-      "descriptor_digest": "bd5cbadab67ac28df8dd80b5f1587362c95b4b4e203b3fad8c2091f0973bd7c4",
+      "descriptor_digest": "bda25129fd1c69d6509678f2b48de6c63b4007c74984c20066d1b2f76c495569",
       "fixture_id": "matmul-f32-2x3x2-v1",
-      "image_digest": "347037c38f1fc46c88fbf64c020022be80091d1c45ee9d8bed48d05a63a51b9c",
+      "image_digest": "a76ba792a62b47c6eadb407e9d8a7fe3857caa6ea627f48f443e7b178450856e",
       "levels": {
         "a": "proven",
         "b": "proven",
@@ -140,9 +140,9 @@
           0.5
         ]
       ],
-      "descriptor_digest": "03ae5f80267f7cf8d4f25e9469b8d07909f34431ad5376b30f6e2ae6df9c4821",
+      "descriptor_digest": "6cb992c666778ada9b107253e22a90e87721e13a7d6da640bb5160169cef2cf5",
       "fixture_id": "softmax-f32-2x2-extreme-v1",
-      "image_digest": "9d6a0d6ba6ab3feb18dbcbdef3505495dc2b79c9c024b9c58b15d3967f12266c",
+      "image_digest": "11e7027d86f6c09fca3657db5fd46378144c8b19ff917f08dce82db3f2375391",
       "levels": {
         "a": "proven",
         "b": "proven",
@@ -159,7 +159,7 @@
     "matmul",
     "softmax"
   ],
-  "source_commit": "1e4f05cd1756a39ca771f04238746b3ad0af97ab",
+  "source_commit": "cd12ca2626ee80e1afd039733cb731ee5162efc6",
   "target": "apple_gpu",
-  "toolchain_fingerprint": "8fc1d23998d37cbc5af2b785b6e73b26c81f21309eefaf50711d361153025c1d"
+  "toolchain_fingerprint": "5dcb86457d9936523b31787541a70303486ad1089e6361a58d19ad42fe9a8f23"
 }

docs/audit/evidence/e2e_spine/apple_gpu/apple7/resources.json

@@ -48,7 +48,7 @@
       },
       "source_fingerprint": {
         "state": "measured",
-        "value": "sha256:12c337036dea501ba6804a7dae969830844d9304ec7e605d2faea27b465e7ee7"
+        "value": "sha256:5b63fddfc10a23525ba1ab5f6aabd0b7eb645543850b010c9761b9bc978f201d"
       },
       "spill_count": {
         "state": "no_public_api",
@@ -76,7 +76,7 @@
           2
         ]
       ],
-      "image_digest": "347037c38f1fc46c88fbf64c020022be80091d1c45ee9d8bed48d05a63a51b9c",
+      "image_digest": "a76ba792a62b47c6eadb407e9d8a7fe3857caa6ea627f48f443e7b178450856e",
       "placement": {
         "fixture": {
           "gpu_placement_proven": true,
@@ -89,7 +89,7 @@
           "status_symbol": "tessera_apple_gpu_bmm_f32_status"
         }
       },
-      "resource_fingerprint": "74b60f63ca214b9e242022b500911ee076e0132bcb5360bc6a898fc478a9088b",
+      "resource_fingerprint": "e1ae90e6a9d76f0c83ff120df784846e021955285df126a843814de5bd965b44",
       "route": "apple_gpu_bmm_f32_batch1",
       "timing_shape": [
         1,
@@ -107,7 +107,7 @@
           2
         ]
       ],
-      "image_digest": "9d6a0d6ba6ab3feb18dbcbdef3505495dc2b79c9c024b9c58b15d3967f12266c",
+      "image_digest": "11e7027d86f6c09fca3657db5fd46378144c8b19ff917f08dce82db3f2375391",
       "placement": {
         "fixture": {
           "execution_width": 32,
@@ -136,7 +136,7 @@
           ]
         }
       },
-      "resource_fingerprint": "d1f5c16643311e04f5b0955b9d46008653a93977c08ea9ac7a8ad3037bb74b73",
+      "resource_fingerprint": "580a8891e929c11af20cfc0a6760f17e02de2b1858326dc297d951b21b1f2aa8",
       "route": "apple_gpu_msl_softmax_f32",
       "timing_shape": [
         64,