perf(apple): the low-precision MoE path jumped the arbiter and was 12-60x slower

[gstoner]

Found while investigating the MoE telemetry flake. It is a separate defect from #719 and needs no runtime edit or ledger re-seal.

The finding

The MoE SwiGLU composite had three implementations and no single place that chose between them. single_fused, lowp and composed were fall-through blocks, each ending in except Exception: pass, so which one ran was not predictable from the inputs and a failure between them was invisible.

The defect is narrower and worse than untidy structure: lowp sat ahead of the arbiter and preempted it unconditionally for any uniform f16/bf16 operand, so production_route_for never got to decide for the common low-precision inference shape.

Measured on this M1 Max, best of 5 after warm-up, milliseconds:

(T,K,H,N,E)	dtype	single_fused	lowp	composed
64,128,256,128,4	f32	13.23	—	1.27
64,128,256,128,4	f16	—	15.04	1.26
256,256,256,256,8	f32	32.00	—	1.89
256,256,256,256,8	f16	—	35.66	1.84
1024,512,512,512,8	f16	—	1571.06	26.31

composed wins every case and the gap widens with size (10× → 17× → 60×). It is also the more accurate: 6.3e-8 relative error against an fp32 reference where lowp is 2.6e-4, because composed accumulates in fp32.

So the low-precision default — the common inference path — was about 12–60× slower and 4000× less accurate than the route sitting directly below it. And lowp has no ledger row at all: preferred by default, never measured into the arbiter.

The fix restores the arbiter rather than removing it

My first attempt deleted the production_route_for call. The suite caught it (test_moe_dispatch_consumes_the_strict_exact_row) and was right to — that would have made the ledger row a declaration nothing reads (Decision #29), and would have "fixed" a slow route by deleting the mechanism whose job is to choose between routes (Decision #28).

Checking the ledger directly settled it: on the committed ledger it answers composed, correctly. It was never the problem. The problem was a path wired ahead of it.

_apple_moe_select_route is now the one selection point, returning (route, reason):

• the arbiter decides between the routes it has evidence about;
• lowp is opt-in (TESSERA_APPLE_MOE_LOWP=1) until it earns a ledger row;
• quant outranks everything — per-GEMM quantization is the one thing the single-kernel paths cannot express;
• choosing a route measured slower than the default lands in the dispatch fallback log under the op's name (Decision Apple GPU Tier-3: conv2d via MPSGraph convolution2D #21), so a machine on a slow lane can be found rather than guessed at;
• the three except Exception: pass swallows are gone, so a failure inside a chosen route is a diagnostic.

Do we need all three pathways?

On this evidence, no — but keep them reachable. Neither alternative earns a default. Both are waiting on rewrites that could change that: the fused kernel is one-thread-per-token and needs a threadgroup-cooperative version, and lowp's single command buffer may win on a resident streaming lane where the composed path's three dispatches dominate. The rule this sets is that such a path must be measured into the arbiter, not wired ahead of it.

Tests

tests/unit/test_moe_route_selection.py (10): composed is the default for every dtype; quant outranks the opt-ins; both alternatives are reachable but only by name; an opt-in the shape cannot honour says so; every reason is non-empty; choosing a slower route is recorded; the arbiter is still consulted (with the op and shape key asserted); an unreadable ledger is non-fatal with a reason; a ledger choice out of the fused kernel's range is declined rather than dispatched into.

Timings are deliberately not asserted — that would be a flaky perf gate on shared CI. The measurement lives in the code comment and the Apple queue entry, where it can be re-run deliberately.

Sweep

M1 Max, unsandboxed: 3 failed / 16568 passed, and none of the three is from this change — two are test_lowp_moe_composite_execute_compare, the separate process-wide telemetry defect (an unrelated op reports no device time either), and one is test_resolver_finds_repo_built_opt_without_env_or_path, which fails because my build directory is named build-apple. mypy clean.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-04T21:04:23.288381Z	0626620	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0626620507

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".