apple: scheduled reduction (first synthesized family) + close APPLE-DEVICE-EVENT-1 - #571
Merged
Merged
Conversation
…EVICE-EVENT-1 Two slices, closing the last WS-1 family and the open device-timing gap. == Scheduled reduction (E2E-REAL-5) == Apple GPU consumes the shared `schedule.reduce -> tile.reduce_kernel` artifact. This family is different in kind from its siblings: matmul f32 delegates to MPS and softmax binds a hand-written kernel, but reduction is genuinely COMPILER-EMITTED — the Decision #28 tier-1 synthesizer (`emit/apple_msl.py::synthesize_pointwise_reduce_msl`) produces the MSL and the source-carrying `tessera_apple_gpu_synth_pointwise_reduce_f32` ABI runs it. It is the first Apple family where the MLIR boundary and the Python synthesizer meet on one artifact — the seam CLAUDE.md names as the real Apple gap. * Content-addressed source: the package records the kernel's SHA-256 and carries the MSL itself as Target IR (`tessera_apple.gpu.msl_kernel`, not a call stub); the runtime re-derives and VERIFIES the digest before dispatch, so a descriptor cannot execute a kernel other than the one it was built from. Synthesis is deterministic in the reduce kind alone, which is what makes that check meaningful rather than decorative. * Placement asserted, not inferred: `run_pointwise_reduce` silently falls back to a NumPy reference on failure, so the submit path requires a `metal_runtime` provenance and raises otherwise (APPLE-PLACEMENT-ABI-1). * Envelope: f32, rank-reducing, LAST AXIS only (inner == 1), sum/mean/max. The kernel gives one thread per row and folds the trailing extent, so an interior axis fails closed in both owners rather than being reordered. gfx1151 keeps its arbitrary-axis support and a test asserts the Apple bound does not leak. Evidence: six exact-device configurations (sum/mean/max x rank-2/rank-3) run on Metal with native_gpu placement and match NumPy; 21 scheduled-kernel tests pass. == APPLE-DEVICE-EVENT-1 closed == The GPU matmul route ran through `MPSGraph runWithMTLCommandQueue:`, which owns and commits its own command buffer, so no object existed on which to observe a device interval and the whole apple_gpu packet was forced onto `kernel_wall`. `mpsg_run_bmm` now encodes into an explicitly owned MPSCommandBuffer under the ALREADY-EXISTING shared `MPSGraphTimingBracket` — the same one gather/transpose/ row-op/BSMM use, so this reuses one implementation rather than adding a second (Decision #31). The recorder no longer hard-codes `device_event_available: False` with a reason that had become false. It probes the device interval per family and claims the `device_event` domain only when EVERY family in scope published one; otherwise it falls back to kernel_wall for all families rather than mixing domains. The sealed packet is RE-RECORDED, not re-stamped: editing apple_gpu_runtime.mm correctly tripped the packet's source fingerprint, and the fix is a fresh measurement. It now seals `required_timing_domains = ["device_event", "end_to_end"]` with both families stable at 31x80 sampling — the 15x50 default drifted 5.2% against a 4% bar for the device cohort, so the sampling was raised rather than the bar lowered. What the stronger domain immediately revealed: matmul is 0.595 ms device inside 1.810 ms end-to-end, and softmax is 0.0235 ms device inside 1.014 ms. These lanes are host-overhead dominated — ~23 us of GPU work inside a millisecond of wall time — which kernel_wall could not have shown. Cost check, since a .mm edit is not free: the strict route ledger's fingerprint was ALREADY invalid before this change (pinned 74eb6e95 vs live 9b12af9a, inert since 2026-07-27), so no live selector evidence was destroyed. End-to-end matmul timing is unchanged (0.95 -> 0.89-0.96 ms across runs, inside p90 noise). `test_both_apple_identities_are_sealed_independently` asserted both lanes report kernel_wall — an assertion that encoded the limitation itself. It now asserts each lane pairs exactly one clock with end_to_end, that apple_cpu (Accelerate, no Metal command buffer) cannot reach device_event, and that any device_event claim is backed by a measured interval. == Verification (M1 Max / apple7) == 205 passed / 17 skipped across the scheduled, e2e-spine, fleet, lineage and Apple attention/MLA/MPSGraph suites; the single failure is the proven pre-existing gfx1151 test that needs AMD clang. Simulated runtime-only host (`env -u TESSERA_OPT`): 117 passed, 0 failed. ruff clean; mypy 0/455; three lit fixtures pass; generated docs in sync. Benchmark extended with reduce rows — all 13 rows native and oracle-checked; synthesized reduce is the fastest family at 0.37 ms. Operational note, because it cost real debugging time: after rebuilding the dylib, ~20 device tests failed with "Apple runtime is missing <symbol>" although nm showed the symbol present. The runtime publishes the dylib into a temp cache that was stale until the next process republished it. Not a regression, but indistinguishable from one at first glance. NVIDIA and ROCm are not applicable: every compiler branch is target-guarded to apple_gpu, and the runtime change is Apple-only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0aceb76997
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Self-review of the reduce consumer found an unlocked edge: a rank-1 reduction produces a rank-0 result, exercising the scalar output binding and the `out_shape == ()` path in the submit reshape. Verified working end-to-end on Metal (err 9.5e-07 for a 128-element sum) and now covered by the exact-device parametrization rather than left to chance. The rank-0 type also has a distinct spelling — `tensor<f32>`, not the `"x".join(...)` form, which degenerates to a malformed `tensor<xf32>` for an empty shape. Building it correctly is what makes the case a real proof rather than a construction error dressed as coverage. Tolerance was checked rather than assumed while reviewing: across 40 seeds the worst reduce error is 0.40x the asserted tolerance (sum at 64x128), so the existing bound has ~2.5x margin and is not flaky. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Self-review caught `_probe_device_event` defined and never called — exactly the unconsumed declaration Decision #29 rejects ("a declaration with no consumer is worse than a missing one: it reads as a closed contract in review while carrying nothing"). The live path uses `_device_time_reader` + `_device_event_medians_ns`, which collect the interval per sample rather than probing once, so the probe was a leftover from an earlier shape of the change. Verified after removal: the recorder still seals `required_timing_domains = ["device_event", "end_to_end"]` and the fleet suite passes 12/12. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…utcomes Two PR #571 review findings. == P2: `max` violated its own declared NaN contract (real, confirmed on device) == The synthesizer emitted `max(acc, v)` for amax/amin. Metal's `max`/`min` are IEEE maxNum/minNum-style and SUPPRESS a NaN operand, so: [1, NaN, 3] -> GPU 3 vs numpy NaN [NaN, NaN, NaN] -> GPU -inf vs numpy NaN The reduce Schedule artifact literally declares `nan_mode = "propagate"`, and `_PW_REDUCE_KINDS`' own numpy reference propagates, so the kernel disagreed with both its contract and its declared oracle. The all-NaN case is the one that bites: missing data silently became a finite extreme that a downstream argmax or clamp would act on. Fixed at the single source — `fusion_core.py::_PW_REDUCE_KINDS`, whose only kernel consumer is `emit/apple_msl.py` — so extrema propagate explicitly and once `acc` is NaN it stays NaN, matching numpy. `sum`/`mean` needed no change: IEEE addition already propagates. All four kinds now match numpy exactly, including +/-inf rows. `test_apple_gpu_scheduled_reduce_propagates_nan` locks it on device, asserting both numeric equality and that an all-NaN row does not become -inf. 294 fusion-lane tests pass, so the shared-table change regresses nothing. == P1: sibling-backend coordination records (AGENTS.md:81-85) == The shared reduce vocabulary, the shared scheduled-kernel gate, and the runtime dispatcher were touched, so all four architecture queues must record an outcome. ROCm, NVIDIA and x86 now carry an explicit `APPLE-SCHEDULED-REDUCE-NAN-2026-08-16` entry: not applicable with an architecture-specific reason (`_PW_REDUCE_KINDS` has exactly one kernel consumer, `emit/apple_msl.py`; each sibling's reduction goes through its own `tile.reduce_kernel` lowering). Each also carries the forward-looking caution that CUDA `fmaxf` and x86 `maxps`/`maxss` suppress NaN identically, so a future port must propagate explicitly rather than inherit the bug, plus a note that gfx1151's arbitrary-axis reduction is deliberately retained where Apple admits last-axis only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PR #571 review, P1: the packet attributed its device_event evidence to a136395 — this branch's PARENT, which still uses `runWithMTLCommandQueue:` and hard-codes kernel_wall. The intervals had been measured from an uncommitted working tree, so the generated fleet dashboard marked a commit release-ready for evidence its code cannot produce. That is a real evidence-integrity defect: the sealed packet's whole job is to identify the tree that was tested. Re-recorded from a verifiably CLEAN tree at b439989, which contains both the owned-command-buffer BMM timing and the NaN-propagation fix (checked by reading the file out of that commit, not by assuming ancestry). `source_commit` now names a tree that can actually produce these intervals, and the packet still seals `required_timing_domains = ["device_event", "end_to_end"]`. Recording order that keeps this honest, since the mechanism stamps `git rev-parse HEAD` at record time: commit the code, verify the tree is clean, record, then commit the packet alone. Recording before committing silently attributes the measurement to the parent. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two slices: the last open WS-1 family, and the open device-timing gap.
1. Scheduled reduction (E2E-REAL-5) — the first compiler-emitted Apple family
Apple GPU now consumes the shared
schedule.reduce → tile.reduce_kernelartifact. This family is different in kind from its siblings: matmul f32 delegates to MPS and softmax binds a hand-written kernel, but reduction is genuinely compiler-emitted — the Decision #28 tier-1 synthesizer (emit/apple_msl.py::synthesize_pointwise_reduce_msl) produces the MSL and the source-carryingtessera_apple_gpu_synth_pointwise_reduce_f32ABI runs it.It is the first Apple family where the MLIR boundary and the Python synthesizer meet on one artifact — the seam
CLAUDE.mdnames as the real Apple gap ("two disconnected compilers").tessera_apple.gpu.msl_kernel, not a call stub). The runtime re-derives and verifies the digest before dispatch, so a descriptor cannot execute a kernel other than the one it was built from. Synthesis is deterministic in the reduce kind alone, which is what makes that check meaningful rather than decorative.run_pointwise_reducesilently falls back to a NumPy reference on failure, so the submit path requires ametal_runtimeprovenance and raises otherwise (APPLE-PLACEMENT-ABI-1).inner == 1),sum/mean/max. The kernel gives one thread per row and folds the trailing extent, so an interior axis fails closed in both owners rather than being reordered. gfx1151 keeps its arbitrary-axis support, and a test asserts the Apple bound does not leak to siblings.Evidence: six exact-device configurations (sum/mean/max × rank-2/rank-3) run on Metal with
native_gpuplacement and match NumPy.2. APPLE-DEVICE-EVENT-1 — closed
The GPU matmul route ran through
MPSGraph runWithMTLCommandQueue:, which owns and commits its own command buffer, so no object existed on which to observe a device interval and the wholeapple_gpupacket was forced ontokernel_wall.mpsg_run_bmmnow encodes into an explicitly ownedMPSCommandBufferunder the already-existing sharedMPSGraphTimingBracket— the same one gather/transpose/row-op/BSMM use, so this reuses one implementation rather than adding a second (Decision #31).Measured, not asserted. The recorder no longer hard-codes
device_event_available: Falsewith a reason that had become false. It probes the device interval per family and claimsdevice_eventonly when every family in scope published one; otherwise it falls back tokernel_wallfor all families rather than mixing domains.The sealed packet is re-recorded, not re-stamped. Editing
apple_gpu_runtime.mmcorrectly tripped the packet's source fingerprint, and the fix is a fresh measurement. It now sealsrequired_timing_domains: ["device_event", "end_to_end"], both families stable at 31×80 sampling — the 15×50 default drifted 5.2% against a 4% bar for the device cohort, so the sampling was raised rather than the bar lowered.What the stronger domain immediately revealed
Softmax spends ~23 µs on the GPU inside a millisecond of wall time — these lanes are host-overhead dominated, which
kernel_wallcould not have shown. That is a materially better question to optimize than any kernel timing.Cost check, since a
.mmedit is not freeThe strict route ledger's fingerprint was already invalid before this change (pinned
74eb6e95…vs live9b12af9a…, inert since 2026-07-27), so no live selector evidence was destroyed. End-to-end matmul timing is unchanged (0.95 → 0.89–0.96 ms across runs, inside p90 noise).A test that encoded the limitation
test_both_apple_identities_are_sealed_independentlyasserted both lanes reportkernel_wall. It now asserts each lane pairs exactly one clock withend_to_end, thatapple_cpu(Accelerate — no Metal command buffer) cannot reachdevice_event, and that anydevice_eventclaim is backed by a measured interval.Benchmark — all WS-1 families
Every row native and oracle-checked (Apple7, 20 reps, end-to-end):
apple_gpu_bmm_f32_batch1apple_gpu_simdgroup_gemm_f16apple_softmax_native_libraryapple_gpu_synth_pointwise_reduce_f32apple_gpu_flash_attn_variant_f32apple_gpu_flash_attn_bwd_split_f32_gradsStill
selector_eligible: false— one process, no paired interleaving.Verification (M1 Max / apple7)
env -u TESSERA_OPT): 117 passed, 0 failed.Operational note, because it cost real debugging time
After rebuilding the dylib, ~20 device tests failed with
Apple runtime is missing <symbol>althoughnmshowed the symbol present. The runtime publishes the dylib into a temp cache that stays stale until the next process republishes it. Not a regression — but indistinguishable from one at first glance.Sibling backends
NVIDIA and ROCm are not applicable: every compiler branch is target-guarded to
apple_gpu, and the runtime change is Apple-only. A test asserts gfx1151's arbitrary-axis reduction is unaffected.🤖 Generated with Claude Code