Skip to content

graph_ir/runtime: honour scalar_side so a left-side scalar literal lowers correctly - #589

Merged
gstoner merged 3 commits into
mainfrom
claude/gracious-grothendieck-a2c44b
Aug 19, 2026
Merged

gstoner merged 3 commits into
mainfrom
claude/gracious-grothendieck-a2c44b

Conversation

@gstoner

@gstoner gstoner commented Aug 19, 2026

Copy link
Copy Markdown
Owner

What

_OpExtractor._try_map_binop (graph_ir.py:1946) lifts a literal operand out of a BinOp into the scalar attribute and records which side it came from in scalar_side. Nothing in python/, src/, or tools/ read that attribute — a Decision #29 violation (a declaration with no consumer).

Why it isn't merely dead

The extractor lifts scalars for tessera.sub and tessera.div, not just the commutative add/mul. So 2.0 - x and x - 2.0 emitted IROps identical except for the unread key:

return 2.0 - x   ->  ('tessera.sub', ['%x'], {'scalar': 2.0, 'scalar_side': 'left'})
return x - 2.0   ->  ('tessera.sub', ['%x'], {'scalar': 2.0, 'scalar_side': 'right'})

Every consumer of scalar binds it as the right operand, and _apple_gpu_dispatch_mpsgraph_binary covers ten non-commutative opcodes (sub, div, pow, mod, floor_div, atan2, and all six comparisons):

_apple_gpu_dispatch_mpsgraph_binary(
    "tessera.sub", [np.array([1.,2.,4.])], {"scalar": 2.0, "scalar_side": "left"}, np)
-> [-1,  0,  2]     # x - 2
   [ 1,  0, -2]     # truth for 2 - x

_apple_gpu_execute_artifact (runtime.py:32345) forwards each op's kwargs verbatim to the lane handler, so the path is connected end to end.

Currently latent, not live. These graphs report compile_bundle.executable=False and fall back to eager Python — the correct answers today come from not executing the IR at all.

The fix

The dispatcher honours scalar_side, swapping operands before the Metal/numpy lane split. Absence means "right" — the definition of the scalar= kwarg on the eager op surface that this module's own {"scalar": s} call sites rely on — and any other value raises rather than being guessed (Decision #21).

Audited all scalar consumers; only one needed changing:

Consumer Ops with the scalar form Needs the side?
runtime._apple_gpu_dispatch_mpsgraph_binary 24 opcodes, 10 non-commutative yes — fixed
matmul_pipeline._execute_op add, mul no (commute)
runtime._execute_runtime_cpu_op add, mul no (commute)
runtime._execute_rocm_compiled_binary no (requires two real operands)

Why not delete the attribute instead

That was the cheaper-looking option, and I implemented it first. It fails on its own precondition: the tracer that E2E-REAL-6 puts in _OpExtractor's place cannot express infix scalar binops at all —

2.0 - x          TypeError: unsupported operand type(s) for -: 'float' and 'Tracer'
x - 2.0          TypeError: unsupported operand type(s) for -: 'Tracer' and 'float'
ops.sub(2.0, x)  TesseraTraceError: non-Tracer positional operand

Tracer has no arithmetic dunders and record_op refuses non-Tracer positional operands. There is no successor to hand the case to, and refusing the lift turned y = 2.0 - x as an intermediate from a correct answer into a hard error on every target. When _OpExtractor is retired, dropping scalar_side becomes a mechanical cleanup guarded by the tests here.

Tests

tests/unit/test_binop_scalar_side.py — 75 tests over four sections:

  • Consumer contract — the negative fixture (left-side scalar on every non-commutative opcode), a guard that left and right actually differ per opcode (otherwise the fixtures would pass under a consumer that ignored the side), absent-means-right, and rejection of an unknown value.
  • Producer — the side is stated on both branches for all four binops.
  • Producer→consumer round-trip — the extractor's own kwargs fed into the dispatcher. This is the assertion that would have caught the bug: each half was self-consistent and only the join was wrong.
  • Tracer fails closed — pins that the successor frontend refuses scalar operands outright, so retiring _OpExtractor cannot silently inherit this.

Reverting only the consumer fix fails 23 of the 75.

Verification

Mac (M1 Max), Homebrew python3 3.14.6:

  • pytest tests/unit -m "not slow"13943 passed, 48 failed. All 48 fail identically with these changes stashed — every one RuntimeError: requires a fresh Tessera Apple GPU runtime dylib (no build/ in this worktree). Zero new failures.
  • mypy python/tessera/ → clean, 467 files (ratchet 0).
  • ruff check → clean. scripts/check_generated_docs.sh → 26 docs in sync (test_coverage regenerated via its CLI, not hand-edited).

Limits of this evidence: with no dylib present, the dispatcher tests exercised its numpy fallback rather than the live MPSGraph symbol — the swap happens before that branch so both are covered by construction, but the Metal lane itself is unproven on this run. No lit suite and no ROCm/CUDA lane was run; this change touches neither.

Follow-up

A separate, independent crash surfaced while investigating: JitFn._establish_tracer_authority indexes legacy.functions[0] unguarded, so any function with an unlowerable intermediate statement raises a bare IndexError. Reproducible on main, unrelated to this diff, and shipping as a stacked PR on top of this one.

🤖 Generated with Claude Code

…wers correctly

`_OpExtractor._try_map_binop` lifts a literal operand out of a BinOp into the
`scalar` attribute and records which side it came from in `scalar_side`.
Nothing in python/, src/, or tools/ read that attribute — a Decision #29
violation (a declaration with no consumer).

It was not merely dead. The extractor lifts scalars for `tessera.sub` and
`tessera.div`, not just the commutative `add`/`mul`, so `2.0 - x` and `x - 2.0`
emitted IROps that were identical except for the unread key. Every consumer of
`scalar` binds it as the RIGHT operand, and
`runtime._apple_gpu_dispatch_mpsgraph_binary` covers ten non-commutative
opcodes (sub, div, pow, mod, floor_div, atan2, and all six comparisons). Fed
the extractor's own kwargs it computed `x - 2.0` for both spellings —
sign-flipped for `sub`, reciprocal for `div`, with no diagnostic.
`_apple_gpu_execute_artifact` forwards an op's kwargs verbatim to the lane
handler, so the whole path is connected.

The bug is latent today only because these graphs report
`compile_bundle.executable=False` and fall back to eager Python; the correct
answers come from not executing the IR at all.

Fix: the dispatcher honours `scalar_side`, swapping the operands before the
Metal/numpy lane split. Absence means "right" — the definition of the `scalar=`
kwarg on the eager op surface that this module's own `{"scalar": s}` call sites
rely on — and any other value raises rather than being guessed (Decision #21).
The two commutative-only consumers (`matmul_pipeline._execute_op`,
`runtime._execute_runtime_cpu_op`) implement the scalar form for `add`/`mul`
alone, so operand order cannot change their result; the ROCm binary lane
requires two real operands. No other consumer needed a change.

Deleting the attribute instead was considered and rejected: the tracer that
E2E-REAL-6 puts in `_OpExtractor`'s place cannot express infix scalar binops at
all (`Tracer` has no arithmetic dunders, and `record_op` refuses non-Tracer
positional operands), so there is no successor to hand the case to, and
refusing the lift turned `y = 2.0 - x` as an intermediate from a correct answer
into a hard error on every target.

tests/unit/test_binop_scalar_side.py pins all four sides of the contract: the
negative fixture (left-side scalar across every non-commutative opcode), a
guard that left and right actually differ per opcode, a producer→consumer
round-trip feeding the extractor's own kwargs to the dispatcher — the assertion
that would have caught this, since each half was self-consistent and only the
join was wrong — and the tracer's fail-closed behaviour, so retiring
`_OpExtractor` cannot silently inherit the bug. Reverting just the consumer fix
fails 23 of the 75.

Verified on the Mac (M1 Max): 13943 passed, 48 failed — all 48 fail identically
with these changes stashed, every one `requires a fresh Tessera Apple GPU
runtime dylib` (no build/ in this worktree). mypy clean over 467 files, ruff
clean, generated-doc drift gate in sync. With no dylib present the dispatcher
tests exercised its numpy fallback rather than the live MPSGraph symbol; the
swap happens before that branch so both are covered by construction, but the
Metal lane itself is unproven on this run. No lit suite and no ROCm/CUDA lane
was run — this change touches neither.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b3a19362f8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread python/tessera/runtime.py
gstoner and others added 2 commits August 19, 2026 12:32
…r backends

CI (Linux) failed 18 of the new tests. Root cause is NOT the operand-ordering
fix: on non-Darwin hosts `_load_apple_gpu_runtime` compiles
`runtime/apple_gpu_runtime_stub.cpp`, whose binary switch implements opcodes 0-8
and whose `default:` arm assigns `out[i] = x`. So `mod`(9), `floor_div`(10), the
six comparisons(11-16), and the logical/bitwise ops(17-22) silently return the
LEFT operand there instead of computing anything — including for calls that
carry no `scalar_side` at all, which behave identically on main. The Mac loads
the real MPSGraph symbol, so the first run could not see it.

Consumer tests now run through BOTH dispatcher lanes via a `lane` fixture:

* `host_reference` — forces the symbol lookup to miss, which is the
  dispatcher's own documented fallback. Deterministic on every host, so operand
  ordering stays covered in CI. The swap happens before the lane split, so this
  exercises exactly the code under test.
* `live_kernel` — Darwin-gated, keeping real Metal coverage where the opcode
  table is fully implemented. Gated rather than probed because probing would
  mean asserting the very thing these tests assert; the skip reason names the
  stub gap so it is not mistaken for absent coverage.

Mac: 123 passed (both lanes). Simulated non-Darwin: 75 passed, 48 skipped, 0
failed.

The stub defect is left for its own change — it is pre-existing, independent of
operand ordering, and a Decision #21 violation in its own right (a lowering the
backend cannot carry must diagnose, never silently return an operand).

Also records the AGENTS.md cross-backend assessment this PR owed, under sync key
`SCALAR-SIDE-ORDERING-2026-08-19`:

* apple  — parity validated on Metal; follow-up required for the portable stub.
* nvidia — not applicable; no NVIDIA path consumes the `scalar` kwarg (verified
  by an exhaustive sweep for `get("scalar"`/`["scalar"]`/`get("other"`).
* rocm   — not applicable; `_execute_rocm_compiled_binary` binds both operands
  positionally and raises when fewer are present, so the lifted-scalar form
  cannot reach the gfx1151 lane. Fails closed by construction.
* x86    — not applicable; `_execute_x86_compiled_binary` raises the same way.

No device evidence is produced or claimed for nvidia, rocm, or x86.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…re boundary

`test_apple_test_inventory.py::test_inline_apple_capability_gates_are_globally_inventoryed`
failed: the `lane` fixture's inline `pytest.skip` naming a Darwin host is an
APPLE-TEST-1 inline capability gate, which that ratchet requires be migrated to
the centralized boundary rather than written at the test site.

The `live_kernel` parameter now carries `pytest.mark.hardware_apple_gpu`, so
`tests/conftest.py`'s `require_apple_metal()` owns the gate. Same coverage, no
inline gate: `host_reference` still runs everywhere and carries the operand-
ordering contract in CI.

My error, not a new defect — I pushed the previous commit without re-running the
full suite after restructuring the file, and this reproduces locally.

Full unit sweep on the Mac after this change: 13991 passed, 48 failed, and the
failure set is byte-identical to the pre-existing baseline (all 48 the missing
Apple runtime dylib). Zero new failures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@gstoner

gstoner commented Aug 19, 2026

Copy link
Copy Markdown
Owner Author

Record the scalar-side contract in every backend plan — AGENTS.md:L81-L85

Valid finding, now addressed in 127d454. I had not read AGENTS.md (CLAUDE.md doesn't reference it), so the cross-backend assessment was genuinely missing.

Sync key SCALAR-SIDE-ORDERING-2026-08-19, recorded in all four plans:

Backend Outcome Basis
apple parity validated, + one follow-up required _apple_gpu_dispatch_mpsgraph_binary is the changed consumer. M1 Max with the real tessera_apple_gpu_mpsgraph_binary_f32 symbol; 123 pass across both lanes.
nvidia not applicable No NVIDIA path consumes the scalar kwarg. Verified by an exhaustive sweep for get("scalar" / ["scalar"] / get("other" across python/tessera/ — the only consumers are the Apple dispatcher, _execute_runtime_cpu_op, and matmul_pipeline._execute_op.
rocm not applicable, fails closed by construction _execute_rocm_compiled_binary binds both operands positionally and raises "binary requires two operands (a, b)", so the lifted-scalar form cannot reach the gfx1151 lane.
x86 not applicable, fails closed by construction _execute_x86_compiled_binary raises "binary math requires two operands (a, b)" the same way.

No device evidence is produced or claimed for nvidia, rocm, or x86.

A real defect this surfaced, filed separately

CI red-flagged 18 of the new tests, and the cause was not the ordering fix. On non-Darwin hosts _load_apple_gpu_runtime compiles apple_gpu_runtime_stub.cpp, whose binary switch implements opcodes 0–8 and whose default: arm is:

default: out[i] = x; break;

So mod(9), floor_div(10), the six comparisons(11–16), and the logical/bitwise ops(17–22) silently return the left operand on every non-Apple host — including for calls carrying no scalar_side at all, which behave identically on main. Because the symbol exists, the dispatcher takes the kernel branch instead of its numpy fallback, so the wrong values come back as if computed.

That is a Decision #21 violation in its own right, pre-existing, and independent of operand ordering. It is not fixed here — it deserves its own change with its own backend assessment. The follow-up is recorded in docs/audit/backend/apple/todo.md.

Consumer tests now run through both dispatcher lanes: host_reference (forces the documented fallback; deterministic on every host, so ordering stays covered in CI) and live_kernel (carries hardware_apple_gpu, so require_apple_metal() owns the gate per APPLE-TEST-1). Mac: 123 passed both lanes. Simulated non-Darwin: 75 passed, 48 skipped, 0 failed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant