Skip to content

examples/advanced/mla: rework to drive the shipped Apple GPU MLA decode - #40

Merged
gstoner merged 1 commit into
mainfrom
examples-mla-rework
May 30, 2026
Merged

gstoner merged 1 commit into
mainfrom
examples-mla-rework

Conversation

@gstoner

@gstoner gstoner commented May 30, 2026

Copy link
Copy Markdown
Owner

Reworks the FlashMLA example to drive the MLA decode work that landed in the
runtime
, cross-checked against a numpy reference. Runs on the GPU on Apple
Silicon; numpy fallback elsewhere.

What

mla.run_gpu_decode_demo(cfg) exercises, from this example's own config:

  1. Weight absorption (runtime._apple_gpu_mla_absorb_decode) — verified
    numerically identical to the explicit decoupled-RoPE decode (the core MLA
    identity).
  2. Paged single-sequence decode (tessera.cache.MLAPagedDecoder).
  3. GPU-resident multi-step decode loop (tessera.cache.ResidentMLADecoder) —
    weights once, one command buffer/step, token-only readback.
  4. Concurrent block-paged serving (tessera.cache.MLABlockPagedCache).

The smoke now prints (on Apple Silicon):

OK mla tiny: (2, 8, 64) kv_reduction 0.75 apple_cpu cpu_accelerate
OK mla gpu-decode: metal absorbed==explicit True paged==ref True block_paged==ref True resident_tokens 4 kv_cache_ratio 7.2x

Files

  • mla/gpu_decode.py (new) — the demo + numpy references, returns GPUDecodeSummary.
  • mla/__init__.py, tests/smoke_random.py, README.md — wired + documented.
  • tests/unit/test_example_mla_gpu_decode.py (new) — regression coverage in the
    main suite (2 tests).

Verification (local, Apple Silicon)

  • example smoke: both lines OK; unit test: 2/2 pass
  • mypy ratchet: clean (only the pre-existing environmental torch-import error)

CI on this repo is uniformly red on main (Python 3.8–3.11 matrix, missing
optional deps) — same state PRs #17#39 merged through. The local signal above
is green.

🤖 Generated with Claude Code

Wires the FlashMLA example to the MLA decode surfaces that landed in the runtime,
cross-checked against a numpy reference (runs on the GPU on Apple Silicon, numpy
fallback elsewhere).

- mla/gpu_decode.py: run_gpu_decode_demo(cfg) exercises, from this example's
  config: (1) weight absorption (runtime._apple_gpu_mla_absorb_decode) verified
  numerically identical to the explicit decoupled-RoPE path; (2) paged
  single-sequence decode (cache.MLAPagedDecoder); (3) the GPU-resident multi-step
  decode loop (cache.ResidentMLADecoder); (4) concurrent block-paged serving
  (cache.MLABlockPagedCache). Returns a GPUDecodeSummary.
- mla/__init__.py: export run_gpu_decode_demo + GPUDecodeSummary.
- tests/smoke_random.py: prints + asserts the gpu-decode demo (absorbed==explicit,
  paged==ref, block_paged==ref, 4 resident tokens, kv_cache_ratio).
- README.md: documents the shipped Apple GPU MLA decode demo.
- tests/unit/test_example_mla_gpu_decode.py: regression coverage in the main suite.

On Apple Silicon the smoke now prints:
  OK mla gpu-decode: metal absorbed==explicit True paged==ref True
  block_paged==ref True resident_tokens 4 kv_cache_ratio 7.2x

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@gstoner
gstoner merged commit 4986808 into main May 30, 2026
7 of 25 checks passed
@gstoner
gstoner deleted the examples-mla-rework branch May 30, 2026 17:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant