examples/advanced/mla: rework to drive the shipped Apple GPU MLA decode - #40
Merged
Merged
Conversation
Wires the FlashMLA example to the MLA decode surfaces that landed in the runtime, cross-checked against a numpy reference (runs on the GPU on Apple Silicon, numpy fallback elsewhere). - mla/gpu_decode.py: run_gpu_decode_demo(cfg) exercises, from this example's config: (1) weight absorption (runtime._apple_gpu_mla_absorb_decode) verified numerically identical to the explicit decoupled-RoPE path; (2) paged single-sequence decode (cache.MLAPagedDecoder); (3) the GPU-resident multi-step decode loop (cache.ResidentMLADecoder); (4) concurrent block-paged serving (cache.MLABlockPagedCache). Returns a GPUDecodeSummary. - mla/__init__.py: export run_gpu_decode_demo + GPUDecodeSummary. - tests/smoke_random.py: prints + asserts the gpu-decode demo (absorbed==explicit, paged==ref, block_paged==ref, 4 resident tokens, kv_cache_ratio). - README.md: documents the shipped Apple GPU MLA decode demo. - tests/unit/test_example_mla_gpu_decode.py: regression coverage in the main suite. On Apple Silicon the smoke now prints: OK mla gpu-decode: metal absorbed==explicit True paged==ref True block_paged==ref True resident_tokens 4 kv_cache_ratio 7.2x Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reworks the FlashMLA example to drive the MLA decode work that landed in the
runtime, cross-checked against a numpy reference. Runs on the GPU on Apple
Silicon; numpy fallback elsewhere.
What
mla.run_gpu_decode_demo(cfg)exercises, from this example's own config:runtime._apple_gpu_mla_absorb_decode) — verifiednumerically identical to the explicit decoupled-RoPE decode (the core MLA
identity).
tessera.cache.MLAPagedDecoder).tessera.cache.ResidentMLADecoder) —weights once, one command buffer/step, token-only readback.
tessera.cache.MLABlockPagedCache).The smoke now prints (on Apple Silicon):
Files
mla/gpu_decode.py(new) — the demo + numpy references, returnsGPUDecodeSummary.mla/__init__.py,tests/smoke_random.py,README.md— wired + documented.tests/unit/test_example_mla_gpu_decode.py(new) — regression coverage in themain suite (2 tests).
Verification (local, Apple Silicon)
torch-import error)🤖 Generated with Claude Code