Gumiho #3: speculative-decoding throughput/latency benchmark (standard JSON)

[gstoner]

Builds out item #3 — a wall-clock throughput/latency benchmark for the Gumiho example, emitting the stable benchmark_gemm.py JSON schema so tools/roofline_tools/ can ingest it.

benchmarks/apple_gpu/benchmark_gumiho.py

Two op groups, both standard-schema rows (with extra decode-specific fields):

op="gumiho_decode" — algorithmic speculative speedup. Tokens committed per target forward pass for vanilla (1.0), speculative_untrained, speculative_trained. This is the model-size-independent win (rows carry tokens_per_step, speedup_vs_vanilla, mean_accepted_length, tokens_per_sec).

op="gumiho_serial_draft" — resident vs per-op wall-clock. The serial draft run GPU-resident (one command buffer/token) vs the per-op path (one dispatch + sync per op); speedup_vs_per_op from removing the per-op CPU↔GPU sync.

Recent M-series run:

row	metric
gumiho_decode vanilla	1.0 tokens/pass, 58.7 tok/s
gumiho_decode speculative_untrained	1.86× tokens/pass
gumiho_decode speculative_trained	4.33× tokens/pass, 136.9 tok/s (2.3× wall-clock — 4.3 tokens/step amortizes the verify)
gumiho_serial_draft resident vs per_op	6.04 ms vs 15.83 ms = 2.62× wall-clock

Honest framing: the decode tokens_per_step ratio is the real, model-size-independent speculative win; at toy scale the draft isn't cheaper than the target, but here the trained run still wins wall-clock because acceptance is high. The resident-vs-per-op win is real even at toy scale (per-op sync dominates). Headline numbers drift across machines/toolchains — the schema is the stable contract.

Tests

tests/unit/test_benchmark_gumiho.py (2): validates the JSON schema + the expected op/mode rows, and that trained > vanilla tokens/pass. Best-effort driver — --target numpy runs anywhere; Metal modes skip off Darwin. ruff + lint_docs clean.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0707224972

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Avoid probing Metal for numpy-only benchmark runs

When the benchmark is run on non-Darwin CI with --target numpy (as the new smoke test and README promise), this still constructs ResidentSerialDraft, whose constructor calls DeviceTensor.is_metal() and loads/builds the Apple GPU runtime before falling back. In environments without a working C++ toolchain or runtime stub this makes the numpy path fail instead of emitting JSON, even though the non-Darwin skip only excludes non-numpy targets. Guard this resident/per-op section for numpy/non-Metal hosts or use a pure host path that does not probe the Metal runtime.

Useful? React with 👍 / 👎.