chore[skiplog]: Qwen3.5 perf benchmark suite (reasoning-budget, ppTPS, desktop + mobile)

[donriddo]

🎯 What problem does this PR solve?

The WB team needs throughput numbers (TTFT, TPS, ppTPS) for Qwen3.5-0.8B and 2B across quantizations Q4_0, Q4_1, Q4_K_M, Q6_K, Q8_0 and reasoning-budget -1/0, on both desktop and mobile including KV-cache types on mobile — plus the ability to catch regressions between addon versions.

📝 How does it solve it?

Coverage

• Models: Qwen3.5-0.8B + 2B (5 quants each), keep Qwen3-1.7B as a desktop comparison baseline, drop Qwen3-4B. No PyTorch.
• Reasoning budget -1 and 0; single ~512-token prompt (verified at ~518 templated tokens against the Qwen3.5 tokenizer).
• Mobile KV-cache types f16, q8_0, q4_0, plus TurboQuant/PolarQuant (tbq3_0/pq3_0, tbq4_0/pq4_0, pq3_0, pq4_0); desktop runs GPU, mobile runs both gpu and cpu.

Report — unified renderer (render-report.js), one identical table per device (desktop + 5 mobile):

• Columns: TTFT (ms) · TPS · ppTPS · Tokens, each as mean ± stddev across repeats (desktop 5, mobile 3).
• Header records addon version, prompt size, repeats, and the detected desktop GPU — version + GPU are stamped into the run's artifacts so they're accurate and survive a later re-render.
• Crashed rows for unsupported combos (e.g. quantized KV cache on Adreno GPUs, or TurboQuant/PolarQuant on iOS Metal and Samsung GPU — run anyway, detected, reported).
• Best configuration per device (highest TPS, highest ppTPS).

Cross-run comparison (regression detection)

• summarize_only re-renders a previous run's report in ~1 min, skipping the ~6h benchmarks.
• compare_run_id adds Δ TTFT / TPS / ppTPS columns vs a baseline run (downloads both runs' artifacts; no re-run needed). The baseline's version is read from its stamp, so the comparison is never mislabelled.

Mobile execution

• Sharded one group per (model × KV-cache type) = 70 shards (2 sizes × 5 quants × 7 KV-cache types), run as 7 sequential KV-cache batches to fit the Device Farm per-test ceiling and avoid pool/disk exhaustion. 3 measured repetitions per config.
• The 70 shard files and the workflow's test_groups are generated from one source of truth (test/integration/_benchmark-matrix.js) and are not committed. CI regenerates them before the Device Farm bundle and hard-fails if any are missing or have drifted from the matrix, so the benchmark can never run against a stale or partial shard set.
• Deliberately absent from test-groups.json; scheduled only via the workflow's test_groups override.

Workflow inputs (no per-run configurability of the matrix — it's fixed in the scripts):
ref, run_desktop, run_mobile, summarize_only, artifact_run_id, compare_run_id. The shared integration-mobile-test-llm-llamacpp.yml gains two additive optional inputs (job_timeout_minutes default 120, artifact_suffix default empty) — backward-compatible for other addon callers.

🧪 How was it tested?

• npx standard clean; validate-mobile-tests.js in sync; verify:benchmark-shards confirms the matrix, the generated integration.auto.cjs (shard-file refs and run-function names), and the workflow test_groups are all in lockstep, so a generator change can't silently desync the Device Farm grep.
• Generation pipeline verified locally: from a fresh checkout (shards absent) test:integration:generate regenerates everything with zero drift in the committed integration.auto.cjs; the mobile-only benchmark shards skip cleanly on desktop.
• Validated end-to-end across every input combination with real runs (full, desktop-only, mobile-only, re-render, comparison).
  • Mobile run on the 3-KV-cache matrix (pre-TurboQuant, 30 shards) — all 6 batches green on 5 real devices: https://github.com/tetherto/qvac/actions/runs/27231785449 — 594 data rows at mobile=3, mean ± stddev, tokens, best-config per device; 0 rows with anomalous variance; 83 Crashed rows all on the Adreno quantized-KV path (78/83 are gpu + quantized).
  • Full 70-shard run including the TurboQuant/PolarQuant KV-cache types and the Pixel 10 device is in progress: https://github.com/tetherto/qvac/actions/runs/27292978920 — the TurboQuant batches are expected to show Crashed on iOS (Metal) and Samsung GPU, since those kernels are Vulkan + CPU only.
  • Desktop sweep (Desktop (Tesla T4), desktop=5) validated in the earlier full run: https://github.com/tetherto/qvac/actions/runs/27183153829.
  • Per-batch wall-clock ran 54–114 min under the 180-min cap; the 7 KV-cache batches run sequentially.

💥 Known findings from the runs (data, not code issues)

• Adreno GPU (Samsung S25/S26) crashes on all gpu + kv=q4_0 and gpu + kv=q8_0 — confirmed and reported as Crashed. CPU path handles quantized KV fine.
• Mobile thermal throttling: on some mobile configs successive repeats get slower (e.g. ppTPS 850 -> 492 -> 428 across 3 reps), which widens the ± stddev on those rows. This is genuine sustained-load throttling on real devices, not measurement error — the stddev reflects it honestly.
• Pixel 9 Pro GPU TTFT/ppTPS are notably weaker than the other devices across quants, consistent with a Vulkan/driver characteristic; CPU results are plausible.

📦 Notes

• Benchmark/test infrastructure only — no addon index.js/native or public-API change, so no version bump or CHANGELOG entry ([skiplog]).
• Pairs with #2382 (workflow infra, already merged).

[gianni-cor]

just for mobile, can you run the bench on both CPU and GPU?

[github-actions]

Tier-based Approval Status

**PR Tier:** TIER1

**Current Status:** ❌ PENDING

**Requirements:**
- 1 Team Member approval ✅ (1/1)
- 1 Team Lead OR Management approval ❌ (0/1)



---
*This comment is automatically updated when reviews change.*

[donriddo]

just for mobile, can you run the bench on both CPU and GPU?

Already does. mobile.config.json sets "devices": ["gpu", "cpu"] and benchmark-perf.test.js loops over both, so each model and quant runs on CPU and GPU.

[donriddo]

Would be possible, in addition of the raw table results of the .md file for the consolidated report, to produce a rendered .html file with graphs to compare different categories (e.g. avg/stdev bars graph with comparing KV cache types and aggregating other matrix parameters/devices, then other graph comparing weight quantization and aggregating other parameters, etc.). That would be easier to inspect quickly.

Maybe to choose a meaningful configuration, show the graphs per device or when keeping other parameters on their default values. E.g. a graph showing how tokens per second vary when varying KV cache type but keeping all other parameters as default for S25, etc.

The point I want to make is that a compact graphical representation of the results would be easy for us to re-view quickly, rather than having to compare hundreds of raw entries of the .md table.

You have a point. Let me see how easy this would be to do. If not, it can come as a follow up.

[jesusmb1995]

Only possible .html report missing, maybe can be done in follow up.

[donriddo]

/review