Skip to content

QVAC-17095 feat: add Whisper performance benchmark workflows#1690

Merged
GustavoA1604 merged 2 commits into
tetherto:mainfrom
ogad-tether:qvac-17095-whisper-performance-benchmarks
Apr 22, 2026
Merged

QVAC-17095 feat: add Whisper performance benchmark workflows#1690
GustavoA1604 merged 2 commits into
tetherto:mainfrom
ogad-tether:qvac-17095-whisper-performance-benchmarks

Conversation

@ogad-tether

@ogad-tether ogad-tether commented Apr 21, 2026

Copy link
Copy Markdown
Contributor

Note: be concise and prefer bullet points.

🎯 What problem does this PR solve?

  • Adds a Whisper desktop RTF benchmarking path across the supported CI runner matrix so performance findings can be collected per backend and runner.
  • Ensures manual benchmark and benchmark-workflow runs can exercise the branch addon/prebuilds instead of silently falling back to the published package.

📝 How does it solve it?

  • Adds a desktop Whisper RTF benchmark harness, matrix runner, and aggregation tooling that emit structured JSON artifacts for cross-runner comparison.
  • Updates the desktop Whisper workflow and adds a dedicated benchmark-orchestration workflow to run prebuilds, benchmark-only desktop E2E paths, and publish one consolidated findings report.

🧪 How was it tested?

  • npx standard "packages/qvac-lib-infer-whispercpp/test/benchmark/rtf-benchmark.test.js" "packages/qvac-lib-infer-whispercpp/scripts/run-rtf-benchmark-matrix.js" "scripts/perf-report/aggregate-whisper-rtf.js"
  • node --check packages/qvac-lib-infer-whispercpp/test/benchmark/rtf-benchmark.test.js
  • node --check packages/qvac-lib-infer-whispercpp/scripts/run-rtf-benchmark-matrix.js
  • node --check scripts/perf-report/aggregate-whisper-rtf.js
  • ruby -e "require 'yaml'; YAML.load_file('.github/workflows/integration-test-qvac-lib-infer-whispercpp.yml'); YAML.load_file('.github/workflows/benchmark-performance-qvac-lib-infer-whispercpp.yml'); YAML.load_file('.github/workflows/benchmark-qvac-lib-infer-whispercpp.yml'); puts 'ok'"
  • Successful desktop benchmark run: https://github.com/tetherto/qvac/actions/runs/24777285232
  • Full upstream verification rerun on reduced branch: https://github.com/tetherto/qvac/actions/runs/24785460222/

📊 Desktop Benchmark Results

Desktop results below come from the successful benchmark run:

Device Platform Model GPU Backend Mean RTF P50 P95 Mean Wall (ms) Tokens/s
ai-run-linux-gpu linux-x64 ggml-tiny cpu cpu 0.0648 0.0647 0.0651 1307 52.1
ai-run-linux-gpu linux-x64 ggml-tiny gpu cuda 0.0647 0.0647 0.0648 1305 52.2
macos-14-xlarge darwin-arm64 ggml-tiny cpu cpu 0.0254 0.0252 0.0261 512 133.2
macos-14-xlarge darwin-arm64 ggml-tiny gpu coreml 0.1001 0.1000 0.1020 2016 33.8
macos-15-large darwin-x64 ggml-tiny cpu cpu 0.0639 0.0636 0.0674 1288 53.0
macos-15-large darwin-x64 ggml-tiny gpu coreml 1.2256 1.1552 1.3663 24679 2.8
ubuntu-22.04 linux-x64 ggml-tiny cpu cpu 0.0521 0.0472 0.0677 1051 66.9
ubuntu-24.04-arm linux-arm64 ggml-tiny cpu cpu 0.1392 0.1392 0.1392 2803 24.3
windows-2022 win32-x64 ggml-tiny cpu cpu 0.0483 0.0485 0.0498 975 69.9
windows-2022 win32-x64 ggml-tiny gpu directml 0.0476 0.0463 0.0511 960 71.2

@ogad-tether ogad-tether requested review from a team as code owners April 21, 2026 12:27
@ogad-tether ogad-tether self-assigned this Apr 21, 2026
@ogad-tether ogad-tether force-pushed the qvac-17095-whisper-performance-benchmarks branch from 380a91b to 24f59a3 Compare April 21, 2026 15:46
@GustavoA1604

Copy link
Copy Markdown
Contributor

/review

@github-actions

Copy link
Copy Markdown
Contributor

Tier-based Approval Status

**PR Tier:** TIER1

**Current Status:** ✅ APPROVED

**Requirements:**
- 1 Team Member approval ✅ (1/1)
- 1 Team Lead OR Management approval ✅ (1/1)



---
*This comment is automatically updated when reviews change.*

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants