Skip to content

QVAC-17801 feat: add mobile Parakeet RTF reporting#1723

Merged
GustavoA1604 merged 17 commits into
mainfrom
feat/parakeet-mobile-rtf-report
Apr 29, 2026
Merged

QVAC-17801 feat: add mobile Parakeet RTF reporting#1723
GustavoA1604 merged 17 commits into
mainfrom
feat/parakeet-mobile-rtf-report

Conversation

@ogad-tether

@ogad-tether ogad-tether commented Apr 23, 2026

Copy link
Copy Markdown
Contributor

🎯 What problem does this PR solve?

  • Parakeet performance reporting needed a mobile Device Farm path that uses the shared QVAC perf-report pipeline.
  • The earlier branch-local mobile RTF extractor was brittle and did not match the working mobile workflow pattern.
  • Desktop and mobile performance data should be visible together in CI artifacts so cross-platform regressions are easier to review.

📝 How does it solve it?

  • Wires Parakeet mobile tests into scripts/test-utils/performance-reporter.js.
  • Emits [PERF_REPORT_START] / [PERF_CHUNK] markers from mobile Device Farm logs and extracts them with scripts/perf-report/extract-from-log.js.
  • Renders mobile summaries with scripts/perf-report/render-step-summary.js.
  • Extends scripts/perf-report/aggregate-parakeet-rtf.js so the unified Parakeet report can include desktop RTF artifacts and mobile performance-report.json artifacts.
  • Adds a unified report job that uploads parakeet-unified-performance-report.{md,json,html}.
  • Keeps Android on the broader mobile model coverage, and quarantines only iOS Sortformer GPU after Device Farm showed that case backgrounds/crashes the app.
  • Addresses PR bot findings by exporting generated mobile wrappers and checking out trusted workflow code in the privileged pull_request_target combine job.

🧪 How was it tested?

✅ Validation Matrix

Area Matrix / Device(s) Result Evidence
Focused mobile workflow Android + iOS Device Farm ✅ Passed run 25103627279
Android mobile Samsung Galaxy S25 Ultra, Google Pixel 9 ✅ Passed perf-report-parakeet-Android-56, console-logs-parakeet-Android, devicefarm-logs-parakeet-Android
iOS mobile Apple iPhone 16 Pro, Apple iPhone 17 Pro ✅ Passed perf-report-parakeet-iOS-56, console-logs-parakeet-iOS, devicefarm-logs-parakeet-iOS
Desktop integration / RTF Linux, macOS, Windows matrix ✅ Passed in full PR validation rtf-results-* artifacts
Unified report generation Desktop + mobile artifact aggregation ✅ Passed in On PR workflow parakeet-unified-performance-report

📊 Latest Mobile Perf Report Check

Platform Device Coverage in report Required metrics present Notes
Android Samsung Galaxy S25 Ultra CTC CPU/GPU, EOU CPU/GPU, Sortformer CPU/GPU, TDT CPU/GPU ✅ RTF, wall, TPS, encoder, decoder, audio perf-report-parakeet-Android-56
Android Google Pixel 9 CTC CPU/GPU, EOU CPU/GPU, Sortformer CPU/GPU ✅ RTF, wall, TPS, encoder, decoder, audio TDT rows were not present for this device artifact
iOS Apple iPhone 16 Pro CTC CPU/GPU, EOU CPU/GPU, Sortformer CPU, TDT CPU/GPU ✅ RTF, wall, TPS, encoder, decoder, audio iOS Sortformer GPU quarantined
iOS Apple iPhone 17 Pro CTC CPU/GPU, EOU CPU/GPU, Sortformer CPU, TDT CPU/GPU ✅ RTF, wall, TPS, encoder, decoder, audio iOS Sortformer GPU quarantined

Latest Sample Metrics

Platform Device Case Avg RTF Avg TPS
Android Samsung Galaxy S25 Ultra TDT GPU 0.101 28.07
Android Samsung Galaxy S25 Ultra CTC GPU 0.090 31.32
Android Samsung Galaxy S25 Ultra EOU GPU 0.166 8.38
Android Samsung Galaxy S25 Ultra Sortformer GPU 0.072 n/a
Android Google Pixel 9 CTC GPU 0.131 21.60
Android Google Pixel 9 EOU GPU 0.186 7.49
Android Google Pixel 9 Sortformer GPU 0.095 n/a
iOS Apple iPhone 17 Pro TDT GPU 0.100 28.27
iOS Apple iPhone 17 Pro CTC GPU 0.097 29.10
iOS Apple iPhone 17 Pro EOU GPU 0.318 4.54
iOS Apple iPhone 17 Pro Sortformer CPU 0.079 n/a
iOS Apple iPhone 16 Pro TDT GPU 0.159 17.83
iOS Apple iPhone 16 Pro CTC GPU 0.143 19.82
iOS Apple iPhone 16 Pro EOU GPU 0.415 3.47
iOS Apple iPhone 16 Pro Sortformer CPU 0.122 n/a

Made with Cursor

Run Parakeet RTF benchmarks through the mobile Device Farm workflow and combine desktop and mobile artifacts into a single report so cross-platform performance is visible in one place.

Made-with: Cursor
@ogad-tether ogad-tether requested review from a team as code owners April 23, 2026 11:38
Comment thread packages/qvac-lib-infer-parakeet/test/mobile/rtf-benchmark.cjs Fixed
@github-actions

github-actions Bot commented Apr 23, 2026

Copy link
Copy Markdown
Contributor

Tier-based Approval Status

**PR Tier:** TIER1

**Current Status:** ✅ APPROVED

**Requirements:**
- 1 Team Member approval ✅ (1/1)
- 1 Team Lead OR Management approval ✅ (1/1)



---
*This comment is automatically updated when reviews change.*

@ogad-tether ogad-tether self-assigned this Apr 23, 2026
@ogad-tether ogad-tether changed the title feat: add mobile Parakeet RTF reporting QVAC-17801 feat: add mobile Parakeet RTF reporting Apr 23, 2026
Mirror OCR's mobile reporting approach by isolating the RTF benchmark into dedicated Device Farm perf runs while keeping the regular mobile suite separate, then only extract benchmark artifacts from the perf lane.

Made-with: Cursor
Replace bash mapfile usage with portable while-read loops for macOS runners and refresh AWS credentials before the long Device Farm monitor and log download phases so the split perf/regular workflow can run to completion.

Made-with: Cursor
Make the addon-side mobile wrappers read testFilter.txt and skip non-selected tests with zero-failure summaries so the perf and regular Device Farm lanes can actually execute different subsets without requiring framework changes.

Made-with: Cursor
Comment thread packages/qvac-lib-infer-parakeet/test/mobile/rtf-benchmark.cjs Fixed
@kinsta

kinsta Bot commented Apr 28, 2026

Copy link
Copy Markdown

Preview deployments for qvac-docs-staging ⚡️

Status Branch preview Commit preview
✅ Ready Visit preview Visit preview

Commit: e1d112780c0538dd76fc309ccbb08a02f1d23506

Deployment ID: 60a09812-bd03-4a6f-858b-284fc74ad388

Static site name: qvac-docs-staging-fazwv

Comment thread packages/qvac-lib-infer-parakeet/test/mobile/integration.auto.cjs Fixed
Comment thread .github/workflows/on-pr-qvac-lib-infer-parakeet.yml Fixed
Comment thread packages/qvac-lib-infer-parakeet/test/mobile/integration.auto.cjs Fixed
Comment thread packages/qvac-lib-infer-parakeet/test/mobile/integration.auto.cjs Fixed
Comment thread packages/qvac-lib-infer-parakeet/test/mobile/integration.auto.cjs Fixed
Comment thread packages/qvac-lib-infer-parakeet/test/mobile/integration.auto.cjs Fixed
Comment thread packages/qvac-lib-infer-parakeet/test/mobile/integration.auto.cjs Fixed
Comment thread packages/qvac-lib-infer-parakeet/test/mobile/integration.auto.cjs Fixed
@ogad-tether

Copy link
Copy Markdown
Contributor Author

/review

@GustavoA1604

Copy link
Copy Markdown
Contributor

/review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants