[Parakeet] QVAC-13814 feat: add automated benchmarks for parakeet ctc, eou and sortformer models by sharmaraju352 · Pull Request #991 · tetherto/qvac

sharmaraju352 · 2026-03-19T06:36:35Z

Summary

Add per-model benchmark config files (config-ctc.yaml, config-eou.yaml, config-sortformer.yaml) with model-appropriate defaults (timeouts, streaming mode, WER/CER toggles)
Update the CI workflow to support an all option that benchmarks every model type in a single matrix run
Add trigger-benchmark.sh for convenient local invocation of benchmark workflows via gh CLI
Fix benchmark server to pass named model paths (ctcModelPath, eouEncoderPath, sortformerPath) at config top level as required by the addon

Successful Workflow Runs

All three new model types verified end-to-end from this branch:

Model Type	Jobs	Run	Status
CTC	cpu-batch, gpu-batch, cpu-streaming	#23284309748	✅ Success
EOU	cpu-streaming, gpu-streaming, cpu-batch	#23284559433	✅ Success
Sortformer	cpu-batch, gpu-batch	#23284886683	✅ Success

Details

Config files

Config	Model Type	Key Differences
`config-ctc.yaml`	CTC	60s timeout, WER/CER enabled, batch mode
`config-eou.yaml`	EOU	120s timeout, WER/CER enabled, streaming mode enabled
`config-sortformer.yaml`	Sortformer	180s timeout, WER/CER disabled (diarization model), batch mode

Workflow changes

Added all to model_type dropdown (runs TDT + CTC + EOU + Sortformer in one matrix = 14 parallel jobs)
Model-specific test matrices:
- CTC: English CPU batch, GPU batch, CPU streaming (3 jobs)
- EOU: English CPU streaming, GPU streaming, CPU batch (3 jobs)
- Sortformer: English CPU batch, GPU batch (2 jobs, no streaming/multilingual)
- TDT: Full suite including multilingual FLEURS (6 jobs, unchanged)
Config step now selects the correct model-specific config template per job
Prebuilds step is non-fatal; falls back to published npm package when CI prebuilds unavailable
Python pinned to 3.13 (3.14 breaks datasets library Pickler)

Benchmark server changes

Added getNamedPaths() to resolve model-type-specific file paths (ctcModelPath, eouEncoderPath, sortformerPath, etc.)
Named paths spread at config top level as required by the addon's _hasNamedPaths() / activate() interface
Fixed sortformer model download URL to use public cgus community HuggingFace repo

Trigger script

trigger-benchmark.sh — Trigger benchmark for a single model type or all types (-t ctc|eou|sortformer|tdt|all), with -m max samples, -W watch, -b branch options

Test plan

Validated all config YAML files parse correctly against Pydantic config schema
Validated workflow YAML structure (triggers, matrix generation, step references)
Tested matrix generation produces correct job variants for each model_type input
Verified trigger script passes bash syntax check and --help works
Ran existing pytest test_config.py suite — all 7 tests pass
CTC benchmark: 3/3 jobs passed (run)
EOU benchmark: 3/3 jobs passed (run)
Sortformer benchmark: 2/2 jobs passed (run)

…odels Add per-model benchmark config files (config-ctc.yaml, config-eou.yaml, config-sortformer.yaml) with appropriate defaults for each model type. Update the CI workflow to support an 'all' option that runs benchmarks for every model type in a single matrix, and add a weekly schedule trigger (Sunday 04:00 UTC) for automated regression benchmarking. Add trigger scripts (trigger-benchmark.sh, trigger-benchmark-all.sh) for convenient local invocation of benchmark workflows via gh CLI. Made-with: Cursor

When CI prebuilds are not available (no successful prebuilds workflow run), fall back to installing @qvac/transcription-parakeet from npm instead of failing the entire benchmark job. Made-with: Cursor

Python 3.14 changed Pickler._batch_setitems() signature which breaks the datasets library. Pin to 3.13 until upstream compatibility is fixed. Made-with: Cursor

The addon requires model-type-specific named paths (e.g. ctcModelPath, eouEncoderPath, sortformerPath) when activating non-TDT models. Add getNamedPaths() that resolves the correct file paths per model type and spreads them into the parakeetConfig passed to the addon constructor. Made-with: Cursor

The addon reads ctcModelPath/eouEncoderPath/sortformerPath from the top-level config object (this._config), not from parakeetConfig. Made-with: Cursor

The tetherto/sortformer-4spk-v2-onnx HuggingFace repo is gated and returns an invalid file. Use the public cgus community repo that the integration tests already rely on. Made-with: Cursor

trigger-benchmark.sh already supports -t all, making the separate trigger-benchmark-all.sh unnecessary. Made-with: Cursor

github-actions · 2026-03-19T08:37:15Z

Tier-based Approval Status

**PR Tier:** TIER1

**Current Status:** ✅ APPROVED

**Requirements:**
- 1 Team Member approval ✅ (1/1)
- 1 Team Lead OR Management approval ✅ (1/1)



---
*This comment is automatically updated when reviews change.*

Per review feedback — "automated" means triggered via workflow_dispatch, not periodic autonomous runs. Made-with: Cursor

…r script - Change MODEL_TYPE fallback from 'all' to 'tdt' to match the workflow_dispatch UI default - Replace unreachable $? check (dead code under set -e) with proper if-not construct in trigger-benchmark.sh Made-with: Cursor

ogad-tether

LGTM. Splitting configs per model type, wiring named paths into the benchmark server, and the matrix '''all''' option are all clear wins. Python 3.13 pin for the benchmark workflow is a sensible fix for the datasets stack.

ogad-tether · 2026-03-23T14:54:33Z

Sortformer weights now pull from the cgus Hugging Face repo — worth a one-line note in the benchmark README or PR description (license/provenance, and whether you plan to mirror under tetherto later) so future readers know that dependency is intentional.

sharmaraju352 · 2026-03-24T09:59:13Z

/review

sharmaraju352 · 2026-03-24T10:10:38Z

/review

* fix: statically link parakeet prebuilds Made-with: Cursor * fix: restore parakeet linux runtime loading Made-with: Cursor * fix: address parakeet apple prebuild failures Made-with: Cursor * chore: remove parakeet release notes file Made-with: Cursor * fix: use static requires for mobile bare-pack bundling The _resolve() helper used computed require paths that bare-pack could not statically trace, so the addon modules were missing from the mobile bundle. Use static string literals for mobile paths (traced by bare-pack) and variable paths for desktop (skipped by bare-pack since ../../ doesn't exist in the mobile layout). Made-with: Cursor * feat[notask]: add download profiler for registry blob performance diagnostics (#1040) * feat[notask]: add download profiler for registry blob performance diagnostics Made-with: Cursor * fix: move profiler deps from devDependencies to dependencies Made-with: Cursor * doc: add profile command and example to client README Made-with: Cursor * fix: show full peer keys in profiler output for troubleshooting Made-with: Cursor * fix: validate parseInt results for interval and timeout CLI flags Made-with: Cursor --------- Co-authored-by: Proletter <40578159+Proletter@users.noreply.github.com> Co-authored-by: Simon Iribarren <simon.ig13@gmail.com> * fix: resolve dependabot alerts for registry-server transitive deps (#1093) * fix(registry-server): PBKDF2 for passphrase-derived keys (CodeQL #9) (#1065) * fix(registry-server): derive passphrase keys with PBKDF2 Replace single-pass SHA-256 with PBKDF2-HMAC-SHA256 (310k iterations) for deterministic test keys; addresses CodeQL js/insufficient-password-hash. * chore(registry-server): remove passphrase migration note from guide --------- Co-authored-by: Proletter <40578159+Proletter@users.noreply.github.com> * fix[notask]: lazy-load Node builtins in profiler for Bare runtime compatibility (#1096) * fix[notask]: sanitize SSE output to prevent reflected XSS (#1027) Co-authored-by: Marco <1369747+elchiapp@users.noreply.github.com> * [Parakeet] QVAC-13814 feat: add automated benchmarks for parakeet ctc, eou and sortformer models (#991) * feat: add automated benchmarks for parakeet ctc, eou and sortformer models Add per-model benchmark config files (config-ctc.yaml, config-eou.yaml, config-sortformer.yaml) with appropriate defaults for each model type. Update the CI workflow to support an 'all' option that runs benchmarks for every model type in a single matrix, and add a weekly schedule trigger (Sunday 04:00 UTC) for automated regression benchmarking. Add trigger scripts (trigger-benchmark.sh, trigger-benchmark-all.sh) for convenient local invocation of benchmark workflows via gh CLI. Made-with: Cursor * fix: make prebuilds step non-fatal with npm fallback When CI prebuilds are not available (no successful prebuilds workflow run), fall back to installing @qvac/transcription-parakeet from npm instead of failing the entire benchmark job. Made-with: Cursor * fix: use python 3.13 for benchmark client compatibility Python 3.14 changed Pickler._batch_setitems() signature which breaks the datasets library. Pin to 3.13 until upstream compatibility is fixed. Made-with: Cursor * fix: add named model paths in benchmark server for ctc/eou/sortformer The addon requires model-type-specific named paths (e.g. ctcModelPath, eouEncoderPath, sortformerPath) when activating non-TDT models. Add getNamedPaths() that resolves the correct file paths per model type and spreads them into the parakeetConfig passed to the addon constructor. Made-with: Cursor * fix: spread named paths at config top level, not inside parakeetConfig The addon reads ctcModelPath/eouEncoderPath/sortformerPath from the top-level config object (this._config), not from parakeetConfig. Made-with: Cursor * fix: use public cgus repo for sortformer model download The tetherto/sortformer-4spk-v2-onnx HuggingFace repo is gated and returns an invalid file. Use the public cgus community repo that the integration tests already rely on. Made-with: Cursor * chore: remove redundant trigger-benchmark-all.sh trigger-benchmark.sh already supports -t all, making the separate trigger-benchmark-all.sh unnecessary. Made-with: Cursor * chore: remove scheduled cron trigger from benchmark workflow Per review feedback — "automated" means triggered via workflow_dispatch, not periodic autonomous runs. Made-with: Cursor * fix: correct workflow fallback default and remove dead code in trigger script - Change MODEL_TYPE fallback from 'all' to 'tdt' to match the workflow_dispatch UI default - Replace unreachable $? check (dead code under set -e) with proper if-not construct in trigger-benchmark.sh Made-with: Cursor --------- Co-authored-by: Raju <raju.sharma> * fix[notask]: replace global streaming state with per-instance map in whispercpp (#1079) The streaming processor used three process-global variables (g_streamingMtx, g_streamingInstance, g_streamingProcessor) which limited the entire process to a single streaming session and risked dangling-pointer access if the owning AddonJs instance was destroyed without cleanup. Replace with an unordered_map keyed by AddonJs* so each addon instance independently owns its streaming session, eliminating the race condition and enabling concurrent streaming across multiple instances. Made-with: Cursor Co-authored-by: Raju <raju.sharma> * chore[notask]: replace deprecated istanbul with nyc in decoder-audio (#1082) * chore[notask]: replace deprecated istanbul with nyc in decoder-audio The istanbul package has been deprecated since 2016 and carries known vulnerable transitive dependencies (minimatch ReDoS, uglify-js ReDoS). Replace with nyc ^17.1.0 (the actively maintained successor) and update coverage scripts to use nyc CLI syntax. Made-with: Cursor * fix[notask]: fix nyc coverage report command to use .nyc_output directory The nyc report command expects coverage data in .nyc_output/ rather than reading from --temp-dir directly. Copy brittle's coverage-final.json into .nyc_output/ before running nyc report so the HTML report generates cleanly without format warnings. Made-with: Cursor --------- Co-authored-by: Raju <raju.sharma> * Updated dependencies with android-arm64 fix (#1095) Co-authored-by: gianni <gianfranco.cordella@tether.io> * fix[notask]: sanitize error messages to prevent filesystem path leakage (#1084) Error messages in whispercpp and parakeet validateModelFiles() included full filesystem paths (e.g. "Model file doesn't exist: /home/user/..."). When surfaced via API responses this reveals internal server layout. Log the full path at debug/error level for operators, but throw generic messages without paths to callers. Made-with: Cursor Co-authored-by: Raju <raju.sharma> * fix[notask]: wrap job ID counter at MAX_SAFE_INTEGER to prevent precision loss (#1085) The _nextJobId counter in WhisperInterface and ParakeetInterface was incremented without bounds. After 2^53 increments, JavaScript loses integer precision and job ID collisions become possible. Replace raw += 1 with nextSafeId() that wraps back to 1 at Number.MAX_SAFE_INTEGER, preserving Number type compatibility for existing consumers. Made-with: Cursor Co-authored-by: Raju <raju.sharma> * fix: catch unhandled rejections in mobile integration runtime Register Bare.on('unhandledRejection') and Bare.on('uncaughtException') handlers to prevent the runtime from aborting (SIGABRT) when network errors escape the promise chain during model downloads. Made-with: Cursor * fix: bundle audio samples and resolve asset paths for mobile tests Add sample-16k.wav, French.raw, and croatian.raw to testAssets so integration tests can run transcription on mobile without downloading. Update getTestPaths to resolve samplesDir from the bundled asset manifest on mobile instead of a non-existent writableRoot/samples path. Made-with: Cursor * chore: bump parakeet to 0.2.4 Made-with: Cursor * chore: bump parakeet to 0.2.5 Made-with: Cursor --------- Co-authored-by: Raju <raju.sharma> Co-authored-by: Yury Samarin <yuri.a.samarin@gmail.com> Co-authored-by: Proletter <40578159+Proletter@users.noreply.github.com> Co-authored-by: Simon Iribarren <simon.ig13@gmail.com> Co-authored-by: Marco <1369747+elchiapp@users.noreply.github.com> Co-authored-by: Raju Sharma <sharmaraju352@gmail.com> Co-authored-by: Juan Pablo Garibotti Arias <juan.arias@bitfinex.com> Co-authored-by: gianni <gianfranco.cordella@tether.io> Co-authored-by: GustavoA1604 <54457676+GustavoA1604@users.noreply.github.com>

…, eou and sortformer models (#991) * feat: add automated benchmarks for parakeet ctc, eou and sortformer models Add per-model benchmark config files (config-ctc.yaml, config-eou.yaml, config-sortformer.yaml) with appropriate defaults for each model type. Update the CI workflow to support an 'all' option that runs benchmarks for every model type in a single matrix, and add a weekly schedule trigger (Sunday 04:00 UTC) for automated regression benchmarking. Add trigger scripts (trigger-benchmark.sh, trigger-benchmark-all.sh) for convenient local invocation of benchmark workflows via gh CLI. Made-with: Cursor * fix: make prebuilds step non-fatal with npm fallback When CI prebuilds are not available (no successful prebuilds workflow run), fall back to installing @qvac/transcription-parakeet from npm instead of failing the entire benchmark job. Made-with: Cursor * fix: use python 3.13 for benchmark client compatibility Python 3.14 changed Pickler._batch_setitems() signature which breaks the datasets library. Pin to 3.13 until upstream compatibility is fixed. Made-with: Cursor * fix: add named model paths in benchmark server for ctc/eou/sortformer The addon requires model-type-specific named paths (e.g. ctcModelPath, eouEncoderPath, sortformerPath) when activating non-TDT models. Add getNamedPaths() that resolves the correct file paths per model type and spreads them into the parakeetConfig passed to the addon constructor. Made-with: Cursor * fix: spread named paths at config top level, not inside parakeetConfig The addon reads ctcModelPath/eouEncoderPath/sortformerPath from the top-level config object (this._config), not from parakeetConfig. Made-with: Cursor * fix: use public cgus repo for sortformer model download The tetherto/sortformer-4spk-v2-onnx HuggingFace repo is gated and returns an invalid file. Use the public cgus community repo that the integration tests already rely on. Made-with: Cursor * chore: remove redundant trigger-benchmark-all.sh trigger-benchmark.sh already supports -t all, making the separate trigger-benchmark-all.sh unnecessary. Made-with: Cursor * chore: remove scheduled cron trigger from benchmark workflow Per review feedback — "automated" means triggered via workflow_dispatch, not periodic autonomous runs. Made-with: Cursor * fix: correct workflow fallback default and remove dead code in trigger script - Change MODEL_TYPE fallback from 'all' to 'tdt' to match the workflow_dispatch UI default - Replace unreachable $? check (dead code under set -e) with proper if-not construct in trigger-benchmark.sh Made-with: Cursor --------- Co-authored-by: Raju <raju.sharma>

* fix: statically link parakeet prebuilds Made-with: Cursor * fix: restore parakeet linux runtime loading Made-with: Cursor * fix: address parakeet apple prebuild failures Made-with: Cursor * chore: remove parakeet release notes file Made-with: Cursor * fix: use static requires for mobile bare-pack bundling The _resolve() helper used computed require paths that bare-pack could not statically trace, so the addon modules were missing from the mobile bundle. Use static string literals for mobile paths (traced by bare-pack) and variable paths for desktop (skipped by bare-pack since ../../ doesn't exist in the mobile layout). Made-with: Cursor * feat[notask]: add download profiler for registry blob performance diagnostics (#1040) * feat[notask]: add download profiler for registry blob performance diagnostics Made-with: Cursor * fix: move profiler deps from devDependencies to dependencies Made-with: Cursor * doc: add profile command and example to client README Made-with: Cursor * fix: show full peer keys in profiler output for troubleshooting Made-with: Cursor * fix: validate parseInt results for interval and timeout CLI flags Made-with: Cursor --------- Co-authored-by: Proletter <40578159+Proletter@users.noreply.github.com> Co-authored-by: Simon Iribarren <simon.ig13@gmail.com> * fix: resolve dependabot alerts for registry-server transitive deps (#1093) * fix(registry-server): PBKDF2 for passphrase-derived keys (CodeQL #9) (#1065) * fix(registry-server): derive passphrase keys with PBKDF2 Replace single-pass SHA-256 with PBKDF2-HMAC-SHA256 (310k iterations) for deterministic test keys; addresses CodeQL js/insufficient-password-hash. * chore(registry-server): remove passphrase migration note from guide --------- Co-authored-by: Proletter <40578159+Proletter@users.noreply.github.com> * fix[notask]: lazy-load Node builtins in profiler for Bare runtime compatibility (#1096) * fix[notask]: sanitize SSE output to prevent reflected XSS (#1027) Co-authored-by: Marco <1369747+elchiapp@users.noreply.github.com> * [Parakeet] QVAC-13814 feat: add automated benchmarks for parakeet ctc, eou and sortformer models (#991) * feat: add automated benchmarks for parakeet ctc, eou and sortformer models Add per-model benchmark config files (config-ctc.yaml, config-eou.yaml, config-sortformer.yaml) with appropriate defaults for each model type. Update the CI workflow to support an 'all' option that runs benchmarks for every model type in a single matrix, and add a weekly schedule trigger (Sunday 04:00 UTC) for automated regression benchmarking. Add trigger scripts (trigger-benchmark.sh, trigger-benchmark-all.sh) for convenient local invocation of benchmark workflows via gh CLI. Made-with: Cursor * fix: make prebuilds step non-fatal with npm fallback When CI prebuilds are not available (no successful prebuilds workflow run), fall back to installing @qvac/transcription-parakeet from npm instead of failing the entire benchmark job. Made-with: Cursor * fix: use python 3.13 for benchmark client compatibility Python 3.14 changed Pickler._batch_setitems() signature which breaks the datasets library. Pin to 3.13 until upstream compatibility is fixed. Made-with: Cursor * fix: add named model paths in benchmark server for ctc/eou/sortformer The addon requires model-type-specific named paths (e.g. ctcModelPath, eouEncoderPath, sortformerPath) when activating non-TDT models. Add getNamedPaths() that resolves the correct file paths per model type and spreads them into the parakeetConfig passed to the addon constructor. Made-with: Cursor * fix: spread named paths at config top level, not inside parakeetConfig The addon reads ctcModelPath/eouEncoderPath/sortformerPath from the top-level config object (this._config), not from parakeetConfig. Made-with: Cursor * fix: use public cgus repo for sortformer model download The tetherto/sortformer-4spk-v2-onnx HuggingFace repo is gated and returns an invalid file. Use the public cgus community repo that the integration tests already rely on. Made-with: Cursor * chore: remove redundant trigger-benchmark-all.sh trigger-benchmark.sh already supports -t all, making the separate trigger-benchmark-all.sh unnecessary. Made-with: Cursor * chore: remove scheduled cron trigger from benchmark workflow Per review feedback — "automated" means triggered via workflow_dispatch, not periodic autonomous runs. Made-with: Cursor * fix: correct workflow fallback default and remove dead code in trigger script - Change MODEL_TYPE fallback from 'all' to 'tdt' to match the workflow_dispatch UI default - Replace unreachable $? check (dead code under set -e) with proper if-not construct in trigger-benchmark.sh Made-with: Cursor --------- Co-authored-by: Raju <raju.sharma> * fix[notask]: replace global streaming state with per-instance map in whispercpp (#1079) The streaming processor used three process-global variables (g_streamingMtx, g_streamingInstance, g_streamingProcessor) which limited the entire process to a single streaming session and risked dangling-pointer access if the owning AddonJs instance was destroyed without cleanup. Replace with an unordered_map keyed by AddonJs* so each addon instance independently owns its streaming session, eliminating the race condition and enabling concurrent streaming across multiple instances. Made-with: Cursor Co-authored-by: Raju <raju.sharma> * chore[notask]: replace deprecated istanbul with nyc in decoder-audio (#1082) * chore[notask]: replace deprecated istanbul with nyc in decoder-audio The istanbul package has been deprecated since 2016 and carries known vulnerable transitive dependencies (minimatch ReDoS, uglify-js ReDoS). Replace with nyc ^17.1.0 (the actively maintained successor) and update coverage scripts to use nyc CLI syntax. Made-with: Cursor * fix[notask]: fix nyc coverage report command to use .nyc_output directory The nyc report command expects coverage data in .nyc_output/ rather than reading from --temp-dir directly. Copy brittle's coverage-final.json into .nyc_output/ before running nyc report so the HTML report generates cleanly without format warnings. Made-with: Cursor --------- Co-authored-by: Raju <raju.sharma> * Updated dependencies with android-arm64 fix (#1095) Co-authored-by: gianni <gianfranco.cordella@tether.io> * fix[notask]: sanitize error messages to prevent filesystem path leakage (#1084) Error messages in whispercpp and parakeet validateModelFiles() included full filesystem paths (e.g. "Model file doesn't exist: /home/user/..."). When surfaced via API responses this reveals internal server layout. Log the full path at debug/error level for operators, but throw generic messages without paths to callers. Made-with: Cursor Co-authored-by: Raju <raju.sharma> * fix[notask]: wrap job ID counter at MAX_SAFE_INTEGER to prevent precision loss (#1085) The _nextJobId counter in WhisperInterface and ParakeetInterface was incremented without bounds. After 2^53 increments, JavaScript loses integer precision and job ID collisions become possible. Replace raw += 1 with nextSafeId() that wraps back to 1 at Number.MAX_SAFE_INTEGER, preserving Number type compatibility for existing consumers. Made-with: Cursor Co-authored-by: Raju <raju.sharma> * fix: catch unhandled rejections in mobile integration runtime Register Bare.on('unhandledRejection') and Bare.on('uncaughtException') handlers to prevent the runtime from aborting (SIGABRT) when network errors escape the promise chain during model downloads. Made-with: Cursor * fix: bundle audio samples and resolve asset paths for mobile tests Add sample-16k.wav, French.raw, and croatian.raw to testAssets so integration tests can run transcription on mobile without downloading. Update getTestPaths to resolve samplesDir from the bundled asset manifest on mobile instead of a non-existent writableRoot/samples path. Made-with: Cursor * chore: bump parakeet to 0.2.4 Made-with: Cursor * chore: bump parakeet to 0.2.5 Made-with: Cursor --------- Co-authored-by: Raju <raju.sharma> Co-authored-by: Yury Samarin <yuri.a.samarin@gmail.com> Co-authored-by: Proletter <40578159+Proletter@users.noreply.github.com> Co-authored-by: Simon Iribarren <simon.ig13@gmail.com> Co-authored-by: Marco <1369747+elchiapp@users.noreply.github.com> Co-authored-by: Raju Sharma <sharmaraju352@gmail.com> Co-authored-by: Juan Pablo Garibotti Arias <juan.arias@bitfinex.com> Co-authored-by: gianni <gianfranco.cordella@tether.io> Co-authored-by: GustavoA1604 <54457676+GustavoA1604@users.noreply.github.com>

Raju added 2 commits March 19, 2026 12:06

fix: make prebuilds step non-fatal with npm fallback

ece2e69

When CI prebuilds are not available (no successful prebuilds workflow run), fall back to installing @qvac/transcription-parakeet from npm instead of failing the entire benchmark job. Made-with: Cursor

sharmaraju352 requested review from a team as code owners March 19, 2026 06:53

Raju added 4 commits March 19, 2026 12:31

fix: use python 3.13 for benchmark client compatibility

d511901

Python 3.14 changed Pickler._batch_setitems() signature which breaks the datasets library. Pin to 3.13 until upstream compatibility is fixed. Made-with: Cursor

fix: spread named paths at config top level, not inside parakeetConfig

5905d82

The addon reads ctcModelPath/eouEncoderPath/sortformerPath from the top-level config object (this._config), not from parakeetConfig. Made-with: Cursor

fix: use public cgus repo for sortformer model download

6884c1c

The tetherto/sortformer-4spk-v2-onnx HuggingFace repo is gated and returns an invalid file. Use the public cgus community repo that the integration tests already rely on. Made-with: Cursor

sharmaraju352 changed the title ~~feat: add automated benchmarks for parakeet ctc, eou and sortformer models~~ [Parakeet] QVAC-13814 feat: add automated benchmarks for parakeet ctc, eou and sortformer models Mar 19, 2026

chore: remove redundant trigger-benchmark-all.sh

d5b7cef

trigger-benchmark.sh already supports -t all, making the separate trigger-benchmark-all.sh unnecessary. Made-with: Cursor

ishanvohra2 previously approved these changes Mar 19, 2026

View reviewed changes

GustavoA1604 requested changes Mar 19, 2026

View reviewed changes

Comment thread .github/workflows/benchmark-qvac-lib-infer-parakeet.yml Outdated

chore: remove scheduled cron trigger from benchmark workflow

0fe11b8

Per review feedback — "automated" means triggered via workflow_dispatch, not periodic autonomous runs. Made-with: Cursor

sharmaraju352 dismissed ishanvohra2’s stale review via 0fe11b8 March 20, 2026 11:33

Raju and others added 2 commits March 20, 2026 17:08

Merge branch 'main' into feat/parakeet-benchmark-all-models

e548b49

sharmaraju352 added verify tier1 labels Mar 23, 2026

GustavoA1604 approved these changes Mar 23, 2026

View reviewed changes

ogad-tether approved these changes Mar 23, 2026

View reviewed changes

Merge branch 'main' into feat/parakeet-benchmark-all-models

bdf67c0

sharmaraju352 merged commit 844e0ee into main Mar 24, 2026
22 checks passed

sharmaraju352 deleted the feat/parakeet-benchmark-all-models branch March 24, 2026 10:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Parakeet] QVAC-13814 feat: add automated benchmarks for parakeet ctc, eou and sortformer models#991

[Parakeet] QVAC-13814 feat: add automated benchmarks for parakeet ctc, eou and sortformer models#991
sharmaraju352 merged 11 commits into
mainfrom
feat/parakeet-benchmark-all-models

sharmaraju352 commented Mar 19, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Mar 19, 2026 •

edited

Loading

Uh oh!

Uh oh!

ogad-tether left a comment

Uh oh!

ogad-tether commented Mar 23, 2026

Uh oh!

sharmaraju352 commented Mar 24, 2026

Uh oh!

sharmaraju352 commented Mar 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

sharmaraju352 commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Successful Workflow Runs

Details

Config files

Workflow changes

Benchmark server changes

Trigger script

Test plan

Uh oh!

github-actions Bot commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tier-based Approval Status

Uh oh!

Uh oh!

ogad-tether left a comment

Choose a reason for hiding this comment

Uh oh!

ogad-tether commented Mar 23, 2026

Uh oh!

sharmaraju352 commented Mar 24, 2026

Uh oh!

sharmaraju352 commented Mar 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sharmaraju352 commented Mar 19, 2026 •

edited

Loading

github-actions Bot commented Mar 19, 2026 •

edited

Loading