Repository navigation

Whisper/Qwen3-ASR PR 4: Whisper large-v3-turbo and large-v3 out of Beta, Qwen3-ASR 1.7B and 0.6B realtime - #318

Merged

dghosef merged 170 commits into

mainfrom

model/asr

Oct 27, 2026

Merged

Whisper/Qwen3-ASR PR 4: Whisper large-v3-turbo and large-v3 out of Beta, Qwen3-ASR 1.7B and 0.6B realtime#318
dghosef merged 170 commits into
mainfrom
model/asr

Commits on Oct 6, 2026
- whisper: read encoder dims, alignment heads and mel params from the checkpoint
  merceod
  committed
  b6d85cf
- whisper: add a native batched audio encoder
  merceod
  committed
  daeb373
- whisper: add timestamp decoding rules as a staged per-row state
  merceod
  committed
  58f651a
- whisper: capture the encoder per batch size, detect language, apply timestamp rules
  merceod
  committed
  992fdb3
- whisper: native encoder, language detection walks, initial_prompt and timestamps
  merceod
  committed
  9d897dc
- qwen3_asr: add the model package
  merceod
  committed
  237acc1
- qwen3_asr: read the AuT and Qwen3 configs from the checkpoint
  merceod
  committed
  75090c1
- qwen3_asr: packed AuT encoder and Qwen3 decoder submodules
  merceod
  committed
  25c6459
- qwen3_asr: add the Qwen3-ASR model graph and prompt
  merceod
  committed
  aa23710
- registry: add whisper_large_v3_turbo, qwen3_asr and qwen3_asr_realtime
  merceod
  committed
  881f05b
- cli: default configs and next steps for the ASR models
  merceod
  committed
  a854e22
- configs: add whisper_large_v3_turbo
  merceod
  committed
  96bcc22
- configs: add qwen3_asr
  merceod
  committed
  4ecff30
- configs: add qwen3_asr_realtime
  merceod
  committed
  891929d
- benchmark: register the new ASR model types
  merceod
  committed
  f9927d7
- benchmark: accept the new ASR model keys in asr_eval
  merceod
  committed
  94847e8
- benchmark: build the shared LibriSpeech ASR set from openslr
  merceod
  committed
  b0eb897
- benchmark: protocol ASR benchmark over /v1/audio/transcriptions
  merceod
  committed
  187e09b
- benchmark: streaming partial-latency benchmark over /v1/realtime
  merceod
  committed
  a68d071
- benchmark: reference-implementation transcripts for WER parity
  merceod
  committed
  e0f5d26
- docs: Whisper turbo/v3 and Qwen3-ASR rows in the model table
  merceod
  committed
  cfeb68d
- tests: Whisper config, encoder parity, state machine and wiring
  merceod
  committed
  4d5e1c8
- tests: pin Whisper timestamp rules to HF's logits processor
  merceod
  committed
  498b48b
- tests: Qwen3-ASR prompt, submodules, state machine and wiring
  merceod
  committed
  3875c39
- tests: GPU parity for Whisper on real weights
  merceod
  committed
  f58fe38
- tests: GPU parity for Qwen3-ASR on real weights
  merceod
  committed
  a2af50a
- api: add the Qwen3-ASR transcription adapter
  merceod
  committed
  e0468ce
- tests: cover the Qwen3-ASR OpenAI adapter
  merceod
  committed
  412ddd5
- whisper: carry the timestamp rule state as a tensor and advance it in the forward
  merceod
  committed
  08041b9
- whisper: route the rule state with the token instead of a host history
  merceod
  committed
  779cb29
- whisper: loop the timestamp rule state back through the decode walk
  merceod
  committed
  5557aef
- tests: advance reproduces rule_state token by token
  merceod
  committed
  ac6b3f4
- tests: routed timestamp state in the decoder and the state machine
  merceod
  committed
  ea97347
- api: report Qwen3-ASR's detected language as an ISO code
  merceod
  committed
  3cf1870
- tests: Qwen3-ASR adapter maps language names to codes
  merceod
  committed
  4993fac
- benchmark: realtime bench speaks vLLM's WebSocket dialect and reports delta latency
  merceod
  committed
  e626090
- whisper: opt the submodules out of the engine's torch.compile
  merceod
  committed
  b641019
- qwen3_asr: opt the submodules out of the engine's torch.compile
  merceod
  committed
  9061a35
- benchmark: realtime bench tolerates close frames
  merceod
  committed
  5182d79
- whisper: send the encoder samples and make the log-mel the encoder's first GPU step
  merceod
  committed
  e162500
- whisper: encoder submodule pads the window and makes the log-mel batch on the GPU
  merceod
  committed
  3d5b6b3
- qwen3_asr: send the encoder samples, the prompt only needs the frame count
  merceod
  committed
  fdcf954
- qwen3_asr: encoder submodule makes the log-mel on the GPU
  merceod
  committed
  2a4d966
- tests: Whisper encoder takes samples and the tiny config uses a 1 s window
  merceod
  committed
  20a6dd1
- tests: Qwen3-ASR encoder takes samples
  merceod
  committed
  4ec2325
- tests: Whisper GPU parity checks the submodule's log-mel against the CPU transform
  merceod
  committed
  c447815
- benchmark: vLLM realtime final text is the joined deltas
  merceod
  committed
  196b06d
- whisper: word timestamps from the alignment heads (median filter, DTW, word grouping)
  merceod
  committed
  acb7a8a
- whisper: name the align walk
  merceod
  committed
  1485a3c
- whisper: teacher-forced cross-attention pass on the served decoder weights
  merceod
  committed
  2980300
- whisper: align walk in the decoder and a float32 log-mel under the engine's cast
  merceod
  committed
  efb87f5
- qwen3_asr: log-mel stays float32 under the engine's cast
  merceod
  committed
  7a543ae
- whisper: run the align walk after decoding when word timestamps are requested
  merceod
  committed
  7024eb2
- tests: word-timestamp alignment against HF attentions, DTW and word grouping
  merceod
  committed
  855e5d5
- tests: align walk through the state machine and the decoder submodule
  merceod
  committed
  6499b8d
- tests: Whisper GPU parity checks word starts against HF token timestamps
  merceod
  committed
  e205911
- docs: speech recognition page (Whisper and Qwen3-ASR scaffold, walks, long-form, realtime, benchmarks)
  merceod
  committed
  484c06c
- docs: link the speech recognition page
  merceod
  committed
  ddffe45
- whisper: select the prefill's last rows from the row lengths (a one-token language-detection prompt is planned as a decode step)
  merceod
  committed
  89361ea
- tests: decoder prefill selects each request's last row itself
  merceod
  committed
  8dd6774
- benchmark: realtime bench keeps the final transcripts
  merceod
  committed
  0d68a6c
- tests: word-timestamp parity reads HF's output as a mapping
  merceod
  committed
  41bff96
- tests: word-timestamp parity compares starts with HF's previous-token ends
  merceod
  committed
  5fe93e7
- docs: serve the ASR API process with one OpenMP thread
  merceod
  committed
  1e9b11e
- docs: serve the ASR API process with a passive OpenMP wait policy
  merceod
  committed
  45f95f6
- benchmark: keep the transcript text from vLLM's realtime chunk lines
  merceod
  committed
  e91bf19
- whisper: deliver the decode loop's tokens in one message at the end
  merceod
  committed
  a67b1ba
- tests: the Whisper decode loop accumulates its tokens for the client
  merceod
  committed
  5f7d98d
- qwen3_asr: deliver the decode loop's tokens in one message at the end
  merceod
  committed
  9c052bb
- docs: ASR tokens reach the client once per window
  merceod
  committed
  22c9592
- benchmark: reword a data split comment
  merceod
  committed
  6774586
- docs: plainer wording on the ASR page
  merceod
  committed
  2ef39cc
- docs: plainer wording in the ASR model entries
  merceod
  committed
  46b3fcd
- qwen3_asr: reword config comments
  merceod
  committed
  7f649ab
- qwen3_asr: reword comments and the audio count error
  merceod
  committed
  5fb27e2
- qwen3_asr: reword submodule comments
  merceod
  committed
  09aaf53
- registry: reword the ASR entries
  merceod
  committed
  b412469
- whisper: reword encoder comments
  merceod
  committed
  04f9820
- whisper: reword a timestamp rule comment
  merceod
  committed
  4b0c38a
- whisper: reword the walk comments
  merceod
  committed
  6e1635b
- whisper: reword submodule comments
  merceod
  committed
  ee97a19
- whisper: reword model comments and the audio count error
  merceod
  committed
  02881ac
- tests: reword parity test comments
  merceod
  committed
  99e8a73
- tests: reword timestamp test docstrings
  merceod
  committed
  fd7332a
- tests: reword Whisper test comments
  merceod
  committed
  1a3002b
- tests: match the new audio count message
  merceod
  committed
  322af1e
- whisper: use PagedKVConfig after the engine rename
  merceod
  committed
  be76e1c
- qwen3_asr: use PagedKVConfig after the engine rename
  merceod
  committed
  7997dd7
- components: check the ASR request kwargs once
  merceod
  committed
  01a6ab5
- whisper: prompt length per walk, state keyed by the handle, greedy detection
  merceod
  committed
  8b74e19
- whisper: check request kwargs and the concurrency cap in the config
  merceod
  committed
  66a8c0e
- qwen3_asr: check request kwargs and the concurrency cap in the config
  merceod
  committed
  4543462
- api: Qwen3-ASR adapter streams through stream_text
  merceod
  committed
  3182aa2
- cli: keep kokoro in the OpenAI example list
  merceod
  committed
  578ac16
- configs: cap whisper_large_v3_turbo at 64 concurrent requests
  merceod
  committed
  bab4291
- configs: cap whisper_large at 64 concurrent requests
  merceod
  committed
  2626cb0
- configs: cap whisper_large_tp2 at 64 concurrent requests
  merceod
  committed
  1adcbf5
- configs: cap qwen3_asr at 64 requests and give it 2048 KV pages
  merceod
  committed
  0f5f951
- configs: cap qwen3_asr_realtime at 64 concurrent requests
  merceod
  committed
  ae09247
- docs: the concurrency caps on the ASR page
  merceod
  committed
  ba8ead3
- tests: Whisper state by handle, prompt length per walk, greedy detection, cap
  merceod
  committed
  86956d9
- tests: Qwen3-ASR stream_text, cap and kwargs checks
  merceod
  committed
  fd98c3f
- tests: the request kwargs checker
  merceod
  committed
  0d70c25
- registry: keep main's waypoint and Qwen3-TTS entries beside the ASR ones
  merceod
  committed
  ed80181
- move language and task computation into the process_prompt
  NSagan271
  committed
  58d632a
- engine: pre_plan as a serve-config key, off for whisper turbo
  
  NSagan271
  and
  claude
  committed
  7536f93
Commits on Oct 8, 2026
- whisper: align with the detected language
  vasilevklart
  committed
  dc614a8
- whisper: count the sink page in the cross cache
  vasilevklart
  committed
  3818a65
- asr: bound seed to int64
  vasilevklart
  committed
  767a787
- qwen3_asr: no compression-ratio retries
  vasilevklart
  committed
  ace45bb
- api: stop long-form windows on disconnect
  vasilevklart
  committed
  66dc67b
- qwen3_asr: refuse <|audio_pad|> in the prompt
  vasilevklart
  committed
  2cb0a7b
- whisper: keep the align walk within 448 positions
  vasilevklart
  committed
  b4a7692
- docs: Qwen3-ASR windowing, ASR route list
  vasilevklart
  committed
  974e5cb
- ci: run the ASR tests in CPU Core
  vasilevklart
  committed
  b36f484
- Merge main into model/asr
  vasilevklart
  committed
  8e16585
- shm: don't send from reused host buffers
  vasilevklart
  committed
  230ad2f
Commits on Oct 9, 2026
- shm: drop host copies after the step's sends
  vasilevklart
  committed
  90ded70
- Merge main into model/asr
  vasilevklart
  committed
  5ea6af2
Commits on Oct 17, 2026
- Merge origin/main into model/asr
  dghosef
  committed
  84e927a
Commits on Oct 20, 2026
- Merge origin/main into model/asr
  
  dghosef
  and
  claude
  committed
  b3e8248
- API server: wake non-streaming handlers on completion instead of a 5 ms poll
  
  dghosef
  and
  claude
  committed
  12ddf77
- Conductor and API message thread: wake on a message instead of a fixed 1 ms sleep
  
  dghosef
  and
  claude
  committed
  a33d315
- whisper: count the decode captures' padding pages in the startup check; tests after the main merge
  
  dghosef
  and
  claude
  committed
  8a710f6
- test: register the woken request under its submission key; stub communicator answers poll_for_messages
  
  dghosef
  and
  claude
  committed
  82fe498
- qwen3_asr: import order after the accelerator-graph rename
  
  dghosef
  and
  claude
  committed
  5f21188
- qwen3_asr: collapse repetition loops in the transcript, as the reference SDK does
  
  dghosef
  and
  claude
  committed
  38ab1d2
- transcriptions: parallel long form decodes the tail a timed window left unfinished
  
  dghosef
  and
  claude
  committed
  c1cffc8
- test: the tick-exception stub communicator answers poll_for_messages (the conductor's message wait)
  
  dghosef
  and
  claude
  committed
  0e61337
- ci: test/asr needs a GPU and real weights; not part of CPU Core
  
  dghosef
  and
  claude
  committed
  a1c237e
- transcriptions: no tail pass once the client is gone; streamed parallel windows decode untimed
  
  dghosef
  and
  claude
  committed
  f4ba498
- whisper: back to main's decoder-cache sizing; padding rows hold no pages
  
  dghosef
  and
  claude
  committed
  601e127
- qwen3_asr: collapse repetition loops in the finished transcript only
  
  dghosef
  and
  claude
  committed
  2f0778a
- API server, conductor: guard the waits between passes; a testable wake backstop
  
  dghosef
  and
  claude
  committed
  cdbb473
- docs: Qwen3-ASR long-file capacity, Whisper's cap check, the pre_plan serve key
  
  dghosef
  and
  claude
  committed
  a886a0d
- Merge origin/main into model/asr
  
  dghosef
  and
  claude
  committed
  9bfb139
- test: worker-metrics run-loop fake sets _enable_pre_plan, which this PR's Worker resolves in __init__
  
  dghosef
  and
  claude
  committed
  13cb3f2
Commits on Oct 21, 2026
- qwen3_asr: splice the audio embeddings in by slice, not by a device mask
  
  dghosef
  and
  claude
  committed
  b112666
- request_kwargs: an integer field that overflows (inf, 1e400) is a ValueError naming the field
  
  dghosef
  and
  claude
  committed
  5118369
- whisper: numba is part of the asr extra; warn without it, compile at load
  
  dghosef
  and
  claude
  committed
  0e0d62d
- asr benchmarks and parity tests: data under data/asr, not an undocumented MSTAR_WS_ROOT
  
  dghosef
  and
  claude
  committed
  0707ac6
- worker: resolve_enable_pre_plan, tested directly; a non-boolean pre_plan is refused
  
  dghosef
  and
  claude
  committed
  2cea7ca
- qwen3_asr: ASR_TEXT_TAG defined once, in a module the API server imports without the model
  
  dghosef
  and
  claude
  committed
  d08a300
- whisper: word timestamps on a TP-sharded decoder are a 400 at ingest
  
  dghosef
  and
  claude
  committed
  5c62b05
- transcriptions: submit a parallel window's tail as soon as that window is in
  
  dghosef
  and
  claude
  committed
  7d26005
- qwen3_asr: the per-request window cap follows max_audio_seconds instead of a constant
  
  dghosef
  and
  claude
  committed
  6a62e79
- docs: Whisper delivers a window's transcript at loop end; qwen3_asr_realtime in the clients table
  
  dghosef
  and
  claude
  committed
  1face27
Commits on Oct 24, 2026
- Merge main into model/asr
  vasilevklart
  committed
  2c077f7
- whisper, qwen3_asr: decode lengths on main's per-request loop caps
  
  dghosef
  and
  claude
  committed
  246d285
- whisper: split zh/ja/th/lo/my/yue words per character, as openai-whisper
  
  dghosef
  and
  claude
  committed
  8ee690b
- qwen3_asr: decode tokens go to the client inline, not accumulated to the loop's end
  
  dghosef
  and
  claude
  committed
  cf65a75
- docs/asr: serve ASR on the shared-memory arena
  
  dghosef
  and
  claude
  committed
  5c1dbb4
- docs/asr: drop the unverified thread-pool hangs claim
  
  dghosef
  and
  claude
  committed
  f78e7c0
- whisper, qwen3_asr: cap the API server's torch threads at 4 (PREPROCESS_TORCH_THREADS)
  
  dghosef
  and
  claude
  committed
  580bfcd
- docs/asr: Qwen3-ASR long-file capacity from a measured run
  
  dghosef
  and
  claude
  committed
  2fc4745
- api server, conductor: drop the early wake between message-loop passes
  
  dghosef
  and
  claude
  committed
  383eed7
- whisper: merge punctuation as openai-whisper's merge_punctuations
  
  dghosef
  and
  claude
  committed
  48b8e69
- test/asr: compare the served model's greedy text to the reference
  
  dghosef
  and
  claude
  committed
  0292f1f
- whisper: stage prefill's rule state and row lengths through a pinned buffer
  
  dghosef
  and
  claude
  committed
  c2d1e69
- qwen3_asr: check_stop_batched takes the engine's step_walk
  
  dghosef
  and
  claude
  committed
  4b6cae8
- whisper, qwen3_asr: fixes from the clean review of this round
  
  dghosef
  and
  claude
  committed
  dc5b770
- whisper tests, docs/asr: fixes from the GPU run; drop the OMP_WAIT_POLICY launch step
  
  dghosef
  and
  claude
  committed
  5b95432
- qwen3_asr: cap the shipped config at 12 concurrent requests, and document the trade-off
  
  dghosef
  and
  claude
  committed
  0a479b7
Commits on Oct 26, 2026
- test/asr: run the Qwen3-ASR SDK reference in a subprocess with its own transformers
  
  dghosef
  and
  claude
  committed
  b2b793c
Commits on Oct 27, 2026
- Merge origin/main into model/asr
  
  dghosef
  and
  claude
  committed
  509382b
- Read pre_plan through the knobs: block (R1.1)
  
  dghosef
  and
  claude
  committed
  2b74310
- Qwen3-ASR's LLM node is a TokenStepNode (R1.2)
  
  dghosef
  and
  claude
  committed
  9acb78e
- One serving.rst knobs row, listing pre_plan (R2.1); base-default wording in TokenStepNode
  
  dghosef
  and
  claude
  committed
  d5d92f8
- Parallel transcription tail cut off the event loop (P3)
  
  dghosef
  and
  claude
  committed
  7b11999
- Parallel transcription tail: only the cut leaves the event loop (R3.1)
  
  dghosef
  and
  claude
  committed
  e3c824b

Commits

Commits on Oct 6, 2026

Commits on Oct 8, 2026

Commits on Oct 9, 2026

Commits on Oct 17, 2026

Commits on Oct 20, 2026

Commits on Oct 21, 2026