FastConformer ASR 3/3: Parakeet and Nemotron ASR, the transducer as a graph loop

.claude/skills/engine-resources/SKILL.md

@@ -18,7 +18,7 @@ Entries marked *open PR* are not on `main` yet. Check the checkout; if your port
 | | `RaggedCrossAttentionSpec` | one contiguous key range per query segment |
 | | `RaggedBlockCausalAttentionSpec` (#381, *open PR*) | each block attends its span's prefix of blocks |
 | a fixed-size cache per request: a sink plus a sliding window, optionally after a shared read-only prefix | bounded KV (`BoundedKVSpec`, #383, *open PR*) | rows at any position in one batch; per-row source; a per-distance score term |
-| fixed-size opaque state per request (recurrent, conv tails) | `RecurrentStatePool` | slots read and written through the step's addressing (`gather` / `scatter_`, #383, *open PR*) |
+| fixed-size opaque state per request (recurrent, conv tails) | `RecurrentStatePool` | slots read and written through the step's addressing (`gather` / `scatter_`) |
 | frame rings (Waypoint) | `RingKVManager` | to be merged into bounded KV |
 
 A resource that holds per-request state gives models operations driven by the step (`gather` / `scatter_`, `attend` over the current plan). It never asks a model to index its storage or to hold and select rows of its plans.

AGENTS.md

@@ -44,7 +44,7 @@ For example, nothing under `mstar/model/` should:
 - **Allocate, free, or evict resource state.** No calls into `PageArena`, `PageAllocator`, `CacheStream`, `CPUPagePool`, or `WorkspacePool`, and no direct `admit`/`commit`. Declare resources in `Model.get_node_resources()` and declare each step's effect on them in `NodeSubmodule.declare_step()`. The engine runs the lifecycle.
 - **Choose which requests form a batch.** Batch *selection* is the micro-scheduler's job: implement `can_batch` and `forward_batched`, set `input_seq_len` in `prepare_inputs`, and let it decide. (Note: given a batch, mechanically stacking or concatenating tensors in `preprocess` is normal and expected, and some audio submodules also pad to a maximum sequence length themselves for capture compatibility).
 - **Manage streams or synchronize.** No `torch.Stream`, `torch.Event`, `torch.accelerator.synchronize`, or device-specific equivalents under `mstar/model/`.
-- **Index a resource's storage or hold its plans.** No `pool.block(...).index_select` / `index_copy_`, host-side slot lookups, or keeping `resource.current` to select rows from later. The resource exposes the access, driven by the step (e.g., `RecurrentStatePool.gather` / `scatter_` ([#383](https://github.com/mstar-project/mstar/pull/383), open PR), `attend` over the current plan), so eager and captured paths address rows the same way.
+- **Index a resource's storage or hold its plans.** No `pool.block(...).index_select` / `index_copy_`, host-side slot lookups, or keeping `resource.current` to select rows from later. The resource exposes the access, driven by the step (e.g., `RecurrentStatePool.gather` / `scatter_`, `attend` over the current plan), so eager and captured paths address rows the same way.
 
 Things that look model-specific but belong to the system:
 

benchmark/asr_eval.py

@@ -1,7 +1,7 @@
 """ASR evaluation script: compute Word Error Rate (WER) on LibriSpeech test-clean.
 
 Runs inference against a running mstar server and compares transcription output
-against ground-truth transcripts.  Supports Whisper-large-v3 and Higgs Audio.
+against ground-truth transcripts, for the models in ``_ASR_MODELS``.
 
 Usage:
     python -m benchmark.asr_eval \\
@@ -27,7 +27,10 @@
 from benchmark.dataset import LibriSpeechEvalDataset
 from benchmark.request import RequestInput
 
-_ASR_MODELS = {ModelType.WHISPER_LARGE, ModelType.HIGGS_AUDIO}
+_ASR_MODELS = {
+    ModelType.WHISPER_LARGE, ModelType.HIGGS_AUDIO,
+    ModelType.CANARY, ModelType.CANARY_QWEN, ModelType.PARAKEET, ModelType.NEMOTRON_ASR,
+}
 
 
 async def _transcribe(

benchmark/base.py

@@ -418,6 +418,33 @@ def get_model_kwargs(self, request_type: RequestType):
         }
 
 
+class _FastConformerAsr(Model):
+    """NVIDIA's FastConformer speech recognizers: audio in, a transcript out, greedy."""
+
+    def get_supported_modalities(self):
+        return {RequestType.A2T}
+
+
+class Canary(_FastConformerAsr):
+    def get_hf_url(self):
+        return "nvidia/canary-1b-v2"
+
+
+class CanaryQwen(_FastConformerAsr):
+    def get_hf_url(self):
+        return "nvidia/canary-qwen-2.5b"
+
+
+class Parakeet(_FastConformerAsr):
+    def get_hf_url(self):
+        return "nvidia/parakeet-tdt-0.6b-v3"
+
+
+class NemotronAsr(_FastConformerAsr):
+    def get_hf_url(self):
+        return "nvidia/nemotron-3.5-asr-streaming-0.6b"
+
+
 class ModelType(Enum):
     BAGEL = "bagel"
     ORPHEUS = "orpheus"
@@ -431,6 +458,10 @@ class ModelType(Enum):
     VJEPA2AC = "vjepa2ac"
     WHISPER_LARGE = "whisper_large"
     HIGGS_AUDIO = "higgs_audio"
+    CANARY = "canary"
+    CANARY_QWEN = "canary_qwen"
+    PARAKEET = "parakeet"
+    NEMOTRON_ASR = "nemotron_asr"
     QWEN3_5 = "qwen3.5"
     QWEN3_8 = "qwen3.8"
 
@@ -459,6 +490,14 @@ def inst(self, **kwargs) -> Model:
             return WhisperLarge(**kwargs)
         if self == ModelType.HIGGS_AUDIO:
             return HiggsAudio(**kwargs)
+        if self == ModelType.CANARY:
+            return Canary(**kwargs)
+        if self == ModelType.CANARY_QWEN:
+            return CanaryQwen(**kwargs)
+        if self == ModelType.PARAKEET:
+            return Parakeet(**kwargs)
+        if self == ModelType.NEMOTRON_ASR:
+            return NemotronAsr(**kwargs)
         if self == ModelType.QWEN3_5:
             return Qwen3_5_Dense(**kwargs)
         if self == ModelType.QWEN3_8:

benchmark/run_asr_benchmark.sh

@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-# ASR benchmark script: throughput and latency for Whisper-large-v3 and Higgs Audio.
+# ASR benchmark script: throughput and latency for the ASR models below.
 #
 # Usage:
 #   ./benchmark/run_asr_benchmark.sh --url http://localhost:8000 [options]
@@ -11,7 +11,8 @@
 #   --url  <base_url>       mstar server base URL
 #
 # Optional:
-#   --model  <name>         Run only this model (whisper_large|higgs_audio)
+#   --model  <name>         Run only this model (whisper_large|higgs_audio|canary|
+#                           canary_qwen|parakeet|nemotron_asr)
 #   --num-requests  <n>     Total requests per model (default: 100)
 #   --batch-size  <b>       Batch size for offline profiling (default: 4)
 #   --max-concurrency <c>   Concurrency cap for closed-loop profiling (default: 4)
@@ -52,7 +53,7 @@ if [[ -z "$URL" ]]; then
   exit 1
 fi
 
-ASR_MODELS=(whisper_large higgs_audio)
+ASR_MODELS=(whisper_large higgs_audio canary canary_qwen parakeet nemotron_asr)
 
 if [[ -n "$MODEL" ]]; then
   ASR_MODELS=("$MODEL")
@@ -101,14 +102,14 @@ done
 echo ""
 echo "All ASR benchmarks complete.  Results in: ${OUTPUT_DIR}"
 
-# Print a quick comparison table if all three models ran
+# Print a quick comparison table of the models that ran
 if [[ -z "$MODEL" ]] && command -v python &>/dev/null; then
   export _ASR_OUTPUT_DIR="$OUTPUT_DIR"
   python - <<'PYEOF'
 import json, os, sys
 
 output_dir = os.environ.get("_ASR_OUTPUT_DIR", "./asr_benchmark_results")
-models = ["whisper_large", "higgs_audio"]
+models = ["whisper_large", "higgs_audio", "canary", "canary_qwen", "parakeet", "nemotron_asr"]
 
 rows = []
 for m in models:

configs/canary.yaml

@@ -0,0 +1,14 @@
+model: "canary"
+max_seq_len: 1024
+# The decoder caches hold 64 requests at their limits (4 context pages for a
+# 40.96 s clip, 8 self-attention pages at the position limit), plus the page
+# each cache keeps as its sink.
+max_concurrent_requests: 64
+resources:
+  kv_cache:
+    max_num_pages: 513
+  cross_kv_cache:
+    max_num_pages: 257
+node_groups:
+  - node_names: [audio_encoder, decoder]
+    ranks: [0]

configs/canary_qwen.yaml

@@ -0,0 +1,11 @@
+model: "canary_qwen"
+max_seq_len: 2048
+# The KV cache holds 64 requests at max_seq_len (16 pages each) plus its sink
+# page; the conductor queues the rest rather than have the worker offload.
+max_concurrent_requests: 64
+resources:
+  kv_cache:
+    max_num_pages: 1025
+node_groups:
+  - node_names: [audio_encoder, llm]
+    ranks: [0]

configs/nemotron_asr.yaml

@@ -0,0 +1,17 @@
+model: "nemotron_asr"
+max_seq_len: 8192
+# A request holds one decoder state slot from its first decode iteration until
+# it is torn down; the pool holds max_concurrent_requests of them plus its sink
+# slot, and the conductor queues the rest.
+max_concurrent_requests: 128
+resources:
+  transducer_state:
+    max_slots: 129
+node_groups:
+  - node_names: [audio_encoder, transducer]
+    ranks: [0]
+# Constructor options (forwarded to ParakeetModel.__init__):
+# model_kwargs:
+#   lookahead: 13  # right attention context in 80 ms frames: 0, 3, 6 or 13;
+#                  # wider is more accurate, and the widest is the default
+#   compile_step: false  # decode with eager kernels: start-up a few seconds faster, decoding ~2.5x slower

configs/parakeet.yaml

@@ -0,0 +1,15 @@
+model: "parakeet"
+max_seq_len: 8192
+# A request holds one decoder state slot from its first decode iteration until
+# it is torn down; the pool holds max_concurrent_requests of them plus its sink
+# slot, and the conductor queues the rest.
+max_concurrent_requests: 128
+resources:
+  transducer_state:
+    max_slots: 129
+node_groups:
+  - node_names: [audio_encoder, transducer]
+    ranks: [0]
+# Constructor options (forwarded to ParakeetModel.__init__):
+# model_kwargs:
+#   compile_step: false  # decode with eager kernels: start-up a few seconds faster, decoding ~2.5x slower

docs/adding_models.rst

@@ -565,6 +565,18 @@ The spec types are:
        ``enable_min_p``, ``max_repetition_window``, ``min_tokens_stop_ids`` and
        ``enable_top_p_first`` are capabilities of the same kind. See
        ``SamplingReqConfig`` above.
+   * - ``RecurrentStateSpec(config=RecurrentStateConfig(...))``
+     - Fixed-size per-request state that carries between steps (an LSTM's ``h`` and
+       ``c``, a cursor, a conv tail): one slot per request, ``blocks`` naming a shape and
+       dtype for each piece, ``num_layers`` copies of each, ``max_slots`` slots with slot 0
+       the sink that padding rows address. ``declare_step`` gives each row a
+       ``RecurrentStep`` segment with span >= 1. The forward reads its rows with
+       ``pool.gather(name, start=...)`` and writes them back with
+       ``pool.scatter_(name, value)``; both follow the step's plan, so there is no
+       addressing to fetch, and eager and captured steps reach the same slots. A row with
+       no saved state (a request's first step, or a padding row) reads ``start``, zeros
+       when none is given. ``gather`` / ``scatter_`` need the sink, so leave
+       ``disable_sink_slot`` unset. Parakeet's transducer decode is the example.
 
 Orpheus declares four specs for its one autoregressive node. Its ``snac_decoder`` node
 appears in no spec, so it receives no resources:
@@ -621,6 +633,9 @@ appears in no spec, so it receives no resources:
    * - Decoder of an encoder-decoder model
      - The three or four specs above, plus a second ``KVSpec`` for the encoder context
        and a ``CrossAttentionSpec`` over that second cache.
+   * - A node that carries per-request state between steps of a loop (an RNN-T
+       decoder, recurrent attention)
+     - A ``RecurrentStateSpec``, read and written with ``gather`` / ``scatter_``.
    * - A node that samples with two different sets of parameters
      - Two ``SamplerSpec`` objects that both name the node, under different keys.
    * - An encoder tower that attends within the segments of one packed forward
@@ -2025,9 +2040,12 @@ Sizing the caches is only half of it: ``configs/whisper_large.yaml`` also sets
 ``max_concurrent_requests: 64``, so the conductor queues further requests instead of the
 worker offloading decoding requests to admit them. ``WhisperModel.validate_config_yaml``
 refuses at startup a cap that either cache cannot hold after the YAML's ``resources:``
-overrides.
+overrides, through ``mstar.model.utils.check_kv_pools_fit``, which any model with KV caches
+can call the same way.
 
-Whisper's ``declare_step`` shows how a write-once context is expressed with spans. The
+The decoder node's ``declare_step`` shows how a write-once context is expressed with spans.
+It lives in ``EncoderDecoderNode`` (``mstar/model/components/encoder_decoder.py``), which
+Whisper's and Canary's decoder nodes share; Whisper adds only its token suppression. The
 context segments have a non-zero span in the prefill step that writes them, and a span of
 0 in every later step. The ``commit`` in the prefill step converts the reservation into
 resident pages that the later steps read.

docs/clients.rst

@@ -244,7 +244,8 @@ Endpoints and model coverage:
      - speech models that publish a voice list through their speech adapter (``kokoro``, ``orpheus``; 404 otherwise)
      - The ``voice`` ids the served model accepts, plus its default.
    * - ``POST /v1/audio/transcriptions``
-     - ``whisper_large``, ``higgs_audio``, ``omnilingual_asr_ctc_*``
+     - ``whisper_large``, ``higgs_audio``, ``omnilingual_asr_ctc_*``, ``canary``, ``canary_qwen``, ``parakeet``,
+       ``nemotron_asr``
      - Speech-to-text (multipart upload; ``json`` / ``text`` / ``verbose_json`` /
        ``srt`` / ``vtt``; streaming via ``transcript.text.delta`` events).
    * - ``WS /v1/realtime?intent=transcription``
@@ -302,7 +303,9 @@ Per-model notes:
   ``initial_prompt``), and ``response_format``
   ``verbose_json`` / ``srt`` / ``vtt`` (or ``timestamp_granularities[]``) asks the
   model for timestamps. Whisper's language and timestamp tokens travel in the
-  text stream and are lifted into ``language`` / ``segments`` by the server; a
+  text stream and are lifted into ``language`` / ``segments`` by the server;
+  Higgs-Audio emits no timestamps, so its ``verbose_json`` / ``srt`` / ``vtt`` get
+  one segment per window, timed to the windows. A
   streaming client receives only the spoken words. Uploads longer than the
   model's clip (30 s for Whisper) are served as consecutive windows. By default
   they run in order, openai-whisper style: each window gets the transcript so
@@ -326,8 +329,17 @@ Per-model notes:
   clip of up to 20 s is one request; a longer upload is cut into windows of about equal
   length, at most 16 s, at quiet points and transcribed in parallel (``window_seconds``
   in ``extra_body``, 5 to 40, sets both lengths). There are no timestamps: ``srt`` / ``vtt``
-  get one cue spanning the file. Nothing detects the language: ``verbose_json``'s
+  get one cue per window, timed to the windows. Nothing detects the language: ``verbose_json``'s
   ``language`` is the request's ``language`` as given, or null.
+- **Canary / Canary-Qwen / Parakeet / Nemotron ASR** — no conditioning text (``prompt`` is
+  dropped) and no timestamps: ``timestamp_granularities`` is a 400, and ``srt`` / ``vtt`` /
+  ``verbose_json`` get one segment per window. Canary takes ``language`` (the speech's,
+  English by default) and, through ``extra_body``, ``target_language`` or
+  ``task="translate"`` (into English) and ``punctuation``; the other three only transcribe,
+  and refuse a ``task`` other than ``transcribe``. Canary-Qwen is English only. Parakeet detects the language itself and refuses a ``language`` outside the 25 it knows;
+  Nemotron takes a locale (``en-US``), a bare code or ``auto``. Uploads longer than the
+  model's clip (40 s for Canary and Canary-Qwen, 10 min for Parakeet and Nemotron) run as
+  parallel windows.
 - **Realtime transcription** — every ``chunk_seconds`` (default 2 s) of new audio the
   session re-transcribes everything heard so far as one engine request whose assistant
   turn is prefilled with the previous hypothesis minus its last ``unfixed_tokens``

docs/installation.rst

@@ -155,6 +155,12 @@ Model families and some output formats need extra packages, exposed as pip *extr
        package metadata allows ``m-star`` to be published on PyPI. **Also needs**
        ``flash-attn-4``, which is installed separately —
        see `flash-attn-4 (Waypoint, FA3 sm90 kernel)`_.
+   * - ``.[asr]``
+     - Speech recognition models: ``transformers``, ``flashinfer-python``, ``safetensors``,
+       ``einops``, ``soundfile`` (WAV / FLAC / OGG / MP3 uploads), ``torchcodec``
+       (FFmpeg-backed decoding of the other formats), ``huggingface-hub``,
+       ``mooncake-transfer-engine``, and ``datasets`` / ``jiwer`` for
+       ``benchmark/asr_eval.py``.
    * - ``.[audio]``
      - ``soundfile`` — only needed to return **non-WAV** audio containers (mp3/flac/…)
        from the OpenAI/SDK audio surfaces. WAV/PCM output works without it.