model: Add NVIDIA Canary-1B-v2 to Transformers

docs/source/en/_toctree.yml

@@ -1093,6 +1093,8 @@
         title: Audio Spectrogram Transformer
       - local: model_doc/bark
         title: Bark
+      - local: model_doc/canary
+        title: Canary
       - local: model_doc/clap
         title: CLAP
       - local: model_doc/cohere_asr

docs/source/en/model_doc/canary.md

@@ -0,0 +1,149 @@
+<!--Copyright 2026 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+*This model was published in HF papers on 2025-09-17 and contributed to Hugging Face Transformers on 2026-07-29.*
+
+<div class="flex flex-wrap space-x-1">
+<img alt="SDPA" src="https://img.shields.io/badge/SDPA-DE3412?style=flat&logo=pytorch&logoColor=white">
+</div>
+
+# Canary
+
+## Overview
+
+Canary-1B-v2 was proposed in [Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and AST](https://huggingface.co/papers/2509.14128) by Monica Sekoyan, Nithin Rao Koluguri, Nune Tadevosyan, Piotr Zelasko, Travis Bartley, Nikolay Karpov, Jagadeesh Balam, and Boris Ginsburg.
+
+The abstract from the paper is the following:
+
+*This report introduces Canary-1B-v2, a fast, robust multilingual model for Automatic Speech Recognition (ASR) and Speech-to-Text Translation (AST). Built with a FastConformer encoder and Transformer decoder, it supports 25 European languages. The model was trained on 1.7M hours of total data samples, including Granary and NeMo ASR Set 3.0, with non-speech audio added to reduce hallucinations for ASR and AST. We describe its two-stage pre-training and fine-tuning process with dynamic data balancing, as well as experiments with an nGPT encoder. Results show nGPT scales well with massive data, while FastConformer excels after fine-tuning. For timestamps, Canary-1B-v2 uses the NeMo Forced Aligner (NFA) with an auxiliary CTC model, providing reliable segment-level timestamps for ASR and AST. Evaluations show Canary-1B-v2 outperforms Whisper-large-v3 on English ASR while being 10× faster, and delivers competitive multilingual ASR and AST performance against larger models like Seamless-M4T-v2-large and LLM-based systems. We also release Parakeet-TDT-0.6B-v3, a successor to v2, offering multilingual ASR across the same 25 languages with just 600M parameters.*
+
+Canary reuses the [Fast Conformer](https://huggingface.co/papers/2305.05084) encoder from [Parakeet](./parakeet.md) (loaded through [`ParakeetEncoder`] / [`ParakeetEncoderConfig`]) and pairs it with a Transformer decoder that uses fixed sinusoidal positional embeddings, cross-attention to the encoder outputs and tied input/output embeddings. The task is selected through a decoder prompt prefix built by [`CanaryProcessor`] of the form `<|startofcontext|> <|startoftranscript|> <|emo:undefined|> <source_lang> <target_lang> <pnc|nopnc> <|noitn|> <|notimestamp|> <|nodiarize|>`, where `source_lang == target_lang` selects transcription and otherwise selects translation.
+
+The original implementation can be found in [NVIDIA NeMo](https://github.com/NVIDIA/NeMo). A model checkpoint is available at [harshaljanjani/canary-1b-v2-hf](https://huggingface.co/harshaljanjani/canary-1b-v2-hf).
+
+This model was contributed by [Harshal Janjani](https://huggingface.co/harshaljanjani).
+
+## Usage
+
+### Transcription
+
+The simplest way to transcribe audio is with `apply_transcription_request`, which builds the multitask decoder prompt for you (it is a convenience wrapper for `apply_chat_template`).
+
+```python
+from datasets import load_dataset, Audio
+from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq
+
+processor = AutoProcessor.from_pretrained("harshaljanjani/canary-1b-v2-hf")
+model = AutoModelForSpeechSeq2Seq.from_pretrained("harshaljanjani/canary-1b-v2-hf", device_map="auto")
+
+ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
+ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
+
+inputs = processor.apply_transcription_request(audio=ds[0]["audio"]["array"], source_language="en").to(model.device)
+generated_ids = model.generate(**inputs, max_new_tokens=128)
+print(processor.decode(generated_ids, skip_special_tokens=True)[0])
+```
+
+### Translation
+
+Set `target_language` to a different language than `source_language` for speech-to-text translation.
+
+```python
+inputs = processor.apply_transcription_request(
+    audio=ds[0]["audio"]["array"], source_language="en", target_language="de"
+).to(model.device)
+generated_ids = model.generate(**inputs, max_new_tokens=128)
+print(processor.decode(generated_ids, skip_special_tokens=True)[0])
+```
+
+### Batch inference
+
+Pass a list of audios and, optionally, a list of `source_language` / `target_language`.
+
+```python
+audios = [ds[0]["audio"]["array"], ds[1]["audio"]["array"]]
+inputs = processor.apply_transcription_request(
+    audio=audios, source_language="en", target_language=["en", "de"]
+).to(model.device)
+generated_ids = model.generate(**inputs, max_new_tokens=128)
+for text in processor.decode(generated_ids, skip_special_tokens=True):
+    print(text)
+```
+
+### Torch compile
+
+For autoregressive transcription, `torch.compile` accelerates the per-token forward passes inside `generate` by providing a `CompileConfig` object.
+
+```python
+from transformers import CompileConfig
+
+inputs = processor.apply_transcription_request(audio=ds[0]["audio"]["array"], source_language="en").to(model.device)
+compile_config = CompileConfig()
+
+# Warmup
+for _ in range(3):
+    _ = model.generate(**inputs, max_new_tokens=128, cache_implementation="static", compile_config=compile_config)
+
+# Apply model
+generated_ids = model.generate(**inputs, max_new_tokens=128, cache_implementation="static", compile_config=compile_config)
+print(processor.decode(generated_ids, skip_special_tokens=True)[0])
+```
+
+### Training
+
+Canary can be trained with the loss outputted by the model. Build the decoder sequence from the multitask prompt followed by the target text, and mask the prompt in the labels.
+
+```python
+import torch
+
+model.train()
+audio = ds[0]["audio"]["array"]
+transcription = "mister Quilter is the apostle of the middle classes, and we are glad to welcome his gospel."
+
+prompt = processor.apply_transcription_request(audio=audio, source_language="en").to(model.device)
+target_ids = processor(audio=audio, text=transcription)["decoder_input_ids"].to(model.device)
+decoder_input_ids = torch.cat([prompt["decoder_input_ids"], target_ids], dim=1)
+labels = decoder_input_ids.clone()
+labels[:, : prompt["decoder_input_ids"].shape[1]] = -100
+
+outputs = model(
+    input_features=prompt["input_features"],
+    attention_mask=prompt["attention_mask"],
+    decoder_input_ids=decoder_input_ids,
+    labels=labels,
+)
+outputs.loss.backward()
+```
+
+> [!NOTE]
+> Segment-level timestamps for Canary-1B-v2 are produced by the external NeMo Forced Aligner (NFA) with an auxiliary CTC model, not by the decoder, so they are not part of the `generate` output.
+
+## CanaryConfig
+
+[[autodoc]] CanaryConfig
+
+## CanaryProcessor
+
+[[autodoc]] CanaryProcessor
+
+## CanaryModel
+
+[[autodoc]] CanaryModel
+    - forward
+
+## CanaryForConditionalGeneration
+
+[[autodoc]] CanaryForConditionalGeneration
+    - forward

src/transformers/audio_utils.py

@@ -445,6 +445,104 @@ def make_list_of_audio_chat_template(
     return make_list_of_audio(audio)
 
 
+def make_audio_chat_template_content(audio_item) -> dict:
+    """
+    Build a chat-template content dict for a single audio item.
+
+    Args:
+        audio_item (`str` or array-like):
+            A single audio item. Strings are treated as local paths or URLs; other values (numpy/torch arrays) are
+            forwarded directly.
+
+    Returns:
+        `dict`: A chat-template content dict, e.g. `{"type": "audio", "path": ...}` for strings or
+        `{"type": "audio", "audio": ...}` otherwise.
+    """
+    if isinstance(audio_item, str):
+        return {"type": "audio", "path": audio_item}
+    return {"type": "audio", "audio": audio_item}
+
+
+def resolve_language(language: str | None, code_to_name: dict[str, str], return_code: bool = True) -> str | None:
+    """
+    Map a language code or name to its canonical form, with validation.
+
+    Accepts either a language code (e.g. ``"zh"``, ``"en"``) or a full name (e.g. ``"Chinese"``, ``"English"``) and
+    returns the canonical code or name depending on ``return_code``. ``None`` passes through unchanged (auto-detect).
+
+    Args:
+        language (`str` or `None`):
+            The language code or full name to resolve. ``None`` is returned unchanged.
+        code_to_name (`dict[str, str]`):
+            Mapping from language code to full language name for the model's supported languages.
+        return_code (`bool`, *optional*, defaults to `True`):
+            Whether to return the canonical language ``code``. If ``False``, returns the full language ``name``.
+
+    Returns:
+        `str` or `None`: The canonical language code or name, or ``None`` if ``language`` is ``None``.
+
+    Raises:
+        `ValueError`: If the language is not recognized.
+    """
+    if language is None:
+        return None
+
+    language_lower = language.lower()
+    # Try code lookup first, then full-name lookup (both case-insensitive)
+    for code, name in code_to_name.items():
+        if language_lower == code.lower() or language_lower == name.lower():
+            return code if return_code else name
+
+    raise ValueError(
+        f"Unsupported language: {language!r}. Use a language code "
+        f"(e.g. 'en', 'zh') or full name (e.g. 'English', 'Chinese'). "
+        f"Supported codes: {sorted(code_to_name.keys())}. "
+        f"Supported names: {sorted(set(code_to_name.values()))}."
+    )
+
+
+def prepare_language_inputs(
+    language: str | list[str] | None,
+    batch_size: int,
+    code_to_name: dict[str, str],
+    allow_broadcast: bool = False,
+    return_code: bool = True,
+) -> list[str | None]:
+    """
+    Broadcast and validate a language argument to match ``batch_size``.
+
+    Accepts language codes (e.g. ``"zh"``, ``"en"``) or full names (e.g. ``"Chinese"``, ``"English"``). Each value is
+    resolved to its canonical form via [`resolve_language`].
+
+    Args:
+        language (`str`, `list[str]`, or `None`):
+            The language hint(s). A single value is broadcast to the whole batch; a list must match ``batch_size``
+            (unless ``allow_broadcast`` is set). ``None`` disables language hints for the whole batch.
+        batch_size (`int`):
+            The number of samples in the batch.
+        code_to_name (`dict[str, str]`):
+            Mapping from language code to full language name for the model's supported languages.
+        allow_broadcast (`bool`, *optional*, defaults to `False`):
+            Whether a single-element list may be broadcast to the whole batch.
+        return_code (`bool`, *optional*, defaults to `True`):
+            Whether to return canonical language ``code``s. If ``False``, returns full language ``name``s.
+
+    Returns:
+        `list[str | None]`: The resolved language for each sample.
+    """
+    if language is None:
+        return [None] * batch_size
+    if isinstance(language, str):
+        return [resolve_language(language, code_to_name, return_code)] * batch_size
+    if isinstance(language, (list, tuple)):
+        if allow_broadcast and len(language) == 1 and batch_size > 1:
+            return [resolve_language(language[0], code_to_name, return_code)] * batch_size
+        if len(language) != batch_size:
+            raise ValueError(f"Got {len(language)} language(s) for {batch_size} sample(s); counts must match.")
+        return [resolve_language(lang, code_to_name, return_code) for lang in language]
+    raise TypeError("`language` must be a string, a list of strings, or `None`.")
+
+
 def hertz_to_mel(freq: float | np.ndarray, mel_scale: str = "htk") -> float | np.ndarray:
     """
     Convert frequency from hertz to mels.

src/transformers/convert_slow_tokenizer.py

@@ -1876,6 +1876,17 @@ def tokenizer(self, proto):
         return tokenizer
 
 
+class CanaryConverter(ParakeetConverter):
+    def __init__(self, vocab_file=None, *args):
+        super().__init__(vocab_file, *args)
+        # Only the `<|...|>` (type 4) control tokens are special; other type-4 pieces (digits, `<pad>`) are plain text.
+        self.special_tokens = {
+            piece.piece
+            for piece in self.proto.pieces
+            if piece.type == 4 and piece.piece.startswith("<|") and piece.piece.endswith("|>")
+        }
+
+
 def bytes_to_unicode():
     """
     Returns list of utf-8 byte and a mapping to unicode strings. We specifically avoids mapping to whitespace/control

src/transformers/models/__init__.py

@@ -56,6 +56,7 @@
     from .bros import *
     from .byt5 import *
     from .camembert import *
+    from .canary import *
     from .canine import *
     from .chameleon import *
     from .chinese_clip import *

src/transformers/models/audioflamingo3/processing_audioflamingo3.py

@@ -16,7 +16,7 @@
 
 import numpy as np
 
-from ...audio_utils import AudioInput, make_list_of_audio_chat_template
+from ...audio_utils import AudioInput, make_audio_chat_template_content, make_list_of_audio_chat_template
 from ...feature_extraction_utils import BatchFeature
 from ...processing_utils import ProcessingKwargs, ProcessorMixin, Unpack
 from ...tokenization_utils_base import TextInput
@@ -236,9 +236,7 @@ def apply_transcription_request(
                     "role": "user",
                     "content": [
                         {"type": "text", "text": prompt_text},
-                        {"type": "audio", "path": audio_item}
-                        if isinstance(audio_item, str)
-                        else {"type": "audio", "audio": audio_item},
+                        make_audio_chat_template_content(audio_item),
                     ],
                 }
             ]

src/transformers/models/auto/auto_mappings.py

@@ -73,6 +73,7 @@
         ("bridgetower_vision_model", "BridgeTowerVisionConfig"),
         ("bros", "BrosConfig"),
         ("camembert", "CamembertConfig"),
+        ("canary", "CanaryConfig"),
         ("canine", "CanineConfig"),
         ("chameleon", "ChameleonConfig"),
         ("chameleon_vqgan", "ChameleonVQVAEConfig"),
@@ -1016,6 +1017,7 @@
         ("blip-2", "Blip2Processor"),
         ("bridgetower", "BridgeTowerProcessor"),
         ("bros", "BrosProcessor"),
+        ("canary", "CanaryProcessor"),
         ("chameleon", "ChameleonProcessor"),
         ("chinese_clip", "ChineseCLIPProcessor"),
         ("clap", "ClapProcessor"),

src/transformers/models/auto/feature_extraction_auto.py

@@ -37,6 +37,7 @@
 MISSING_FEATURE_EXTRACTOR_MAPPING_NAMES = OrderedDict(
     [
         ("audioflamingo3", "WhisperFeatureExtractor"),
+        ("canary", "ParakeetFeatureExtractor"),
         ("csm", "EncodecFeatureExtractor"),
         ("data2vec-audio", "Wav2Vec2FeatureExtractor"),
         ("glmasr", "WhisperFeatureExtractor"),

src/transformers/models/auto/modeling_auto.py

@@ -77,6 +77,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
         ("bridgetower", "BridgeTowerModel"),
         ("bros", "BrosModel"),
         ("camembert", "CamembertModel"),
+        ("canary", "CanaryModel"),
         ("canine", "CanineModel"),
         ("chameleon", "ChameleonModel"),
         ("chinese_clip", "ChineseCLIPModel"),
@@ -1314,6 +1315,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
 
 MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING_NAMES = OrderedDict(
     [
+        ("canary", "CanaryForConditionalGeneration"),
         ("cohere_asr", "CohereAsrForConditionalGeneration"),
         ("dia", "DiaForConditionalGeneration"),
         ("granite_speech", "GraniteSpeechForConditionalGeneration"),

src/transformers/models/auto/tokenization_auto.py

@@ -90,6 +90,7 @@
         ("bros", "BertTokenizer" if is_tokenizers_available() else None),
         ("byt5", "ByT5Tokenizer"),
         ("camembert", "CamembertTokenizer" if is_tokenizers_available() else None),
+        ("canary", "TokenizersBackend" if is_tokenizers_available() else None),
         ("canine", "CanineTokenizer"),
         ("chinese_clip", "BertTokenizer" if is_tokenizers_available() else None),
         ("clap", "RobertaTokenizer"),

src/transformers/models/canary/__init__.py

@@ -0,0 +1,28 @@
+# Copyright 2026 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from typing import TYPE_CHECKING
+
+from ...utils import _LazyModule
+from ...utils.import_utils import define_import_structure
+
+
+if TYPE_CHECKING:
+    from .configuration_canary import *
+    from .modeling_canary import *
+    from .processing_canary import *
+else:
+    import sys
+
+    _file = globals()["__file__"]
+    sys.modules[__name__] = _LazyModule(__name__, _file, define_import_structure(_file), module_spec=__spec__)