model: Add NVIDIA Canary-1B-v2 to Transformers

docs/source/en/_toctree.yml

@@ -1091,6 +1091,8 @@
         title: Audio Spectrogram Transformer
       - local: model_doc/bark
         title: Bark
+      - local: model_doc/canary
+        title: Canary
       - local: model_doc/clap
         title: CLAP
       - local: model_doc/cohere_asr

docs/source/en/model_doc/canary.md

@@ -0,0 +1,147 @@
+<!--Copyright 2026 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+*This model was published in HF papers on 2025-09-17 and contributed to Hugging Face Transformers on 2026-07-09.*
+
+<div class="flex flex-wrap space-x-1">
+<img alt="SDPA" src="https://img.shields.io/badge/SDPA-DE3412?style=flat&logo=pytorch&logoColor=white">
+</div>
+
+# Canary
+
+## Overview
+
+Canary-1B-v2 was proposed in [Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and AST](https://huggingface.co/papers/2509.14128) by Monica Sekoyan, Nithin Rao Koluguri, Nune Tadevosyan, Piotr Zelasko, Travis Bartley, Nikolay Karpov, Jagadeesh Balam, and Boris Ginsburg.
+
+The abstract from the paper is the following:
+
+*This report introduces Canary-1B-v2, a fast, robust multilingual model for Automatic Speech Recognition (ASR) and Speech-to-Text Translation (AST). Built with a FastConformer encoder and Transformer decoder, it supports 25 European languages. The model was trained on 1.7M hours of total data samples, including Granary and NeMo ASR Set 3.0, with non-speech audio added to reduce hallucinations for ASR and AST. We describe its two-stage pre-training and fine-tuning process with dynamic data balancing, as well as experiments with an nGPT encoder. Results show nGPT scales well with massive data, while FastConformer excels after fine-tuning. For timestamps, Canary-1B-v2 uses the NeMo Forced Aligner (NFA) with an auxiliary CTC model, providing reliable segment-level timestamps for ASR and AST. Evaluations show Canary-1B-v2 outperforms Whisper-large-v3 on English ASR while being 10× faster, and delivers competitive multilingual ASR and AST performance against larger models like Seamless-M4T-v2-large and LLM-based systems. We also release Parakeet-TDT-0.6B-v3, a successor to v2, offering multilingual ASR across the same 25 languages with just 600M parameters.*
+
+Canary reuses the [Fast Conformer](https://huggingface.co/papers/2305.05084) encoder from [Parakeet](./parakeet.md) (loaded through [`ParakeetEncoder`] / [`ParakeetEncoderConfig`]) and pairs it with a Transformer decoder that uses fixed sinusoidal positional embeddings, cross-attention to the encoder outputs and tied input/output embeddings. The task is selected through a decoder prompt prefix built by [`CanaryProcessor`] of the form `<|startofcontext|> <|startoftranscript|> <|emo:undefined|> <source_lang> <target_lang> <pnc|nopnc> <|noitn|> <|notimestamp|> <|nodiarize|>`, where `source_lang == target_lang` selects transcription and otherwise selects translation.
+
+The original implementation can be found in [NVIDIA NeMo](https://github.com/NVIDIA/NeMo). A model checkpoint is available at [harshaljanjani/canary-1b-v2-hf](https://huggingface.co/harshaljanjani/canary-1b-v2-hf).
+
+## Usage
+
+### Transcription
+
+The simplest way to transcribe audio is with `apply_transcription_request`, which builds the multitask decoder prompt for you (it is a convenience wrapper for `apply_chat_template`).
+
+```python
+from datasets import load_dataset, Audio
+from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq
+
+processor = AutoProcessor.from_pretrained("harshaljanjani/canary-1b-v2-hf")
+model = AutoModelForSpeechSeq2Seq.from_pretrained("harshaljanjani/canary-1b-v2-hf", device_map="auto")
+
+ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
+ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
+
+inputs = processor.apply_transcription_request(audio=ds[0]["audio"]["array"], source_language="en").to(model.device)
+generated_ids = model.generate(**inputs, max_new_tokens=128)
+print(processor.batch_decode(generated_ids, skip_special_tokens=True)[0])
+```
+
+### Translation
+
+Set `target_language` to a different language than `source_language` for speech-to-text translation.
+
+```python
+inputs = processor.apply_transcription_request(
+    audio=ds[0]["audio"]["array"], source_language="en", target_language="de"
+).to(model.device)
+generated_ids = model.generate(**inputs, max_new_tokens=128)
+print(processor.batch_decode(generated_ids, skip_special_tokens=True)[0])
+```
+
+### Batch inference
+
+Pass a list of audios and, optionally, a list of `source_language` / `target_language`.
+
+```python
+audios = [ds[0]["audio"]["array"], ds[1]["audio"]["array"]]
+inputs = processor.apply_transcription_request(
+    audio=audios, source_language="en", target_language=["en", "de"]
+).to(model.device)
+generated_ids = model.generate(**inputs, max_new_tokens=128)
+for text in processor.batch_decode(generated_ids, skip_special_tokens=True):
+    print(text)
+```
+
+### Torch compile
+
+For autoregressive transcription, `torch.compile` accelerates the per-token forward passes inside `generate` by providing a `CompileConfig` object.
+
+```python
+from transformers import CompileConfig
+
+inputs = processor.apply_transcription_request(audio=ds[0]["audio"]["array"], source_language="en").to(model.device)
+compile_config = CompileConfig()
+
+# Warmup
+for _ in range(3):
+    _ = model.generate(**inputs, max_new_tokens=128, cache_implementation="static", compile_config=compile_config)
+
+# Apply model
+generated_ids = model.generate(**inputs, max_new_tokens=128, cache_implementation="static", compile_config=compile_config)
+print(processor.batch_decode(generated_ids, skip_special_tokens=True)[0])
+```
+
+### Training
+
+Canary can be trained with the loss outputted by the model. Build the decoder sequence from the multitask prompt followed by the target text, and mask the prompt in the labels.
+
+```python
+import torch
+
+model.train()
+audio = ds[0]["audio"]["array"]
+transcription = "mister Quilter is the apostle of the middle classes, and we are glad to welcome his gospel."
+
+prompt = processor.apply_transcription_request(audio=audio, source_language="en").to(model.device)
+target_ids = processor(audio=audio, text=transcription)["decoder_input_ids"].to(model.device)
+decoder_input_ids = torch.cat([prompt["decoder_input_ids"], target_ids], dim=1)
+labels = decoder_input_ids.clone()
+labels[:, : prompt["decoder_input_ids"].shape[1]] = -100
+
+outputs = model(
+    input_features=prompt["input_features"],
+    attention_mask=prompt["attention_mask"],
+    decoder_input_ids=decoder_input_ids,
+    labels=labels,
+)
+outputs.loss.backward()
+```
+
+> [!NOTE]
+> Segment-level timestamps for Canary-1B-v2 are produced by the external NeMo Forced Aligner (NFA) with an auxiliary CTC model, not by the decoder, so they are not part of the `generate` output.
+
+## CanaryConfig
+
+[[autodoc]] CanaryConfig
+
+## CanaryProcessor
+
+[[autodoc]] CanaryProcessor
+
+## CanaryModel
+
+[[autodoc]] CanaryModel
+    - forward
+
+## CanaryForConditionalGeneration
+
+[[autodoc]] CanaryForConditionalGeneration
+    - forward

src/transformers/convert_slow_tokenizer.py

@@ -1876,6 +1876,17 @@ def tokenizer(self, proto):
         return tokenizer
 
 
+class CanaryConverter(ParakeetConverter):
+    def __init__(self, vocab_file=None, *args):
+        super().__init__(vocab_file, *args)
+        # Only the `<|...|>` (type 4) control tokens are special; other type-4 pieces (digits, `<pad>`) are plain text.
+        self.special_tokens = {
+            piece.piece
+            for piece in self.proto.pieces
+            if piece.type == 4 and piece.piece.startswith("<|") and piece.piece.endswith("|>")
+        }
+
+
 def bytes_to_unicode():
     """
     Returns list of utf-8 byte and a mapping to unicode strings. We specifically avoids mapping to whitespace/control

src/transformers/models/__init__.py

@@ -56,6 +56,7 @@
     from .bros import *
     from .byt5 import *
     from .camembert import *
+    from .canary import *
     from .canine import *
     from .chameleon import *
     from .chinese_clip import *

src/transformers/models/auto/auto_mappings.py

@@ -73,6 +73,7 @@
         ("bridgetower_vision_model", "BridgeTowerVisionConfig"),
         ("bros", "BrosConfig"),
         ("camembert", "CamembertConfig"),
+        ("canary", "CanaryConfig"),
         ("canine", "CanineConfig"),
         ("chameleon", "ChameleonConfig"),
         ("chameleon_vqgan", "ChameleonVQVAEConfig"),
@@ -1008,6 +1009,7 @@
         ("blip-2", "Blip2Processor"),
         ("bridgetower", "BridgeTowerProcessor"),
         ("bros", "BrosProcessor"),
+        ("canary", "CanaryProcessor"),
         ("chameleon", "ChameleonProcessor"),
         ("chinese_clip", "ChineseCLIPProcessor"),
         ("clap", "ClapProcessor"),

src/transformers/models/auto/feature_extraction_auto.py

@@ -37,6 +37,7 @@
 MISSING_FEATURE_EXTRACTOR_MAPPING_NAMES = OrderedDict(
     [
         ("audioflamingo3", "WhisperFeatureExtractor"),
+        ("canary", "ParakeetFeatureExtractor"),
         ("csm", "EncodecFeatureExtractor"),
         ("data2vec-audio", "Wav2Vec2FeatureExtractor"),
         ("glmasr", "WhisperFeatureExtractor"),

src/transformers/models/auto/modeling_auto.py

@@ -77,6 +77,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
         ("bridgetower", "BridgeTowerModel"),
         ("bros", "BrosModel"),
         ("camembert", "CamembertModel"),
+        ("canary", "CanaryModel"),
         ("canine", "CanineModel"),
         ("chameleon", "ChameleonModel"),
         ("chinese_clip", "ChineseCLIPModel"),
@@ -1310,6 +1311,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
 
 MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING_NAMES = OrderedDict(
     [
+        ("canary", "CanaryForConditionalGeneration"),
         ("cohere_asr", "CohereAsrForConditionalGeneration"),
         ("dia", "DiaForConditionalGeneration"),
         ("granite_speech", "GraniteSpeechForConditionalGeneration"),

src/transformers/models/auto/tokenization_auto.py

@@ -90,6 +90,7 @@
         ("bros", "BertTokenizer" if is_tokenizers_available() else None),
         ("byt5", "ByT5Tokenizer"),
         ("camembert", "CamembertTokenizer" if is_tokenizers_available() else None),
+        ("canary", "TokenizersBackend" if is_tokenizers_available() else None),
         ("canine", "CanineTokenizer"),
         ("chinese_clip", "BertTokenizer" if is_tokenizers_available() else None),
         ("clap", "RobertaTokenizer"),

src/transformers/models/canary/__init__.py

@@ -0,0 +1,28 @@
+# Copyright 2026 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from typing import TYPE_CHECKING
+
+from ...utils import _LazyModule
+from ...utils.import_utils import define_import_structure
+
+
+if TYPE_CHECKING:
+    from .configuration_canary import *
+    from .modeling_canary import *
+    from .processing_canary import *
+else:
+    import sys
+
+    _file = globals()["__file__"]
+    sys.modules[__name__] = _LazyModule(__name__, _file, define_import_structure(_file), module_spec=__spec__)

src/transformers/models/canary/configuration_canary.py

@@ -0,0 +1,132 @@
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
+#           This file was automatically generated from src/transformers/models/canary/modular_canary.py.
+#               Do NOT edit this file manually as any edits will be overwritten by the generation of
+#             the file from the modular. If any change should be done, please apply the change to the
+#                          modular_canary.py file directly. One of our CI enforces this.
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
+# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from huggingface_hub.dataclasses import strict
+
+from ...configuration_utils import PreTrainedConfig
+from ...utils import auto_docstring
+from ..parakeet import ParakeetEncoderConfig
+
+
+@auto_docstring(checkpoint="harshaljanjani/canary-1b-v2-hf")
+@strict
+class CanaryConfig(PreTrainedConfig):
+    r"""
+    encoder_config (`Union[dict, ParakeetEncoderConfig]`, *optional*):
+        The config object or dictionary of the FastConformer encoder ([`ParakeetEncoderConfig`]).
+    vocab_size (`int`, *optional*, defaults to 16384):
+        Vocabulary size of the Canary decoder.
+    d_model (`int`, *optional*, defaults to 1024):
+        Dimensionality of the decoder layers and the pooler layer.
+    decoder_layers (`int`, *optional*, defaults to 8):
+        Number of decoder layers.
+    decoder_attention_heads (`int`, *optional*, defaults to 8):
+        Number of attention heads for each attention layer in the decoder.
+    decoder_ffn_dim (`int`, *optional*, defaults to 4096):
+        Dimensionality of the "intermediate" (often named feed-forward) layer in the decoder.
+    decoder_layerdrop (`float`, *optional*, defaults to 0.0):
+        The LayerDrop probability for the decoder. See the [LayerDrop paper](https://huggingface.co/papers/1909.11556)
+        for more details.
+    activation_function (`str`, *optional*, defaults to `"relu"`):
+        The non-linear activation function in the decoder feed-forward layers.
+    max_target_positions (`int`, *optional*, defaults to 1024):
+        The maximum sequence length that the decoder might ever be used with.
+    dropout (`float`, *optional*, defaults to 0.1):
+        The dropout probability for the decoder embeddings, attention output, and feed-forward layers.
+    activation_dropout (`float`, *optional*, defaults to 0.1):
+        The dropout ratio for activations inside the decoder feed-forward layer.
+    scale_embedding (`bool`, *optional*, defaults to `False`):
+        Whether to scale the decoder token embeddings by `sqrt(d_model)`.
+    use_cache (`bool`, *optional*, defaults to `True`):
+        Whether the model should return the last key/values attentions.
+    is_encoder_decoder (`bool`, *optional*, defaults to `True`):
+        Whether the model is used as an encoder/decoder model.
+    tie_word_embeddings (`bool`, *optional*, defaults to `True`):
+        Whether to tie the decoder input embeddings and the language modeling head.
+    pad_token_id (`int`, *optional*, defaults to 2):
+        Padding token id.
+    bos_token_id (`int`, *optional*, defaults to 4):
+        Beginning of stream token id (`<|startoftranscript|>`).
+    eos_token_id (`int`, *optional*, defaults to 3):
+        End of stream token id (`<|endoftext|>`).
+    decoder_start_token_id (`int`, *optional*, defaults to 7):
+        The token id that starts decoding (`<|startofcontext|>`, the first token of the multitask prompt).
+
+    Example:
+
+    ```python
+    >>> from transformers import CanaryForConditionalGeneration, CanaryConfig
+
+    >>> # Initializing a Canary configuration
+    >>> configuration = CanaryConfig()
+
+    >>> # Initializing a model from the configuration
+    >>> model = CanaryForConditionalGeneration(configuration)
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```
+    """
+
+    model_type = "canary"
+    keys_to_ignore_at_inference = ["past_key_values"]
+    sub_configs = {"encoder_config": ParakeetEncoderConfig}
+    attribute_map = {
+        "hidden_size": "d_model",
+        "num_attention_heads": "decoder_attention_heads",
+        "num_hidden_layers": "decoder_layers",
+    }
+
+    encoder_config: dict | PreTrainedConfig | None = None
+    vocab_size: int = 16384
+    d_model: int = 1024
+    decoder_layers: int = 8
+    decoder_attention_heads: int = 8
+    decoder_ffn_dim: int = 4096
+    decoder_layerdrop: float | int = 0.0
+    activation_function: str = "relu"
+    max_target_positions: int = 1024
+    dropout: float | int = 0.1
+    attention_dropout: float | int = 0.1
+    activation_dropout: float | int = 0.1
+    scale_embedding: bool = False
+    use_cache: bool = True
+    is_encoder_decoder: bool = True
+    tie_word_embeddings: bool = True
+    pad_token_id: int | None = 2
+    bos_token_id: int | None = 4
+    eos_token_id: int | None = 3
+    decoder_start_token_id: int | None = 7
+    initializer_range: float = 0.02
+
+    def __post_init__(self, **kwargs):
+        if isinstance(self.encoder_config, dict):
+            self.encoder_config = ParakeetEncoderConfig(**self.encoder_config)
+        elif self.encoder_config is None:
+            self.encoder_config = ParakeetEncoderConfig(
+                num_hidden_layers=32,
+                num_mel_bins=128,
+                scale_input=False,
+                layerdrop=0.0,
+                dropout_positions=0.0,
+            )
+        super().__post_init__(**kwargs)
+
+
+__all__ = ["CanaryConfig"]