[canary] Refactor: `PromptedAudioToTextLhotseDataset` and `EncDecMultiTaskModel`#8247

Merged

pzelasko merged 7 commits intocanary-2from canary-2-dataset-refactor

Jan 26, 2024

+1,142-217

Commits on Jan 25, 2024

Create a separate CanaryDataset and use it inside transformer_bpe_models.py. Ditches token_sequence_format.
pzelasko
committed

Commits on Jan 26, 2024

[canary] Refactor: move changes in transformer_bpe_models.py to Canar… (#8252 )
pzelasko
authored
Rename CanaryDataset to PromptedAudioToTextLhotseDataset; add prompt_format_fn argument; clean-up the _canary_prompt_format function a bit
pzelasko
committed
Move tokenization into prompt_format_fn, fix usage, add docs
pzelasko
committed
Backward-compatible utterance validation
pzelasko
committed
Improve type annotations
pzelasko
committed
config and prompt_fn registration changes from review
pzelasko
committed