Support vibevoice_asr (audio diffusion pipeline)

[justinchuby]

Overview

vibevoice_asr is Microsoft's speech recognition/synthesis model using a 4-stage diffusion pipeline. Available models: microsoft/VibeVoice-ASR (512K downloads), microsoft/VibeVoice-1.5B (63K downloads).

model_type: vibevoice
HF archs: VibeVoiceForASRTraining, VibeVoiceForConditionalGeneration

Architecture (4 stages)

Audio input
  → Semantic Tokenizer  (causal EnCodec-style CNN, ratios=[8,5,5,4,2,2])
  → Acoustic Tokenizer  (separate causal CNN encoder, same structure)
  → Qwen2 LM Decoder    (hidden=1536–3584, 28 layers — reusable)
  → DDPM Diffusion Head (20 inference steps, cosine schedule, latent_size=64)
  → Audio output

Component	Type	Reusable?
Semantic tokenizer	Causal CNN (EnCodec-style)	❌ New
Acoustic tokenizer	Causal CNN (EnCodec-style)	❌ New
Qwen2 LM decoder	Standard transformer	✅ Reuse Qwen2CausalLMModel
DDPM diffusion head	Diffusion (20 steps, cosine)	❌ New

Complexity: High

This is not a standard transformer model — it is a diffusion-based audio pipeline. New infrastructure required:

1. Causal convolutional audio encoder (EnCodec-style, depthwise conv with downsampling ratios)
2. VQ-VAE / audio tokenizer for semantic and acoustic token quantization
3. DDPM diffusion head — vibepod_diffusion_head, cosine noise schedule, 20-step inference loop
4. Multi-stage pipeline — unlike standard input_ids → logits, this has 4 distinct stages with different I/O contracts

Estimated effort: 4–6 weeks (new component family for audio diffusion).

Dependencies

• Audio diffusion infrastructure (partially exists for image diffusion — see diffusion models skill)
• Causal CNN encoder component (new, similar to audio codec encoders)
• DDPM inference loop support in ONNX graph
• New AudioDiffusionTask or similar task class

Priority: P2

Lower download counts relative to complexity. The Qwen2 backbone is reusable, but represents only a small fraction of the total work. Recommend deferring until audio diffusion infrastructure is established (potentially after image diffusion is more mature).

Related

• microsoft/VibeVoice-Realtime-0.5B (364K downloads) — same architecture, smaller
• microsoft/VibeVoice-ASR-HF (257K downloads) — HF-compatible variant
• Issue Support protein/biology models: MaskedLMTask + ESMFold + Evolla #80 (protein/biology models) — independent