Skip to content

feat: add LFM2-Audio / LFM2.5-Audio speech input and output support - #2601

Merged
Tianlei Wu (tianleiwu) merged 24 commits into
microsoft:mainfrom
ykhrustalev:ykhrustalev/add-lfm2-audio-support
Sep 24, 2026
Merged

Tianlei Wu (tianleiwu) merged 24 commits into
microsoft:mainfrom
ykhrustalev:ykhrustalev/add-lfm2-audio-support

Conversation

@ykhrustalev

@ykhrustalev Yuri Khrustalev (ykhrustalev) commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem
LFM2-Audio and LFM2.5-Audio cannot be used here: nothing turns a sound into something the model can read or makes room for it in the prompt, nothing lets the model answer in speech, and the model builder cannot read their checkpoints, which have no model type and nest the LFM2 decoder inside the config.

Solution

  • Adds an lfm2_audio pipeline (speech encoder → embedding → LFM2 decoder) with a C++ processor that computes NeMo's log-mel front end and swaps each <|audio|> marker for one placeholder per encoder frame; clips are resampled with a port of torchaudio's windowed sinc, as the reference does
  • Adds speech output (model.audio_output): the depthformer turns each decoder hidden state into a frame of eight audio codes whose embedding is the decoder's next input, switching between text and speech as generate_sequential and generate_interleaved do (search.audio_interleaved, audio_temperature, audio_top_k); the sequence keeps one placeholder per frame and the codes come from get_output("audio_codes")
  • Reads these checkpoints in the model builder, by folder or by name, as lfm2_audio or the text-only lfm2_audio_text; a text-only build stops at <|audio_start|> and <|text_end|>
  • Requires one marker per clip and one prompt at a time; refuses beam search and adapter_path (an adapter trained on these checkpoints is keyed lfm.*, so PEFT would attach untrained weights with only a warning)
  • Touches shared code in two places: Tokenizer::Decode returns an empty string for no tokens instead of dying with SIGFPE, and Images.open / Audios.open no longer keep pointers into a growing path list
  • Adds a tutorial: build, precisions, modes, speech output and its waveform, known limitations

Testing

  • A tiny model under test/models/lfm2-audio runs the whole pipeline in CI against a plain ONNX Runtime replay, speech output included: tokens and audio codes must match the replay of the reference's text/speech switching exactly
  • Against liquid-audio, fp32: ASR and chat agree to 6e-4 in the logits with every token the same on all three published checkpoints; with the audio taken greedily on both sides, TTS and interleaved answers match in every text token and every audio frame (200 items on the reference's question.wav)
  • LiquidAI's published depthformer graph never applies its rotary embedding and builds the tables for the wrong theta, so seven of its eight codebooks disagree with the PyTorch model; the tutorial has the six-line fix, after which all eight agree to 1e-5
  • Speech made with the model card's sampling reads back through this runtime's own ASR as the text the model wrote

Follow-up

Copilot AI lite review requested due to automatic review settings September 20, 2026 20:13
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Comment thread test/python/models/test_lfm2_audio_models.py Fixed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Adapter loading, checkpoint memory usage, and beam-batching issues remain unresolved, along with documentation clarifications.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 High severity · 4 Low severity

Open (5)
What changed in this PR

Adds LFM2-Audio and LFM2.5-Audio speech-input support, model loading, preprocessing, runtime integration, documentation, and tests. Also fixes Python media path lifetime handling.

Changes:

  • Adds log-mel processing, prompt expansion, and per-clip speech encoding.
  • Adds nested checkpoint builder support and model registration.
  • Adds end-to-end tests, fixtures, tutorials, and multi-path media fixes.
File Summary
test/​python/​test_onnxruntime_genai_api.py Tests multi-path media loading.
test/​python/​models/​test_lfm2_audio_models.py Tests audio processing and generation.
test/​python/​create/​create_dummy_lfm2_audio_models.py Generates dummy audio models.
test/​python/​builder/​test_qmoe_weights.py Updates builder test stubs.
test/​python/​builder/​test_lfm2_audio.py Tests audio checkpoint loading.
test/​models/​lfm2-audio/​tokenizer_config.json Adds tokenizer fixture configuration.
test/​models/​lfm2-audio/​genai_config.json Adds pipeline fixture configuration.
test/​cpp/​lfm2_audio_frames_test.cpp Tests frame and prompt accounting.
test/​CMakeLists.txt Registers audio tests.
src/​python/​python.cpp Fixes media path pointer lifetimes.
src/​python/​py/​models/​README.md Documents audio decoder support.
src/​python/​py/​models/​builders/​lfm2.py Loads nested audio checkpoints.
src/​python/​py/​models/​builders/​__init__.py Exports audio builder support.
src/​python/​py/​models/​builder.py Dispatches audio model builds.
src/​models/​preprocessing/​multi_modal_processor.cpp Registers the audio processor.
src/​models/​preprocessing/​lfm2_audio_processor.h Declares audio processing interfaces.
src/​models/​preprocessing/​lfm2_audio_processor.cpp Implements audio preprocessing.
src/​models/​multi_modal.h Declares speech state handling.
src/​models/​multi_modal.cpp Runs and concatenates speech features.
src/​models/​model.cpp Creates audio multimodal models.
src/​models/​model_type.h Registers audio model types.
src/​generator/​generators.cpp Disables unsupported rewind behavior.
src/​config.h Adds audio-length configuration.
src/​config.cpp Parses audio-length configuration.
README.md Updates supported architectures.
examples/​python/​lfm2-audio-tutorial.md Documents building and using audio models.
examples/​python/​common.py Adds audio prompt markers.
.gitignore Includes audio test fixtures.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/python/py/models/builders/lfm2.py
Comment thread examples/python/lfm2-audio-tutorial.md Outdated
Comment thread src/models/preprocessing/lfm2_audio_processor.cpp Outdated
Comment thread src/python/py/models/builders/lfm2.py Outdated
Comment thread src/python/py/models/builders/lfm2.py Outdated
LFM2-Audio pairs a FastConformer speech encoder with an LFM2 decoder.
Nothing here could read its checkpoints or turn a sound into something
the model could read.

- Adds audio handling for LFM2-Audio: turn the sound into log-mel frames
  the way the original does, hand them to the speech encoder, and splice
  what comes back into the prompt.
- Swaps each `<|audio|>` marker in the prompt for the right number of
  empty slots, so the sound and the text line up. The prompt has to carry
  exactly one marker per clip, in clip order; any other count is refused
  up front rather than guessed at. One prompt at a time: a one-item list
  works, longer lists are refused instead of silently losing the text.
- Reads these checkpoints in the model builder. They have no model type
  and no transformers class: the text half sits nested inside the config
  next to the encoder and the speech-output parts, so it is picked out by
  hand and everything else left alone.
- Registers the two new model names, so the loader knows how to run them:
  one for the speech model, one for its text half.
- Several clips in one prompt share the prompt but not the encoder run:
  the published encoder was exported for one clip at a time, so each clip
  is run on its own frames and the results joined in order.
- Adds a tutorial. Known gap: the model can also speak, and that is not
  supported here; it answers in text only.
@ykhrustalev
Yuri Khrustalev (ykhrustalev) force-pushed the ykhrustalev/add-lfm2-audio-support branch from 6dad651 to 40ee98e Compare September 20, 2026 20:43
@ykhrustalev
Yuri Khrustalev (ykhrustalev) marked this pull request as draft September 20, 2026 22:38
Review fixes:
- Apply the LoRA adapter named by `adapter_path` when loading an
  LFM2-Audio decoder. The override returned before the base loader's
  adapter step, so the adapter was dropped without a word. The step is
  now a method the base loader and this one share.
- Move the nested-config reader onto the model class and drop the
  module-level architecture name, per the model-builder rules.
- Correct the comments and the tutorial: the clips of one prompt are
  staged in one padded tensor, but the encoder runs once per clip, on
  that clip's own frames. Nothing says they share a run any more.
- Bind the subscript CodeQL read as a statement with no effect.

Wider tests, to match what the vision model has:
- A text-only audio export runs through the LFM2 runtime, alongside the
  vision and MoE cases already covered there.
- Frame and token accounting as a standalone arithmetic file, checked
  against the reference implementation when it is installed, mirroring
  the vision token tests.
- Opening a clip from bytes matches opening it from a path.
- Clips above the encoder's rate are resampled down to it.
- Every documented front-end override is honoured, including the one
  that changes how many placeholders a clip is worth.
- A clip recorded below the encoder's rate is refused with a message
  naming the clip and saying to resample it first: the decoder only
  resamples downwards, which nothing said before.
Passing the checkpoint by name (`-m LiquidAI/LFM2.5-Audio-1.5B`) leaves
the input path empty, and the loader handed that empty string to the Hub
as a repository name. Only `-i <directory>` worked. Fall back to the
name the config was read from, and cover both ways round.

Found by exporting every LFM2 audio checkpoint LiquidAI publishes.
Noted in the tutorial which of them the published encoder belongs to,
since the other two need their own, and that audio below the encoder's
rate has to be resampled before it gets here.
Both had grown to a hundred lines covering several unrelated jobs.
The processor now decodes a clip, builds the prompt tokens, and stages
the audio tensors in three methods named after those jobs; the speech
state resolves its bindings, runs one clip, and joins the results in
three more. No behaviour changes.

Also covers reading a checkpoint that `save_pretrained` split into
several files, which an fp32 fine-tune of this model would be.
`-p int4` quantizes the decoder and nothing else. The embedding table
and the speech encoder are put together by hand, so they stay at
whatever they were built or downloaded at, and together they outweigh
the quantized decoder: 1787 MB for a build where only 730 MB of it was
quantized.

Measured all five combinations on three clips. Every int4 build wrote
the same words as every other, so a half-width embedding table and the
published 4-bit encoder cost nothing and take the build to 1178 MB. The
only difference anywhere is the int4 decoder against fp32, which dropped
one comma.

Also: the encoder variants refer to their own data file by name, so
renaming the pair breaks the build with an external-data error. Say so,
and stop the example renaming it.
These checkpoints answer in speech as well as text, chosen by the system
prompt. Asked for speech, the model emits <|audio_start|> and everything
after it is audio codes meant for the depthformer, which this runtime
does not have. Those positions were being read off the text head, which
gave fluent, plausible nonsense with nothing to say it was wrong: a TTS
prompt returned eleven tokens of invented sentence.

<|audio_start|> now goes into the stop tokens when the decoder is built,
so an answer that turns to speech ends there instead, keeping whatever
text came before it.

Of the model's three modes, only transcription was ever exercised. Now
measured on the real model: transcription is unchanged and still matches
the reference token for token; a speech prompt returns nothing at all
rather than nonsense; and the interleaved prompt returns text, which on
a ten second clip was the reference's interleaved text exactly, all 29
tokens of it, before the reference carried on into speech. The tutorial
says which prompt gives what, and that the temperatures quoted for the
model are for the speech stream this runtime does not produce.
The earlier note was measured with the audio taken greedily, which is
not how the model card runs interleaved generation, and it overstated
the case twice over: it said this runtime returns the text up to the
first switch to speech, when the switch comes after six tokens and the
whole text still came through; and it said the reference carries on past
where this stops, which was an artefact of that sampling.

Generated the speech properly this time, with the sampling the model
card gives, and decoded it. The reference answered with 29 text tokens
and 48 audio frames; those frames are 3.8 seconds of real speech, which
our own transcription reads back as the opening of the same answer. This
runtime gives the same 29 tokens. With the audio taken greedily the
reference instead runs to 54 tokens, so the text does depend on the
speech, and the agreement is not a guarantee. Says so now.
Asked the reference to speak a known sentence in two of its voices, and
read the result back with our own transcription: both came back as the
sentence exactly, punctuation and all. The two voices are different
waveforms, and the male one carries more of its energy low down than the
female one. So the speech this runtime leaves on the table is good, not
a curiosity, and the earlier partial read-back was a truncated clip
rather than poor audio.

Also: a speech prompt yields no text at all, not merely a short one.
The entire text stream is the token that starts the audio followed by
the end of the turn.
Stopping when the model starts speaking covered the case where the whole
answer is spoken, but not the interleaved one, where the model writes an
answer, marks the text finished, and says the rest aloud. Asked a spoken
question that way, the answer came back followed by thirty-odd tokens of
invented whitespace read off the text head at positions meant for the
depthformer.

Both tokens that hand the turn over to speech now end generation. Asked
"What is the capital of France?", the interleaved answer is now exactly
"The capital of France is Paris" -- the text the model wrote before it
began speaking, and what it then spoke. Transcription and a turn that
stays in text are unchanged.

Found by listening to the speech instead of only counting its frames:
synthesising a question, asking it back, and reading the answer aloud.
… fire

Went through every way the audio code can refuse something and asked
which were actually reachable. The tensors the processor returns can be
replaced before they reach the generator, so the checks that they still
agree with each other are reachable, and five of them now have tests:
a mel tensor of the wrong rank, a clip count that disagrees with the
placeholders, a frame count per clip that disagrees with the tensor, and
a clip claiming frames past the end of it, which would otherwise be
copied out of bounds. Beam search with several clips is refused too,
since they share one buffer.

One check could never fire: it compared the sum of the per-clip token
counts against a total derived from those same counts upstream. Removed.

Also covers the three front-end settings that change the numbers rather
than the shapes, where a default quietly ignoring the override would
have looked perfectly healthy.
Went looking for a way to reach the last unreached check, that the
decoder hands back mono, and found a worse problem instead. The decoder
downmixes one and two channels correctly, but deinterleaves anything
wider as though it were stereo: a six channel clip arrives three times
too long and garbled, and transcribes to nonsense. Silently.

It cannot be caught here. Neither setting of the downmix flag reports
how many channels a file had, and nothing else in the decoder's
interface exposes it, so there is no way to tell six channels from two.
The tutorial now says to mix down to mono or stereo first.

The test pins the behaviour rather than blessing it: it asserts stereo
keeps its length and six channels still come out three times too long,
and says in its own message that a fix upstream should make it fail.
A TTS prompt stops on its first token, and decoding the nothing that follows
the prompt ended the process with SIGFPE in the detokenizer, for any tokenizer.
Beam search failed as a shape mismatch inside the encoder with one clip, so it
is refused outright. The LFM2-Audio config is now read only once AutoConfig has
refused the checkpoint, and the tutorial's example carries the stop tokens.
Comment thread test/python/models/test_lfm2_audio_models.py Fixed
A fine-tune of these checkpoints keys its adapter "lfm.*", which matches nothing
in the Lfm2ForCausalLM the builder loads. PEFT only warns, so the export carried
untrained LoRA weights and the build still succeeded. The shared load_adapter
step existed only to reach that path, so base.py goes back to what main has.
The audio decoder's resampler is a different filter from the torchaudio one
liquid-audio uses: on the reference's own 24 kHz Japanese clip the log-mel frames
differed by up to 2 and one homophone came out differently. Clips are now decoded
at their own rate and resampled with a port of torchaudio's windowed sinc, which
matches token for token and also takes audio below 16 kHz.
…ence

Step-by-step text logits against liquid-audio: ASR and plain chat agree to 6e-4
and every token matches on the reference's own question.wav and typed questions;
TTS matches at its single text step in all four voices. Interleaved matches for
the six tokens before the reference first feeds audio back, then degenerates into
the non-breaking-space token on three of four questions, so the tutorial now says
to leave the system prompt out for a text answer instead.
The depthformer turns each decoder hidden state into a frame of eight audio codes,
and the frame's embedding is the decoder's next input, with liquid-audio's switching
between text and speech: on <|audio_start|> and the end-of-audio code, and by count
with search.audio_interleaved. The sequence keeps one placeholder per frame and the
codes come from get_output("audio_codes").

With the audio taken greedily on both sides every text token and audio frame matches
liquid-audio, once the published depthformer graph is corrected: it never applied its
rotary embedding and built the tables for the wrong theta. The tutorial has the fix.
@ykhrustalev Yuri Khrustalev (ykhrustalev) changed the title feat: add LFM2-Audio / LFM2.5-Audio speech input support feat: add LFM2-Audio / LFM2.5-Audio speech input and output support Sep 21, 2026
… measured

All four published depthformers have the rotary defect and take the same fix; the
q4 depthformer and the q4 and q8 detokenizers lose enough to quantization that fp32
or fp16 are the ones to use, and the q4 and q8 audio embeddings are not quantized at
all. Greedy-audio parity with liquid-audio holds item for item on Linux x64 and on
macOS arm64.
The recipe for refreshing tokenizer.zip sat in a comment, which CodeQL reads as
commented-out code. It is now a helper, with a test that rebuilding twice gives the
same bytes and the contents that are checked in.
Speech output crashed the moment it sampled its first audio frame on any
execution provider that does not keep the decoder's tensors in device
memory. SampleFrame and WritePendingFrame wrapped hidden_states and
inputs_embeds with the model's device interface, but genai allocates
those from p_device_inputs_, which is plain CPU memory for WebGPU
without graph capture. The EP was then handed a host pointer as if it
were one of its buffers and aborted inside its own copy.

Pick the interface from the tensor's memory info instead, the way
nemotron_speech.cpp already does for encoder outputs, as a shared
DeviceForTensor next to ByteWrapTensor.

Measured on an A10 (Linux) and an M3 Ultra (macOS): interleaved mode and
TTS on WebGPU now match the liquid-audio reference token for token and
frame for frame, and CPU is unchanged.
…d them

CPU was the only provider the tutorial spoke for. CUDA matches it once
every graph runs on the GPU, WebGPU only for the modes that take no
audio, and a half-and-half configuration crashes rather than falling
back, so say all of that where the other limitations are.
Two of the three claims in the providers note were wrong once the
failures were reduced to minimal cases. The WebGPU attention failure is
not about starting from an empty cache but about the present outgrowing
its buffer, on every step. Speech input is not blocked by the encoder
graph, which loads and runs on WebGPU at ORT_ENABLE_BASIC; it is blocked
by this runtime putting the features buffer on the decoder's device.

Point at the two issues so a reader can see where each fix has to land.
@ykhrustalev
Yuri Khrustalev (ykhrustalev) marked this pull request as ready for review September 21, 2026 23:12
Brings in the Moonshine streaming ASR support (microsoft#2302). One conflict, in
test_onnxruntime_genai_api.py, where main renamed the streaming ASR
fixture next to the Images.open/Audios.open regression test this branch
adds; both kept.

@tianleiwu Tianlei Wu (tianleiwu) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The LFM2-Audio feature is comprehensive and generally well tested, and the latest revision addresses the earlier adapter, builder-structure, and documentation concerns. I am requesting changes for two memory-safety areas: submodel outputs must not be bound to buffers from a different execution provider, and configuration-derived indices/span lengths must be validated before memory access. I also left one model-builder convention comment.

Comment thread src/models/multi_modal.cpp
Comment thread src/models/lfm2_audio_output.cpp
Comment thread src/models/lfm2_audio_output.cpp
Comment thread src/python/py/models/builders/lfm2.py Outdated
…uching memory

Review feedback on microsoft#2601:

- audio_token_id must lie inside the vocabulary: Lfm2AudioOutput indexes
  the placeholder logits with it and the processor writes it into
  input_ids. Only its sign was checked.
- codebook_size must be positive, and each depthformer run checks it
  against the logits the graph actually produced, since the width can
  be dynamic. A negative value became a huge span length and an
  oversized one read past the logits. The audio embedding output is
  now also checked to hold one embedding per codebook before
  EmbedFrame sums them.
- A speech or embedding graph that its own session_options leave on
  CPU is refused when a buffer it would be handed is device memory,
  rather than writing host bytes over it (microsoft#2604). The embedding graph
  is checked at load against the decoder's inputs; with audio, both
  graphs are checked against the features buffer. WebGPU without graph
  capture keeps its inputs on the host, so its text-only path with CPU
  sub-models is still accepted. OpenVINO and QNN allocate memory a CPU
  session can use and are not refused.
- The builder sets config.name_or_path instead of the private field.
- Depthformer logits that are not float get their own error instead of
  being reported as too few "float logits".
- New cases for the checks that had none: float16 logits, an audio
  embedding that returns fewer rows than codebooks, and a negative
  codebook count that a symbolic codebook dimension would otherwise
  match.
- The audio-time placement check now runs through the real pipeline
  rather than only the C++ matrix. On CUDA, text-only prompts run with
  the encoder left on CPU and an audio prompt is refused. On WebGPU, a
  CPU export's config keeps running text-only prompts and refuses
  audio. Both run where the EP is installed.

@tianleiwu Tianlei Wu (tianleiwu) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The LFM2-Audio feature is comprehensive and generally well tested. I re-checked the current head and the earlier builder/documentation concerns are no longer present. I still recommend one follow-up on the per-clip speech encoder path: the temporary output tensor is allocated with the decoder allocator while it is rebound as the speech session output, so mixed-provider configurations should be rejected or allocated on the speech session’s device before execution.

Comment thread src/models/multi_modal.cpp
@tianleiwu

Copy link
Copy Markdown
Contributor

/azp run macOS arm64 Build

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@tianleiwu
Tianlei Wu (tianleiwu) merged commit 9e83936 into microsoft:main Sep 24, 2026
16 checks passed
@ykhrustalev

Copy link
Copy Markdown
Contributor Author

Tianlei Wu (@tianleiwu) thank you for collaboration

Yuri Khrustalev (ykhrustalev) added a commit to ykhrustalev/onnxruntime-genai that referenced this pull request Sep 24, 2026
…ment

Conflicts:
	src/models/multi_modal.h

Both sides added members to MultiModalLanguageModel: this branch the per
sub-model device pointers, main (microsoft#2601) the LFM2-Audio output sessions. Kept
both.

Also carried the fix over to code that microsoft#2601 added: Lfm2AudioSpeechState
allocated its per-clip output and wrapped the features buffer with the
decoder's device (model_.p_device_), the pattern this branch replaces. It now
uses p_session_device_, the same change PixtralVisionState received. With the
speech session following the decoder both are the same device, so nothing
changes for the layouts CheckLfm2AudioSessionDevices admits today.
Yuri Khrustalev (ykhrustalev) added a commit to ykhrustalev/onnxruntime-genai that referenced this pull request Sep 25, 2026
microsoft#2601 refused a GPU decoder next to an embedding or encoder that its own
session_options keep on CPU, which is what a CPU export's
genai_config.json describes: those sessions were handed buffers on the
decoder's device and wrote host bytes over them (microsoft#2604). microsoft#2605 allocates
each sub-model's buffers on its own session's device and copies at the
hand-off, per clip for LFM2-Audio's encoder too, so the check now only
refuses configurations that work.

- Remove CheckLfm2AudioSessionDevices and its two call sites.
- Replace its C++ tests and the three Python tests that expected the
  refusal with tests that compare each layout with an all-CPU run on
  CUDA and WebGPU: the encoder, the embedding or both on CPU, with no
  clip, one or two, and interleaved speech output with the depthformer
  and audio embedding on CPU as well.
- Rewrite the tutorial's provider notes: a CPU export's config runs as
  it is once the decoder moves, and WebGPU needs the encoder and the
  audio_output graphs on CPU.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants