Discuss how to design omni model

[tc-mb]

Hi, I'm a member of the MIniCPM-V & MIniCPM-o teams.
In January of this year, we launched the MIniCPM-o 2.6 model. I'd like to merge the Omni model's capabilities into llama.cpp. I've already implemented a version, and the code used for iPad display is based on this modified CPP code. However, when I tried to merge it, I discovered that llama.cpp was undergoing code refactoring, and the code logic was already significantly different. Therefore, I haven't submitted a complete Omni code yet.

I still hope to bring Omni capabilities to the open source community through llama.cpp. However, the Omni model is quite complex, and the structure diagram is placed at the end, making modifications difficult and prone to accuracy issues. I'd like to ask the above questions in advance to minimize major structural changes during the Omni merge.

I noticed that in llama.cpp, mtmd now includes image capabilities, and whisper has also been added. It contains many of the basic components required by Omni (such as clips). However, Omni implements "live streaming" capabilities, which are significantly different in terms of upper-level scheduling. Especially since MTMD currently uses the input_chunks model, which significantly differs from the streaming model in terms of scheduling. Submitting code to MTMD as usual is extremely difficult. This is the main reason I'm asking this question.

I'd like to ask the following questions:

1. Does the llama.cpp community have any plans for Omni capabilities?
2. Will the Omni capabilities planned for llama.cpp also be included in MTMD?
3. Can I keep the Omni code in a separate folder to keep development simple and contribute it to the community? (I noticed that TTS is also provided in a separate folder.)
4. Should I copy and paste the visual components needed for Omni in MTMD or reference them whenever possible?

I look forward to your answers and appreciate your help. @ggerganov

MiniCPM-o 2.6 live streaming structure

[ggerganov]

Hi @tc-mb, thanks for starting the discussion. It will take me some time to respond in a meaningful way. In the meantime, tagging @ngxson in case he has some thoughts or suggestions how to best support this.

[tc-mb]

Thank you for your reply. I'll patiently await your response.

[tc-mb]

@ngson if you could take the time to review this discussion. I'd appreciate your feedback.

[tc-mb]

Also, I'll share some actual iPad deployment results (using a modified version of llama.cpp for inference).
https://www.youtube.com/watch?v=JFJg9KZ_iZk&t=3s

[ngxson]

Hey @tc-mb , to response to your questions:

• Does the llama.cpp community have any plans for Omni capabilities?

• Will the Omni capabilities planned for llama.cpp also be included in MTMD?

Yes, we already support audio+image input for some Omni models. The video support is a bit more tricky as each model seems to have a different strategy to interleave image and audio tokens, so we still haven't yet support this. Helps on this part would be appreciated.

For the audio output capability, we're still working on this. Models generating audio based on semantic/acoustic tokens (like Mimi decoder) are trivial to add, but diffusion-based models (which generates mel spectrogram) will be much more difficult.

For multi-head output from the backbone, this actually requires a new API from libllama, so probably we should discuss this separately with @ggerganov

• Can I keep the Omni code in a separate folder to keep development simple and contribute it to the community? (I noticed that TTS is also provided in a separate folder.)

• Should I copy and paste the visual components needed for Omni in MTMD or reference them whenever possible?

It can be something under tools/mtmd/omni/...

You can include internal components from clip.h and mtmd.h

[ngxson]

Note that the "Omni-Modality Streaming Backbone" is just a normal LLM, so it can run using libllama, I think it will require a special output head instead of lm_head

IIRC the "Streaming speech decoder" is the diffusion-based model which generates mel spectrogram (in MiniCPM-Omni). It should be implemented inside libmtmd

Please confirm if what I said is correct @tc-mb