add support for MammothModa2 model

[HonestDeng]

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

Resolve #314 , add support for MammothModa2 model https://github.com/bytedance/mammothmoda

Test Plan

Machine:

• H200(140GB) x 1

Parallel:

• TP: None

Image:

• Size: 1024 x 1024
• DiT Step: 50

2. Image Summery

Machine:

• H200(140GB) x 1

Parallel:

• TP: None

Image:

• Size: 1024 x 1024

Test Result

Image in the left side is generated by MammothModa2 official implementation while the right side from vllm-omni:

This table shows performance in two implementations:

Stages	official-impl	vllm-omni
AR stage	83.529s	74.06s
DiT stage	10.320s	9.65s

Transfer time: 4.012ms

We get better performance.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

[hsliuustc0106]

Hi, will the model be ready before 1230 release?

[HonestDeng]

Yes.

The MammothModa2-Preview is combined Qwen25-VL(with extra gen-experts in MLP layers) with an DiT module for image generation. Now I have already implemented the Qwen25-VL part of MammothModa2-Preview by reusing vllm code, such as Qwen2Attention, Qwen2MLP, and we can takes text and image as input to generate text token.

Now I'm currently working on DiT parts. Hopefully I will finish DiT parts in this weekend and review my code before 1230.

I'm not quite familiar in supporting new models. If there is any problem in my code, please correct me. Thanks!

[hsliuustc0106]

Yes.

The MammothModa2-Preview is combined Qwen25-VL(with extra gen-experts in MLP layers) with an DiT module for image generation. Now I have already implemented the Qwen25-VL part of MammothModa2-Preview by reusing vllm code, such as Qwen2Attention, Qwen2MLP, and we can takes text and image as input to generate text token.

Now I'm currently working on DiT parts. Hopefully I will finish DiT parts in this weekend and review my code before 1230.

I'm not quite familiar in supporting new models. If there is any problem in my code, please correct me. Thanks!

the model seems quite similar to Qwen-Image strcuture with a qwen-vl for encoding and a DiT module for image generation.

[princepride]

python3 examples/offline_inference/mammothmodal2_preview/run_mammothmoda2_t2i.py   --model bytedance-research/MammothModa2-Preview --stage-c
onfig ./vllm_omni/model_executor/stage_configs/mammoth_moda2.yaml   --prompt "A stylish woman riding a motorcycle 
in NYC, movie poster style"   --height 1024   --width 1024   --num-inference-steps 50   --text-guidance-scale 4.0 
  --out output.png
/proj-tango-pvc/users/zhipeng.wang/workspace/vllm-omni/vllm_omni/__init__.py:32: RuntimeWarning: Failed to import version from _version.py: No module named 'vllm_omni._version'
This typically happens in development mode before building.
Using fallback version 'dev'.
  from .version import __version__, __version_tuple__  # isort:skip
Traceback (most recent call last):
  File "/proj-tango-pvc/users/zhipeng.wang/workspace/vllm-omni/examples/offline_inference/mammothmodal2_preview/run_mammothmoda2_t2i.py", line 241, in <module>
    main()
  File "/proj-tango-pvc/users/zhipeng.wang/workspace/vllm-omni/examples/offline_inference/mammothmodal2_preview/run_mammothmoda2_t2i.py", line 186, in main
    gen_cfg = load_t2i_generation_config(args.model)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/proj-tango-pvc/users/zhipeng.wang/workspace/vllm-omni/examples/offline_inference/mammothmodal2_preview/run_mammothmoda2_t2i.py", line 57, in load_t2i_generation_config
    raise FileNotFoundError(f"Config not found: {gen_cfg_path}")
FileNotFoundError: Config not found: bytedance-research/MammothModa2-Preview/t2i_generation_config.json

I got an error when using huggingface id, PTAL.

[princepride]

PTAL

[princepride]

@HonestDeng pre-commit failed, PTAL

[HonestDeng]

Now huggingface id has been supported and pre-commit run successfully.

[princepride]

Roughly LGTM, after testing, I will approve it.

[HonestDeng]

Roughly LGTM, after testing, I will approve it.

Thanks

[lishunyang12]

Looks like good progress — active iteration with @hsliuustc0106 and @princepride. I'll defer to them on the remaining items.

[hsliuustc0106]

Looks like good progress — active iteration with @hsliuustc0106 and @princepride. I'll defer to them on the remaining items.

Later, we can move to v2.5 and fix the remaining issues

[hsliuustc0106]

lgtm