[ConfigRefactor] GLM-Image by JaredforReal · Pull Request #2977 · vllm-project/vllm-omni

JaredforReal · 2026-04-21T06:47:07Z

Migrate GLM-Image to the new declarative config system (PipelineConfig + DeployConfig), fixing a broken two-stage pipeline where only the diffusion stage was loaded.

Model type detection failure — GLM-Image is a diffusers-style repo with model_index.json at root but no config.json. _auto_detect_model_type() only checked for config.json, so it returned None and the system fell back to single-stage diffusion.
async_chunk defaulted to True — The legacy deploy YAML didn't set async_chunk, and merge_pipeline_deploy would raise ValueError since no GLM-Image stage declares async-chunk processors.
Legacy YAML format — The deploy YAML used the old nested engine_args:/runtime:/stage_type: format with topology fields that now belong in PipelineConfig.

Changes

PipelineConfig.diffusers_class_name — New field lets pipelines declare their diffusers _class_name. The model type detector now checks model_index.json and matches against registered pipelines, eliminating the need for a separate _DIFFUSERS_CLASS_TO_CONFIG mapping table.
StagePipelineConfig.model_subdir / tokenizer_subdir — Moved from deploy YAML to pipeline topology. These are structural properties (AR config lives in vision_language_encoder/), not deployment knobs. Injected into engine_args by _build_engine_args.
deploy/glm_image.yaml — Rewritten to the new flat format with async_chunk: false, containing only deployment knobs (GPU placement, memory, sampling params). All topology fields removed.
pipeline.py — Added diffusers_class_name, model_subdir, tokenizer_subdir, requires_multimodal_data, and model_arch on the diffusion stage.

Test plan

Online: vllm serve zai-org/GLM-Image --port 8000 --host 0.0.0.0 — verify both AR and diffusion stages initialize (check logs for model loading messages from both stages)
Try override with --stage-overrides '{"0": {"gpu_memory_utilization": 0.65}}'
Offline: python examples/offline_inference/glm_image/end2end.py --model-path <path-to-GLM-Image> --prompt "A cat sitting on the table" --output cat.png --height 1024 --width 1024 --num-inference-steps 50 --enable-diffusion-pipeline-profiler

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
The test results. Please paste the results comparison before and after, or the e2e results.
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
(Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

Signed-off-by: JaredforReal <w13431838023@gmail.com>

chatgpt-codex-connector · 2026-04-21T06:47:14Z

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

lishunyang12 · 2026-04-21T06:51:17Z

Are there any doc that you need you update as well?

Copilot

Pull request overview

Refactors GLM-Image configuration to the new “frozen pipeline topology + deploy YAML” split introduced by the config refactor work, and updates offline example entrypoints to reference the new deploy config location.

Changes:

Removed legacy stage_configs/glm_image*.yaml configs and introduced vllm_omni/deploy/glm_image.yaml.
Added a frozen GLM-Image PipelineConfig (model_executor/models/glm_image/pipeline.py) and registered it in pipeline_registry.py.
Updated offline inference examples to use the new deploy YAML path by default.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
vllm_omni/model_executor/stage_configs/glm_image_muilticonnector.yaml	Removes legacy MultiConnector stage config YAML.
vllm_omni/model_executor/stage_configs/glm_image.yaml	Removes legacy GLM-Image stage config YAML.
vllm_omni/model_executor/models/glm_image/pipeline.py	Adds frozen two-stage GLM-Image pipeline topology.
vllm_omni/deploy/glm_image.yaml	Adds deploy YAML for GLM-Image stages (resources + sampling defaults).
vllm_omni/config/pipeline_registry.py	Registers the new `glm_image` pipeline for lazy loading.
examples/offline_inference/glm_image/run_t2i.sh	Points default config to `vllm_omni/deploy/glm_image.yaml`.
examples/offline_inference/glm_image/run_i2i.sh	Points default config to `vllm_omni/deploy/glm_image.yaml`.
examples/offline_inference/glm_image/end2end.py	Updates default config path fallback to the deploy YAML.
examples/offline_inference/glm_image/README.md	Updates config-path examples (but still has one lingering legacy path).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-21T06:55:18Z

+            final_output_type="image",
+            model_arch="GlmImagePipeline",
+            custom_process_input_func="vllm_omni.model_executor.stage_input_processors.glm_image.ar2diffusion",
+            omni_kv_config={"need_recv_cache": False},


omni_kv_config is set on this StagePipelineConfig, but it is currently never propagated into stage engine_args by merge_pipeline_deploy (and StagePipelineConfig.omni_kv_config is otherwise unused). Either move this into deploy YAML as omni_kv_config (stage engine extra) or update the merge logic to carry it into yaml_engine_args, otherwise this setting has no effect.

Suggested change

omni_kv_config={"need_recv_cache": False},

lishunyang12 · 2026-04-21T06:56:57Z

#2072 Make sure that in 5 level use cases, configs can take affect on this model, especicially stage-config overrides.

Signed-off-by: JaredforReal <w13431838023@gmail.com>

…utils Signed-off-by: JaredforReal <w13431838023@gmail.com>

JaredforReal · 2026-04-21T08:25:32Z

@lishunyang12 @hsliuustc0106 @princepride cc

lishunyang12 · 2026-04-21T09:22:46Z

Test results have been shown offline. Waiting for CI green.

Signed-off-by: JaredforReal <w13431838023@gmail.com>

lishunyang12 · 2026-04-21T10:17:39Z

PTAL @xiaohajiayou Can you help check if this pr has the override precedence issue you mentioned?

hsliuustc0106 · 2026-04-21T13:19:18Z

resolve conflicts please

hsliuustc0106 · 2026-04-21T13:19:42Z

@gcanlin please take care of pipeline yamls for different hardwares

gcanlin · 2026-04-21T13:23:11Z

@gcanlin please take care of pipeline yamls for different hardwares

I'm verifying the models that we have supported. For GLM-Image, @lyj-jjj Could you please take a look?

lyj-jjj · 2026-04-21T15:04:51Z

@gcanlin Okay, tomorrow I will test on glm-image.

Signed-off-by: Jared Wen <w13431838023@gmail.com>

Signed-off-by: JaredforReal <w13431838023@gmail.com>

hsliuustc0106

lgtm

hsliuustc0106 · 2026-04-22T04:23:44Z

one suggestion: please rm the glm-image folder under examples and update the glm-image recipe later

Signed-off-by: JaredforReal <w13431838023@gmail.com>

JaredforReal · 2026-04-22T05:18:25Z

@hsliuustc0106 examples removed

hsliuustc0106 · 2026-04-22T07:09:25Z

please remember to update the recipe

Signed-off-by: JaredforReal <w13431838023@gmail.com> Signed-off-by: Jared Wen <w13431838023@gmail.com> Co-authored-by: SYLAR <125541396+lishunyang12@users.noreply.github.com>

JaredforReal added 2 commits April 21, 2026 13:56

init

69ae9b2

Signed-off-by: JaredforReal <w13431838023@gmail.com>

update config

f711005

Signed-off-by: JaredforReal <w13431838023@gmail.com>

JaredforReal requested a review from hsliuustc0106 as a code owner April 21, 2026 06:47

Copilot AI review requested due to automatic review settings April 21, 2026 06:47

Copilot started reviewing on behalf of JaredforReal April 21, 2026 06:47 View session

JaredforReal changed the title ~~[ConfigRefactor] GLM-Image~~ [WIP][ConfigRefactor] GLM-Image Apr 21, 2026

Copilot AI reviewed Apr 21, 2026

View reviewed changes

JaredforReal added 2 commits April 21, 2026 15:15

resolve diffusers-style models

3d9681e

Signed-off-by: JaredforReal <w13431838023@gmail.com>

move around model/tokenizer subdir, remove dependency on entrypoints.…

8d38ead

…utils Signed-off-by: JaredforReal <w13431838023@gmail.com>

JaredforReal changed the title ~~[WIP][ConfigRefactor] GLM-Image~~ [ConfigRefactor] GLM-Image Apr 21, 2026

lishunyang12 added merge-test label to trigger buildkite merge test CI ready label to trigger buildkite CI labels Apr 21, 2026

Merge branch 'main' into config

d28bb3e

add __init__ to fix CI error

118821d

Signed-off-by: JaredforReal <w13431838023@gmail.com>

This was referenced Apr 21, 2026

[Config Refactor 3a/N] Image diffusion pipeline configs #2987

Closed

[Config Refactor] HunyuanImage3 pipeline configs #2989

Open

hsliuustc0106 mentioned this pull request Apr 21, 2026

[Docs]Add recipe for GLM-Image on 2x A800 GPUs and 1x A800 GPU #2950

Open

JaredforReal added 3 commits April 22, 2026 10:11

Merge branch 'main' into config

124e4b0

Signed-off-by: Jared Wen <w13431838023@gmail.com>

pre-commit

7af5777

Signed-off-by: JaredforReal <w13431838023@gmail.com>

pre-commit

6870c6d

Signed-off-by: JaredforReal <w13431838023@gmail.com>

Merge branch 'main' into config

a12c144

lishunyang12 added this to the v0.20.0 milestone Apr 22, 2026

hsliuustc0106 approved these changes Apr 22, 2026

View reviewed changes

remove examples for glm image

b79c30a

Signed-off-by: JaredforReal <w13431838023@gmail.com>

hsliuustc0106 merged commit 5542332 into vllm-project:main Apr 22, 2026
6 of 8 checks passed

ptarasiewiczNV mentioned this pull request Apr 23, 2026

[Bugfix] GLM-Image: fix noisy / washed-out t2i output (#3034) ptarasiewiczNV/vllm-omni#1

Closed

5 tasks

Conversation

JaredforReal commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Test plan

Uh oh!

chatgpt-codex-connector Bot commented Apr 21, 2026

Uh oh!

lishunyang12 commented Apr 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

lishunyang12 commented Apr 21, 2026

Uh oh!

JaredforReal commented Apr 21, 2026

Uh oh!

lishunyang12 commented Apr 21, 2026

Uh oh!

lishunyang12 commented Apr 21, 2026

Uh oh!

hsliuustc0106 commented Apr 21, 2026

Uh oh!

hsliuustc0106 commented Apr 21, 2026

Uh oh!

gcanlin commented Apr 21, 2026

Uh oh!

lyj-jjj commented Apr 21, 2026

Uh oh!

hsliuustc0106 left a comment

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 commented Apr 22, 2026

Uh oh!

JaredforReal commented Apr 22, 2026

Uh oh!

hsliuustc0106 commented Apr 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

JaredforReal commented Apr 21, 2026 •

edited

Loading