[Bugfix] GLM-Image: fix noisy / washed-out t2i output (#3034)

[ptarasiewiczNV]

Purpose

Fix vllm-omni#3034: zai-org/GLM-Image served via vllm serve --omni returns a noisy / near-uniform white image for the minimal curl from the recipe:

curl -sS http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"A beautiful landscape painting"}]}'

Two independent regressions contribute, both fixed here.

1. Route t2i requests through the multimodal processor.
OmniOpenAIServingChat only attached mm_processor_kwargs to the tprompt when the user supplied extra_body.height / extra_body.width. OmniInputPreprocessor._process_text then gated its multimodal branch on elif mm_processor_kwargs: (truthiness), so when the field was omitted the default {} was falsy and routing fell back to plain _tokenize_prompt. That bypassed GLM-Image's HF processor and the image-generation scaffold <|image|>PROMPT<sop>H W<eop><sop>h w<eop><|dit_token_N|> it emits — the AR never entered image-gen mode and collapsed to a handful of repeated VQ codes (unique=15/1281, no terminal EOS), which the DiT denoised into uniform white. Now serving_chat always attaches mm_processor_kwargs (possibly empty) for image-modality requests, and _process_text switches from truthiness to presence ("mm_processor_kwargs" in parsed_content) so an explicitly-empty dict correctly routes through the multimodal processor.

2. Make the AR max_tokens compute cover the default target size.
PR #2320 dropped max_tokens: 1281 from the GLM-Image stage config and moved the compute into _apply_request_overrides, but gated it on height is not None and width is not None. For the bare-curl request (no extra_body) the gate skipped the compute and max_tokens fell through to max_model_len - seq_len (~131k), producing an upstream IndexError. Now when the user didn't pass h/w we fall back to any stage's default h/w (GLM-Image stage-1 yaml declares height: 1024, width: 1024) so the compute fires for the bare-curl too. The implicit gate becomes "a stage declares h/w in its sampling params" — LLM-only / audio pipelines skip, no architecture check needed. Also fixes a latent getattr(explicit_fields, "max_tokens", None) bug — explicit_fields is a set, so the getattr always returned None and silently overwrote user-provided max_tokens.

Test Plan

Reproduce the bug's minimal curl and compare the returned PNG against the same run on main:

# Launch (2× 48 GB GPUs; A6000 tested)
vllm serve zai-org/GLM-Image --omni --port 8091

# In another shell
curl -sS -X POST http://localhost:8091/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"A beautiful landscape painting"}]}' \
  -o /tmp/resp.json

python - <<'PY'
import base64, io, json, numpy as np
from PIL import Image
j = json.load(open("/tmp/resp.json"))
url = j["choices"][0]["message"]["content"][0]["image_url"]["url"]
im = Image.open(io.BytesIO(base64.b64decode(url.split(",", 1)[1]))).convert("RGB")
a = np.asarray(im)
print(f"{im.size} mean={a.mean():.2f} std={a.std():.2f} min={a.min()} max={a.max()}")
im.save("/tmp/out.png")
PY

Also tested with an extra_body-specified override and with a second prompt ("A red apple on a wooden table").

Test Result

Same prompt and seed=42 before/after:

	mean	std	min	max	unique AR codes	EOS emitted
Before	249	15	135	255	15 / 1281	❌
After	117	71	0	255	139 / 1281	✅

Before: 1024×1024 PNG that is uniform near-white, no scene content.
After: coherent landscape (mountain + lake + wildflowers).

Second prompt sanity-check ("A red apple on a wooden table") also renders a coherent image.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
• The test results. Please paste the results comparison before and after, or the e2e results.
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
• (Optional) Release notes update. If your change is user-facing, please update the release notes draft.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 25c4e15e19

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[hsliuustc0106]

BLOCKING:

• Test Coverage — This is a bugfix PR but lacks automated regression tests. The manual curl test evidence is thorough, but we need at least unit tests to prevent future regressions:

  1. Test for with explicitly-empty to verify presence-based routing (not truthiness)
  2. Test for to verify the height/width fallback logic when user doesn't specify them
  3. Test for to verify max_tokens is always computed for AR + image-diffusion pipelines

The changes are well-documented and the manual testing evidence is convincing, but automated tests are required for maintainability.

[ptarasiewiczNV]

BLOCKING:

• Test Coverage — This is a bugfix PR but lacks automated regression tests. The manual curl test evidence is thorough, but we need at least unit tests to prevent future regressions:

  1. Test for with explicitly-empty to verify presence-based routing (not truthiness)
  2. Test for to verify the height/width fallback logic when user doesn't specify them
  3. Test for to verify max_tokens is always computed for AR + image-diffusion pipelines

The changes are well-documented and the manual testing evidence is convincing, but automated tests are required for maintainability.

Thanks, @hsliuustc0106, for the review. I have added four regression tests covering each of the three points:

• Presence-based routing (empty mm_processor_kwargs) — tests/inputs/test_preprocess.py::TestProcessTextMmProcessorKwargsRouting:

  • test_empty_mm_processor_kwargs_routes_to_multimodal — explicit {"mm_processor_kwargs": {}} routes through _process_multimodal, not _tokenize_prompt.
  • test_missing_mm_processor_kwargs_routes_to_tokenize — control: missing key still falls through to plain tokenize.

• h/w fallback when user doesn't specify them — tests/entrypoints/openai_api/test_serving_chat_sampling_params.py::TestApplyRequestOverridesGLMImage::test_falls_back_to_diffusion_stage_defaults_when_no_extra_body — with no extra_body and stage-1 params carrying height=1024, width=1024,
  _apply_request_overrides pulls h/w from the diffusion stage defaults; max_tokens==1281 and extra_args["target_h/w"]==1024.

• max_tokens always computed for AR + image-diffusion — covered by the existing test_dynamic_max_tokens_overrides_user_value (user-supplied value is overridden by compute) plus a new test_explicit_null_max_tokens_still_computes which guards the case where max_tokens is set to null.

[ptarasiewiczNV]

Closing for rebased: #3189 after #3084 fixed one of the two addressed issues.