[Jungle430]

Purpose

Related to #7841.

GLM-Image’s AR stage produces discrete image token IDs, but the current alias mapping parses token_ids as TEXT. This PR introduces an independent TOKEN_IDS type and enforces canonical output-type names.

• Remove _MODALITY_ALIASES; accept only text, image, audio, latent, and token_ids. Preserve None defaults and + / , combinations. Reject legacy aliases, unknown names, case variants, whitespace, and empty components.
• Preserve explicit empty config values so invalid names fail during LLM output-processor initialization, before requests are accepted. Engine replicas may already have started at this point.
• Wire the new type through default output naming and DELTA retention. Generic model_outputs and hidden follow the shared default-key naming rule, without a token_ids-specific exception. Keep dim-0 accumulation and the precedence of existing per-key strategies.
• Retain GLM-Image’s original configuration. Generated IDs still travel through cumulative_token_ids, and source-image IDs remain in ids.prior_image; no ar2diffusion() changes are needed.

Existing latent models are unchanged. Normalization only renames fields; it does not convert or validate tensor values. Independent auxiliary outputs need distinct, explicit keys. A richer per-field contract for modality, representation, and update rules is follow-up work.

Test Plan

• Cover parsing, config propagation, startup failure/cleanup, shared field normalization, preservation of explicitly named auxiliary fields, accumulation, DELTA retention, and GLM-Image T2I/I2I stage handoff.
• Compare NVIDIA T2I/I2I results against the base using the same model revision, inputs, and seed.
• When present, auxiliary hidden is normalized to token_ids without changing its tensor values. GLM-Image’s bridge ignores this auxiliary field and reads generated IDs from cumulative_token_ids. The default GPU path omits auxiliary hidden output; NPU and some optional runner/cache paths may still emit it and require separate validation.

Pytest Validation Report

Tested commit: bbfd7fc54. Test date: 2026-09-23.

Environment

Item	Value
Hardware	1 × NVIDIA RTX 2000 Ada, 16 GB
NVIDIA driver	580.159.04
Python	3.12.3
vLLM	0.30.0
PyTorch	2.13.0+cu130
Transformers	5.14.1
pytest	9.1.1
Selection	Six related test files; core_model and cpu markers

Test Results

Check	Scope / coverage	Result
Related unit suite	Six files selected with core_model and cpu; canonical parsing, enum/name routing, config propagation, startup validation/cleanup, payload normalization, accumulation, DELTA retention, and GLM-Image T2I/I2I handoff.	368 passed, 1 skipped, 19 warnings; exit code 0; 9.07 s.
Skipped case	qwen3_omni_moe pipeline registry case requires an HF config to resolve.	1 skipped; no test failures.

This is the focused suite for this change. The 19 warnings comprise 14 PyTorch deprecations, one vLLM/vLLM-Omni development-version warning, three deprecated diffusion-config warnings, and one FlashInfer/Cutlass deprecation.

Pytest command

python -m pytest -vv -ra -m "core_model and cpu" --run-level=core_model \
  tests/engine/test_output_modality.py \
  tests/engine/test_output_processor.py \
  tests/engine/test_stage_metadata.py \
  tests/engine/test_stage_engine_args.py \
  tests/engine/test_async_omni_engine_stage_init.py \
  tests/model_executor/stage_input_processors/test_glm_image.py

GLM-Image Regression Report

PR commit: bbfd7fc54. Baseline: 59db6a428. Test date: 2026-09-23.

Environment

Item	Value
Hardware	1 × NVIDIA H100, 80 GB
NVIDIA driver	580.126.09
Python	3.12.3
vLLM	0.30.0
PyTorch	2.13.0+cu130
Transformers	5.14.1
Validation	Online HTTP inference; archived request, response, PNG and RGB-pixel comparison

GLM-Image regression setup

Setting	Value
Model	zai-org/GLM-Image
Pinned model revision	2c433cc0cbc293bde2ac8ca9624f279b5d23fcf4
Execution	Both stages on GPU 0; AR and Diffusion in eager mode; async_chunk=false.
AR memory / context	gpu_memory_utilization=0.5, max_model_len=32768, max_num_seqs=1.
Generation	1024 × 1024; 50 diffusion steps; guidance_scale=1.5; seed=42.
Comparison controls	Same environment, model files, deployment YAML, client scripts and per-case request JSON; only the source commit changed.
Endpoint	POST /v1/chat/completions

End-to-end results

Case	Base	PR	PNG bytes / RGB pixels	Maximum pixel difference
T2I	HTTP 200; 50.82 s	HTTP 200; 52.49 s	Identical / identical	0
I2I	HTTP 200; 46.55 s	HTTP 200; 47.77 s	Identical / identical	0

All four runs exited with code 0 and returned a decodable 1024 × 1024 PNG. Returned image payloads match the saved PNG files. File bytes, decoded RGB pixels and SHA-256 hashes were independently checked from the archived artifacts. The timings are individual request measurements, not a performance benchmark.

Image comparison

T2I — text-to-image:

Base (59db6a428)	PR (bbfd7fc54)

I2I — the same source image was used for both runs:

Source image	Base (59db6a428)	PR (bbfd7fc54)

The comparisons above use the original archived PNGs. Each PR/base pair is byte-for-byte identical; the maximum decoded RGB pixel difference is 0.

Reproduce the fixed-input NVIDIA comparison (inline code)

Use a clean, dedicated checkout containing both tested commits and an editable vLLM-Omni installation. Activate the environment with the versions listed above; the client also uses requests and Pillow. Run both commits in the same environment on the same GPU, restarting the service between them.

This condensed client sends the same request JSON as the archived test scripts. The reported timings come from the original runs. No evidence archive is required to use the code below.

1. Prepare the pinned model, fixed input and deployment config. Run from the repository root; adjust the model directory as needed. This YAML retains the tested NVIDIA settings and omits unused NPU overrides.

export GLM_REG_DIR="$HOME/glm-image-regression"
export GLM_MODEL_DIR="$HOME/models/GLM-Image"
mkdir -p "$GLM_REG_DIR"

hf download zai-org/GLM-Image \
  --revision 2c433cc0cbc293bde2ac8ca9624f279b5d23fcf4 \
  --local-dir "$GLM_MODEL_DIR"

curl --fail --location \
  https://github.com/user-attachments/assets/7be7ca5b-6920-4bf7-bfc2-52eb33fbefbb \
  --output "$GLM_REG_DIR/i2i-source.png"

cat > "$GLM_REG_DIR/deploy.yaml" <<'YAML'
async_chunk: false
stages:
- stage_id: 0
  max_num_seqs: 1
  gpu_memory_utilization: 0.5
  enforce_eager: true
  trust_remote_code: true
  enable_prefix_caching: false
  max_num_batched_tokens: 32768
  devices: '0'
  default_sampling_params:
    temperature: 0.9
    top_p: 0.75
    top_k: 16512
    stop_token_ids:
    - 16385
    max_tokens: 4353
    seed: 42
    detokenize: false
  max_model_len: 32768
- stage_id: 1
  max_num_batched_tokens: 32768
  max_num_seqs: 1
  enforce_eager: true
  trust_remote_code: true
  enable_prefix_caching: false
  devices: '0'
  default_sampling_params:
    seed: 42
    num_inference_steps: 50
    guidance_scale: 1.5
    height: 1024
    width: 1024
YAML

2. Save the client below. It checks the input hash, sends both requests, and saves requests, responses and PNGs.

cat > "$GLM_REG_DIR/client.py" <<'PY'
import base64
import hashlib
import io
import json
import os
import sys
from pathlib import Path

import requests
from PIL import Image

label = sys.argv[1]
assert label in {"pr", "base"}
root = Path(os.environ["GLM_REG_DIR"])
source = (root / "i2i-source.png").read_bytes()
assert hashlib.sha256(source).hexdigest() == "4b48207e037f09758ae36bf780bd171e61549a79cfcde40a8c05d94c2828dc71"
image_url = "data:image/png;base64," + base64.b64encode(source).decode()
cases = {
    "t2i": "A Vincent van Gogh style impressionist painting.",
    "i2i": [
        {"type": "text", "text": "Transform this modern, geometric image into a Vincent van Gogh style impressionist painting."},
        {"type": "image_url", "image_url": {"url": image_url}},
    ],
}
session = requests.Session()
session.trust_env = False
session.get("http://127.0.0.1:8091/health", timeout=10).raise_for_status()
for case, content in cases.items():
    out = root / label / case
    out.mkdir(parents=True, exist_ok=True)
    payload = {
        "model": "glm-image", "stream": False, "seed": 42,
        "messages": [{"role": "user", "content": content}],
        "extra_body": {"height": 1024, "width": 1024, "num_inference_steps": 50, "guidance_scale": 1.5},
    }
    (out / "request.json").write_text(json.dumps(payload, indent=2))
    response = session.post("http://127.0.0.1:8091/v1/chat/completions", json=payload, timeout=(10, 900))
    (out / "response.json").write_bytes(response.content)
    assert response.status_code == 200, response.text[:3000]
    content = response.json()["choices"][0]["message"]["content"]
    assert isinstance(content, list)
    images = [item for item in content if item.get("type") == "image_url"]
    assert len(images) == 1
    url = images[0]["image_url"]["url"]
    assert url.startswith("data:image/png;base64,")
    png = base64.b64decode(url.split(",", 1)[1], validate=True)
    image = Image.open(io.BytesIO(png))
    image.load()
    assert image.size == (1024, 1024)
    (out / "output.png").write_bytes(png)
    print(label, case, "PASS", "PNG SHA-256:", hashlib.sha256(png).hexdigest(), flush=True)
PY

3. Start the PR server in the first terminal.

set -o pipefail
serve_glm() {
  git rev-parse HEAD > "$GLM_REG_DIR/$1-commit.txt"
  CUDA_VISIBLE_DEVICES=0 HF_HOME="$HOME/.cache/huggingface" PYTHONUNBUFFERED=1 \
    vllm serve "$GLM_MODEL_DIR" --omni \
    --host 127.0.0.1 --port 8091 --served-model-name glm-image \
    --deploy-config "$GLM_REG_DIR/deploy.yaml" \
    --stage-init-timeout 1200 --init-timeout 1800 \
    2>&1 | tee "$GLM_REG_DIR/$1-server.log"
}

git checkout --detach bbfd7fc54b637bc58bc9e84179d1b73262235e17
serve_glm pr

After Application startup complete, activate the same environment in a second terminal and run:

export GLM_REG_DIR="$HOME/glm-image-regression"
python "$GLM_REG_DIR/client.py" pr

4. Repeat on the baseline. Stop the PR server with Ctrl+C in the first terminal and wait for its GPU workers to exit. Keep the environment and configuration unchanged, then run there:

git checkout --detach 59db6a4285e9086391efc85a6eff6030a6294878
serve_glm base

After startup completes, run in the second terminal:

python "$GLM_REG_DIR/client.py" base

5. Compare the requests and outputs. Run in the second terminal:

python - <<'PY'
import json
import os
from pathlib import Path

from PIL import Image, ImageChops

root = Path(os.environ["GLM_REG_DIR"])
for case in ("t2i", "i2i"):
    base, pr = root / "base" / case, root / "pr" / case
    assert json.loads((base / "request.json").read_text()) == json.loads((pr / "request.json").read_text())
    assert (base / "output.png").read_bytes() == (pr / "output.png").read_bytes()
    with Image.open(base / "output.png") as a, Image.open(pr / "output.png") as b:
        assert a.size == b.size == (1024, 1024)
        difference = ImageChops.difference(a.convert("RGB"), b.convert("RGB"))
        maximum = max(high for low, high in difference.getextrema())
        assert maximum == 0
    print(case, "PASS: identical requests, PNG bytes and RGB pixels; max difference =", maximum)
PY

The recorded runs passed all comparisons for both cases. These checks establish PR/base parity for the fixed inputs; they do not measure general image quality or performance.

AR → Diffusion handoff

Both commits recorded the same handoff values:

Case	AR output tokens	Target prior tokens passed to DiT	Source-image IDs
T2I	1281	4096	Absent, as expected
I2I	1025	4096	Present (prior_image_ids=yes)

Scope and log notes

Path / mode	Coverage in this report
NVIDIA online T2I / I2I	End-to-end PR/base comparison on one H100; both stages on GPU 0, eager mode, async_chunk=false, stream=false. One fixed input and seed per case.
Optional auxiliary hidden payload	CPU unit tests cover normalization and GLM-Image handoff with synthetic payloads. Runner/cache paths that emit this field were not exercised end to end.
NPU	Not tested on NPU hardware. The synthetic payload tests do not validate the NPU runner.
CUDA Graph	Not covered; both tested stages used eager mode.
Multi-GPU / distributed placement	Not covered; both tested stages shared one GPU.
Streaming / async-chunk execution	No end-to-end coverage claimed; the regression used full-response HTTP requests and disabled async chunks.

Both versions logged the same processor / model_type configuration-probing errors during startup, then completed startup, inference and shutdown successfully. Neither server log contains a traceback or OOM.

Other Validation

• Applicable non-mypy checks and test-file mypy pass. The five production mypy errors are reproducible on the upstream base.
• Hardware and execution-mode coverage limits are listed above.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/entrypoints.md, docs/design/module/input_output_modality_contracts.md.

Module owners: @alex-jw-brooks @linyueqian @NickCao

Routing: @alex-jw-brooks via module of the changed files, semantic router; @linyueqian via module of the changed files; @NickCao via module of the changed files

@Jungle430, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit 1680ae2c747f produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[Jungle430]

I reviewed the diff, focusing on:

• Canonical-name parsing and startup validation, including None defaults, compound types, and propagation of explicitly empty configuration values.
• TOKEN_IDS field normalization, accumulation, DELTA retention, and the precedence of existing per-key strategies.
• GLM-Image stage handoff: generated IDs still travel through cumulative_token_ids, and source-image IDs remain in ids.prior_image, without changes to the bridge.

Validation on the current commit bbfd7fc54: the related pytest suite reports 368 passed, 1 skipped. GLM-Image T2I and I2I both passed PR/base comparisons, with identical PNG bytes and RGB pixels for both cases.

End-to-end validation covers eager execution on a single NVIDIA H100. Coverage limits for other paths are documented in the PR, alongside result images and reproduction code.

cc @tzhouam

[tzhouam]

@vllm-omni-review-bot

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Direct under experiment vllm-omni-strict-5050-20260829.

[vllm-omni-review-bot]

Omni ReviewBot review

PR description

This change makes engine_output_type accept only the lowercase names text, image, audio, latent, and token_ids, and it treats token_ids as its own modality instead of a text alias. Empty or omitted values stay distinct: None still defaults to text, while an explicit empty string is kept in stage engine args and fails when the LLM output processor is built. Generic producer keys such as model_outputs and hidden are renamed to the declared type, including token_ids, and those tensors stay accumulated across DELTA steps; GLM-Image’s AR-to-diffusion bridge still reads generated IDs from cumulative_token_ids and source-image IDs from ids.prior_image.

Change flow

flowchart TD
    A["[EXISTING] StagePipelineConfig.engine_output_type"]:::existing
    B["[CHANGED] _build_engine_args keeps empty strings"]:::changed
    C["[REMOVED] _MODALITY_ALIASES"]:::removed
    D["[CHANGED] OutputModality.from_string canonical parse"]:::changed
    E["[NEW] TOKEN_IDS type and DELTA retention"]:::new
    F["[CHANGED] MultimodalOutputProcessor default keys"]:::changed
    G["[EXISTING] GLM-Image ar2diffusion reads cumulative_token_ids"]:::existing
    A --> B --> D
    C --> D
    D --> E --> F --> G
    classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
    classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
    classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
    classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

Loading

CI at bbfd7fc54b63 (2026-09-28T02:33:59.379129+00:00): required check(s) blocking: buildkite/vllm-omni (missing), and -5 more.

No actionable findings.

[tzhouam]

Reviewed at bbfd7fc54. I found no correctness issues, only two stale docstrings outside the diff that still show a spelling this PR now rejects:

• vllm_omni/config/model.py:105: the engine_output_type docstring gives "latents" as an example value.
• vllm_omni/engine/arg_utils.py:159: the same "latents" example on OmniEngineArgs.engine_output_type.

With the aliases removed, OutputModality.from_string("latents") raises ValueError: Unknown modality. Anyone who copies the documented example into a custom pipeline config will fail at startup. Suggest changing both to "latent", and possibly listing "token_ids" alongside it.

What I checked and found fine: no engine_output_type value in the repo (code, examples, benchmarks, tests, docs, YAML/JSON configs) uses a removed alias. The other from_string callers are unaffected. The GLM-Image token_ids payload-key change doesn't affect ar2diffusion. The is not None change in _build_engine_args correctly reaches startup validation, and the replica cleanup is covered by the new test.

[Jungle430]

@tzhouam Fixed both docstrings in ad7a9408: replaced latents with latent and added token_ids to the examples.

[Jungle430]

@tzhouam The GitHub Actions checks have passed after merging the latest main. Need to trigger the Buildkite pipelines.

[Jungle430]

@tzhouam The main Buildkite pipeline (buildkite/vllm-omni) has passed. I suspect the AMD and NPU failures shown below are shared CI issues unrelated to this PR, as the same errors also occur in #8064.

• AMD

#7843

#8064

• NPU

#7843

#8064

[vllm-omni-review-bot]

Omni ReviewBot review

PR description

This PR adds a first-class TOKEN_IDS output modality and stops treating token_ids as a TEXT alias, so discrete image/codebook IDs get their own payload key and retention rules. Parsing now accepts only exact lowercase canonical names (text, image, audio, latent, token_ids), including +/, compounds, while rejecting legacy aliases, case variants, whitespace, and empty components. Explicit empty engine_output_type values are preserved through stage engine-args so invalid names fail when the LLM output processor is built, with replica cleanup covered by tests. GLM-Image keeps reading generated IDs from cumulative_token_ids and source-image IDs from ids.prior_image; the AR→Diffusion bridge is unchanged.

Change flow

flowchart TD
  A["[EXISTING] Stage engine_output_type<br/>pipeline / deploy config"]:::existing --> B["[CHANGED] _build_engine_args<br/>preserve explicit empty values"]:::changed
  B --> C["[CHANGED] OutputModality.from_string<br/>canonical names only; aliases removed"]:::changed
  C --> D["[NEW] OutputModality.TOKEN_IDS<br/>default key + CONCAT_DIM0 + non-drainable"]:::new
  D --> E["[CHANGED] MultimodalOutputProcessor<br/>naming / DELTA retention"]:::changed
  E --> F["[EXISTING] GLM-Image ar2diffusion<br/>cumulative_token_ids + ids.prior_image"]:::existing
  classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
  classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
  classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

Loading

CI at 536de30c1679 (2026-09-30T22:49:52.052671+00:00): all required and operator-watched checks reported green. Observed Buildkite: buildkite/vllm-omni (passed), buildkite/vllm-omni-amd-ci (failed), buildkite/vllm-omni-npu-ci (failed), and 1 more.

No actionable findings.

---

_{🤖 This review was generated by InferMatrix Copilot, an open-source repo-maintenance agent for PR review, CI debugging and issue triage. Try it on your own repo, and ⭐ star it if it helped!}

[hsliuustc0106]

@Jungle430 this PR is labeled ready + high priority, but CI is failing on the latest commit:

• buildkite/vllm-omni-amd-ci
• buildkite/vllm-omni-npu-ci

Could you please take a look and push an update to get CI green? Once the checks pass we can proceed with review/merge. Thanks!

[vllm-omni-review-bot]

Omni ReviewBot review

PR description

This PR makes engine_output_type accept only canonical lowercase names and adds a real TOKEN_IDS modality instead of treating token_ids as a text alias. Invalid or legacy names now fail when the LLM output processor is built at stage init, while explicit empty strings are preserved so that failure still happens there and replicas are shut down. GLM-Image keeps emitting generated IDs through cumulative_token_ids / ids.prior_image; generic model_outputs/hidden fields simply rename to the declared default key, including token_ids.

Change flow

flowchart TD
  A["[EXISTING] Stage engine_output_type<br/>pipeline / deploy / engine args"]:::existing --> B["[CHANGED] _build_engine_args<br/>preserve explicit empty values"]:::changed
  B --> C["[CHANGED] OutputModality.from_string<br/>canonical names only"]:::changed
  C --> D["[NEW] TOKEN_IDS modality<br/>naming + DELTA retention"]:::new
  D --> E["[CHANGED] MultimodalOutputProcessor<br/>default key + accumulate"]:::changed
  E --> F["[EXISTING] GLM-Image ar2diffusion<br/>cumulative_token_ids / prior_image"]:::existing
  classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
  classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
  classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

Loading

CI at 1680ae2c747f (2026-10-04T03:36:20.495849+00:00): required check(s) blocking: pre-commit (pending), buildkite/vllm-omni (missing), docs/readthedocs.org:vllm-omni (pending).

No actionable findings.

---

_{🤖 This review was generated by InferMatrix Copilot, an open-source repo-maintenance agent for PR review, CI debugging and issue triage. Try it on your own repo, and ⭐ star it if it helped!}

[Jungle430]

@hsliuustc0106 Thanks for checking. I’ve merged the latest main and resolved the conflicts in 1680ae2c7. This includes #8330 for the NPU circular-import failure and #8342 for the AMD test-routing, AuK numerical-comparison, and CPU timeout issues.

The linked failed builds ran on the previous commit. Could you please trigger the Buildkite pipelines on the updated commit so we can verify whether any failures remain? The GitHub Actions checks have passed.

cc @tzhouam

[tzhouam]

LGTM

[Jungle430]

@tzhouam need to trigger buildkite NPU ci again, pass it then merge.