[EPD] Metadata-only image and video inputs for GLM-5.3-Flash by grYe99 · Pull Request #64071 · vllm-project/vllm

grYe99 · 2026-10-27T15:17:15Z

Overview

GLM-5.3-Flash can now take metadata-only image and video inputs in Python EPD, the same out-of-band placeholder path as Qwen-VL (#50390) and audio models (#57887). The encoder publishes image_grid_thw, and for video video_grid_thw plus one timestamp per temporal patch. The consumer rebuilds placeholders from those fields and receives embeddings through the EC connector.

No open PR covers this. #61418 is routing-metadata adapters, not EPD media metadata.

Claims

An EC consumer no longer reruns GLM-5.3-Flash vision preprocessing when the encoder published the placeholder metadata.
On a cold 1E+1PD pair this is about 1.14× request throughput and 0.86× TTFT versus forwarding the raw image. The same ratio holds after each arm has warmed its own CUDA graphs.
On 32 greedy image requests and 16 greedy video requests, prompt token counts and answer text matched the raw-media path on every request. Image and video throughput are separate tables. The image ratio above is not a video result.

Validation

Not a duplicate of an open PR (search above).

AI assistance was used. The submitting human should review the diff.

Parser contract on the same H200 pod, against the installed tree with this patch (pytest 9.1.1, torchvision 0.28.0):

python3 -m pytest -q tests/models/glm5next/test_epd_metadata.py

4 passed in 15.54s.

End-to-end on 8× H200, zai-org checkpoint /models/GLM-5.3-Flash (bf16/fp8), vLLM 0.32.0 plus this patch. The PD worker loads the full checkpoint, 76.5 GiB per TP=4 rank, so the PD side is TP=4 on GPU 4–7. The encoder is a dedicated ec_producer and runs encoder-only. In the throughput runs it was also started at TP=4 on GPU 0–3, and each encoder worker loaded 1.17 GiB; a later startup of this patch with --tensor-parallel-size 1 loaded 1.93 GiB and used 4579 MiB on one GPU. Encoder TP=4 was not required for the vision tower. ECExampleConnector over local disk was used because ECMooncakeConnector producers require tensor_parallel_size=1, and the producer was originally launched with the full-model command. That check does not rule out Mooncake for an encoder-only producer. The tables below were measured with encoder TP=4 and ECExampleConnector. Max model len 8192, prefix caching left on for the PD instance, CUDA graphs on, concurrency 64, 64 distinct 768×768 images, 64 output tokens, greedy. Each arm was a fresh process start plus one warmup request, then two measured passes. Zero failed requests. The metadata proxy logged 129 rewrites; the raw proxy (--no-rewrite) logged 0.

arm	pass	req/s	output tok/s	mean TTFT	mean TPOT
raw	cold	5.46	349	7575 ms	59.0 ms
raw	warm	10.91	698	3573 ms	31.0 ms
metadata-only	cold	6.22	398	6530 ms	50.1 ms
metadata-only	warm	12.51	800	3076 ms	26.9 ms

Logs: the run was on a private pod; the table is the Serving Benchmark Result block from vllm bench serve.

Video throughput was rerun with vllm bench serve after the first 16-request burst proved to be one unschedulable wave (raw warm slower than raw cold, metadata cold dominated by a few multi-second prefills). This run is a different pod, same image and patch. Each arm is a fresh 1E+1PD process and a fresh EC cache. 64 distinct 2-second 320×240 clips at 8 fps, ShareGPT dataset, 64 output tokens, greedy, concurrency 8. One probe measured 248 prompt tokens, and 8192 / 248 caps concurrency at 8. Pass 1 is preceded by 8 warmups of the first clip so Triton JIT is outside the measurement. Pass 2 and pass 3 are repeats with no extra warmup. Zero failed requests. Mean TTFT stays next to the median, and P99 TTFT stays within about 1.5× the mean. The metadata proxy logged 201 rewrites; the raw proxy logged 0. On each paired pass, metadata-only is slightly ahead (about 1.07–1.11× output tok/s and 0.86–0.92× mean TTFT). Repeating a warm pass on the same arm moves output tok/s by 9–12%, which is the same size as that gap, so this table is not a video speedup claim.

arm	pass	req/s	output tok/s	mean TTFT	mean TPOT
raw	1	4.82	308	849 ms	12.8 ms
raw	2	6.81	436	562 ms	9.7 ms
raw	3	6.01	385	673 ms	10.4 ms
metadata-only	1	5.18	331	780 ms	12.1 ms
metadata-only	2	7.35	471	481 ms	9.6 ms
metadata-only	3	6.69	428	581 ms	9.7 ms

Correctness used the same 1E+1PD stack and checkpoint, greedy, max_tokens=32, one request at a time. Raw (--no-rewrite) first, then metadata-only.

Image: 32 distinct 768×768 images. The metadata proxy logged 32 rewrites.

image	prompt tokens	answer text
match	32/32	32/32

Video: 16 distinct 2-second 320×240 clips (8 fps). The metadata proxy logged 16 rewrites. The throughput table above is a separate 64-clip vllm bench serve run, not this greedy check.

video	prompt tokens	answer text
match	16/16	16/16

Image and video placeholder reconstruction, without a serving stack, is test_glm5_metadata_only_roundtrip in tests/models/multimodal/processing/test_glm5next.py. On the H200 pod, with this patch and the local /models/GLM-5.3-Flash processor:

python3 -m pytest --confcutdir=tests/models/multimodal/processing \
  tests/models/multimodal/processing/test_glm5next.py \
  -k test_glm5_metadata_only_roundtrip -q

2 passed in 24.56s (image and video). Each case builds the prompt from raw media, publishes the EC metadata, and checks that metadata-only inputs reproduce the same prompt token ids and placeholders.

Details

Glm4vMultiModalDataParser.embedding_fields marks the grid and video timestamps as placeholder metadata and the embeds as values, so allow_missing_mm_embeddings drops only the embeds on an EC consumer. Timestamps are computed with the existing GLM-5.3 frame sampler and stored on the processed video item, which is what the encoder publishes and what the prompt replacement reads. Image replacement was already grid-sized; it now works when the request carries only image_grid_thw.

Video still needs those timestamps because a GLM placeholder includes one timestamp string per temporal patch, not only grid.prod(). Requests that omit the metadata stay on the raw-media path.

Pull Request Checklist

I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).
AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

mergify · 2026-10-27T15:17:50Z

Documentation preview: https://vllm--64071.org.readthedocs.build/en/64071/

Isotr0py

Also cc @gty111 @JaredforReal

Isotr0py · 2026-10-27T15:38:01Z

+### Vision inputs
+
+GLM-5.3-Flash accepts metadata-only image and video inputs through the Python
+EPD proxy. The encoder publishes `image_grid_thw` for each image, and
+`video_grid_thw` plus one timestamp per temporal patch for each video. The
+consumer rebuilds the placeholders from those fields and receives the
+embeddings through its EC connector, without repeating vision preprocessing.
+


I think this too model-specific in feature doc.

Removed the GLM-5.3-Flash section from docs/features/disagg_encoder.md. The feature doc no longer calls out one model.

Isotr0py · 2026-10-27T16:12:29Z

+    def _published_video_timestamps(self, video_item) -> list[int]:
+        """Timestamps the encoder publishes so a consumer can skip sampling."""
+        video, metadata = video_item
+        return self._get_video_second_idx_glm46v(metadata, len(video))
+
+    def _placeholder_from_timestamps(
+        self,
+        timestamps: Sequence[float] | torch.Tensor,
+        grid_thw: torch.Tensor,
+    ) -> list[int]:
+        """Video placeholder from a grid and the timestamps that size it."""
+        hf_processor = self.get_hf_processor()
+        tokenizer = self.get_tokenizer()
+        hf_config = self.get_hf_config()
+        merge_length = hf_processor.image_processor.merge_size**2
+        if isinstance(timestamps, torch.Tensor):
+            timestamp_values = timestamps.flatten().tolist()
+        else:
+            timestamp_values = list(timestamps)
+
+        frames_idx_token = [
+            tokenizer.encode(f"{float(ts):.1f} seconds", add_special_tokens=False)
+            for ts in timestamp_values
+        ]
+        _, height, width = grid_thw
+        num_tokens_per_frame = int(height * width) // merge_length
+        frame_embed_token_id = hf_processor.image_token_id
+        placeholder = [hf_config.video_start_token_id]
+        for frame_idx in frames_idx_token:
+            placeholder.append(hf_config.image_start_token_id)
+            placeholder.extend([frame_embed_token_id] * num_tokens_per_frame)
+            placeholder.append(hf_config.image_end_token_id)
+            placeholder.extend(frame_idx)
+        placeholder.append(hf_config.video_end_token_id)
+        return placeholder


I think this is duplicate with _construct_video_placeholder?

Done in 287204d
_construct_video_placeholder now only selects the timestamps and calls _placeholder_from_timestamps. The token assembly is not duplicated. The metadata-only path uses that same helper.

Isotr0py · 2026-10-27T16:14:09Z

+    def _apply_hf_processor_main(
+        self,
+        mm_items: MultiModalDataItems,
+        hf_kwargs: Mapping[str, object],
+    ):
+        processed = super()._apply_hf_processor_main(mm_items, hf_kwargs)
+        video_items = mm_items.get("video")
+        if (
+            video_items is not None
+            and video_items.get_count() > 0
+            and not isinstance(video_items, DictEmbeddingItems)
+            and "timestamps" not in processed
+        ):
+            # Publish the same timestamps the prompt replacement encodes, so an
+            # EC consumer can rebuild the placeholder without the frames.
+            processed["timestamps"] = [
+                self._glm_info()._published_video_timestamps(video_items.get(i))
+                for i in range(video_items.get_count())
+            ]
+        return processed


I think we should update these methods at vllm/model_executor/models/glm4_1v.py instead? GLM5.3-Flash's mm processing is quite similar to GLM4.6V.

Done in 287204d
Moved the metadata-only parser, timestamp publishing, and prompt update onto Glm4vProcessingInfo / Glm4vMultiModalProcessor. GLM-5.3-Flash inherits them. Its frame sampler still overrides _get_video_second_idx_glm46v, which this path calls.

gty111

Two suggestions on video timestamp handling:

gty111 · 2026-10-28T00:08:22Z

-        num_tokens_per_frame = int(H * W) // merge_length
-        placeholder = []
-        placeholder.append(bov_token_id)
+        _, height, width = grid_thw


Could we validate len(timestamp_values) == grid_thw[0] here and add a negative test? Otherwise, inconsistent metadata produces a placeholder count that does not match the encoder output.

Done in 002d127

_placeholder_from_timestamps now checks len(timestamps) == video_grid_thw[0] before it builds the placeholder. A mismatch raises ValueError instead of emitting a placeholder whose token count cannot match the encoder rows.

The negative case is test_placeholder_rejects_timestamp_count_mismatch in tests/models/multimodal/processing/test_glm5next.py. I reran that check on the current patch with the local zai-org/GLM-5.3-Flash processor: two timestamps against grid_t=4 raised the error.

gty111 · 2026-10-28T00:08:22Z

+            and "timestamps" not in processed
+        ):
+            processed["timestamps"] = [
+                self.info._video_timestamps(*video_items.get(i))


Could we derive timestamps from the actual frame indices used by the HF processor, where available, rather than recomputing sampling from the original metadata? This would avoid drift if sampling options or padding behavior change.

Done in 002d127

When frames_indices is already on the video metadata, the timestamp helpers use those indices instead of sampling again from fps and duration, including when do_sample_frames is still true. After call_hf_processor, indices recorded on the VideoMetadata are copied back onto the item, so the timestamps published for EPD follow the frames the processor kept. If the processor did not record indices, the previous sampler is unchanged.

test_video_timestamps_follow_supplied_frame_indices covers this. On the current patch with the local GLM-5.3-Flash processor, indices [0, 8, 80, 88] produced timestamps [0, 10]; resampling that clip without those indices produced 32 timestamps.

gty111 · 2026-10-28T00:13:02Z

ECExampleConnector over local disk, not Mooncake: ECMooncakeConnector producers require tensor_parallel_size=1, and this checkpoint does not fit one H200.

Would you mind sharing the encoder launch command? My understanding is that a dedicated ec_producer automatically enables encoder-only mode. In our TP=1 test on latest main, the vision weights were ~1.05 GiB and startup GPU memory was ~3.97 GiB without eager mode. Could you clarify why encoder TP=4 was needed here?

grYe99 · 2026-10-28T02:23:17Z

@gty111
Encoder launch command for the image run:

CUDA_VISIBLE_DEVICES=0,1,2,3 python3 -m vllm.entrypoints.cli.main serve /models/GLM-5.3-Flash \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --max-num-seqs 64 \
  --max-num-batched-tokens 8192 \
  --limit-mm-per-prompt '{"image":1,"video":0}' \
  --port 19534 \
  --ec-transfer-config '{"ec_connector":"ECExampleConnector","ec_role":"ec_producer","ec_connector_extra_config":{"shared_storage_path":"/ygr-sc/bench-glm53-epd-metadata/ec_cache"}}'

No separate --mm-encoder-only flag. The video throughput run used the same command except --limit-mm-per-prompt '{"image":0,"video":1}'.

Your reading of current main is right. A dedicated ec_producer turned this process into encoder-only mode. The encoder log says "this instance runs the multi-modal encoder only". TP=4 was not required for the encoder. We set it because ECMooncakeConnector requires the producer to use tensor_parallel_size=1, and at the time we were still treating the producer like a full-model server. The full GLM-5.3-Flash checkpoint does not fit one H200, so both sides were launched with ECExampleConnector at TP=4. The 76.46 GiB/rank figure is the PD worker, which is why the PD side stays at TP=4.

I reran the encoder alone on the current patch with --tensor-parallel-size 1 and --limit-mm-per-prompt '{"image":1,"video":1}', same otherwise as the command above. It became healthy as encoder-only. Model loading took 1.93 GiB, and the process used 4579 MiB on that one GPU. That is the same range as the ~1.05 GiB vision weights and ~3.97 GiB startup you measured. The published image and video throughput numbers were still measured with the TP=4 encoder above; this TP=1 run is a startup check, not a new throughput table.

gty111

Two issues found in the updated patch:

gty111 · 2026-10-28T02:55:59Z

-            # The loader already picked the frames and the processor keeps them
-            # as they are, so the frame count is what was handed in and
-            # ``frames_indices`` only supplies the timestamp values.
+        if "frames_indices" in metadata or not do_sample_frames:


Input frames_indices may describe frames before HF sampling. With 32 frames at 8 fps, frames_indices=list(range(32)), and do_sample_frames=True, I reproduced 16 timestamps for grid_t=4 on the raw-video path. The sampled-index writeback only covers the old Glm4vProcessor branch, not GLM-5.3's direct path. Could we synchronize the actual sampled indices there too and add this roundtrip case?

Done in fb25f7e.

Input frames_indices with do_sample_frames=True is the pre-sample range. The HF processor returns a new VideoMetadata whose frames_indices are the frames it kept; the object we passed in is left unchanged. GLM-5.3's direct path now copies those returned indices back onto the item before timestamps are built. The older Glm4vProcessor branch reads the returned metadata the same way.

test_glm5_raw_video_timestamps_follow_sampled_frames is the 32-frame, 8 fps, frames_indices=range(32) case. On this patch it produces grid_t=4 and timestamps [0, 1, 2, 3], and the metadata-only prompt matches the raw one.

gty111 · 2026-10-28T02:55:59Z

+    def _parse_image_data(self, data):
+        if isinstance(data, dict):
+            required, optional = self.embedding_field_sets("image")
+            return DictEmbeddingItems(


Could we validate supplied embeddings in both dictionary branches: rank, hidden size, and row count against the grid? With expected_hidden_size=16 and grid [1,4,4] (merge size 2), this parser accepts shapes [4,1], [4], and [3,16]. The ordinary tensor path rejects the wrong hidden size, while this path defers failures to model execution. Metadata-only inputs should remain valid.

Done in fb25f7e.

Both dictionary branches now check a supplied embedding before building DictEmbeddingItems: rank 2, hidden size against expected_hidden_size when that is set, and row count against prod(grid_thw) / merge_size**2. With expected_hidden_size=16 and grid [1, 4, 4] (merge size 2), shapes [4, 1], [4], and [3, 16] are rejected for image and video. Omitting the embedding tensor is still valid when allow_missing_mm_embeddings is set.

Covered by test_glm_dict_embeds_must_match_grid_and_hidden_size.

gty111

One remaining issue with multi-image embedding validation:

gty111 · 2026-10-28T06:45:47Z

+                f"grid_thw must have shape (N, 3), got {tuple(grid_thw.shape)}"
+            )
+        merge = self._spatial_merge_size**2
+        return int(grid.prod().item()) // merge


Could we sum the per-item row counts instead? grid.prod() multiplies across images, so two grids [1,4,4] with merge size 2 require 64 rows here instead of 8, rejecting valid embeddings. Suggested fix: int((grid.prod(dim=-1) // merge).sum().item()). A multi-image test would cover this.

Done in 55c590b.

_embed_row_count now sums each item's prod(t, h, w) / merge_size**2 instead of multiplying the whole grid tensor. Two [1, 4, 4] grids with merge size 2 accept 8 rows and reject 64. Image and video dict embeddings share this count. The multi-image case is in test_glm_dict_embeds_must_match_grid_and_hidden_size.

gty111

LGTM. Thanks for the work.

Isotr0py · 2026-10-29T04:57:59Z

/ci run

github-actions · 2026-10-29T05:00:06Z

✅ Triggered Buildkite CI #98208 for commit 19c5c9029612.

ℹ️ This PR is 180 commits behind upstream main. Merged upstream main into this branch as 19c5c9029612 before starting CI. Pull the branch before pushing more changes.

grYe99 · 2026-10-29T08:46:46Z

/ci run

github-actions · 2026-10-29T08:47:06Z

✅ Triggered Buildkite CI #98242 for commit c569fbea3cca.

grYe99 · 2026-10-29T08:54:39Z

@Isotr0py Thanks for the approval and kicking off CI.

Buildkite #98208 failed on computer-cpu-multimodal-processor (shards 1/2/4) in test_glm4_1v (test_video_loader_consistency / test_processor_override), not in the GLM-5.3 tests:

ValueError: Video timestamp count does not match video_grid_thw[0]: 122 timestamps for grid_t=5/10

Root cause: this PR started preferring metadata frames_indices whenever present. GLM-4.1V's opencv / processor path still carries the full pre-sample frames_indices while do_sample_frames=True, so timestamps stayed at ~122 against a sampled grid_t. The stricter _placeholder_from_timestamps check then failed. GLM-5.3 was fine because we write back the HF-sampled indices.

Fix in c569fbea3c (rebased onto latest main, dropped the CI merge commit):

restore if not do_sample_frames: for reusing frames_indices
after a successful HF sampled-index write-back, set do_sample_frames=False so GLM-5.3 still uses those indices

Re-triggered CI: https://buildkite.com/vllm/ci/builds/98242

Let an EC consumer size image and video placeholders from the encoder's grid and timestamps instead of running vision preprocessing again. Signed-off-by: grYe99 <guorongye99@gmail.com>

Move the placeholder metadata parser onto GLM-4V so GLM-5.3-Flash inherits it, and build video placeholders from the existing constructor. Signed-off-by: grYe99 <guorongye99@gmail.com>

InputProcessingContext is frozen, so the consumer-path check has to replace call_hf_processor on the class. Signed-off-by: grYe99 <guorongye99@gmail.com>

…context" This reverts commit b4594d7. Signed-off-by: grYe99 <guorongye99@gmail.com>

Compare raw and metadata-only prompt tokens without patching the frozen processor context. Signed-off-by: grYe99 <guorongye99@gmail.com>

Placeholder construction now requires one timestamp per temporal patch. When the processor or loader already selected frames, those indices are used instead of sampling again. Signed-off-by: grYe99 <guorongye99@gmail.com>

The direct path was publishing timestamps from the pre-sample frame range. Dict embeddings now have to match the grid and hidden size; metadata-only inputs stay valid. Signed-off-by: grYe99 <guorongye99@gmail.com>

grid.prod() across a batch of grids rejects a valid multi-image embedding. Each item's token count is computed first, then added. Signed-off-by: grYe99 <guorongye99@gmail.com>

Preferring frames_indices whenever present made GLM-4.1V reuse the pre-sample range and emit far more timestamps than video_grid_thw. Restore the do_sample_frames gate, and clear that flag after HF write-back so GLM-5.3 still keeps the sampled indices. Signed-off-by: grYe99 <guorongye99@gmail.com>

In-place write-back of do_sample_frames=False poisoned the caller's mm_data dict. A later re-parse (cache path in processing_correctness) then ran HF without sampling on the original odd-length video, so grid_t exceeded the timestamp count from the written-back indices. Copy the metadata dict into mm_items instead. Signed-off-by: grYe99 <guorongye99@gmail.com>

Feed HF-sampled frame metadata into the timestamp helper via overrides instead of mutating mm_items. test_processing_correctness reuses the same items for baseline then cache; persisting do_sample_frames=False against a full-length video made pass-2 HF emit mismatched timestamps for odd-length videos. Signed-off-by: grYe99 <guorongye99@gmail.com>

grYe99 · 2026-10-29T13:03:24Z

@Isotr0py Follow-up after #98242.

That run failed on GLM-5.3 test_processing_correctness (2 timestamps for grid_t=3). After c569fbe set do_sample_frames=False on write-back, shared mm_items were reused for baseline+cache while the video tensor stayed full-length, so pass-2 HF disagreed on grid_t.

Post-approval bugfixes now on the branch (force-pushed while rebasing from 55d3260 to e33a639):

c569fbe / bd9a1af: restore if not do_sample_frames: (fixes the glm4_1v 122-vs-grid_t issue from #98208).
55d3260 / 89fa863: copy metadata before write-back — protects caller mm_data, not enough for processing_correctness.
e33a639: stop mutating mm_items; sampled-frame overrides only feed the timestamp helper.

Risk vs your approved tip (19c5c902): (1) is a real semantic fix for GLM-4.1V pre-sample indices; (2)(3) are mutation hygiene and should not change EPD metadata-only publish/consume behavior. Local CPU processor tests for GLM-4.1V / GLM-5.3 processing_correctness / glm5next / epd are green on e33a639.

grYe99 · 2026-10-29T13:03:36Z

/ci run

github-actions · 2026-10-29T13:04:03Z

✅ Triggered Buildkite CI #98301 for commit 029db6e0279b.

ℹ️ This PR is 7 commits behind upstream main. Merged upstream main into this branch as 029db6e0279b before starting CI. Pull the branch before pushing more changes.

aoshen02 · 2026-10-30T03:14:39Z

pre-commit fail

Narrow MultiModalKwargsItem and NestedTensors types in test_glm5_raw_video_timestamps_follow_sampled_frames, which landed in vllm-project#64071 concurrently with vllm-project#64342 enabling stricter mypy for tests/models/multimodal. Signed-off-by: Micah Williamson <micah.williamson@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>

…oject#64071) Signed-off-by: grYe99 <guorongye99@gmail.com> Co-authored-by: grYe99 <guorongye99@gmail.com> Co-authored-by: vllm-agent <claw@inferact.ai> Signed-off-by: Dao Le <Dao007forever@gmail.com>

grYe99 requested review from AndreasKaratzas, DarkLight1337 and ywang96 as code owners October 27, 2026 15:17

claude Bot reviewed Oct 27, 2026

View reviewed changes

mergify Bot added documentation Improvements or additions to documentation multi-modality Related to multi-modality (#4194) glm labels Oct 27, 2026

DarkLight1337 requested a review from Isotr0py October 27, 2026 15:24

Isotr0py reviewed Oct 27, 2026

View reviewed changes

gty111 reviewed Oct 28, 2026

View reviewed changes

gty111 approved these changes Oct 28, 2026

View reviewed changes

grYe99 force-pushed the feat/glm53flash-epd-metadata-only branch from 55c590b to 097536d Compare October 28, 2026 08:07

Isotr0py approved these changes Oct 29, 2026

View reviewed changes

Isotr0py enabled auto-merge (squash) October 29, 2026 04:58

github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 29, 2026

auto-merge was automatically disabled October 29, 2026 08:37
Head branch was pushed to by a user without write access

grYe99 force-pushed the feat/glm53flash-epd-metadata-only branch from 19c5c90 to c569fbe Compare October 29, 2026 08:37

grYe99 force-pushed the feat/glm53flash-epd-metadata-only branch from c569fbe to 55d3260 Compare October 29, 2026 11:57

grYe99 added 2 commits October 29, 2026 20:18

[EPD] Send GLM-5.3-Flash placeholder metadata out of band

9fd772c

Let an EC consumer size image and video placeholders from the encoder's grid and timestamps instead of running vision preprocessing again. Signed-off-by: grYe99 <guorongye99@gmail.com>

[EPD] Share GLM vision metadata-only with GLM-4V

7c85ffd

Move the placeholder metadata parser onto GLM-4V so GLM-5.3-Flash inherits it, and build video placeholders from the existing constructor. Signed-off-by: grYe99 <guorongye99@gmail.com>

grYe99 added 9 commits October 29, 2026 20:18

[EPD] Patch GLM metadata roundtrip around a frozen processor context

f6fbe9f

InputProcessingContext is frozen, so the consumer-path check has to replace call_hf_processor on the class. Signed-off-by: grYe99 <guorongye99@gmail.com>

Revert "[EPD] Patch GLM metadata roundtrip around a frozen processor …

ce664c5

…context" This reverts commit b4594d7. Signed-off-by: grYe99 <guorongye99@gmail.com>

[EPD] Move the GLM metadata roundtrip next to the GLM-5.3 tests

8e97edc

Compare raw and metadata-only prompt tokens without patching the frozen processor context. Signed-off-by: grYe99 <guorongye99@gmail.com>

[EPD] Use HF-sampled GLM frame indices and check embed shapes

93e2a03

The direct path was publishing timestamps from the pre-sample frame range. Dict embeddings now have to match the grid and hidden size; metadata-only inputs stay valid. Signed-off-by: grYe99 <guorongye99@gmail.com>

[EPD] Sum GLM embed rows per image instead of multiplying grids

2f8afbf

grid.prod() across a batch of grids rejects a valid multi-image embedding. Each item's token count is computed first, then added. Signed-off-by: grYe99 <guorongye99@gmail.com>

grYe99 force-pushed the feat/glm53flash-epd-metadata-only branch from 55d3260 to e33a639 Compare October 29, 2026 12:31

Merge branch 'main' into feat/glm53flash-epd-metadata-only

029db6e

jeejeelee merged commit a17e79a into vllm-project:main Oct 30, 2026
135 checks passed

aoshen02 mentioned this pull request Oct 30, 2026

[CI] Fix mypy errors in test_glm5next.py #64644

Merged

AndreasKaratzas mentioned this pull request Oct 30, 2026

[Test][UMBP][6/10] Add embedded runtime and MORI integration coverage #61794

Merged

Conversation

grYe99 commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Claims

Validation

Details

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

mergify Bot commented Oct 27, 2026

Uh oh!

Isotr0py left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grYe99 Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grYe99 Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gty111 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grYe99 Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grYe99 Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gty111 commented Oct 28, 2026

Uh oh!

grYe99 commented Oct 28, 2026

Uh oh!

gty111 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grYe99 Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gty111 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gty111 left a comment

Choose a reason for hiding this comment

Uh oh!

Isotr0py commented Oct 29, 2026

Uh oh!

github-actions Bot commented Oct 29, 2026

Uh oh!

grYe99 commented Oct 29, 2026

grYe99 commented Oct 27, 2026 •

edited

Loading

grYe99 Oct 27, 2026 •

edited

Loading

grYe99 Oct 27, 2026 •

edited

Loading

grYe99 Oct 28, 2026 •

edited

Loading

grYe99 Oct 28, 2026 •

edited

Loading

grYe99 Oct 28, 2026 •

edited

Loading

grYe99 commented Oct 29, 2026 •

edited

Loading