feat(grpc): send preprocessed multimodal data to vLLM with hashing and structured tokens by CatherineSue · Pull Request #570 · smg-project/smg

feat(multimodal): add blake3 image hasher module

CatherineSue

committed

171d3aa

feat(proto): expand vLLM MultimodalInputs with preprocessed data fields

CatherineSue

committed

8022cce

feat(grpc): add mm_hashes to MultimodalData and expand into_vllm_proto

CatherineSue

committed

2e66130

feat(grpc): switch vLLM to preprocessed multimodal path with hashing

CatherineSue

committed

67f8bfb

feat(multimodal): add blake3 hash to ImageFrame at decode time

CatherineSue

committed

8c23931

refactor(grpc): replace ProcessedMessages.multimodal_images with multimodal_data

CatherineSue

committed

ba808a0

refactor(grpc): collapse multimodal pipeline to single-phase in preparation.rs

CatherineSue

committed

37cec93

style: rustfmt

CatherineSue

committed

86a5b96

feat(grpc): add batched_keys to MultimodalInputs proto and Rust pipeline

CatherineSue

committed

c52abf7

feat(grpc): expand TRT-LLM proto with preprocessed multimodal fields

CatherineSue

committed

5ee072c

Revert "feat(grpc): expand TRT-LLM proto with preprocessed multimodal fields"

CatherineSue

committed

8df1931

feat(grpc): add field_layouts and flat_keys to multimodal pipeline

CatherineSue

committed

18f38a8

feat(multimodal): add Qwen3-VL model processor spec

CatherineSue

committed

542f887

fix(multimodal): use int64 for patches_per_image to avoid torch.uint32 pickle crash

CatherineSue

committed

e6b6558

feat(multimodal): structured prompt tokens for Llama4 and pass PreprocessedImages to prompt_replacements

CatherineSue

committed

632288d

style: rustfmt

CatherineSue

committed

2c3ee5e

refactor(proto): remove unused image_data from vLLM MultimodalInputs

CatherineSue

committed

62b4846

fix(multimodal): use chunks_exact and strict length check in extract_aspect_ratios

CatherineSue

committed

c6b75ff

nit: comments cleanup

CatherineSue

committed

3595db9

fix(grpc): emit patch-only placeholder offsets for sglang multimodal

CatherineSue

committed

0728a34

fix(multimodal): align QwenVL placeholder_token_id with pad_token_id

CatherineSue

committed

2713a29

refactor(grpc): compute sglang patch offsets during token expansion

CatherineSue

committed

02fd148

Navigation Menu

feat(grpc): send preprocessed multimodal data to vLLM with hashing and structured tokens - #570

feat(grpc): send preprocessed multimodal data to vLLM with hashing and structured tokens#570
CatherineSue merged 22 commits into
mainfrom
chang/grpc-mm

Commits on Mar 1, 2026

feat(multimodal): add blake3 image hasher module

feat(proto): expand vLLM MultimodalInputs with preprocessed data fields

feat(grpc): add mm_hashes to MultimodalData and expand into_vllm_proto

feat(grpc): switch vLLM to preprocessed multimodal path with hashing

feat(multimodal): add blake3 hash to ImageFrame at decode time

refactor(grpc): replace ProcessedMessages.multimodal_images with multimodal_data

refactor(grpc): collapse multimodal pipeline to single-phase in preparation.rs

style: rustfmt

feat(grpc): add batched_keys to MultimodalInputs proto and Rust pipeline

feat(grpc): expand TRT-LLM proto with preprocessed multimodal fields

Revert "feat(grpc): expand TRT-LLM proto with preprocessed multimodal fields"

feat(grpc): add field_layouts and flat_keys to multimodal pipeline

feat(multimodal): add Qwen3-VL model processor spec

fix(multimodal): use int64 for patches_per_image to avoid torch.uint32 pickle crash

feat(multimodal): structured prompt tokens for Llama4 and pass PreprocessedImages to prompt_replacements

style: rustfmt

refactor(proto): remove unused image_data from vLLM MultimodalInputs

fix(multimodal): use chunks_exact and strict length check in extract_aspect_ratios

nit: comments cleanup

fix(grpc): emit patch-only placeholder offsets for sglang multimodal

fix(multimodal): align QwenVL placeholder_token_id with pad_token_id

refactor(grpc): compute sglang patch offsets during token expansion