Add VLM support to the dynamic-batching inference server - #6260

Merged

cspades merged 45 commits into

NVIDIA:mainfrom

RPrenger:vlm-inference

Aug 20, 2026

Commits on Aug 15, 2026

Add VLM support to the dynamic-batching inference server
RPrenger
committed
a392b42
NeMo-RL working branch.

cspades
authored and
RPrenger
committed
2402b1d
Match vLLM multimodal API.

cspades
authored and
RPrenger
committed
23fad30
Contain SUBMIT_REQUEST errors so one bad payload doesn't kill the engine
RPrenger
committed
5080610
Clean up VLM request data if _add_request rejects the request
RPrenger
committed
e43f374
Reject incomplete static-tiling image payloads at the wire boundary
RPrenger
committed
c373111
Preserve VLM subtype and image data across suspend/resume
RPrenger
committed
c7833b4
Fix placeholder token bug with RL + Gym.

cspades
authored and
RPrenger
committed
0299f03
Address PR comments for cspades' changes.

cspades
authored and
RPrenger
committed
5ee057b
Apply black/isort autoformatter to the current changeset
RPrenger
committed
25dccea
Guard the truncation branch for final_position_ids being None
RPrenger
committed
bdf13de
Stop threading a non-image-aware arange into the language model as position_ids
RPrenger
committed
592fd9c
Restore h/w kwargs on pixel_shuffle to keep existing callers working
RPrenger
committed
d513661
Fail loudly for audio / video kwargs on LLaVAModel.forward instead of silently ignoring
RPrenger
committed
3a39431
Raise instead of assert on image-embedding count mismatch
RPrenger
committed
4418ab6
Fix RADIO class_token_len override chain and self._class_token_len
RPrenger
committed
bd99491
Fold two D2H syncs in packed_seq_params rebuild into one
RPrenger
committed
591f560
Default inference_wrapper_cls to GPTInferenceWrapper on MegatronLLM / MegatronAsyncLLM
RPrenger
committed
151c8ce
Make _generate_impl.multi_modal_data_list optional (backward-compat)
RPrenger
committed
76a89ef
Match the decode-branch rank in current_image_token_mask to the prefill branch
RPrenger
committed
2aba70e
Short-circuit current_image_embeddings on decode steps
RPrenger
committed
fdff099
Skip VLM context helpers entirely on text-only workloads
RPrenger
committed
3ed1164
Collapse per-image .item() syncs to a single tolist() in expand_image_tokens
RPrenger
committed
78adb52
Skip num_tiles.sum() D2H sync on dynamic-resolution admissions
RPrenger
committed
7909b4f
Reject HTTP redirects in the image_url fetch path
RPrenger
committed
643a615
Note the n>1 image re-preprocessing cost as a known limitation
RPrenger
committed
084c7be
Drop the wire-side static-tiling helper to remove megatron/core → examples/ import
RPrenger
committed
67c7239
Stop mutating sys.path inside megatron/core
RPrenger
committed
5e61e13
Assert MoE hybrid config has num_moe_experts set in get_hybrid_layer_spec_te
RPrenger
committed
d96819c
Give the CLS token a position embedding in the learned_absolute ViT path
RPrenger
committed
ebc35b7
Move KimiLearned2DPosEmbed weight to device on both forward paths
RPrenger
committed
a972aa2
Pin apply_chat_template return_dict=False to keep the tensor/list return shape
RPrenger
committed
b20d25e
Declare Pillow as a megatron-core dev dep for VLM image preprocessing
RPrenger
committed
4eb95a4
Guard the torchvision import in image_preprocessing with an install-me error
RPrenger
committed
4915df7
Re-run isort/black on files touched in this round of review fixes
RPrenger
committed
1af7efc
Refresh uv.lock so it matches the Pillow entry added to pyproject.toml
RPrenger
committed
a9464c7

Commits on Aug 17, 2026

Merge branch 'main' into vlm-inference
RPrenger
authored
f1180d4
Lazy-import add_inference_args in vlm_dynamic_inference
RPrenger
committed
43a3650
Add docstring to _NoRedirectHandler.http_error_301 for pylint C0116
RPrenger
committed
9e56ee8
Address unit-test regressions from run 31997499116
RPrenger
committed
54e2c47
Reformat test_llava_sound.py after skip-decorator addition
RPrenger
committed
93f19e7
Also gate packed_seq_params on the vision-model call
RPrenger
committed
17bd43a
Guard third VLM dict + skip config lookup for text-only requests
RPrenger
committed
7d2cb12
Stop passing dead mtp_source_loss_mask kwarg to language model
RPrenger
committed
11abf32

Commits on Aug 18, 2026

Merge branch 'main' into vlm-inference
RPrenger
authored
cffcdbc