[Model] Add MiniCPM-V 4.6 support

[AgainstEntropy]

Motivation

Add support for MiniCPM-V 4.6, the next iteration of the MiniCPM-V series. Compared to 4.5 (Qwen3 + SigLip + Perceiver-style resampler) the architecture changes are:

• Mid-ViT compression: a 2x2 window-attention + 2x2 spatial fold inserted inside the SigLip encoder at config.insert_layer_id.
• Post-encoder MLP merger (MiniCPMV_Merger) replacing the legacy Perceiver resampler.
• Qwen3.5 hybrid backbone in the LLM tower.
• config.downsample_mode toggles "16x" (mid-ViT + post merger, default) vs "4x" (skip mid-ViT, keep 4x more visual tokens).

Usage

Cookbook PR: #24876

sglang serve --model-path openbmb/MiniCPM-V-4_6 \
  --trust-remote-code \
  --dtype bfloat16 \
  --mem-fraction-static 0.15 \
  --mamba-scheduler-strategy extra_buffer \
  --reasoning-parser qwen3 \
  --host 0.0.0.0 --port 30000

Note --dtype bfloat16 is required here ckpt's config.json may not declare a torch_dtype / dtype; without --dtype bfloat16, sglang's default dtype-resolution path triggers a bf16/fp16 conditional-load mismatch in the GDN causal_conv1d Triton kernel. Reproduced on the current sgl-kernel 0.4.2.post1.

Example image+text request (OpenAI-compatible):

curl http://127.0.0.1:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openbmb/MiniCPM-V-4_6",
    "messages": [{"role": "user", "content": [
      {"type": "image_url", "image_url": {"url": "https://example.com/cat.jpg"}},
      {"type": "text", "text": "What is in this image?"}
    ]}],
    "chat_template_kwargs": {"enable_thinking": false}
  }'

chat_template_kwargs.enable_thinking toggles 4.6's reasoning mode: true (default) emits a <think>...</think> block routed to reasoning_content; false skips reasoning and routes the answer straight into content. Multi-image and video are passed as additional image_url / video_url parts in the same content list.

See more in the MiniCPM-V 4.6 cookbook.

Accuracy Tests

python benchmark/mmmu/bench_sglang.py --port 30000 --concurrency 32 --max-new-tokens 2048

MMMU val (900): T=0.6 sampling, #24084 extractor, ~37-39% across trials (consistent with the reference number on the same ckpt from MiniCPM team).

Speed Tests and Profiling

sglang.bench_serving on the open test ckpt, 1× H200, BF16, --mem-fraction-static 0.5, no TP/DP.

• Text-only (random 1000 input / 1000 output tokens):

Concurrency	Req/s	Input tok/s	Output tok/s	Median TTFT	Median TPOT	Median E2E
1	1.34	816	565	104 ms	1.44 ms	590 ms
100	21.24	10,675	10,628	185 ms	9.16 ms	4.3 s

• Vision (random 720p image + 128 input / 1024 output tokens, --chunked-prefill-size -1):

Concurrency	Req/s	Input tok/s	Output tok/s	Median TTFT	Median TPOT	Median E2E
1	0.97	75	411	403 ms	1.44 ms	898 ms
100	2.78	222	1,419	34.3 s	1.45 ms	35.3 s

Full commands and numbers will be in the MiniCPM-V 4.6 cookbook.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[AgainstEntropy]

/tag-and-rerun-ci

[JustinTong0323]

Great Job!!

[AgainstEntropy]

/rerun-failed-ci, try again