[PyTorch-MPS] Document verified model support by WindChimeRan · Pull Request #859 · vllm-project/vllm-metal

WindChimeRan · 2026-09-28T08:01:49Z

Builds on #858, which adds verified Qwen3 support. Document verified Hunyuan Dense and OLMo2 support through upstream's auto model selection, without model-specific attention patches like #664.

Add the PyTorch MPS column to the model table; leave unverified models blank.

Both checkpoints matched 24/24 CPU transformers reference greedy tokens.

Benchmark

Model	Path	Output tok/s ↑	Mean TTFT (ms) ↓	Mean TPOT (ms) ↓
Hunyuan-1.8B-Instruct	MLX	384.0	771.5	68.0
	PyTorch MPS	288.4 (-24.9%)	1684.1	86.0
OLMo-2-0425-1B-Instruct	MLX	543.9	360.3	48.2
	PyTorch MPS	412.4 (-24.2%)	887.1	61.4

M5 Pro 64 GiB; stack c517256, vLLM 0.30 / PyTorch 2.13 / MLX 0.32.1. Same unquantized checkpoint and BF16 weights/KV per pair; greedy Sonnet 1024/128, 100 requests, 10/s, C32; context/batch limit 2048, 2,340 allocator blocks with 16-token attention blocks, prefix cache off, async on both paths. Two fresh-server runs per path in reversed order; identical token counts and no preemptions. Percentages are throughput changes versus MLX.

Copilot

Copilot review overview

🟡 Changes recommended

Newly admitted model configurations can fail automatic cache sizing because the memory estimator assumes an intermediate_size field.

Review effort: Balanced
Findings: 1 · 1

Open (2)

Handle decoder configs without intermediate_size in memory estimation · New
Clarify Hunyuan MPS requires unquantized weights · New

What changed in this PR

Broadens the opt-in PyTorch MPS backend beyond Qwen3 using upstream model selection.

Changes:

Replaces the model-name restriction with capability and attention-head checks.
Documents verified MPS support for Qwen3, Hunyuan Dense, and OLMo 2.

File	Description
`vllm_metal/pytorch_backend/worker.py`	Adds capability-based model admission checks.
`vllm_metal/envs.py`	Updates the backend comment only.
`docs/supported_models.md`	Adds MPS support columns, leaving unverified entries blank.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

ericcurtin · 2026-10-04T08:26:11Z

Thanks! Please fix Copilot's findings: determine_available_memory assumes intermediate_size (breaks OPT/GPT-2 layouts this now admits), and the Hunyuan example checkpoint is MLX-only. Rebase after #858. Setting to draft, please reopen when ready for re-review.

ericcurtin · 2026-10-05T21:46:54Z

Thanks! Blocked on #858 (stacked), so setting to draft. Please rebase onto main once it lands and reopen for re-review.

ericcurtin · 2026-10-07T08:32:14Z

Thanks! Blocked on #858 (stacked), so setting to draft. Please rebase onto main once it lands and reopen for re-review.

Signed-off-by: Ranran <ranranhaoranzhang@gmail.com>

ricky-chaoju

Hunyuan on MPS needs --generation-config vllm here: its generation_config.json sets repetition_penalty 1.05, so even temperature=0 requests get a 400. Might be worth a note in the table.

ricky-chaoju · 2026-10-14T02:03:01Z

@WindChimeRan I have an exam later today, so I'll take a look at #880 tonight.

Signed-off-by: Ranran <ranranhaoranzhang@gmail.com>

WindChimeRan · 2026-10-14T05:02:05Z

Thanks! reproduced red->greed locally with the notes. @ricky-chaoju

No worry on the review. Omni works will mainly depend on PR1 (already merged)

Stacked on #859. Add opt-in MPS support for dense LFM2/LFM2.5 through upstream `Lfm2ForCausalLM` and MRv2. - Reuse vLLM's Torch ShortConv implementation through its pluggable-layer hook. - Reuse scheduler-owned state copying/clearing and the existing cache allocation. - Mark LFM support in the PyTorch MPS model column. LFM2.5-1.2B FP16 matched 24/24 CPU-reference greedy tokens; prefix reuse and chunked prefill pass. BF16 normalization rounding changed 18/512 fixed-history greedy choices versus eager (ΔNLL +0.00203). Validation: 3,857 non-slow tests passed; lint and types pass. Cache stress completed with 11 preemptions and 80 cached-prefix tokens. No new kernels or dependencies. Explicit async scheduling passes concurrent HTTP, cancellation and stop-token checks in FP16. BF16 batch/preemption rounding can change near-tied greedy choices. ## Benchmark | Model | Path | Output tok/s ↑ | Mean TTFT (ms) ↓ | Mean TPOT (ms) ↓ | | --- | --- | ---: | ---: | ---: | | LFM2.5-1.2B-Instruct | MLX | 806.8 | 255.8 | 30.7 | | | PyTorch MPS | 573.0 (-29.0%) | 549.9 | 43.1 | M5 Pro 64 GiB; stack `c517256`, vLLM 0.30 / PyTorch 2.13 / MLX 0.32.1. Same unquantized checkpoint and BF16 weights/KV per pair; greedy Sonnet 1024/128, 100 requests, 10/s, C32; context/batch limit 2048, 2,340 allocator blocks with 16-token attention blocks, prefix cache off, **async on both paths**. Two fresh-server runs per path in reversed order; identical token counts and no preemptions. Percentages are throughput changes versus MLX. --------- Signed-off-by: Ranran <ranranhaoranzhang@gmail.com>

WindChimeRan added this pull request to stack #860 September 28, 2026 08:02

WindChimeRan mentioned this pull request Sep 28, 2026

[PyTorch-MPS][MRv2] Support Hunyuan Dense and OLMo2 via upstream models #857

Closed

WindChimeRan force-pushed the stack/mps-upstream-models branch from 994a675 to 676fdfe Compare September 28, 2026 09:18

WindChimeRan mentioned this pull request Sep 28, 2026

[PyTorch-MPS][MRv2] Use upstream Triton sampling directly on MPS #865

Closed

WindChimeRan force-pushed the stack/mps-upstream-models branch from 676fdfe to c6b41fa Compare September 28, 2026 09:30

WindChimeRan removed this pull request from stack #860 September 28, 2026 20:31

WindChimeRan added this pull request to stack #871 September 28, 2026 20:32

WindChimeRan force-pushed the stack/mps-upstream-models branch from c6b41fa to 79c1ff7 Compare September 28, 2026 22:20

WindChimeRan mentioned this pull request Sep 28, 2026

[PyTorch-MPS][MRv2] Add LFM2/LFM2.5 ShortConv support #880

Merged

WindChimeRan force-pushed the stack/mps-upstream-models branch 2 times, most recently from 0ae344b to 67aa993 Compare October 3, 2026 16:03

WindChimeRan force-pushed the stack/mps-upstream-models branch 3 times, most recently from 1ac6ec3 to 40c42e8 Compare October 4, 2026 06:04

WindChimeRan marked this pull request as ready for review October 4, 2026 06:36

ericcurtin requested a balanced review from Copilot October 4, 2026 07:00

Copilot started reviewing on behalf of ericcurtin October 4, 2026 07:01

View session

Copilot AI reviewed Oct 4, 2026

View reviewed changes

Comment thread vllm_metal/pytorch_backend/worker.py Resolved

Comment thread docs/supported_models.md Resolved

ericcurtin marked this pull request as draft October 4, 2026 08:26

WindChimeRan force-pushed the stack/mps-upstream-models branch 4 times, most recently from 249e9a6 to e8c4dfd Compare October 4, 2026 22:49

WindChimeRan force-pushed the stack/mps-upstream-models branch 3 times, most recently from 99a3ead to da18035 Compare October 5, 2026 18:36

WindChimeRan marked this pull request as ready for review October 5, 2026 20:11

WindChimeRan force-pushed the stack/mps-upstream-models branch from ac9ce1e to edd4c6e Compare October 5, 2026 20:29

ericcurtin marked this pull request as draft October 5, 2026 21:46

WindChimeRan marked this pull request as ready for review October 5, 2026 21:49

ericcurtin marked this pull request as draft October 7, 2026 08:32

WindChimeRan force-pushed the stack/mps-upstream-models branch from edd4c6e to edf8667 Compare October 7, 2026 20:46

WindChimeRan force-pushed the stack/mps-upstream-models branch from edf8667 to 418933b Compare October 7, 2026 21:25

WindChimeRan marked this pull request as ready for review October 8, 2026 04:01

WindChimeRan force-pushed the stack/mps-upstream-models branch 2 times, most recently from e870c85 to fbd5835 Compare October 8, 2026 05:37

Base automatically changed from stack/mps-mrv2-base to main October 13, 2026 21:21

WindChimeRan added 3 commits October 13, 2026 16:21

Document verified models for the PyTorch MPS backend

7980e12

Signed-off-by: Ranran <ranranhaoranzhang@gmail.com>

Use the shared Hunyuan checkpoint in the model table

1bb6291

Signed-off-by: Ranran <ranranhaoranzhang@gmail.com>

Clarify that blank model cells describe MPS verification

5eeb73b

Signed-off-by: Ranran <ranranhaoranzhang@gmail.com>

WindChimeRan force-pushed the stack/mps-upstream-models branch from ac10bfd to 5eeb73b Compare October 13, 2026 21:21

ricky-chaoju approved these changes Oct 14, 2026

View reviewed changes

Document Hunyuan MPS generation defaults

f689390

Signed-off-by: Ranran <ranranhaoranzhang@gmail.com>

WindChimeRan merged commit 9c7b463 into main Oct 14, 2026
3 checks passed

Navigation Menu

Repository navigation

[PyTorch-MPS] Document verified model support - #859

[PyTorch-MPS] Document verified model support#859
WindChimeRan merged 4 commits into
mainfrom
stack/mps-upstream-models

WindChimeRan commented Sep 28, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

ericcurtin commented Oct 4, 2026

Uh oh!

ericcurtin commented Oct 5, 2026

Uh oh!

ericcurtin commented Oct 7, 2026

Uh oh!

ricky-chaoju left a comment

Uh oh!

ricky-chaoju commented Oct 14, 2026

Uh oh!

WindChimeRan commented Oct 14, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

WindChimeRan commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

ericcurtin commented Oct 4, 2026

Uh oh!

ericcurtin commented Oct 5, 2026

Uh oh!

ericcurtin commented Oct 7, 2026

Uh oh!

ricky-chaoju left a comment

Choose a reason for hiding this comment

Uh oh!

ricky-chaoju commented Oct 14, 2026

Uh oh!

WindChimeRan commented Oct 14, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

WindChimeRan commented Sep 28, 2026 •

edited

Loading