Skip to content

[UMBP][5/10] Add MORI embedded DRAM runtime and transfer handling - #13

Draft
charxwu wants to merge 227 commits into
review/umbp-split-2-lookupfrom
review/umbp-split-3a-embedded
Draft

[UMBP][5/10] Add MORI embedded DRAM runtime and transfer handling#13
charxwu wants to merge 227 commits into
review/umbp-split-2-lookupfrom
review/umbp-split-3a-embedded

Conversation

@charxwu

@charxwu charxwu commented Oct 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

This PR is split from the original full PR: vllm-project#58568

Register UMBPStoreConnector and add the worker-private MORI embedded DRAM runtime.
Register GPU KV buffers and move them with bulk and ranged MORI I/O, without a connector-owned host staging copy.
Handle timeouts, cancellation, eviction retry, lookup sockets, and DRAM configuration.

Implementation only. Pipeline parallelism is rejected. Usage and limits are in docs/features/umbp_embedded_dram.md. CPU contract tests and MORI integration coverage are in the next PR.

Commit: 063282ba40. Review slice of vllm-project/vllm#58568 for RFC #57187. The earlier combined embedded draft is vllm-project/vllm#58568.

AI assistance was used to draft this description. The submitting human reviews the code.

This was referenced Oct 14, 2026
Draft
Draft
@charxwu
charxwu added this pull request to stack #18 October 15, 2026 09:05
@charxwu
charxwu force-pushed the review/umbp-split-3a-embedded branch 2 times, most recently from 0f213f4 to ac33086 Compare October 21, 2026 07:06
@charxwu
charxwu force-pushed the review/umbp-split-3a-embedded branch from ac33086 to 49f4077 Compare October 22, 2026 18:07
@charxwu
charxwu force-pushed the review/umbp-split-3a-embedded branch 2 times, most recently from 22ce3c1 to 804c649 Compare October 25, 2026 06:01
kfirtoledo and others added 19 commits October 26, 2026 11:16
@kfirtoledo @orozery
[KV Offload] Load small CPU-tier hits synchronously within the step (v…
fe5fe8b 
…llm-project#61961)

Signed-off-by: Kfir Toledo <kfir.toledo@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
@pepmach @claude @vllm-agent
[Bugfix][KV Offload] Do not store sliding-window chunks with null blo…
1d3f05a 
…cks (vllm-project#63652)

Signed-off-by: Stan <273962421+pepmach@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
@BaoYunkai @claude @shen-shanshan
[Bugfix][ROCm] Match the DeepSeek-V4.1 FP8 config in the shared-exper…
c3941c0 
…t FSE compatibility check (vllm-project#63099)

Signed-off-by: BaoYunkai <269084791+BaoYunkai@users.noreply.github.com>
Co-authored-by: BaoYunkai <269084791+BaoYunkai@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Shanshan Shen <467638484@qq.com>
@nilig @vllm-agent
[Bugfix][KV Offload][P2P] Exclude rc_gda lanes from the P2P NIXL agent (
71294c2 
vllm-project#63498)

Signed-off-by: nilig <nili.ifergan@gmail.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
@snadampal @erezzarum
@seanwestfall @claude
[docker] add vllm-openai-efa target (vllm-project#58768)
ffcabb8 
Signed-off-by: Sunita Nadampalli <nadampal@amazon.com>
Co-authored-by: Erez Zarum <erezz@amazon.com>
Co-authored-by: Sean Westfall <sean.westfall@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@charxwu @tjtanaa
[Test][UMBP][4/10] Cover async lookup, hybrid cache, events, and metr…
51d43b3 
…ics (vllm-project#61792)

Signed-off-by: CharlesWu <Charles.Wu@amd.com>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
@amirfr3 @claude
[Metrics][KV Offload] Fill vllm:kv_offload_config_info with the CPU c…
5b06b5f 
…ache capacity (vllm-project#53902)

Signed-off-by: Amir Friedman <Amir.Friedman1@ibm.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@nilig @vllm-agent
[Bugfix][KV Offload][P2P] Mark the session dead when its ConnectAck s…
f682f1d 
…end fails (vllm-project#63664)

Signed-off-by: nilig <nili.ifergan@gmail.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
@Vegetog @codex
[Bugfix][KV Offload] Preserve new loads during same-step resume (vllm…
a64fc94 
…-project#60574)

Signed-off-by: Vegetog <110553275+Vegetog@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
@Sy0307 @vllm-agent
[Bugfix][Multimodal] Release consumed MRV2 CPU inputs (vllm-project#5…
373614f 
…7368)

Signed-off-by: Sy03 <1370724210@qq.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
@shen-shanshan @vllm-agent
[ROCm][Perf] Make MXFP4 the default indexer K cache dtype for DeepSee…
d0d681e 
…k-V4 (vllm-project#63669)

Signed-off-by: shen-shanshan <467638484@qq.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
@mgoin
[Perf][DeepSeek V4] Declare MXFP8 MoE activations and run them as blo…
fcb7870 
…ck FP8 on Hopper (vllm-project#61875)
@sudhanshu112233shukla @vllm-agent
[Bugfix] Preserve fp16 in legacy fused MoE LoRA (vllm-project#59931)
2f1cb15 
Signed-off-by: sudhanshu shukla <sudhanshu112233shukla@users.noreply.github.com>
Co-authored-by: sudhanshu shukla <sudhanshu112233shukla@users.noreply.github.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
@Vedant-Agarwal @claude @vllm-agent
[Bugfix][KV Offload][OBJ] Namespace object keys by blocks_per_chunk (v…
8c6e770 
…llm-project#63655)

Signed-off-by: Vedant Agarwal <vedantagwl10@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
@hlin99 @NickLucche
[docs] correct NIXL hybrid SSM heterogeneous TP support (vllm-project…
74340c7 
…#63036)

Signed-off-by: Tony Lin <tony.lin@intel.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
@Levius-Fubuki @vllm-agent
[Bugfix][KV Offload] Roll back failed P2P NIXL registration (vllm-pro…
8a4b018 
…ject#63628)

Signed-off-by: levius <2114377220@qq.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
@javieratomic @claude @vllm-agent
[Bugfix][KV Offload] Hold promotions after a cache reset until jobs r…
14aefcd 
…eport (vllm-project#62929)

Signed-off-by: Javier López <59578422+javieratomic@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
@liranschour @vllm-agent
[Bugfix][KV Offload][P2P] Never block in ZmqConnection.send() (vllm-p…
51d2a85 
…roject#63708)

Signed-off-by: Liran Schour <lirans@il.ibm.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
@OnePunchMonk @claude
@jeejeelee @vllm-agent
[Bugfix][LoRA] Fix double pinning of 3D MoE LoRA and EP dummy LoRA si…
14d5931 
…zing (vllm-project#61636)

Signed-off-by: onepunchmonk <aggarwal.avaya@yahoo.com>
Signed-off-by: OnePunchMonk <aggarwal.avaya@yahoo.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
Hritik003 and others added 30 commits October 28, 2026 03:42
@Hritik003 @vllm-agent
[Bugfix][Frontend] fix Anthropic inline-system probe crashing startup (…
b766f15 
…vllm-project#64020)

Signed-off-by: Hritik Raj <hritik.raj@nutanix.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
@yiminghub2024 @cursoragent @vllm-agent
[Doc] Add autotrust/JEV-27B-VL multimodal recipe (vLLM >= 0.32.0) (vl…
9bc1c8b 
…lm-project#64180)

Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: yiminghub2024 <yiminghub2024@users.noreply.github.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
@AndreasKaratzas @claude
[CI][Bugfix] Pin the ColUltraVBERT test checkpoint to the revision vL…
7e3e3f6 
…LM supports (vllm-project#64189)

Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
Co-authored-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
@AlanFokCo @aoshen02
[Feature][Core] Add opt-in multi-worker CUDA checkpoint sleep (vllm-p…
88d3995 
…roject#64186)

Signed-off-by: AlanFokCo <alanfok2868@gmail.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
@lucifer1004 Kimi Code @claude
[Perf][SM120][MLA] Use compact FP8 KV rows for NoPE models (vllm-proj…
dbcf568 
…ect#58305)

Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@vadiklyutiy @claude @gau-nernst
[Bugfix] Merge supported KV cache layouts across pipeline stages inst…
edad8b0 
…ead of asserting they match (vllm-project#61294)

Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
@vadiklyutiy @claude
[Bugfix] Omit null annotations and usage.prompt_tokens_details in…
f60edcc 
… chat completions (vllm-project#64177)

Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
@jperezdealgaba
[Security] Fix ReDoS in validation error path sanitizer (vllm-project…
665bfc8 
…#52120)

Signed-off-by: jperezde <jperezde@redhat.com>
@jperezdealgaba
[Security] Bound FunAudioChat runtime audio to the profiled duration (v…
26302eb 
…llm-project#57411)

Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com>
@bvolpato @codex
[Bugfix][Pooling] Reject prompt-wide truncation for Jina ranking (vll…
e82e6d4 
…m-project#55200)

Signed-off-by: Bruno Volpato <brunocvcunha@gmail.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
@chaojun-zhang @jikunshang
[CI][XPU] Deselect Mamba deep-sleep case in Intel CI (vllm-project#64057
08a236f 
)

Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
@Yejing-Lai @vllm-agent
[XPU][CI] Fix test_tp1_fp8_fusions rms_quant_fusion on XPU (vllm-proj…
41dd82e 
…ect#63975)

Signed-off-by: Lai, Yejing <yejing.lai@intel.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
@yma11 @vllm-agent
[XPU] bump xpu-kernels to v0.1.16.2 (vllm-project#61956)
fd3469f 
Signed-off-by: Yan Ma <yan.ma@intel.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
@nbommi19 @claude
[Perf][DSv4.1] Tile the NVFP4 compressed-KV dequant-gather (vllm-proj…
6abdff5 
…ect#63018)

Signed-off-by: Nitin Sai Bommi <nbommi@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@paolo-perrone @DarkLight1337
[Frontend] Support /v1/completions in run-batch (vllm-project#64097)
9fc019a 
Signed-off-by: paoloap <58889145+paoloap-py@users.noreply.github.com>
Co-authored-by: paoloap <58889145+paoloap-py@users.noreply.github.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
@amenomori
[Bugfix][Spec Decode] Align adaptive draft prefill graph padding (vll…
5118d99 
…m-project#62796)

Signed-off-by: Yusen Gao <amenomorikou@gmail.com>
@RuibinCheung @cursoragent
@shen-shanshan @vllm-agent
[ROCm][MiniMax-M3.1] Fuse MXFP8 activation quant into the Gemma input…
1cb31be 
…_layernorm for qkv_proj (vllm-project#63637)

Signed-off-by: RuibinCheung <ruibzhan@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Shanshan Shen <467638484@qq.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
@tjtanaa
Merge branch 'main' into review/umbp-split-3a-embedded
ac1322b
@Juntian777 @claude
@ivanium @codex
[KV Connector][NIXL] Select push mode for NixlConnector via transfer_…
066d979 
…mode (vllm-project#64230)

Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
@gcanlin @claude @mergify
[Bugfix][DSA] Reserve the decode indexer logits in the memory profile (…
317081f 
…vllm-project#62860)

Signed-off-by: Canlin <canlinguosdu@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
@subhashpolisetti
[Bugfix][Benchmark] Restore --input-len/--output-len/--prefix-len in …
deb1a97 
…vllm bench throughput (vllm-project#64227)

Signed-off-by: subhash polisetti <subhashr161347@gmail.com>
@nrenvo @vllm-agent
[Bugfix][Multimodal] Fix DeepSeek-OCR zero-sum patch guard in _encode… (
76c4ad2 
vllm-project#64091)

Signed-off-by: nrenvo <nrenvo@gmail.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
@Juntian777 @claude
[Bugfix][KV Offload] Count bounded-replay SWA allocations in the lazy…
77f58e5 
… walk target (vllm-project#64229)

Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@vllm-agent
Merge branch 'main' into review/umbp-split-3a-embedded
7db1ce3
@Thangnguyenvn98 @claude
[CI] Move Attention, Quantization and MoE kernel tests and their AMD …
5ea8301 
…mirrors from fixed to automatic shards (vllm-project#62077)

Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@vMaroon
[Bugfix][KV Offload][P2P] Stop DEALER monitor before closing its read…
f4fd1ab 
…er (vllm-project#63207)

Signed-off-by: Maroon Ayoub <mayoub@redhat.com>
@bantianyue @taneem-ibrahim
[Doc] Fix long-text embedding example pooler config and GPU pinning (v…
9b9eeb2 
…llm-project#64170)

Signed-off-by: bantianyue <bantianyue@gmail.com>
Co-authored-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
@AnvitDevadiga AnvitDevadiga @Harry-Chen
[Build] keep CUDA PyTorch wheels aligned with image (vllm-project#59198)
ab0e5ef 
Signed-off-by: AnvitDevadiga <anvit.mail@icloud.com>
Co-authored-by: AnvitDevadiga <anvit.mail@icloud.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
@charxwu @codex
[Test][UMBP] Keep lookup socket tests below the Unix path limit
b40ad63 
Use a short temporary socket directory independent of Buildkite's nested
TMPDIR. Preserve both ownership regressions and their existing assertions.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: CharlesWu <Charles.Wu@amd.com>
@vllm-agent
Merge branch 'main' into review/umbp-split-3a-embedded
e16a5fb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.