[Draft][MoE][Offload] Row-level expert weight cache provider (moe_expert_cache_provider=row) by 01554 · Pull Request #47 · 01554/vllm

[MoE][Offload] moe_expert_cache_provider knob (cached|row, default cached) and RowCacheWeightProvider skeleton with the CachedWeightProvider surface

01554

committed

cb7220a

[MoE][Offload] RowCacheWeightProvider: per-token capacity check in plan_chunks, validate explicit ids before mutation, explicit device kwarg (review)

01554

committed

9bdb7ef

[MoE][Offload] RowCacheWeightProvider: own the host source (clone, never alias), owner-stream check on invalidate, accurate copy wording (review)

01554

committed

3137683

[MoE][Offload] RowCacheWeightProvider.prepare: validate before any state change (owner record and counters after validation) (review)

01554

committed

d1c0e63

[MoE][Offload] RowCacheWeightProvider: staged copies on a provider-owned copy stream with release/ready events, forward-generation map; CPU event-order mock and CUDA reuse/race tests

01554

committed

14cba23

[MoE][Offload] RowCacheWeightProvider: last_copies on both paths, invalidate ordering comment matches the contract, CUDA race test proves reader overlap over all six slot tensors, invalidate->prepa…

01554

committed

a38a9c4

[MoE][Offload] provider tests: staging mock expects copy-only wait on invalidate; CUDA tests state allocator behaviour accurately and prove reader overlap for invalidate->prepare too (review)

01554

committed

e8b48e8

tests: captured-graph reader fixture for RowCacheWeightProvider CUDA race tests

01554

committed

e5632e3

tests: feedback-free GPU delay and distinct-sum precheck in race fixture

01554

committed

d54f3c8

[Feature] CachedWeightProvider: optional per-expert second-level scales

01554

committed

6fec9af

tests: report prepare host time vs reader GPU time when the race fixture is inconclusive

01554

committed

ebb0a42

[Bugfix] RowCacheWeightProvider: upload expert maps without a host sync

01554

committed

d26e289

[Feature] RoutedExperts: select the expert cache provider, carry scale_2 through slots

01554

committed

48db118

[Feature] ModelOptNvFp4FusedMoE: opt into the expert cache on the Marlin backend

01554

committed

d83afbb

tests: gate the NVFP4 consumer test on Marlin capability instead of skipping on any ValueError

01554

committed

d112e19

[Feature] NVFP4 expert cache: load expert tensors into pinned host memory, repack Marlin in chunks

01554

committed

1ca5a99

review: release each repacked chunk before the next, validate expert_chunk, alias contract wording, host/pinned checks in tests

01554

committed

abb4871

tests: CPU contract check that a non-positive expert_chunk is rejected before side effects

01554

committed

6d80aaf

tests: gate the NVFP4 consumer test on is_fp4_marlin_supported (the FP4 Marlin path's own predicate)

01554

committed

49cdc04

tests: quantize the NVFP4 consumer fixture with row-major block scales (checkpoint layout)

01554

committed

b2b5ec1

[Feature] expert cache: byte accounting at startup and periodic row-provider stats

01554

committed

67b45f8

[Bugfix] device_loading_context: restore only the parameters it moved

01554

committed

e6f58d6

[Bugfix] device_loading_context: keep the name-based restore; exempt only cache-marked slot buffers

01554

committed

eea1e76

[Bugfix] device_loading_context: scan parameters in a helper so the generator frame holds no Parameter across the yield

01554

committed

d25ae3c

[Feature] expert cache under breakable CUDA graphs

01554

committed

3e7dc47

[Bugfix] moe_runner: resolve the legacy layer placeholder once per op in the break-point wrapper

01554

committed

ee74bb4

tests: legacy-placeholder regression uses a MoERunnerInterface fake and checks thunk replay keeps the fixed name

01554

committed

857e881

tests: build the fake MoE runner by stubbing every abstract member of MoERunnerInterface

01554

committed

9e7c9ee

tests: stub the CUDA-only weak_ref_tensor in the CPU break-point dispatch tests

01554

committed

0b6fe94

[Bugfix] RowCacheWeightProvider: track the owner stream instead of rejecting other streams

01554

committed

69699c5

[Bugfix] RowCacheWeightProvider: serialize resident-map writes with the slot copies across owner changes

01554

committed

b5ac54e

docs: RowCacheWeightProvider consumer contract; periodic stats after the call's counts; host_bytes scope

01554

committed

60ba5b9

Navigation Menu

[Draft][MoE][Offload] Row-level expert weight cache provider (moe_expert_cache_provider=row) - #47

[Draft][MoE][Offload] Row-level expert weight cache provider (moe_expert_cache_provider=row)#47
01554 wants to merge 32 commits into
lab/upstream-37190-basefrom
lab/upstream-expert-row-provider

Commits on Sep 9, 2026

[MoE][Offload] moe_expert_cache_provider knob (cached|row, default cached) and RowCacheWeightProvider skeleton with the CachedWeightProvider surface

[MoE][Offload] RowCacheWeightProvider: per-token capacity check in plan_chunks, validate explicit ids before mutation, explicit device kwarg (review)

[MoE][Offload] RowCacheWeightProvider: own the host source (clone, never alias), owner-stream check on invalidate, accurate copy wording (review)

[MoE][Offload] RowCacheWeightProvider.prepare: validate before any state change (owner record and counters after validation) (review)

[MoE][Offload] RowCacheWeightProvider: staged copies on a provider-owned copy stream with release/ready events, forward-generation map; CPU event-order mock and CUDA reuse/race tests

[MoE][Offload] RowCacheWeightProvider: last_copies on both paths, invalidate ordering comment matches the contract, CUDA race test proves reader overlap over all six slot tensors, invalidate->prepa…

[MoE][Offload] provider tests: staging mock expects copy-only wait on invalidate; CUDA tests state allocator behaviour accurately and prove reader overlap for invalidate->prepare too (review)

tests: captured-graph reader fixture for RowCacheWeightProvider CUDA race tests

tests: feedback-free GPU delay and distinct-sum precheck in race fixture

[Feature] CachedWeightProvider: optional per-expert second-level scales

tests: report prepare host time vs reader GPU time when the race fixture is inconclusive

[Bugfix] RowCacheWeightProvider: upload expert maps without a host sync

[Feature] RoutedExperts: select the expert cache provider, carry scale_2 through slots

[Feature] ModelOptNvFp4FusedMoE: opt into the expert cache on the Marlin backend

tests: gate the NVFP4 consumer test on Marlin capability instead of skipping on any ValueError

[Feature] NVFP4 expert cache: load expert tensors into pinned host memory, repack Marlin in chunks

review: release each repacked chunk before the next, validate expert_chunk, alias contract wording, host/pinned checks in tests

tests: CPU contract check that a non-positive expert_chunk is rejected before side effects

tests: gate the NVFP4 consumer test on is_fp4_marlin_supported (the FP4 Marlin path's own predicate)

tests: quantize the NVFP4 consumer fixture with row-major block scales (checkpoint layout)

[Feature] expert cache: byte accounting at startup and periodic row-provider stats

[Bugfix] device_loading_context: restore only the parameters it moved

[Bugfix] device_loading_context: keep the name-based restore; exempt only cache-marked slot buffers

[Bugfix] device_loading_context: scan parameters in a helper so the generator frame holds no Parameter across the yield

[Feature] expert cache under breakable CUDA graphs

[Bugfix] moe_runner: resolve the legacy layer placeholder once per op in the break-point wrapper

tests: legacy-placeholder regression uses a MoERunnerInterface fake and checks thunk replay keeps the fixed name

tests: build the fake MoE runner by stubbing every abstract member of MoERunnerInterface

tests: stub the CUDA-only weak_ref_tensor in the CPU break-point dispatch tests

[Bugfix] RowCacheWeightProvider: track the owner stream instead of rejecting other streams

[Bugfix] RowCacheWeightProvider: serialize resident-map writes with the slot copies across owner changes

docs: RowCacheWeightProvider consumer contract; periodic stats after the call's counts; host_bytes scope