[Core] Extensible (growable) KV cache by njhill · Pull Request #50779 · vllm-project/vllm

njhill · 2026-08-02T22:36:23Z

**Stacked on #51718. Only the commits above that PR's head belong to this one.
This is a draft — it cannot merge until #51718 lands, after which the base will be retargeted to main.

Summary

Opt-in growable KV cache (--enable-extensible-kv-cache) built on the standardized KV cache layout from #51718.

This is the productization of the demonstration #47363 from @zhuohan123.

vLLM sizes the KV cache from a profiling run, then allocates it up front. Any memory the engine consumes after profiling — CUDA graph pools, the spec-decode logits all-gather, workspace growth — has to be guessed at via
headroom margins, and a wrong guess is either an OOM or wasted HBM.

This change reserves virtual address space for the KV cache capacity but commits physical pages incrementally (CUDA cuMem* / ROCm hipMem*):

Allocate the VA reservation, commit one block per layout segment.
Run warmup and CUDA graph capture.
Measure actual post-warmup free memory and re-size.
extend_kv_cache() maps the remaining pages under the same base pointer.

Because each block keeps a fixed offset within its layout segment and the base pointer never moves, captured graphs and layer views stay valid across the growth — no re-view, no re-capture.

What #51718 buys this

The layout standardization removes most of what made the earlier version of this work invasive. Buffer segmentation is now a pure function of KVCacheLayout.stride_order (new num_outer_segments() helper) instead of per-backend shape/stride/block-dim probing; connector registration views are a plain narrow(0) of the logical [B, H, N, C] view; and the packed / block_stride special cases disappear entirely. The integration is ~30%
smaller than the equivalent against main.

Views are authoritative, not `storage.nbytes()`

The one contract change worth reviewer attention. Under an extensible cache the untyped storage spans the reserved capacity, of which only each view's per-segment block prefix is physically backed. Any code deriving block geometry or registration extents from untyped_storage().nbytes() will compute a size that includes unbacked pages.

Updated accordingly, with the contract documented on KVConnectorBase_V1.register_kv_caches:

NixlBaseConnectorWorker.register_kv_caches — collects per-region committed spans and registers those; falls back to the single per-allocation registration when the storage is fully backed. Block-major packing is
detected from view strides rather than nbytes().
get_kv_cache_block_regions (v1/worker/utils.py) — same, which also covers the CPU-offload connector.

KV-transfer init is deferred until after extend_kv_cache, so connectors only ever register committed memory. On the NIXL path, extend also defragments (release + single-chunk recommit per segment): UCX cannot transfer a VMM region spanning multiple cuMemCreate handles.

Relationship to existing work

[Core] Demo implementation of extensible kv cache memory #47363 (@zhuohan123, "Demo implementation of extensible kv cache memory) — this is the productionization of that demo, not a competing implementation. It adds the V1 + V2 runner integration, the measured post-warmup sizing pass, VMM probe/fallback, sleep-mode support, KV connector support, and a ROCm backend. Happy to fold this into [Core] Demo implementation of extensible kv cache memory #47363 instead if that's preferred.
[6/N][KV-Cache Layout Refactor] Standardize KV cache layout #44458 — strictly a dependency; no overlap in changed behavior.
Searched open PRs for extensible/growable/VMM KV cache work; nothing else overlaps.

Contributors: @zhuohan123 (original demo), @ZJY0516 (packed-storage bounding, sleep gate).

Testing

Unless noted, run against this branch at its current head.

Unit — pytest tests/v1/worker/test_extensible_kv_cache.py tests/utils_/test_extensible_tensor.py tests/v1/engine/test_engine_args.py -v → 29/29 pass (GB200, aarch64).

e2e, 1×GB200, CUDA graphs on — outputs byte-identical to a non-extensible baseline for: standard generation, sleep(level=1)/wake, and --kv-cache-memory-bytes.

e2e, sizing under pressure — Qwen3.5-4B TP4, MTP num_speculative_tokens=100, chunk 256, gpu-memory-utilization=0.92 (a config that OOMs without this change): passes in both eager and cudagraph modes. Sizing 35,277→34,721 tokens (eager), 35,243→34,592 (cudagraph). Functional check 64/43 answers correct.

e2e, NIXL P/D disaggregation — Qwen3-0.6B, HND, block 128, gsm8k via P/D proxy:

config	gsm8k	expected
1P1D eager	0.422	~0.41
1P1D cudagraph	0.418	~0.41
P(TP1)→D(TP2)	0.415	~0.41
P(TP2)→D(TP2)	0.415	~0.41

Model evaluation — the feature is opt-in and does not change model outputs; the byte-identical-output comparison above and the gsm8k table are the evidence for that.

Known gaps

The V2 encoder-cache profiling reservation (last commit) is validated on the equivalent branch against main (Qwen2-VL-2B: 16,384-token budget reserved, 238,148 blocks committed) but has not been re-run on this branch — the cluster battery above has no multimodal configuration. Will re-run before marking ready.
No unit coverage for commit(defragment=True) or the connector sleep gate; both are exercised only by the e2e NIXL runs. Adding these before ready.
Multi-node NIXL (NIC RDMA path with the GDR + POSIX-FD allocation flags) is unvalidated; the repro harness is localhost-only.
The ROCm hipMem* backend is written but untested on AMD hardware.
Encoder-decoder mixed layouts raise rather than falling back.

AI assistance

AI assistance (Claude Code) was used for this change: for the port onto the #44458 layout model, the connector geometry audit, and drafting. I have reviewed every changed line, and ran the tests and cluster validation above
myself.

njhill · 2026-08-04T19:49:16Z

/ci run

github-actions · 2026-08-04T19:49:43Z

✅ Triggered Buildkite CI #82367 for commit a2bfba8a0ad2.

mergify · 2026-08-05T23:52:22Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @njhill.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

njhill · 2026-08-06T01:32:33Z

/ci run

github-actions · 2026-08-06T01:32:48Z

✅ Triggered Buildkite CI #82597 for commit d7f10f2407f9.

njhill · 2026-08-06T20:20:46Z

/ci run

mergify · 2026-08-06T20:51:19Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @njhill.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>

Assisted-by: OpenAI Codex Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>

Add a block-size-aware state-count helper and use it when constructing kernel block views. Keep num_states as the manager-block convenience property and preserve one recurrent state for Mamba. Assisted-by: OpenAI Codex Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>

njhill · 2026-08-19T06:01:27Z

/ci run

github-actions · 2026-08-19T06:01:45Z

✅ Triggered Buildkite CI #84556 for commit eaa39b4ae041.

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>

mergify · 2026-08-19T18:35:31Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @njhill.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

The V2 runner's decode-shaped autotune dummy run reaches trtllm_batch_decode_mla, which MRV1's warmup never exercised; its 21-tactic sweep faults on the dummy decode batch (zeroed block tables). Keep the op on its heuristic fallback -- exactly main's behavior -- until the sweep is made safe. Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ThBqMtrBGwXZnMYHky55on

…umbing Ported from the extensible-kv-cache branch (originally #47363): VmmDriver (CUDA cuMem* / ROCm hipMem* bindings with probe), ExtensibleTensor / ExtensibleKVCacheBuffers (grow-only per-segment prefix commits over a stable VA reservation), enable_extensible_kv_cache config/CLI plumbing, and the register_kv_caches views-are-authoritative contract note. Co-authored-by: Zhuohan Li <zhuohan123@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ho2gVXA5r7PhrM2sk8oJUn Signed-off-by: Nick Hill <nickhill123@gmail.com>

Rework of the extensible-kv-cache V2 integration on the #51718 single-allocation model, substantially simplified by it: - The shared allocate_and_reshape_kv_cache gains a reserve(size, num_segments) hook that stands in for the plain torch.zeros backing allocation; one VMM buffer backs the whole KV cache. - Buffer segmentation falls out of the KVCacheTensor stride model: num_segments = size / (num_blocks * block_stride), i.e. one segment per contiguous run of blocks (1 for block-outermost layouts, one per layer region under layer-compact layouts), replacing per-backend shape/stride/block-dim probing. - Connector registration views are a plain narrow(0) of the logical [B, H, N, C] per-layer views. Ported unchanged: measured post-warmup sizing, deferred KV-transfer init, sleep release/recommit, VMM probe/fallback, elastic-EP skip, and encoder-cache profiling reservation. Co-authored-by: Zhuohan Li <zhuohan123@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ho2gVXA5r7PhrM2sk8oJUn Signed-off-by: Nick Hill <nickhill123@gmail.com>

- NIXL register_kv_caches and get_kv_cache_block_regions derive all geometry from the registration views instead of untyped storage nbytes: with the extensible KV cache the storage spans the reserved virtual-address capacity, of which only each view's per-segment block prefix is physically backed. Fully-backed storages keep the single per-allocation registration. - Mooncake likewise registers the byte span the layer views cover, and requires the DMA-BUF registration path (WITH_NVIDIA_PEERMEM=0): nvidia-peermem cannot pin VMM-backed memory. - Extensible KV cache unit tests against the single-allocation surface (kv_cache_num_segments, narrow, shared allocation path). Co-authored-by: Zhuohan Li <zhuohan123@gmail.com> Co-authored-by: zjy0516 <riverclouds.zhu@qq.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ho2gVXA5r7PhrM2sk8oJUn Signed-off-by: Nick Hill <nickhill123@gmail.com>

The V2 encoder profiling path held only one batch of dummy encoder outputs, while at runtime the encoder cache grows to encoder_cache_size tokens of embeddings. KV sizing therefore over-committed for multimodal models. Mirrors the reservation the V1 runner already makes in profile_run(). Co-authored-by: Zhuohan Li <zhuohan123@gmail.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>

Keep engine/core.py and the two model runners thin by relocating the extensible KV cache machinery to the utility modules that own each concern: - kv_cache_utils: configure_kv_cache (KV sizing + applying the result to the config, previously inline in EngineCore and now needed twice), use_extensible_kv_cache (validation + worker VMM probe), and post_warmup_available_memory. - kv_cache_interface: tensor_num_outer_segments, the per-tensor wrapper that was duplicated in both runners. - extensible_tensor: ExtensibleKVCacheBuilder, replacing the duplicated per-tensor VMM setup, plus ensure_blocks()/extend() carrying the mechanism docs. - gpu/kv_connector: get_deferred_kv_connector, the narrow-then-register step the V2 runner performs after extend_kv_cache. No functional change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>

Three ROCm-driven fixes; no behaviour change on CUDA. Granule floor. ROCm reports a 4 KiB minimum allocation granularity where CUDA reports 2 MiB (measured 4 KiB on ROCm 7.2.2, 7.13 and 7.14 alike), so the per-granule bookkeeping in _VirtualBuffer would hold 26.2M entries for a 100 GiB cache -- ~1.9 GB of Python objects and ~2.8 s just to build the set. Commit in units of at least 2 MiB instead, always a multiple of what the driver requires. On CUDA this is a no-op; a 1 TiB cache stays at 524k granules / 33 MB. Disable the feature on HIP runtimes before 7.12. hipMemSetAccess there rejects a validly mapped range once a reservation holds several differently-sized allocations, which is exactly how a growable KV cache is committed. Without this, an AMD user passing --enable-extensible-kv-cache got `GPU driver error 1: invalid argument` at engine init; now the engine warns and falls back to the standard allocator. This is ROCm's rocm-systems#2516, fixed by rocm-systems#2451 -- merged to develop on 2026-01-26 and present in release/therock-7.12 and later, but absent from every release/rocm-rel-7.x branch, so no shipped ROCm below the TheRock line has it. Gating on version rather than probing is deliberate: the defect is order-dependent (measured 19/20 failures at 8 mapped segments, but 0/20 at two), so a probe cheap enough to run at startup passes on affected runtimes and would give false assurance. Return early from commit() before releasing. With defragment=True and a request at or below the committed size, the old ordering released the mapping and then re-committed the smaller size, discarding live KV data. Not reachable today, but elastic EP re-runs warmup -- and hence ensure_blocks -- while the cache holds live data, so the failure mode would be silent corruption rather than a crash. Pinned by a test. Verified: 25 units on CUDA and ROCm. CUDA e2e unchanged across default, extensible, sleep/wake and deferred-connector paths, with byte-identical output and the gate inert. On ROCm 7.2.2 the engine now falls back gracefully instead of failing to start. On ROCm 7.13 the gate opens (runtime_version (7, 13), no reason reported) and the mapping sequence that fails 0/20 on 7.2.2 passes 20/20. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>

Builds on the extensible KV cache: since it commits physical pages only after warmup, it can size the cache from memory measured at that point rather than from a profiling estimate plus a hand-tuned safety fraction. That is more accurate, so make it the default where the driver supports it and stop reserving an unused fraction of the device for it. Utilization ----------- `gpu_memory_utilization` gains a default of 1.0 when the extensible KV cache is active: the unused fraction exists to cover what profiling mis-estimates, and measured sizing does not need it. Its margin is the explicit post-warmup buffer instead. It remains a fraction of the device's *total* memory in all cases, never rescaled against free memory. Rescaling would quietly break the documented case of two instances at 0.5 each: the second would see half the memory free and take a quarter of the device. Two consequences had to be handled: - The budget can now exceed memory that was free when the initial snapshot was taken, since this worker's own context and communication buffers are already resident. `post_warmup_available` therefore clamps to what is measured free, and `request_memory` tolerates the shortfall only for the extensible path, where exceeding free memory over-reserves address space rather than over-committing pages. - Falling back to profiling-based sizing has to undo the raised default, so `CacheConfig.disable_extensible_kv_cache` restores it in one place, used by both the config-level and driver-level fallbacks. Sharing a device ---------------- Refusing to start whenever another process holds GPU memory is too strict; a process fitting inside this instance's headroom is not necessarily a problem. But the default assumes exclusive use, so sharing should be deliberate: warn when utilization was set explicitly, raise when it was left to default, and keep raising when the requested fraction is not free at all. Detecting a resident process needs a process query, not a memory reading: by the time a worker reads free memory it has created its own context and comm buffers, so used memory is not attributable. Added `Platform.get_foreign_device_processes`, implemented for NVML, returning None where a platform cannot enumerate processes -- treated as unknown, so missing telemetry never refuses startup. ROCm inherits the None default; `amdsmi_get_gpu_process_list` is the equivalent. Processes sharing this process's session are excluded, since the engine process may itself hold a context on device 0 from capability queries. Post-warmup buffer ------------------ The buffer absorbs what the measurement cannot see: fragmentation, activation peaks at shapes warmup did not exercise, workspaces that grow at runtime. Those scale with the workload rather than the device, so it is a fraction of the profiled transient activation peak with an absolute floor, rather than the previous fixed 150 MiB -- which was ~0.08% of a 185 GiB device and 2.5% of a 6 GiB one. The fraction is not yet fitted; it wants a sweep against a long-running workload, which is the only thing that exercises what it covers. Explicit requests are mandatory ------------------------------- `enable_extensible_kv_cache` becomes tri-state so "requested" is distinguishable from "defaulted". An explicit request fails startup where it cannot be honored, rather than silently sizing the cache the less accurate way; left unset it falls back. Since the default is now on, the incompatible cases are detected at config time -- KV connectors on the V1 runner, and sleep mode alongside a KV connector -- so they disable the feature before workers spawn instead of failing during startup or at first sleep. The gate also excludes non-CUDA/ROCm platforms locally, without a collective RPC. Both tri-state fields follow the sentinel-default pattern CacheConfig already uses for `block_size`, and their derived `user_specified_*` flags are excluded from compute_hash so the compilation cache stays independent of how the values were supplied. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Gate ordering (CI build 82729) ------------------------------ The driver gate originally resolved in the engine after worker memory profiling, and `disable_extensible_kv_cache` only reset the engine's config copy. Workers therefore profiled against the extensible defaults (util=1.0) even on platforms taking the fallback path, and the estimate-based sizing consumed the whole device: every AMD CI job (ROCm 7.2 fails the VMM version gate) OOMed with ~100% of VRAM allocated to KV cache. Each worker now resolves the gate against its own driver before taking the memory snapshot, and non-CUDA/ROCm platforms are excluded at config time (before workers spawn) since CPU sizing reads the same utilization value. The post-warmup buffer also gains a proportional component (2% of the KV budget, still capped at 10%): some allocations happen only after warmup (first spec-decode rejection batch, multimodal frame staging), and the 150 MiB floor was not covering them on devices where the old default left ~8% idle. Signed-off-by: Nick Hill <nickhill123@gmail.com>

Sizing the KV cache from measured post-warmup memory removes the reason most of these caps existed. They were set to leave slack for memory the profiling estimate could not see -- NCCL buffers, CUDA graphs, quantized scratch space -- which measurement now accounts for directly. Worse, the caps have become actively harmful: measured sizing subtracts an explicit buffer where the estimate relied on the unused fraction as an implicit margin, and it counts CUDA graph memory that the estimate omits unless VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS is set, so a low cap can leave no room for a cache at all. Three CI jobs failed exactly that way. Removed from 84 test files (116 call sites plus constants, helper parameters and CLI arguments that existed only to carry the value). Retained where the value is load-bearing: - The argument itself is under test: test_gpu_utilization, test_kv_cache_utils, test_utils, test_config_utils, test_mp_reducer, test_nixl_rocm_gpu_mem_diag, test_lmcache_integration, and the torchrun examples, which deliberately vary it per rank. - Deliberate device sharing, where the default cannot apply because another engine is alive: test_logprobs (module-scope fixtures), test_custom_offline, test_spec_decode, test_batch_invariance, test_speculators_correctness, test_mtp_parallel_load, test_shutdown, test_elastic_ep, test_transcription_api_correctness, and test_rocm_compressed_tensors_w4a16 ("shared nodes"). - Synthetic configs in unit tests that never allocate, left untouched. Adjusted rather than removed: - test_full_cudagraph llm_pair: the fixture holds two engines (full and piecewise cudagraph modes) alive concurrently at 0.43 each; the sweep misread them as sequential. Reinstated (CI build 82729). - test_logprobs_mode / test_prompt_logprobs_mode: 0.05 of a 16 GiB device is 0.8 GiB, all of which goes to weights (0.52) and CUDA graphs (0.23). 0.1 keeps the sharing intact while leaving the cache room to exist. The other shared-device values are an order of magnitude above the level at which this bites, so they stand. - test_encoder_instance_zero_kv_cache: 0.7 left the consumer 2.12 GiB against the 2.0 GiB minimum for max_model_len. ec_producer's 0.01 stays, being what forces the zero-block assertion. - test_compile_correctness[test_setting3]: 0.98 left 0.52 GiB against a 0.66 GiB minimum; removing the cap raises the budget, the default now being 1.0. conftest's ROCm memory-settle helper read CacheConfig.gpu_memory_utilization as a class attribute, which is now the unresolved sentinel, so `1.0 - None` raised on every ROCm vllm_runner test that did not pass the argument. It now takes the concrete default. Verified: 1540 tests collect across the changed files (the six that do not are missing optional deps, datasets and lm_eval, unrelated to this change). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ThBqMtrBGwXZnMYHky55on Signed-off-by: Nick Hill <nickhill123@gmail.com>

Measured post-warmup KV sizing only accounts for memory that warmup actually allocates: torch's allocator retains warmup's high-water segments, and the measurement reads free memory after warmup. The multimodal encoder and the encoder cache budget were exercised only inside memory_profiling, which drops those allocations with empty_cache() on exit; warmup itself was decoder-only, so the post-warmup measurement credited that memory to the KV cache and the first runtime encoder batch OOMed inside the vision tower. Re-run the worst-case encoder pass in warmup_kernels with the other scheduler-realistic steps. It runs unconditionally since it also JIT-warms the vision tower's kernels (removing the first-vision-request spike on any sizing path). On small GPUs the encoder worst case may not fit alongside everything else warmup keeps resident (it fit during profiling, which ran it nearly alone). Rather than leaving a latent runtime OOM, halve the per-step encoder token budget until the batch fits (floored at one worst-case item) and hand the reduced budget to the scheduler via CompilationTimes, so it never batches more encoder tokens than warmup proved to fit. The extensible KV cache is V2-only, so the V1 runner needs no equivalent. Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ThBqMtrBGwXZnMYHky55on

njhill · 2026-08-19T20:33:55Z

/ci run

github-actions · 2026-08-19T20:34:14Z

✅ Triggered Buildkite CI #84683 for commit 362af7697617.

mergify · 2026-08-19T21:47:16Z

Documentation preview: https://vllm--50779.org.readthedocs.build/en/50779/

njhill · 2026-09-11T17:11:30Z

Replaced by #56492.

mergify Bot added frontend mrv2 Model Runner V2 specific kv-connector labels Aug 2, 2026

njhill force-pushed the njhill/extensible-kv-cache branch from bbd90e8 to 34382e3 Compare August 2, 2026 23:53

mergify Bot added the nvidia label Aug 4, 2026

github-project-automation Bot added this to NVIDIA Aug 4, 2026

njhill force-pushed the njhill/extensible-kv-cache branch from ab371fd to aaee56d Compare August 4, 2026 19:05

mergify Bot added the cpu Related to CPU backends label Aug 4, 2026

njhill force-pushed the njhill/extensible-kv-cache branch 2 times, most recently from fab567e to a2bfba8 Compare August 4, 2026 19:41

mergify Bot added the needs-rebase label Aug 5, 2026

njhill force-pushed the njhill/extensible-kv-cache branch from a2bfba8 to d7f10f2 Compare August 6, 2026 01:31

mergify Bot added llama Related to Llama models multi-modality Related to multi-modality (#4194) qwen Related to Qwen models gpt-oss Related to GPT-OSS models labels Aug 6, 2026

github-project-automation Bot added this to gpt-oss Issues & Enhancements Aug 6, 2026

github-project-automation Bot moved this to To Triage in gpt-oss Issues & Enhancements Aug 6, 2026

LucasWilkinson force-pushed the lwilkinson/kv-layout/core-standardize branch from d458dc6 to 040cb42 Compare August 6, 2026 02:06

njhill force-pushed the njhill/extensible-kv-cache branch from d7f10f2 to 4961f67 Compare August 6, 2026 17:58

mergify Bot removed the needs-rebase label Aug 6, 2026

njhill force-pushed the njhill/extensible-kv-cache branch from 4961f67 to 5516aaf Compare August 6, 2026 20:16

LucasWilkinson force-pushed the lwilkinson/kv-layout/core-standardize branch from b6fcd8e to 296e5f6 Compare August 6, 2026 20:49

LucasWilkinson added 8 commits August 19, 2026 04:18

ci fixes

266695a

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>

cleanup

39fb44e

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>

cleanups

3dd85d0

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>

Simplify KV cache layout view construction

cb6df74

Assisted-by: OpenAI Codex Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>

fixes

226b77a

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>

fix precommit

6497030

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>

simplify

6cba887

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>

get rid of global kv-cache layout variable

4564f49

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>

njhill and others added 11 commits August 19, 2026 12:48

temp: enable MRV2 by default for non-pooling models

362af76

njhill mentioned this pull request Aug 21, 2026

[Model Runner V2] Reserve CUDA graph memory #53306

Merged

GirasoleY mentioned this pull request Sep 2, 2026

[Kernel][PCP] Publish replicated KV updates through PyTorch symmetric memory #52863

Open

njhill mentioned this pull request Sep 11, 2026

[Core] Extensible KV cache #56492

Draft

Uh oh!

Conversation

njhill commented Aug 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What #51718 buys this

Views are authoritative, not storage.nbytes()

Relationship to existing work

Testing

Known gaps

AI assistance

Uh oh!

njhill commented Aug 4, 2026

Uh oh!

github-actions Bot commented Aug 4, 2026

Uh oh!

mergify Bot commented Aug 5, 2026

Uh oh!

njhill commented Aug 6, 2026

Uh oh!

github-actions Bot commented Aug 6, 2026

Uh oh!

njhill commented Aug 6, 2026

Uh oh!

mergify Bot commented Aug 6, 2026

Uh oh!

njhill commented Aug 19, 2026

Uh oh!

github-actions Bot commented Aug 19, 2026

Uh oh!

mergify Bot commented Aug 19, 2026

Uh oh!

njhill commented Aug 19, 2026

Uh oh!

github-actions Bot commented Aug 19, 2026

Uh oh!

mergify Bot commented Aug 19, 2026

Uh oh!

njhill commented Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

njhill commented Aug 2, 2026 •

edited

Loading

Views are authoritative, not `storage.nbytes()`