[lukealonso]

Research-only Kimi memory and decode experiments: bounded vision/projector execution, activation/residual storage reuse, auxiliary projection sharding, optional aligned decode GEMMs, dense-weight L2 prefetch and split KDA projection overlap.

Depends on #924 for shared projection and collective interfaces, and #857/#842 for correctness prerequisites. Integrated into dev/karmic-kraken; optional-path GPU and performance qualification remains incomplete and was accepted by the maintainer.

Validation: 168 CPU tests passed, 9 GPU cases deselected across test_eagle3.py, test_mm_preprocess.py, test_l2_prefetch.py, test_exl3.py, test_sequence_parallel.py, the MLA output-gate dispatch cases, and full/sliding-window cache capability merging. Relevant pre-commit hooks, including mypy, pass. Exact checkpoint-dimension validation from #857/#842 is included. The complete stack preserves the original optional implementations plus review fixes, with no circular dependency between these drafts.

GPU validation and performance qualification are pending. Before integration, measure each optional feature independently against the same checkpoint, topology, shapes, graph mode and baseline; require numerical correctness, changed-input graph replay, aliasing/compile compatibility and allocation stability. Storage-donation paths still require real torch.compile and graph coverage. No speedup or serving qualification is claimed.

The existing #592 targets dev/infernal-invocation and its older hook/API layout. This preserves the prepared Karmic implementation and memory changes extracted from #857 rather than adding another patch to that lineage. L2 prefetch and projection overlap remain opt-in.

AI assistance was used for separation, review fixes and validation.

[coderabbitai]

Warning

Review limit reached

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 22a9a302-4001-4ab0-938d-e5165cbcce34

📥 Commits

Reviewing files that changed from the base of the PR and between d8cdca5 and 8751db7.

📒 Files selected for processing (52)

• tests/distributed/test_b12x_pcie_all_reduce.py
• tests/distributed/test_dcp_direct_a2a_lse_reduce.py
• tests/model_executor/layers/test_shared_experts_stream.py
• tests/model_executor/test_b12x_warmup.py
• tests/models/kimi_k3/test_attn_res.py
• tests/models/kimi_k3/test_eagle3.py
• tests/models/kimi_k3/test_kda.py
• tests/models/kimi_k3/test_l2_prefetch.py
• tests/models/kimi_k3/test_latent_moe_tail.py
• tests/models/kimi_k3/test_mla_prefill_context.py
• tests/models/kimi_k3/test_mm_preprocess.py
• tests/models/kimi_k3/test_sequence_parallel.py
• tests/models/kimi_k3/test_vision_projector.py
• tests/models/test_vision.py
• tests/quantization/test_exl3.py
• tests/v1/attention/test_mla_context_chunks.py
• tests/v1/attention/test_mla_dcp_prefetch.py
• tests/v1/attention/test_mla_noncausal.py
• tests/v1/attention/test_mla_prefill_quant_output.py
• tests/v1/executor/test_b12x_startup.py
• tests/v1/worker/test_cp_utils.py
• vllm/distributed/communication_op.py
• vllm/distributed/device_communicators/b12x_dcp.py
• vllm/distributed/device_communicators/b12x_pcie_all_reduce.py
• vllm/distributed/device_communicators/cuda_communicator.py
• vllm/distributed/parallel_state.py
• vllm/envs.py
• vllm/model_executor/layers/attention/attention.py
• vllm/model_executor/layers/attention/mla_attention.py
• vllm/model_executor/layers/fused_moe/runner/moe_runner.py
• vllm/model_executor/layers/linear.py
• vllm/model_executor/models/kimi_k25_vit.py
• vllm/model_executor/models/vision.py
• vllm/model_executor/warmup/b12x_prepare.py
• vllm/models/glm5next/nvidia/l2_prefetch.py
• vllm/models/kimi_k3/common/mm_preprocess.py
• vllm/models/kimi_k3/nvidia/kda.py
• vllm/models/kimi_k3/nvidia/l2_prefetch.py
• vllm/models/kimi_k3/nvidia/latent_moe_runner.py
• vllm/models/kimi_k3/nvidia/low_latency_gemm.py
• vllm/models/kimi_k3/nvidia/mla.py
• vllm/models/kimi_k3/nvidia/model.py
• vllm/models/kimi_k3/nvidia/ops/attn_res.py
• vllm/models/kimi_k3/nvidia/tp_projection.py
• vllm/v1/attention/backend.py
• vllm/v1/attention/backends/mla/prefill/b12x.py
• vllm/v1/attention/backends/mla/prefill/registry.py
• vllm/v1/attention/backends/mla/rocm_aiter_mla.py
• vllm/v1/attention/ops/dcp.py
• vllm/v1/attention/ops/dcp_prefetch.py
• vllm/v1/attention/selector.py
• vllm/v1/kv_cache_interface.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}