Skip to content

Close RWKV-7 production serving gaps - #12

Merged
btlqql merged 2 commits into
mainfrom
btlqql/rwkv7-production-gaps
Jul 31, 2026
Merged

btlqql merged 2 commits into
mainfrom
btlqql/rwkv7-production-gaps

Conversation

@btlqql

@btlqql btlqql commented Jul 31, 2026

Copy link
Copy Markdown
Owner

Summary

  • Add block-aware recurrent-state prefix caching for pure and hybrid RWKV-7 models.
  • Add standard vLLM runtime LoRA support with a reproducible two-adapter
    load/switch/remove benchmark.
  • Add selective online INT8 targeting, compact gate reporting, and positive
    plus negative exact-card evidence.
  • Mark verified LoRA support in the model table and document precise capability
    boundaries.

Why

The initial native RWKV-7 integration served and quantized the model, but
production follow-up still needed non-vacuous prefix reuse, standard runtime
adapters, and fail-closed quantization claims. This patch reuses vLLM's existing
Mamba cache, LoRA, quantization, and V1 engine infrastructure without forking
scheduler or cache-engine subsystems.

Validation

  • RTX 4080: tests/model_executor/test_rwkv7.py - 32 passed, 2 skipped.
  • RTX 4080: model and attention registry selection - 4 passed.
  • V100 environment, GPU hidden because both shared cards were occupied: 5 new
    interface/configuration tests passed.
  • Pure RWKV-7 prefix cache: bsz=4, prompt=512, 496 cached tokens per request;
    cold/hit and Torch/auto output exact.
  • Hybrid RWKV-7 prefix cache: bsz=2, prompt=2064, 2048 cached tokens per request;
    cold/hit and Torch/auto output exact.
  • Pure and hybrid runtime LoRA: rank-8 two-adapter switching, removal, and exact
    base restoration passed.
  • Real 1.5B checkpoint, RTX 4080, bsz=1: lm_head-only online INT8 reduced
    model memory from 2.85 to 2.72 GiB, reached 1.0224x fp16 throughput, and
    retained 1.0 token agreement.
  • The same selective INT8 row at bsz=4 failed speed and token gates; it is
    recorded as a negative result and is not enabled by default.
  • Ruff format/check, markdownlint, JSONL parsing, Python compilation, and
    git diff --check passed.

Scope and upstream relationship

This is a downstream follow-up based directly on synchronized official vLLM
history. It does not open a competing official-vLLM PR while
vllm-project/vllm#50077 remains open. Full W8/W4 rows remain
memory/compatibility evidence unless all card-local speed and token gates pass.

btlqql added 2 commits July 31, 2026 18:17
Define fail-closed prefix-cache, runtime LoRA, quantization, and exact-card validation gates.

Assisted-by: OpenAI Codex
Add block-aware pure and hybrid prefix caching, runtime LoRA support, selective quantization gates, and exact-card evidence.

Assisted-by: OpenAI Codex
@btlqql btlqql added the ready Run required pull request checks label Jul 31, 2026
@btlqql
btlqql marked this pull request as ready for review July 31, 2026 14:35
@btlqql
btlqql merged commit 302d1d5 into main Jul 31, 2026
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready Run required pull request checks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant