docs(routing): cover pressure-aware selection and the hash index in cache-aware routing

[slin1237]

cache_aware changed substantially in v1.10 and v1.11, and the page still described the old algorithm: a fleet-wide "imbalanced" switch, and misses routed to "most cache capacity". It also claimed the tree "mirrors backend KV cache 100%". The changes it missed:

• Pressure-aware spill gate and knobs (feat(router): pressure-aware tuning knobs for cache-aware selection smg#2117, #2161). Balance is now a per-request spill gate, alongside KV-usage triggers, overlap decay, and selection temperature. Misses and spills go to the lowest expected wait.
• Hash index mode (#2213). --cache-index hash with --cache-boundaries and --cache-ttl-secs.
• Shared per-model tree budget (#2137).
• Cache partitions (#2435), plus the new tree and decision metrics (#2134, #2515).
• KV-event recovery. SGLang event replay after gaps (#2673, #2672), and bounds on the event indexer (#2095).

Changes

• concepts/routing/cache-aware.md (rewritten from the v1.11 code)
  • Index modes: one table for KV-event, token tree, string tree, and hash. It says which requests carry token IDs, and how body buffering interacts with each mode.
  • Approximate trees: tree size and eviction under the shared per-model budget, and purging when a worker is removed.
  • New sections: Hash Index Mode and Cache Partitions (cache_salt, extra_key, LoRA; HTTP only).
  • Routing Algorithm:
    • A 7-step decision flow: eligibility, KV-pressure triggers, per-index candidates, decay and temperature, spill gate, miss, record.
    • A "Changed in v1.10.0" note.
    • Expected-wait selection and a tuning-knob table with aliases.
    • Which backends report which load signals.
    • A warning about label-created per-model policies, which use built-in defaults.
  • Block size: its two roles.
  • Monitoring: only real metrics now, with type and labels, plus debug logs and a signals-to-watch table.
  • Mesh sync: rewritten. It is best effort, event-driven and hash indexes are not synced, and ZMQ workers are excluded.
  • Removed: an unattributed benchmark table and TTFT percentages, invented memory and alert numbers, and two diagrams of the old algorithm.
• getting-started/kv-events-cache-aware.md
  • Corrections:
    • The event index is chosen per model.
    • --cache-index hash bypasses the event index.
    • --cache-threshold defaults to 0.3 and does not apply to event-driven routing.
    • The real debug targets and log lines.
    • TokenSpeed uses --prefix-granularity.
    • The nonexistent "worker config file" is removed.
  • Added:
    • The SGLang replay_endpoint.
    • "Recovery after gaps" and "Bounding the index" sections.
    • A rewritten tuning table, monitoring, and updated caveats.
    • A two-worker walkthrough, so the repeat-prompt check means something.

Verification

• Checked against smg v1.11.0 (3be823a7):
  • policies/cache_aware.rs, policies/cache_namespace.rs, policies/mod.rs (built-in defaults verified)
  • crates/kv_index, worker/kv_event_monitor.rs, mesh/adapters/tree_sync.rs
  • grpc_servicer/smg_grpc_servicer/{sglang,vllm,tokenspeed}/kv_events.py, main.rs, observability/metrics.rs
• Engine flags were checked against local SGLang, vLLM, and TokenSpeed checkouts.
• pnpm lint, pnpm check (0 errors), and pnpm build pass.
• The link check reports only request-streaming.md (docs: document request body streaming and upstream HTTP/2 #47) and overload-protection.md (docs(reliability): document worker overload protection and load monitoring #37), which are added by sibling PRs.
• Both pages render with status 200 under wrangler pages dev.

This is one of a series of docs PRs that bring the site up to smg v1.11.0, one per area.

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

Next included review available in 23 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available. Your 65 included PR review attempts over the past 7 days set your current allowance at 4 reviews per hour.

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 3b470f24-0d9c-4da8-8220-e60e08152ec3

📥 Commits

Reviewing files that changed from the base of the PR and between 7362b40 and 03b857c.

📒 Files selected for processing (2)

• src/lib/content/concepts/routing/cache-aware.md
• src/lib/content/getting-started/kv-events-cache-aware.md

---

_{Comment @coderabbitai help to get the list of available commands.}