feat: Tier 1 optimizations — streaming tool fix, frequency-aware cache, block reuse

[raullenchai]

Summary

Implements the top 3 Tier 1 optimizations for MiniMax-M2.5 + OpenClaw on M3 Ultra, building on the merged Tier 0 work (GC control, pinned prefix cache, schema hardening).

1. Fix Streaming Tool Truncation

• Problem: The fallback in stream_chat_completion() only checked for "<tool_call>" in accumulated text. MiniMax uses <minimax:tool_call>, Llama uses <function=, and Mistral uses [TOOL_CALLS] — so the fallback never triggered for these parsers, and incomplete tool calls were silently lost.
• Fix: Added has_pending_tool_call() to ToolParser base class with parser-specific overrides. Server fallback now uses tool_parser.has_pending_tool_call() instead of hardcoded string check.
• Bonus: Hardened MiniMax extract_tool_calls() with partial regex patterns (INVOKE_PARTIAL, PARAM_PARTIAL) to extract tool calls even when </parameter> or </invoke> closing tags are missing due to truncated streams.

2. Frequency-Aware Cache Eviction (LRU-LFU Hybrid)

• Problem: Pure LRU eviction doesn't account for block reuse frequency. A system prompt block accessed 100 times gets evicted just as easily as a one-shot block if it hasn't been touched recently.
• Fix: Added access_count field to CacheBlock, incremented on every touch(). New popleft_lfu() method examines a window of 8 LRU candidates and picks the one with the lowest access_count. Used in allocate_block() and evict_lru_blocks() when prefix caching is enabled.
• Result: High-frequency blocks (system prompts, common prefixes) survive much longer under memory pressure.

3. KV Block Reuse Ordering

• Problem: When multiple blocks share the same hash (e.g., in hybrid models), get_block() returned an arbitrary block via next(iter(blocks.values())).
• Fix: Added get_best_block() to BlockHashToBlockMap that returns the block with the highest access_count. Used in get_computed_blocks() for better cache locality.

Files Changed

File	Changes
vllm_mlx/server.py	Use tool_parser.has_pending_tool_call() in streaming fallback
vllm_mlx/tool_parsers/abstract_tool_parser.py	Add has_pending_tool_call() base method
vllm_mlx/tool_parsers/minimax_tool_parser.py	Override has_pending_tool_call(), add partial regex patterns for truncated input
vllm_mlx/tool_parsers/llama_tool_parser.py	Override has_pending_tool_call() for <function=
vllm_mlx/tool_parsers/mistral_tool_parser.py	Override has_pending_tool_call() for [TOOL_CALLS]
vllm_mlx/paged_cache.py	access_count on CacheBlock, popleft_lfu(), get_best_block(), frequency-aware eviction
README.md	Update optimization roadmap — Tier 0 marked merged, Tier 1 top 3 marked merged

Test plan

• pytest tests/test_paged_cache.py tests/test_prefix_cache.py tests/test_tool_parsers.py — all 136 tests pass
• Full test suite: 902 passed (20 pre-existing async infra failures, unrelated)
• Manual: Start server with --tool-call-parser minimax --enable-auto-tool-choice, send streaming tool call requests, verify fallback triggers on truncated streams
• Benchmark: Run benchmark_minmax.py to compare prefix cache Turn4/Turn1 ratio and tool call reliability vs Tier 0 baseline

🤖 Generated with Claude Code