Enable adaptive verify window for ngram/draft speculation

[michaelneale]

What this fixes

N-gram speculation on a split runtime could be slower than no speculation
despite high per-token acceptance. On a 2-node WAN split (Sydney↔Melbourne,
~20ms RTT, MiniMax-M2.7) N-gram ON measured ~40% slower than OFF, with an 0.89
token accept rate but 231 early-reject windows each paying a 2-round-trip
recovery.

Root cause

The adaptive verify window was never enabled on the split-serving path:
to_embedded_openai_args hardcoded adaptive_speculative_window = false. With
a fixed window the window never shrank after an early reject
(window_shrinks = 0 in telemetry), so a sustained reject storm kept proposing
at full depth and kept paying the full recovery cost per committed token. Over
WAN those extra round-trips dominate.

Change

Enable the adaptive window whenever speculation actually proposes a window
(ngram or draft mode). The existing shrink_adaptive_window logic then
narrows the window toward the observed accept depth after an early reject,
cutting recovery frequency. Native-MTP-only decode is unaffected.

Adds a regression test asserting N-gram speculation turns the adaptive window
on.

Validation

• cargo test -p mesh-llm-host-runtime --lib (new test + existing pass)
• cargo clippy -p mesh-llm-host-runtime --lib -- -D warnings
• cargo fmt --all --check
• Live A/B on a 2-node split with metrics-server pending (will confirm
  window_shrinks > 0 and throughput recovery).

Summary by CodeRabbit

• New Features

  • Enabled adaptive speculative verification windows for N-gram and draft speculation modes, allowing verification behavior to respond dynamically.

• Bug Fixes

  • Corrected embedded inference configuration so adaptive verification is enabled when supported speculation modes are selected.

• Tests

  • Added regression coverage to verify adaptive windows are enabled for N-gram speculation.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c7fe57a8-e8ff-4ca5-9847-f300504f1bc6

📥 Commits

Reviewing files that changed from the base of the PR and between 2c5dacf and b683fa8.

📒 Files selected for processing (2)

• crates/mesh-llm-host-runtime/src/inference/skippy/resolver/tests.rs
• crates/mesh-llm-host-runtime/src/inference/skippy/resolver/translation.rs

---

📝 Walkthrough

Walkthrough

Changes

Speculative window behavior

Layer / File(s)	Summary
Adaptive window mapping and regression test crates/mesh-llm-host-runtime/src/inference/skippy/resolver/translation.rs, crates/mesh-llm-host-runtime/src/inference/skippy/resolver/tests.rs	Embedded OpenAI arguments now enable adaptive speculative verification for ngram and draft modes, with a regression test covering N-gram speculation.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested labels: experimental

Suggested reviewers: i386

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: enabling the adaptive verification window for ngram and draft speculation.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/ngram-adaptive-window

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[michaelneale]

Live A/B validation (2-node WAN split)

Validated on a faithful 2-node split — M5 (Sydney, coordinator) + RTX 6000 Ada
48GB (Melbourne worker), ~25ms RTT, meshllm/MiniMax-M2.7-UD-Q4_K_XL-layers,
agentic coding prompt OSL 512, acceptance captured via metrics-server on the
coordinator (stage-0).

Condition	tok/s	window behaviour	early-reject windows	recovery ms
ngram OFF (baseline)	~17.0	—	—	—
ngram ON, before this PR	~15.0	fixed 12, shrinks=0	231	18,971
ngram ON, with this PR	~15.8	adaptive: shrinks=141, grows=138, final=9, min=3	141	13,298

The fix does what it claims: the adaptive window now actually engages
(window_shrinks 0 → 141), early-reject windows drop 231 → 141, and recovery
cost drops 19.0s → 13.3s, recovering throughput 15.0 → 15.8 tok/s.

Honest scope

This is a correctness fix (the adaptive window should have been on for the
split path and never was), not a latency win. Even with it, ngram-cache
speculation on a reasoning MoE over a 25ms WAN link is still below ngram-OFF
(~15.8 vs ~17.0).

Why: tolerating latency requires committing many tokens per round-trip.
ngram-cache predicts shallowly here (accept ~0.65, window converges to ~9), and
each early-reject still costs 2 serial WAN round-trips. Shrinking reduces waste
but also reduces committed-tokens-per-hop — it caps the downside, it can't reach
a net win. The real WAN latency win needs native MTP (high acceptance at a large
window), tracked separately.

Recommendation: merge as a general correctness/robustness improvement (the
window should adapt), but it should not be read as making ngram a WAN speedup.
Follow-ups that would actually help recovery cost regardless of acceptance:
fuse restore+repair into one round-trip; skip the restore round-trip for
single-token repair.

[michaelneale]

Caveat: this makes WAN splits slightly worse — do not merge if WAN is the priority

We care about WAN split serving, and the live A/B shows the adaptive window is a
net negative there for N-gram on a reasoning MoE:

• ngram OFF: ~17.0 tok/s
• ngram ON + this fix: ~15.8 tok/s (still below OFF)

The fix is mechanically correct (the window now adapts instead of being stuck),
but on a high-latency link it optimizes the wrong thing: shrinking commits
fewer tokens per round-trip, when latency tolerance needs more tokens per
round-trip. It caps the downside of a bad proposer but cannot make N-gram a WAN
win.

So: treat this as a latent correctness fix, not a WAN improvement. If anything,
enabling N-gram speculation at all on a WAN split should be discouraged until we
have native MTP (high acceptance at a large window) or cheaper recovery
(fuse restore+repair into one round-trip). Recommend not enabling N-gram by
default on WAN-topology splits.

[michaelneale]

Follow-up: the real WAN cost is a redundant round-trip, not KV rollback

Traced the misprediction path in code. The KV rollback itself is cheap — a local
suffix trim (restore_checkpoint → trim speculative suffix back to the checkpoint
position), and the checkpoint is piggybacked on the VerifyWindow message (no extra
hop). So the recovery cost is not rollback work.

It's the synchronization: on a reject the coordinator sends RestoreSession
and blocks on its ACK across the WAN, then sends the repair decode and blocks
again. A rejected window = verify RT + restore RT + repair RT = 3 serialized WAN
traversals to commit what plain decode does in 1. At ~25ms RTT that's where the
~13–19s aggregate recovery went — almost all wire-wait, barely any compute.

Implication for the next recovery fix (higher value than window shrinking):
fuse restore+repair into one message/round-trip. Since the downstream restore
is a trivial local trim, there's no reason to wait for a separate restore-ACK
before the repair — send "trim KV to checkpoint X, then decode from token Y" and
block once. That halves the per-reject penalty (3 RT → 2 RT) on every
misprediction, regardless of acceptance or model. Adaptive shrinking (this PR)
reduces the count of round-trips; fusing reduces the cost of each, and works
even when acceptance is low — i.e. exactly the WAN case.

[ndizazzo]

Superseded by #1026