Add optional simple fallback for history ngram proposer misses

[danielwinterw]

Stacked on #1037 (feat/suffix-ngram-proposer). Only the last commit is new.

What

Adds an opt-in fallback chain for the standalone N-gram proposers: when the
configured history proposer (cache or suffix) proposes nothing, retry the
same window with llama.cpp's simple proposer so there is always some draft in
flight.

New config key, default off:

[models.speculative]
ngram_proposer = "suffix"
ngram_fallback = "simple"

Why

A suffix miss currently means zero draft, which costs one token per network
round trip. On a WAN split that round trip dominates, so a mediocre draft
beats no draft. This raises the worst-case floor without touching the hit
path: a history-proposer hit is returned unchanged, and the fallback only
runs on a miss.

Expected impact: roughly neutral on LAN, possibly slightly negative (we
measured simple at 13.6 vs 15 tok/s baseline on freeform chat locally).
Estimated +10-20% on WAN from the raised floor. WAN numbers are estimates,
not measurements; the LAN numbers are mechanism microbenchmarks, not quality
results.

Design notes

• Threaded end to end the same way as ngram_proposer: string config surface
  and validation in mesh-llm-config, schema + control behavior, resolver into
  NgramProposalConfig.fallback_simple, dispatch in
  speculative/standalone.rs. The resolved-plan field is serde-defaulted so
  existing JSON plans keep parsing.
• The fallback uses a hardcoded 2-token minimum match instead of the primary
  proposer's bounds. Suffix runs at min 5, which simple would rarely satisfy;
  llama.cpp's simple proposer takes only a minimum, and 2 is its useful floor.
  Fine to make configurable later if anyone needs it.
• ngram_fallback = "simple" with a simple primary is rejected in validation
  since it can never do anything.
• Draft source telemetry reports "simple" when the fallback produced the
  draft, via the existing ConfiguredNgramProposal.source field, so
  acceptance can be split by source when measuring.

Tests

Co-located in standalone.rs: fallback fires on a history-proposer miss,
does not replace a hit, stays off by default, and a fallback miss still
reports the primary source. Plan validation test for the simple-primary
rejection. Docs updated in USAGE.md and the CONFIGURATION.md matrix.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 632e4adf-a80c-406c-be95-bc6abfc92a8f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/ngram-fallback-chain

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}