Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
9df67ed
Enable adaptive verify window for ngram/draft speculation
michaelneale Jul 19, 2026
141ef39
Replace speculative rollback with positional MTP n-gram pipelining
i386 Jul 20, 2026
c06bca6
Pipeline speculative verify windows across latency
i386 Jul 20, 2026
55db388
Fix positional correction and adaptive pipeline depth
i386 Jul 20, 2026
0b3720c
Continuously refill the speculative horizon
i386 Jul 20, 2026
20e457a
Productionize pipelined MTP n-gram speculation
i386 Jul 20, 2026
87de12b
Fix speculative docs and UI formatting
i386 Jul 20, 2026
5b824cb
Remove stale speculative projections and fix CI
i386 Jul 20, 2026
296c949
Handle fragmented direct-return fallback replies
i386 Jul 21, 2026
2af07bc
Replace speculative repair with fixed-depth positional pipeline
i386 Jul 21, 2026
68f4502
Expose split-stage compute overlap telemetry
i386 Jul 21, 2026
7f8ef6c
Lock split topology placement
i386 Jul 21, 2026
9160b31
Document locked split topology
i386 Jul 21, 2026
1d02631
Address locked topology review feedback
i386 Jul 21, 2026
bd940a2
Merge locked split topology for deterministic benchmarks
i386 Jul 21, 2026
479a4e1
Fix SPEED-Bench timing JSONL output
i386 Jul 22, 2026
c8227ef
Bound benchmark telemetry finalization
i386 Jul 22, 2026
6006848
Hash SPEED-Bench request and response pairs
i386 Jul 22, 2026
87e691a
mesh: stop re-applying formation-time RTT gate to operational stage s…
michaelneale Jul 22, 2026
34bd3b8
skippy: raise return-sink ready timeout 5s->20s for cold WAN bridge s…
michaelneale Jul 22, 2026
c05355a
runtime: relaunch withdrawn splits when peers return instead of endin…
michaelneale Jul 22, 2026
05dae2a
Merge remote-tracking branch 'origin/main' into agent/positional-mtp-…
michaelneale Jul 22, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 10 additions & 15 deletions SKIPPY_PROTOCOL_TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,28 +155,23 @@ chunk and every normal decode token.

## E. State Lifecycle Efficiency

- [ ] Reduce checkpoint and restore round trips for speculative decode.
- Avoid checkpointing every `VerifySpan` when a cheaper journal or suffix
trim can preserve correctness.
- Use `SKIP_VERIFY_CHECKPOINT` only when the repair path is proven safe.

- [ ] Add speculative journals.
- Record enough per-stage state to commit or discard a verify span without a
full restore where possible.

- [ ] Improve suffix trim semantics.
- Make rejected speculative suffix rollback cheaper than full session restore.
- Validate recurrent-state families separately from transformer-only KV.
- [x] Make speculative positions authoritative in stage-state v9.
- Decode and `VerifyWindow` messages carry the absolute position each stage
must have before execution.
- Stages ahead of that position rewind attention KV locally.
- Speculation never checkpoints, restores, trims by control message, or
replays a rejected prefix.
- Recurrent-state stages reject positional speculation instead of falling
back to the removed checkpoint protocol.

- [ ] Add checkpoint hierarchy.
- Keep coarse prompt checkpoints and lightweight decode-span checkpoints.
- Keep coarse prompt checkpoints for non-speculative session/cache workflows.
- Evict checkpoint state with explicit memory accounting.

- [ ] Track state-control latency separately from activation latency.
- Emit checkpoint, restore, trim, prefix-restore, and decode-fuse timings per
stage.
- Use these metrics to decide whether direct final replies or journaled
rollback should land first.
- Keep speculative position-rewind counts and trimmed-token totals separate.

## Validation Gates

Expand Down
7 changes: 4 additions & 3 deletions crates/llama-spec-bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Local target/draft speculative decoding checker and benchmark.
`llama-spec-bench` compares a target GGUF model with a draft GGUF model on a
prompt set. It checks tokenizer compatibility, verifies speculative output
against baseline target decoding, measures acceptance behavior, and reports
projected verification costs.
the measured target and draft costs of the serial diagnostic loop.

## Architecture Role

Expand Down Expand Up @@ -46,7 +46,7 @@ sequenceDiagram
D-->>B: proposed tokens
B->>T: verify proposals
T-->>B: accepted prefix or rejection
B->>B: update acceptance and projection stats
B->>B: update acceptance stats
end
B->>B: compare against baseline target tokens
```
Expand Down Expand Up @@ -79,7 +79,8 @@ speculative output mismatch makes the command fail.
- accepted/rejected draft token counts
- acceptance rate and mean accepted tokens per window
- baseline and speculative decode timing
- projected rollback and scratch verification costs
- measured target verification and draft proposal costs
- measured speculative throughput and speedup versus target-only decoding
- per-prompt text previews and mismatch index

The default corpus path is
Expand Down
Loading
Loading