speculative: draft-mtp for dots3note

[cdanis]

Overview

draft-mtp support for dots3note model arch.

Not compatible with vision/mtmd -- hits existing TODO in common/speculative.cpp.

Additional information

Tested, gives sizable performance boost on Strix Halo (10t/s -> 14-16t/s). Acceptance rate is pretty good on some coding prompts I tried.

I slot print_timing: id  3 | task 0 | draft acceptance = 0.74800 (  561 accepted /   750 generated), mean len =  3.24
I spec common_specu: statistics        draft-mtp: #calls(b,g,a) =    1    250    250, #gen drafts =    250, #acc drafts =   217, #gen tokens =    750, #acc tokens =   561, #mean acc len = 3.24, #acc rate/pos = (0.868, 0.744, 0.632), dur(b,g,a) = 0.002, 3828.419, 0.336 ms

Tested w/ both ROCm and Vulkan.

Also tested CUDA with layer split against 2x RTX A5000, ofc heavy offload (-ncmoe 22), but still meaningful boost:

2.19.802.231 I slot print_timing: id  1 | task 1 | draft acceptance = 0.70285 (  641 accepted /   912 generated), mean len =  3.11
2.19.802.312 I spec common_specu: statistics        draft-mtp: #calls(b,g,a) =    1    304    304, #gen drafts =    304, #acc drafts =   255, #gen tokens =    912, #acc tokens =   641, #mean acc len = 3.11, #acc rate/pos = (0.839, 0.691, 0.579), dur(b,g,a) = 0.001, 5448.805, 0.702 ms

Requirements

• I have read and agree with the contributing guidelines: YES
• AI usage disclosure: YES: coding, repo exploration

[cdanis]

Marking this draft while I do more code review and testing -- happy to accept comments meanwhile though :)

[cdanis]

@ngxson since you wrote #27060 would you have time to review this? Or maybe @fairydreaming

[ngxson]

your PR says that it's related to dots3note, but file changes extended to 10 more models. I cannot review this PR as-is

per contribution guidelines, unrelated changes require separate PR.

[cdanis]

your PR says that it's related to dots3note, but file changes extended to 10 more models. I cannot review this PR as-is

per contribution guidelines, unrelated changes require separate PR.

sorry, I had done a TODO to refactor out some code instead of copy-n-paste for the 10th time. changes to other models reverted, now patch is just dots3note and core files like llama-graph & speculative.

[nikwen]

TODO to refactor out some code instead of copy-n-paste for the 10th time

Feel free to submit that as a follow-up PR. Would be appreciated. :)