mtp: use inp_out_ids for skipping logit computation

[am17an]

When doing a follow-up decode for the draft model, we were always doing the logits computation even though it is not required. Thanks for comment at #23230 (comment) for pointing this out

Overview

Additional information

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: NO

[Bushido76]

@am17an Thank you very much for your very quick reaction and first implementation. I'm a newby regarding the llama.cpp software architecture, but I did implement and compile your implementation locally. Unfortunately - at least for now - it seems to slow down the token processing speed (tg128) by around 15-20%. With the "background" of the fix (avoiding unneccesary logit computation) - imo the opposite should be the case.

Testing on an AMD Unified Memory APU (Strix Halo, Vulkan/RADV) ... passing zero-sized tensors down the graph pipeline seems to cause severe pipeline stalls and driver overhead in Vulkan, compared to cleanly stopping graph expansion on the host side when n_outputs == 0.

[am17an]

@Bushido76 indeed there is probably some error in how you measure. This PR avoids extra computations, so either it should 0 or positive effect.

[engrtipusultan]

@Bushido76 indeed there is probably some error in how you measure. This PR avoids extra computations, so either it should 0 or positive effect.

Yes it is certainly faster.

Before this PR. Your modified bench with extra variance.
https://gist.github.com/engrtipusultan/3c5985c6ff58a56bb5f7fc0c02e26f40

Before Bench

 bash  python3 mtp-bench.py --url http://127.0.0.1:52497
  code_python        pred= 192 draft= 132 acc= 124 rate=0.939 tok/s=16.6
  code_cpp           pred=  54 draft=  36 acc=  36 rate=1.000 tok/s=17.2
  explain_concept    pred= 192 draft= 189 acc=  95 rate=0.503 tok/s=11.5
  summarize          pred=  48 draft=  40 acc=  28 rate=0.700 tok/s=13.8
  qa_factual         pred= 192 draft= 144 acc= 118 rate=0.819 tok/s=15.2
  translation        pred=  17 draft=  12 acc=  10 rate=0.833 tok/s=16.1
  creative_short     pred=  30 draft=  32 acc=  14 rate=0.438 tok/s=10.7
  stepwise_math      pred= 192 draft= 144 acc= 118 rate=0.819 tok/s=15.2
  long_code_review   pred= 192 draft= 201 acc=  90 rate=0.448 tok/s=10.8
  rust_boilerplate   pred= 192 draft= 140 acc= 120 rate=0.857 tok/s=15.6
  yaml_automation    pred= 192 draft= 163 acc= 109 rate=0.669 tok/s=13.5
  json_extraction    pred= 106 draft=  70 acc=  70 rate=1.000 tok/s=17.4
  js_async_retry     pred= 192 draft= 140 acc= 120 rate=0.857 tok/s=15.6
  regex_parsing      pred= 192 draft= 133 acc= 124 rate=0.932 tok/s=16.5
  system_tuning_explanation pred= 192 draft= 162 acc= 110 rate=0.679 tok/s=13.6
  creative_unpredictable pred= 192 draft= 202 acc=  89 rate=0.441 tok/s=10.8

Aggregate: {
  "n_requests": 16,
  "total_predicted": 2367,
  "total_draft": 1940,
  "total_draft_accepted": 1375,
  "aggregate_accept_rate": 0.7088,
  "wall_s_total": 202.54
}

PR Bench.

Details

 bash  python3 mtp-bench.py --url http://127.0.0.1:39879
  code_python        pred= 192 draft= 132 acc= 124 rate=0.939 tok/s=17.4
  code_cpp           pred=  54 draft=  36 acc=  36 rate=1.000 tok/s=18.2
  explain_concept    pred= 192 draft= 189 acc=  95 rate=0.503 tok/s=12.2
  summarize          pred=  48 draft=  40 acc=  28 rate=0.700 tok/s=14.5
  qa_factual         pred= 192 draft= 144 acc= 118 rate=0.819 tok/s=15.7
  translation        pred=  17 draft=  12 acc=  10 rate=0.833 tok/s=15.9
  creative_short     pred=  30 draft=  32 acc=  14 rate=0.438 tok/s=10.5
  stepwise_math      pred= 192 draft= 144 acc= 118 rate=0.819 tok/s=15.3
  long_code_review   pred= 192 draft= 201 acc=  90 rate=0.448 tok/s=11.4
  rust_boilerplate   pred= 192 draft= 140 acc= 120 rate=0.857 tok/s=16.3
  yaml_automation    pred= 192 draft= 163 acc= 109 rate=0.669 tok/s=14.1
  json_extraction    pred= 106 draft=  70 acc=  70 rate=1.000 tok/s=18.2
  js_async_retry     pred= 192 draft= 140 acc= 120 rate=0.857 tok/s=16.3
  regex_parsing      pred= 192 draft= 133 acc= 124 rate=0.932 tok/s=17.0
  system_tuning_explanation pred= 192 draft= 162 acc= 110 rate=0.679 tok/s=14.1
  creative_unpredictable pred= 192 draft= 202 acc=  89 rate=0.441 tok/s=11.3

Aggregate: {
  "n_requests": 16,
  "total_predicted": 2367,
  "total_draft": 1940,
  "total_draft_accepted": 1375,
  "aggregate_accept_rate": 0.7088,
  "wall_s_total": 194.94
}

[am17an]

can I get another approval?