Generate: assistant should be greedy in assisted decoding #30778
Conversation
| greedy_wrong_parameter_msg.format(flag_name="top_p", flag_value=self.top_p), | ||
| UserWarning, | ||
| ) | ||
| if self.min_p is not None: |
There was a problem hiding this comment.
This check was missing 👼
amyeroberts
left a comment
amyeroberts
left a comment
There was a problem hiding this comment.
Thanks for fixing!
Will this result in different outputs for the users? If so, we might want to make sure to run any slow tests.
Question for the future: would we ever want to enable a samplng strategy for the assistant?
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Nope :) The only difference is execution time. (The output is entirely defined by the main model and its generation config, both untouched in this PR)
Perhaps. A potential strategy for improvement is to sample several continuations and try them all as candidates. But that would require significant changes over the current code 👀 |
Co-authored-by: amyeroberts <[email protected]>
@amyeroberts my answer did not age well (#32867 -- some randomness was lost as a result of this change) |
3 participants
What does this PR do?
Restores a previous behavior -- forces the assistant in assisted decoding to do greedy decoding.
While providing this answer, I noticed that we dropped a property in the refactor that was present in the original implementation: the assistant should be greedy.
Why? We want the assistant to maximize the number of matches with the original model, to result in the maximum speedup. Remember: the main model always calls the shots, the output text is never changed regardless of the flags we set for the assistant. By forcing the assistant to be greedy, we pick its most likely tokens. If the assistant model has a distribution aligned with the main model, these tokens will also be the main model's most likely tokens, increasing the number of matches.
When running my original benchmarks (https://github.com/gante/huggingface-demos/tree/main/experiments/faster_generation), this resulted in a ~20% speedup @ temperature=0.5 over the current version of the code.