Align with huggingface beam search

[hashming]

main modifications

1. sampler
   • always sample 2 * num_beams tokens, even if beam_width < num_beams
2. scheduler
   • update: keep at-most num_beams finished seqs (beam_hyps) and at-most num_beams running seqs in seq_group.
   • move _decode and _stop function to scheduler (from llm_engine).
3. sequence
   • move "data" attributes from Sequence to SequenceData

example

• prompt: "What is deep learning?"
• sampling_params: SamplingParams(temperature = 0, use_beam_search = True, n = 5, max_tokens = 100)
• model & tokenizer: A finetuned Llama
• original beam search returns:

 * Learning is deep type of machine learning that involves using building and neural networks to model and solve complex problems.
 * Deep learning is a subset of artificial intelligence that involves the use of artificial neural networks to model and solve complex problems.
 * It's a type of Artificial allows training artificial neural networks to recognize patterns in data. problems that require
 * the not a word machine learning that involves building and training neural networks to model and generalize knowledge problems that Deep learning is inspired by the structure and function of the human brain and is used for a variety of applications including natural language processing, image and speech recognition, and decision making.
 * A is a the subfield of artificial intelligence that involves the use of artificial neural networks to model and solve complex problems. Deep learning is inspired by the structure and function of the human brain and is used for a variety of applications including natural language processing, image and speech recognition, and decision making.

• modified beam search:

 * Deep learning is a subfield of artificial intelligence that involves the use of artificial neural networks to model and solve problems that require high-level processing, such as natural language processing, image and speech recognition, and decision making.
 * Deep learning is a subset of artificial intelligence that involves the use of artificial neural networks to model and solve complex problems. It is inspired by the structure and function of the human brain and is used for a variety of applications such as natural language processing, image and speech recognition, and decision making.
 * Deep learning is a subfield of artificial intelligence that involves the use of artificial neural networks to model and solve complex problems. Deep learning is inspired by the structure and function of the human brain and is used for a variety of applications including natural language processing, image and speech recognition, and decision making.
 * Deep learning is a subset of artificial intelligence that involves the use of artificial neural networks to model and solve complex problems. It is inspired by the structure and function of the human brain and is used for a variety of applications such as natural language processing, image and speech recognition, and autonomous decision making.
 * Deep learning is a subfield of artificial intelligence that involves the use of artificial neural networks to model and solve problems that require high-level processing, such as natural language processing, image and speech recognition, and decision making. Deep learning is inspired by the structure and function of the human brain and is used to model and solve complex problems.

[hashming]

related issue: #344 #644

[zhuohan123]

@hsm1997 Thank you for your great contribution! The changes you make are a bit complicated. Can we schedule a chat to discuss about this PR? I cannot find your email address. Can you send me an email at zhuohan[at]berkeley.edu? Thanks again!

[leiwen83]

A minor fix with this PR:

diff --git a/vllm/core/scheduler.py b/vllm/core/scheduler.py
index e2ca127..11eea5e 100644
--- a/vllm/core/scheduler.py
+++ b/vllm/core/scheduler.py
@@ -342,7 +342,7 @@ class Scheduler:
                     continue

                 # schedule next-beam tasks
-                pending = pending[:sampling_params.n]
+                pending = pending[:sampling_params.best_of]
                 running_ids = [
                     seq.seq_id for seq in seq_group.get_seqs(
                         status=SequenceStatus.RUNNING)

We need to keep the history with best_of setting, or we may lose the highest score, since n in sampling_params only means for the output token, not in the searching stage.

[zhuohan123]

Close this PR since #857 is merged. Thanks @hsm1997 again for finding the issue and the draft PR!