[Core] Add multi-step support to LLMEngine #7789
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge). To run full CI, you can do one of these:
🚀 |
|
/ready |
github-actions
bot
added
the
ready
benchmarks/benchmark_throughput.py
Outdated
| num_scheduler_steps=8, | ||
| use_v2_block_manager=True, |
There was a problem hiding this comment.
Good catch, added as arg params
examples/offline_inference.py
Outdated
| @@ -11,7 +11,7 @@ | |||
| sampling_params = SamplingParams(temperature=0.8, top_p=0.95) | |||
|
|
|||
| # Create an LLM. | |||
| llm = LLM(model="facebook/opt-125m") | |||
| llm = LLM(model="facebook/opt-125m", tensor_parallel_size=2) | |||
There was a problem hiding this comment.
reverted
tests/multi_step/test_correctness.py
Outdated
| async def test_multi_step_async_llm(example_prompts, model: str, tp_size: int, | ||
| pp_size: int, eager_mode: int, | ||
| num_scheduler_steps: int, | ||
| num_prompts: int): |
There was a problem hiding this comment.
Revert this as unrelated
There was a problem hiding this comment.
reverted
alexm-neuralmagic
force-pushed
the
multi_step_llm_engine
branch
from
a2ee9d3
to
ec2c589
Compare
Signed-off-by: Alvant <[email protected]>
This PR adds multi-step support to the LLM Engine class. The new multi-step methods that were originally added to async_llm_engine were moved to llm_engine, since async_llm_engine inherits from llm_engine.