[LLM Batch][4/N] vLLM engine stage

[comaniac]

Why are these changes needed?

This PR introduces vLLM engine stage with the following features:

• Decoding models.
• Embedding models.
• vLLM v0 and vLLM v1.
• TP and PP.

This PR doesn't support:

• Vision models.
• End to end vLLM processor.

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[GeneDer]

LGTM!

[comaniac]

@kouroshHakha CI green. PTAL

[kouroshHakha]

Just small comments around test org + some nits:

[kouroshHakha]

so what I am saying is we need to a bit of file reorg to make the BUILD files cleaner. What do you think about this suggestion:

Make the folder structure as following:

python/ray/llm/tests
-- batch 
BUILD
----- gpu
-------- stages
----------- vllm_engine_stage_gpu.py
----- cpu
------- stages
...

in the BUILD do sth like

py_test_module_list(
  files = glob(["cpu/*/test_*.py"]),
  size = "small",
  tags = ["exclusive", "team:llm"],
  deps = ["//:ray_lib"],
)

py_test_module_list(
  files = glob(["gpu/*/test_*.py"]),
  size = "large",
  tags = ["exclusive", "gpu", "team:llm"],
  deps = ["//:ray_lib"],
)

[kouroshHakha]

Let's goooooo

[pcmoritz]

I'll approve this to get unblocked :)

Let's make sure we have a docs page and good documentation once this is ready :)