[Refactor] [1/N] to simplify the vLLM serving architecture

[chaunceyjiang]

Purpose

Refer #28218

Split vllm/entrypoints/openai/api_server.py based on different functionalities.

Test Plan

(APIServer pid=126097) INFO 12-01 11:13:27 [api_server.py:1504] Starting vLLM API server 0 on http://0.0.0.0:8000
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:38] Available routes are:
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /v1/load_lora_adapter, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /v1/unload_lora_adapter, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /classify, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /v1/embeddings, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /score, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /v1/score, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /rerank, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /v1/rerank, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /v2/rerank, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /pooling, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /adapters, Methods: POST
(APIServer pid=126097) INFO 12-01 11:13:27 [launcher.py:46] Route: /adapters/{adapter_name}, Methods: DELETE

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[markmc]

For reference, from @DarkLight1337 in #27882

I propose restructuring the code for online serving into the following modules:

• vllm.entrypoints.serve.core: Contains the code for setting up the async client and FastAPI app. May include some common APIs such as health check.
• vllm.entrypoints.serve.openai: Contains only the code for OpenAI endpoints (e.g. Completions API, Chat Completions API, Responses API)
• vllm.entrypoints.serve.anthropic: Contains only the code for Anthropic endpoints (Messages API)
• vllm.entrypoints.serve.jina: Contains only the code for JinaAI endpoints (Reranker API)
• vllm.entrypoints.serve.vllm: Contains only the code for vLLM endpoints (e.g. Tokenize API, Pooling API, dev mode endpoints)

In vllm.entrypoints.serve, we can have the actual entrypoint which uses .core to build the server, then incrementally attach endpoints to the FastAPI app by importing relevant functions from the API-specific submodules.

[esmeetu]

@chaunceyjiang @markmc @DarkLight1337 Just opened RFC #28218 , might be related to this refactor. Would love to hear your thoughts!

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.

[DarkLight1337]

/gemini review

[gemini-code-assist]

Code Review

This pull request refactors the vLLM serving architecture by splitting the monolithic vllm/entrypoints/openai/api_server.py into smaller, more focused modules under vllm/entrypoints/serve/. This is a good improvement for maintainability. I've found two issues: one critical regression where some refactored API endpoints are not registered, and one high-severity issue where an API response type has changed, which could be a breaking change for clients.

[chaunceyjiang]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a significant and well-executed refactoring of the vLLM serving architecture by splitting the monolithic api_server.py into a more modular structure with functionality-specific routers. This greatly improves the maintainability and readability of the code. The code has been moved logically into new modules under vllm/entrypoints/serve/. While reviewing the changes, I identified a critical issue with state synchronization for the elastic endpoint scaling feature that could cause problems in a multi-worker production environment. My detailed feedback is in the review comment.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @chaunceyjiang.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork