[Feature] Add --max-unfinished-requests apiserver parameter

[chaunceyjiang]

Purpose

Feature implementation #18826

CLOSE #18826

CLOSE #21352

Add new CLI argument --max-unfinished-requests to limit concurrent unfinished requests across all API servers. When the limit is exceeded, new requests are rejected with 503 Service Unavailable.

Test Plan

 vllm serve /mnt/data3/models/MiniMax/MiniMax-M2.5 -tp 4 --tool-call-parser minimax_m2 --enable-auto-tool-choice --reasoning-parser minimax_m2   --trust-remote-code  --max_unfinished_requests 10 --api-server-count 4

Test Result

(ApiServer_1 pid=3380147) INFO:     127.0.0.1:52948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(ApiServer_1 pid=3380147) INFO:     127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(ApiServer_1 pid=3380147) INFO:     127.0.0.1:53012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(ApiServer_1 pid=3380147) INFO:     127.0.0.1:53012 - "POST /v1/chat/completions HTTP/1.1" 503 Service Unavailable
(ApiServer_1 pid=3380147) INFO:     127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 503 Service Unavailable
(ApiServer_2 pid=3380148) INFO:     127.0.0.1:52970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(ApiServer_3 pid=3380149) INFO:     127.0.0.1:52964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(ApiServer_3 pid=3380149) INFO:     127.0.0.1:52990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(ApiServer_2 pid=3380148) INFO:     127.0.0.1:53022 - "POST /v1/chat/completions HTTP/1.1" 200 OK

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request implements a global request limit across multiple API servers using shared memory. It introduces the --max-unfinished-requests parameter and updates the load_aware_call decorator to reject requests with a 503 error when the limit is reached. Review feedback identifies several critical issues: the shared memory array is not updated during request completion, leading to stale data; a race condition exists in the limit-checking logic; and the shared array should use a lock to ensure consistent reads during summation.

[chaunceyjiang]

#27064 (comment)

Hi @orozery PTAL.

[chaunceyjiang]

/cc @DarkLight1337 PTAL.