[Core] Add max-waiting-queue-length parameter to reject requests when queue is full

[chudyandrej]

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

Implements --max-waiting-queue-length parameter to allow vLLM to reject new requests when the waiting queue reaches a specified limit, providing better load management for production environments.

Addresses: #2901 #3168 #4190

Key changes:

• Added max_waiting_queue_length field to SchedulerConfig with CLI argument support
• Implemented queue length validation in Scheduler.add_seq_group() with custom SchedulerWaitingQueueFullError
• Added HTTP 503 error handling across OpenAI-compatible serving endpoints (serving_chat.py, serving_completion.py, serving_engine.py)
• Integrated error propagation through AsyncLLMEngine

Benefits:

• Prevents memory exhaustion from unbounded request queuing
• Provides graceful degradation with clear HTTP 503 responses
• Enables load balancing by allowing clients to route to less loaded instances

Test Plan

# 1. Install and activate venv
source .venv/bin/activate
pip install -e .

# 2. Run unit tests for scheduler queue limiting
python -m pytest tests/core/test_scheduler.py::test_scheduler_max_waiting_queue_length -v
python -m pytest tests/core/test_scheduler.py::test_scheduler_max_waiting_queue_length_disabled -v

# 3. Test CLI argument parsing and help text
vllm serve --help | grep -A 3 "max-waiting-queue-length"

# 4. Test runtime behavior with queue limit
vllm serve meta-llama/Llama-2-7b-hf --max-waiting-queue-length 1 --max-num-seqs 1
# Send multiple concurrent requests to trigger queue limit

Test Result

Unit Tests:
tests/core/test_scheduler.py::test_scheduler_max_waiting_queue_length PASSED
tests/core/test_scheduler.py::test_scheduler_max_waiting_queue_length_disabled PASSED

CLI Help Output:
--max-waiting-queue-length MAX_WAITING_QUEUE_LENGTH
                        Maximum number of requests that can be in the waiting
                        queue. When the queue reaches this limit, new requests
                        will be rejected with HTTP 503 error. If None, no
                        limit is enforced. (default: None)

Behavior Verification:
- ✅ Queue limiting works correctly when enabled (rejects with HTTP 503)
- ✅ Feature disabled by default (backwards compatible)
- ✅ Error handling propagates through all serving endpoints
- ✅ Custom exception provides clear error messages

(Optional) Documentation Update

No documentation update required - CLI help text is auto-generated from SchedulerConfig docstrings, ensuring consistency with the implementation.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[gemini-code-assist]

Code Review

This pull request introduces a valuable feature for managing server load by limiting the waiting queue length. My review focuses on improving the correctness and robustness of the error handling. I've identified some unreachable code in the exception handling logic and a bug where streaming responses would not receive the correct HTTP 503 error when the queue is full. Addressing these points will make the implementation more robust and prevent unexpected behavior in production.

[robertgshaw2-redhat]

Hello. Thank you for your PR.

V0 is in process of being deprecated. I think this is a useful feature, so I would be happy to review it in the V1 codepath.

[chaunceyjiang]

About two months ago, I submitted an RFC: #18826.
Since there is a significant difference between V1 and V0—V1 has two processes, P0 and P1—currently P1 cannot catch custom errors and pass them to P0, such as SchedulerWaitingQueueFullError. I've implemented part of the custom error handling locally, and it seems that quite a few code changes are needed.
Therefore, I haven't started the implementation yet.
Additionally, the DP (Data Parallel) scenario may also need to be considered.

[chudyandrej]

@chaunceyjiang Thanks for your comment; this complexity I totally missed. That's indeed a building block that is currently missing. Do you have some quick workaround in your mind that can unblock this PR? Or do you believe that cross-process error reporting needs to be implemented first?

[chudyandrej]

I can imagine a counter on the serving layer. Something like

async def create_chat_completion(...):
      try:
          await self._validate_queue_capacity()  # ← Fast validation
          self._active_requests_count += 1       # ← Increment

          generator = self.engine_client.generate(...)  # ← Call engine

          # Process results...

      finally:
          self._active_requests_count -= 1       # ← Always decrement

[chaunceyjiang]

Hi @chudyandrej, I’ve submitted a PR (#21352) that fully implements error propagation — custom errors can now be passed from P1 to P0. If you don’t mind, I’d like to add you to the co-authors list. Then we can shift our focus to reviewing #21352. What do you think?

[chudyandrej]

Hi @chudyandrej, I’ve submitted a PR (#21352) that fully implements error propagation — custom errors can now be passed from P1 to P0. If you don’t mind, I’d like to add you to the co-authors list. Then we can shift our focus to reviewing #21352. What do you think?

Sounds good. Okay, so let's close this one.