[Core][Feat] Add max-waiting-queue-time parameter to reject requests

[chaunceyjiang]

Purpose

Add max-waiting-queue-time parameter to reject requests

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request introduces a max-waiting-queue-time parameter to reject requests when the server is under high load. This is achieved by tracking the average queue time of recent requests using a new QueueTimeTracker class. The feature is well-integrated, with the new parameter added to EngineArgs and propagated down to the serving layers. The logic to check the queue time and reject requests with a 503 error is implemented in OpenAIServing. The QueueTimeTracker itself uses a sliding window with time-based decay, which is a solid approach. My review found one area for improvement in the QueueTimeTracker implementation regarding an unused variable, which I've commented on.