fix(router): distinguish overload from unavailable [DYN-3279]

docs/components/router/router-configuration.md

@@ -60,7 +60,7 @@ Queue limits are configured per discovered worker endpoint with
 configured value multiplied by the current number of discovered endpoints.
 Limits are checked against current usage before adding the incoming request,
 so the request that crosses a limit is accepted and the next queued request is
-rejected with HTTP 503 and the effective total. Worker removal does not evict
+rejected with HTTP 529 and the effective total. Worker removal does not evict
 queued requests; new arrivals reject until usage drains or capacity returns.
 DRR charges the uncached-token snapshot captured at enqueue, while raw, cached,
 and uncached snapshots remain unchanged for limits, WSPT, counters, and later
@@ -94,7 +94,7 @@ no separate first-stage admission queue or global cross-class cap.
 
 This is intentionally not behavior preserving. Class limits are worker-scaled
 and class-local rather than global; rejection returns the structured
-policy-class HTTP 503 response rather than the previous overload 429 path; and
+policy-class HTTP 529 response rather than the previous overload 429 path; and
 it does not exclude the entire router instance. The previous flat
 `default_policy_class` and `uncached_isl_policy_class_tiers` schema is not
 accepted, and ordinary physical classes are no longer direct header

docs/fault-tolerance/request-rejection.md

@@ -2,7 +2,7 @@
 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 title: Request Rejection
-subtitle: Sheds load with HTTP 503 when all workers exceed admission-control thresholds, preventing cascading failures and OOMs.
+subtitle: Sheds load with HTTP 529 when all workers exceed admission-control thresholds, preventing cascading failures and OOMs.
 ---
 
 This document describes how Dynamo implements request rejection to prevent system overload and maintain service stability under high load conditions.
@@ -15,7 +15,7 @@ Request rejection (also known as load shedding) is a fault tolerance mechanism t
 - Degraded latency for all requests
 - Out-of-memory conditions on GPU workers
 
-When all workers exceed their configured busy thresholds, new requests receive an HTTP 503 (Service Unavailable) response, signaling clients to retry later.
+When all workers exceed their configured busy thresholds, new requests receive an HTTP 529 response, signaling clients to retry later.
 
 ## Architecture
 
@@ -34,7 +34,7 @@ When all workers exceed their configured busy thresholds, new requests receive a
                                          │ If all workers busy
                                          ▼
                                 ┌─────────────────────┐
-                                │   HTTP 503 Error    │
+                                │   HTTP 529 Error    │
                                 │ "All workers busy"  │
                                 └─────────────────────┘
 ```
@@ -157,26 +157,26 @@ Workers publish these metrics for monitoring:
 3. If configured, router retrieves list of free (non-busy) instances
 4. If no free instances exist (but instances are registered):
    - Request is rejected with `PipelineError::ServiceOverloaded`
-   - HTTP 503 response is returned to client
+   - HTTP 529 response is returned to client
 
 ### Error Response
 
 When requests are rejected, clients receive:
 
 ```http
-HTTP/1.1 503 Service Unavailable
+HTTP/1.1 529
 Content-Type: application/json
 
 {
-  "message": "Service temporarily unavailable: All workers are busy, please retry later",
-  "type": "service_unavailable",
-  "code": 503
+  "message": "Service temporarily overloaded",
+  "type": "Overloaded",
+  "code": 529
 }
 ```
 
 ### Client Retry Strategy
 
-Clients should implement exponential backoff when receiving 503 responses:
+Clients should implement exponential backoff when receiving 529 responses:
 
 ```python
 import time
@@ -185,7 +185,7 @@ import random
 def send_with_retry(request, max_retries=5):
     for attempt in range(max_retries):
         response = send_request(request)
-        if response.status_code != 503:
+        if response.status_code != 529:
             return response
 
         # Exponential backoff with jitter
@@ -205,7 +205,7 @@ Track rejection behavior with these metrics:
   - Labels:
     - `model`: The model name being served
     - `endpoint`: The API endpoint that received the request (e.g., `chat_completions`, `completions`, `embeddings`)
-  - This metric is incremented when the router returns a `ResourceExhausted` error because all workers are busy. The rejected request is surfaced to the client as an HTTP 503 response.
+  - This metric is incremented when the router returns a `ResourceExhausted` error because all workers are busy. The rejected request is surfaced to the client as an HTTP 529 response.
 
 **Example metrics output:**
 ```text
@@ -312,7 +312,7 @@ the engine while a slot is free; once all `N` engine slots are busy, further
 requests go into the small overflow queue of size `Q`; when the engine **and**
 the queue are both full the worker rejects the request with
 `Server overloaded: worker at capacity`. The frontend maps this rejection to
-`ResourceExhausted` → **HTTP 503**, and temporarily marks the worker overloaded
+`ResourceExhausted` → **HTTP 529**, and temporarily marks the worker overloaded
 so it is skipped on the next routing decision (cleared automatically on the next
 metric recompute). The effective hard cap is **N + Q** in-flight requests per
 worker. The overflow channel is sized to `Q-1` because the single dispatcher

docs/getting-started/introduction.md

@@ -177,7 +177,7 @@ Kubernetes comes with some fault tolerance functionalities, but LLM deployment r
 - **Router and Frontend** -- Dynamo supports launching multiple frontend + router replicas for improved fault tolerance by sharing router states.
 - **Request Migration** -- When a worker fails during request processing, Dynamo can migrate in-progress requests to healthy workers while preserving partial generation state and maintaining seamless token flow to clients.
 - **Request Cancellation** -- Dynamo supports canceling in-flight requests through the AsyncEngineContext trait, which provides graceful stop signals and hierarchical cancellation propagation through request chains.
-- **Request Rejection (Load Shedding)** -- When workers are overloaded, Dynamo rejects new requests with HTTP 503 responses based on configurable thresholds for KV cache utilization and prefill tokens.
+- **Request Rejection (Load Shedding)** -- When workers are overloaded, Dynamo rejects new requests with HTTP 529 responses based on configurable thresholds for KV cache utilization and prefill tokens.
 
 ### Observability
 

docs/getting-started/introduction.zh-CN.md

@@ -145,7 +145,7 @@ Kubernetes 自带一些容错功能，但 LLM 部署需要专门的容错和韧
 - **Router and Frontend** -- Dynamo 支持启动多个 frontend + router 副本，并通过共享 router 状态提升容错能力。
 - **Request Migration** -- 当 worker 在请求处理过程中失败时，Dynamo 可以将进行中的请求迁移到健康 worker，同时保留部分生成状态，并维持面向客户端的无缝 token 流。
 - **Request Cancellation** -- Dynamo 支持通过 AsyncEngineContext trait 取消进行中的请求，该 trait 提供优雅停止信号和通过请求链传播的层级取消。
-- **Request Rejection (Load Shedding)** -- 当 worker 过载时，Dynamo 会基于 KV 缓存利用率和 prefill token 的可配置阈值，以 HTTP 503 响应拒绝新请求。
+- **Request Rejection (Load Shedding)** -- 当 worker 过载时，Dynamo 会基于 KV 缓存利用率和 prefill token 的可配置阈值，以 HTTP 529 响应拒绝新请求。
 
 ### 可观测性
 

lib/llm/src/discovery/worker_monitor.rs

@@ -498,7 +498,7 @@ impl KvWorkerMonitor {
     /// Returns true iff the user explicitly configured at least one threshold.
     ///
     /// When false, all three per-field checks are skipped in `is_overloaded` and
-    /// rejection is fully disabled. Callers that gate 503 responses on overload
+    /// rejection is fully disabled. Callers that gate 529 responses on overload
     /// detection should check this before enabling the gate.
     pub fn is_configured(&self) -> bool {
         self.thresholds.read().unwrap().is_configured()

lib/llm/src/http/service/busy_threshold.rs

@@ -5,8 +5,8 @@
 //!
 //! The busy thresholds control when workers are marked as "busy" based on their
 //! KV cache block utilization and prefill token utilization. When all workers
-//! for a model exceed their thresholds, new requests are rejected with a 503
-//! Service Unavailable response.
+//! for a model exceed their thresholds, new requests are rejected with a 529
+//! response.
 //!
 //! ## Endpoints
 //!

lib/llm/src/http/service/error.rs

@@ -4,6 +4,10 @@
 use axum::http::StatusCode;
 use thiserror::Error;
 
+pub(crate) fn overload_status_code() -> StatusCode {
+    StatusCode::from_u16(529).expect("529 is a valid HTTP status code")
+}
+
 /// Implementation of the Completion Engines served by the HTTP service should
 /// map their custom errors to to this error type if they wish to return error
 /// codes besides 500.
@@ -26,8 +30,10 @@ pub struct HttpError {
 pub enum SanitizedError {
     /// 499 Client Closed Request.
     Cancelled,
-    /// 503 Service Unavailable.
+    /// 529 Site Is Overloaded.
     Overloaded,
+    /// 503 Service Unavailable.
+    Unavailable,
     /// 500 Internal Server Error.
     Internal,
     /// Preserve a backend-reported 5xx status code while replacing the
@@ -67,7 +73,8 @@ impl SanitizedError {
         match self {
             // 499 is not IANA-registered but is widely used (nginx).
             SanitizedError::Cancelled => StatusCode::from_u16(499).unwrap(),
-            SanitizedError::Overloaded => StatusCode::SERVICE_UNAVAILABLE,
+            SanitizedError::Overloaded => overload_status_code(),
+            SanitizedError::Unavailable => StatusCode::SERVICE_UNAVAILABLE,
             SanitizedError::Internal => StatusCode::INTERNAL_SERVER_ERROR,
             SanitizedError::PreserveServerError(code) => {
                 debug_assert!(
@@ -87,6 +94,7 @@ impl SanitizedError {
         match self {
             SanitizedError::Cancelled => "request_cancelled",
             SanitizedError::Overloaded => "overloaded_error",
+            SanitizedError::Unavailable => "overloaded_error",
             SanitizedError::Internal => "api_error",
             SanitizedError::PreserveServerError(status) => match status.as_u16() {
                 503 | 529 => "overloaded_error",
@@ -100,6 +108,7 @@ impl SanitizedError {
         match self {
             SanitizedError::Cancelled => "request_cancelled",
             SanitizedError::Overloaded => "service_unavailable",
+            SanitizedError::Unavailable => "service_unavailable",
             SanitizedError::Internal => "internal_server_error",
             SanitizedError::PreserveServerError(status) => match status.as_u16() {
                 503 | 529 => "service_unavailable",
@@ -121,6 +130,7 @@ impl std::fmt::Display for SanitizedError {
         match self {
             SanitizedError::Cancelled => f.write_str("Request cancelled"),
             SanitizedError::Overloaded => f.write_str("Service temporarily overloaded"),
+            SanitizedError::Unavailable => f.write_str("Service temporarily unavailable"),
             SanitizedError::Internal | SanitizedError::PreserveServerError(_) => {
                 f.write_str("Internal server error")
             }
@@ -132,6 +142,15 @@ impl std::fmt::Display for SanitizedError {
 mod tests {
     use super::*;
 
+    #[test]
+    fn local_statuses_distinguish_overload_from_unavailable() {
+        assert_eq!(SanitizedError::Overloaded.status().as_u16(), 529);
+        assert_eq!(
+            SanitizedError::Unavailable.status(),
+            StatusCode::SERVICE_UNAVAILABLE
+        );
+    }
+
     #[test]
     fn preserve_server_error_503_maps_to_overload_types() {
         // Backend-asserted 503 must surface as the spec-correct overload

lib/llm/src/http/service/metrics.rs

@@ -36,6 +36,13 @@ pub fn request_was_rejected(err: &(dyn std::error::Error + 'static)) -> bool {
     dynamo_runtime::error::match_error_chain(err, REJECTION, NON_REJECTION)
 }
 
+/// Check whether an error chain indicates that no backend worker is available.
+pub fn request_was_unavailable(err: &(dyn std::error::Error + 'static)) -> bool {
+    const UNAVAILABLE: &[DynamoErrorType] = &[DynamoErrorType::Unavailable];
+    const AVAILABLE: &[DynamoErrorType] = &[];
+    dynamo_runtime::error::match_error_chain(err, UNAVAILABLE, AVAILABLE)
+}
+
 /// Check whether an error chain indicates the request was cancelled.
 pub fn request_was_cancelled(err: &(dyn std::error::Error + 'static)) -> bool {
     const CANCELLATION: &[DynamoErrorType] = &[DynamoErrorType::Cancelled];
@@ -378,7 +385,7 @@ pub enum ErrorType {
     Validation,
     /// Model or resource not found (404)
     NotFound,
-    /// Service overloaded, too many requests (503)
+    /// Service overloaded or rate limited (429 or 529)
     Overload,
     /// Request cancelled by client or timeout
     Cancelled,

lib/llm/src/http/service/openai.rs

@@ -79,7 +79,7 @@ pub const ANNOTATION_REQUEST_ID: &str = "request_id";
 
 const VALIDATION_PREFIX: &str = "Validation: ";
 
-use super::error::SanitizedError;
+use super::error::{SanitizedError, overload_status_code};
 
 pub(super) fn rl_router(
     drt: Arc<dynamo_runtime::DistributedRuntime>,
@@ -115,6 +115,7 @@ pub(crate) struct ErrorMessage {
 fn map_error_code_to_error_type(code: StatusCode) -> String {
     match code.canonical_reason() {
         Some(reason) => reason.to_string(),
+        None if code.as_u16() == 529 => "Overloaded".to_string(),
         // 499 is not IANA-registered (nginx convention for client-closed-request),
         // so canonical_reason() returns None. Use the de facto standard name.
         None if code.as_u16() == 499 => "Client Closed Request".to_string(),
@@ -136,11 +137,12 @@ fn classify_error_for_metrics(code: StatusCode, message: &str) -> ErrorType {
         StatusCode::NOT_FOUND => ErrorType::NotFound, // 404
         StatusCode::NOT_IMPLEMENTED => ErrorType::NotImplemented, // 501
         StatusCode::TOO_MANY_REQUESTS => ErrorType::Overload, // 429
-        StatusCode::SERVICE_UNAVAILABLE => ErrorType::Overload, // 503
+        StatusCode::SERVICE_UNAVAILABLE => ErrorType::Overload, // 503 unavailable or legacy overload
         StatusCode::INTERNAL_SERVER_ERROR => ErrorType::Internal, // 500
-        _ if code.as_u16() == 499 => ErrorType::Cancelled, // 499 Client Closed Request
-        _ if code.is_client_error() => ErrorType::Validation, // other 4xx
-        _ => ErrorType::Internal,                     // everything else
+        _ if code.as_u16() == 529 => ErrorType::Overload,       // 529
+        _ if code.as_u16() == 499 => ErrorType::Cancelled,      // 499 Client Closed Request
+        _ if code.is_client_error() => ErrorType::Validation,   // other 4xx
+        _ => ErrorType::Internal,                               // everything else
     }
 }
 
@@ -365,7 +367,7 @@ impl ErrorMessage {
     /// with the details of the error.
     pub fn from_anyhow(err: anyhow::Error, alt_msg: &str) -> ErrorResponse {
         if let Some(rejection) = find_queue_rejection_in_chain(err.as_ref()) {
-            let code = StatusCode::SERVICE_UNAVAILABLE;
+            let code = overload_status_code();
             return (
                 code,
                 Json(ErrorMessage {
@@ -377,14 +379,22 @@ impl ErrorMessage {
             );
         }
 
-        // Check for ResourceExhausted anywhere in the error chain → HTTP 503
+        // Check for ResourceExhausted anywhere in the error chain → HTTP 529
         if super::metrics::request_was_rejected(err.as_ref()) {
             return ErrorMessage::sanitized_with_details(
                 SanitizedError::Overloaded,
                 format!("{err:#}"),
             );
         }
 
+        // No backend workers are currently routable → HTTP 503.
+        if super::metrics::request_was_unavailable(err.as_ref()) {
+            return ErrorMessage::sanitized_with_details(
+                SanitizedError::Unavailable,
+                format!("{err:#}"),
+            );
+        }
+
         // InvalidArgument (top-level OR Backend) → 400.
         if let Some(dynamo_err) = find_invalid_argument_in_chain(err.as_ref()) {
             return (
@@ -3381,7 +3391,9 @@ mod tests {
             .build()
             .into();
         let response = ErrorMessage::from_anyhow(err, BACKUP_ERROR_MESSAGE);
-        assert_eq!(response.0, StatusCode::SERVICE_UNAVAILABLE);
+        assert_eq!(response.0.as_u16(), 529);
+        assert_eq!(response.1.code, 529);
+        assert_eq!(response.1.error_type, "Overloaded");
         assert_eq!(response.1.message, "Service temporarily overloaded");
         assert!(
             !response.1.message.contains("All workers are busy"),
@@ -3390,7 +3402,23 @@ mod tests {
     }
 
     #[test]
-    fn queue_rejection_maps_to_structured_http_503() {
+    fn unavailable_error_response_from_anyhow() {
+        use dynamo_runtime::error::{DynamoError, ErrorType};
+
+        let err: anyhow::Error = DynamoError::builder()
+            .error_type(ErrorType::Unavailable)
+            .message("No workers available for endpoint test/worker/generate")
+            .build()
+            .into();
+        let response = ErrorMessage::from_anyhow(err, BACKUP_ERROR_MESSAGE);
+
+        assert_eq!(response.0, StatusCode::SERVICE_UNAVAILABLE);
+        assert_eq!(response.1.code, StatusCode::SERVICE_UNAVAILABLE.as_u16());
+        assert_eq!(response.1.message, "Service temporarily unavailable");
+    }
+
+    #[test]
+    fn queue_rejection_maps_to_structured_http_529() {
         use dynamo_kv_router::scheduling::{QueueLimitKind, QueueRejection};
 
         let rejection = QueueRejection {
@@ -3402,8 +3430,9 @@ mod tests {
         let response =
             ErrorMessage::from_anyhow(anyhow::Error::new(rejection), BACKUP_ERROR_MESSAGE);
 
-        assert_eq!(response.0, StatusCode::SERVICE_UNAVAILABLE);
-        assert_eq!(response.1.code, StatusCode::SERVICE_UNAVAILABLE.as_u16());
+        assert_eq!(response.0.as_u16(), 529);
+        assert_eq!(response.1.code, 529);
+        assert_eq!(response.1.error_type, "Overloaded");
         assert_eq!(
             response.1.details.as_deref(),
             Some(&serde_json::json!({
@@ -4488,6 +4517,10 @@ mod tests {
             classify_error_for_metrics(StatusCode::SERVICE_UNAVAILABLE, "Overloaded"),
             ErrorType::Overload
         );
+        assert_eq!(
+            classify_error_for_metrics(overload_status_code(), "Overloaded"),
+            ErrorType::Overload
+        );
         assert_eq!(
             classify_error_for_metrics(StatusCode::INTERNAL_SERVER_ERROR, "Panic"),
             ErrorType::Internal

lib/llm/src/http/service/openapi_docs.rs

@@ -169,6 +169,13 @@ fn create_operation_for_route(method: &str, path: &str) -> utoipa::openapi::path
             .build(),
     );
 
+    operation = operation.response(
+        "529",
+        ResponseBuilder::new()
+            .description("Service overloaded")
+            .build(),
+    );
+
     operation.build()
 }