docs(fault-tolerance): clarify migration after shutdown grace expires

docs/fern/pages/developer-guide/knowledge-base/concepts/fault-tolerance/graceful-shutdown-architecture.md

@@ -40,12 +40,15 @@ for sig in (signal.SIGTERM, signal.SIGINT):
 ```
 
 The `graceful_shutdown()` function:
+
 1. Logs the shutdown signal
 2. Unregisters all endpoints from discovery
 3. Waits for a configurable grace period (`DYN_GRACEFUL_SHUTDOWN_GRACE_PERIOD_SECS`, default 5s)
-4. Calls `runtime.shutdown()` to invalidate endpoints and stop accepting new requests
-5. Waits for in-flight requests (based on `graceful_shutdown` per endpoint)
-6. Returns to allow cleanup to proceed
+4. Awaits the optional `drain_callback` and `pre_shutdown_callback`, in that order
+5. Sets `shutdown_event`, if provided, to initiate cancellation of unfinished requests
+6. Awaits the optional `cleanup_callback` before runtime teardown
+7. Calls `runtime.shutdown()` to initiate runtime shutdown
+8. Returns while the runtime waits for request handlers to finish, including with an error (based on `graceful_shutdown` per endpoint)
 
 The aggregate wait in `runtime.shutdown()` is bounded by
 `DYN_RUNTIME_GRACEFUL_SHUTDOWN_TIMEOUT_SECS`, which defaults to 900 seconds
@@ -101,7 +104,7 @@ drain into a restart loop.
 
 Backend workers always use `graceful_shutdown=True`, meaning they wait for in-flight requests to complete until the engine is stopped. Request migration is configured at the **frontend** level via `--migration-limit`:
 
-- When migration is enabled at the frontend, disconnected streams from failed workers are automatically retried on healthy workers
+- When migration is enabled at the frontend, requests interrupted by worker failure or graceful shutdown after grace expires are retried on healthy workers, subject to the retry budget and request limits
 - Workers don't need to know about migration configuration - they simply complete their work or signal incomplete streams
 - See [Request Migration Architecture](request-migration-architecture.md) for details on how migration works
 
@@ -206,7 +209,7 @@ Kubernetes uses health endpoints to determine pod readiness:
 
 - **During shutdown**: Endpoints become unavailable
 - **Readiness probe fails**: Traffic stops routing to the pod
-- **Graceful draining**: Existing requests complete
+- **Graceful draining**: Existing requests have time to complete
 
 ## Related Documentation
 

docs/fern/pages/developer-guide/knowledge-base/concepts/fault-tolerance/request-migration-architecture.md

@@ -59,13 +59,13 @@ The migration system handles two distinct failure scenarios:
 
 #### 2. Ongoing Request Migration (Mid-Stream Disconnection)
 
-**Scenario**: Connection lost during active generation after partial responses have been received.
+**Scenario**: The connection is lost or an ongoing request ends with a migratable error.
 
-**Error Pattern**: Stream termination detected before generation completion.
+**Error Pattern**: A migratable stream error is reported before generation completes.
 
 **Migration Process**:
 
-1. **Failure Detection**: The system detects the stream disconnection through error monitoring.
+1. **Failure Detection**: The system detects the connection loss or migratable request error.
 
 2. **State Preservation**: At this point, the request's token sequence contains both the original prompt tokens and all successfully generated tokens from the failed worker.
 

docs/fern/pages/kubernetes/fault-tolerance/graceful-shutdown.md

@@ -7,6 +7,8 @@ subtitle: Let workers finish in-flight requests and release resources cleanly wh
 
 When Kubernetes terminates a pod (rollout, scale-down, node drain), Dynamo workers stop accepting new requests, keep serving in-flight ones through a grace period, then release engine and connection resources before exiting. This is **on by default** — every component handles `SIGTERM`/`SIGINT` and drains automatically. The steps below tune *how long* it waits and make sure interrupted requests are recovered.
 
+Once shutdown proceeds past the grace period and any backend-specific draining, workers initiate cancellation of unfinished requests. The Frontend can migrate these requests to a healthy worker when migration is enabled and policy permits it; otherwise the client receives an error. Exhausted retries or an exceeded sequence-length cap can prevent recovery.
+
 The knobs are three timeouts plus enabling migration. The default flow: endpoints unregister from discovery immediately, workers serve for a short grace period, then endpoints drain (bounded by a timeout) before resources are cleaned up.
 
 > **How it works:** the signal handlers, the `graceful_shutdown()` sequence, per-backend `cleanup()` code, and error-initiated shutdown are documented in [Graceful Shutdown Architecture](../../developer-guide/knowledge-base/concepts/fault-tolerance/graceful-shutdown-architecture.md).
@@ -54,7 +56,7 @@ The defaults are sound for most deployments. Raise the relevant timeout only for
 
 <Step title="Enable migration so drained requests retry">
 
-Draining lets *current* requests finish, but a request interrupted by an unexpected worker loss still needs somewhere to go. Enable [request migration](request-migration.md) on the Frontend so disconnected streams are retried on healthy workers. Set `DYN_MIGRATION_LIMIT` in the Frontend `env:` (or `--migration-limit` in its `args:`):
+Draining lets *current* requests finish during the grace period set by `DYN_GRACEFUL_SHUTDOWN_GRACE_PERIOD_SECS`, but a request interrupted after the shutdown grace period expires or by an unexpected worker loss still needs somewhere to go. Enable [request migration](request-migration.md) on the Frontend for rolling upgrades and scale-downs as well, so eligible interrupted requests are retried on healthy workers. Set `DYN_MIGRATION_LIMIT` in the Frontend `env:` (or `--migration-limit` in its `args:`):
 
 ```yaml
   - name: Frontend
@@ -90,7 +92,7 @@ During Frontend shutdown, `/health` returns 503 so readiness routing stops, whil
 finish or `DYN_HTTP_GRACEFUL_SHUTDOWN_TIMEOUT_SECS` expires.
 
 During worker shutdown, endpoints unregister and stop receiving new work while admitted requests
-complete. If a pod receives `SIGKILL` before draining finishes, increase
+have time to complete. If a pod receives `SIGKILL` before draining finishes, increase
 `terminationGracePeriodSeconds` (step 1) or lower the relevant internal timeout (step 2).
 
 </Step>

docs/fern/pages/kubernetes/fault-tolerance/request-migration.md

@@ -2,10 +2,10 @@
 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 title: Request Migration
-subtitle: Keep in-flight generations alive across worker failures by retrying them on a healthy worker.
+subtitle: Keep in-flight generations alive across worker failures and shutdown interruptions by retrying them on a healthy worker.
 ---
 
-When a worker fails mid-generation, Dynamo can migrate the in-progress request to a healthy worker and continue from the exact point of failure — no tokens lost or duplicated, and no interruption visible to the client. Migration is configured once on the **Frontend** and applies globally to every model it serves.
+When a worker fails mid-generation or its shutdown grace period expires, Dynamo can migrate the in-progress request to a healthy worker and continue from the exact point of failure — no tokens lost or duplicated, and no interruption visible to the client. Migration is configured once on the **Frontend** and applies globally to every model it serves.
 
 Migration is **off by default** (`--migration-limit 0`). The steps below turn it on, optionally bound its memory use, and verify it is working.