Skip to content

docs(fault-tolerance): clarify migration after shutdown grace expires #14872

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kthui merged 2 commits into from
Sep 18, 2026
Merged

docs(fault-tolerance): clarify migration after shutdown grace expires #14872

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
15a63a7
docs(fault-tolerance): clarify migration after shutdown grace expires
kthui Sep 15, 2026
d4a074e
docs(fault-tolerance): clarify shutdown sequence and timing
kthui Sep 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions ...guide/knowledge-base/concepts/fault-tolerance/graceful-shutdown-architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ The `graceful_shutdown()` function:
2. Unregisters all endpoints from discovery
3. Waits for a configurable grace period (`DYN_GRACEFUL_SHUTDOWN_GRACE_PERIOD_SECS`, default 5s)
4. Calls `runtime.shutdown()` to invalidate endpoints and stop accepting new requests
5. Waits for in-flight requests (based on `graceful_shutdown` per endpoint)
5. Waits for request handlers to finish, including with an error (based on `graceful_shutdown` per endpoint)
6. Returns to allow cleanup to proceed

@jh-nv jh-nv Sep 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shutdown sequence here is missing the shutdown_event step that is directly relevant to the migration behavior this PR documents. After the grace period, graceful_shutdown_with_discovery() may first run drain_callback / pre_shutdown_callback, then sets shutdown_event, and only later calls runtime.shutdown(). Backend handlers use shutdown_event to abort unfinished generation with EngineShutdown, which the migrator treats as migratable. Could we update this sequence to reflect that ordering?

@kthui kthui Sep 18, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this. Updated the sequence in d4a074e74d to include the optional drain/pre-shutdown callbacks and show shutdown_event being set before runtime.shutdown().


The aggregate wait in `runtime.shutdown()` is bounded by
Expand Down Expand Up @@ -101,7 +101,7 @@ drain into a restart loop.

Backend workers always use `graceful_shutdown=True`, meaning they wait for in-flight requests to complete until the engine is stopped. Request migration is configured at the **frontend** level via `--migration-limit`:

- When migration is enabled at the frontend, disconnected streams from failed workers are automatically retried on healthy workers
- When migration is enabled at the frontend, requests interrupted by worker failure or graceful shutdown after grace expires are retried on healthy workers, subject to the retry budget and request limits
- Workers don't need to know about migration configuration - they simply complete their work or signal incomplete streams
- See [Request Migration Architecture](request-migration-architecture.md) for details on how migration works

Expand Down Expand Up @@ -206,7 +206,7 @@ Kubernetes uses health endpoints to determine pod readiness:

- **During shutdown**: Endpoints become unavailable
- **Readiness probe fails**: Traffic stops routing to the pod
- **Graceful draining**: Existing requests complete
- **Graceful draining**: Existing requests have time to complete

## Related Documentation

Expand Down
6 changes: 3 additions & 3 deletions ...guide/knowledge-base/concepts/fault-tolerance/request-migration-architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,13 +59,13 @@ The migration system handles two distinct failure scenarios:

#### 2. Ongoing Request Migration (Mid-Stream Disconnection)

**Scenario**: Connection lost during active generation after partial responses have been received.
**Scenario**: The connection is lost or an ongoing request ends with a migratable error.

**Error Pattern**: Stream termination detected before generation completion.
**Error Pattern**: A migratable stream error is reported before generation completes.

**Migration Process**:

1. **Failure Detection**: The system detects the stream disconnection through error monitoring.
1. **Failure Detection**: The system detects the connection loss or migratable request error.

2. **State Preservation**: At this point, the request's token sequence contains both the original prompt tokens and all successfully generated tokens from the failed worker.

Expand Down
6 changes: 4 additions & 2 deletions docs/fern/pages/kubernetes/fault-tolerance/graceful-shutdown.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ subtitle: Let workers finish in-flight requests and release resources cleanly wh

When Kubernetes terminates a pod (rollout, scale-down, node drain), Dynamo workers stop accepting new requests, keep serving in-flight ones through a grace period, then release engine and connection resources before exiting. This is **on by default** — every component handles `SIGTERM`/`SIGINT` and drains automatically. The steps below tune *how long* it waits and make sure interrupted requests are recovered.

Expiry of `DYN_GRACEFUL_SHUTDOWN_GRACE_PERIOD_SECS` initiates cancellation of unfinished requests. The Frontend can migrate these requests to a healthy worker when migration is enabled and policy permits it; otherwise the client receives an error. Exhausted retries or an exceeded sequence-length cap can prevent recovery.
Comment thread
kthui marked this conversation as resolved.
Resolved
Comment thread
kthui marked this conversation as resolved.
Resolved

@jh-nv jh-nv Sep 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a little too precise about the timing. Grace-period expiry does not necessarily immediately initiate cancellation: graceful_shutdown_with_discovery() can run backend-specific drain/pre-shutdown callbacks before setting shutdown_event. For example, TRT-LLM prefill can drain for up to another 30s before the shutdown event is set. Could we phrase this as cancellation happening when shutdown proceeds past the grace/backend-specific drain phase, rather than exactly when the grace period expires?

@kthui kthui Sep 18, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, the original wording was too precise about grace expiry. Updated it to: “Once shutdown proceeds past the grace period and any backend-specific draining, workers initiate cancellation of unfinished requests.”

This accounts for the extra draining time after grace, as you suggested.


The knobs are three timeouts plus enabling migration. The default flow: endpoints unregister from discovery immediately, workers serve for a short grace period, then endpoints drain (bounded by a timeout) before resources are cleaned up.

> **How it works:** the signal handlers, the `graceful_shutdown()` sequence, per-backend `cleanup()` code, and error-initiated shutdown are documented in [Graceful Shutdown Architecture](../../developer-guide/knowledge-base/concepts/fault-tolerance/graceful-shutdown-architecture.md).
Expand Down Expand Up @@ -54,7 +56,7 @@ The defaults are sound for most deployments. Raise the relevant timeout only for

<Step title="Enable migration so drained requests retry">

Draining lets *current* requests finish, but a request interrupted by an unexpected worker loss still needs somewhere to go. Enable [request migration](request-migration.md) on the Frontend so disconnected streams are retried on healthy workers. Set `DYN_MIGRATION_LIMIT` in the Frontend `env:` (or `--migration-limit` in its `args:`):
Draining lets *current* requests finish during the grace period set by `DYN_GRACEFUL_SHUTDOWN_GRACE_PERIOD_SECS`, but a request interrupted after the shutdown grace period expires or by an unexpected worker loss still needs somewhere to go. Enable [request migration](request-migration.md) on the Frontend for rolling upgrades and scale-downs as well, so eligible interrupted requests are retried on healthy workers. Set `DYN_MIGRATION_LIMIT` in the Frontend `env:` (or `--migration-limit` in its `args:`):

```yaml
- name: Frontend
Expand Down Expand Up @@ -90,7 +92,7 @@ During Frontend shutdown, `/health` returns 503 so readiness routing stops, whil
finish or `DYN_HTTP_GRACEFUL_SHUTDOWN_TIMEOUT_SECS` expires.

During worker shutdown, endpoints unregister and stop receiving new work while admitted requests
complete. If a pod receives `SIGKILL` before draining finishes, increase
have time to complete. If a pod receives `SIGKILL` before draining finishes, increase
`terminationGracePeriodSeconds` (step 1) or lower the relevant internal timeout (step 2).

</Step>
Expand Down
4 changes: 2 additions & 2 deletions docs/fern/pages/kubernetes/fault-tolerance/request-migration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
title: Request Migration
subtitle: Keep in-flight generations alive across worker failures by retrying them on a healthy worker.
subtitle: Keep in-flight generations alive across worker failures and shutdown interruptions by retrying them on a healthy worker.
---

When a worker fails mid-generation, Dynamo can migrate the in-progress request to a healthy worker and continue from the exact point of failure — no tokens lost or duplicated, and no interruption visible to the client. Migration is configured once on the **Frontend** and applies globally to every model it serves.
When a worker fails mid-generation or its shutdown grace period expires, Dynamo can migrate the in-progress request to a healthy worker and continue from the exact point of failure — no tokens lost or duplicated, and no interruption visible to the client. Migration is configured once on the **Frontend** and applies globally to every model it serves.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

Migration is **off by default** (`--migration-limit 0`). The steps below turn it on, optionally bound its memory use, and verify it is working.

Expand Down
Loading