diff --git a/README.md b/README.md index 11fd1032dd2f..cf84f4ab3d5a 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ limitations under the License. [![Discord](https://dcbadge.limes.pink/api/server/D92uqZRjCZ?style=flat)](https://discord.gg/D92uqZRjCZ) [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/ai-dynamo/dynamo) -| **[Roadmap](https://github.com/ai-dynamo/dynamo/issues/762)** | **[Support matrix](https://github.com/ai-dynamo/dynamo/blob/main/docs/reference/support-matrix.md)** | **[Documentation](https://docs.nvidia.com/dynamo/latest/index.html)** | **[Examples](https://github.com/ai-dynamo/dynamo/tree/main/examples)** | **[Prebuilt containers](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-dynamo/collections/ai-dynamo)** | **[Design Proposals](https://github.com/ai-dynamo/enhancements)** | **[Blogs](https://developer.nvidia.com/blog/tag/nvidia-dynamo)** +| **[Roadmap](https://github.com/ai-dynamo/dynamo/issues/2486)** | **[Support matrix](https://github.com/ai-dynamo/dynamo/blob/main/docs/reference/support-matrix.md)** | **[Documentation](https://docs.nvidia.com/dynamo/latest/index.html)** | **[Examples](https://github.com/ai-dynamo/dynamo/tree/main/examples)** | **[Prebuilt containers](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-dynamo/collections/ai-dynamo)** | **[Design Proposals](https://github.com/ai-dynamo/enhancements)** | **[Blogs](https://developer.nvidia.com/blog/tag/nvidia-dynamo)** # NVIDIA Dynamo @@ -56,9 +56,9 @@ Dynamo is designed to be inference engine agnostic (supports TRT-LLM, vLLM, SGLa | Feature | vLLM | SGLang | TensorRT-LLM | | ------------------------------------------------------------------------------------------------- | ---- | ------ | ------------ | -| [**Disaggregated Serving**](/docs/architecture/disagg_serving.md) | ✅ | ✅ | ✅ | -| [**Conditional Disaggregation**](/docs/architecture/disagg_serving.md#conditional-disaggregation) | 🚧 | 🚧 | 🚧 | -| [**KV-Aware Routing**](/docs/architecture/kv_cache_routing.md) | ✅ | ✅ | ✅ | +| [**Disaggregated Serving**](/docs/design_docs/disagg_serving.md) | ✅ | ✅ | ✅ | +| [**Conditional Disaggregation**](/docs/design_docs/disagg_serving.md#conditional-disaggregation) | 🚧 | 🚧 | 🚧 | +| [**KV-Aware Routing**](/docs/router/kv_cache_routing.md) | ✅ | ✅ | ✅ | | [**Load Based Planner**](docs/planner/load_planner.md) | 🚧 | 🚧 | 🚧 | | [**SLA-Based Planner**](docs/planner/sla_planner.md) | ✅ | ✅ | ✅ | | [**KVBM**](docs/kvbm/kvbm_architecture.md) | ✅ | 🚧 | ✅ | diff --git a/benchmarks/router/README.md b/benchmarks/router/README.md index 2b17066d911b..b82096bb1e3b 100644 --- a/benchmarks/router/README.md +++ b/benchmarks/router/README.md @@ -116,7 +116,7 @@ To see all available router arguments, run: python -m dynamo.frontend --help ``` -For detailed explanations of router arguments (especially KV cache routing parameters), see the [KV Cache Routing documentation](../../docs/architecture/kv_cache_routing.md). +For detailed explanations of router arguments (especially KV cache routing parameters), see the [KV Cache Routing documentation](../../docs/router/kv_cache_routing.md). #### Disaggregated Serving with Automatic Prefill Routing @@ -125,7 +125,7 @@ When you launch prefill workers using `run_engines.sh --prefill`, the frontend a - Uses KV-aware routing regardless of the frontend's `--router-mode` setting - Seamlessly integrates with your decode workers for token generation -No additional configuration is needed - simply launch both decode and prefill workers, and the system handles the rest. See the [KV Cache Routing documentation](../../docs/architecture/kv_cache_routing.md#disaggregated-serving-prefill-and-decode) for more details. +No additional configuration is needed - simply launch both decode and prefill workers, and the system handles the rest. See the [KV Cache Routing documentation](../../docs/router/kv_cache_routing.md#disaggregated-serving-prefill-and-decode) for more details. **Note**: If you're unsure whether your backend engines correctly emit KV events for certain models (e.g., hybrid models like gpt-oss or nemotron nano 2), use the `--no-kv-events` flag to disable KV event tracking and use approximate KV indexing instead: diff --git a/components/README.md b/components/README.md index 18b88c9ee748..3aa6cc0ea828 100644 --- a/components/README.md +++ b/components/README.md @@ -31,7 +31,7 @@ Each engine provides launch scripts for different deployment patterns in their r ## Core Components -### [Backends](src/dynamo/) +### [Backends](backends/) The backends directory contains inference engine integrations and implementations, with a key focus on: diff --git a/components/backends/sglang/deploy/README.md b/components/backends/sglang/deploy/README.md index b2ebff3da983..8b77cd973ebc 100644 --- a/components/backends/sglang/deploy/README.md +++ b/components/backends/sglang/deploy/README.md @@ -144,7 +144,7 @@ All templates use **DeepSeek-R1-Distill-Llama-8B** as the default model. But you ## Further Reading -- **Deployment Guide**: [Creating Kubernetes Deployments](../../../../docs/kubernetes/create_deployment.md) +- **Deployment Guide**: [Creating Kubernetes Deployments](../../../../docs/kubernetes/deployment/create_deployment.md) - **Quickstart**: [Deployment Quickstart](../../../../docs/kubernetes/README.md) - **Platform Setup**: [Dynamo Cloud Installation](../../../../docs/kubernetes/installation_guide.md) - **Examples**: [Deployment Examples](../../../../docs/examples/README.md) diff --git a/components/backends/trtllm/deploy/README.md b/components/backends/trtllm/deploy/README.md index 0c3c78659bac..7dafa6d84d70 100644 --- a/components/backends/trtllm/deploy/README.md +++ b/components/backends/trtllm/deploy/README.md @@ -153,7 +153,7 @@ args: ### 3. Deploy -See the [Create Deployment Guide](../../../../docs/kubernetes/create_deployment.md) to learn how to deploy the deployment file. +See the [Create Deployment Guide](../../../../docs/kubernetes/deployment/create_deployment.md) to learn how to deploy the deployment file. First, create a secret for the HuggingFace token. ```bash @@ -258,7 +258,7 @@ For detailed configuration instructions, see the [KV cache transfer guide](../.. ## Request Migration -You can enable [request migration](../../../../docs/architecture/request_migration.md) to handle worker failures gracefully by adding the migration limit argument to worker configurations: +You can enable [request migration](../../../../docs/fault_tolerance/request_migration.md) to handle worker failures gracefully by adding the migration limit argument to worker configurations: ```yaml args: @@ -277,11 +277,11 @@ Configure the `model` name and `host` based on your deployment. ## Further Reading -- **Deployment Guide**: [Creating Kubernetes Deployments](../../../../docs/kubernetes/create_deployment.md) +- **Deployment Guide**: [Creating Kubernetes Deployments](../../../../docs/kubernetes/deployment/create_deployment.md) - **Quickstart**: [Deployment Quickstart](../../../../docs/kubernetes/README.md) - **Platform Setup**: [Dynamo Cloud Installation](../../../../docs/kubernetes/installation_guide.md) - **Examples**: [Deployment Examples](../../../../docs/examples/README.md) -- **Architecture Docs**: [Disaggregated Serving](../../../../docs/architecture/disagg_serving.md), [KV-Aware Routing](../../../../docs/architecture/kv_cache_routing.md) +- **Architecture Docs**: [Disaggregated Serving](../../../../docs/design_docs/disagg_serving.md), [KV-Aware Routing](../../../../docs/router/kv_cache_routing.md) - **Multinode Deployment**: [Multinode Examples](../../../../docs/backends/trtllm/multinode/multinode-examples.md) - **Speculative Decoding**: [Llama 4 + Eagle Guide](../../../../docs/backends/trtllm/llama4_plus_eagle.md) - **Kubernetes CRDs**: [Custom Resources Documentation](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) diff --git a/components/backends/vllm/deploy/README.md b/components/backends/vllm/deploy/README.md index dd7c9f143889..84d156cb5570 100644 --- a/components/backends/vllm/deploy/README.md +++ b/components/backends/vllm/deploy/README.md @@ -224,7 +224,7 @@ All templates use **Qwen/Qwen3-0.6B** as the default model, but you can use any ## Request Migration -You can enable [request migration](../../../../docs/architecture/request_migration.md) to handle worker failures gracefully by adding the migration limit argument to worker configurations: +You can enable [request migration](../../../../docs/fault_tolerance/request_migration.md) to handle worker failures gracefully by adding the migration limit argument to worker configurations: ```yaml args: @@ -234,12 +234,12 @@ args: ## Further Reading -- **Deployment Guide**: [Creating Kubernetes Deployments](../../../../docs/kubernetes/create_deployment.md) +- **Deployment Guide**: [Creating Kubernetes Deployments](../../../../docs/kubernetes/deployment/create_deployment.md) - **Quickstart**: [Deployment Quickstart](../../../../docs/kubernetes/README.md) - **Platform Setup**: [Dynamo Cloud Installation](../../../../docs/kubernetes/installation_guide.md) - **SLA Planner**: [SLA Planner Quickstart Guide](../../../../docs/planner/sla_planner_quickstart.md) - **Examples**: [Deployment Examples](../../../../docs/examples/README.md) -- **Architecture Docs**: [Disaggregated Serving](../../../../docs/architecture/disagg_serving.md), [KV-Aware Routing](../../../../docs/architecture/kv_cache_routing.md) +- **Architecture Docs**: [Disaggregated Serving](../../../../docs/design_docs/disagg_serving.md), [KV-Aware Routing](../../../../docs/router/kv_cache_routing.md) ## Troubleshooting diff --git a/components/src/dynamo/router/README.md b/components/src/dynamo/router/README.md index 79e00dcc21ba..c2b38bb792e5 100644 --- a/components/src/dynamo/router/README.md +++ b/components/src/dynamo/router/README.md @@ -3,7 +3,7 @@ # Standalone Router -A backend-agnostic standalone KV-aware router service for Dynamo deployments. For details on how KV-aware routing works, see the [KV Cache Routing documentation](/docs/architecture/kv_cache_routing.md). +A backend-agnostic standalone KV-aware router service for Dynamo deployments. For details on how KV-aware routing works, see the [KV Cache Routing documentation](/docs/router/kv_cache_routing.md). ## Overview @@ -29,7 +29,7 @@ python -m dynamo.router \ - `--endpoint`: Full endpoint path for workers in the format `namespace.component.endpoint` (e.g., `dynamo.prefill.generate`) **Router Configuration:** -For detailed descriptions of all KV router configuration options including `--block-size`, `--kv-overlap-score-weight`, `--router-temperature`, `--no-kv-events`, `--router-replica-sync`, `--router-snapshot-threshold`, `--router-reset-states`, and `--no-track-active-blocks`, see the [KV Cache Routing documentation](/docs/architecture/kv_cache_routing.md). +For detailed descriptions of all KV router configuration options including `--block-size`, `--kv-overlap-score-weight`, `--router-temperature`, `--no-kv-events`, `--router-replica-sync`, `--router-snapshot-threshold`, `--router-reset-states`, and `--no-track-active-blocks`, see the [KV Cache Routing documentation](/docs/router/kv_cache_routing.md). ## Architecture @@ -43,7 +43,7 @@ Clients query the `find_best_worker` endpoint to determine which worker should p ## Example: Manual Disaggregated Serving (Alternative Setup) > [!Note] -> **This is an alternative advanced setup.** The recommended approach for disaggregated serving is to use the frontend's automatic prefill routing, which activates when you register workers with `ModelType.Prefill`. See the [KV Cache Routing documentation](/docs/architecture/kv_cache_routing.md#disaggregated-serving-prefill-and-decode) for the default setup. +> **This is an alternative advanced setup.** The recommended approach for disaggregated serving is to use the frontend's automatic prefill routing, which activates when you register workers with `ModelType.Prefill`. See the [KV Cache Routing documentation](../../../../docs/router/kv_cache_routing.md#disaggregated-serving-prefill-and-decode) for the default setup. > > Use this manual setup if you need explicit control over prefill routing configuration or want to manage prefill and decode routers separately. @@ -103,6 +103,6 @@ See [`components/src/dynamo/vllm/handlers.py`](../vllm/handlers.py) for a refere ## See Also -- [KV Cache Routing Architecture](/docs/architecture/kv_cache_routing.md) - Detailed explanation of KV-aware routing +- [KV Cache Routing Architecture](/docs/router/kv_cache_routing.md) - Detailed explanation of KV-aware routing - [Frontend Router](../frontend/README.md) - Main HTTP frontend with integrated routing - [Router Benchmarking](/benchmarks/router/README.md) - Performance testing and tuning diff --git a/deploy/cloud/pre-deployment/README.md b/deploy/cloud/pre-deployment/README.md index 9bcb79e589ff..a134bfeb7179 100644 --- a/deploy/cloud/pre-deployment/README.md +++ b/deploy/cloud/pre-deployment/README.md @@ -21,7 +21,7 @@ This directory contains a pre-deployment check script that verifies your Kuberne - For NCCL tests, please refer to the [NCCL tests](https://docs.nebius.com/kubernetes/gpu/nccl-test#run-tests) for more details. -- For NIXL benchmark, please refer to the [NIXL benchmark pre-deployment checks](/deploy/cloud/pre-deployment/nixl/README.md) for more details. +For the latest pre-deployment check instructions, see the [main branch version of this README](https://github.com/ai-dynamo/dynamo/blob/main/deploy/cloud/pre-deployment/README.md). ## Usage diff --git a/deploy/inference-gateway/README.md b/deploy/inference-gateway/README.md index 3d1c2b5b331b..36eb254b37d2 100644 --- a/deploy/inference-gateway/README.md +++ b/deploy/inference-gateway/README.md @@ -16,7 +16,7 @@ Currently, these setups are only supported with the kGateway based Inference Gat - [Prerequisites](#prerequisites) - [Installation Steps](#installation-steps) -- [Usage](#usage) +- [Usage](#6-usage) ## Prerequisites @@ -160,7 +160,7 @@ You can configure the plugin by setting environment vars in your [values-dynamo- - Set `DYNAMO_OVERLAP_SCORE_WEIGHT` to weigh how heavily the score uses token overlap (predicted KV cache hits) versus other factors (load, historical hit rate). Higher weight biases toward reusing workers with similar cached prefixes. - Set `DYNAMO_ROUTER_TEMPERATURE` to soften or sharpen the selection curve when combining scores. Low temperature makes the router pick the top candidate deterministically; higher temperature lets lower-scoring workers through more often (exploration). - Set `DYNAMO_USE_KV_EVENTS=false` if you want to disable KV event tracking while using kv-routing - - See the [KV cache routing design](../../docs/architecture/kv_cache_routing.md) for details. + - See the [KV cache routing design](../../docs/router/kv_cache_routing.md) for details. diff --git a/deploy/logging/README.md b/deploy/logging/README.md index 6bed61da48b5..2423989d9964 100644 --- a/deploy/logging/README.md +++ b/deploy/logging/README.md @@ -1,3 +1,3 @@ # Dynamo Logging on Kubernetes -For detailed documentation on collecting and visualizing logs on Kubernetes, see [docs/kubernetes/logging.md](../../docs/kubernetes/logging.md). +For detailed documentation on collecting and visualizing logs on Kubernetes, see [docs/kubernetes/observability/logging.md](../../docs/kubernetes/observability/logging.md). diff --git a/deploy/metrics/k8s/README.md b/deploy/metrics/k8s/README.md index d4ac85c0b6eb..514b544b81cb 100644 --- a/deploy/metrics/k8s/README.md +++ b/deploy/metrics/k8s/README.md @@ -1,3 +1,3 @@ # Dynamo Metrics Collection on Kubernetes -For detailed documentation on collecting and visualizing metrics on Kubernetes, see [docs/kubernetes/metrics.md](../../../docs/kubernetes/metrics.md). +For detailed documentation on collecting and visualizing metrics on Kubernetes, see [docs/kubernetes/observability/metrics.md](../../../docs/kubernetes/observability/metrics.md). diff --git a/docs/_sections/architecture.rst b/docs/_sections/architecture.rst deleted file mode 100644 index 13e6dad0a587..000000000000 --- a/docs/_sections/architecture.rst +++ /dev/null @@ -1,11 +0,0 @@ -Overview -============ - -.. include:: ../architecture/architecture.md - :parser: myst_parser.sphinx_ - -.. toctree:: - :hidden: - - Overview - Disaggregated Serving <../architecture/disagg_serving> diff --git a/docs/_sections/backends.rst b/docs/_sections/backends.rst index 653f2ee770bd..e77774f41055 100644 --- a/docs/_sections/backends.rst +++ b/docs/_sections/backends.rst @@ -1,42 +1,9 @@ -.. - SPDX-FileCopyrightText: Copyright (c) 2024-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. - SPDX-License-Identifier: Apache-2.0 - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. - Backends ======== -NVIDIA Dynamo supports multiple inference backends to provide flexibility and performance optimization for different use cases and model architectures. Backends are the underlying engines that execute AI model inference, each optimized for specific scenarios, hardware configurations, and performance requirements. - -Overview --------- - -Dynamo's multi-backend architecture allows you to: - -* **Choose the optimal engine** for your specific workload and hardware -* **Switch between backends** without changing your application code -* **Leverage specialized optimizations** from each backend -* **Scale flexibly** across different deployment scenarios - -Supported Backends ------------------- - -Dynamo currently supports the following high-performance inference backends: - .. toctree:: :maxdepth: 1 vLLM <../backends/vllm/README> SGLang <../backends/sglang/README> - TensorRT-LLM <../backends/trtllm/README> + TensorRT-LLM <../backends/trtllm/README> \ No newline at end of file diff --git a/docs/_sections/k8s_deployment.rst b/docs/_sections/k8s_deployment.rst new file mode 100644 index 000000000000..33cebc46bc28 --- /dev/null +++ b/docs/_sections/k8s_deployment.rst @@ -0,0 +1,10 @@ +Deployment Guide +================ + +.. toctree:: + :hidden: + + Kubernetes Quickstart <../kubernetes/README> + Detailed Installation Guide <../kubernetes/installation_guide> + Dynamo Operator <../kubernetes/dynamo_operator> + Minikube Setup <../kubernetes/deployment/minikube> diff --git a/docs/_sections/k8s_multinode.rst b/docs/_sections/k8s_multinode.rst new file mode 100644 index 000000000000..3a1c7cff2c40 --- /dev/null +++ b/docs/_sections/k8s_multinode.rst @@ -0,0 +1,8 @@ +Multinode +========= + +.. toctree:: + :hidden: + + Multinode Deployments <../kubernetes/deployment/multinode-deployment> + Grove <../kubernetes/grove> diff --git a/docs/_sections/k8s_observability.rst b/docs/_sections/k8s_observability.rst new file mode 100644 index 000000000000..af7c6ff66d95 --- /dev/null +++ b/docs/_sections/k8s_observability.rst @@ -0,0 +1,8 @@ +Observability +============= + +.. toctree:: + :hidden: + + Metrics <../kubernetes/observability/metrics> + Logging <../kubernetes/observability/logging> diff --git a/docs/_sections/observability.rst b/docs/_sections/observability.rst new file mode 100644 index 000000000000..f91973e7d115 --- /dev/null +++ b/docs/_sections/observability.rst @@ -0,0 +1,9 @@ +Observability +============= + +.. toctree:: + :hidden: + + Metrics <../observability/metrics> + Logging <../observability/logging> + Health Checks <../observability/health-checks> \ No newline at end of file diff --git a/docs/guides/tool-calling.md b/docs/agents/tool-calling.md similarity index 100% rename from docs/guides/tool-calling.md rename to docs/agents/tool-calling.md diff --git a/docs/backends/sglang/README.md b/docs/backends/sglang/README.md index 8252ad2411fb..82ca386f774f 100644 --- a/docs/backends/sglang/README.md +++ b/docs/backends/sglang/README.md @@ -34,9 +34,9 @@ git checkout $(git describe --tags $(git rev-list --tags --max-count=1)) | Feature | SGLang | Notes | |---------|--------|-------| -| [**Disaggregated Serving**](../../architecture/disagg_serving.md) | ✅ | | -| [**Conditional Disaggregation**](../../architecture/disagg_serving.md#conditional-disaggregation) | 🚧 | WIP [PR](https://github.com/sgl-project/sglang/pull/7730) | -| [**KV-Aware Routing**](../../architecture/kv_cache_routing.md) | ✅ | | +| [**Disaggregated Serving**](../../design_docs/disagg_serving.md) | ✅ | | +| [**Conditional Disaggregation**](../../design_docs/disagg_serving.md#conditional-disaggregation) | 🚧 | WIP [PR](https://github.com/sgl-project/sglang/pull/7730) | +| [**KV-Aware Routing**](../../router/kv_cache_routing.md) | ✅ | | | [**SLA-Based Planner**](../../planner/sla_planner.md) | ✅ | | | [**Multimodal EPD Disaggregation**](multimodal_epd.md) | ✅ | | | [**KVBM**](../../kvbm/kvbm_architecture.md) | ❌ | Planned | @@ -55,7 +55,7 @@ Dynamo SGLang uses SGLang's native argument parser, so **most SGLang engine argu | Argument | Description | Default | SGLang Equivalent | |----------|-------------|---------|-------------------| | `--endpoint` | Dynamo endpoint in `dyn://namespace.component.endpoint` format | Auto-generated based on mode | N/A | -| `--migration-limit` | Max times a request can migrate between workers for fault tolerance. See [Request Migration Architecture](../../../docs/architecture/request_migration.md). | `0` (disabled) | N/A | +| `--migration-limit` | Max times a request can migrate between workers for fault tolerance. See [Request Migration Architecture](../../fault_tolerance/request_migration.md). | `0` (disabled) | N/A | | `--dyn-tool-call-parser` | Tool call parser for structured outputs (takes precedence over `--tool-call-parser`) | `None` | `--tool-call-parser` | | `--dyn-reasoning-parser` | Reasoning parser for CoT models (takes precedence over `--reasoning-parser`) | `None` | `--reasoning-parser` | | `--use-sglang-tokenizer` | Use SGLang's tokenizer instead of Dynamo's | `False` | N/A | @@ -83,7 +83,7 @@ When a user cancels a request (e.g., by disconnecting from the frontend), the re > [!WARNING] > ⚠️ SGLang backend currently does not support cancellation during remote prefill phase in disaggregated mode. -For more details, see the [Request Cancellation Architecture](../../architecture/request_cancellation.md) documentation. +For more details, see the [Request Cancellation Architecture](../../fault_tolerance/request_cancellation.md) documentation. ## Installation diff --git a/docs/backends/sglang/multimodal_epd.md b/docs/backends/sglang/multimodal_epd.md index 0131804ae901..f4bd19566b6d 100644 --- a/docs/backends/sglang/multimodal_epd.md +++ b/docs/backends/sglang/multimodal_epd.md @@ -23,7 +23,7 @@ git checkout $(git describe --tags $(git rev-list --tags --max-count=1)) ### Components -- workers: For aggregated serving, we have two workers, [MultimodalEncodeWorker](src/dynamo/sglang/request_handlers/multimodal_encode_worker_handler.py) for encoding and [MultimodalWorker](src/dynamo/sglang/request_handlers/multimodal_worker_handler.py) for prefilling and decoding. +- workers: For aggregated serving, we have two workers, [MultimodalEncodeWorkerHandler](../../../components/src/dynamo/sglang/request_handlers/multimodal/encode_worker_handler.py) for encoding and [MultimodalWorkerHandler](../../../components/src/dynamo/sglang/request_handlers/multimodal/worker_handler.py) for prefilling and decoding. - processor: Tokenizes the prompt and passes it to the MultimodalEncodeWorker. ### Workflow @@ -109,7 +109,7 @@ You should see a response similar to this: ### Components -- workers: For disaggregated serving, we have three workers, [MultimodalEncodeWorker](src/dynamo/sglang/request_handlers/multimodal_encode_worker_handler.py) for encoding, [MultimodalWorker](src/dynamo/sglang/request_handlers/multimodal_worker_handler.py) for decoding, and [MultimodalPrefillWorker](src/dynamo/sglang/request_handlers/multimodal_worker_handler.py) for prefilling. +- workers: For disaggregated serving, we have three workers, [MultimodalEncodeWorkerHandler](../../../components/src/dynamo/sglang/request_handlers/multimodal/encode_worker_handler.py) for encoding, [MultimodalWorkerHandler](../../../components/src/dynamo/sglang/request_handlers/multimodal/worker_handler.py) for decoding, and [MultimodalPrefillWorkerHandler](../../../components/src/dynamo/sglang/request_handlers/multimodal/worker_handler.py) for prefilling. - processor: Tokenizes the prompt and passes it to the MultimodalEncodeWorker. ### Workflow diff --git a/docs/backends/trtllm/README.md b/docs/backends/trtllm/README.md index 906eb88c150c..b041773a9ce6 100644 --- a/docs/backends/trtllm/README.md +++ b/docs/backends/trtllm/README.md @@ -52,9 +52,9 @@ git checkout $(git describe --tags $(git rev-list --tags --max-count=1)) | Feature | TensorRT-LLM | Notes | |---------|--------------|-------| -| [**Disaggregated Serving**](../../../docs/architecture/disagg_serving.md) | ✅ | | -| [**Conditional Disaggregation**](../../../docs/architecture/disagg_serving.md#conditional-disaggregation) | 🚧 | Not supported yet | -| [**KV-Aware Routing**](../../../docs/architecture/kv_cache_routing.md) | ✅ | | +| [**Disaggregated Serving**](../../../docs/design_docs/disagg_serving.md) | ✅ | | +| [**Conditional Disaggregation**](../../../docs/design_docs/disagg_serving.md#conditional-disaggregation) | 🚧 | Not supported yet | +| [**KV-Aware Routing**](../../../docs/router/kv_cache_routing.md) | ✅ | | | [**SLA-Based Planner**](../../../docs/planner/sla_planner.md) | ✅ | | | [**Load Based Planner**](../../../docs/planner/load_planner.md) | 🚧 | Planned | | [**KVBM**](../../../docs/kvbm/kvbm_architecture.md) | ✅ | | @@ -220,13 +220,13 @@ Dynamo with TensorRT-LLM supports two methods for transferring KV cache in disag ## Request Migration -You can enable [request migration](../../../docs/architecture/request_migration.md) to handle worker failures gracefully. Use the `--migration-limit` flag to specify how many times a request can be migrated to another worker: +You can enable [request migration](../../../docs/fault_tolerance/request_migration.md) to handle worker failures gracefully. Use the `--migration-limit` flag to specify how many times a request can be migrated to another worker: ```bash python3 -m dynamo.trtllm ... --migration-limit=3 ``` -This allows a request to be migrated up to 3 times before failing. See the [Request Migration Architecture](../../../docs/architecture/request_migration.md) documentation for details on how this works. +This allows a request to be migrated up to 3 times before failing. See the [Request Migration Architecture](../../../docs/fault_tolerance/request_migration.md) documentation for details on how this works. ## Request Cancellation @@ -240,7 +240,7 @@ When a user cancels a request (e.g., by disconnecting from the frontend), the re | **Disaggregated (Decode-First)** | ✅ | ✅ | | **Disaggregated (Prefill-First)** | ✅ | ✅ | -For more details, see the [Request Cancellation Architecture](../../../docs/architecture/request_cancellation.md) documentation. +For more details, see the [Request Cancellation Architecture](../../fault_tolerance/request_cancellation.md) documentation. ## Client diff --git a/docs/backends/vllm/README.md b/docs/backends/vllm/README.md index 423d0589d9f8..9f2b901f2025 100644 --- a/docs/backends/vllm/README.md +++ b/docs/backends/vllm/README.md @@ -35,9 +35,9 @@ git checkout $(git describe --tags $(git rev-list --tags --max-count=1)) | Feature | vLLM | Notes | |---------|------|-------| -| [**Disaggregated Serving**](../../../docs/architecture/disagg_serving.md) | ✅ | | -| [**Conditional Disaggregation**](../../../docs/architecture/disagg_serving.md#conditional-disaggregation) | 🚧 | WIP | -| [**KV-Aware Routing**](../../../docs/architecture/kv_cache_routing.md) | ✅ | | +| [**Disaggregated Serving**](../../../docs/design_docs/disagg_serving.md) | ✅ | | +| [**Conditional Disaggregation**](../../../docs/design_docs/disagg_serving.md#conditional-disaggregation) | 🚧 | WIP | +| [**KV-Aware Routing**](../../../docs/router/kv_cache_routing.md) | ✅ | | | [**SLA-Based Planner**](../../../docs/planner/sla_planner.md) | ✅ | | | [**Load Based Planner**](../../../docs/planner/load_planner.md) | 🚧 | WIP | | [**KVBM**](../../../docs/kvbm/kvbm_architecture.md) | ✅ | | @@ -153,7 +153,7 @@ Below we provide a selected list of advanced deployments. Please open up an issu ### Kubernetes Deployment -For complete Kubernetes deployment instructions, configurations, and troubleshooting, see [vLLM Kubernetes Deployment Guide](/components/backends/vllm/deploy/README.md) +For complete Kubernetes deployment instructions, configurations, and troubleshooting, see [vLLM Kubernetes Deployment Guide](../../../components/backends/vllm/deploy/README.md) ## Configuration @@ -178,17 +178,17 @@ When using KV-aware routing, ensure deterministic hashing across processes to av ```bash vllm serve ... --enable-prefix-caching --prefix-caching-algo sha256 ``` -See the high-level notes in [KV Cache Routing](../../../docs/architecture/kv_cache_routing.md) on deterministic event IDs. +See the high-level notes in [KV Cache Routing](../../../docs/router/kv_cache_routing.md) on deterministic event IDs. ## Request Migration -You can enable [request migration](../../../docs/architecture/request_migration.md) to handle worker failures gracefully. Use the `--migration-limit` flag to specify how many times a request can be migrated to another worker: +You can enable [request migration](../../../docs/fault_tolerance/request_migration.md) to handle worker failures gracefully. Use the `--migration-limit` flag to specify how many times a request can be migrated to another worker: ```bash python3 -m dynamo.vllm ... --migration-limit=3 ``` -This allows a request to be migrated up to 3 times before failing. See the [Request Migration Architecture](../../../docs/architecture/request_migration.md) documentation for details on how this works. +This allows a request to be migrated up to 3 times before failing. See the [Request Migration Architecture](../../../docs/fault_tolerance/request_migration.md) documentation for details on how this works. ## Request Cancellation @@ -201,4 +201,4 @@ When a user cancels a request (e.g., by disconnecting from the frontend), the re | **Aggregated** | ✅ | ✅ | | **Disaggregated** | ✅ | ✅ | -For more details, see the [Request Cancellation Architecture](../../../docs/architecture/request_cancellation.md) documentation. +For more details, see the [Request Cancellation Architecture](../../../docs/fault_tolerance/request_cancellation.md) documentation. diff --git a/docs/benchmarks/benchmarking.md b/docs/benchmarks/benchmarking.md index c9c7202b7d9f..a981ecb4b6a4 100644 --- a/docs/benchmarks/benchmarking.md +++ b/docs/benchmarks/benchmarking.md @@ -521,3 +521,18 @@ The built-in Python workflow connects to endpoints, benchmarks with aiperf, and 3. **Direct module usage**: Use individual Python modules (`benchmarks.utils.benchmark`, `benchmarks.utils.plot`) for granular control over each step of the benchmarking process. The Python benchmarking module provides a complete end-to-end benchmarking experience with full control over the workflow. + +--- + +## Testing with Mocker Backend + +For development and testing purposes, Dynamo provides a [mocker backend](../../components/src/dynamo/mocker/) that simulates LLM inference without requiring actual GPU resources. This is useful for: + +- **Testing deployments** without expensive GPU infrastructure +- **Developing and debugging** router, planner, or frontend logic +- **CI/CD pipelines** that need to validate infrastructure without model execution +- **Benchmarking framework validation** to ensure your setup works before using real backends + +The mocker backend mimics the API and behavior of real backends (vLLM, SGLang, TensorRT-LLM) but generates mock responses instead of running actual inference. + +See the [mocker directory](../../components/src/dynamo/mocker/) for usage examples and configuration options. diff --git a/docs/conf.py b/docs/conf.py index d95a535b7ccc..b4f50c9ed9b2 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -70,7 +70,7 @@ } ], "switcher": { - "json_url": "../versions1.json", + "json_url": "versions1.json", "version_match": release, }, "extra_head": { diff --git a/docs/architecture/architecture.md b/docs/design_docs/architecture.md similarity index 99% rename from docs/architecture/architecture.md rename to docs/design_docs/architecture.md index 6aa885a55ca0..1c53cf7bc3cc 100644 --- a/docs/architecture/architecture.md +++ b/docs/design_docs/architecture.md @@ -53,7 +53,7 @@ To address the growing demands of distributed inference serving, NVIDIA introduc The following diagram outlines Dynamo's high-level architecture. To enable large-scale distributed and disaggregated inference serving, Dynamo includes five key features: - [Dynamo Disaggregated Serving](disagg_serving.md) -- [Dynamo Smart Router](kv_cache_routing.md) +- [Dynamo Smart Router](../router/kv_cache_routing.md) - [Dynamo KV Cache Block Manager](../kvbm/kvbm_intro.rst) - [Planner](../planner/planner_intro.rst) - [NVIDIA Inference Transfer Library (NIXL)](https://github.com/ai-dynamo/nixl/blob/main/docs/nixl.md) diff --git a/docs/architecture/disagg_serving.md b/docs/design_docs/disagg_serving.md similarity index 100% rename from docs/architecture/disagg_serving.md rename to docs/design_docs/disagg_serving.md diff --git a/docs/architecture/distributed_runtime.md b/docs/design_docs/distributed_runtime.md similarity index 100% rename from docs/architecture/distributed_runtime.md rename to docs/design_docs/distributed_runtime.md diff --git a/docs/architecture/dynamo_flow.md b/docs/design_docs/dynamo_flow.md similarity index 100% rename from docs/architecture/dynamo_flow.md rename to docs/design_docs/dynamo_flow.md diff --git a/docs/development/backend-guide.md b/docs/development/backend-guide.md index 76861455a038..f6e0ee8e75f9 100644 --- a/docs/development/backend-guide.md +++ b/docs/development/backend-guide.md @@ -74,7 +74,7 @@ The `model_type` can be: - `model_name`: The name to call the model. Your incoming HTTP requests model name must match this. Defaults to the hugging face repo name or the folder name. - `context_length`: Max model length in tokens. Defaults to the model's set max. Only set this if you need to reduce KV cache allocation to fit into VRAM. - `kv_cache_block_size`: Size of a KV block for the engine, in tokens. Defaults to 16. -- `migration_limit`: Maximum number of times a request may be [migrated to another Instance](../architecture/request_migration.md). Defaults to 0. +- `migration_limit`: Maximum number of times a request may be [migrated to another Instance](../fault_tolerance/request_migration.md). Defaults to 0. - `user_data`: Optional dictionary containing custom metadata for worker behavior (e.g., LoRA configuration). Defaults to None. See `components/backends` for full code examples. @@ -116,7 +116,7 @@ In the P/D disaggregated setup you would have `deepseek-distill-llama8b.prefill. A Python worker may need to be shut down promptly, for example when the node running the worker is to be reclaimed and there isn't enough time to complete all ongoing requests before the shutdown deadline. -In such cases, you can signal incomplete responses by raising a `GeneratorExit` exception in your generate loop. This will immediately close the response stream, signaling to the frontend that the stream is incomplete. With request migration enabled (see the [`migration_limit`](../architecture/request_migration.md) parameter), the frontend will automatically migrate the partially completed request to another worker instance, if available, to be completed. +In such cases, you can signal incomplete responses by raising a `GeneratorExit` exception in your generate loop. This will immediately close the response stream, signaling to the frontend that the stream is incomplete. With request migration enabled (see the [`migration_limit`](../fault_tolerance/request_migration.md) parameter), the frontend will automatically migrate the partially completed request to another worker instance, if available, to be completed. > [!WARNING] > We will update the `GeneratorExit` exception to a new Dynamo exception. Please expect minor code breaking change in the near future. @@ -139,7 +139,7 @@ class RequestHandler: When `GeneratorExit` is raised, the frontend receives the incomplete response and can seamlessly continue generation on another available worker instance, preserving the user experience even during worker shutdowns. -For more information about how request migration works, see the [Request Migration Architecture](../architecture/request_migration.md) documentation. +For more information about how request migration works, see the [Request Migration Architecture](../fault_tolerance/request_migration.md) documentation. ## Request Cancellation @@ -161,4 +161,4 @@ class RequestHandler: The context parameter is optional - if your generate method doesn't include it in its signature, Dynamo will call your method without the context argument. -For detailed information about request cancellation, including async cancellation monitoring and context propagation patterns, see the [Request Cancellation Architecture](../architecture/request_cancellation.md) documentation. +For detailed information about request cancellation, including async cancellation monitoring and context propagation patterns, see the [Request Cancellation Architecture](../fault_tolerance/request_cancellation.md) documentation. diff --git a/docs/architecture/request_cancellation.md b/docs/fault_tolerance/request_cancellation.md similarity index 100% rename from docs/architecture/request_cancellation.md rename to docs/fault_tolerance/request_cancellation.md diff --git a/docs/architecture/request_migration.md b/docs/fault_tolerance/request_migration.md similarity index 100% rename from docs/architecture/request_migration.md rename to docs/fault_tolerance/request_migration.md diff --git a/docs/hidden_toctree.rst b/docs/hidden_toctree.rst index ad1fbbbffb8e..637cbf3dfd95 100644 --- a/docs/hidden_toctree.rst +++ b/docs/hidden_toctree.rst @@ -25,10 +25,9 @@ api/nixl_connect/README.md kubernetes/api_reference.md - kubernetes/create_deployment.md + kubernetes/deployment/create_deployment.md kubernetes/fluxcd.md - kubernetes/gke_setup.md kubernetes/grove.md kubernetes/model_caching_with_fluid.md kubernetes/README.md @@ -36,12 +35,12 @@ observability/metrics.md kvbm/vllm-setup.md kvbm/trtllm-setup.md - guides/tool-calling.md + agents/tool-calling.md - architecture/kv_cache_routing.md + router/kv_cache_routing.md planner/load_planner.md - architecture/request_migration.md - architecture/request_cancellation.md + fault_tolerance/request_migration.md + fault_tolerance/request_cancellation.md backends/trtllm/multinode/multinode-examples.md backends/trtllm/multinode/multinode-multimodal-example.md @@ -66,14 +65,16 @@ examples/README.md examples/runtime/hello_world/README.md - architecture/distributed_runtime.md - architecture/dynamo_flow.md + design_docs/distributed_runtime.md + design_docs/dynamo_flow.md backends/vllm/deepseek-r1.md backends/vllm/gpt-oss.md backends/vllm/multi-node.md backends/vllm/prometheus.md + benchmarks/kv-router-ab-testing.md + .. TODO: architecture/distributed_runtime.md and architecture/dynamo_flow.md have some outdated names/references and need a refresh. diff --git a/docs/index.rst b/docs/index.rst index b0fa8a4dca56..22035c10e442 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -43,22 +43,28 @@ Quickstart Quickstart Installation <_sections/installation> Support Matrix - Architecture <_sections/architecture> Examples <_sections/examples> .. toctree:: :hidden: :caption: Kubernetes Deployment - Quickstart (K8s) <../kubernetes/README.md> - Detailed Installation Guide <../kubernetes/installation_guide.md> - Creating Deployments <../kubernetes/create_deployment.md> - API Reference <../kubernetes/api_reference.md> - Dynamo Operator <../kubernetes/dynamo_operator.md> - Metrics <../kubernetes/metrics.md> - Logging <../kubernetes/logging.md> - Multinode <../kubernetes/multinode-deployment.md> - Minikube Setup <../kubernetes/minikube.md> + Deployment Guide <_sections/k8s_deployment> + Observability (K8s) <_sections/k8s_observability> + Multinode <_sections/k8s_multinode> + +.. toctree:: + :hidden: + :caption: User Guides + + Tool Calling + Multimodality Support + Finding Best Initial Configs + Benchmarking + Tuning Disaggregated Performance + Writing Python Workers in Dynamo + Observability (Local) <_sections/observability> + Glossary .. toctree:: :hidden: @@ -71,13 +77,9 @@ Quickstart .. toctree:: :hidden: - :caption: Developer Guide - - Benchmarking Guide - KV Router A/B Testing - SLA Planner (Autoscaling) Quickstart - Logging - Health Checks - Tuning Disaggregated Serving Performance - Writing Python Workers in Dynamo - Glossary + :caption: Design Docs + + Overall Architecture + Architecture Flow + Disaggregated Serving + Distributed Runtime diff --git a/docs/kubernetes/README.md b/docs/kubernetes/README.md index fb2408ba8239..508ec5b2c18a 100644 --- a/docs/kubernetes/README.md +++ b/docs/kubernetes/README.md @@ -15,14 +15,26 @@ See the License for the specific language governing permissions and limitations under the License. --> -# Deploying Inference Graphs to Kubernetes +# Deploying Dynamo on Kubernetes High-level guide to Dynamo Kubernetes deployments. Start here, then dive into specific guides. -## Pre-deployment Checks +## Important Terminology + +**Kubernetes Namespace**: The K8s namespace where your DynamoGraphDeployment resource is created. +- Used for: Resource isolation, RBAC, organizing deployments +- Example: `dynamo-system`, `dynamo-cloud`, `team-a-namespace` + +**Dynamo Namespace**: The logical namespace used by Dynamo components for service discovery via etcd. +- Used for: Runtime component communication, service discovery +- Specified in: `.spec.services..dynamoNamespace` field +- Example: `my-llm`, `production-model`, `dynamo-dev` -Before deploying the platform, it is recommended to run the pre-deployment checks to ensure the cluster is ready for deployment. Please refer to the [pre-deployment checks](/deploy/cloud/pre-deployment/README.md) for more details. +These are independent. A single Kubernetes namespace can host multiple Dynamo namespaces, and vice versa. +## Pre-deployment Checks + +Before deploying the platform, it is recommended to run the pre-deployment checks to ensure the cluster is ready for deployment. Please refer to the [pre-deployment checks](../../deploy/cloud/pre-deployment/README.md) for more details. ## 1. Install Platform First @@ -31,7 +43,7 @@ Before deploying the platform, it is recommended to run the pre-deployment check export NAMESPACE=dynamo-system export RELEASE_VERSION=0.x.x # any version of Dynamo 0.3.2+ listed at https://github.com/ai-dynamo/dynamo/releases -# 2. Install CRDs +# 2. Install CRDs (skip if on shared cluster where CRDs already exist) helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-crds-${RELEASE_VERSION}.tgz helm install dynamo-crds dynamo-crds-${RELEASE_VERSION}.tgz --namespace default @@ -40,22 +52,29 @@ helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-$ helm install dynamo-platform dynamo-platform-${RELEASE_VERSION}.tgz --namespace ${NAMESPACE} --create-namespace ``` -For more details or customization options (including multinode deployments), see **[Installation Guide for Dynamo Kubernetes Platform](/docs/kubernetes/installation_guide.md)**. +**For Shared/Multi-Tenant Clusters:** + +If your cluster has namespace-restricted Dynamo operators, add this flag to step 3: +```bash +--set dynamo-operator.namespaceRestriction.enabled=true +``` + +For more details or customization options (including multinode deployments), see **[Installation Guide for Dynamo Kubernetes Platform](./installation_guide.md)**. ## 2. Choose Your Backend Each backend has deployment examples and configuration options: -| Backend | Available Configurations | -|---------|--------------------------| -| **[vLLM](/components/backends/vllm/deploy/README.md)** | Aggregated, Aggregated + Router, Disaggregated, Disaggregated + Router, Disaggregated + Planner, Disaggregated Multi-node | -| **[SGLang](/components/backends/sglang/deploy/README.md)** | Aggregated, Aggregated + Router, Disaggregated, Disaggregated + Planner, Disaggregated Multi-node | -| **[TensorRT-LLM](/components/backends/trtllm/deploy/README.md)** | Aggregated, Aggregated + Router, Disaggregated, Disaggregated + Router, Disaggregated Multi-node | +| Backend | Aggregated | Aggregated + Router | Disaggregated | Disaggregated + Router | Disaggregated + Planner | Disaggregated Multi-node | +|--------------|:----------:|:-------------------:|:-------------:|:----------------------:|:-----------------------:|:------------------------:| +| **[SGLang](../../components/backends/sglang/deploy/README.md)** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| **[TensorRT-LLM](../../components/backends/trtllm/deploy/README.md)** | ✅ | ✅ | ✅ | ✅ | 🚧 | ✅ | +| **[vLLM](../../components/backends/vllm/deploy/README.md)** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ## 3. Deploy Your First Model ```bash -export NAMESPACE=dynamo-cloud +export NAMESPACE=dynamo-system kubectl create namespace ${NAMESPACE} # to pull model from HF @@ -75,6 +94,8 @@ kubectl port-forward svc/vllm-agg-frontend 8000:8000 -n ${NAMESPACE} curl http://localhost:8000/v1/models ``` +For SLA-based autoscaling, see [SLA Planner Quick Start Guide](../planner/sla_planner_quickstart.md). + ## Understanding Dynamo's Custom Resources Dynamo provides two main Kubernetes Custom Resources for deploying models: @@ -103,15 +124,15 @@ A lower-level interface that defines your complete inference pipeline: Use this when you need fine-grained control or have already completed profiling. -Refer to the [API Reference and Documentation](/docs/kubernetes/api_reference.md) for more details. +Refer to the [API Reference and Documentation](./api_reference.md) for more details. ## 📖 API Reference & Documentation For detailed technical specifications of Dynamo's Kubernetes resources: -- **[API Reference](/docs/kubernetes/api_reference.md)** - Complete CRD field specifications for all Dynamo resources -- **[Create Deployment](/docs/kubernetes/create_deployment.md)** - Step-by-step deployment creation with DynamoGraphDeployment -- **[Operator Guide](/docs/kubernetes/dynamo_operator.md)** - Dynamo operator configuration and management +- **[API Reference](./api_reference.md)** - Complete CRD field specifications for all Dynamo resources +- **[Create Deployment](./deployment/create_deployment.md)** - Step-by-step deployment creation with DynamoGraphDeployment +- **[Operator Guide](./dynamo_operator.md)** - Dynamo operator configuration and management ### Choosing Your Architecture Pattern @@ -194,13 +215,13 @@ Key customization points include: ## Additional Resources -- **[Examples](/examples/README.md)** - Complete working examples -- **[Create Custom Deployments](/docs/kubernetes/create_deployment.md)** - Build your own CRDs -- **[Operator Documentation](/docs/kubernetes/dynamo_operator.md)** - How the platform works -- **[Helm Charts](/deploy/helm/README.md)** - For advanced users -- **[GitOps Deployment with FluxCD](/docs/kubernetes/fluxcd.md)** - For advanced users -- **[Logging](/docs/kubernetes/logging.md)** - For logging setup -- **[Multinode Deployment](/docs/kubernetes/multinode-deployment.md)** - For multinode deployment -- **[Grove](/docs/kubernetes/grove.md)** - For grove details and custom installation -- **[Monitoring](/docs/kubernetes/metrics.md)** - For monitoring setup -- **[Model Caching with Fluid](/docs/kubernetes/model_caching_with_fluid.md)** - For model caching with Fluid +- **[Examples](../examples/README.md)** - Complete working examples +- **[Create Custom Deployments](./deployment/create_deployment.md)** - Build your own CRDs +- **[Operator Documentation](./dynamo_operator.md)** - How the platform works +- **[Helm Charts](../../deploy/helm/README.md)** - For advanced users +- **[GitOps Deployment with FluxCD](./fluxcd.md)** - For advanced users +- **[Logging](./observability/logging.md)** - For logging setup +- **[Multinode Deployment](./deployment/multinode-deployment.md)** - For multinode deployment +- **[Grove](./grove.md)** - For grove details and custom installation +- **[Monitoring](./observability/metrics.md)** - For monitoring setup +- **[Model Caching with Fluid](./model_caching_with_fluid.md)** - For model caching with Fluid diff --git a/docs/kubernetes/create_deployment.md b/docs/kubernetes/deployment/create_deployment.md similarity index 64% rename from docs/kubernetes/create_deployment.md rename to docs/kubernetes/deployment/create_deployment.md index 3d552ad5bbca..a0d2877d0a64 100644 --- a/docs/kubernetes/create_deployment.md +++ b/docs/kubernetes/deployment/create_deployment.md @@ -1,25 +1,79 @@ # Creating Kubernetes Deployments The scripts in the `components//launch` folder like [agg.sh](../../../components/backends/vllm/launch/agg.sh) demonstrate how you can serve your models locally. -The corresponding YAML files like [agg.yaml](../../../components/backends/vllm/deploy/agg.yaml) show you how you could create a kubernetes deployment for your inference graph. - +The corresponding YAML files like [agg.yaml](../../../components/backends/vllm/deploy/agg.yaml) show you how you could create a Kubernetes deployment for your inference graph. This guide explains how to create your own deployment files. ## Step 1: Choose Your Architecture Pattern +Before choosing a template, understand the different architecture patterns: + +### Aggregated Serving (agg.yaml) + +**Pattern**: Prefill and decode on the same GPU in a single process. + +**Suggested to use for**: +- Small to medium models (under 70B parameters) +- Development and testing +- Low to moderate traffic +- Simplicity is prioritized over maximum throughput + +**Tradeoffs**: +- Simpler setup and debugging +- Lower operational complexity +- GPU utilization may not be optimal (prefill and decode compete for resources) +- Lower throughput ceiling compared to disaggregated + +**Example**: [`agg.yaml`](../../../components/backends/vllm/deploy/agg.yaml) + +### Aggregated + Router (agg_router.yaml) + +**Pattern**: Load balancer routing across multiple aggregated worker instances. + +**Suggested to use for**: +- Medium traffic requiring high availability +- Need horizontal scaling +- Want some load balancing without disaggregation complexity + +**Tradeoffs**: +- Better scalability than plain aggregated +- High availability through multiple replicas +- Still has GPU underutilization issues of aggregated serving +- More complex than plain aggregated but simpler than disaggregated + +**Example**: [`agg_router.yaml`](../../../components/backends/vllm/deploy/agg_router.yaml) + +### Disaggregated Serving (disagg_router.yaml) + +**Pattern**: Separate prefill and decode workers with specialized optimization. + +**Suggested to use for**: +- Production-style deployments +- High throughput requirements +- Large models (70B+ parameters) +- Maximum GPU utilization needed + +**Tradeoffs**: +- Maximum performance and throughput +- Better GPU utilization (prefill and decode specialized) +- Independent scaling of prefill and decode +- More complex setup and debugging +- Requires understanding of prefill/decode separation + +**Example**: [`disagg_router.yaml`](../../../components/backends/vllm/deploy/disagg_router.yaml) + +### Quick Selection Guide + Select the architecture pattern as your template that best fits your use case. -For example, when using the `VLLM` inference backend: +For example, when using the `vLLM` backend: -- **Development / Testing** - Use [`agg.yaml`](/components/backends/vllm/deploy/agg.yaml) as the base configuration. +- **Development / Testing**: Use [`agg.yaml`](../../../components/backends/vllm/deploy/agg.yaml) as the base configuration. -- **Production with Load Balancing** - Use [`agg_router.yaml`](/components/backends/vllm/deploy/agg_router.yaml) to enable scalable, load-balanced inference. +- **Production with Load Balancing**: Use [`agg_router.yaml`](../../../components/backends/vllm/deploy/agg_router.yaml) to enable scalable, load-balanced inference. -- **High Performance / Disaggregated Deployment** - Use [`disagg_router.yaml`](/components/backends/vllm/deploy/disagg_router.yaml) for maximum throughput and modular scalability. +- **High Performance / Disaggregated Deployment**: Use [`disagg_router.yaml`](../../../components/backends/vllm/deploy/disagg_router.yaml) for maximum throughput and modular scalability. ## Step 2: Customize the Template @@ -90,7 +144,7 @@ Consult the corresponding sh file. Each of the python commands to launch a compo The front end is launched with "python3 -m dynamo.frontend [--http-port 8000] [--router-mode kv]" Each worker will launch `python -m dynamo.YOUR_INFERENCE_BACKEND --model YOUR_MODEL --your-flags `command. -If you are a Dynamo contributor the [dynamo run guide](/docs/reference/cli.md) for details on how to run this command. +If you are a Dynamo contributor the [dynamo run guide](../../reference/cli.md) for details on how to run this command. ## Step 3: Key Customization Points diff --git a/docs/kubernetes/minikube.md b/docs/kubernetes/deployment/minikube.md similarity index 94% rename from docs/kubernetes/minikube.md rename to docs/kubernetes/deployment/minikube.md index 6047353dc165..7a4a1cd65767 100644 --- a/docs/kubernetes/minikube.md +++ b/docs/kubernetes/deployment/minikube.md @@ -58,5 +58,5 @@ kubectl get storageclass ## Next Steps -Once your local environment is set up, you can proceed with the [Dynamo Kubernetes Platform installation guide](./installation_guide.md) to deploy the platform to your local cluster. +Once your local environment is set up, you can proceed with the [Dynamo Kubernetes Platform installation guide](../installation_guide.md) to deploy the platform to your local cluster. diff --git a/docs/kubernetes/multinode-deployment.md b/docs/kubernetes/deployment/multinode-deployment.md similarity index 100% rename from docs/kubernetes/multinode-deployment.md rename to docs/kubernetes/deployment/multinode-deployment.md diff --git a/docs/kubernetes/dynamo_operator.md b/docs/kubernetes/dynamo_operator.md index be5c9e5de754..effa1b3b322b 100644 --- a/docs/kubernetes/dynamo_operator.md +++ b/docs/kubernetes/dynamo_operator.md @@ -23,11 +23,57 @@ Dynamo operator is a Kubernetes operator that simplifies the deployment, configu For the complete technical API reference for Dynamo Custom Resource Definitions, see: -**📖 [Dynamo CRD API Reference](/docs/kubernetes/api_reference.md)** +**📖 [Dynamo CRD API Reference](./api_reference.md)** ## Installation -[See installation steps](/docs/kubernetes/installation_guide.md#overview) +### Quick Install with Helm + +```bash +# Set environment +export NAMESPACE=dynamo-system +export RELEASE_VERSION=0.x.x # any version of Dynamo 0.3.2+ listed at https://github.com/ai-dynamo/dynamo/releases + +# Install Platform (includes operator) +helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-${RELEASE_VERSION}.tgz +helm install dynamo-platform dynamo-platform-${RELEASE_VERSION}.tgz --namespace ${NAMESPACE} --create-namespace +``` + +For namespace-restricted installations (shared clusters): +```bash +helm install dynamo-platform dynamo-platform-${RELEASE_VERSION}.tgz \ + --namespace ${NAMESPACE} \ + --create-namespace \ + --set dynamo-operator.namespaceRestriction.enabled=true +``` + +### Building from Source + +```bash +# Set environment +export NAMESPACE=dynamo-system +export DOCKER_SERVER=your-registry.com/ # your container registry +export IMAGE_TAG=latest + +# Build operator image +cd deploy/cloud/operator +docker build -t $DOCKER_SERVER/dynamo-operator:$IMAGE_TAG . +docker push $DOCKER_SERVER/dynamo-operator:$IMAGE_TAG +cd - + +# Install CRDs +cd deploy/cloud/helm +helm install dynamo-crds ./crds/ --namespace default + +# Install platform with custom operator image +helm install dynamo-platform ./platform/ \ + --namespace ${NAMESPACE} \ + --create-namespace \ + --set "dynamo-operator.controllerManager.manager.image.repository=${DOCKER_SERVER}/dynamo-operator" \ + --set "dynamo-operator.controllerManager.manager.image.tag=${IMAGE_TAG}" +``` + +For detailed installation options, see the [Installation Guide](./installation_guide.md) ## Development diff --git a/docs/kubernetes/fluxcd.md b/docs/kubernetes/fluxcd.md index 013e3460566f..9487afc5cc10 100644 --- a/docs/kubernetes/fluxcd.md +++ b/docs/kubernetes/fluxcd.md @@ -1,10 +1,10 @@ # GitOps Deployment with FluxCD -This section describes how to use FluxCD for GitOps-based deployment of Dynamo inference graphs. GitOps enables you to manage your Dynamo deployments declaratively using Git as the source of truth. We'll use the [aggregated vLLM example](/docs/backends/vllm/README.md) to demonstrate the workflow. +This section describes how to use FluxCD for GitOps-based deployment of Dynamo inference graphs. GitOps enables you to manage your Dynamo deployments declaratively using Git as the source of truth. We'll use the [aggregated vLLM example](../backends/vllm/README.md) to demonstrate the workflow. ## Prerequisites -- A Kubernetes cluster with [Dynamo Cloud](/docs/kubernetes/installation_guide.md) installed +- A Kubernetes cluster with [Dynamo Cloud](./installation_guide.md) installed - [FluxCD](https://fluxcd.io/flux/installation/) installed in your cluster - A Git repository to store your deployment configurations @@ -18,7 +18,7 @@ The GitOps workflow for Dynamo deployments consists of three main steps: ## Step 1: Build and Push Dynamo Cloud Operator -First, follow to [See Install Dynamo Cloud](/docs/kubernetes/installation_guide.md). +First, follow to [See Install Dynamo Cloud](./installation_guide.md). ## Step 2: Create Initial Deployment diff --git a/docs/kubernetes/gke_setup.md b/docs/kubernetes/gke_setup.md deleted file mode 100644 index feb2c9c393a0..000000000000 --- a/docs/kubernetes/gke_setup.md +++ /dev/null @@ -1,169 +0,0 @@ -# GKE Workload Identity and Artifact Registry Setup Guide - -This guide explains how to set up Workload Identity in GKE and configure access to Google Artifact Registry. - -## Prerequisites - -- Google Cloud SDK installed -- Access to a GKE cluster -- Required permissions to create and manage service accounts - -## Project Setup - -Set your project: -```bash -export NAMESPACE=your-k8s-namespace -export RELEASE=your-helm-release-name - -export PROJECT=$(gcloud config get-value project) -# set the cluster related info (you can list cluster using gcloud container clusters list) -export CLUSTER_NAME=your-cluster-name -export CLUSTER_REGION=$(gcloud container clusters list --filter="name=${CLUSTER_NAME}" --format="get(location)") -gcloud config set project ${PROJECT} -# Retrieve the Workload Identifier Namespace associated with your cluster: -export CLUSTER_WIN=$(gcloud container clusters describe ${CLUSTER_NAME} \ - --region=${CLUSTER_REGION} \ - --format="value(workloadIdentityConfig.workloadPool)") -``` - -```{important} -Make sure Workload Identity is enabled in your cluster! -``` - - -## Service Account Creation and Configuration - -1. Create a service account for Workload Identity: - -Go to the GCP console and create a new service account (or reuse an existing one) - -```bash -gcloud iam service-accounts create workload-identity-sa\ - --display-name="workload identity service account" \ - --description="Service account to use for Workload Identity in GKE" -export SA=workload-identity-sa@${PROJECT}.iam.gserviceaccount.com -``` - -2. Configure Workload Identity bindings for Kubernetes service accounts: -```bash - -gcloud iam service-accounts add-iam-policy-binding \ - ${SA} \ - --role roles/iam.workloadIdentityUser \ - --member "serviceAccount:${CLUSTER_WIN}[${NAMESPACE}/${RELEASE}-dynamo-operator-controller-manager]" - -gcloud iam service-accounts add-iam-policy-binding \ - ${SA} \ - --role roles/iam.workloadIdentityUser \ - --member "serviceAccount:${CLUSTER_WIN}[${NAMESPACE}/${RELEASE}-dynamo-operator-image-builder]" - -gcloud iam service-accounts add-iam-policy-binding \ - ${SA} \ - --role roles/iam.workloadIdentityUser \ - --member "serviceAccount:${CLUSTER_WIN}[${NAMESPACE}/${RELEASE}-dynamo-operator-component]" -``` - -## Artifact Registry Access - -### Option 1: Project-Level Access - -Grant read and write access at the project level: -```bash -# Grant reader role -gcloud projects add-iam-policy-binding ${PROJECT} \ - --member="serviceAccount:${SA}" \ - --role="roles/artifactregistry.reader" - -# Grant writer role -gcloud projects add-iam-policy-binding ${PROJECT} \ - --member="serviceAccount:${SA}" \ - --role="roles/artifactregistry.writer" -``` - -### Option 2: Repository-Level Access - -Grant access to specific repository: -```bash -gcloud artifacts repositories add-iam-policy-binding your-artifact-repository \ - --location=${CLUSTER_REGION} \ - --project=${PROJECT} \ - --member="serviceAccount:${SA}" \ - --role="roles/artifactregistry.reader" -``` - -## GKE Node Access to Artifact Registry - -This is needed to make sure pods can pull images from Artifact Registry without needing to specify an imagePullSecret - -### For GKE Autopilot - -```bash -# Get project number -export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT} --format='value(projectNumber)') - -# Grant access to the default compute service account -gcloud projects add-iam-policy-binding ${PROJECT} \ - --member="serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com" \ - --role="roles/artifactregistry.reader" -``` - -### For Standard GKE - -```bash -# Get node service account -export NODE_SERVICE_ACCOUNT=$(gcloud container clusters describe ${CLUSTER_NAME} \ - --region ${CLUSTER_REGION} \ - --format="get(nodeConfig.serviceAccount)") - -# Grant access to node service account -gcloud projects add-iam-policy-binding ${PROJECT} \ - --member="serviceAccount:${NODE_SERVICE_ACCOUNT}" \ - --role="roles/artifactregistry.reader" -``` - -## Adding annotations to enable Workload Identity - -This is an example of values.yaml used to deploy Dynamo Cloud using custom GCP annotations to enable Workload Identity. - -```yaml - -dynamo-operator: - ... - controllerManager: - serviceAccount: - create: true - annotations: - iam.gke.io/gcp-service-account: your-sa@your-gcp-project.iam.gserviceaccount.com - ... - dynamo: - components: - serviceAccount: - annotations: - iam.gke.io/gcp-service-account: your-sa@your-gcp-project.iam.gserviceaccount.com - ... - -.... -``` - -You can use it during helm installation: - -```bash -helm upgrade --install ${RELEASE} platform/ -f values.yaml --namespace ${NAMESPACE} -``` - -## Important Notes - -1. **Prerequisites for Image Pulling**: - - Workload Identity must be enabled on your GKE cluster - - GKE nodes' service account must have the `artifactregistry.reader` role - -2. **Troubleshooting**: - - If pods can't pull images, verify both Workload Identity and node service account configurations - - Check service account annotations on Kubernetes service accounts - - Verify IAM bindings are correctly set up - -## References - -- [GKE Workload Identity Documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) -- [Artifact Registry Authentication](https://cloud.google.com/artifact-registry/docs/docker/authentication) -- [IAM Roles for Artifact Registry](https://cloud.google.com/artifact-registry/docs/access-control) \ No newline at end of file diff --git a/docs/kubernetes/grove.md b/docs/kubernetes/grove.md index 177f19c0b6c0..5839f1518670 100644 --- a/docs/kubernetes/grove.md +++ b/docs/kubernetes/grove.md @@ -93,8 +93,8 @@ For KAI Scheduler, see the [KAI Scheduler Deployment Guide](https://github.com/N For installation instructions, see the [Grove Installation Guide](https://github.com/NVIDIA/grove/blob/main/docs/installation.md). -For practical examples of Grove-based multinode deployments in action, see the [Multinode Deployment Guide](multinode-deployment.md), which demonstrates multi-node disaggregated serving scenarios. +For practical examples of Grove-based multinode deployments in action, see the [Multinode Deployment Guide](./deployment/multinode-deployment.md), which demonstrates multi-node disaggregated serving scenarios. For the latest updates on Grove, refer to the [official project on GitHub](https://github.com/NVIDIA/grove). -Dynamo Cloud also allows you to install Grove and KAI Scheduler as part of the platform installation. See the [Dynamo Cloud Deployment Installation Guide](installation_guide.md) for more details. \ No newline at end of file +Dynamo Cloud also allows you to install Grove and KAI Scheduler as part of the platform installation. See the [Dynamo Cloud Deployment Installation Guide](./installation_guide.md) for more details. \ No newline at end of file diff --git a/docs/kubernetes/installation_guide.md b/docs/kubernetes/installation_guide.md index fc4ea82aeb20..62505170175c 100644 --- a/docs/kubernetes/installation_guide.md +++ b/docs/kubernetes/installation_guide.md @@ -19,18 +19,42 @@ limitations under the License. Deploy and manage Dynamo inference graphs on Kubernetes with automated orchestration and scaling, using the Dynamo Kubernetes Platform. -## Quick Start Paths +## Before You Start -Platform is installed using Dynamo Kubernetes Platform [helm chart](/deploy/cloud/helm/platform/README.md). +Determine your cluster environment: -**Path A: Production Install** -Install from published artifacts on your existing cluster → [Jump to Path A](#path-a-production-install) +**Shared/Multi-Tenant Cluster** (K8s cluster with existing Dynamo artifacts): +- CRDs already installed cluster-wide - skip CRD installation step +- Must use namespace-restricted installation (see note in installation steps) -**Path B: Local Development** -Set up Minikube first → [Minikube Setup](minikube.md) → Then follow Path A +**Dedicated Cluster** (full cluster admin access): +- You install CRDs yourself +- Can use cluster-wide operator (default) -**Path C: Custom Development** -Build from source for customization → [Jump to Path C](#path-c-custom-development) +**Local Development** (Minikube, testing): +- See [Minikube Setup](deployment/minikube.md) first, then follow installation steps below + +To check if CRDs already exist: +```bash +kubectl get crd | grep dynamo +# If you see dynamographdeployments, dynamocomponentdeployments, etc., CRDs are already installed +``` + +## Installation Paths + +Platform is installed using Dynamo Kubernetes Platform [helm chart](../../deploy/cloud/helm/platform/README.md). + +**Path A: Pre-built Artifacts** +- Use case: Production deployment, shared or dedicated clusters +- Source: NGC published Helm charts +- Time: ~10 minutes +- Jump to: [Path A](#path-a-production-install) + +**Path B: Custom Build from Source** +- Use case: Contributing to Dynamo, using latest features from main branch, customization +- Requirements: Docker build environment +- Time: ~30 minutes +- Jump to: [Path B](#path-b-custom-build-from-source) All helm install commands could be overridden by either setting the values.yaml file or by passing in your own values.yaml: @@ -48,31 +72,39 @@ helm install ... ## Prerequisites +Verify before proceeding: + +- Kubernetes cluster v1.24+ access +- kubectl v1.24+ installed and configured +- Helm v3.0+ installed +- Cluster type determined (shared vs dedicated) +- CRD status checked if on shared cluster +- NGC credentials if using NVIDIA images (optional for public images) + +Estimated time: 5-30 minutes depending on path + ```bash -# Required tools +# Check required tools kubectl version --client # v1.24+ helm version # v3.0+ -docker version # Running daemon +docker version # Running daemon (for Path D only) -# Set your inference runtime image +# Set your release version export RELEASE_VERSION=0.x.x # any version of Dynamo 0.3.2+ listed at https://github.com/ai-dynamo/dynamo/releases -export DYNAMO_IMAGE=nvcr.io/nvidia/ai-dynamo/vllm-runtime:${RELEASE_VERSION} -# Also available: sglang-runtime, tensorrtllm-runtime ``` -> [!TIP] -> No cluster? See [Minikube Setup](minikube.md) for local development. +> No cluster? See [Minikube Setup](deployment/minikube.md) for local development. ## Path A: Production Install -Install from [NGC published artifacts](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-dynamo/collections/ai-dynamo/artifacts) in 3 steps. +Install from [NGC published artifacts](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-dynamo/collections/ai-dynamo/artifacts). ```bash # 1. Set environment export NAMESPACE=dynamo-system export RELEASE_VERSION=0.x.x # any version of Dynamo 0.3.2+ listed at https://github.com/ai-dynamo/dynamo/releases -# 2. Install CRDs +# 2. Install CRDs (skip if on shared cluster where CRDs already exist) helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-crds-${RELEASE_VERSION}.tgz helm install dynamo-crds dynamo-crds-${RELEASE_VERSION}.tgz --namespace default @@ -81,10 +113,27 @@ helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-$ helm install dynamo-platform dynamo-platform-${RELEASE_VERSION}.tgz --namespace ${NAMESPACE} --create-namespace ``` +**For Shared/Multi-Tenant Clusters:** + +If your cluster has namespace-restricted Dynamo operators, you MUST add namespace restriction to your installation: + +```bash +# Add this flag to the helm install command above +--set dynamo-operator.namespaceRestriction.enabled=true +``` + +Note: Use the full path `dynamo-operator.namespaceRestriction.enabled=true` (not just `namespaceRestriction.enabled=true`). + +If you see this validation error, you need namespace restriction: +``` +VALIDATION ERROR: Cannot install cluster-wide Dynamo operator. +Found existing namespace-restricted Dynamo operators in namespaces: ... +``` + > [!TIP] -> For multinode deployments, you need to enable Grove and Kai Scheduler. +> For multinode deployments, you need to enable Grove and KAI Scheduler. > You might chose to install them manually or through the dynamo-platform helm install command. -> When using the dynamo-platform helm install command, Grove and Kai Scheduler are NOT installed by default. You can enable their installation by setting the following flags in the helm install command: +> When using the dynamo-platform helm install command, Grove and KAI Scheduler are NOT installed by default. You can enable their installation by setting the following flags in the helm install command: ```bash --set "grove.enabled=true" @@ -111,9 +160,11 @@ helm install dynamo-platform dynamo-platform-${RELEASE_VERSION}.tgz --namespace → [Verify Installation](#verify-installation) -## Path B: Custom Development +## Path B: Custom Build from Source + +Build and deploy from source for customization, contributing to Dynamo, or using the latest features from the main branch. -Build and deploy from source for customization. +Note: This gives you access to the latest unreleased features and fixes on the main branch. ```bash # 1. Set environment @@ -190,16 +241,43 @@ kubectl get pods -n ${NAMESPACE} ``` 2. **Explore Backend Guides** - - [vLLM Deployments](/components/backends/vllm/deploy/README.md) - - [SGLang Deployments](/components/backends/sglang/deploy/README.md) - - [TensorRT-LLM Deployments](/components/backends/trtllm/deploy/README.md) + - [vLLM Deployments](../../components/backends/vllm/deploy/README.md) + - [SGLang Deployments](../../components/backends/sglang/deploy/README.md) + - [TensorRT-LLM Deployments](../../components/backends/trtllm/deploy/README.md) 3. **Optional:** - - [Set up Prometheus & Grafana](metrics.md) + - [Set up Prometheus & Grafana](./observability/metrics.md) - [SLA Planner Quickstart Guide](../planner/sla_planner_quickstart.md) (for SLA-aware scheduling and autoscaling) ## Troubleshooting +**"VALIDATION ERROR: Cannot install cluster-wide Dynamo operator"** + +``` +VALIDATION ERROR: Cannot install cluster-wide Dynamo operator. +Found existing namespace-restricted Dynamo operators in namespaces: ... +``` + +Cause: Attempting cluster-wide install on a shared cluster with existing namespace-restricted operators. + +Solution: Add namespace restriction to your installation: +```bash +--set dynamo-operator.namespaceRestriction.enabled=true +``` + +Note: Use the full path `dynamo-operator.namespaceRestriction.enabled=true` (not just `namespaceRestriction.enabled=true`). + +**CRDs already exist** + +Cause: Installing CRDs on a cluster where they're already present (common on shared clusters). + +Solution: Skip step 2 (CRD installation), proceed directly to platform installation. + +To check if CRDs exist: +```bash +kubectl get crd | grep dynamo +``` + **Pods not starting?** ```bash kubectl describe pod -n ${NAMESPACE} @@ -232,8 +310,7 @@ just add the following to the helm install command: ## Advanced Options -- [Helm Chart Configuration](/deploy/cloud/helm/platform/README.md) -- [GKE-specific setup](gke_setup.md) -- [Create custom deployments](create_deployment.md) -- [Dynamo Operator details](dynamo_operator.md) +- [Helm Chart Configuration](../../deploy/cloud/helm/platform/README.md) +- [Create custom deployments](./deployment/create_deployment.md) +- [Dynamo Operator details](./dynamo_operator.md) - [Model Express Server details](https://github.com/ai-dynamo/modelexpress) diff --git a/docs/kubernetes/logging.md b/docs/kubernetes/observability/logging.md similarity index 98% rename from docs/kubernetes/logging.md rename to docs/kubernetes/observability/logging.md index 802dcc5dbdf5..abce5b81b890 100644 --- a/docs/kubernetes/logging.md +++ b/docs/kubernetes/observability/logging.md @@ -17,7 +17,7 @@ This guide demonstrates how to set up logging for Dynamo in Kubernetes using Gra ### 1. Dynamo Cloud Kubernetes Operator -This guide assumes you have installed Dynamo Cloud Kubernetes Operator. For more information, see [Dynamo Cloud Operator](./README.md). +This guide assumes you have installed Dynamo Cloud Kubernetes Operator. For more information, see [Dynamo Cloud Operator](../README.md). ### 2. Kube-prometheus diff --git a/docs/kubernetes/metrics.md b/docs/kubernetes/observability/metrics.md similarity index 93% rename from docs/kubernetes/metrics.md rename to docs/kubernetes/observability/metrics.md index a7e31572ee64..8e241760f254 100644 --- a/docs/kubernetes/metrics.md +++ b/docs/kubernetes/observability/metrics.md @@ -28,7 +28,7 @@ helm install prometheus -n monitoring --create-namespace prometheus-community/ku > The commands enumerated below assume you have installed the kube-prometheus-stack with the installation method listed above. Depending on your installation configuration of the monitoring stack, you may need to modify the `kubectl` commands that follow in this document accordingly (e.g modifying Namespace or Service names accordingly). ### Install Dynamo Operator -Before setting up metrics collection, you'll need to have the Dynamo operator installed in your cluster. Follow our [Installation Guide](/docs/kubernetes/installation_guide.md) for detailed instructions on deploying the Dynamo operator. +Before setting up metrics collection, you'll need to have the Dynamo operator installed in your cluster. Follow our [Installation Guide](../installation_guide.md) for detailed instructions on deploying the Dynamo operator. Make sure to set the `prometheusEndpoint` to the Prometheus endpoint you installed in the previous step. ```bash @@ -53,7 +53,7 @@ If the output is empty, you need to install the dcgm-exporter. For more informat Let's start by deploying a simple vLLM aggregated deployment: ```bash -export NAMESPACE=dynamo # namespace where dynamo operator is installed +export NAMESPACE=dynamo-system # namespace where dynamo operator is installed pushd components/backends/vllm/deploy kubectl apply -f agg.yaml -n $NAMESPACE popd @@ -64,8 +64,8 @@ This will create two components: - A Worker component exposing metrics on its system port Both components expose a `/metrics` endpoint following the OpenMetrics format, but with different metrics appropriate to their roles. For details about: -- Deployment configuration: See the [vLLM README](/docs/backends/vllm/README.md) -- Available metrics: See the [metrics guide](/docs/observability/metrics.md) +- Deployment configuration: See the [vLLM README](../../backends/vllm/README.md) +- Available metrics: See the [metrics guide](../../observability/metrics.md) ### Validate the Deployment @@ -87,7 +87,7 @@ curl localhost:8000/v1/chat/completions \ }' ``` -For more information about validating the deployment, see the [vLLM README](../backends/vllm/README.md). +For more information about validating the deployment, see the [vLLM README](../../backends/vllm/README.md). ## Set Up Metrics Collection @@ -137,7 +137,7 @@ Visit http://localhost:9090 and try these example queries: - `dynamo_frontend_requests_total` - `dynamo_frontend_time_to_first_token_seconds_bucket` -![Prometheus UI showing Dynamo metrics](../images/prometheus-k8s.png) +![Prometheus UI showing Dynamo metrics](../../images/prometheus-k8s.png) ### In Grafana ```bash @@ -155,4 +155,4 @@ Visit http://localhost:3000 and log in with the credentials captured above. Once logged in, find the Dynamo dashboard under General. -![Grafana dashboard showing Dynamo metrics](../images/grafana-k8s.png) +![Grafana dashboard showing Dynamo metrics](../../images/grafana-k8s.png) diff --git a/docs/multimodal/multimodal_intro.md b/docs/multimodal/multimodal_intro.md new file mode 100644 index 000000000000..500ace092fab --- /dev/null +++ b/docs/multimodal/multimodal_intro.md @@ -0,0 +1,53 @@ + + +# Multimodal Inference in Dynamo: + +You can find example workflows and reference implementations for deploying a multimodal model using Dynamo in [multimodal examples](https://github.com/ai-dynamo/dynamo/tree/main/examples/multimodal). + +## EPD vs. PD Disaggregation +Dynamo supports two primary approaches for processing multimodal inputs, which differ in how the initial media encoding step is handled relative to the main LLM inference engine. + +### 1. EPD (Encode-Prefill-Decode) Disaggregation +The EPD approach introduces an explicit separation of the media encoding step, maximizing the utilization of specialized hardware and increasing overall system efficiency for large multimodal models. + +* **Media Input:** Image, video, audio, or an embedding URL is provided. +* **Process Flow:** + 1. A dedicated **Encode Worker** is launched separately to handle the embedding extraction from the media input. + 2. The extracted embeddings are transferred to the main engine via the **NVIDIA Inference Xfer Library (NIXL)**. + 3. The main **Engine** performs the remaining **Prefill Decode Disaggregation** steps to generate the output. +* **Benefit:** This disaggregation allows for the decoupling of media encoding hardware/resources from the main LLM serving engine, making the serving of large multimodal models more efficient. + +### 2. PD (Prefill-Decode) Disaggregation + +The PD approach is a more traditional, aggregated method where the inference engine handles the entire process. +* **Media Input:** Image, video, or audio is loaded. +* **Process Flow:** + 1. The main **Engine** receives the media input. + 2. The Engine executes the full sequence: **Encode + Prefill + Decode**. +* **Note:** In this approach, the encoding step is executed within the same pipeline as the prefill and decode phases. + +## Inference Framework Support Matrix + +Dynamo supports multimodal capabilities across leading LLM inference backends, including **vLLM**, **TensorRT-LLM (TRT-LLM)**, and **SGLang**. The table below details the current support level for EPD/PD and various media types for each stack. + +| Stack | EPD Support | PD Support | Image | Video | Audio | +| --------- | --------- | --------- | --------- |---------| --------- | +| **vLLM** | ✅ | ✅ | ✅ | ✅ | 🚧 | +| **TRT-LLM** | ✅ (Currently via precomputed Embeddings URL) | ✅ | ✅ | ❌ | ❌ | +| **SGLang** | ✅ | ❌ | ✅ | ❌ | ❌ | diff --git a/docs/observability/health-checks.md b/docs/observability/health-checks.md index 8405c964eb16..9e77f3202b18 100644 --- a/docs/observability/health-checks.md +++ b/docs/observability/health-checks.md @@ -195,6 +195,6 @@ date: Wed, 03 Sep 2025 13:42:45 GMT ## Related Documentation -- [Distributed Runtime Architecture](../architecture/distributed_runtime.md) -- [Dynamo Architecture Overview](../architecture/architecture.md) +- [Distributed Runtime Architecture](../design_docs/distributed_runtime.md) +- [Dynamo Architecture Overview](../design_docs/architecture.md) - [Backend Guide](../development/backend-guide.md) diff --git a/docs/observability/logging.md b/docs/observability/logging.md index 668902bc5f2b..6b28061170dc 100644 --- a/docs/observability/logging.md +++ b/docs/observability/logging.md @@ -185,7 +185,7 @@ curl -d '{"model": "Qwen/Qwen3-0.6B", "max_completion_tokens": 2049, "messages": ## Related Documentation -- [Distributed Runtime Architecture](../architecture/distributed_runtime.md) -- [Dynamo Architecture Overview](../architecture/architecture.md) +- [Distributed Runtime Architecture](../design_docs/distributed_runtime.md) +- [Dynamo Architecture Overview](../design_docs/architecture.md) - [Backend Guide](../development/backend-guide.md) -- [Log Aggregation in Kubernetes](../kubernetes/logging.md) +- [Log Aggregation in Kubernetes](../kubernetes/observability/logging.md) diff --git a/docs/observability/metrics.md b/docs/observability/metrics.md index adbc63618773..26897f19ed08 100644 --- a/docs/observability/metrics.md +++ b/docs/observability/metrics.md @@ -31,7 +31,7 @@ Dynamo automatically exposes metrics with the `dynamo_` name prefixes. It also a **Specialized Component Metrics**: Components can also expose additional metrics specific to their functionality. For example, a `preprocessor` component exposes metrics with the `dynamo_preprocessor_*` prefix. See the [Available Metrics section](../../deploy/metrics/README.md#available-metrics) for details on specialized component metrics. -**Kubernetes Integration**: For comprehensive Kubernetes deployment and monitoring setup, see the [Kubernetes Metrics Guide](../kubernetes/metrics.md). This includes Prometheus Operator setup, metrics collection configuration, and visualization in Grafana. +**Kubernetes Integration**: For comprehensive Kubernetes deployment and monitoring setup, see the [Kubernetes Metrics Guide](../kubernetes/observability/metrics.md). This includes Prometheus Operator setup, metrics collection configuration, and visualization in Grafana. ## Metrics Hierarchy @@ -94,8 +94,8 @@ The metrics system includes a pre-configured Grafana dashboard for visualizing s ## Related Documentation -- [Distributed Runtime Architecture](../architecture/distributed_runtime.md) -- [Dynamo Architecture Overview](../architecture/architecture.md) +- [Distributed Runtime Architecture](../design_docs/distributed_runtime.md) +- [Dynamo Architecture Overview](../design_docs/architecture.md) - [Backend Guide](../development/backend-guide.md) - [Metrics Implementation Examples](../../deploy/metrics/README.md#implementation-examples) - [Complete Metrics Setup Guide](../../deploy/metrics/README.md) \ No newline at end of file diff --git a/docs/performance/aiconfigurator.md b/docs/performance/aiconfigurator.md new file mode 100644 index 000000000000..353bd23001a9 --- /dev/null +++ b/docs/performance/aiconfigurator.md @@ -0,0 +1,155 @@ + + +# Finding Best Initial Configs using AIConfigurator + +[AIConfigurator](https://github.com/ai-dynamo/aiconfigurator/tree/main) is a performance optimization tool that helps you find the optimal configuration for deploying LLMs with Dynamo. It automatically determines the best number of prefill and decode workers, parallelism settings, and deployment parameters to meet your SLA targets while maximizing throughput. + +## Why Use AIConfigurator? + +When deploying LLMs with Dynamo, you need to make several critical decisions: +- **Aggregated vs Disaggregated**: Which architecture gives better performance for your workload? +- **Worker Configuration**: How many prefill and decode workers to deploy? +- **Parallelism Settings**: What tensor/pipeline parallel configuration to use? +- **SLA Compliance**: How to meet your TTFT and TPOT targets? + +AIConfigurator answers these questions in seconds, providing: +- Optimal configurations that meet your SLA requirements +- Ready-to-deploy Dynamo configuration files +- Performance comparisons between different deployment strategies +- Up to 1.7x better throughput compared to manual configuration + +## Quick Start + +```bash +# Install +pip3 install aiconfigurator + +# Find optimal configuration +aiconfigurator cli default \ + --model QWEN3_32B \ # Model name (QWEN3_32B, LLAMA3.1_70B, etc.) + --total_gpus 32 \ # Number of available GPUs + --system h200_sxm \ # GPU type (h100_sxm, h200_sxm, a100_sxm) + --isl 4000 \ # Input sequence length (tokens) + --osl 500 \ # Output sequence length (tokens) + --ttft 300 \ # Target Time To First Token (ms) + --tpot 10 \ # Target Time Per Output Token (ms) + --save_dir ./dynamo-configs + +# Deploy +kubectl apply -f ./dynamo-configs/disagg/top1/disagg/k8s_deploy.yaml +``` + +## Example Output + +```text +******************************************************************************** +* Dynamo aiconfigurator Final Results * +******************************************************************************** + ---------------------------------------------------------------------------- + Input Configuration & SLA Target: + Model: QWEN3_32B (is_moe: False) + Total GPUs: 32 + Best Experiment Chosen: disagg at 812.92 tokens/s/gpu (1.70x better) + ---------------------------------------------------------------------------- + Overall Best Configuration: + - Best Throughput: 812.92 tokens/s/gpu + - User Throughput: 120.23 tokens/s/user + - TTFT: 276.76ms + - TPOT: 8.32ms + ---------------------------------------------------------------------------- + Pareto Frontier: + QWEN3_32B Pareto Frontier: tokens/s/gpu vs tokens/s/user + ┌────────────────────────────────────────────────────────────────────────┐ +1600.0┤ •• disagg │ + │ ff agg │ + │ xx disagg best │ + │ │ +1333.3┤ f │ + │ ff │ + │ ff • │ + │ f •••••••• │ +1066.7┤ f •• │ + │ fff •••••••• │ + │ f •• │ + │ f •••• │ + 800.0┤ fffff •••x │ + │ fff •• │ + │ fff • │ + │ fffff •• │ + 533.3┤ ffff •• │ + │ ffff •• │ + │ fffffff ••••• │ + │ ffffff •• │ + 266.7┤ fffff ••••••••• │ + │ ffffffffff │ + │ f │ + │ │ + 0.0┤ │ + └┬─────────────────┬─────────────────┬────────────────┬─────────────────┬┘ + 0 60 120 180 240 +tokens/s/gpu tokens/s/user + +1. **Performance Comparison**: Shows disaggregated vs aggregated serving performance +2. **Optimal Configuration**: The best configuration that meets your SLA targets +3. **Deployment Files**: Ready-to-use Dynamo configuration files + +## Key Features + +### Fast Profiling Integration +```bash +# Use with Dynamo's SLA planner (20-30 seconds vs hours) +python3 -m benchmarks.profiler.profile_sla \ + --config ./components/backends/trtllm/deploy/disagg.yaml \ + --backend trtllm \ + --use-ai-configurator \ + --aic-system h200_sxm \ + --aic-model-name QWEN3_32B +``` + +### Custom Configuration +```bash +# For advanced users: define custom search space +aiconfigurator cli exp --yaml_path custom_config.yaml +``` + +## Common Use Cases + +```bash +# Strict SLAs (low latency) +aiconfigurator cli default --model QWEN2.5_7B --total_gpus 8 --system h200_sxm --ttft 100 --tpot 5 + +# High throughput (relaxed latency) +aiconfigurator cli default --model QWEN3_32B --total_gpus 32 --system h200_sxm --ttft 1000 --tpot 50 +``` + +## Supported Configurations + +**Models**: GPT, LLAMA2/3, QWEN2.5/3, Mixtral, DEEPSEEK_V3 +**GPUs**: H100, H200, A100, B200 (preview), GB200 (preview) +**Backend**: TensorRT-LLM (vLLM and SGLang coming soon) + +## Additional Options + +```bash +# Web interface +aiconfigurator webapp # Visit http://127.0.0.1:7860 + +# Docker +docker run -it --rm nvcr.io/nvidia/aiconfigurator:latest \ + aiconfigurator cli default --model LLAMA3.1_70B --total_gpus 16 --system h100_sxm +``` + +## Troubleshooting + +**Model name mismatch**: Use exact model name that matches your deployment +**GPU allocation**: Verify available GPUs match `--total_gpus` +**Performance variance**: Results are estimates - benchmark actual deployment + +## Learn More + +- [Dynamo Installation Guide](/docs/kubernetes/installation_guide.md) +- [SLA Planner Quick Start Guide](/docs/planner/sla_planner_quickstart.md) +- [Benchmarking Guide](/docs/benchmarks/benchmarking.md) \ No newline at end of file diff --git a/docs/planner/sla_planner.md b/docs/planner/sla_planner.md index 26f1f9010362..4c33fb6463d2 100644 --- a/docs/planner/sla_planner.md +++ b/docs/planner/sla_planner.md @@ -21,7 +21,7 @@ The SLA (Service Level Agreement)-based planner is an intelligent autoscaling sy - **Planner**: Queries Prometheus and adjusts worker scaling every adjustment interval - **Workers**: prefill and backend workers handle inference -The adjustment interval can be defined in the planner manifest as an argument. The default interval value can be found in this [file](/components/planner/src/dynamo/planner/defaults.py). +The adjustment interval can be defined in the planner manifest as an argument. The default interval value can be found in this [file](/components/src/dynamo/planner/defaults.py). ```mermaid flowchart LR diff --git a/docs/planner/sla_planner_quickstart.md b/docs/planner/sla_planner_quickstart.md index 0d846e3ed9ad..9feb2f45c14c 100644 --- a/docs/planner/sla_planner_quickstart.md +++ b/docs/planner/sla_planner_quickstart.md @@ -15,7 +15,7 @@ The deployment process consists of two mandatory phases: 2. **SLA Planner Deployment** (5-10 minutes) - Enables autoscaling > [!TIP] -> **Fast Profiling with AI Configurator**: For TensorRT-LLM users, we provide AI Configurator (AIC) that can complete profiling in 20-30 seconds using performance simulation instead of real deployments. Support for vLLM and SGLang coming soon. See [AI Configurator section](/docs/benchmarks/pre_deployment_profiling.md#running-the-profiling-script-with-aiconfigurator) in the Profiling Guide. +> **Fast Profiling with AI Configurator**: For TensorRT-LLM users, we provide AI Configurator (AIC) that can complete profiling in 20-30 seconds using performance simulation instead of real deployments. Support for vLLM and SGLang coming soon. See [AI Configurator section](/docs/benchmarks/pre_deployment_profiling.md#running-the-profiling-script-with-ai-configurator) in the Profiling Guide. ```mermaid flowchart TD @@ -38,7 +38,7 @@ flowchart TD Before deploying the SLA planner, ensure: - **Dynamo platform installed** (see [Installation Guide](/docs/kubernetes/installation_guide.md)) -- **[kube-prometheus-stack](/docs/kubernetes/metrics.md) installed and running.** By default, the prometheus server is deployed in the `monitoring` namespace. If it is deployed to a different namespace, set `dynamo-operator.dynamo.metrics.prometheusEndpoint="http://prometheus-kube-prometheus-prometheus..svc.cluster.local:9090"`. +- **[kube-prometheus-stack](/docs/kubernetes/observability/metrics.md) installed and running.** By default, the prometheus server is not deployed in the `monitoring` namespace. If it is deployed to a different namespace, set `dynamo-operator.dynamo.metrics.prometheusEndpoint="http://prometheus-kube-prometheus-prometheus..svc.cluster.local:9090"`. - **Benchmarking resources setup** (see [Kubernetes utilities for Dynamo Benchmarking and Profiling](../../deploy/utils/README.md)) The script will create a `dynamo-pvc` with `ReadWriteMany` access, if your cluster's default storageClassName does not allow `ReadWriteMany`, you need to specify a different storageClassName in `deploy/utils/manifests/pvc.yaml` which does support `ReadWriteMany`. diff --git a/docs/reference/cli.md b/docs/reference/cli.md index dc5ab413e2ad..fe842e7561c2 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -152,19 +152,19 @@ The KV-aware routing arguments: ### Request Migration -In a [Distributed System](#distributed-system), you can enable [request migration](../architecture/request_migration.md) to handle worker failures gracefully. Use the `--migration-limit` flag to specify how many times a request can be migrated to another worker: +In a [Distributed System](#distributed-system), you can enable [request migration](../fault_tolerance/request_migration.md) to handle worker failures gracefully. Use the `--migration-limit` flag to specify how many times a request can be migrated to another worker: ```bash dynamo-run in=dyn://... out= ... --migration-limit=3 ``` -This allows a request to be migrated up to 3 times before failing. See the [Request Migration Architecture](../architecture/request_migration.md) documentation for details on how this works. +This allows a request to be migrated up to 3 times before failing. See the [Request Migration Architecture](../fault_tolerance/request_migration.md) documentation for details on how this works. ### Request Cancellation When using the HTTP interface (`in=http`), if the HTTP request connection is dropped by the client, Dynamo automatically cancels the downstream request to the worker. This ensures that computational resources are not wasted on generating responses that are no longer needed. -For detailed information about how request cancellation works across the system, see the [Request Cancellation Architecture](../architecture/request_cancellation.md) documentation. +For detailed information about how request cancellation works across the system, see the [Request Cancellation Architecture](../fault_tolerance/request_cancellation.md) documentation. ## Development diff --git a/docs/architecture/kv_cache_routing.md b/docs/router/kv_cache_routing.md similarity index 99% rename from docs/architecture/kv_cache_routing.md rename to docs/router/kv_cache_routing.md index a42adac842a8..44b192e9c759 100644 --- a/docs/architecture/kv_cache_routing.md +++ b/docs/router/kv_cache_routing.md @@ -255,7 +255,7 @@ python -m dynamo.frontend --router-mode kv --port 8002 --router-replica-sync >[!Note] > If you need to start with a fresh state, you have two options: -> 1. **Recommended**: Use a different namespace/component (see [Distributed Runtime](distributed_runtime.md)) which will start a new stream and NATS object store path +> 1. **Recommended**: Use a different namespace/component (see [Distributed Runtime](/docs/design_docs/distributed_runtime.md)) which will start a new stream and NATS object store path > 2. **Use with caution**: Launch a router with the `--router-reset-states` flag, which will purge the entire stream and radix snapshot. This should only be done when launching the first router replica in a component, as it can bring existing router replicas into an inconsistent state. ## Understanding KV Cache diff --git a/examples/README.md b/examples/README.md index 2571ccbd8ee5..b52ff56a3f57 100644 --- a/examples/README.md +++ b/examples/README.md @@ -30,17 +30,24 @@ Learn fundamental Dynamo concepts through these introductory examples: - **[Disaggregated Serving](basics/disaggregated_serving/README.md)** - Prefill/decode separation for enhanced performance and scalability - **[Multi-node](basics/multinode/README.md)** - Distributed inference across multiple nodes and GPUs +## Framework Support + +These examples show how Dynamo broadly works using major inference engines. + +If you want to see advanced, framework-specific deployment patterns and best practices, check out the [Components Workflows](../components/backends/) directory: +- **[vLLM](../components/backends/vllm/)** – vLLM-specific deployment and configuration +- **[SGLang](../components/backends/sglang/)** – SGLang integration examples and workflows +- **[TensorRT-LLM](../components/backends/trtllm/)** – TensorRT-LLM workflows and optimizations + ## Deployment Examples Platform-specific deployment guides for production environments: - **[Amazon EKS](deployments/EKS/)** - Deploy Dynamo on Amazon Elastic Kubernetes Service - **[Azure AKS](deployments/AKS/)** - Deploy Dynamo on Azure Kubernetes Service +- **[Amazon ECS](deployments/ECS/)** - Deploy Dynamo on Amazon Elastic Container Service - **[Router Standalone](deployments/router_standalone/)** - Standalone router deployment patterns -- **Amazon ECS** - _Coming soon_ - **Google GKE** - _Coming soon_ -- **Ray** - _Coming soon_ -- **NVIDIA Cloud Functions (NVCF)** - _Coming soon_ ## Runtime Examples @@ -68,11 +75,4 @@ Before running any examples, ensure you have: - **Python 3.9++** - For client scripts and utilities - **Kubernetes cluster** - For any cloud deployment/K8s examples -## Framework Support - -These examples show how Dynamo broadly works using major inference engines. -If you want to see advanced, framework-specific deployment patterns and best practices, check out the [Components Workflows](../components/backends/) directory: -- **[vLLM](../components/backends/vllm/)** – vLLM-specific deployment and configuration -- **[SGLang](../components/backends/sglang/)** – SGLang integration examples and workflows -- **[TensorRT-LLM](../components/backends/trtllm/)** – TensorRT-LLM workflows and optimizations diff --git a/examples/basics/multinode/README.md b/examples/basics/multinode/README.md index fed574774c3d..557a2a8c3c0c 100644 --- a/examples/basics/multinode/README.md +++ b/examples/basics/multinode/README.md @@ -4,8 +4,8 @@ This example demonstrates running Dynamo across multiple nodes with **KV-aware r For more information about the core concepts, see: -- [Dynamo Disaggregated Serving](../../../docs/architecture/disagg_serving.md) -- [KV Cache Routing Architecture](../../../docs/architecture/kv_cache_routing.md) +- [Dynamo Disaggregated Serving](../../../docs/design_docs/disagg_serving.md) +- [KV Cache Routing Architecture](../../../docs/router/kv_cache_routing.md) ## Architecture Overview @@ -65,7 +65,7 @@ This is particularly beneficial for: - **Similar queries**: Common prefixes are computed once and reused - **Batch processing**: Related requests can be routed to workers with shared context -For detailed technical information about how KV routing works, see the [KV Cache Routing Architecture documentation](../../../docs/architecture/kv_cache_routing.md). +For detailed technical information about how KV routing works, see the [KV Cache Routing Architecture documentation](../../../docs/router/kv_cache_routing.md). ## Prerequisites @@ -461,7 +461,7 @@ python -m dynamo.frontend \ --router-temperature 0.0 # Temperature for probabilistic routing (0 = deterministic) ``` -For more advanced configuration options including custom worker selection, block size tuning, and alternative indexing strategies, see the [KV Cache Routing documentation](../../../docs/architecture/kv_cache_routing.md). +For more advanced configuration options including custom worker selection, block size tuning, and alternative indexing strategies, see the [KV Cache Routing documentation](../../../docs/router/kv_cache_routing.md). ## Cleanup diff --git a/examples/custom_backend/cancellation/README.md b/examples/custom_backend/cancellation/README.md index bd1a12dcf8a6..659991dcc58f 100644 --- a/examples/custom_backend/cancellation/README.md +++ b/examples/custom_backend/cancellation/README.md @@ -88,4 +88,4 @@ python3 client.py --middle - Both modes demonstrate the same cancellation behavior - The middle server shows how to properly forward context in proxy scenarios -For more details on the request cancellation architecture, refer to the [architecture documentation](../../../docs/architecture/request_cancellation.md). +For more details on the request cancellation architecture, refer to the [architecture documentation](../../../docs/fault_tolerance/request_cancellation.md). diff --git a/tests/planner/README.md b/tests/planner/README.md index 289db840a7c1..6c834751a6d8 100644 --- a/tests/planner/README.md +++ b/tests/planner/README.md @@ -165,7 +165,7 @@ Test complete scaling behavior including Kubernetes deployment and load generati **Prerequisites:** -- **[kube-prometheus-stack](../../docs/kubernetes/metrics.md) installed and running.** The SLA planner requires Prometheus to observe metrics and make scaling decisions. +- **[kube-prometheus-stack](../../docs/kubernetes/observability/metrics.md) installed and running.** The SLA planner requires Prometheus to observe metrics and make scaling decisions. - Ensure the Dynamo operator was installed with the Prometheus endpoint configured (see [SLA Planner Quickstart Guide](../../docs/planner/sla_planner_quickstart.md#prerequisites) for details). **Prepare the test deployment manifest:**