[docs][serve][llm] Add comprehensive architecture documentation for Ray Serve LLM

doc/source/serve/llm/architecture/index.md

@@ -1,10 +1,13 @@
 # Architecture
 
-Technical details and design documentation for Ray Serve LLM.
+Technical documentation for Ray Serve LLM architecture, components, and patterns.
 
 ```{toctree}
 :maxdepth: 1
 
+Architecture overview <overview>
+Core components <core>
+Serving patterns <serving-patterns/index>
 Request routing <routing-policies>
 ```
 

doc/source/serve/llm/architecture/overview.md

@@ -0,0 +1,218 @@
+(serve-llm-architecture-overview)=
+# Architecture overview
+
+Ray Serve LLM is a framework that specializes Ray Serve primitives for distributed LLM serving workloads. This guide explains the core components, serving patterns, and routing policies that enable scalable and efficient LLM inference.
+
+## What Ray Serve LLM provides
+
+Ray Serve LLM takes the performance of a single inference engine (such as vLLM) and extends it to support:
+
+- **Horizontal scaling**: Replicate inference across multiple GPUs on the same node or across nodes.
+- **Advanced distributed strategies**: Coordinate multiple engine instances for prefill-decode disaggregation, data parallel attention, and expert parallelism.
+- **Modular deployment**: Separate infrastructure logic from application logic for clean, maintainable deployments.
+
+Ray Serve LLM excels at highly distributed multi-node inference workloads where the unit of scale spans multiple nodes:
+
+- **Pipeline parallelism across nodes**: Serve large models that don't fit on a single node.
+- **Disaggregated prefill and decode**: Scale prefill and decode phases independently for better resource utilization.
+- **Cluster-wide parallelism**: Combine data parallel attention with expert parallelism for serving large-scale sparse MoE architectures such as Deepseek-v3, GPT OSS, etc.
+
+
+## Ray Serve primitives
+
+Before diving into the architecture, you should understand these Ray Serve primitives:
+
+- **Deployment**: A class that defines the unit of scale.
+- **Replica**: An instance of a deployment which corresponds to a Ray actor. Multiple replicas can be distributed across a cluster.
+- **Deployment handle**: An object that allows one replica to call into replicas of other deployments.
+
+For more details, see the {ref}`Ray Serve core concepts <serve-key-concepts>`.
+
+## Core components
+
+Ray Serve LLM provides two primary components that work together to serve LLM workloads:
+
+### LLMServer
+
+`LLMServer` is a Ray Serve _deployment_ that manages a single inference engine instance. _Replicas_ of this _deployment_ can operate in three modes:
+
+- **Isolated**: Each _replica_ handles requests independently (horizontal scaling).
+- **Coordinated within deployment**: Multiple _replicas_ work together (data parallel attention).
+- **Coordinated across deployments**: Replicas coordinate with different deployments (prefill-decode disaggregation).
+
+
+The following example demonstrates the sketch of how to use `LLMServer` standalone:
+
+```python
+from ray import serve
+from ray.serve.llm import LLMConfig
+from ray.serve.llm.deployment import LLMServer
+
+llm_config = LLMConfig(...)
+
+# Get deployment options (placement groups, etc.)
+serve_options = LLMServer.get_deployment_options(llm_config)
+
+# Decorate with serve options
+server_cls = serve.deployment(LLMServer).options(
+    stream=True, **serve_options)
+
+# Bind the decorated class to its constructor parameters
+server_app = server_cls.bind(llm_config)
+
+# Run the application
+serve_handle = serve.run(server_app)
+
+# Use the deployment handle
+result = serve_handle.chat.remote(request=...).result()
+```
+
+#### Physical placement
+
+`LLMServer` controls physical placement of its constituent actors through placement groups. By default, it uses:
+
+- `{CPU: 1}` for the replica actor itself (no GPU resources).
+- `world_size` number of `{GPU: 1}` bundles for the GPU workers.
+
+The `world_size` is computed as `tensor_parallel_size × pipeline_parallel_size`. The vLLM engine allocates TP and PP ranks based on bundle proximity, prioritizing TP ranks on the same node.
+
+The PACK strategy tries to place all resources on a single node, but provisions different nodes when necessary. This works well for most deployments, though heterogeneous model deployments might occasionally run TP across nodes.
+
+```{figure} ../images/placement.png
+---
+width: 600px
+name: placement
+---
+Physical placement strategy for GPU workers
+```
+
+#### Engine management
+
+When `LLMServer` starts, it:
+
+1. Creates a vLLM engine client.
+2. Spawns a background process that uses Ray's distributed executor backend.
+3. Uses the parent actor's placement group to instantiate child GPU worker actors.
+4. Executes the model's forward pass on these GPU workers.
+
+```{figure} ../images/llmserver.png
+---
+width: 600px
+name: llmserver
+---
+Illustration of `LLMServer` managing vLLM engine instance.
+```
+
+### OpenAiIngress
+
+`OpenAiIngress` provides an OpenAI-compatible FastAPI ingress that routes traffic to the appropriate model. It handles:
+
+- **Standard endpoint definitions**: `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, etc.
+- **Request routing logic**: The execution of custom router logic (for example, prefix-aware or session-aware routing).
+- **Model multiplexing**: LoRA adapter management and routing.
+
+The following example shows a complete deployment with `OpenAiIngress`:
+
+```python
+from ray import serve
+from ray.serve.llm import LLMConfig
+from ray.serve.llm.deployment import LLMServer
+from ray.serve.llm.ingress import OpenAiIngress, make_fastapi_ingress
+
+llm_config = LLMConfig(...)
+
+# Construct the LLMServer deployment
+serve_options = LLMServer.get_deployment_options(llm_config)
+server_cls = serve.deployment(LLMServer).options(**serve_options)
+llm_server = server_cls.bind(llm_config)
+
+# Get ingress default options
+ingress_options = OpenAiIngress.get_deployment_options([llm_config])
+
+# Decorate with FastAPI app
+ingress_cls = make_fastapi_ingress(OpenAiIngress)
+
+# Make it a serve deployment with the right options
+ingress_cls = serve.deployment(ingress_cls, **ingress_options)
+
+# Bind with llm_server deployment handle
+ingress_app = ingress_cls.bind([llm_server])
+
+# Run the application
+serve.run(ingress_app)
+```
+
+:::{note}
+You can create your own ingress deployments and connect them to existing LLMServer deployments. This is useful when you want to customize request tracing, authentication layers, etc.
+:::
+
+#### Network topology and RPC patterns
+
+When the ingress makes an RPC call to `LLMServer` through the deployment handle, it can reach any replica across any node. However, the default request router prioritizes replicas on the same node to minimize cross-node RPC overhead, which is insignificant in LLM serving applications (only a few milliseconds impact on TTFT at high concurrency).
+
+The following figure illustrates the data flow:
+
+```{figure} ../images/llmserver-ingress-rpc.png
+---
+width: 600px
+name: llmserver-ingress-rpc
+---
+Request routing from ingress to LLMServer replicas. Solid lines represent preferred local RPC calls; dashed lines represent potential cross-node RPC calls when local replicas are busy.
+```
+
+#### Scaling considerations
+
+**Ingress-to-LLMServer ratio**: The ingress event loop can become the bottleneck at high concurrency. In such situations, upscaling the number of ingress replicas can mitigate CPU contention. We recommend keeping at least a 2:1 ratio between the number of ingress replicas and LLMServer replicas. This architecture allows the system to dynamically scale the component that is the bottleneck.
+
+**Autoscaling coordination**: To maintain proper ratios during autoscaling, configure `target_ongoing_requests` proportionally:
+
+- Profile your vLLM configuration to find the maximum concurrent requests (for example, 64 requests).
+- Choose an ingress-to-LLMServer ratio (for example, 2:1).
+- Set LLMServer's `target_ongoing_requests` to say 75% of max capacity (for example, 48).
+- Set ingress's `target_ongoing_requests` to maintain the ratio (for example, 24).
+
+## Architecture patterns
+
+Ray Serve LLM supports several deployment patterns for different scaling scenarios:
+
+### Data parallel attention pattern
+
+Create multiple inference engine instances that process requests in parallel while coordinating across expert layers and sharding requests across attention layers. Useful for serving sparse MoE models for high-throughput workloads.
+
+**When to use**: High request volume, kv-cache limited, need to maximize throughput.
+
+See: {doc}`serving-patterns/data-parallel`
+
+### Prefill-decode disaggregation
+
+Separate prefill and decode phases to optimize resource utilization and scale each phase independently.
+
+**When to use**: Prefill-heavy workloads where there's tension between prefill and decode, cost optimization with different GPU types.
+
+See: {doc}`serving-patterns/prefill-decode`
+
+### Custom request routing
+
+Implement custom routing logic for specific optimization goals such as cache locality or session affinity.
+
+**When to use**: Workloads with repeated prompts, session-based interactions, or specific routing requirements.
+
+See: {doc}`routing-policies`
+
+## Design principles
+
+Ray Serve LLM follows these key design principles:
+
+1. **Engine-agnostic**: Support multiple inference engines (vLLM, SGLang, etc.) through the `LLMEngine` protocol.
+2. **Composable patterns**: Combine serving patterns (data parallel attention, prefill-decode, custom routing) for complex deployments.
+3. **Builder pattern**: Use builders to construct complex deployment graphs declaratively.
+4. **Separation of concerns**: Keep infrastructure logic (placement, scaling) separate from application logic (routing, processing).
+5. **Protocol-based extensibility**: Define clear protocols for engines, servers, and ingress to enable custom implementations.
+
+## See also
+
+- {doc}`core` - Technical implementation details and extension points
+- {doc}`serving-patterns/index` - Detailed serving pattern documentation
+- {doc}`routing-policies` - Request routing architecture and patterns
+- {doc}`../user-guides/index` - Practical deployment guides
+

doc/source/serve/llm/architecture/routing-policies.md

@@ -54,8 +54,9 @@ Ray Serve LLM provides multiple request routing policies to optimize for differe
 ### Default routing: Power of Two Choices
 
 The default router uses the Power of Two Choices algorithm to:
-1. Randomly sample two replicas
-2. Route to the replica with fewer ongoing requests
+
+1. Randomly sample two replicas.
+2. Route to the replica with fewer ongoing requests.
 
 This provides good load balancing with minimal coordination overhead.
 
@@ -64,10 +65,11 @@ This provides good load balancing with minimal coordination overhead.
 The `PrefixCacheAffinityRouter` optimizes for workloads with shared prefixes by routing requests with similar prefixes to the same replicas. This improves KV cache hit rates in vLLM's Automatic Prefix Caching (APC).
 
 The routing strategy:
-1. **Check load balance**: If replicas are balanced (queue difference < threshold), use prefix matching
-2. **High match rate (≥10%)**: Route to replicas with highest prefix match
-3. **Low match rate (<10%)**: Route to replicas with lowest cache utilization
-4. **Fallback**: Use Power of Two Choices when load is imbalanced
+
+1. **Check load balance**: If replicas are balanced (queue difference < threshold), use prefix matching.
+2. **High match rate (≥10%)**: Route to replicas with highest prefix match.
+3. **Low match rate (<10%)**: Route to replicas with lowest cache utilization.
+4. **Fallback**: Use Power of Two Choices when load is imbalanced.
 
 For more details, see {ref}`prefix-aware-routing-guide`.
 
@@ -111,13 +113,15 @@ Centralized metric store pattern for custom routing
 ```
 
 **Pros:**
-- Simple implementation - no need to modify deployment logic for recording replica statistics
-- Request metrics are immediately available
-- Strong consistency guarantees
+
+- Simple implementation - no need to modify deployment logic for recording replica statistics.
+- Request metrics are immediately available.
+- Strong consistency guarantees.
 
 **Cons:**
-- A single actor can become a bottleneck in high-throughput applications where TTFT is impacted by the RPC call (~1000s of requests/s)
-- Requires an additional network hop for every routing decision
+
+- A single actor can become a bottleneck in high-throughput applications where TTFT is impacted by the RPC call (~1000s of requests/s).
+- Requires an additional network hop for every routing decision.
 
 ### Pattern 2: Metrics broadcasted from Serve controller
 
@@ -156,51 +160,54 @@ Broadcast metrics pattern for custom routing
 ```
 
 **Pros:**
-- Scalable to higher throughput
-- No additional RPC overhead per routing decision
-- Distributed routing decision making
+
+- Scalable to higher throughput.
+- No additional RPC overhead per routing decision.
+- Distributed routing decision making.
 
 **Cons:**
-- Time lag between the request router's view of statistics and the ground truth state of the replicas
-- Eventual consistency - routers may base decisions on slightly stale data
-- More complex implementation requiring coordination with the Serve controller
+
+- Time lag between the request router's view of statistics and the ground truth state of the replicas.
+- Eventual consistency - routers may base decisions on slightly stale data.
+- More complex implementation requiring coordination with the Serve controller.
 
 
-- **Use Pattern 1 (Centralized store)** when you need strong consistency, have moderate throughput requirements, or want simpler implementation
-- **Use Pattern 2 (Broadcast metrics)** when you need very high throughput, can tolerate eventual consistency, or want to minimize per-request overhead
+- **Use Pattern 1 (Centralized store)** when you need strong consistency, have moderate throughput requirements, or want simpler implementation.
+- **Use Pattern 2 (Broadcast metrics)** when you need very high throughput, can tolerate eventual consistency, or want to minimize per-request overhead.
 
 ## Custom routing policies
 
 You can implement custom routing policies by extending Ray Serve's [`RequestRouter`](../../api/doc/ray.serve.request_router.RequestRouter.rst) base class. For detailed examples and step-by-step guides on implementing custom routers, see {ref}`custom-request-router-guide`.
 
 Key methods to implement:
-- [`choose_replicas()`](../../api/doc/ray.serve.request_router.RequestRouter.choose_replicas.rst): Select which replicas should handle a request
-- [`on_request_routed()`](../../api/doc/ray.serve.request_router.RequestRouter.on_request_routed.rst): Update the router state after a request is routed
-- [`on_replica_actor_died()`](../../api/doc/ray.serve.request_router.RequestRouter.on_replica_actor_died.rst): Clean up the state when a replica dies
+
+- [`choose_replicas()`](../../api/doc/ray.serve.request_router.RequestRouter.choose_replicas.rst): Select which replicas should handle a request.
+- [`on_request_routed()`](../../api/doc/ray.serve.request_router.RequestRouter.on_request_routed.rst): Update the router state after a request is routed.
+- [`on_replica_actor_died()`](../../api/doc/ray.serve.request_router.RequestRouter.on_replica_actor_died.rst): Clean up the state when a replica dies.
 
 ### Utility mixins
 
 Ray Serve provides mixin classes that add common functionality to routers. See the {ref}`custom-request-router-guide` for examples:
 
-- [`LocalityMixin`](../../api/doc/ray.serve.request_router.LocalityMixin.rst): Prefers replicas on the same node to reduce network latency
-- [`MultiplexMixin`](../../api/doc/ray.serve.request_router.MultiplexMixin.rst): Tracks which models are loaded on each replica for LoRA deployments
-- [`FIFOMixin`](../../api/doc/ray.serve.request_router.FIFOMixin.rst): Ensures FIFO ordering of requests
+- [`LocalityMixin`](../../api/doc/ray.serve.request_router.LocalityMixin.rst): Prefers replicas on the same node to reduce network latency.
+- [`MultiplexMixin`](../../api/doc/ray.serve.request_router.MultiplexMixin.rst): Tracks which models are loaded on each replica for LoRA deployments.
+- [`FIFOMixin`](../../api/doc/ray.serve.request_router.FIFOMixin.rst): Ensures FIFO ordering of requests.
 
 
 
 ### Router lifecycle
 
 The typical lifecycle of request routers includes the following stages:
 
-1. **Initialization**: Router created with list of replicas
-2. **Request routing**: `choose_replicas()` called for each request
-3. **Callback**: `on_request_routed()` called after successful routing
-4. **Replica failure**: `on_replica_actor_died()` called when replica dies
-5. **Cleanup**: Router cleaned up when deployment is deleted
+1. **Initialization**: Router created with list of replicas.
+2. **Request routing**: `choose_replicas()` called for each request.
+3. **Callback**: `on_request_routed()` called after successful routing.
+4. **Replica failure**: `on_replica_actor_died()` called when replica dies.
+5. **Cleanup**: Router cleaned up when deployment is deleted.
 
 #### Async operations
 
-Routers should use async operations for best performance, for example:
+Routers should use async operations for best performance. The following example demonstrates the recommended pattern:
 
 ```python
 # Recommended pattern: Async operation
@@ -216,7 +223,7 @@ async def choose_replicas(self, ...):
 
 #### State management
 
-For routers with state, use appropriate synchronization, for example:
+For routers with state, use appropriate synchronization. The following example shows the recommended pattern:
 
 ```python
 class StatefulRouter(RequestRouter):