feat: [AMD Support via SemiAnalysis][2/4] vLLM MoRI-IO integration

CLAUDE.md

@@ -119,7 +119,7 @@ Which rank serves what is the frontend's call too, one method per consumer: `wor
 
 ### Ports
 
-Fixed ports are constants in `src/srtctl/ports.py`. Every port a worker process binds is a `PortKind` in the same module (`name`, `base`, `stride`, `per_node`) and is handed out by `NodePortAllocator.next(kind, node, size)` in `src/srtctl/core/topology.py`: per-node counters for listeners only bound on that node (`HTTP_PORTS`, `BOOTSTRAP_PORTS`, `DP_RPC_PORTS`, `DIST_INIT_PORTS`) and global counters for side channels peers address (`SYS_PORTS`, `KV_EVENTS_PORTS`, `NIXL_PORTS`, `KVBM_ZMQ_PORTS`, `NCCL_PORTS`, `VLLM_SCAN_PORTS`, `TRTLLM_DIST_INIT_PORTS`, `SIDECAR_GRPC_PORTS`). `size > 1` reserves a block when the engine adds a rank offset to the port it is given. `endpoints_to_processes` allocates the generic kinds and stores them on `Process`; a backend adds its engine-specific kinds with `dataclasses.replace` on the way out (SGLang `nccl_port` and `dist_init_port`, vLLM `vllm_scan_port`, TRT-LLM `trtllm_dist_init_port`). `do_sweep` builds the one allocator per job, seeding the sidecar gRPC base from `dynamo.sidecar_port`.
+Fixed ports are constants in `src/srtctl/ports.py`. Every port a worker process binds is a `PortKind` in the same module (`name`, `base`, `stride`, `per_node`) and is handed out by `NodePortAllocator.next(kind, node, size)` in `src/srtctl/core/topology.py`: per-node counters for listeners only bound on that node (`HTTP_PORTS`, `BOOTSTRAP_PORTS`, `DP_RPC_PORTS`, `DIST_INIT_PORTS`) and global counters for side channels peers address (`SYS_PORTS`, `KV_EVENTS_PORTS`, `NIXL_PORTS`, `KVBM_ZMQ_PORTS`, `NCCL_PORTS`, `VLLM_SCAN_PORTS`, `MORIIO_HANDSHAKE_PORTS`, `MORIIO_NOTIFY_PORTS`, `TRTLLM_DIST_INIT_PORTS`, `SIDECAR_GRPC_PORTS`). `size > 1` reserves a block when the engine adds a rank offset to the port it is given (MoRI-IO adds the local DP and TP rank to its notify base). `endpoints_to_processes` allocates the generic kinds and stores them on `Process`; a backend adds its engine-specific kinds with `dataclasses.replace` on the way out (SGLang `nccl_port` and `dist_init_port`, vLLM `vllm_scan_port` or, for a discovery connector, `moriio_handshake_port` and `moriio_notify_port`, TRT-LLM `trtllm_dist_init_port`). `do_sweep` builds the one allocator per job, seeding the sidecar gRPC base from `dynamo.sidecar_port`.
 
 Rules that follow: a new listener is a new `PortKind` plus a `Process` field, allocated in the topology builder, never `some_base + (sys_port - DYN_SYSTEM_PORT_BASE)` or any other arithmetic on another port at command-build time. Consumers read the field; a `None` means the topology was built without that kind, and a consumer that needs it raises rather than guessing. A worker's own address is `get_hostname_ip(node, runtime.network_interface)`, not upstream's interface guess. Fixed scan bases such as `VLLM_PORT` exist only to keep co-located `get_open_port()` scans apart; if a connector allocates inside forked children, leave the base unset so the kernel assigns ports. Frontends bind `FRONTEND_PUBLIC_PORT` (8000), or `FRONTEND_INTERNAL_PORT` (8180) behind nginx when `enable_multiple_frontends` is set. Service ports are `ServiceKind` defaults overridden by `options`. `tests/test_port_allocator.py` asserts over every example recipe that no two processes on a node share a port and global kinds never repeat.
 
@@ -371,7 +371,7 @@ with patch.dict(os.environ, H100Rack.slurm_env()):
    - `get_process_environment(process)` - Per-process env derived from `Process` ports (side channels, scan bases)
    - `mooncake_kv_store` / `get_mooncake_worker_env(...)` - the Mooncake block and its worker env; `None` / `{}` without one
    - `failover` / `get_failover_environment(...)` - shadow engine recovery; `None` / `{}` without it
-   - `should_set_cuda_visible_devices(process)` - `True` unless the engine takes its devices on the command line
+   - `should_set_visible_devices()` - `True` unless the engine takes its devices on the command line; the variable is the cluster's `visible_devices_env`
    - `get_served_model_name(default)`
 3. Export from `backends/__init__.py`
 4. Add polymorphic deserialization in `BackendConfigField` in `schema.py`

docs/architecture.md

@@ -296,7 +296,7 @@ class BackendProtocol(Protocol):
     def failover(self) -> VLLMFailoverConfig | None: ...
     def get_mooncake_worker_env(self, infra_node_ip, local_hostname) -> dict[str, str]: ...
     def get_failover_environment(self, process, job_id) -> dict[str, str]: ...
-    def should_set_cuda_visible_devices(self, process) -> bool: ...
+    def should_set_visible_devices(self) -> bool: ...
 
     def get_srun_config(self) -> SrunConfig: ...  # launch_per_endpoint, sequential_node_start, mpi
     def get_config_for_mode(self, mode: str) -> dict[str, Any]: ...
@@ -723,6 +723,8 @@ consumer derives one port from another.
 | nccl                  | 17500  | 1      | global   | SGLang servers                          |
 | dist_init             | 8300   | 1      | per node | SGLang multi-node endpoints (leader)    |
 | vllm_scan             | 20000  | 50     | global   | vLLM get_open_port() scan range         |
+| moriio_handshake      | 40000  | 1      | global   | vLLM MoRI-IO workers (peer handshake)   |
+| moriio_notify         | 41000  | 1      | global   | vLLM MoRI-IO workers (block per rank)   |
 | trtllm_dist_init      | 29500  | 1      | global   | TRT-LLM endpoints (leader's MASTER_PORT)|
 +-----------------------+--------+--------+----------+----------------------------------------+
 | Frontend public 8000, internal 8180 (behind nginx); etcd 2379, NATS 4222: fixed constants |

docs/config-reference.md

@@ -134,6 +134,8 @@ The `srtslurm.yaml` file can contain the following fields:
 | `gpus_per_node`                 | int    | Default GPUs per node (applied to recipes that omit `resources.gpus_per_node`) |
 | `default_gpu_type`              | string | Default `resources.gpu_type` for recipes that omit it |
 | `network_interface`             | string | Network interface for NCCL                            |
+| `visible_devices_env`           | string | Worker GPU-subset mask; defaults to `CUDA_VISIBLE_DEVICES` |
+| `default_gpu_exporter`          | dict/null | Cluster GPU exporter; defaults to DCGM, explicit null disables it |
 | `srtctl_root`                   | string | Root directory for srtctl                             |
 | `output_dir`                    | string | Custom output directory (overrides srtctl_root/outputs) |
 | `model_paths`                   | dict   | Model path aliases                                    |
@@ -241,6 +243,22 @@ model:
 
 ## engine
 
+GPU scheduling uses upstream's existing cluster settings. For eight-GPU
+allocations on GRES-only clusters, set `use_gpus_per_node_directive: false`
+and `default_sbatch_directives: {gres: "gpu:8"}`.
+
+### GPU visibility on AMD
+
+Set `visible_devices_env: ROCR_VISIBLE_DEVICES` in the cluster profile for ROCm
+workers. GPU subsets then use only that mask, without applying a second mask to
+already-renumbered devices. Set `default_gpu_exporter: null` to disable the
+NVIDIA GPU exporter, or configure an exporter image, port, and command once for
+the cluster. Other telemetry is unchanged; an explicit recipe exporter wins.
+
+For vLLM builds without `--device-ids`, set `engine.set_visible_devices: true`.
+This is one explicit boolean, not automatic vLLM version detection. The default
+is false: vLLM binds devices with `--device-ids`. There is no CUDA-named alias.
+
 `engine:` names the inference engine that builds every worker role's command. A bare string is the common form; a mapping carries the engine-wide knobs, the fields that are not per role:
 
 ```yaml
@@ -271,7 +289,7 @@ Valid types are `sglang`, `vllm`, `trtllm`, and `mocker`. Everything that is per
 | Engine | Engine-wide knobs |
 | --- | --- |
 | `sglang-router` | none beyond `type` |
-| `vllm` | `connector` (default `nixl`), `dp_launch_mode`, `vllm_serve_binary`, `set_cuda_visible_devices`, `allow_prefill_decode_colocation`, `allow_prefill_decode_colocation_across_nodes` |
+| `vllm` | `connector` (default `nixl`), `dp_launch_mode`, `vllm_serve_binary`, `set_visible_devices`, `allow_prefill_decode_colocation`, `allow_prefill_decode_colocation_across_nodes` |
 | `trtllm` | `served_model_name`, `publish_metrics`, `publish_events_and_metrics`, `sequential_node_start`, `numa_memory_bind`, `numa_cpu_bind` |
 | `mocker` | the simulation parameters: `engine_type`, `speedup_ratio`, `decode_speedup_ratio`, `num_gpu_blocks_override`, `max_num_seqs`, `max_num_batched_tokens`, `block_size`, `data_parallel_size`, ... |
 
@@ -749,6 +767,8 @@ Compare with `frontend.type: dynamo` + `engine: vllm`, which keeps Dynamo as the
 
 `type: vllm-router` launches the official vLLM Router in front of direct `vllm serve` workers. It supports aggregate replicas and disaggregated P/D topologies without Dynamo or NATS/etcd. See [vLLM Router](vllm-router.md) for complete topology examples and the division of responsibility between the upstream vLLM backend topology and Router adapter.
 
+`engine.connector: moriio` (AMD MoRI-IO on ROCm) switches the same frontend to the Router's discovery mode. srtctl launches one Router on the head node with `--kv-connector moriio --vllm-discovery-address 0.0.0.0:36367` and no worker URLs, gives every prefill and decode worker a role-aware `MoRIIOConnector` `--kv-transfer-config` that carries the Router's address, the worker's own routable IP and HTTP port, and the handshake and notify listeners the port allocator reserved for it, and waits on the Router's `/health`, which answers 503 until a prefill and a decode have registered. Both roles must run the connector, the topology must be prefill/decode, and `frontend.enable_multiple_frontends` must be `false`. See [MoRI-IO discovery](vllm-router.md#mori-io-discovery).
+
 ---
 
 ## benchmark

docs/schema-reference.md

@@ -393,6 +393,7 @@ Native Tachometer collection for an observability-enabled run.
 | `storage_subdir` | str | `'tachometer'` |  |
 | `extra_metadata` | dict[str, str] | `{}` |  |
 | `default_exporters` | bool | `True` |  |
+| `default_gpu_exporter` | [TelemetryExporterConfig](#telemetryexporterconfig) \| None | `<lambda>()` | Resolved from srtslurm.yaml at load time; never read global config here. |
 | `dcgm_exporter` | [TelemetryExporterConfig](#telemetryexporterconfig) \| None | `None` |  |
 | `node_exporter` | [TelemetryExporterConfig](#telemetryexporterconfig) \| None | `None` |  |
 | `process_exporter` | [TelemetryExporterConfig](#telemetryexporterconfig) \| None | `None` |  |
@@ -607,8 +608,8 @@ vLLM protocol - implements BackendProtocol.
 | Key | Type | Default | Description |
 |---|---|---|---|
 | `type` | one of `'vllm'` | `'vllm'` |  |
-| `set_cuda_visible_devices` | bool | `False` | Legacy device binding for vLLM builds without --device-ids. |
-| `connector` | str \| None | `'nixl'` | Default KV connector: "nixl", "lmcache", "kvbm", or a raw JSON string for --kv-transfer-config. Can be overridden per role by setting "connector" in roles.<role>.args; connector_for_mode resolves it. dynamo 1.0.0+: translated to --kv-transfer-config (--connector was removed). |
+| `set_visible_devices` | bool | `False` | Use an environment mask instead of the engine's --device-ids option. |
+| `connector` | str \| None | `'nixl'` | Default KV connector: "nixl", "lmcache", "kvbm", "moriio", or a raw JSON string for --kv-transfer-config. Can be overridden per role by setting "connector" in roles.<role>.args; connector_for_mode resolves it. "moriio" (ROCm MoRI-IO) registers workers with the vLLM Router and needs frontend.type: vllm-router. dynamo 1.0.0+: translated to --kv-transfer-config (--connector was removed). |
 | `failover` | [VLLMFailoverConfig](#vllmfailoverconfig) \| None | `None` | Shadow engine recovery: when set, every worker runs shadow_engines standby engines on its GPUs next to an implied `gms` service that owns the weights. Dynamo frontend only. |
 | `allow_prefill_decode_colocation` | bool | `False` | Allow prefill and decode workers to share one node when the combined GPU request fits within gpus_per_node. Defaults off to preserve existing P/D node separation. |
 | `allow_prefill_decode_colocation_across_nodes` | bool | `False` | Extend P/D colocation to multi-node topologies. When enabled together with allow_prefill_decode_colocation, workers are packed contiguously across the minimum number of nodes instead of reserving separate P/D node pools. Defaults off to preserve the original one-node-only policy. |
@@ -662,6 +663,8 @@ Top-level keys of `srtslurm.yaml`. Recipes inherit these defaults and resolve al
 | `gpus_per_node` | int \| None | `None` |  |
 | `default_gpu_type` | str \| None | `None` | Default for ``ResourceConfig.gpu_type`` when the recipe omits it. Lets one recipe move between clusters of different GPU types without an edit. |
 | `network_interface` | str \| None | `None` |  |
+| `visible_devices_env` | str | `'CUDA_VISIBLE_DEVICES'` | GPU-subset mask passed to workers; ROCm clusters use ROCR_VISIBLE_DEVICES. |
+| `default_gpu_exporter` | [TelemetryExporterConfig](#telemetryexporterconfig) \| None | `<lambda>()` | Recipe exporter settings win. Explicit null disables the GPU default only. |
 | `use_gpus_per_node_directive` | bool | `True` |  |
 | `use_segment_sbatch_directive` | bool | `True` |  |
 | `use_exclusive_sbatch_directive` | bool | `False` |  |

docs/vllm-router.md

@@ -142,6 +142,43 @@ P/D base must represent the same number of local DP ranks. srtctl derives and
 validates that value; do not set it manually unless it exactly matches the
 allocated topology.
 
+### MoRI-IO discovery
+
+`engine.connector: moriio` runs the P/D pair over AMD's MoRI-IO connector (ROCm). MoRI-IO workers do not take peer addresses on the command line; they register with the Router over a ZMQ discovery endpoint and learn each other's transfer addresses from it. srtctl therefore launches the Router in discovery mode and builds each worker's connector config from the realized topology instead of listing URLs:
+
+```yaml
+frontend:
+  type: vllm-router
+  enable_multiple_frontends: false
+
+engine:
+  type: vllm
+  connector: moriio
+roles:
+  prefill:
+    nodes: 1
+    workers: 1
+    gpus: 1
+  decode:
+    nodes: colocate
+    workers: 1
+    gpus: 1
+```
+
+| Piece | srtctl sets | Why |
+| --- | --- | --- |
+| Router command | `--vllm-pd-disaggregation --kv-connector moriio --vllm-discovery-address 0.0.0.0:36367`, no `--prefill`/`--decode` | Discovery mode in vllm-project/router (`RouterArgs.vllm_discovery_address`); the address is the one listener workers register with |
+| `kv_connector` / `kv_role` | `MoRIIOConnector`; `kv_producer` on prefill, `kv_consumer` on decode | The connector's role follows the worker's mode |
+| `proxy_ip`, `proxy_ping_port` | the head node's IP on the cluster interface, `36367` | Where the worker registers; hence one Router, on the head node |
+| `http_port`, `host_ip` | the worker's allocated HTTP port and its own routable IP | What the Router routes to and what peers connect to; upstream falls back to its own interface guess for `host_ip`, which is wrong on multi-homed nodes |
+| `handshake_port`, `notify_port` | allocated per process by `NodePortAllocator` (`moriio_handshake`, `moriio_notify` kinds); both reserve one port per local rank | vLLM adds rank offsets to both bases, so colocated workers need disjoint blocks. Defaults start at 26000/27000, below Linux's default ephemeral range, to avoid MoRI's other dynamically allocated listeners |
+| `read_mode` | `true` | The Router's discovery flow drives reads |
+| `VLLM_PORT` | unset for these workers | MoRI-IO opens its other listeners inside the TP child processes, which inherit `VLLM_PORT`; a fixed scan base there hands several ranks the same unbound port, so vLLM takes ephemeral ports from the kernel |
+
+Readiness is the Router's `/health`: it answers 503 `Waiting for discovered workers` until a prefill and a decode have registered, then 200 (vllm-project/router 43140bc8e2). This is a workaround for a gap in the Router, not a MoRI-IO property: in discovery mode the Router keeps registered workers in a separate registry but still serves `/workers` and `/get_server_info` from the static URL list, which is empty, so the count-based `/workers` gate srtctl uses for static routing has nothing to count. `/health` is the only endpoint that reads the discovery registry, and it only proves one worker of each role, so with several workers per role the benchmark can start before the Router knows all of them; the per-worker `/health` gate that follows proves the workers are up, not that they registered. The upstream fix is for `/workers` to merge the discovery registry (`src/routers/router_manager.rs`); when it does, `VLLMRouterFrontend.probe_ready` loses its discovery branch and the count-based probe covers both modes.
+
+Rules the recipe must satisfy: both `prefill` and `decode` run the connector (an engine-wide `connector: moriio`, or the same value in both roles' `args`), the topology is prefill/decode, `frontend.enable_multiple_frontends` is `false`, and `frontend.orchestrator_placement` is `head`. `srtctl dry-run` rejects anything else, and any other `frontend.type` with this connector. Upstream key names are those `moriio_common.py` reads (vllm-project/vllm 9679173788). `examples/vllm/vllm-router-moriio-disagg.yaml` is the reference recipe.
+
 ## Multiple Router processes
 
 With `enable_multiple_frontends: true`, srtctl starts nginx on the public port

examples/README.md

@@ -7,7 +7,7 @@ Small, runnable starting points, one per frontend and topology. Every example se
 | Backend | Dynamo frontend | Native router | Router-free direct |
 | --- | --- | --- | --- |
 | SGLang | `sglang/dynamo-agg.yaml`, `sglang/dynamo-disagg.yaml` | `sglang/sglang-router-agg.yaml`, `sglang/sglang-router-disagg.yaml` | `sglang/sglang-direct-agg.yaml` |
-| vLLM | `vllm/dynamo-agg.yaml`, `vllm/dynamo-disagg.yaml` | `vllm/vllm-router-agg.yaml`, `vllm/vllm-router-disagg.yaml` | `vllm/vllm-direct-agg.yaml` |
+| vLLM | `vllm/dynamo-agg.yaml`, `vllm/dynamo-disagg.yaml` | `vllm/vllm-router-agg.yaml`, `vllm/vllm-router-disagg.yaml`, `vllm/vllm-router-moriio-disagg.yaml` (ROCm, MoRI-IO discovery) | `vllm/vllm-direct-agg.yaml` |
 | TRT-LLM | `trtllm/dynamo-agg.yaml`, `trtllm/dynamo-disagg.yaml` | `trtllm/trtllm-serve-disagg.yaml` | `trtllm/trtllm-serve-agg.yaml` |
 | Mocker | `mocker/dynamo-agg.yaml` | | |