[AMD Support][1/4] AMD hardware and required cluster runtime

docs/config-reference.md

@@ -135,6 +135,8 @@ The `srtslurm.yaml` file can contain the following fields:
 | `gpus_per_node`                 | int    | Default GPUs per node (applied to recipes that omit `resources.gpus_per_node`) |
 | `default_gpu_type`              | string | Default `resources.gpu_type` for recipes that omit it |
 | `network_interface`             | string | Network interface for NCCL                            |
+| `visible_devices_env`           | string | Worker GPU-subset mask; defaults to `CUDA_VISIBLE_DEVICES` |
+| `default_gpu_exporter`          | dict/null | Cluster GPU exporter; defaults to DCGM, explicit null disables it |
 | `srtctl_root`                   | string | Root directory for srtctl                             |
 | `output_dir`                    | string | Custom output directory (overrides srtctl_root/outputs) |
 | `model_paths`                   | dict   | Model path aliases                                    |
@@ -223,6 +225,22 @@ model:
 
 ## engine
 
+GPU scheduling uses upstream's existing cluster settings. For eight-GPU
+allocations on GRES-only clusters, set `use_gpus_per_node_directive: false`
+and `default_sbatch_directives: {gres: "gpu:8"}`.
+
+### GPU visibility on AMD
+
+Set `visible_devices_env: ROCR_VISIBLE_DEVICES` in the cluster profile for ROCm
+workers. GPU subsets then use only that mask, without applying a second mask to
+already-renumbered devices. Set `default_gpu_exporter: null` to disable the
+NVIDIA GPU exporter, or configure an exporter image, port, and command once for
+the cluster. Other telemetry is unchanged; an explicit recipe exporter wins.
+
+For vLLM builds without `--device-ids`, set `engine.set_visible_devices: true`.
+This is one explicit boolean, not automatic vLLM version detection. The default
+is false: vLLM binds devices with `--device-ids`. There is no CUDA-named alias.
+
 `engine:` names the inference engine that builds every worker role's command. A bare string is the common form; a mapping carries the engine-wide knobs, the fields that are not per role:
 
 ```yaml
@@ -253,7 +271,7 @@ Valid types are `sglang`, `vllm`, `trtllm`, and `mocker`. Everything that is per
 | Engine | Engine-wide knobs |
 | --- | --- |
 | `sglang-router` | none beyond `type` |
-| `vllm` | `connector` (default `nixl`), `dp_launch_mode`, `vllm_serve_binary`, `set_cuda_visible_devices`, `allow_prefill_decode_colocation`, `allow_prefill_decode_colocation_across_nodes` |
+| `vllm` | `connector` (default `nixl`), `dp_launch_mode`, `vllm_serve_binary`, `set_visible_devices`, `allow_prefill_decode_colocation`, `allow_prefill_decode_colocation_across_nodes` |
 | `trtllm` | `served_model_name`, `publish_metrics`, `publish_events_and_metrics`, `sequential_node_start`, `numa_memory_bind`, `numa_cpu_bind` |
 | `mocker` | the simulation parameters: `engine_type`, `speedup_ratio`, `decode_speedup_ratio`, `num_gpu_blocks_override`, `max_num_seqs`, `max_num_batched_tokens`, `block_size`, `data_parallel_size`, ... |
 

docs/schema-reference.md

@@ -380,6 +380,7 @@ Native Tachometer collection for an observability-enabled run.
 | `storage_subdir` | str | `'tachometer'` |  |
 | `extra_metadata` | dict[str, str] | `{}` |  |
 | `default_exporters` | bool | `True` |  |
+| `default_gpu_exporter` | [TelemetryExporterConfig](#telemetryexporterconfig) \| None | `<lambda>()` | Resolved from srtslurm.yaml at load time; never read global config here. |
 | `dcgm_exporter` | [TelemetryExporterConfig](#telemetryexporterconfig) \| None | `None` |  |
 | `node_exporter` | [TelemetryExporterConfig](#telemetryexporterconfig) \| None | `None` |  |
 | `process_exporter` | [TelemetryExporterConfig](#telemetryexporterconfig) \| None | `None` |  |
@@ -567,7 +568,7 @@ vLLM protocol - implements BackendProtocol.
 | Key | Type | Default | Description |
 |---|---|---|---|
 | `type` | one of `'vllm'` | `'vllm'` |  |
-| `set_cuda_visible_devices` | bool | `False` | Legacy device binding for vLLM builds without --device-ids. |
+| `set_visible_devices` | bool | `False` | Use an environment mask instead of the engine's --device-ids option. |
 | `connector` | str \| None | `'nixl'` | Default KV connector: "nixl", "lmcache", or a raw JSON string for --kv-transfer-config. Can be overridden per role by setting "connector" in roles.<role>.args. dynamo 1.0.0+: translated to --kv-transfer-config (--connector was removed). |
 | `allow_prefill_decode_colocation` | bool | `False` | Allow prefill and decode workers to share one node when the combined GPU request fits within gpus_per_node. Defaults off to preserve existing P/D node separation. |
 | `allow_prefill_decode_colocation_across_nodes` | bool | `False` | Extend P/D colocation to multi-node topologies. When enabled together with allow_prefill_decode_colocation, workers are packed contiguously across the minimum number of nodes instead of reserving separate P/D node pools. Defaults off to preserve the original one-node-only policy. |
@@ -612,6 +613,8 @@ Top-level keys of `srtslurm.yaml`. Recipes inherit these defaults and resolve al
 | `gpus_per_node` | int \| None | `None` |  |
 | `default_gpu_type` | str \| None | `None` | Default for ``ResourceConfig.gpu_type`` when the recipe omits it. Lets one recipe move between clusters of different GPU types without an edit. |
 | `network_interface` | str \| None | `None` |  |
+| `visible_devices_env` | str | `'CUDA_VISIBLE_DEVICES'` | GPU-subset mask passed to workers; ROCm clusters use ROCR_VISIBLE_DEVICES. |
+| `default_gpu_exporter` | [TelemetryExporterConfig](#telemetryexporterconfig) \| None | `<lambda>()` | Recipe exporter settings win. Explicit null disables the GPU default only. |
 | `use_gpus_per_node_directive` | bool | `True` |  |
 | `use_segment_sbatch_directive` | bool | `True` |  |
 | `use_exclusive_sbatch_directive` | bool | `False` |  |

src/srtctl/backends/vllm.py

@@ -263,8 +263,8 @@ class VLLMProtocol:
     # vLLM server CLI config per mode
     vllm_config: VLLMServerConfig | None = None
 
-    # Legacy device binding for vLLM builds without --device-ids.
-    set_cuda_visible_devices: bool = False
+    # Use an environment mask instead of the engine's --device-ids option.
+    set_visible_devices: bool = False
 
     # Default KV connector: "nixl", "lmcache", or a raw JSON string for --kv-transfer-config.
     # Can be overridden per role by setting "connector" in roles.<role>.args.
@@ -745,14 +745,9 @@ def _dp_rank_gpu_groups(self, mode: WorkerMode, gpu_indices: frozenset[int]) ->
             for offset in range(0, len(sorted_gpus), gpus_per_rank)
         ]
 
-    def should_set_cuda_visible_devices(self, process: Process) -> bool:
-        """Whether worker_stage should set CUDA_VISIBLE_DEVICES.
-
-        Newer vLLM builds should use ``--device-ids`` instead. Older builds
-        before https://github.com/vllm-project/vllm/pull/45026 should set
-        CUDA_VISIBLE_DEVICES.
-        """
-        return self.set_cuda_visible_devices
+    def should_set_visible_devices(self) -> bool:
+        """Whether worker launch should set the cluster-configured GPU mask."""
+        return self.set_visible_devices
 
     def endpoints_to_processes(
         self,
@@ -1146,7 +1141,7 @@ def build_worker_command(
                             process.node,
                         )
             _log_overridden_recipe_flags(overridden, srtslurm_owned, process.node)
-            if not self.set_cuda_visible_devices:
+            if not self.should_set_visible_devices():
                 device_ids = ",".join(str(i) for i in sorted(process.gpu_indices))
                 if device_ids:
                     cmd.extend(["--device-ids", device_ids])
@@ -1180,7 +1175,7 @@ def build_worker_command(
             kv_transfer_cfg = _connector_to_kv_transfer_config(connector)
             cmd.extend(["--kv-transfer-config", kv_transfer_cfg])
 
-        if not self.set_cuda_visible_devices:
+        if not self.should_set_visible_devices():
             device_ids = ",".join(str(i) for i in sorted(process.gpu_indices))
             if device_ids:
                 cmd.extend(["--device-ids", device_ids])

src/srtctl/cli/mixins/worker_stage.py

@@ -11,6 +11,7 @@
 import shlex
 from collections import defaultdict
 from concurrent.futures import ThreadPoolExecutor, as_completed
+from pathlib import Path
 from typing import TYPE_CHECKING, Any
 
 from srtctl.core.fingerprint import generate_capture_script
@@ -98,6 +99,21 @@ def _build_worker_preamble(self) -> str | None:
 
         return " && ".join(parts)
 
+    def _visible_device_environment(self, process: "Process") -> dict[str, str]:
+        """Build a vendor-native device mask when the backend needs one."""
+        should_set_devices = getattr(self.backend, "should_set_visible_devices", lambda: True)
+        force_mask = getattr(self.config.dynamo, "sidecar", False) is True and self.backend.type == "vllm"
+        if not (force_mask or should_set_devices()) or len(process.gpu_indices) >= self.runtime.gpus_per_node:
+            return {}
+        return {self.runtime.visible_devices_env: process.cuda_visible_devices}
+
+    def _container_log_path(self, filename: str) -> Path:
+        """Return a worker-visible path under the runtime log mount."""
+        container_log_dir = self.runtime.container_mounts.get(self.runtime.log_dir)
+        if container_log_dir is None:
+            raise RuntimeError(f"Runtime log directory is not mounted in the container: {self.runtime.log_dir}")
+        return container_log_dir / filename
+
     def _apply_kvbm_endpoint_env(self, env_to_set: dict[str, str], endpoint_processes: list["Process"]) -> None:
         """Fill KVBM leader ZMQ settings for an endpoint.
 
@@ -164,7 +180,7 @@ def start_worker(self, process: "Process", endpoint_processes: list["Process"])
 
         # Log and config files
         worker_log = self.runtime.log_dir / f"{process.node}_{mode}_w{index}.out"
-        config_dump = self.runtime.log_dir / f"{process.node}_config.json"
+        config_dump = self._container_log_path(f"{process.node}_config.json")
 
         # Profiling setup
         profiling = self.config.profiling
@@ -229,10 +245,7 @@ def __missing__(self, key: str) -> str:
             profile_dir = str(self.runtime.log_dir / "profiles")
             env_to_set.update(profiling.get_env_vars(mode, profile_dir))
 
-        should_set_cvd = getattr(self.backend, "should_set_cuda_visible_devices", lambda _process: True)
-        force_cvd = getattr(self.config.dynamo, "sidecar", False) is True and self.backend.type == "vllm"
-        if (force_cvd or should_set_cvd(process)) and len(process.gpu_indices) < self.runtime.gpus_per_node:
-            env_to_set["CUDA_VISIBLE_DEVICES"] = process.cuda_visible_devices
+        env_to_set.update(self._visible_device_environment(process))
 
         # Add backend-specific process environment variables (e.g., unique ports)
         env_to_set.update(self.backend.get_process_environment(process))
@@ -329,7 +342,7 @@ def start_endpoint_worker(self, endpoint_processes: list["Process"]) -> ManagedP
 
         # Log and config files (use leader node in name)
         worker_log = self.runtime.log_dir / f"{leader.node}_{mode}_w{index}.out"
-        config_dump = self.runtime.log_dir / f"{leader.node}_config.json"
+        config_dump = self._container_log_path(f"{leader.node}_config.json")
 
         # Profiling setup
         profiling = self.config.profiling
@@ -392,10 +405,7 @@ def start_endpoint_worker(self, endpoint_processes: list["Process"]) -> ManagedP
             profile_dir = str(self.runtime.log_dir / "profiles")
             env_to_set.update(profiling.get_env_vars(mode, profile_dir))
 
-        should_set_cvd = getattr(self.backend, "should_set_cuda_visible_devices", lambda _process: True)
-        force_cvd = getattr(self.config.dynamo, "sidecar", False) is True and self.backend.type == "vllm"
-        if (force_cvd or should_set_cvd(leader)) and len(leader.gpu_indices) < self.runtime.gpus_per_node:
-            env_to_set["CUDA_VISIBLE_DEVICES"] = leader.cuda_visible_devices
+        env_to_set.update(self._visible_device_environment(leader))
 
         # Add mooncake worker env vars if configured (SGLang only). For MPI-style
         # endpoint launching we use the leader node's IP — mooncake's per-worker

src/srtctl/cli/submit.py

@@ -232,6 +232,9 @@ def show_config_details(config: SrtConfig) -> None:
     environment variables (global and backend per-mode) so users can verify their
     config is correct before submitting.
     """
+    visible_devices_env = get_srtslurm_setting("visible_devices_env", "CUDA_VISIBLE_DEVICES")
+    console.print(f"GPU subset visibility variable: {visible_devices_env}")
+
     if config.frontend.type == "dynamo" and not config.dynamo.sidecar:
         from srtctl.backends.trtllm import TRTLLMProtocol
 

src/srtctl/core/config.py

@@ -251,6 +251,12 @@ def resolve_config_with_defaults(user_config: dict[str, Any], cluster_config: di
         model["path"] = resolved_path
         logger.debug(f"Resolved model alias '{model_path}' -> '{resolved_path}'")
 
+    # Resolve the cluster GPU exporter once, before container aliases. Keep
+    # recipe overrides and the existing default_exporters opt-out authoritative.
+    if "default_gpu_exporter" in cluster_config:
+        tachometer = config.setdefault("observability", {}).setdefault("tachometer", {})
+        tachometer.setdefault("default_gpu_exporter", copy.deepcopy(cluster_config["default_gpu_exporter"]))
+
     # Resolve every container alias in one pass (model.container,
     # frontend.container_image / nginx_container, benchmark.container_image,
     # exporter images, mooncake_kv_store.container, services, ...).

src/srtctl/core/runtime.py

@@ -238,6 +238,7 @@ class RuntimeContext:
     # HuggingFace model support - True if model.path was "hf:model/name"
     is_hf_model: bool = False
     gpu_type: str | None = None
+    visible_devices_env: str = "CUDA_VISIBLE_DEVICES"
 
     # Container mounts: host_path -> container_path
     container_mounts: dict[Path, Path] = field(default_factory=dict)
@@ -286,9 +287,12 @@ def from_config(
         # Compute run_name
         run_name = f"{config.name}_{job_id}"
 
-        # Resolve node IPs
-        head_node_ip = get_hostname_ip(nodes.head)
-        infra_node_ip = get_hostname_ip(nodes.infra)
+        # Resolve node IPs on the cluster-selected fabric. Some systems expose
+        # a public default route and a separate private control/data plane; the
+        # latter is what containers on peer Slurm nodes can reliably reach.
+        network_interface = get_srtslurm_setting("network_interface", "eth0")
+        head_node_ip = get_hostname_ip(nodes.head, network_interface)
+        infra_node_ip = get_hostname_ip(nodes.infra, network_interface)
 
         # Compute log directory using FormattablePath or default logic
         # Check for SRTCTL_OUTPUT_DIR from sbatch script first (ensures consistency)
@@ -406,6 +410,7 @@ def from_config(
         environment = config.dynamo.get_wheel_environment()
         environment.update(config.environment)
 
+        visible_devices_env = get_srtslurm_setting("visible_devices_env", "CUDA_VISIBLE_DEVICES")
         temp_context = cls(
             job_id=job_id,
             run_name=run_name,
@@ -417,7 +422,8 @@ def from_config(
             container_image=container_image,
             gpus_per_node=config.resources.gpus_per_node,
             gpu_type=config.resources.gpu_type,
-            network_interface=get_srtslurm_setting("network_interface", "eth0"),
+            network_interface=network_interface,
+            visible_devices_env=visible_devices_env,
             container_mounts={},
             srun_options=dict(config.srun_options),
             environment=environment,
@@ -443,7 +449,8 @@ def from_config(
             container_image=container_image,
             gpus_per_node=config.resources.gpus_per_node,
             gpu_type=config.resources.gpu_type,
-            network_interface=get_srtslurm_setting("network_interface", "eth0"),
+            network_interface=network_interface,
+            visible_devices_env=visible_devices_env,
             container_mounts=container_mounts,
             srun_options=dict(config.srun_options),
             environment=environment,

src/srtctl/core/schema.py

@@ -299,6 +299,10 @@ class ClusterConfig:
     # recipe move between clusters of different GPU types without an edit.
     default_gpu_type: str | None = None
     network_interface: str | None = None
+    # GPU-subset mask passed to workers; ROCm clusters use ROCR_VISIBLE_DEVICES.
+    visible_devices_env: str = "CUDA_VISIBLE_DEVICES"
+    # Recipe exporter settings win. Explicit null disables the GPU default only.
+    default_gpu_exporter: "TelemetryExporterConfig | None" = field(default_factory=lambda: DEFAULT_DCGM_EXPORTER)
     use_gpus_per_node_directive: bool = True
     use_segment_sbatch_directive: bool = True
     use_exclusive_sbatch_directive: bool = False
@@ -1235,6 +1239,8 @@ class TachometerConfig:
     storage_subdir: str = "tachometer"
     extra_metadata: dict[str, str] = field(default_factory=dict)
     default_exporters: bool = True
+    # Resolved from srtslurm.yaml at load time; never read global config here.
+    default_gpu_exporter: TelemetryExporterConfig | None = field(default_factory=lambda: DEFAULT_DCGM_EXPORTER)
     dcgm_exporter: TelemetryExporterConfig | None = None
     node_exporter: TelemetryExporterConfig | None = None
     process_exporter: TelemetryExporterConfig | None = None
@@ -1243,10 +1249,10 @@ class TachometerConfig:
 
     @property
     def resolved_dcgm_exporter(self) -> TelemetryExporterConfig | None:
-        """User-configured DCGM exporter, else the built-in default."""
+        """Recipe exporter, else the resolved cluster default."""
         if self.dcgm_exporter is not None:
             return self.dcgm_exporter
-        return DEFAULT_DCGM_EXPORTER if self.default_exporters else None
+        return self.default_gpu_exporter if self.default_exporters else None
 
     @property
     def resolved_node_exporter(self) -> TelemetryExporterConfig | None:

src/srtctl/frontends/sglang.py

@@ -35,6 +35,8 @@ class SGLangRouterFrontend(StaticRouterFrontend):
     # Preserve the historical launch shape used by dry-run/topology callers
     # that construct the frontend before populating worker processes.
     allow_empty_workers: ClassVar[bool] = True
+    # Static registration expires if model loading outlasts the router startup window.
+    wait_for_workers_before_start: ClassVar[bool] = True
 
     def get_managed_frontend_args(
         self,
@@ -62,8 +64,7 @@ def worker_scheme(self, backend: Any, mode: str) -> str:
         return "grpc" if backend.is_grpc_mode(mode) else "http"
 
     def resolve_worker_host(self, node: str, network_interface: str | None) -> str:
-        del network_interface
-        return get_hostname_ip(node)
+        return get_hostname_ip(node, network_interface)
 
     def start_process(self, **kwargs: Any) -> Any:
         return start_srun_process(**kwargs)