feat: [AMD Support via SemiAnalysis][1/4] AMD hardware and required cluster runtime

[cquil11]

Scope

Enable AMD hardware and make native srt-slurm jobs work on our Slurm clusters. This combines the former hardware and required-runtime PRs; functionality needed to launch the deployment is no longer hidden in a separate prerequisite layer.

• Cluster configuration owns GPU visibility, GPU-exporter defaults, network interface, mounts, model paths, and Slurm options. NVIDIA defaults are preserved. There is no vendor enum, accelerator dispatch module, or new GRES convention.
• Resolve GPU-exporter defaults once at config load, with explicit recipe overrides taking precedence.
• Apply the configured visibility variable to allocated GPU subsets; preserve full-node and backend device-binding behavior. vLLM has one set_visible_devices option.
• Respect the configured interface for control-plane and SGLang worker addresses, and forward recipe Slurm options to native routers.
• Write worker configuration inside the existing container log mount.
• Static routers that need ready HTTP workers opt into the existing health polling before launch. SGLang enables it so slow model loading cannot exhaust initial registration. Worker URLs are resolved once and reused.

Upstream scheduling escape hatches handle GRES: use_gpus_per_node_directive: false and default_sbatch_directives: {gres: "gpu:8"}. Cluster host setup uses the existing lifecycle. No engine patch, new deployment system, automatic network fallback reordering, or Dynamo installation workaround is added.

Review map

• core/schema.py, core/config.py, core/runtime.py: cluster defaults and runtime context.
• cli/mixins/worker_stage.py, backends/vllm.py: GPU binding and container-visible config.
• frontends/, services/implicit.py: configured addresses, Slurm options, and native startup.
• Example profile, dry-run, docs, and focused behavioral tests accompany those changes.

Manual stack

#431 AMD hardware + required cluster runtime → #432 vLLM/MoRI → #433 ATOM/AToMesh → #476 optional observation and diagnostics.

The core runtime does not depend on #476. The InferenceX CI launcher uses its optional srtctl wait command to stream logs and report allocation completion; it is not part of GPU/worker/router enablement. ATOM uses Mooncake and has no engine dependency on MoRI.

Optional deployment fork #16, Dynamo installation fork #15, and Infera fork #3 are outside this validation.

E2E validation

InferenceX #2555 targets the seven-point MI355X DeepSeek V4 AgentX sweep plus three default evaluations.

Current full validation: InferenceX run 35231968727, port commit 15e61d4e865c9eb8c64009f57ec0948e4c934e12, fork runtime c63427249ab996bd38d364cad1629aa5aa8877ef. This preserves the maintainer-selected DSPARK baseline #3188, including all seven points and real-acceptance evaluation. Not yet green.

Previous attempt 35181888533 passed four throughput points and c16 GSM8K (97.35% versus 91% threshold). Three points stopped during host preparation, and two evals exposed a JSON-environment templating defect in the optional runtime layer. The replacement removes that formatting, uses native eval artifacts, and restores the existing GPU-drain time allowance. Serving settings are unchanged. #3170 is closed as superseded.

The reorganized revision is not E2E validated. These upstream PRs are synchronized for review; a complete green run and artifact inspection are still pending.

Latest completed runs before this cleanup/rebase, retained as historical evidence only:

• SGLang AgentX full sweep
• SGLang fixed-sequence full sweep
• ATOM/AToMesh full sweep
• vLLM MiniMax M3 validation

All four fork PRs target upstream main and are manually stacked: #431 → #432 → #433 → #476. Each later branch includes the preceding layers; merge in order and rebase the remaining PRs.

[ishandhanani]

The net diff is small (9 files, ~180 lines) and the scope discipline is good: GPU scheduling stays on use_gpus_per_node_directive / default_sbatch_directives, and the sidecar special case in worker_stage.py was only moved, not added. The concern is that the design center, accelerator_vendor, is a closed enum whose two effects (pick a visibility env var, suppress the DCGM default) are both hardcoded in Python, when each already has a cluster-config pattern in this repo. srtslurm is meant to be portable across clusters by editing srtslurm.yaml, not by adding vendor branches. Inline comments have the specifics.

Suggested shape instead of the enum:

• visible_devices_env: str = "CUDA_VISIBLE_DEVICES" on ClusterConfig, read once into RuntimeContext, same as network_interface. An AMD profile sets ROCR_VISIBLE_DEVICES. No accelerator.py, no ValueError path, no Literal to keep in sync across three files.
• default_gpu_exporter: TelemetryExporterConfig | None on ClusterConfig, following default_health_check / default_host_setup. An AMD profile points it at AMD's device-metrics-exporter once; NVIDIA clusters keep the built-in DCGM default; observability.tachometer.dcgm_exporter in a recipe still wins. The schema property then stops reading global config.

Not in the diff, but worth noting for a PR titled base AMD support:

• src/srtctl/core/fingerprint.py:330 and :917 shell out to nvidia-smi; :353-363 call nvcc and torch.cuda.nccl.version(). On AMD the fingerprint and lockfile silently record GPU, CUDA, and NCCL as unavailable.
• srtslurm.yaml.example is not updated here (#432 adds the line).
• The CLAUDE.md hunk is a trailing-newline removal only.

Housekeeping:

• The first commit touched ~30 files and the second reverted nearly all of them. Please squash so the history matches what lands.
• CI conventional-title is red on the [AMD Support][1/3] prefix.

[ishandhanani]

bet

[ishandhanani]

Merged main into this branch (5b61ec6) so it lands cleanly on top of #486, #487, and #489. Three files conflicted and were resolved by hand; the rest of the branch is unchanged.

What moved, and why:

• should_set_visible_devices() is now a BackendProtocol member. refactor: ports through NodePortAllocator, backends through BackendProtocol #489 made every backend question go through the protocol (no getattr(backend, ...) in src/, enforced by tests/test_backend_protocol.py). The rename from this PR stays; SGLang, TRT-LLM, and the mocker answer True, vLLM answers from set_visible_devices.
• _visible_device_environment reads self.backend.should_set_visible_devices() and self.config.dynamo.sidecar directly. Same behavior as before, without the getattr defaults. Both worker launch paths build the mask from runtime.visible_devices_env through this one helper.
• Static router pre-start probe reads config.health_check and passes runtime.srun_options directly. The MockConfig test fake gained health_check: None, which is how tests opt out of the probe; the probe itself is covered by test_sglang_router_starts_only_after_workers_are_healthy.
• SGLangRouterFrontend.resolve_worker_host passes the network interface. The base class already did after refactor(frontends): the frontend class owns registration, validation, worker shape, and endpoints #487; the override now matches.

Validation on the merged head: full suite 3034 passed on Python 3.13 and 3.10, make lint scope clean, schema docs in sync, and srtctl dry-run on all 26 example recipes differs from main only by the GPU subset visibility variable line this PR adds.