Add the services: block for sidecars and standalone Mooncake stores

CLAUDE.md

@@ -183,6 +183,24 @@ backend:
 
 **Validation:** In disaggregated mode, srtslurm rejects configs that set `mooncake_kv_store` without `disaggregation-transfer-backend: mooncake` on `sglang_config.prefill` or `sglang_config.decode`. This catches the common misconfiguration where the master process gets launched but workers fall back to default transport.
 
+### Services
+
+The top-level `services:` list declares long-running processes launched next to the job (see `docs/services.md`). Each entry has a `type` that selects a `ServiceKind` registered in `src/srtctl/services/` with `@register_service("<name>")`; the kind supplies defaults (command, start phase, criticality) and the env it injects, and `ServiceStageMixin` (`src/srtctl/cli/mixins/service_stage.py`) launches every kind the same way: resolve `placement.node` to physical nodes, optional clone/build of `source`, one `srun` per node, optional TCP `readiness` gate, `ManagedProcess` into the shared registry. `start_services("before_workers")` runs after the Mooncake master; `start_services("after_frontend")` runs after the frontend is healthy.
+
+```yaml
+services:
+  - name: store
+    type: mooncake-store       # generic (default) | mooncake-store
+    placement:
+      node: workers            # head | infra | prefill | decode | agg | workers
+    env:
+      MOONCAKE_GLOBAL_SEGMENT_SIZE: 100gb
+    readiness:
+      port: 8800
+```
+
+Adding a kind: subclass `ServiceKind`, set `default_command` / `default_start` / `default_critical`, override `validate`, `container_fallback`, `default_environment`, `forced_environment` as needed, decorate, and import it from `src/srtctl/services/__init__.py`. `srtctl dry-run` prints every service; add a `tests/test_dry_run.py` case when a kind adds visible fields.
+
 ### Host Setup
 
 `host_setup` runs commands on each node's **bare host, outside the container**, before any

docs/README.md

@@ -57,3 +57,4 @@ Once allocated, workers launch inside containers, discover each other through ET
 - [Profiling](profiling.md) - Performance analysis with torch/nsys
 - [Analyzing Results](analyzing.md) - Dashboard and visualization
 - [SGLang Router](sglang-router.md) - Alternative to Dynamo for PD disaggregation
+- [Services](services.md) - Sidecars and standalone stores launched next to the job

docs/SUMMARY.md

@@ -19,6 +19,7 @@
 - [SGLang Router](sglang-router.md)
 - [vLLM Router](vllm-router.md)
 - [Mooncake KV Store](mooncake-kv-store.md)
+- [Services](services.md)
 
 ## Benchmarking
 

docs/config-reference.md

@@ -31,6 +31,7 @@ This page is the prose guide: what each block means, how the pieces interact, an
 - [srun_options](#srun_options)
 - [setup_script](#setup_script)
 - [host_setup](#host_setup)
+- [services](#services)
 - [enable_config_dump](#enable_config_dump)
 - [Complete Examples](#complete-examples)
 
@@ -1754,6 +1755,53 @@ host_setup:
 
 ---
 
+## services
+
+Long-running processes srtctl launches and tracks next to the workers, frontend, and benchmark client. One list covers generic sidecars (an experimental router built from a PR) and typed services (a standalone Mooncake store per worker node). Full reference: [services.md](services.md).
+
+```yaml
+services:
+  - name: my-sidecar
+    type: generic                # generic (default) | mooncake-store
+    command:
+      - python3
+      - -m
+      - my_package.my_sidecar
+    args:
+      - --port
+      - "9000"
+    container: my-image          # alias or path; default: job container
+    env:
+      MY_FLAG: "1"
+    placement:
+      node: head                 # head | infra | prefill | decode | agg | workers
+    start: after_frontend        # after_frontend | before_workers
+    readiness:
+      port: 9000
+      timeout_seconds: 120
+    inherit_discovery_env: true  # ETCD_ENDPOINTS / NATS_SERVER
+    critical: false
+```
+
+| Field | Type | Default | Description |
+| --- | --- | --- | --- |
+| `name` | string | required | Unique; names `service_<name>.out` and the tracked process |
+| `type` | string | `generic` | Registered service kind; supplies defaults and injected env |
+| `command` | list[string] | type default | Argv, not shell-interpreted; required for `generic` |
+| `args` | list[string] | `[]` | Appended to `command` |
+| `container` | string | type fallback, then job container | Image or `srtslurm.yaml` alias |
+| `env` | dict | `{}` | Service environment; placeholders like `{node_ip}` are substituted |
+| `placement.node` | string | `head` | One instance for `head`/`infra`; one per node for `prefill`/`decode`/`agg`/`workers` |
+| `start` | string | type default | `after_frontend` (generic) or `before_workers` (mooncake-store) |
+| `readiness` | object | none | `port` + `timeout_seconds`; the job waits for it on every service node |
+| `inherit_discovery_env` | bool | `true` | Inject the Dynamo discovery env |
+| `critical` | bool | type default | A crash fails the run when true |
+| `source`, `build_command` | object, list[string] | none | Clone an immutable git rev and build once before launch; single-node placements only |
+| `build_timeout_seconds` | int | `1800` | `build_command` is killed when this runs out so a hung build cannot hold the allocation |
+| `preamble`, `cpus_per_task`, `cpu_bind`, `srun_options` | | none | Pass-through launch knobs for this service |
+
+---
+
 ## enable_config_dump
 
 Enable dumping worker configuration to JSON for debugging.

docs/mooncake-kv-store.md

@@ -9,6 +9,7 @@ First-class support for [Mooncake](https://github.com/kvcache-ai/Mooncake) as th
 - [Quick Start (vLLM)](#quick-start-vllm)
 - [What srtslurm Owns vs What You Set](#what-srtslurm-owns-vs-what-you-set)
 - [Configuration Reference](#configuration-reference)
+- [Standalone Store Services](#standalone-store-services)
 - [Master Metrics Endpoint](#master-metrics-endpoint)
 - [Validation](#validation)
 - [Common Configurations](#common-configurations)
@@ -173,6 +174,29 @@ backend:
 
 Older Mooncake versions do not recognize this option, so leave it out of those recipes. The existing `--eviction_high_watermark_ratio` controls memory eviction; the `--nof_...` option independently controls the NVMe-over-Fabrics SSD tier.
 
+## Standalone Store Services
+
+Mooncake can run the Store as a standalone process per node, so workers use embedded clients with `MOONCAKE_GLOBAL_SEGMENT_SIZE=0` while dedicated stores own the DRAM segments. In srtslurm that is a `services:` entry with `type: mooncake-store`: it starts after the master is healthy and before workers, gets `MOONCAKE_MASTER`, `MOONCAKE_TE_META_DATA_SERVER`, and `MOONCAKE_LOCAL_HOSTNAME` from the runtime, and defaults its container to `mooncake_kv_store.container`.
+
+```yaml
+services:
+  - name: store
+    type: mooncake-store
+    placement:
+      node: workers          # or prefill / decode for per-role segment sizes
+    args:
+      - --port
+      - "8800"
+    env:
+      MOONCAKE_PROTOCOL: rdma
+      MOONCAKE_DEVICE: "mlx5_0,mlx5_1"
+      MOONCAKE_GLOBAL_SEGMENT_SIZE: 100gb
+    readiness:
+      port: 8800
+```
+
+The worker side stays in the backend's per-mode env (`prefill_environment` / `decode_environment`). See [Services](services.md#example-standalone-mooncake-stores) for the full shape, per-role entries, and the co-location rules.
+
 ## Master Metrics Endpoint
 
 The `mooncake_master` admin HTTP server is always exposed on port `8702` on the infra node and starts before workers do (srtslurm waits for it). It serves:

docs/schema-reference.md

@@ -33,6 +33,7 @@ Top-level keys of a recipe YAML.
 | `enable_config_dump` | bool | `True` |  |
 | `setup_script` | str \| None | `None` | Custom setup script (runs before dynamo install and worker startup) e.g. "custom-setup.sh" -> runs /configs/custom-setup.sh |
 | `host_setup` | [HostSetupConfig](#hostsetupconfig) | `HostSetupConfig()` | Commands run on each node's bare host, outside the container, before any worker starts. Cluster-wide default lives in srtslurm.yaml as default_host_setup; a recipe that sets this block replaces that default. |
+| `services` | list[[ServiceConfig](#serviceconfig)] | `[]` | Long-running processes launched next to the job: generic sidecars (an experimental router built from a PR) and typed ones (a standalone Mooncake store per worker node). See docs/services.md. |
 | `identity` | [IdentityConfig](#identityconfig) | `IdentityConfig()` | Virtual identity — declares what *should* be running (verified against fingerprint) |
 | `reporting` | [ReportingConfig](#reportingconfig) \| None | `None` | Reporting configuration (status API, future: logs to S3, etc.) |
 
@@ -261,6 +262,31 @@ Commands run on the bare host of each allocated node, outside the container.
 | `ignore_failure` | bool | `False` | When True, a failing node logs a warning instead of failing the job. |
 | `timeout_seconds` | int | `300` | Per-node wall-clock budget for commands and for teardown. |
 
+### ServiceConfig
+
+One entry of the top-level ``services:`` list.
+
+| Key | Type | Default | Description |
+|---|---|---|---|
+| `name` | str | required | Unique label; names the log file (``service_<name>.out``) and the tracked process. |
+| `type` | str | `'generic'` | Service kind. ``generic`` (default) launches exactly what you wrote; ``mooncake-store`` runs a standalone Mooncake Store wired to the managed master. See ``docs/services.md`` for the kinds. |
+| `command` | list[str] \| None | `None` | Argv to launch (not shell-interpreted). Required for ``generic``; typed kinds supply a default. |
+| `args` | list[str] | `[]` | Extra argv appended to ``command``. |
+| `container` | str \| None | `None` | Container image or ``srtslurm.yaml`` alias. Defaults to the kind's fallback (Mooncake's ``mooncake_kv_store.container``), then the job container. |
+| `env` | dict[str, str] | `{}` | Environment for the service process, on top of what the kind injects. |
+| `source` | [ServiceSourceConfig](#servicesourceconfig) \| None | `None` | Optional git source to clone before ``build_command`` and ``command`` run. Single-node placements only. |
+| `build_command` | list[str] \| None | `None` | Argv run once inside the service container, from the clone, before ``command`` starts. Only meaningful with ``source``. |
+| `placement` | [ServicePlacementConfig](#serviceplacementconfig) | `ServicePlacementConfig()` | Where the service runs. Default ``head``. |
+| `start` | str \| None | `None` | ``after_frontend`` (default for ``generic``) or ``before_workers`` (default for ``mooncake-store``). |
+| `readiness` | [ServiceReadinessConfig](#servicereadinessconfig) \| None | `None` | Optional TCP port gate; the job waits for it on every service node before continuing. |
+| `inherit_discovery_env` | bool | `True` | Inject ``ETCD_ENDPOINTS`` / ``NATS_SERVER`` so the service can register with the job's Dynamo discovery plane. |
+| `critical` | bool \| None | `None` | When true a crash fails the run, like a worker dying. Default false for ``generic`` (a dead sidecar costs its own log, not the run) and true for ``mooncake-store``. Set true for anything in the live request path. |
+| `preamble` | str \| None | `None` | Shell run inside the container before ``command`` (``ulimit`` and friends). |
+| `cpus_per_task` | int \| None | `None` | Optional ``srun --cpus-per-task``. |
+| `cpu_bind` | str \| None | `None` | Optional ``srun --cpu-bind``. |
+| `srun_options` | dict[str, str] | `{}` | Extra srun options for this service only. |
+| `build_timeout_seconds` | int | `1800` | Kill ``build_command`` after this many seconds. |
+
 ### IdentityConfig
 
 Virtual identity for runtime verification and reproduction.
@@ -326,6 +352,33 @@ Configuration for a metrics exporter deployed on worker nodes.
 | `port` | int | required |  |
 | `command` | str \| None | `None` |  |
 
+### ServiceSourceConfig
+
+Git source to build a service from before launching it.
+
+| Key | Type | Default | Description |
+|---|---|---|---|
+| `git` | str | required | Repository URL to clone. |
+| `rev` | str | required | Immutable ref to check out: a commit SHA, a tag, or ``refs/pull/<n>/head`` for an unmerged PR. Branch names are rejected because they move out from under a build. |
+| `path` | str \| None | `None` | Optional subdirectory of the clone that ``build_command`` and ``command`` run from. Defaults to the repository root. |
+
+### ServicePlacementConfig
+
+Where a service runs.
+
+| Key | Type | Default | Description |
+|---|---|---|---|
+| `node` | str | `'head'` | ``head`` or ``infra`` (one instance), ``prefill`` / ``decode`` / ``agg`` (one instance per distinct physical node that role's workers use), or ``workers`` (one instance per worker node). |
+
+### ServiceReadinessConfig
+
+TCP readiness gate: the launch blocks until ``port`` accepts connections on every service node.
+
+| Key | Type | Default | Description |
+|---|---|---|---|
+| `port` | int | required | TCP port the service listens on. |
+| `timeout_seconds` | int | `120` | How long to wait per node before failing the job. |
+
 ### IdentityModelConfig
 
 Virtual model identity for runtime verification.