Add the services: block for sidecars and standalone Mooncake stores

[ishandhanani]

Stacked on #395 (Track 2 step 7 of #385). Draft until the stack below it merges.

What

One top-level services: list for every long-running process launched next to the job: an experimental router built from a PR, a standalone Mooncake Store per worker node, a debugging HTTP server. Each entry has a type that selects a registered ServiceKind (@register_service, same pattern as @register_benchmark):

• generic (default): launches exactly the argv written. Starts after the frontend, non-critical.
• mooncake-store: standalone Mooncake Store wired to the managed master. Default command, starts before workers, critical, container falls back to mooncake_kv_store.container, MOONCAKE_MASTER / MOONCAKE_TE_META_DATA_SERVER / MOONCAKE_LOCAL_HOSTNAME injected.

The kind supplies defaults and the env it injects. ServiceStageMixin launches every kind the same way: placement.node resolves to physical nodes (head, infra, prefill, decode, agg, workers), optional clone and build of an immutable git source, one srun per node, an optional TCP readiness gate, and a ManagedProcess in the shared registry for crash detection and teardown.

services:
  - name: thunderagent-router
    source:
      git: https://github.com/ai-dynamo/dynamo
      rev: refs/pull/14000/head
    build_command:
      - bash
      - -lc
      - "cd lib/bindings/python && maturin develop --uv && cd ../../.. && pip install -e ."
    command:
      - python3
      - -m
      - dynamo.thunderagent_router
    critical: true

  - name: store-decode
    type: mooncake-store
    placement:
      node: decode
    env:
      MOONCAKE_GLOBAL_SEGMENT_SIZE: 400gb
    readiness:
      port: 8800

Folds two open PRs onto the 2.0 shape

• feat(auxiliary-services): generic sidecar config for both execution paths #374 (auxiliary_services): the generic sidecar with source / build_command / inherit_discovery_env / critical, the head-node clone-on-bare-host then build-in-container flow, and the dry-run panel. Its --bash direct-runner half is gone with that path ([2.0] Remove the --bash direct-host execution path #386). Renamed to services: with container (alias-resolved by the shared walker) instead of container_image. Its unrelated kv_events_config: true aggregated-mode fix is carried as its own commit with the regression test.
• feat(mooncake): manage standalone store services #265 (mooncake_kv_store.standalone): the standalone Store services, as a typed service instead of a backend sub-block with a placements: {prefill, decode, aggregated} map. Per-role segment sizes are two entries placed on prefill and decode; preamble, cpus_per_task, cpu_bind, srun_options, the {node} / {node_ip} / {role} placeholders, and the TCP readiness check all carry over. Two services that listen on the same port on one node are rejected before launch (the co-location conflict feat(mooncake): manage standalone store services #265 checked). The worker-side backend.mooncake_kv_store (master, worker env, vLLM store_config) is unchanged.

Both external PRs can close in favour of this one once it lands; credit to their authors in the file history.

Also

• examples/features/services.yaml: a runnable HTTP log browser on the head node gated on its port.
• docs/services.md, a ## services section in docs/config-reference.md, a pointer section in docs/mooncake-kv-store.md, CLAUDE.md notes, regenerated docs/schema-reference.md.
• Tests: tests/test_services.py (schema, kinds, stage: placement resolution, env layering, source build, readiness failure teardown, port collisions), dry-run cases.

Cleanup guarantees

Nothing a service launches outlives the job. Every srun the stage starts, including the one-shot clone and build steps, is registered with the ProcessRegistry the moment it exists (not when the stage returns), so cleanup on normal exit, a failed stage, the SIGTERM handler, and the crash monitor all reach it. The stage also terminates what it started on any exception, including SystemExit from the signal handler. The clone step is bounded by its per-command git timeouts and the build by build_timeout_seconds (default 1800); a step that overruns is killed. A service that dies before its readiness port answers fails at once with its exit code instead of waiting out the timeout. Slurm step cancellation kills the whole step cgroup, so forked children inside the container go with the service.

Validation

• Full suite green (1808 passed); lint, format, schema-docs drift check, every example validates.
• sa-b200, Qwen3-0.6B: examples/features/services.yaml ran as job 11995: the log-browser service launched on the head node, its readiness gate on port 9911 passed after 21s, the benchmark completed with rollups written, and the service was torn down with the job on the cleanup path; the mooncake-store shape dry-run validated on a disagg recipe (no Mooncake image on that cluster to run it).
• Cleanup, live: job 12004 completed normally and sacct shows the service step cancelled with the workers and router at teardown; job 12005 was scancelled while the service was up, every step ended (service step CANCELLED), and a check on the compute node afterwards found no http.server, no engine processes, and no steps still allocated.

[ishandhanani]

Consolidated into #407, which carries these same commits as one PR against main (this description is reproduced there as one of its parts). Closing to keep review in one place; the branch stays for per-step reference via stack #398.