refactor: standardize srt-slurm v2.2.1 and migrate recipes

.github/AGENT_OPERATIONS.md

@@ -96,7 +96,7 @@ Multinode disaggregated results add `prefill_gpu_energy_j`, `decode_gpu_energy_j
 
 Every power result — valid or invalid, single-node or multinode — carries `power_metric_schema_version`. Version 2 defines each unprefixed `joules_per_*` field as whole-deployment GPU-board energy over the named denominator; role-scoped energy uses the explicit `prefill_*` / `decode_*` keys. Rows without the field predate the whole-deployment switch and their unprefixed joules are not comparable across topologies.
 
-For srt-slurm recipes, `telemetry: {provider: dcgm-power}` enables official energy collection. `runners/launch_gb200-nv.sh`, `runners/launch_gb300-nv.sh`, and `runners/launch_h200-dgxc-slurm.sh` are the source of truth for `POWER_SRT_SLURM_PIN`. CI derives `POWER_PRODUCER_SHA` from the launcher stamp. `utils/test_gb300_power_official_contract.py` exercises launcher routing; the aggregate-power and AgentX power tests validate telemetry, provenance, and lifecycle behavior. These local tests do not prove hardware power collection. Eligible recipe-gated `dynamo-sglang` dcgm-power lanes are validated.
+For srt-slurm recipes, `telemetry.enabled: true` with `telemetry.dcgm_exporter` enables official energy collection. The Git submodule pointer at `utils/srt-slurm` is the source of truth for the shared srt-slurm commit, used by both power and non-power NVIDIA lanes. TileRT is the single documented fork exception. CI derives `POWER_PRODUCER_SHA` from the launcher stamp. The aggregate-power and AgentX power tests validate telemetry and provenance. These local tests do not prove hardware power collection. Eligible recipe-gated `dynamo-sglang` dcgm-power lanes are validated.
 
 Power audit artifacts are named `power_audit_<result>` and contain `power_validation_<result>.json` for single-node runs or `power_validation_<result>_*.json` for multinode runs. They are uploaded even when validation fails.
 

.github/workflows/test-process-result.yml

@@ -15,13 +15,13 @@ on:
       - 'runners/test_native_collector_barriers.py'
       - 'runners/test_native_collector_receipts.py'
       - 'utils/test_native_multinode_power.py'
-      - 'benchmarks/multi_node/srt-slurm-recipes/sglang/deepseek-v4/**/*.yaml'
-      - 'benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/**/*.yaml'
-      - 'benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k2.6/b200-fp4/**/*.yaml'
-      - 'benchmarks/multi_node/srt-slurm-recipes/sglang/deepseek-v4/agentic/agg-h200-tp8-mtp-kvoffload.yaml'
-      - 'benchmarks/multi_node/srt-slurm-recipes/sglang/glm5.2/agentic/disagg-h200-2p2d-pcp8-tp8-dp8-mtp.yaml'
-      - 'benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/gb200-*/**/*.yaml'
-      - 'benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/gb300-*/**/*.yaml'
+      - 'benchmarks/multi_node/srt-slurm-recipes/dsv4/sglang/**/*.yaml'
+      - 'benchmarks/multi_node/srt-slurm-recipes/dsv4/vllm/**/*.yaml'
+      - 'benchmarks/multi_node/srt-slurm-recipes/kimik2.6/vllm/b200-fp4/**/*.yaml'
+      - 'benchmarks/multi_node/srt-slurm-recipes/dsv4/sglang/h200-fp8/agentx/agg-tp8-mtp-kvoffload.yaml'
+      - 'benchmarks/multi_node/srt-slurm-recipes/glm5.2/sglang/h200-fp8/agentx/disagg-2p2d-pcp8-tp8-dp8-mtp.yaml'
+      - 'benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-*/**/*.yaml'
+      - 'benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-*/**/*.yaml'
       - 'configs/nvidia-master.yaml'
       - 'runners/launch_gb200-nv.sh'
       - 'runners/launch_gb300-nv.sh'

.gitmodules

@@ -1,3 +1,6 @@
 [submodule "utils/aiperf"]
 	path = utils/aiperf
 	url = https://github.com/SemiAnalysisAI/aiperf.git
+[submodule "utils/srt-slurm"]
+	path = utils/srt-slurm
+	url = https://github.com/NVIDIA/srt-slurm.git

benchmarks/multi_node/amd_utils/models.yaml

@@ -403,7 +403,7 @@ DeepSeek-V4-Pro-DI:
   # DeepSeek-V4 built-in MTP: load via the EAGLE spec path with eagle-topk 1 (a single
   # MTP chain). NOT NEXTN -- the V3/R1 NEXTN loader crashes the dsv4 decode server at
   # init on the V4 architecture. Mirrors the sglang dsv4 recipes
-  # (srt-slurm-recipes/sglang/deepseek-v4/*: EAGLE + eagle-topk 1, in-checkpoint draft,
+  # (srt-slurm-recipes/dsv4/sglang/*: EAGLE + eagle-topk 1, in-checkpoint draft,
   # no draft-model-path). Consumed by build_server_config only when DECODE_MTP_SIZE > 0
   # (spec-decoding: mtp); a no-op for the spec-none base scenario (DECODE_MTP_SIZE=0).
   # build_server_config appends --speculative-num-steps/--speculative-num-draft-tokens

benchmarks/multi_node/srt-slurm-recipes/RECIPES.md

@@ -1,92 +1,79 @@
-# Registering Recipes from srtslurm
+# srt-slurm recipes
 
-For disaggregated multi-node configurations (`dynamo-sglang`, `dynamo-trt`), recipes are stored in the external [srtslurm](https://github.com/NVIDIA/srt-slurm) repository. This doc covers staging those recipes in InferenceX.
+**English** | [中文](./RECIPES_zh.md)
 
-## 1. Locate source recipes in srtslurm
+InferenceX owns the recipes in this directory. Every NVIDIA srt-slurm launcher uses `setup_srt_slurm()` in [`runners/slurm_utils.sh`](../../../runners/slurm_utils.sh), makes a job-local Git clone of the pinned submodule, and copies this entire tree into `recipes/`. The shared helper records the actual revision in `srt-slurm-sha.txt`; power lanes copy that revision into `power-producer-sha.txt` for result validation.
 
-```bash
-# Example: H200 sglang disagg recipes
-ls /path/to/srtslurm/recipes/h200/
-# 1k1k/  8k1k/
-```
+The shared version is the Git submodule pointer at [`utils/srt-slurm`](../../../utils/srt-slurm), currently [v2.2.1](https://github.com/NVIDIA/srt-slurm/releases/tag/v2.2.1) (`984180e5b8755aef85e9995048b5a16cb5336bce`). Update that submodule pointer when upgrading, then run the recipe and integration checks. Do not add model-specific checkout branches to launchers.
 
-## 2. Recipe structure
-
-Each recipe YAML contains:
-- `name`: Recipe identifier
-- `model`: Model path/container info
-- `resources`: GPU type, prefill/decode node/worker counts
-- `backend.sglang_config`: Prefill and decode configuration (tp-size, dp-size, ep-size, dp-attention, etc.)
-- `benchmark`: ISL/OSL and concurrency settings
-
-## 3. Add config to nvidia-master.yaml
-
-```yaml
-dsr1-fp8-h200-dynamo-sglang:
-  image: lmsysorg/sglang:v0.5.8-cu130-runtime
-  model: deepseek-ai/DeepSeek-R1-0528
-  model-prefix: dsr1
-  runner: cluster:h200-dgxc
-  precision: fp8
-  framework: dynamo-sglang
-  multinode: true
-  disagg: true
-  scenarios:
-    fixed-seq-len:
-    - isl: 1024
-      osl: 1024
-      search-space:
-      - conc-list: [1, 4, 16, 32, 64, 128, 256, 512]
-        prefill:
-        num-worker: 1
-        tp: 8
-        ep: 1
-        dp-attn: false
-        additional-settings:
-        - "CONFIG_FILE=recipes/h200/1k1k/bs128-agg-tp.yaml"
-      decode:
-        num-worker: 0
-        tp: 8
-        ep: 1
-        dp-attn: false
-```
+InferenceX requires srt-slurm 2.0 or newer and `schema: 2` recipes. Legacy recipe layouts are unsupported; migrate them before adding them to this tree.
+
+## Directory and filename convention
+
+Store every recipe at `<model-prefix>/<engine>/<gpu>-<precision>/<workload>/<recipe>.yaml`:
 
-## 4. Field mapping (srtslurm → nvidia-master.yaml)
-
-| srtslurm field | nvidia-master.yaml field |
-|----------------|-------------------------|
-| `resources.prefill_workers` | `prefill.num-worker` |
-| `resources.decode_workers` | `decode.num-worker` |
-| `sglang_config.prefill.tp-size` | `prefill.tp` |
-| `sglang_config.prefill.ep-size` | `prefill.ep` |
-| `sglang_config.prefill.enable-dp-attention` | `prefill.dp-attn` |
-| `benchmark.concurrencies` (parsed) | `conc-list` |
-| Recipe file path | `additional-settings: CONFIG_FILE=...` |
-
-## 5. Common patterns
-
-- **Aggregated (AGG)**: Single node, `num-worker: 1` for prefill, `num-worker: 0` for decode
-- **TEP (Tensor-Expert Parallel)**: `dp-attn: false`, `ep: 1`
-- **DEP (Data-Expert Parallel)**: `dp-attn: true`, `ep: 8` (typically)
-- **Low latency**: More decode workers (e.g., 9), lower concurrencies
-- **High throughput**: Fewer decode workers, higher concurrencies
-
-## 6. Add perf-changelog entry
-
-```yaml
-- config-keys:
-    - dsr1-fp8-h200-dynamo-sglang
-  description:
-    - "Add DSR1 FP8 H200 Dynamo SGLang disaggregated multinode configuration"
-    - "Image: lmsysorg/sglang:v0.5.8-cu130-runtime"
-    - "Recipes sourced from srtslurm repo (recipes/h200/)"
-  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX
+```text
+dsr1/sglang/b200-fp4/8k1k/disagg-stp-mtp-variants.yaml
+glm5.2/sglang/h200-fp8/agentx/disagg-1p1d-pcp8-tp8-dp8-mtp6-hicache.yaml
+qwen3.5/trtllm/gb300-fp4/agentx/disagg-1p7d-dep4-tep8-c7-b1-mtp-kvoffload.yaml
 ```
 
-## 7. Validate
+- Use the master config's `model-prefix` and `precision` labels. Engines are `sglang`, `vllm`, `trtllm`, and `tilert`; frontend selection remains explicit inside the recipe. Hardware directories use GPU types such as `b200` and `gb300`, rather than cluster names.
+- Workloads are `1k1k`, `8k1k`, or `agentx`. Existing bundles spanning several fixed sequence lengths use `fixed-seq-len`; keep their override selectors intact.
+- Use lowercase, hyphen-separated filenames beginning with `agg` or `disagg`. Include topology and the settings that distinguish sibling recipes, such as parallelism, batch size, concurrency, MTP, offload, or cache configuration. Avoid dates, numbered latency/throughput labels, and repeating the model or hardware already in the path.
+- In topology names, `1p4d` denotes prefill/decode worker counts, not necessarily physical nodes. Role-qualified `p-tp4` and `d-tp8` identify prefill/decode TP; `b` denotes batch size and `c` concurrency. The YAML is authoritative for runtime settings.
+- Name override bundles `*-variants.yaml`. Keep distinct sweep entry files separate even when their contents match: recipe paths participate in eval grouping. The Qwen3.5 `*-stp-sweep.yaml` and `*-mtp-sweep.yaml` pair preserves that existing distinction.
+- Update `CONFIG_FILE` and `EVAL_CONFIG_FILE` references in active and deprecated master configs, launcher path rules, workflow filters, and local documentation together when moving a file. Preserve upstream source URLs as provenance and leave historical performance-changelog entries unchanged. No aliases for the old layout are provided.
+
+Shared runtime assets stay under `configs/` beside the model directories; they are not standalone recipes. The four files in `configs/dsv4-moe-load-balancer-configs/` are copied verbatim from NVIDIA/srt-slurm commit `deb1dfd9934398664f92d194169c183e009da83b`, preserving the EPLB initial expert assignments used by 17 DSV4 TRT recipes. `setup_srt_slurm()` stages them into the job checkout's `configs/` directory for the recipes' bind mounts. Keeping a recipe in this tree does not activate it; the master configs determine the benchmark matrix.
+
+## TileRT exception
+
+For `FRAMEWORK=tilert`, `setup_srt_slurm()` fetches the SemiAnalysisAI/srt-slurm fork directly at `6bc3f306bdafa1edfb5dded2fcda8f1ccede1bde` into the job checkout. This is the schema-2 TileRT port in [SemiAnalysisAI/srt-slurm#13](https://github.com/SemiAnalysisAI/srt-slurm/pull/13). It is the only alternate checkout; its pin lives in that helper because the TileRT backend and router are absent from the NVIDIA pin. TileRT uses the same schema-2 recipe layout and native post-eval dispatch as NVIDIA. TileRT jobs need network access to the fork at setup time. Remove the fork exception once those features are available upstream.
+
+## Schema 2 and master configuration
+
+Recipes use `schema: 2`, `engine`, and `roles`. Each worker role owns its node count, worker count, GPU allocation, environment, and engine arguments. `resources` retains GPU hardware facts. `placement` controls the frontend and benchmark location, `services` describes auxiliary processes, and `dynamo.source` selects the Dynamo package or source revision.
+
+| Recipe field | `configs/nvidia-master.yaml` field |
+|---|---|
+| `roles.prefill.workers` | `prefill.num-worker` |
+| `roles.decode.workers` | `decode.num-worker` |
+| `roles.prefill.args.tp-size` (SGLang) | `prefill.tp` |
+| `roles.prefill.args.ep-size` (SGLang) | `prefill.ep` |
+| `roles.prefill.args.enable-dp-attention` | `prefill.dp-attn` |
+| `benchmark.concurrencies` | `conc-list` |
+| Recipe path, optionally with an override selector | `additional-settings: CONFIG_FILE=recipes/...yaml` |
+
+Keep the recipe and master configuration synchronized. The launcher executes the recipe; the master configuration supplies result labels and scheduling metadata. For aggregate recipes use `roles.agg`; `roles.decode.nodes: colocate` shares prefill nodes and contributes no additional worker nodes to scheduling.
+
+All referenced recipes must be checked in: srt-slurm 2 ships curated examples instead of the historical `recipes/` archive. The initial migration restores 204 previously external recipes and two still-referenced AgentX recipes from InferenceX history. Master-config paths follow the layout above; existing override selectors are preserved.
+
+## Migration and validation
+
+Install the shared pin in an isolated environment, then use its CLI:
 
 ```bash
+# Verify each supported recipe directory before rewriting it.
+srtctl migrate --verify -f benchmarks/multi_node/srt-slurm-recipes/dsr1/sglang
+srtctl migrate --in-place -f benchmarks/multi_node/srt-slurm-recipes/dsr1/sglang
+# Repeat for the other model/engine directories.
+# Use the pinned TileRT fork for glm5.1/tilert/.
+python -m pytest utils/matrix_logic/ -q
 python -m infx.matrix.generate full-sweep \
   --config-files configs/nvidia-master.yaml \
-  --framework dynamo-sglang
+  --framework dynamo-sglang dynamo-trt dynamo-vllm --multi-node
 ```
+
+Validate recipes with the exact launcher pin, including all override variants. For a path-only reorganization, compare generated matrices before and after with the path mapping applied; all other fields, including eval selection and node counts, must match. A passing local schema check does not replace the full hardware sweep and evals.
+
+The initial migration also resolves compatibility issues that `srtctl migrate` cannot fix itself:
+
+- SGLang Model Gateway recipes use `frontend.type: sglang-router`; in v2.2.1, `sglang` selects a direct worker without a router.
+- Duplicate YAML keys retain the value selected by the former PyYAML loader.
+- DCGM telemetry uses `collect_interval_ms: 1000` instead of `provider` and `default_frequency`. The collector derives its shutdown budget; an explicit ten-second budget is too short for the current validator. Dedicated discovery-service placement is preserved from the original recipes. The pinned upstream runtime rejects telemetry with dedicated infrastructure nodes; this remains a power compatibility blocker rather than changing the original topology to satisfy validation. H200 custom recipes declare a default concurrency that the launcher replaces before submission.
+- DeepSeek-V4 vLLM benchmarks use the supported `custom_tokenizer` loader. Retired `warmup_req_rate: inf` fields are removed; the current upstream client uses its fixed warmup rate of 250 requests per second.
+- The power reader accepts both generations of samples CSV while validating utilization values and continuing to compute board energy from watts.
+- Post-eval selection uses native `post_eval.command` and `post_eval.passthrough_env` with [`srt_eval.sh`](../srt_eval.sh). TRT AgentX recipes declare their existing Dynamo fork with `dynamo.source.git`; launchers no longer rewrite the srt-slurm source.
+
+Append a new entry to the physical end of `perf-changelog.yaml` for every recipe or runtime change. Preserve all historical bytes. Validate the PR with `full-sweep-fail-fast`, including evals, before following the repository's review and artifact-reuse merge process.