refactor(srt): compose AgentX recipes and select power from the master row

.agents/skills/debug-runs/SKILL.md

@@ -111,9 +111,10 @@ Steps:
    of `python -m infx.launch run` (`inferencex-e2e/infx/launch/drivers/srt/`): it binds the
    matrix row's `srt-recipe` (`inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/<srt-recipe-dir>/<srt-recipe>`)
    with `python -m infx.srt_slurm.single_node prepare`, renders the job-local `srtslurm.yaml`
-   from the cluster's record, then submits it through srtctl. Read the recipe for the image
-   (`model.container`), server args and env, and the cluster record for mounts and the job env
-   (`IMAGE`, `TP`, `PRECISION`, `SPEC_DECODING`, `CONC`, …). On Slurm clusters, use
+   from the cluster's record, then submits it through srtctl. Read the bound `recipe.yaml` that
+   `prepare` writes for the image (`model.container`, the master `image`), server args and env,
+   and the cluster record for mounts and the job env (`IMAGE`, `TP`, `PRECISION`,
+   `SPEC_DECODING`, `CONC`, …). On Slurm clusters, use
    `salloc` or `srun` with the squash image. On the **bare-metal `-tw` pools, use `docker run`**
    on the node directly without `srun`.
 3. **Always diff against a working node or working SKU** for reference. Most node failures

.claude/commands/add-model-hardware.md

@@ -98,10 +98,10 @@ adjusted for this hardware's quirks."
 Copy the sibling recipe (`base:` plus one `override_*` variant per matrix point) and adjust.
 Engine flags are `roles.agg.args` keys without the leading `--`; env vars go in
 `roles.agg.env`; each variant names its `CONC` (and `KV_OFFLOADING` for AgentX) in
-`benchmark.env`. Fixed-sequence recipes are fragments. Omit `model.path`/`container`/`precision`,
-`benchmark.type`/`command` and the client env, which the shared block and binder supply (see
-`RECIPES.md`). AgentX recipes need `model.container` equal to the master-config `image`. Things
-that vary and must be checked against the sibling /
+`benchmark.env`. Recipes, fixed-sequence and AgentX, are fragments. Omit
+`model.path`/`container`/`precision`, `benchmark.type`/`command` and the client env, which the
+shared block and binder supply (see `RECIPES.md`). Things that vary and must be checked against
+the sibling /
 the model's `recipes.vllm.ai` page:
 - **Mandatory model flags** (carry from the sibling): block size, parser flags
   (`--tool-call-parser` / `--reasoning-parser`), `--language-model-only` for text-only sweeps,

.claude/commands/nuke.md

@@ -68,7 +68,8 @@ Each PR triggers a full GPU sweep, so surface the total PR count explicitly.
 ## Step 4 — create one PR per family
 
 Use these helpers (write them to /tmp) for precise, per-config-key edits. A blind
-`sed` is unsafe because the same old tag appears under many keys.
+`sed` is unsafe because the same old tag appears under many keys. Only the master `image:`
+changes: the srt-slurm binder writes it into the recipe.
 
 `/tmp/edit_image.py`:
 ```python
@@ -104,56 +105,13 @@ block += ["  description:", f'    - "{desc}"', "  pr-link: PRLINK_PLACEHOLDER"]
 open(f,'w').write(content + '\n' + '\n'.join(block) + '\n')
 ```
 
-`/tmp/edit_recipe_container.py` (single-node runs go through the srt-slurm
-recipe each master search-space row names in `srt-recipe:`, and
-`inferencex-e2e/infx/srt_slurm/single_node.py` rejects the run unless the recipe's
-`model.container` equals the master `image:` exactly):
-```python
-#!/usr/bin/env python3
-# Usage: edit_recipe_container.py <master_yaml> <new_image> <key1> [key2 ...]
-import os, re, sys
-f, new_image, keys = sys.argv[1], sys.argv[2], sys.argv[3:]
-# srt-recipe names a file under the key's srt-recipe-dir, below this root
-root = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(f))), "benchmarks/single_node/srt-slurm-recipes")
-master = open(f).read().split('\n')
-recipes = set()
-for key in keys:
-    kre = re.compile(r'^' + re.escape(key) + r':\s*$')
-    start = next((i for i,l in enumerate(master) if kre.match(l)), None)
-    if start is None: sys.exit(f"ERROR: key not found: {key}")
-    directory, names = None, set()
-    for j in range(start+1, len(master)):
-        if re.match(r'^[A-Za-z0-9._-]+:\s*$', master[j]): break  # next top-level key
-        d = re.match(r'^\s+srt-recipe-dir:\s*(\S+)\s*$', master[j])
-        if d: directory = d.group(1)
-        names.update(re.findall(r'(?<![\w-])srt-recipe:\s*([^\s,}:]+)', master[j]))
-    if directory is None or not names: sys.exit(f"ERROR: no srt-recipe-dir/srt-recipe for key {key}")
-    recipes.update(os.path.join(directory, n) for n in names)
-for r in sorted(os.path.join(root, x) for x in recipes):
-    lines = open(r).read().split('\n')
-    in_model, hit = False, False
-    for i, l in enumerate(lines):
-        if re.match(r'^\s*model:\s*$', l): in_model, indent = True, len(l) - len(l.lstrip()); continue
-        if in_model and l.strip() and len(l) - len(l.lstrip()) <= indent: in_model = False
-        m = re.match(r'^(\s+)container:\s*(.+?)\s*$', l) if in_model else None
-        if m:
-            lines[i] = f"{m.group(1)}container: {new_image}"; hit = True
-            print(f"{r}: {m.group(2)} -> {new_image}"); break
-    if not hit: sys.exit(f"ERROR: no model.container in {r}")
-    open(r, 'w').write('\n'.join(lines))
-```
-
-If another key's `srt-recipe-dir` plus `srt-recipe` resolves to the same recipe file, stop and ask the user:
-bumping it would break that other key's `model.container == image` check.
-
 For each family, run strictly sequentially because git checkouts can't be parallel:
 
 ```bash
 git checkout main -q && git reset --hard origin/main -q
 branch="klaud/<basekey>-<TAG>"
 git checkout -b "$branch" -q
 python3 /tmp/edit_image.py <master.yaml> <NEW_IMAGE> <key> [<key>-mtp]
-python3 /tmp/edit_recipe_container.py <master.yaml> <NEW_IMAGE> <key> [<key>-mtp]
 python3 /tmp/append_changelog.py inferencex-e2e/perf-changelog.yaml "<DESC>" <key> [<key>-mtp]
 git add -A
 git commit -q -m "[Klaud Cold] Update <basekey>[ (+mtp)] <PHRASE> to <TAG>"

.github/AGENT_OPERATIONS.md

@@ -74,7 +74,7 @@ Multinode disaggregated results add `prefill_gpu_energy_j`, `decode_gpu_energy_j
 
 Every power result, valid or invalid, carries `power_metric_schema_version`. Version 2 defines each unprefixed `joules_per_*` field as whole-deployment GPU-board energy over the named denominator; role-scoped energy uses the explicit `prefill_*` / `decode_*` keys. Rows without the field predate the whole-deployment switch and their unprefixed joules are not comparable across topologies.
 
-For srt-slurm recipes, `telemetry.enabled: true` with `telemetry.dcgm_exporter` enables official energy collection. The Git submodule pointer at `inferencex-e2e/utils/srt-slurm` is the source of truth for every srt-slurm job, including TileRT. CI derives `POWER_PRODUCER_SHA` from the launcher stamp. The aggregate-power and AgentX power tests validate telemetry and provenance. These local tests do not prove hardware power collection. Eligible recipe-gated `dynamo-sglang` dcgm-power lanes are validated.
+A multinode master row with `power: true` enables official energy collection: the launcher merges `inferencex-e2e/configs/srt-recipes/telemetry-dcgm.yaml` into its srt-slurm recipe. The Git submodule pointer at `inferencex-e2e/utils/srt-slurm` is the source of truth for every srt-slurm job, including TileRT. CI derives `POWER_PRODUCER_SHA` from the launcher stamp. The aggregate-power and AgentX power tests validate telemetry and provenance. These local tests do not prove hardware power collection. Eligible `dynamo-sglang` dcgm-power lanes are validated.
 
 Power audit artifacts are named `power_audit_<result>` and contain the multinode `power_validation_<result>_*.json` sidecars. They are uploaded even when validation fails.
 

.github/klaud-candidate-prompt.md

@@ -34,9 +34,10 @@ Check the exact selected launch path for existing engine patches as well as prop
 
 Edit only the family's master image and its already referenced, unshared recipe's
 source-backed compatibility flags/environment or srt-slurm YAML image/backend settings.
-Match model.container and identity.container.image to the master. Preserve model, precision,
-topology, speculation, workload/dataset, duration, resources, recipe references, all points
-and default evals. No broad tuning, shared code/workflows or other-family edits.
+Recipes never set model.container or identity.container.image; the binder writes both from
+the master image. Preserve model, precision, topology, speculation, workload/dataset,
+duration, resources, recipe references, all points and default evals. No broad tuning,
+shared code/workflows or other-family edits.
 ZERO runtime engine/serving-stack patching: no source/site-packages/container rewrites,
 overlays, monkey patches or forked/rebuilt engine wheels, including pre-existing patches.
 If the shipped stack needs one, report incompatible. Preserve nodes:N, MTP chat templates,

.github/workflows/benchmark-multinode-tmpl.yml

@@ -145,6 +145,7 @@ env:
   EVAL_SUITE: ${{ inputs.eval-suite }}
   EVAL_CONC: ${{ inputs.eval-conc }}
   EVAL_LIMIT: ${{ inputs.eval-limit }}
+  POWER: ${{ fromJSON(inputs.config).power && '1' || '0' }}
   REQUIRE_POWER: ${{ inputs.require-power && '1' || '0' }}
   POWER_PRODUCER_SHA: ${{ inputs.power-producer-sha }}
   # Agentic-coding env. Fixed-seq-len jobs leave these empty.

AGENTS.md

@@ -111,7 +111,7 @@ Deleting a test that fails these questions needs no replacement. Do not preserve
 - Every change that can affect benchmark performance and every recipe addition or modification requires a new `inferencex-e2e/perf-changelog.yaml` entry. The file is append-only and byte-sensitive. Preserve all existing bytes and separator whitespace, and append only at the tail.
 - One PR maps to one `inferencex-e2e/perf-changelog.yaml` block. By convention, a PR with a perf changelog entry is a change that affects inference performance or otherwise requires a rerun to collect up-to-date results. That block may list one or more `config-keys` to run, but do not add a second block for the same PR, even when the change evolved across commits. Revise the PR's existing block so it describes the final state of the change.
 - New `inferencex-e2e/perf-changelog.yaml` entries must be English-only. Do not add Chinese translations or bilingual descriptions; the bilingual README and GitHub-content rules do not apply to these entries. Leave historical entries unchanged.
-- Multi-node srt-slurm changes update the recipe YAML and matching master config together. Fixed-sequence recipes are fragments that get the master image, model, precision and lengths from the binder; for AgentX image bumps, `model.container` must equal `image`.
+- Multi-node srt-slurm changes update the recipe YAML and matching master config together. srt-slurm recipes, fixed-sequence and AgentX, are fragments that get the master image, model, precision (and fixed-sequence lengths) from the binder.
 - Every speculative fixed-sequence benchmark renders prompts with the chat template: the binder (`inferencex-e2e/infx/srt_slurm/workload.py::bind_workload`) sets `benchmark.env.USE_CHAT_TEMPLATE: "true"` for single-node recipes that speculate, which `python3 -m infx.bench fixed-seq` (run by `srt_fixed_sequence.sh`) turns into `--use-chat-template` for the benchmark client.
 - Benchmarks create no new directories under `/workspace`. Root containers must not leave root-owned files in shared AMD runner workspaces.
 - Generated configuration is not runtime proof. Run the narrowest local check, then the applicable smoke, sweep, or eval procedure from the [task routing table](inferencex-e2e/docs/index.md#task-routing).