[2.0] The full 2.0 stack: schema v2, services, graceful shutdown, observability, MCP, and the migrator (#386-#406 and after) - #407
Merged
Conversation
srt-slurm 2.0 is the Slurm piece only. The direct-host lifecycle that rendered a Docker launcher from a recipe (srtctl apply --bash) is removed so every later 2.0 change has one implementation to write instead of two. Deleted: - src/srtctl/render/ (direct plan, host runner, container runner, and the direct stage mixins) - src/srtctl/templates/direct_container.sh.j2 - docs/direct-host.md - tests/test_direct_host_runner.py, tests/test_direct_plan.py, tests/test_direct_runner.py Edited: - cli/submit.py: drop the --bash flag, render_bash_script, and the bash-mode argument checks; the Slurm submit path is unchanged - tests/test_submit_cli.py: replace the direct-container test with a check that --bash is now rejected by argparse - docs (cli.md, README.md, docs/README.md, ruter.md, SUMMARY.md) and two comments in telemetry_stage.py, core/telemetry.py, ruter/normalize.py Nothing under core/power/ or the power hooks is touched. The path can return later as a separate execution target; nothing in 2.0 core depends on it. Part of the 2.0 plan: #385
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
The 563-file recipes/ tree was a benchmark-results archive, not a set of starting points. It is removed. examples/ now holds one small, runnable configuration per frontend and topology, all serving Qwen3-0.6B on one node so the files differ only in the frontend and prefill/decode layout: sglang/ dynamo-agg dynamo-disagg sglang-router-agg sglang-router-disagg vllm/ dynamo-agg dynamo-disagg vllm-router-agg vllm-router-disagg vllm-direct-agg trtllm/ dynamo-agg dynamo-disagg trtllm-serve-agg trtllm-serve-disagg mocker/ dynamo-agg features/ sweep override profiling examples/README.md documents the matrix, the srtslurm.yaml aliases the files rely on, and how to validate them. Wiring (carried from #338): CI's validate step, the Makefile target, the interactive selector, CODEOWNERS, docs, and tests point at examples/. validate_config_file now expands top-level `sweep:` files the same way it expands override files, so CI validates every file under examples/ including features/sweep.yaml. tests/test_configs.py checks that every topology example loads as a plain config, that every example validates, and that the documented matrix is present on disk. Part of the 2.0 plan: #385
test_e2e.py encoded the old 6P+2D two-node Qwen3-32B layout. The example-dependent tests now parametrize over every topology example, and the shared-node disaggregation checks assert the 1P+1D TP1 layout (decode_nodes: 0) that all four disagg examples use.
Current vLLM rejects the flag (argparse exit 2), which killed the direct vLLM example on the first cluster run.
dynamo.mocker derives its model name from --model-path, which for the /model container mount is "model", while sa-bench asks for the model path basename (SrtConfig.served_model_name). With a local model path the benchmark 404s with "Model not found"; it only worked for hf: paths. Pass --model-name explicitly. Surfaced by examples/mocker/dynamo-agg.yaml on sa-b200 (job 11846).
docs/config-reference.md is hand-written prose and drifts from the schema (the power docs merged last week already reference a field that #317 removed). This adds a generated, CI-checked field-level reference so the truth lives in the code. - src/srtctl/core/schema_docs.py walks the dataclass tree rooted at SrtConfig plus the four backend types and ClusterConfig, and renders one Markdown table per dataclass: YAML key (honoring marshmallow data_key), type (nested dataclasses linked, Literals expanded), default, and a description taken from the class docstring `Attributes:` block or the comment on the field. - `srtctl schema-docs` writes docs/schema-reference.md; `--check` exits 1 when the checked-in file is stale. `make schema-docs-check` and a CI lint step run the check; tests/test_schema_docs.py enforces it in the test suite too and covers the renderer and the CLI. - docs/config-reference.md points at the generated file as authoritative for keys, types, and defaults; SUMMARY.md links it; CLAUDE.md tells contributors to regenerate after any schema change. Part of the 2.0 plan: #385
Recipes gain a top-level `schema:` key (attribute `schema_version`, YAML key `schema` via marshmallow data_key). Absent means 1, the pre-2.0 layout; `schema: 2` is the 2.0 layout. Both load; unknown versions are rejected at load time. This is the dispatch point the structural 2.0 changes hang off, and it gives downstream recipes a way to say which layout they target instead of failing on unknown fields. - SrtConfig.schema_version with CURRENT_SCHEMA_VERSION / SUPPORTED_SCHEMA_VERSIONS - generate_override_configs and resolve_override_yaml carry a file-level `schema` (declared beside `base`) into every expanded variant - src/srtctl/core/migrate.py: ruamel round-trip migration that inserts or bumps `schema: 2` and preserves comments, key order, and quoting. Structural v1 -> v2 rewrites plug into _migrate_1_to_2 in later steps. - `srtctl migrate -f recipe.yaml [--in-place | --output PATH]` - all 17 examples migrated to `schema: 2` with the new command - docs: config-reference `## schema`, regenerated schema-reference Part of the 2.0 plan: #385
Downstream runners edit recipes with sed against v1 indentation and field
names, which breaks silently under any re-nesting. This gives them a
supported, path-based way to tweak a recipe from a script before the 2.0
structural changes land.
- src/srtctl/core/overrides.py: dotted paths with [N] indexes and quoted
segments; values parse as YAML scalars or lists, mappings stay literal
strings (engine flags take JSON); apply to a plain, sweep, or override
document. On override files a --set is written into base and every
override_*/zip_override_* variant (one-element list broadcast for zip
groups) so no variant can shadow it; --unset removes from all of them
and is a no-op on missing paths.
- cli: --set/--unset on apply, dry-run, preflight, and resolve-override.
Applied in materialize_config_path to the raw document (ruamel round
trip, comments preserved) before cluster defaults, observability
expansion, sweep expansion, and schema validation, so an explicit --set
wins and {placeholder} values still expand. The overridden document is
what gets copied as config.yaml into the job directory; the source file
is never modified. Every --json record lists applied_overrides.
- docs/cli.md: flags, semantics, examples.
Part of the 2.0 plan: #385
Container-alias resolution was seven hand-written blocks in resolve_config_with_defaults, one per image key, and the Mooncake master container had no block at all. Replace them with a single walk over the recipe: any string under a `container`, `container_image`, `image`, or `nginx_container` key that names a `containers:` alias is resolved; literal paths and registry URIs pass through. Free-form maps (environment, *_environment, env, args, engine config blocks, container_mounts) and the identity block are skipped, so a key that happens to be called `image` in user data is never rewritten. A new recipe block that names an image (e.g. a future services: list) resolves with no resolver code. - src/srtctl/core/config.py: resolve_container_aliases() + CONTAINER_ALIAS_KEYS - tests/test_container_aliases.py: every image key in one pass, skip list, literal/URI/unknown pass-through, new-block coverage, mutate-in-place notes - docs/config-reference.md: document the containers resolver Part of the 2.0 plan: #385
gpu_type and gpus_per_node describe the cluster, not the deployment, yet every recipe repeated them. Make both inheritable so one recipe can move between clusters unchanged: - resources.gpu_type is now optional (str | None); a recipe that omits it inherits srtslurm.yaml default_gpu_type. - resources.gpus_per_node inherits the cluster gpus_per_node when omitted (else the existing default of 4). - ClusterConfig gains default_gpu_type. Applied in resolve_config_with_defaults; an explicit recipe value always wins. The fields stay valid in recipes so a run is self-describing for result rollups. No consumer required gpu_type to be non-None. Docs: config-reference resources + cluster tables; regenerated schema-reference. Part of the 2.0 plan: #385
…pe enum An unknown benchmark.type (a typo like `gsm8k-bench`, or a removed type) loaded fine and only failed deep in the benchmark stage after a full allocation. SrtConfig.__post_init__ now checks benchmark.type against the runner registry plus the special `manual` type, so a bad type fails at load / dry-run instead. The registry import is lazy and guarded, so an import hiccup never blocks a load. The BenchmarkType enum listed 10 of the 13 registered types and had zero references anywhere; it is removed. The registry (register_benchmark / list_benchmarks) is the single source of truth for valid types. This is the safe half of the benchmark step: it validates the type. The per-type field split (rejecting stray fields like use_chat_template on a custom benchmark) is deferred to the discriminated-union follow-up. Part of the 2.0 plan: #385
A recipe can now describe a worker role in one block:
roles:
prefill: {nodes: 2, workers: 6, gpus: 2, env: {...}, args: {...}}
decode: {nodes: 0, workers: 2, gpus: 2, env: {...}, args: {...}}
instead of spreading it across resources.prefill_workers /
gpus_per_prefill, backend.prefill_environment, and
backend.<engine>_config.prefill. src/srtctl/core/roles.py normalizes a
roles: block into those existing internal fields before schema load
(expand_roles), wired into resolve_config_with_defaults and
SrtConfig.from_yaml so every load path is covered and no downstream
consumer changes. Role names are prefill/decode/agg; agg maps to the
aggregated env/config keys; args route to the engine's config by
backend.type; extra_args to <mode>_extra_args.
roles_from_legacy is the inverse (used to prove equivalence and for the
future migrator). Mixing roles: with the fields it expands into is
rejected.
The legacy layout still loads unchanged, so v1 recipes keep working, and
`srtctl migrate` still only stamps schema: 2 (the comment-preserving
legacy->roles rewrite over the downstream corpus is deferred to the
golden-CI migration release). The examples are converted to roles:,
except features/override.yaml which stays legacy to show v1 still loads.
Part of the 2.0 plan: #385
The reference and module docstring used flow-style mappings, which read like JSON. roles.<role>.env and roles.<role>.args are plain mappings and are written exactly like the legacy backend fields they normalize into.
One placement vocabulary replaces the per-block knobs:
frontend: {placement: {node: head | first_decode | dedicated}}
benchmark: {placement: {node: head | last_decode | dedicated}}
infra: {placement: {node: head | dedicated}}
node: dedicated reserves a node for that component (and implies the head
location, which the legacy validation already required); any other value
is a location string. src/srtctl/core/placement.py normalizes these
blocks into the existing fields (frontend.orchestrator_placement /
dedicated_node, benchmark.client_placement / client_dedicated_node,
infra.etcd_nats_dedicated_node) before schema load, wired into
resolve_config_with_defaults and SrtConfig.from_yaml alongside
expand_roles. No consumer changes; legacy fields still load; mixing
placement: with the fields it fills is rejected.
Part of the 2.0 plan: #385
One top-level list for every long-running process launched next to the job. Each entry has a type that selects a registered ServiceKind (@register_service, like @register_benchmark): generic launches exactly the argv written; mooncake-store runs a standalone Mooncake Store wired to the managed master. The kind supplies defaults (command, start phase, criticality) and the env it injects; ServiceStageMixin launches every kind the same way: placement.node -> physical nodes (head, infra, prefill, decode, agg, workers), optional clone/build of an immutable git source, one srun per node, optional TCP readiness gate, ManagedProcess into the shared registry. start: before_workers runs after the Mooncake master; after_frontend (default) runs once the frontend is healthy. Folds the two open sidecar PRs onto the 2.0 shape: the generic sidecar with source/build_command/inherit_discovery_env/critical from #374 (its --bash direct-runner half is gone with that path), and the standalone Mooncake Store services from #265 as a typed service instead of a backend.mooncake_kv_store.standalone sub-block with a per-role placements map. Per-role segment sizes are now two entries placed on prefill and decode; two services on one node with the same port are rejected before launch. Adds examples/features/services.yaml (an HTTP log browser gated on its port), docs/services.md, a config-reference section, dry-run output, the regenerated schema reference, and tests for the schema, kinds, and stage.
The global-bool shortcut only matched prefill and decode, so an aggregated topology never got --kv-events-config and the sidecar's cache-overlap score stayed at zero. Carried over from #374 with its regression test.
… on death Close the windows where a service process could run untracked: - start_services takes the ProcessRegistry and registers each service process the moment its srun exists, not after the whole stage returns. A SIGTERM during a readiness wait now finds every launched process in the registry; the stage also terminates what it started on any BaseException before re-raising. - The one-shot clone and build sruns are ManagedProcesses in the registry too (non-critical), and wait under a wall-clock bound: the clone by the sum of its per-command git timeouts, the build by a new build_timeout_seconds (default 1800). A step that overruns is killed and the job fails pointing at its log instead of holding the allocation. - The readiness gate polls in short slices and checks the process between them, so a service that dies before opening its port fails at once with its exit code rather than after the full timeout. Documented under a Cleanup section in docs/services.md.
One shape for "this code, from git", shared by services[].source and the
new dynamo.source:
dynamo:
source:
git: https://github.com/ai-dynamo/dynamo # default when only rev is set
rev: refs/pull/14000/head # commit, tag, or PR head
dynamo.source takes exactly one of git+rev (with optional patches, the
legacy cargo_patches), pypi (the legacy version), or wheel. It maps onto the
legacy fields in DynamoConfig.__post_init__, so the install code and every
downstream consumer keep reading hash / version / wheel unchanged, and v1
recipes still load. Combining source with a legacy field is rejected.
Testing an unmerged Dynamo PR was impossible before: hash needed a SHA,
and a plain clone does not carry PR refs. The cached install now clones
the configured repo (forks work) and fetches a non-commit ref by name.
A ref like refs/pull/N/head moves, so srtctl apply resolves every unpinned
source.rev with git ls-remote (peeled tags preferred) and records the
commit as source.sha in the submitted config.yaml, comments preserved and
the recipe on disk untouched. The job builds exactly the commit the
lockfile names and the /configs/dynamo-wheels cache is keyed by it; two
runs of one recipe cannot silently build different code because the PR
was pushed to while the job queued. dry-run and preflight never touch the
network. A resolution failure warns and leaves the ref unpinned rather
than blocking the submit; --json lists what was pinned as pinned_sources.
Adds examples/features/dynamo-source.yaml (Dynamo built from the v1.4.2
tag), docs, dry-run output, tests for the shape, resolution, pinning, the
Dynamo mapping, and the submit path, and the regenerated schema reference.
services[].readiness names exactly one probe: tcp (a port accepts a connection; `port:` alone is shorthand), http (GET a path on a port and expect a status), or log (a regular expression matched against the service's log file), plus timeout_seconds and interval_seconds. The generic wait loop in core/readiness.py re-runs the probe until it passes, the deadline expires, or the process dies, so a crashed service fails at once instead of after the full timeout. The global health_check block and the worker and frontend health checks are untouched, per the schema design: this fixes the silent-dead-sidecar class without touching the 377 downstream recipes that set health_check. The services example now gates on an http probe against the log browser's directory listing.
…spatch
The RUN_EVAL / EVAL_ONLY evaluation forwards a built-in list of workflow
variables into the eval process. Downstream runners extended that list by
patching srtctl's source text (patch_srt_eval_dispatch.py, anchored on
exact lines in do_sweep.py). post_eval.passthrough_env makes the extension
config, and post_eval.command replaces the built-in lm-eval runner argv
with placeholders for {endpoint} and {infmax_workspace}, so a runner that
rewrites bench.sh can point at its own script instead. MODEL_NAME and
EVAL_CONC are still set by srtctl. Shown in dry-run when set.
…s not use Every runner now declares the BenchmarkConfig fields it reads as config_fields, next to a shared set every type may set (client placement, sweep, aiperf plumbing, post-processing). SrtConfig rejects a schema: 2 recipe that sets a field outside shared + its type's fields, naming the stray fields and the accepted ones; a schema 1 recipe gets a warning and keeps loading. Before this, a field the runner never read (isl on gsm8k, num_shots on sa-bench) was a silent no-op. Checked against the full recipe corpus: 555 historical in-repo recipes plus 481 downstream InferenceMAX recipes, 1179 variants after override expansion. The rule rejects exactly one (a manual-type recipe carrying isl/osl/concurrencies); every other field set per type is accepted.
…grate --verify srtctl migrate now rewrites the legacy layout, not just the schema key, on a ruamel round-trip document so comments, key order, and quoting survive and moved keys keep their comments: - resources.<role>_nodes/_workers, gpus_per_<role>, backend.<mode>_environment, backend.<engine>_config.<mode>, backend.<mode>_extra_args -> roles.<role> - frontend.orchestrator_placement/dedicated_node, benchmark.client_placement/ client_dedicated_node, infra.etcd_nats_dedicated_node -> placement.node - dynamo.hash/cargo_patches/wheel/version -> dynamo.source (top_of_tree is left with a note; it has no immutable rev) - benchmark fields the type never reads are removed (schema 2 rejects them) Every variant of an override file (base, override_*, zip_override_*) is rewritten, since a half-migrated file would collide roles with the legacy keys on merge. A backend block emptied by the folds keeps an explicit type instead of disappearing, because a zip variant may null its way back to the default and that only works while the base has the key. -f accepts directories and repeats; --in-place rewrites many files at once. srtctl migrate --verify migrates in memory, expands plain, override, and sweep files, resolves both documents through the loader, and compares the dumps (masking dynamo.source, which only records spelling, and benchmark fields the type never reads). CI runs it over the examples and the 555 historical recipes extracted from the last commit that carried recipes/ (make golden-check does the same locally). Results: examples 19/19 identical. Historical corpus 553 identical, 0 mismatched, 2 skipped because v1 itself rejects benchmark.type 'gsm8k-bench'. Downstream InferenceMAX corpus (481, run locally; the repo is private): 370 identical, 0 mismatched, 105 skipped because v1 itself rejects telemetry.provider or benchmark.tokenizer_mode, 6 unreadable because the recipe has a duplicate YAML key (reported by name).
ishandhanani
requested review from
alec-flowers,
hjjq,
kedarpotdar-nv,
kyleliang-nv and
qiching
as code owners
September 8, 2026 04:00
This was referenced Sep 13, 2026
[NOT RATE MATCHED]Add NVFP4 WideEP disaggregated DEP8/DEP16/DEP32 recipes for Qwen3.5-397B-A17B
#167
Closed
This was referenced Sep 13, 2026
Merged
DeepEP assertion failure during decode worker warmup: num_max_dispatch_tokens_per_rank too small
#42
Closed
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Roadmap: #385. This is the whole 2.0 migration as one PR against
main, so it can be tested and merged as a unit. Parts 1 to 16 are the commits of GitHub stack #398 (#386 through #406), rebased onto the currentmain; those PRs stay available for per-step review of the same commits, and every part below reproduces that PR's description. Parts 17 to 21 landed on this branch afterwards: typed and implicit services (etcd, NATS, the Mooncake master, the exporters), graceful shutdown for every step, the observability cleanup, the MCP job tools and agent skill, and the fixes the cluster runs surfaced.How to review. Read the commits in order; each part is one commit or a small group and stands alone. The stack order is the dependency order: the removals first, then the schema version key and
--set, then each 2.0 authoring surface, then the migrator that ties them together.dedicated,external,enabled: false)srtctl skillmigrate --in-placeon unreadable filesValidation of the combined branch. Full suite 1866 passed; lint, format, and the schema-docs drift check clean; every example validates and migrates to itself (19/19 golden). Behavioural changes were run end-to-end on sa-b200 with Qwen3-0.6B (job ids are in the parts below): every example frontend x topology,
--set/--unset, cluster GPU defaults,roles:, theservices:example with tcp and http readiness probes plus ascancelmid-run leaving the node clean, and Dynamo built from a pinned tag on a cold cache then reused from the cache.srtctl migrate --verifyover the 555 historical recipes: 553 identical, 0 mismatched, 2 skipped because v1 rejects them. Over the 481 downstream InferenceMAX recipes, run locally: 370 identical, 0 mismatched, 105 skipped because v1 itself rejectstelemetry.provider/benchmark.tokenizer_mode, 6 unreadable because of duplicate YAML keys, each reported by name.Compatibility. Every v1 recipe still loads unchanged;
schema: 2opts into the stricter rules (per-type benchmark fields).--set/--unset,name:at top level, thehealth_checkshape, theJob <id> submitted!line, the override format, and thebenchmark.{type,isl,osl,concurrencies}paths are all preserved for downstream runners. Power telemetry (src/srtctl/core/power/, thetelemetry:block) is untouched.Fixes after consolidation
Found by the first runs of the combined branch with tachometer on by default (#358 landed on
mainunderneath the stack):Annotatedthroughstr(get_origin(...)), which only matches on newer Pythons, so the drift check failed on the 3.10 CI interpreter. Identity check now; the generated file is byte-identical on 3.10 and 3.13. The golden-equality job also fetched the historical recipes commit by short SHA, whichgit fetchcannot resolve; full SHA now.--prometheus-portis passed, and nativesglang.launch_serverworkers only serve/metricswith--enable-metrics, so underfrontend.type: sglangevery worker and frontend scrape failed for the whole run (job 12440: 287+288 worker errors, 52 frontend errors). srtctl now adds--prometheus-port 29000 --prometheus-host 0.0.0.0to the gateway and--enable-metricsto native workers unless the recipe set them, and tachometer targets worker leaders on their HTTP port and the gateway on its Prometheus port. Job 12461: 36k and 39k rows from the two workers (374 sglang metric families), 14k rows from the gateway (137smg_*families), both from the moment they came up.srun, which srun turns into a step abort that SIGKILLs the task (probed on sa-b200 with and withoutexec: the task never saw SIGTERM). Short runs therefore produced no parquet and the perf dashboard reportedscrapes=0. Named steps (start_srun_process(step_name=...)) are now signalled withscancel --signal=TERM --full <job>.<step>and giventerminate_timeout(90 s for tachometer) before the old path is used. Job 12461:Sent SIGTERM to step 12461.16 (tachometer),final.parquetwith 242k rows written and ingested by the dashboard. The bash wrapper alsoexecs the final command so the task is the process itself.Part 17: Services own the discovery plane, the Mooncake master, and the exporters
Everything that is not a worker or the frontend is a service launched by one stage. Three things that had bespoke launch paths are services now, and they are implied by the rest of the recipe:
frontend.type: dynamoimpliesetcdandnats(phaseinfra, on the infra node);backend.mooncake_kv_storeimpliesmooncake-master(phasebefore_workers); tachometer impliesdcgm-exporterandnode-exporteron every worker node (phaseafter_frontend).srtctl dry-runlists them next to the declared services, markedimplied by:, with the kind's built command, container, default readiness ports, and options. A declared entry with the same name takes over:external: <address>uses an instance that already runs (nothing launches; workers and the frontend get its address).infra:andbackend.mooncake_kv_storeare v1 spellings that still load;infra.placementleaves the placement vocabulary.srtctl migratefoldsinfra:into etcd/nats entries (spelled out asplacement.node: infrain override variants that undo a basetrue; left alone under a static frontend, where the dedicated flag still reserves a node and the payload knob was inert) andbackend.mooncake_kv_storeinto amooncake-masterentry plus Mooncake env on every role. Golden verification compares services by effect. New kinds:etcd,nats,mooncake-master,dcgm-exporter,node-exporter(src/srtctl/services/); kinds may build their own command, declare default readiness ports and placement, skip the bash wrapper (distroless images get their env throughsrun --export), and accept validatedoptions.examples/features/infra-services.yamlis the runnable version. Docs:docs/services.md(Implicit Services),docs/config-reference.md,docs/mooncake-kv-store.md.Cluster. sa-b200 job 12807 (
examples/features/infra-services.yaml, Dynamo + SGLang, 2 nodes): etcd and NATS on the dedicated nodec012with the NATS config file, workers and exporters onc013, etcd and NATS shut down cleanly at the end (closed etcd server,JetStream Shutdown). Job 12808 (examples/features/services.yaml, SGLang router): both exporters and the declared sidecar as services, no discovery plane launched.Part 18: Graceful shutdown for workers and frontends; tiered registry cleanup
SIGTERM aimed at an
srunclient aborts the step and SIGKILLs the task, so nothing ever got to shut down cleanly. Every long-running step is now named (step_nameonstart_srun_processandManagedProcess), and cleanup delivers SIGTERM withscancel --signal=TERM --full <job>.<step>: workers (per-process and MPI endpoint, 30 s), frontends (Dynamo, SGLang router, vLLM router, trtllm-serve orchestrator, nginx, 20 s), services (30 s), tachometer (90 s).ProcessRegistry.cleanup()is two-phase per tier: SIGTERM to every process of a tier at once (reverse registration order), wait up to each process's own timeout, escalate to SIGKILL, then the next tier. Tier 0 is workers, frontends, and sidecars; tier 1 the Mooncake master and stores; tier 2 etcd and NATS, so nothing deregisters from a plane that is already gone and a job with dozens of workers finishes cleanup in about one timeout. Steps are listed with onesqueuecall per cleanup; without Slurm tools on PATH (tests, the mock) the step path declines quietly.Native
sglang.launch_serverworkers drain in about five seconds after SIGTERM and then treat their own exit as a crash: py-spy dumps (which need root) and a 60 s wait for CUDA coredumps that are never produced unlessSGLANG_CUDA_COREDUMP=1. Workers underengine: sglangnow defaultSGLANG_CUDA_COREDUMP_BEFORE_CRASH=0andSGLANG_PYSPY_DUMP_BEFORE_CRASH=0; a recipe that opts into coredumps keeps the wait.Cluster. Job 12807: all six tier-0 steps signalled at 18:49:59, etcd and NATS at 18:50:11 after the tier had exited; Dynamo SGLang workers exited within the tier. Job 12808:
SIGTERM received ... Gracefully exiting ... Remaining requests 0in both native SGLang worker logs, then the coredump wait that the env defaults now skip.Part 19: Observability: tachometer is the only scrape path; legacy Python analysis deleted
Roadmap track "all scraping goes through tachometer". Removed:
src/ingest/metrics_prometheus.py(reader for theraw_prometheus.jsonlthe deleted #351 scraper wrote; the ingest's metrics source is now tachometer parquet first, then AIPerf's own exports); the in-flight batch-metrics snapshotter with its log parser and matplotlib renderer and thereporting.live_metricsknob; the Streamlit dashboard and the srtlog log-parsing stack underanalysis/(5300 lines),docs/analyzing.md,benchmark.export_node_metricsand the post-process CSV export (no historical or downstream recipe set it); srtlog parsing inside the S3 upload container, which now only installs awscli and syncs the log directory. Kept:srtctl.analysis.host_sampler(reads/procfor what no/metricsendpoint publishes) and the per-run HTML perf dashboard (src/ingest+src/visualization), fed by the tachometer parquet. Power telemetry untouched. 8381 lines deleted.The
tachometer-scraperbinaries attached to the GitHub releases (whatmake setupinstalls) were built before the writer gained thetimestamp_nscolumn (#350), so the dashboard ingest crashed on every fresh install (job 12807:Field "timestamp_ns" does not exist in schema). The ingest now derives timestamps fromtime_since_startanchored at the scraper start intachometer.out, or at the parquet mtime. The release workflow reuses the previous release's scraper binaries unless the merged PR touchedsrc/tachometer/; aworkflow_dispatchrebuild (ormake tachometer-scraper) is what refreshes them.Part 20: MCP job lifecycle tools and an in-package agent skill
srtctl-mcpgainssubmit_job(srtctl apply -y --jsonwith--set/--unset/--tags),dry_run,job_status(sacct row, job metadata, the orchestrator's current stage,[ERROR]lines, benchmark rollup, sweep-log tail),job_logs(list or tail, confined to the job's log directory),list_jobs, andcancel_job. They only do anything where Slurm is; the schema tools keep working anywhere and still never read host-sidesrtslurm.yaml. The server now imports under the mcp SDK the lock file pins (2.x renamedFastMCP).srtctl skill --target claude|codex|cursor [--root DIR] [--print]installssrc/srtctl/skills/SKILL.md, one document teaching an agent the 2.0 recipe shape, dry-run before apply,--set,migrate --verify, where a run's logs and artifacts live, how cleanup behaves, and the MCP tools.Part 21: Fixes from the cluster runs
srtctl migrate --in-place -f <dir>reports a recipe with duplicate YAML keys by name and continues with the rest of the directory (it used to abort with a traceback after rewriting the files before it) and exits 1 with a migrated / not-migrated count.make golden-checkhad a mangledmrmin its recipe.Merged with main through 2026-09-12
The branch carries
mainup to 4d3e8bd (#402, #355, #359, #365, #402, #408, #409, #410, #413, #414, #415, #421, #422). Two of those met this PR's design head-on and were resolved so both intents hold:mainthese are launched by the tachometer stage; here every exporter is a service.process-exporteris a service kind, implied while tachometer runs,placement.node: all, host-native from theconfigs/process-exporterbinarymake setupinstalls (no container, no mounts), skipped with a warning when the binary is missing, container launch when a recipe declares acontainer.ServiceKindgainedhost_native,prepare(writes the group file), andskip_reason. The node exporter's built command carries the stat/vmstat/pressure/meminfo_numa/processes collectors.run_benchmarkare kept.stop_tachometernow terminates through the Slurm step (ManagedProcess.terminatewithstep_name); SIGTERM to the srun client, whichmaindoes, aborts the step and SIGKILLs the scraper before it compacts. The grace is the recipe'sshutdown_grace_secs.Caching (#344) tested, not taken
Roadmap track 5 said to test #344 (digest-pinned container cache) on a real cluster before building on it. On sa-b200 (enroot 3.5.0)
srtctlfrom that branch reportsContainer cache miss: importing sha256:..., the Pyxis import fails, and the run falls back to native handling as designed; but enroot itself rejects every@sha256:reference ([ERROR] Invalid image reference, with and withoutdocker://, with and without--container-save; tag references with--container-savework), so the native fallback fails the same way and the job dies at the firstsrun(job 12809). The cache cannot fire on that cluster at all, and #344 also swallows the Pyxis stderr on the failed import. Not built on; nothing from it is in this PR.Downstream
NVIDIA/InferenceMAX: a draft PR migrates the 481 recipes under
benchmarks/multi_node/srt-slurm-recipes/withsrtctl migrate --in-placefrom this branch (376 verified identical, 0 mismatched, 99 unverifiable because current srt-slurm rejects them as v1 too, 6 not migrated because of duplicate YAML keys). It is held until this PR merges and the runners' srt-slurm pin moves.Validation of parts 17 to 21
Full suite 1902 passed on Python 3.10 and 3.13; lint, format, and the schema-docs drift check clean; every example validates;
srtctl migrate --verify: examples 20/20, historical 553 identical / 0 mismatched / 2 skipped, downstream 376 / 0 / 99 skipped / 6 unreadable. Cluster runs above: sa-b200 jobs 12807 and 12808 (both COMPLETED), 12809 (#344 test).Part 1: [2.0] Remove the --bash direct-host execution path (#386, branch
idhanani/srt2-01-remove-bash-path)Summary
First PR of the 2.0 stack (plan: #385, Track 1 item 4). Removes the
--bashdirect-host lifecycle so 2.0 is the Slurm path only. Pure deletion plus the CLI flag and doc references; no behavior change forsrtctl apply,dry-run, or any Slurm stage.Deleted:
src/srtctl/render/(direct plan, host runner, container runner, direct stage mixins),templates/direct_container.sh.j2,docs/direct-host.md, the threetest_direct_*suites.Edited:
cli/submit.py(drop--bash,render_bash_script, and the bash-mode argument checks),tests/test_submit_cli.py(the direct-container test becomes a check that--bashis rejected), five docs, and three stale comments.Untouched: everything under
core/power/, the power hooks,measurement_window.py.Validation
ruff checkandruff format --checkcleanpytest tests: 1656 passed, 2 skipped, 6 deselectedStack
--bashexamples/matrixPart 2: [2.0] Replace the recipes archive with a curated examples matrix (#387, branch
idhanani/srt2-02-examples-matrix)Summary
Second PR of the 2.0 stack (plan: #385, Track 1 item 1). Stacked on #386; the diff against that branch is what to review.
Removes
recipes/(563 files) and replaces it withexamples/, one small runnable configuration per frontend and topology. Every example serves Qwen3-0.6B on one node, so the files differ only in the frontend and the prefill/decode layout and a full matrix run finishes in minutes.examples/README.mddocuments the matrix, the twosrtslurm.yamlalias kinds the files rely on (qwen3-0.6b;sglang,vllm,trtllm), and how to validate.Supersedes #338 and carries its wiring (CI validate step, Makefile target, interactive selector, CODEOWNERS, docs, tests).
Code change
validate_config_filenow expands top-levelsweep:files the same way it expandsbase:/override files, so CI validates every file underexamples/. Previously sweep files were rejected as having an unknownsweepfield.Validation
validate_config_file(sweep and override files expanded to their variants)ruffclean onsrc/srtctl/Stack
--bashPart 3: [2.0] Generate docs/schema-reference.md from the config dataclasses (#388, branch
idhanani/srt2-03-schema-docs)Summary
Third PR of the 2.0 stack (plan: #385, Track 1 item 2). Stacked on #387; the diff against that branch is what to review.
docs/config-reference.mdis hand-written prose and drifts from the schema (the power docs merged last week already reference atelemetry.providerfield that #317 removed). This adds a generated, CI-checked field-level reference so keys, types, and defaults live in the code.src/srtctl/core/schema_docs.pywalks the dataclass tree rooted atSrtConfig, the four backend types, andClusterConfig, and renders one Markdown table per dataclass: YAML key (honoring marshmallowdata_key, sogpus_per_prefillrather than_explicit_gpus_per_prefill), type (nested dataclasses linked,Literals expanded), default, and a description taken from the class docstringAttributes:block or the#comment on the field.srtctl schema-docswritesdocs/schema-reference.md(562 lines, 38 sections).srtctl schema-docs --checkexits 1 when the checked-in file is stale.make schema-docs-check(part ofmake check), andtests/test_schema_docs.py::test_checked_in_schema_reference_is_current.docs/config-reference.mdnow points at the generated file as authoritative for keys, types, and defaults;SUMMARY.mdlinks it;CLAUDE.mdtells contributors to regenerate after any schema change and where to put field descriptions so they land in the table.Validation
ruff,ruff format --check, andtyclean on the new moduletests/test_schema_docs.py(9 tests: drift check, determinism, data_key handling, docstring and comment descriptions, backend and cluster sections, CLI write/check/stale)srtctl schema-docs --checkpasses on the committed fileStack
--bashPart 4: [2.0] Add the recipe schema version key and srtctl migrate (#389, branch
idhanani/srt2-04-schema-version)Summary
Fourth PR of the 2.0 stack (plan: #385, Track 2 step 1). Stacked on #388; the diff against that branch is what to review.
Recipes gain a top-level
schema:key. Absent means1(the pre-2.0 layout);schema: 2is the 2.0 layout. Both versions load onmain; unknown versions are rejected at load time. This is the dispatch point the structural 2.0 changes (roles:,placement:,services:) hang off, and it lets downstream recipes declare which layout they target instead of failing on unknown fields.SrtConfig.schema_version(YAML keyschemavia marshmallowdata_key), plusCURRENT_SCHEMA_VERSIONandSUPPORTED_SCHEMA_VERSIONS.schema:besidebase:;generate_override_configsandresolve_override_yamlcarry it into every expanded variant. Sweep files carry it through expansion unchanged.srtctl migrate -f recipe.yaml [--in-place | --output PATH]: a ruamel round-trip migration that inserts or bumpsschema: 2and preserves comments, key order, and quoting. Structural v1 to v2 rewrites plug into_migrate_1_to_2in later steps.examples/is the v2 reference corpus.config-reference.mdgains a## schemasection;schema-reference.mdregenerated (the new field shows up automatically).Validation
ruffclean; targeted suites 391 passed; full suite greentests/test_schema_version.py(14 tests): absent/2/unknown versions, propagation through override variants, validation of override and sweep files with the key, migration idempotence and comment preservation, override and lock files keep top-level sections, CLI stdout / in-place / output modes, every example declares the current versionStack
--bashPart 5: [2.0] Add --set KEY=VALUE and --unset KEY recipe overrides (#390, branch
idhanani/srt2-05-set-unset)Summary
Fifth PR of the 2.0 stack (plan: #385, Track 2 step 2). Stacked on #389; the diff against that branch is what to review.
Downstream runners edit recipes with
sedanchored on v1 indentation and field names, which breaks silently under any re-nesting. This adds a supported, path-based override so scripts can tweak a recipe without editing the YAML, before the structural 2.0 changes land.srtctl apply -f recipe.yaml --set health_check.max_attempts=720 --unset sbatch_directives.exclude srtctl apply -f recipe.yaml --set 'backend.sglang_config.decode.speculative-config={"method": "eagle"}' srtctl dry-run -f recipe.yaml --set benchmark.concurrencies=[4,8][N]indexes a list, quotes protect segments with dots. Values parse as YAML scalars or lists; mappings stay literal strings because engine flags take JSON.materialize_config_pathto the raw document (ruamel round trip, comments kept) before cluster defaults, observability expansion, sweep expansion, and validation, so an explicit--setalways wins and{placeholder}values still expand.baseand everyoverride_*/zip_override_*variant (one-element-list broadcast for zip groups), so no variant can shadow it.--unsetremoves from all of them and is a no-op on missing paths.config.yamlin the job directory; the source file is never modified. Every--jsonrecord listsapplied_overrides.apply,dry-run,preflight, andresolve-override.Validation
ruffclean; full suite greentests/test_overrides.py(24 tests): path grammar and rejections, value typing incl. JSON-as-string and placeholders, nested creation, list indexes, lenient unset, override-file semantics, comment preservation, dry-run,--setbeatingdefault_health_check,--jsonrecord and writtenconfig.yamlunder--mock,resolve-override --stdoutStack
--bash--set/--unsetPart 6: [2.0] Resolve every container alias with one recipe walker (#391, branch
idhanani/srt2-06-alias-resolver)Summary
Sixth PR of the 2.0 stack (plan: #385, Track 2 step 3). Stacked on #390; the diff against that branch is what to review.
Container-alias resolution was seven copy-pasted blocks in
resolve_config_with_defaults, one per image key (model.container,frontend.container_image,frontend.nginx_container,benchmark.container_image, the Tachometer and power exporter images), andbackend.mooncake_kv_store.containerhad no block at all. This replaces them with one walk over the recipe.container,container_image,image, ornginx_containerkey that names acontainers:alias is resolved; literal paths and registry URIs pass through.environment,*_environment,env,args, engine config blocks,container_mounts) and theidentityblock are skipped, so a key that happens to be namedimagein user data is never rewritten.services:list) resolves with no resolver code. This is the seam Track 3 and step 7 need.Validation
ruffclean; full suite: 1759 passedtests/test_container_aliases.py: every image key in one pass (incl. the newly-covered Mooncake container), the skip list, literal / registry-URI / unknown-alias pass-through, a syntheticservices:block, in-place mutation with per-resolution notes, and no-containers-map behaviour. The existing nginx / benchmark / exporter alias tests intest_configs.pystill pass unchanged.Stack
--bash--set/--unsetPart 7: [2.0] Default resources.gpu_type and gpus_per_node from srtslurm.yaml (#392, branch
idhanani/srt2-07-cluster-gpu-defaults)Summary
Seventh PR of the 2.0 stack (plan: #385, Track 2 step 9). Stacked on #391; the diff against that branch is what to review.
gpu_typeandgpus_per_nodedescribe the cluster, not the deployment, yet 100% of recipes repeat them. This makes both inheritable so one recipe can move between clusters unchanged.resources.gpu_typeis now optional (str | None); a recipe that omits it inheritssrtslurm.yamldefault_gpu_type.resources.gpus_per_nodeinherits the clustergpus_per_nodewhen omitted (else the existing default of4).ClusterConfiggainsdefault_gpu_type.Applied in
resolve_config_with_defaults; an explicit recipe value always wins. The fields stay valid in recipes so a run is self-describing for result rollups. No consumer requiredgpu_typeto be non-None (it is metadata plus the TRT-LLM numactlin (...)check, which handlesNone). Additive: existing recipes are unaffected.Validation
ruffclean; full suite: 1763 passedtests/test_configs.py::TestClusterGpuDefaults: inherit when omitted, recipe value wins, loads with neither recipe nor cluster valueStack
...7 of the stack; see #386-#391 for the earlier PRs.
Part 8: [2.0] Reject unknown benchmark.type at load; delete dead BenchmarkType enum (#393, branch
idhanani/srt2-08-benchmark-validation)Summary
Eighth PR of the 2.0 stack (plan: #385, Track 2 step 4, safe half). Stacked on #392.
An unknown
benchmark.type(a typo likegsm8k-bench, or a removed type) loaded fine and only failed deep in the benchmark stage after a full allocation.SrtConfig.__post_init__now validatesbenchmark.typeagainst the runner registry plus the specialmanualtype, so a bad type fails at load / dry-run. The import is lazy and guarded, so a registry import hiccup never blocks a load.The
BenchmarkTypeenum listed 10 of the 13 registered types and had zero references; it is removed. The registry (register_benchmark/list_benchmarks) is the single source of truth.Scope note: this is the safe half of the benchmark step. It validates the type. The per-type field split (rejecting stray fields such as
use_chat_templateon acustombenchmark, via a discriminated union) is deferred to a follow-up, because a wrong per-type allowlist would hard-error a valid recipe and needs the golden corpus to land safely.Validation
ruffclean; full suite: 1766 passedtests/test_configs.py::TestBenchmarkTypeValidation: registered + manual types load,gsm8k-benchrejected at load, enum is goneStack
8 of the stack; see #386-#392 for the earlier PRs.
Part 9: [2.0] Add the roles: authoring surface for recipe topology (#394, branch
idhanani/srt2-09-roles)Summary
Ninth PR of the 2.0 stack (plan: #385, Track 2 step 5, authoring surface). Stacked on #393.
A recipe can now describe a worker role in one block instead of spreading it across
resources,backend.*_environment, andbackend.<engine>_config.*:src/srtctl/core/roles.py::expand_rolesnormalizesroles:into the existing internal fields (resources.prefill_workers/gpus_per_prefill,backend.prefill_environment,backend.<engine>_config.prefill) before schema load, wired intoresolve_config_with_defaultsandSrtConfig.from_yamlso every load path is covered and no downstream consumer changes.prefill/decode/agg;aggmaps to theaggregatedenv/config keys;argsroute to the engine's config bybackend.type;extra_argsto<mode>_extra_args(TRT-LLM).roles_from_legacyis the inverse, proving the two forms load identically and providing the transform the future migrator will use.roles:with the fields it expands into is rejected.v1 stays valid. The legacy layout still loads unchanged, so v1 recipes keep working.
srtctl migratestill only stampsschema: 2; the comment-preserving legacy-to-roles:rewrite over the 481-recipe downstream corpus is deferred to the golden-CI migration release, because that structural surgery is where the risk concentrates. Both forms are valid v2.The 16 topology examples are converted to
roles:;features/override.yamlstays legacy to demonstrate v1 still loads.Validation
ruffclean; full suite: 1776 passedtests/test_roles.py(10): expand produces the legacy layout, roles and legacy load to identicalSrtConfig, round-trip throughroles_from_legacy,agg→aggregated, engine-config key by backend type, TRT-LLMextra_args, mixing rejected, unknown role/spec key rejected, no-op withoutroles:,decode_nodes: 0survivesvalidate_config_file; the e2e disagg tests pass against the converted examplesStack
9 of the stack; see #386-#393 for the earlier PRs.
envandargsare ordinary block-style YAML mappings, written exactly like the legacybackend.prefill_environment/backend.sglang_config.prefillblocks they normalize into. No inline{}or JSON needed; the earlier flow-style snippet here was only shorthand.Part 10: [2.0] Add the placement: vocabulary for frontend, benchmark, and infra (#395, branch
idhanani/srt2-10-placement)Summary
Tenth PR of the 2.0 stack (plan: #385, Track 2 step 6). Stacked on #394.
One
placement:vocabulary replaces the per-block placement knobs:node: dedicatedreserves a node for that component and implies the head location (which the legacy validation already required for a dedicated node); any other value is a location string.src/srtctl/core/placement.py::expand_placementnormalizes these into the existing internal fields (frontend.orchestrator_placement/dedicated_node,benchmark.client_placement/client_dedicated_node,infra.etcd_nats_dedicated_node) before schema load, wired in next toexpand_roles. No consumer changes; the legacy fields still load; mixingplacement:with the fields it fills is rejected.Validation
ruffclean; full suite: 1784 passedtests/test_placement.py: frontend/benchmark location +dedicated, infra head/dedicated, placement and legacy load to identicalSrtConfig, mixing rejected, invalid values rejected, no-op withoutplacement:Cluster: dry-run on sa-b200 with
--set frontend.placement.node=head --set infra.placement.node=headresolved cleanly (placement expands through the CLI path). Dedicated-node placement needs multi-node to exercise and was not run.\n\n## Stack10 of the stack; see #386-#394 for the earlier PRs.
Part 11: Add the services: block for sidecars and standalone Mooncake stores (#399, branch
idhanani/srt2-11-services)What
One top-level
services:list for every long-running process launched next to the job: an experimental router built from a PR, a standalone Mooncake Store per worker node, a debugging HTTP server. Each entry has atypethat selects a registeredServiceKind(@register_service, same pattern as@register_benchmark):generic(default): launches exactly the argv written. Starts after the frontend, non-critical.mooncake-store: standalone Mooncake Store wired to the managed master. Default command, starts before workers, critical, container falls back tomooncake_kv_store.container,MOONCAKE_MASTER/MOONCAKE_TE_META_DATA_SERVER/MOONCAKE_LOCAL_HOSTNAMEinjected.The kind supplies defaults and the env it injects.
ServiceStageMixinlaunches every kind the same way:placement.noderesolves to physical nodes (head,infra,prefill,decode,agg,workers), optional clone and build of an immutable gitsource, onesrunper node, an optional TCPreadinessgate, and aManagedProcessin the shared registry for crash detection and teardown.Folds two open PRs onto the 2.0 shape
auxiliary_services): the generic sidecar withsource/build_command/inherit_discovery_env/critical, the head-node clone-on-bare-host then build-in-container flow, and the dry-run panel. Its--bashdirect-runner half is gone with that path ([2.0] Remove the --bash direct-host execution path #386). Renamed toservices:withcontainer(alias-resolved by the shared walker) instead ofcontainer_image. Its unrelatedkv_events_config: trueaggregated-mode fix is carried as its own commit with the regression test.mooncake_kv_store.standalone): the standalone Store services, as a typed service instead of a backend sub-block with aplacements: {prefill, decode, aggregated}map. Per-role segment sizes are two entries placed onprefillanddecode;preamble,cpus_per_task,cpu_bind,srun_options, the{node}/{node_ip}/{role}placeholders, and the TCP readiness check all carry over. Two services that listen on the same port on one node are rejected before launch (the co-location conflict feat(mooncake): manage standalone store services #265 checked). The worker-sidebackend.mooncake_kv_store(master, worker env, vLLMstore_config) is unchanged.Both external PRs can close in favour of this one once it lands; credit to their authors in the file history.
Also
examples/features/services.yaml: a runnable HTTP log browser on the head node gated on its port.docs/services.md, a## servicessection indocs/config-reference.md, a pointer section indocs/mooncake-kv-store.md, CLAUDE.md notes, regenerateddocs/schema-reference.md.tests/test_services.py(schema, kinds, stage: placement resolution, env layering, source build, readiness failure teardown, port collisions), dry-run cases.Cleanup guarantees
Nothing a service launches outlives the job. Every srun the stage starts, including the one-shot clone and build steps, is registered with the
ProcessRegistrythe moment it exists (not when the stage returns), so cleanup on normal exit, a failed stage, the SIGTERM handler, and the crash monitor all reach it. The stage also terminates what it started on any exception, includingSystemExitfrom the signal handler. The clone step is bounded by its per-command git timeouts and the build bybuild_timeout_seconds(default 1800); a step that overruns is killed. A service that dies before its readiness port answers fails at once with its exit code instead of waiting out the timeout. Slurm step cancellation kills the whole step cgroup, so forked children inside the container go with the service.Validation
examples/features/services.yamlran as job 11995: thelog-browserservice launched on the head node, itsreadinessgate on port 9911 passed after 21s, the benchmark completed with rollups written, and the service was torn down with the job on the cleanup path; themooncake-storeshape dry-run validated on a disagg recipe (no Mooncake image on that cluster to run it).sacctshows the service step cancelled with the workers and router at teardown; job 12005 wasscancelled while the service was up, every step ended (service stepCANCELLED), and a check on the compute node afterwards found nohttp.server, no engine processes, and no steps still allocated.Part 12: Add dynamo.source and pin source revs to commits at submit (#400, branch
idhanani/srt2-12-dynamo-source)What
One shape for "this code, from git", shared by
services[].source(#399) and the newdynamo.source:dynamo.sourcetakes exactly one ofgit+rev(with optionalpatches, the oldcargo_patches),pypi(the oldversion), orwheel. It maps onto the legacyDynamoConfigfields at load, so the install code and every downstream consumer keep readinghash/version/wheelunchanged and v1 recipes still load. Combiningsourcewith a legacy field is rejected.Testing an unmerged Dynamo PR was not possible before:
hashneeded a commit SHA, and a plain clone does not carry PR refs. The cached source install now clones the configured repository and fetches a non-commit ref by name before checkout.Pinning at submit
A ref like
refs/pull/14000/headmoves.srtctl applyresolves every unpinnedsource.rev(Dynamo and services alike) withgit ls-remote, preferring the peeled commit for tags, and records it assource.shain the submittedconfig.yaml. Comments are preserved and the recipe on disk is untouched. The job builds exactly the commit the lockfile names, and the/configs/dynamo-wheelscache is keyed by that commit, so two runs of one recipe cannot silently build different code because the PR was pushed to while the job sat in the queue.dry-run,preflight, andresolve-overridenever touch the network.--jsonoutput lists what was pinned underpinned_sources;dry-runprints the Dynamo source and whether it is pinned.Also
examples/features/dynamo-source.yaml: Dynamo built from thev1.4.2tag (the same release the PyPI examples install).--set dynamo.source.rev=refs/pull/<n>/headturns it into a PR test.## dynamoindocs/config-reference.md, the pinning note indocs/services.md, regenerateddocs/schema-reference.md.tests/test_source.py(shape validation,git ls-remoteresolution incl. peeled tags and failures, pinning across plain and override-format documents with comments kept, the Dynamo mapping and install script for unpinned refs, pinned SHAs, forks, and patches, and the submit-pathmaterialize_config_pathbehaviour) plus dry-run cases.Validation
git ls-remoteresolution ofv1.4.2andrefs/pull/14000/headagainst ai-dynamo/dynamo from this branch.srtctl applypinnedv1.4.2to2ecbdfdfand wrote theshainto the submittedconfig.yaml(--jsonlisted it underpinned_sources); the workers did a cold source build at that commit inside the SGLang container (rustup, cargo, maturin), populated/configs/dynamo-wheels/2ecbdfdf..., and the aggregated Dynamo benchmark completed. Whole job, build included: 7m45s.Dynamo installed from source (2ecbdfdf...)) and the benchmark completed. Whole job: 3m06s.Part 13: Add the readiness probe vocabulary for services: tcp, http, log (#403, branch
idhanani/srt2-13-readiness)What
services[].readinessgains the probe vocabulary from the schema design: exactly one ofplus
timeout_seconds(default 120) andinterval_seconds(default 2). The generic wait loop incore/readiness.pyre-runs the probe until it passes, the deadline expires, or the process dies, so a crashed service fails the job at once with its exit code instead of after the full timeout. The port-collision check now keys on the port atcporhttpprobe implies.Per the design, the global
health_checkblock and the worker and frontend health checks are untouched: this fixes the silent-dead-sidecar class (NVIDIA/InferenceMAX#271) without touching the downstream recipes that sethealth_check.Also
examples/features/services.yamlnow gates on anhttpprobe against the log browser's directory listing.tests/test_readiness.pycovers the schema (shorthand, exactly-one, invalid regex/path/interval), each probe, and the wait loop with a fake clock (retries, fail-fast on death, clipped final sleep).tests/test_services.pyupdated for the probe API.docs/services.md, config-reference row, regenerated schema reference.Validation
http://<node>:9911/ -> 200came ready after 40s, the benchmark completed, and the job finished clean (3m25s).Part 14: Add post_eval: passthrough_env and a command override for the eval dispatch (#404, branch
idhanani/srt2-14-post-eval)What
The
RUN_EVAL/EVAL_ONLYevaluation forwards a built-in list of workflow variables into the eval process. Downstream runners extended that list by patching srtctl's source text (patch_srt_eval_dispatch.py, anchored on exact lines indo_sweep.py), the only runner patch that edits srtctl source. This makes it config:passthrough_envextends the built-in list (RUN_EVAL,EVAL_ONLY,MODEL,ISL,OSL,PREFILL_TP, ...). Names are validated as identifiers.commandreplaces the lm-eval runner argv, with{endpoint}and{infmax_workspace}placeholders, so a runner that rewritesbench.shpoints at its own script instead of patching.MODEL_NAMEandEVAL_CONCare still set by srtctl.srtctl dry-runprints the effective dispatch when the block is set.Validation
tests/test_post_eval.pydrives_run_post_evalwith a mocked srun and asserts the forwarded env and the substituted command.Part 15: Split benchmark fields per type: schema 2 rejects fields the type does not use (#405, branch
idhanani/srt2-15-benchmark-fields)What
BenchmarkConfigis one flat dataclass with every type's fields on it, so a field the runner never reads (islongsm8k,num_shotsonsa-bench) loads fine and silently does nothing. Each runner now declares the fields it reads asconfig_fields, next to a shared set every type may set (client_placement,client_dedicated_node,colocate_with_frontend,sweep,aiperf_package,aiperf_args,export_node_metrics).schema: 2recipe that sets a field outside shared + its type's fields is rejected at load, naming the stray fields and the accepted ones.config_fields; a test asserts every declared field exists onBenchmarkConfig.Checked against the whole corpus
The rule was run over the 555 historical in-repo recipes and the 481 downstream InferenceMAX recipes, 1179 variants after override expansion, each forced to
schema: 2. It rejects exactly one: amanual-type recipe carryingisl/osl/concurrencies. Every other field set per type is accepted. (The same pass surfaced two pre-existing, unrelated load failures in the downstream corpus: 98 recipes settelemetry.provider/default_frequency, which the currentTelemetryConfigdoes not have, and 6 setbenchmark.tokenizer_mode. Those fail onmaintoday and are not touched here.)Validation
tests/test_benchmark_fields.py: declared fields exist, each type accepts the field sets its recipes use, shared fields work for every type, schema 2 rejects with the expected message, schema 1 warns, defaults never count as set.Part 16: Migrate v1 recipes to the v2 layout and prove equality with srtctl migrate --verify (#406, branch
idhanani/srt2-16-migrate)What
srtctl migratenow rewrites the legacy layout, not just theschema:key, on a ruamel round-trip document so comments, key order, and quoting survive and moved keys keep their comments:resources.<role>_nodes/_workers,gpus_per_<role>,backend.<mode>_environment,backend.<engine>_config.<mode>,backend.<mode>_extra_argsroles.<role>.{nodes, workers, gpus, env, args, extra_args}frontend.orchestrator_placement/dedicated_node,benchmark.client_placement/client_dedicated_node,infra.etcd_nats_dedicated_nodeplacement.nodedynamo.hash/cargo_patches/wheel/versiondynamo.source.{rev, patches, wheel, pypi}(top_of_treeis left with a note; it has no immutable rev)Every variant of an override file (
base,override_*,zip_override_*) is rewritten, since a half-migrated file would collideroleswith the legacy keys on merge. Abackend:emptied by the folds keeps an explicittypeinstead of disappearing, because a zip variant maynullits way back to the default and that only works while the base has the key.-faccepts directories and repeats;--in-placerewrites many files at once.Golden equality
srtctl migrate --verify -f <paths>migrates in memory, expands plain, override, and sweep files, resolves both documents through the loader, and compares the dumps (maskingdynamo.source, which only records spelling, and benchmark fields the type never reads). Exit 1 on any mismatch. A new CI jobgolden-equalityruns it overexamples/and the 555 historical recipes extracted from the last commit that carriedrecipes/(make golden-checkdoes the same locally).examples/(19)benchmark.type: gsm8k-bench, unknown on main)telemetry.provider/benchmark.tokenizer_mode, unknown on main)The skipped and unreadable rows are pre-existing recipe or schema gaps that fail on
maintoday, independent of this stack; the report names each one so they can be fixed downstream.Also
## schemaand## rolesindocs/config-reference.mddescribe the rewrite and--verify.tests/test_migrate.pycovers each fold with comments preserved, idempotency, override and zip variants,dynamo.sourcecases, verify outcomes, every example being golden, and the directory forms of--verifyand--in-place.