From 9cd522a07d2a1283bd12c05fef647268277fa3d6 Mon Sep 17 00:00:00 2001 From: Alec Flowers Date: Tue, 27 Oct 2026 21:32:59 -0700 Subject: [PATCH 1/6] docs(agents): focus the root AGENTS.md on Dynamo development Reorganize the root agent guide around what a developer needs: the repository map, the engine repositories Dynamo integrates with, the skills index, and how to build and test efficiently. - Add vLLM, SGLang, TensorRT-LLM, and TokenSpeed to the ecosystem table, with the backend and sidecar paths for each. - Regroup the skills index by task and refresh each summary from the skill's frontmatter. - Add published nightly images, wheels, and Helm chart, a local-dev container loop, and Rust and pytest practices that keep builds and test runs short. - Replace the PR conventions and docs rules with pointers to the pr-checklist skill, CONTRIBUTING.md, and the docs skills. - Move the optimization-loop harness and issue-reporting guidance into agent-docs/guides/optimization/optimize-loop.md, and update the synthesize-user-workload reference to it. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Alec Flowers --- .../skills/synthesize-user-workload/SKILL.md | 2 +- AGENTS.md | 401 ++++++++---------- .../guides/optimization/optimize-loop.md | 52 +++ 3 files changed, 238 insertions(+), 217 deletions(-) diff --git a/.agents/skills/synthesize-user-workload/SKILL.md b/.agents/skills/synthesize-user-workload/SKILL.md index 78d5a5c5e46b..9cc82298d4db 100644 --- a/.agents/skills/synthesize-user-workload/SKILL.md +++ b/.agents/skills/synthesize-user-workload/SKILL.md @@ -201,5 +201,5 @@ supporting context for `perf-analyzer`, `hypothesis-generator`, and `hypothesis- End with an operator handoff line before any deployment starts: state that the interview is complete, give the contract path, and tell the operator that the engagement now runs a long unattended loop — without goal mode the -session pauses for input at every turn end — and that this is the moment to arm it (AGENTS.md, Long-Running Runs, +session pauses for input at every turn end — and that this is the moment to arm it (`agent-docs/guides/optimization/optimize-loop.md`, Long-Running Runs, has the condition template; fill in the budget). diff --git a/AGENTS.md b/AGENTS.md index d6dbdf5e315e..7963166526c1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,8 +8,8 @@ SPDX-License-Identifier: Apache-2.0 ## Overview Dynamo is NVIDIA's open-source, datacenter-scale distributed inference framework. It is -the orchestration layer **above** inference engines (SGLang, TensorRT-LLM, vLLM), not a -replacement for them: it turns a cluster of GPUs into one coordinated inference system. +the orchestration layer **above** inference engines (SGLang, TensorRT-LLM, vLLM, TokenSpeed), +not a replacement for them: it turns a cluster of GPUs into one coordinated inference system. Core capabilities are disaggregated prefill/decode serving, KV-aware routing, multi-tier KV cache management (KVBM: GPU → CPU → SSD → remote), SLA-driven autoscaling (Planner), in-flight fault tolerance, and a Kubernetes operator for deployment. @@ -20,71 +20,96 @@ KV-block-manager engines. A **Python extensibility layer** (the `ai-dynamo` wheel, bound to the Rust core through PyO3/maturin) holds the frontend, backends, planner, and profiler. A **Kubernetes layer** (`deploy/`) holds the operator, Helm charts, and gateway integration. Treat any change that crosses -these boundaries as non-trivial. Dynamo also sits inside a wider `ai-dynamo` ecosystem of -sibling repos (below) that it integrates with rather than vendors. +these boundaries as non-trivial. -## Agent Instruction Files +## Repository Map + +| Path | Contents | +|------|----------| +| `lib/` | Rust workspace crates: `runtime`, `llm`, `kv-router`, `kvbm-*`, `mocker`, `sidecar/*`, and more (see the root [`Cargo.toml`](Cargo.toml) `[workspace] members`), plus `bindings/python` — the PyO3 extension crate, built via maturin and deliberately excluded from the workspace | +| `components/src/dynamo/` | Python packages: `frontend`, `planner`, `router`, `vllm`/`sglang`/`trtllm`/`tokenspeed` backends, `mocker`, `profiler`, and more | +| `deploy/` | Kubernetes `operator`, Helm charts, `inference-gateway` ext-proc, `observability` | +| `container/` | Dockerfiles and build scripts for runtime and dev images | +| `docs/fern/` | Fern docs site: `pages/` holds every page, the rest is site config (`index.yml`, `docs.yml`, `main.css`, `components/`, `scripts/`, `translations/`) | +| `examples/`, `recipes/` | Runnable examples and deployment recipes | +| `benchmarks/`, `tests/` | Benchmark harnesses and the top-level pytest suite | +| `.ai/` | Agent topic guidelines: `ais-perf-model-guidelines.md` (canonical AISimulate interface), `bash-launch-guidelines.md`, `ci-guidelines.md`, `linear-ticket-refs.md`, `pytest-guidelines.md`, `python-guidelines.md`, `test-model-size-guardrails.md` | +| `.agents/skills/` | Agent skills (see [Skills](#skills)) | +| `agents/`, `agent-docs/` | Role contracts, guides, and rules for the recipe optimization loop | + +## Ecosystem -- `AGENTS.md` is the canonical and only source of agent instructions at every - repository scope. -- Every `AGENTS.md` must have a sibling `CLAUDE.md` containing exactly - `@AGENTS.md` so Claude loads the same instructions. -- Do not put independent instructions in `CLAUDE.md`, and do not make - `AGENTS.md` a symlink to `CLAUDE.md`. -- Add, move, or remove each `AGENTS.md` and `CLAUDE.md` pair together. +Repositories this repo integrates with: + +| Repo | Role | +|------|------| +| [vLLM](https://github.com/vllm-project/vllm) | Inference engine; Dynamo backend in `components/src/dynamo/vllm/`, gRPC sidecar in `lib/sidecar/vllm/` | +| [SGLang](https://github.com/sgl-project/sglang) | Inference engine; Dynamo backend in `components/src/dynamo/sglang/`, gRPC sidecar in `lib/sidecar/sglang/` | +| [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) | Inference engine; Dynamo backend in `components/src/dynamo/trtllm/`, gRPC sidecar in `lib/sidecar/trtllm/` | +| [TokenSpeed](https://github.com/lightseekorg/tokenspeed) | Inference engine; Dynamo backend in `components/src/dynamo/tokenspeed/` | +| [NIXL](https://github.com/ai-dynamo/nixl) | High-throughput inference data-transfer library (KV-cache transfer over RDMA/NVLink) that underpins disaggregated serving | +| [AIPerf](https://github.com/ai-dynamo/aiperf) | Benchmarking and load-generation tool used by the benchmarking guides | +| [AISimulate](https://pypi.org/project/aisimulate/) | Predicts serving behavior and searches deployment configurations offline without requiring a GPU cluster | +| [ModelExpress](https://github.com/ai-dynamo/modelexpress) | Streams model weights GPU-to-GPU via NIXL for fast replica cold-start | +| [Grove](https://github.com/ai-dynamo/grove) | Kubernetes operator for topology-aware gang scheduling | ## Skills Skills live canonically in `.agents/skills/`; `skills/` and `.claude/skills/` are symlinks -to it — edit only the canonical copy. Reach for the right group first: +to it — edit only the canonical copy. -**For developing Dynamo:** +**Build, debug, and measure:** -- `debug-session` — structured bug investigation with a persistent worklog -- `dep-create` — create or update Dynamo Enhancement Proposals as GitHub issues -- `dep-status` — check DEP status and list DEPs by lifecycle state or area -- `dep-update` — advance DEP lifecycle: triage, PIC assignment, review, approval +- `debug-session` — investigate a bug with a persistent worklog: capture the environment, rebuild, reproduce +- `dynamo-frontend-benchmark` — benchmark and profile the frontend against `dynamo.mocker` workers +- `dynamo-kv-replay-parity` — byte-parity and paired performance campaigns for offline KV-aware replay - `dynamo-clone-hotpath-audit` — audit Rust hot-path `.clone()` calls -- `dynamo-docs` — Fern docs-site content per the style guide -- `dynamo-frontend-benchmark` — benchmark/profile the frontend against mock workers -- `fern-components` — Fern MDX component library and usage guidance -- `fern-navigation` — Fern navigation and site-structure configuration guidance -- `dynamo-kv-replay-parity` — validate offline KV replay parity and performance -- `dynamo-agent-harness` — drive persistent Claude Code, Codex, or OpenCode sessions through Dynamo over ACP +- `dynamo-agent-harness` — drive Claude Code, Codex, or OpenCode sessions through a Dynamo endpoint over ACP + +**Pull requests, review, and CI:** + +- `pr-checklist` — check a PR for design fit, tests, code quality, and description, when making or reviewing one - `graham-code-review` — strict Rust/systems review in Graham King's style -- `dynamo-issue-first` — start from a tracked issue and reference it from the PR -- `pr-checklist` — check a PR when making it or reviewing it -- `pr-monitor` — CI health check, failure root-cause, and skip analysis +- `visual-review` — interactive HTML review dashboard for a PR or diff +- `pr-monitor` — a PR's CI status, why full CI has or has not triggered, and failure root cause against `main` +- `dynamo-issue-first` — start work from a tracked issue and reference it from the PR +- `repo-codeowners` — who reviews a change, fixing the `codeowners` check, changing review routing - `dynamo-release-cherry-pick` — land a fix on a release branch: cherry-pick, release-only fix, or external Cherry-Pick Request -- `repo-codeowners` — who reviews a change, fixing a failing `codeowners` check, changing review routing -- `visual-review` — interactive HTML code-review dashboards with diagrams and annotated diffs - -When reviewing frontend or runtime changes, also read the corresponding -[frontend review prompt](.github/review-prompts/frontend.md) or -[runtime review prompt](.github/review-prompts/runtime.md) for additional CODEOWNERS guidance. - -Before changing request or response fields, validation, preprocessing, backend translation, or -response projection, read the -[protocol field handling contract](docs/fern/pages/developer-guide/knowledge-base/modular-components/frontend/protocol-field-handling.md). -Identify every semantic owner and preserve the field's documented end-to-end behavior. - -**For deploying and operating Dynamo:** - -- `synthesize-user-workload` — interview the user, capture their confirmed baseline DGD, and create the canonical workload contract -- `author-baseline-dgd` — draft a baseline DGD from interview requirements when no recipe matches, for the user's confirmation -- `consult-perf-knowledge` — select one evidence-backed optimization proposal and write its reasoning record -- `create-optimization-hypothesis` — materialize a performance consultation as a challenger-ready DGD draft -- `perform-adversarial-review` — challenge a generated DGD candidate before it consumes GPU time -- `deploy-dynamo-recipe` — deploy one assigned Kubernetes DGD and verify it with an API smoke test -- `configure-aiperf-benchmark` — freeze and render a comparable AIPerf workload for a deployed candidate -- `run-aiperf-benchmark` — execute and collect one run-scoped AIPerf Kubernetes benchmark -- `analyze-aiperf-results` — validate AIPerf evidence, evaluate SLOs, and compare valid same-series runs -- `find-serving-recipe` — walk the ordered recipe catalogs with provenance gates and write a recipe dossier -- `report-skillpack-issue` — file a sanitized, operator-approved GitHub issue for a defect in the pack itself -- `dynamo-router-starter` — start/patch router modes with smoke checks -- `dynamo-interconnect-check` — validate NIXL/UCX/NCCL readiness for disaggregation + +**Design proposals (DEPs):** + +- `dep-create` — propose a feature or architecture change as a Dynamo Enhancement Proposal issue +- `dep-status` — check where a DEP stands, or list DEPs by lifecycle state or area +- `dep-update` — move a DEP through triage, PIC assignment, review, and approval + +**Docs** (any change under `docs/`, `examples/`, or `recipes/`): + +- `dynamo-docs` — add, update, move, or remove docs-site content per the style guide +- `fern-components` — pick and use Fern MDX components on a page +- `fern-navigation` — Fern site navigation and structure in `docs.yml` + +**Deploy and operate:** + +- `find-serving-recipe` — find the best known serving configuration for a model, hardware, and workload, with provenance +- `dynamo-router-starter` — start or patch router modes and smoke-test the endpoint +- `dynamo-interconnect-check` — validate NIXL/UCX/NCCL readiness for disaggregated serving - `troubleshoot-dynamo` — diagnose failed or unhealthy deployments +**Recipe optimization loop** — start from +[`agent-docs/guides/optimization/optimize-loop.md`](agent-docs/guides/optimization/optimize-loop.md), +which sequences these skills: + +- `synthesize-user-workload` — interview the user, capture their baseline DGD, and write the workload contract +- `author-baseline-dgd` — draft a baseline DGD when no recipe matches, for the user's confirmation +- `deploy-dynamo-recipe` — deploy one assigned DGD and smoke-test its API +- `configure-aiperf-benchmark` — freeze a comparable AIPerf workload for a deployed candidate +- `run-aiperf-benchmark` — run and collect one AIPerf Kubernetes benchmark +- `analyze-aiperf-results` — validate AIPerf results and compare them against SLOs and prior runs +- `consult-perf-knowledge` — select one evidence-backed optimization proposal +- `create-optimization-hypothesis` — turn a proposal into a challenger-ready DGD draft +- `perform-adversarial-review` — challenge a DGD draft before it consumes GPU time +- `report-skillpack-issue` — file a sanitized, operator-approved issue for a defect in these skills or guides + **Adding a skill:** the folder name must equal the frontmatter `name` (kebab-case); the `description` is third person, states what the skill does and when to use it, and is at most 1024 characters; include `license: Apache-2.0` and a `metadata:` block with `author` @@ -93,99 +118,26 @@ exactly. All of this is enforced by `scripts/validate_skills.py` (pre-commit hoo `validate-skills`). Changes under `.agents/skills/` are also validated by NVSkills CI — a maintainer comments `/nvskills-ci` on the PR. -## Improving These Instructions - -If these skills, instructions, role contracts, or this file misled you, blocked you, contradicted what you verified -live, or left a component or situation uncovered, do not route around it silently: invoke the -`report-skillpack-issue` skill (`.agents/skills/report-skillpack-issue/`), which every role contract declares and -which owns the full procedure. Dispatched roles record drafts in `/analysis/skillpack-defects.md` and return -them; only the top-level agent, with operator approval, files, at most one new issue per session with findings -batched, plus comments on duplicates. If your harness cannot surface the skill, follow this minimum, which the skill -also enforces: search existing reports by title, body, and comments first; identify yourself as an AI agent with -your driver model and the skills commit; sanitize every emitted string (title, body, comment, search term: no -workload details, traffic numbers, cluster or namespace names, company names, or credentials); keep drafts as -run-scoped files, never in a shared temp path; and show the operator the exact draft and file only on their -approval, using the `[AGENT]: ` title prefix and verifying the label landed. - -## Optimization Role Dispatch - -When the first user message starts a new Dynamo recipe optimization run, FIRST read -`agent-docs/guides/optimization/optimize-loop.md` end to end - the individual SKILL.md files are auto-discoverable, -but the loop's sequencing, state machine, and stopping rules live only in that guide - then dispatch -`user_interviewer` before any other specialized role. It must invoke `synthesize-user-workload` and produce a validated -`/user_workload.yaml` plus an immutable `/inputs/user_provided_dgd.yaml` copied from the baseline the -user supplied or explicitly confirmed. Do not dispatch `recipe_deployer`, `perf_analyzer`, `hypothesis_generator`, or -`hypothesis_challenger` until both exact paths and SHA256 values are available. Pass both inputs directly to -`recipe_deployer`; pass the same immutable workload path and hash to every later role. Do not insert a recipe -exploration or selection step after the interview: the baseline-source ladder -(`agents/user-interviewer/AGENTS.md`) is the only place selection or authoring happens, always with the user's -explicit confirmation, and always before the loop starts. - -## Long-Running Runs And Harness Compatibility - -An optimization loop is long-running, unattended work. Know which harness you are in: in a SINGLE-SHOT harness -(headless `-p`/print mode, one-turn API calls), background-job completion notifications can never reach you - the -session is gone when your turn ends. There, poll synchronously with bounded loops and never park the engagement on -a wake-up you cannot receive; parking is only valid where the harness can re-invoke you (interactive sessions, goal -mode). An interactive harness ends its turn whenever the agent stops -calling tools — a turn that ends on narrated intent ("now I'll test disagg") silently stalls the loop until a human -notices. Two rules: - -1. **Operators: launch unattended runs inside your harness's goal mode.** Goal mode is an operator action at launch, - not something these instructions can enable mid-run. On Codex CLI, wrap the run in `/goal` with a token budget. On - Claude Code (v2.1.139+), wrap it in `/goal`; its completion condition is model-evaluated and may include a bound - such as "or stop after N turns" as part of the condition text (a soft limit, not a hard budget). A validated - condition template: "Test every lever family that is testable within the authorized budget. Never stop because a - report exists. Parked on pending asks with nothing else testable is a valid pause, not completion. Valid stops: an - operator-granted stop-request, the authorized budget exhausted, access lost, or operator interrupt." Always name - the budget (GPU-hours, wall-clock, failed-deployment limit) in the condition; a bare "never stop" silently relies - on credential expiry as its budget. Tell the operator at the START of any optimization - engagement — not only when they say "unattended" — that this is long-running work and how to arm goal mode; the - user-interviewer's contract handoff is the natural moment. Arm goal mode only AFTER the contract questions are - answered: a goal hook armed while questions are outstanding forces the run past them onto its own defaults. The template's parked-on-asks pause assumes a - reachable operator: for runs where the operator will be away, instruct the agent not to park on asks (asks are - logged and the loop continues) and keep only the hard stops. Blocking question tools suspend the turn BEFORE the - goal hook can evaluate, so one blocking question can hang an unattended run for hours; harnesses that support - tool restrictions should disallow blocking question tools in goal mode. -2. **Never end a turn on narrated intent during a loop.** Either perform the next step in the same turn, launch it as - background work that will re-invoke you, or return the specific blocking question you need answered. - -**Harness tiers.** These roles and skills are developed and tested on Claude Code and Codex CLI. Isolated role -configurations currently ship for Codex only (`.codex/agents/*.toml`); on Claude Code the roles run in-context within -one session (no `.claude/agents/` configurations yet), so adversarial review there is same-context review, not an -independent reviewer. The skills follow the Agent Skills open standard and load on other compliant harnesses (for -example, OpenCode includes `.agents/skills/` among its standard skill search paths), with the same in-context role -caveat plus two more degradations: no native goal mode (run lights-out sessions under an external loop), and every -rule in this pack is prompt-enforced, so discipline depends on the driver model. If you hit an instruction gap on any -harness, prepare a sanitized issue describing the gap and ask your operator to approve filing it on this repository. +## Build and Test -## Ecosystem +### Published builds -Sibling repositories this repo integrates with: +Use a published build when you do not need to change it. Nightly CI builds `main` and +publishes the artifacts below; see +[Nightly Releases](docs/fern/pages/reference/general/nightly-releases.md) for details. -| Repo | Role | -|------|------| -| [NIXL](https://github.com/ai-dynamo/nixl) | High-throughput inference data-transfer library (KV-cache transfer over RDMA/NVLink) that underpins disaggregated serving | -| [AIPerf](https://github.com/ai-dynamo/aiperf) | Benchmarking and load-generation tool used by the benchmarking guides | -| [AISimulate](https://pypi.org/project/aisimulate/) | Predicts serving behavior and searches deployment configurations offline without requiring a GPU cluster | -| [ModelExpress](https://github.com/ai-dynamo/modelexpress) | Streams model weights GPU-to-GPU via NIXL for fast replica cold-start | -| [Grove](https://github.com/ai-dynamo/grove) | Kubernetes operator for topology-aware gang scheduling | +| Artifact | Where | +|----------|-------| +| Runtime images | `nvcr.io/nvidia/ai-dynamo/{vllm,sglang,tensorrtllm}-runtime-nightly`, tag `latest` or `YYYYMMDD-`, plus a `-efa` variant of each | +| Component images | `nvcr.io/nvidia/ai-dynamo/{kubernetes-operator,dynamo-frontend,dynamo-planner}-nightly` | +| Python wheels | `ai-dynamo`, `ai-dynamo-runtime`, and `kvbm` on `https://pypi.nvidia.com/`: `uv pip install --pre --extra-index-url https://pypi.nvidia.com/ ai-dynamo` | +| Helm chart | `dynamo-platform-nightly` on NGC | +| Stable and prerelease builds, Rust crates | [Release Artifacts](docs/fern/pages/reference/general/release-artifacts.mdx) | -## Repository Map +Pin the dated tag to reproduce a result; `latest` moves every night. To see the engine +version in an image, run `docker run --rm pip show vllm`. -| Path | Contents | -|------|----------| -| `lib/` | Rust workspace crates: `runtime`, `llm`, `kv-router`, `kvbm-*`, `mocker`, and more (see the root [`Cargo.toml`](Cargo.toml) `[workspace] members`), plus `bindings/python` — the PyO3 extension crate, built via maturin and deliberately excluded from the workspace | -| `components/src/dynamo/` | Python packages: `frontend`, `planner`, `router`, `vllm`/`sglang`/`trtllm` backends, `mocker`, `profiler`, and more | -| `deploy/` | Kubernetes `operator`, Helm charts, `inference-gateway` ext-proc, `observability` | -| `container/` | Dockerfiles and build scripts for runtime and dev images | -| `docs/fern/` | Fern docs site: `pages/` holds every page, the rest is site config (`index.yml`, `docs.yml`, `main.css`, `components/`, `scripts/`, `translations/`). Read [`docs/fern/AGENTS.md`](docs/fern/AGENTS.md) before editing, and [`docs/fern/pages/AGENTS.md`](docs/fern/pages/AGENTS.md) before adding a page | -| `examples/`, `recipes/` | Runnable examples and deployment recipes — also covered by [`docs/fern/AGENTS.md`](docs/fern/AGENTS.md) | -| `benchmarks/`, `tests/` | Benchmark harnesses and the top-level pytest suite | -| `.ai/` | Agent topic guidelines: `ais-perf-model-guidelines.md` (canonical AISimulate interface), `bash-launch-guidelines.md`, `ci-guidelines.md`, `linear-ticket-refs.md`, `pytest-guidelines.md`, `python-guidelines.md`, `test-model-size-guardrails.md` | -| `.agents/skills/` | Agent skills (see [Skills](#skills)) | - -## Build +### Build from source System prerequisites (Rust toolchain, `uv`, system libraries) and the VS Code / Cursor devcontainer are covered in [the contribution guide](docs/fern/pages/community/contributing/overview.md). @@ -201,96 +153,113 @@ uv pip install -e . python3 -m dynamo.frontend --help # verify ``` -Rust-only: +Python edits under `components/src/` take effect without a rebuild. After a Rust change +that Python uses, rerun `maturin develop --uv` in `lib/bindings/python`. + +To work against an engine, build a `local-dev` image and mount your worktree into it. +[`container/README.md`](container/README.md) describes every target. ```bash -cargo build # whole workspace -cargo build -p dynamo-llm # one crate +container/render.py --framework vllm --target local-dev --output-short-filename +docker build --build-arg USER_UID=$(id -u) --build-arg USER_GID=$(id -g) \ + -t dynamo:latest-vllm-local-dev -f container/rendered.Dockerfile . +container/run.sh --mount-workspace -it --image dynamo:latest-vllm-local-dev ``` -## Test +### Rust: build and test efficiently + +Most Rust builds wait on one serial chain of crates that ends in `dynamo-llm`. More +cores, sccache, or a remote cache barely help; the cost is set by which crates you build +and which test binaries you link. Approximate times on one workstation (October 2026), +for relative cost: + +| After you edit | Command | Time | +|----------------|---------|------| +| a sidecar | `cargo build -p dynamo-vllm-sidecar` | 7 s | +| `dynamo-llm` | `cargo check -p dynamo-llm` | 16 s | +| `dynamo-llm` | `cargo test -p dynamo-llm --test ` | 46 s | +| `dynamo-llm` | `cargo test -p dynamo-llm --lib` | 61 s | +| `dynamo-llm` | `cargo test -p dynamo-llm` (every target) | 75 s | +| nothing, in a new worktree | the first sidecar build | 4 min | + +The edit loop: ```bash -cargo test # Rust -pytest -m unit tests/ # Python unit tests +# 1. Type-check only the crate you are editing, until it compiles. +cargo check -p dynamo-kv-router + +# 2. Run only the tests you need: name one target, then filter by test path. +cargo test -p dynamo-kv-router --lib scheduling:: +cargo test -p dynamo-llm --test http-service + +# 3. Before you push, format and lint the crates you touched as CI does. +cargo fmt --all +cargo clippy --locked --no-deps -p dynamo-kv-router --all-targets -- -D warnings ``` +- **Keep one `-p` set per area.** Cargo resolves features across the packages you name, + so switching between `-p a` and `-p a -p b` can recompile `dynamo-llm` with different + features. For the sidecars, always use + `-p dynamo-vllm-sidecar -p dynamo-sglang-sidecar -p dynamo-trtllm-sidecar -p dynamo-lpx-sidecar -p dynamo-sidecar-testkit`. +- **Name the test target.** `cargo test -p dynamo-llm` with no target builds the unit-test + harness (a second compile of `dynamo-llm`), every integration-test binary, and the doc + tests. Use `--lib` or `--test `. +- **Keep build flags stable.** [`.cargo/config.toml`](.cargo/config.toml) sets the + `rustflags`. A `RUSTFLAGS` variable replaces them and recompiles every crate, and so + does a change to the dev profile or the toolchain. To make debug builds smaller, set + `CARGO_PROFILE_DEV_DEBUG=line-tables-only` once and leave it set. +- **Reuse your worktree.** A new worktree starts with an empty build cache; switching + branches in an existing one rebuilds only the crates that changed. +- **Measure performance on optimized builds.** Debug timings mean nothing. Use + `--release` (`maturin develop --uv --release` for the bindings). To profile with perf or + Nsight, build with `--profile profiling`, which is release with symbols; + `-p dynamo-llm --features nvtx` adds NVTX ranges. +- **Run the feature-gated lanes you touch.** CI also checks some crates with non-default + features, such as `dynamo-llm --no-default-features` and + `dynamo-kv-router --features standalone-indexer`. The exact commands are in + [`.github/workflows/pre-merge.yml`](.github/workflows/pre-merge.yml). + On macOS, run `dynamo-llm` checks and targeted tests with `--no-default-features` unless the target explicitly requires `block-manager`. The default feature enables Linux/CUDA-oriented NIXL, NUMA, and `O_DIRECT` code that is not a valid general-purpose macOS validation path. -Markers are strict (`--strict-markers`); the full marker list lives in -[`pyproject.toml`](pyproject.toml) `[tool.pytest.ini_options]`, including GPU gating -(`gpu_0` … `gpu_8`). Read [`.ai/pytest-guidelines.md`](.ai/pytest-guidelines.md) and +### Python tests + +```bash +# Preview what a selection runs before you run it. +pytest --collect-only -q -m "integration and pre_merge" tests/router + +# No GPU: unit tests, then integration tests against dynamo.mocker workers. +pytest -m "unit and pre_merge" +pytest -m "integration and pre_merge" + +# Tests marked parallel-safe, spread across cores. +pytest -m "pre_merge and parallel and gpu_0" -n auto --dist=loadscope + +# One test while you iterate. +pytest tests/router/.py -k + +# GPU end to end, one backend at a time (also sglang, trtllm). +pytest -m "vllm and e2e and gpu_1" +``` + +`dynamo.mocker` stands in for an engine, so router, planner, and frontend changes can be +tested end to end without a GPU. Markers are strict (`--strict-markers`); the full marker +list lives in [`pyproject.toml`](pyproject.toml) `[tool.pytest.ini_options]`, including GPU +gating (`gpu_0` … `gpu_8`). Read [`tests/README.md`](tests/README.md), +[`.ai/pytest-guidelines.md`](.ai/pytest-guidelines.md), and [`.ai/test-model-size-guardrails.md`](.ai/test-model-size-guardrails.md) before writing tests. -## Lint +### Lint ```bash -pre-commit run --all-files # all hooks (run `pre-commit install` first; it also installs the DCO commit-msg hook) -cargo fmt --all && cargo clippy --workspace +pre-commit run --files # hooks on what you changed +pre-commit run --all-files # every hook; run `pre-commit install` first (it also installs the DCO commit-msg hook) ``` -## PR and Commit Conventions - -- Keep changes focused and reviewable. -- Use Conventional Commit PR titles: `type(scope): summary`. Accepted types: - `feat`, `fix`, `docs`, `test`, `ci`, `refactor`, `perf`, `chore`, `revert`, - `style`, and `build`. -- PR descriptions follow [`.github/pull_request_template.md`](.github/pull_request_template.md): - Overview, Key Changes, Validation, Issues and Ongoing Work, and the checklist. Run the - `pr-checklist` skill when making or reviewing a PR. -- Every PR references the tracked issue it implements, in the title, description, or - branch name (`Closes #123`, `Closes DYN-1234`, `user/dyn-1234-description`). The - `PR Issue Link` check verifies the reference; it is advisory until `release/1.7.0` is - cut, then required. Reference forms and remediation are in the `dynamo-issue-first` skill. -- Sign every commit with DCO: `git commit -s`. -- For fork PRs that qualify for automatic CI approval, GitHub must report every commit's - signature as `Verified`. A DCO sign-off alone is not enough. Signing commits does not - grant automatic approval; a maintainer can approve the current head manually with - `/ok to test `. - The only signature exception is a full-CI request from `glamr-agent` on its own PR. - The commenter, PR author, and head repository owner must all be `glamr-agent`, - checked by GitHub account ID. The request's full SHA must match the PR's current - head. Other automatic approvals still require verified signatures. -- Do not hand-edit a generated artifact — change its source and regenerate. A - generated file says so in a `do not edit` marker, and its generator has a - `--check` mode that fails when the committed output is stale. Resolve a - conflict in a generated file by regenerating rather than editing the - conflict — a hand-resolved artifact passes review and then fails the next - `--check` — and resolve one in an aggregate list, such as a coverage set or - a filter list, as the union of both sides. -- Do not hand-edit the root `CODEOWNERS` — it is generated. To change review - routing, edit `.github/codeowners/areas.yaml` and regenerate; CI gates 100% - coverage and `CODEOWNERS`↔`areas.yaml` drift. See - `.github/codeowners/README.md`. To check who reviews your PR: - `python .github/codeowners/who_owns.py --codeowners CODEOWNERS --changed` - (`--people` expands teams to members for org members). - If the `codeowners` check fails after adding a new directory, claim it with - one line under the owning area in `areas.yaml`, regenerate, and commit both - files together. External contributors earn area-scoped co-ownership via - `.github/codeowners/external_contributors.yaml`. The `repo-codeowners` - skill automates all of this. -- Full CI on a PR runs only after a maintainer comments `/ok to test ` with the short - SHA of the latest commit; copy-pr-bot then creates the `pull-request/N` branch that - triggers it. For an eligible fork PR, automatic approval posts this command after the - checks above pass. Fix CI failures before requesting human review. -- Architecture changes require a Dynamo Enhancement Proposal (DEP), filed as a GitHub - issue on `ai-dynamo/dynamo` with `dep:*` labels (the `dep-create` skill automates this). - -See [the contribution guide](docs/fern/pages/community/contributing/overview.md) for the full workflow -(issue sizing, CODEOWNERS, review process). - -## Docs, Examples, Recipes - -Any change under `docs/`, `examples/`, or `recipes/` must follow -[`docs/fern/AGENTS.md`](docs/fern/AGENTS.md) and the -[documentation style guide](docs/fern/pages/community/contributing/documentation/documentation-style-guide.md): SPDX headers, Fern -frontmatter (no body `# H1`), GitHub-style admonitions, and backend casing -(vLLM / SGLang / TensorRT-LLM). The deterministic subset is enforced pre-merge. - -The docs site is tab-based: `docs/fern/pages/` splits into `kubernetes/` and `cli/` (parallel -guides for two readers), plus `use-cases/`, `recipes/`, `developer-guide/`, `reference/`, `blog/`, -and `community/`. Read [`docs/fern/pages/AGENTS.md`](docs/fern/pages/AGENTS.md) to pick the right -tab before adding a page — a misplaced page costs a move plus a redirect. +## Pull Requests + +Run the `pr-checklist` skill when you make or review a pull request. +[`CONTRIBUTING.md`](CONTRIBUTING.md) covers DCO sign-off, commit titles, and how full CI is +triggered. diff --git a/agent-docs/guides/optimization/optimize-loop.md b/agent-docs/guides/optimization/optimize-loop.md index d4f8f3fd7824..5b3185631621 100644 --- a/agent-docs/guides/optimization/optimize-loop.md +++ b/agent-docs/guides/optimization/optimize-loop.md @@ -48,6 +48,58 @@ LOOP itself has no recipe-discovery or recipe-selection step. When using Codex multi-agent mode, dispatch registered roles through `.codex/config.toml`. Each launcher must read and follow its corresponding `agents//AGENTS.md` contract. +## Long-Running Runs And Harness Compatibility + +An optimization loop is long-running, unattended work. Know which harness you are in: in a SINGLE-SHOT harness +(headless `-p`/print mode, one-turn API calls), background-job completion notifications can never reach you - the +session is gone when your turn ends. There, poll synchronously with bounded loops and never park the engagement on +a wake-up you cannot receive; parking is only valid where the harness can re-invoke you (interactive sessions, goal +mode). An interactive harness ends its turn whenever the agent stops +calling tools — a turn that ends on narrated intent ("now I'll test disagg") silently stalls the loop until a human +notices. Two rules: + +1. **Operators: launch unattended runs inside your harness's goal mode.** Goal mode is an operator action at launch, + not something these instructions can enable mid-run. On Codex CLI, wrap the run in `/goal` with a token budget. On + Claude Code (v2.1.139+), wrap it in `/goal`; its completion condition is model-evaluated and may include a bound + such as "or stop after N turns" as part of the condition text (a soft limit, not a hard budget). A validated + condition template: "Test every lever family that is testable within the authorized budget. Never stop because a + report exists. Parked on pending asks with nothing else testable is a valid pause, not completion. Valid stops: an + operator-granted stop-request, the authorized budget exhausted, access lost, or operator interrupt." Always name + the budget (GPU-hours, wall-clock, failed-deployment limit) in the condition; a bare "never stop" silently relies + on credential expiry as its budget. Tell the operator at the START of any optimization + engagement — not only when they say "unattended" — that this is long-running work and how to arm goal mode; the + user-interviewer's contract handoff is the natural moment. Arm goal mode only AFTER the contract questions are + answered: a goal hook armed while questions are outstanding forces the run past them onto its own defaults. The template's parked-on-asks pause assumes a + reachable operator: for runs where the operator will be away, instruct the agent not to park on asks (asks are + logged and the loop continues) and keep only the hard stops. Blocking question tools suspend the turn BEFORE the + goal hook can evaluate, so one blocking question can hang an unattended run for hours; harnesses that support + tool restrictions should disallow blocking question tools in goal mode. +2. **Never end a turn on narrated intent during a loop.** Either perform the next step in the same turn, launch it as + background work that will re-invoke you, or return the specific blocking question you need answered. + +**Harness tiers.** These roles and skills are developed and tested on Claude Code and Codex CLI. Isolated role +configurations currently ship for Codex only (`.codex/agents/*.toml`); on Claude Code the roles run in-context within +one session (no `.claude/agents/` configurations yet), so adversarial review there is same-context review, not an +independent reviewer. The skills follow the Agent Skills open standard and load on other compliant harnesses (for +example, OpenCode includes `.agents/skills/` among its standard skill search paths), with the same in-context role +caveat plus two more degradations: no native goal mode (run lights-out sessions under an external loop), and every +rule in this pack is prompt-enforced, so discipline depends on the driver model. If you hit an instruction gap on any +harness, prepare a sanitized issue describing the gap and ask your operator to approve filing it on this repository. + +## Improving These Instructions + +If these skills, instructions, role contracts, or this file misled you, blocked you, contradicted what you verified +live, or left a component or situation uncovered, do not route around it silently: invoke the +`report-skillpack-issue` skill (`.agents/skills/report-skillpack-issue/`), which every role contract declares and +which owns the full procedure. Dispatched roles record drafts in `/analysis/skillpack-defects.md` and return +them; only the top-level agent, with operator approval, files, at most one new issue per session with findings +batched, plus comments on duplicates. If your harness cannot surface the skill, follow this minimum, which the skill +also enforces: search existing reports by title, body, and comments first; identify yourself as an AI agent with +your driver model and the skills commit; sanitize every emitted string (title, body, comment, search term: no +workload details, traffic numbers, cluster or namespace names, company names, or credentials); keep drafts as +run-scoped files, never in a shared temp path; and show the operator the exact draft and file only on their +approval, using the `[AGENT]: ` title prefix and verifying the label landed. + ## 1. Interview, Capture The DGD, And Synthesize The Workload Immediately dispatch `user-interviewer` with the user's first optimization message and any supplied attachments. It From fe3453fed544e19cadc6e01b7217137e78f96f27 Mon Sep 17 00:00:00 2001 From: Alec Flowers Date: Wed, 28 Oct 2026 11:17:21 -0700 Subject: [PATCH 2/6] docs(agents): leave optimize-loop.md and synthesize-user-workload unchanged Keep this PR to the root AGENTS.md. Restore both files to main. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Alec Flowers --- .../skills/synthesize-user-workload/SKILL.md | 2 +- .../guides/optimization/optimize-loop.md | 52 ------------------- 2 files changed, 1 insertion(+), 53 deletions(-) diff --git a/.agents/skills/synthesize-user-workload/SKILL.md b/.agents/skills/synthesize-user-workload/SKILL.md index 9cc82298d4db..78d5a5c5e46b 100644 --- a/.agents/skills/synthesize-user-workload/SKILL.md +++ b/.agents/skills/synthesize-user-workload/SKILL.md @@ -201,5 +201,5 @@ supporting context for `perf-analyzer`, `hypothesis-generator`, and `hypothesis- End with an operator handoff line before any deployment starts: state that the interview is complete, give the contract path, and tell the operator that the engagement now runs a long unattended loop — without goal mode the -session pauses for input at every turn end — and that this is the moment to arm it (`agent-docs/guides/optimization/optimize-loop.md`, Long-Running Runs, +session pauses for input at every turn end — and that this is the moment to arm it (AGENTS.md, Long-Running Runs, has the condition template; fill in the budget). diff --git a/agent-docs/guides/optimization/optimize-loop.md b/agent-docs/guides/optimization/optimize-loop.md index 5b3185631621..d4f8f3fd7824 100644 --- a/agent-docs/guides/optimization/optimize-loop.md +++ b/agent-docs/guides/optimization/optimize-loop.md @@ -48,58 +48,6 @@ LOOP itself has no recipe-discovery or recipe-selection step. When using Codex multi-agent mode, dispatch registered roles through `.codex/config.toml`. Each launcher must read and follow its corresponding `agents//AGENTS.md` contract. -## Long-Running Runs And Harness Compatibility - -An optimization loop is long-running, unattended work. Know which harness you are in: in a SINGLE-SHOT harness -(headless `-p`/print mode, one-turn API calls), background-job completion notifications can never reach you - the -session is gone when your turn ends. There, poll synchronously with bounded loops and never park the engagement on -a wake-up you cannot receive; parking is only valid where the harness can re-invoke you (interactive sessions, goal -mode). An interactive harness ends its turn whenever the agent stops -calling tools — a turn that ends on narrated intent ("now I'll test disagg") silently stalls the loop until a human -notices. Two rules: - -1. **Operators: launch unattended runs inside your harness's goal mode.** Goal mode is an operator action at launch, - not something these instructions can enable mid-run. On Codex CLI, wrap the run in `/goal` with a token budget. On - Claude Code (v2.1.139+), wrap it in `/goal`; its completion condition is model-evaluated and may include a bound - such as "or stop after N turns" as part of the condition text (a soft limit, not a hard budget). A validated - condition template: "Test every lever family that is testable within the authorized budget. Never stop because a - report exists. Parked on pending asks with nothing else testable is a valid pause, not completion. Valid stops: an - operator-granted stop-request, the authorized budget exhausted, access lost, or operator interrupt." Always name - the budget (GPU-hours, wall-clock, failed-deployment limit) in the condition; a bare "never stop" silently relies - on credential expiry as its budget. Tell the operator at the START of any optimization - engagement — not only when they say "unattended" — that this is long-running work and how to arm goal mode; the - user-interviewer's contract handoff is the natural moment. Arm goal mode only AFTER the contract questions are - answered: a goal hook armed while questions are outstanding forces the run past them onto its own defaults. The template's parked-on-asks pause assumes a - reachable operator: for runs where the operator will be away, instruct the agent not to park on asks (asks are - logged and the loop continues) and keep only the hard stops. Blocking question tools suspend the turn BEFORE the - goal hook can evaluate, so one blocking question can hang an unattended run for hours; harnesses that support - tool restrictions should disallow blocking question tools in goal mode. -2. **Never end a turn on narrated intent during a loop.** Either perform the next step in the same turn, launch it as - background work that will re-invoke you, or return the specific blocking question you need answered. - -**Harness tiers.** These roles and skills are developed and tested on Claude Code and Codex CLI. Isolated role -configurations currently ship for Codex only (`.codex/agents/*.toml`); on Claude Code the roles run in-context within -one session (no `.claude/agents/` configurations yet), so adversarial review there is same-context review, not an -independent reviewer. The skills follow the Agent Skills open standard and load on other compliant harnesses (for -example, OpenCode includes `.agents/skills/` among its standard skill search paths), with the same in-context role -caveat plus two more degradations: no native goal mode (run lights-out sessions under an external loop), and every -rule in this pack is prompt-enforced, so discipline depends on the driver model. If you hit an instruction gap on any -harness, prepare a sanitized issue describing the gap and ask your operator to approve filing it on this repository. - -## Improving These Instructions - -If these skills, instructions, role contracts, or this file misled you, blocked you, contradicted what you verified -live, or left a component or situation uncovered, do not route around it silently: invoke the -`report-skillpack-issue` skill (`.agents/skills/report-skillpack-issue/`), which every role contract declares and -which owns the full procedure. Dispatched roles record drafts in `/analysis/skillpack-defects.md` and return -them; only the top-level agent, with operator approval, files, at most one new issue per session with findings -batched, plus comments on duplicates. If your harness cannot surface the skill, follow this minimum, which the skill -also enforces: search existing reports by title, body, and comments first; identify yourself as an AI agent with -your driver model and the skills commit; sanitize every emitted string (title, body, comment, search term: no -workload details, traffic numbers, cluster or namespace names, company names, or credentials); keep drafts as -run-scoped files, never in a shared temp path; and show the operator the exact draft and file only on their -approval, using the `[AGENT]: ` title prefix and verifying the label landed. - ## 1. Interview, Capture The DGD, And Synthesize The Workload Immediately dispatch `user-interviewer` with the user's first optimization message and any supplied attachments. It From d368d0d2e87e994af67853624269648b07534ba8 Mon Sep 17 00:00:00 2001 From: Alec Flowers Date: Wed, 28 Oct 2026 12:26:50 -0700 Subject: [PATCH 3/6] docs(agents): point to README and use verified build and test commands - Replace the overview with a pointer to README.md and drop the engine paths that repeat the repository map. - Re-measure the Rust cost table on a clean worktree. - Add the setup needs found by running the commands: protoc 3.15+, Python 3.11+, and CI's maturin features for the router tests. - Run Python tests by file or test ID with python3 -m pytest instead of marker sweeps, and note the etcd, NATS, and model prerequisites. - Fix the wheel and EFA image rows. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Alec Flowers --- AGENTS.md | 119 +++++++++++++++++++++++++++--------------------------- 1 file changed, 59 insertions(+), 60 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 34edbf91be78..08b620afd974 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -5,28 +5,15 @@ SPDX-License-Identifier: Apache-2.0 # Dynamo — Agent Guide -## Overview - -Dynamo is NVIDIA's open-source, datacenter-scale distributed inference framework. It is -the orchestration layer **above** inference engines (SGLang, TensorRT-LLM, vLLM, TokenSpeed), -not a replacement for them: it turns a cluster of GPUs into one coordinated inference system. -Core capabilities are disaggregated prefill/decode serving, KV-aware routing, multi-tier -KV cache management (KVBM: GPU → CPU → SSD → remote), SLA-driven autoscaling (Planner), -in-flight fault tolerance, and a Kubernetes operator for deployment. - -The stack is deliberately layered and large. A **Rust core** (a Cargo workspace of -twenty-plus crates, mostly under `lib/`) holds the runtime, LLM, routing, and -KV-block-manager engines. A **Python -extensibility layer** (the `ai-dynamo` wheel, bound to the Rust core through PyO3/maturin) -holds the frontend, backends, planner, and profiler. A **Kubernetes layer** (`deploy/`) -holds the operator, Helm charts, and gateway integration. Treat any change that crosses -these boundaries as non-trivial. +[`README.md`](README.md) explains what Dynamo is and what it supports. This guide covers +working in the repo: where code lives, which skill fits a task, and how to build and test +a change. ## Repository Map | Path | Contents | |------|----------| -| `lib/` | Rust workspace crates: `runtime`, `llm`, `kv-router`, `kvbm-*`, `mocker`, `sidecar/*`, and more (see the root [`Cargo.toml`](Cargo.toml) `[workspace] members`), plus `bindings/python` — the PyO3 extension crate, built via maturin and deliberately excluded from the workspace | +| `lib/` | Rust workspace crates: `runtime`, `llm`, `kv-router`, `kvbm-*`, `mocker`, `sidecar/*` (engine gRPC sidecars), and more (see the root [`Cargo.toml`](Cargo.toml) `[workspace] members`), plus `bindings/python` — the PyO3 extension crate, built via maturin and deliberately excluded from the workspace | | `components/src/dynamo/` | Python packages: `frontend`, `planner`, `router`, `vllm`/`sglang`/`trtllm`/`tokenspeed` backends, `mocker`, `profiler`, and more | | `deploy/` | Kubernetes `operator`, Helm charts, `inference-gateway` ext-proc, `observability` | | `container/` | Dockerfiles and build scripts for runtime and dev images | @@ -43,10 +30,7 @@ Repositories this repo integrates with: | Repo | Role | |------|------| -| [vLLM](https://github.com/vllm-project/vllm) | Inference engine; Dynamo backend in `components/src/dynamo/vllm/`, gRPC sidecar in `lib/sidecar/vllm/` | -| [SGLang](https://github.com/sgl-project/sglang) | Inference engine; Dynamo backend in `components/src/dynamo/sglang/`, gRPC sidecar in `lib/sidecar/sglang/` | -| [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) | Inference engine; Dynamo backend in `components/src/dynamo/trtllm/`, gRPC sidecar in `lib/sidecar/trtllm/` | -| [TokenSpeed](https://github.com/lightseekorg/tokenspeed) | Inference engine; Dynamo backend in `components/src/dynamo/tokenspeed/` | +| [vLLM](https://github.com/vllm-project/vllm), [SGLang](https://github.com/sgl-project/sglang), [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM), [TokenSpeed](https://github.com/lightseekorg/tokenspeed) | Inference engines that Dynamo serves | | [NIXL](https://github.com/ai-dynamo/nixl) | High-throughput inference data-transfer library (KV-cache transfer over RDMA/NVLink) that underpins disaggregated serving | | [AIPerf](https://github.com/ai-dynamo/aiperf) | Benchmarking and load-generation tool used by the benchmarking guides | | [AISimulate](https://pypi.org/project/aisimulate/) | Predicts serving behavior and searches deployment configurations offline without requiring a GPU cluster | @@ -129,9 +113,9 @@ publishes the artifacts below; see | Artifact | Where | |----------|-------| -| Runtime images | `nvcr.io/nvidia/ai-dynamo/{vllm,sglang,tensorrtllm}-runtime-nightly`, tag `latest` or `YYYYMMDD-`, plus a `-efa` variant of each | +| Runtime images | `nvcr.io/nvidia/ai-dynamo/{vllm,sglang,tensorrtllm}-runtime-nightly`, tag `latest` or `YYYYMMDD-`; add `-efa` to either tag for the AWS EFA build | | Component images | `nvcr.io/nvidia/ai-dynamo/{kubernetes-operator,dynamo-frontend,dynamo-planner}-nightly` | -| Python wheels | `ai-dynamo`, `ai-dynamo-runtime`, and `kvbm` on `https://pypi.nvidia.com/`: `uv pip install --pre --extra-index-url https://pypi.nvidia.com/ ai-dynamo` | +| Python wheels | `ai-dynamo` and `ai-dynamo-runtime` (Python 3.11+) on `https://pypi.nvidia.com/`: `uv pip install --pre --extra-index-url https://pypi.nvidia.com/ ai-dynamo` | | Helm chart | `dynamo-platform-nightly` on NGC | | Stable and prerelease builds, Rust crates | [Release Artifacts](docs/fern/pages/reference/general/release-artifacts.mdx) | @@ -140,22 +124,31 @@ version in an image, run `docker run --rm pip show vllm`. ### Build from source -System prerequisites (Rust toolchain, `uv`, system libraries) and the VS Code / Cursor -devcontainer are covered in [the contribution guide](docs/fern/pages/community/contributing/overview.md). +Install the system packages, Rust, and `uv` from +[Building from Source](docs/fern/pages/developer-guide/advanced-customizations/building-from-source.md), +plus: + +- **protoc 3.15 or newer.** Ubuntu 22.04's `protobuf-compiler` is 3.12, and the `dynamo-llm` + build fails on `relay.proto`. CI installs protoc 30.2 from the + [protobuf releases](https://github.com/protocolbuffers/protobuf/releases). +- **Python 3.11 or newer.** `uv venv` finds or downloads one; Ubuntu 22.04's `python3` is 3.10. Python dev build (bindings + wheel, editable): ```bash uv venv .venv && source .venv/bin/activate uv pip install pip 'maturin[patchelf]' -cd lib/bindings/python && maturin develop --uv && cd - +cd lib/bindings/python +maturin develop --uv --features kv-indexer,slot-tracker,select-service,mm-routing,ais-forward-pass,request-trace-s3 +cd - uv pip install -e lib/gpu_memory_service uv pip install -e . python3 -m dynamo.frontend --help # verify ``` -Python edits under `components/src/` take effect without a rebuild. After a Rust change -that Python uses, rerun `maturin develop --uv` in `lib/bindings/python`. +The features match the CI wheel. Without `kv-indexer`, router tests that start a standalone +indexer fail. Python edits under `components/src/` take effect without a rebuild. After a +Rust change that Python uses, rerun the `maturin develop` command. To work against an engine, build a `local-dev` image and mount your worktree into it. [`container/README.md`](container/README.md) describes every target. @@ -171,17 +164,20 @@ container/run.sh --mount-workspace -it --image dynamo:latest-vllm-local-dev Most Rust builds wait on one serial chain of crates that ends in `dynamo-llm`. More cores, sccache, or a remote cache barely help; the cost is set by which crates you build -and which test binaries you link. Approximate times on one workstation (October 2026), -for relative cost: +and which test binaries you link. Measured on a 24-core workstation (October 2026, +rustc 1.96.1), for relative cost: | After you edit | Command | Time | |----------------|---------|------| -| a sidecar | `cargo build -p dynamo-vllm-sidecar` | 7 s | -| `dynamo-llm` | `cargo check -p dynamo-llm` | 16 s | -| `dynamo-llm` | `cargo test -p dynamo-llm --test ` | 46 s | -| `dynamo-llm` | `cargo test -p dynamo-llm --lib` | 61 s | -| `dynamo-llm` | `cargo test -p dynamo-llm` (every target) | 75 s | -| nothing, in a new worktree | the first sidecar build | 4 min | +| `dynamo-kv-router` | `cargo check -p dynamo-kv-router` | 2 s | +| `dynamo-kv-router` | `cargo test -p dynamo-kv-router --lib scheduling::` (277 tests) | 11 s | +| a sidecar | `cargo build` with the sidecar `-p` set below | 10–15 s | +| `dynamo-llm` | `cargo check -p dynamo-llm` | 17 s | +| `dynamo-llm` | `cargo test -p dynamo-llm --test http-service`, build and run | 39 s | +| `dynamo-llm` | `cargo test -p dynamo-llm --lib`: build, then run 3,731 tests | 70 s + 22 s | +| `dynamo-llm` | `cargo test -p dynamo-llm`: build all 44 test targets, before running any | 97 s | +| `dynamo-llm` | `maturin develop --uv` in `lib/bindings/python` | 94 s | +| nothing, in a new worktree | first `dynamo-llm` test build / sidecar set / bindings | 4 / 5.5 / 5.6 min | The edit loop: @@ -192,6 +188,7 @@ cargo check -p dynamo-kv-router # 2. Run only the tests you need: name one target, then filter by test path. cargo test -p dynamo-kv-router --lib scheduling:: cargo test -p dynamo-llm --test http-service +cargo test -p dynamo-llm --lib -- # several filters at once # 3. Before you push, format and lint the crates you touched as CI does. cargo fmt --all @@ -199,26 +196,28 @@ cargo clippy --locked --no-deps -p dynamo-kv-router --all-targets -- -D warnings ``` - **Keep one `-p` set per area.** Cargo resolves features across the packages you name, - so switching between `-p a` and `-p a -p b` can recompile `dynamo-llm` with different - features. For the sidecars, always use + so each `-p` set gets its own build of the shared crates; the first build of a new set + costs a rebuild. For the sidecars, always use `-p dynamo-vllm-sidecar -p dynamo-sglang-sidecar -p dynamo-trtllm-sidecar -p dynamo-lpx-sidecar -p dynamo-sidecar-testkit`. - **Name the test target.** `cargo test -p dynamo-llm` with no target builds the unit-test - harness (a second compile of `dynamo-llm`), every integration-test binary, and the doc - tests. Use `--lib` or `--test `. + harness (a second compile of `dynamo-llm`), all 44 test binaries, and the doc tests. + Use `--lib` or `--test `. - **Keep build flags stable.** [`.cargo/config.toml`](.cargo/config.toml) sets the - `rustflags`. A `RUSTFLAGS` variable replaces them and recompiles every crate, and so - does a change to the dev profile or the toolchain. To make debug builds smaller, set - `CARGO_PROFILE_DEV_DEBUG=line-tables-only` once and leave it set. + `rustflags`. A `RUSTFLAGS` variable replaces them and recompiles every dependency, and + so does changing it back; a change to the dev profile or the toolchain does the same. + Set `CARGO_PROFILE_DEV_DEBUG=line-tables-only` once and leave it set: a cold + `dynamo-llm` test build took 4 min with it and 5.5 min with full debug info. - **Reuse your worktree.** A new worktree starts with an empty build cache; switching branches in an existing one rebuilds only the crates that changed. - **Measure performance on optimized builds.** Debug timings mean nothing. Use `--release` (`maturin develop --uv --release` for the bindings). To profile with perf or Nsight, build with `--profile profiling`, which is release with symbols; `-p dynamo-llm --features nvtx` adds NVTX ranges. -- **Run the feature-gated lanes you touch.** CI also checks some crates with non-default +- **Run only the feature lanes you touch.** CI also checks some crates with non-default features, such as `dynamo-llm --no-default-features` and - `dynamo-kv-router --features standalone-indexer`. The exact commands are in - [`.github/workflows/pre-merge.yml`](.github/workflows/pre-merge.yml). + `dynamo-kv-router --features standalone-indexer`; the exact commands are in + [`.github/workflows/pre-merge.yml`](.github/workflows/pre-merge.yml). Each feature set is + a separate build: `dynamo-llm --no-default-features` takes 4 min the first time. On macOS, run `dynamo-llm` checks and targeted tests with `--no-default-features` unless the target explicitly requires `block-manager`. The default feature enables Linux/CUDA-oriented NIXL, @@ -226,26 +225,26 @@ NUMA, and `O_DIRECT` code that is not a valid general-purpose macOS validation p ### Python tests -```bash -# Preview what a selection runs before you run it. -pytest --collect-only -q -m "integration and pre_merge" tests/router - -# No GPU: unit tests, then integration tests against dynamo.mocker workers. -pytest -m "unit and pre_merge" -pytest -m "integration and pre_merge" +Run the tests that cover your change, by file or test ID; leave marker sweeps to CI. +Use `python3 -m pytest` so the venv's pytest and packages are used. -# Tests marked parallel-safe, spread across cores. -pytest -m "pre_merge and parallel and gpu_0" -n auto --dist=loadscope +```bash +# One test while you iterate, then the related tests in its file. +python3 -m pytest "tests/router/test_router_e2e_with_mockers.py::test_router_decisions[local_indexer-tcp]" -q +python3 -m pytest tests/router/test_router_e2e_with_mockers.py -k "test_router_decisions and not disagg" -q -# One test while you iterate. -pytest tests/router/.py -k +# GPU end to end: the same, on the backend's serve tests. +python3 -m pytest tests/serve/test_vllm.py -k guided_decoding -s -# GPU end to end, one backend at a time (also sglang, trtllm). -pytest -m "vllm and e2e and gpu_1" +# Before you push, list the pre-merge CPU tests in the area you changed. +python3 -m pytest --collect-only -q -m "pre_merge and gpu_0" tests/router ``` `dynamo.mocker` stands in for an engine, so router, planner, and frontend changes can be -tested end to end without a GPU. Markers are strict (`--strict-markers`); the full marker +tested end to end without a GPU. Outside the dev containers, install the test plugins with +`uv pip install -r container/deps/requirements.test.txt`, and put `etcd` and `nats-server` on +`PATH`: integration fixtures start their own instances. Tests download their Hugging Face +model unless you pass `--models-dir`. Markers are strict (`--strict-markers`); the full marker list lives in [`pyproject.toml`](pyproject.toml) `[tool.pytest.ini_options]`, including GPU gating (`gpu_0` … `gpu_8`). Read [`tests/README.md`](tests/README.md), [`.ai/pytest-guidelines.md`](.ai/pytest-guidelines.md), and From 0871608ec264867dc211111831c3d5760bbf0f00 Mon Sep 17 00:00:00 2001 From: Alec Flowers Date: Thu, 29 Oct 2026 11:58:30 -0700 Subject: [PATCH 4/6] docs(agents): drop the skill-authoring rules from AGENTS.md scripts/validate_skills.py enforces them, so the root guide does not need to repeat them. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Alec Flowers --- AGENTS.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 08b620afd974..ae919df02466 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -95,14 +95,6 @@ which sequences these skills: - `perform-adversarial-review` — challenge a DGD draft before it consumes GPU time - `report-skillpack-issue` — file a sanitized, operator-approved issue for a defect in these skills or guides -**Adding a skill:** the folder name must equal the frontmatter `name` (kebab-case); the -`description` is third person, states what the skill does and when to use it, and is at -most 1024 characters; include `license: Apache-2.0` and a `metadata:` block with `author` -and `tags`. List the skill in this section — the index must match `.agents/skills/` -exactly. All of this is enforced by `scripts/validate_skills.py` (pre-commit hook -`validate-skills`). Changes under `.agents/skills/` are also validated by NVSkills CI — -a maintainer comments `/nvskills-ci` on the PR. - ## Build and Test ### Published builds From 618ca06ccda9b58481b78cfdcd9a83636a1b030c Mon Sep 17 00:00:00 2001 From: Alec Flowers Date: Thu, 29 Oct 2026 12:53:31 -0700 Subject: [PATCH 5/6] docs(agents): cut the Rust build rules to the ones that change a command Drops the Cargo explanation, the numbers already in the cost table, and the debug-info tip. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Alec Flowers --- AGENTS.md | 33 +++++++++++---------------------- 1 file changed, 11 insertions(+), 22 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index ae919df02466..e2cb76a7aeb7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -187,29 +187,18 @@ cargo fmt --all cargo clippy --locked --no-deps -p dynamo-kv-router --all-targets -- -D warnings ``` -- **Keep one `-p` set per area.** Cargo resolves features across the packages you name, - so each `-p` set gets its own build of the shared crates; the first build of a new set - costs a rebuild. For the sidecars, always use +- **Name the test target** (`--lib` or `--test `). Plain `cargo test -p dynamo-llm` + builds all 44 test binaries. +- **Don't set `RUSTFLAGS`.** It replaces the flags in [`.cargo/config.toml`](.cargo/config.toml) + and rebuilds every dependency, and unsetting it rebuilds them again. +- **Build the sidecars with one `-p` set:** `-p dynamo-vllm-sidecar -p dynamo-sglang-sidecar -p dynamo-trtllm-sidecar -p dynamo-lpx-sidecar -p dynamo-sidecar-testkit`. -- **Name the test target.** `cargo test -p dynamo-llm` with no target builds the unit-test - harness (a second compile of `dynamo-llm`), all 44 test binaries, and the doc tests. - Use `--lib` or `--test `. -- **Keep build flags stable.** [`.cargo/config.toml`](.cargo/config.toml) sets the - `rustflags`. A `RUSTFLAGS` variable replaces them and recompiles every dependency, and - so does changing it back; a change to the dev profile or the toolchain does the same. - Set `CARGO_PROFILE_DEV_DEBUG=line-tables-only` once and leave it set: a cold - `dynamo-llm` test build took 4 min with it and 5.5 min with full debug info. -- **Reuse your worktree.** A new worktree starts with an empty build cache; switching - branches in an existing one rebuilds only the crates that changed. -- **Measure performance on optimized builds.** Debug timings mean nothing. Use - `--release` (`maturin develop --uv --release` for the bindings). To profile with perf or - Nsight, build with `--profile profiling`, which is release with symbols; - `-p dynamo-llm --features nvtx` adds NVTX ranges. -- **Run only the feature lanes you touch.** CI also checks some crates with non-default - features, such as `dynamo-llm --no-default-features` and - `dynamo-kv-router --features standalone-indexer`; the exact commands are in - [`.github/workflows/pre-merge.yml`](.github/workflows/pre-merge.yml). Each feature set is - a separate build: `dynamo-llm --no-default-features` takes 4 min the first time. + A different set changes features and rebuilds `dynamo-llm`. +- **Reuse your worktree.** A new one starts with an empty build cache (last row of the table). +- **Profiling:** `--profile profiling` is release with symbols; `-p dynamo-llm --features nvtx` + adds NVTX ranges. +- CI's non-default feature checks are in [`pre-merge.yml`](.github/workflows/pre-merge.yml). + Each is a separate cold build (about 4 min), so run only the one you touched. On macOS, run `dynamo-llm` checks and targeted tests with `--no-default-features` unless the target explicitly requires `block-manager`. The default feature enables Linux/CUDA-oriented NIXL, From 508e8058c76cc8a6b722b09b022637bc03dfe269 Mon Sep 17 00:00:00 2001 From: Alec Flowers Date: Thu, 29 Oct 2026 12:59:43 -0700 Subject: [PATCH 6/6] docs(agents): trim the ecosystem, repo map, prerequisites, tests, and lint sections - Ecosystem: one sentence of links instead of a table of product descriptions. - Repository map: drop file lists that ls shows and that go stale. - Prerequisites: state the versions without the Ubuntu explanation. - Python tests: drop the marker details that tests/README.md and .ai/pytest-guidelines.md cover, and the --collect-only example. - Lint: pre-commit install once, hooks run on commit. - Shorten the opening, the Rust intro, and the macOS note. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Alec Flowers --- AGENTS.md | 63 ++++++++++++++++++++----------------------------------- 1 file changed, 23 insertions(+), 40 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index e2cb76a7aeb7..a714920139c2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -5,37 +5,34 @@ SPDX-License-Identifier: Apache-2.0 # Dynamo — Agent Guide -[`README.md`](README.md) explains what Dynamo is and what it supports. This guide covers -working in the repo: where code lives, which skill fits a task, and how to build and test -a change. +[`README.md`](README.md) explains what Dynamo is and what it supports. ## Repository Map | Path | Contents | |------|----------| -| `lib/` | Rust workspace crates: `runtime`, `llm`, `kv-router`, `kvbm-*`, `mocker`, `sidecar/*` (engine gRPC sidecars), and more (see the root [`Cargo.toml`](Cargo.toml) `[workspace] members`), plus `bindings/python` — the PyO3 extension crate, built via maturin and deliberately excluded from the workspace | +| `lib/` | Rust crates, including `runtime`, `llm`, `kv-router`, `kvbm-*`, `mocker`, `sidecar/*` (engine gRPC sidecars), and `bindings/python` (the PyO3 extension, built with maturin, outside the Cargo workspace) | | `components/src/dynamo/` | Python packages: `frontend`, `planner`, `router`, `vllm`/`sglang`/`trtllm`/`tokenspeed` backends, `mocker`, `profiler`, and more | | `deploy/` | Kubernetes `operator`, Helm charts, `inference-gateway` ext-proc, `observability` | | `container/` | Dockerfiles and build scripts for runtime and dev images | -| `docs/fern/` | Fern docs site: `pages/` holds every page, the rest is site config (`index.yml`, `docs.yml`, `main.css`, `components/`, `scripts/`, `translations/`) | +| `docs/fern/` | Fern docs site; pages are in `pages/` | | `examples/`, `recipes/` | Runnable examples and deployment recipes | | `benchmarks/`, `tests/` | Benchmark harnesses and the top-level pytest suite | -| `.ai/` | Agent topic guidelines: `ais-perf-model-guidelines.md` (canonical AISimulate interface), `bash-launch-guidelines.md`, `ci-guidelines.md`, `linear-ticket-refs.md`, `pytest-guidelines.md`, `python-guidelines.md`, `test-model-size-guardrails.md` | +| `.ai/` | Topic guidelines for agents (pytest, Python, CI, launch scripts, AISimulate); read the matching one before writing that kind of code | | `.agents/skills/` | Agent skills (see [Skills](#skills)) | | `agents/`, `agent-docs/` | Role contracts, guides, and rules for the recipe optimization loop | ## Ecosystem -Repositories this repo integrates with: - -| Repo | Role | -|------|------| -| [vLLM](https://github.com/vllm-project/vllm), [SGLang](https://github.com/sgl-project/sglang), [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM), [TokenSpeed](https://github.com/lightseekorg/tokenspeed) | Inference engines that Dynamo serves | -| [NIXL](https://github.com/ai-dynamo/nixl) | High-throughput inference data-transfer library (KV-cache transfer over RDMA/NVLink) that underpins disaggregated serving | -| [AIPerf](https://github.com/ai-dynamo/aiperf) | Benchmarking and load-generation tool used by the benchmarking guides | -| [AISimulate](https://pypi.org/project/aisimulate/) | Predicts serving behavior and searches deployment configurations offline without requiring a GPU cluster | -| [ModelExpress](https://github.com/ai-dynamo/modelexpress) | Streams model weights GPU-to-GPU via NIXL for fast replica cold-start | -| [Grove](https://github.com/ai-dynamo/grove) | Kubernetes operator for topology-aware gang scheduling | +Dynamo serves [vLLM](https://github.com/vllm-project/vllm), +[SGLang](https://github.com/sgl-project/sglang), +[TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM), and +[TokenSpeed](https://github.com/lightseekorg/tokenspeed). Related projects: +[NIXL](https://github.com/ai-dynamo/nixl) (KV-cache transfer), +[AIPerf](https://github.com/ai-dynamo/aiperf) (load generation), +[AISimulate](https://pypi.org/project/aisimulate/) (offline configuration search), +[ModelExpress](https://github.com/ai-dynamo/modelexpress) (GPU-to-GPU weight streaming), and +[Grove](https://github.com/ai-dynamo/grove) (Kubernetes gang scheduling). ## Skills @@ -120,10 +117,9 @@ Install the system packages, Rust, and `uv` from [Building from Source](docs/fern/pages/developer-guide/advanced-customizations/building-from-source.md), plus: -- **protoc 3.15 or newer.** Ubuntu 22.04's `protobuf-compiler` is 3.12, and the `dynamo-llm` - build fails on `relay.proto`. CI installs protoc 30.2 from the - [protobuf releases](https://github.com/protocolbuffers/protobuf/releases). -- **Python 3.11 or newer.** `uv venv` finds or downloads one; Ubuntu 22.04's `python3` is 3.10. +- protoc 3.15 or newer (CI uses 30.2 from the + [protobuf releases](https://github.com/protocolbuffers/protobuf/releases)) +- Python 3.11 or newer (`uv venv` can download one) Python dev build (bindings + wheel, editable): @@ -154,10 +150,7 @@ container/run.sh --mount-workspace -it --image dynamo:latest-vllm-local-dev ### Rust: build and test efficiently -Most Rust builds wait on one serial chain of crates that ends in `dynamo-llm`. More -cores, sccache, or a remote cache barely help; the cost is set by which crates you build -and which test binaries you link. Measured on a 24-core workstation (October 2026, -rustc 1.96.1), for relative cost: +Times on a 24-core workstation (October 2026, rustc 1.96.1), for relative cost: | After you edit | Command | Time | |----------------|---------|------| @@ -200,9 +193,7 @@ cargo clippy --locked --no-deps -p dynamo-kv-router --all-targets -- -D warnings - CI's non-default feature checks are in [`pre-merge.yml`](.github/workflows/pre-merge.yml). Each is a separate cold build (about 4 min), so run only the one you touched. -On macOS, run `dynamo-llm` checks and targeted tests with `--no-default-features` unless the -target explicitly requires `block-manager`. The default feature enables Linux/CUDA-oriented NIXL, -NUMA, and `O_DIRECT` code that is not a valid general-purpose macOS validation path. +On macOS, build and test `dynamo-llm` with `--no-default-features` unless you need `block-manager`. ### Python tests @@ -216,28 +207,20 @@ python3 -m pytest tests/router/test_router_e2e_with_mockers.py -k "test_router_d # GPU end to end: the same, on the backend's serve tests. python3 -m pytest tests/serve/test_vllm.py -k guided_decoding -s - -# Before you push, list the pre-merge CPU tests in the area you changed. -python3 -m pytest --collect-only -q -m "pre_merge and gpu_0" tests/router ``` `dynamo.mocker` stands in for an engine, so router, planner, and frontend changes can be tested end to end without a GPU. Outside the dev containers, install the test plugins with `uv pip install -r container/deps/requirements.test.txt`, and put `etcd` and `nats-server` on `PATH`: integration fixtures start their own instances. Tests download their Hugging Face -model unless you pass `--models-dir`. Markers are strict (`--strict-markers`); the full marker -list lives in [`pyproject.toml`](pyproject.toml) `[tool.pytest.ini_options]`, including GPU -gating (`gpu_0` … `gpu_8`). Read [`tests/README.md`](tests/README.md), -[`.ai/pytest-guidelines.md`](.ai/pytest-guidelines.md), and -[`.ai/test-model-size-guardrails.md`](.ai/test-model-size-guardrails.md) before writing -tests. +model unless you pass `--models-dir`. Before writing tests, read +[`tests/README.md`](tests/README.md), [`.ai/pytest-guidelines.md`](.ai/pytest-guidelines.md), +and [`.ai/test-model-size-guardrails.md`](.ai/test-model-size-guardrails.md). ### Lint -```bash -pre-commit run --files # hooks on what you changed -pre-commit run --all-files # every hook; run `pre-commit install` first (it also installs the DCO commit-msg hook) -``` +Run `pre-commit install` once; the hooks, including the DCO sign-off check, then run on every +commit. To check before committing, run `pre-commit run --files `. ## Pull Requests