diff --git a/.agents/skills/analyze-aiperf-results/SKILL.md b/.agents/skills/analyze-aiperf-results/SKILL.md index 026d76fe7aab..9c1222b26f67 100644 --- a/.agents/skills/analyze-aiperf-results/SKILL.md +++ b/.agents/skills/analyze-aiperf-results/SKILL.md @@ -14,6 +14,11 @@ metadata: # Analyze AIPerf Results + + Audit benchmark evidence before interpreting performance. Preserve raw files unchanged and do not claim an unmeasured server-side cause. diff --git a/.agents/skills/configure-aiperf-benchmark/SKILL.md b/.agents/skills/configure-aiperf-benchmark/SKILL.md index 7ef9fe5f3953..bbd2c3148adc 100644 --- a/.agents/skills/configure-aiperf-benchmark/SKILL.md +++ b/.agents/skills/configure-aiperf-benchmark/SKILL.md @@ -15,6 +15,11 @@ metadata: # Configure AIPerf Benchmark + + Create a reproducible benchmark that answers the current performance question without changing the deployed candidate. Freeze semantics only for runs used in the same direct comparison. diff --git a/.agents/skills/consult-perf-knowledge/SKILL.md b/.agents/skills/consult-perf-knowledge/SKILL.md index 5d723743e220..21db0efc8447 100644 --- a/.agents/skills/consult-perf-knowledge/SKILL.md +++ b/.agents/skills/consult-perf-knowledge/SKILL.md @@ -17,6 +17,11 @@ metadata: # Consult Performance Knowledge + + Turn the current audited performance finding into one documented configuration proposal. Write the reasoning record; do not edit a deployment manifest, deploy anything, or run AIPerf. diff --git a/.agents/skills/create-optimization-hypothesis/SKILL.md b/.agents/skills/create-optimization-hypothesis/SKILL.md index 07f3e587e765..389b13c1d3c8 100644 --- a/.agents/skills/create-optimization-hypothesis/SKILL.md +++ b/.agents/skills/create-optimization-hypothesis/SKILL.md @@ -17,6 +17,11 @@ metadata: # Create Optimization Hypothesis + + Materialize an already-reasoned proposal. Treat `knowledge-consult.md` as a flexible reasoning record, not a rigid schema. Do not select a different lever, broaden the proposal, deploy, benchmark, or approve it. diff --git a/.agents/skills/deploy-dynamo-recipe/SKILL.md b/.agents/skills/deploy-dynamo-recipe/SKILL.md index dd91720bc183..f6471bd3a0d6 100644 --- a/.agents/skills/deploy-dynamo-recipe/SKILL.md +++ b/.agents/skills/deploy-dynamo-recipe/SKILL.md @@ -13,6 +13,11 @@ metadata: # Deploy Dynamo Recipe + + ## Purpose Deploy exactly one assigned Dynamo Kubernetes DGD and return a small smoke-test artifact. This skill does not search @@ -66,7 +71,10 @@ Create exactly one directory for the assigned candidate: Create `applied_manifests/` beneath it. Copy the assigned DGD and every explicitly handed-off support manifest used by the deployment into that directory with stable names such as `deploy.yaml`, `model-cache.yaml`, -`model-download.yaml`, and `model-validate.yaml`. Never modify the handed-off source files. +`model-download.yaml`, and `model-validate.yaml` — normalizing the filename at copy time. A recipe may ship +variant-specific manifests (`recipes/deepseek-v4/*` ship `model-download-fp8.yaml` and `model-download-nvfp4.yaml`): +select the one matching the assigned DGD's precision and copy it as `model-download.yaml`. Few recipes ship a +validation job at all. Copy what the handoff actually contains. Never modify the handed-off source files. Update these run-scoped copies in place when a compatibility fix is required, then reapply them. Record every change and reason in `deployment_ledger.json`; do not retain numbered intermediate copies. After a successful smoke test, @@ -157,7 +165,9 @@ If the effective cluster context differs from what `/manifest.yaml` re cluster-context entry before mutating anything. Read each support manifest's `kind` and `metadata.name`; never infer a Kubernetes resource name from its filename. Set -`DOWNLOAD_JOB` and `VALIDATE_JOB` from the corresponding Job manifests. +`DOWNLOAD_JOB` and `VALIDATE_JOB` from the corresponding Job manifests. The run-scoped copies are already normalized to +the stable filenames above, so the applies below reference those names directly; skip a block when the recipe ships no +such manifest. ```bash set -euo pipefail diff --git a/.agents/skills/perform-adversarial-review/SKILL.md b/.agents/skills/perform-adversarial-review/SKILL.md index 7e6ccea449ad..c1bb94730d03 100644 --- a/.agents/skills/perform-adversarial-review/SKILL.md +++ b/.agents/skills/perform-adversarial-review/SKILL.md @@ -17,6 +17,11 @@ metadata: # Perform Adversarial Review + + Try to falsify a proposed optimization experiment before it consumes GPU time. Review the proposal; do not generate a second one, edit its draft, deploy it, or run AIPerf. diff --git a/.agents/skills/run-aiperf-benchmark/SKILL.md b/.agents/skills/run-aiperf-benchmark/SKILL.md index 5c135ecc5513..1409305bf86f 100644 --- a/.agents/skills/run-aiperf-benchmark/SKILL.md +++ b/.agents/skills/run-aiperf-benchmark/SKILL.md @@ -14,6 +14,11 @@ metadata: # Run AIPerf Benchmark + + Execute the configured Job and preserve operational evidence. Do not interpret performance. Read `agent-docs/rules/execution/deployment.md`, `agent-docs/rules/execution/logging.md`, diff --git a/.agents/skills/synthesize-user-workload/SKILL.md b/.agents/skills/synthesize-user-workload/SKILL.md index 416fcc0e0c42..78d5a5c5e46b 100644 --- a/.agents/skills/synthesize-user-workload/SKILL.md +++ b/.agents/skills/synthesize-user-workload/SKILL.md @@ -16,6 +16,11 @@ metadata: # Synthesize User Workload + + Create the durable workload contract and user-provided baseline DGD that every later optimization role receives. Do not search for or select a recipe, deploy, benchmark, or propose tuning changes. diff --git a/.github/ISSUE_TEMPLATE/agent-reported.yml b/.github/ISSUE_TEMPLATE/agent-reported.yml index 6ae857ce622a..0463d7120907 100644 --- a/.github/ISSUE_TEMPLATE/agent-reported.yml +++ b/.github/ISSUE_TEMPLATE/agent-reported.yml @@ -1,3 +1,6 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + name: Agent-reported instruction gap description: For AI agents using the optimization skills — report instructions that misled, blocked, or contradicted live verification. title: "[AGENT]: " diff --git a/agent-docs/guides/deployment/kubernetes-recipe-workflow.md b/agent-docs/guides/deployment/kubernetes-recipe-workflow.md index 80d5edaea938..36b138246978 100644 --- a/agent-docs/guides/deployment/kubernetes-recipe-workflow.md +++ b/agent-docs/guides/deployment/kubernetes-recipe-workflow.md @@ -21,9 +21,18 @@ retained beyond the deployment ledger. ```bash kubectl --context "${KUBE_CONTEXT}" get namespace "${NAMESPACE}" -kubectl --context "${KUBE_CONTEXT}" get crd | grep -i dynamo || { echo "Dynamo CRDs missing"; exit 1; } -kubectl --context "${KUBE_CONTEXT}" get storageclass -kubectl --context "${KUBE_CONTEXT}" get nodes -o wide +# CRD presence gate: a Forbidden here is tolerated because deploy-dynamo-recipe's server +# dry-run re-checks it authoritatively; a confirmed absence stops before any mutation. +crds="$(kubectl --context "${KUBE_CONTEXT}" get crd 2>&1 || true)" +case "${crds}" in + *Forbidden*) echo "WARN: cluster-scope CRD list forbidden for this identity; deferring to server dry-run" ;; + *dynamographdeployment*) : ;; + *) echo "Dynamo CRDs missing"; exit 1 ;; +esac +# Advisory reads: storage classes and node inventory inform sizing but a namespace-scoped +# identity may lack cluster-scope list rights. Record a Forbidden as a run limitation; do not fail. +kubectl --context "${KUBE_CONTEXT}" get storageclass || echo "WARN: storageclass list forbidden; record as limitation" +kubectl --context "${KUBE_CONTEXT}" get nodes -o wide || echo "WARN: node list forbidden; record as limitation" ``` Check secrets only by name. Never print, decode, or persist secret values. @@ -42,7 +51,10 @@ kubectl --context "${KUBE_CONTEXT}" get pvc -n "${NAMESPACE}" ``` Run model download and validation jobs when present. Read each Job name from its manifest's `metadata.name`; never infer -the resource name from the filename. +the resource name from the filename. A recipe may ship variant-specific download manifests (for example +`model-download-fp8.yaml` and `model-download-nvfp4.yaml`); select the one matching the assigned DGD and copy it into +`applied_manifests/` as `model-download.yaml`, per `deploy-dynamo-recipe`. Skip a block when the recipe ships no such +job. ```bash kubectl --context "${KUBE_CONTEXT}" apply -f "${DEPLOY_ROOT}/applied_manifests/model-download.yaml" -n "${NAMESPACE}" @@ -59,7 +71,7 @@ Apply the selected DGD from its run-scoped copy: kubectl --context "${KUBE_CONTEXT}" apply -f "${DEPLOY_ROOT}/applied_manifests/deploy.yaml" -n "${NAMESPACE}" kubectl --context "${KUBE_CONTEXT}" get dynamographdeployment -n "${NAMESPACE}" kubectl --context "${KUBE_CONTEXT}" get pods -n "${NAMESPACE}" -o wide -kubectl get svc -n "${NAMESPACE}" +kubectl --context "${KUBE_CONTEXT}" get svc -n "${NAMESPACE}" ``` ## Readiness Signals diff --git a/agent-docs/guides/knob-tuning/vllm.md b/agent-docs/guides/knob-tuning/vllm.md index 56f7c1be068a..a0b1dfe15751 100644 --- a/agent-docs/guides/knob-tuning/vllm.md +++ b/agent-docs/guides/knob-tuning/vllm.md @@ -87,13 +87,16 @@ guidance is in [`memory.md`](../model-sizing/memory.md) and [`parallelism.md`](. Exact API surfaces may differ by version. - **Chunked prefill** (`enable_chunked_prefill`) — splits long prefill into chunks so prefill and decode interleave. - Strongly recommended `true` for throughput workloads. recent vLLM versions enable it by default for standard generation; verify against the running version - models. + Strongly recommended `true` for throughput workloads. Recent vLLM versions enable it by default for standard + generation models; verify against the running version. - **`max_num_seqs`** — maximum sequences per iteration; primary throughput/latency knob, analogous to `max_batch_size` in TensorRT-LLM. - **`max_num_batched_tokens`** — maximum total tokens admitted per scheduling step, analogous to `max_num_tokens` in - TensorRT-LLM. Without chunked prefill, the largest prompt must fit within this budget. An approximate sizing relation - is `max_num_batched_tokens ≈ max_num_seqs × avg_seq_len`. + TensorRT-LLM. Without chunked prefill, the largest prompt must fit within this budget. This is a *per-step* budget, + not total KV capacity: in steady-state decode each running sequence contributes about one token per step, so the + budget only has to cover the decode batch plus one prefill chunk. Start in the low thousands — raise it to finish + prefill in fewer steps for throughput and TTFT, lower it to keep prefill from crowding decode for smoother + inter-token latency — and size total in-flight tokens with `max_num_seqs` and `max_model_len` instead. - **`max_model_len`** — per-request sequence cap. Fix it to the workload maximum rather than tuning it unless GPU memory is tight after weights; see [`memory.md`](../model-sizing/memory.md). - **`gpu_memory_utilization`** — fraction of total GPU memory for weights, KV cache, and buffers, with a default of 0.9; diff --git a/agent-docs/guides/optimization/optimize-loop.md b/agent-docs/guides/optimization/optimize-loop.md index 24c72c9a3ef1..3fbc2fc4b309 100644 --- a/agent-docs/guides/optimization/optimize-loop.md +++ b/agent-docs/guides/optimization/optimize-loop.md @@ -33,6 +33,11 @@ rules: # Optimize Loop + + Use this workflow for an end-to-end Dynamo configuration optimization job. The baseline DGD comes from the interview's baseline-source ladder (`agents/user-interviewer/AGENTS.md`): supplied by the user, or a recipe or authored draft the user explicitly confirmed. `user-interviewer` captures the confirmed baseline and hands it diff --git a/agents/hypothesis-challenger/AGENTS.md b/agents/hypothesis-challenger/AGENTS.md index 12a631cb2a9b..522c81791695 100644 --- a/agents/hypothesis-challenger/AGENTS.md +++ b/agents/hypothesis-challenger/AGENTS.md @@ -27,6 +27,11 @@ skills: # Hypothesis Challenger + + You are the independent adversarial reviewer between hypothesis generation and GPU spend. Assume the proposal may be wrong, redundant, or misleading until its evidence, diff, and risks survive review. diff --git a/agents/hypothesis-generator/AGENTS.md b/agents/hypothesis-generator/AGENTS.md index f38f89558063..e2b8b8375cfa 100644 --- a/agents/hypothesis-generator/AGENTS.md +++ b/agents/hypothesis-generator/AGENTS.md @@ -34,6 +34,11 @@ skills: # Hypothesis Generator + + You are the evidence-driven configuration hypothesis generator for the Dynamo optimization loop. You own the first proposal after `perf-analyzer` finishes, not its approval or execution. diff --git a/agents/perf-analyzer/AGENTS.md b/agents/perf-analyzer/AGENTS.md index fd75fff40cdb..6b95051835bd 100644 --- a/agents/perf-analyzer/AGENTS.md +++ b/agents/perf-analyzer/AGENTS.md @@ -32,6 +32,11 @@ skills: # Perf Analyzer + + You own the complete AIPerf lifecycle for one already-deployed candidate. The candidate must have a successful `/smoke_test_artifact.json`, a complete deployment ledger, and durable config-engagement evidence before benchmarking begins. Define the performance question before selecting the benchmark. diff --git a/agents/recipe-deployer/AGENTS.md b/agents/recipe-deployer/AGENTS.md index ec25c40cf915..b844ae971d7a 100644 --- a/agents/recipe-deployer/AGENTS.md +++ b/agents/recipe-deployer/AGENTS.md @@ -22,6 +22,11 @@ skills: # Recipe Deployer + + You are the mechanical deployer for one assigned Dynamo Kubernetes DGD. Input ownership: diff --git a/agents/user-interviewer/AGENTS.md b/agents/user-interviewer/AGENTS.md index d9bef7c4e4e6..d259058d9c3b 100644 --- a/agents/user-interviewer/AGENTS.md +++ b/agents/user-interviewer/AGENTS.md @@ -18,6 +18,11 @@ skills: # User Interviewer + + You are the first specialized role for every new Dynamo recipe optimization run. Receive the user's initial message before deployment, benchmarking, or hypothesis work begins. diff --git a/docs/fern/pages/recipes/feature-benchmarks/agentic-coding-throughput-stack-kimi-k2-5.mdx b/docs/fern/pages/recipes/feature-benchmarks/agentic-coding-throughput-stack-kimi-k2-5.mdx index 920aae58e7a9..1c7608614d58 100644 --- a/docs/fern/pages/recipes/feature-benchmarks/agentic-coding-throughput-stack-kimi-k2-5.mdx +++ b/docs/fern/pages/recipes/feature-benchmarks/agentic-coding-throughput-stack-kimi-k2-5.mdx @@ -19,7 +19,7 @@ Four configurations run Dynamo + TensorRT-LLM on 6x GB200 nodes (24 GPUs, MNNVL) Model nvidia/Kimi-K2.5-NVFP4 GPUs 24x GB200 (6 nodes, MNNVL) Runtime TensorRT-LLM -Workload Mooncake-style agentic coding trace (~200K-token context, multi-turn), one-hour replay +Workload Mooncake-style agentic coding trace (~200K-token context, multi-turn), 30-minute replay Metrics tok/s/user, tok/s/GPU, goodput at TTFT 5s / ITL 10ms Held constant Model, runtime, GPU count, trace, duration, and goodput thresholds across all configurations @@ -27,6 +27,11 @@ Four configurations run Dynamo + TensorRT-LLM on 6x GB200 nodes (24 GPUs, MNNVL) ## Results + +The values below come from the earlier one-hour qualification. The current manifests cap each measured run at 30 +minutes, so these numbers require requalification before they can be treated as expected 30-minute results. + + The disaggregated configuration with KV-aware routing, Eagle3 decoding, and KV offloading achieves the best system throughput and interactivity. Each row is that configuration's chosen operating point on the source Pareto plot — concurrency differs by row and the values are approximate plot readings, so read them as per-configuration operating points rather than an equal-load sweep: | Configuration | Concurrency | tok/s/user (avg) | tok/s/GPU | @@ -81,7 +86,7 @@ aiperf profile -m nvidia/Kimi-K2.5-NVFP4 \ --url http://:8000 \ --streaming --extra-inputs ignore_eos:true \ --concurrency <8|24|32> --random-seed 42 \ - --benchmark-duration 3600 --concurrency-ramp-duration 60 \ + --benchmark-duration 1800 --concurrency-ramp-duration 60 \ --goodput "time_to_first_token:5000 inter_token_latency:10" ```