ai-dynamo · dmitry-tokarev-nv · Aug 21, 2026 · Aug 21, 2026 · Aug 21, 2026 · coderabbitai
@@ -14,6 +14,11 @@ metadata:
 
 # Analyze AIPerf Results
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
 Audit benchmark evidence before interpreting performance. Preserve raw files unchanged and do not claim an unmeasured
 server-side cause.
 

@@ -15,6 +15,11 @@ metadata:
 
 # Configure AIPerf Benchmark
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
 Create a reproducible benchmark that answers the current performance question without changing the deployed candidate.
 Freeze semantics only for runs used in the same direct comparison.
 

@@ -17,6 +17,11 @@ metadata:
 
 # Consult Performance Knowledge
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
 Turn the current audited performance finding into one documented configuration proposal. Write the reasoning record;
 do not edit a deployment manifest, deploy anything, or run AIPerf.
 

@@ -17,6 +17,11 @@ metadata:
 
 # Create Optimization Hypothesis
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
 Materialize an already-reasoned proposal. Treat `knowledge-consult.md` as a flexible reasoning record, not a rigid
 schema. Do not select a different lever, broaden the proposal, deploy, benchmark, or approve it.
 

@@ -13,6 +13,11 @@ metadata:
 
 # Deploy Dynamo Recipe
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
 ## Purpose
 
 Deploy exactly one assigned Dynamo Kubernetes DGD and return a small smoke-test artifact. This skill does not search
@@ -66,7 +71,10 @@ Create exactly one directory for the assigned candidate:
 
 Create `applied_manifests/` beneath it. Copy the assigned DGD and every explicitly handed-off support manifest used by
 the deployment into that directory with stable names such as `deploy.yaml`, `model-cache.yaml`,
-`model-download.yaml`, and `model-validate.yaml`. Never modify the handed-off source files.
+`model-download.yaml`, and `model-validate.yaml` — normalizing the filename at copy time. A recipe may ship
+variant-specific manifests (`recipes/deepseek-v4/*` ship `model-download-fp8.yaml` and `model-download-nvfp4.yaml`):
+select the one matching the assigned DGD's precision and copy it as `model-download.yaml`. Few recipes ship a
+validation job at all. Copy what the handoff actually contains. Never modify the handed-off source files.
 
 Update these run-scoped copies in place when a compatibility fix is required, then reapply them. Record every change
 and reason in `deployment_ledger.json`; do not retain numbered intermediate copies. After a successful smoke test,
@@ -157,7 +165,9 @@ If the effective cluster context differs from what `<EXP_ROOT>/manifest.yaml` re
 cluster-context entry before mutating anything.
 
 Read each support manifest's `kind` and `metadata.name`; never infer a Kubernetes resource name from its filename. Set
-`DOWNLOAD_JOB` and `VALIDATE_JOB` from the corresponding Job manifests.
+`DOWNLOAD_JOB` and `VALIDATE_JOB` from the corresponding Job manifests. The run-scoped copies are already normalized to
+the stable filenames above, so the applies below reference those names directly; skip a block when the recipe ships no
+such manifest.
 
 ```bash
 set -euo pipefail

@@ -17,6 +17,11 @@ metadata:
 
 # Perform Adversarial Review
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
 Try to falsify a proposed optimization experiment before it consumes GPU time. Review the proposal; do not generate a
 second one, edit its draft, deploy it, or run AIPerf.
 

@@ -14,6 +14,11 @@ metadata:
 
 # Run AIPerf Benchmark
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
 Execute the configured Job and preserve operational evidence. Do not interpret performance.
 
 Read `agent-docs/rules/execution/deployment.md`, `agent-docs/rules/execution/logging.md`,

@@ -16,6 +16,11 @@ metadata:
 
 # Synthesize User Workload
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
 Create the durable workload contract and user-provided baseline DGD that every later optimization role receives. Do
 not search for or select a recipe, deploy, benchmark, or propose tuning changes.
 

@@ -1,3 +1,6 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
 name: Agent-reported instruction gap
 description: For AI agents using the optimization skills — report instructions that misled, blocked, or contradicted live verification.
 title: "[AGENT]: "

@@ -21,9 +21,18 @@ retained beyond the deployment ledger.
 
 ```bash
 kubectl --context "${KUBE_CONTEXT}" get namespace "${NAMESPACE}"
-kubectl --context "${KUBE_CONTEXT}" get crd | grep -i dynamo || { echo "Dynamo CRDs missing"; exit 1; }
-kubectl --context "${KUBE_CONTEXT}" get storageclass
-kubectl --context "${KUBE_CONTEXT}" get nodes -o wide
+# CRD presence gate: a Forbidden here is tolerated because deploy-dynamo-recipe's server
+# dry-run re-checks it authoritatively; a confirmed absence stops before any mutation.
+crds="$(kubectl --context "${KUBE_CONTEXT}" get crd 2>&1 || true)"
+case "${crds}" in
+  *Forbidden*) echo "WARN: cluster-scope CRD list forbidden for this identity; deferring to server dry-run" ;;
+  *dynamographdeployment*) : ;;
+  *) echo "Dynamo CRDs missing"; exit 1 ;;
+esac
+# Advisory reads: storage classes and node inventory inform sizing but a namespace-scoped
+# identity may lack cluster-scope list rights. Record a Forbidden as a run limitation; do not fail.
+kubectl --context "${KUBE_CONTEXT}" get storageclass || echo "WARN: storageclass list forbidden; record as limitation"
+kubectl --context "${KUBE_CONTEXT}" get nodes -o wide || echo "WARN: node list forbidden; record as limitation"
 ```
 
 Check secrets only by name. Never print, decode, or persist secret values.
@@ -42,7 +51,10 @@ kubectl --context "${KUBE_CONTEXT}" get pvc -n "${NAMESPACE}"
 ```
 
 Run model download and validation jobs when present. Read each Job name from its manifest's `metadata.name`; never infer
-the resource name from the filename.
+the resource name from the filename. A recipe may ship variant-specific download manifests (for example
+`model-download-fp8.yaml` and `model-download-nvfp4.yaml`); select the one matching the assigned DGD and copy it into
+`applied_manifests/` as `model-download.yaml`, per `deploy-dynamo-recipe`. Skip a block when the recipe ships no such
+job.
 
 ```bash
 kubectl --context "${KUBE_CONTEXT}" apply -f "${DEPLOY_ROOT}/applied_manifests/model-download.yaml" -n "${NAMESPACE}"
@@ -59,7 +71,7 @@ Apply the selected DGD from its run-scoped copy:
 kubectl --context "${KUBE_CONTEXT}" apply -f "${DEPLOY_ROOT}/applied_manifests/deploy.yaml" -n "${NAMESPACE}"
 kubectl --context "${KUBE_CONTEXT}" get dynamographdeployment -n "${NAMESPACE}"
 kubectl --context "${KUBE_CONTEXT}" get pods -n "${NAMESPACE}" -o wide
-kubectl get svc -n "${NAMESPACE}"
+kubectl --context "${KUBE_CONTEXT}" get svc -n "${NAMESPACE}"
 ```
 
 ## Readiness Signals

@@ -87,13 +87,16 @@ guidance is in [`memory.md`](../model-sizing/memory.md) and [`parallelism.md`](.
 Exact API surfaces may differ by version.
 
 - **Chunked prefill** (`enable_chunked_prefill`) — splits long prefill into chunks so prefill and decode interleave.
-  Strongly recommended `true` for throughput workloads. recent vLLM versions enable it by default for standard generation; verify against the running version
-  models.
+  Strongly recommended `true` for throughput workloads. Recent vLLM versions enable it by default for standard
+  generation models; verify against the running version.
 - **`max_num_seqs`** — maximum sequences per iteration; primary throughput/latency knob, analogous to
   `max_batch_size` in TensorRT-LLM.
 - **`max_num_batched_tokens`** — maximum total tokens admitted per scheduling step, analogous to `max_num_tokens` in
-  TensorRT-LLM. Without chunked prefill, the largest prompt must fit within this budget. An approximate sizing relation
-  is `max_num_batched_tokens ≈ max_num_seqs × avg_seq_len`.
+  TensorRT-LLM. Without chunked prefill, the largest prompt must fit within this budget. This is a *per-step* budget,
+  not total KV capacity: in steady-state decode each running sequence contributes about one token per step, so the
+  budget only has to cover the decode batch plus one prefill chunk. Start in the low thousands — raise it to finish
+  prefill in fewer steps for throughput and TTFT, lower it to keep prefill from crowding decode for smoother
+  inter-token latency — and size total in-flight tokens with `max_num_seqs` and `max_model_len` instead.
 - **`max_model_len`** — per-request sequence cap. Fix it to the workload maximum rather than tuning it unless GPU memory
   is tight after weights; see [`memory.md`](../model-sizing/memory.md).
 - **`gpu_memory_utilization`** — fraction of total GPU memory for weights, KV cache, and buffers, with a default of 0.9;

@@ -33,6 +33,11 @@ rules:
 
 # Optimize Loop
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
 Use this workflow for an end-to-end Dynamo configuration optimization job. The user supplies the baseline DGD;
 `user-interviewer` captures it and hands it directly to `recipe-deployer`. There is no recipe-discovery or
 recipe-selection step in this workflow.

@@ -27,6 +27,11 @@ skills:
 
 # Hypothesis Challenger
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
 You are the independent adversarial reviewer between hypothesis generation and GPU spend. Assume the proposal may be
 wrong, redundant, or misleading until its evidence, diff, and risks survive review.
 

@@ -34,6 +34,11 @@ skills:
 
 # Hypothesis Generator
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
 You are the evidence-driven configuration hypothesis generator for the Dynamo optimization loop. You own the first
 proposal after `perf-analyzer` finishes, not its approval or execution.
 

@@ -32,6 +32,11 @@ skills:
 
 # Perf Analyzer
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
 You own the complete AIPerf lifecycle for one already-deployed candidate. The candidate must have a successful
 `<DEPLOY_ROOT>/smoke_test_artifact.json`, a complete deployment ledger, and durable config-engagement evidence before
 benchmarking begins. Define the performance question before selecting the benchmark.

@@ -22,6 +22,11 @@ skills:
 
 # Recipe Deployer
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
 You are the mechanical deployer for one assigned Dynamo Kubernetes DGD.
 
 Input ownership:

@@ -17,6 +17,11 @@ skills:
 
 # User Interviewer
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
 You are the first specialized role for every new Dynamo recipe optimization run. Receive the user's initial message
 before deployment, benchmarking, or hypothesis work begins.
 

@@ -19,14 +19,19 @@ Four configurations run Dynamo + TensorRT-LLM on 6x GB200 nodes (24 GPUs, MNNVL)
 <span><b>Model</b> nvidia/Kimi-K2.5-NVFP4</span>
 <span><b>GPUs</b> 24x GB200 (6 nodes, MNNVL)</span>
 <span><b>Runtime</b> TensorRT-LLM</span>
-<span><b>Workload</b> Mooncake-style agentic coding trace (~200K-token context, multi-turn), one-hour replay</span>
+<span><b>Workload</b> Mooncake-style agentic coding trace (~200K-token context, multi-turn), 30-minute replay</span>
 <span><b>Metrics</b> tok/s/user, tok/s/GPU, goodput at TTFT 5s / ITL 10ms</span>
 <span><b>Held constant</b> Model, runtime, GPU count, trace, duration, and goodput thresholds across all configurations</span>
 </div>
 </div>
 
 ## Results
 
+<Warning>
+The values below come from the earlier one-hour qualification. The current manifests cap each measured run at 30
+minutes, so these numbers require requalification before they can be treated as expected 30-minute results.
+</Warning>
+
 The disaggregated configuration with KV-aware routing, Eagle3 decoding, and KV offloading achieves the best system throughput and interactivity. Each row is that configuration's chosen operating point on the source Pareto plot — concurrency differs by row and the values are approximate plot readings, so read them as per-configuration operating points rather than an equal-load sweep:
 
 | Configuration | Concurrency | tok/s/user (avg) | tok/s/GPU |
@@ -81,7 +86,7 @@ aiperf profile -m nvidia/Kimi-K2.5-NVFP4 \
   --url http://<frontend>:8000 \
   --streaming --extra-inputs ignore_eos:true \
   --concurrency <8|24|32> --random-seed 42 \
-  --benchmark-duration 3600 --concurrency-ramp-duration 60 \
+  --benchmark-duration 1800 --concurrency-ramp-duration 60 \
   --goodput "time_to_first_token:5000 inter_token_latency:10"
 ```