dagil-nvidia · May 27, 2026 · May 20, 2026 · May 23, 2026 · May 23, 2026 · May 23, 2026
@@ -7,6 +7,11 @@ disable-model-invocation: true
 
 # Start Debug Session
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: CC-BY-4.0
+-->
+
 Create a structured debugging session for an issue in the Dynamo ecosystem.
 
 ## Step 1: Get the Bug Report

@@ -5,6 +5,11 @@ description: Create or update Dynamo Enhancement Proposals as GitHub issues, inc
 
 # Skill: Create a DEP as a GitHub Issue
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: CC-BY-4.0
+-->
+
 ## Purpose
 
 Create a new Dynamo Enhancement Proposal (DEP) as a GitHub Issue on

@@ -5,6 +5,11 @@ description: Check Dynamo Enhancement Proposal issue status, list DEPs by lifecy
 
 # Skill: Check DEP Status
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: CC-BY-4.0
+-->
+
 ## Purpose
 
 List DEP issues with their current status, area, PIC, and approval

@@ -5,6 +5,11 @@ description: Update Dynamo Enhancement Proposal lifecycle state in GitHub, inclu
 
 # Skill: Update DEP Lifecycle
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: CC-BY-4.0
+-->
+
 ## Purpose
 
 Update DEP status through its lifecycle — triage, review, approve,

@@ -5,6 +5,11 @@ description: Maintain Dynamo Fern docs site -- add, update, move, or remove page
 
 # Dynamo Docs Maintenance
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: CC-BY-4.0
+-->
+
 Unified skill for adding, updating, moving, and removing pages on the Dynamo Fern documentation site.
 
 ## Branch Rule

@@ -6,6 +6,11 @@ user-invocable: true
 
 # File a Dynamo Bug Issue
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: CC-BY-4.0
+-->
+
 Use the current conversation context to file a well-structured bug report against `ai-dynamo/dynamo` via the `gh` CLI.
 
 ## Instructions

@@ -3,6 +3,13 @@ name: "graham-code-review"
 description: "Use this agent when you need a code review in the style of Graham King from the ai-dynamo/dynamo project. This agent embodies Graham's coding standards, review patterns, and technical preferences learned from his PRs and code reviews. Particularly useful for reviewing Rust code, systems-level changes, and code touching the dynamo runtime, networking, or performance-critical paths."
 ---
 
+# Graham Code Review
+
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: CC-BY-4.0
+-->
+
 You are a senior systems engineer specializing in Rust, distributed systems, and performance-critical infrastructure code for the ai-dynamo/dynamo project.
 
 This skill is most appropriate for these areas. Be strict if the code touches these. Outside these areas, lean toward suggestions rather than blocking issues:
@@ -91,4 +98,3 @@ Unless explicitly told otherwise, review **only the recently written/modified co
 VERY IMPORTANT: Before finalizing findings, make one more focused pass over each changed hunk for all the review rules above, and for each of the sections above: comment hygiene, concurrency / async patterns, naming section, and the tests section.
 
 ALWAYS REPORT ALL FINDINGS.
-
@@ -7,6 +7,11 @@ disable-model-invocation: true
 
 # PR CI Monitor
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: CC-BY-4.0
+-->
+
 Perform a full health check on a Dynamo pull request. Takes a PR number as argument (e.g., `/dynamo:pr-monitor 6554`).
 
 ## Step 1: PR Overview

@@ -1,5 +1,10 @@
 # Tool Parser Generator Skill
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: CC-BY-4.0
+-->
+
 A Claude Code skill for adding tool calling support to dynamo by analyzing HuggingFace model chat templates.
 
 ## Overview

@@ -6,6 +6,11 @@ license: "Apache-2.0"
 
 # Tool Parser Generator Skill
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: CC-BY-4.0
+-->
+
 Add support for new models' tool calling formats by analyzing their chat templates and generating appropriate parser implementations for dynamo.
 
 ## When to Use This Skill

@@ -1,5 +1,10 @@
 # Parser Integration Guide
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: CC-BY-4.0
+-->
+
 Step-by-step guide for integrating new parsers or configurations into dynamo.
 
 ## Option 1: Add Configuration Preset (Most Common)

@@ -1,5 +1,10 @@
 # Tool Call Parser Patterns Reference
 
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: CC-BY-4.0
+-->
+
 Quick reference for common tool call patterns found in LLM chat templates.
 
 ## Pattern Categories

@@ -0,0 +1,86 @@
+---
+name: dynamo-interconnect-check
+description: Validate that a Dynamo deployment's NIXL/UCX/NCCL interconnect is ready for disaggregated serving over RDMA/NVLink. Use after recipe-runner brings a deployment up (especially disagg/multi-node) to confirm the KV transport is correct; use troubleshoot for diagnosing already-failed pods.
+---
+
+# Dynamo Interconnect Check
+
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: CC-BY-4.0
+-->
+
+## Goal
+
+Confirm that the transport disaggregated serving depends on actually works. A
+deployment can pass an endpoint smoke test while disagg is silently wrong: if
+NIXL/UCX cannot reach the peer worker over RDMA or NVLink, KV transfer falls
+back to a slow or broken path. Catch that with read-only checks before trusting
+a disagg deployment or its benchmark numbers.
+
+This skill is read-only. It never mutates the cluster and never prints secrets.
+
+## When To Use
+
+- After `dynamo-recipe-runner` deploys a **disagg** or multi-node recipe.
+- Before reporting disagg throughput/latency, so numbers reflect the real
+  transport.
+- When agg works but disagg is slow, hangs, or returns wrong output and you
+  suspect the fabric rather than the model.
+
+For diagnosing pods that are already crashing or unschedulable, use
+`dynamo-troubleshoot` first.
+
+## Workflow
+
+### 1. Check Transport Env Vars On The Recipe
+
+```bash
+python3 scripts/check_interconnect.py env recipes/<model>/<framework>/<mode>
+```
+
+Reports which NIXL/UCX/NCCL transport variables are set and flags
+disagg-critical ones (e.g. `UCX_TLS`, `UCX_NET_DEVICES`, `NCCL_IB_HCA`) that are
+absent. Missing here is only a warning — they may be baked into the image — so
+confirm with the node and NIXL checks. See
+`references/interconnect-env-vars.md` for what each variable does.
+
+### 2. Check Node Capabilities
+
+Locally on a GPU node, or inside a running worker pod:
+
+```bash
+python3 scripts/check_interconnect.py node \
+  --namespace "${NAMESPACE}" --pod <worker-pod>
+```
+
+Probes (read-only) for: InfiniBand devices and Active links, GPUDirect RDMA
+(`nvidia_peermem`), GDRCopy, and NVLink in the GPU topology. Missing tools are
+reported as `skipped`, not failures.
+
+### 3. Validate NIXL Reachability
+
+```bash
+python3 scripts/check_interconnect.py nixl \
+  --namespace "${NAMESPACE}" --pod <worker-pod>
+```
+
+Looks for NIXL test tooling in the pod and surfaces the exact next step to run a
+pairwise prefill↔decode transfer test. A full cross-pod transfer test requires
+two scheduled GPU pods on the fabric.
+
+## Output Contract
+
+Each check returns `ok` / `warn` / `fail` / `skipped` with a one-line detail,
+plus a rolled-up verdict on disagg transport readiness. Report:
+
+- transport env vars present vs. disagg-critical ones missing
+- RDMA / GPUDirect / NVLink capability status
+- whether NIXL reachability was validated, and the next command if not
+- a clear statement of whether disagg can be trusted, or what to fix first
+
+## References
+
+- `references/interconnect-env-vars.md` — NIXL/UCX/NCCL env var catalog and IB
+  capability checklist.
+- Use `scripts/check_interconnect.py` for all read-only checks.
@@ -0,0 +1,62 @@
+{
+  "skill": "dynamo-interconnect-check",
+  "cases": [
+    {
+      "id": "confirm-nixl-before-trusting-disagg",
+      "question": "I just brought up a disaggregated recipe. Confirm the NIXL/UCX transport is actually working before I trust the throughput numbers.",
+      "expected_skill": "dynamo-interconnect-check",
+      "expected_script": "scripts/check_interconnect.py",
+      "ground_truth": "Run read-only interconnect checks (transport env vars on the recipe, then runtime NIXL/UCX/NCCL reachability over RDMA/NVLink) and report whether KV transport is on the fast path.",
+      "expected_behavior": [
+        "Check transport env vars on the recipe",
+        "Run read-only runtime interconnect checks",
+        "Report whether NIXL/UCX can reach peer workers over RDMA/NVLink",
+        "Never mutate the cluster or print secrets"
+      ]
+    },
+    {
+      "id": "agg-ok-disagg-slow-suspect-fabric",
+      "question": "Aggregated serving works fine but disaggregated is slow and sometimes hangs — I suspect the fabric, not the model.",
+      "expected_skill": "dynamo-interconnect-check",
+      "expected_script": "scripts/check_interconnect.py",
+      "ground_truth": "Validate the disagg transport path; a deployment can pass an endpoint smoke test while KV transfer silently falls back to a slow/broken path.",
+      "expected_behavior": [
+        "Check transport configuration and runtime reachability",
+        "Determine if KV transfer is on a fallback path",
+        "Report fabric findings"
+      ]
+    },
+    {
+      "id": "rdma-nvlink-readiness-multinode",
+      "question": "Check RDMA/NVLink readiness for my multi-node disagg deployment.",
+      "expected_skill": "dynamo-interconnect-check",
+      "expected_script": "scripts/check_interconnect.py",
+      "ground_truth": "Run the interconnect readiness checks for a multi-node disagg deployment and report transport health.",
+      "expected_behavior": [
+        "Run interconnect readiness checks for multi-node disagg",
+        "Report RDMA/NVLink transport health"
+      ]
+    },
+    {
+      "id": "neg-deploy-disagg-recipe",
+      "question": "Deploy a disaggregated recipe for DeepSeek on my cluster.",
+      "expected_skill": "dynamo-recipe-runner",
+      "ground_truth": "Bringing a recipe up belongs to dynamo-recipe-runner; interconnect-check runs afterward.",
+      "expected_behavior": ["dynamo-interconnect-check stays silent; dynamo-recipe-runner handles it"]
+    },
+    {
+      "id": "neg-disagg-pods-crashing",
+      "question": "My disagg prefill pods are crashing and unschedulable.",
+      "expected_skill": "dynamo-troubleshoot",
+      "ground_truth": "Already-crashing/unschedulable pods are failure diagnosis, which belongs to dynamo-troubleshoot.",
+      "expected_behavior": ["dynamo-interconnect-check stays silent; dynamo-troubleshoot handles it"]
+    },
+    {
+      "id": "neg-switch-router-mode",
+      "question": "Change my router to least-loaded mode.",
+      "expected_skill": "dynamo-router-starter",
+      "ground_truth": "Router mode work belongs to dynamo-router-starter.",
+      "expected_behavior": ["dynamo-interconnect-check stays silent; dynamo-router-starter handles it"]
+    }
+  ]
+}
@@ -0,0 +1,67 @@
+# Dynamo Interconnect Env Vars & IB Capability Checklist
+
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: CC-BY-4.0
+-->
+
+Disaggregated serving moves KV cache between prefill and decode workers over
+NIXL (which uses UCX underneath), and tensor/expert-parallel workers exchange
+data over NCCL. Both ride the same fabric — InfiniBand/RoCE RDMA across nodes
+and NVLink within a node. If that fabric is misconfigured, a deployment still
+serves `/v1/models` but disagg is slow or wrong. The variables below decide
+which transport gets used.
+
+## NIXL / UCX (KV cache transport)
+
+| Variable | Disagg-critical | Purpose |
+|----------|:---:|---------|
+| `UCX_TLS` | yes | Allowed transports. Include `rc`/IB and `cuda_ipc`/`cuda_copy` for RDMA + NVLink. If it resolves to `tcp` only, KV transfer crawls. |
+| `UCX_NET_DEVICES` | yes | Pins UCX to a specific HCA/port (e.g. `mlx5_0:1`). Unset or wrong device falls back to the management NIC. |
+| `UCX_IB_GPU_DIRECT_RDMA` | yes | Enables GPUDirect RDMA so the NIC DMAs straight to/from GPU memory. |
+| `UCX_RNDV_SCHEME` | no | Rendezvous scheme for large messages (`put_zcopy` / `get_zcopy` / `auto`). |
+| `NIXL_PLUGIN_DIR` | no | Backend plugin search path; only for non-default install layouts. |
+
+## NCCL (collective transport)
+
+| Variable | Disagg-critical | Purpose |
+|----------|:---:|---------|
+| `NCCL_IB_HCA` | yes | IB HCAs NCCL may use (e.g. `mlx5_0,mlx5_1`). |
+| `NCCL_SOCKET_IFNAME` | yes | NIC for NCCL bootstrap/rendezvous (e.g. `eth0`, `ib0`). A wrong guess hangs init. |
+| `NCCL_IB_DISABLE` | yes | Must be `0`/unset to use InfiniBand; `1` forces sockets. |
+| `NCCL_NET_GDR_LEVEL` | no | GPUDirect RDMA aggressiveness (`SYS`/`PHB`/`PIX`). |
+| `NCCL_P2P_LEVEL` | no | NVLink/PCIe peer-to-peer level for intra-node collectives. |
+| `NCCL_IB_GID_INDEX` | no | GID index for RoCE/EFA fabrics; not needed on classic IB. |
+| `NCCL_DEBUG` | no | Set to `INFO` to print the transport NCCL actually selected. |
+
+## IB / GPUDirect / NVLink capability checklist
+
+The `node` check probes these read-only; here is what each tells you:
+
+- **`/dev/infiniband` + `ibv_devinfo -l`** — RDMA devices are exposed to the
+  pod. Empty means no RDMA (often a missing device plugin or `hostNetwork`/
+  resource request).
+- **`ibstat` → `State: Active` / `LinkUp`** — the IB/RoCE port is actually up.
+  `Down`/`Polling` means cabling, subnet manager, or fabric issues.
+- **`nvidia_peermem` (or legacy `nv_peer_mem`) loaded** — GPUDirect RDMA kernel
+  support; without it RDMA stages through host memory.
+- **`/dev/gdrdrv`** — GDRCopy present, used for low-latency small transfers.
+- **`nvidia-smi topo -m` showing `NV#`** — NVLink links between GPUs for
+  intra-node KV/collective traffic; `PIX`/`PXB` rows show GPU↔NIC affinity,
+  which should line up with `UCX_NET_DEVICES` / `NCCL_IB_HCA`.
+
+## Validating NIXL reachability
+
+Capabilities being present does not prove two workers can talk. To actually
+exercise the path, run a pairwise transfer between a prefill pod and a decode
+pod (different nodes for the RDMA path, same node for NVLink) using the NIXL
+test/bench tooling shipped in the worker image, e.g.:
+
+```bash
+# in the worker image; exact binary/flags depend on the NIXL build
+kubectl exec -n "${NAMESPACE}" <prefill-pod> -- nixlbench --help
+```
+
+If the transfer fails or silently uses TCP, fix the env vars and capabilities
+above before trusting any disagg result. Set `NCCL_DEBUG=INFO` and inspect UCX
+logs to confirm the selected transport.