chore(skills): add tags frontmatter to four bring-up skills

skills/dynamo-interconnect-check/SKILL.md

@@ -1,6 +1,15 @@
 ---
 name: dynamo-interconnect-check
 description: Validate that a Dynamo deployment's NIXL/UCX/NCCL interconnect is ready for disaggregated serving over RDMA/NVLink. Use after recipe-runner brings a deployment up (especially disagg/multi-node) to confirm the KV transport is correct; use troubleshoot for diagnosing already-failed pods.
+license: Apache-2.0
+metadata:
+  author: Dan Gil <dagil@nvidia.com>
+  tags:
+    - dynamo
+    - nixl
+    - rdma
+    - disagg
+    - validation
 ---
 
 # Dynamo Interconnect Check
@@ -10,7 +19,7 @@ SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All
 SPDX-License-Identifier: CC-BY-4.0
 -->
 
-## Goal
+## Purpose
 
 Confirm that the transport disaggregated serving depends on actually works. A
 deployment can pass an endpoint smoke test while disagg is silently wrong: if
@@ -20,6 +29,13 @@ a disagg deployment or its benchmark numbers.
 
 This skill is read-only. It never mutates the cluster and never prints secrets.
 
+## Prerequisites
+
+- Python 3.10+ on the operator machine.
+- `kubectl exec` access to a worker pod in the target Dynamo deployment.
+- Read access to the recipe directory (`recipes/<model>/<framework>/<mode>`).
+- For node-capability checks: tools like `ibstat`, `nvidia-smi`, `lsmod` available in the worker pod image (missing tools are reported as `skipped`, not failures).
+
 ## When To Use
 
 - After `dynamo-recipe-runner` deploys a **disagg** or multi-node recipe.
@@ -31,7 +47,7 @@ This skill is read-only. It never mutates the cluster and never prints secrets.
 For diagnosing pods that are already crashing or unschedulable, use
 `dynamo-troubleshoot` first.
 
-## Workflow
+## Instructions
 
 ### 1. Check Transport Env Vars On The Recipe
 
@@ -69,6 +85,44 @@ Looks for NIXL test tooling in the pod and surfaces the exact next step to run a
 pairwise prefill↔decode transfer test. A full cross-pod transfer test requires
 two scheduled GPU pods on the fabric.
 
+## Available Scripts
+
+| Script | Purpose | Arguments |
+|---|---|---|
+| `scripts/check_interconnect.py env` | Inspect NIXL/UCX/NCCL env vars on a recipe | positional recipe path |
+| `scripts/check_interconnect.py node` | Probe InfiniBand, GPUDirect RDMA, GDRCopy, NVLink on a node or pod | `--namespace`, `--pod` |
+| `scripts/check_interconnect.py nixl` | Surface NIXL transfer-test readiness for a pod | `--namespace`, `--pod` |
+
+Invoke via the agentskills.io `run_script()` protocol:
+
+```python
+run_script("scripts/check_interconnect.py", args=["env", "recipes/qwen3-coder-480b/sglang/disagg"])
+run_script("scripts/check_interconnect.py", args=["node", "--namespace", "dynamo-demo", "--pod", "qwen-worker-0"])
+```
+
+## Examples
+
+Verify a disagg recipe's transport env shape before deploy:
+
+```bash
+python3 scripts/check_interconnect.py env recipes/qwen3-coder-480b/sglang/disagg
+```
+
+After deploy, validate a worker pod's fabric:
+
+```bash
+python3 scripts/check_interconnect.py node \
+  --namespace dynamo-demo --pod qwen-worker-0
+python3 scripts/check_interconnect.py nixl \
+  --namespace dynamo-demo --pod qwen-worker-0
+```
+
+Equivalent through the agent protocol:
+
+```python
+run_script("scripts/check_interconnect.py", args=["nixl", "--namespace", "dynamo-demo", "--pod", "qwen-worker-0"])
+```
+
 ## Output Contract
 
 Each check returns `ok` / `warn` / `fail` / `skipped` with a one-line detail,
@@ -79,6 +133,22 @@ plus a rolled-up verdict on disagg transport readiness. Report:
 - whether NIXL reachability was validated, and the next command if not
 - a clear statement of whether disagg can be trusted, or what to fix first
 
+## Limitations
+
+- Read-only fabric probe; does not run a full pairwise NIXL transfer (requires two scheduled GPU pods and the in-pod NIXL test tools).
+- `skipped` results for missing tools (`ibstat`, `nvidia-smi`, `lsmod`) are inconclusive, not a pass.
+- Env-var check inspects the recipe text; values injected at runtime via initContainers or operator-applied envs are not detected.
+- Single-node agg deployments do not exercise the transport — this skill is for disagg / multi-node validation.
+
+## Troubleshooting
+
+| Symptom | Likely cause | Next step |
+|---|---|---|
+| `env` reports all critical vars missing | Vars baked into image or injected by operator | Run the `node` check inside the worker pod to verify actual env |
+| `node` reports no Active IB link | Fabric down or HCA not provisioned to the node | Contact cluster admin; verify `kubectl describe node` shows `nvidia.com/gpu` and IB labels |
+| `nvidia_peermem` missing | GPUDirect RDMA module not loaded | Ask cluster admin to load `nvidia-peermem`; without it, NIXL falls back to staged copies |
+| `nixl` finds no test tools | Worker image lacks NIXL test harness | Use a NIXL-enabled image or run the standalone transfer test from a debug pod |
+
 ## References
 
 - `references/interconnect-env-vars.md` — NIXL/UCX/NCCL env var catalog and IB

skills/dynamo-interconnect-check/scripts/check_interconnect.py

@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 
@@ -27,6 +28,12 @@
 from pathlib import Path
 from typing import Any
 
+# Tunables and conventional return codes (kept here to avoid magic numbers).
+DEFAULT_PROBE_TIMEOUT_SEC = 20
+# POSIX-conventional return codes used when the wrapper itself fails before
+# the probed binary can produce a real one.
+RETURNCODE_COMMAND_NOT_FOUND = 127  # binary not found in PATH or pod
+
 # Transport-relevant env vars, grouped by subsystem. ``disagg`` marks the ones
 # whose absence most often makes multi-node disaggregated serving fall back to a
 # slow or incorrect transport. Names are distinctive enough to match anywhere in
@@ -108,7 +115,7 @@ class Check:
     detail: str
 
 
-def run(cmd: list[str], timeout: int = 20) -> dict[str, Any]:
+def run(cmd: list[str], timeout: int = DEFAULT_PROBE_TIMEOUT_SEC) -> dict[str, Any]:
     """Run a command read-only, never raising on failure or a missing binary."""
     try:
         proc = subprocess.run(
@@ -216,7 +223,7 @@ def check_env(target: Path) -> list[Check]:
 def classify_node_probe(name: str, res: dict[str, Any]) -> Check:
     """Turn a raw probe result into a triaged Check."""
     out = (res["out"] or "").strip()
-    if res["rc"] == 127:
+    if res["rc"] == RETURNCODE_COMMAND_NOT_FOUND:
         return Check(name, "skipped", "tool/path not present in this environment")
     if res["rc"] != 0:
         return Check(name, "warn", (res["err"] or "non-zero exit").strip()[:200])
@@ -279,7 +286,7 @@ def check_nixl(
         ]
     )
     found = (probe["out"] or "").strip()
-    if probe["rc"] == 127 or not found:
+    if probe["rc"] == RETURNCODE_COMMAND_NOT_FOUND or not found:
         return [
             Check(
                 "nixl:binary",

skills/dynamo-interconnect-check/skill-card.md

@@ -0,0 +1,38 @@
+## Description: <br>
+Validate that a Dynamo deployment's NIXL/UCX/NCCL interconnect is ready for disaggregated serving over RDMA/NVLink. <br>
+
+This skill is ready for commercial/non-commercial use. <br>
+
+## Owner: NVIDIA <br>
+
+### License/Terms of Use: <br>
+Apache-2.0 <br>
+## Use Case: <br>
+Developers and infrastructure engineers use this skill to confirm that the NIXL/UCX/NCCL transport fabric is correctly configured for disaggregated serving before trusting benchmark numbers or production traffic. <br>
+
+### Deployment Geography for Use: <br>
+Global <br>
+
+## Known Risks and Mitigations: <br>
+Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. <br>
+Mitigation: Review and scan skill before deployment. <br>
+
+## Reference(s): <br>
+- [Interconnect Env Vars & IB Capability Checklist](references/interconnect-env-vars.md) <br>
+- [Dynamo GitHub Repository](https://github.com/ai-dynamo/dynamo) <br>
+
+
+## Skill Output: <br>
+**Output Type(s):** [Shell commands, Analysis] <br>
+**Output Format:** [Structured JSON with ok/warn/fail/skipped verdicts] <br>
+**Output Parameters:** [1D] <br>
+**Other Properties Related to Output:** [None] <br>
+
+## Skill Version(s): <br>
+1.2.0 (source: pyproject.toml) <br>
+
+## Ethical Considerations: <br>
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>
+
+(For Release on NVIDIA Platforms Only) <br>
+Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail). <br>

skills/dynamo-interconnect-check/skill.oms.sig

@@ -0,0 +1 @@
+{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAiZHluYW1vLWludGVyY29ubmVjdC1jaGVjayIsCiAgICAgICJkaWdlc3QiOiB7CiAgICAgICAgInNoYTI1NiI6ICJiYTkyYTc1OTQ4ZWRhZGIwMmE0YTcyOGYwYmZiOWZiMmUzZWNmNzA3OTAwZTE4Mjk2YmQ2Mzg1ODg0NjY4ZjM0IgogICAgICB9CiAgICB9CiAgXSwKICAicHJlZGljYXRlVHlwZSI6ICJodHRwczovL21vZGVsX3NpZ25pbmcvc2lnbmF0dXJlL3YxLjAiLAogICJwcmVkaWNhdGUiOiB7CiAgICAic2VyaWFsaXphdGlvbiI6IHsKICAgICAgImhhc2hfdHlwZSI6ICJzaGEyNTYiLAogICAgICAiYWxsb3dfc3ltbGlua3MiOiBmYWxzZSwKICAgICAgImlnbm9yZV9wYXRocyI6IFsKICAgICAgICAiLmdpdGh1YiIsCiAgICAgICAgIi5naXRpZ25vcmUiLAogICAgICAgICIuZ2l0YXR0cmlidXRlcyIsCiAgICAgICAgIi5naXQiCiAgICAgIF0sCiAgICAgICJtZXRob2QiOiAiZmlsZXMiCiAgICB9LAogICAgInJlc291cmNlcyI6IFsKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAibmFtZSI6ICJTS0lMTC5tZCIsCiAgICAgICAgImRpZ2VzdCI6ICIzYjAzNTg2NzUyMGE4ZDRlZmZhMDYyYTNhN2MyOTEzNjlkZWM0Mjk5Y2JmYmZhODg4MDE2NTVlOTMyNTk3OWIzIgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogImV2YWxzL2V2YWxzLmpzb24iLAogICAgICAgICJkaWdlc3QiOiAiZjVhYTA4MDYxNTAyZjNiMDgwMmYzNjRkMWVjOWQ0NTEyNTMyZmVmYzZlN2MwMDYwMDE4OGQ2NjgzMzlmYjljMCIKICAgICAgfSwKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAibmFtZSI6ICJyZWZlcmVuY2VzL2ludGVyY29ubmVjdC1lbnYtdmFycy5tZCIsCiAgICAgICAgImRpZ2VzdCI6ICIxMjAzMjQzMjE2ODJiODJjMWI4NTA2NjY2OTBkN2ZlYjRhNDc1YzZkOGExYWE1MTk2YWY2MmRlNzU3YjFmY2U0IgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogInNjcmlwdHMvY2hlY2tfaW50ZXJjb25uZWN0LnB5IiwKICAgICAgICAiZGlnZXN0IjogIjEyNjlhOWIzNzNjOTIyMDcyNjhhZDNlNTg1NWYwNWVlZGU3ODA0MGUwY2Y1MDhkZDQ5NjFlZWUyZTJiMWM4MWQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAic2tpbGwtY2FyZC5tZCIsCiAgICAgICAgImRpZ2VzdCI6ICJjZDkxMGQ3ZjFhOGY1Zjg5YmIzOTQwMDc2MTQzNjlhMWIwOTgzZjhiMmFkZjI4MzcxNGQwZDViNDE0NWI1YThmIgogICAgICB9CiAgICBdCiAgfQp9","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGYCMQCR9QQZPoMX4x/ITIjCFfhhcJJcS1qdhzYEAxStkKDZCEZWbFDo2QwSwq3Nvh8S+fcCMQDdNDM0a4haFv7XpwflvJbpGUD++YIMqmEkjHNu+kpAxghm2pezztpSOMsHnb++BxU=","keyid":""}]}}

skills/dynamo-recipe-runner/SKILL.md

@@ -1,6 +1,18 @@
 ---
 name: dynamo-recipe-runner
 description: Select, validate, patch, and deploy existing NVIDIA Dynamo Kubernetes recipes. Use for model/backend/GPU/deployment-mode recipe bring-up; use router-starter for router-only mode work and troubleshoot for broken deployments.
+license: Apache-2.0
+metadata:
+  author: Dan Gil <dagil@nvidia.com>
+  tags:
+    - dynamo
+    - kubernetes
+    - recipes
+    - bring-up
+  permissions:
+    - file_read
+    - network
+    - kubectl_exec
 ---
 
 # Dynamo Recipe Runner
@@ -10,13 +22,22 @@ SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All
 SPDX-License-Identifier: CC-BY-4.0
 -->
 
-## Goal
+## Purpose
 
 Get from user intent to a working Dynamo recipe endpoint with minimal back and
 forth. Do not create new guide content. Operate on the existing `recipes/`
 tree, patch the smallest necessary set of manifests, deploy when the user has
 cluster access, and prove success with an OpenAI-compatible smoke request.
 
+## Prerequisites
+
+- Python 3.10+ on the operator machine.
+- `kubectl` configured with a working cluster context.
+- Cluster has a default storage class for model-cache PVCs.
+- Hugging Face token stored in a Kubernetes secret named `hf-token-secret`
+  (or equivalent) in the target namespace.
+- Read access to the `recipes/` tree in the ai-dynamo/dynamo repository.
+
 ## Required Inputs
 
 Collect or infer these before changing manifests:
@@ -31,7 +52,7 @@ Collect or infer these before changing manifests:
 If a required value is missing and cannot be inferred from the selected recipe,
 ask for only that value.
 
-## Workflow
+## Instructions
 
 ### 1. Preflight
 
@@ -120,6 +141,40 @@ If `dynamo-router-starter` is also installed, prefer its `scripts/check_router_h
 for the full OpenAI-compatible smoke test. If this fails, switch to
 `dynamo-troubleshoot`.
 
+## Available Scripts
+
+| Script | Purpose | Arguments |
+|---|---|---|
+| `scripts/recipe_tool.py list` | Enumerate available recipes, optionally filtered | `--query`, `--framework`, `--mode`, `--format` |
+| `scripts/recipe_tool.py validate` | Validate a recipe directory before apply | positional recipe path |
+
+Invoke via the agentskills.io `run_script()` protocol:
+
+```python
+run_script("scripts/recipe_tool.py", args=["list", "--framework", "sglang", "--format", "table"])
+run_script("scripts/recipe_tool.py", args=["validate", "recipes/nemotron-3-super-fp8/sglang/agg"])
+```
+
+## Examples
+
+List sglang recipes that fit a single 8xB200 node:
+
+```bash
+python3 scripts/recipe_tool.py list --framework sglang --format table
+```
+
+Validate a specific recipe and resolve blockers before applying:
+
+```bash
+python3 scripts/recipe_tool.py validate recipes/nemotron-3-super-fp8/sglang/agg
+```
+
+Equivalent through the agent protocol:
+
+```python
+run_script("scripts/recipe_tool.py", args=["validate", "recipes/nemotron-3-super-fp8/sglang/agg"])
+```
+
 ## Output Contract
 
 Return:
@@ -131,6 +186,23 @@ Return:
 - unresolved blockers, if any
 - next troubleshooting step when deployment does not become healthy
 
+## Limitations
+
+- Operates on the existing `recipes/` tree only. Does not author new manifests.
+- Cluster-mutating apply steps require `kubectl` permission to the target namespace.
+- Smoke-test depth is intentionally minimal; for full router/endpoint coverage use `dynamo-router-starter`.
+- Multi-node disagg transport correctness is out of scope; use `dynamo-interconnect-check` after deploy.
+
+## Troubleshooting
+
+| Symptom | Likely cause | Next step |
+|---|---|---|
+| `kubectl` cluster unreachable | Context not set or VPN down | Return exact commands instead of running them; resume when cluster is reachable |
+| `validate` reports missing storage class | Cluster has no default `StorageClass` | Patch `storageClassName` on the model-cache manifest before applying |
+| Model-cache job stuck `Pending` | PVC unbound or HF secret missing | Inspect PVC events; create or rename the HF secret to match the recipe |
+| Worker pods `ImagePullBackOff` | Stale image tag or missing pull secret | Patch the image tag; verify image pull secret in the namespace |
+| `/v1/models` 4xx/5xx after deploy | Frontend not ready or wrong service port | Wait for pods Ready; re-run port-forward; switch to `dynamo-troubleshoot` if it persists |
+
 ## References
 
 - Read `references/k8s-recipe-workflow.md` for command templates and readiness checks.

skills/dynamo-recipe-runner/scripts/recipe_tool.py

@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0