diff --git a/skills/dynamo-interconnect-check/SKILL.md b/skills/dynamo-interconnect-check/SKILL.md index 031fcca6dbe6..8ee38293b75a 100644 --- a/skills/dynamo-interconnect-check/SKILL.md +++ b/skills/dynamo-interconnect-check/SKILL.md @@ -1,6 +1,15 @@ --- name: dynamo-interconnect-check description: Validate that a Dynamo deployment's NIXL/UCX/NCCL interconnect is ready for disaggregated serving over RDMA/NVLink. Use after recipe-runner brings a deployment up (especially disagg/multi-node) to confirm the KV transport is correct; use troubleshoot for diagnosing already-failed pods. +license: Apache-2.0 +metadata: + author: Dan Gil + tags: + - dynamo + - nixl + - rdma + - disagg + - validation --- # Dynamo Interconnect Check @@ -10,7 +19,7 @@ SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All SPDX-License-Identifier: CC-BY-4.0 --> -## Goal +## Purpose Confirm that the transport disaggregated serving depends on actually works. A deployment can pass an endpoint smoke test while disagg is silently wrong: if @@ -20,6 +29,13 @@ a disagg deployment or its benchmark numbers. This skill is read-only. It never mutates the cluster and never prints secrets. +## Prerequisites + +- Python 3.10+ on the operator machine. +- `kubectl exec` access to a worker pod in the target Dynamo deployment. +- Read access to the recipe directory (`recipes///`). +- For node-capability checks: tools like `ibstat`, `nvidia-smi`, `lsmod` available in the worker pod image (missing tools are reported as `skipped`, not failures). + ## When To Use - After `dynamo-recipe-runner` deploys a **disagg** or multi-node recipe. @@ -31,7 +47,7 @@ This skill is read-only. It never mutates the cluster and never prints secrets. For diagnosing pods that are already crashing or unschedulable, use `dynamo-troubleshoot` first. -## Workflow +## Instructions ### 1. Check Transport Env Vars On The Recipe @@ -69,6 +85,44 @@ Looks for NIXL test tooling in the pod and surfaces the exact next step to run a pairwise prefill↔decode transfer test. A full cross-pod transfer test requires two scheduled GPU pods on the fabric. +## Available Scripts + +| Script | Purpose | Arguments | +|---|---|---| +| `scripts/check_interconnect.py env` | Inspect NIXL/UCX/NCCL env vars on a recipe | positional recipe path | +| `scripts/check_interconnect.py node` | Probe InfiniBand, GPUDirect RDMA, GDRCopy, NVLink on a node or pod | `--namespace`, `--pod` | +| `scripts/check_interconnect.py nixl` | Surface NIXL transfer-test readiness for a pod | `--namespace`, `--pod` | + +Invoke via the agentskills.io `run_script()` protocol: + +```python +run_script("scripts/check_interconnect.py", args=["env", "recipes/qwen3-coder-480b/sglang/disagg"]) +run_script("scripts/check_interconnect.py", args=["node", "--namespace", "dynamo-demo", "--pod", "qwen-worker-0"]) +``` + +## Examples + +Verify a disagg recipe's transport env shape before deploy: + +```bash +python3 scripts/check_interconnect.py env recipes/qwen3-coder-480b/sglang/disagg +``` + +After deploy, validate a worker pod's fabric: + +```bash +python3 scripts/check_interconnect.py node \ + --namespace dynamo-demo --pod qwen-worker-0 +python3 scripts/check_interconnect.py nixl \ + --namespace dynamo-demo --pod qwen-worker-0 +``` + +Equivalent through the agent protocol: + +```python +run_script("scripts/check_interconnect.py", args=["nixl", "--namespace", "dynamo-demo", "--pod", "qwen-worker-0"]) +``` + ## Output Contract Each check returns `ok` / `warn` / `fail` / `skipped` with a one-line detail, @@ -79,6 +133,22 @@ plus a rolled-up verdict on disagg transport readiness. Report: - whether NIXL reachability was validated, and the next command if not - a clear statement of whether disagg can be trusted, or what to fix first +## Limitations + +- Read-only fabric probe; does not run a full pairwise NIXL transfer (requires two scheduled GPU pods and the in-pod NIXL test tools). +- `skipped` results for missing tools (`ibstat`, `nvidia-smi`, `lsmod`) are inconclusive, not a pass. +- Env-var check inspects the recipe text; values injected at runtime via initContainers or operator-applied envs are not detected. +- Single-node agg deployments do not exercise the transport — this skill is for disagg / multi-node validation. + +## Troubleshooting + +| Symptom | Likely cause | Next step | +|---|---|---| +| `env` reports all critical vars missing | Vars baked into image or injected by operator | Run the `node` check inside the worker pod to verify actual env | +| `node` reports no Active IB link | Fabric down or HCA not provisioned to the node | Contact cluster admin; verify `kubectl describe node` shows `nvidia.com/gpu` and IB labels | +| `nvidia_peermem` missing | GPUDirect RDMA module not loaded | Ask cluster admin to load `nvidia-peermem`; without it, NIXL falls back to staged copies | +| `nixl` finds no test tools | Worker image lacks NIXL test harness | Use a NIXL-enabled image or run the standalone transfer test from a debug pod | + ## References - `references/interconnect-env-vars.md` — NIXL/UCX/NCCL env var catalog and IB diff --git a/skills/dynamo-interconnect-check/scripts/check_interconnect.py b/skills/dynamo-interconnect-check/scripts/check_interconnect.py old mode 100644 new mode 100755 index 95eea04226a5..7bfc79e2dcd5 --- a/skills/dynamo-interconnect-check/scripts/check_interconnect.py +++ b/skills/dynamo-interconnect-check/scripts/check_interconnect.py @@ -1,3 +1,4 @@ +#!/usr/bin/env python3 # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 @@ -27,6 +28,12 @@ from pathlib import Path from typing import Any +# Tunables and conventional return codes (kept here to avoid magic numbers). +DEFAULT_PROBE_TIMEOUT_SEC = 20 +# POSIX-conventional return codes used when the wrapper itself fails before +# the probed binary can produce a real one. +RETURNCODE_COMMAND_NOT_FOUND = 127 # binary not found in PATH or pod + # Transport-relevant env vars, grouped by subsystem. ``disagg`` marks the ones # whose absence most often makes multi-node disaggregated serving fall back to a # slow or incorrect transport. Names are distinctive enough to match anywhere in @@ -108,7 +115,7 @@ class Check: detail: str -def run(cmd: list[str], timeout: int = 20) -> dict[str, Any]: +def run(cmd: list[str], timeout: int = DEFAULT_PROBE_TIMEOUT_SEC) -> dict[str, Any]: """Run a command read-only, never raising on failure or a missing binary.""" try: proc = subprocess.run( @@ -216,7 +223,7 @@ def check_env(target: Path) -> list[Check]: def classify_node_probe(name: str, res: dict[str, Any]) -> Check: """Turn a raw probe result into a triaged Check.""" out = (res["out"] or "").strip() - if res["rc"] == 127: + if res["rc"] == RETURNCODE_COMMAND_NOT_FOUND: return Check(name, "skipped", "tool/path not present in this environment") if res["rc"] != 0: return Check(name, "warn", (res["err"] or "non-zero exit").strip()[:200]) @@ -279,7 +286,7 @@ def check_nixl( ] ) found = (probe["out"] or "").strip() - if probe["rc"] == 127 or not found: + if probe["rc"] == RETURNCODE_COMMAND_NOT_FOUND or not found: return [ Check( "nixl:binary", diff --git a/skills/dynamo-interconnect-check/skill-card.md b/skills/dynamo-interconnect-check/skill-card.md new file mode 100644 index 000000000000..9d680cc42915 --- /dev/null +++ b/skills/dynamo-interconnect-check/skill-card.md @@ -0,0 +1,38 @@ +## Description:
+Validate that a Dynamo deployment's NIXL/UCX/NCCL interconnect is ready for disaggregated serving over RDMA/NVLink.
+ +This skill is ready for commercial/non-commercial use.
+ +## Owner: NVIDIA
+ +### License/Terms of Use:
+Apache-2.0
+## Use Case:
+Developers and infrastructure engineers use this skill to confirm that the NIXL/UCX/NCCL transport fabric is correctly configured for disaggregated serving before trusting benchmark numbers or production traffic.
+ +### Deployment Geography for Use:
+Global
+ +## Known Risks and Mitigations:
+Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills.
+Mitigation: Review and scan skill before deployment.
+ +## Reference(s):
+- [Interconnect Env Vars & IB Capability Checklist](references/interconnect-env-vars.md)
+- [Dynamo GitHub Repository](https://github.com/ai-dynamo/dynamo)
+ + +## Skill Output:
+**Output Type(s):** [Shell commands, Analysis]
+**Output Format:** [Structured JSON with ok/warn/fail/skipped verdicts]
+**Output Parameters:** [1D]
+**Other Properties Related to Output:** [None]
+ +## Skill Version(s):
+1.2.0 (source: pyproject.toml)
+ +## Ethical Considerations:
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+ +(For Release on NVIDIA Platforms Only)
+Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail).
diff --git a/skills/dynamo-interconnect-check/skill.oms.sig b/skills/dynamo-interconnect-check/skill.oms.sig new file mode 100644 index 000000000000..f4008ecbfae6 --- /dev/null +++ b/skills/dynamo-interconnect-check/skill.oms.sig @@ -0,0 +1 @@ +{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAiZHluYW1vLWludGVyY29ubmVjdC1jaGVjayIsCiAgICAgICJkaWdlc3QiOiB7CiAgICAgICAgInNoYTI1NiI6ICJiYTkyYTc1OTQ4ZWRhZGIwMmE0YTcyOGYwYmZiOWZiMmUzZWNmNzA3OTAwZTE4Mjk2YmQ2Mzg1ODg0NjY4ZjM0IgogICAgICB9CiAgICB9CiAgXSwKICAicHJlZGljYXRlVHlwZSI6ICJodHRwczovL21vZGVsX3NpZ25pbmcvc2lnbmF0dXJlL3YxLjAiLAogICJwcmVkaWNhdGUiOiB7CiAgICAic2VyaWFsaXphdGlvbiI6IHsKICAgICAgImhhc2hfdHlwZSI6ICJzaGEyNTYiLAogICAgICAiYWxsb3dfc3ltbGlua3MiOiBmYWxzZSwKICAgICAgImlnbm9yZV9wYXRocyI6IFsKICAgICAgICAiLmdpdGh1YiIsCiAgICAgICAgIi5naXRpZ25vcmUiLAogICAgICAgICIuZ2l0YXR0cmlidXRlcyIsCiAgICAgICAgIi5naXQiCiAgICAgIF0sCiAgICAgICJtZXRob2QiOiAiZmlsZXMiCiAgICB9LAogICAgInJlc291cmNlcyI6IFsKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAibmFtZSI6ICJTS0lMTC5tZCIsCiAgICAgICAgImRpZ2VzdCI6ICIzYjAzNTg2NzUyMGE4ZDRlZmZhMDYyYTNhN2MyOTEzNjlkZWM0Mjk5Y2JmYmZhODg4MDE2NTVlOTMyNTk3OWIzIgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogImV2YWxzL2V2YWxzLmpzb24iLAogICAgICAgICJkaWdlc3QiOiAiZjVhYTA4MDYxNTAyZjNiMDgwMmYzNjRkMWVjOWQ0NTEyNTMyZmVmYzZlN2MwMDYwMDE4OGQ2NjgzMzlmYjljMCIKICAgICAgfSwKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAibmFtZSI6ICJyZWZlcmVuY2VzL2ludGVyY29ubmVjdC1lbnYtdmFycy5tZCIsCiAgICAgICAgImRpZ2VzdCI6ICIxMjAzMjQzMjE2ODJiODJjMWI4NTA2NjY2OTBkN2ZlYjRhNDc1YzZkOGExYWE1MTk2YWY2MmRlNzU3YjFmY2U0IgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogInNjcmlwdHMvY2hlY2tfaW50ZXJjb25uZWN0LnB5IiwKICAgICAgICAiZGlnZXN0IjogIjEyNjlhOWIzNzNjOTIyMDcyNjhhZDNlNTg1NWYwNWVlZGU3ODA0MGUwY2Y1MDhkZDQ5NjFlZWUyZTJiMWM4MWQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAic2tpbGwtY2FyZC5tZCIsCiAgICAgICAgImRpZ2VzdCI6ICJjZDkxMGQ3ZjFhOGY1Zjg5YmIzOTQwMDc2MTQzNjlhMWIwOTgzZjhiMmFkZjI4MzcxNGQwZDViNDE0NWI1YThmIgogICAgICB9CiAgICBdCiAgfQp9","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGYCMQCR9QQZPoMX4x/ITIjCFfhhcJJcS1qdhzYEAxStkKDZCEZWbFDo2QwSwq3Nvh8S+fcCMQDdNDM0a4haFv7XpwflvJbpGUD++YIMqmEkjHNu+kpAxghm2pezztpSOMsHnb++BxU=","keyid":""}]}} \ No newline at end of file diff --git a/skills/dynamo-recipe-runner/SKILL.md b/skills/dynamo-recipe-runner/SKILL.md index e2ad040b3e75..9c3e35430a8f 100644 --- a/skills/dynamo-recipe-runner/SKILL.md +++ b/skills/dynamo-recipe-runner/SKILL.md @@ -1,6 +1,18 @@ --- name: dynamo-recipe-runner description: Select, validate, patch, and deploy existing NVIDIA Dynamo Kubernetes recipes. Use for model/backend/GPU/deployment-mode recipe bring-up; use router-starter for router-only mode work and troubleshoot for broken deployments. +license: Apache-2.0 +metadata: + author: Dan Gil + tags: + - dynamo + - kubernetes + - recipes + - bring-up + permissions: + - file_read + - network + - kubectl_exec --- # Dynamo Recipe Runner @@ -10,13 +22,22 @@ SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All SPDX-License-Identifier: CC-BY-4.0 --> -## Goal +## Purpose Get from user intent to a working Dynamo recipe endpoint with minimal back and forth. Do not create new guide content. Operate on the existing `recipes/` tree, patch the smallest necessary set of manifests, deploy when the user has cluster access, and prove success with an OpenAI-compatible smoke request. +## Prerequisites + +- Python 3.10+ on the operator machine. +- `kubectl` configured with a working cluster context. +- Cluster has a default storage class for model-cache PVCs. +- Hugging Face token stored in a Kubernetes secret named `hf-token-secret` + (or equivalent) in the target namespace. +- Read access to the `recipes/` tree in the ai-dynamo/dynamo repository. + ## Required Inputs Collect or infer these before changing manifests: @@ -31,7 +52,7 @@ Collect or infer these before changing manifests: If a required value is missing and cannot be inferred from the selected recipe, ask for only that value. -## Workflow +## Instructions ### 1. Preflight @@ -120,6 +141,40 @@ If `dynamo-router-starter` is also installed, prefer its `scripts/check_router_h for the full OpenAI-compatible smoke test. If this fails, switch to `dynamo-troubleshoot`. +## Available Scripts + +| Script | Purpose | Arguments | +|---|---|---| +| `scripts/recipe_tool.py list` | Enumerate available recipes, optionally filtered | `--query`, `--framework`, `--mode`, `--format` | +| `scripts/recipe_tool.py validate` | Validate a recipe directory before apply | positional recipe path | + +Invoke via the agentskills.io `run_script()` protocol: + +```python +run_script("scripts/recipe_tool.py", args=["list", "--framework", "sglang", "--format", "table"]) +run_script("scripts/recipe_tool.py", args=["validate", "recipes/nemotron-3-super-fp8/sglang/agg"]) +``` + +## Examples + +List sglang recipes that fit a single 8xB200 node: + +```bash +python3 scripts/recipe_tool.py list --framework sglang --format table +``` + +Validate a specific recipe and resolve blockers before applying: + +```bash +python3 scripts/recipe_tool.py validate recipes/nemotron-3-super-fp8/sglang/agg +``` + +Equivalent through the agent protocol: + +```python +run_script("scripts/recipe_tool.py", args=["validate", "recipes/nemotron-3-super-fp8/sglang/agg"]) +``` + ## Output Contract Return: @@ -131,6 +186,23 @@ Return: - unresolved blockers, if any - next troubleshooting step when deployment does not become healthy +## Limitations + +- Operates on the existing `recipes/` tree only. Does not author new manifests. +- Cluster-mutating apply steps require `kubectl` permission to the target namespace. +- Smoke-test depth is intentionally minimal; for full router/endpoint coverage use `dynamo-router-starter`. +- Multi-node disagg transport correctness is out of scope; use `dynamo-interconnect-check` after deploy. + +## Troubleshooting + +| Symptom | Likely cause | Next step | +|---|---|---| +| `kubectl` cluster unreachable | Context not set or VPN down | Return exact commands instead of running them; resume when cluster is reachable | +| `validate` reports missing storage class | Cluster has no default `StorageClass` | Patch `storageClassName` on the model-cache manifest before applying | +| Model-cache job stuck `Pending` | PVC unbound or HF secret missing | Inspect PVC events; create or rename the HF secret to match the recipe | +| Worker pods `ImagePullBackOff` | Stale image tag or missing pull secret | Patch the image tag; verify image pull secret in the namespace | +| `/v1/models` 4xx/5xx after deploy | Frontend not ready or wrong service port | Wait for pods Ready; re-run port-forward; switch to `dynamo-troubleshoot` if it persists | + ## References - Read `references/k8s-recipe-workflow.md` for command templates and readiness checks. diff --git a/skills/dynamo-recipe-runner/scripts/recipe_tool.py b/skills/dynamo-recipe-runner/scripts/recipe_tool.py old mode 100644 new mode 100755 index e4c35a6d96d1..4869af5ac441 --- a/skills/dynamo-recipe-runner/scripts/recipe_tool.py +++ b/skills/dynamo-recipe-runner/scripts/recipe_tool.py @@ -1,3 +1,4 @@ +#!/usr/bin/env python3 # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 diff --git a/skills/dynamo-recipe-runner/skill-card.md b/skills/dynamo-recipe-runner/skill-card.md new file mode 100644 index 000000000000..7e6668d85eb8 --- /dev/null +++ b/skills/dynamo-recipe-runner/skill-card.md @@ -0,0 +1,39 @@ +## Description:
+Select, validate, patch, and deploy existing NVIDIA Dynamo Kubernetes recipes. Use for model/backend/GPU/deployment-mode recipe bring-up; use router-starter for router-only mode work and troubleshoot for broken deployments.
+ +This skill is ready for commercial/non-commercial use.
+ +## Owner: NVIDIA
+ +### License/Terms of Use:
+Apache 2.0
+## Use Case:
+Developers and infrastructure engineers use this skill to select, configure, and deploy NVIDIA Dynamo inference recipes on Kubernetes clusters, minimizing manual manifest editing and deployment steps.
+ +### Deployment Geography for Use:
+Global
+ +## Known Risks and Mitigations:
+Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills.
+Mitigation: Review and scan skill before deployment.
+ +## Reference(s):
+- [Kubernetes Recipe Workflow](references/k8s-recipe-workflow.md)
+- [Dynamo Recipes](https://github.com/ai-dynamo/dynamo/tree/main/recipes)
+- [Dynamo Documentation](https://docs.nvidia.com/dynamo/)
+ + +## Skill Output:
+**Output Type(s):** [Shell commands, Configuration instructions, Analysis]
+**Output Format:** [Markdown with inline bash code blocks]
+**Output Parameters:** [1D]
+**Other Properties Related to Output:** [None]
+ +## Skill Version(s):
+1.2.0 (source: pyproject.toml)
+ +## Ethical Considerations:
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+ +(For Release on NVIDIA Platforms Only)
+Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail).
diff --git a/skills/dynamo-recipe-runner/skill.oms.sig b/skills/dynamo-recipe-runner/skill.oms.sig new file mode 100644 index 000000000000..dee9204b43f6 --- /dev/null +++ b/skills/dynamo-recipe-runner/skill.oms.sig @@ -0,0 +1 @@ +{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAiZHluYW1vLXJlY2lwZS1ydW5uZXIiLAogICAgICAiZGlnZXN0IjogewogICAgICAgICJzaGEyNTYiOiAiMWVhOTliMTZlZmI5YzAwN2U1NWE5YTA3OTRiOWUzYWZiNTVjYjYwYjlhNmMxYzkwOWQwODZhYjY2OTU1ZjE2OSIKICAgICAgfQogICAgfQogIF0sCiAgInByZWRpY2F0ZVR5cGUiOiAiaHR0cHM6Ly9tb2RlbF9zaWduaW5nL3NpZ25hdHVyZS92MS4wIiwKICAicHJlZGljYXRlIjogewogICAgInJlc291cmNlcyI6IFsKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogIjcyYjczYmVlOTBlMjY2YjBjMTFmYjA4NTJhOTJkNjY1MDMyMDk0NzlhMDEzNWY0MjExYzY4M2EyODFjODA3ZmMiLAogICAgICAgICJuYW1lIjogIlNLSUxMLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiMmYzMDM1OWY5MWYwNGFjMmQ2ZGM4NGJiYjU3ZWE0MDkwNzVhMTNlOWQ0NjM2MjI2OTAxNDNhNGMzZWZiY2NiOSIsCiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIKICAgICAgfSwKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogIjg0MzcxNjAxYmExMzA3MjVhODU1NTA3YjFhMjc4ZjQ5ZGNiNWVkNjI0NTAxNDA5ZTM0ZWFlNWIxMDU1ZDA3MzEiLAogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvazhzLXJlY2lwZS13b3JrZmxvdy5tZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogIjlmYjJjZTg5YWE1MzllZWI2ZjQwNDc0ZTYxN2I0ZGFkNzVkZGIyYTFmODFlOTEyNzYyMTVjZGU2MDQ3MTZlODEiLAogICAgICAgICJuYW1lIjogInNjcmlwdHMvcmVjaXBlX3Rvb2wucHkiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICIzNGMwMzQ3YmRkYmNkODkwNWE2NjQ3ZGQ2MTQ2YmU2MDUxMTczZDQyMTgwNmViMDBiOTkyNDUwZmQzZjZiZTA0IiwKICAgICAgICAibmFtZSI6ICJza2lsbC1jYXJkLm1kIgogICAgICB9CiAgICBdLAogICAgInNlcmlhbGl6YXRpb24iOiB7CiAgICAgICJhbGxvd19zeW1saW5rcyI6IGZhbHNlLAogICAgICAiaWdub3JlX3BhdGhzIjogWwogICAgICAgICIuZ2l0YXR0cmlidXRlcyIsCiAgICAgICAgIi5naXQiLAogICAgICAgICIuZ2l0aHViIiwKICAgICAgICAiLmdpdGlnbm9yZSIKICAgICAgXSwKICAgICAgIm1ldGhvZCI6ICJmaWxlcyIsCiAgICAgICJoYXNoX3R5cGUiOiAic2hhMjU2IgogICAgfQogIH0KfQ==","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGQCMA0KTeGaqHB2ImHF/tgzT1v/6uk3FuBWdV8nQuurILSm7My0zwsR/fPO94kGjTz/QwIwXD8kjfb9ytJWVnzGsEc9Z3CmAR7z2DXy0E4oJATIpj6Hq2mNbCxsphcWQp2rTqv3","keyid":""}]}} \ No newline at end of file diff --git a/skills/dynamo-router-starter/SKILL.md b/skills/dynamo-router-starter/SKILL.md index c8b4cdcc77fd..2d8d89d9957f 100644 --- a/skills/dynamo-router-starter/SKILL.md +++ b/skills/dynamo-router-starter/SKILL.md @@ -1,6 +1,14 @@ --- name: dynamo-router-starter description: Start or patch Dynamo router modes and run router endpoint smoke checks. Use for round-robin, KV-aware, least-loaded, or device-aware routing setup; use recipe-runner for recipe deployment and troubleshoot for failure diagnosis. +license: Apache-2.0 +metadata: + author: Dan Gil + tags: + - dynamo + - router + - smoke-test + - bring-up --- # Dynamo Router Starter @@ -10,12 +18,19 @@ SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All SPDX-License-Identifier: CC-BY-4.0 --> -## Goal +## Purpose Make Dynamo routing feel easy by getting a baseline router mode running, enabling KV-aware routing when appropriate, and proving the endpoint works. Keep the user focused on exact commands and success signals, not router internals. +## Prerequisites + +- Python 3.10+ with the `dynamo` package importable (`python3 -m dynamo.frontend --help` works). +- For Kubernetes runs: `kubectl` configured with access to the target namespace and a deployed Dynamo recipe. +- Network reachability to the frontend service (port-forward or direct). +- A model already loaded into at least one worker (`/v1/models` returns at least one entry). + ## Required Inputs Collect or infer: @@ -26,7 +41,7 @@ Collect or infer: - whether workers publish KV events; if not, use approximate KV mode - model name for smoke requests, if `/v1/models` cannot discover it -## Workflow +## Instructions ### 1. Establish A Baseline @@ -89,6 +104,40 @@ When comparing round-robin vs KV routing: If the endpoint is unhealthy or workers are missing, switch to `dynamo-troubleshoot`. +## Available Scripts + +| Script | Purpose | Arguments | +|---|---|---| +| `scripts/check_router_health.py` | Smoke-test `/v1/models` and one chat completion against a Dynamo frontend | `--base-url`, `--retries`, `--timeout` | + +Invoke via the agentskills.io `run_script()` protocol: + +```python +run_script("scripts/check_router_health.py", args=["--base-url", "http://127.0.0.1:8000"]) +``` + +## Examples + +Local KV-routed frontend on port 8000, then smoke-test it: + +```bash +python3 -m dynamo.frontend --router-mode kv --http-port 8000 & +python3 scripts/check_router_health.py --base-url http://127.0.0.1:8000 +``` + +Kubernetes-deployed frontend reachable via port-forward: + +```bash +kubectl port-forward svc/qwen-vllm-disagg-frontend 8000:8000 -n dynamo-demo & +python3 scripts/check_router_health.py --base-url http://127.0.0.1:8000 --retries 3 +``` + +Equivalent through the agent protocol: + +```python +run_script("scripts/check_router_health.py", args=["--base-url", "http://127.0.0.1:8000", "--retries", "3"]) +``` + ## Output Contract Return: @@ -100,6 +149,21 @@ Return: - any limitation, such as approximate KV mode or missing worker KV events - next command to run for a fuller comparison +## Limitations + +- Smoke test is one chat completion; it is not a benchmark. Use `dynamo-benchmark` for throughput/latency numbers. +- KV-aware mode without worker KV-event publication degrades to approximate mode; this skill flags but does not fix the underlying worker config. +- Mode comparisons require matched workloads; cross-mode latency claims need separate benchmark runs. + +## Troubleshooting + +| Symptom | Likely cause | Next step | +|---|---|---| +| `/v1/models` returns empty list | No worker registered with the frontend | Verify worker pods are Ready; confirm they connect to the same etcd/NATS | +| Smoke chat request times out | Frontend up, workers not serving | Switch to `dynamo-troubleshoot`; inspect worker logs | +| KV mode hangs | Workers do not publish KV cache events | Set `DYN_ROUTER_USE_KV_EVENTS=false` (approximate mode) | +| Connection refused on port-forward | Port-forward dropped or wrong service name | Re-run port-forward; verify the frontend service name matches the recipe | + ## References - Read `references/router-modes.md` for the compact mode/env map. diff --git a/skills/dynamo-router-starter/scripts/check_router_health.py b/skills/dynamo-router-starter/scripts/check_router_health.py old mode 100644 new mode 100755 index 9fa0d12aa925..f7fd8bbd3d34 --- a/skills/dynamo-router-starter/scripts/check_router_health.py +++ b/skills/dynamo-router-starter/scripts/check_router_health.py @@ -1,3 +1,4 @@ +#!/usr/bin/env python3 # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 @@ -14,9 +15,27 @@ import urllib.request from typing import Any +# Tunables and contract values (kept here to avoid magic numbers in the body). +DEFAULT_BASE_URL = "http://127.0.0.1:8000" +DEFAULT_PROMPT = "Say hello from Dynamo in one short sentence." +DEFAULT_MAX_TOKENS = 32 +DEFAULT_RETRIES = 5 +DEFAULT_RETRY_SLEEP_SEC = 2.0 +DEFAULT_HTTP_TIMEOUT_SEC = 20.0 +HTTP_OK = 200 + +# Process exit codes used to distinguish smoke-test outcomes. +EXIT_OK = 0 +EXIT_MODELS_UNAVAILABLE = 2 +EXIT_NO_MODEL_DISCOVERED = 3 +EXIT_CHAT_FAILED = 4 + def request_json( - method: str, url: str, payload: dict[str, Any] | None = None, timeout: float = 20 + method: str, + url: str, + payload: dict[str, Any] | None = None, + timeout: float = DEFAULT_HTTP_TIMEOUT_SEC, ) -> tuple[int, Any]: # Only talk to real HTTP(S) endpoints; urlopen otherwise happily opens # file:// and other local schemes if a bad --base-url is passed. @@ -64,15 +83,13 @@ def choose_model(models_body: Any) -> str | None: def main() -> int: parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--base-url", default="http://127.0.0.1:8000") + parser.add_argument("--base-url", default=DEFAULT_BASE_URL) parser.add_argument("--model") - parser.add_argument( - "--prompt", default="Say hello from Dynamo in one short sentence." - ) - parser.add_argument("--max-tokens", type=int, default=32) + parser.add_argument("--prompt", default=DEFAULT_PROMPT) + parser.add_argument("--max-tokens", type=int, default=DEFAULT_MAX_TOKENS) parser.add_argument("--skip-chat", action="store_true") - parser.add_argument("--retries", type=int, default=5) - parser.add_argument("--retry-sleep", type=float, default=2.0) + parser.add_argument("--retries", type=int, default=DEFAULT_RETRIES) + parser.add_argument("--retry-sleep", type=float, default=DEFAULT_RETRY_SLEEP_SEC) args = parser.parse_args() base_url = args.base_url.rstrip("/") @@ -82,7 +99,7 @@ def main() -> int: models_body = None for attempt in range(1, args.retries + 1): models_status, models_body = request_json("GET", f"{base_url}/v1/models") - if models_status == 200: + if models_status == HTTP_OK: break time.sleep(args.retry_sleep) @@ -91,21 +108,21 @@ def main() -> int: {"name": "models", "status": models_status, "body": models_body, "model": model} ) - if models_status != 200: + if models_status != HTTP_OK: print(json.dumps(result, indent=2)) - return 2 + return EXIT_MODELS_UNAVAILABLE if args.skip_chat: result["ok"] = True print(json.dumps(result, indent=2)) - return 0 + return EXIT_OK if not model: result["checks"].append( {"name": "chat", "status": "skipped", "reason": "No model discovered"} ) print(json.dumps(result, indent=2)) - return 3 + return EXIT_NO_MODEL_DISCOVERED payload = { "model": model, @@ -116,9 +133,9 @@ def main() -> int: "POST", f"{base_url}/v1/chat/completions", payload ) result["checks"].append({"name": "chat", "status": chat_status, "body": chat_body}) - result["ok"] = chat_status == 200 + result["ok"] = chat_status == HTTP_OK print(json.dumps(result, indent=2)) - return 0 if result["ok"] else 4 + return EXIT_OK if result["ok"] else EXIT_CHAT_FAILED if __name__ == "__main__": diff --git a/skills/dynamo-router-starter/skill-card.md b/skills/dynamo-router-starter/skill-card.md new file mode 100644 index 000000000000..74f58df05298 --- /dev/null +++ b/skills/dynamo-router-starter/skill-card.md @@ -0,0 +1,37 @@ +## Description:
+Start or patch Dynamo router modes and run router endpoint smoke checks.
+ +This skill is ready for commercial/non-commercial use.
+ +## Owner: NVIDIA
+ +### License/Terms of Use:
+Apache-2.0
+## Use Case:
+Developers and infrastructure engineers use this skill to configure Dynamo routing modes (round-robin, KV-aware, least-loaded, device-aware) and verify endpoint health via smoke tests during cluster bring-up.
+ +### Deployment Geography for Use:
+Global
+ +## Known Risks and Mitigations:
+Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills.
+Mitigation: Review and scan skill before deployment.
+ +## Reference(s):
+- [Router Modes Reference](references/router-modes.md)
+ + +## Skill Output:
+**Output Type(s):** [Shell commands, Configuration instructions]
+**Output Format:** [Markdown with inline bash code blocks]
+**Output Parameters:** [1D]
+**Other Properties Related to Output:** [None]
+ +## Skill Version(s):
+1.2.0 (source: pyproject.toml)
+ +## Ethical Considerations:
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+ +(For Release on NVIDIA Platforms Only)
+Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail).
diff --git a/skills/dynamo-router-starter/skill.oms.sig b/skills/dynamo-router-starter/skill.oms.sig new file mode 100644 index 000000000000..143fabaf0253 --- /dev/null +++ b/skills/dynamo-router-starter/skill.oms.sig @@ -0,0 +1 @@ +{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAiZHluYW1vLXJvdXRlci1zdGFydGVyIiwKICAgICAgImRpZ2VzdCI6IHsKICAgICAgICAic2hhMjU2IjogIjg0ZmViZDU3N2NlZjQ2YTk5ZDBhMTFjMThlMTcyZWExOTRhYTgyMDM2MTk2MjYwYzQ2NzA5ZGU5M2FhYTkzNzciCiAgICAgIH0KICAgIH0KICBdLAogICJwcmVkaWNhdGVUeXBlIjogImh0dHBzOi8vbW9kZWxfc2lnbmluZy9zaWduYXR1cmUvdjEuMCIsCiAgInByZWRpY2F0ZSI6IHsKICAgICJyZXNvdXJjZXMiOiBbCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICI0ZDA3OGNlNGM1ZDhjYTM4MzJhZGViZTdlZTBmMGNhZjc2ODZiYWRhMDM0MjQwMWZlNjBmZTM2YjgzMTJjZWQ4IiwKICAgICAgICAibmFtZSI6ICJTS0lMTC5tZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogIjMyZTdmMWI2NTBlMTU5OWE2YjY4NDY4YWRiMmNjMjdlYzY5YWVhMWNmNzVkOTRjNjY2NTg4ZjI2MWI3MmM2ZGMiLAogICAgICAgICJuYW1lIjogImV2YWxzL2V2YWxzLmpzb24iCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICIyMTI2ZGUyYTYzN2ZiMzhjY2NmNzczMzNlMDNmNzMwN2UxMjhhNGY3YjA4OGE0MmQwY2MzOWI5M2U4MzQwNmYzIiwKICAgICAgICAibmFtZSI6ICJyZWZlcmVuY2VzL3JvdXRlci1tb2Rlcy5tZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogIjcyMzIxZjQ0Mjc0NjFiZmJhNDk3MjI3ZDFlNTU0YzJmZjgzZWZiYzM5ZDRlZDNmMzZmMmM3NWM4NDA0MTBkZTIiLAogICAgICAgICJuYW1lIjogInNjcmlwdHMvY2hlY2tfcm91dGVyX2hlYWx0aC5weSIKICAgICAgfSwKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogImNhMjQ4NzdiODliYTM5NzJiNzc3YjE5YzQ3NjIyMjYyOTczZmI0YzVkNzhkYzVkMWU3ZGQyNWNlOWZlY2VmYzUiLAogICAgICAgICJuYW1lIjogInNraWxsLWNhcmQubWQiCiAgICAgIH0KICAgIF0sCiAgICAic2VyaWFsaXphdGlvbiI6IHsKICAgICAgImlnbm9yZV9wYXRocyI6IFsKICAgICAgICAiLmdpdGlnbm9yZSIsCiAgICAgICAgIi5naXRodWIiLAogICAgICAgICIuZ2l0YXR0cmlidXRlcyIsCiAgICAgICAgIi5naXQiCiAgICAgIF0sCiAgICAgICJhbGxvd19zeW1saW5rcyI6IGZhbHNlLAogICAgICAibWV0aG9kIjogImZpbGVzIiwKICAgICAgImhhc2hfdHlwZSI6ICJzaGEyNTYiCiAgICB9CiAgfQp9","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGYCMQCnsedaVvqBHWgArjKjF2MINkaInqnvqPTooWA+EISOj9R3R62xRz14jf4s3mt5WQcCMQDXMJ1dENslRRHJFbAlB5Fgp+36lJeJ+ry0/2lsPPH8bN9R1cYKNlkb+xJoZ+RmwH0=","keyid":""}]}} \ No newline at end of file diff --git a/skills/dynamo-troubleshoot/SKILL.md b/skills/dynamo-troubleshoot/SKILL.md index d7ca88b455ce..e7cb5e7d8079 100644 --- a/skills/dynamo-troubleshoot/SKILL.md +++ b/skills/dynamo-troubleshoot/SKILL.md @@ -1,6 +1,14 @@ --- name: dynamo-troubleshoot description: Diagnose failed or unhealthy Dynamo deployments. Use when pods, model-cache jobs, PVCs, workers, frontend/router health, endpoints, or benchmark jobs fail; use recipe-runner/router-starter before this for normal bring-up. +license: Apache-2.0 +metadata: + author: Dan Gil + tags: + - dynamo + - kubernetes + - troubleshooting + - day-2 --- # Dynamo Troubleshoot @@ -10,13 +18,20 @@ SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All SPDX-License-Identifier: CC-BY-4.0 --> -## Goal +## Purpose Turn a Dynamo failure into a clear problem class, strongest signal, and next action. Start with read-only evidence, avoid secrets, and fix one layer at a time. -## Workflow +## Prerequisites + +- Python 3.10+ on the operator machine. +- `kubectl` configured with read access to the target namespace. +- Permission to read pods, events, jobs, PVCs, and `DynamoGraphDeployment` resources (NOT secrets). +- Network reachability to the cluster API server. + +## Instructions ### 1. Collect A Read-Only Bundle @@ -78,6 +93,40 @@ Prefer the smallest reversible change: After each fix, rerun the relevant readiness check before moving deeper. +## Available Scripts + +| Script | Purpose | Arguments | +|---|---|---| +| `scripts/collect_dynamo_debug_bundle.py` | Collect a read-only debug bundle (pods, events, jobs, PVCs, CR status) | `--namespace`, `--deployment-name`, `--output-dir` | + +Invoke via the agentskills.io `run_script()` protocol: + +```python +run_script("scripts/collect_dynamo_debug_bundle.py", args=["--namespace", "dynamo-demo"]) +``` + +## Examples + +Collect everything in a namespace for triage: + +```bash +python3 scripts/collect_dynamo_debug_bundle.py --namespace dynamo-demo +``` + +Scope to a single failing deployment: + +```bash +python3 scripts/collect_dynamo_debug_bundle.py \ + --namespace dynamo-demo \ + --deployment-name qwen-vllm-disagg +``` + +Equivalent through the agent protocol: + +```python +run_script("scripts/collect_dynamo_debug_bundle.py", args=["--namespace", "dynamo-demo", "--deployment-name", "qwen-vllm-disagg"]) +``` + ## Output Contract Return: @@ -90,6 +139,22 @@ Return: - what was ruled out - whether it is safe to continue deployment or benchmarking +## Limitations + +- Read-only. Never mutates the cluster; remediation commands are returned, not executed. +- Will not collect secrets or print Hugging Face tokens; some failure modes (auth) may need user-side inspection. +- Bundle size grows with deployment size; on very large namespaces, scope with `--deployment-name`. +- Does not validate disagg transport — use `dynamo-interconnect-check` for that. + +## Troubleshooting + +| Symptom | Likely cause | Next step | +|---|---|---| +| `kubectl` returns Forbidden on events/pods | Service account lacks read RBAC | Ask operator for read-only role binding on the namespace | +| Bundle missing `DynamoGraphDeployment` status | Operator not installed or different namespace | Verify `dynamo-platform` operator is installed and watching the namespace | +| Model-download job in `Pending` | PVC unbound or HF secret missing | Fix PVC binding or create the named HF secret, then rerun the job | +| Worker pods `CrashLoopBackOff` | Image/runtime mismatch or GPU not available | Inspect container logs; check `nvidia.com/gpu` allocatable on nodes | + ## References - Read `references/failure-decision-tree.md` for bucket-specific checks. diff --git a/skills/dynamo-troubleshoot/scripts/collect_dynamo_debug_bundle.py b/skills/dynamo-troubleshoot/scripts/collect_dynamo_debug_bundle.py old mode 100644 new mode 100755 index 7d819add65a5..2b1822a1dfba --- a/skills/dynamo-troubleshoot/scripts/collect_dynamo_debug_bundle.py +++ b/skills/dynamo-troubleshoot/scripts/collect_dynamo_debug_bundle.py @@ -1,3 +1,4 @@ +#!/usr/bin/env python3 # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 @@ -14,6 +15,14 @@ from pathlib import Path from typing import Any +# Tunables and conventional return codes (kept here to avoid magic numbers). +DEFAULT_KUBECTL_TIMEOUT_SEC = 30 +DEFAULT_LOG_TAIL_LINES = 200 +# POSIX-conventional return codes used when the wrapper itself fails before +# kubectl can produce a real one. +RETURNCODE_COMMAND_NOT_FOUND = 127 # `kubectl` not installed +RETURNCODE_TIMED_OUT = 124 # subprocess timeout + # `kubectl describe` and pod logs can echo secret env values (HF tokens, # bearer tokens, passwords). Scrub them before anything is written to disk so # the bundle honors its no-secrets contract. @@ -46,11 +55,16 @@ def run(cmd: list[str], timeout: int) -> dict[str, Any]: "stderr": proc.stderr, } except FileNotFoundError as exc: - return {"cmd": cmd, "returncode": 127, "stdout": "", "stderr": str(exc)} + return { + "cmd": cmd, + "returncode": RETURNCODE_COMMAND_NOT_FOUND, + "stdout": "", + "stderr": str(exc), + } except subprocess.TimeoutExpired as exc: return { "cmd": cmd, - "returncode": 124, + "returncode": RETURNCODE_TIMED_OUT, "stdout": exc.stdout or "", "stderr": exc.stderr or f"Timed out after {timeout}s", } @@ -129,8 +143,8 @@ def main() -> int: default=None, help="Output dir; defaults to a private mkdtemp dynamo-debug-* directory", ) - parser.add_argument("--tail", type=int, default=200) - parser.add_argument("--timeout", type=int, default=30) + parser.add_argument("--tail", type=int, default=DEFAULT_LOG_TAIL_LINES) + parser.add_argument("--timeout", type=int, default=DEFAULT_KUBECTL_TIMEOUT_SEC) args = parser.parse_args() if args.outdir: diff --git a/skills/dynamo-troubleshoot/skill-card.md b/skills/dynamo-troubleshoot/skill-card.md new file mode 100644 index 000000000000..5a85a2d6fd12 --- /dev/null +++ b/skills/dynamo-troubleshoot/skill-card.md @@ -0,0 +1,38 @@ +## Description:
+Diagnose failed or unhealthy Dynamo deployments. Use when pods, model-cache jobs, PVCs, workers, frontend/router health, endpoints, or benchmark jobs fail; use recipe-runner/router-starter before this for normal bring-up.
+ +This skill is ready for commercial/non-commercial use.
+ +## Owner: NVIDIA
+ +### License/Terms of Use:
+Apache 2.0
+## Use Case:
+Developers and platform engineers use this skill to diagnose and resolve failures in Dynamo Kubernetes deployments, including pod crashes, PVC issues, model-download job failures, GPU scheduling problems, and endpoint health checks.
+ +### Deployment Geography for Use:
+Global
+ +## Known Risks and Mitigations:
+Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills.
+Mitigation: Review and scan skill before deployment.
+ +## Reference(s):
+- [Failure Decision Tree](references/failure-decision-tree.md)
+- [Dynamo Documentation](https://docs.nvidia.com/dynamo/)
+ + +## Skill Output:
+**Output Type(s):** [Analysis, Shell commands, Configuration instructions]
+**Output Format:** [Markdown with inline bash code blocks]
+**Output Parameters:** [1D]
+**Other Properties Related to Output:** [None]
+ +## Skill Version(s):
+1.2.0 (source: pyproject.toml)
+ +## Ethical Considerations:
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+ +(For Release on NVIDIA Platforms Only)
+Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail).
diff --git a/skills/dynamo-troubleshoot/skill.oms.sig b/skills/dynamo-troubleshoot/skill.oms.sig new file mode 100644 index 000000000000..f50a20f62634 --- /dev/null +++ b/skills/dynamo-troubleshoot/skill.oms.sig @@ -0,0 +1 @@ +{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAiZHluYW1vLXRyb3VibGVzaG9vdCIsCiAgICAgICJkaWdlc3QiOiB7CiAgICAgICAgInNoYTI1NiI6ICJhYjc4OGFiZGE4OGM0N2Q2YzZiMjU1ZmZlNGRhYTFmMjc1ZGU2YzhiZGFjYzAzZGYwMzg1N2ZmOTNkNTA2ZmMxIgogICAgICB9CiAgICB9CiAgXSwKICAicHJlZGljYXRlVHlwZSI6ICJodHRwczovL21vZGVsX3NpZ25pbmcvc2lnbmF0dXJlL3YxLjAiLAogICJwcmVkaWNhdGUiOiB7CiAgICAic2VyaWFsaXphdGlvbiI6IHsKICAgICAgImlnbm9yZV9wYXRocyI6IFsKICAgICAgICAiLmdpdGlnbm9yZSIsCiAgICAgICAgIi5naXRhdHRyaWJ1dGVzIiwKICAgICAgICAiLmdpdCIsCiAgICAgICAgIi5naXRodWIiCiAgICAgIF0sCiAgICAgICJtZXRob2QiOiAiZmlsZXMiLAogICAgICAiaGFzaF90eXBlIjogInNoYTI1NiIsCiAgICAgICJhbGxvd19zeW1saW5rcyI6IGZhbHNlCiAgICB9LAogICAgInJlc291cmNlcyI6IFsKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiYmM5NDMyYWJlZjA2ZDRiNGQzNzhmNTE1NDQwN2Y5MWFhNTFiNWJkZmYxNDI2NmY1MDFhZjY1N2MyMWE1ZDdlMCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogIlNLSUxMLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICI0OTJjMmFjOTI2MTU4ZDQ2MjZkOGYwYzM2ZjNmNDM0Mjc5M2FjYWM0OTIyNGY2NzgwNGQ3YWYxY2EzZDEyNDQ3IiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIKICAgICAgfSwKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiMjY0YWEwYTM4ZjZlYmNlNTVhZjVlYjUxNTZlYjRmZWYwYWUzN2YwYTQ5YmU3N2QwNDliYTZiZmMwMTQzNzM3YiIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvZmFpbHVyZS1kZWNpc2lvbi10cmVlLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICIxMjAwZWJjN2YxMWUxMDdhZWI0NDM2MDE4Mjc2OTY0NTZiMDU5NjU4MWNlNzA3NTc4NWQzNWQ0ZTRjZDA5YTZlIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAic2NyaXB0cy9jb2xsZWN0X2R5bmFtb19kZWJ1Z19idW5kbGUucHkiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiZGlnZXN0IjogIjY2M2MzOTk2MTU4YzU2NWFmYWEzN2I3MDMxZTgwYjYxNDlkODE3YzFmMmMyOWM2YzkxZDgwZmZjNTJiYzUyNDQiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAibmFtZSI6ICJza2lsbC1jYXJkLm1kIgogICAgICB9CiAgICBdCiAgfQp9","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGYCMQDJyd8EAzKQvyOQNfUNcwX6jcfkQHDKRRP7smgXpBZOrrgcc6MC+6qKHg5rXFVGurwCMQCOpx2iiG274qbMv4p1MpJry8WHT3jjjQxN9+6/pX7/Z5MYkcZPKPUdeFf3f5xLYQY=","keyid":""}]}} \ No newline at end of file