From 2597b6669ea1e3a408b18e3c8deeca83242119ba Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Muhammet=20Eren=20Karaku=C5=9F?= Date: Sat, 28 Feb 2026 14:40:07 +0300 Subject: [PATCH 1/5] feat(skills): add Psyche network monitor skill with multi-run support MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rework of #35 based on maintainer feedback — removes the tool entirely and ships only as a skill. No changes to tools/, toolsets.py, model_tools.py, or tests/. - Dynamic multi-run discovery via HuggingFace API (no hardcoded runs) - Fallback strategy: HF API → Solana RPC → Psyche CLI - Run state reference aligned with official Psyche documentation - Command reference with curl + python3 parsing examples - Participation guide for mining pool, compute, and code contributions --- skills/psyche/DESCRIPTION.md | 3 + skills/psyche/network-monitor/SKILL.md | 172 ++++++++++++++++ .../network-monitor/references/commands.md | 184 ++++++++++++++++++ .../network-monitor/references/run-states.md | 47 +++++ 4 files changed, 406 insertions(+) create mode 100644 skills/psyche/DESCRIPTION.md create mode 100644 skills/psyche/network-monitor/SKILL.md create mode 100644 skills/psyche/network-monitor/references/commands.md create mode 100644 skills/psyche/network-monitor/references/run-states.md diff --git a/skills/psyche/DESCRIPTION.md b/skills/psyche/DESCRIPTION.md new file mode 100644 index 0000000000000..ccb817f555052 --- /dev/null +++ b/skills/psyche/DESCRIPTION.md @@ -0,0 +1,3 @@ +--- +description: Skills for monitoring and interacting with the Psyche decentralized AI training network — training runs, checkpoints, mining pool, and on-chain state via Solana. +--- diff --git a/skills/psyche/network-monitor/SKILL.md b/skills/psyche/network-monitor/SKILL.md new file mode 100644 index 0000000000000..72908d9adfbb6 --- /dev/null +++ b/skills/psyche/network-monitor/SKILL.md @@ -0,0 +1,172 @@ +--- +name: network-monitor +description: Monitor Psyche decentralized AI training network — discover runs, track checkpoints, check mining pool status, and query on-chain state. No API keys required. +version: 1.0.0 +author: Eren Karakus +license: MIT +metadata: + hermes: + tags: [Psyche, Nous Research, Decentralized AI, Solana, Training, HuggingFace, Mining Pool] + related_skills: [] +--- + +# Psyche Network Monitor + +Monitor the Psyche decentralized AI training network built by Nous Research. Dynamically discover training runs, track model checkpoints, check mining pool status, and query on-chain state. + +**Zero dependencies. No API keys. Uses curl + python3 + existing Hermes tools.** + +## What is Psyche? + +Psyche is a decentralized AI model training platform coordinated on the Solana blockchain. It distributes training across independent GPU nodes worldwide using DisTrO (distributed training optimization with ~3x bandwidth reduction) and Iroh-based P2P networking. Anyone can contribute compute or funds to training runs. + +## When to Use + +This skill should be loaded when: +- User asks about Psyche Network, its training runs, or models +- User wants to check the status of a Psyche training run +- User asks about mining pool contributions or how to participate +- User wants to find or download Psyche model checkpoints from HuggingFace + +## Quick Reference + +| Action | Command | +|--------|---------| +| List all runs/models | `curl -s "https://huggingface.co/api/models?author=PsycheFoundation"` | +| Get model details | `curl -s "https://huggingface.co/api/models/PsycheFoundation/"` | +| List checkpoint files | `curl -s "https://huggingface.co/api/models/PsycheFoundation//tree/main"` | +| On-chain run state | `run-manager json-dump-run --rpc --run-id ` | + +For detailed command syntax and parsing examples, see `references/commands.md`. +For run state definitions, see `references/run-states.md`. + +## Procedure + +Follow this multi-run discovery workflow. Do not hardcode model names — always discover dynamically. + +### Step 1: Discover All Runs + +Query the HuggingFace API to list all models under the PsycheFoundation organization: + +```bash +curl -s "https://huggingface.co/api/models?author=PsycheFoundation" | \ + python3 -c " +import sys, json +models = json.load(sys.stdin) +for m in sorted(models, key=lambda x: x.get('lastModified',''), reverse=True): + print(f\"{m['id']:55s} modified: {m.get('lastModified','N/A')[:10]} downloads: {m.get('downloads',0)}\") +" +``` + +This returns all training run models with their last modification date. The most recently modified model is likely the currently active run. + +### Step 2: Get Run Details + +Once you identify a model of interest, fetch its full metadata: + +```bash +curl -s "https://huggingface.co/api/models/PsycheFoundation/" | \ + python3 -c " +import sys, json +m = json.load(sys.stdin) +print(f\"Model: {m['id']}\") +print(f\"Tags: {', '.join(m.get('tags', []))}\") +print(f\"Downloads: {m.get('downloads', 0)}\") +print(f\"Last Modified: {m.get('lastModified', 'N/A')}\") +siblings = m.get('siblings', []) +print(f\"Files: {len(siblings)}\") +for s in siblings[:15]: + print(f\" {s['rfilename']}\") +if len(siblings) > 15: + print(f\" ... and {len(siblings)-15} more files\") +" +``` + +### Step 3: List Checkpoint Files + +To see all checkpoint files for a model (useful for downloading specific training snapshots): + +```bash +curl -s "https://huggingface.co/api/models/PsycheFoundation//tree/main" | \ + python3 -c " +import sys, json +files = json.load(sys.stdin) +for f in files: + size_mb = f.get('size', 0) / 1024 / 1024 + print(f\"{f['path']:50s} {size_mb:>8.1f} MB\") +" +``` + +### Step 4: On-Chain State (Optional) + +If the user has the Psyche CLI installed or wants deeper on-chain data, query Solana directly: + +```bash +# Requires: psyche-solana-client (from https://github.com/PsycheFoundation/psyche) +run-manager json-dump-run --rpc "$SOLANA_RPC_URL" --run-id +``` + +If the CLI is not available, use the Solana JSON-RPC API as a fallback (see `references/commands.md`). + +## Fallback Strategy + +If the primary data source fails, follow this fallback chain: + +1. **HuggingFace API** (primary) — Most reliable for model/checkpoint data +2. **Solana RPC** (secondary) — For on-chain run state and participant data +3. **Psyche CLI** (`run-manager`) — If installed locally, provides the richest run data +4. **Web search** — Last resort for general Psyche network status + +If a source returns an error: +- HuggingFace 404 → The model ID may have changed. Re-run Step 1 to discover current models. +- HuggingFace timeout → Retry once, then fall back to `web_search` for cached info. +- Solana RPC timeout → Suggest the user set `$SOLANA_RPC_URL` to a private RPC endpoint. + +## Participation Guide + +### Mining Pool (Financial Contribution) + +1. Install a Solana wallet (Phantom, Solflare, or `solana-keygen new`) +2. Navigate to https://psyche.network +3. Connect wallet and deposit SOL to the training pool +4. Pool capacity is limited — if full, check back periodically + +### Compute Contribution (GPU) + +Requires NVIDIA GPU with sufficient VRAM: +- Minimum: RTX 4090 (24 GB) +- Recommended: A100 (40/80 GB), H100 (80 GB) + +Setup: https://docs.psyche.network + +### Code Contribution + +- Psyche core: https://github.com/PsycheFoundation/psyche (Rust + TypeScript) +- Hermes Agent: https://github.com/NousResearch/hermes-agent +- Atropos RL: https://github.com/NousResearch/atropos + +## Key Links + +| Resource | URL | +|----------|-----| +| Dashboard | https://psyche.network | +| Documentation | https://docs.psyche.network | +| GitHub | https://github.com/PsycheFoundation/psyche | +| Forum | https://forum.nousresearch.com | +| HuggingFace Org | https://huggingface.co/PsycheFoundation | +| Discord | https://discord.gg/NousResearch | + +## Pitfalls + +- **Do not hardcode model names.** Always use Step 1 to discover current models. New runs and models are added regularly. +- **Mining pool capacity.** The pool is frequently full. If `psyche.network` shows the pool as full, advise the user to check back later. +- **Public Solana RPC rate limits.** The default `https://api.mainnet-beta.solana.com` has strict rate limits. For repeated queries, recommend a private RPC provider or set `$SOLANA_RPC_URL`. +- **Checkpoint size.** Large models (40B+) have checkpoint files in the tens of GB. Warn the user about download size before initiating. +- **Testnet vs Mainnet.** Some runs may be on Solana devnet. Check the run documentation or dashboard for the correct network. + +## Verification + +After completing any query, verify the results: +- Model listing returns a JSON array with `id` fields → success +- Model details contain `siblings` (file list) → model exists and has uploaded checkpoints +- If zero models returned, re-check the API URL or use `web_search "PsycheFoundation HuggingFace"` as fallback diff --git a/skills/psyche/network-monitor/references/commands.md b/skills/psyche/network-monitor/references/commands.md new file mode 100644 index 0000000000000..0b971bb2c30fd --- /dev/null +++ b/skills/psyche/network-monitor/references/commands.md @@ -0,0 +1,184 @@ +# Psyche Network Monitor — Command Reference + +Detailed command syntax and parsing examples for querying Psyche network data. + +## HuggingFace API + +All endpoints are public, no authentication required. + +### List All Models + +```bash +curl -s "https://huggingface.co/api/models?author=PsycheFoundation" +``` + +**Response:** JSON array. Each element contains: +- `id` — Full model ID (e.g., `PsycheFoundation/consilience-40b-CqX3FUm4`) +- `modelId` — Same as `id` +- `lastModified` — ISO 8601 timestamp +- `downloads` — Total download count +- `tags` — List of tags (architecture, library, etc.) +- `pipeline_tag` — Model pipeline type + +**Parse to table:** + +```bash +curl -s "https://huggingface.co/api/models?author=PsycheFoundation" | \ + python3 -c " +import sys, json +models = json.load(sys.stdin) +print(f'{'Model ID':55s} {'Modified':12s} {'Downloads':>10s}') +print('-' * 82) +for m in sorted(models, key=lambda x: x.get('lastModified',''), reverse=True): + print(f\"{m['id']:55s} {m.get('lastModified','N/A')[:10]:12s} {m.get('downloads',0):>10d}\") +print(f'\\nTotal: {len(models)} models') +" +``` + +### Get Model Details + +```bash +curl -s "https://huggingface.co/api/models/PsycheFoundation/" +``` + +**Response:** JSON object with full metadata including `siblings` (file list), `cardData`, `config`, etc. + +### List Files (Tree) + +```bash +curl -s "https://huggingface.co/api/models/PsycheFoundation//tree/main" +``` + +**Response:** JSON array. Each element contains: +- `path` — File path relative to repo root +- `size` — File size in bytes +- `type` — `file` or `directory` +- `oid` — Git object ID (SHA) + +**Parse with size:** + +```bash +curl -s "https://huggingface.co/api/models/PsycheFoundation//tree/main" | \ + python3 -c " +import sys, json +files = json.load(sys.stdin) +total = 0 +for f in sorted(files, key=lambda x: x.get('size',0), reverse=True): + size = f.get('size', 0) + total += size + if size > 1024*1024: + print(f\" {size/1024/1024:>10.1f} MB {f['path']}\") + else: + print(f\" {size/1024:>10.1f} KB {f['path']}\") +print(f'\\nTotal: {total/1024/1024/1024:.1f} GB across {len(files)} files') +" +``` + +### List Commits (Version History) + +```bash +curl -s "https://huggingface.co/api/models/PsycheFoundation//commits/main" | \ + python3 -c " +import sys, json +commits = json.load(sys.stdin) +for c in commits[:10]: + print(f\"{c['date'][:16]} {c['title']}\") +" +``` + +--- + +## Solana RPC + +For on-chain queries. The default public endpoint has rate limits; for production use, set `$SOLANA_RPC_URL` to a private RPC provider. + +### Check Account Balance + +```bash +curl -s -X POST "${SOLANA_RPC_URL:-https://api.mainnet-beta.solana.com}" \ + -H "Content-Type: application/json" \ + -d '{ + "jsonrpc": "2.0", + "id": 1, + "method": "getBalance", + "params": [""] + }' | python3 -c " +import sys, json +r = json.load(sys.stdin) +lamports = r.get('result', {}).get('value', 0) +print(f'Balance: {lamports / 1e9:.4f} SOL') +" +``` + +### Get Account Info (Run State) + +```bash +curl -s -X POST "${SOLANA_RPC_URL:-https://api.mainnet-beta.solana.com}" \ + -H "Content-Type: application/json" \ + -d '{ + "jsonrpc": "2.0", + "id": 1, + "method": "getAccountInfo", + "params": ["", {"encoding": "base64"}] + }' +``` + +**Note:** The account data is binary (Borsh-serialized). For human-readable output, prefer the Psyche CLI `json-dump-run` command below. + +--- + +## Psyche CLI (run-manager) + +Requires the Psyche client. Install via Docker or build from source. + +### Install + +```bash +# Option 1: Docker +docker pull ghcr.io/psychefoundation/psyche:latest + +# Option 2: Build from source +git clone https://github.com/PsycheFoundation/psyche +cd psyche +cargo build --release --bin psyche-solana-client +``` + +### Dump Run State + +```bash +run-manager json-dump-run \ + --rpc "${SOLANA_RPC_URL:-https://api.mainnet-beta.solana.com}" \ + --run-id +``` + +Returns full JSON dump of the run's on-chain state: participants, round info, epoch data, configuration. + +### Dump User State + +```bash +run-manager json-dump-user \ + --rpc "${SOLANA_RPC_URL:-https://api.mainnet-beta.solana.com}" \ + --run-id \ + --address +``` + +Returns the user's participation data: collateral, rewards, training history. + +### Pause / Resume a Run + +```bash +# Pause +run-manager set-paused \ + --rpc \ + --run-id \ + --wallet-private-key-path + +# Resume +run-manager set-paused \ + --rpc \ + --run-id \ + --resume \ + --wallet-private-key-path +``` + +**Note:** Pause/resume requires the run coordinator's wallet. This is an admin operation. diff --git a/skills/psyche/network-monitor/references/run-states.md b/skills/psyche/network-monitor/references/run-states.md new file mode 100644 index 0000000000000..aa13c44198fea --- /dev/null +++ b/skills/psyche/network-monitor/references/run-states.md @@ -0,0 +1,47 @@ +# Psyche Run States + +Canonical run state definitions from the Psyche Network coordinator program. Use these to interpret on-chain run data and advise users. + +Source: https://docs.psyche.network/explain/general-workflow.html + +## State Machine + +``` +WaitingForMembers → Warmup → RoundTrain ↔ RoundWitness → Cooldown + ↑ | + └────────────────── (next epoch) ────────────────────┘ +``` + +An epoch cycles through: WaitingForMembers → Warmup → (RoundTrain ↔ RoundWitness) × N rounds → Cooldown → back to WaitingForMembers for the next epoch. + +## State Reference + +| State | Meaning | What to Tell the User | +|-------|---------|----------------------| +| **WaitingForMembers** | Run is initialized, coordinator is waiting for enough clients to connect before training can start. | "The run is waiting for participants. Not enough GPU nodes have joined yet." | +| **Warmup** | Sufficient clients have joined. Nodes are downloading the model and loading it onto GPUs. Training has not started. | "Nodes are preparing — downloading the model and loading weights onto GPUs. Training will begin shortly." | +| **RoundTrain** | Active training round. Each client trains on its assigned data batch using the shared seed and round/epoch indices. | "Training is actively running. Nodes are processing their assigned data batches." | +| **RoundWitness** | Validation phase. Witness nodes send proofs to the coordinator to verify training integrity. | "The network is verifying training results. Witness nodes are submitting proofs." | +| **Cooldown** | Final phase of an epoch. The coordinator waits for the cooldown period to elapse before starting the next epoch. | "This epoch is complete. The network is in cooldown before the next epoch begins." | +| **Paused** | Run has been paused by the coordinator via `run-manager set-paused`. Can be resumed. | "This run is currently paused by the coordinator. It may resume later." | + +## Transitions + +- **WaitingForMembers → Warmup**: Minimum client threshold met. +- **Warmup → RoundTrain**: All clients have downloaded the model and are ready. +- **RoundTrain → RoundWitness**: Training round complete, entering validation. +- **RoundWitness → RoundTrain**: Validation passed, starting next training round within the same epoch. +- **RoundWitness → Cooldown**: All rounds in the epoch are complete. +- **Cooldown → WaitingForMembers**: Cooldown period elapsed, starting new epoch. Clients may join or leave. +- **Any → Paused**: Coordinator pauses the run (admin action). +- **Paused → WaitingForMembers**: Coordinator resumes the run. + +## Diagnostic Guide + +When interpreting run state for the user: + +1. If the run is in **WaitingForMembers** for a long time → The run may need more participants, or the minimum threshold is high. +2. If the run is in **Warmup** for a long time → Large model download in progress (40B+ models can take significant time on slower connections). +3. If the run alternates rapidly between **RoundTrain** and **RoundWitness** → Normal operation. Training is progressing healthily. +4. If the run is **Paused** → Check the Psyche dashboard or Discord for announcements about why. +5. If you cannot determine the state → Use `run-manager json-dump-run` for the full on-chain state dump. From edca3d7b3d49a680e314f183567e4aab45fefa12 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Muhammet=20Eren=20Karaku=C5=9F?= Date: Sat, 28 Feb 2026 14:53:12 +0300 Subject: [PATCH 2/5] fix(skills): use createdAt instead of lastModified in model list endpoint The HuggingFace /models list endpoint returns createdAt but not lastModified. The lastModified field is only available on the individual model detail endpoint. Updated parsing examples accordingly. --- skills/psyche/network-monitor/SKILL.md | 6 +++--- skills/psyche/network-monitor/references/commands.md | 12 +++++++----- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/skills/psyche/network-monitor/SKILL.md b/skills/psyche/network-monitor/SKILL.md index 72908d9adfbb6..0d01000ab07f6 100644 --- a/skills/psyche/network-monitor/SKILL.md +++ b/skills/psyche/network-monitor/SKILL.md @@ -53,12 +53,12 @@ curl -s "https://huggingface.co/api/models?author=PsycheFoundation" | \ python3 -c " import sys, json models = json.load(sys.stdin) -for m in sorted(models, key=lambda x: x.get('lastModified',''), reverse=True): - print(f\"{m['id']:55s} modified: {m.get('lastModified','N/A')[:10]} downloads: {m.get('downloads',0)}\") +for m in sorted(models, key=lambda x: x.get('createdAt',''), reverse=True): + print(f\"{m['id']:55s} created: {m.get('createdAt','N/A')[:10]} downloads: {m.get('downloads',0)}\") " ``` -This returns all training run models with their last modification date. The most recently modified model is likely the currently active run. +This returns all training run models sorted by creation date. To get the last modification date for a specific model, use Step 2. ### Step 2: Get Run Details diff --git a/skills/psyche/network-monitor/references/commands.md b/skills/psyche/network-monitor/references/commands.md index 0b971bb2c30fd..c51eb00230233 100644 --- a/skills/psyche/network-monitor/references/commands.md +++ b/skills/psyche/network-monitor/references/commands.md @@ -15,11 +15,13 @@ curl -s "https://huggingface.co/api/models?author=PsycheFoundation" **Response:** JSON array. Each element contains: - `id` — Full model ID (e.g., `PsycheFoundation/consilience-40b-CqX3FUm4`) - `modelId` — Same as `id` -- `lastModified` — ISO 8601 timestamp +- `createdAt` — ISO 8601 timestamp - `downloads` — Total download count - `tags` — List of tags (architecture, library, etc.) - `pipeline_tag` — Model pipeline type +Note: `lastModified` is only available on the individual model detail endpoint (see below). + **Parse to table:** ```bash @@ -27,10 +29,10 @@ curl -s "https://huggingface.co/api/models?author=PsycheFoundation" | \ python3 -c " import sys, json models = json.load(sys.stdin) -print(f'{'Model ID':55s} {'Modified':12s} {'Downloads':>10s}') +print(f'{'Model ID':55s} {'Created':12s} {'Downloads':>10s}') print('-' * 82) -for m in sorted(models, key=lambda x: x.get('lastModified',''), reverse=True): - print(f\"{m['id']:55s} {m.get('lastModified','N/A')[:10]:12s} {m.get('downloads',0):>10d}\") +for m in sorted(models, key=lambda x: x.get('createdAt',''), reverse=True): + print(f\"{m['id']:55s} {m.get('createdAt','N/A')[:10]:12s} {m.get('downloads',0):>10d}\") print(f'\\nTotal: {len(models)} models') " ``` @@ -41,7 +43,7 @@ print(f'\\nTotal: {len(models)} models') curl -s "https://huggingface.co/api/models/PsycheFoundation/" ``` -**Response:** JSON object with full metadata including `siblings` (file list), `cardData`, `config`, etc. +**Response:** JSON object with full metadata including `lastModified`, `siblings` (file list), `cardData`, `config`, etc. ### List Files (Tree) From 65e3f8cb63a4859454c766fccd2cd0f23e4dbaca Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Muhammet=20Eren=20Karaku=C5=9F?= Date: Sat, 28 Feb 2026 14:56:42 +0300 Subject: [PATCH 3/5] fix(skills): fix syntax error, add missing states, clarify CLI invocation - Fix Python f-string quote collision in commands.md parse example - Add Uninitialized and Finished states to run-states.md (from Psyche glossary) - Clarify psyche-solana-client vs run-manager invocation pattern --- .../network-monitor/references/commands.md | 21 ++++++++++++-- .../network-monitor/references/run-states.md | 29 ++++++++++++------- 2 files changed, 37 insertions(+), 13 deletions(-) diff --git a/skills/psyche/network-monitor/references/commands.md b/skills/psyche/network-monitor/references/commands.md index c51eb00230233..3a8782d2f6a45 100644 --- a/skills/psyche/network-monitor/references/commands.md +++ b/skills/psyche/network-monitor/references/commands.md @@ -29,7 +29,7 @@ curl -s "https://huggingface.co/api/models?author=PsycheFoundation" | \ python3 -c " import sys, json models = json.load(sys.stdin) -print(f'{'Model ID':55s} {'Created':12s} {'Downloads':>10s}') +print(f'{\"Model ID\":55s} {\"Created\":12s} {\"Downloads\":>10s}') print('-' * 82) for m in sorted(models, key=lambda x: x.get('createdAt',''), reverse=True): print(f\"{m['id']:55s} {m.get('createdAt','N/A')[:10]:12s} {m.get('downloads',0):>10d}\") @@ -129,9 +129,9 @@ curl -s -X POST "${SOLANA_RPC_URL:-https://api.mainnet-beta.solana.com}" \ --- -## Psyche CLI (run-manager) +## Psyche CLI -Requires the Psyche client. Install via Docker or build from source. +The Psyche CLI binary is `psyche-solana-client`. The `run-manager` subcommand handles run management operations. Install via Docker or build from source. ### Install @@ -145,6 +145,21 @@ cd psyche cargo build --release --bin psyche-solana-client ``` +### Invocation + +```bash +# If built from source: +cargo run --release --bin psyche-solana-client -- run-manager + +# Or if the binary is in PATH: +psyche-solana-client run-manager + +# Docker: +docker run ghcr.io/psychefoundation/psyche run-manager +``` + +The examples below use the short form `run-manager` for readability. Replace with the full invocation above based on your setup. + ### Dump Run State ```bash diff --git a/skills/psyche/network-monitor/references/run-states.md b/skills/psyche/network-monitor/references/run-states.md index aa13c44198fea..0f5daa0e6f6b2 100644 --- a/skills/psyche/network-monitor/references/run-states.md +++ b/skills/psyche/network-monitor/references/run-states.md @@ -2,37 +2,44 @@ Canonical run state definitions from the Psyche Network coordinator program. Use these to interpret on-chain run data and advise users. -Source: https://docs.psyche.network/explain/general-workflow.html +Sources: +- https://docs.psyche.network/explain/general-workflow.html +- https://docs.psyche.network/explain/glossary.html ## State Machine ``` -WaitingForMembers → Warmup → RoundTrain ↔ RoundWitness → Cooldown - ↑ | - └────────────────── (next epoch) ────────────────────┘ +Uninitialized → WaitingForMembers → Warmup → RoundTrain ↔ RoundWitness → Cooldown + ↑ | + └────────────────── (next epoch) ────────────────────┘ + → Finished ``` -An epoch cycles through: WaitingForMembers → Warmup → (RoundTrain ↔ RoundWitness) × N rounds → Cooldown → back to WaitingForMembers for the next epoch. +Full lifecycle: Uninitialized → WaitingForMembers → Warmup → (RoundTrain ↔ RoundWitness) × N rounds → Cooldown → WaitingForMembers (next epoch) or Finished (training complete). ## State Reference | State | Meaning | What to Tell the User | |-------|---------|----------------------| +| **Uninitialized** | Default starting state. The run has been created on-chain but not yet configured or started. | "This run exists on-chain but has not been initialized yet." | | **WaitingForMembers** | Run is initialized, coordinator is waiting for enough clients to connect before training can start. | "The run is waiting for participants. Not enough GPU nodes have joined yet." | | **Warmup** | Sufficient clients have joined. Nodes are downloading the model and loading it onto GPUs. Training has not started. | "Nodes are preparing — downloading the model and loading weights onto GPUs. Training will begin shortly." | | **RoundTrain** | Active training round. Each client trains on its assigned data batch using the shared seed and round/epoch indices. | "Training is actively running. Nodes are processing their assigned data batches." | | **RoundWitness** | Validation phase. Witness nodes send proofs to the coordinator to verify training integrity. | "The network is verifying training results. Witness nodes are submitting proofs." | | **Cooldown** | Final phase of an epoch. The coordinator waits for the cooldown period to elapse before starting the next epoch. | "This epoch is complete. The network is in cooldown before the next epoch begins." | | **Paused** | Run has been paused by the coordinator via `run-manager set-paused`. Can be resumed. | "This run is currently paused by the coordinator. It may resume later." | +| **Finished** | Training run has completed all planned epochs. No further training will occur. | "This training run is complete. Check HuggingFace for the final model checkpoint." | ## Transitions +- **Uninitialized → WaitingForMembers**: Run configured and started by coordinator. - **WaitingForMembers → Warmup**: Minimum client threshold met. - **Warmup → RoundTrain**: All clients have downloaded the model and are ready. - **RoundTrain → RoundWitness**: Training round complete, entering validation. - **RoundWitness → RoundTrain**: Validation passed, starting next training round within the same epoch. - **RoundWitness → Cooldown**: All rounds in the epoch are complete. - **Cooldown → WaitingForMembers**: Cooldown period elapsed, starting new epoch. Clients may join or leave. +- **Cooldown → Finished**: All planned epochs complete. Training is done. - **Any → Paused**: Coordinator pauses the run (admin action). - **Paused → WaitingForMembers**: Coordinator resumes the run. @@ -40,8 +47,10 @@ An epoch cycles through: WaitingForMembers → Warmup → (RoundTrain ↔ RoundW When interpreting run state for the user: -1. If the run is in **WaitingForMembers** for a long time → The run may need more participants, or the minimum threshold is high. -2. If the run is in **Warmup** for a long time → Large model download in progress (40B+ models can take significant time on slower connections). -3. If the run alternates rapidly between **RoundTrain** and **RoundWitness** → Normal operation. Training is progressing healthily. -4. If the run is **Paused** → Check the Psyche dashboard or Discord for announcements about why. -5. If you cannot determine the state → Use `run-manager json-dump-run` for the full on-chain state dump. +1. If the run is **Uninitialized** → The run account exists but hasn't been started. It may be newly created or misconfigured. +2. If the run is in **WaitingForMembers** for a long time → The run may need more participants, or the minimum threshold is high. +3. If the run is in **Warmup** for a long time → Large model download in progress (40B+ models can take significant time on slower connections). +4. If the run alternates rapidly between **RoundTrain** and **RoundWitness** → Normal operation. Training is progressing healthily. +5. If the run is **Paused** → Check the Psyche dashboard or Discord for announcements about why. +6. If the run is **Finished** → Training is complete. Direct the user to HuggingFace for the final model. +7. If you cannot determine the state → Use `run-manager json-dump-run` for the full on-chain state dump. From c392dc9206e8bd6e424362880cfc3ad8030735cc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Muhammet=20Eren=20Karaku=C5=9F?= Date: Sun, 1 Mar 2026 12:42:45 +0300 Subject: [PATCH 4/5] fix(skills): use Psyche API as primary run discovery, HF only for checkpoint lookup MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Replace HuggingFace-based run discovery with psyche.network/api/runs as primary source - HuggingFace now used only for checkpoint search (best-effort, two-pass: org-scoped then broad) - Fix /api/runs parsing: response is {"runs":[...]} not a direct array - Fix status field handling: status is an object {"type":"active"} not a string - Add source labels ([tracked], [on-chain], [best-effort], [unverified]) for data provenance - Add context-dependent fallback chains (discovery vs detail) - Handle "API 200 but run not in list" scenario (allowlist filtering) - Document that Solana RPC is reference-only (Borsh parse not feasible for agent) - Remove incorrect "Finished → check HuggingFace" guidance from run-states - Add disambiguation rules for multiple HF matches in broad search --- skills/psyche/network-monitor/SKILL.md | 236 ++++++++++---- .../network-monitor/references/commands.md | 300 ++++++++++++------ .../network-monitor/references/run-states.md | 4 +- 3 files changed, 386 insertions(+), 154 deletions(-) diff --git a/skills/psyche/network-monitor/SKILL.md b/skills/psyche/network-monitor/SKILL.md index 0d01000ab07f6..907d792b3bd3a 100644 --- a/skills/psyche/network-monitor/SKILL.md +++ b/skills/psyche/network-monitor/SKILL.md @@ -1,7 +1,7 @@ --- name: network-monitor -description: Monitor Psyche decentralized AI training network — discover runs, track checkpoints, check mining pool status, and query on-chain state. No API keys required. -version: 1.0.0 +description: Monitor Psyche decentralized AI training network — discover runs via Psyche API, look up checkpoints on HuggingFace (best-effort), check system status, and query on-chain state via CLI. No API keys required. +version: 2.0.0 author: Eren Karakus license: MIT metadata: @@ -12,7 +12,7 @@ metadata: # Psyche Network Monitor -Monitor the Psyche decentralized AI training network built by Nous Research. Dynamically discover training runs, track model checkpoints, check mining pool status, and query on-chain state. +Monitor the Psyche decentralized AI training network built by Nous Research. Discover training runs, check system status, look up model checkpoints, and query on-chain state. **Zero dependencies. No API keys. Uses curl + python3 + existing Hermes tools.** @@ -20,107 +20,218 @@ Monitor the Psyche decentralized AI training network built by Nous Research. Dyn Psyche is a decentralized AI model training platform coordinated on the Solana blockchain. It distributes training across independent GPU nodes worldwide using DisTrO (distributed training optimization with ~3x bandwidth reduction) and Iroh-based P2P networking. Anyone can contribute compute or funds to training runs. +## Scope + +**This skill reports runs tracked by psyche.network by default.** + +The Psyche API uses an allowlist filter, so not all on-chain runs may appear (e.g., users' private runs, test runs). For full on-chain coverage of a specific run, the Psyche CLI (`run-manager`) is required with a known run-id. + +Always communicate this scope to the user when reporting results. + ## When to Use This skill should be loaded when: - User asks about Psyche Network, its training runs, or models - User wants to check the status of a Psyche training run - User asks about mining pool contributions or how to participate -- User wants to find or download Psyche model checkpoints from HuggingFace +- User wants to find or download Psyche model checkpoints ## Quick Reference -| Action | Command | -|--------|---------| -| List all runs/models | `curl -s "https://huggingface.co/api/models?author=PsycheFoundation"` | -| Get model details | `curl -s "https://huggingface.co/api/models/PsycheFoundation/"` | -| List checkpoint files | `curl -s "https://huggingface.co/api/models/PsycheFoundation//tree/main"` | -| On-chain run state | `run-manager json-dump-run --rpc --run-id ` | +| Action | Method | Source Label | +|--------|--------|-------------| +| List tracked runs | `curl -s "https://psyche.network/api/runs"` | `[tracked]` | +| System status | `curl -s "https://psyche.network/api/status"` | `[tracked]` | +| Checkpoint search (narrow) | `curl -s "https://huggingface.co/api/models?search=&author=PsycheFoundation"` | `[best-effort]` | +| Checkpoint search (broad) | `curl -s "https://huggingface.co/api/models?search="` | `[best-effort]` | +| On-chain run detail | `run-manager json-dump-run --run-id ` | `[on-chain]` | For detailed command syntax and parsing examples, see `references/commands.md`. For run state definitions, see `references/run-states.md`. +## Source Labels + +Always tag each piece of data with its source to prevent false certainty: + +| Label | Source | Reliability | +|-------|--------|------------| +| `[tracked]` | psyche.network/api/runs | High — official dashboard data | +| `[tracked:not-found]` | psyche.network/api/runs (run absent from list) | High — confirmed not in tracked set; may exist on-chain | +| `[on-chain]` | Psyche CLI (json-dump-run) | High — direct blockchain data | +| `[best-effort]` | HuggingFace search | Medium — checkpoint may not be found, false matches possible | +| `[unverified]` | web_search | Low — may be outdated, unverified | + +Examples: +- "moe-10b-a1b-8k-wsd-lr3e4-1t: active, 1T tokens `[tracked]`" +- "Checkpoint: PsycheFoundation/moe-10b on HuggingFace `[best-effort]`" +- "This run is not in the tracked list `[tracked:not-found]`. If you know the run-id, on-chain query via CLI is available." + ## Procedure -Follow this multi-run discovery workflow. Do not hardcode model names — always discover dynamically. +Follow this workflow. Do not hardcode model names or run IDs — always discover dynamically. -### Step 1: Discover All Runs +### Step 1: Run Discovery -Query the HuggingFace API to list all models under the PsycheFoundation organization: +Query the Psyche API to list all tracked training runs: ```bash -curl -s "https://huggingface.co/api/models?author=PsycheFoundation" | \ +curl -s "https://psyche.network/api/runs" | \ python3 -c " import sys, json -models = json.load(sys.stdin) -for m in sorted(models, key=lambda x: x.get('createdAt',''), reverse=True): - print(f\"{m['id']:55s} created: {m.get('createdAt','N/A')[:10]} downloads: {m.get('downloads',0)}\") +data = json.load(sys.stdin) +runs = data.get('runs', []) +print(f'{\"Run ID\":40s} {\"Status\":20s} {\"Arch\":15s}') +print('-' * 78) +for r in runs: + status = r.get('status', {}) + status_str = status.get('type', '?') if isinstance(status, dict) else str(status) + print(f\"{r.get('id','?'):40s} {status_str:20s} {r.get('arch','?'):15s}\") +print(f'\nTotal: {len(runs)} tracked runs') " ``` -This returns all training run models sorted by creation date. To get the last modification date for a specific model, use Step 2. +**Response shape:** `{"runs": [...], "totalTokens": "...", "totalTokensPerSecondActive": "...", "error": null}`. +Each run's `status` is an object like `{"type": "active"}` or `{"type": "completed", "at": {...}}` — extract `status.type` for the state string. + +This returns runs tracked by the Psyche dashboard. **Not all on-chain runs appear here** due to the allowlist filter. -### Step 2: Get Run Details +### Step 2: System Status (Optional) -Once you identify a model of interest, fetch its full metadata: +Check overall Psyche network health: ```bash -curl -s "https://huggingface.co/api/models/PsycheFoundation/" | \ +curl -s "https://psyche.network/api/status" | \ python3 -c " import sys, json -m = json.load(sys.stdin) -print(f\"Model: {m['id']}\") -print(f\"Tags: {', '.join(m.get('tags', []))}\") -print(f\"Downloads: {m.get('downloads', 0)}\") -print(f\"Last Modified: {m.get('lastModified', 'N/A')}\") -siblings = m.get('siblings', []) -print(f\"Files: {len(siblings)}\") -for s in siblings[:15]: - print(f\" {s['rfilename']}\") -if len(siblings) > 15: - print(f\" ... and {len(siblings)-15} more files\") +s = json.load(sys.stdin) +for k, v in s.items(): + print(f'{k}: {v}') " ``` -### Step 3: List Checkpoint Files +### Step 3: Run Details -To see all checkpoint files for a model (useful for downloading specific training snapshots): +From the Step 1 response, extract details for a specific run of interest. The `/api/runs` response wraps runs in `{"runs": [...]}`. Per-run fields include: `id`, `name`, `description`, `status` (object: `{"type": "active"}`, `{"type": "completed", "at": {...}}`, etc.), `arch`, `totalTokens`, `size`, `lastUpdate`, `trainingStep`. ```bash -curl -s "https://huggingface.co/api/models/PsycheFoundation//tree/main" | \ +curl -s "https://psyche.network/api/runs" | \ python3 -c " import sys, json -files = json.load(sys.stdin) -for f in files: - size_mb = f.get('size', 0) / 1024 / 1024 - print(f\"{f['path']:50s} {size_mb:>8.1f} MB\") +data = json.load(sys.stdin) +runs = data.get('runs', []) +target = '' +match = [r for r in runs if r.get('id') == target or r.get('name') == target] +if match: + print(json.dumps(match[0], indent=2)) +else: + print(f'Run \"{target}\" not found in tracked list.') + print('It may be a private/test run or outside the allowlist.') + print('If you have the run-id, try: run-manager json-dump-run --run-id ') " ``` -### Step 4: On-Chain State (Optional) +**If API returns 200 but the run is not in the list**, tell the user: +"This run is not in the tracked list `[tracked:not-found]`. It may be a private/test run or outside the allowlist. If you know the run-id, on-chain query via CLI is available." + +### Step 4: Checkpoint Lookup + +The `/api/runs` endpoint does **not** include checkpoint type information. Checkpoint types (Hub, P2P, Gcs, P2PGcs, Ephemeral) are only available on-chain. Use the following best-effort strategy to find checkpoints: + +#### 4a. Narrow Search (PsycheFoundation org) + +Search with run-id first (always present), then run-name as fallback (name may be empty): + +```bash +# Search by run-id (primary key) +curl -s "https://huggingface.co/api/models?search=&author=PsycheFoundation" + +# If no results, search by run-name (secondary key) +curl -s "https://huggingface.co/api/models?search=&author=PsycheFoundation" +``` + +#### 4b. Broad Search (org-unrestricted fallback) + +If the narrow search returns no results, some checkpoints may be uploaded to individual users' HuggingFace accounts rather than PsycheFoundation: + +```bash +# Broad search by run-id +curl -s "https://huggingface.co/api/models?search=" + +# If no results, broad search by run-name +curl -s "https://huggingface.co/api/models?search=" +``` + +If results are found outside PsycheFoundation, clearly indicate this to the user: +"Checkpoint found under `/` (not PsycheFoundation) `[best-effort]`" + +**Multiple matches:** The broad search may return multiple results. In this case: +- Prefer exact name matches +- If multiple candidates remain, list all and let the user choose +- Tag all results with `[best-effort]` + +#### 4c. CLI Checkpoint Backend (if run-id is known and CLI is available) + +```bash +run-manager json-dump-run \ + --rpc "${SOLANA_RPC_URL:-https://api.mainnet-beta.solana.com}" \ + --run-id +``` + +This reveals the checkpoint backend type (Hub, P2P, Gcs, P2PGcs, Ephemeral) from on-chain data. Tag results with `[on-chain]`. + +#### 4d. Not Found + +If no checkpoint is found through any method, tell the user: +"This run's checkpoints are not visible on HuggingFace. Checkpoints may be stored via P2P, GCS, or may be ephemeral. Check the Psyche CLI or dashboard for details." + +**Never say "checkpoint not found = run doesn't exist or is broken."** -If the user has the Psyche CLI installed or wants deeper on-chain data, query Solana directly: +### Step 5: On-Chain Deep Dive (Optional) + +If the user has the Psyche CLI installed and knows the run-id: ```bash -# Requires: psyche-solana-client (from https://github.com/PsycheFoundation/psyche) -run-manager json-dump-run --rpc "$SOLANA_RPC_URL" --run-id +# Full on-chain state dump +run-manager json-dump-run \ + --rpc "${SOLANA_RPC_URL:-https://api.mainnet-beta.solana.com}" \ + --run-id + +# User participation data +run-manager json-dump-user \ + --rpc "${SOLANA_RPC_URL:-https://api.mainnet-beta.solana.com}" \ + --run-id \ + --address ``` -If the CLI is not available, use the Solana JSON-RPC API as a fallback (see `references/commands.md`). +Tag results with `[on-chain]`. + +**Note:** Chain Mode requires a known run-id. The CLI cannot discover runs — there is no `run-manager list` command. ## Fallback Strategy -If the primary data source fails, follow this fallback chain: +Fallback order depends on context. The CLI requires a run-id, so it cannot help with run discovery. + +**Important:** The API may return 200 but not list a specific run (due to allowlist filtering). This is not the same as "API unreachable" but the same fallback applies. -1. **HuggingFace API** (primary) — Most reliable for model/checkpoint data -2. **Solana RPC** (secondary) — For on-chain run state and participant data -3. **Psyche CLI** (`run-manager`) — If installed locally, provides the richest run data -4. **Web search** — Last resort for general Psyche network status +### Run Discovery (run-id unknown) + +``` +psyche.network/api/runs (primary, tracked runs) + | if unreachable OR run not in list +web_search (last resort, non-deterministic) [unverified] +``` -If a source returns an error: -- HuggingFace 404 → The model ID may have changed. Re-run Step 1 to discover current models. -- HuggingFace timeout → Retry once, then fall back to `web_search` for cached info. -- Solana RPC timeout → Suggest the user set `$SOLANA_RPC_URL` to a private RPC endpoint. +### Run Detail (run-id known) + +``` +psyche.network/api/runs (primary, tracked runs) + | if unreachable OR run not in list +Psyche CLI: run-manager json-dump-run [on-chain] + | if CLI not installed +web_search (last resort, non-deterministic) [unverified] +``` + +**Why not Solana RPC as a fallback?** `getProgramAccounts` and `getAccountInfo` return raw Borsh-serialized binary data. PDA derivation (run-id to coordinator account) and Borsh deserialization are not practically feasible for an AI agent. Solana RPC commands are documented in `references/commands.md` as a reference only, not as a practical fallback. ## Participation Guide @@ -152,21 +263,26 @@ Setup: https://docs.psyche.network | Dashboard | https://psyche.network | | Documentation | https://docs.psyche.network | | GitHub | https://github.com/PsycheFoundation/psyche | -| Forum | https://forum.nousresearch.com | +| Forum | https://forum.psyche.network | | HuggingFace Org | https://huggingface.co/PsycheFoundation | | Discord | https://discord.gg/NousResearch | ## Pitfalls -- **Do not hardcode model names.** Always use Step 1 to discover current models. New runs and models are added regularly. +- **Do not hardcode model names or run IDs.** Always use Step 1 to discover current runs. New runs are added regularly. +- **HuggingFace does NOT equal all runs.** Only ~33% of tracked runs have HF checkpoints. A run existing on the Psyche API but not on HuggingFace is normal — checkpoints may use P2P, GCS, or be ephemeral. +- **API may return 200 but not list a run.** The backend uses an allowlist filter. Private, test, or untracked runs will not appear. This does not mean the run doesn't exist. +- **Checkpoint type varies per run.** Hub (HuggingFace), P2P, Gcs, P2PGcs, Ephemeral. The `/api/runs` endpoint does not expose checkpoint type — only on-chain data (via CLI) reveals this. - **Mining pool capacity.** The pool is frequently full. If `psyche.network` shows the pool as full, advise the user to check back later. - **Public Solana RPC rate limits.** The default `https://api.mainnet-beta.solana.com` has strict rate limits. For repeated queries, recommend a private RPC provider or set `$SOLANA_RPC_URL`. - **Checkpoint size.** Large models (40B+) have checkpoint files in the tens of GB. Warn the user about download size before initiating. -- **Testnet vs Mainnet.** Some runs may be on Solana devnet. Check the run documentation or dashboard for the correct network. +- **CLI cannot discover runs.** There is no `run-manager list` command. All CLI subcommands require a known run-id. ## Verification -After completing any query, verify the results: -- Model listing returns a JSON array with `id` fields → success -- Model details contain `siblings` (file list) → model exists and has uploaded checkpoints -- If zero models returned, re-check the API URL or use `web_search "PsycheFoundation HuggingFace"` as fallback +After completing any query, verify the results against these scenarios: + +1. **Run found on API, checkpoint on HF under PsycheFoundation** — Narrow search finds it. Report with `[tracked]` for run, `[best-effort]` for checkpoint. +2. **Run found on API, checkpoint on HF under another user** — Broad search finds it. Report the non-PsycheFoundation source explicitly with `[best-effort]`. +3. **Run found on API, no checkpoint on HF** — This is normal. Report the run as `[tracked]` and note: "Checkpoints may be P2P, GCS, or ephemeral. Not all runs publish to HuggingFace." +4. **Run not found on API** — Report `[tracked:not-found]` and suggest CLI if run-id is known. diff --git a/skills/psyche/network-monitor/references/commands.md b/skills/psyche/network-monitor/references/commands.md index 3a8782d2f6a45..0247583d50cce 100644 --- a/skills/psyche/network-monitor/references/commands.md +++ b/skills/psyche/network-monitor/references/commands.md @@ -2,136 +2,96 @@ Detailed command syntax and parsing examples for querying Psyche network data. -## HuggingFace API +--- + +## 1. Psyche API (Primary) -All endpoints are public, no authentication required. +The Psyche dashboard API is the primary data source for run discovery. All endpoints are public, no authentication required. -### List All Models +### List Tracked Runs ```bash -curl -s "https://huggingface.co/api/models?author=PsycheFoundation" +curl -s "https://psyche.network/api/runs" ``` -**Response:** JSON array. Each element contains: -- `id` — Full model ID (e.g., `PsycheFoundation/consilience-40b-CqX3FUm4`) -- `modelId` — Same as `id` -- `createdAt` — ISO 8601 timestamp -- `downloads` — Total download count -- `tags` — List of tags (architecture, library, etc.) -- `pipeline_tag` — Model pipeline type +**Response:** JSON object `{"runs": [...], "totalTokens": "...", "totalTokensPerSecondActive": "...", "error": null}`. + +Each run element contains: +- `id` — Run identifier (string) +- `name` — Human-readable run name (string, may be empty) +- `description` — Run description (string) +- `status` — Status object: `{"type": "active"}`, `{"type": "paused"}`, `{"type": "completed", "at": {...}}`, `{"type": "waitingForMembers"}`. Extract `status.type` for the state string. +- `arch` — Model architecture (HfDeepseek, HfLlama, HfAuto, Torchtitan) +- `totalTokens` — Total tokens for training +- `size` — Model size +- `lastUpdate` — Last update timestamp +- `trainingStep` — Current training step -Note: `lastModified` is only available on the individual model detail endpoint (see below). +**Note:** This endpoint uses an allowlist filter. Not all on-chain runs appear. Checkpoint type is **not** included. **Parse to table:** ```bash -curl -s "https://huggingface.co/api/models?author=PsycheFoundation" | \ +curl -s "https://psyche.network/api/runs" | \ python3 -c " import sys, json -models = json.load(sys.stdin) -print(f'{\"Model ID\":55s} {\"Created\":12s} {\"Downloads\":>10s}') -print('-' * 82) -for m in sorted(models, key=lambda x: x.get('createdAt',''), reverse=True): - print(f\"{m['id']:55s} {m.get('createdAt','N/A')[:10]:12s} {m.get('downloads',0):>10d}\") -print(f'\\nTotal: {len(models)} models') +data = json.load(sys.stdin) +runs = data.get('runs', []) +print(f'{\"Run ID\":40s} {\"Status\":20s} {\"Arch\":15s}') +print('-' * 78) +for r in runs: + status = r.get('status', {}) + status_str = status.get('type', '?') if isinstance(status, dict) else str(status) + print(f\"{r.get('id','?'):40s} {status_str:20s} {r.get('arch','?'):15s}\") +print(f'\nTotal: {len(runs)} tracked runs') " ``` -### Get Model Details - -```bash -curl -s "https://huggingface.co/api/models/PsycheFoundation/" -``` - -**Response:** JSON object with full metadata including `lastModified`, `siblings` (file list), `cardData`, `config`, etc. - -### List Files (Tree) +### System Status ```bash -curl -s "https://huggingface.co/api/models/PsycheFoundation//tree/main" +curl -s "https://psyche.network/api/status" ``` -**Response:** JSON array. Each element contains: -- `path` — File path relative to repo root -- `size` — File size in bytes -- `type` — `file` or `directory` -- `oid` — Git object ID (SHA) +**Response:** JSON object with system-level information including Coordinator Program ID and Mining Pool Program ID. -**Parse with size:** +**Parse:** ```bash -curl -s "https://huggingface.co/api/models/PsycheFoundation//tree/main" | \ +curl -s "https://psyche.network/api/status" | \ python3 -c " import sys, json -files = json.load(sys.stdin) -total = 0 -for f in sorted(files, key=lambda x: x.get('size',0), reverse=True): - size = f.get('size', 0) - total += size - if size > 1024*1024: - print(f\" {size/1024/1024:>10.1f} MB {f['path']}\") - else: - print(f\" {size/1024:>10.1f} KB {f['path']}\") -print(f'\\nTotal: {total/1024/1024/1024:.1f} GB across {len(files)} files') +s = json.load(sys.stdin) +for k, v in s.items(): + print(f'{k}: {v}') " ``` -### List Commits (Version History) +### Find Specific Run ```bash -curl -s "https://huggingface.co/api/models/PsycheFoundation//commits/main" | \ +curl -s "https://psyche.network/api/runs" | \ python3 -c " import sys, json -commits = json.load(sys.stdin) -for c in commits[:10]: - print(f\"{c['date'][:16]} {c['title']}\") +data = json.load(sys.stdin) +runs = data.get('runs', []) +target = '' +match = [r for r in runs if r.get('id') == target or r.get('name') == target] +if match: + print(json.dumps(match[0], indent=2)) +else: + print(f'Run \"{target}\" not found in tracked list.') + print('It may be a private/test run or outside the allowlist.') " ``` --- -## Solana RPC - -For on-chain queries. The default public endpoint has rate limits; for production use, set `$SOLANA_RPC_URL` to a private RPC provider. - -### Check Account Balance - -```bash -curl -s -X POST "${SOLANA_RPC_URL:-https://api.mainnet-beta.solana.com}" \ - -H "Content-Type: application/json" \ - -d '{ - "jsonrpc": "2.0", - "id": 1, - "method": "getBalance", - "params": [""] - }' | python3 -c " -import sys, json -r = json.load(sys.stdin) -lamports = r.get('result', {}).get('value', 0) -print(f'Balance: {lamports / 1e9:.4f} SOL') -" -``` - -### Get Account Info (Run State) - -```bash -curl -s -X POST "${SOLANA_RPC_URL:-https://api.mainnet-beta.solana.com}" \ - -H "Content-Type: application/json" \ - -d '{ - "jsonrpc": "2.0", - "id": 1, - "method": "getAccountInfo", - "params": ["", {"encoding": "base64"}] - }' -``` - -**Note:** The account data is binary (Borsh-serialized). For human-readable output, prefer the Psyche CLI `json-dump-run` command below. - ---- +## 2. Psyche CLI (On-Chain) -## Psyche CLI +The Psyche CLI binary is `psyche-solana-client`. The `run-manager` subcommand handles run queries. **Most subcommands require a known run-id** (see table below). -The Psyche CLI binary is `psyche-solana-client`. The `run-manager` subcommand handles run management operations. Install via Docker or build from source. +There is no `run-manager list` command — CLI cannot discover runs. ### Install @@ -160,6 +120,21 @@ docker run ghcr.io/psychefoundation/psyche run-manager The examples below use the short form `run-manager` for readability. Replace with the full invocation above based on your setup. +### Available Subcommands + +| Subcommand | Purpose | Requires run-id | +|-----------|---------|----------------| +| JsonDumpRun | Full on-chain run state dump | Yes | +| JsonDumpUser | User participation data | Yes + address | +| CanJoin | Check if a wallet can join a run | Yes | +| CreateRun | Create a new run (admin) | No | +| CloseRun | Close a run (admin) | Yes | +| SetPaused | Pause/resume a run (admin) | Yes | +| UpdateConfig | Update run config (admin) | Yes | +| Checkpoint | Trigger checkpoint (admin) | Yes | +| DownloadResults | Download training results | Yes | +| UploadData | Upload training data | Yes | + ### Dump Run State ```bash @@ -168,7 +143,7 @@ run-manager json-dump-run \ --run-id ``` -Returns full JSON dump of the run's on-chain state: participants, round info, epoch data, configuration. +Returns full JSON dump of the run's on-chain state: participants, round info, epoch data, configuration, and **checkpoint backend type** (Hub, P2P, Gcs, P2PGcs, Ephemeral). ### Dump User State @@ -199,3 +174,144 @@ run-manager set-paused \ ``` **Note:** Pause/resume requires the run coordinator's wallet. This is an admin operation. + +--- + +## 3. HuggingFace API (Checkpoint Search — Best Effort) + +HuggingFace is used **only for checkpoint lookup**, not for run discovery. All endpoints are public, no authentication required. + +**Important:** Not all runs publish checkpoints to HuggingFace. Checkpoint types include Hub (HF), P2P, Gcs, P2PGcs, and Ephemeral. Only Hub-type checkpoints appear on HuggingFace. + +### Narrow Search (PsycheFoundation org) + +Search by run-id first (always present), then run-name (may be empty): + +```bash +# By run-id (primary) +curl -s "https://huggingface.co/api/models?search=&author=PsycheFoundation" + +# By run-name (secondary, if run-id search returns no results) +curl -s "https://huggingface.co/api/models?search=&author=PsycheFoundation" +``` + +### Broad Search (org-unrestricted fallback) + +If narrow search returns no results, checkpoints may be under individual users' HuggingFace accounts: + +```bash +# By run-id (primary) +curl -s "https://huggingface.co/api/models?search=" + +# By run-name (secondary) +curl -s "https://huggingface.co/api/models?search=" +``` + +**Multiple matches:** Broad search may return unrelated models. Disambiguation rules: +- Prefer exact name match with run-id or run-name +- If multiple candidates, list all and let the user choose +- Tag all results with `[best-effort]` +- If found outside PsycheFoundation, note the actual org/user + +### Parse Model List + +```bash +curl -s "https://huggingface.co/api/models?search=&author=PsycheFoundation" | \ + python3 -c " +import sys, json +models = json.load(sys.stdin) +if not models: + print('No models found for this search query.') +else: + for m in sorted(models, key=lambda x: x.get('createdAt',''), reverse=True): + print(f\"{m['id']:55s} created: {m.get('createdAt','N/A')[:10]} downloads: {m.get('downloads',0)}\") + print(f'\nTotal: {len(models)} models') +" +``` + +### Get Model Details + +```bash +curl -s "https://huggingface.co/api/models/PsycheFoundation/" | \ + python3 -c " +import sys, json +m = json.load(sys.stdin) +print(f\"Model: {m['id']}\") +print(f\"Tags: {', '.join(m.get('tags', []))}\") +print(f\"Downloads: {m.get('downloads', 0)}\") +print(f\"Last Modified: {m.get('lastModified', 'N/A')}\") +siblings = m.get('siblings', []) +print(f\"Files: {len(siblings)}\") +for s in siblings[:15]: + print(f\" {s['rfilename']}\") +if len(siblings) > 15: + print(f\" ... and {len(siblings)-15} more files\") +" +``` + +### List Checkpoint Files (Tree) + +```bash +curl -s "https://huggingface.co/api/models/PsycheFoundation//tree/main" | \ + python3 -c " +import sys, json +files = json.load(sys.stdin) +total = 0 +for f in sorted(files, key=lambda x: x.get('size',0), reverse=True): + size = f.get('size', 0) + total += size + if size > 1024*1024: + print(f\" {size/1024/1024:>10.1f} MB {f['path']}\") + else: + print(f\" {size/1024:>10.1f} KB {f['path']}\") +print(f'\nTotal: {total/1024/1024/1024:.1f} GB across {len(files)} files') +" +``` + +--- + +## 4. Solana RPC (Reference Only) + +**These commands are documented for reference only.** Solana RPC returns raw Borsh-serialized binary data that requires PDA derivation and schema-specific deserialization. This is not practically feasible for an AI agent. **Prefer the Psyche CLI for on-chain queries.** + +For production use, set `$SOLANA_RPC_URL` to a private RPC provider. The default public endpoint (`https://api.mainnet-beta.solana.com`) has strict rate limits. + +### Check Account Balance + +```bash +curl -s -X POST "${SOLANA_RPC_URL:-https://api.mainnet-beta.solana.com}" \ + -H "Content-Type: application/json" \ + -d '{ + "jsonrpc": "2.0", + "id": 1, + "method": "getBalance", + "params": [""] + }' | python3 -c " +import sys, json +r = json.load(sys.stdin) +lamports = r.get('result', {}).get('value', 0) +print(f'Balance: {lamports / 1e9:.4f} SOL') +" +``` + +### Get Account Info + +```bash +curl -s -X POST "${SOLANA_RPC_URL:-https://api.mainnet-beta.solana.com}" \ + -H "Content-Type: application/json" \ + -d '{ + "jsonrpc": "2.0", + "id": 1, + "method": "getAccountInfo", + "params": ["", {"encoding": "base64"}] + }' +``` + +**Note:** The account data is Borsh-serialized binary. PDA derivation (run-id to coordinator account address) requires Solana cryptographic operations. For human-readable output, use the Psyche CLI `json-dump-run` command instead. + +### Known Program IDs + +| Program | Address | +|---------|---------| +| Coordinator | `4SHugWqSXwKE5fqDchkJcPEqnoZE22VYKtSTVm7axbT7` | +| Mining Pool | `PsyMP8fXEEMo2C6C84s8eXuRUrvzQnZyquyjipDRohf` | diff --git a/skills/psyche/network-monitor/references/run-states.md b/skills/psyche/network-monitor/references/run-states.md index 0f5daa0e6f6b2..40c15caf2a75c 100644 --- a/skills/psyche/network-monitor/references/run-states.md +++ b/skills/psyche/network-monitor/references/run-states.md @@ -28,7 +28,7 @@ Full lifecycle: Uninitialized → WaitingForMembers → Warmup → (RoundTrain | **RoundWitness** | Validation phase. Witness nodes send proofs to the coordinator to verify training integrity. | "The network is verifying training results. Witness nodes are submitting proofs." | | **Cooldown** | Final phase of an epoch. The coordinator waits for the cooldown period to elapse before starting the next epoch. | "This epoch is complete. The network is in cooldown before the next epoch begins." | | **Paused** | Run has been paused by the coordinator via `run-manager set-paused`. Can be resumed. | "This run is currently paused by the coordinator. It may resume later." | -| **Finished** | Training run has completed all planned epochs. No further training will occur. | "This training run is complete. Check HuggingFace for the final model checkpoint." | +| **Finished** | Training run has completed all planned epochs. No further training will occur. | "This training run is complete. Checkpoint source varies by run (HuggingFace, GCS, P2P, or ephemeral). Not all runs publish checkpoints to HuggingFace." | ## Transitions @@ -52,5 +52,5 @@ When interpreting run state for the user: 3. If the run is in **Warmup** for a long time → Large model download in progress (40B+ models can take significant time on slower connections). 4. If the run alternates rapidly between **RoundTrain** and **RoundWitness** → Normal operation. Training is progressing healthily. 5. If the run is **Paused** → Check the Psyche dashboard or Discord for announcements about why. -6. If the run is **Finished** → Training is complete. Direct the user to HuggingFace for the final model. +6. If the run is **Finished** → Training is complete. Checkpoint source varies by run — use the checkpoint lookup procedure (SKILL.md Step 4) to search HuggingFace (best-effort) or CLI for the checkpoint backend type. Not all runs publish to HuggingFace. 7. If you cannot determine the state → Use `run-manager json-dump-run` for the full on-chain state dump. From 3f07ff31e3e2d5508fc254071d6c5b12c3fe11a1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Muhammet=20Eren=20Karaku=C5=9F?= Date: Sun, 1 Mar 2026 12:50:15 +0300 Subject: [PATCH 5/5] fix(skills): correct "All subcommands" to "Most query subcommands" in pitfalls --- skills/psyche/network-monitor/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/skills/psyche/network-monitor/SKILL.md b/skills/psyche/network-monitor/SKILL.md index 907d792b3bd3a..4c61717979054 100644 --- a/skills/psyche/network-monitor/SKILL.md +++ b/skills/psyche/network-monitor/SKILL.md @@ -276,7 +276,7 @@ Setup: https://docs.psyche.network - **Mining pool capacity.** The pool is frequently full. If `psyche.network` shows the pool as full, advise the user to check back later. - **Public Solana RPC rate limits.** The default `https://api.mainnet-beta.solana.com` has strict rate limits. For repeated queries, recommend a private RPC provider or set `$SOLANA_RPC_URL`. - **Checkpoint size.** Large models (40B+) have checkpoint files in the tens of GB. Warn the user about download size before initiating. -- **CLI cannot discover runs.** There is no `run-manager list` command. All CLI subcommands require a known run-id. +- **CLI cannot discover runs.** There is no `run-manager list` command. Most CLI query subcommands require a known run-id. ## Verification