diff --git a/skills/psyche/DESCRIPTION.md b/skills/psyche/DESCRIPTION.md new file mode 100644 index 0000000000000..ccb817f555052 --- /dev/null +++ b/skills/psyche/DESCRIPTION.md @@ -0,0 +1,3 @@ +--- +description: Skills for monitoring and interacting with the Psyche decentralized AI training network — training runs, checkpoints, mining pool, and on-chain state via Solana. +--- diff --git a/skills/psyche/network-monitor/SKILL.md b/skills/psyche/network-monitor/SKILL.md new file mode 100644 index 0000000000000..4c61717979054 --- /dev/null +++ b/skills/psyche/network-monitor/SKILL.md @@ -0,0 +1,288 @@ +--- +name: network-monitor +description: Monitor Psyche decentralized AI training network — discover runs via Psyche API, look up checkpoints on HuggingFace (best-effort), check system status, and query on-chain state via CLI. No API keys required. +version: 2.0.0 +author: Eren Karakus +license: MIT +metadata: + hermes: + tags: [Psyche, Nous Research, Decentralized AI, Solana, Training, HuggingFace, Mining Pool] + related_skills: [] +--- + +# Psyche Network Monitor + +Monitor the Psyche decentralized AI training network built by Nous Research. Discover training runs, check system status, look up model checkpoints, and query on-chain state. + +**Zero dependencies. No API keys. Uses curl + python3 + existing Hermes tools.** + +## What is Psyche? + +Psyche is a decentralized AI model training platform coordinated on the Solana blockchain. It distributes training across independent GPU nodes worldwide using DisTrO (distributed training optimization with ~3x bandwidth reduction) and Iroh-based P2P networking. Anyone can contribute compute or funds to training runs. + +## Scope + +**This skill reports runs tracked by psyche.network by default.** + +The Psyche API uses an allowlist filter, so not all on-chain runs may appear (e.g., users' private runs, test runs). For full on-chain coverage of a specific run, the Psyche CLI (`run-manager`) is required with a known run-id. + +Always communicate this scope to the user when reporting results. + +## When to Use + +This skill should be loaded when: +- User asks about Psyche Network, its training runs, or models +- User wants to check the status of a Psyche training run +- User asks about mining pool contributions or how to participate +- User wants to find or download Psyche model checkpoints + +## Quick Reference + +| Action | Method | Source Label | +|--------|--------|-------------| +| List tracked runs | `curl -s "https://psyche.network/api/runs"` | `[tracked]` | +| System status | `curl -s "https://psyche.network/api/status"` | `[tracked]` | +| Checkpoint search (narrow) | `curl -s "https://huggingface.co/api/models?search=&author=PsycheFoundation"` | `[best-effort]` | +| Checkpoint search (broad) | `curl -s "https://huggingface.co/api/models?search="` | `[best-effort]` | +| On-chain run detail | `run-manager json-dump-run --run-id ` | `[on-chain]` | + +For detailed command syntax and parsing examples, see `references/commands.md`. +For run state definitions, see `references/run-states.md`. + +## Source Labels + +Always tag each piece of data with its source to prevent false certainty: + +| Label | Source | Reliability | +|-------|--------|------------| +| `[tracked]` | psyche.network/api/runs | High — official dashboard data | +| `[tracked:not-found]` | psyche.network/api/runs (run absent from list) | High — confirmed not in tracked set; may exist on-chain | +| `[on-chain]` | Psyche CLI (json-dump-run) | High — direct blockchain data | +| `[best-effort]` | HuggingFace search | Medium — checkpoint may not be found, false matches possible | +| `[unverified]` | web_search | Low — may be outdated, unverified | + +Examples: +- "moe-10b-a1b-8k-wsd-lr3e4-1t: active, 1T tokens `[tracked]`" +- "Checkpoint: PsycheFoundation/moe-10b on HuggingFace `[best-effort]`" +- "This run is not in the tracked list `[tracked:not-found]`. If you know the run-id, on-chain query via CLI is available." + +## Procedure + +Follow this workflow. Do not hardcode model names or run IDs — always discover dynamically. + +### Step 1: Run Discovery + +Query the Psyche API to list all tracked training runs: + +```bash +curl -s "https://psyche.network/api/runs" | \ + python3 -c " +import sys, json +data = json.load(sys.stdin) +runs = data.get('runs', []) +print(f'{\"Run ID\":40s} {\"Status\":20s} {\"Arch\":15s}') +print('-' * 78) +for r in runs: + status = r.get('status', {}) + status_str = status.get('type', '?') if isinstance(status, dict) else str(status) + print(f\"{r.get('id','?'):40s} {status_str:20s} {r.get('arch','?'):15s}\") +print(f'\nTotal: {len(runs)} tracked runs') +" +``` + +**Response shape:** `{"runs": [...], "totalTokens": "...", "totalTokensPerSecondActive": "...", "error": null}`. +Each run's `status` is an object like `{"type": "active"}` or `{"type": "completed", "at": {...}}` — extract `status.type` for the state string. + +This returns runs tracked by the Psyche dashboard. **Not all on-chain runs appear here** due to the allowlist filter. + +### Step 2: System Status (Optional) + +Check overall Psyche network health: + +```bash +curl -s "https://psyche.network/api/status" | \ + python3 -c " +import sys, json +s = json.load(sys.stdin) +for k, v in s.items(): + print(f'{k}: {v}') +" +``` + +### Step 3: Run Details + +From the Step 1 response, extract details for a specific run of interest. The `/api/runs` response wraps runs in `{"runs": [...]}`. Per-run fields include: `id`, `name`, `description`, `status` (object: `{"type": "active"}`, `{"type": "completed", "at": {...}}`, etc.), `arch`, `totalTokens`, `size`, `lastUpdate`, `trainingStep`. + +```bash +curl -s "https://psyche.network/api/runs" | \ + python3 -c " +import sys, json +data = json.load(sys.stdin) +runs = data.get('runs', []) +target = '' +match = [r for r in runs if r.get('id') == target or r.get('name') == target] +if match: + print(json.dumps(match[0], indent=2)) +else: + print(f'Run \"{target}\" not found in tracked list.') + print('It may be a private/test run or outside the allowlist.') + print('If you have the run-id, try: run-manager json-dump-run --run-id ') +" +``` + +**If API returns 200 but the run is not in the list**, tell the user: +"This run is not in the tracked list `[tracked:not-found]`. It may be a private/test run or outside the allowlist. If you know the run-id, on-chain query via CLI is available." + +### Step 4: Checkpoint Lookup + +The `/api/runs` endpoint does **not** include checkpoint type information. Checkpoint types (Hub, P2P, Gcs, P2PGcs, Ephemeral) are only available on-chain. Use the following best-effort strategy to find checkpoints: + +#### 4a. Narrow Search (PsycheFoundation org) + +Search with run-id first (always present), then run-name as fallback (name may be empty): + +```bash +# Search by run-id (primary key) +curl -s "https://huggingface.co/api/models?search=&author=PsycheFoundation" + +# If no results, search by run-name (secondary key) +curl -s "https://huggingface.co/api/models?search=&author=PsycheFoundation" +``` + +#### 4b. Broad Search (org-unrestricted fallback) + +If the narrow search returns no results, some checkpoints may be uploaded to individual users' HuggingFace accounts rather than PsycheFoundation: + +```bash +# Broad search by run-id +curl -s "https://huggingface.co/api/models?search=" + +# If no results, broad search by run-name +curl -s "https://huggingface.co/api/models?search=" +``` + +If results are found outside PsycheFoundation, clearly indicate this to the user: +"Checkpoint found under `/` (not PsycheFoundation) `[best-effort]`" + +**Multiple matches:** The broad search may return multiple results. In this case: +- Prefer exact name matches +- If multiple candidates remain, list all and let the user choose +- Tag all results with `[best-effort]` + +#### 4c. CLI Checkpoint Backend (if run-id is known and CLI is available) + +```bash +run-manager json-dump-run \ + --rpc "${SOLANA_RPC_URL:-https://api.mainnet-beta.solana.com}" \ + --run-id +``` + +This reveals the checkpoint backend type (Hub, P2P, Gcs, P2PGcs, Ephemeral) from on-chain data. Tag results with `[on-chain]`. + +#### 4d. Not Found + +If no checkpoint is found through any method, tell the user: +"This run's checkpoints are not visible on HuggingFace. Checkpoints may be stored via P2P, GCS, or may be ephemeral. Check the Psyche CLI or dashboard for details." + +**Never say "checkpoint not found = run doesn't exist or is broken."** + +### Step 5: On-Chain Deep Dive (Optional) + +If the user has the Psyche CLI installed and knows the run-id: + +```bash +# Full on-chain state dump +run-manager json-dump-run \ + --rpc "${SOLANA_RPC_URL:-https://api.mainnet-beta.solana.com}" \ + --run-id + +# User participation data +run-manager json-dump-user \ + --rpc "${SOLANA_RPC_URL:-https://api.mainnet-beta.solana.com}" \ + --run-id \ + --address +``` + +Tag results with `[on-chain]`. + +**Note:** Chain Mode requires a known run-id. The CLI cannot discover runs — there is no `run-manager list` command. + +## Fallback Strategy + +Fallback order depends on context. The CLI requires a run-id, so it cannot help with run discovery. + +**Important:** The API may return 200 but not list a specific run (due to allowlist filtering). This is not the same as "API unreachable" but the same fallback applies. + +### Run Discovery (run-id unknown) + +``` +psyche.network/api/runs (primary, tracked runs) + | if unreachable OR run not in list +web_search (last resort, non-deterministic) [unverified] +``` + +### Run Detail (run-id known) + +``` +psyche.network/api/runs (primary, tracked runs) + | if unreachable OR run not in list +Psyche CLI: run-manager json-dump-run [on-chain] + | if CLI not installed +web_search (last resort, non-deterministic) [unverified] +``` + +**Why not Solana RPC as a fallback?** `getProgramAccounts` and `getAccountInfo` return raw Borsh-serialized binary data. PDA derivation (run-id to coordinator account) and Borsh deserialization are not practically feasible for an AI agent. Solana RPC commands are documented in `references/commands.md` as a reference only, not as a practical fallback. + +## Participation Guide + +### Mining Pool (Financial Contribution) + +1. Install a Solana wallet (Phantom, Solflare, or `solana-keygen new`) +2. Navigate to https://psyche.network +3. Connect wallet and deposit SOL to the training pool +4. Pool capacity is limited — if full, check back periodically + +### Compute Contribution (GPU) + +Requires NVIDIA GPU with sufficient VRAM: +- Minimum: RTX 4090 (24 GB) +- Recommended: A100 (40/80 GB), H100 (80 GB) + +Setup: https://docs.psyche.network + +### Code Contribution + +- Psyche core: https://github.com/PsycheFoundation/psyche (Rust + TypeScript) +- Hermes Agent: https://github.com/NousResearch/hermes-agent +- Atropos RL: https://github.com/NousResearch/atropos + +## Key Links + +| Resource | URL | +|----------|-----| +| Dashboard | https://psyche.network | +| Documentation | https://docs.psyche.network | +| GitHub | https://github.com/PsycheFoundation/psyche | +| Forum | https://forum.psyche.network | +| HuggingFace Org | https://huggingface.co/PsycheFoundation | +| Discord | https://discord.gg/NousResearch | + +## Pitfalls + +- **Do not hardcode model names or run IDs.** Always use Step 1 to discover current runs. New runs are added regularly. +- **HuggingFace does NOT equal all runs.** Only ~33% of tracked runs have HF checkpoints. A run existing on the Psyche API but not on HuggingFace is normal — checkpoints may use P2P, GCS, or be ephemeral. +- **API may return 200 but not list a run.** The backend uses an allowlist filter. Private, test, or untracked runs will not appear. This does not mean the run doesn't exist. +- **Checkpoint type varies per run.** Hub (HuggingFace), P2P, Gcs, P2PGcs, Ephemeral. The `/api/runs` endpoint does not expose checkpoint type — only on-chain data (via CLI) reveals this. +- **Mining pool capacity.** The pool is frequently full. If `psyche.network` shows the pool as full, advise the user to check back later. +- **Public Solana RPC rate limits.** The default `https://api.mainnet-beta.solana.com` has strict rate limits. For repeated queries, recommend a private RPC provider or set `$SOLANA_RPC_URL`. +- **Checkpoint size.** Large models (40B+) have checkpoint files in the tens of GB. Warn the user about download size before initiating. +- **CLI cannot discover runs.** There is no `run-manager list` command. Most CLI query subcommands require a known run-id. + +## Verification + +After completing any query, verify the results against these scenarios: + +1. **Run found on API, checkpoint on HF under PsycheFoundation** — Narrow search finds it. Report with `[tracked]` for run, `[best-effort]` for checkpoint. +2. **Run found on API, checkpoint on HF under another user** — Broad search finds it. Report the non-PsycheFoundation source explicitly with `[best-effort]`. +3. **Run found on API, no checkpoint on HF** — This is normal. Report the run as `[tracked]` and note: "Checkpoints may be P2P, GCS, or ephemeral. Not all runs publish to HuggingFace." +4. **Run not found on API** — Report `[tracked:not-found]` and suggest CLI if run-id is known. diff --git a/skills/psyche/network-monitor/references/commands.md b/skills/psyche/network-monitor/references/commands.md new file mode 100644 index 0000000000000..0247583d50cce --- /dev/null +++ b/skills/psyche/network-monitor/references/commands.md @@ -0,0 +1,317 @@ +# Psyche Network Monitor — Command Reference + +Detailed command syntax and parsing examples for querying Psyche network data. + +--- + +## 1. Psyche API (Primary) + +The Psyche dashboard API is the primary data source for run discovery. All endpoints are public, no authentication required. + +### List Tracked Runs + +```bash +curl -s "https://psyche.network/api/runs" +``` + +**Response:** JSON object `{"runs": [...], "totalTokens": "...", "totalTokensPerSecondActive": "...", "error": null}`. + +Each run element contains: +- `id` — Run identifier (string) +- `name` — Human-readable run name (string, may be empty) +- `description` — Run description (string) +- `status` — Status object: `{"type": "active"}`, `{"type": "paused"}`, `{"type": "completed", "at": {...}}`, `{"type": "waitingForMembers"}`. Extract `status.type` for the state string. +- `arch` — Model architecture (HfDeepseek, HfLlama, HfAuto, Torchtitan) +- `totalTokens` — Total tokens for training +- `size` — Model size +- `lastUpdate` — Last update timestamp +- `trainingStep` — Current training step + +**Note:** This endpoint uses an allowlist filter. Not all on-chain runs appear. Checkpoint type is **not** included. + +**Parse to table:** + +```bash +curl -s "https://psyche.network/api/runs" | \ + python3 -c " +import sys, json +data = json.load(sys.stdin) +runs = data.get('runs', []) +print(f'{\"Run ID\":40s} {\"Status\":20s} {\"Arch\":15s}') +print('-' * 78) +for r in runs: + status = r.get('status', {}) + status_str = status.get('type', '?') if isinstance(status, dict) else str(status) + print(f\"{r.get('id','?'):40s} {status_str:20s} {r.get('arch','?'):15s}\") +print(f'\nTotal: {len(runs)} tracked runs') +" +``` + +### System Status + +```bash +curl -s "https://psyche.network/api/status" +``` + +**Response:** JSON object with system-level information including Coordinator Program ID and Mining Pool Program ID. + +**Parse:** + +```bash +curl -s "https://psyche.network/api/status" | \ + python3 -c " +import sys, json +s = json.load(sys.stdin) +for k, v in s.items(): + print(f'{k}: {v}') +" +``` + +### Find Specific Run + +```bash +curl -s "https://psyche.network/api/runs" | \ + python3 -c " +import sys, json +data = json.load(sys.stdin) +runs = data.get('runs', []) +target = '' +match = [r for r in runs if r.get('id') == target or r.get('name') == target] +if match: + print(json.dumps(match[0], indent=2)) +else: + print(f'Run \"{target}\" not found in tracked list.') + print('It may be a private/test run or outside the allowlist.') +" +``` + +--- + +## 2. Psyche CLI (On-Chain) + +The Psyche CLI binary is `psyche-solana-client`. The `run-manager` subcommand handles run queries. **Most subcommands require a known run-id** (see table below). + +There is no `run-manager list` command — CLI cannot discover runs. + +### Install + +```bash +# Option 1: Docker +docker pull ghcr.io/psychefoundation/psyche:latest + +# Option 2: Build from source +git clone https://github.com/PsycheFoundation/psyche +cd psyche +cargo build --release --bin psyche-solana-client +``` + +### Invocation + +```bash +# If built from source: +cargo run --release --bin psyche-solana-client -- run-manager + +# Or if the binary is in PATH: +psyche-solana-client run-manager + +# Docker: +docker run ghcr.io/psychefoundation/psyche run-manager +``` + +The examples below use the short form `run-manager` for readability. Replace with the full invocation above based on your setup. + +### Available Subcommands + +| Subcommand | Purpose | Requires run-id | +|-----------|---------|----------------| +| JsonDumpRun | Full on-chain run state dump | Yes | +| JsonDumpUser | User participation data | Yes + address | +| CanJoin | Check if a wallet can join a run | Yes | +| CreateRun | Create a new run (admin) | No | +| CloseRun | Close a run (admin) | Yes | +| SetPaused | Pause/resume a run (admin) | Yes | +| UpdateConfig | Update run config (admin) | Yes | +| Checkpoint | Trigger checkpoint (admin) | Yes | +| DownloadResults | Download training results | Yes | +| UploadData | Upload training data | Yes | + +### Dump Run State + +```bash +run-manager json-dump-run \ + --rpc "${SOLANA_RPC_URL:-https://api.mainnet-beta.solana.com}" \ + --run-id +``` + +Returns full JSON dump of the run's on-chain state: participants, round info, epoch data, configuration, and **checkpoint backend type** (Hub, P2P, Gcs, P2PGcs, Ephemeral). + +### Dump User State + +```bash +run-manager json-dump-user \ + --rpc "${SOLANA_RPC_URL:-https://api.mainnet-beta.solana.com}" \ + --run-id \ + --address +``` + +Returns the user's participation data: collateral, rewards, training history. + +### Pause / Resume a Run + +```bash +# Pause +run-manager set-paused \ + --rpc \ + --run-id \ + --wallet-private-key-path + +# Resume +run-manager set-paused \ + --rpc \ + --run-id \ + --resume \ + --wallet-private-key-path +``` + +**Note:** Pause/resume requires the run coordinator's wallet. This is an admin operation. + +--- + +## 3. HuggingFace API (Checkpoint Search — Best Effort) + +HuggingFace is used **only for checkpoint lookup**, not for run discovery. All endpoints are public, no authentication required. + +**Important:** Not all runs publish checkpoints to HuggingFace. Checkpoint types include Hub (HF), P2P, Gcs, P2PGcs, and Ephemeral. Only Hub-type checkpoints appear on HuggingFace. + +### Narrow Search (PsycheFoundation org) + +Search by run-id first (always present), then run-name (may be empty): + +```bash +# By run-id (primary) +curl -s "https://huggingface.co/api/models?search=&author=PsycheFoundation" + +# By run-name (secondary, if run-id search returns no results) +curl -s "https://huggingface.co/api/models?search=&author=PsycheFoundation" +``` + +### Broad Search (org-unrestricted fallback) + +If narrow search returns no results, checkpoints may be under individual users' HuggingFace accounts: + +```bash +# By run-id (primary) +curl -s "https://huggingface.co/api/models?search=" + +# By run-name (secondary) +curl -s "https://huggingface.co/api/models?search=" +``` + +**Multiple matches:** Broad search may return unrelated models. Disambiguation rules: +- Prefer exact name match with run-id or run-name +- If multiple candidates, list all and let the user choose +- Tag all results with `[best-effort]` +- If found outside PsycheFoundation, note the actual org/user + +### Parse Model List + +```bash +curl -s "https://huggingface.co/api/models?search=&author=PsycheFoundation" | \ + python3 -c " +import sys, json +models = json.load(sys.stdin) +if not models: + print('No models found for this search query.') +else: + for m in sorted(models, key=lambda x: x.get('createdAt',''), reverse=True): + print(f\"{m['id']:55s} created: {m.get('createdAt','N/A')[:10]} downloads: {m.get('downloads',0)}\") + print(f'\nTotal: {len(models)} models') +" +``` + +### Get Model Details + +```bash +curl -s "https://huggingface.co/api/models/PsycheFoundation/" | \ + python3 -c " +import sys, json +m = json.load(sys.stdin) +print(f\"Model: {m['id']}\") +print(f\"Tags: {', '.join(m.get('tags', []))}\") +print(f\"Downloads: {m.get('downloads', 0)}\") +print(f\"Last Modified: {m.get('lastModified', 'N/A')}\") +siblings = m.get('siblings', []) +print(f\"Files: {len(siblings)}\") +for s in siblings[:15]: + print(f\" {s['rfilename']}\") +if len(siblings) > 15: + print(f\" ... and {len(siblings)-15} more files\") +" +``` + +### List Checkpoint Files (Tree) + +```bash +curl -s "https://huggingface.co/api/models/PsycheFoundation//tree/main" | \ + python3 -c " +import sys, json +files = json.load(sys.stdin) +total = 0 +for f in sorted(files, key=lambda x: x.get('size',0), reverse=True): + size = f.get('size', 0) + total += size + if size > 1024*1024: + print(f\" {size/1024/1024:>10.1f} MB {f['path']}\") + else: + print(f\" {size/1024:>10.1f} KB {f['path']}\") +print(f'\nTotal: {total/1024/1024/1024:.1f} GB across {len(files)} files') +" +``` + +--- + +## 4. Solana RPC (Reference Only) + +**These commands are documented for reference only.** Solana RPC returns raw Borsh-serialized binary data that requires PDA derivation and schema-specific deserialization. This is not practically feasible for an AI agent. **Prefer the Psyche CLI for on-chain queries.** + +For production use, set `$SOLANA_RPC_URL` to a private RPC provider. The default public endpoint (`https://api.mainnet-beta.solana.com`) has strict rate limits. + +### Check Account Balance + +```bash +curl -s -X POST "${SOLANA_RPC_URL:-https://api.mainnet-beta.solana.com}" \ + -H "Content-Type: application/json" \ + -d '{ + "jsonrpc": "2.0", + "id": 1, + "method": "getBalance", + "params": [""] + }' | python3 -c " +import sys, json +r = json.load(sys.stdin) +lamports = r.get('result', {}).get('value', 0) +print(f'Balance: {lamports / 1e9:.4f} SOL') +" +``` + +### Get Account Info + +```bash +curl -s -X POST "${SOLANA_RPC_URL:-https://api.mainnet-beta.solana.com}" \ + -H "Content-Type: application/json" \ + -d '{ + "jsonrpc": "2.0", + "id": 1, + "method": "getAccountInfo", + "params": ["", {"encoding": "base64"}] + }' +``` + +**Note:** The account data is Borsh-serialized binary. PDA derivation (run-id to coordinator account address) requires Solana cryptographic operations. For human-readable output, use the Psyche CLI `json-dump-run` command instead. + +### Known Program IDs + +| Program | Address | +|---------|---------| +| Coordinator | `4SHugWqSXwKE5fqDchkJcPEqnoZE22VYKtSTVm7axbT7` | +| Mining Pool | `PsyMP8fXEEMo2C6C84s8eXuRUrvzQnZyquyjipDRohf` | diff --git a/skills/psyche/network-monitor/references/run-states.md b/skills/psyche/network-monitor/references/run-states.md new file mode 100644 index 0000000000000..40c15caf2a75c --- /dev/null +++ b/skills/psyche/network-monitor/references/run-states.md @@ -0,0 +1,56 @@ +# Psyche Run States + +Canonical run state definitions from the Psyche Network coordinator program. Use these to interpret on-chain run data and advise users. + +Sources: +- https://docs.psyche.network/explain/general-workflow.html +- https://docs.psyche.network/explain/glossary.html + +## State Machine + +``` +Uninitialized → WaitingForMembers → Warmup → RoundTrain ↔ RoundWitness → Cooldown + ↑ | + └────────────────── (next epoch) ────────────────────┘ + → Finished +``` + +Full lifecycle: Uninitialized → WaitingForMembers → Warmup → (RoundTrain ↔ RoundWitness) × N rounds → Cooldown → WaitingForMembers (next epoch) or Finished (training complete). + +## State Reference + +| State | Meaning | What to Tell the User | +|-------|---------|----------------------| +| **Uninitialized** | Default starting state. The run has been created on-chain but not yet configured or started. | "This run exists on-chain but has not been initialized yet." | +| **WaitingForMembers** | Run is initialized, coordinator is waiting for enough clients to connect before training can start. | "The run is waiting for participants. Not enough GPU nodes have joined yet." | +| **Warmup** | Sufficient clients have joined. Nodes are downloading the model and loading it onto GPUs. Training has not started. | "Nodes are preparing — downloading the model and loading weights onto GPUs. Training will begin shortly." | +| **RoundTrain** | Active training round. Each client trains on its assigned data batch using the shared seed and round/epoch indices. | "Training is actively running. Nodes are processing their assigned data batches." | +| **RoundWitness** | Validation phase. Witness nodes send proofs to the coordinator to verify training integrity. | "The network is verifying training results. Witness nodes are submitting proofs." | +| **Cooldown** | Final phase of an epoch. The coordinator waits for the cooldown period to elapse before starting the next epoch. | "This epoch is complete. The network is in cooldown before the next epoch begins." | +| **Paused** | Run has been paused by the coordinator via `run-manager set-paused`. Can be resumed. | "This run is currently paused by the coordinator. It may resume later." | +| **Finished** | Training run has completed all planned epochs. No further training will occur. | "This training run is complete. Checkpoint source varies by run (HuggingFace, GCS, P2P, or ephemeral). Not all runs publish checkpoints to HuggingFace." | + +## Transitions + +- **Uninitialized → WaitingForMembers**: Run configured and started by coordinator. +- **WaitingForMembers → Warmup**: Minimum client threshold met. +- **Warmup → RoundTrain**: All clients have downloaded the model and are ready. +- **RoundTrain → RoundWitness**: Training round complete, entering validation. +- **RoundWitness → RoundTrain**: Validation passed, starting next training round within the same epoch. +- **RoundWitness → Cooldown**: All rounds in the epoch are complete. +- **Cooldown → WaitingForMembers**: Cooldown period elapsed, starting new epoch. Clients may join or leave. +- **Cooldown → Finished**: All planned epochs complete. Training is done. +- **Any → Paused**: Coordinator pauses the run (admin action). +- **Paused → WaitingForMembers**: Coordinator resumes the run. + +## Diagnostic Guide + +When interpreting run state for the user: + +1. If the run is **Uninitialized** → The run account exists but hasn't been started. It may be newly created or misconfigured. +2. If the run is in **WaitingForMembers** for a long time → The run may need more participants, or the minimum threshold is high. +3. If the run is in **Warmup** for a long time → Large model download in progress (40B+ models can take significant time on slower connections). +4. If the run alternates rapidly between **RoundTrain** and **RoundWitness** → Normal operation. Training is progressing healthily. +5. If the run is **Paused** → Check the Psyche dashboard or Discord for announcements about why. +6. If the run is **Finished** → Training is complete. Checkpoint source varies by run — use the checkpoint lookup procedure (SKILL.md Step 4) to search HuggingFace (best-effort) or CLI for the checkpoint backend type. Not all runs publish to HuggingFace. +7. If you cannot determine the state → Use `run-manager json-dump-run` for the full on-chain state dump.