docs(rl): restore missing RL Training docs page

website/docs/getting-started/learning-path.md

@@ -28,7 +28,7 @@ First-time users almost always want `hermes setup --portal` — one OAuth covers
 |---|---|---|---|
 | **Beginner** | Get up and running, have basic conversations, use built-in tools | [Installation](/getting-started/installation) → [Quickstart](/getting-started/quickstart) → [CLI Usage](/user-guide/cli) → [Configuration](/user-guide/configuration) | ~1 hour |
 | **Intermediate** | Set up messaging bots, use advanced features like memory, cron jobs, and skills | [Sessions](/user-guide/sessions) → [Messaging](/user-guide/messaging) → [Tools](/user-guide/features/tools) → [Skills](/user-guide/features/skills) → [Memory](/user-guide/features/memory) → [Cron](/user-guide/features/cron) | ~2–3 hours |
-| **Advanced** | Build custom tools, create skills, train models with RL, contribute to the project | [Architecture](/developer-guide/architecture) → [Adding Tools](/developer-guide/adding-tools) → [Creating Skills](/developer-guide/creating-skills) → [Contributing](/developer-guide/contributing) | ~4–6 hours |
+| **Advanced** | Build custom tools, create skills, train models with RL, contribute to the project | [Architecture](/developer-guide/architecture) → [Adding Tools](/developer-guide/adding-tools) → [Creating Skills](/developer-guide/creating-skills) → [RL Training](/user-guide/features/rl-training) → [Contributing](/developer-guide/contributing) | ~4–6 hours |
 
 ## By Use Case
 
@@ -104,9 +104,10 @@ Use reinforcement learning to fine-tune model behavior with Hermes Agent's RL tr
 
 1. [Quickstart](/getting-started/quickstart)
 2. [Configuration](/user-guide/configuration)
-3. [Atropos RL Environments](https://github.com/NousResearch/atropos) (external)
-4. [Provider Routing](/user-guide/features/provider-routing)
-5. [Architecture](/developer-guide/architecture)
+3. [RL Training](/user-guide/features/rl-training)
+4. [Atropos RL Environments](https://github.com/NousResearch/atropos) (external)
+5. [Provider Routing](/user-guide/features/provider-routing)
+6. [Architecture](/developer-guide/architecture)
 
 :::tip
 RL training works best when you already understand the basics of how Hermes Agent handles conversations and tool calls. Run through the Beginner path first if you're new.
@@ -140,6 +141,7 @@ Not sure what's available? Here's a quick directory of major features:
 | **Browser** | Web browsing and scraping | [Browser](/user-guide/features/browser) |
 | **Hooks** | Event-driven callbacks and middleware | [Hooks](/user-guide/features/hooks) |
 | **Batch Processing** | Process multiple inputs in bulk | [Batch Processing](/user-guide/features/batch-processing) |
+| **RL Training** | Fine-tune models with reinforcement learning | [RL Training](/user-guide/features/rl-training) |
 | **Provider Routing** | Route requests across multiple LLM providers | [Provider Routing](/user-guide/features/provider-routing) |
 
 ## What to Read Next

website/docs/integrations/index.md

@@ -106,4 +106,5 @@ See the [Messaging Gateway overview](/user-guide/messaging) for the platform com
 
 ## Training & Evaluation
 
+- **[RL Training](/user-guide/features/rl-training)** — Generate trajectory data from agent sessions for reinforcement learning and model fine-tuning. Supports Atropos environments with customizable reward functions.
 - **[Batch Processing](/user-guide/features/batch-processing)** — Run the agent across hundreds of prompts in parallel, generating structured ShareGPT-format trajectory data for training data generation or evaluation.

website/docs/user-guide/features/overview.md

@@ -47,6 +47,7 @@ Hermes Agent includes a rich set of capabilities that extend far beyond basic ch
 - **[Memory Providers](memory-providers.md)** — Plug in external memory backends (Honcho, OpenViking, Mem0, Hindsight, Holographic, RetainDB, ByteRover, Supermemory) for cross-session user modeling and personalization beyond the built-in memory system.
 - **[API Server](api-server.md)** — Expose Hermes as an OpenAI-compatible HTTP endpoint. Connect any frontend that speaks the OpenAI format — Open WebUI, LobeChat, LibreChat, and more.
 - **[IDE Integration (ACP)](acp.md)** — Use Hermes inside ACP-compatible editors such as VS Code, Zed, and JetBrains. Chat, tool activity, file diffs, and terminal commands render inside your editor.
+- **[RL Training](/user-guide/features/rl-training)** — Generate trajectory data from agent sessions for reinforcement learning and model fine-tuning.
 - **[Batch Processing](batch-processing.md)** — Run the agent over many prompts or tasks in parallel from the CLI, with structured outputs and trajectory capture suitable for evals or downstream training pipelines.
 
 ## Customization

website/docs/user-guide/features/rl-training.md

@@ -0,0 +1,234 @@
+---
+sidebar_position: 13
+title: "RL Training"
+description: "Reinforcement learning on agent behaviors with Tinker-Atropos — environment discovery, training, and evaluation"
+---
+
+# RL Training
+
+Hermes Agent includes an integrated RL (Reinforcement Learning) training pipeline built on **Tinker-Atropos**. This enables training language models on environment-specific tasks using GRPO (Group Relative Policy Optimization) with LoRA adapters, orchestrated entirely through the agent's tool interface.
+
+## Overview
+
+The RL training system consists of three components:
+
+1. **Atropos** — A trajectory API server that coordinates environment interactions, manages rollout groups, and computes advantages
+2. **Tinker** — A training service that handles model weights, LoRA training, sampling/inference, and optimizer steps
+3. **Environments** — Python classes that define tasks, scoring, and reward functions (e.g., GSM8K math problems)
+
+The agent can discover environments, configure training parameters, launch training runs, and monitor metrics — all through a set of `rl_*` tools.
+
+## Requirements
+
+RL training requires:
+
+- **Python >= 3.11** (Tinker package requirement)
+- **TINKER_API_KEY** — API key for the Tinker training service
+- **WANDB_API_KEY** — API key for Weights & Biases metrics tracking
+- The `tinker-atropos` submodule (at `tinker-atropos/` relative to the Hermes root)
+
+```bash
+# Set up API keys
+hermes config set TINKER_API_KEY your-tinker-key
+hermes config set WANDB_API_KEY your-wandb-key
+```
+
+When both keys are present and Python >= 3.11 is available, the `rl` toolset is automatically enabled.
+
+## Available Tools
+
+| Tool | Description |
+|------|-------------|
+| `rl_list_environments` | Discover available RL environments |
+| `rl_select_environment` | Select an environment and load its config |
+| `rl_get_current_config` | View configurable and locked fields |
+| `rl_edit_config` | Modify configurable training parameters |
+| `rl_start_training` | Launch a training run (spawns 3 processes) |
+| `rl_check_status` | Monitor training progress and WandB metrics |
+| `rl_stop_training` | Stop a running training job |
+| `rl_get_results` | Get final metrics and model weights path |
+| `rl_list_runs` | List all active and completed runs |
+| `rl_test_inference` | Quick inference test using OpenRouter |
+
+## Workflow
+
+### 1. Discover Environments
+
+```
+List the available RL environments
+```
+
+The agent calls `rl_list_environments()` which scans `tinker-atropos/tinker_atropos/environments/` using AST parsing to find Python classes inheriting from `BaseEnv`. Each environment defines:
+
+- **Dataset loading** — where training data comes from (e.g., HuggingFace datasets)
+- **Prompt construction** — how to format items for the model
+- **Scoring/verification** — how to evaluate model outputs and assign rewards
+
+### 2. Select and Configure
+
+```
+Select the GSM8K environment and show me the configuration
+```
+
+The agent calls `rl_select_environment("gsm8k_tinker")`, then `rl_get_current_config()` to see all parameters.
+
+Configuration fields are divided into two categories:
+
+**Configurable fields** (can be modified):
+- `group_size` — Number of completions per item (default: 16)
+- `batch_size` — Training batch size (default: 128)
+- `wandb_name` — WandB run name (auto-set to `{env}-{timestamp}`)
+- Other environment-specific parameters
+
+**Locked fields** (infrastructure settings, cannot be changed):
+- `tokenizer_name` — Model tokenizer (e.g., `Qwen/Qwen3-8B`)
+- `rollout_server_url` — Atropos API URL (`http://localhost:8000`)
+- `max_token_length` — Maximum token length (8192)
+- `max_num_workers` — Maximum parallel workers (2048)
+- `total_steps` — Total training steps (2500)
+- `lora_rank` — LoRA adapter rank (32)
+- `learning_rate` — Learning rate (4e-5)
+- `max_token_trainer_length` — Max tokens for trainer (9000)
+
+### 3. Start Training
+
+```
+Start the training run
+```
+
+The agent calls `rl_start_training()` which:
+
+1. Generates a YAML config file merging locked settings with configurable overrides
+2. Creates a unique run ID
+3. Spawns three processes:
+   - **Atropos API server** (`run-api`) — trajectory coordination
+   - **Tinker trainer** (`launch_training.py`) — LoRA training + FastAPI inference server on port 8001
+   - **Environment** (`environment.py serve`) — the selected environment connecting to Atropos
+
+The processes start with staggered delays (5s for API, 30s for trainer, 90s more for environment) to ensure proper initialization order.
+
+### 4. Monitor Progress
+
+```
+Check the status of training run abc12345
+```
+
+The agent calls `rl_check_status(run_id)` which reports:
+
+- Process status (running/exited for each of the 3 processes)
+- Running time
+- WandB metrics (step, reward mean, percent correct, eval accuracy)
+- Log file locations for debugging
+
+:::note Rate Limiting
+Status checks are rate-limited to once every **30 minutes** per run ID. This prevents excessive polling during long-running training jobs that take hours.
+:::
+
+### 5. Stop or Get Results
+
+```
+Stop the training run
+# or
+Get the final results for run abc12345
+```
+
+`rl_stop_training()` terminates all three processes in reverse order (environment → trainer → API). `rl_get_results()` retrieves final WandB metrics and training history.
+
+## Inference Testing
+
+Before committing to a full training run, you can test if an environment works correctly using `rl_test_inference`. This runs a few steps of inference and scoring using OpenRouter — no Tinker API needed, just an `OPENROUTER_API_KEY`.
+
+```
+Test the selected environment with inference
+```
+
+Default configuration:
+- **3 steps × 16 completions = 48 rollouts per model**
+- Tests 3 models at different scales for robustness:
+  - `qwen/qwen3-8b` (small)
+  - `z-ai/glm-4.7-flash` (medium)
+  - `minimax/minimax-m2.7` (large)
+- Total: ~144 rollouts
+
+This validates:
+- Environment loads correctly
+- Prompt construction works
+- Inference response parsing is robust across model scales
+- Verifier/scoring logic produces valid rewards
+
+## Tinker API Integration
+
+The trainer uses the [Tinker](https://tinker.computer) API for model training operations:
+
+- **ServiceClient** — Creates training and sampling clients
+- **Training client** — Handles forward-backward passes with importance sampling loss, optimizer steps (Adam), and weight checkpointing
+- **Sampling client** — Provides inference using the latest trained weights
+
+The training loop:
+1. Fetches a batch of rollouts from Atropos (prompt + completions + scores)
+2. Converts to Tinker Datum objects with padded logprobs and advantages
+3. Runs forward-backward pass with importance sampling loss
+4. Takes an optimizer step (Adam: lr=4e-5, β1=0.9, β2=0.95)
+5. Saves weights and creates a new sampling client for next-step inference
+6. Logs metrics to WandB
+
+## Architecture Diagram
+
+```mermaid
+flowchart LR
+    api["Atropos API<br/>run-api<br/>port 8000"]
+    env["Environment<br/>BaseEnv implementation"]
+    infer["OpenAI / sglang<br/>inference API<br/>port 8001"]
+    trainer["Tinker Trainer<br/>LoRA training + FastAPI"]
+
+    env <--> api
+    env --> infer
+    api -->|"batches: tokens, scores, logprobs"| trainer
+    trainer -->|"serves inference"| infer
+```
+
+## Creating Custom Environments
+
+To create a new RL environment:
+
+1. Create a Python file in `tinker-atropos/tinker_atropos/environments/`
+2. Define a class that inherits from `BaseEnv`
+3. Implement the required methods:
+   - `load_dataset()` — Load your training data
+   - `get_next_item()` — Provide the next item to the model
+   - `score_answer()` — Score model outputs and assign rewards
+   - `collect_trajectories()` — Collect and return trajectories
+4. Optionally define a custom config class inheriting from `BaseEnvConfig`
+
+Study the existing `gsm8k_tinker.py` as a template. The agent can help you create new environments — it can read existing environment files, inspect HuggingFace datasets, and write new environment code.
+
+## WandB Metrics
+
+Training runs log to Weights & Biases with these key metrics:
+
+| Metric | Description |
+|--------|-------------|
+| `train/loss` | Training loss (importance sampling) |
+| `train/learning_rate` | Current learning rate |
+| `reward/mean` | Mean reward across groups |
+| `logprobs/mean` | Mean reference logprobs |
+| `logprobs/mean_training` | Mean training logprobs |
+| `logprobs/diff` | Logprob drift (reference - training) |
+| `advantages/mean` | Mean advantage values |
+| `advantages/std` | Advantage standard deviation |
+
+## Log Files
+
+Each training run generates log files in `~/.hermes/logs/rl_training/`:
+
+```
+logs/
+├── api_{run_id}.log        # Atropos API server logs
+├── trainer_{run_id}.log    # Tinker trainer logs
+├── env_{run_id}.log        # Environment process logs
+└── inference_tests/        # Inference test results
+    ├── test_{env}_{model}.jsonl
+    └── test_{env}_{model}.log
+```
+
+These are invaluable for debugging when training fails or produces unexpected results.

website/i18n/zh-Hans/docusaurus-plugin-content-docs/current/user-guide/features/overview.md

@@ -43,7 +43,7 @@ Hermes Agent 包含一套丰富的能力，远超基础聊天范畴。从持久
 - **[记忆提供商](memory-providers.md)** — 接入外部记忆后端（Honcho、OpenViking、Mem0、Hindsight、Holographic、RetainDB、ByteRover、Supermemory），实现跨会话用户建模和超越内置记忆系统的个性化。
 - **[API 服务器](api-server.md)** — 将 Hermes 作为兼容 OpenAI 的 HTTP 端点暴露。连接任何支持 OpenAI 格式的前端——Open WebUI、LobeChat、LibreChat 等。
 - **[IDE 集成（ACP）](acp.md)** — 在兼容 ACP 的编辑器（如 VS Code、Zed 和 JetBrains）中使用 Hermes。聊天、工具活动、文件 diff 和终端命令均在编辑器内渲染。
-- **[强化学习训练](rl-training.md)** — 从 Agent 会话中生成轨迹数据，用于强化学习和模型微调。
+- **[强化学习训练](/user-guide/features/rl-training)** — 从 Agent 会话中生成轨迹数据，用于强化学习和模型微调。
 
 ## 自定义
 

website/sidebars.ts

@@ -121,6 +121,13 @@ const sidebars: SidebarsConfig = {
             'user-guide/features/subscription-proxy',
           ],
         },
+        {
+          type: 'category',
+          label: 'Advanced',
+          items: [
+            'user-guide/features/rl-training',
+          ],
+        },
         {
           type: 'category',
           label: 'Skills',