diff --git a/documentation/docs/tutorials/recipes-tutorial.md b/documentation/docs/tutorials/recipes-tutorial.md index 446bc4d6abde..214782918b98 100644 --- a/documentation/docs/tutorials/recipes-tutorial.md +++ b/documentation/docs/tutorials/recipes-tutorial.md @@ -2,7 +2,7 @@ description: Learn how to create and use goose recipes with this comprehensive tutorial covering prompts, parameters, and MCP servers --- -# goose Recipes +# Recipes goose recipes are files that contain all the details to allow goose to do one specific task. Since they are contained in just one file, they are easy to share through all the normal ways we share files, including version management systems like git. Let's get started with the simplest recipe possible. diff --git a/documentation/docs/tutorials/rpi.md b/documentation/docs/tutorials/rpi.md new file mode 100644 index 000000000000..56c7286d1cdd --- /dev/null +++ b/documentation/docs/tutorials/rpi.md @@ -0,0 +1,272 @@ +--- +title: Research → Plan → Implement Pattern +description: How to use RPI, a context engineering technique, on complex software projects +--- + +import CodeBlock from '@theme/CodeBlock'; +import researchDoc from '../../static/files/thoughts/research/2025-12-22-llm-tool-selection-strategy.raw'; +import planDoc from '../../static/files/thoughts/plans/2025-12-23-remove-tool-selection-strategy.raw'; + + +Most people use AI agents by jumping straight to execution: "refactor this code", "remove this feature", "add this new feature". While sometimes this works well, especially for smaller changes or codebases, it often falls apart on complex changes. + +**RPI (Research, Plan, Implement)** is a mental model that proposes a different way of working with AI agents. This approach trades speed for clarity, predictability, and correctness. + +This tutorial walks through how RPI works via a real demonstration. By the end, you should be able to run this same workflow on your own codebase. + +## Prerequisites + +
+1. Import RPI Recipes + +Copy the snippet below and paste it in your terminal. This will download the main RPI recipes and their subrecipes and save them into the global recipe directory. + +```sh +mkdir -p ~/.config/goose/recipes/subrecipes + +curl -sL https://raw.githubusercontent.com/block/goose/main/documentation/src/pages/recipes/data/recipes/rpi-research.yaml -o ~/.config/goose/recipes/rpi-research.yaml +curl -sL https://raw.githubusercontent.com/block/goose/main/documentation/src/pages/recipes/data/recipes/rpi-plan.yaml -o ~/.config/goose/recipes/rpi-plan.yaml +curl -sL https://raw.githubusercontent.com/block/goose/main/documentation/src/pages/recipes/data/recipes/rpi-implement.yaml -o ~/.config/goose/recipes/rpi-implement.yaml +curl -sL https://raw.githubusercontent.com/block/goose/main/documentation/src/pages/recipes/data/recipes/rpi-iterate.yaml -o ~/.config/goose/recipes/rpi-iterate.yaml + +curl -sL https://raw.githubusercontent.com/block/goose/main/documentation/src/pages/recipes/data/recipes/subrecipes/rpi-codebase-locator.yaml -o ~/.config/goose/recipes/subrecipes/rpi-codebase-locator.yaml +curl -sL https://raw.githubusercontent.com/block/goose/main/documentation/src/pages/recipes/data/recipes/subrecipes/rpi-codebase-analyzer.yaml -o ~/.config/goose/recipes/subrecipes/rpi-codebase-analyzer.yaml +curl -sL https://raw.githubusercontent.com/block/goose/main/documentation/src/pages/recipes/data/recipes/subrecipes/rpi-pattern-finder.yaml -o ~/.config/goose/recipes/subrecipes/rpi-pattern-finder.yaml +``` +
+ +
+2. Add Custom Slash Commands + +Now that the recipes are imported, to quickly invoke them in-session [add custom slash commands](/docs/guides/recipes/session-recipes/#custom-recipe-commands) for each of the following recipes: + +| Recipe | Slash Command | +|--------|---------------| +| RPI Research Codebase | `research` | +| RPI Create Plan | `plan` | +| RPI Implement Plan | `implement` | +| RPI Iterate | `iterate` | +
+ +## RPI Workflow + +In goose, we use a structured RPI workflow using recipes to systematically tackle complex codebase changes. The workflow consists of slash commands that guide goose through disciplined phases of work: + +1. `research` – Document what exists today. No opinions. +2. `plan` - Design the change with clear phases and success criteria. +3. `implement` - Execute the plan step by step with verification. +4. `iterate` – (optional) Adjust the plan if necessary. + +```md +┌─────────────────────────────────────────────────────────────────────────────-┐ +│ RPI WORKFLOW │ +├─────────────────────────────────────────────────────────────────────────────-┤ +│ │ +│ /research "topic" │ +│ │ │ +│ ├──► Spawns parallel sub-agents: │ +│ │ • find_files (rpi-codebase-locator) │ +│ │ • analyze_code (rpi-codebase-analyzer) │ +│ │ • find_patterns (rpi-pattern-finder) │ +│ │ │ +│ └──► Output: thoughts/research/YYYY-MM-DD-HHmm-topic.md │ +│ │ +│ /plan "feature/task" │ +│ │ │ +│ ├──► Reads research docs │ +│ ├──► Asks clarifying questions │ +│ ├──► Proposes design options │ +│ │ │ +│ └──► Output: thoughts/plans/YYYY-MM-DD-HHmm-description.md │ +│ │ +│ /implement "plan path" │ +│ │ │ +│ ├──► Executes phase by phase │ +│ ├──► Runs verification after each phase │ +│ ├──► Updates checkboxes in plan │ +│ │ │ +│ └──► Working code │ +│ │ +│ /iterate "plan path" + feedback │ +│ │ │ +│ ├──► Researches only what changed │ +│ ├──► Updates the plan surgically │ +│ │ │ +│ └──► Updated plan │ +│ │ +└─────────────────────────────────────────────────────────────────────────────-┘ +``` + +All RPI outputs live in a predictable place: + +```md +thoughts/ +├── research/ +│ └── YYYY-MM-DD-HHmm-topic.md +└── plans/ + └── YYYY-MM-DD-HHmm-description.md +``` + +## The Task + +In this tutorial, I want to remove an existing feature from a large codebase. + +This isn't a small change. The feature touches: + +- Core Rust code +- TypeScript +- Configuration +- Tests +- Documentation + +This is the kind of task where agents often struggle, not because they're incapable, but because the work spans too much context to safely "just do it." + +So instead of jumping to implementation, we'll use RPI. + +## Session 1: Research + +The concept of planning before implementation has become a widely accepted practice. However, planning without research can lead to assumptions that come back to bite you. So, in RPI, we begin with research. + +I start the prompt with the `/research` command followed by a topic written in natural language + +``` +/research "look through the cloned goose repo and research how the LLM Tool Discovery is implemented" +``` + + +This command invokes the **[RPI Research Codebase](https://raw.githubusercontent.com/block/goose/refs/heads/main/documentation/src/pages/recipes/data/recipes/rpi-research.yaml)** recipe, whose job is very strict: + +- Document what exists +- Do not suggest changes +- Do not critique +- Do not plan + +With this recipe, goose automatically spawns three parallel subagents: + +- `find_files`: Uses the codebase locator to figure out where relevant files live. +- `analyze_code`: Reads those files fully and documents how they work. +- `find_patterns`: Looks for similar features or conventions elsewhere in the repo. + +These subagents run independently and report back. You don't have to orchestrate that yourself. + +:::info A course correction +After goose began researching, I noticed that it was researching "tool discovery" in general. But I only wanted to remove a specific feature called Tool Selection Strategy. So I stopped goose and reran research with a more accurate topic. + +That wasn't a failure. In fact, this is exactly why research exists. Had I told goose to "remove the LLM Tool Discovery feature", it may have removed our other tool discovery methods as well. Fortunately, catching these types of mistakes early is cheap and easy to recover from. +::: + +The output from the `/research` session was a detailed research document: + +
+./thoughts/research/2025-12-22-llm-tool-selection-strategy.md + + +{researchDoc} + + +
+ +This is a large, structured file that includes: +- Git metadata +- File and line references +- Flow descriptions +- Key components +- Open questions + +Think of it as a technical map of the feature as it exists today. Nothing was changed yet, and that's intentional. The only goal was shared understanding. + +As the human in the loop, be sure to review the research! This will inform the plan, so you want to make sure this is accurate. + +## Session 2: Plan + +Once research is complete, you move to planning. + +:::tip Sessions +It's important to do each phase in a new session to keep the LLM laser focused on only the task at hand. One goal per session! +::: + +``` +/plan a removal of the Tool Selection Strategy feature +``` + +The **[RPI Create Plan](https://raw.githubusercontent.com/block/goose/refs/heads/main/documentation/src/pages/recipes/data/recipes/rpi-plan.yaml)** recipe starts by reading the research document goose created. + +Then it did three key things: + +1. Asked clarifying questions +For example: + - Full removal vs deprecation? + - How should config cleanup behave? + - Should OpenAPI artifacts be regenerated? + - Where do related tests live? + +2. Presented design options + Where there were multiple reasonable approaches, goose laid them out and asked me to choose. + +3. Produced a phased implementation plan + + +The output was a detailed plan: + +
+thoughts/plans/2025-12-23-remove-tool-selection-strategy.md + + +{planDoc} + + +
+ +This plan includes: +- 10 explicit phases +- Exact file paths +- Code snippets showing what to remove +- Automated success criteria +- Manual verification steps +- Checkboxes for tracking progress + +At this point, the plan became the source of truth. The key shift here is that we've moved from understanding to decision making, but we're still not touching code. + +The plan is explicit enough that someone else could execute it. That's not an accident. Remember that the implementation will be in a fresh new session, so the plan must have enough context to actually execute it. + +Again, you as the human need to step in here to review the plan and make sure it's solid. If there's anything amiss, instead of starting over you can run the **[RPI Iterate Plan](https://raw.githubusercontent.com/block/goose/refs/heads/main/documentation/src/pages/recipes/data/recipes/rpi-iterate.yaml)** plan (`/iterate`) with details on what's wrong. goose will then read the existing plan, research only what needs rethinking, propose targeted updates, and edit the plan accordingly. + +## Session 3: Implement + +Only after research and planning are complete should you move to implementation. Pass in the plan document. + +``` +/implement thoughts/plans/2025-12-23-remove-tool-selection-strategy.md +``` + +The **[RPI Implement Plan](https://raw.githubusercontent.com/block/goose/refs/heads/main/documentation/src/pages/recipes/data/recipes/rpi-implement.yaml)** recipe is intentionally boring. In fact, I fell asleep while goose was running it. Implementation should feel mechanical. If it feels creative, something upstream is missing. But knowing that you have a rock solid plan, I advise you to go do something else with your time while goose works (unless there are manual steps in the plan). + +It will read the plan completely, execute the phases in order, run verification after each phase, and update the checkboxes directly in the plan file as it goes. + +That last bit was really helpful because my context window filled up partway through and goose was able to compact and pick up right where it left off because of the status updates in the plan. + +## Final Result + +For 10 phases of work that spanned 32 files, the Research phase took 9 minutes, the Plan phase took 4 minutes, and the Implement phase took 39 minutes. So in total, this took just shy of an hour... 52 minutes to be exact. This included goose working and testing as well as me answering questions. + +Definitely not a fast process. BUT! When I put up [this PR](https://github.com/block/goose/pull/6250), the build passed and the separate Code Review Agent didn't have a single comment. That's just how well done the work was. + +Had I done this without AI, it would have likely taken me several hours of work as the feature was complex and deeply integrated. And had I had AI jump straight to implementation, I have no doubt it would have surely drifted and messed something up. + +So while RPI is slower than having AI get right to work, the quality is top notch. A very worthy tradeoff. + + +## When to Use RPI + +For basic tasks, RPI may be overkill. Especially because it's not a quick process. However, if you need to do a complex task that spans multiple files, it's a great choice. + +You can use RPI for: +- Refactors +- Migrations +- Feature additions +- Large upgrades +- Incident cleanup +- Documentation overhauls + +Try it for yourself! + diff --git a/documentation/plugins/custom-webpack.cjs b/documentation/plugins/custom-webpack.cjs index eac2a005a38b..3f1a6b36e2c2 100644 --- a/documentation/plugins/custom-webpack.cjs +++ b/documentation/plugins/custom-webpack.cjs @@ -1,18 +1,17 @@ module.exports = function () { return { - name: 'custom-yaml-loader', - configureWebpack() { - return { - module: { - rules: [ - { - test: /\.ya?ml$/, - use: 'yaml-loader', - }, - ], - }, - }; + name: 'custom-webpack-loaders', + configureWebpack(config) { + config.module.rules.push({ + test: /\.ya?ml$/, + use: 'yaml-loader', + }); + config.module.rules.push({ + test: /\.raw$/, + type: 'asset/source', + }); + return {}; }, }; }; - \ No newline at end of file + diff --git a/documentation/static/files/thoughts/plans/2025-12-23-remove-tool-selection-strategy.raw b/documentation/static/files/thoughts/plans/2025-12-23-remove-tool-selection-strategy.raw new file mode 100644 index 000000000000..f5cdfa3fef59 --- /dev/null +++ b/documentation/static/files/thoughts/plans/2025-12-23-remove-tool-selection-strategy.raw @@ -0,0 +1,654 @@ +# Remove Tool Selection Strategy Feature - Implementation Plan + +## Overview +Complete removal of the "Tool Selection Strategy" feature (also known as "LLM Tool Router" or "Smart Tool Routing"). This experimental feature used a secondary LLM call to dynamically select which tools to present to the main LLM based on the user's query. + +## Current State Analysis +The feature is controlled by `GOOSE_ENABLE_ROUTER` config parameter (default: false). It consists of: +- Core implementation files (4 Rust modules + 1 prompt template) +- Integration points in agent, prompt manager, and server routes +- UI settings section in desktop app +- CLI configuration dialog +- Documentation pages + +### Key Discoveries: +- Feature is disabled by default and marked as "experimental/preview" +- Only tested with Claude models +- Has telemetry tracking in posthog +- Snapshot tests in `prompt_manager.rs` use `.with_router_enabled(true)` and will need updating +- `agent.rs` test references `tool_route_manager` but only for context setup, not testing router functionality + +## Desired End State +- All Tool Selection Strategy code removed from codebase +- No `GOOSE_ENABLE_ROUTER` config handling (ignored if present in user config) +- No `router__llm_search` tool +- No `/agent/update_router_tool_selector` API endpoint +- No UI settings for tool selection strategy +- No CLI configuration for router strategy +- Documentation removed/updated +- All tests pass, linting passes + +## What We're NOT Doing +- Cleaning up existing `GOOSE_ENABLE_ROUTER` entries from user config files (will be ignored) +- Adding deprecation warnings +- Keeping any stub code + +## Implementation Approach +Remove in dependency order: core files first, then integration points, then UI/CLI, then documentation. This minimizes compilation errors during the process. + +--- + +## Phase 1: Remove Core Implementation Files + +### Overview +Delete the core Rust modules that implement the router functionality. + +### Changes Required: + +#### 1. Delete core router files +**Files to delete:** +- `crates/goose/src/agents/router_tool_selector.rs` +- `crates/goose/src/agents/router_tools.rs` +- `crates/goose/src/agents/tool_route_manager.rs` +- `crates/goose/src/agents/tool_router_index_manager.rs` +- `crates/goose/src/prompts/router_tool_selector.md` + +#### 2. Update module declarations +**File**: `crates/goose/src/agents/mod.rs` +**Changes**: Remove module declarations for deleted files + +```rust +// REMOVE these lines: +mod router_tool_selector; +mod router_tools; +mod tool_route_manager; +mod tool_router_index_manager; +``` + +### Success Criteria: + +#### Automated Verification: +- [x] Files deleted +- [x] `cargo build -p goose` compiles (will fail until Phase 2 completes) + +**Implementation Note**: Phase 1 will cause compilation errors. Proceed immediately to Phase 2. + +--- + +## Phase 2: Update Agent Integration + +### Overview +Remove all router-related code from `agent.rs` and related files. + +### Changes Required: + +#### 1. Update agent.rs +**File**: `crates/goose/src/agents/agent.rs` +**Changes**: + +Remove imports: +```rust +// REMOVE: +use crate::agents::router_tools::ROUTER_LLM_SEARCH_TOOL_NAME; +use crate::agents::tool_route_manager::ToolRouteManager; +use crate::agents::tool_router_index_manager::ToolRouterIndexManager; +``` + +Remove from Agent struct: +```rust +// REMOVE field: +pub tool_route_manager: Arc, +``` + +Remove from Agent::new(): +```rust +// REMOVE: +tool_route_manager: Arc::new(ToolRouteManager::new()), +``` + +Remove method `disable_router_for_recipe`: +```rust +// REMOVE entire method: +pub async fn disable_router_for_recipe(&self) { + self.tool_route_manager.disable_router_for_recipe().await; +} +``` + +Remove from `dispatch_tool_call` - the `ROUTER_LLM_SEARCH_TOOL_NAME` branch: +```rust +// REMOVE this else-if branch: +} else if tool_call.name == ROUTER_LLM_SEARCH_TOOL_NAME { + match self + .tool_route_manager + .dispatch_route_search_tool(tool_call.arguments.unwrap_or_default()) + .await + { + Ok(tool_result) => tool_result, + Err(e) => return (request_id, Err(e)), + } +} +``` + +Remove from `add_extension` - the router indexing logic: +```rust +// REMOVE this block: +// If LLM tool selection is functional, index the tools +if self.tool_route_manager.is_router_functional().await { + let selector = self.tool_route_manager.get_router_tool_selector().await; + if let Some(selector) = selector { + let selector = Arc::new(selector); + if let Err(e) = ToolRouterIndexManager::update_extension_tools( + &selector, + &self.extension_manager, + &extension.name(), + "add", + ) + .await + { + return Err(ExtensionError::SetupError(format!( + "Failed to index tools for extension {}: {}", + extension.name(), + e + ))); + } + } +} +``` + +Remove `list_tools_for_router` method: +```rust +// REMOVE entire method: +pub async fn list_tools_for_router(&self) -> Vec { + ... +} +``` + +Remove from `remove_extension` - the router de-indexing logic: +```rust +// REMOVE this block: +// If LLM tool selection is functional, remove tools from the index +if self.tool_route_manager.is_router_functional().await { + let selector = self.tool_route_manager.get_router_tool_selector().await; + if let Some(selector) = selector { + ToolRouterIndexManager::update_extension_tools( + &selector, + &self.extension_manager, + name, + "remove", + ) + .await?; + } +} +``` + +Remove `update_router_tool_selector` method: +```rust +// REMOVE entire method: +pub async fn update_router_tool_selector( + &self, + provider: Option>, + reindex_all: Option, +) -> Result<()> { + ... +} +``` + +Remove from `reply_internal` stream - the `record_tool_requests` call: +```rust +// REMOVE: +self.tool_route_manager + .record_tool_requests(&requests_to_record) + .await; +``` + +#### 2. Update extension.rs (PlatformExtensionContext) +**File**: `crates/goose/src/agents/extension.rs` +**Changes**: Remove `tool_route_manager` field from `PlatformExtensionContext` if present + +Search for any references to `tool_route_manager` in this file and remove them. + +#### 3. Update extension_manager_extension.rs +**File**: `crates/goose/src/agents/extension_manager_extension.rs` +**Changes**: Remove any references to `tool_route_manager` + +### Success Criteria: + +#### Automated Verification: +- [x] `cargo build -p goose` compiles (may still fail until Phase 3) + +--- + +## Phase 3: Update Prompt Manager + +### Overview +Remove router-related prompt building logic. + +### Changes Required: + +#### 1. Update prompt_manager.rs +**File**: `crates/goose/src/agents/prompt_manager.rs` +**Changes**: + +Remove import: +```rust +// REMOVE: +use crate::agents::router_tools::llm_search_tool_prompt; +``` + +Remove from `SystemPromptContext`: +```rust +// REMOVE field: +#[serde(skip_serializing_if = "Option::is_none")] +tool_selection_strategy: Option, +``` + +Remove from `SystemPromptBuilder`: +```rust +// REMOVE field: +router_enabled: bool, +``` + +Remove `with_router_enabled` method: +```rust +// REMOVE entire method: +pub fn with_router_enabled(mut self, enabled: bool) -> Self { + self.router_enabled = enabled; + self +} +``` + +Update `build` method - remove router_enabled from context: +```rust +// REMOVE from SystemPromptContext construction: +tool_selection_strategy: self.router_enabled.then(llm_search_tool_prompt), +``` + +Update builder initialization: +```rust +// REMOVE from SystemPromptBuilder initialization: +router_enabled: false, +``` + +#### 2. Update system.md template +**File**: `crates/goose/src/prompts/system.md` +**Changes**: Remove the `{{tool_selection_strategy}}` placeholder line + +```markdown + +{{tool_selection_strategy}} +``` + +#### 3. Update snapshot tests +**File**: `crates/goose/src/agents/prompt_manager.rs` (tests section) +**Changes**: Remove `.with_router_enabled(true)` from tests + +In `test_one_extension`: +```rust +// REMOVE: +.with_router_enabled(true) +``` + +In `test_typical_setup`: +```rust +// REMOVE: +.with_router_enabled(true) +``` + +#### 4. Update/regenerate snapshots +**Files to update:** +- `crates/goose/src/agents/snapshots/goose__agents__prompt_manager__tests__one_extension.snap` +- `crates/goose/src/agents/snapshots/goose__agents__prompt_manager__tests__typical_setup.snap` + +Run `cargo test -p goose prompt_manager` with `INSTA_UPDATE=1` to regenerate snapshots, or manually remove the "LLM Tool Selection Instructions" section from each snapshot. + +### Success Criteria: + +#### Automated Verification: +- [x] `cargo build -p goose` compiles +- [x] `cargo test -p goose` passes (after snapshot updates) + +--- + +## Phase 4: Update Server Routes + +### Overview +Remove the `/agent/update_router_tool_selector` endpoint. + +### Changes Required: + +#### 1. Update agent.rs routes +**File**: `crates/goose-server/src/routes/agent.rs` +**Changes**: + +Remove request struct: +```rust +// REMOVE: +#[derive(Deserialize, utoipa::ToSchema)] +pub struct UpdateRouterToolSelectorRequest { + session_id: String, +} +``` + +Remove handler function: +```rust +// REMOVE entire function: +#[utoipa::path( + post, + path = "/agent/update_router_tool_selector", + ... +)] +async fn update_router_tool_selector( + ... +) -> Result, StatusCode> { + ... +} +``` + +Remove route from router: +```rust +// REMOVE from routes() function: +.route( + "/agent/update_router_tool_selector", + post(update_router_tool_selector), +) +``` + +### Success Criteria: + +#### Automated Verification: +- [x] `cargo build -p goose-server` compiles +- [x] `cargo test -p goose-server` passes + +--- + +## Phase 5: Update CLI Configuration + +### Overview +Remove the router configuration dialog from CLI. + +### Changes Required: + +#### 1. Update configure.rs +**File**: `crates/goose-cli/src/commands/configure.rs` +**Changes**: + +Remove the router strategy menu item from `configure_settings_dialog`: +```rust +// REMOVE this item: +.item( + "goose_router_strategy", + "Router Tool Selection Strategy", + "Experimental: configure a strategy for auto selecting tools to use", +) +``` + +Remove the match arm: +```rust +// REMOVE: +"goose_router_strategy" => { + configure_goose_router_strategy_dialog()?; +} +``` + +Remove the entire `configure_goose_router_strategy_dialog` function: +```rust +// REMOVE entire function: +pub fn configure_goose_router_strategy_dialog() -> anyhow::Result<()> { + ... +} +``` + +### Success Criteria: + +#### Automated Verification: +- [x] `cargo build -p goose-cli` compiles +- [x] `cargo test -p goose-cli` passes + +--- + +## Phase 6: Update Telemetry ✅ COMPLETE + +### Overview +Remove router-related telemetry. + +### Changes Required: + +#### 1. Update posthog.rs +**File**: `crates/goose/src/posthog.rs` +**Changes**: + +Remove the router telemetry: +```rust +// REMOVE this block: +if let Ok(router_enabled) = config.get_param::("GOOSE_ENABLE_ROUTER") { + event + .insert_prop("setting_router_enabled", router_enabled) + .ok(); +} +``` + +### Success Criteria: + +#### Automated Verification: +- [x] `cargo build -p goose` compiles + +--- + +## Phase 7: Update Desktop UI ✅ COMPLETE + +### Overview +Remove the Tool Selection Strategy settings section from the desktop app. + +### Changes Required: + +#### 1. Delete UI component directory +**Directory to delete**: `ui/desktop/src/components/settings/tool_selection_strategy/` + +#### 2. Update ChatSettingsSection.tsx +**File**: `ui/desktop/src/components/settings/chat/ChatSettingsSection.tsx` +**Changes**: + +Remove import: +```typescript +// REMOVE: +import { ToolSelectionStrategySection } from '../tool_selection_strategy/ToolSelectionStrategySection'; +``` + +Remove the Card containing ToolSelectionStrategySection: +```tsx +{/* REMOVE entire Card: */} + + + Tool Selection Strategy (preview) + + Experimental: configure how Goose selects tools for your requests, useful when there are + many tools. Only tested with Claude models currently. + + + + + + +``` + +#### 3. Regenerate OpenAPI types +Run: `just generate-openapi` + +This will update `ui/desktop/openapi.json` and `ui/desktop/src/api/types.gen.ts` to remove the `UpdateRouterToolSelectorRequest` type. + +### Success Criteria: + +#### Automated Verification: +- [x] `cd ui/desktop && npm run lint` passes +- [x] `cd ui/desktop && npm run typecheck` passes +- [x] OpenAPI types regenerated + +#### Manual Verification: +- [ ] Desktop app Settings > Chat page loads without errors +- [ ] No "Tool Selection Strategy" section visible + +--- + +## Phase 8: Update Documentation ✅ COMPLETE + +### Overview +Remove documentation for the removed feature. + +### Changes Required: + +#### 1. Delete tool-router.md +**File to delete**: `documentation/docs/guides/managing-tools/tool-router.md` + +#### 2. Update managing-tools index +**File**: `documentation/docs/guides/managing-tools/index.md` +**Changes**: + +Remove the Tool Selection Strategy card: +```tsx +{/* REMOVE: */} + +``` + +#### 3. Update environment-variables.md +**File**: `documentation/docs/guides/environment-variables.md` +**Changes**: + +Remove the `GOOSE_ENABLE_ROUTER` row from the table: +```markdown + +| `GOOSE_ENABLE_ROUTER` | Enables [intelligent tool selection strategy](/docs/guides/managing-tools/tool-router) | "true", "false" | "false" | +``` + +Remove from the example section: +```bash +# REMOVE: +# Enable intelligent tool selection +export GOOSE_ENABLE_ROUTER=true +``` + +#### 4. Check for other documentation references +Search for any other references to "tool selection", "router", or "GOOSE_ENABLE_ROUTER" in documentation and remove them. + +### Success Criteria: + +#### Automated Verification: +- [x] No remaining references to GOOSE_ENABLE_ROUTER in documentation + +#### Manual Verification: +- [ ] No references to Tool Selection Strategy in docs + +--- + +## Phase 9: Update Tests ✅ COMPLETE (merged into earlier phases) + +### Overview +Update any remaining tests that reference the removed functionality. + +### Changes Required: + +#### 1. Update agent.rs tests +**File**: `crates/goose/tests/agent.rs` +**Changes**: + +In `extension_manager_tests::setup_agent_with_extension_manager`, remove the `tool_route_manager` from context setup: +```rust +// REMOVE from PlatformExtensionContext: +tool_route_manager: Some(Arc::downgrade(&agent.tool_route_manager)), +``` + +#### 2. Update PlatformExtensionContext references +Search for any other test files that set up `PlatformExtensionContext` with `tool_route_manager` and remove that field. + +### Success Criteria: + +#### Automated Verification: +- [x] `cargo test` passes for all crates +- [x] `./scripts/clippy-lint.sh` passes + +--- + +## Phase 10: Final Cleanup and Verification ✅ COMPLETE + +### Overview +Final verification that all changes are complete and correct. + +### Changes Required: + +#### 1. Search for any remaining references +Run these searches to ensure nothing was missed: +```bash +rg "tool_route" --type rust +rg "router_tool" --type rust +rg "RouterToolSelector" --type rust +rg "ROUTER_LLM_SEARCH" --type rust +rg "llm_search_tool" --type rust +rg "GOOSE_ENABLE_ROUTER" --type rust +rg "tool_selection_strategy" --type rust +rg "ToolSelectionStrategy" --type ts --type tsx +``` + +#### 2. Run full test suite +```bash +cargo fmt +cargo build +cargo test +./scripts/clippy-lint.sh +``` + +#### 3. Run UI checks +```bash +cd ui/desktop +npm run lint +npm run build +npm test +``` + +### Success Criteria: + +#### Automated Verification: +- [x] All searches return no results (except in thoughts/research) +- [x] `cargo fmt` - no changes +- [x] `cargo build` - succeeds +- [x] `cargo test` - all tests pass +- [x] `./scripts/clippy-lint.sh` - passes +- [x] UI lint/typecheck - passes + +#### Manual Verification: +- [ ] Start goose CLI - works normally +- [ ] Start goose desktop - works normally +- [ ] Settings page loads without errors +- [ ] No console errors related to removed feature + +--- + +## Testing Strategy + +### Unit Tests: +- Snapshot tests in `prompt_manager.rs` need regeneration (Phase 3) +- Agent tests need `tool_route_manager` references removed (Phase 9) + +### Integration Tests: +- Full `cargo test` after all phases +- Desktop app manual testing after Phase 7 + +### Regression Testing: +- Ensure normal tool calling still works +- Ensure extension add/remove still works +- Ensure all goose modes (auto, approve, smart_approve, chat) still work + +--- + +## Rollback Plan +If issues are discovered: +1. Git revert the changes +2. The feature was disabled by default, so no user impact from keeping it + +--- + +## Notes +- The research document at `thoughts/research/2025-12-22-llm-tool-selection-strategy.md` should be kept for historical reference +- Users with `GOOSE_ENABLE_ROUTER=true` in their config will simply have the setting ignored diff --git a/documentation/static/files/thoughts/research/2025-12-22-llm-tool-selection-strategy.raw b/documentation/static/files/thoughts/research/2025-12-22-llm-tool-selection-strategy.raw new file mode 100644 index 000000000000..77e706426543 --- /dev/null +++ b/documentation/static/files/thoughts/research/2025-12-22-llm-tool-selection-strategy.raw @@ -0,0 +1,318 @@ +--- +date: 2025-12-22T23:43:05-06:00 +git_commit: 2f876725d3c08f821358e1391a7daadf468193d8 +branch: remove-llm-tool-discovery +repository: goose +topic: "LLM Tool Selection Strategy (Tool Discovery Feature)" +tags: [research, codebase, tools, router, llm-selection, experimental] +status: complete +--- + +# Research: LLM Tool Selection Strategy + +## Research Question +How is the "Tool Selection Strategy" feature implemented? This is the experimental feature that uses "LLM-based intelligence to select the most relevant tools based on the user query context." + +## Summary + +The **Tool Selection Strategy** is an experimental feature (preview) that dynamically filters which tools are presented to the LLM based on the user's query. Instead of sending all tools from all extensions to the LLM, it: + +1. Provides a single `router__llm_search` tool to the main LLM +2. When invoked, uses a secondary LLM call to search indexed tools and return only relevant ones +3. Tracks recently used tools to include them automatically + +This saves context window space when many extensions are enabled. + +**Configuration**: `GOOSE_ENABLE_ROUTER` (boolean, default: false) + +## Detailed Findings + +### 1. Feature Toggle - Configuration + +The feature is controlled by the `GOOSE_ENABLE_ROUTER` config parameter: + +**File: `crates/goose/src/agents/tool_route_manager.rs:79-86`** +```rust +pub async fn is_router_enabled(&self) -> bool { + if *self.router_disabled_override.lock().await { + return false; + } + + let config = Config::global(); + if let Ok(config_value) = config.get_param::("GOOSE_ENABLE_ROUTER") { + return config_value.to_lowercase() == "true"; + } + + // Default to false if neither is set + false +} +``` + +**UI Toggle: `ui/desktop/src/components/settings/tool_selection_strategy/ToolSelectionStrategySection.tsx`** +- Displays "Disabled" (default) and "Enabled" radio options +- Updates `GOOSE_ENABLE_ROUTER` config via upsert +- Calls `/agent/update_router_tool_selector` endpoint to reinitialize + +### 2. Core Components + +#### 2.1 ToolRouteManager +**File: `crates/goose/src/agents/tool_route_manager.rs`** + +Central manager that: +- Holds the `RouterToolSelector` instance +- Checks if router is enabled/functional +- Dispatches search tool calls +- Provides tools for router mode + +```rust +pub struct ToolRouteManager { + router_tool_selector: Mutex>>>, + router_disabled_override: Mutex, // For recipes that need all tools +} +``` + +Key methods: +- `is_router_enabled()` - Checks config +- `is_router_functional()` - Enabled AND selector initialized +- `dispatch_route_search_tool()` - Handles `router__llm_search` calls +- `list_tools_for_router()` - Returns search tool + recently used tools + +#### 2.2 RouterToolSelector Trait & LLMToolSelector +**File: `crates/goose/src/agents/router_tool_selector.rs`** + +```rust +#[async_trait] +pub trait RouterToolSelector: Send + Sync { + async fn select_tools(&self, params: JsonObject) -> Result, ErrorData>; + async fn index_tools(&self, tools: &[Tool], extension_name: &str) -> Result<(), ErrorData>; + async fn remove_tool(&self, tool_name: &str) -> Result<(), ErrorData>; + async fn record_tool_call(&self, tool_name: &str) -> Result<(), ErrorData>; + async fn get_recent_tool_calls(&self, limit: usize) -> Result, ErrorData>; +} +``` + +**LLMToolSelector** implementation: +- Stores tool strings indexed by extension name +- Uses an LLM provider to search tools based on query +- Tracks last 100 tool calls for "recently used" feature + +#### 2.3 The Search Tool Definition +**File: `crates/goose/src/agents/router_tools.rs`** + +```rust +pub const ROUTER_LLM_SEARCH_TOOL_NAME: &str = "router__llm_search"; + +pub fn llm_search_tool() -> Tool { + Tool::new( + ROUTER_LLM_SEARCH_TOOL_NAME.to_string(), + r#"Searches for relevant tools based on the user's messages. + Format a query to search for the most relevant tools... + Extension name is not optional, it is required. + The returned result will be a list of tool names, descriptions, and schemas..."#, + // Schema requires: extension_name (string), query (string), optional k (integer) + ) +} +``` + +#### 2.4 Tool Indexing Manager +**File: `crates/goose/src/agents/tool_router_index_manager.rs`** + +Handles indexing/removing tools when extensions are added/removed: + +```rust +impl ToolRouterIndexManager { + pub async fn update_extension_tools( + selector: &Arc>, + extension_manager: &ExtensionManager, + extension_name: &str, + action: &str, // "add" or "remove" + ) -> Result<()> +} +``` + +### 3. Flow: How Tool Selection Works + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ INITIALIZATION │ +├─────────────────────────────────────────────────────────────────┤ +│ 1. User enables "Tool Selection Strategy" in settings │ +│ 2. GOOSE_ENABLE_ROUTER = "true" saved to config │ +│ 3. /agent/update_router_tool_selector called │ +│ 4. ToolRouteManager.update_router_tool_selector(): │ +│ a. Creates LLMToolSelector with provider │ +│ b. Indexes all tools from all enabled extensions │ +│ c. Stores selector in router_tool_selector mutex │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ TOOL LISTING (Router Mode) │ +├─────────────────────────────────────────────────────────────────┤ +│ When agent.list_tools() is called with router enabled: │ +│ │ +│ Instead of returning ALL tools, returns: │ +│ 1. router__llm_search tool │ +│ 2. Recently used tools (last 20 calls) │ +│ 3. Platform tools (extension manager, etc.) │ +│ │ +│ This dramatically reduces context sent to LLM │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ RUNTIME: User Query │ +├─────────────────────────────────────────────────────────────────┤ +│ 1. User: "list files in current directory" │ +│ 2. LLM sees router__llm_search tool in available tools │ +│ 3. LLM invokes: router__llm_search( │ +│ extension_name: "developer", │ +│ query: "list files directory" │ +│ ) │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ SEARCH EXECUTION │ +├─────────────────────────────────────────────────────────────────┤ +│ Agent.dispatch_tool_call() routes to: │ +│ ToolRouteManager.dispatch_route_search_tool() │ +│ → LLMToolSelector.select_tools() │ +│ │ +│ LLMToolSelector: │ +│ 1. Gets indexed tool strings for extension │ +│ 2. Renders router_tool_selector.md prompt template │ +│ 3. Calls LLM provider with tool list + query │ +│ 4. Parses response for "Tool: X\nDescription: Y\nSchema: Z" │ +│ 5. Returns matching tools as Content │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ LLM RECEIVES RESULTS │ +├─────────────────────────────────────────────────────────────────┤ +│ Main LLM receives tool definitions in response │ +│ Can now invoke the actual tool (e.g., developer__shell) │ +│ Tool call is recorded for "recently used" feature │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### 4. System Prompt Integration + +**File: `crates/goose/src/agents/prompt_manager.rs`** + +When router is enabled, the system prompt includes special instructions: + +```rust +tool_selection_strategy: self.router_enabled.then(llm_search_tool_prompt), +``` + +**File: `crates/goose/src/agents/router_tools.rs:41-57`** +```rust +pub fn llm_search_tool_prompt() -> String { + format!( + r#"# LLM Tool Selection Instructions + Important: the user has opted to dynamically enable tools, so although an extension could be enabled, \ + please invoke the llm search tool to actually retrieve the most relevant tools to use according to the user's messages. + ... + By dynamically enabling tools, you (goose) as the agent save context window space and allow the user to dynamically retrieve the most relevant tools. + "#, + // Lists platform extension tools that are always available + ) +} +``` + +This is injected into `system.md` via `{{tool_selection_strategy}}`. + +### 5. LLM Search Prompt Template + +**File: `crates/goose/src/prompts/router_tool_selector.md`** +```markdown +You are a tool selection assistant. Your task is to find the most relevant tools based on the user's query. + +Given the following tools: +{{ tools }} + +Find the most relevant tools for the query: {{ query }} + +Return the tools in this exact format for each tool: +Tool: +Description: +Schema: +``` + +### 6. Server Endpoint + +**File: `crates/goose-server/src/routes/agent.rs`** + +```rust +#[utoipa::path( + post, + path = "/agent/update_router_tool_selector", + ... +)] +async fn update_router_tool_selector( + State(state): State>, + Json(payload): Json, +) -> Result, StatusCode> { + let agent = state.get_agent_for_route(payload.session_id).await?; + agent + .update_router_tool_selector(None, Some(true)) // reindex_all = true + .await + .map_err(...)?; + + Ok(Json("Tool selection strategy updated successfully".to_string())) +} +``` + +### 7. Recipe Override + +Recipes can disable the router to ensure all tools are available: + +**File: `crates/goose/src/agents/tool_route_manager.rs:31-34`** +```rust +pub async fn disable_router_for_recipe(&self) { + *self.router_disabled_override.lock().await = true; + *self.router_tool_selector.lock().await = None; +} +``` + +## Code References + +### Core Implementation +- `crates/goose/src/agents/tool_route_manager.rs` - Main manager, config check, dispatch +- `crates/goose/src/agents/router_tool_selector.rs` - `RouterToolSelector` trait, `LLMToolSelector` impl +- `crates/goose/src/agents/router_tools.rs` - `router__llm_search` tool definition, prompt function +- `crates/goose/src/agents/tool_router_index_manager.rs` - Tool indexing on extension add/remove + +### Integration Points +- `crates/goose/src/agents/agent.rs` - `dispatch_tool_call()` routes `ROUTER_LLM_SEARCH_TOOL_NAME` +- `crates/goose/src/agents/prompt_manager.rs` - `with_router_enabled()`, injects prompt +- `crates/goose-server/src/routes/agent.rs` - `/agent/update_router_tool_selector` endpoint + +### UI +- `ui/desktop/src/components/settings/tool_selection_strategy/ToolSelectionStrategySection.tsx` - Settings toggle + +### Prompts +- `crates/goose/src/prompts/router_tool_selector.md` - LLM search prompt template +- `crates/goose/src/prompts/system.md` - `{{tool_selection_strategy}}` placeholder + +## Key Design Patterns + +1. **Two-stage LLM calls**: Main LLM calls search tool → Search LLM finds relevant tools → Main LLM uses them +2. **Extension-based indexing**: Tools indexed by extension name for filtered searches +3. **Recently used caching**: Last 20 tool calls automatically included (no search needed) +4. **Override mechanism**: Recipes can disable router to get all tools +5. **Lazy initialization**: Selector only created when enabled AND provider available + +## Limitations Noted in UI + +- "Only tested with Claude models currently" (from UI description) +- Experimental/preview feature + +## Open Questions + +1. **Performance**: What's the latency impact of the secondary LLM call for tool search? +2. **Accuracy**: How well does the search LLM match tools to queries in practice? +3. **Token savings**: What's the actual context window savings for typical extension counts? +4. **Model compatibility**: Why only tested with Claude? What breaks with other models?