feat(agent): describe images for text-only models

[AnthonyRonning]

Summary

• keep Goose native read_image behavior for vision-capable Agent models
• use a hidden, thinking-disabled Gemma 4 completion to turn images into text-only tool results for nonvision models, guided by explicit task context
• derive vision support from the advanced model catalog and lock the Agent model after the first message
• bound and isolate the helper request, strip raw images on every nonvision outcome, and account for helper usage without corrupting primary context-window metrics

Validation

• pre-commit suite: production frontend build, 77 frontend tests, and 103 Rust tests
• TypeScript, ESLint, Prettier, cargo fmt --check, and Clippy
• provider-boundary test confirms Gemma model selection, image serialization, no tools, and thinking disabled
• rebuilt macOS app E2E: GLM receives a useful text description for a remote image; Gemma keeps the native image path without a context parameter
• independent final Rust and frontend review passes

---

Summary by CodeRabbit

• New Features

  • Added support for detecting model vision capabilities, including aliases.
  • Improved image-reading tools for models with and without vision support.
  • Locked the selected agent model after a session begins.
  • Added vision capability information to agent message requests.

• Bug Fixes

  • Prevented sessions from switching models after messages are sent.
  • Disabled unsupported image and reasoning features when unavailable.

• Tests

  • Added coverage for model locking, vision capability resolution, image handling, and session state detection.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 8183f655-90f2-4b16-9caf-cf6262c10d8b

📥 Commits

Reviewing files that changed from the base of the PR and between 6df18c8 and e1c9012.

📒 Files selected for processing (9)

• frontend/src-tauri/src/agent.rs
• frontend/src-tauri/src/agent/developer_tools.rs
• frontend/src-tauri/src/agent/shell_permission.rs
• frontend/src/components/AgentMode.tsx
• frontend/src/services/agentModels.test.ts
• frontend/src/services/agentModels.ts
• frontend/src/services/agentRuntimeService.ts
• frontend/src/services/agentTimeline.test.ts
• frontend/src/services/agentTimeline.ts

---

📝 Walkthrough

Walkthrough

The agent now locks a session’s model after messages begin, sends resolved vision capability, and configures image tooling for direct vision or bounded contextual descriptions. Frontend model selection, backend requests, usage tracking, and shell classifier parameters were updated with tests.

Changes

Vision-aware agent flow

Layer / File(s)	Summary
Model capability and selector state frontend/src/services/agentModels.ts, frontend/src/services/agentTimeline.ts, frontend/src/components/AgentMode.tsx, frontend/src/services/agentRuntimeService.ts, frontend/src/services/*test.ts	Model and alias vision metadata are resolved, started sessions lock model selection, and outgoing requests include visionCapable.
Session model locking and vision wiring frontend/src-tauri/src/agent.rs	Session setup validates model changes, accepts vision capability, and passes it into developer-client configuration.
Contextual image tool fallback frontend/src-tauri/src/agent/developer_tools.rs	read_image supports direct image inspection for vision models and bounded text descriptions, including metadata preservation, usage recording, and coverage tests.
Thinking-disabled classifier parameters frontend/src-tauri/src/agent/shell_permission.rs	Shell permission classification uses the shared thinking-disabled request-parameter helper and updated tests.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

• OpenSecretCloud/Maple#495: Adds model catalog and alias vision metadata used by this capability-resolution flow.
• OpenSecretCloud/Maple#620: Modifies the shared developer-tool and shell-permission plumbing touched here.

Sequence Diagram(s)

sequenceDiagram
  participant AgentMode
  participant agentRuntimeService
  participant agent_send_message
  participant MapleDeveloperClient
  participant ImageDescriptionProvider
  AgentMode->>agentRuntimeService: send visionCapable and selected model
  agentRuntimeService->>agent_send_message: submit agent request
  agent_send_message->>MapleDeveloperClient: configure vision-aware developer tools
  MapleDeveloperClient->>ImageDescriptionProvider: describe image when primary model lacks vision
  ImageDescriptionProvider-->>MapleDeveloperClient: return contextual text

Loading

Poem

A rabbit hops through model gates,
And locks the choice when chatting starts.
Images bloom or turn to words,
With bounded prompts and careful hearts.
Vision flags dance down the stream—
Carrots salute the clever scheme! 🐇

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: adding image description support for text-only agent models.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex-maple-contextual-image-helper

---

_{Comment @coderabbitai help to get the list of available commands.}

[cloudflare-workers-and-pages]

Deploying maple with    Cloudflare Pages

Latest commit:	e1c9012
Status:	✅  Deploy successful!
Preview URL:	https://468bbfd7.maple-ca8.pages.dev
Branch Preview URL:	https://codex-maple-contextual-image.maple-ca8.pages.dev

View logs

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.