Skip to content

chore: suggest using text/markdown when fetching content #5854

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alexhancock merged 1 commit into from
Nov 25, 2025
Merged

chore: suggest using text/markdown when fetching content #5854

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 13 additions & 7 deletions crates/goose-mcp/src/computercontroller/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -492,13 +492,19 @@ impl ComputerControllerServer {
let save_as = params.save_as;

// Fetch the content
let response = self.http_client.get(url).send().await.map_err(|e| {
ErrorData::new(
ErrorCode::INTERNAL_ERROR,
format!("Failed to fetch URL: {}", e),
None,
)
})?;
let response = self
.http_client
.get(url)
.header("Accept", "text/markdown, */*")
.send()
.await
.map_err(|e| {
ErrorData::new(
ErrorCode::INTERNAL_ERROR,
format!("Failed to fetch URL: {}", e),
None,
)
})?;

let status = response.status();
if !status.is_success() {
Expand Down
2 changes: 2 additions & 0 deletions crates/goose-mcp/src/developer/rmcp_developer.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,6 +325,8 @@ impl ServerHandler for DeveloperServer {
**Important**: Each shell command runs in its own process. Things like directory changes or
sourcing files do not persist between tool calls. So you may need to repeat them each time by
stringing together commands.

If fetching web content, consider adding Accept: text/markdown header
Copy link
Contributor

@simonsickle simonsickle Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add the reason why to add the header? Right now it would take some extra context to know why it’s needed or when it’s appropriate, so maybe inlining that context here is good?

Copy link
Collaborator Author

@michaelneale michaelneale Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah - I suspect so. I am always wary of "adding more" to a prompt that effectively ends up in a system prompt, so am trying to be brief - any ideas of the densest possible way to say that? (I am also not sure if the shell() tool is the right place, but there is no developer tool that is "fetch" - there are a ton of extensions that are, and the computer controller one...). Maybe don't even need it here and over time models will know (via pre-train) to do this when appropriate? (ie don't touch this one, but leave the one in computer controller) @simonsickle ?

Copy link
Contributor

@simonsickle simonsickle Nov 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought about this a bit more. When I have written rules, I typically like to follow the following pattern: (a) short, (b) conditional, (c) with explicit carve-outs

so, my attempt here following that pattern is

“For HTTP GETs that retrieve human-readable documentation or prose (e.g., READMEs, guides, specs, blog posts), include ‘Accept: text/markdown, /’ to request markdown when available (token-efficient, easy to parse). Do not add this header for API/JSON requests, binary/media downloads, or HTML scraping.”

this tells the LLMs when and why we should use the header but also why we should not use the header (if you ask it to scrape HTML or call a JSON endpoint specifically).

So we have any cli benchmarks we could test this with?

Copy link
Collaborator Author

@michaelneale michaelneale Nov 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is tbench - but still working on getting that a) regular and reliable (it costs about $200+ each run and takes some time) and b) let it run on branches too (may be overkillf or that). That seems a bit much wording for that prompt, would rather it be part of a bigger developer MCP refactoring (ie take things out, add other things). Perhaps it is time to bite the bullet and have a built in fetch tool? (I expect that is now common?)

Copy link
Collaborator

@alexhancock alexhancock Nov 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting discussion here, but in the interest of testing this out I am going to merge with it as-is. We can always iterate on prompts.

"#};

let windows_specific = indoc! {r#"
Expand Down
Loading