Provider scenario tests #3688
Provider scenario tests #3688
Conversation
| CallToolResult, Implementation, InitializeResult, ListPromptsResult, ListResourcesResult, | ||
| ListToolsResult, ReadResourceResult, ServerCapabilities, ToolsCapability, | ||
| }; | ||
| use mcp_core::{Tool, ToolError}; |
There was a problem hiding this comment.
Can you import Tool from rmcp::model instead?
Sorry, I should have removed the old definition when I finished the migration between those types here https://github.com/block/goose/pull/3617/files
I'll do that as a small PR, but good to start importing the new types
There was a problem hiding this comment.
yeah, it's been a moving target - when I merged in master I had to update all the references again :)
There was a problem hiding this comment.
would you mind if we do that in a follow up? it gives me some weird type errors
jamadeo
left a comment
jamadeo
left a comment
There was a problem hiding this comment.
Very cool. So if I understand correctly, in replay mode, this will replay the provider-specific recordings, but through a mock provider? And then in recording mode we can actually test the providers end-to-end.
| return Ok(()); | ||
| } | ||
|
|
||
| let configs_to_test: Vec<_> = if let Some(to_skip) = providers_to_skip { |
There was a problem hiding this comment.
you can get an iterator from an Option<Vec> with option.iter().flatt() -- useful for this kind of thing
| } | ||
| } | ||
|
|
||
| // Rest of the function remains the same... |
|
|
||
| #[tokio::test] | ||
| async fn test_weather_tool() -> Result<()> { | ||
| // Google tells me it only knows about the weather in the US, so we skip it. |
There was a problem hiding this comment.
I suppose we could ask it about the weather somewhere it knows about
There was a problem hiding this comment.
we could, but I am guessing it will still refuse to call the tool? I'm just trying to get this in first and then we can make it work for more and more things
* main: blog: streamlining detection development w/ recipes (#3689) fix: have option for cli providers to use their configured or default model (#3683) docs: new blog post and corrections to an old one on goosehints (#3657) Resolve sub recipe path relative to the parent recipe path (#3642) Speed up recipe loading from deeplinks and various fixes (#3662) fix cmd + , not opening settings (#3694) Add warning when JSON env parsing fails. (#3696) chore: refactor session naming into provider (#3678) feat (ui): File picker for scheduling recipes default to recipe dir (#3611) fix: address issue with streamable http interactions via mcp (#3693) Provider scenario tests (#3688) Fix conversations before they hit the LLM (#3660) cli: add detailed instruction for WSL users (#3496) feat: recipe runs will now prompt for missing extension secrets (#3668) fix: pricing integration tests -> trying more runs for cache and retries (#3546)
* main: fix: Ensures final output tool is available when using vector tool search (#3701) chore: adding in some new models token limits (#3685) blog: streamlining detection development w/ recipes (#3689) fix: have option for cli providers to use their configured or default model (#3683) docs: new blog post and corrections to an old one on goosehints (#3657) Resolve sub recipe path relative to the parent recipe path (#3642) Speed up recipe loading from deeplinks and various fixes (#3662) fix cmd + , not opening settings (#3694) Add warning when JSON env parsing fails. (#3696) chore: refactor session naming into provider (#3678) feat (ui): File picker for scheduling recipes default to recipe dir (#3611) fix: address issue with streamable http interactions via mcp (#3693) Provider scenario tests (#3688)
Co-authored-by: Douwe Osinga <[email protected]> Signed-off-by: Adam Tarantino <[email protected]>
Have scenario tests per provider. We are still missing quite some coverage, but it is a start!