Scenario tests #3430
Merged
Scenario tests #3430
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
July 15, 2025 15:50
Collaborator
|
nice - love this idea. Was thinking it is "mock provider" but that isn't the right word - it is a provider simulator for scenarios. Will help a lot (can even have sessions from real providers in case some have unique behavior, eg openai with multiple parallel tool calls etc) |
Collaborator
Author
|
yeah, I was just thinking, we should enhance this such that certain scenarios specify which model they need. If you then try to record and you don't have that model configured it will fail. that way if we have a fix for say gemini, the author of that fix can show us that it works by adding a scenario, even if we don't have access to gemini |
jamadeo
approved these changes
Jul 16, 2025
Collaborator
jamadeo
left a comment
jamadeo
left a comment
There was a problem hiding this comment.
Really nice. Once merged I can add a variant that streams tokens to test all of that.
|
|
||
| #[derive(Debug, Clone)] | ||
| pub struct ScenarioResult { | ||
| pub messages: Vec<goose::message::Message>, |
Collaborator
There was a problem hiding this comment.
might as well import this
| (Ok(provider), Ok(model)) => (provider, model), | ||
| _ => { | ||
| if std::env::var("GITHUB_ACTIONS").is_ok() { | ||
| panic!("You forgot to add the file {} to the repository. This file is required when running tests in CI.", file_path); |
Collaborator
There was a problem hiding this comment.
is this error in the right place?
katzdave
added a commit
that referenced
this pull request
Jul 22, 2025
* 'main' of github.com:block/goose: (23 commits) fix: add fallback id to messages if none provided (#3584) feat: migrate ErrorData from internal mcp crates to rmcp version (#3586) fix: adjust subrecipe description to allow running tests (#3585) Scenario tests (#3430) feat: migrate JsonRpcMessage/Request/Response/Error/Notification from internal mcp crates to rmcp versions (#3564) Restore recipe parameters functionality (#3530) feat: Enhanced loading states with thinking icons and flying bird animation (#3478) Agent loop defensive (#3554) chore: remove needless clone() in goose/providers (#2528) Add recipe install warning (#3537) Replace mcp_core::resource::* with rmcp types (#3563) Add YouTube video embed to using-goosehints.md (#3560) fix: ensure retry-config and success-criteria are populated in openapi spec (#3575) fix: use sequential when sub recipe task is 1. (#3573) fix: track message id to keep like with like (#3572) Replace mcp_core::prompt with rmcp::model types (#3561) feat (ui): close recipe modals with esc key (#3568) feat: recipes can retry with success criteria (#3474) Env var to set Ollama request timeout (#3516) Updating docs to match new UI (#3552) ...
michaelneale
added a commit
that referenced
this pull request
Jul 23, 2025
* main: docs: desktop recipe format (#3594) Fix model display name not being updated immediately after leaving settings (#3587) Added option to summarize the chat when an error is triggered (#3598) Remove mcp_macros and unused types (#3581) fix: add fallback id to messages if none provided (#3584) feat: migrate ErrorData from internal mcp crates to rmcp version (#3586) fix: adjust subrecipe description to allow running tests (#3585) Scenario tests (#3430) feat: migrate JsonRpcMessage/Request/Response/Error/Notification from internal mcp crates to rmcp versions (#3564) Restore recipe parameters functionality (#3530) feat: Enhanced loading states with thinking icons and flying bird animation (#3478) Agent loop defensive (#3554) chore: remove needless clone() in goose/providers (#2528) Add recipe install warning (#3537) Replace mcp_core::resource::* with rmcp types (#3563) Add YouTube video embed to using-goosehints.md (#3560) fix: ensure retry-config and success-criteria are populated in openapi spec (#3575)
taylorkmho
added a commit
that referenced
this pull request
Jul 23, 2025
* main: feat: subagent turn count, simple agent loop (#3597) feat: subagent independent extension manager (#3596) Improve session history loading resiliency (#3588) Added logging and changed default route case to not redirect to home when there is an invalid route (#3610) fix: chat sidebar layout overlapping content occasionally (#3590) fix: loading shared sessions (#3607) docs: use installer component for tutorial, add updating extensions section (#3608) fix: show token alert popover during agent responses and agent failure cases (#3536) reuse the cancellation token in the agent level (#3599) Docs: Move MongoDB tutorial to MCP section (#3602) docs: desktop recipe format (#3594) Fix model display name not being updated immediately after leaving settings (#3587) Added option to summarize the chat when an error is triggered (#3598) Remove mcp_macros and unused types (#3581) fix: add fallback id to messages if none provided (#3584) feat: migrate ErrorData from internal mcp crates to rmcp version (#3586) fix: adjust subrecipe description to allow running tests (#3585) Scenario tests (#3430) feat: migrate JsonRpcMessage/Request/Response/Error/Notification from internal mcp crates to rmcp versions (#3564) Restore recipe parameters functionality (#3530)
lifeizhou-ap
added a commit
that referenced
this pull request
Jul 24, 2025
* main: (28 commits) fix: multi-tool calls in streamed openai-compatible responses (#3609) feat: subagent turn count, simple agent loop (#3597) feat: subagent independent extension manager (#3596) Improve session history loading resiliency (#3588) Added logging and changed default route case to not redirect to home when there is an invalid route (#3610) fix: chat sidebar layout overlapping content occasionally (#3590) fix: loading shared sessions (#3607) docs: use installer component for tutorial, add updating extensions section (#3608) fix: show token alert popover during agent responses and agent failure cases (#3536) reuse the cancellation token in the agent level (#3599) Docs: Move MongoDB tutorial to MCP section (#3602) docs: desktop recipe format (#3594) Fix model display name not being updated immediately after leaving settings (#3587) Added option to summarize the chat when an error is triggered (#3598) Remove mcp_macros and unused types (#3581) fix: add fallback id to messages if none provided (#3584) feat: migrate ErrorData from internal mcp crates to rmcp version (#3586) fix: adjust subrecipe description to allow running tests (#3585) Scenario tests (#3430) feat: migrate JsonRpcMessage/Request/Response/Error/Notification from internal mcp crates to rmcp versions (#3564) ...
atarantino
pushed a commit
to atarantino/goose
that referenced
this pull request
Aug 5, 2025
Co-authored-by: Douwe Osinga <[email protected]> Signed-off-by: Adam Tarantino <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
First set up for scenario tests