chore: cover code mode with end to end provider tests #6183
Conversation
michaelneale
requested review from
codefromthecrypt,
Copilot and
tlongwell-block
There was a problem hiding this comment.
Pull request overview
This PR extends the provider test script to support testing the new code execution mode alongside the existing developer extension mode. The test can now be run in two modes: normal mode (direct tool calls) and code execution mode (JS batching), allowing verification that code execution mode works equivalently to the stock developer extension across all providers.
- Adds
--code-execflag to enable code execution mode testing - Configures mode-specific settings for builtins and success patterns
- Parameterizes test assertions to work with both modes
scripts/test_providers.sh
Outdated
| case $arg in | ||
| --code-exec) | ||
| CODE_EXEC_MODE=true | ||
| shift |
There was a problem hiding this comment.
The shift command has no effect here because the loop iterates over "$@" directly, not positional parameters. Remove this line as it serves no purpose in this context.
| shift |
scripts/test_providers.sh
Outdated
| if [ "$CODE_EXEC_MODE" = true ]; then | ||
| echo "Mode: code_execution (JS batching)" | ||
| BUILTINS="developer,code_execution" | ||
| SUCCESS_PATTERN="execute_code | code_execution|read_module | code_execution" |
There was a problem hiding this comment.
The regex pattern mixes literal pipes (with spaces) and regex OR operator (pipe without spaces), making it unclear. Consider using explicit grouping like "(execute_code | code_execution)|(read_module | code_execution)" to make the intent clearer, or escape the literal pipes if they should be matched literally.
| SUCCESS_PATTERN="execute_code | code_execution|read_module | code_execution" | |
| SUCCESS_PATTERN="(execute_code | code_execution)|(read_module | code_execution)" |
| case $arg in | ||
| --code-exec) | ||
| CODE_EXEC_MODE=true | ||
| ;; |
There was a problem hiding this comment.
The shift command inside the loop will cause the loop to skip arguments. Since you're iterating over "$@" with "for arg in", the shift modifies the positional parameters but doesn't affect the loop iteration. Remove the shift command as it serves no purpose here and could cause unexpected behavior if additional arguments are added in the future.
| ;; |
codefromthecrypt
left a comment
codefromthecrypt
left a comment
There was a problem hiding this comment.
ah so the main change here is we are still doing "list the files...", just that we are adding code execution to do it which forces the other extensions inside. Same test
|
Nice work @michaelneale ! |
Attempting to cover the new code mode with same tests for providers. Should work the same as stock developer extension