allow skipping providers in test_providers.sh #6778
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DOsinga
approved these changes
Jan 28, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a mechanism to selectively skip provider smoke tests via an environment variable, to keep CI/local runs useful when certain providers are temporarily unavailable.
Changes:
- Documented
SKIP_PROVIDERSandSKIP_BUILDenvironment variables in the script header. - Added
should_skip_provider()helper to parseSKIP_PROVIDERS. - Skipped provider execution early in the provider loop when configured.
| if [ -z "$SKIP_PROVIDERS" ]; then | ||
| return 1 | ||
| fi | ||
| IFS=',' read -ra SKIP_LIST <<< "$SKIP_PROVIDERS" |
There was a problem hiding this comment.
should_skip_provider populates SKIP_LIST as a global array, which can unintentionally leak state outside the function; declare it as a local array (e.g., local -a ...) to keep the function side-effect free.
Suggested change
| IFS=',' read -ra SKIP_LIST <<< "$SKIP_PROVIDERS" | |
| local -a SKIP_LIST | |
| local IFS=',' | |
| read -ra SKIP_LIST <<< "$SKIP_PROVIDERS" |
Comment on lines
+122
to
+127
| # Skip provider if it's in SKIP_PROVIDERS | ||
| if should_skip_provider "$PROVIDER"; then | ||
| echo "⊘ Skipping provider: ${PROVIDER} (SKIP_PROVIDERS)" | ||
| echo "---" | ||
| continue | ||
| fi |
There was a problem hiding this comment.
If SKIP_PROVIDERS ends up skipping every provider, the script will currently report success (and print "All tests passed!") even though no tests ran; track whether any models were executed and fail (or at least print a clear warning and exit non-zero in CI) when the run is empty.
zanesq
added a commit
that referenced
this pull request
Jan 28, 2026
…ensions * 'main' of github.com:block/goose: chore: re-sync package-lock.json (#6783) upgrade electron to 39.3.0 (#6779) allow skipping providers in test_providers.sh (#6778) fix: enable custom model entry for OpenRouter provider (#6761) Remove codex skills flag support (#6775) Improve mcp test (#6671) Feat/anthropic custom headers (#6774) Fix/GitHub copilot error handling 5845 (#6771) fix(ui): respect width parameter in MCP app size-changed notifications (#6376) fix: address compilation issue in main (#6776)
lifeizhou-ap
added a commit
that referenced
this pull request
Jan 28, 2026
* main: (47 commits) Upgrade error handling (#6747) Fix/filter audience 6703 local (#6773) chore: re-sync package-lock.json (#6783) upgrade electron to 39.3.0 (#6779) allow skipping providers in test_providers.sh (#6778) fix: enable custom model entry for OpenRouter provider (#6761) Remove codex skills flag support (#6775) Improve mcp test (#6671) Feat/anthropic custom headers (#6774) Fix/GitHub copilot error handling 5845 (#6771) fix(ui): respect width parameter in MCP app size-changed notifications (#6376) fix: address compilation issue in main (#6776) Upgrade GitHub Actions for Node 24 compatibility (#6699) fix(google): preserve thought signatures in streaming responses (#6708) added reduce motion support for css animations and streaming text (#6551) fix: Re-enable subagents for Gemini models (#6513) fix(google): use parametersJsonSchema for full JSON Schema support (#6555) fix: respect GOOSE_CLI_MIN_PRIORITY for shell streaming output (#6558) feat: add requires_auth flag for custom providers without authentication (#6705) fix: normalize extension names consistently in ExtensionManager (#6529) ...
michaelneale
added a commit
that referenced
this pull request
Jan 29, 2026
* main: (30 commits) Different approach to determining final confidence level of prompt injection evaluation outcomes (#6729) fix: read_resource_tool deadlock causing test_compaction to hang (#6737) Upgrade error handling (#6747) Fix/filter audience 6703 local (#6773) chore: re-sync package-lock.json (#6783) upgrade electron to 39.3.0 (#6779) allow skipping providers in test_providers.sh (#6778) fix: enable custom model entry for OpenRouter provider (#6761) Remove codex skills flag support (#6775) Improve mcp test (#6671) Feat/anthropic custom headers (#6774) Fix/GitHub copilot error handling 5845 (#6771) fix(ui): respect width parameter in MCP app size-changed notifications (#6376) fix: address compilation issue in main (#6776) Upgrade GitHub Actions for Node 24 compatibility (#6699) fix(google): preserve thought signatures in streaming responses (#6708) added reduce motion support for css animations and streaming text (#6551) fix: Re-enable subagents for Gemini models (#6513) fix(google): use parametersJsonSchema for full JSON Schema support (#6555) fix: respect GOOSE_CLI_MIN_PRIORITY for shell streaming output (#6558) ...
This was referenced Jan 29, 2026
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Allow skipping some providers, which we might want to temporarily do if they go down or need API key balance reloads.