Skip to content

allow skipping providers in test_providers.sh #6778

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jamadeo merged 2 commits into from
Jan 28, 2026
Merged

allow skipping providers in test_providers.sh #6778

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9860f2b
Allow skipping
jamadeo Jan 28, 2026
eb4e1aa
add to the action
jamadeo Jan 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/pr-smoke-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@ jobs:
HOME: /tmp/goose-home
GOOSE_DISABLE_KEYRING: 1
SKIP_BUILD: 1
SKIP_PROVIDERS: ${{ vars.SKIP_PROVIDERS || '' }}
run: |
# Ensure the HOME directory structure exists
mkdir -p $HOME/.local/share/goose/sessions
Expand Down Expand Up @@ -194,6 +195,7 @@ jobs:
HOME: /tmp/goose-home
GOOSE_DISABLE_KEYRING: 1
SKIP_BUILD: 1
SKIP_PROVIDERS: ${{ vars.SKIP_PROVIDERS || '' }}
run: |
mkdir -p $HOME/.local/share/goose/sessions
mkdir -p $HOME/.config/goose
Expand Down
28 changes: 28 additions & 0 deletions scripts/test_providers.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@
# Usage:
# ./test_providers.sh # Normal mode (direct tool calls)
# ./test_providers.sh --code-exec # Code execution mode (JS batching)
#
# Environment variables:
# SKIP_PROVIDERS Comma-separated list of providers to skip (e.g., "tetrate,xai")
# SKIP_BUILD Skip the cargo build step if set

CODE_EXEC_MODE=false
for arg in "$@"; do
Expand Down Expand Up @@ -91,13 +95,37 @@ is_allowed_failure() {
return 1
}

should_skip_provider() {
local provider="$1"
if [ -z "$SKIP_PROVIDERS" ]; then
return 1
fi
IFS=',' read -ra SKIP_LIST <<< "$SKIP_PROVIDERS"
Copy link

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should_skip_provider populates SKIP_LIST as a global array, which can unintentionally leak state outside the function; declare it as a local array (e.g., local -a ...) to keep the function side-effect free.

Suggested change
IFS=',' read -ra SKIP_LIST <<< "$SKIP_PROVIDERS"
local -a SKIP_LIST
local IFS=','
read -ra SKIP_LIST <<< "$SKIP_PROVIDERS"

Copilot uses AI. Check for mistakes.
for skip in "${SKIP_LIST[@]}"; do
# Trim whitespace
skip=$(echo "$skip" | xargs)
if [ "$skip" = "$provider" ]; then
return 0
fi
done
return 1
}

RESULTS=()
HARD_FAILURES=()

for provider_config in "${PROVIDERS[@]}"; do
# Split on " -> " to get provider and models
PROVIDER="${provider_config%% -> *}"
MODELS_STR="${provider_config#* -> }"

# Skip provider if it's in SKIP_PROVIDERS
if should_skip_provider "$PROVIDER"; then
echo "⊘ Skipping provider: ${PROVIDER} (SKIP_PROVIDERS)"
echo "---"
continue
fi
Comment on lines +122 to +127
Copy link

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If SKIP_PROVIDERS ends up skipping every provider, the script will currently report success (and print "All tests passed!") even though no tests ran; track whether any models were executed and fail (or at least print a clear warning and exit non-zero in CI) when the run is empty.

Copilot uses AI. Check for mistakes.

# Split models on "|"
IFS='|' read -ra MODELS <<< "$MODELS_STR"
for MODEL in "${MODELS[@]}"; do
Expand Down