test: drop unavailable and preview Gemini smoke models - #10355
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 33e45c8e0d
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| { name: 'gemini-3-pro-preview', flaky: true }, | ||
| 'gemini-3-flash-preview', | ||
| ], | ||
| models: [{ name: 'gemini-2.5-flash', flaky: true }], |
There was a problem hiding this comment.
With GOOGLE_API_KEY set, this list now discovers only a flaky Google case; registerTests() sends flaky cases through the catch-and-warn path in ui/desktop/tests/integration/test_providers_lib.ts lines 315-325, so any runGoose failure for Google is swallowed. The previous list still had non-flaky Google models, but after this change the provider smoke job can stay green even when Google auth/API/model wiring is completely broken; keep at least one available Google model non-flaky or update the default coverage rather than leaving all Google smoke coverage allowed to fail.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Addressed in 3a501da: gemini-2.5-flash is now non-flaky, so the Google provider remains required smoke coverage while unavailable and preview models are excluded.
|
Looking for a solid google model to put in the smoke |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3a501da4d5
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d2af265df8
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| { name: 'gemini-3-pro-preview', flaky: true }, | ||
| 'gemini-3-flash-preview', | ||
| ], | ||
| models: [{ name: 'gemini-2.5-flash', flaky: true }, 'gemini-3.5-flash'], |
There was a problem hiding this comment.
Gemini 2.5 Flash will be removed from the Gemini API on October 16, 2026. Which is a ways away but I wonder if it should also be removed now?
| const shellToolPattern = /(shell \| developer)|(▸.*shell)/; | ||
| return shellToolPattern.test(output) && output.includes(tokenA) && output.includes(tokenB); | ||
| } | ||
| ); | ||
|
|
||
| const shellToolPattern = /(shell \| developer)|(▸.*shell)/; |
There was a problem hiding this comment.
I'm just curious about this change and why shellToolPattern is redefined here? Couldn't shellToolPattern move above const output on L40 and then just use that from there on rather than defining it twice?
- feat: allow askai bot to read issues and PRs (aaif-goose#10398) - fix: Fix Gemini OAuth onboarding (aaif-goose#10363) - chore: remove unavailable models from cerebras catalog (aaif-goose#10407) - feat(cli): Fix output.rs merge conflicts - chore(deps): bump actions/stale from 10.3.0 to 10.4.0 (aaif-goose#10371) - chore(deps): bump dorny/paths-filter from 4.0.1 to 4.0.2 (aaif-goose#10370) - chore(deps): bump actions/cache from 6.0.0 to 6.1.0 (aaif-goose#10374) - fix: treat empty-string finish_reason as non-terminal in streaming (aaif-goose#10258) - chore(deps): bump step-security/harden-runner from 2.19.4 to 2.20.0 (aaif-goose#10372) - chore(deps): bump docker/login-action from 4.2.0 to 4.4.0 (aaif-goose#10373) - test: drop unavailable and preview Gemini smoke models (aaif-goose#10355) - fix(ui): restore pnpm 11 project configuration (aaif-goose#10395) - Fix/compact disabled during approval (aaif-goose#10089) - fix(prompt): make prompt timestamps timezone-explicit (aaif-goose#10209) - fix: preserve reasoning_content for DeepSeek thinking mode on multi-turn tool calls (aaif-goose#10366) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
3 participants
Summary
gemini-2.5-profrom Google smoke tests because the API reports it is no longer availablegemini-3-pro-previewandgemini-3-flash-previewsmoke coveragegemini-2.5-flashas the Google smoke-test modelVerification
pnpm exec prettier --check tests/integration/test_providers_lib.tspnpm exec eslint tests/integration/test_providers_lib.ts