fix(ci): mark openai/gpt-5 smoke test as flaky - #9027
Conversation
…om failing the build - Mark `openai / gpt-5` as `flaky: true` in `ui/desktop/tests/integration/test_providers_lib.ts` so the existing flaky-test handler (longer 90s vitest timeout + caught `runGoose` rejection) treats timeouts as allowed instead of red CI. - Addresses repeated `Smoke Tests (Code Execution)` failures on `main` and PR branches where gpt-5 spends the turn calling `list_functions` / `get_function_details` (Memory, Todo) and never reaches `code_execution` before the 55s `runGoose` timeout. The faster non-code-exec smoke test (which gpt-5 passes in ~5s) is unaffected. - Mirrors the existing pattern (commit 097e1bc, "fix(ci): prevent flaky smoke test timeouts from failing the build") used for gpt-3.5-turbo, qwen3-coder:exacto, gemini-2.5-flash, gemini-3-pro-preview, and nemotron-3-nano. Signed-off-by: Taylor Ho <taylorkmho@gmail.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 185c6187cf
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| 'gpt-4o', | ||
| 'gpt-4o-mini', | ||
| { name: 'gpt-3.5-turbo', flaky: true }, | ||
| { name: 'gpt-5', flaky: true }, |
There was a problem hiding this comment.
Marking gpt-5 as flaky in the shared provider list changes behavior for both integration suites, not just code-exec: test_providers.test.ts and test_providers_code_exec.test.ts both consume discoverTestCases() from this file, and registerTests converts flaky failures into warnings. This means any future real failure in the normal smoke test for openai/gpt-5 will no longer fail CI, which weakens regression detection outside the timeout scenario this commit targets.
Useful? React with 👍 / 👎.
tellaho
changed the title
* main: feat: move goose2 provider catalog behind ACP layer (#9030) fix: use python3 in developer extension instructions for macOS/Linux compatibility (#8784) fix(acp): synchronously reap ACP child to avoid SIGCHLD race (#9023) fix goose2 small-window chat and settings layouts (#9019) docs: improve goose2 AGENTS.md (#9028) agents: add CLAUDE.mds to mirror AGENTS.mds (#9029) remove skill categories (#9008) fix: 8531 - elicitation fixes (#8999) feat(chat): group consecutive tool calls into one summarized chain card (#8995) fix(ci): mark openai/gpt-5 smoke test as flaky (#9027) goose2 distribution bundling (#8911) Add "Trimmed trailing whitespace" message to moim whitelist (#8847)
Signed-off-by: Taylor Ho <taylorkmho@gmail.com>
2 participants
Summary
The
Smoke Tests (Code Execution)job has been failing repeatedly onmain(and on PR branches once they merge inmain) on theinvokes code_execution tool — 'openai' / 'gpt-5'test case. The model spends its turn callinglist_functionsandget_function_detailsforMemoryandTodo, never reachingcode_executionbefore the 55srunGoosetimeout fires.This change marks
openai / gpt-5asflaky: true, mirroring the established pattern from #8837 (fix(ci): prevent flaky smoke test timeouts from failing the build) already used forgpt-3.5-turbo,qwen/qwen3-coder:exacto,gemini-2.5-flash,gemini-3-pro-preview, andnvidia/nemotron-3-nano-30b-a3b:free.For flaky entries:
runGooserejection fires firstconsole.warn(`Flaky test ... failed (allowed): ${err}`)instead of failing the buildThe faster non-code-exec smoke test (which gpt-5 passes in ~5s) is unaffected — only how timeouts are treated changes.
Evidence
main: https://github.com/aaif-goose/goose/actions/runs/25366266244 (gpt-5 timed out at 55007ms in code-exec)main: 25329705071, 25325931805, 25319479744, 25297284836, 25296973706Test plan
Smoke Testsjob (regular) still passes — gpt-5 succeeds there in ~5sSmoke Tests (Code Execution)job no longer fails the build when gpt-5 times out; check the run for aFlaky test openai/gpt-5 failed (allowed)warning if it does time out