Skip to content

fix mcp integration test flakiness #4871

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jamadeo merged 8 commits into from
Oct 1, 2025
Merged

fix mcp integration test flakiness #4871

jamadeo merged 8 commits into from
Oct 1, 2025

Conversation

@jamadeo
Copy link
Collaborator

@jamadeo jamadeo commented Sep 29, 2025 •
edited
Loading

Concurrent cargo run commands don't work so well. This changes it to build the binary and then use that.

@jamadeo
Copy link
Collaborator Author

jamadeo commented Sep 29, 2025

Thanks @michaelneale for fixing the breaking test. The passing tests are still somewhat flaky for me when I run them locally, although I've yet to see them fail on CI in this way.

I tracked it down to the fact that it uses cargo run to run the MCP replay process. It seems that when cargo locks itself, the process is killed somewhere and the connection closed.

@alexhancock
Copy link
Collaborator

alexhancock commented Sep 29, 2025

when they flake, how do they flake?

i've experienced difficulty with recording (other output included etc), but not once we have a clean set of recordings made

@jamadeo
Copy link
Collaborator Author

jamadeo commented Sep 30, 2025

when they flake, how do they flake?

@alexhancock this affected replay mode, where the concurrent cargo run commands might mean one or more of the server processes exits without doing anything. The error you'd see is something like "connection closed before init response" or something like that.

@jamadeo jamadeo changed the title disable mcp integration tests until we fix flakiness fix mcp integration test flakiness Sep 30, 2025
@alexhancock
Copy link
Collaborator

alexhancock commented Sep 30, 2025

when they flake, how do they flake?

@alexhancock this affected replay mode, where the concurrent cargo run commands might mean one or more of the server processes exits without doing anything. The error you'd see is something like "connection closed before init response" or something like that.

I see

@jamadeo jamadeo merged commit 3b9a3b9 into main Oct 1, 2025
11 checks passed
@jamadeo jamadeo deleted the jackamadeo/disable-mcp-integration branch October 1, 2025 13:17
zanesq added a commit that referenced this pull request Oct 1, 2025
@zanesq
Merge branch 'main' of github.com:block/goose into zane/fix-extension…
5164f56 
…-deeplink

* 'main' of github.com:block/goose: (57 commits)
  Don't set agent props twice (#4872)
  fix: conversation fixer merges assistant text blocks and drops empty text messages (#4898)
  Batch fetch remaining issues for documentation updates
  fix: session timestamps (#4913)
  feat: lazy infinite scroller for session history view (#4922)
  chore: properly identify when to try oauth (#4918)
  Make token counter safer (#4924)
  Rename Hacktoberfest Blog to Hacktoberfest Content (#4926)
  Include Session ID appropriately in UI (#4901)
  fix mcp integration test flakiness (#4871)
  Add type field to empty schemas for anthropic (#4911)
  docs: remove subagents from experimental (#4907)
  CLI: dont show logs to user (#4902)
  patching the security scanner to redact environment variables (#4908)
  rmcp upgrade (#4792)
  feat: Use the screen, goose (#4905)
  added claude-sonnet-4-5 (#4906)
  Tiny: fix github casing  (#4903)
  remove anyOf from create_task tool (#4897)
  chore(deps): bump tracing-subscriber from 0.3.19 to 0.3.20 (#4442)
  ...
zanesq added a commit that referenced this pull request Oct 1, 2025
@zanesq
Merge branch 'main' of github.com:block/goose into zane/fix-auto-scroll
9a79394 
* 'main' of github.com:block/goose:
  Don't set agent props twice (#4872)
  fix: conversation fixer merges assistant text blocks and drops empty text messages (#4898)
  Batch fetch remaining issues for documentation updates
  fix: session timestamps (#4913)
  feat: lazy infinite scroller for session history view (#4922)
  chore: properly identify when to try oauth (#4918)
  Make token counter safer (#4924)
  Rename Hacktoberfest Blog to Hacktoberfest Content (#4926)
  Include Session ID appropriately in UI (#4901)
  fix mcp integration test flakiness (#4871)
  Add type field to empty schemas for anthropic (#4911)
  docs: remove subagents from experimental (#4907)
  CLI: dont show logs to user (#4902)
  patching the security scanner to redact environment variables (#4908)
  rmcp upgrade (#4792)
  feat: Use the screen, goose (#4905)

# Conflicts:
#	ui/desktop/src/components/ui/scroll-area.tsx
michaelneale added a commit that referenced this pull request Oct 2, 2025
@michaelneale
Merge branch 'main' into micn/g3
8bc8273 
* main:
  Fix auto scroll to bottom during chat (#4923)
  Fix Typo, Add Description to Hacktoberfest Content Issue Template (#4931)
  Don't set agent props twice (#4872)
  fix: conversation fixer merges assistant text blocks and drops empty text messages (#4898)
  Batch fetch remaining issues for documentation updates
  fix: session timestamps (#4913)
  feat: lazy infinite scroller for session history view (#4922)
  chore: properly identify when to try oauth (#4918)
  Make token counter safer (#4924)
  Rename Hacktoberfest Blog to Hacktoberfest Content (#4926)
  Include Session ID appropriately in UI (#4901)
  fix mcp integration test flakiness (#4871)
michaelneale added a commit that referenced this pull request Oct 2, 2025
@michaelneale
Merge branch 'main' into micn/tunnel-access
b1fddbf 
* main: (44 commits)
  Add PR template (#4934)
  Using --resume with --name should still accept session IDs (#4937)
  Fix auto scroll to bottom during chat (#4923)
  Fix Typo, Add Description to Hacktoberfest Content Issue Template (#4931)
  Don't set agent props twice (#4872)
  fix: conversation fixer merges assistant text blocks and drops empty text messages (#4898)
  Batch fetch remaining issues for documentation updates
  fix: session timestamps (#4913)
  feat: lazy infinite scroller for session history view (#4922)
  chore: properly identify when to try oauth (#4918)
  Make token counter safer (#4924)
  Rename Hacktoberfest Blog to Hacktoberfest Content (#4926)
  Include Session ID appropriately in UI (#4901)
  fix mcp integration test flakiness (#4871)
  Add type field to empty schemas for anthropic (#4911)
  docs: remove subagents from experimental (#4907)
  CLI: dont show logs to user (#4902)
  patching the security scanner to redact environment variables (#4908)
  rmcp upgrade (#4792)
  feat: Use the screen, goose (#4905)
  ...
zanesq added a commit that referenced this pull request Oct 2, 2025
@zanesq
Merge branch 'main' of github.com:block/goose into zane/create-recipe…
fe28b24 
…-unification

* 'main' of github.com:block/goose: (24 commits)
  Fix auto scroll to bottom during chat (#4923)
  Fix Typo, Add Description to Hacktoberfest Content Issue Template (#4931)
  Don't set agent props twice (#4872)
  fix: conversation fixer merges assistant text blocks and drops empty text messages (#4898)
  Batch fetch remaining issues for documentation updates
  fix: session timestamps (#4913)
  feat: lazy infinite scroller for session history view (#4922)
  chore: properly identify when to try oauth (#4918)
  Make token counter safer (#4924)
  Rename Hacktoberfest Blog to Hacktoberfest Content (#4926)
  Include Session ID appropriately in UI (#4901)
  fix mcp integration test flakiness (#4871)
  Add type field to empty schemas for anthropic (#4911)
  docs: remove subagents from experimental (#4907)
  CLI: dont show logs to user (#4902)
  patching the security scanner to redact environment variables (#4908)
  rmcp upgrade (#4792)
  feat: Use the screen, goose (#4905)
  added claude-sonnet-4-5 (#4906)
  Tiny: fix github casing  (#4903)
  ...

# Conflicts:
#	ui/desktop/src/components/BaseChat.tsx
HikaruEgashira pushed a commit to HikaruEgashira/goose that referenced this pull request Oct 3, 2025
@jamadeo @michaelneale @HikaruEgashira
fix mcp integration test flakiness (block#4871)
2fcae67 
Co-authored-by: Michael Neale <[email protected]>
Signed-off-by: HikaruEgashira <[email protected]>
This was referenced Oct 8, 2025
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alexhancock alexhancock alexhancock approved these changes

@DOsinga DOsinga DOsinga approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@jamadeo @alexhancock @DOsinga @michaelneale