Skip to content

test: isolate flaky smoke assertions - #1978

Merged
kevincodex1 merged 1 commit into
Twigpine:mainfrom
jatmn:fix/flaky-test-isolation
Jul 16, 2026
Merged

kevincodex1 merged 1 commit into
Twigpine:mainfrom
jatmn:fix/flaky-test-isolation

Conversation

@jatmn

@jatmn jatmn commented Jul 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • isolate the plan-mode legacy-reminder assertion from leaked feature-flag state
  • restore the real provider module around domain-check tests while preserving OpenAI/Gemini classification coverage

Validation

  • bun test --feature=UNATTENDED_RETRY src/tools/WebFetchTool/domainCheck.test.ts src/utils/messages/planMode.test.ts --rerun-each 20
  • bun run smoke
  • bun run test:full (scoped tests no longer fail; existing unrelated environment/baseline failures remain)

Final reviewed SHA: 1743c5c

Summary by CodeRabbit

  • Tests
    • Improved test isolation for domain access checks and plan mode behavior.
    • Prevented environment settings and feature flags from leaking between tests.
    • Ensured provider behavior is reset consistently after each test.

@coderabbitai

coderabbitai Bot commented Jul 16, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Changes

The test updates isolate provider module mocks in domain blocklist tests and isolate plan-mode environment and shared mutation state across tests.

Test isolation

Layer / File(s) Summary
Provider mock lifecycle
src/tools/WebFetchTool/domainCheck.test.ts
Dynamic provider imports and centralized cleanup replace per-test provider-selection mocks for OpenAI, Gemini, and first-party cases.
Plan-mode lifecycle isolation
src/utils/messages/planMode.test.ts
Tests acquire the shared mutation lock, pin the interview-phase environment variable, and restore state during cleanup.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

Suggested labels: bug

Suggested reviewers: kevincodex1

🚥 Pre-merge checks | ✅ 6 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Risk Surface Disclosed ⚠️ Warning PR touches provider-routing test code, but the description only covers isolation/validation and never explicitly flags the routing risk surface or whether it’s a blocker. Add a maintainer note stating the change is test-only, provider-routing behavior is unchanged, and no blocker/risk was introduced.
✅ Passed checks (6 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
No Hidden Policy Change ✅ Passed PASS: HEAD only changes two test files to isolate flaky state; no product, trust, routing, telemetry/network, or permission-policy code changes are hidden.
Title check ✅ Passed The title is concise and matches the diff’s focus on isolating flaky test assertions.
Description check ✅ Passed The description covers the summary and testing, but it omits the Impact and Notes sections from the template.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@jatmn jatmn added the bug Something isn't working label Jul 16, 2026
@jatmn jatmn self-assigned this Jul 16, 2026
@jatmn
jatmn marked this pull request as ready for review July 16, 2026 01:25

@jatmn jatmn left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kevincodex1 LGTM
1 of probably many.

@kevincodex1 kevincodex1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice one!

@kevincodex1
kevincodex1 merged commit 7ad96e9 into Twigpine:main Jul 16, 2026
5 checks passed
@jatmn
jatmn deleted the fix/flaky-test-isolation branch July 16, 2026 02:48
@devNull-bootloader

Copy link
Copy Markdown
Contributor

@coderabbitai why are PR checks failing

@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

CodeRabbit chat interactions are restricted to organization members for this repository. Ask an organization member to interact with CodeRabbit, or set chat.allow_non_org_members: true in your configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants