Skip to content

test(run_agent): split monolithic test_run_agent.py to fix CI shard timeout - #66

Merged
Kyzcreig merged 2 commits into
mainfrom
fix/test-run-agent-split
Jun 21, 2026
Merged

test(run_agent): split monolithic test_run_agent.py to fix CI shard timeout#66
Kyzcreig merged 2 commits into
mainfrom
fix/test-run-agent-split

Conversation

@Kyzcreig

@Kyzcreig Kyzcreig commented Jun 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes the recurring Fleet CI-fail alert — tests/run_agent/test_run_agent.py
was timing out the sharded test job (RED on main and on every PR).

Root cause

scripts/run_tests_parallel.py runs one subprocess per file and caps each at
_DEFAULT_FILE_TIMEOUT_SECONDS = 140.0. It shards across files but cannot split
within one, so the slowest single file sets a hard floor on the matrix.
test_run_agent.py had grown to 6,619 lines / 378 tests and blew past 140s — it
was SIGKILL'd at ~10% complete, so a shard with 4269 passed, 0 failed still went
RED purely on that one file's timeout (✗ … (378 tests, 140.0s) →
1 file where no tests ran (timeout before collection)).

Fix — split the monolith (not bump the cap)

Bumping the timeout is a band-aid that re-reds on the next growth and lengthens the
critical-path shard. Instead, split into 8 theme-focused modules matching the
directory's existing per-theme layout:

file tests
test_run_agent_api_kwargs.py 90
test_run_agent_tool_exec.py 61
test_run_agent_conversation.py 59
test_run_agent_misc.py 51
test_run_agent_reasoning.py 45
test_run_agent_streaming.py 28
test_run_agent_providers.py 26
test_run_agent_init_memory.py 18

Shared fixtures (agent, agent_with_memory_tool) → conftest.py (auto-inject);
mock-builders (_mock_response, _make_chunk, …) → _run_agent_helpers.py.
Pure move — zero test-body or production-code edits. Repointed the one sibling
(test_partial_stream_finish_reason.py) that imported helpers from the old module.

Faithful-split proof (behavior contract, not snapshot)

  • Collection count for tests/run_agent/ unchanged: 1671 before and after.
  • The 8 new files collect exactly the original 378 (51+45+18+28+59+90+61+26).
  • Full dir green via the CI runner: 116 files, 1668 passed, 0 failed.
  • Slowest single file now 50s (was >140s → killed); every file well under cap.

Follow-up (separate PR, not here)

A durable guard in run_tests_parallel.py so no future fat file re-reds the matrix:
auto-split a file whose recorded duration approaches the cap, + a loud advisory when
any single file crosses ~75% of the cap. Spec'd; intentionally out of scope for this
hotfix.

@Kyzcreig
test(run_agent): split monolithic test_run_agent.py to fix CI shard t…
3d908d2 
…imeout

tests/run_agent/test_run_agent.py had grown to 6,619 lines / 378 tests and
exceeded the 140s per-file wall-clock cap in scripts/run_tests_parallel.py
(the runner spawns one subprocess per file and cannot split within a file, so
the slowest single file pins the whole sharded matrix). The file was being
SIGKILL'd at ~10% complete, turning a green 4269-pass shard RED on a timeout —
red on main and on every PR, firing the Fleet CI-fail alert.

Split the monolith into 8 theme-focused modules (reasoning, streaming, api_kwargs,
tool_exec, conversation, providers, init_memory, misc), moving the shared
fixtures (agent, agent_with_memory_tool) into conftest.py and the mock-builder
helpers into _run_agent_helpers.py. Pure move — zero test-body or production
edits. Repointed the one sibling (test_partial_stream_finish_reason.py) that
imported helpers from the old module.

Faithful-split proof:
- collection count for tests/run_agent/ unchanged: 1671 before and after.
- the 8 new files collect exactly the original 378 (51+45+18+28+59+90+61+26).
- full dir green via the CI runner: 116 files, 1668 passed, 0 failed.
- slowest single file now 50s (was >140s -> killed); max well under the cap.

Matches the repo's existing per-theme test layout in tests/run_agent/ and the
'refactor god-files into clean modules' guidance in AGENTS.md.
@github-actions

github-actions Bot commented Jun 21, 2026 •
edited
Loading

Copy link
Copy Markdown

🔎 Lint report: fix/test-run-agent-split vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 11218 on HEAD, 11217 on base (🆕 +1)

🆕 New issues (15):

Rule Count
unresolved-attribute 9
unresolved-import 4
invalid-assignment 1
invalid-argument-type 1
First entries 
tests/run_agent/test_run_agent_streaming.py:10: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
tests/run_agent/test_credits_notices_toggle.py:76: [invalid-assignment] invalid-assignment: Object of type `None` is not assignable to attribute `_credits_session_start_micros` of type `int`
tests/run_agent/test_run_agent_init_memory.py:190: [unresolved-attribute] unresolved-attribute: Object of type `AIAgent` has no attribute `max_tokens`
tests/run_agent/test_run_agent_init_memory.py:100: [unresolved-attribute] unresolved-attribute: Object of type `AIAgent` has no attribute `api_mode`
tests/run_agent/test_run_agent_init_memory.py:246: [unresolved-attribute] unresolved-attribute: Object of type `AIAgent` has no attribute `session_id`
tests/run_agent/test_run_agent_init_memory.py:211: [unresolved-attribute] unresolved-attribute: Object of type `AIAgent` has no attribute `_cache_ttl`
tests/run_agent/test_run_agent_init_memory.py:101: [unresolved-attribute] unresolved-attribute: Object of type `AIAgent` has no attribute `_use_prompt_caching`
tests/run_agent/test_run_agent_tool_exec.py:15: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
tests/run_agent/test_run_agent_tool_exec.py:1056: [invalid-argument-type] invalid-argument-type: Argument to function `_paths_overlap` is incorrect: Expected `Path`, found `Path | None`
tests/run_agent/test_run_agent_providers.py:488: [unresolved-attribute] unresolved-attribute: Unresolved attribute `provider` on type `AIAgent`
tests/run_agent/test_run_agent_init_memory.py:228: [unresolved-attribute] unresolved-attribute: Object of type `AIAgent` has no attribute `valid_tool_names`
tests/run_agent/test_run_agent_reasoning.py:9: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
tests/run_agent/test_run_agent_misc.py:129: [unresolved-attribute] unresolved-attribute: Object of type `AIAgent` has no attribute `model`
tests/run_agent/test_run_agent_conversation.py:1014: [unresolved-attribute] unresolved-attribute: Unresolved attribute `status_code` on type `Exception`
tests/run_agent/test_run_agent_misc.py:14: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`

✅ Fixed issues (13):

Rule Count
unresolved-attribute 11
unresolved-import 1
invalid-argument-type 1
First entries 
tests/run_agent/test_run_agent.py:4093: [unresolved-attribute] unresolved-attribute: Unresolved attribute `status_code` on type `Exception`
tests/run_agent/test_run_agent.py:5729: [unresolved-attribute] unresolved-attribute: Unresolved attribute `provider` on type `AIAgent`
tests/run_agent/test_credits_notices_toggle.py:76: [unresolved-attribute] unresolved-attribute: Unresolved attribute `_credits_session_start_micros` on type `AIAgent`
tests/run_agent/test_run_agent.py:910: [unresolved-attribute] unresolved-attribute: Object of type `AIAgent` has no attribute `max_tokens`
tests/run_agent/test_run_agent.py:931: [unresolved-attribute] unresolved-attribute: Object of type `AIAgent` has no attribute `_cache_ttl`
tests/run_agent/test_run_agent.py:820: [unresolved-attribute] unresolved-attribute: Object of type `AIAgent` has no attribute `api_mode`
tests/run_agent/test_run_agent.py:821: [unresolved-attribute] unresolved-attribute: Object of type `AIAgent` has no attribute `_use_prompt_caching`
tests/run_agent/test_run_agent.py:948: [unresolved-attribute] unresolved-attribute: Object of type `AIAgent` has no attribute `valid_tool_names`
tests/run_agent/test_run_agent.py:966: [unresolved-attribute] unresolved-attribute: Object of type `AIAgent` has no attribute `session_id`
tests/run_agent/test_run_agent.py:195: [unresolved-attribute] unresolved-attribute: Object of type `AIAgent` has no attribute `model`
tests/run_agent/test_run_agent.py:20: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
run_agent.py:2931: [unresolved-attribute] unresolved-attribute: Object of type `Self@get_credits_spent_micros` has no attribute `_credits_session_start_micros`
tests/run_agent/test_run_agent.py:3072: [invalid-argument-type] invalid-argument-type: Argument to function `_paths_overlap` is incorrect: Expected `Path`, found `Path | None`

Unchanged: 5846 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

@greptile-apps

greptile-apps Bot commented Jun 21, 2026 •
edited
Loading

Copy link
Copy Markdown

Greptile Summary

This PR splits the 6,619-line test_run_agent.py monolith into 8 theme-focused modules to fix CI shard timeouts caused by the per-file 140-second wall-clock cap in run_tests_parallel.py. Shared fixtures move to conftest.py and mock-builders to _run_agent_helpers.py; no test logic or production code is changed.

  • Deleted test_run_agent.py (378 tests, 6,619 lines) and redistributed all tests verbatim across test_run_agent_{api_kwargs,tool_exec,conversation,misc,reasoning,streaming,providers,init_memory}.py — parametrized counts confirmed at 378 total.
  • Added _run_agent_helpers.py for shared mock-builders and updated conftest.py with the agent / agent_with_memory_tool fixtures; test_partial_stream_finish_reason.py import path corrected to the new module.
  • Added .gitleaks.toml to suppress false-positive secret-scan hits on the obviously-fake API key literals used throughout the test fixtures.

Confidence Score: 5/5

Safe to merge — this is a pure reorganisation of test code with no changes to production logic, test assertions, or fixture behaviour.

All 378 tests are accounted for (parametrized counts verified), every new file imports correctly from the shared helpers and conftest, and the only production-adjacent change is the corrected import in test_partial_stream_finish_reason.py. Nothing in the split alters what is actually tested.

No files require special attention.

Important Files Changed
Filename Overview
tests/run_agent/test_run_agent.py 6,619-line monolith deleted after its contents were split into 8 theme-focused modules plus conftest.py and _run_agent_helpers.py.
tests/run_agent/_run_agent_helpers.py New shared helper module extracted verbatim from the old monolith; provides _mock_response, _make_chunk, _mock_tool_call, _mock_assistant_msg, _make_tc_delta, and _make_tool_defs used by all split test files.
tests/run_agent/conftest.py Added agent and agent_with_memory_tool fixtures moved from the old monolith; contains a dead _make_tool_defs import (already flagged in a prior review thread) but otherwise correct.
.gitleaks.toml New gitleaks config that allowlists short, specific stopwords for known fake test-fixture API keys; the short length of two stopwords (4 chars each) is a minor theoretical broadness concern but author reports negative-injection testing.
tests/run_agent/test_partial_stream_finish_reason.py Single import line updated from old monolith to new _run_agent_helpers module; straightforward correctness fix.
tests/run_agent/test_run_agent_api_kwargs.py 90 tests covering build_api_kwargs, system-prompt building, tool-use enforcement, and max-tokens; clean split with correct imports from _run_agent_helpers and conftest fixtures.
tests/run_agent/test_run_agent_tool_exec.py 61 tests covering tool execution, dispatch, and error handling; clean split.
tests/run_agent/test_run_agent_conversation.py 59 tests covering conversation loop behaviour; clean split, imports and fixture injection intact.
tests/run_agent/test_run_agent_misc.py 51 tests covering normalization, redaction, masking, safe-writer, and snapshot helpers; clean split.
tests/run_agent/test_run_agent_reasoning.py 43 function definitions (45 parameterised cases) covering reasoning-block extraction and injection; clean split.
tests/run_agent/test_run_agent_streaming.py 28 tests for streaming chunk handling; clean split.
tests/run_agent/test_run_agent_providers.py 26 tests for provider-specific behaviour (Anthropic, OpenRouter, Gemini); clean split.
tests/run_agent/test_run_agent_init_memory.py 18 tests covering AIAgent init, interrupt, and memory nudge/context; clean split.

Flowchart

 
%%{init: {'theme': 'neutral'}}%%
flowchart TD
    OLD["test_run_agent.py\n(6,619 lines / 378 tests)\n❌ killed by 140 s cap"]

    OLD -->|split| SHARED
    OLD -->|split| F1
    OLD -->|split| F2
    OLD -->|split| F3
    OLD -->|split| F4
    OLD -->|split| F5
    OLD -->|split| F6
    OLD -->|split| F7
    OLD -->|split| F8

    subgraph SHARED ["Shared infrastructure"]
        HELPERS["_run_agent_helpers.py\n_mock_response / _make_chunk\n_mock_tool_call / _make_tc_delta\n_make_tool_defs"]
        CONF["conftest.py\nagent fixture\nagent_with_memory_tool fixture"]
    end

    subgraph MODULES ["8 theme-focused modules (all < 50 s)"]
        F1["test_run_agent_api_kwargs.py\n90 tests"]
        F2["test_run_agent_tool_exec.py\n61 tests"]
        F3["test_run_agent_conversation.py\n59 tests"]
        F4["test_run_agent_misc.py\n51 tests"]
        F5["test_run_agent_reasoning.py\n45 tests"]
        F6["test_run_agent_streaming.py\n28 tests"]
        F7["test_run_agent_providers.py\n26 tests"]
        F8["test_run_agent_init_memory.py\n18 tests"]
    end

    HELPERS --> F1 & F2 & F3 & F4 & F5 & F6 & F7 & F8
    CONF --> F1 & F2 & F3 & F4 & F5 & F6 & F7 & F8

    SIBLING["test_partial_stream_finish_reason.py"]
    HELPERS --> SIBLING
Loading 
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    OLD["test_run_agent.py\n(6,619 lines / 378 tests)\n❌ killed by 140 s cap"]

    OLD -->|split| SHARED
    OLD -->|split| F1
    OLD -->|split| F2
    OLD -->|split| F3
    OLD -->|split| F4
    OLD -->|split| F5
    OLD -->|split| F6
    OLD -->|split| F7
    OLD -->|split| F8

    subgraph SHARED ["Shared infrastructure"]
        HELPERS["_run_agent_helpers.py\n_mock_response / _make_chunk\n_mock_tool_call / _make_tc_delta\n_make_tool_defs"]
        CONF["conftest.py\nagent fixture\nagent_with_memory_tool fixture"]
    end

    subgraph MODULES ["8 theme-focused modules (all < 50 s)"]
        F1["test_run_agent_api_kwargs.py\n90 tests"]
        F2["test_run_agent_tool_exec.py\n61 tests"]
        F3["test_run_agent_conversation.py\n59 tests"]
        F4["test_run_agent_misc.py\n51 tests"]
        F5["test_run_agent_reasoning.py\n45 tests"]
        F6["test_run_agent_streaming.py\n28 tests"]
        F7["test_run_agent_providers.py\n26 tests"]
        F8["test_run_agent_init_memory.py\n18 tests"]
    end

    HELPERS --> F1 & F2 & F3 & F4 & F5 & F6 & F7 & F8
    CONF --> F1 & F2 & F3 & F4 & F5 & F6 & F7 & F8

    SIBLING["test_partial_stream_finish_reason.py"]
    HELPERS --> SIBLING
Loading

Reviews (2): Last reviewed commit: "chore(secret-scan): allowlist fake test-..." | Re-trigger Greptile

greptile-apps[bot]
greptile-apps Bot reviewed Jun 21, 2026
View reviewed changes
Comment thread tests/run_agent/conftest.py

from run_agent import AIAgent # noqa: E402

from tests.run_agent._run_agent_helpers import _make_tool_defs # noqa: E402,F401

@greptile-apps greptile-apps Bot Jun 21, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 _make_tool_defs is imported here but never referenced inside conftest.py — every test file that needs it already imports directly from _run_agent_helpers. Pytest conftest doesn't re-export module-level names as fixtures, so this import is dead code suppressed only by # noqa: F401. Removing it keeps the file self-consistent and avoids misleading future readers into thinking it's somehow available via conftest injection.

Suggested change
from tests.run_agent._run_agent_helpers import _make_tool_defs # noqa: E402,F401
from tests.run_agent._run_agent_helpers import _make_tool_defs # noqa: E402

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@Kyzcreig
chore(secret-scan): allowlist fake test-fixture API keys in gitleaks
cf64ac4 
Splitting test_run_agent.py re-introduced its long-standing placeholder api_key
fixtures as 'added' lines, so the diff-scoped gitleaks scan flagged them as new
(they were never real — the OpenAI/Anthropic clients are mocked, no network call).

Add a .gitleaks.toml that extends the default ruleset and allowlists ONLY the
distinctive tokens of those known fakes via stopwords:
  - 1234567890  (test-key-1234567890 constructor stub)
  - 2mno        (gsk_ab...2mno / sk-ant...2mno redaction fixtures)
  - mnop        (sk-or-...mnop key-masking fixture)

Verified against the CI-pinned gitleaks 8.18.4: the PR range scans clean, and a
freshly-injected high-entropy api_key in the same test file is still CAUGHT — the
allowlist exempts only the known fakes, not the rule.
@Kyzcreig
Kyzcreig merged commit 8f7c231 into main Jun 21, 2026
37 checks passed
@Kyzcreig
Kyzcreig deleted the fix/test-run-agent-split branch June 21, 2026 00:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@greptile-apps greptile-apps[bot] greptile-apps[bot] left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Kyzcreig