test(cli): fix order-dependent test_resume_quiet_stderr flake at the source - #68872
Closed
mehmetkr-31 wants to merge 1 commit into
Closed
test(cli): fix order-dependent test_resume_quiet_stderr flake at the source#68872mehmetkr-31 wants to merge 1 commit into
mehmetkr-31 wants to merge 1 commit into
Conversation
…source test_session_not_found_goes_to_stdout_in_full_mode passes in isolation but fails in a full tests/cli run. Two independent leaks from the same neighbor test conspire: 1. test_cli_init.py's _make_cli() reloads cli.py while prompt_toolkit is stubbed with MagicMocks and never reloads it back, so sys.modules['cli'] is left with a mock _pt_print/_PT_ANSI and cli._cprint silently no-ops for every later test. Fixed by reloading cli once more with the real modules visible (try/finally). 2. prompt_toolkit's print_formatted_text caches its Output on the process-global default AppSession the first time it renders without an explicit output=. Under capsys (which swaps sys.stdout per test), the first CLI test to emit through _cprint locks that cache onto its own captured stdout, so later capsys tests read an empty buffer. Fixed with an autouse fixture in a new tests/cli/conftest.py that resets the cached output around each test. Neither change touches production code or the flaky test's own assertion. Related to NousResearch#59358 (which addresses the same flaky test by mocking _cprint in the assertion instead; this fixes the two underlying leaks at the source and does not modify test_resume_quiet_stderr.py). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Merged via PR #74553 (commit c15aab9, your authorship preserved, contributor mapping added via contributors/emails/). Excellent root-cause work — both legs verified (stale mock globals after cli reload + prompt_toolkit AppSession output cache pinned to a dead capsys buffer), reproduced both orders pre-fix, 723/723 tests/cli green post-fix. The release.py AUTHOR_MAP hunk was rerouted to contributors/emails/ (the dict is frozen). Thanks @mehmetkr-31! |
This was referenced Jul 30, 2026
randlee
pushed a commit
to randlee/hermes-agent
that referenced
this pull request
Aug 11, 2026
…ontributors/emails/ The NousResearch#68873 salvage re-added a line to the frozen dict; the canonical mapping (contributors/emails/mehmet.kar@std.yildiz.edu.tr) already exists from the NousResearch#68872 salvage.
33hodl
pushed a commit
to 33hodl/hermes-agent
that referenced
this pull request
Aug 12, 2026
…ontributors/emails/ The NousResearch#68873 salvage re-added a line to the frozen dict; the canonical mapping (contributors/emails/mehmet.kar@std.yildiz.edu.tr) already exists from the NousResearch#68872 salvage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tests/cli/test_resume_quiet_stderr.py::TestResumeQuietStderr::test_session_not_found_goes_to_stdout_in_full_modepasses in isolation and in its own file but fails in a fulltests/clirun — a long-standing order-dependent flake. This fixes the two independent leaks at their source rather than papering over the assertion.Root cause (two independent leaks, both from
test_cli_init.py::_make_cli)clileft with mock prompt_toolkit globals._make_cli()reloadscli.pyinsidepatch.dict(sys.modules, prompt_toolkit_stubs)and never reloads it back.patch.dictrestoressys.moduleson exit, but not the names the reloaded module already bound — sosys.modules["cli"]keeps aMagicMock_pt_print/_PT_ANSI, andcli._cprintsilently no-ops for every later test.prompt_toolkit output cache pinned to a stale capture buffer.
print_formatted_text(used by_cprint) caches itsOutputon the process-global defaultAppSessionthe first time it renders with no explicitoutput=, and never re-readssys.stdout. Undercapsys(which swapssys.stdoutper test), the first CLI test to emit through_cprint— e.g. anyTestBusyInputModetest that runs/queueand prints a "Queued: …" line — locks that cache onto its captured stdout. Every latercapsystest asserting on_cprintoutput then reads an empty buffer. Confirmed by tracing the cached output to aPlainTextOutputbound to the polluter's capture fd while the victim'ssys.stdoutwas a different object.Each leak alone is enough to blank the victim's captured stdout; with leak 1 active the polluter's
_cprintis a mock and never populates the PT cache, so fixing 1 exposes 2 — both must be addressed.Fix (test-only, no production change)
tests/cli/test_cli_init.py: wrap_make_cli's stubbed reload intry/finallyand reloadclionce more with the real modules visible, sosys.modules["cli"]rebinds clean PT globals.tests/cli/conftest.py(new): an autouse fixture that resets the defaultAppSession's cached_outputaround each test, so every CLI test re-creates a fresh prompt_toolkit output bound to its ownsys.stdout. This fixes the whole class ofcapsys+ prompt_toolkit flakes, not just this one test.Neither change touches production code or the flaky test's own assertion.
Verification
Relation to #59358
#59358 targets the same flaky test and identifies the same prompt_toolkit/capsys interaction, but fixes it by
patch("cli._cprint")and asserting on the mock's call args — a test-local assertion workaround that leaves both underlying leaks in place (othercapsystests remain exposed, and the_pt_printmock leak is untouched). This PR instead removes the two leaks at the source and does not modifytest_resume_quiet_stderr.py, so the two changes don't conflict; if #59358 lands first, its assertion change simply becomes redundant. Flagging for the maintainer to pick the preferred approach.🤖 Generated with Claude Code