test(gateway): drop a wait that was never synchronising anything - #89481
test(gateway): drop a wait that was never synchronising anything#89481jackulau wants to merge 1 commit into
Conversation
No blocking issues found. |
|
Thanks for the review. Taking the nit seriously sent me back to the code, and it The premise does not holdThe description says the enqueue "does not happen inline — the hook hands off to So I checked it instead of reasoning about it. Removing the wait entirely 8 for 8. If the assertion were racing a spawned task, a zero-wait run would be Which makes the flake something elseIf the enqueue is inline, then a CI run that reports A bounded poll cannot fix any of those. It would wait 2 seconds and then fail What I would like to do about itWhichever the maintainers prefer:
I am happy to do either, and I would lean towards (1) plus a separate issue for On your actual nit, for completeness: you were right that an event or awaiting |
test_runner_goal_hook_enqueues_into_the_key_the_adapter_drains slept a fixed 50ms between calling the post-turn goal hook and asserting the enqueue landed. It has been flaking in CI on unrelated PRs with: AssertionError: continuation enqueued under a different key than the adapter drains: pending keys=[] expected=agent:main:slack:channel:C1:1718600000.000100 An earlier revision of this change replaced the sleep with a bounded poll, on the theory that the enqueue was performed by a task the hook spawns and the assertion was racing the event loop. That theory is wrong. _post_turn_goal_continuation contains no create_task, no ensure_future and no call_soon; every step is awaited, and it ends in a synchronous _enqueue_fifo. _defer_goal_status_notice_after_delivery, the one call that looks deferred, either registers an adapter callback and returns or awaits _deliver() inline, and it sends a status notice rather than enqueueing. Removing the wait entirely passes 15 out of 15 runs. If the assertion were racing a spawned task, the zero-wait run is the one that would fail every time. So the sleep was not too short, it was measuring nothing, and the poll would only have made a failure arrive two seconds later. Which means the CI flake is not a timing bug in this test at all. `pending keys=[]` cannot be a lost race; it means the hook took a path that never enqueued: one of the early returns (no session id, mgr.is_active() false, should_continue false, no prompt) or the `except Exception` around the enqueue, which swallows into a logger.debug. This change therefore does not claim to fix the flake. It removes a wait that made the test look synchronised when it was not, and leaves a comment saying why adding a sleep back would be the wrong response to the next failure.
3d61418 to
c4faa25
Compare
|
Went with option (1) and pushed it, since leaving the PR standing on a premise I What is on the branch now:
Still worth someone opening an investigation into those enqueue-path returns. |
What does this PR do?
tests/gateway/test_goal_continuation_drain.py::test_runner_goal_hook_enqueues_into_the_key_the_adapter_drainsslept a fixed 50ms between calling the post-turn goal hook and asserting the
enqueue landed. This removes that wait. It does not claim to fix the CI
flake, and the rewrite below explains why, because an earlier revision of this
PR did claim exactly that and was wrong.
What this PR used to say, and why it was wrong
The original version replaced the sleep with a bounded poll, on the stated
theory that "the enqueue does not happen inline, the hook hands off to a task,
so the assertion is racing the event loop".
There is no such handoff.
_post_turn_goal_continuationcontains noasyncio.create_task, noensure_futureand nocall_soon; every step in itis awaited, and it ends in a synchronous
self._enqueue_fifo(...)._defer_goal_status_notice_after_delivery, the one call in there that looksdeferred, either registers an adapter callback and returns or awaits
_deliver()directly, and what it delivers is a status notice rather than theenqueue.
So I measured it instead of arguing about it. Removing the wait entirely
(no sleep, no poll) on current
main:If the assertion were racing a spawned task, the zero-wait run is precisely the
one that would fail every time. The 50ms sleep was not too short. It was
measuring nothing.
Credit to @Enough1122, whose review nit ("an event or awaiting the task would be
strictly deterministic") is what sent me back to the code. The answer turned out
to be that there is no task to await, so no production test hook is needed
either.
What that means for the flake
The CI failure this test produces is:
If the enqueue is inline,
pending keys=[]cannot be a lost race. It means thehook took a path that never enqueued at all: one of the early returns (no
session id,
mgr.is_active()false,should_continuefalse, no prompt), or theexcept Exceptionwrapped around the enqueue, which swallows the error into alogger.debug. A bounded poll cannot fix any of those. It would wait twoseconds and fail with the same message.
My current suspicion is goals-DB contention upstream of
mgr.is_active(), sincethe test writes a goal via
GoalManagerand then depends on reading it back,but I have not proven that and am not claiming it here. Someone should still
open an investigation into the enqueue-path returns; I did not want the
observed CI failure marked resolved by a change that cannot resolve it.
Related Issue
None. This came out of a flake observed on unrelated PRs.
Type of Change
Changes Made
tests/gateway/test_goal_continuation_drain.py: delete theawait asyncio.sleep(0.05)between the hook call and the assertion, replacing it with a comment that
records why the hook is already synchronous, and why adding a sleep back would
be the wrong response to the next failure here.
No production code is touched, and no other test changes.
How to Test
pytest tests/gateway/test_goal_continuation_drain.py -q— 2 passed.for i in $(seq 1 15); do pytest tests/gateway/...::test_runner_goal_hook_enqueues_into_the_key_the_adapter_drains -q; donegives 15/15 pass with no wait present.
_post_turn_goal_continuationingateway/run.pyand grep it forcreate_task,ensure_future,call_soon: none are present, which is thestatic half of the same claim.
ruff check tests/gateway/test_goal_continuation_drain.py— clean.Checklist
Code
fix(scope):,feat(scope):, etc.)pytest tests/ -qand all tests passDocumentation & Housekeeping
docs/, docstrings) — or N/Acli-config.yaml.exampleif I added/changed config keys — or N/ACONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — or N/AScreenshots / Logs