Repository navigation
test(sse): run every executor's first model per format on a deep-frozen body (#18692 option 2) - #18851
Open
AndresDiagoM wants to merge 3 commits into
Open
test(sse): run every executor's first model per format on a deep-frozen body (#18692 option 2)#18851AndresDiagoM wants to merge 3 commits into
AndresDiagoM wants to merge 3 commits into
Conversation
…en body Follow-up to cheaper-inference#18692 (option 2): drive each executor getExecutor() can return, once per request format it serves (openai, openai-responses, claude, gemini), with a stubbed fetch, withEgressGuard and a deep-frozen body. Writers into the caller's body and executors the stubs cannot drive to fetch are reported through t.diagnostic, never pinned; the test only fails if the guard is left installed, a corpus body is not frozen, or a format reaches fetch in no executor at all.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is option 2 from #18692, following the split suggested there. #18819 added the guard, and this PR adds the light survey on top of it. Everything is under
tests/, in one new file:tests/unit/executor-frozen-body.test.ts(547 lines). It contains one test,every executor's first model per format runs on a deep-frozen body inside the egress guard.execute()never writes its input (would you want it?) #18692 did: registry aliases,PROVIDERS, the dashboard catalog and three dynamic*-compatible-*nodes, deduplicated. Ids that refuse chat are skipped and counted.globalThis.fetchstubbed. The run happens inside thetests/_setup/egressGuard.tsguard from test(setup): opt-in egress guard for in-process tests, plus the #16631 Claude OAuth frozen-body case #18819, and executors are also constructed inside it.What fails and what only reports
The test fails only when the test itself has stopped working:
The rest goes to
t.diagnosticand never fails the test:When a case runs past 5 s it is aborted. A timed-out run that ignores the abort could otherwise reach the network after the survey, once the guard and the real fetch are back. So the survey waits up to 10 s for such runs to settle. While any is still running, the guard stays installed and
fetchstays stubbed for the rest of the test process, and the test reports it as a diagnostic.What it reports today on
release/v3.8.53These are the in-place writers already listed in #18692. As you asked there, I'm not filing them as bugs; the test only lists them. Of the 45 cases that don't reach the stub, 7 are those writers, whose frozen write throws first. The other 38 are web and session executors that stop with a local HTTP error, either because the generic stub can't answer their session bootstrap or because they reject the body (for example a forced
tool_choice), CLI and http2/wreq-js transports the guard blocks, and one credential that needs a userId. The output lists them by reason.Verification
egress-guard.test.tsandclaude-oauth-frozen-body-egress-guard-16631.test.ts: 14/14 pass, withDATA_DIRisolated.git diff --quietclean afterwards:MoonshotExecutor.transformRequest: the test still passes, and the diagnostic now namesmoonshot/openaiandkimi/openaiatmoonshot.ts:145.no executor sent a openai request to the stubbed fetch.moonshotmade to hang and ignore the abort: the test passes and reportsstill running 10000ms after their abort, egress left blocked: moonshot.complexity15 andsonarjs/cognitive-complexity15 is clean, and so are prettier and the repo's ESLint run. The file is under the 1440-line cap.npm run test:unitlocally: on this Windows machine it takes about 3 hours. CI is the full run.Left out of the full harness on purpose
*16631*tests.Refs #18692.