Skip to content

test(sse): run every executor's first model per format on a deep-frozen body (#18692 option 2) - #18851

Open
AndresDiagoM wants to merge 3 commits into
cheaper-inference:release/v3.8.53from
AndresDiagoM:test/executor-frozen-body
Open

AndresDiagoM wants to merge 3 commits into
cheaper-inference:release/v3.8.53from
AndresDiagoM:test/executor-frozen-body

Conversation

@AndresDiagoM

Copy link
Copy Markdown
Contributor

⚠️ base-red inherited: #18791

Summary

This is option 2 from #18692, following the split suggested there. #18819 added the guard, and this PR adds the light survey on top of it. Everything is under tests/, in one new file: tests/unit/executor-frozen-body.test.ts (547 lines). It contains one test, every executor's first model per format runs on a deep-frozen body inside the egress guard.

What fails and what only reports

The test fails only when the test itself has stopped working:

  • The guard is still installed after the run.
  • A corpus body was not actually frozen.
  • For one of the four formats, no executor reached the stubbed fetch. That means the enumeration or the stub broke, and without this check the test would pass without checking anything.

The rest goes to t.diagnostic and never fails the test:

  • Writers: the executor, the format, the error and the first repo stack frame.
  • Unreached cases, grouped by reason: HTTP status of the local error response, which transport the guard blocked, or the thrown message.

When a case runs past 5 s it is aborted. A timed-out run that ignores the abort could otherwise reach the network after the survey, once the guard and the real fetch are back. So the survey waits up to 10 s for such runs to settle. While any is still running, the guard stays installed and fetch stays stubbed for the rest of the test process, and the test reports it as a diagnostic.

What it reports today on release/v3.8.53

379 executor/format cases (+20 ids that refuse chat), 334 reached the stubbed fetch: openai 305, openai-responses 10, claude 16, gemini 3
WRITES: pollinations/openai          'model'               open-sse/executors/pollinations.ts:60
WRITES: opencode-zen/openai          tool 'name'           open-sse/translator/helpers/toolCallHelper.ts:78
WRITES: opencode-zen/openai-responses adds 'summary'       open-sse/executors/opencode.ts:218
WRITES: opencode-go/openai           tool 'name'           open-sse/translator/helpers/toolCallHelper.ts:78
WRITES: opencode-go/openai-responses adds 'summary'        open-sse/executors/opencode.ts:218
WRITES: vertex/claude                adds 'anthropic_version' open-sse/executors/vertex.ts:477
WRITES: nvidia/openai                tool_calls[].'id'     open-sse/executors/default.ts:154

These are the in-place writers already listed in #18692. As you asked there, I'm not filing them as bugs; the test only lists them. Of the 45 cases that don't reach the stub, 7 are those writers, whose frozen write throws first. The other 38 are web and session executors that stop with a local HTTP error, either because the generic stub can't answer their session bootstrap or because they reject the body (for example a forced tool_choice), CLI and http2/wreq-js transports the guard blocks, and one credential that needs a userId. The output lists them by reason.

Verification

  • Five runs (three before the linger change, two after) gave identical diagnostics. The test takes about 4 to 6 s, and about 10 to 16 s wall time including node start.
  • Run together with egress-guard.test.ts and claude-oauth-frozen-body-egress-guard-16631.test.ts: 14/14 pass, with DATA_DIR isolated.
  • I made three temporary changes to check the test isn't empty. Each was reverted, with git diff --quiet clean afterwards:
    • An in-place write added to MoonshotExecutor.transformRequest: the test still passes, and the diagnostic now names moonshot/openai and kimi/openai at moonshot.ts:145.
    • The stub's reach detection broken on purpose: every format reaches 0, and the test fails with no executor sent a openai request to the stubbed fetch.
    • moonshot made to hang and ignore the abort: the test passes and reports still running 10000ms after their abort, egress left blocked: moonshot.
  • ESLint with complexity 15 and sonarjs/cognitive-complexity 15 is clean, and so are prettier and the repo's ESLint run. The file is under the 1440-line cap.
  • I did not run the full npm run test:unit locally: on this Windows machine it takes about 3 hours. CI is the full run.

Left out of the full harness on purpose

  • The other two credential profiles. The Claude OAuth block stays covered by the two *16631* tests.
  • Six of the seven body shapes, the Vertex transport variants and the opencode/muse-spark special ids.
  • The second pass on a plain copy, diffed against a snapshot.
  • Every pin table.

Refs #18692.

…en body

Follow-up to cheaper-inference#18692 (option 2): drive each executor getExecutor() can return, once per
request format it serves (openai, openai-responses, claude, gemini), with a stubbed fetch,
withEgressGuard and a deep-frozen body. Writers into the caller's body and executors the
stubs cannot drive to fetch are reported through t.diagnostic, never pinned; the test only
fails if the guard is left installed, a corpus body is not frozen, or a format reaches fetch
in no executor at all.
@diegosouzapw diegosouzapw added the deferred-v3.8.54 Feat PRs held for v3.8.54. Keep open. Do not merge into v3.8.53. label Nov 4, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deferred-v3.8.54 Feat PRs held for v3.8.54. Keep open. Do not merge into v3.8.53.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants