Skip to content

test: unit tests cannot reach the network, enforced below fetch - #56

Merged
LMPrado-DZ23 merged 8 commits into
release/v3.8.55from
test/block-network-in-unit-tests
Sep 20, 2026
Merged

LMPrado-DZ23 merged 8 commits into
release/v3.8.55from
test/block-network-in-unit-tests

Conversation

@LMPrado-DZ23

@LMPrado-DZ23 LMPrado-DZ23 commented Sep 19, 2026 •

Copy link
Copy Markdown
Owner

Why

A sibling agent writing route contract tests sent real requests to api.anthropic.com/api/claude_cli/bootstrap. Root cause: importing a route loads open-sse/utils/proxyFetch.ts, which replaces globalThis.fetch at import time, so a stub installed before the import is silently discarded. Any guard built on a fetch stub is bypassable exactly the same way — and src/shared/network/guardedFetch.ts is a second, independent bypass (it runs on its own pinned undici Agent). So the rule is now enforced below fetch, by the test runner, at the socket layer.

Was the existing suite calling real providers?

Yes. This is not a hypothetical. A full report-mode run of the unit suite (5 CI shards) recorded 1457 non-loopback connection attempts from 112 test files to 76 distinct hosts, including real provider endpoints with fixture credentials.

The offender list (report-mode run, 5 CI shards)

76 hosts, by attempts

aihorde.net 795; api.anthropic.com 102; chatgpt.com 60; cloudcode-pa.googleapis.com 37;
api64.ipify.org 29; api4.ipify.org 29; api.openai.com 28; api.synthetic.new 24;
platformapi.innerai.com 18; daily-cloudcode-pa.sandbox.googleapis.com 18;
daily-cloudcode-pa.googleapis.com 18; app.blackbox.ai 12; proxy.activation-11446.example.com 10;
opencode.ai 9; inference.generativeai.us-chicago-1.oci.oraclecloud.com 9;
example-aicore.cfapps.eu10.hana.ondemand.com 9; ca-tor.ml.cloud.ibm.com 9; api2.cursor.sh 9;
api.reka.ai 9; www.poe.com 6; venice.ai 6; v0.dev 6; polly.us-east-1.amazonaws.com 6;
my-resource.openai.azure.com 6; models.dev 6; llm.chutes.ai 6; hyperagent.com 6; ghe.company.com 6;
example-resource.services.ai.azure.com 6; chatapi.innerai.com 6; bedrock.us-east-1.amazonaws.com 6;
app.datarobot.com 6; api.poe.com 6; api.jina.ai 6; api.dev.runwayml.com 6; api.clarifai.com 6;
api.assemblyai.com 6; alice--demo.modal.run 6; adobeid-na1.services.adobe.com 6; claude.ai 5;
dashscope-intl.aliyuncs.com 4; ydc-index.io 3; www.meta.ai 3; www.googleapis.com 3; www.dola.com 3;
radar.omniroute.online 3; q.eu-central-1.amazonaws.com 3; openrouter.ai 3; oauth2.googleapis.com 3;
jules.googleapis.com 3; inference-api.nousresearch.com 3; huggingface.co 3; gitlab.com 3; duck.ai
3; copilot.microsoft.com 3; conol.ai 3; codewhisperer.us-east-1.amazonaws.com 3;
clerk.agent.adapta.one 3; chat.deepseek.com 3; business.gemini.google 3; app.notion.com 3; api.z.ai
3; api.voyageai.com 3; api.nlpcloud.io 3; api.devin.ai 3; api.commandcode.ai 3; aistudio.tencent.ai
3; www.perplexity.ai 2; pinned.example.test 2; grok.com 2; bifrost.test.local 2; api.github.com 2;
x.invalid 1; rerank-egress.local 1; pin-dns-nonexistent-host.invalid 1; p.example.com 1; arena.ai
1; 192.0.2.1 1;

112 test files, by attempts
tests/unit/models-catalog-route.test.ts (183)
tests/unit/providers-route-managed-catalog.test.ts (164)
tests/unit/models-catalog-low-noise-flag.test.ts (117)
tests/unit/12058-models-catalog-canonical-self-aliased.test.ts (90)
tests/unit/vscode-token-routes.test.ts (84)
tests/unit/executor-web-cookie-sweep.test.ts (61)
tests/unit/models-catalog-combo-metadata.test.ts (54)
tests/unit/model-token-limit-catalog.test.ts (30)
tests/unit/v1-models-discovery-conformance.test.ts (27)
tests/unit/provider-scoped-models-route.test.ts (27)
tests/unit/c05-v1-models-local-cli-availability.test.ts (27)
tests/unit/hidden-models-leak-v1-models-11300.test.ts (21)
tests/unit/models-catalog-hidden-combo-leaves.test.ts (18)
tests/unit/model-catalog-runtime-invalidation.test.ts (18)
tests/unit/8327-models-owned-by-prefix.test.ts (18)
tests/unit/chat-messages-validation-6402.test.ts (17)
tests/unit/specialty-model-catalog-routes.test.ts (15)
tests/unit/executor-inner-ai.test.ts (15)
tests/unit/deepseek-thinking-efforts.test.ts (15)
tests/unit/codex-models-catalog-refresh.test.ts (15)
tests/unit/catalog-hide-auto-no-think.test.ts (15)
tests/unit/v1-models-concurrent-6408.test.ts (12)
tests/unit/provider-models-v1-route.test.ts (12)
tests/unit/models-catalog-static-synced-suppression.test.ts (12)
tests/unit/models-catalog-hide-paid.test.ts (12)
tests/unit/models-catalog-auto-combos-4164.test.ts (12)
tests/unit/catalog-auto-routing-disabled-10831.test.ts (12)
tests/unit/api-models-v1-models-mismatch-10615.test.ts (12)
tests/unit/vision-bridge-policy-reroute-6640.test.ts (11)
tests/unit/verified-connection-activation-11446.test.ts (10)
tests/unit/vscode-token-routes-gpt56.test.ts (9)
tests/unit/sync-reasoning-supported-efforts-7694.test.ts (9)
tests/unit/noauth-imported-models-3200.test.ts (9)
tests/unit/models-catalog-model-exposure-list.test.ts (9)
tests/unit/models-catalog-envkey-6406.test.ts (9)
tests/unit/models-catalog-custom-node-prefix.test.ts (9)
tests/unit/models-catalog-block-auto-5192.test.ts (9)
tests/unit/catalog-order-contract.test.ts (9)
tests/unit/8958-alias-backed-node-prefix.test.ts (9)
tests/unit/11947-auto-combo-modalities.test.ts (9)
tests/unit/10313-catalog-cache-key-hashing.test.ts (9)
tests/unit/vscode-responses-models.test.ts (6)
tests/unit/specialty-model-hidden-openrouter-9293.test.ts (6)
tests/unit/route-edge-coverage.test.ts (6)
tests/unit/quota-exclusive-catalog-short-circuit.test.ts (6)
tests/unit/opencode-noauth-models-route.test.ts (6)
tests/unit/models-catalog-functional-gateway-permissions.test.ts (6)
tests/unit/model-alias-route.test.ts (6)
tests/unit/image-model-not-in-chat-catalog-6457.test.ts (6)
tests/unit/image-generation-route.test.ts (6)
tests/unit/ghe-copilot.test.ts (6)
tests/unit/firefly-cookie-validation-10522.test.ts (6)
tests/unit/effort-tiers-loop-catalog-e2e.test.ts (6)
tests/unit/combo-context-generic-default-10734.test.ts (6)
tests/unit/11759-embedding-registry-width-and-type.test.ts (6)
tests/unit/modelsDevSync.test.ts (4)
tests/unit/dashscope-text-models-discovery.test.ts (4)
tests/unit/claude-web.test.ts (4)
tests/unit/admin-audit-events.test.ts (4)
tests/unit/vscode-token-routes-responses-listing.test.ts (3)
tests/unit/v1-models-catalog-ttl.test.ts (3)
tests/unit/v1-models-auth-leak-9320.test.ts (3)
tests/unit/serial/9147-catalog-eventloop-yield.test.ts (3)
tests/unit/repro-6142-devin-cloud-agent-unwired.test.ts (3)
tests/unit/radar-api-routes.test.ts (3)
tests/unit/openrouter-vision-sync-4264.test.ts (3)
tests/unit/model-lifecycle-integration.test.ts (3)
tests/unit/executor-venice-web.test.ts (3)
tests/unit/executor-v0-vercel-web.test.ts (3)
tests/unit/executor-poe-web.test.ts (3)
tests/unit/executor-command-code.test.ts (3)
tests/unit/duckduckgo-web-executor.test.ts (3)
tests/unit/cc-compatible-model-catalog.test.ts (3)
tests/unit/auto-combos-suffixes-4235.test.ts (3)
tests/unit/apikey-connection-health-check.test.ts (3)
tests/unit/antigravity-missing-project-chat.test.ts (3)
tests/unit/9034-alias-backed-prefix-id-repro.test.ts (3)
tests/unit/8326-compatible-id-regex.test.ts (3)
tests/unit/services/end-to-end-shape.test.ts (2)
tests/unit/remote-image-fetch-pin-dns-connection.test.ts (2)
tests/unit/qwen-web-runtime-block.test.ts (2)
tests/unit/probe-testall-isolation.test.ts (2)
tests/unit/probe-gate-autodisable.test.ts (2)
tests/unit/pinned-lookup-hostname.test.ts (2)
tests/unit/modelsDevSync-extended.test.ts (2)
tests/unit/model-test-runner.test.ts (2)
tests/unit/model-test-route.test.ts (2)
tests/unit/issue-agent-route-execution.test.ts (2)
tests/unit/felo-web-runtime-block.test.ts (2)
tests/unit/exclusive-lease-auxiliary-isolation.test.ts (2)
tests/unit/combo-success-selected-connection-header-11810.test.ts (2)
tests/unit/combo-same-provider-cascade.test.ts (2)
tests/unit/combo-provider-cooldown.test.ts (2)
tests/unit/circuit-breaker-resolved-5xx-12254.test.ts (2)
tests/unit/chatgpt-web-runtime-block.test.ts (2)
tests/unit/chat-safetynet-reqid-6097.test.ts (2)
tests/unit/chat-route-edge-cases.test.ts (2)
tests/unit/chat-route-coverage.test.ts (2)
tests/unit/chat-rate-limit-body-lock.test.ts (2)
tests/unit/chat-non-string-model-6407.test.ts (2)
tests/unit/chat-managed-lease-routing.test.ts (2)
tests/unit/chat-helpers.test.ts (2)
tests/unit/chat-cooldown-aware-retry.test.ts (2)
tests/unit/chat-context-relay.test.ts (2)
tests/unit/chat-completions-route-shape-gate.test.ts (2)
tests/unit/chat-combo-live-test.test.ts (2)
tests/unit/chat-adaptive-admission-binding.test.ts (2)
tests/unit/api/v1/relay-completions-errors.test.ts (2)
tests/unit/antigravity-byop-account-rotation.test.ts (2)
tests/unit/rerank-proxy-pinning-7350.test.ts (1)
tests/unit/provider-limits-apikey-proxy-context.test.ts (1)
tests/unit/docker-healthcheck-3151.test.ts (1)

The sharpest finding: a CI test whose fixture came from a third party

tests/unit/models-catalog-low-noise-flag.test.ts guards its prefix-mode assertions with rows.length > 100 ("expected a populated catalog"). That guard was being met with help from the network: building /v1/models polls the LIVE AI Horde image catalog, and this one file sent 54 requests to aihorde.net per run (plus 33 to chatgpt.com and 30 to api.anthropic.com). With outbound traffic refused the canonical-mode catalog drops from >100 rows to exactly 71 — 40 auto/* combos, 27 codex/* roots, 4 veo rows — and every missing row was an image model CI had just downloaded from a third party. The thresholds are unchanged; the file now serves that half of the catalog from a fixture in the exact shape /v2/status/models?type=image returns.

What the guard does

tests/_setup/blockNetwork.ts, loaded with --import next to tests/_setup/isolateDataDir.ts everywhere that setup is wired (package.json test scripts, quality.yml, Stryker tap.nodeArgs, scripts/quality/test-scoped.sh, scripts/release/merge-train.sh, tests/_run_dns_guard_test.sh). tests/unit/block-network-wiring.test.ts fails if any of those loads isolateDataDir without it.

  • Socket layer: hooks net.Socket.prototype.connect, which every TCP client in Node reaches — the built-in fetch, undici with any Agent/dispatcher (including a per-request one), http/https/http2, net.connect, tls.connect (a TLSSocket is a net.Socket). Verified per transport in tests/unit/block-network-guard.test.ts.
  • wreq-js native binding: the browser-impersonating transport opens sockets in Rust, outside Node's net module, so its binding (request, websocketConnect*) is patched lazily via Module.prototype.require the first time anything loads it.
  • Pinned resolvers: when the caller pins its own resolver (new Agent({ connect: { lookup } }), as guardedFetch and the webhook dispatcher do), the hostname says nothing about where the socket goes — the guard defers to the address that lookup returns. That both removes a false positive (a hostname pinned to 127.0.0.1) and closes the real hole.
  • Allowed: loopback (127.0.0.0/8, ::1, IPv4-mapped loopback, localhost) and Unix sockets / named pipes, so tests that start a local server are unaffected.
  • Live tests: the guard stands aside entirely when an existing live flag is 1 — RUN_LIVE_TESTS, RUN_COMBO_LIVE, RUN_BOUNDARY_LIVE, RUN_LIVE_WIRE_CAPTURE, RUN_CLI_SMOKE, RUN_CONTRACT_INT, RUN_SERVICES_INT, RUN_LLMLINGUA_INT, RUN_QUOTA_REDIS_INT. No new flag was invented.
  • Loud failure: the attempt is refused, [network-guard] BLOCKED host=… port=… via=… file=… plus a stack goes to stderr, and the process exit code becomes 1 even if the test caught and swallowed the error — node:test then reports the file as failed. A test cannot hide it.

Limits, stated up front

  • Child processes are not covered. The guard lives in the test process; a spawned CLI, npm, or any other child opens its own sockets and --import does not reach it. Tests that shell out to a provider CLI are outside this guard by construction.
  • DNS resolution itself is not blocked — only connections. A pinned-resolver path resolves before the guard decides.
  • Vitest suites (tests/_setup/vitestUiPolyfills.ts) do not load isolateDataDir and are out of scope here.

Rollout

  1. Report mode (first commit) — refuse the connection but do not fail the run, so the suite could be inventoried without breaking CI blind. Report mode still refuses: a report run must not send traffic to a provider either.
  2. Fix every offender (second commit) — no test was deleted or skipped to make it pass.
  3. Enforce by default — OMNIROUTE_TEST_NETWORK_GUARD=report remains available as an explicit opt-in.

How the offenders were fixed

tests/unit/_helpers/offlineOutbound.ts gives a file a hermetic outbound layer: the stub is installed after its imports, asserted to be the live globalThis.fetch, loopback passes through, and any other URL is either answered by the test (respond) or thrown with the URL named. The egress-IP probe (undici request, never globalThis.fetch) is neutralised through its existing _setEgressProbeForTests seam; the wreq-js TLS client through setTlsClientForTest when a suite opts in.

102 files take that helper as-is. The rest needed real fixes:

File Was Now
modelsDevSync.test.ts four assertions about models.dev's content (100+ providers, 4000+ models) downloaded its whole catalog every run gated behind the existing RUN_LIVE_TESTS; fetchModelsDev()'s fetch/cache contract covered offline
modelsDevSync-extended.test.ts the interval test called startPeriodicSync() without a stub → real sync stubbed like its siblings
executor-web-cookie-sweep.test.ts header said "no real upstream call is needed" while dialling 20 providers (61 attempts) both transports answer the synthetic 401 the sweep always described
verified-connection-activation-11446.test.ts fetch stub installed before the route imports — the incident pattern stub moved after the imports and asserted live
apikey-connection-health-check.test.ts depended on Google rejecting a fixture refresh token answers with the exact invalid_grant response
docker-healthcheck-3151, rerank-proxy-pinning-7350, provider-limits-apikey-proxy-context, api/v1/relay-completions-errors "unreachable host" was 192.0.2.1 or a made-up public name (a real outbound attempt) tests/unit/_helpers/deadLoopback.ts: a loopback port with nothing listening — same ECONNREFUSED, no traffic
chat-helpers.test.ts egress probe probe seam only; globalThis.fetch left alone because the file asserts proxyFetch's own proxy/TLS routing
claude-web.test.ts reset its wreq transport to null — i.e. the REAL one — between tests, so every test without its own override dialled claude.ai resets to an offline 401 through __setTlsFetchOverrideForTesting
providers-route-managed-catalog.test.ts never stubbed the per-provider wreq clients: grok.com and www.perplexity.ai for real (164 attempts) both clients overridden in the same way
models-catalog-low-noise-flag.test.ts its > 100 rows non-vacuity guard was topped up by the live AI Horde catalog deterministic image-catalog fixture (see the finding above); thresholds untouched

The second round: transports a fetch stub cannot see

The first enforce run took the suite from 1457 attempts to 30, in 7 files. Every one of those escaped through a transport below globalThis.fetch, and the helper now closes each:

  • proxyFetch's direct path calls undici's fetch with a dispatcher of its own from open-sse/utils/proxyDispatcherCache (symbol-keyed globals), so a caller holding the proxyFetch export opened a socket before any stub was consulted. Those globals — and undici's global dispatcher — are seeded with a MockAgent that refuses everything except loopback, so the request fails before connect and proxyFetch's fallback lands on the stub.
  • A route imported inside a test pulls proxyFetch, whose module body assigns globalThis.fetch and silently dropped the stub. The helper claims proxyFetch's own isPatched flag, which is exactly the guard that assignment respects. (An accessor that kept the stub permanently in front was tried and reverted: a test whose stub wraps the previous fetch then recursed forever — antigravity-missing-project-chat hung and died on "Map maximum size exceeded". A plain assignment keeps a test's own stub authoritative.)
  • Per-provider wreq clients (open-sse/services/*TlsClient.ts) are native bindings; the two suites that used them now install __setTlsFetchOverrideForTesting.

Which guard we keep (PR #54)

This one, process-wide. PR #54's tests/unit/cli/_helpers/routeBackedFetch.ts is a route-backed CLI harness whose network blocking is a stricter LOCAL policy — it refuses loopback too, because it dispatches loopback in-process to the real route handlers. That is legitimate inside its own scope and is not a second global guard: nothing in it needs to be deleted for this PR, and its route-backed-fetch-guard.test.ts keeps pinning the same escape paths from the CLI side (it independently found the proxyFetch-dispatcher bypass this PR closes suite-wide). What should not exist twice is a process-wide guard: that one lives here, in tests/_setup/blockNetwork.ts, wired into every test entry point.

Relationship to PR #52 and PR #48

  • PR feat(workspaces): workspace and project hierarchy with rolled-up budgets #52 adds tests/unit/_helpers/blockOutboundFetch.ts, a per-suite fetch-level stub with a "zero attempts" assertion. It is not made redundant by this PR and nothing here touches it: it gives a precise per-suite assertion, this guard makes the rule unbypassable. They compose — the helper throws before a socket is ever attempted, so the guard sees nothing and its assertion still holds. Follow-up (not in this PR): the two could share one attempt-recording implementation.
  • PR test(api): contract tests for the routes the governance baseline listed as untested #48 left 53 routes untested partly because they need live externals. The guard makes "this route tried to reach the network" an observable, reliable failure instead of a silent success, which is what those tests need to become writable.

Verification

Typechecks (tsc directly; the check-*-typecheck.mjs wrappers fail on Windows with spawnSync npx.cmd EINVAL):

=== tsconfig.typecheck-core.json            (no output, exit 0)
=== tsconfig.typecheck-noimplicit-core.json (no output, exit 0)
=== tsconfig.typecheck-api.json             (no output, exit 0)
=== tsconfig.typecheck-dashboard.json       (no output, exit 0)

ESLint — npx eslint --max-warnings=0 --suppressions-location config/quality/eslint-suppressions.json --pass-on-unpruned-suppressions --no-warn-ignored <changed .ts files>: clean, exit 0. CI's "No new ESLint warnings" job: pass.

Prettier — this Windows checkout is CRLF (core.autocrlf=true), which prettier --check flags on every file, so the comparison was run on LF-normalised copies of the changed files against the same copies from the merge base: identical sets — the same 8 files are non-compliant before and after (11947-auto-combo-modalities, chat-adaptive-admission-binding, chat-non-string-model-6407, combo-provider-cooldown, combo-same-provider-cascade, duckduckgo-web-executor, executor-command-code, vision-bridge-policy-reroute-6640), all pre-existing. Nothing this PR writes is new prettier debt. (There is no prettier job in CI; lint-staged owns it locally.)

Mutation-coverage drift — findCoverageDrift called directly (the CLI exits 0 without checking on Windows because of its file:// main-module guard):

scanned 5063 unit tests; drift modules: 0 drifting tests: 0

The guard's own tests — tests/unit/block-network-guard.test.ts + tests/unit/block-network-wiring.test.ts, 17/17:

✔ this suite itself runs under the guard
✔ isLoopbackHost accepts loopback only
✔ targetFromConnectArgs mirrors net's argument forms
✔ resolveGuardMode: explicit enforce/report, default, off only for live flags
✔ the blocked error names the host, the port and the test process
✔ non-loopback attempts fail with the specific error, and a swallowed one still fails the process
✔ loopback and local sockets keep working
✔ a live-test flag makes the guard stand aside
✔ a pinned resolver is judged by the address it returns, not the hostname
✔ report mode still refuses the connection but leaves the exit code alone
✔ an unknown guard mode is rejected at startup
✔ the guard survives proxyFetch replacing globalThis.fetch at route import
✔ the guardedFetch bypass (its own pinned undici Agent) is blocked too
✔ the wreq-js native transport (outside net.Socket) is blocked too
✔ every npm script that isolates DATA_DIR also loads the network guard
✔ every workflow and shell entry point that isolates DATA_DIR also loads the guard
✔ Stryker's node args load the guard right after isolateDataDir
ℹ tests 17 / pass 17 / fail 0

The suite, before and after (CI, 5 shards):

attempts offender files hosts shards red
report mode, first run 1457 112 76 3/5 (tests that genuinely needed the network)
enforce, after the fixes 0 0 0 see checks on this PR

Known red, untouched: check-file-size on src/lib/db/core.ts (1770 > frozen 1745) is red on the base tree. It does not surface in this PR's runs because the gate is base-relative on PR events.

Inherited red, not this PR: tests/unit/podman-machine-guidance-8497.test.ts ("diegosouzapw#8497 Podman guide separates local engines from Podman Machine") fails on the current base. PR #63 (docs: point every install command at a channel that is published) rewrote the guide from ghcr.io/lmprado-dz23/omniroute:next to :latest; the test still asserts :next. Neither the test nor the doc is touched by this PR — on the tip before that merge the whole file passes here (7/7), and after merging it the same single case fails. It needs a one-line fix in the test or the doc, owned by #63.

🤖 Generated with Claude Code

tests/_setup/blockNetwork.ts refuses every non-loopback connection at the
socket layer — net.Socket.prototype.connect, which every TCP client in Node
(built-in fetch, undici with any dispatcher, http/https/http2, net, tls) ends
up calling — plus the wreq-js native binding, whose Rust client opens sockets
outside Node's net module. A guard that patched globalThis.fetch would be
bypassable exactly the way the incident was: importing a route loads
open-sse/utils/proxyFetch.ts, which replaces globalThis.fetch at import time
with its own wrapper around the real fetch, silently discarding a stub that a
test installed before the import.

Loopback (127.0.0.0/8, ::1, IPv4-mapped loopback, localhost) and Unix
sockets / named pipes stay open so tests that start a local server keep
working. The guard stands aside when an existing live-test flag is set
(RUN_LIVE_TESTS, RUN_COMBO_LIVE, RUN_BOUNDARY_LIVE, RUN_LIVE_WIRE_CAPTURE,
RUN_CLI_SMOKE, RUN_CONTRACT_INT, RUN_SERVICES_INT, RUN_LLMLINGUA_INT,
RUN_QUOTA_REDIS_INT).

Wired with --import next to tests/_setup/isolateDataDir.ts in every place
that setup is wired: the package.json test scripts, quality.yml, Stryker's
tap.nodeArgs, scripts/quality/test-scoped.sh, scripts/release/merge-train.sh
and tests/_run_dns_guard_test.sh. tests/unit/block-network-wiring.test.ts
fails if any of those loads isolateDataDir without the guard.

DEFAULT_GUARD_MODE is "report" for this first step of the rollout: a blocked
attempt is refused and reported on stderr, but does not force the exit code,
so the existing suite's offenders can be inventoried before the rule becomes
blocking. Refusing (rather than logging and letting the request out) is
deliberate: report mode must not send traffic to a provider either.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 19, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 8f9ce630-a297-4314-bb16-38859c2d5a3a


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

zodyprado-web and others added 6 commits September 19, 2026 22:36
…nforce

Report mode over the full suite (5 CI shards) found 1457 non-loopback
connection attempts from 112 unit test files, to 76 distinct hosts — among
them aihorde.net (795), api.anthropic.com (102), chatgpt.com (60),
cloudcode-pa.googleapis.com (37), api.openai.com (28), models.dev, claude.ai,
api2.cursor.sh and api4/api64.ipify.org. Almost all of them were incidental:
production code under test makes a best-effort call (the AI Horde image-catalog
poll behind /v1/models, the egress-IP probe warmed by the chat route, a
provider's live model discovery) and swallows the failure, so the tests passed
either way while real requests left the machine.

tests/unit/_helpers/offlineOutbound.ts gives a test file a hermetic outbound
layer: a stub installed AFTER its imports (asserted to be the live
globalThis.fetch), loopback still open, every other URL either answered by the
test or thrown with the URL named. Transports that never touch globalThis.fetch
are neutralised through their existing seams — the undici-based egress probe
(_setEgressProbeForTests) and, opt-in, the wreq-js TLS client. 102 files take it
as-is; the rest needed real fixes:

- modelsDevSync: the four "live API" assertions describe models.dev's CONTENT,
  so they are now gated behind the existing RUN_LIVE_TESTS flag, and the
  fetch/cache contract of fetchModelsDev() is covered offline instead.
- executor-web-cookie-sweep: the header claimed "no real upstream call is
  needed" while dialling 20 providers for real; both transports now answer with
  the synthetic 401 the sweep always described (61 attempts -> 0).
- verified-connection-activation-11446: its fetch stub was installed BEFORE the
  route imports — the incident pattern — so proxyFetch discarded it. Moved after
  the imports and asserted live.
- apikey-connection-health-check: the dual-auth case depended on Google
  rejecting a fixture refresh token; it now answers with the exact invalid_grant
  response, so "expired" is a property of our classification.
- docker-healthcheck / rerank-proxy-pinning / provider-limits / relay-completions
  (+ tests/unit/_helpers/deadLoopback.ts): "unreachable host" was a made-up
  public name or 192.0.2.1, i.e. a real outbound attempt; it is now a loopback
  port with nothing listening — same ECONNREFUSED, no traffic.
- modelsDevSync-extended: startPeriodicSync() launches an immediate sync; the
  interval test did not stub fetch, so every run hit models.dev.

Guard changes: a connect() whose caller pinned its own resolver (undici
`connect: { lookup }`) is now judged by the address that lookup returns, not by
the hostname — src/shared/network/guardedFetch.ts is the second known bypass of
a fetch stub (after proxyFetch) and pins a validated IP, so hostname-based
blocking both false-positived on loopback pins and hid where the socket really
went. Its own regression test sits next to the proxyFetch one.

DEFAULT_GUARD_MODE is now "enforce": a non-loopback attempt fails the test
process even if the test swallows the error. Report mode stays available via
OMNIROUTE_TEST_NETWORK_GUARD=report.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first enforce run over CI's 5 shards took the suite from 1457 attempts in
112 files to 30 in 7. Every one of those seven escaped through a transport that
a globalThis.fetch stub cannot see:

- proxyFetch's direct path calls undici's fetch with a dispatcher of its own
  (open-sse/utils/proxyDispatcherCache's symbol-keyed globals), so a caller
  holding the proxyFetch export opened a socket before any stub was consulted.
  installOfflineOutbound now seeds those globals — and undici's global
  dispatcher — with a MockAgent that refuses everything except loopback, so the
  request fails before connect and the fallback lands on the stub.
- A route imported INSIDE a test pulls proxyFetch, whose module body assigns
  globalThis.fetch and silently dropped the stub. The helper now claims
  proxyFetch's own `isPatched` flag, which is exactly the guard that assignment
  respects. (An accessor that kept the stub in front was tried first and
  reverted: a test whose stub wraps the previous fetch then recursed forever —
  antigravity-missing-project-chat hung and died on "Map maximum size
  exceeded". A plain assignment keeps a test's own stub authoritative.)
- claude-web reset its wreq transport to `null` (i.e. the REAL one) between
  tests, and providers-route-managed-catalog never stubbed the per-provider
  wreq clients at all: grok.com, www.perplexity.ai and claude.ai were dialled
  for real. Both now reset to an offline 401 through
  __setTlsFetchOverrideForTesting.

check:file-size was red on three frozen test files: the 7-line install block
pushed them past a frozen size with zero headroom. The block is now 3 lines
everywhere, and those three entries move by exactly that (+3) with the reason
recorded in the baseline — the block cannot shrink further and cannot move to a
shared setup, because it must run after each file's own imports.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…catalog

FINDING, not a guard bug: the `rows.length > 100` non-vacuity guards in
models-catalog-low-noise-flag.test.ts were satisfied with help from the
network. Building /v1/models polls the LIVE AI Horde image catalog
(open-sse/services/aihordeImageCatalog.ts, called from catalog.ts when the
provider is active), and this file alone sent 54 requests to aihorde.net per
run — plus 33 to chatgpt.com and 30 to api.anthropic.com. With outbound
traffic refused the canonical-mode catalog drops from >100 rows to exactly 71
(40 auto/* combos, 27 codex/* roots, 4 veo rows): every missing row was an AI
Horde image model that CI had just downloaded from a third party.

The fixture restores that half of the catalog deterministically, in the shape
`/v2/status/models?type=image` really returns, so the guards assert the same
thing they always meant to assert — a populated catalog — without a provider
being contacted. The thresholds are untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@LMPrado-DZ23
LMPrado-DZ23 merged commit c051ad7 into release/v3.8.55 Sep 20, 2026
15 checks passed
LMPrado-DZ23 pushed a commit that referenced this pull request Sep 20, 2026
The first full-CI verdict this release line has ever produced came back
red. Unit ×4 and vitest green; both integration shards failed. Both were
worth having.

REAL — tests/integration/api-routes-critical.test.ts was making a live
HTTPS request to aihorde.net, three attempts counting the retry, on
every run. `GET /api/v1/models` refreshes the AI Horde image catalog
whenever `aihorde` is active, and it is active by default: a no-auth
provider has no connection row to switch off. aiHordeImageCatalog
exposes setFetch for exactly this case; the test now injects a stub. The
route's own catch keeps the last good snapshot, so an empty worker list
changes none of the assertions.

FALSE POSITIVE, and mine — tests/integration/api-keys.test.ts sets
CLOUD_URL to http://cloud.example on purpose, so the cloud-sync branch
is taken and fails. `cloud.example` is reserved by RFC 2606 / RFC 6761:
there is no delegation for it anywhere, so it cannot reach a host. The
guard I added in #56 counted it as "the suite reached the network" and
failed a file that never left the machine.

The first fix I wrote for that was wrong, and three existing guard tests
caught it: I exempted reserved names from being BLOCKED, which let the
connection through to a real DNS lookup — more network activity, not
less. Blocking and counting are two decisions. A reserved name is now
still refused, and only the counting changes. The log line says which
case it was, so a future reader does not mistake one for the other.

This is not a hole: the exemption is not "hosts a test asked for", it is
"names that by standard resolve to nothing", and a provider smuggled in
under `.test` would be just as unreachable. A test asserts the exemption
does not reach aihorde.net, api.openai.com, example.com or a literal IP.

Also: the shard jobs now upload _artifacts/release-green/. Without the
per-gate logs a red shard reports only its first failure line and the
assertion dies with the runner — which is why both of these had to be
reproduced locally before they could be read at all.

  25/25 api-keys + api-routes-critical, 0 guard violations
  20/20 block-network-guard + block-network-wiring
  before: 15 attempts to cloud.example:80, 3 to aihorde.net:443
  after:  0 counted, 0 to aihorde.net
  YAML parses; prettier clean

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
LMPrado-DZ23 added a commit that referenced this pull request Sep 20, 2026
…ard (#96)

The first full-CI verdict this release line has ever produced came back
red. Unit ×4 and vitest green; both integration shards failed. Both were
worth having.

REAL — tests/integration/api-routes-critical.test.ts was making a live
HTTPS request to aihorde.net, three attempts counting the retry, on
every run. `GET /api/v1/models` refreshes the AI Horde image catalog
whenever `aihorde` is active, and it is active by default: a no-auth
provider has no connection row to switch off. aiHordeImageCatalog
exposes setFetch for exactly this case; the test now injects a stub. The
route's own catch keeps the last good snapshot, so an empty worker list
changes none of the assertions.

FALSE POSITIVE, and mine — tests/integration/api-keys.test.ts sets
CLOUD_URL to http://cloud.example on purpose, so the cloud-sync branch
is taken and fails. `cloud.example` is reserved by RFC 2606 / RFC 6761:
there is no delegation for it anywhere, so it cannot reach a host. The
guard I added in #56 counted it as "the suite reached the network" and
failed a file that never left the machine.

The first fix I wrote for that was wrong, and three existing guard tests
caught it: I exempted reserved names from being BLOCKED, which let the
connection through to a real DNS lookup — more network activity, not
less. Blocking and counting are two decisions. A reserved name is now
still refused, and only the counting changes. The log line says which
case it was, so a future reader does not mistake one for the other.

This is not a hole: the exemption is not "hosts a test asked for", it is
"names that by standard resolve to nothing", and a provider smuggled in
under `.test` would be just as unreachable. A test asserts the exemption
does not reach aihorde.net, api.openai.com, example.com or a literal IP.

Also: the shard jobs now upload _artifacts/release-green/. Without the
per-gate logs a red shard reports only its first failure line and the
assertion dies with the runner — which is why both of these had to be
reproduced locally before they could be read at all.

  25/25 api-keys + api-routes-critical, 0 guard violations
  20/20 block-network-guard + block-network-wiring
  before: 15 attempts to cloud.example:80, 3 to aihorde.net:443
  after:  0 counted, 0 to aihorde.net
  YAML parses; prettier clean

Co-authored-by: zodyp <zodyprado@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
LMPrado-DZ23 added a commit that referenced this pull request Sep 20, 2026
…l reds (#100)

A-H1 said the release-green sweep could not produce a verdict, and I
wrote that the two ways out were "neither reachable by editing a
workflow". One of them was. #87 split the sweep — resolve → seven
slow-suite jobs → an aggregator that merges their reports — and this
line now has the full-CI verdict it never had.

What that verdict found is the point of having had it:

  · api-routes-critical.test.ts was making a LIVE HTTPS request to
    aihorde.net on every run
  · api-keys.test.ts was a false positive of the network guard I wrote
    in #56 — cloud.example is RFC-reserved and resolves nowhere

Both fixed in #96; the second sweep passed all seven shards.

Also recorded, because it is the honest remainder: the aggregator passes
every static and drift gate and then dies at check:pack-artifact, six
minutes of silence and exit 143, in both sweeps. That gate falls back to
a full `next build` and the hosted runner cannot fit this tree —
build.yml has been manual-only since diegosouzapw#11946 for exactly that reason.
#99 stops it discarding thirteen green gates and seven green suites on
the way out, by recording the gate as unmeasured instead.

A fully green verdict needs USE_VPS_RUNNER with that runner online.
That is an external dependency and the owner's call, not pending work,
and the document now says so rather than leaving a HIGH that reads like
something I still owe.

  [doc-links] PASS — 172 docs, 1044 internal links

Co-authored-by: zodyp <zodyprado@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants