Skip to content

test(rail): boundary coupling measurement script and multi-format provider-journey contract (rail 3.8.55) - #16205

Merged
diegosouzapw merged 2 commits into
release/v3.8.52from
test/rail-boundary-measure-journey
Oct 10, 2026
Merged

diegosouzapw merged 2 commits into
release/v3.8.52from
test/rail-boundary-measure-journey

Conversation

@diegosouzapw

Copy link
Copy Markdown
Collaborator

Summary

Rail 3.8.55, Task 13: boundary remeasurement and an expanded provider-journey E2E, ahead of the v4 split of open-sse/ (engine) from src/lib/* (modules). No production code changes.

  1. scripts/quality/measure-boundary.mjs (npm run quality:boundary): a reproducible, deterministic measurement of src ⇄ open-sse coupling, replacing the hand-built tables from the characterization tasks.
  2. Multi-format provider journey: two companion contract suites to provider-journey.contract.test.ts (test(e2e): contract test for the full provider journey #8330). One covers the Claude format, one covers the Gemini client format. Both run against a fake upstream on an ephemeral 127.0.0.1 port, with no external network.

What changed

File Change
scripts/quality/measure-boundary.mjs New. It scans src/**/*.ts(x) and open-sse/**/*.ts(x) and skips node_modules, .next, tests/, __tests__/ and *.test/*.spec files. A small lexer strips comments, string contents, template text and regex literals. The script then extracts static imports, re-exports, side-effect imports, import() calls and require() calls. It resolves @/, @omniroute/open-sse[/…] and relative paths, including ../../src/... escapes. Edges are counted as unique (importing file → imported file) pairs, grouped by subsystem of origin (src/lib/<name>, src/app/api/<route>, src/<dir>, open-sse/<dir>). Output is --json or --md. The JSON also includes openSseToSrcEdges, the full engine→app list, which is the v4 remediation input. Only node:fs, node:path and node:url are used; there is no child process and no shell.
tests/unit/measure-boundary.test.ts New. 7 tests on a fixture tree in a temp dir. They cover aliases, a bare @omniroute/open-sse import, a relative escape, import(), require(), type-only imports, commented-out imports, test/__tests__/node_modules/.next exclusion, dedupe, subsystem aggregation, determinism, Markdown output, and the CLI.
package.json, config/quality/gate-manifest.json quality:boundary alias, registered as separately-invoked. It is a report only, not a gate.
tests/integration/_providerJourneyHarness.ts New. Contains the fake upstream, which records every request, and a loopback shim. The shim serves the sync route's self-fetch from the in-process /models route, so the real sync path runs.
tests/integration/provider-journey-claude.contract.test.ts New. It walks this path: anthropic-compatible node → connection → real model sync (upstream /models receives x-api-key) → /v1/models publishes <prefix>/<model> with owned_by = prefix and never the UUID (#8327) → POST /v1/messages, both JSON and SSE. The upstream receives /v1/messages with the raw model id, the connection key, and the system and user turns. The client receives a Claude message or an Anthropic event stream, with no OpenAI shapes.
tests/integration/provider-journey-gemini.contract.test.ts New. It walks this path: openai-compatible node → connection → real sync (Bearer key) → /v1/models owned_by = prefix → /v1beta/models/<prefix>/<model>:generateContent. The upstream receives /v1/chat/completions with the raw id and the system and user messages, and no contents. The client receives candidates[0].content.parts[0].text with finishReason: STOP. /v1beta/models lists the model.

Both suites are collected by the existing tests/integration/*.test.ts glob (test:integration, test:integration:ci).

Measurement at the release tip (cc84e9426a)

Metric Value
src files scanned 3452
open-sse files scanned 1848
src → open-sse edges 1492
open-sse → src edges 745 (372 distinct engine files)

The engine→app direction is far from zero. The source subsystems are open-sse/services (284), handlers (258), executors (63), mcp-server (59), utils (50), config (16) and translator (15). The top targets are src/shared (266), src/lib/db (146), src/lib/usageDb (45), src/lib/usage (37), src/lib/providers (33), src/domain (26) and src/sse (21). The full per-edge list is the v4 remediation list.

Cross-check against the Task 12 hand tables: files importing open-sse match exactly (memory 21, guardrails 7, skills 7, a2a 3, oauth 18). Distinct open-sse consumers also match exactly (memory 5, guardrails 3, skills 12, oauth 6). The only difference is a2a: the script finds 0 open-sse consumers against Task 12's 1, because that consumer is a vitest file, and test files are excluded here by design.

Findings

Validation

Gate Command Result
Unit (new) node --import tsx/esm --test tests/unit/measure-boundary.test.ts 7/7 pass (red before the script existed)
Integration (new + existing journey) node --import tsx/esm --import ./open-sse/utils/setupPolyfill.ts --import ./tests/_setup/isolateDataDir.ts --test --test-force-exit --test-concurrency=1 tests/integration/provider-journey{,-claude,-gemini}.contract.test.ts tests/unit/measure-boundary.test.ts 34 tests: 27 pass, 0 fail, 6 skipped (live block, RUN_CONTRACT_INT), 1 todo
Mutation check Return the raw OpenAI response from the /v1beta generate route STEP 5 of the Gemini suite fails as expected (reverted)
ESLint npx eslint <changed files> 0 problems
Typecheck npm run typecheck:core clean
Dead code npm run check:dead-code OK: 370 (baseline 377)
Test discovery npm run check:test-discovery OK
Gate manifest node scripts/quality/gate-manifest.mjs OK (137 aliases)
Pre-commit lint-staged, docs-sync, any-budget, tracked-artifacts, ai-attribution pass on both commits

⚠️ base-red inherited: #15306

…cript

scripts/quality/measure-boundary.mjs scans src/**/*.ts(x) and open-sse/**/*.ts(x)
(skipping node_modules, .next, tests/, __tests__/ and *.test/*.spec files), lexes out
comments/strings/regex literals, extracts static, re-export, side-effect, dynamic
import() and require() specifiers, resolves @/, @omniroute/open-sse and relative paths,
and counts unique cross-boundary file edges in both directions, grouped by subsystem of
origin. Emits --json or --md, deterministically ordered, plus the full open-sse -> src
edge list as the v4 remediation input. Exposed as `npm run quality:boundary` and
registered in the gate manifest as separately-invoked (report only, not a gate).

Rail 3.8.55, Task 13.
…ites

The #8330 journey covers one OpenAI-compatible provider and stops at /v1/models.
Two companion suites extend it to other wire formats and add the legs that only
break across module boundaries, against a fake upstream on an ephemeral 127.0.0.1
port (no external network):

- provider-journey-claude: anthropic-compatible node -> connection -> REAL model
  sync (sync route -> /models route -> upstream /models) -> /v1/models publishes
  <prefix>/<model> with owned_by = prefix -> /v1/messages (JSON and SSE) reaches the
  upstream as an Anthropic call for the raw model id with the connection key, and the
  client gets a Claude message / Anthropic event stream.
- provider-journey-gemini: openai-compatible node, same sync/catalog legs, then
  /v1beta/models/<prefix>/<model>:generateContent is translated to an OpenAI Chat
  Completions upstream call and back to candidates[].

A TODO step records a gap found on the way: /v1beta/models still names
custom-provider models by the raw provider-node UUID (#8327 residue on the
Gemini catalog surface).

Rail 3.8.55, Task 13.
@github-actions

Copy link
Copy Markdown
Contributor

CI Coverage Report

  • Coverage job: skipped
  • PR test policy: success

Coverage artifact was not available for this run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant