feat(router-eval): add retained optimization gate

[KooshaPari]

Summary

• add router-eval aggregation, Pareto frontier, AIQ scoring, and retained report artifacts
• add Bun CLI modes for JSONL, SQLite replay, baseline comparison, search, patch compare, trends, and retained regression checks
• add focused fixtures and unit coverage for the router-eval gate surface

Validation

• npm exec --yes tsx -- --test --test-concurrency=1 tests/unit/router-eval.test.ts tests/unit/router-eval-cli.test.ts tests/unit/router-eval-check.test.ts tests/unit/router-eval-trends.test.ts tests/unit/router-eval-compare.test.ts tests/unit/router-eval-search.test.ts tests/unit/router-eval-patch-compare.test.ts tests/unit/router-eval-e2e-chain.test.ts
• bunx --bun oxlint src/lib/routerEval/index.ts scripts/router-eval/index.ts scripts/router-eval/trends.ts scripts/router-eval/compare.ts scripts/router-eval/search.ts scripts/router-eval/patch-compare.ts scripts/check/check-router-eval-regression.ts tests/unit/router-eval.test.ts tests/unit/router-eval-cli.test.ts tests/unit/router-eval-check.test.ts tests/unit/router-eval-trends.test.ts tests/unit/router-eval-compare.test.ts tests/unit/router-eval-search.test.ts tests/unit/router-eval-patch-compare.test.ts tests/unit/router-eval-e2e-chain.test.ts
• npm run check:router-eval

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive router evaluation framework (router-eval), adding scripts for running, comparing, searching, and checking regressions, along with core library logic, fixtures, and unit tests. The review feedback highlights several critical issues: a potential crash (unhandled ENOENT) in the regression check script when patch files are missing on failure, a TypeScript compilation error in src/lib/routerEval/index.ts from passing an extra argument to asRecord, a potential NaN bug when calculating cost deltas with a zero baseline, a CLI hanging issue when running interactively without piped input, and a Repository Style Guide violation regarding the placement of scripts under the unauthorized scripts/router-eval/ directory.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

If the regression gate fails (result.status !== 0), runPatchGate is skipped, meaning ensureReadable is never called on the patch files. However, writeRetainedRun is still called and attempts to copy the patch files if they are specified in args. If these files do not exist or are unreadable, fs.copyFileSync will throw an unhandled ENOENT exception, crashing the script instead of exiting cleanly with the regression exit code.

Validating and ensuring readability of the patch files at the start of main() resolves this robustness issue.

function main(): void {
  const args = readArgs();
  ensureReadable(args.baseline, "baseline corpus");
  ensureReadable(args.candidate, "candidate corpus");
  if (args.baselinePatch || args.candidatePatch) {
    if (!args.baselinePatch || !args.candidatePatch) {
      console.error("[router-eval] --baseline-patch and --candidate-patch must be provided together");
      process.exit(2);
    }
    ensureReadable(args.baselinePatch, "baseline patch");
    ensureReadable(args.candidatePatch, "candidate patch");
  }
  fs.mkdirSync(path.dirname(args.output), { recursive: true });
  fs.mkdirSync(path.dirname(args.jsonOutput), { recursive: true });

[gemini-code-assist]

The helper function asRecord is defined to accept only a single argument (value: unknown). Passing a second argument {} here will cause a TypeScript compilation error under strict type checking. Please remove the redundant second argument.

Suggested change

	const routeInput = asRecord(value.routeInput, {});
	const routeInput = asRecord(value.routeInput);

[gemini-code-assist]

If baselineCost is 0 and cost is positive, costIncrease is evaluated as Infinity. Multiplying costIncrease * baselineCost (Infinity * 0) results in NaN for delta.costUsd in the comparison report.

Since costIncrease * baselineCost mathematically simplifies to cost - baselineCost, calculating the absolute cost delta directly avoids this edge-case NaN bug and simplifies the logic.

Suggested change

	costUsd: costIncrease * baselineCost,
	costUsd: cost - baselineCost,

[gemini-code-assist]

According to the Repository Style Guide (Rule 1), all scripts must be placed strictly inside one of the allowed scripts/ subfolders: build/, dev/, check/, docs/, i18n/, or ad-hoc/. The new scripts/router-eval/ subfolder is not in this list. Please move these scripts (e.g., compare.ts, index.ts, patch-compare.ts, search.ts, trends.ts) to an allowed subfolder (such as scripts/ad-hoc/router-eval/) and update the corresponding scripts in package.json to adhere to the style guide.

References

1. ALL maintenance, debugging, generation, or experimental scripts MUST be placed strictly inside one of the scripts/ subfolders (build/, dev/, check/, docs/, i18n/, ad-hoc/). ^(link)

[gemini-code-assist]

When running the script interactively without any piped input, calling await readJsonl() without arguments will cause the CLI to hang indefinitely waiting for stdin EOF. Checking process.stdin.isTTY allows the script to exit cleanly with an error message when no input is provided.

  const candidate: RouterObservation[] = args.input
    ? await readJsonl(args.input)
    : hasCandidateDb
      ? readDb(resolveDbPath(args.db), args.since, args.limit, args.dbSource)
      : process.stdin.isTTY
        ? []
        : await readJsonl();

[diegosouzapw]

Thank you, @KooshaPari 🙏 — closing as deferred, not rejected. This PR is the router-eval optimization gate (~6k LOC) — a 3.9.0-scale eval subsystem. It's one piece of the native-router backend workstream tracked in #5670, where I've cataloged all of these as the staged backlog for the 3.9.0/4.0 window (see my comment there). Merging these subsystem pieces individually into the hot release/v3.8.x branch would commit the core to a major rewrite before the design is agreed — so the plan is: lock the migration plan (#6081) as the RFC on #5670, then re-open focused slices against the 3.9.0 branch. The engineering is solid; this is purely about sequencing a big subsystem behind an agreed design. 🚀