Skip to content

[WRONG BRANCH] fix(routing): enforce hard policy cost ceilings on unknown estimates #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
luvs01 wants to merge 1 commit into from
Closed

[WRONG BRANCH] fix(routing): enforce hard policy cost ceilings on unknown estimates #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions src/routing/evaluator.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -304,14 +304,22 @@ export function evaluatePolicyProfile(
const excludedByUnknown = unknown && profile.unknownEvidence.capability === "exclude";
const costLimit = profile.limits.maxEstimatedCostUsd;
const estimatedCost = evidence.cost?.estimatedUsd;
// A hard ceiling requires evidence that the request fits below it. Routing
// with an unknown estimate would otherwise turn the ceiling into the
// profile's (permissive by default) unknown-cost scoring preference.
const unknownCostAtLimit = costLimit !== undefined
&& (typeof estimatedCost !== "number" || !Number.isFinite(estimatedCost));
Comment on lines +310 to +311

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Document fail-closed behavior for unknown cost ceilings

Update the routing documentation to describe this new precedence: because routeModel currently constructs cost evidence without request usage, any live profile with maxEstimatedCostUsd now rejects every candidate with unknown-price, regardless of unknownEvidence.cost. The documented example combines this limit with cost: "penalize", while the reference still says the cap applies when evidence is known, so users following it will receive NoEligiblePolicyCandidateError/404 responses unexpectedly. Revise the English reference and translations to explain the fail-closed behavior and correct the example.

AGENTS.md reference: src/AGENTS.md:L28-L28

Useful? React with 👍 / 👎.

const overCostLimit = costLimit !== undefined
&& typeof estimatedCost === "number"
&& Number.isFinite(estimatedCost)
&& estimatedCost > costLimit;
if (unknownCostAtLimit) {
exclusions.push({ code: "unknown-price", detail: "maxEstimatedCostUsd" });
}
if (overCostLimit) {
exclusions.push({ code: "cost-limit", detail: "maxEstimatedCostUsd" });
}
let eligible = !unsatisfied && !excludedByUnknown && !overCostLimit;
let eligible = !unsatisfied && !excludedByUnknown && !unknownCostAtLimit && !overCostLimit;

// Health scoring (RI-06): live hard cooldown is authoritative and
// excludes; unknown health follows the profile's unknownEvidence policy;
Expand Down Expand Up @@ -349,7 +357,7 @@ export function evaluatePolicyProfile(
// above; unknown cost follows the profile's unknownEvidence policy.
const cost = evidence.cost;
let costValue = cost ? costScore(cost) : null;
if (costValue === null && profile.unknownEvidence.cost === "exclude") {
if (costValue === null && profile.unknownEvidence.cost === "exclude" && !unknownCostAtLimit) {
exclusions.push({ code: "unknown-price" });
eligible = false;
} else if (costValue === null && profile.unknownEvidence.cost === "penalize") {
Expand Down
24 changes: 24 additions & 0 deletions tests/cost-scoring.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import { tmpdir } from "node:os";
import { join } from "node:path";
import { costEvidenceForCandidate, costScore } from "../src/routing/cost";
import { evaluatePolicyProfile, COST_UNKNOWN_PENALTY_SCORE } from "../src/routing/evaluator";
import { NoEligiblePolicyCandidateError, routeModel } from "../src/router";
import type { OcxConfig } from "../src/types";

let testDir = "";
Expand Down Expand Up @@ -100,6 +101,29 @@ describe("cost-aware scoring (RI-08)", () => {
expect(result.selectedIndex).toBeNull();
});

test("runtime routing fails closed when a hard cost ceiling cannot be estimated", () => {
const limited = config({
routingProfiles: {
cost: {
candidates: [{ provider: "anthropic", model: "claude-opus-5" }],
limits: { maxEstimatedCostUsd: 0.000001 },
},
},
});

expect(() => routeModel(limited, "policy/cost")).toThrow(NoEligiblePolicyCandidateError);
try {
routeModel(limited, "policy/cost");
} catch (error) {
const trace = (error as NoEligiblePolicyCandidateError).trace;
expect(trace.candidates[0]).toMatchObject({
eligible: false,
cost: { incomplete: true, limitUsd: 0.000001 },
exclusions: [{ code: "unknown-price", detail: "maxEstimatedCostUsd" }],
});
}
});
Comment on lines +104 to +125

@coderabbitai coderabbitai Bot Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Cover the "exclude" and non-finite branches.

The test does not set unknownEvidence.cost, so it does not guarantee coverage of the duplicate-suppression branch at src/routing/evaluator.ts Line 360. Set that policy explicitly to "exclude" and retain the single-entry exclusion assertion.

Also add cases for Number.NaN, Number.POSITIVE_INFINITY, and Number.NEGATIVE_INFINITY to cover the non-finite predicate at src/routing/evaluator.ts Lines 310-315.

As per path instructions, src/** behavior changes require focused regression coverage under tests/**.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/cost-scoring.test.ts` around lines 104 - 125, Update the runtime
cost-ceiling test around routeModel to explicitly configure unknownEvidence.cost
as "exclude" while preserving the single unknown-price exclusion assertion. Add
focused tests covering maxEstimatedCostUsd values of Number.NaN,
Number.POSITIVE_INFINITY, and Number.NEGATIVE_INFINITY, asserting the expected
failed-closed routing behavior and trace details.

Source: Path instructions


test("unknown cost follows the profile policy (exclude / penalize / allow)", async () => {
const strict = config({
routingProfiles: {
Expand Down
Loading