diff --git a/docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md b/docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md index 4cc210a5e1..3e40886e5f 100644 --- a/docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md +++ b/docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md @@ -33,17 +33,28 @@ all five, and auto-optimize routing by cost. transport only; request-time credential reads go through the KV. 2. **Auto model discovery + governed virtual pools**: discovery runs with the orchestrator's own `discover_all_models()` against the KV credentials. - OpenCode and Noema admit only zero-priced routes. Strix admits provider- - diverse priced routes only when discovery supplies finite, nonnegative - prompt and completion prices plus an explicit currency; missing or malformed - price evidence fails closed. The gateway's + OpenCode and Noema admit only zero-priced routes. Strix admits two explicit + evidence tiers: zero-priced first, then routes with finite, + nonnegative prompt and completion prices plus an explicit currency. Routes + without a complete published price vector remain counted for audit but are + not admitted to CI review. A missing pair is never relabeled free or + price-attested; a partial price vector, malformed numeric value, conflicting + free marker, or missing currency for a published vector fails closed. The gateway's `orchestrator/free` virtual id fails closed (`400 invalid_model`) unless an enabled zero-cost agent exists. Strix uses `orchestrator/auto`; its catalog - may admit priced routes only through the same evidence-bearing policy, never - through a direct-provider model identifier. -3. **ZDR-first selection**: `scripts/ci/zdr_policy.py` defines ZDR the way - OpenRouter does ("a provider will not store your data for any period of - time"; zero retention also implies no training) and is deliberately + may admit priced routes only through this evidence-bearing + policy, never through a direct-provider model identifier. + The auto pool probes the free catalog first. Only when every selected free + route rejects the real runtime request contract does it rebuild once from + fully price-attested routes and record the rejected primary attempt. This is + evidence-triggered failover, not an arbitrary free/paid mixing ratio. + Both stages share one twelve-route startup budget: no more than eight routes + enter the free primary stage and only its remaining capacity may enter priced + fallback. Full discovery counts remain in policy evidence, and the transient + priced catalog is removed immediately after loading. +3. **ZDR-first within each cost tier**: `scripts/ci/zdr_policy.py` defines ZDR + the way OpenRouter does ("a provider will not store your data for any period + of time"; zero retention also implies no training) and is deliberately conservative: any provider whose zero-retention guarantee cannot be attested from a machine-readable, dated source is treated as non-ZDR, mirroring OpenRouter's stance on unascertained policies. The @@ -51,10 +62,15 @@ all five, and auto-optimize routing by cost. fetched when egress allows it and is authoritative for the `openrouter` scope; otherwise the dated static attestation table is used, never a fabricated policy. - `scripts/ci/contextual_orchestrator_review_policy.py` turns the price- - evidenced discovery report into a ZDR-prioritized, provider-family-diverse agents - catalog (primary/secondary NVIDIA keys share one outage-domain family), - capped in size, in the orchestrator's own `ModelAgent` schema. + For private targets, ZDR admission is applied before choosing the cost tier. + A discovered but non-ZDR free route therefore cannot suppress an attested + priced route; when an admitted free tier exists it remains the exclusive + primary, and the admitted priced tier remains fallback-only. + `scripts/ci/contextual_orchestrator_review_policy.py` turns the discovery + report into a free-first, cost-evidence-ranked, ZDR-prioritized, + provider-family-diverse agents catalog (primary/secondary NVIDIA keys share + one outage-domain family), capped in size, in the orchestrator's own + `ModelAgent` schema. 4. **Wiring**: `pr-review-autofix.yml` and the Required OpenCode dispatch provision the sidecar with the five secrets before OpenCode runs and point every model/diagnosis candidate at `contextual-orchestrator/orchestrator/free`; @@ -66,17 +82,18 @@ all five, and auto-optimize routing by cost. discovered free routes all shared the OpenRouter outage domain, which the gateway correctly collapsed to one provider attempt. Strix therefore uses the provider-diverse pool supplied by all five configured credentials. - Provider diversity and provider cost evidence remain delegated to the + Provider diversity and cost-evidence classification remain delegated to the gateway rather than embedding a second routing policy in GitHub Actions. - Strix has no external fallback and private targets pass - visibility through to the gateway's ZDR requirement. Noema reviewer identity - remains `NOEMA_REVIEW_TOKEN` / GitHub App / OIDC and is still never - `github.token`; Autofix mutation still requires `PR_REVIEW_MERGE_TOKEN` / + Strix has no external fallback and private targets pass visibility through + to the gateway's ZDR requirement. Noema reviewer identity remains + `NOEMA_REVIEW_TOKEN` / GitHub App / OIDC and is still never `github.token`; + Autofix mutation still requires `PR_REVIEW_MERGE_TOKEN` / `OPENCODE_APPROVE_TOKEN` / the exchanged OpenCode app token, never `github.token`; model subprocesses still run with `GITHUB_TOKEN`/`GH_TOKEN`/OIDC request env stripped. 5. **Evidence**: the sidecar writes a discovery report, the policy report (pool, - counts, ZDR sources, feed-used flag, selected routes), and exports + total/free/priced/unknown counts, selected counts by admitted cost tier, ZDR + sources, feed-used flag, selected routes), and exports `CONTEXTUAL_ORCHESTRATOR_EVIDENCE`; these are auditable per run. 6. **Review request envelope**: the library keeps its generic 64 KiB default, while this loopback, bearer-authenticated, per-job sidecar configures a @@ -91,10 +108,12 @@ all five, and auto-optimize routing by cost. - The autofix/OpenCode review paths no longer hard-code any provider base URL or model id; upstream model selection is delegated to the orchestrator's discovery under the zero-cost pool. Strix uses the separately governed auto - pool without treating absent price metadata as paid-route evidence. + pool without treating absent price metadata as either free or paid-route + evidence. - Strix delegates selection to `orchestrator/auto`. Its correctness-first pool remains distinct from the zero-cost OpenCode/Noema pool, while private-target - ZDR admission remains fail-closed. + ZDR admission remains fail-closed. Unknown-cost routes remain auditable but + ineligible; free and fully price-attested routes are the only review routes. - Workers need egress to the five provider model-list hosts and, when reachable, `https://openrouter.ai/api/v1/endpoints/zdr`; the feed failure path is graceful (static table). diff --git a/scripts/ci/contextual_orchestrator_review_launcher.py b/scripts/ci/contextual_orchestrator_review_launcher.py index f7bd07b343..abb9af3b21 100644 --- a/scripts/ci/contextual_orchestrator_review_launcher.py +++ b/scripts/ci/contextual_orchestrator_review_launcher.py @@ -42,6 +42,8 @@ # for a required CI gate. With at most twelve sequential candidates, startup is # bounded below the sidecar's three-minute readiness deadline. REVIEW_PREFLIGHT_TIMEOUT_SECONDS = 10 +REVIEW_PREFLIGHT_MAX_TOTAL_ROUTES = 12 +REVIEW_PREFLIGHT_PRIMARY_ROUTE_LIMIT = 8 class ReviewPreflightError(RuntimeError): @@ -222,6 +224,26 @@ def _preflight_review_agents( return viable, report +def _preflight_with_fallback( + primary_agents: list[object], fallback_agents: list[object], *, client: Any +) -> tuple[list[object], dict[str, object], bool]: + """Use the priced catalog only after every primary route rejects.""" + try: + viable, report = _preflight_review_agents(primary_agents, client=client) + return viable, report, False + except ReviewPreflightError as primary_error: + if not fallback_agents: + raise + try: + viable, report = _preflight_review_agents(fallback_agents, client=client) + except ReviewPreflightError as fallback_error: + fallback_error.report["primary_attempt"] = primary_error.report + raise + report["primary_attempt"] = primary_error.report + report["fallback_reason"] = "primary_routes_unavailable" + return viable, report, True + + def _write_json(path: str, payload: object) -> None: """Write one deterministic UTF-8 JSON evidence file.""" Path(path).write_text( @@ -229,6 +251,79 @@ def _write_json(path: str, payload: object) -> None: ) +def _bounded_primary_catalog_limit( + requested_limit: int, *, pool: str, has_free_rows: bool +) -> int: + """Return the primary-stage route limit within one startup budget.""" + if requested_limit < 1: + raise ValueError("ORCHESTRATOR_CATALOG_LIMIT must be positive") + total_limit = min(requested_limit, REVIEW_PREFLIGHT_MAX_TOTAL_ROUTES) + if pool == "auto" and has_free_rows: + return min(total_limit, REVIEW_PREFLIGHT_PRIMARY_ROUTE_LIMIT) + return total_limit + + +def _bounded_fallback_catalog_limit( + requested_limit: int, *, primary_count: int +) -> int: + """Return remaining priced-fallback capacity after primary selection.""" + if requested_limit < 1: + raise ValueError("ORCHESTRATOR_CATALOG_LIMIT must be positive") + total_limit = min(requested_limit, REVIEW_PREFLIGHT_MAX_TOTAL_ROUTES) + if primary_count < 0 or primary_count > total_limit: + raise ValueError("primary route count exceeds the preflight budget") + return total_limit - primary_count + + +def _with_discovery_counts( + report: dict[str, object], rows: list[dict[str, Any]] +) -> dict[str, object]: + """Copy a stage report while restoring full discovery-tier counts.""" + enriched = dict(report) + enriched.update( + { + "total_routes": len(rows), + "total_free_routes": sum(row.get("cost_evidence") == "free" for row in rows), + "total_priced_routes": sum(row.get("cost_evidence") == "priced" for row in rows), + "total_unknown_routes": sum(row.get("cost_evidence") == "unknown" for row in rows), + } + ) + return enriched + + +def _zdr_admitted_rows( + rows: list[dict[str, Any]], + *, + require_zdr: bool, + zdr_endpoints: frozenset[str], + checker: Any, +) -> list[dict[str, Any]]: + """Return rows that can enter the selected privacy boundary.""" + if not require_zdr: + return list(rows) + return [ + row + for row in rows + if checker( + str(row["provider"]), + model=str(row["model"]), + zdr_endpoints=zdr_endpoints, + ) + ] + + +def _load_temporary_agents( + path: str, catalog_agents: list[dict[str, Any]], *, loader: Any +) -> list[object]: + """Load one transient catalog and remove it on every exit path.""" + catalog_path = Path(path) + _write_json(str(catalog_path), {"agents": catalog_agents}) + try: + return list(loader(str(catalog_path))) + finally: + catalog_path.unlink(missing_ok=True) + + def main(argv: list[str] | None = None) -> int: """Bootstrap the KV, discover and preflight free models, then serve. @@ -269,8 +364,10 @@ def main(argv: list[str] | None = None) -> int: ) from contextual_orchestrator.server import SecurityConfig, serve from scripts.ci.contextual_orchestrator_review_policy import ( + PolicyError, _load_zdr_endpoints, build_zdr_prioritized_catalog, + is_zdr_model, parse_discovery_report, ) @@ -306,14 +403,43 @@ def main(argv: list[str] | None = None) -> int: rows = _report_rows(selected_models, free_route_identities) _write_json(args.discovery_out, {"models": rows}) zdr_endpoints = _load_zdr_endpoints(args.zdr_endpoints) + normalized_rows = parse_discovery_report({"models": rows}) + free_rows = [ + row for row in normalized_rows if row.get("cost_evidence") == "free" + ] + priced_rows = [ + row for row in normalized_rows if row.get("cost_evidence") == "priced" + ] + admitted_free_rows = _zdr_admitted_rows( + free_rows, + require_zdr=args.require_zdr, + zdr_endpoints=zdr_endpoints, + checker=is_zdr_model, + ) + admitted_priced_rows = _zdr_admitted_rows( + priced_rows, + require_zdr=args.require_zdr, + zdr_endpoints=zdr_endpoints, + checker=is_zdr_model, + ) + requested_catalog_limit = int(os.environ.get("ORCHESTRATOR_CATALOG_LIMIT", "12")) + primary_limit = _bounded_primary_catalog_limit( + requested_catalog_limit, pool=args.pool, has_free_rows=bool(admitted_free_rows) + ) + primary_rows = ( + (admitted_free_rows or admitted_priced_rows) + if args.pool == "auto" + else normalized_rows + ) result = build_zdr_prioritized_catalog( - parse_discovery_report({"models": rows}), - limit=int(os.environ.get("ORCHESTRATOR_CATALOG_LIMIT", "12")), + primary_rows, + limit=primary_limit, family_cap=int(os.environ.get("ORCHESTRATOR_CATALOG_FAMILY_CAP", "4")), zdr_endpoints=zdr_endpoints, require_zdr=args.require_zdr, pool=args.pool, ) + result["report"] = _with_discovery_counts(result["report"], normalized_rows) Path(args.catalog_out).write_text( json.dumps({"agents": result["agents"]}, indent=2, sort_keys=True) + "\n", encoding="utf-8", @@ -321,6 +447,42 @@ def main(argv: list[str] | None = None) -> int: _write_json(args.report_out, result["report"]) agents = load_agents(args.catalog_out) + primary_report = result["report"] + fallback_result = None + fallback_agents: list[object] = [] + fallback_limit = _bounded_fallback_catalog_limit( + requested_catalog_limit, primary_count=len(result["agents"]) + ) + if ( + args.pool == "auto" + and admitted_free_rows + and admitted_priced_rows + and fallback_limit + ): + try: + fallback_result = build_zdr_prioritized_catalog( + admitted_priced_rows, + limit=fallback_limit, + family_cap=int(os.environ.get("ORCHESTRATOR_CATALOG_FAMILY_CAP", "4")), + zdr_endpoints=zdr_endpoints, + require_zdr=args.require_zdr, + pool="auto", + ) + except PolicyError: + fallback_result = None + if fallback_result is not None: + fallback_result["report"] = _with_discovery_counts( + fallback_result["report"], normalized_rows + ) + fallback_result["report"]["primary_selected_count"] = primary_report[ + "selected_count" + ] + fallback_result["report"]["primary_selection"] = primary_report["selected"] + fallback_agents = _load_temporary_agents( + f"{args.catalog_out}.priced", + fallback_result["agents"], + loader=load_agents, + ) client = ModelClient( timeout=REVIEW_PREFLIGHT_TIMEOUT_SECONDS, max_output_tokens=REVIEW_MAX_OUTPUT_TOKENS, @@ -328,10 +490,21 @@ def main(argv: list[str] | None = None) -> int: temperature=REVIEW_TEMPERATURE, ) try: - agents, preflight_report = _preflight_review_agents(agents, client=client) + agents, preflight_report, fallback_used = _preflight_with_fallback( + agents, fallback_agents, client=client + ) except ReviewPreflightError as exc: _write_json(args.preflight_out, exc.report) raise SystemExit(f"review sidecar preflight failed: {exc}") from None + if fallback_used and fallback_result is not None: + Path(args.catalog_out).write_text( + json.dumps({"agents": fallback_result["agents"]}, indent=2, sort_keys=True) + + "\n", + encoding="utf-8", + ) + result = fallback_result + result["report"]["fallback_reason"] = "primary_routes_unavailable" + _write_json(args.report_out, result["report"]) _write_json(args.preflight_out, preflight_report) client = ModelClient( diff --git a/scripts/ci/contextual_orchestrator_review_policy.py b/scripts/ci/contextual_orchestrator_review_policy.py index 01bea6cc46..6d2a6a86bb 100644 --- a/scripts/ci/contextual_orchestrator_review_policy.py +++ b/scripts/ci/contextual_orchestrator_review_policy.py @@ -1,21 +1,9 @@ -"""Turn an orchestrator discovery report into a ZDR-prioritized ``orchestrator/free`` catalog. - -The contextual-orchestrator server exposes the fail-closed zero-cost pool under -the virtual model id ``orchestrator/free``; it only resolves when at least one -enabled agent is an explicitly zero-priced (``cost:free``) model -(``contextual_orchestrator/orchestrator.py`` ``_is_free_agent``). This module, - -1. reads the same ``discover-models`` report the orchestrator prints, -2. keeps only free (zero-cost), known-provider chat routes, -3. orders them ZDR-compliant first, then non-ZDR free, with a provider-family - cap so a single outage domain cannot monopolize the pool, and -4. writes an ``agents`` JSON catalog in the orchestrator's own - ``ModelAgent.to_config()`` schema so the vendored sidecar can - ``load_agents()`` it unchanged. - -Everything is stdlib-only and offline-testable; the live OpenRouter ZDR feed is -an optional input file so CI either attests real ZDR endpoints or falls back to -the static ``scripts/ci/zdr_policy.py`` table (never fabricated). +"""Build governed contextual-orchestrator review catalogs from discovery evidence. + +``orchestrator/free`` remains strictly zero-priced. ``orchestrator/auto`` is +free-first and then uses fully price-attested routes. Models without a complete +price vector remain visible in audit counts but are never admitted to CI review. +Partial, malformed, or contradictory price vectors fail closed. """ from __future__ import annotations @@ -23,9 +11,8 @@ import argparse import json import math -import os -import sys import re +import sys from collections import Counter from pathlib import Path from typing import Any, Iterable, Mapping @@ -40,8 +27,6 @@ route_key, ) -# NVIDIA primary and secondary keys share one outage domain for provider-family -# diversity, mirroring contextual_orchestrator ``_provider_family``. PROVIDER_FAMILIES: Mapping[str, str] = { "nvidia_nim": "nvidia_nim", "nvidia_nim_sub": "nvidia_nim", @@ -50,30 +35,29 @@ DEFAULT_CATALOG_LIMIT = 12 DEFAULT_FAMILY_CAP = 4 +COST_FREE = "free" +COST_PRICED = "priced" +COST_UNKNOWN = "unknown" +_COST_EVIDENCE_RANK: Mapping[str, int] = { + COST_FREE: 0, + COST_PRICED: 1, + COST_UNKNOWN: 2, +} + _AGENT_ID_RE = re.compile(r"^[a-z][a-z0-9]*_[a-z0-9]+(?:_[a-z0-9]+)*$") +class PolicyError(ValueError): + """Raised when discovery evidence cannot produce a governed catalog.""" + + def provider_family(provider_name: str) -> str: - """Return the outage-domain family for a provider (itself when ungrouped).""" + """Return the outage-domain family for a provider.""" return PROVIDER_FAMILIES.get(provider_name, provider_name) def _normalize_agent_id(candidate: str, provider_name: str) -> str: - """Return a valid two-or-more-word snake_case agent id. - - The orchestrator requires agent ids to match the org naming convention - (``[tool.contextual_orchestrator] object_name_pattern = - two_or_more_words_snake_case``). Discovery emits ids like - ``openrouter_deepseek_deepseek_r1_free``; this normalizes any provider/model - string that does not already comply. - - Args: - candidate: Candidate agent id from the discovery report. - provider_name: Provider identifier used as the first segment fallback. - - Returns: - A naming-compliant snake_case id. - """ + """Return a two-or-more-word snake_case agent identifier.""" slug = re.sub(r"[^a-zA-Z0-9]+", "_", candidate).strip("_").lower() parts = [part for part in slug.split("_") if part] if len(parts) == 1: @@ -82,12 +66,12 @@ def _normalize_agent_id(candidate: str, provider_name: str) -> str: def _route_key(provider_name: str, model: str) -> str: - """Return the ``provider/model`` key used by the ZDR endpoints feed.""" + """Return the provider/model key used by the ZDR feed.""" return route_key(provider_name, model) def _is_valid_is_free(value: object) -> bool: - """Return whether a report row carries an explicit free marker.""" + """Return whether a row carries an explicit scalar free marker.""" if isinstance(value, str): return value.strip() != "" if isinstance(value, bool): @@ -97,12 +81,8 @@ def _is_valid_is_free(value: object) -> bool: return False -class PolicyError(ValueError): - """Raised when a discovery report cannot produce a usable catalog.""" - - def _validated_price(value: object, *, route: str, field: str) -> float: - """Return a finite nonnegative published price or reject the route.""" + """Return a finite nonnegative published price or reject it.""" if isinstance(value, bool) or not isinstance(value, (int, float)): raise PolicyError(f"model {route} lacks numeric {field} evidence") price = float(value) @@ -111,25 +91,48 @@ def _validated_price(value: object, *, route: str, field: str) -> float: return price -def parse_discovery_report(report: Mapping[str, Any]) -> list[dict[str, Any]]: - """Validate and extract the model rows from a ``discover-models`` report. +def _normalize_cost_evidence( + *, + route: str, + is_free: bool, + prompt_price: object, + completion_price: object, + currency_code: object, +) -> tuple[str, float | None, float | None, str | None]: + """Classify complete free, priced, or wholly unavailable price evidence. + + A provider that publishes neither price component is retained for audit but + is not eligible for review routing. A partial vector is ambiguous and + rejected. Free markers remain authoritative only when any accompanying + published vector is complete, valid, and zero-priced. + """ + if prompt_price is None and completion_price is None: + return (COST_UNKNOWN, None, None, None) - Args: - report: Parsed JSON report with a top-level ``models`` list; each row - must carry ``provider``, ``model``, and ``is_free``. + normalized_prompt = _validated_price( + prompt_price, route=route, field="prompt_price_per_1k" + ) + normalized_completion = _validated_price( + completion_price, route=route, field="completion_price_per_1k" + ) + if not isinstance(currency_code, str) or not currency_code.strip(): + raise PolicyError(f"model {route} lacks currency_code evidence") + if is_free and (normalized_prompt != 0 or normalized_completion != 0): + raise PolicyError(f"model {route} conflicts with its free price marker") + return ( + COST_FREE if is_free else COST_PRICED, + normalized_prompt, + normalized_completion, + currency_code.strip().upper(), + ) - Returns: - A list of normalized row dicts with ``provider``, ``model``, ``agent_id`` - (fallback to provider_model), and an explicit boolean ``is_free``. - Raises: - PolicyError: If the report has no ``models`` list, or any row lacks a - provider/model, or names a provider outside the ZDR policy table so - routing cannot silently bypass the policy. - """ +def parse_discovery_report(report: Mapping[str, Any]) -> list[dict[str, Any]]: + """Validate and normalize a contextual-orchestrator discovery report.""" rows = report.get("models") if not isinstance(rows, list): raise PolicyError("discovery report must contain a 'models' list") + normalized: list[dict[str, Any]] = [] for row in rows: if not isinstance(row, dict): @@ -141,42 +144,56 @@ def parse_discovery_report(report: Mapping[str, Any]) -> list[dict[str, Any]]: if not model or not isinstance(model, str): raise PolicyError("discovery model row is missing 'model'") if provider not in PROVIDER_CREDENTIAL_NAMES: - raise PolicyError(f"provider {provider!r} is not registered in the ZDR policy table") - candidate_id = row.get("agent_id") or f"{provider}_{model}" + raise PolicyError( + f"provider {provider!r} is not registered in the ZDR policy table" + ) if not _is_valid_is_free(row.get("is_free")): - raise PolicyError(f"model {provider}/{model} lacks an explicit is_free marker") + raise PolicyError( + f"model {provider}/{model} lacks an explicit is_free marker" + ) + is_free = is_free_route(row.get("is_free")) route = f"{provider}/{model}" - prompt_price = row.get("prompt_price_per_1k") - completion_price = row.get("completion_price_per_1k") - currency_code = row.get("currency_code") - if not is_free: - prompt_price = _validated_price( - prompt_price, route=route, field="prompt_price_per_1k" - ) - completion_price = _validated_price( - completion_price, route=route, field="completion_price_per_1k" + cost_evidence, prompt_price, completion_price, currency_code = ( + _normalize_cost_evidence( + route=route, + is_free=is_free, + prompt_price=row.get("prompt_price_per_1k"), + completion_price=row.get("completion_price_per_1k"), + currency_code=row.get("currency_code"), ) - if not isinstance(currency_code, str) or not currency_code.strip(): - raise PolicyError(f"model {route} lacks currency_code evidence") - currency_code = currency_code.strip().upper() + ) + candidate_id = row.get("agent_id") or f"{provider}_{model}" normalized.append( { "provider": provider, "model": model, "agent_id": str(candidate_id), "is_free": is_free, + "cost_evidence": cost_evidence, "prompt_price_per_1k": prompt_price, "completion_price_per_1k": completion_price, "currency_code": currency_code, "base_url": row.get("base_url") or PROVIDER_BASE_URLS[provider], - "credential_key": row.get("credential_key") or PROVIDER_CREDENTIAL_NAMES[provider], - "auth_scheme": row.get("auth_scheme") or PROVIDER_AUTH_SCHEMES[provider], + "credential_key": row.get("credential_key") + or PROVIDER_CREDENTIAL_NAMES[provider], + "auth_scheme": row.get("auth_scheme") + or PROVIDER_AUTH_SCHEMES[provider], } ) return normalized +def _cost_evidence(row: Mapping[str, Any]) -> str: + """Return a validated cost-evidence tier from a normalized row.""" + evidence = row.get("cost_evidence") + if evidence in _COST_EVIDENCE_RANK: + return str(evidence) + # Backward compatibility for callers that build normalized-like rows by + # hand rather than using parse_discovery_report(). + return COST_FREE if row.get("is_free") is True else COST_UNKNOWN + + def build_zdr_prioritized_catalog( rows: Iterable[Mapping[str, Any]], *, @@ -186,71 +203,47 @@ def build_zdr_prioritized_catalog( require_zdr: bool = False, pool: str = "free", ) -> dict[str, Any]: - """Select and rank governed routes into a ZDR-first, family-diverse catalog. - - Ranking is deterministic and evidence-based, never heuristic cost guesses: - free (zero-cost, attested by discovery price metadata) routes always outrank - any priced route; among free routes ZDR-compliant routes outrank the rest; - within a tier the provider/model order is stable. A provider-family cap - prevents the primary and secondary NVIDIA keys from absorbing the whole - pool. Priced routes are excluded because the org's CI review path is pinned - to the ``orchestrator/free`` pool. - - Args: - rows: Normalized discovery rows (see ``parse_discovery_report``). - limit: Maximum number of catalog agents (orchestrator default 12). - family_cap: Maximum agents per provider outage-domain family. - zdr_endpoints: ``provider/model`` route keys from the OpenRouter ZDR - feed; authoritative when non-empty for the openrouter scope. - require_zdr: Admit only routes with attested ZDR evidence. Intended for - private/internal target repositories; an empty ZDR pool fails closed. - - Returns: - A dict with ``agents`` (orchestrator ``ModelAgent.to_config()`` rows, - ZDR-first) and ``report`` (counts + evidence for audit). - - Raises: - PolicyError: If no free, known-provider route remains (the catalog - would fail closed at serve time anyway, but this makes the failure - early and explainable). - """ - catalog_rows: list[dict[str, Any]] = [] - per_family: Counter[str] = Counter() - zdr_count = 0 - - def family_is_open(family: str) -> bool: - """Return whether a provider family still has catalog capacity.""" - return per_family[family] < family_cap - + """Select a free-first, ZDR-aware, provider-family-diverse catalog.""" if pool not in {"free", "auto"}: raise PolicyError(f"unsupported review pool {pool!r}") + all_rows = list(rows) - all_free_rows = [row for row in all_rows if row["is_free"]] - candidate_rows = all_free_rows if pool == "free" else all_rows + all_free_rows = [row for row in all_rows if _cost_evidence(row) == COST_FREE] + all_priced_rows = [row for row in all_rows if _cost_evidence(row) == COST_PRICED] + all_unknown_rows = [row for row in all_rows if _cost_evidence(row) == COST_UNKNOWN] + candidate_rows = ( + all_free_rows if pool == "free" else [*all_free_rows, *all_priced_rows] + ) eligible_rows = [ row for row in candidate_rows if not require_zdr or is_zdr_model( - row["provider"], model=row["model"], zdr_endpoints=zdr_endpoints + str(row["provider"]), + model=str(row["model"]), + zdr_endpoints=zdr_endpoints, ) ] eligible_rows.sort( key=lambda row: ( + _COST_EVIDENCE_RANK[_cost_evidence(row)], 0 if is_zdr_model( - row["provider"], model=row["model"], zdr_endpoints=zdr_endpoints + str(row["provider"]), + model=str(row["model"]), + zdr_endpoints=zdr_endpoints, ) else 1, - 0 if row["is_free"] else 1, - row["provider"], - row["model"], + str(row["provider"]), + str(row["model"]), ) ) - picked: list[dict[str, Any]] = [] - for _order, row in enumerate(eligible_rows): - family = provider_family(row["provider"]) - if not family_is_open(family): + + per_family: Counter[str] = Counter() + picked: list[Mapping[str, Any]] = [] + for row in eligible_rows: + family = provider_family(str(row["provider"])) + if per_family[family] >= family_cap: continue per_family[family] += 1 picked.append(row) @@ -264,27 +257,32 @@ def family_is_open(family: str) -> bool: f"orchestrator/{pool} would fail closed" ) + catalog_rows: list[dict[str, Any]] = [] + zdr_count = 0 for rank, row in enumerate(picked): + provider = str(row["provider"]) + model = str(row["model"]) + evidence = _cost_evidence(row) zdr = is_zdr_model( - row["provider"], model=row["model"], zdr_endpoints=zdr_endpoints + provider, model=model, zdr_endpoints=zdr_endpoints ) if zdr: zdr_count += 1 catalog_rows.append( { - "id": _normalize_agent_id(row["agent_id"], row["provider"]), - "model": row["model"], + "id": _normalize_agent_id(str(row["agent_id"]), provider), + "model": model, "base_url": row["base_url"], "api_key_env": "", "credential_key": row["credential_key"], "tags": [ "review", - "cost:free" if row["is_free"] else "cost:priced", + f"cost:{evidence}", "zdr" if zdr else "non-zdr", ], "priority": -rank, "disabled": False, - "provider_name": row["provider"], + "provider_name": provider, "provider_exclusions": [], "local_credential_key": "", "auth_scheme": row["auth_scheme"], @@ -294,19 +292,31 @@ def family_is_open(family: str) -> bool: } ) + selected_evidence = [_cost_evidence(row) for row in picked] return { "agents": catalog_rows, "report": { "pool": f"orchestrator/{pool}", "total_routes": len(all_rows), "total_free_routes": len(all_free_rows), + "total_priced_routes": len(all_priced_rows), + "total_unknown_routes": len(all_unknown_rows), "zdr_required": require_zdr, "selected_count": len(catalog_rows), - "free_selected_count": sum(1 for row in picked if row["is_free"]), - "priced_selected_count": sum(1 for row in picked if not row["is_free"]), + "free_selected_count": selected_evidence.count(COST_FREE), + "priced_selected_count": selected_evidence.count(COST_PRICED), + "unknown_selected_count": selected_evidence.count(COST_UNKNOWN), "zdr_selected_count": zdr_count, "zdr_sources": sorted( - {provider_zdr_scope(row["provider"]).source for row in picked if is_zdr_model(row["provider"], model=row["model"], zdr_endpoints=zdr_endpoints)} + { + provider_zdr_scope(str(row["provider"])).source + for row in picked + if is_zdr_model( + str(row["provider"]), + model=str(row["model"]), + zdr_endpoints=zdr_endpoints, + ) + } ), "zdr_endpoints_feed_used": bool(zdr_endpoints), "selected": [ @@ -314,7 +324,12 @@ def family_is_open(family: str) -> bool: "provider": row["provider"], "model": row["model"], "agent_id": entry["id"], - "zdr": is_zdr_model(row["provider"], model=row["model"], zdr_endpoints=zdr_endpoints), + "cost_evidence": _cost_evidence(row), + "zdr": is_zdr_model( + str(row["provider"]), + model=str(row["model"]), + zdr_endpoints=zdr_endpoints, + ), } for row, entry in zip(picked, catalog_rows) ], @@ -323,16 +338,7 @@ def family_is_open(family: str) -> bool: def _load_zdr_endpoints(path: str | None) -> frozenset[str]: - """Load ``provider/model`` ZDR route keys from the OpenRouter ZDR feed. - - Args: - path: Path to the feed JSON (shape ``{\"data\": [{\"name\", \"model_name\", - \"provider_name\", \"supports_implicit_caching\"}]}``); None yields no - feed evidence. - - Returns: - A frozen set of ``f\"{provider_name}/{model_name}\"`` keys. - """ + """Load exact provider/model keys from an OpenRouter ZDR feed file.""" if not path: return frozenset() payload = json.loads(Path(path).read_text(encoding="utf-8")) @@ -357,63 +363,47 @@ def build_catalog_from_paths( require_zdr: bool = False, pool: str = "free", ) -> dict[str, Any]: - """Build and persist the ZDR-prioritized ``orchestrator/free`` catalog. - - Args: - discovery_path: Path to the ``discover-models`` JSON report. - out_path: Where to write the agents catalog JSON. - report_path: Where to write the audit/evidence JSON report. - limit: Maximum number of catalog agents. - family_cap: Maximum agents per provider outage-domain family. - zdr_endpoints_path: Optional OpenRouter ZDR feed JSON path. - require_zdr: Admit only attested ZDR routes and fail closed otherwise. - - Returns: - The return value of ``build_zdr_prioritized_catalog`` (both files were - also written). - """ + """Build and persist an agents catalog and its audit report.""" report = json.loads(Path(discovery_path).read_text(encoding="utf-8")) - zdr_endpoints = _load_zdr_endpoints(zdr_endpoints_path) result = build_zdr_prioritized_catalog( parse_discovery_report(report), limit=limit, family_cap=family_cap, - zdr_endpoints=zdr_endpoints, + zdr_endpoints=_load_zdr_endpoints(zdr_endpoints_path), require_zdr=require_zdr, pool=pool, ) Path(out_path).write_text( - json.dumps({"agents": result["agents"]}, indent=2) + "\n", encoding="utf-8" + json.dumps({"agents": result["agents"]}, indent=2) + "\n", + encoding="utf-8", + ) + Path(report_path).write_text( + json.dumps(result["report"], indent=2) + "\n", + encoding="utf-8", ) - Path(report_path).write_text(json.dumps(result["report"], indent=2) + "\n", encoding="utf-8") return result def _build_parser() -> argparse.ArgumentParser: - """Build the CLI parser with the required catalog and report paths.""" + """Build the command-line parser for catalog generation.""" parser = argparse.ArgumentParser( - description="Build a ZDR-prioritized orchestrator/free agents catalog from a discovery report." + description="Build a governed contextual-orchestrator review catalog." + ) + parser.add_argument( + "--discovery-report", required=True, help="Path to the discovery JSON" ) - parser.add_argument("--discovery-report", required=True, help="Path to the discover-models JSON report") - parser.add_argument("--out", required=True, help="Path to write the agents catalog JSON") - parser.add_argument("--report", required=True, help="Path to write the audit evidence JSON") + parser.add_argument("--out", required=True, help="Path to write agents JSON") + parser.add_argument("--report", required=True, help="Path to write audit JSON") parser.add_argument("--limit", type=int, default=DEFAULT_CATALOG_LIMIT) parser.add_argument("--family-cap", type=int, default=DEFAULT_FAMILY_CAP) - parser.add_argument("--zdr-endpoints", default=None, help="Optional OpenRouter /api/v1/endpoints/zdr JSON path") - parser.add_argument("--require-zdr", action="store_true", help="Fail closed unless every selected route has attested ZDR evidence") + parser.add_argument("--zdr-endpoints", default=None) + parser.add_argument("--require-zdr", action="store_true") parser.add_argument("--pool", choices=("free", "auto"), default="free") return parser def main(argv: list[str] | None = None) -> int: - """CLI entry point; returns 1 on PolicyError/unusable catalog (fail closed). - - Args: - argv: CLI arguments (defaults to ``sys.argv[1:]``). - - Returns: - 0 on success; 1 when the catalog cannot be built. - """ + """Run the catalog CLI and return one on policy or input failure.""" args = _build_parser().parse_args(argv) try: build_catalog_from_paths( diff --git a/tests/test_contextual_orchestrator_review_live_discovery_contract.py b/tests/test_contextual_orchestrator_review_live_discovery_contract.py new file mode 100644 index 0000000000..32af68a30a --- /dev/null +++ b/tests/test_contextual_orchestrator_review_live_discovery_contract.py @@ -0,0 +1,154 @@ +"""Regression tests for live provider catalogs with unavailable price metadata.""" + +from __future__ import annotations + +import pytest + +from scripts.ci import contextual_orchestrator_review_policy as policy + + +FREE_MODEL = "qwen/qwen3-coder:free" +PRICED_MODEL = "anthropic/claude-sonnet-4.6" +PRICED_ZDR_FEED = frozenset({f"openrouter/{PRICED_MODEL}"}) + + +def _live_discovery_report() -> dict[str, object]: + """Mirror the price shapes returned by the current provider list APIs.""" + return { + "models": [ + { + "provider": "openrouter", + "model": FREE_MODEL, + "agent_id": "openrouter_qwen3_coder_free", + "is_free": True, + "prompt_price_per_1k": 0.0, + "completion_price_per_1k": 0.0, + "currency_code": "USD", + }, + { + "provider": "openrouter", + "model": PRICED_MODEL, + "agent_id": "openrouter_claude_sonnet_46", + "is_free": False, + "prompt_price_per_1k": 0.003, + "completion_price_per_1k": 0.015, + "currency_code": "USD", + }, + { + "provider": "openai", + "model": "gpt-5.6", + "agent_id": "openai_gpt_56", + "is_free": False, + "prompt_price_per_1k": None, + "completion_price_per_1k": None, + "currency_code": "USD", + }, + { + "provider": "nvidia_nim", + "model": "meta/llama-3.3-70b-instruct", + "agent_id": "nvidia_nim_llama_33_70b", + "is_free": False, + "prompt_price_per_1k": None, + "completion_price_per_1k": None, + "currency_code": "USD", + }, + { + "provider": "nvidia_nim_sub", + "model": "nvidia/llama-3.1-nemotron-ultra-253b-v1", + "agent_id": "nvidia_nim_sub_nemotron_ultra", + "is_free": False, + "prompt_price_per_1k": None, + "completion_price_per_1k": None, + "currency_code": "USD", + }, + { + "provider": "bytez", + "model": "Qwen/Qwen3-Coder-480B-A35B-Instruct", + "agent_id": "bytez_qwen3_coder_480b", + "is_free": False, + "prompt_price_per_1k": None, + "completion_price_per_1k": None, + "currency_code": "USD", + }, + ] + } + + +def test_live_catalog_keeps_complete_unknown_prices_as_unknown() -> None: + """Unavailable price vectors are fallback evidence, not a catalog-wide error.""" + rows = policy.parse_discovery_report(_live_discovery_report()) + evidence = {row["model"]: row["cost_evidence"] for row in rows} + + assert evidence[FREE_MODEL] == "free" + assert evidence[PRICED_MODEL] == "priced" + assert evidence["gpt-5.6"] == "unknown" + assert evidence["meta/llama-3.3-70b-instruct"] == "unknown" + assert evidence["Qwen/Qwen3-Coder-480B-A35B-Instruct"] == "unknown" + + +def test_auto_pool_excludes_unknown_cost_and_remains_provider_diverse() -> None: + """A review route must be free or carry a complete published price vector.""" + result = policy.build_zdr_prioritized_catalog( + policy.parse_discovery_report(_live_discovery_report()), + limit=6, + family_cap=2, + zdr_endpoints=PRICED_ZDR_FEED, + pool="auto", + ) + + agents = result["agents"] + assert agents[0]["model"] == FREE_MODEL + assert "cost:free" in agents[0]["tags"] + assert "cost:priced" in next( + agent for agent in agents if agent["model"] == PRICED_MODEL + )["tags"] + + assert all("cost:unknown" not in agent["tags"] for agent in agents) + assert result["report"]["total_unknown_routes"] == 4 + assert result["report"]["unknown_selected_count"] == 0 + + +def test_free_pool_never_admits_priced_or_unknown_routes() -> None: + """The existing orchestrator/free contract remains strictly zero-priced.""" + result = policy.build_zdr_prioritized_catalog( + policy.parse_discovery_report(_live_discovery_report()), + pool="free", + ) + + assert [agent["model"] for agent in result["agents"]] == [FREE_MODEL] + assert result["report"]["unknown_selected_count"] == 0 + assert result["report"]["priced_selected_count"] == 0 + + +def test_partial_price_vectors_still_fail_closed() -> None: + """One missing price component is malformed evidence, not an unknown vector.""" + report = _live_discovery_report() + openai = next(row for row in report["models"] if row["provider"] == "openai") + openai["completion_price_per_1k"] = 0.01 + + with pytest.raises(policy.PolicyError, match="lacks numeric prompt_price_per_1k"): + policy.parse_discovery_report(report) + + +@pytest.mark.parametrize( + ("prompt_price", "completion_price", "currency_code"), + [(None, 0.0, "USD"), (-1.0, 0.0, "USD"), (0.0, 0.0, None), (0.1, 0.0, "USD")], +) +def test_free_marker_rejects_malformed_or_conflicting_price_vectors( + prompt_price: object, completion_price: object, currency_code: object +) -> None: + """A free marker cannot bypass validation of provider-published prices.""" + report = _live_discovery_report() + free = next(row for row in report["models"] if row["is_free"] is True) + free["prompt_price_per_1k"] = prompt_price + free["completion_price_per_1k"] = completion_price + free["currency_code"] = currency_code + + with pytest.raises(policy.PolicyError): + policy.parse_discovery_report(report) + + +def test_legacy_normalized_rows_have_conservative_cost_fallbacks() -> None: + """Old normalized callers remain free-only or unknown, never inferred priced.""" + assert policy._cost_evidence({"is_free": True}) == "free" + assert policy._cost_evidence({"is_free": False}) == "unknown" diff --git a/tests/test_contextual_orchestrator_review_policy.py b/tests/test_contextual_orchestrator_review_policy.py index 12f4990daa..a2abc8c1ed 100644 --- a/tests/test_contextual_orchestrator_review_policy.py +++ b/tests/test_contextual_orchestrator_review_policy.py @@ -10,6 +10,11 @@ from scripts.ci import zdr_policy ZDR_FEED = frozenset({"openrouter/deepseek/deepseek-r1:free"}) +FREE_PRICE = { + "prompt_price_per_1k": 0.0, + "completion_price_per_1k": 0.0, + "currency_code": "USD", +} def _report() -> dict[str, object]: @@ -20,30 +25,35 @@ def _report() -> dict[str, object]: "model": "deepseek/deepseek-r1:free", "agent_id": "or_ds_r1", "is_free": True, + **FREE_PRICE, }, { "provider": "nvidia_nim", "model": "nvidia/nemotron-3-nano-30b-a3b", "agent_id": "nim_nano_free", "is_free": True, + **FREE_PRICE, }, { "provider": "nvidia_nim_sub", "model": "meta/llama-3.3-70b-instruct", "agent_id": "nimsec_70b", "is_free": True, + **FREE_PRICE, }, { "provider": "openai", "model": "gpt-4o-mini", "agent_id": "openai_gpt_4o_mini", "is_free": True, + **FREE_PRICE, }, { "provider": "bytez", "model": "qwen2.5-coder", "agent_id": "bytez_qwen25_coder", "is_free": True, + **FREE_PRICE, }, { "provider": "openai", @@ -88,6 +98,17 @@ def test_is_valid_is_free_rejects_non_scalar_markers() -> None: assert policy._is_valid_is_free(0) is True +def test_free_marker_without_a_price_vector_remains_unknown() -> None: + """A provider label cannot replace missing prompt/completion price evidence.""" + assert policy._normalize_cost_evidence( + route="openrouter/example", + is_free=True, + prompt_price=None, + completion_price=None, + currency_code=None, + ) == (policy.COST_UNKNOWN, None, None, None) + + def test_load_zdr_endpoints_skips_rows_without_provider_or_model(tmp_path) -> None: """Feed rows missing a provider or model cannot pollute the route keys.""" feed = tmp_path / "zdr.json" @@ -280,20 +301,21 @@ def test_build_catalog_applies_family_cap() -> None: """A family cap keeps one outage domain from absorbing the pool.""" report = { "models": [ - {"provider": "nvidia_nim", "model": f"m{i}", "agent_id": f"nim_a{i}", "is_free": True} + {"provider": "nvidia_nim", "model": f"m{i}", "agent_id": f"nim_a{i}", "is_free": True, **FREE_PRICE} for i in range(6) ] + [ { "provider": "nvidia_nim_sub", "model": f"s{i}", - "agent_id": f"nim_b{i}", - "is_free": True, + "agent_id": f"nim_b{i}", + "is_free": True, + **FREE_PRICE, } for i in range(6) ] + [ - {"provider": "openai", "model": f"o{i}", "agent_id": f"oa_{i}", "is_free": True} + {"provider": "openai", "model": f"o{i}", "agent_id": f"oa_{i}", "is_free": True, **FREE_PRICE} for i in range(3) ] } @@ -312,7 +334,7 @@ def test_build_catalog_respects_limit() -> None: """The catalog never exceeds the configured agent limit.""" report = { "models": [ - {"provider": "openai", "model": f"m{i}", "agent_id": f"oa_{i}", "is_free": True} + {"provider": "openai", "model": f"m{i}", "agent_id": f"oa_{i}", "is_free": True, **FREE_PRICE} for i in range(20) ] } diff --git a/tests/test_contextual_orchestrator_review_runtime_preflight.py b/tests/test_contextual_orchestrator_review_runtime_preflight.py index a61caeee0e..baa1a8df76 100644 --- a/tests/test_contextual_orchestrator_review_runtime_preflight.py +++ b/tests/test_contextual_orchestrator_review_runtime_preflight.py @@ -4,6 +4,7 @@ from contextlib import redirect_stdout import io +import json import runpy from pathlib import Path import sys @@ -120,6 +121,123 @@ def test_preflight_fails_closed_when_every_route_rejects() -> None: preflight([agent], client=client) +def test_preflight_uses_priced_fallback_only_after_primary_routes_reject() -> None: + """A live primary route wins; priced fallback is evidence-triggered only.""" + namespace = _load_launcher() + preflight = namespace["_preflight_with_fallback"] + primary = SimpleNamespace( + id="openrouter_free", provider_name="openrouter", model="free/model" + ) + fallback = SimpleNamespace( + id="openrouter_priced", provider_name="openrouter", model="priced/model" + ) + client = _ProbeClient( + {primary.id: TimeoutError("unavailable"), fallback.id: _openai_text("OK")} + ) + + viable, report, fallback_used = preflight( + [primary], [fallback], client=client + ) + + assert viable == [fallback] + assert fallback_used is True + assert report["fallback_reason"] == "primary_routes_unavailable" + assert report["primary_attempt"]["ready_count"] == 0 + assert [call[0] for call in client.calls] == [primary, fallback] + + ready_client = _ProbeClient( + {primary.id: _openai_text("OK"), fallback.id: _openai_text("unused")} + ) + viable, report, fallback_used = preflight( + [primary], [fallback], client=ready_client + ) + assert viable == [primary] + assert fallback_used is False + assert "fallback_reason" not in report + assert [call[0] for call in ready_client.calls] == [primary] + + failing_client = _ProbeClient( + {primary.id: TimeoutError("unavailable"), fallback.id: RuntimeError("rejected")} + ) + with pytest.raises(namespace["ReviewPreflightError"]) as failure: + preflight([primary], [fallback], client=failing_client) + assert failure.value.report["ready_count"] == 0 + assert failure.value.report["primary_attempt"]["ready_count"] == 0 + + +def test_preflight_stage_limits_share_one_startup_budget() -> None: + """Free-first and priced-fallback probes share one bounded route budget.""" + namespace = _load_launcher() + primary = namespace["_bounded_primary_catalog_limit"]( + 99, pool="auto", has_free_rows=True + ) + fallback = namespace["_bounded_fallback_catalog_limit"]( + 99, primary_count=primary + ) + assert (primary, fallback) == (8, 4) + assert primary + fallback == namespace["REVIEW_PREFLIGHT_MAX_TOTAL_ROUTES"] + + +def test_zdr_admission_selects_priced_tier_when_free_routes_are_not_private() -> None: + """Privacy admission precedes the free-first tier decision.""" + namespace = _load_launcher() + admit = namespace["_zdr_admitted_rows"] + rows = [ + {"provider": "openrouter", "model": "free/non-private"}, + {"provider": "openrouter", "model": "priced/private"}, + ] + + def checker(provider: str, *, model: str, zdr_endpoints: frozenset[str]) -> bool: + return f"{provider}:{model}" in zdr_endpoints + + admitted = admit( + rows, + require_zdr=True, + zdr_endpoints=frozenset({"openrouter:priced/private"}), + checker=checker, + ) + assert admitted == [rows[1]] + + +def test_discovery_counts_survive_stage_specific_policy_reports() -> None: + """Fallback selection preserves full discovery cost-tier evidence.""" + namespace = _load_launcher() + base = {"selected_count": 1, "selected": [{"model": "priced/model"}]} + rows = [ + {"cost_evidence": "free"}, + {"cost_evidence": "priced"}, + {"cost_evidence": "priced"}, + {"cost_evidence": "unknown"}, + ] + enriched = namespace["_with_discovery_counts"](base, rows) + assert base == {"selected_count": 1, "selected": [{"model": "priced/model"}]} + assert [enriched[key] for key in ( + "total_routes", "total_free_routes", "total_priced_routes", "total_unknown_routes" + )] == [4, 1, 2, 1] + + +def test_temporary_fallback_catalog_is_removed_after_loading(tmp_path: Path) -> None: + """The price-only handoff file is removed after success and failure.""" + helper = _load_launcher()["_load_temporary_agents"] + path = tmp_path / "review-catalog.json.priced" + agents = [{"id": "priced_route"}] + + def loader(value: str) -> list[object]: + assert json.loads(Path(value).read_text(encoding="utf-8")) == {"agents": agents} + return [SimpleNamespace(id="priced_route")] + + assert [agent.id for agent in helper(str(path), agents, loader=loader)] == ["priced_route"] + assert not path.exists() + + def failing_loader(value: str) -> list[object]: + assert Path(value).exists() + raise RuntimeError("loader rejected catalog") + + with pytest.raises(RuntimeError, match="loader rejected catalog"): + helper(str(path), agents, loader=failing_loader) + assert not path.exists() + + def test_preflight_transport_is_bounded_and_provider_neutral() -> None: """Sequential route probes must fit inside the sidecar startup budget.""" launcher = _LAUNCHER.read_text(encoding="utf-8") @@ -137,7 +255,7 @@ def test_sidecar_preserves_diagnostics_and_probes_the_real_gateway() -> None: launcher = _LAUNCHER.read_text(encoding="utf-8") sidecar = _SIDECAR.read_text(encoding="utf-8") - assert "_preflight_review_agents(agents, client=client)" in launcher + assert "_preflight_with_fallback(" in launcher assert "preflight-out" in launcher assert "max_output_tokens=REVIEW_MAX_OUTPUT_TOKENS" in launcher assert "temperature=REVIEW_TEMPERATURE" in launcher