fix(discovery): gate OpenRouter evidence_only/zdr_capable per model, not per provider

[seonghobae]

The bug (confirmed via a 5-agent investigation)

ProviderModelSource.evidence_only (default False) was hardcoded True for OpenRouter's provider source declaration only, at module load, before any HTTP call — applying to every discovered OpenRouter model unconditionally, regardless of that specific model's own evidence. OpenRouter was the only one of six provider sources with this override; every other provider (openai, opencode_zen, nvidia_nim, nvidia_nim_sub, bytez, configured_gateway) uses the False default.

That flag gated three downstream chokepoints, all firing regardless of the individual model's real evidence:

• zdr_capable=not model.evidence_only and matches(model.model_id) — forced False for every OpenRouter row even when it genuinely matched OpenRouter's own ZDR feed.
• result = [replace(model, evidence_only=source.evidence_only) for model in discovered] — stamped the provider-level flag onto every row.
• is_routable_discovered_model short-circuited before any capability check.
• agent_from_discovered hard-refused with ValueError("evidence-only model cannot become a serving agent").

The critical part: per-model ZDR evidence for OpenRouter already existed and was actively computed in this same file — _merge_openrouter_zdr_metadata/_merge_openrouter_provider_privacy fetch OpenRouter's authoritative /api/v1/endpoints/zdr feed and per-endpoint provider privacy policy, building a real per-model zdr_capable match. But that computed match was thrown away for OpenRouter's own rows — used exclusively to donate ZDR status to other providers' rows that happen to share the same canonical model id (e.g. an nvidia_nim row). OpenRouter never got to use its own evidence for its own serving eligibility.

This contradicts docs/planning/adrs/0032-model-group-cost-aware-discovery.md (accepted, still the standing design for this exact subsystem):

Privacy discovery is also model-specific rather than inferred from price... discovery does not turn either ambiguous state into blanket non-support.

The blanket evidence_only=True was added later (952996ec, a bare one-line commit message, no rationale, no accompanying ADR amendment) and never reconciled against ADR 0032. Notably, git show 952996ec~1 shows OpenRouter was not blanket-excluded before that commit — the regression flipped evidence_only from the False default to a hardcoded True with no documented reason.

What changed

contextual_orchestrator/model_discovery.py:

1. Removed the static evidence_only=True from the OpenRouter ProviderModelSource declaration — it now leaves it at the False default like every other provider.
2. _apply_discovered_model_evidence now computes evidence_only/zdr_capable per model for OpenRouter's own rows, reusing the exact same feed-match mechanism (zdr_model_ids, from OpenRouter's /api/v1/endpoints/zdr) already used to donate evidence to other providers:
   • A row becomes evidence_only=False/zdr_capable=True only when its own model id is present in the fetched ZDR feed.
   • Absence from the feed, an empty feed, or a total feed-fetch failure all leave the row evidence_only=True/zdr_capable=False — fail-closed, never a blanket default independent of that model's own feed coverage.
3. Updated stale comments (PROVIDER_MODEL_SOURCES declaration and discover_all_models) that asserted OpenRouter "is never selected as an inference upstream here."

tests/test_model_discovery.py:

• Fixed test_default_sources_request_openrouter_full_modality_catalog (source-level evidence_only is now False).
• Fixed the three zdr_capable expectation tests that were flipped in lockstep with 952996ec to assert the wrong blanket-exclusion behavior (test_discover_all_models_applies_model_zdr_evidence_to_other_sources, test_discover_all_models_does_not_match_a_shared_zdr_model_suffix, test_discover_all_models_rejects_an_ambiguous_zdr_model_suffix) — each now asserts OpenRouter's own exact-feed-match row correctly becomes zdr_capable=True.
• Added test_discover_all_models_openrouter_model_absent_from_zdr_feed_stays_evidence_only — a real negative case: an OpenRouter model with no attestation on the feed stays evidence_only=True/unroutable, confirming the fix doesn't turn every OpenRouter row into a serving agent.
• Added test_discover_all_models_openrouter_zdr_feed_failure_keeps_every_row_evidence_only — a total feed-fetch failure leaves every OpenRouter row evidence_only=True, confirming it never silently falls back to the new False default.

CHANGELOG.md and docs/product-technical-gap-baseline.md: new entries documenting the fix and reconciling the earlier (now-superseded) "OpenRouter is deliberately evidence_only=True... never serves inference" claims recorded in ADR 0041 and an earlier gap-baseline entry. ADR 0041's actual decision (no models_dev_provider_id join for OpenRouter, since it reports its own real pricing) is unaffected — only the incidental blanket-exclusion claim is superseded.

Verification

• tests/test_model_discovery.py: 81/81 passed.
• Related suites (test_model_discovery_boundaries.py, test_chat_model_capability_isolation.py, test_auto_discovery_server.py, test_provider_bootstrap.py, test_review_gateway.py, test_batch_embeddings.py, test_discovery_bootstrap_selection.py): 129/129 passed.
• Full suite (python -m pytest tests -q): 2825 passed, 2 failed. Both failures are pre-existing sandbox environment gaps unrelated to this change (test_privacy_policy_analysis.py::test_pinned_mcp_client_renders_and_closes_camoufox_tab — installed mcp package lacks Client attribute; test_psychometric_routing.py::test_fast_mlsirm_fit_uses_judge_acceptance_item_for_context_score — fast_mlsirm Rust extension not installed in this sandbox). Neither touches model_discovery.py or anything downstream of evidence_only/zdr_capable.
• interrogate on the changed file: 100% (20/20).
• Manually traced every downstream consumer of evidence_only/zdr_capable for OpenRouter (is_routable_discovered_model, agent_from_discovered, general_free_serving_candidates, select_cheapest_discovered_agent/select_top_n_cheapest_discovered_agents, refresh_price_book, privacy_tags_for_discovered, the CLI report in __main__.py) to confirm fail-closed is preserved: a model with no real ZDR evidence can never become servable through this change, only models with genuine per-model evidence gain a changed outcome.

Companion fix (not in this PR)

ContextualWisdomLab/.github's scripts/ci/contextual_orchestrator_review_launcher.py has a separate, independent blanket filter at _routable_discovered_models() (~line 150) that also strips every OpenRouter row before .github's own already-correct per-route ZDR mechanism (zdr_policy.py's is_zdr_model()/openrouter_endpoints_feed) ever gets to evaluate them. That is being handled separately in .github, not touched here.

---

🤖 Generated with Claude Code

https://claude.ai/code/session_015Gs7KmNvH75nxz1sL8mKjw

---

Generated by Claude Code

---

[coderabbitai]

Warning

Review limit reached

Next included review available in 12 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 4bd74017-211d-4a2a-bad4-2b9bdeb7332c

📥 Commits

Reviewing files that changed from the base of the PR and between c107e3e and 4c976bb.

📒 Files selected for processing (4)

• CHANGELOG.md
• contextual_orchestrator/model_discovery.py
• docs/product-technical-gap-baseline.md
• tests/test_model_discovery.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[devin-ai-integration]

Devin Review found 5 potential issues.

[seonghobae]

Closing as superseded by #949, which merged into main at 8cd99f1 moments ago.

#949 independently fixes the same root bug this PR addresses (removes the blanket evidence_only=True override on OpenRouter's ProviderModelSource, restoring per-model evidence-driven routing) and amends ADR 0032 to the same effect. It also ships a broader fix: a new spend_admitted field on DiscoveredModel, gated by apply_openrouter_spend_admission() / openrouter_paid_inference_available(), threaded through runtime auto-discovery, provider-catalog bootstrap, and durable catalog persistence. That's exactly the gap Devin flagged on this PR ("Unfunded OpenRouter models enter routing" — a paid ZDR-listed model bypassing openrouter_paid_inference_available and being routed to despite exhausted/unverifiable account credit).

Since #949 is a strict superset of this PR's scope and is already merged, closing this one without merging. Branch fix/openrouter-per-model-evidence is left in place.

---

Generated by Claude Code