Skip to content
Merged
Show file tree
Hide file tree
Changes from 90 commits
Commits
Show all changes
100 commits
Select commit Hold shift + click to select a range
7e3b65e
test: reproduce embedding deployment selected for chat synthesis
seonghobae Aug 20, 2026
817eaec
fix: isolate chat discovery from embedding endpoints
seonghobae Aug 20, 2026
c765ce4
test: cover chat capability isolation defenses
seonghobae Aug 20, 2026
6c2d263
docs: record embedding-to-chat incident boundary
seonghobae Aug 20, 2026
35c7bef
docs: normalize incident references to APA 7
seonghobae Aug 20, 2026
f01da9f
test: reproduce stale embedding agent runtime selection
seonghobae Aug 20, 2026
266561c
fix: enforce chat capability at runtime boundaries
seonghobae Aug 20, 2026
4b50126
refactor: share chat capability classifier
seonghobae Aug 20, 2026
bc004d0
fix: activate runtime chat capability guards
seonghobae Aug 20, 2026
8bf5db1
test: tighten runtime guard coverage surface
seonghobae Aug 20, 2026
d33d1d5
test: cover runtime guard installation idempotence
seonghobae Aug 20, 2026
7ae2d4d
docs: record stale-agent runtime containment
seonghobae Aug 20, 2026
db7b209
fix: integrate chat capability checks explicitly
seonghobae Aug 20, 2026
7739ca0
docs: normalize capability incident formatting
seonghobae Aug 20, 2026
3571869
ci: repair chat capability guards with source integration
seonghobae Aug 20, 2026
14571a5
ci: remove completed one-shot repair workflow
seonghobae Aug 20, 2026
c41aedb
Merge protected main into chat capability isolation
seonghobae Aug 20, 2026
480d6ef
test: reproduce embedding deployment selected for chat synthesis
seonghobae Aug 20, 2026
54b4403
fix: isolate chat discovery from embedding endpoints
seonghobae Aug 20, 2026
e557400
test: cover chat capability isolation defenses
seonghobae Aug 20, 2026
be2f191
docs: record embedding-to-chat incident boundary
seonghobae Aug 20, 2026
c1fc1ca
docs: normalize incident references to APA 7
seonghobae Aug 20, 2026
392ce6c
test: reproduce stale embedding agent runtime selection
seonghobae Aug 20, 2026
99f5c41
fix: enforce chat capability at runtime boundaries
seonghobae Aug 20, 2026
136426d
refactor: share chat capability classifier
seonghobae Aug 20, 2026
050b514
fix: activate runtime chat capability guards
seonghobae Aug 20, 2026
fa980bc
test: tighten runtime guard coverage surface
seonghobae Aug 20, 2026
3998ad1
test: cover runtime guard installation idempotence
seonghobae Aug 20, 2026
feeac9e
docs: record stale-agent runtime containment
seonghobae Aug 20, 2026
668cf86
fix: integrate chat capability checks explicitly
seonghobae Aug 20, 2026
64b4e70
docs: normalize capability incident formatting
seonghobae Aug 20, 2026
543cdc7
ci: verify final chat capability guards
seonghobae Aug 20, 2026
95a2649
fix: complete explicit chat capability guards
seonghobae Aug 20, 2026
92efe38
refactor: keep ModelClient internal to orchestrator module
seonghobae Aug 20, 2026
df20c19
test: separate chat transport from general synthesis roles
seonghobae Aug 20, 2026
8e86976
ci: verify transport and synthesis capability separation
seonghobae Aug 20, 2026
4fb286f
chore: remove superseded transport-role repair workflow
seonghobae Aug 20, 2026
ea2ca8d
test: remove out-of-scope transport-role experiment
seonghobae Aug 20, 2026
a30c390
test: define honest provider-diverse discovery bootstrap
seonghobae Aug 20, 2026
6578dc5
test: bind NIM keys to one provider family
seonghobae Aug 20, 2026
3f15912
test: reproduce chat transport and role boundary conflation
seonghobae Aug 20, 2026
1ca4015
fix: rank unknown prices honestly and diversify bootstrap providers
seonghobae Aug 20, 2026
33185ad
ci: verify chat transport and role boundaries safely
seonghobae Aug 20, 2026
a5a3840
chore: remove branch-mutating repair workflow
seonghobae Aug 20, 2026
cabb1b0
test: remove discarded transport-role experiment
seonghobae Aug 20, 2026
3e75b1e
test: reproduce transport and agent-role capability conflation
seonghobae Aug 20, 2026
104ba94
ci: verify transport and ordinary-agent capability boundaries
seonghobae Aug 20, 2026
9984ad2
chore: remove branch-mutating capability workflow
seonghobae Aug 20, 2026
0ec353d
fix: separate chat transport from synthesis roles
seonghobae Aug 20, 2026
67b4341
test: reproduce non-chat passthrough and batch bypasses
seonghobae Aug 20, 2026
06da0c0
fix(discovery): wire provider-diverse bootstrap into CLI
seonghobae Aug 20, 2026
50b879b
ci: verify passthrough and batch chat capability guards
seonghobae Aug 20, 2026
a806e39
fix: replace chat capability repair workflow with reviewed source
seonghobae Aug 20, 2026
a424c9d
test: preserve unknown model identifiers without capability fabrication
seonghobae Aug 20, 2026
d71762d
fix: avoid inferring guard roles from unrelated suffixes
seonghobae Aug 20, 2026
f03d226
Merge protected main into chat capability isolation
seonghobae Aug 20, 2026
6bd4e0c
ci: verify final chat capability contract
seonghobae Aug 20, 2026
290838c
chore: remove completed capability repair workflow
seonghobae Aug 20, 2026
a063113
test(discovery): expose ambiguous price and duplicate pool gaps
seonghobae Aug 20, 2026
2539ad5
fix(discovery): reject untrusted duplicate price evidence
seonghobae Aug 20, 2026
a78d3a6
fix(discovery): keep duplicate evidence comparison total
seonghobae Aug 20, 2026
75e15af
test(discovery): expose overflow, corrupt-row, and currency gaps
seonghobae Aug 20, 2026
183e7a8
fix(discovery): fail closed on malformed and incomparable price evidence
seonghobae Aug 20, 2026
c01733b
fix(cost): reject corrupt persisted price rows
seonghobae Aug 20, 2026
669d97f
ci: finalize chat transport and role capability boundaries
seonghobae Aug 20, 2026
c08a8a6
ci: finalize endpoint transport and agent role separation
seonghobae Aug 20, 2026
70757f6
chore: remove superseded capability repair workflow v2
seonghobae Aug 20, 2026
8c06c74
chore: remove superseded capability repair workflow v3
seonghobae Aug 20, 2026
8885ef2
ci: finalize chat capability boundaries on exact branch
seonghobae Aug 20, 2026
d1e06d5
fix: close chat capability review gaps
seonghobae Aug 20, 2026
22a3270
test: make unknown-identifier regression directly runnable
seonghobae Aug 20, 2026
60be2e4
test: make transport-role regression directly runnable
seonghobae Aug 20, 2026
e91a108
test: cover legacy completions and direct execution
seonghobae Aug 20, 2026
a4aef39
test: make chat-capability regression directly runnable
seonghobae Aug 20, 2026
f13ee66
docs: register chat capability regression checks
seonghobae Aug 20, 2026
88fee97
fix: exclude shieldgemma from general chat roles
seonghobae Aug 20, 2026
0777e14
test(cost): reject persisted partial price rows
seonghobae Aug 20, 2026
386c9a0
fix: reject incomplete persisted price rows
seonghobae Aug 20, 2026
5d989c6
feat: add contextual review gateway bootstrap
seonghobae Aug 20, 2026
04761e4
fix: preserve priced discovery and review routing
seonghobae Aug 20, 2026
3a0e9f4
fix: isolate review gateway chat candidates
seonghobae Aug 20, 2026
84a8999
fix: keep partial discovery prices unknown
seonghobae Aug 20, 2026
9041e78
fix: resolve review gateway auth through KV
seonghobae Aug 20, 2026
0071751
fix: keep auth outside provider bootstrap gate
seonghobae Aug 20, 2026
6c10b49
Merge embedding capability isolation into discovery stack
seonghobae Aug 20, 2026
6b603ef
docs: ground provider-diverse routing in research
seonghobae Aug 20, 2026
172a890
test: cover model discovery rejection paths
seonghobae Aug 21, 2026
bfdcdd0
fix: reject gte embedding models from chat roles
seonghobae Aug 21, 2026
d3a27db
fix: keep foreign currency prices out of ranking
seonghobae Aug 21, 2026
8d31fa5
merge: preserve provider-diverse gateway discovery
seonghobae Aug 21, 2026
17a5e42
Merge main into contextual review gateway
seonghobae Aug 24, 2026
e2ac971
Merge remote-tracking branch 'origin/main' into local-790
seonghobae Aug 25, 2026
9f848a8
Merge remote-tracking branch 'origin/main' into feat-opencode-review-…
seonghobae Aug 25, 2026
c649cdb
fix: preserve non-chat capability routing and price integrity
seonghobae Aug 25, 2026
b826099
Merge remote-tracking branch 'origin/main' into codex/pr790-review
seonghobae Aug 25, 2026
1c3b52a
Merge branch 'feat/opencode-review-gateway' of https://github.com/Con…
seonghobae Aug 25, 2026
ee216d0
fix: keep one canonical price parser
seonghobae Aug 25, 2026
26c8f31
Merge remote-tracking branch 'origin/main' into codex/pr790-review
seonghobae Aug 25, 2026
13bc939
docs: list chat capability check
seonghobae Aug 25, 2026
6acd0fb
fix: validate review gateway agent limit at CLI
seonghobae Aug 25, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,6 @@ Run an evaluation against that server with `--temperature 0` for repeatable judg
Model-based conduct verification requires `fast-mlsirm` in the same runtime and fails closed when it is absent or broken; fast-mlsirm sends its judge completion through this contextual-orchestrator gateway, so no direct provider fallback is used. “Same runtime” means that the exact interpreter used for the live run can import both packages: install both checkouts into one environment (prefer editable installs), or expose both source roots with `PYTHONPATH` during a source run. Before a live judge benchmark, run `python -m contextual_orchestrator check-fast-mlsirm` with that exact interpreter. It prints the interpreter, package version, transitive-import status, and contextual contract check, and exits nonzero on a missing dependency or contract mismatch. Do not run the preflight in one virtual environment and the judge in another. See [ADR 0001](docs/planning/adrs/0001-fail-closed-model-judgment.md).

The agent pool is manageable at runtime: `POST`/`PATCH`/`DELETE` on `/api/v1/agent_pools/default/worker_agents[/{id}]` add, govern, and remove model-group members. Pass `--agents-db PATH` (or `CONTEXTUAL_ORCHESTRATOR_AGENTS_DB`) to persist those changes to a stdlib sqlite file — stored changes overlay the seed agents file at startup, and removals write disabled tombstones so they survive restarts; without it the pool is in-memory as before.

Beyond the local MLX/llama.cpp discovery above, `python -m contextual_orchestrator discover-models [--agents-db PATH]` discovers models from remote providers (OpenAI, OpenRouter, NVIDIA NIM ×2 keys, Bytez) for any subset of their KV-registered credentials, and can persist them into the same `--agents-db` sqlite file, added disabled by default. See [docs/kv-credentials.md](docs/kv-credentials.md#multi-provider-auto-discovery) for the credential-name table and cost-based auto-selection.

Seed the credential into the KV once at bootstrap:
Expand Down Expand Up @@ -304,6 +303,10 @@ python tests/test_admin_contract.py
python tests/test_conventions.py
python tests/test_api_contract.py
python tests/test_security_hardening.py
python tests/test_chat_model_capability_isolation.py
python tests/test_chat_transport_role_separation.py
python tests/test_chat_capability_unknown_identifiers.py
python tests/test_chat_passthrough_capability_isolation.py
python tests/test_repository_security_metadata.py
python tests/test_product_planning_contract.py
python tests/test_plugin_driven_artifacts.py
Expand Down
6 changes: 3 additions & 3 deletions contextual_orchestrator/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
agent_id_for,
discover_all_models,
refresh_price_book,
select_top_n_cheapest_discovered_agents,
select_bootstrap_discovered_agents,
)
from .orchestrator import (
CONTEXTUAL_ORCHESTRATOR_CONTRACT_V1,
Expand Down Expand Up @@ -211,7 +211,7 @@ def _discover_models_command(argv: list[str]) -> None:
type=_non_negative_int,
default=0,
metavar="N",
help="Enable the N cheapest discovered agents in --agents-db (auto-optimization bootstrap; "
help="Enable a price-honest, provider-diverse discovered agent pool in --agents-db (auto-optimization bootstrap; "
"requires --agents-db; 0 disables, the default, leaving every discovered agent inert).",
)
args = parser.parse_args(argv)
Expand All @@ -229,7 +229,7 @@ def _discover_models_command(argv: list[str]) -> None:
)
bootstrap.sync_discovered_agents([agent_from_discovered(model) for model in discovered])
if args.enable_cheapest:
for model in select_top_n_cheapest_discovered_agents(discovered, price_book, args.enable_cheapest):
for model in select_bootstrap_discovered_agents(discovered, price_book, args.enable_cheapest):
agent_id = agent_id_for(model)
bootstrap.patch_agent("default", agent_id, {"status": "active"})
enabled_agent_ids.append(agent_id)
Expand Down
96 changes: 96 additions & 0 deletions contextual_orchestrator/chat_capability.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
"""Classify chat transport compatibility and ordinary agent-role eligibility.

Provider catalogs mix endpoint-only models with models served through an
OpenAI-compatible chat transport. Transport compatibility is not the same as
fitness for an ordinary thinker, worker, verifier, or synthesizer role: audio
and policy-classification models can use chat transport, while embedding,
reranking, transcription, moderation-endpoint, image-generation, realtime, and
speech-only models cannot.
"""

from __future__ import annotations

import re

_MODEL_TOKEN_RE = re.compile(r"[a-z0-9]+")
_TRANSPORT_INCOMPATIBLE_EXACT_TOKENS = frozenset(
{
"bge",
"clip",
"dall",
"e5",
"embed",
"embedding",
"embeddings",
"gte",
"image",
"images",
"moderation",
"realtime",
"rerank",
"reranker",
"siglip",
"sora",
"speech",
"transcribe",
"transcription",
"tts",
"whisper",
}
)
_TRANSPORT_INCOMPATIBLE_PREFIXES = (
"embed",
"moderat",
"rerank",
"transcrib",
)
Comment thread
seonghobae marked this conversation as resolved.
Resolved


def is_chat_compatible_model_id(model_id: str) -> bool:
"""Return whether an identifier can use the ordinary chat transport.

The classifier rejects only identifiers that clearly advertise an endpoint
family incompatible with chat messages. Audio-capable and safety-classifier
models remain transport-compatible because providers serve some of them over
``/chat/completions``.
"""
tokens = _model_tokens(model_id)
return _is_transport_compatible_tokens(tokens)


def _is_transport_compatible_tokens(tokens: tuple[str, ...]) -> bool:
"""Judge transport compatibility from already-normalized model tokens."""
if not tokens:
return False
for token in tokens:
if token in _TRANSPORT_INCOMPATIBLE_EXACT_TOKENS:
return False
if token.startswith(_TRANSPORT_INCOMPATIBLE_PREFIXES):
return False
return True


def _model_tokens(model_id: str) -> tuple[str, ...]:
"""Normalize one provider-prefixed model identifier into lowercase tokens."""
if not isinstance(model_id, str):
return ()
return tuple(_MODEL_TOKEN_RE.findall(model_id.casefold()))


def is_general_chat_agent_model_id(model_id: str) -> bool:
"""Return whether a chat model may enter ordinary orchestration roles.

Explicit guard and safety models can use chat transport but are specialized
policy classifiers, not general answer synthesizers. This negative role gate
does not infer reasoning, coding, vision, or verification capabilities.
"""
tokens = _model_tokens(model_id)
if not tokens or not _is_transport_compatible_tokens(tokens):
return False
return not any(
token == "safety"
or token == "guard"
or token == "shieldgemma"
or token.startswith("nemoguard")
for token in tokens
)
24 changes: 22 additions & 2 deletions contextual_orchestrator/cost_ledger.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
import threading
from dataclasses import dataclass, field
from decimal import ROUND_HALF_UP, Decimal
import math
import time
from typing import Any, Dict, List, Optional, Protocol
import uuid
Expand Down Expand Up @@ -162,11 +163,30 @@ def get_price(self, provider: str, model: str) -> Optional[PriceEntry]:
raw = self._config.get(_PRICE_CATEGORY, _price_key(provider, "*"), None)
if raw is None:
return None
if not isinstance(raw, dict):
return None
try:
if (
"prompt_price_per_1k" not in raw
or "completion_price_per_1k" not in raw
):
return None
prompt_price = float(raw["prompt_price_per_1k"])
completion_price = float(raw["completion_price_per_1k"])
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
except (OverflowError, TypeError, ValueError):
return None
if (
not math.isfinite(prompt_price)
or not math.isfinite(completion_price)
or prompt_price < 0
or completion_price < 0
):
return None
Comment thread
seonghobae marked this conversation as resolved.
Resolved
return PriceEntry(
provider_name=raw.get("provider_name", provider),
model_name=raw.get("model_name", model),
prompt_price_per_1k=float(raw.get("prompt_price_per_1k", 0.0)),
completion_price_per_1k=float(raw.get("completion_price_per_1k", 0.0)),
prompt_price_per_1k=prompt_price,
completion_price_per_1k=completion_price,
Comment thread
seonghobae marked this conversation as resolved.
Resolved
currency_code=raw.get("currency_code", self.default_currency),
)

Expand Down
Loading
Loading