feat: route model groups across modalities - #837
Merged
seonghobae merged 2 commits intoAug 25, 2026
Merged
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Comment on lines
+189
to
+191
| capabilities = tuple( | ||
| dict.fromkeys(_CAPABILITY_NAMES.get(value, value) for value in (*source.capabilities, *inputs, *outputs)) | ||
| ) |
Contributor
There was a problem hiding this comment.
🔴 Input-only vision models treated as image generators
The capabilities tuple merges input modalities alongside outputs, so a vision model that only accepts image input gets the plain image capability tag. Selection matches on that tag, so /v1/images/generations requests can route to models that cannot generate images.
Prompt for agents
In _parse_openai_compatible (contextual_orchestrator/model_discovery.py), the capabilities tuple is derived from (*source.capabilities, *inputs, *outputs), which folds provider-declared INPUT modalities into the generation capability set. Because agent_from_discovered copies these capabilities into the agent's plain tags, and _capability_agents in orchestrator.py selects members with `capability in agent.tags`, an image-input (vision) model ends up matching the 'image' generation capability. ADR 0026 explicitly promises that provider-declared direction is retained so 'an input-capable vision model is not mistaken for an image generator'. The direction-tagged form (input:<modality>/output:<modality>) is preserved, but capability matching ignores it. Decide the correct semantics for each capability (image/video/speech should match on OUTPUT modality; transcription/audio-in cases may need input) and make capability derivation and/or the _capability_agents selection direction-aware so input-only modalities do not grant a generation capability.
Was this helpful? React with 👍 or 👎 to provide feedback.
Comment on lines
+5039
to
+5052
| if path in capability_routes: | ||
| _validate_capability_request(path, body) | ||
| capability, endpoint, binary = capability_routes[path] | ||
| result = self._run( | ||
| lambda: orchestrator.proxy_capability( | ||
| body, capability=capability, endpoint=endpoint, binary=binary | ||
| ) | ||
| ) | ||
| if binary: | ||
| raw, content_type = result | ||
| self._send_bytes(raw, content_type) | ||
| else: | ||
| self._send(result) | ||
| return |
Contributor
There was a problem hiding this comment.
🟡 Unavailable capability returns 500 not 503
When no enabled member supports the requested capability, proxy_capability raises RuntimeError, which this dispatch lets fall through to the generic handler as a 500. The API contract documents 503 for these endpoints, and the embeddings path maps the same failure to 503.
Prompt for agents
The capability route dispatch in server.py do_POST calls orchestrator.proxy_capability, which (via _capability_agents) raises RuntimeError when no enabled member supports the capability or group, and also raises RuntimeError when all candidates fail. These propagate to do_POST's generic `except Exception` handler and produce a 500 internal_error. api_contract.py declares 503 'No capable model group member is available' for these endpoints, and _validate_embeddings_model already maps the same select_capability_agent failure to a 503. Wrap the proxy_capability call (or catch RuntimeError from _capability_agents) and raise RequestError(503, ...) so the documented status is returned when no capable member exists.
Was this helpful? React with 👍 or 👎 to provide feedback.
Comment on lines
+2806
to
+2810
| requested_group = ( | ||
| canonical_group_name(model_name) | ||
| if model_name is not None and model_name not in exact_models | ||
| else None | ||
| ) |
Contributor
There was a problem hiding this comment.
📝 Info: Unknown single-word model gives confusing 400
_capability_agents calls canonical_group_name(model_name) eagerly for any model not served exactly. A single-token unknown model makes it raise ValueError, surfaced to the client as a group-naming message rather than a clear 'model not found'. _require_pool_model guards this call in try/except; this path does not.
Was this helpful? React with 👍 or 👎 to provide feedback.
Comment on lines
+1215
to
+1232
| def proxy_send_bytes( | ||
| self, agent: ModelAgent, endpoint: str, payload: dict[str, Any] | ||
| ) -> tuple[bytes, str]: | ||
| """Passthrough a provider response whose body is binary media.""" | ||
| if agent.base_url.startswith("mock://"): | ||
| return b"mock audio", "audio/mpeg" | ||
| api_key = _provider_credential(agent) # pragma: no cover | ||
| headers = {"content-type": "application/json"} # pragma: no cover | ||
| if api_key: # pragma: no cover | ||
| headers["authorization"] = f"{agent.auth_scheme} {api_key}" | ||
| request = urllib.request.Request( # pragma: no cover | ||
| self._provider_url(agent, f"/{endpoint.lstrip('/')}"), | ||
| data=json.dumps(payload).encode("utf-8"), | ||
| headers=headers, | ||
| method="POST", | ||
| ) | ||
| with self._open_provider(request, self._validate_provider(agent)) as response: # pragma: no cover | ||
| return response.read(), response.headers.get_content_type() |
Contributor
There was a problem hiding this comment.
📝 Info: Binary media path has no transient retry
proxy_send_bytes issues one provider request with no retry, unlike proxy_send which uses _send_raw_with_retry. Binary endpoints get no per-request transient retry; only cross-member failover in proxy_capability remains.
Was this helpful? React with 👍 or 👎 to provide feedback.
…overy' into feat/multimodal-model-group-orchestration
seonghobae
merged commit Aug 25, 2026
b353e6a
into
feat/model-group-cost-aware-discovery
1 of 2 checks passed
Comment on lines
+187
to
+188
| inputs = tuple(value for value in architecture.get("input_modalities", ()) if isinstance(value, str)) | ||
| outputs = tuple(value for value in architecture.get("output_modalities", ()) if isinstance(value, str)) |
Contributor
There was a problem hiding this comment.
🔴 Null modality field crashes entire model discovery
A provider row whose architecture.input_modalities or output_modalities is JSON null makes the generator iterate None and raise TypeError (the () default applies only when the key is absent). discover_provider_models catches only URLError/TimeoutError/ValueError and discover_all_models catches only ProviderDiscoveryError, so one malformed row aborts discovery for every provider.
Suggested change
| inputs = tuple(value for value in architecture.get("input_modalities", ()) if isinstance(value, str)) | |
| outputs = tuple(value for value in architecture.get("output_modalities", ()) if isinstance(value, str)) | |
| raw_inputs = architecture.get("input_modalities") | |
| inputs = tuple(v for v in raw_inputs if isinstance(v, str)) if isinstance(raw_inputs, list) else () | |
| raw_outputs = architecture.get("output_modalities") | |
| outputs = tuple(v for v in raw_outputs if isinstance(v, str)) if isinstance(raw_outputs, list) else () |
Was this helpful? React with 👍 or 👎 to provide feedback.
seonghobae
added a commit
that referenced
this pull request
Aug 26, 2026
* feat(routing): add operator-managed model groups * fix(api): enforce model group create semantics * fix(routing): preserve eligibility and REST contracts * docs(groups): remove ephemeral model example * fix(groups): route advertised aliases end to end * ci: hourly OpenCode maintenance agent routed through this gateway (#835) * ci: add hourly OpenCode maintenance agent routed through this gateway The scheduled job boots the contextual-orchestrator gateway with the five org provider secrets seeded into the process-local KV registry (bootstrap transport only), auto-discovers chat-capable models, assigns the ox-alpha measured-routing group across OpenRouter + OpenCode Zen aliases, then runs the pinned OpenCode CLI pointed at http://127.0.0.1:8000/v1 with model 'ox-alpha' so the agent's own traffic exercises group routing. The agent works the PR queue (review -> fix -> recheck -> merge), root-causes failing checks, and advances docs/product-technical-gap-baseline.md when the PR queue is empty. COPILOT_GITHUB_TOKEN is not used; the existing review-agent key scheme is untouched. * fix(ci): remove ephemeral model binding * fix(ci): install gateway and grant branch writes * fix(groups): preserve internal dispatch default * fix: isolate model group routing evidence * fix: preserve conduct semantics for model groups * ci: minimize hourly loop permissions and installs * fix: measure streamed group routing * feat: route model groups across modalities (#837) * feat: normalize and edit model groups (#838) * fix: preserve capability routing contracts * release: v0.2.0 — model groups, cost-aware discovery, changelog baseline Bump 0.1.0 -> 0.2.0 and add the canonical Keep-a-Changelog file with the 0.1.0 baseline and the 0.2.0 additions (operator-managed model groups, measured group routing, OpenCode Zen discovery + free-tier classification, Strix B105 root-cause remediation). * fix(discovery): tolerate null modality arrays * docs: specify model group product and technical contracts * release: align v0.2.0 changelog and lock metadata * feat: stream orchestrated reasoning summaries * fix: keep free reasoning streams fail closed * fix: preserve free routing evidence and analytics * fix: distinguish failed Responses streams in analytics * fix: pin structured free judge to selected agent * fix: lock container dependencies and virtual capabilities * fix: keep free passthrough on zero-cost models * fix: prune removed routing measurements * refactor: remove unreachable responses passthrough branch * fix: route virtual models across media capabilities * fix: retain reset candidate routing rows * fix: preserve Responses instructions in workflows * test: align model-group missing-member code with canonical agent_not_found #831 unified worker-agent not-found errors on agent_not_found; the model-group CRUD contract now asserts that same canonical code. * fix(api): model-group creation returns canonical agent_not_found for unknown members * ci: route hourly OpenCode loop through auto * fix: reject unsupported orchestrated structured output * docs: correct Responses stream options error * fix: retrieve URL-encoded model identifiers * fix: measure free capability and failover routing * fix: resolve model group review findings * fix: close model group integration gaps * fix: align provider inventory and session cache scope * fix: harden compose secrets and repeated reasoning summaries * fix: contain binary response disconnects * fix: ground Zen free discovery in structured costs * fix: keep group judge within allowed members * fix: preserve free catalog evidence across reloads * docs: assign unique model-group ADR number * fix: preserve catalog capability evidence * fix: preserve declared Bytez endpoint capabilities * fix: validate batch model identity at ingress * fix(api): normalize missing model group errors * docs(prd): align product bets with model groups * fix(discovery): filter chat-only Bytez transports * docs(adr): reserve model-group decision identifier * feat: replace routing heuristics with measured evidence ledgers (#847) * feat: replace routing heuristics with measured evidence ledgers Remove DOMAIN_HINTS/COMPLEX_HINTS keyword tables; route via eligibility contracts, declaration priority/capability fit/cosine affinity over operator-declared metadata, and measured intra-group quality then EWMA tokens-per-second. Add structured fail-closed triage gas with content-hash verdict caching and real-time fast-mlsirm judging on direct routes that feeds a Beta-Bernoulli quality ledger with in-budget failover. ADR 0027 + doctoring APA 7 references (Jacobson 1988; Gelman et al. 2013; Karpukhin et al. 2020; Ong et al. 2024; Chen et al. 2023; Zheng et al. 2023; Jeon et al. 2021). Gap baseline added at docs/product-technical-gap-baseline.md. * fix: keep routing evidence units and capability boundaries honest * fix(admin): tolerate unavailable model-group state * docs(loop): require PRD and measured web capacity * fix(admin): remove retired policy hint metric * fix(routing): validate evidence before mutation * fix(discovery): remove model-name free inference guidance * fix: complete #834 model-group persistence on the normalized agent-pool schema - model_group/model_group_member relations compose with main's normalized agent_pool (no JSON shadow); save() maintains membership, load_all() restores group_name via join. - Legacy payload promotion reads agent_pool_legacy_payloads during the migration window and drops it after promoting group names. - DB-naming gate now extracts SQL from AST string constants so prose in comments can no longer produce false identifier violations. - Batch runner signature aligned (messages, mode, model) with the merged LocalBatchBackend; stream-route fake accepts the merged caller kwargs. * test: case-sensitive DDL pattern so prose cannot fake identifiers * docs: refresh protected-main gap evidence * docs: record exact provider regression suite * docs: correct exact provider head * fix(admin): make guidance customer-actionable * fix(admin): remove internal terms from customer copy * docs(ui): record customer-copy screenshot audit * test(ci): lock hourly orchestrator loop contract * fix: replace synthetic admin status with truthful empty states * docs: refresh exact-head remediation baseline * fix(stream): cancel orchestration after Responses disconnect * test: align boundary contracts with current routing * docs: complete public boundary docstrings * docs: refresh exact-head product gap queue * test: enforce complete public docstrings * ci: enforce protected exact-head merge loop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Verification
pytest -q tests/test_api_contract.py tests/test_model_discovery.py tests/test_model_group.py tests/test_multimodal_model_group_http.py tests/test_embeddings_model_pool_http_honesty.py(58 passed)Research and standards