Skip to content
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
262b4eb
fix: orchestrate structured provider features
seonghobae Aug 21, 2026
1f6b665
fix: bound local structured synthesis output
seonghobae Aug 21, 2026
16276d6
fix: preserve structured orchestration request contracts
seonghobae Aug 21, 2026
503e639
fix: preserve responses optional fields
seonghobae Aug 21, 2026
836a1a7
fix: keep structured judge calls bounded
seonghobae Aug 21, 2026
ee5cf34
fix: enforce spend budgets for structured provider calls
seonghobae Aug 21, 2026
79f2ab2
Merge remote-tracking branch 'origin/fix/auto-reasoning-effort-contra…
seonghobae Aug 21, 2026
e2a74a5
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae Aug 21, 2026
7cf488e
ci: verify and repair PR 805 runtime accounting
seonghobae Aug 21, 2026
a87af91
fix: close structured orchestration review gaps
seonghobae Aug 21, 2026
5c536de
chore: remove self-modifying review workflow
seonghobae Aug 21, 2026
e20d88d
test: stage PR 805 passthrough accounting repair
seonghobae Aug 21, 2026
207e25e
ci: execute the remaining PR 805 accounting repair
seonghobae Aug 21, 2026
2544b86
fix: account ordinary provider passthrough
seonghobae Aug 21, 2026
53f2aa4
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae Aug 21, 2026
3eb73f9
Merge remote-tracking branch 'origin/fix/auto-reasoning-effort-contra…
seonghobae Aug 21, 2026
daa508d
test: stage request-scoped sampling isolation repair
seonghobae Aug 21, 2026
db4e1eb
fix: preserve native responses orchestration lineage
seonghobae Aug 21, 2026
e494e4b
ci: harden the staged sampling repair runner
seonghobae Aug 21, 2026
85f9baf
ci: execute request-scoped sampling isolation repair
seonghobae Aug 21, 2026
d4c11ad
fix: isolate request sampling settings
seonghobae Aug 21, 2026
02e306d
ci: keep tests workflow read-only
seonghobae Aug 21, 2026
dbd7c3d
Merge remote PR 805 fixes with native Responses orchestration
seonghobae Aug 21, 2026
4f79492
test: close structured orchestration quality gaps
seonghobae Aug 21, 2026
f1d61d6
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae Aug 21, 2026
51b22d8
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae Aug 21, 2026
afe102c
test: stage Responses multimodal synthesis repair
seonghobae Aug 21, 2026
41680a1
ci: verify Responses multimodal synthesis repair
seonghobae Aug 21, 2026
387ee19
fix: preserve Responses multimodal synthesis
seonghobae Aug 21, 2026
8a30eb3
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae Aug 21, 2026
2da526d
ci: remove completed Responses repair workflow
seonghobae Aug 21, 2026
d01311e
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae Aug 21, 2026
3a60530
Merge gateway control stack with native Responses orchestration
seonghobae Aug 21, 2026
87421f0
fix: account Responses passthrough usage
seonghobae Aug 21, 2026
1207d55
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae Aug 21, 2026
afd683c
refactor: preserve provider usage totals
seonghobae Aug 21, 2026
0fe5cd3
fix: preserve structured contracts on HTTP
seonghobae Aug 21, 2026
6589dfa
fix: retain the gateway model identity
seonghobae Aug 21, 2026
c90c1f6
fix: preserve local Responses contracts and usage
seonghobae Aug 21, 2026
1103487
fix: preserve local response schemas and usage accounting
seonghobae Aug 21, 2026
ad142aa
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae Aug 21, 2026
976dca7
fix: record provider-reported structured usage
seonghobae Aug 21, 2026
b2c71d1
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae Aug 21, 2026
6e915c2
fix: include in-flight workflow spend in final budget gate
seonghobae Aug 21, 2026
c203207
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae Aug 21, 2026
7a0c61b
fix: enforce structured verification and provider contracts
seonghobae Aug 21, 2026
714e1c5
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae Aug 21, 2026
aa8a5fe
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae Aug 21, 2026
e47242c
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae Aug 21, 2026
b86ddbe
fix: persist model judge spend
seonghobae Aug 21, 2026
ea37a1e
fix: contain provider errors in structured flows
seonghobae Aug 21, 2026
47ff9d9
fix: preserve Responses text format precedence
seonghobae Aug 21, 2026
a47a0c8
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae Aug 21, 2026
d0a2aa4
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae Aug 21, 2026
5764fb9
fix: preserve Responses metadata for providers
seonghobae Aug 21, 2026
39c3c4d
fix: emit native Responses usage fields
seonghobae Aug 21, 2026
82bdbb4
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae Aug 21, 2026
68bdda6
fix: attribute every metered workflow call
seonghobae Aug 21, 2026
9a3f32d
refactor: remove judge ledger string coupling
seonghobae Aug 21, 2026
1d11e7d
fix: bound raw workflow retention
seonghobae Aug 21, 2026
e975711
Merge remote-tracking branch 'origin/fix/auto-reasoning-effort-contra…
seonghobae Aug 21, 2026
3bd723c
fix: synchronize runtime analytics snapshots
seonghobae Aug 21, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
267 changes: 231 additions & 36 deletions contextual_orchestrator/orchestrator.py

Large diffs are not rendered by default.

132 changes: 93 additions & 39 deletions contextual_orchestrator/server.py
Comment thread
seonghobae marked this conversation as resolved.
Resolved
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
sse_stream_body,
)

# OpenAI request params forwarded verbatim to the provider on passthrough.
# OpenAI request params forwarded to the final provider after orchestration.
OPENAI_PASSTHROUGH_PARAM_KEYS = {
"temperature", "top_p", "max_tokens", "max_completion_tokens", "n", "stop",
"seed", "presence_penalty", "frequency_penalty", "logit_bias", "logprobs",
Expand All @@ -50,7 +50,7 @@
# Assistants-style tool_resources — named unsupported (not unknown_fields).
"tool_resources",
}
# Provider features the multi-agent verifier cannot merge -> single-agent passthrough.
# Provider features requiring structured finalization after the multi-agent workflow.
PASSTHROUGH_TRIGGER_KEYS = {"response_format", "tools", "tool_choice", "functions", "function_call"}
ALLOWED_CHAT_KEYS = {
"model", "messages", "orchestration", "orchestration_mode", "mode",
Expand All @@ -66,7 +66,7 @@
"max_output_tokens",
# Tool-loop budget — accepted only for explicit unsupported error (no multi-step tool loop).
"max_tool_calls",
# Gateway cost/routing control plane (stripped before provider passthrough).
# Gateway cost/routing control plane (stripped before final provider transport).
"attribution", "routing",
# previous_response_id / conversation / truncation / include fail closed
# with named unsupported errors. Official text.format is validated
Expand Down Expand Up @@ -5071,25 +5071,52 @@ def do_POST(self) -> None: # noqa: N802
presence_penalty = sampling["presence_penalty"]
frequency_penalty = sampling["frequency_penalty"]
# Explicit JSON null on trigger keys is omit-equivalent (SDK optional
# defaults) — do not force single-agent passthrough for null-only keys.
# defaults) — do not force a structured workflow for null-only keys.
if any(
key in body and body.get(key) is not None
for key in PASSTHROUGH_TRIGGER_KEYS
):
# response_format / tools cannot be merged across agents;
# proxy the full request to one agent and return it verbatim.
# response_format / tools are finalized after the conducted
# workflow so the provider response shape remains intact.
started_at = time.perf_counter()
proxied = self._run(
lambda: orchestrator.proxy_completion(body, endpoint="chat/completions")
model_client = orchestrator.client
previous_max_tokens = model_client.max_output_tokens
previous_temperature = model_client.default_temperature
previous_top_p = model_client.default_top_p
previous_presence = model_client.default_presence_penalty
previous_frequency = model_client.default_frequency_penalty
if max_tokens is not None:
model_client.max_output_tokens = max_tokens
if temperature is not None:
model_client.default_temperature = temperature
if top_p is not None:
model_client.default_top_p = top_p
if presence_penalty is not None:
model_client.default_presence_penalty = presence_penalty
if frequency_penalty is not None:
model_client.default_frequency_penalty = frequency_penalty
try:
proxied = self._run(
lambda: orchestrator.proxy_completion(body, endpoint="chat/completions")
)
finally:
model_client.max_output_tokens = previous_max_tokens
model_client.default_temperature = previous_temperature
model_client.default_top_p = previous_top_p
model_client.default_presence_penalty = previous_presence
model_client.default_frequency_penalty = previous_frequency
Comment thread
seonghobae marked this conversation as resolved.
Resolved
event_detail = {
"endpoint_path": "/v1/chat/completions",
"actor_scope": "inference",
"status_code": 200,
"duration_ms": round((time.perf_counter() - started_at) * 1000, 2),
}
orchestrator.record_analytics_event(
"chat_completion_orchestrated_provider", event_detail
)
orchestrator.record_analytics_event(
"chat_completion_passthrough",
{
"endpoint_path": "/v1/chat/completions",
"actor_scope": "inference",
"status_code": 200,
"duration_ms": round((time.perf_counter() - started_at) * 1000, 2),
},
{**event_detail, "event_alias_for": "chat_completion_orchestrated_provider"},
)
Comment thread
seonghobae marked this conversation as resolved.
Resolved
self._send(proxied)
return
Expand Down Expand Up @@ -5386,8 +5413,8 @@ def do_POST(self) -> None: # noqa: N802
self._send(_response_payload(retrieved, include_trace=True))
return
if path == "/v1/responses":
# The Responses API has no chat-completions verifier equivalent,
# so every request is proxied to one agent verbatim.
# Every Responses request enters the conduct-based multi-agent
# workflow and is shaped by the selected synthesizer.
_reject_unknown_keys(body, ALLOWED_RESPONSES_KEYS)
# Fail-closed shape checks before passthrough so buyers never
# get a 200 after shipping invalid OpenAI-shaped metadata/input.
Expand All @@ -5403,14 +5430,10 @@ def do_POST(self) -> None: # noqa: N802
if "stream_options" in body:
_validate_responses_stream_options(body)
# Sampling knobs: type/range fail-closed before provider passthrough.
if "temperature" in body:
_validate_completions_temperature(body)
if "top_p" in body:
_validate_completions_top_p(body)
if "presence_penalty" in body:
_validate_completions_presence_penalty(body)
if "frequency_penalty" in body:
_validate_completions_frequency_penalty(body)
responses_temperature = _validate_completions_temperature(body)
responses_top_p = _validate_completions_top_p(body)
responses_presence_penalty = _validate_completions_presence_penalty(body)
responses_frequency_penalty = _validate_completions_frequency_penalty(body)
if "n" in body:
_validate_responses_n(body)
if "seed" in body:
Expand All @@ -5421,12 +5444,9 @@ def do_POST(self) -> None: # noqa: N802
_validate_responses_logit_bias(body)
if "logprobs" in body or "top_logprobs" in body:
_validate_responses_logprobs(body)
if "max_tokens" in body:
_validate_completions_max_tokens(body)
if "max_completion_tokens" in body:
_validate_chat_max_completion_tokens(body)
if "max_output_tokens" in body:
_validate_responses_max_output_tokens(body)
responses_max_tokens = _validate_completions_max_tokens(body)
responses_max_completion_tokens = _validate_chat_max_completion_tokens(body)
responses_max_output_tokens = _validate_responses_max_output_tokens(body)
Comment thread
seonghobae marked this conversation as resolved.
Resolved
Comment thread
seonghobae marked this conversation as resolved.
Resolved
if "max_tool_calls" in body:
_validate_responses_max_tool_calls(body)
_validate_openai_sdk_control_fields(body, endpoint_path="/v1/responses")
Expand Down Expand Up @@ -5533,7 +5553,7 @@ def do_POST(self) -> None: # noqa: N802
"input must be a non-empty string or non-empty array on /v1/responses",
)
# stream=false / omit → non-SSE JSON response (honest no-stream path).
# stream=true is not implemented for Responses passthrough.
# stream=true is not implemented for Responses finalization.
# String/0-1 forms coerce via shared bool helper (parity with chat).
if "stream" in body:
stream = _coerce_optional_bool(
Expand All @@ -5548,17 +5568,51 @@ def do_POST(self) -> None: # noqa: N802
"stream is not supported on /v1/responses",
)
started_at = time.perf_counter()
proxied = self._run(
lambda: orchestrator.proxy_completion(body, endpoint="responses")
model_client = orchestrator.client
previous_max_tokens = model_client.max_output_tokens
previous_temperature = model_client.default_temperature
previous_top_p = model_client.default_top_p
previous_presence = model_client.default_presence_penalty
previous_frequency = model_client.default_frequency_penalty
response_max_tokens = (
responses_max_tokens
if responses_max_tokens is not None
else responses_max_completion_tokens
if responses_max_completion_tokens is not None
else responses_max_output_tokens
)
Comment thread
seonghobae marked this conversation as resolved.
Resolved
if response_max_tokens is not None:
model_client.max_output_tokens = response_max_tokens
if responses_temperature is not None:
model_client.default_temperature = responses_temperature
if responses_top_p is not None:
model_client.default_top_p = responses_top_p
if responses_presence_penalty is not None:
model_client.default_presence_penalty = responses_presence_penalty
if responses_frequency_penalty is not None:
model_client.default_frequency_penalty = responses_frequency_penalty
try:
proxied = self._run(
lambda: orchestrator.proxy_completion(body, endpoint="responses")
)
finally:
model_client.max_output_tokens = previous_max_tokens
model_client.default_temperature = previous_temperature
model_client.default_top_p = previous_top_p
model_client.default_presence_penalty = previous_presence
model_client.default_frequency_penalty = previous_frequency
event_detail = {
"endpoint_path": "/v1/responses",
"actor_scope": "inference",
"status_code": 200,
"duration_ms": round((time.perf_counter() - started_at) * 1000, 2),
}
orchestrator.record_analytics_event(
"responses_orchestrated_provider", event_detail
)
orchestrator.record_analytics_event(
"responses_passthrough",
{
"endpoint_path": "/v1/responses",
"actor_scope": "inference",
"status_code": 200,
"duration_ms": round((time.perf_counter() - started_at) * 1000, 2),
},
{**event_detail, "event_alias_for": "responses_orchestrated_provider"},
)
if body.get("stream") is True:
self._send_sse(responses_sse_body(proxied))
Expand Down
8 changes: 8 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,14 @@ The useful split is quality-latency, not separate products:
- Low-latency routing: select one worker for the current query or turn.
- Deep orchestration: create a multi-step workflow when the task needs decomposition, independent attempts, verification, or synthesis.

Structured provider features do not create a third single-agent path. A
non-null `response_format`, tools/function declaration, or Responses request
enters the conducted workflow and reaches one final provider only after the
Thinker/Worker/Verifier/Synthesizer evidence has been assembled. The final
provider call preserves the validated wire feature; it is a transport boundary,
not a bypass of orchestration. JSON object and JSON schema are both covered by
[ADR 0011](planning/adrs/0011-structured-provider-features-stay-orchestrated.md).

TRINITY contributes the compact coordinator idea: a small model representation plus a lightweight head can choose agent and role over multiple turns. Its Thinker, Worker, and Verifier contracts are practical enough to implement directly.

Conductor contributes the workflow representation: each step is a natural-language subtask, an assigned worker, and an access list of prior step outputs. This is the key piece for preventing every worker from being dragged into the same transcript while still allowing deliberate collaboration.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
---
id: "0011"
title: "Keep structured provider features inside multi-agent orchestration"
status: accepted
proposed_date: "2026-08-21"
accepted_date: "2026-08-21"
deciders:
- "repository maintainer"
consulted:
- "LineageWeave integration"
- "contextual-orchestrator runtime"
informed:
- "API consumers"
affected_components:
- "contextual_orchestrator/orchestrator.py"
- "contextual_orchestrator/server.py"
- "tests/test_openai_passthrough.py"
effort: M
supersedes: null
superseded-by: null
related:
- path: "docs/planning/adrs/0001-fail-closed-model-judgment.md"
relation: constrains
- path: "docs/planning/adrs/0002-explicit-local-mlx-evaluation.md"
relation: extends
- path: "docs/architecture.md"
relation: implements
success_criteria:
- metric: "structured requests using the multi-agent workflow"
target: "100% of non-null response_format, tools, tool_choice, functions, function_call, and Responses requests"
measurement_window: "every structured-output regression run"
source: "tests/test_openai_passthrough.py"
- metric: "Responses json_schema translation"
target: "Responses text.format json_schema reaches the final provider as the equivalent Chat response_format without losing the schema"
measurement_window: "every Responses structured-output regression run"
source: "tests/test_openai_passthrough.py"
- metric: "multimodal context preservation"
target: "image_url input remains available to the final synthesis request"
measurement_window: "every multimodal structured-output regression run"
source: "tests/test_openai_passthrough.py"
---

# Keep structured provider features inside multi-agent orchestration

## Context

The OpenAI-compatible boundary previously treated `response_format`, tools, and
the Responses API as a provider passthrough. That made a request look
successful while skipping the Thinker/Worker/Verifier/Synthesizer workflow.
Structured output and multimodal requests are still product work, not an
exception to the orchestration contract. A consumer must receive the same
workflow evidence, verification boundary, session lineage, and cost accounting
as a plain chat request.

## Decision

1. A non-null structured provider feature is an orchestration trigger, never a
silent single-agent downgrade.
2. The request enters the existing conducted workflow. Intermediate steps use
the original messages, including multimodal content, and the final
synthesizer performs the provider-facing structured completion.
3. The final provider payload preserves validated tools and structured-output
fields. Responses `text.format` is translated to the equivalent Chat
`response_format` for the internal provider call, then the result is mapped
back to the Responses shape.
4. `json_object` and `json_schema` are both first-class structured workflows.
Schema validation remains fail-closed at the HTTP boundary; a provider
success is not treated as semantic schema validity.
5. The workflow response exposes bounded orchestration metadata by default.
Prompts, answers, images, tool arguments, secrets, and unbounded raw traces
are not put into telemetry or the default response.
6. Tool execution loops are not fabricated by this decision. Provider tools are
preserved for the final synthesized call; an actual execute-observe-replan
loop requires a separate ADR and contract.

## Research basis

This decision applies the existing research-grounded architecture rather than
inventing a provider-specific exception:

- Fugu distinguishes a low-latency routed call from a quality-oriented deep
workflow and keeps the worker pool configurable.
- TRINITY supplies the Thinker, Worker, and Verifier role boundary used before
synthesis.
- Conductor supplies explicit workflow steps and access-controlled context.

The canonical references are maintained in `docs/architecture.md` and
`docs/papers/README.md`. OpenAI's Chat Completions and Responses API contracts
define the wire-shape translation, not the orchestration policy.

## Consequences

* Good: structured and multimodal requests no longer bypass verification and
orchestration evidence.
* Good: JSON object and JSON schema requests share one tested policy instead of
diverging into transport-specific single-agent paths.
* Good: the final provider retains the capability fields it must interpret.
* Bad: structured requests consume more provider calls and can take longer than
a plain routed request.
* Bad: tool execution remains a follow-up capability rather than being implied
by merely forwarding a tool declaration.

## Confirmation

Run the focused structured-output and orchestration tests. Confirm that the
Responses JSON-schema test preserves the schema, the tools test reports the
conducted workflow, and the multimodal test retains `image_url` through final
synthesis. Do not claim provider-side semantic schema validity from HTTP 200.

## References

Nielsen, S., Cetin, E., Schwendeman, P., Sun, Q., Xu, J., & Tang, Y. (2025).
*Learning to orchestrate agents in natural language with the Conductor*.
https://doi.org/10.48550/arXiv.2512.04388

Sakana AI. (2026, June 22). *Sakana Fugu: One model to command them all*.
https://sakana.ai/fugu-release/

Xu, J., Sun, Q., Schwendeman, P., Nielsen, S., Cetin, E., & Tang, Y. (2025).
*Trinity: An evolved LLM coordinator*. https://doi.org/10.48550/arXiv.2512.04695

OpenAI. (n.d.-a). *Create chat completion*. OpenAI Platform.
https://platform.openai.com/docs/api-reference/chat/create

OpenAI. (n.d.-b). *Create a model response*. OpenAI Platform.
https://platform.openai.com/docs/api-reference/responses/create
Loading
Loading