Repository navigation
fix: orchestrate structured provider features #805
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
seonghobae
merged 62 commits into
fix/auto-reasoning-effort-contract-rebased
from
fix/structured-output-orchestration
Aug 21, 2026
Merged
Changes from 8 commits
Commits
Show all changes
62 commits
Select commit
Hold shift + click to select a range
262b4eb
fix: orchestrate structured provider features
seonghobae 1f6b665
fix: bound local structured synthesis output
seonghobae 16276d6
fix: preserve structured orchestration request contracts
seonghobae 503e639
fix: preserve responses optional fields
seonghobae 836a1a7
fix: keep structured judge calls bounded
seonghobae ee5cf34
fix: enforce spend budgets for structured provider calls
seonghobae 79f2ab2
Merge remote-tracking branch 'origin/fix/auto-reasoning-effort-contra…
seonghobae e2a74a5
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae 7cf488e
ci: verify and repair PR 805 runtime accounting
seonghobae a87af91
fix: close structured orchestration review gaps
seonghobae 5c536de
chore: remove self-modifying review workflow
seonghobae e20d88d
test: stage PR 805 passthrough accounting repair
seonghobae 207e25e
ci: execute the remaining PR 805 accounting repair
seonghobae 2544b86
fix: account ordinary provider passthrough
seonghobae 53f2aa4
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae 3eb73f9
Merge remote-tracking branch 'origin/fix/auto-reasoning-effort-contra…
seonghobae daa508d
test: stage request-scoped sampling isolation repair
seonghobae db4e1eb
fix: preserve native responses orchestration lineage
seonghobae e494e4b
ci: harden the staged sampling repair runner
seonghobae 85f9baf
ci: execute request-scoped sampling isolation repair
seonghobae d4c11ad
fix: isolate request sampling settings
seonghobae 02e306d
ci: keep tests workflow read-only
seonghobae dbd7c3d
Merge remote PR 805 fixes with native Responses orchestration
seonghobae 4f79492
test: close structured orchestration quality gaps
seonghobae f1d61d6
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae 51b22d8
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae afe102c
test: stage Responses multimodal synthesis repair
seonghobae 41680a1
ci: verify Responses multimodal synthesis repair
seonghobae 387ee19
fix: preserve Responses multimodal synthesis
seonghobae 8a30eb3
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae 2da526d
ci: remove completed Responses repair workflow
seonghobae d01311e
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae 3a60530
Merge gateway control stack with native Responses orchestration
seonghobae 87421f0
fix: account Responses passthrough usage
seonghobae 1207d55
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae afd683c
refactor: preserve provider usage totals
seonghobae 0fe5cd3
fix: preserve structured contracts on HTTP
seonghobae 6589dfa
fix: retain the gateway model identity
seonghobae c90c1f6
fix: preserve local Responses contracts and usage
seonghobae 1103487
fix: preserve local response schemas and usage accounting
seonghobae ad142aa
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae 976dca7
fix: record provider-reported structured usage
seonghobae b2c71d1
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae 6e915c2
fix: include in-flight workflow spend in final budget gate
seonghobae c203207
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae 7a0c61b
fix: enforce structured verification and provider contracts
seonghobae 714e1c5
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae aa8a5fe
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae e47242c
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae b86ddbe
fix: persist model judge spend
seonghobae ea37a1e
fix: contain provider errors in structured flows
seonghobae 47ff9d9
fix: preserve Responses text format precedence
seonghobae a47a0c8
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae d0a2aa4
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae 5764fb9
fix: preserve Responses metadata for providers
seonghobae 39c3c4d
fix: emit native Responses usage fields
seonghobae 82bdbb4
Merge remote-tracking branch 'origin/fix/structured-output-orchestrat…
seonghobae 68bdda6
fix: attribute every metered workflow call
seonghobae 9a3f32d
refactor: remove judge ledger string coupling
seonghobae 1d11e7d
fix: bound raw workflow retention
seonghobae e975711
Merge remote-tracking branch 'origin/fix/auto-reasoning-effort-contra…
seonghobae 3bd723c
fix: synchronize runtime analytics snapshots
seonghobae File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
126 changes: 126 additions & 0 deletions
126
docs/planning/adrs/0011-structured-provider-features-stay-orchestrated.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,126 @@ | ||
| --- | ||
| id: "0011" | ||
| title: "Keep structured provider features inside multi-agent orchestration" | ||
| status: accepted | ||
| proposed_date: "2026-08-21" | ||
| accepted_date: "2026-08-21" | ||
| deciders: | ||
| - "repository maintainer" | ||
| consulted: | ||
| - "LineageWeave integration" | ||
| - "contextual-orchestrator runtime" | ||
| informed: | ||
| - "API consumers" | ||
| affected_components: | ||
| - "contextual_orchestrator/orchestrator.py" | ||
| - "contextual_orchestrator/server.py" | ||
| - "tests/test_openai_passthrough.py" | ||
| effort: M | ||
| supersedes: null | ||
| superseded-by: null | ||
| related: | ||
| - path: "docs/planning/adrs/0001-fail-closed-model-judgment.md" | ||
| relation: constrains | ||
| - path: "docs/planning/adrs/0002-explicit-local-mlx-evaluation.md" | ||
| relation: extends | ||
| - path: "docs/architecture.md" | ||
| relation: implements | ||
| success_criteria: | ||
| - metric: "structured requests using the multi-agent workflow" | ||
| target: "100% of non-null response_format, tools, tool_choice, functions, function_call, and Responses requests" | ||
| measurement_window: "every structured-output regression run" | ||
| source: "tests/test_openai_passthrough.py" | ||
| - metric: "Responses json_schema translation" | ||
| target: "Responses text.format json_schema reaches the final provider as the equivalent Chat response_format without losing the schema" | ||
| measurement_window: "every Responses structured-output regression run" | ||
| source: "tests/test_openai_passthrough.py" | ||
| - metric: "multimodal context preservation" | ||
| target: "image_url input remains available to the final synthesis request" | ||
| measurement_window: "every multimodal structured-output regression run" | ||
| source: "tests/test_openai_passthrough.py" | ||
| --- | ||
|
|
||
| # Keep structured provider features inside multi-agent orchestration | ||
|
|
||
| ## Context | ||
|
|
||
| The OpenAI-compatible boundary previously treated `response_format`, tools, and | ||
| the Responses API as a provider passthrough. That made a request look | ||
| successful while skipping the Thinker/Worker/Verifier/Synthesizer workflow. | ||
| Structured output and multimodal requests are still product work, not an | ||
| exception to the orchestration contract. A consumer must receive the same | ||
| workflow evidence, verification boundary, session lineage, and cost accounting | ||
| as a plain chat request. | ||
|
|
||
| ## Decision | ||
|
|
||
| 1. A non-null structured provider feature is an orchestration trigger, never a | ||
| silent single-agent downgrade. | ||
| 2. The request enters the existing conducted workflow. Intermediate steps use | ||
| the original messages, including multimodal content, and the final | ||
| synthesizer performs the provider-facing structured completion. | ||
| 3. The final provider payload preserves validated tools and structured-output | ||
| fields. Responses `text.format` is translated to the equivalent Chat | ||
| `response_format` for the internal provider call, then the result is mapped | ||
| back to the Responses shape. | ||
| 4. `json_object` and `json_schema` are both first-class structured workflows. | ||
| Schema validation remains fail-closed at the HTTP boundary; a provider | ||
| success is not treated as semantic schema validity. | ||
| 5. The workflow response exposes bounded orchestration metadata by default. | ||
| Prompts, answers, images, tool arguments, secrets, and unbounded raw traces | ||
| are not put into telemetry or the default response. | ||
| 6. Tool execution loops are not fabricated by this decision. Provider tools are | ||
| preserved for the final synthesized call; an actual execute-observe-replan | ||
| loop requires a separate ADR and contract. | ||
|
|
||
| ## Research basis | ||
|
|
||
| This decision applies the existing research-grounded architecture rather than | ||
| inventing a provider-specific exception: | ||
|
|
||
| - Fugu distinguishes a low-latency routed call from a quality-oriented deep | ||
| workflow and keeps the worker pool configurable. | ||
| - TRINITY supplies the Thinker, Worker, and Verifier role boundary used before | ||
| synthesis. | ||
| - Conductor supplies explicit workflow steps and access-controlled context. | ||
|
|
||
| The canonical references are maintained in `docs/architecture.md` and | ||
| `docs/papers/README.md`. OpenAI's Chat Completions and Responses API contracts | ||
| define the wire-shape translation, not the orchestration policy. | ||
|
|
||
| ## Consequences | ||
|
|
||
| * Good: structured and multimodal requests no longer bypass verification and | ||
| orchestration evidence. | ||
| * Good: JSON object and JSON schema requests share one tested policy instead of | ||
| diverging into transport-specific single-agent paths. | ||
| * Good: the final provider retains the capability fields it must interpret. | ||
| * Bad: structured requests consume more provider calls and can take longer than | ||
| a plain routed request. | ||
| * Bad: tool execution remains a follow-up capability rather than being implied | ||
| by merely forwarding a tool declaration. | ||
|
|
||
| ## Confirmation | ||
|
|
||
| Run the focused structured-output and orchestration tests. Confirm that the | ||
| Responses JSON-schema test preserves the schema, the tools test reports the | ||
| conducted workflow, and the multimodal test retains `image_url` through final | ||
| synthesis. Do not claim provider-side semantic schema validity from HTTP 200. | ||
|
|
||
| ## References | ||
|
|
||
| Nielsen, S., Cetin, E., Schwendeman, P., Sun, Q., Xu, J., & Tang, Y. (2025). | ||
| *Learning to orchestrate agents in natural language with the Conductor*. | ||
| https://doi.org/10.48550/arXiv.2512.04388 | ||
|
|
||
| Sakana AI. (2026, June 22). *Sakana Fugu: One model to command them all*. | ||
| https://sakana.ai/fugu-release/ | ||
|
|
||
| Xu, J., Sun, Q., Schwendeman, P., Nielsen, S., Cetin, E., & Tang, Y. (2025). | ||
| *Trinity: An evolved LLM coordinator*. https://doi.org/10.48550/arXiv.2512.04695 | ||
|
|
||
| OpenAI. (n.d.-a). *Create chat completion*. OpenAI Platform. | ||
| https://platform.openai.com/docs/api-reference/chat/create | ||
|
|
||
| OpenAI. (n.d.-b). *Create a model response*. OpenAI Platform. | ||
| https://platform.openai.com/docs/api-reference/responses/create |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.