feat(foundry): add brokered responses mode - #19
Conversation
Signed-off-by: Sertac Ozercan <sozercan@gmail.com>
There was a problem hiding this comment.
Pull request overview
Adds a brokered-only Foundry hosted /responses mode intended to pause on a deterministic function_call and resume via function_call_output, alongside cross-language ABI/config validation and tooling to safely export schema-only brokered tools from Orka Tool CRDs. This extends the shared runtimes/common core with brokered schema types, digest/drift protections, and conformance fixtures/scripts for local and containerized validation.
Changes:
- Introduce brokered tool schemas (
brokeredTools) in the Go/Python ABI, including safe-field enforcement and deterministic schema digests. - Add Foundry brokered conformance fixtures (SDK spike + production brokered-only image) plus transcript generation/verification helpers.
- Extend shared runtime types and add an optional low-level model-loop fallback for brokered tool calling.
Show a summary per file
| File | Description |
|---|---|
| test/foundry-brokered-conformance/README.md | Documents the SDK conformance container fixture and local/live validation steps. |
| test/foundry-brokered-conformance/foundry.agent.yaml.example | Example Foundry hosted-agent manifest for the conformance fixture. |
| test/foundry-brokered-conformance/Dockerfile | Builds a minimal container that runs agentkit-foundry-conformance. |
| test/foundry-brokered-agentkit/README.md | Documents the production AgentKit brokered-only fixture container usage. |
| test/foundry-brokered-agentkit/Dockerfile | Builds a minimal container running agentkit-foundry-brokered with static brokeredTools. |
| test/foundry-brokered-agentkit/agent.yaml | Static agent spec used by the production brokered-only fixture. |
| runtimes/common/tests/test_foundry_transcript_verifier.py | Adds tests for the transcript verifier script behavior. |
| runtimes/common/tests/test_foundry_protocol.py | Adds tests ensuring non-brokered behavior ignores brokered-specific state and routes. |
| runtimes/common/tests/test_foundry_conformance.py | Adds SDK conformance tests for deterministic brokered function-call lifecycle. |
| runtimes/common/tests/test_config_validation.py | Adds extensive brokeredTools validation coverage and digest canonicalization tests. |
| runtimes/common/tests/test_brokered_schema.py | Tests Orka Tool CRD export → safe brokeredTools generation + CLI behavior. |
| runtimes/common/tests/fixtures/foundry_brokered/unknown_previous_response_id_error.json | Golden fixture for unknown previous_response_id error envelope. |
| runtimes/common/tests/fixtures/foundry_brokered/unknown_call_id_error.json | Golden fixture for unknown call_id error envelope. |
| runtimes/common/tests/fixtures/foundry_brokered/tool_policy_rejection_payload.json | Golden payload fixture for policy rejection. |
| runtimes/common/tests/fixtures/foundry_brokered/tool_execution_failure_payload.json | Golden payload fixture for execution failure. |
| runtimes/common/tests/fixtures/foundry_brokered/multiple_function_calls_unsupported_error.json | Golden fixture for deterministic “multiple outputs unsupported” error. |
| runtimes/common/tests/fixtures/foundry_brokered/initial_request.json | Golden initial request fixture for brokered flow. |
| runtimes/common/tests/fixtures/foundry_brokered/function_call_response.json | Golden initial function_call response fixture. |
| runtimes/common/tests/fixtures/foundry_brokered/final_message_response.json | Golden final assistant message response fixture. |
| runtimes/common/tests/fixtures/foundry_brokered/continuation_request.json | Golden continuation request fixture containing function_call_output. |
| runtimes/common/tests/fixtures/foundry_brokered/approval_declined_payload.json | Golden payload fixture for declined approval. |
| runtimes/common/README.md | Documents brokered schema export helper and conformance app entrypoint. |
| runtimes/common/pyproject.toml | Adds Azure Responses SDK dependency and new console scripts for brokered tooling. |
| runtimes/common/agentkit_serve_common/runtime.py | Adds brokered/Responses-oriented runtime result dataclasses and schema digest field. |
| runtimes/common/agentkit_serve_common/foundry_model_loop.py | Adds opt-in low-level brokered model loop (chat-completions) for frameworks without pause/resume. |
| runtimes/common/agentkit_serve_common/foundry_conformance.py | Adds standalone Azure Responses SDK conformance app + CLI entrypoint. |
| runtimes/common/agentkit_serve_common/foundry_brokered_cli.py | Adds brokered-only Foundry /responses serving entrypoint (no direct runtime). |
| runtimes/common/agentkit_serve_common/config.py | Adds brokeredTools schema models, safety validation, and deterministic schema digests. |
| runtimes/common/agentkit_serve_common/brokered.py | Adds Orka Tool CRD → safe brokeredTools export helper + CLI. |
| pkg/agentkit/effective/agent.go | Propagates brokeredTools into effective agent state with deep-copy protection. |
| pkg/agentkit/effective/agent_test.go | Adds tests ensuring brokeredTools parameters are deep-copied safely. |
| pkg/agentkit/config/validate.go | Adds Go-side brokeredTools validation + cross-language canonical digest implementation. |
| pkg/agentkit/config/tool.go | Defines BrokeredTool config type (schema-only) in Go. |
| pkg/agentkit/config/specs.go | Wires brokeredTools into Go AgentConfig schema. |
| pkg/agentkit/config/config_test.go | Adds brokeredTools validation tests and digest cross-language parity tests. |
| pkg/agentkit/abi/render.go | Extends ABI rendering to emit brokeredTools and preserve numeric schema fidelity. |
| pkg/agentkit/abi/render_test.go | Tests ABI render output for brokeredTools and numeric formatting. |
| docs/runtime-capabilities.md | Updates capability docs to describe Foundry brokered deterministic loop and model-loop fallback. |
| docs/foundry-hosted-brokered.md | Adds full documentation for Foundry hosted brokered Responses mode and security invariants. |
| docs/agent-abi.md | Documents brokeredTools in the agent ABI and reader contract expectations. |
| deploy/foundry/scripts/verify_brokered_transcript.py | Adds transcript verifier tool for archived conformance evidence. |
| deploy/foundry/scripts/local_brokered_conformance_container.sh | Adds local build/run + transcript smoke helper for fixtures. |
| deploy/foundry/scripts/foundry_brokered_conformance.sh | Adds live endpoint conformance helper script that produces sanitized transcripts. |
| deploy/foundry/README.md | Documents brokered conformance smoke workflows and new scripts. |
| deploy/foundry/doctor.sh | Adds --brokered-conformance checks for transcript helper prerequisites. |
| .gitignore | Ignores local brokered transcript directories and temporary working dirs. |
| .github/workflows/ci.yml | Extends bash syntax checks and Python compile checks to include new scripts. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 48/49 changed files
- Comments generated: 2
- Review effort level: Low
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7b9f0f6c35
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: Sertac Ozercan <sozercan@gmail.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 268a67c200
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: Sertac Ozercan <sozercan@gmail.com>
Signed-off-by: Sertac Ozercan <sozercan@gmail.com>
Signed-off-by: Sertac Ozercan <sozercan@gmail.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4ad34a9704
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: Sertac Ozercan <sozercan@gmail.com>
Signed-off-by: Sertac Ozercan <sozercan@gmail.com>
Signed-off-by: Sertac Ozercan <sozercan@gmail.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 42006c664c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: Sertac Ozercan <sozercan@gmail.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e1b3db1c8d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: Sertac Ozercan <sozercan@gmail.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f0a3d72ecd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: Sertac Ozercan <sozercan@gmail.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 10addc89f5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: Sertac Ozercan <sozercan@gmail.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 52309dd5b1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if "error" not in parsed or not isinstance(parsed["error"], dict): | ||
| raise ValueError("denied function_call_output.output.error must be an object") |
There was a problem hiding this comment.
In brokered model-loop mode, a denied continuation is canonicalized and passed back to the model as output_json, but this validation only proves that error is an object and leaves any nested fields intact. Fresh evidence beyond the prior top-level shape fix is that {"approved":false,"error":{"code":"denied","message":"no","debug":...}} now passes this branch; the deterministic path ignores debug, while the model-loop resume exposes it to the model, so denied payloads should be restricted to the documented code/message fields or stripped before canonicalization.
Useful? React with 👍 / 👎.
| yield stream.emit_created() | ||
| yield stream.emit_in_progress() | ||
|
|
||
| if _request_tools(request): |
There was a problem hiding this comment.
When the SDK conformance fixture receives a request like {"input":"...","tool_choice":"none"} without a tools array, this check is false and the handler still emits the conformance_read function call. That can make the live brokered smoke pass for a request where the client supplied tool-selection controls, while the production brokered /responses path rejects tool_choice; reject any request-level tool_choice here before emitting conformance evidence.
Useful? React with 👍 / 👎.
| except KeyError: | ||
| return _error("unknown previous_response_id", status=404, code="unknown_previous_response_id") | ||
|
|
||
| item = outputs[0] |
There was a problem hiding this comment.
If the Orka continuation item includes status: "failed" or "in_progress" but still has a parseable output, this handler ignores the item status and consumes the pending state as a successful continuation. In that malformed broker-response scenario AgentKit can finalize the response or resume the model even though the function_call_output was not marked completed; reject non-completed statuses before reading output.
Useful? React with 👍 / 👎.
| message = choice.get("message") | ||
| if not isinstance(message, Mapping): | ||
| raise AgentRunError("model response choice did not include a message", status=502, code="InvalidModelResponse") | ||
| return message |
There was a problem hiding this comment.
In model-loop mode, an OpenAI-compatible provider response whose choice.message.role is user or tool is accepted as long as it has content or tool_calls. With that malformed provider response, AgentKit can either emit a brokered function call or return final text from a non-assistant turn; validate message.role == "assistant" here before accepting tool calls or final content.
Useful? React with 👍 / 👎.
Summary
/responsessupport with static safebrokeredTools, continuation guards, deterministic conformance flow, optional file-backed state, and opt-in model-loop fallbackValidation
uv run --directory runtimes/common --extra dev pytest -q(294 passed)go test ./...make lint(0 issues; existing sibling-path warning still printed)go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.12 .github/workflows/*.ymlbash -n deploy/foundry/doctor.sh deploy/foundry/scripts/*.shcall_...IDs$autoreviewcleanNo transcript artifacts are included in this PR.
Remaining live gates
AGENT_RESPONSES_ENDPOINTis available