feat(verifier): tool-output postcondition validation + correctionPrompt retry (#130) - #157
Merged
Conversation
Plugins can now declare an optional Zod `output` schema on each tool. The bridge validates the tool's return value against the schema; a mismatch is recorded as a structured postcondition marker on the RunTrace and surfaces as a synthetic `tool_postcondition` claim (status='contradicted') in the verifier verdict. That contradicted verdict flips the aggregate to `blocked` and feeds into the existing `correctionPrompt` retry loop in `VerifierService`, so no new retry pathway is introduced — the orchestrator simply re-runs the turn with a German "Tool-Output nicht spec-konform" section in the correction hint, instructing the model to re-call the tool with fixed arguments or pick a different tool. Backwards-compatible: tools without an `output` schema behave exactly as before; `LocalSubAgentTool.handle` returns a `string | LocalSubAgentToolResult` union so existing plugins keep returning strings unchanged. Touches the cross-package run-trace shape: `RunToolCall.postcondition` is mirrored in `@omadia/plugin-api` (KG-side) and `@omadia/channel-sdk` (structural copy lifted in S+10-2). Addresses #130 (Recommendation #4, May 2026 LLM-harness audit). Tests: 14 new (verifierPipeline +2 — blocks-on-violation, backward-compat; correctionPromptPostcondition +3 — section emitted, isolated from other sections, undefined for non-blocked verdicts). Full middleware suite: 2629/2634 pass (3 preexisting privacyV4Bypass fails unrelated to #130). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Weegy
added a commit
that referenced
this pull request
May 28, 2026
Extends the #130 postcondition mechanism to native (top-level orchestrator) tools — until now only sub-agent tools (UploadedToolkit / LocalSubAgentTool) could declare a postcondition. Native tools like the upcoming deterministic-tools plugin (#127, #128) need the same protection. What changed: - `NativeToolHandler` may now return `string | NativeToolResult` where `NativeToolResult = { output, postcondition? }`. Backwards-compatible: every existing plugin returns plain strings unchanged. - `Orchestrator.dispatchToolInner` / `.dispatchTool` normalise the handler result into the structured shape; the slot-loop and `finishSlotInvocation` stamp the postcondition onto `RunToolCall.postcondition` for the trace. - A postcondition violation also flips `isError` on the slot so the Anthropic tool_result is marked as such — the orchestrator already short-circuits the answer when it sees a tool error, and the verifier pipeline (#130 PR #157) then raises a `tool_postcondition` claim that drives the existing correctionPrompt retry loop. Tests: new `runTraceCollectorPostcondition.test.ts` covers the one runtime hand-off (collector accepts + emits postcondition on orchestratorToolCalls) that the TypeScript compiler can't catch. Full middleware suite stays at 2629/2634 pass (3 preexisting privacyV4Bypass fails unrelated to this change). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #130.
Summary
outputschema on each tool. The bridge validates the tool's return against the schema; a mismatch is stamped on theRunTraceasRunToolCall.postcondition.tool_postconditionClaimVerdict (status='contradicted'), flipping the verdict toblocked.VerifierServiceretry loop fires automatically;buildCorrectionPromptadds a new German## Tool-Output nicht spec-konformsection that instructs the model to re-call the tool with corrected args (or pick a different tool).outputschema behave exactly as today.LocalSubAgentTool.handlereturns astring | LocalSubAgentToolResultunion — existing plugins return strings unchanged.What this enables
Once #127 (deterministic calculator tool) and #128 (logic-inference tools) land, they will be the first real consumers of
outputschemas. Until then this PR is a no-op at runtime for existing tools but unblocks the deterministic-tools slice.Files
plugin-api:LocalSubAgentToolResultunion +RunToolCall.postconditionchannel-sdk: mirrorRunToolCall.postcondition(structural copy from S+10-2 lift)harness-orchestrator: bridgeTool result unwrap inlocalSubAgent.dispatch,AskObserver.onSubToolResultevent extension,RunTraceCollectorcopies marker,verifierService.extractPostconditionViolationsharness-verifier:ClaimTypeadds'tool_postcondition',VerifierInput.toolPostconditionViolations, pipeline builds synthetic verdicts before extraction,buildCorrectionPromptemits the new sectiondynamicAgentRuntime: bridgeTool runs the postcondition schema aftertd.run(parsed)and returns the structured shape on mismatchverifierPipeline.test.ts+2 (blocks-on-violation, backward-compat); newcorrectionPromptPostcondition.test.ts+3Test plan
npm run buildclean across plugin-api, channel-sdk, orchestrator, verifiernpx tsc --noEmit -p tsconfig.json→ exit 0privacyV4Bypassfails unrelated to feat(runtime): tool-result postcondition validation in dynamicAgentRuntime #130)Addresses Recommendation #4 from the May 2026 LLM-harness audit.
🤖 Generated with Claude Code