feat(context-engine): post-response grounding-enforcement hook - #50155
feat(context-engine): post-response grounding-enforcement hook#50155arminanton wants to merge 1 commit into
Conversation
Adds an optional, duck-typed grounding-enforcement seam to the conversation loop: after the assistant message is normalized, if the active context engine advertises an "enforce_response" capability (via a capabilities() method), the engine may audit the final answer against its record and return a replacement when the answer is ungrounded. No-op for the built-in ContextCompressor and LCM (they have no capabilities()), duck-typed via getattr so any engine lacking the API is skipped, and fully wrapped so enforcement can never break the turn. This is the host-side seam a grounding-aware context engine (e.g. a verbatim-memory engine) plugs into. 6 tests cover replace/keep verdicts, no-capabilities engine, no engine, tool-call turns (skipped), and empty content (skipped).
…v0.17 evidence - CMX grounding-enforcement hook graduated from deferred to its own draft PR NousResearch#50155 (the single CMX-implementation PR per user rule). Only the private-path schema test remains deferred (cannot be public). - v017_per_pr.txt: per-PR v0.17.0 re-applicability for all 39 feature PRs (37 clean + 2 --3way with forward-compat branches + 0 conflict). - symdiff_cleanclone_output.txt refreshed: PASS with 39 PR heads.
|
Related: #50053 (the additive grounding hook points — |
|
Thanks @alt-glitch — correct, this is the conversation-loop call site invoking |
…open questions Addresses the Council demand for platform/review state, not just local repro. Pulled directly from GitHub for all 42 PRs (PER-PR-PLATFORM-STATUS.txt): - 42/42 OPEN (8 ready-for-review, 34 draft). - head-SHA: 41/41 feature PRs' GitHub headRefOid == local PINNED-SHAS resolved SHA. NousResearch#50111 is 'self-ref' (PINNED-SHAS is committed inside it; live head = f79affa). - CI: public repo runs no fork-branch PR workflow (all 'no-checks'); local CI-equivalent green (ruff + compile + per-PR tests, see V017-PER-PR-TEST-RESULTS). - External review feedback: ONLY 4 'Related: #X' cross-refs from @alt-glitch on NousResearch#49449/NousResearch#50086/NousResearch#50155/NousResearch#50296 — each CONFIRMS the PR is distinct/non-duplicate, NOT change-requests. 0 unaddressed review threads, 0 submitted reviews. - The 6 conflict PRs are MERGEABLE on their own base (origin/main); the v0.17.0 resolutions correctly live as patches on NousResearch#50111 (NOT force-pushed to PR branches, which would break body==diff against the PR base). Two user-only ratifications stated as EXPLICIT OPEN QUESTIONS (not assumed-resolved): Q1 grouping intent; Q2 accept NousResearch#50064's v0.17.0 forward-compat test drop (NousResearch#2647).
…rdict Each open PR replayed onto v0.17.0 + tests run. Verdict per PR (not just coverage): 34 clean apply + tests/code-only; 6 conflict PRs with verified v0.17.0 patches; the 5 with failures root-caused (NousResearch#50066/NousResearch#50086 pre-existing v0.17.0 flake reproduced on clean v0.17.0; NousResearch#50031/NousResearch#50032/NousResearch#50078 declared stack-deps that pass on full overlay). NousResearch#50146 + NousResearch#50155 (secondary co-owners, previously only co-listed) now individually verified: CLEAN apply + 6 passed each. 0 PRs left needing review.
teknium1
left a comment
There was a problem hiding this comment.
Thanks for the focused companion host hook. The underlying extension point is not on current main, so this remains useful work, but the draft needs relocation to match the current conversation-loop finalization path.
Problems
- The proposed post-normalization call is not final-only on current
main:agent/conversation_loop.py:4387-4431handles Codexfinish_reason == "incomplete"after normalization and continues the turn. Calling the engine there withfinal=Truecan audit an intermediate response. - Final text is assembled later:
agent/conversation_loop.py:5137-5142joins truncated parts and strips think blocks, and:5268appends the final message. The proposed call would audit a fragment rather than the delivered result. tests/agent/test_grounding_enforcement_hook.py:14-30mirrors the production block instead of invoking the conversation loop, so it cannot verify integration placement or persistence.
Suggested changes
- Rebase the implementation concept onto the current final-response path after
final_responseis composed atagent/conversation_loop.py:5137-5142and before final-message construction/persistence. - Add real
run_conversation()coverage for replacement, incomplete-response skip, and length-continuation composition. - Coordinate the capability/verdict API with companion PR #50053, which is still open.
Automated hermes-sweeper review.
| _has_tools = bool(getattr(assistant_message, "tool_calls", None)) | ||
| _caps = getattr(_eng, "capabilities", None) | ||
| if (_eng is not None and not _has_tools | ||
| and isinstance(_content, str) and _content.strip() |
There was a problem hiding this comment.
This runs immediately after normalization, but a normalized response is not necessarily final: current main continues Codex finish_reason == "incomplete" responses later in the loop. Move enforcement to the final-response path after continuation/recovery and response-part assembly, otherwise final=True can replace an intermediate answer.
| import types | ||
|
|
||
|
|
||
| def _apply_enforcement(agent, assistant_message, messages): |
There was a problem hiding this comment.
This helper mirrors the implementation, so these tests do not exercise the conversation-loop call site. Please drive run_conversation() with an opt-in fake engine to verify replacement is returned and persisted only for a completed final response.
Summary
Adds the host-side grounding-enforcement seam that a grounding-aware context
engine plugs into. After the assistant message is normalized in the conversation
loop, if the active context engine advertises an
enforce_responsecapability(via a
capabilities()method returning{"enforce_response": True, ...}), theengine may audit the final answer against its record and return a replacement
when the answer is ungrounded.
Design
ContextCompressorand LCM have nocapabilities()method, so they are entirely unaffected.getattr(_eng, "capabilities", None)+ acallablecheck,so any engine lacking the API is silently skipped — no hard dependency on a
particular engine being installed.
This is the single host-side seam for a verbatim-memory / grounding context engine.
The engine implementation itself lives in its own component; this PR is just the
provider-agnostic hook in the conversation loop.
Tests
tests/agent/test_grounding_enforcement_hook.py(6 cases): replace verdict,keep verdict, engine without
capabilities()(no-op), no engine (no-op),tool-call turn (skipped), empty content (skipped). All pass; ruff clean.
Built on
origin/main; applies onto v0.17.0.