fix(verifier): check Odoo record existence for anchored soft claims (#129 golden flake) - #781
Merged
Merged
Conversation
…129 golden flake) golden-eval.yml flaked on `blocked_deterministic_id_absent` (2 of 15 main runs today, ~30-50 % per-sample miss locally). Root cause: the Haiku claim extractor types "die Rechnung INV/2026/0099 ist verbucht" as `id` in some samples and as `qualitative` in others — in BOTH cases with `odoo_record: {model: 'account.move', ref: 'INV/2026/0099'}` populated. `qualitative` claims bypassed the DeterministicChecker entirely and went to the EvidenceJudge, which returned `unverified` → `approved_with_disclaimer` instead of `blocked`. That is not eval noise: a hallucinated invoice number passed with a disclaimer in roughly a third of real turns. Fix, two layers: 1. Pipeline: soft claims anchored on an Odoo record (`hasOdooRecordAnchor`) get `DeterministicChecker.checkRecordExists()` before the judge. A record that does not exist is a contradiction regardless of how the extractor typed the claim; those claims never reach the judge. Existing records (or unverifiable lookups) go to the judge unchanged. 2. Extractor prompt: a record reference is always its own `id` claim, in addition to any qualitative claim about the same record. Verification: new unit tests (checker ×4, pipeline ×3) red→green; full middleware suite 7165/0 fail; golden corpus 15/15 locally; the flaky case went from 10/16 to 12/12 `blocked` against claude-haiku-4-5 with both claim types now contradicted.
Review findings on PR #781: - HIGH: `name` claims were anchored too — an exact `name = "John Doe"` search would refute "Doe, John" and block a correct answer. Anchor is now `qualitative` only, and a textual `ref` must look like a document sequence (contains a digit). - HIGH: `ref` was searched on `name` only; vendor bills keep the supplier number in `ref`, sale orders in `client_order_ref`, … Added `SOFT_ANCHOR_REF_FIELDS` (per-model field list, tried in order) and an allow-list: unknown model ⇒ `unverified`, judge decides. - MEDIUM: anchors already covered by a hard `id` claim in the same turn are no longer re-queried — one contradiction per record, and the hard path's replay guard owns that record's verdict (asymmetry documented). - LOW: `checkRecordExists` on an unanchored claim now says so. - Tests: fail-open (unverified → judge), hard/soft twin dedupe, `name` exclusion, ref-field fallback, allow-list miss, predicate edge cases. - docs/CHANGELOG.md entry (blocking behaviour change). Verification: verifier tests 55/55, middleware suite 7217/0 fail, golden corpus 15/15 locally, flaky case 10/10 `blocked` with exactly one contradiction per run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
golden-eval.ymlflaked onblocked_deterministic_id_absent— 2 of 15mainruns today (12:39, 13:55), alwaysvia=deterministic-contradicted(MISS)with samples like[approved_with_disclaimer, approved_with_disclaimer, blocked].Reproduced locally (16 runs, 6 misses). In the miss samples the Haiku extractor returns the claim with
odoo_record: {model: "account.move", ref: "INV/2026/0099"}but typedqualitativeinstead ofid.qualitativeclaims never reachDeterministicChecker; the judge saysunverified→approved_with_disclaimer.This is a real gap, not eval noise: a hallucinated invoice number passes with a disclaimer in roughly a third of turns.
Fix (two layers)
hasOdooRecordAnchor()+DeterministicChecker.checkRecordExists(): soft claims anchored on an Odoo record get an existence re-query before the judge (readby id /searchbyname = ref, same transport as theidpath). Non-existent record ⇒contradicted⇒blocked, judge never asked. Existing / unverifiable ⇒ judge as before.idclaim in addition to any qualitative claim about the same record.Verification
npm test(middleware)npm run typecheck/typecheck:testnpm run eval:goldenlocallyclaude-haiku-4-5blocked(before: 10/16), both claim types contradictedCloses the flake behind #129; no corpus or
maxRunschanges — the fix is in the checker, not the eval.Review round (omadia-reviewer) →
fbf7aad5nameclaims anchored → exactname = "John Doe"could block correct answersqualitativeonly;refmust look like a document sequence (contains a digit)refsearched onnameonly (vendor billref, SOclient_order_ref, …)SOFT_ANCHOR_REF_FIELDSper-model field list + allow-list; unknown model ⇒unverified(judge)idtwin + qualitative twin share an anchorcheckSoftClaimsnameexclusion, field fallback, allow-list, predicate edges);docs/CHANGELOG.mdentryPost-review: verifier tests 55/55 · suite 7217/0 fail · golden 15/15 · flaky case 10/10
blocked, exactly one contradiction each.