Skip to content

fix(verifier): check Odoo record existence for anchored soft claims (#129 golden flake) - #781

Merged
Weegy merged 3 commits into
mainfrom
fix/golden-eval-id-existence-any-type
Aug 20, 2026
Merged

fix(verifier): check Odoo record existence for anchored soft claims (#129 golden flake)#781
Weegy merged 3 commits into
mainfrom
fix/golden-eval-id-existence-any-type

Conversation

@Weegy

@Weegy Weegy commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Problem

golden-eval.yml flaked on blocked_deterministic_id_absent — 2 of 15 main runs today (12:39, 13:55), always via=deterministic-contradicted(MISS) with samples like [approved_with_disclaimer, approved_with_disclaimer, blocked].

Reproduced locally (16 runs, 6 misses). In the miss samples the Haiku extractor returns the claim with odoo_record: {model: "account.move", ref: "INV/2026/0099"} but typed qualitative instead of id. qualitative claims never reach DeterministicChecker; the judge says unverifiedapproved_with_disclaimer.

This is a real gap, not eval noise: a hallucinated invoice number passes with a disclaimer in roughly a third of turns.

Fix (two layers)

  1. PipelinehasOdooRecordAnchor() + DeterministicChecker.checkRecordExists(): soft claims anchored on an Odoo record get an existence re-query before the judge (read by id / search by name = ref, same transport as the id path). Non-existent record ⇒ contradictedblocked, judge never asked. Existing / unverifiable ⇒ judge as before.
  2. Extractor prompt — a record reference is always its own id claim in addition to any qualitative claim about the same record.

Verification

Check Result
New unit tests (checker ×4, pipeline ×3) red → green
npm test (middleware) 7165 tests, 0 fail
npm run typecheck / typecheck:test clean / ratchet 406 = baseline
npm run eval:golden locally 15/15 PASS, all first-sample
Flaky case ×12 against claude-haiku-4-5 12/12 blocked (before: 10/16), both claim types contradicted

Closes the flake behind #129; no corpus or maxRuns changes — the fix is in the checker, not the eval.

Review round (omadia-reviewer) → fbf7aad5

Finding Resolution
HIGH name claims anchored → exact name = "John Doe" could block correct answers Anchor is qualitative only; ref must look like a document sequence (contains a digit)
HIGH ref searched on name only (vendor bill ref, SO client_order_ref, …) SOFT_ANCHOR_REF_FIELDS per-model field list + allow-list; unknown model ⇒ unverified (judge)
MEDIUM double contradiction when hard id twin + qualitative twin share an anchor Dedupe by anchor key; hard path owns the record verdict
MEDIUM replay-guard asymmetry on soft path Covered by the hard twin; documented in checkSoftClaims
LOW misleading reason string, test gaps, missing CHANGELOG Fixed; +5 tests (fail-open, twin dedupe, name exclusion, field fallback, allow-list, predicate edges); docs/CHANGELOG.md entry

Post-review: verifier tests 55/55 · suite 7217/0 fail · golden 15/15 · flaky case 10/10 blocked, exactly one contradiction each.

Weegy added 3 commits August 20, 2026 16:22
…129 golden flake)

golden-eval.yml flaked on `blocked_deterministic_id_absent` (2 of 15 main
runs today, ~30-50 % per-sample miss locally). Root cause: the Haiku claim
extractor types "die Rechnung INV/2026/0099 ist verbucht" as `id` in some
samples and as `qualitative` in others — in BOTH cases with
`odoo_record: {model: 'account.move', ref: 'INV/2026/0099'}` populated.
`qualitative` claims bypassed the DeterministicChecker entirely and went to
the EvidenceJudge, which returned `unverified` → `approved_with_disclaimer`
instead of `blocked`. That is not eval noise: a hallucinated invoice number
passed with a disclaimer in roughly a third of real turns.

Fix, two layers:

1. Pipeline: soft claims anchored on an Odoo record (`hasOdooRecordAnchor`)
   get `DeterministicChecker.checkRecordExists()` before the judge. A record
   that does not exist is a contradiction regardless of how the extractor
   typed the claim; those claims never reach the judge. Existing records (or
   unverifiable lookups) go to the judge unchanged.
2. Extractor prompt: a record reference is always its own `id` claim, in
   addition to any qualitative claim about the same record.

Verification: new unit tests (checker ×4, pipeline ×3) red→green; full
middleware suite 7165/0 fail; golden corpus 15/15 locally; the flaky case
went from 10/16 to 12/12 `blocked` against claude-haiku-4-5 with both
claim types now contradicted.
Review findings on PR #781:

- HIGH: `name` claims were anchored too — an exact `name = "John Doe"`
  search would refute "Doe, John" and block a correct answer. Anchor is now
  `qualitative` only, and a textual `ref` must look like a document
  sequence (contains a digit).
- HIGH: `ref` was searched on `name` only; vendor bills keep the supplier
  number in `ref`, sale orders in `client_order_ref`, … Added
  `SOFT_ANCHOR_REF_FIELDS` (per-model field list, tried in order) and an
  allow-list: unknown model ⇒ `unverified`, judge decides.
- MEDIUM: anchors already covered by a hard `id` claim in the same turn are
  no longer re-queried — one contradiction per record, and the hard path's
  replay guard owns that record's verdict (asymmetry documented).
- LOW: `checkRecordExists` on an unanchored claim now says so.
- Tests: fail-open (unverified → judge), hard/soft twin dedupe, `name`
  exclusion, ref-field fallback, allow-list miss, predicate edge cases.
- docs/CHANGELOG.md entry (blocking behaviour change).

Verification: verifier tests 55/55, middleware suite 7217/0 fail,
golden corpus 15/15 locally, flaky case 10/10 `blocked` with exactly one
contradiction per run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant