diff --git a/middleware/test/golden/README.md b/middleware/test/golden/README.md index b9380d962..8b6b977cb 100644 --- a/middleware/test/golden/README.md +++ b/middleware/test/golden/README.md @@ -11,34 +11,70 @@ behaviour (LLM weakness #13, version drift) fails here instead of shipping. The assertion target is **not** the raw model string — it is the verifier verdict *class*, which is stable despite generation stochasticity: -- `approved` +- `approved` — via trigger-skip **or** (v2) a deterministic-verified hard claim - `approved_with_disclaimer` (the borderline path, `isBorderlineVerdict`) -- `blocked` — including the two synthetic claim paths: +- `blocked` — via a judge contradiction, a deterministic contradiction (v2), or + the two synthetic claim paths: - `tool_postcondition` (#130) - `citation_missing` (#131) Each corpus entry is a frozen `(userMessage, answer, trace fields, fixture -evidence, expected status)` tuple run through a **real** `VerifierPipeline` — -real `ClaimExtractor` + real `EvidenceJudge` on the Anthropic adapter — with -**fixture-backed** deterministic sources (no Postgres, no Odoo, no orchestrator). -That keeps the job hermetic and cheap while still exercising the two stochastic -stages that actually drift. - -## Scope (v1) and what is v2 - -- **v1 (this):** verifier-stage eval. The stochastic stages run on - `VERIFIER_MODEL` (default `claude-haiku-4-5-20251001`), so that is the model - pinned here — asserting against the orchestrator model would be dishonest, - those stages never call it. Hard-claim verdicts resolve to `unverified` (no - Odoo/graph reader is wired), so no fixture relies on deterministic hard-claim - verification. Consequence: the judge's `verified → approved` path is **not** - exercised in v1 (a triggering answer's hard claim is always `unverified` → - disclaimer), so v1's `approved` fixtures cover only the trigger-skip path. -- **v2 (not built) — tracked in [#639](https://github.com/byte5ai/omadia/issues/639):** - a deterministic Odoo/graph fixture reader (enables a genuine judge/deterministic - `verified → approved` entry and a deterministic-contradiction `blocked` entry), - and a full-turn eval (`input → live orchestrator answer → verdict`) once a - headless turn runner exists. +evidence, fixture Odoo records, expected status)` tuple run through a **real** +`VerifierPipeline` — real `ClaimExtractor` + real `EvidenceJudge` + +real `DeterministicChecker` on the Anthropic adapter — with **fixture-backed** +sources (an in-memory `FixtureOdooReader`, no Postgres, no live Odoo, no +orchestrator). That keeps the job hermetic and cheap while still exercising the +stochastic stages that actually drift. + +Since v2 (#639) an entry may also assert the **path** that produced the class, +not just the class, via `expected.via`: + +- `deterministic-verified` — an Odoo re-query CONFIRMED a hard claim (→ approved). +- `deterministic-contradicted` — an Odoo re-query REFUTED a hard claim (→ blocked). + +This exists because the class alone is not enough for the deterministic +`approved` case: a triggering answer whose extractor returns **zero** claims +also lands in `approved`, so `status: "approved"` on its own would pass for the +wrong reason and hide a checker regression. `via` requires a decided-class +sample to also carry a hard-claim verdict of the required kind — a soft (judge) +or synthetic contradiction, or an empty extraction, does not satisfy it. + +## Scope: v1, v2, and what is still open + +All corpus paths pin **`VERIFIER_MODEL`** (default `claude-haiku-4-5-20251001`) +— the model the stochastic `ClaimExtractor`/`EvidenceJudge` actually run on. +Asserting against the orchestrator model would be dishonest; those stages never +call it. The deterministic checker is **pure code** (no model), so the v2 +paths still pin only `VERIFIER_MODEL` (for the extractor that produces the +claim). + +- **v1 (#129):** verifier-stage eval with `DeterministicChecker({})` — no reader, + so every hard claim resolved `unverified` and the checker's verified/ + contradicted branches had zero coverage. `approved` was reachable **only** via + trigger-skip (no hard signal → extraction skipped). +- **v2 (#639), Gaps 1 & 2 — done (this):** a `FixtureOdooReader` injected into + `DeterministicChecker` lets a corpus entry declare frozen Odoo records the + checker re-queries. This adds: + - `corpus/approved-deterministic.jsonl` — a hard claim the ERP **confirms** → + a genuine `verified → approved` (not trigger-skip). Closes Gap 1. + - `corpus/blocked-deterministic-contradiction.jsonl` — a hard claim the ERP + **refutes** → `blocked`, distinct from the judge-contradiction and the + trace-missing-call paths. Closes Gap 2. + + Both use the value-parse-free **id/ref existence** check (the extractor need + only fill `odoo_record`, not the optional `value`), which keeps the live eval + stable under the majority-of-3 policy. The `amount`/`value` branch (verified + and contradicted) is pinned **deterministically** in `goldenModel.test.ts` + with a scripted extractor stub — no key, no jitter — rather than trusting the + live model to populate `value`. +- **v2 (#639), Gap 3 — still open:** a full-turn eval + (`userMessage → live orchestrator answer → verdict`) pinned to + `ORCHESTRATOR_MODEL`/`SUB_AGENT_MODEL`. Deliberately deferred: it needs a + headless turn runner that does not yet exist, and it is the larger lift called + out as its own slice in #639. Until it lands, a model/prompt regression in the + orchestrator itself (e.g. it stops calling a domain tool) is out of scope + here — the corpus evaluates the verifier on frozen answers, it does not + generate them. ## Running locally @@ -54,13 +90,18 @@ never invokes this runner (it lives outside the `test/**/*.test.ts` glob); only the key-free suites run there: - `test/goldenRunner.test.ts` covers the **pure** harness (`goldenRunner.ts`: - parsing, majority voting, flake-tolerant `runEntry`, summary). That module - imports `@omadia/verifier` as `import type` only, so this suite needs no build. + parsing, majority voting, flake-tolerant `runEntry`, the `via` path assertion, + summary) and the pure `FixtureOdooReader` (`read`/`search`/`search_read` over + declared rows). Both import `@omadia/verifier` as `import type` only, so this + suite needs no build. - `test/goldenModel.test.ts` covers the **model** layer (`goldenModel.ts`: the - real `VerifierPipeline` wiring) with a **stub** `LlmProvider`. The synthetic - block paths (`tool_postcondition`, `citation_missing`) need no model, so this - drives the real pipeline to a verdict — and asserts the trace fields are wired - through — without a key. It does need the workspace `dist/` built. + real `VerifierPipeline` wiring) with a **stub** `LlmProvider`. Two stub flavours: + an empty-content stub drives the synthetic block paths (`tool_postcondition`, + `citation_missing`, no model needed); a scripted `record_claims` stub drives + the real extractor → classify → `DeterministicChecker` + `FixtureOdooReader` + path to a deterministic-verified `approved` and a deterministic-contradicted + `blocked`, and asserts the projected verdict tags — all without a key. It does + need the workspace `dist/` built. ## Type coverage @@ -89,10 +130,11 @@ on `pull_request`, forks never attempt to run it. ## Adding a corpus entry (do this when you ship a new agent type or verdict path) 1. Pick the file under `corpus/` that matches the **expected verdict class** - (`approve.jsonl`, `disclaimer.jsonl`, `blocked-citation.jsonl`, - `blocked-tool-postcondition.jsonl`, `blocked-contradiction.jsonl`), or add a - new `*.jsonl` file for a new class. Every `.jsonl` in `corpus/` is picked up - automatically. + (`approve.jsonl`, `approved-deterministic.jsonl`, `disclaimer.jsonl`, + `blocked-citation.jsonl`, `blocked-tool-postcondition.jsonl`, + `blocked-contradiction.jsonl`, `blocked-deterministic-contradiction.jsonl`), + or add a new `*.jsonl` file for a new class. Every `.jsonl` in `corpus/` is + picked up automatically. 2. Append one JSON object per line with this shape: ```jsonc @@ -112,10 +154,26 @@ on `pull_request`, forks never attempt to run it. "evidence": [ // optional fixture evidence for the judge { "nodeId": "n1", "source": "graph", "content": "…", "title": "…" } ], - "expected": { "status": "blocked" } // required: approved | approved_with_disclaimer | blocked + "odoo": { // optional (v2) — frozen Odoo records the + "records": [ // DeterministicChecker re-queries this turn + { "model": "account.move", "id": 42, + "fields": { "name": "INV/2026/0042", "amount_total": 1234.56 } } + ] + }, + "expected": { + "status": "blocked", // required: approved | approved_with_disclaimer | blocked + "via": "deterministic-contradicted" // optional (v2): deterministic-verified | deterministic-contradicted + } } ``` + The `odoo` fixture is a tiny in-memory Odoo: `FixtureOdooReader` answers the + checker's `read` / `search` / `search_read` calls against these rows, so a + hard claim can resolve `verified` or `contradicted` instead of the reader-less + `unverified`. `fields.name` is what a `search` on an accounting ref matches; + `amount_total` / `invoice_date` back the amount / date checks (see + `deterministicChecker.ts` for the per-model field map). + 3. **The trigger trap — read this before writing a judge fixture.** The pipeline runs the stochastic extractor + judge only when `shouldTriggerVerifier(answer)` fires, and it fires **only on a hard signal in @@ -143,10 +201,33 @@ on `pull_request`, forks never attempt to run it. that **explicitly** states something incompatible (the judge only contradicts on explicit conflict); a contradiction dominates the co-extracted `unverified` hard claim, so the verdict is `blocked`. - - `approved` is reachable in v1 **only** via the trigger-skip path (no signal → - extraction skipped). A judge-*verified* approve is not achievable in v1: any - triggering answer carries a hard claim that resolves `unverified` → disclaimer. - That path is v2 (needs the deterministic Odoo/graph reader). + - `approved` via **trigger-skip** needs an answer with **no** hard signal, so + extraction is skipped (`approve.jsonl`). + - `approved` via **deterministic-verified** (v2) and `blocked` via + **deterministic-contradicted** (v2) both need: a hard signal (use an + **accounting ref** like `INV/2026/0042` — it triggers without forcing a + value-dependent amount claim), `trace.domainToolsCalled: ["query_odoo_*"]` + (so the trace-cross-check does **not** pre-empt the checker), an `odoo` + fixture the checker re-queries, and `expected.via` set. Prefer the id/ref + **existence** check: the fixture holding the record → `verified`; the fixture + lacking it → `contradicted`. It needs only `odoo_record`, not the optional + `value`, so it is stable live. If you need to pin an amount/date `value` + branch, do it in `goldenModel.test.ts` with a scripted extractor stub rather + than a live corpus entry — the live model does not reliably fill `value`. + Always assert `via`, never the class alone: a `deterministic-verified` entry + that asserts only `status: "approved"` would silently pass on an empty + extraction and hide the very regression it guards. + - **`deterministic-verified` needs a soft-claim guard.** Clean `approved` + requires **every** extracted claim to be non-`unverified`, and a triggering + answer means the extractor ran. Beside the hard claim the model may + co-extract a soft/qualitative claim off the same sentence; with no + `evidence` that soft claim resolves `unverified` → the turn drops to + `approved_with_disclaimer` and the entry FAILS for a reason you never + controlled. So a `deterministic-verified` entry should also carry an + `evidence` snippet that confirms the qualitative reading (it is simply never + fetched if no soft claim is emitted). A `deterministic-contradicted` entry + needs no such guard — a contradiction dominates the aggregate regardless of + any co-extracted `unverified` claim. 5. Lines starting with `#` and blank lines are ignored — use them for section headers. 6. Validate the shape without spending tokens: `npm test` runs the parser over diff --git a/middleware/test/golden/corpus/approved-deterministic.jsonl b/middleware/test/golden/corpus/approved-deterministic.jsonl new file mode 100644 index 000000000..46541048a --- /dev/null +++ b/middleware/test/golden/corpus/approved-deterministic.jsonl @@ -0,0 +1,43 @@ +# Expected verdict class: approved, reached through a DETERMINISTIC-VERIFIED hard +# claim (#639 v2, closes Gap 1). This is the path v1 could not reach: with no +# Odoo reader, every triggering answer's hard claim resolved `unverified` and +# landed in approved_with_disclaimer, so the checker's verified -> approved +# branch had zero coverage and a regression that stopped confirming true claims +# was invisible. +# +# How this entry forces the branch: +# - The answer names an Odoo record (accounting-ref INV/2026/0042) => the +# trigger router fires on the accounting-ref signal and the extractor + real +# DeterministicChecker actually run. We use an accounting-ref (NOT a currency +# amount) on purpose: an amount claim needs the extractor to also fill the +# optional `value` field, and a value-less amount would resolve `unverified` +# and mask the class. The amount/value branch is pinned deterministically in +# the key-free goldenModel tests instead. +# - `domainToolsCalled: ["query_odoo_accounting"]` so the trace-cross-check +# (hard claim without an Odoo call => contradicted replay) does NOT pre-empt +# the real checker. +# - The `odoo` fixture holds the record, so FixtureOdooReader answers the +# checker's `search name=INV/2026/0042` with id 42 => verified => approved. +# - `expected.via = deterministic-verified` asserts the PATH, not just the +# class: an approved produced by an empty extraction carries no verified hard +# claim and so FAILS this entry — the class assertion alone would not. +# +# Why the `evidence` snippet is here even though the PATH is deterministic: +# clean `approved` requires EVERY extracted claim to be non-`unverified`, and +# reaching a deterministic verdict means the extractor actually ran. Beside the +# `id` claim the model may ALSO emit a soft/qualitative claim off the same +# sentence ("die Rechnung ist vorhanden"). A soft claim with no evidence +# resolves `unverified` (evidenceJudge: "no evidence available") — one such +# claim drags the whole turn to approved_with_disclaimer and the entry would +# FAIL for a reason the fixture never controlled. The snippet gives the judge a +# fact that confirms that existence reading, so an incidental soft claim lands +# `verified` (still `approved`) instead of sinking the entry. If the model emits +# no soft claim the snippet is simply never fetched. The `via` assertion keeps +# this honest: the HARD id claim, not the judge, must still be what earns the +# class. (The mirror entry blocked-deterministic-contradiction.jsonl needs no +# such snippet — a contradiction dominates the aggregate regardless of any +# co-extracted soft/unverified claim.) +# +# Model pinned: VERIFIER_MODEL (extractor + judge). The deterministic +# verification itself is pure code (FixtureOdooReader + DeterministicChecker). +{"id":"approve_deterministic_id_verified","note":"Accounting-ref triggers; the extracted id claim (account.move / INV/2026/0042) is re-queried against the odoo fixture, the record exists => verified => approved. Closes Gap 1. Evidence snippet guards against an incidental soft claim being unverified.","userMessage":"Existiert die Rechnung INV/2026/0042 in unserem ERP?","answer":"Ja, die Rechnung INV/2026/0042 ist im Odoo-Modell account.move als Datensatz vorhanden.","trace":{"agent":"accounting","domainToolsCalled":["query_odoo_accounting"]},"evidence":[{"nodeId":"odoo-inv-42","source":"odoo","content":"Der Datensatz INV/2026/0042 existiert im Odoo-Modell account.move (Betrag 1.234,56 €, Rechnungsdatum 2026-04-19). Die Rechnung ist im System vorhanden.","title":"account.move INV/2026/0042"}],"odoo":{"records":[{"model":"account.move","id":42,"fields":{"name":"INV/2026/0042","amount_total":1234.56,"invoice_date":"2026-04-19"}}]},"expected":{"status":"approved","via":"deterministic-verified"}} diff --git a/middleware/test/golden/corpus/blocked-deterministic-contradiction.jsonl b/middleware/test/golden/corpus/blocked-deterministic-contradiction.jsonl new file mode 100644 index 000000000..1bfa7547f --- /dev/null +++ b/middleware/test/golden/corpus/blocked-deterministic-contradiction.jsonl @@ -0,0 +1,27 @@ +# Expected verdict class: blocked, reached through a DETERMINISTIC CONTRADICTION +# (#639 v2, closes Gap 2) — the ERP data itself refutes a hard claim. This is +# DISTINCT from the two existing block-via-contradiction paths: +# - judge-contradiction (blocked-contradiction.jsonl): a SOFT claim the +# EvidenceJudge refutes against fixture evidence. +# - trace-missing-call (verifierPipeline): a hard claim whose turn made no +# Odoo call at all. This entry declares `query_odoo_accounting`, so that +# pre-check does NOT fire — the contradiction comes from the checker +# re-querying the fixture and finding the claimed record absent. +# +# How this entry forces the branch: +# - Accounting-ref INV/2026/0099 triggers the router and the extractor + real +# DeterministicChecker run (value-parse-free, same rationale as +# approved-deterministic.jsonl). +# - `domainToolsCalled: ["query_odoo_accounting"]` lets the hard claim reach the +# checker instead of being short-circuited as a replay. +# - The `odoo` fixture holds a DIFFERENT invoice (INV/2026/0042) but not +# INV/2026/0099, so FixtureOdooReader answers `search name=INV/2026/0099` +# with [] => contradicted ("no account.move with name=...") => blocked. The +# decoy row makes this an explicit "the ERP has invoices, just not this one" +# refutation rather than an empty-table artefact. +# - `expected.via = deterministic-contradicted` asserts the hard-claim +# contradiction path specifically (soft/judge and synthetic contradictions +# do not satisfy it — see viaSatisfied). +# +# Model pinned: VERIFIER_MODEL (the extractor). The refutation is pure code. +{"id":"blocked_deterministic_id_absent","note":"Accounting-ref triggers; the extracted id claim (account.move / INV/2026/0099) is re-queried against the odoo fixture, no such record exists => contradicted => blocked. Closes Gap 2, distinct from judge-contradiction.","userMessage":"Ist die Rechnung INV/2026/0099 bereits verbucht?","answer":"Ja, die Rechnung INV/2026/0099 ist im Odoo-Modell account.move verbucht und abgeschlossen.","trace":{"agent":"accounting","domainToolsCalled":["query_odoo_accounting"]},"odoo":{"records":[{"model":"account.move","id":42,"fields":{"name":"INV/2026/0042","amount_total":1234.56}}]},"expected":{"status":"blocked","via":"deterministic-contradicted"}} diff --git a/middleware/test/golden/fixtureOdooReader.ts b/middleware/test/golden/fixtureOdooReader.ts new file mode 100644 index 000000000..858205b5d --- /dev/null +++ b/middleware/test/golden/fixtureOdooReader.ts @@ -0,0 +1,151 @@ +/** + * In-memory Odoo stand-in for the golden-set deterministic path (#639, v2). + * + * v1 (#129) wired `DeterministicChecker({})` — no reader — so every HARD claim + * (amount / id / date / aggregate) resolved `unverified`, and the checker's + * `verified` (→ approved) and `contradicted` (→ blocked) branches had zero + * corpus coverage. This reader closes that gap: a golden entry declares a frozen + * record set, and the real `DeterministicChecker` re-queries it exactly as it + * would a live Odoo. + * + * It is a FIXTURE, not a call-mock. It interprets the three RPC methods the + * checker actually issues — `read`, `search`, `search_read` — against the + * declared rows, so the checker's own branching (field lookup, cent-tolerance + * compare, JS re-aggregation) runs UNCHANGED. Only the database is faked. That + * is deliberate: a mock keyed on the exact request would hide the very branch + * the fixture is meant to exercise (cf. the v1 trigger-skip trap the corpus + * README warns about). + * + * Pure: the only `@omadia/verifier` dependency is a type import (erased at + * runtime), so this file loads in the default `npm test` glob without `dist/`. + */ + +import type { OdooReader } from '@omadia/verifier'; + +/** One frozen Odoo row. `fields` holds every column a checker branch may read + * (e.g. `amount_total`, `invoice_date`, `name`, `employee_id`). */ +export interface FixtureOdooRecord { + model: string; + id: number; + fields: Readonly>; +} + +/** The `odoo` block on a corpus entry: the records this turn's checker sees. */ +export interface OdooFixture { + records: readonly FixtureOdooRecord[]; +} + +/** Narrow the loosely-typed positional args the checker passes. */ +function asArray(v: unknown): unknown[] { + return Array.isArray(v) ? v : []; +} + +/** + * Evaluate an Odoo search domain against a row's fields. Supports the small + * slice the DeterministicChecker emits: an AND-list of `[field, op, value]` + * triples with `=`, `!=`, `in`. An empty domain matches every row. Boolean + * operator tokens (`'&'`, `'|'`) and any unknown operator make the row NOT + * match — conservative on purpose, so a fixture never over-matches silently. + */ +function matchesDomain( + fields: Readonly>, + domain: unknown, +): boolean { + if (!Array.isArray(domain)) return true; // no filter → all rows + for (const clause of domain) { + if (!Array.isArray(clause) || clause.length < 3) return false; + const [field, op, value] = clause as [unknown, unknown, unknown]; + if (typeof field !== 'string') return false; + const actual = fields[field]; + switch (op) { + case '=': + if (actual !== value) return false; + break; + case '!=': + if (actual === value) return false; + break; + case 'in': + if (!Array.isArray(value) || !value.includes(actual)) return false; + break; + default: + return false; + } + } + return true; +} + +function pick( + fields: Readonly>, + names: unknown, +): Record { + const out: Record = {}; + for (const n of asArray(names)) { + if (typeof n === 'string' && n in fields) out[n] = fields[n]; + } + return out; +} + +export class FixtureOdooReader implements OdooReader { + private readonly records: readonly FixtureOdooRecord[]; + + constructor(fixture: OdooFixture) { + this.records = fixture.records; + } + + execute(req: { + model: string; + method: string; + positionalArgs: unknown[]; + kwargs: Record; + }): Promise { + const rows = this.records.filter((r) => r.model === req.model); + switch (req.method) { + case 'read': + return Promise.resolve(this.read(rows, req.positionalArgs)); + case 'search': + return Promise.resolve(this.search(rows, req.positionalArgs)); + case 'search_read': + return Promise.resolve(this.searchRead(rows, req.positionalArgs)); + default: + // Unknown method → undefined. The checker treats a non-array / missing + // result as `unverified`, which is the safe default. + return Promise.resolve(undefined); + } + } + + /** `read([ids], [fields])` → one row per matching id, in id order. */ + private read( + rows: readonly FixtureOdooRecord[], + positionalArgs: unknown[], + ): Array> { + const ids = asArray(positionalArgs[0]); + const fields = positionalArgs[1]; + const out: Array> = []; + for (const id of ids) { + const rec = rows.find((r) => r.id === id); + if (rec) out.push({ id: rec.id, ...pick(rec.fields, fields) }); + } + return out; + } + + /** `search(domain)` → matching ids. */ + private search( + rows: readonly FixtureOdooRecord[], + positionalArgs: unknown[], + ): number[] { + const domain = positionalArgs[0]; + return rows.filter((r) => matchesDomain(r.fields, domain)).map((r) => r.id); + } + + /** `search_read(domain, [fields])` → matching rows' selected fields. */ + private searchRead( + rows: readonly FixtureOdooRecord[], + positionalArgs: unknown[], + ): Array> { + const domain = positionalArgs[0]; + const fields = positionalArgs[1]; + return rows + .filter((r) => matchesDomain(r.fields, domain)) + .map((r) => ({ id: r.id, ...pick(r.fields, fields) })); + } +} diff --git a/middleware/test/golden/goldenModel.ts b/middleware/test/golden/goldenModel.ts index 27b41ebcf..c4e0f37ae 100644 --- a/middleware/test/golden/goldenModel.ts +++ b/middleware/test/golden/goldenModel.ts @@ -27,7 +27,13 @@ import { } from '@omadia/verifier'; import type { LlmProvider } from '@omadia/llm-provider-api'; -import type { EntryRun, GoldenEntry, RunOnce } from './goldenRunner.js'; +import { FixtureOdooReader } from './fixtureOdooReader.js'; +import type { + EntryRun, + GoldenEntry, + RunOnce, + VerdictTag, +} from './goldenRunner.js'; function toVerifierInput(entry: GoldenEntry): VerifierInput { const t = entry.trace; @@ -62,16 +68,23 @@ function fixtureFetcher(evidence: EvidenceSnippet[]): EvidenceFetcher { function buildPipeline( llm: LlmProvider, model: string, - evidence: EvidenceSnippet[], + entry: GoldenEntry, ): VerifierPipeline { + // #639 v2 — when the entry declares `odoo` records, the DeterministicChecker + // re-queries them through a FixtureOdooReader, so a hard claim can resolve + // `verified` (→ approved) or `contradicted` (→ blocked). Without it, hard + // claims resolve `unverified` exactly as in v1 — no Postgres either way. + const deterministic = entry.odoo + ? new DeterministicChecker({ odoo: new FixtureOdooReader(entry.odoo) }) + : new DeterministicChecker({}); return new VerifierPipeline({ extractor: new ClaimExtractor({ llm, model }), - // No odoo/graph reader: hard claims resolve to `unverified` rather than - // hitting Postgres. The verdict-class fixtures never depend on - // deterministic hard-claim verification (that path needs an Odoo fixture - // reader — v2, see README). - deterministic: new DeterministicChecker({}), - judge: new EvidenceJudge({ llm, fetcher: fixtureFetcher(evidence), model }), + deterministic, + judge: new EvidenceJudge({ + llm, + fetcher: fixtureFetcher(entry.evidence ?? []), + model, + }), log: (): void => { /* silent */ }, @@ -100,8 +113,15 @@ export function buildVerifierRunOnce( stream: provider.stream.bind(provider), classifyError: provider.classifyError.bind(provider), }; - const pipeline = buildPipeline(counting, model, entry.evidence ?? []); + const pipeline = buildPipeline(counting, model, entry); const verdict = await pipeline.verify(toVerifierInput(entry)); - return { status: verdict.status, tokens }; + // Project the per-claim verdicts so the runner can assert the PATH (`via`), + // not just the class — an `approved` from an empty extraction carries no + // `verified` tag and so cannot satisfy `deterministic-verified`. + const verdicts: VerdictTag[] = verdict.claims.map((c) => ({ + status: c.status, + claimType: c.claim.type, + })); + return { status: verdict.status, tokens, verdicts }; }; } diff --git a/middleware/test/golden/goldenRunner.ts b/middleware/test/golden/goldenRunner.ts index 336da830b..9f52f7637 100644 --- a/middleware/test/golden/goldenRunner.ts +++ b/middleware/test/golden/goldenRunner.ts @@ -20,11 +20,63 @@ * re-run up to `maxRuns` total and decided by majority. */ -import type { EvidenceSnippet, VerifierVerdict } from '@omadia/verifier'; +import type { + ClaimType, + ClaimVerdict, + EvidenceSnippet, + VerifierVerdict, +} from '@omadia/verifier'; + +import type { OdooFixture } from './fixtureOdooReader.js'; /** The three stable assertion targets — the live `VerifierVerdict` statuses. */ export type StatusName = VerifierVerdict['status']; +/** + * #639 v2 — an OPTIONAL, stronger assertion than the verdict class alone: it + * pins WHICH path produced the class, so an entry cannot pass for the wrong + * reason. + * + * The v1 blind spot was structural: a triggering answer whose extractor + * happens to return zero claims lands in `approved` for FREE — the same + * trigger-skip trap the corpus README warns about, one layer up. Asserting + * only `status === 'approved'` would let that empty-extraction case pass and + * hide a regression where the deterministic checker stopped confirming true + * claims. `via` closes that: the entry passes only if a decided-class sample + * also carried a HARD-claim verdict of the required kind. + * + * - `deterministic-verified` — an Odoo/graph re-query CONFIRMED a hard claim + * (closes Gap 1: a genuine `verified → approved`, not trigger-skip). + * - `deterministic-contradicted` — an Odoo/graph re-query REFUTED a hard claim + * (closes Gap 2: ERP data contradicts the answer → blocked, distinct from + * the judge-contradiction and trace-missing-call paths, both of which this + * predicate excludes — see `viaSatisfied`). + */ +export type ViaTag = 'deterministic-verified' | 'deterministic-contradicted'; + +const VIA_TAGS: readonly ViaTag[] = [ + 'deterministic-verified', + 'deterministic-contradicted', +]; + +/** Claim types the DeterministicChecker owns (vs. the EvidenceJudge's soft + * `name`/`qualitative`). A `via` predicate keys on this so a judge or + * synthetic contradiction can never satisfy a `deterministic-*` assertion. */ +const HARD_CLAIM_TYPES: readonly ClaimType[] = [ + 'amount', + 'id', + 'date', + 'aggregate', +]; + +/** The claim-level shape a `via` predicate inspects — one per `ClaimVerdict` + * the real pipeline emitted. Kept minimal (status + claim type) so the pure + * layer stays value-free of the verifier package. */ +export interface VerdictTag { + status: ClaimVerdict['status']; + claimType: ClaimType; +} + const STATUS_NAMES: readonly StatusName[] = [ 'approved', 'approved_with_disclaimer', @@ -61,13 +113,31 @@ export interface GoldenEntry { * evidence => contradicted => blocked. */ evidence?: EvidenceSnippet[]; - expected: { status: StatusName }; + /** + * #639 v2 — frozen Odoo records the DeterministicChecker re-queries this + * turn (via `FixtureOdooReader`). Present ⇒ hard claims can resolve + * `verified`/`contradicted` instead of the v1 default `unverified`. Absent ⇒ + * `DeterministicChecker({})`, exactly as v1. + */ + odoo?: OdooFixture; + expected: { + status: StatusName; + /** #639 v2 — optional path assertion; see {@link ViaTag}. */ + via?: ViaTag; + }; } export interface EntryRun { status: StatusName; /** input+output tokens spent across every LLM call in this single run. */ tokens: number; + /** + * #639 v2 — the per-claim verdicts the pipeline produced this run, projected + * to {@link VerdictTag}. Optional: a `RunOnce` that predates the `via` + * feature (or a scripted test stub) may omit it, in which case any `via` + * assertion on the entry treats the run as not satisfying the path. + */ + verdicts?: VerdictTag[]; } /** One stochastic execution of a fixture. Injected so the pure layer never @@ -84,6 +154,11 @@ export interface GoldenResult { pass: boolean; /** summed tokens across every sample of this entry. */ tokens: number; + /** #639 v2 — the asserted path, when the entry declared one. */ + via?: ViaTag; + /** #639 v2 — whether a decided-class sample reached the class via `via`. + * Undefined when the entry declared no `via`. */ + viaPass?: boolean; } // --------------------------------------------------------------------------- @@ -127,26 +202,67 @@ export function parseCorpusLine( const id = req('id'); const userMessage = req('userMessage'); const answer = req('answer'); - const expected = o['expected'] as { status?: unknown } | undefined; + const expected = o['expected'] as + | { status?: unknown; via?: unknown } + | undefined; if (!expected || !isStatusName(expected.status)) { throw new Error( `${where}: "expected.status" must be one of ${STATUS_NAMES.join(' | ')}`, ); } + if (expected.via !== undefined && !isViaTag(expected.via)) { + throw new Error( + `${where}: "expected.via" must be one of ${VIA_TAGS.join(' | ')}`, + ); + } const entry: GoldenEntry = { id, userMessage, answer, - expected: { status: expected.status }, + expected: { + status: expected.status, + ...(expected.via !== undefined ? { via: expected.via } : {}), + }, }; if (typeof o['note'] === 'string') entry.note = o['note']; if (o['trace'] !== undefined) entry.trace = o['trace'] as GoldenTrace; if (o['evidence'] !== undefined) { entry.evidence = o['evidence'] as EvidenceSnippet[]; } + if (o['odoo'] !== undefined) { + const odoo = o['odoo'] as { records?: unknown }; + if (!odoo || !Array.isArray(odoo.records)) { + throw new Error(`${where}: "odoo.records" must be an array`); + } + entry.odoo = odoo as OdooFixture; + } return entry; } +function isViaTag(v: unknown): v is ViaTag { + return typeof v === 'string' && (VIA_TAGS as readonly string[]).includes(v); +} + +/** + * Does any of this run's claim verdicts satisfy the required path? A + * `deterministic-*` tag is satisfied only by a HARD-claim verdict of the + * matching status — soft (judge) claims and synthetic + * (`tool_postcondition`/`citation_missing`) claims are excluded by type, and + * the trace-missing-call contradiction is excluded at the fixture level (those + * entries declare a `query_odoo_*` call, so that pre-check never fires). Net: + * a satisfied `deterministic-contradicted` means the checker itself refuted the + * claim against the fixture records. + */ +function viaSatisfied(via: ViaTag, verdicts: VerdictTag[] | undefined): boolean { + if (!verdicts) return false; + const want = via === 'deterministic-verified' ? 'verified' : 'contradicted'; + return verdicts.some( + (v) => + v.status === want && + (HARD_CLAIM_TYPES as readonly string[]).includes(v.claimType), + ); +} + /** Parse a whole `.jsonl` file body. Blank lines and `#`-comment lines skipped. */ export function loadCorpusFromText(text: string, file: string): GoldenEntry[] { const out: GoldenEntry[] = []; @@ -197,27 +313,51 @@ export async function runEntry( runOnce: RunOnce, maxRuns = 3, ): Promise { - const first = await runOnce(entry); - const runs: EntryRun[] = [first]; - if (first.status !== entry.expected.status) { - while (runs.length < maxRuns) { - runs.push(await runOnce(entry)); - } + const via = entry.expected.via; + const runs: EntryRun[] = [await runOnce(entry)]; + // Re-run (up to `maxRuns`) while the entry is not yet decided AND satisfied: + // a first sample that is already green on BOTH status and path costs one + // call; a miss on either buys the same majority-of-3 flake tolerance. Folding + // `via` into the guard means the path assertion is flake-tolerant too — a + // lone empty-extraction sample doesn't sink an otherwise-verified entry. + while (runs.length < maxRuns && !decidedAndSatisfied(runs, entry.expected.status, via)) { + runs.push(await runOnce(entry)); } const statuses = runs.map((r) => r.status); // `majority` returns the sole sample unchanged for a one-element list, so it // doubles as the single-run case — no separate index into `statuses`. const decided = majority(statuses).winner; + const statusPass = decided === entry.expected.status; + // The path assertion is checked on the decided-class samples: at least one + // sample that produced `decided` must have reached it through `via`. + const viaPass = + via === undefined + ? undefined + : runs.some((r) => r.status === decided && viaSatisfied(via, r.verdicts)); return { id: entry.id, expected: entry.expected.status, runs: statuses, decided, - pass: decided === entry.expected.status, + pass: statusPass && viaPass !== false, tokens: runs.reduce((sum, r) => sum + r.tokens, 0), + ...(via !== undefined ? { via, viaPass: viaPass ?? false } : {}), }; } +/** Would the samples so far pass? Used as the re-run stop condition, so the + * cost-aware "green first try = one call" property extends to `via`. */ +function decidedAndSatisfied( + runs: EntryRun[], + expected: StatusName, + via: ViaTag | undefined, +): boolean { + const decided = majority(runs.map((r) => r.status)).winner; + if (decided !== expected) return false; + if (via === undefined) return true; + return runs.some((r) => r.status === decided && viaSatisfied(via, r.verdicts)); +} + export async function runCorpus( entries: GoldenEntry[], runOnce: RunOnce, @@ -243,9 +383,10 @@ export function formatSummary(results: GoldenResult[]): string { const lines: string[] = []; for (const r of results) { const mark = r.pass ? 'PASS' : 'FAIL'; + const via = r.via ? ` via=${r.via}(${r.viaPass ? 'ok' : 'MISS'})` : ''; lines.push( `${mark} ${r.id.padEnd(28)} expected=${r.expected} decided=${r.decided} ` + - `runs=[${r.runs.join(',')}] tokens=${r.tokens}`, + `runs=[${r.runs.join(',')}] tokens=${r.tokens}${via}`, ); } lines.push(''); diff --git a/middleware/test/goldenModel.test.ts b/middleware/test/goldenModel.test.ts index e4139dff9..023882784 100644 --- a/middleware/test/goldenModel.test.ts +++ b/middleware/test/goldenModel.test.ts @@ -59,6 +59,143 @@ function stubProvider(onComplete?: () => void): LlmProvider { }; } +/** + * A provider whose `complete` returns a fixed `record_claims` tool call — it + * lets us drive the REAL ClaimExtractor -> classify -> DeterministicChecker path + * deterministically (no key, no model jitter). We use it to pin the + * hard-claim amount branch and the verdict-tag projection that the live corpus + * (majority-of-3, value-fill dependent) deliberately does not lean on. + */ +function claimStub(claims: unknown[]): LlmProvider { + return { + id: 'stub', + capabilities: { + tools: true, + vision: false, + streaming: false, + promptCaching: false, + forcedToolChoice: true, + parallelToolCalls: false, + interleavedToolUse: false, + }, + complete(_req) { + return Promise.resolve({ + content: [ + { type: 'tool_call', id: 'call_x', name: 'record_claims', input: { claims } }, + ], + finishReason: 'stop', + model: 'stub', + usage: { inputTokens: 5, outputTokens: 5 }, + }); + }, + stream() { + throw new Error('stub: stream() must not be called on the verify path'); + }, + classifyError() { + throw new Error('stub: classifyError() must not be called on the verify path'); + }, + }; +} + +describe('goldenModel/deterministic hard-claim path (#639 v2, key-free)', () => { + // A currency amount triggers the router; the stub extracts it as an amount + // claim pinned to account.move:42; the odoo fixture supplies amount_total. + const answer = 'Der Gesamtbetrag von INV/2026/0042 beträgt 1.234,56 €.'; + const amountClaim = { + text: '1.234,56 €', + type: 'amount', + expected_source: 'odoo', + value: 1234.56, + unit: '€', + odoo_record: { model: 'account.move', id: 42 }, + }; + function entry(amount_total: number): GoldenEntry { + return { + id: 'det', + userMessage: 'Wie hoch ist der Gesamtbetrag von INV/2026/0042?', + answer, + trace: { agent: 'accounting', domainToolsCalled: ['query_odoo_accounting'] }, + odoo: { records: [{ model: 'account.move', id: 42, fields: { amount_total } }] }, + expected: { status: 'approved' }, + }; + } + + it('APPROVES via a deterministic-verified amount and tags the verdict (Gap 1)', async () => { + const r = await buildVerifierRunOnce(claimStub([amountClaim]), 'stub-model')(entry(1234.56)); + assert.equal(r.status, 'approved'); + assert.ok( + r.verdicts?.some((v) => v.status === 'verified' && v.claimType === 'amount'), + 'expected a verified amount tag so `via: deterministic-verified` can be asserted', + ); + }); + + it('BLOCKS via a deterministic contradiction when the ERP amount differs (Gap 2)', async () => { + const r = await buildVerifierRunOnce(claimStub([amountClaim]), 'stub-model')(entry(2000)); + assert.equal(r.status, 'blocked'); + assert.ok( + r.verdicts?.some((v) => v.status === 'contradicted' && v.claimType === 'amount'), + 'expected a contradicted amount tag', + ); + }); + + it('without an odoo fixture the same claim resolves unverified -> disclaimer (v1 behaviour)', async () => { + const noReader: GoldenEntry = { ...entry(1234.56), odoo: undefined }; + const r = await buildVerifierRunOnce(claimStub([amountClaim]), 'stub-model')(noReader); + assert.equal(r.status, 'approved_with_disclaimer'); + }); +}); + +describe('goldenModel/deterministic id-ref path (#639 v2, key-free)', () => { + // Key-free PARITY for the branch the two live corpus entries actually ride: + // an `id` claim with `odoo_record.ref` (no numeric id) drives the REAL + // checkOdooId `search [['name','=',ref]]` path — distinct from the amount + // block above, which no key-free test otherwise exercises. The fixture + // holding the ref => verified => approved; a fixture with only a DECOY row + // => search returns [] => contradicted => blocked. + const answer = + 'Ja, die Rechnung INV/2026/0042 ist im Odoo-Modell account.move als Datensatz vorhanden.'; + const idClaim = { + text: 'INV/2026/0042', + type: 'id', + expected_source: 'odoo', + odoo_record: { model: 'account.move', ref: 'INV/2026/0042' }, + }; + function entry(records: Array<{ model: string; id: number; fields: Record }>): GoldenEntry { + return { + id: 'det-id', + userMessage: 'Existiert die Rechnung INV/2026/0042 in unserem ERP?', + answer, + trace: { agent: 'accounting', domainToolsCalled: ['query_odoo_accounting'] }, + odoo: { records }, + expected: { status: 'approved' }, + }; + } + const present = [ + { model: 'account.move', id: 42, fields: { name: 'INV/2026/0042', amount_total: 1234.56 } }, + ]; + const decoyOnly = [ + { model: 'account.move', id: 7, fields: { name: 'INV/2026/0007', amount_total: 99 } }, + ]; + + it('APPROVES via a deterministic-verified id/ref existence check (Gap 1 path)', async () => { + const r = await buildVerifierRunOnce(claimStub([idClaim]), 'stub-model')(entry(present)); + assert.equal(r.status, 'approved'); + assert.ok( + r.verdicts?.some((v) => v.status === 'verified' && v.claimType === 'id'), + 'expected a verified id tag so `via: deterministic-verified` can be asserted', + ); + }); + + it('BLOCKS via a deterministic contradiction when the ref is absent (Gap 2 path)', async () => { + const r = await buildVerifierRunOnce(claimStub([idClaim]), 'stub-model')(entry(decoyOnly)); + assert.equal(r.status, 'blocked'); + assert.ok( + r.verdicts?.some((v) => v.status === 'contradicted' && v.claimType === 'id'), + 'expected a contradicted id tag', + ); + }); +}); + describe('goldenModel/buildVerifierRunOnce (synthetic paths, key-free)', () => { it('blocks via a recorded tool_postcondition violation (#130)', async () => { const entry: GoldenEntry = { diff --git a/middleware/test/goldenRunner.test.ts b/middleware/test/goldenRunner.test.ts index df4ade0ef..c019374c1 100644 --- a/middleware/test/goldenRunner.test.ts +++ b/middleware/test/goldenRunner.test.ts @@ -24,12 +24,38 @@ import { type GoldenEntry, type RunOnce, type StatusName, + type VerdictTag, + type ViaTag, } from './golden/goldenRunner.js'; +import { FixtureOdooReader } from './golden/fixtureOdooReader.js'; function entry(expected: StatusName, id = 'e'): GoldenEntry { return { id, userMessage: 'u', answer: 'a', expected: { status: expected } }; } +function entryVia(status: StatusName, via: ViaTag, id = 'e'): GoldenEntry { + return { id, userMessage: 'u', answer: 'a', expected: { status, via } }; +} + +/** A RunOnce that yields scripted EntryRuns (status + verdict tags), then + * repeats the last — for exercising the `via` path assertion. */ +function scriptedRuns(seq: EntryRun[]): RunOnce & { calls: () => number } { + let i = 0; + let calls = 0; + const fn = ((): Promise => { + calls++; + const run = seq[Math.min(i, seq.length - 1)] ?? seq[seq.length - 1]!; + i++; + return Promise.resolve(run); + }) as unknown as RunOnce & { calls: () => number }; + fn.calls = (): number => calls; + return fn; +} + +function run(status: StatusName, verdicts?: VerdictTag[]): EntryRun { + return { status, tokens: 10, ...(verdicts ? { verdicts } : {}) }; +} + /** A RunOnce that yields a scripted sequence of statuses, then repeats the last. * Records how many times it was called so the cost-aware path is observable. */ function scriptedRunOnce(seq: StatusName[]): RunOnce & { calls: () => number } { @@ -82,6 +108,47 @@ describe('goldenRunner/parseCorpusLine', () => { it('throws on malformed JSON', () => { assert.throws(() => parseCorpusLine('{not json', 'f.jsonl', 2), /f\.jsonl:2: invalid JSON/); }); + + it('parses a v2 entry with expected.via and an odoo fixture (#639)', () => { + const e = parseCorpusLine( + JSON.stringify({ + id: 'v2', + userMessage: 'q', + answer: 'INV/2026/0042', + trace: { domainToolsCalled: ['query_odoo_accounting'] }, + odoo: { records: [{ model: 'account.move', id: 42, fields: { name: 'INV/2026/0042' } }] }, + expected: { status: 'approved', via: 'deterministic-verified' }, + }), + 'f.jsonl', + 1, + ); + assert.equal(e.expected.via, 'deterministic-verified'); + assert.equal(e.odoo?.records[0]?.id, 42); + }); + + it('throws on an unknown expected.via', () => { + assert.throws( + () => + parseCorpusLine( + JSON.stringify({ id: 'x', userMessage: 'q', answer: 'a', expected: { status: 'approved', via: 'judge-verified' } }), + 'f.jsonl', + 4, + ), + /f\.jsonl:4: "expected\.via" must be one of/, + ); + }); + + it('throws when odoo.records is not an array', () => { + assert.throws( + () => + parseCorpusLine( + JSON.stringify({ id: 'x', userMessage: 'q', answer: 'a', odoo: { records: 'nope' }, expected: { status: 'approved' } }), + 'f.jsonl', + 9, + ), + /f\.jsonl:9: "odoo\.records" must be an array/, + ); + }); }); describe('goldenRunner/loadCorpusFromText', () => { @@ -147,6 +214,139 @@ describe('goldenRunner/runEntry', () => { }); }); +describe('goldenRunner/runEntry via path assertion (#639 v2)', () => { + const VERIFIED_HARD: VerdictTag[] = [{ status: 'verified', claimType: 'id' }]; + const CONTRA_HARD: VerdictTag[] = [{ status: 'contradicted', claimType: 'amount' }]; + + it('passes when the decided class AND the via path are both met', async () => { + const r = await runEntry( + entryVia('approved', 'deterministic-verified'), + scriptedRuns([run('approved', VERIFIED_HARD)]), + ); + assert.equal(r.pass, true); + assert.equal(r.viaPass, true); + assert.equal(r.via, 'deterministic-verified'); + }); + + it('FAILS an approved reached with no verified hard claim — the empty-extraction trap', async () => { + // Status matches but the path does not: an `approved` carrying no verified + // hard verdict (e.g. the extractor returned zero claims) must NOT pass. This + // is exactly the v1 blind spot #639 exists to close. The via-miss also + // drives the re-runs, so we pay the full flake budget before failing. + const runner = scriptedRuns([run('approved'), run('approved'), run('approved')]); + const r = await runEntry(entryVia('approved', 'deterministic-verified'), runner); + assert.equal(r.decided, 'approved'); + assert.equal(r.viaPass, false); + assert.equal(r.pass, false); + assert.equal(runner.calls(), 3); + }); + + it('is flake-tolerant on the path: a lone via-miss is corrected by majority', async () => { + const r = await runEntry( + entryVia('approved', 'deterministic-verified'), + scriptedRuns([run('approved'), run('approved', VERIFIED_HARD), run('approved', VERIFIED_HARD)]), + ); + assert.equal(r.pass, true); + assert.equal(r.viaPass, true); + }); + + it('deterministic-contradicted is satisfied by a contradicted HARD claim', async () => { + const r = await runEntry( + entryVia('blocked', 'deterministic-contradicted'), + scriptedRuns([run('blocked', CONTRA_HARD)]), + ); + assert.equal(r.pass, true); + assert.equal(r.viaPass, true); + }); + + it('a SOFT (judge) contradiction does NOT satisfy deterministic-contradicted', async () => { + // Same blocked class, but the contradiction is on a qualitative claim — the + // judge path, not the deterministic checker. Keeps the two block-via- + // contradiction paths from being conflated. + const soft: VerdictTag[] = [{ status: 'contradicted', claimType: 'qualitative' }]; + const runner = scriptedRuns([run('blocked', soft), run('blocked', soft), run('blocked', soft)]); + const r = await runEntry(entryVia('blocked', 'deterministic-contradicted'), runner); + assert.equal(r.decided, 'blocked'); + assert.equal(r.viaPass, false); + assert.equal(r.pass, false); + }); + + it('a green first sample on both class and path costs exactly one call', async () => { + const runner = scriptedRuns([run('approved', VERIFIED_HARD)]); + await runEntry(entryVia('approved', 'deterministic-verified'), runner); + assert.equal(runner.calls(), 1); + }); +}); + +describe('golden/FixtureOdooReader (#639 v2)', () => { + const reader = new FixtureOdooReader({ + records: [ + { model: 'account.move', id: 42, fields: { name: 'INV/2026/0042', amount_total: 1234.56 } }, + { model: 'account.move', id: 7, fields: { name: 'INV/2026/0007', amount_total: 99 } }, + { model: 'hr.leave', id: 3, fields: { number_of_days: 2, employee_id: 7 } }, + { model: 'hr.leave', id: 4, fields: { number_of_days: 5, employee_id: 7 } }, + ], + }); + + it('read([id],[field]) returns the row with id + selected fields', async () => { + const rows = (await reader.execute({ + model: 'account.move', + method: 'read', + positionalArgs: [[42], ['amount_total']], + kwargs: {}, + })) as Array>; + assert.deepEqual(rows, [{ id: 42, amount_total: 1234.56 }]); + }); + + it('read of a missing id returns [] (checker reads that as "not found")', async () => { + const rows = (await reader.execute({ + model: 'account.move', + method: 'read', + positionalArgs: [[999], ['amount_total']], + kwargs: {}, + })) as unknown[]; + assert.deepEqual(rows, []); + }); + + it('search on name = present ref returns the id; absent ref returns []', async () => { + const hit = (await reader.execute({ + model: 'account.move', + method: 'search', + positionalArgs: [[['name', '=', 'INV/2026/0042']]], + kwargs: { limit: 1 }, + })) as number[]; + assert.deepEqual(hit, [42]); + const miss = (await reader.execute({ + model: 'account.move', + method: 'search', + positionalArgs: [[['name', '=', 'INV/2026/0099']]], + kwargs: { limit: 1 }, + })) as number[]; + assert.deepEqual(miss, []); + }); + + it('search_read filters by an employee_id domain and projects the field', async () => { + const rows = (await reader.execute({ + model: 'hr.leave', + method: 'search_read', + positionalArgs: [[['employee_id', '=', 7]], ['number_of_days']], + kwargs: { limit: 1000 }, + })) as Array>; + assert.equal(rows.length, 2); + assert.deepEqual(rows.map((r) => r.number_of_days).sort(), [2, 5]); + }); + + it('does not leak rows across models', async () => { + const rows = (await reader.execute({ + model: 'account.move', + method: 'search_read', + positionalArgs: [[], ['amount_total']], + kwargs: {}, + })) as unknown[]; + assert.equal(rows.length, 2); // the two account.move rows, not hr.leave + }); +}); + describe('goldenRunner/corpus integrity (#129 acceptance, key-free)', () => { const dir = new URL('./golden/corpus/', import.meta.url); const files = readdirSync(dir).filter((f) => f.endsWith('.jsonl')); @@ -154,9 +354,9 @@ describe('goldenRunner/corpus integrity (#129 acceptance, key-free)', () => { loadCorpusFromText(readFileSync(new URL(f, dir), 'utf8'), f), ); - it('every corpus line parses and there are at least 12 entries', () => { + it('every corpus line parses and there are at least 14 entries', () => { assert.ok(files.length > 0, 'corpus dir must contain .jsonl files'); - assert.ok(entries.length >= 12, `expected >= 12 entries, got ${entries.length}`); + assert.ok(entries.length >= 14, `expected >= 14 entries, got ${entries.length}`); }); it('entry ids are unique', () => { @@ -181,6 +381,28 @@ describe('goldenRunner/corpus integrity (#129 acceptance, key-free)', () => { assert.ok(hasToolPostcondition, 'need a tool_postcondition (#130) fixture'); assert.ok(hasCitationPath, 'need a citation_missing (#131) fixture'); }); + + it('covers the deterministic verified (Gap 1) and contradicted (Gap 2) paths (#639)', () => { + const verifiedEntry = entries.find( + (e) => e.expected.via === 'deterministic-verified', + ); + const contradictedEntry = entries.find( + (e) => e.expected.via === 'deterministic-contradicted', + ); + assert.ok(verifiedEntry, 'need a deterministic-verified -> approved fixture (Gap 1)'); + assert.ok(contradictedEntry, 'need a deterministic-contradicted -> blocked fixture (Gap 2)'); + // Both paths need an odoo fixture to re-query and a declared Odoo call so the + // trace-cross-check does not pre-empt the deterministic checker. + for (const e of [verifiedEntry, contradictedEntry]) { + assert.ok((e.odoo?.records.length ?? 0) > 0, `${e.id}: expected an odoo fixture`); + assert.ok( + e.trace?.domainToolsCalled?.some((t) => t.startsWith('query_odoo_')), + `${e.id}: expected a query_odoo_* call in the trace`, + ); + } + assert.equal(verifiedEntry.expected.status, 'approved'); + assert.equal(contradictedEntry.expected.status, 'blocked'); + }); }); describe('goldenRunner/hasRegression + formatSummary', () => {