fix(verifier): give the judge the sentence a fragment claim was cut from - #831
Merged
Conversation
golden-eval flaked on `blocked_contradiction_role` (judge path). Measured 12 local runs: 2 misses, both with the qualitative claim extracted as the subject-less fragment "in die IT-Abteilung". The EvidenceJudge never sees the answer by design, so it cannot tell who moved where and correctly says `unverified`. With the full sentence as claim text both double-check votes agreed in 100 % of runs — the AND gate is not the problem, the fragment is. Fix: - `Claim.context`: enclosing sentence of the answer, cut deterministically (`.`/`!`/`?` + whitespace, or newline; "01.03.2023" stays intact; 400-char window around the span). Attached by the extractor when it adds something beyond `text`. - EvidenceJudge passes it as a `CONTEXT:` line with an explicit disambiguation-only rule; the contradiction double-check is unchanged. - Extractor prompt: qualitative claims must be self-contained. Verification: +8 unit tests (claimContext boundaries/caps, extractor attaches context, judge prompt includes/omits CONTEXT); verifier tests 65/65; middleware suite 7192/0 fail; golden corpus 15/15 first-sample; role case 12/12 `blocked` with exactly 2 judge calls each (before 10/12).
Review findings on PR #831: - MEDIUM: first occurrence won — a fragment present in two sentences got the wrong subject attached, the one new false-contradiction vector. Now: span in more than one sentence ⇒ no context (old behaviour). - MEDIUM: ordinals/abbreviations ("1. März", "z.B.", "Dr.") split the sentence and dropped the subject. Dot after a number or a known abbreviation is no longer a boundary; context equal to the claim modulo trailing punctuation is suppressed. - LOW: lower-casing can change code-unit length (U+0130) and shift offsets — bail out when lengths differ. - LOW: judge rule no longer lists "date" as something to resolve from CONTEXT and forbids verdicts based on context-only facts. - MAX_CONTEXT_CHARS shared between extractor and judge. - Tests: multi-occurrence, z.B./ordinal/Dr., bullets, needle with trailing dot, U+0130. Verification: verifier tests green, suite 7196/0 fail, golden corpus 2×15/15 first-sample, both judge cases 8/8 blocked.
…-flake # Conflicts: # docs/CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Follow-up to #781.
golden-eval.ymlstill flaked occasionally on the judge-path cases (blocked_contradiction_role, majority-of-3 caught it). Measured locally (12 runs): 2 misses — in both the extractor emitted the qualitative claim as the subject-less fragment "in die IT-Abteilung". TheEvidenceJudgenever sees the answer by design, so it cannot know who moved where and (correctly) returnsunverified.Whenever the claim text was the full sentence, both double-check votes said
contradictedin 100 % of runs — the 2-of-2 AND gate is not the problem and is left untouched.Fix
Claim.context./!/?+ whitespace, or newline →01.03.2023stays intact; 400-char window around the span). Attached by the extractor only when it adds something beyondtext.EvidenceJudgeCONTEXT:line; system rule: disambiguation only, judge the CLAIM as meant in that sentence.Verification
claimContextboundaries/caps, extractor attaches context, judge prompt includes/omits CONTEXT)npm test(middleware)npm run typecheck/typecheck:testnpm run eval:goldenlocallyblocked_contradiction_role×12 againstclaude-haiku-4-5blocked, exactly 2 judge calls each (before: 10/12, 2–4 calls)Review round (omadia-reviewer) →
0bc991eb1. März,z.B.,Dr.) split the sentencetoLowerCase()can shift offsets (U+0130)MAX_CONTEXT_CHARSshared; notedPost-review: suite 7196/0 · golden 2×15/15 first-sample · role+status cases 8/8 each.