fix: wire content_hash pre-check into ingest paths (#254) - #284
fix: wire content_hash pre-check into ingest paths (#254)#284robotrocketscience wants to merge 1 commit into
Conversation
belief_id is sha256(source‖text); content_hash is sha256(text). The ingest path's id-based dedup check (`store.get_belief(bid)`) catches same-source re-ingest only — same content from a different source produced a parallel belief row, which is the structural cause behind the 5.3× row inflation reported in #219. Both ingest entry points now consult `get_belief_by_content_hash` after the id miss. On hit they record a corroboration against the canonical row and skip insert. `triple_extractor._resolve_or_create_belief` returns the canonical id so commit-ingest edges wire to the existing belief. `bulk=True` continues to suppress corroboration writes, matching the same-source semantics. This is the application-level (Option B) path from #254. Schema-level UNIQUE on content_hash (Option A) is deferred — it requires #219's consolidation migration to land first. Existing test_source_label_passed_through asserted the buggy behavior (two ingests of identical content under different source labels both insert); inverted to match the fix. New test_cross_source_dedup covers the canonical id, corroboration recording, and bulk suppression. Closes #254
|
[claim:review:Gylf:2026-04-29T02:17:56Z] |
|
Review: this PR is likely superseded by #283 (just merged). #283 introduced Recommend: rebase on main, verify #254's tests still hold against the #283 implementation, and either close this in favor of #283 or repurpose to add any missing coverage that #283 didn't include. (Side note: pytest 3.13 timeout in CI was on |
|
[release:review:Gylf:2026-04-29T02:18:13Z] |
|
[claim:review:Setr:2026-04-29T02:59:59Z] |
|
[release:review:Setr:2026-04-29T03:00:34Z] |
|
Closing — superseded by #283 which already landed comprehensive |
Summary
Fixes the structural cause behind issue #219's 5.3× row inflation by closing the cross-source dedup gap from #254.
belief_id = sha256(source‖text)butcontent_hash = sha256(text). The ingest path'sstore.get_belief(bid)check catches same-source re-ingest only — the same sentence from a different source produced a parallel belief row.Both ingest entry points now consult
get_belief_by_content_hashafter the id miss. On hit, they record a corroboration against the canonical row and skip insert._resolve_or_create_beliefreturns the canonical id so commit-ingest edges wire to the existing belief.bulk=Truecontinues to suppress corroboration writes, matching same-source semantics.This is the application-level (Option B) path from #254. Schema-level UNIQUE on
content_hash(Option A) is deferred — it requires consolidation of historical duplicates (issue #219) to land first.Tests
tests/test_cross_source_dedup.py: 4 cases covering no-row-inflation, corroboration against canonical id, canonical-id stability across re-ingests, and bulk suppression.tests/test_ingest_jsonl.py::test_source_label_passed_throughpreviously asserted the buggy behavior (two writes for two source labels with same content); inverted to assert the fix (second call inserts 0).Test plan
tests/test_corroborations.pyunchangedCloses #254