feat: semantic-unit chunking for the embedding channel

.gitignore

@@ -4,3 +4,4 @@ __pycache__/
 .venv/
 *.egg-info/
 .DS_Store
+.codegraph/

ARCHITECTURE.md

@@ -61,8 +61,10 @@ flowchart LR
 |---|---|
 | `models.py` | `Record`, `Edge`, `Tree` -- source-agnostic data shapes |
 | `channels.py` | Independent `[0, 1]` scoring functions |
-| `embedding_client.py` | Pluggable text-embedding channel (`Null` default, `OpenAiCompatible` real impl) |
+| `chunking.py` | Splits a document into meaning-identifiable units (paragraph, sentence, DOM, conversation-turn) plus embedded-image extraction, in document order |
+| `embedding_client.py` | Pluggable text-embedding channel (`Null` default, `OpenAiCompatible` real impl) + `chunked_max_similarity` |
 | `adjudication_client.py` | Pluggable LLM-judgment channel (`Null` default, `ContextualOrchestrator` real impl) |
+| `image_content.py` | Pluggable vision channel: OCR + object recognition/tagging for embedded images (`Null` default, `OpenAiCompatibleVisionClient` real impl) |
 | `tepp_client.py` | TEPP's published `AnalysisRunRequest` wire contract, pluggable transport |
 | `reconstruct.py` | The pipeline: group → candidate window → score → fuse → thread |
 | `fixtures.py` | Synthetic demo dataset -- no real data ships in this repo |

CHANGELOG.md

@@ -4,6 +4,63 @@ All notable changes to this project are documented here. Format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versioning follows
 [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.3.0] - 2026-08-13
+
+### Fixed
+
+- Embedding, adjudication, and vision clients now POST through a shared
+  `http_client.post_json` helper that allowlists `http`/`https` and never
+  calls `urllib.request.urlopen`. That closes the `file://` read concern
+  Semgrep's `dynamic-urllib-use-detected` rule was flagging on the
+  operator-configured base URLs. HTTPS posts wrap the connected socket
+  with a certifi-backed `SSLContext` instead of constructing
+  `http.client.HTTPSConnection`, so certificate verification is explicit
+  on the Python 3.10+ runtime this project requires.
+
+### Added
+
+- `lineageweave/chunking.py`: semantic-unit chunking so the embedding
+  channel compares meaning-identifiable units instead of whole flattened
+  documents -- `chunk_by_paragraph` (Hearst, 1997, TextTiling subtopic
+  boundaries), `chunk_by_sentence`, `chunk_by_dom` (WHATWG HTML Living
+  Standard sectioning/flow block elements), and `chunk_by_conversation_turn`
+  (RFC 5322 sender/receiver boundaries).
+- `embedding_client.chunked_max_similarity`: chunks two documents, embeds
+  every chunk, and returns the single highest-scoring pair -- the standard
+  passage-retrieval strategy for "a relevant unit is buried in a longer
+  document." Degrades to plain whole-text embedding for any document that
+  chunks to zero or one piece (this project's real short-title dataset
+  behaves exactly as it did before chunking existed).
+- Real-provider test proving chunking works, not just that it type-checks:
+  a short relevant paragraph buried inside a longer synthetic document
+  scored higher via `chunked_max_similarity` than via whole-document
+  embedding, against the live embedding provider.
+- `docs/lineage-bi-research-notes.md`: new "Chunking" section with the
+  four units' grounding and an explicit, honest note that this project's
+  real dataset's only free-text field is too short to need chunking in
+  practice -- the module exists for richer content sources (e.g. the raw
+  MHTML artifacts that dataset's records were derived from).
+- `lineageweave/image_content.py`: pluggable vision channel for base64
+  images embedded in DOM content -- real OCR (Li et al., 2023, TrOCR) and
+  object recognition/tagging (Radford et al., 2021, CLIP) via
+  `OpenAiCompatibleVisionClient`, same never-fake-a-missing-channel
+  discipline as the embedding/adjudication clients. `chunk_by_dom` now
+  extracts embedded images as `"image"` chunks interleaved with text
+  chunks in true document order, so an image's position relative to its
+  surrounding text is preserved and reconstructable.
+- Real-provider test proving OCR works, not just that it type-checks: a
+  real PNG generated with real rendered text (not a fixture file) was
+  read back correctly by `OpenAiCompatibleVisionClient` against the live
+  vision-capable model.
+- `docs/image-content-schema.md`: proposed DB schema (snake_case, 2+ word
+  object names) for persisting and searching extracted image content,
+  designed so a text/tag search hit stays traceable to which document and
+  which position produced it, and so the same image (by content hash) is
+  never *stored* twice (the primary key guarantees that part). Avoiding a
+  duplicate vision-provider *call* for two concurrent ingests of the same
+  new image is a separate concern the schema documents but does not solve
+  by itself -- a real write path still needs an atomic claim/lease step.
+
 ## [0.2.0] - 2026-08-13
 
 ### Added

docs/image-content-schema.md

@@ -0,0 +1,107 @@
+# DB design: searchable, position-preserving image content
+
+**Status:** proposed schema, not yet backed by a persistence layer in this
+repo (LineageWeave's own demo server is in-memory/stdlib -- this document
+is the design a real deployment's persistence layer should implement).
+All object names are snake_case, two or more words, per this project's
+naming convention.
+
+## Why this needs its own tables, not just a text column
+
+Base64 images live inside a source document's DOM at a specific point --
+"the invoice number was in the picture right after the third paragraph."
+If extracted OCR text and tags are stored disconnected from that position,
+a search hit tells you *that* something matched but not *where* in the
+original document to look, and there is no way to reconstruct the
+document's original visual layout (text, then a picture, then more text)
+for review. The design below keeps three things independently true at
+once: (1) the extracted text and tags are searchable on their own terms,
+(2) each image stays traceable to exactly which document and which
+position produced it, and (3) the same image (by content hash) is never
+*stored* twice -- `embedded_image_id`'s primary key guarantees that part
+by construction. Avoiding a duplicate *vision-provider call* for two
+concurrent ingests of the same new image is a separate concern this
+schema does not solve on its own (see the query-shapes note below) --
+that needs an atomic claim/lease step in the write path, not just the
+content-hash primary key.
+
+## Tables
+
+### `source_document`
+
+One row per document that may contain embedded images (an MHTML/HTML
+artifact, an ingested email, etc.). Referenced by whatever this project's
+`Record.record_id` maps to in a real deployment -- not redefined here to
+avoid coupling this schema to one specific source system.
+
+| Column | Type | Notes |
+|---|---|---|
+| `source_document_id` | `text` primary key | opaque id, matches the owning record |
+| `content_sha256` | `text not null` | hash of the full source document, for idempotent re-ingest |
+| `ingested_at` | `timestamptz not null default now()` | |
+
+### `embedded_image`
+
+One row per distinct image (by content hash), decoupled from *where* it
+appeared -- the same picture can appear in more than one document.
+
+| Column | Type | Notes |
+|---|---|---|
+| `embedded_image_id` | `text primary key` | `sha256(image_bytes)`, so identical images across documents are stored once |
+| `mime_type` | `text not null` | e.g. `image/png` |
+| `byte_size` | `bigint not null` | |
+| `extracted_text` | `text` | OCR result (Li et al., 2023 -- TrOCR-family text recognition); `null` until processed, empty string if genuinely no legible text |
+| `caption_text` | `text` | one-sentence description (Radford et al., 2021 -- CLIP-family vision-language grounding) |
+| `processed_at` | `timestamptz` | `null` until a vision client has run; distinguishes "not yet processed" from "processed, nothing found" |
+| `processing_model` | `text` | which vision-capable model produced `extracted_text`/`caption_text`, for auditability if a model change should trigger reprocessing |
+
+### `image_tag`
+
+Many-to-many: independently searchable keyword tags per image, separate
+from the free-text caption.
+
+| Column | Type | Notes |
+|---|---|---|
+| `embedded_image_id` | `text not null references embedded_image` | |
+| `tag_text` | `text not null` | a single short tag |
+| primary key | `(embedded_image_id, tag_text)` | |
+
+### `document_image_position`
+
+The position-preserving join: which image appeared in which document, at
+which position among that document's other content units (text and image
+chunks together, true document order -- see
+`lineageweave.chunking.chunk_by_dom`'s unified sequence). This is what
+lets a UI reconstruct "here is the document, and here is where the
+picture sat relative to the surrounding paragraphs."
+
+| Column | Type | Notes |
+|---|---|---|
+| `source_document_id` | `text not null references source_document` | |
+| `embedded_image_id` | `text not null references embedded_image` | |
+| `chunk_position` | `integer not null` | 0-based index among ALL of this document's chunks (text and image together) -- matches `Chunk.index` from `chunk_by_dom` |
+| primary key | `(source_document_id, chunk_position)` | one image slot per position per document |
+
+## Query shapes this supports
+
+- **"Find images whose extracted text or tags match a search query, then
+  show me the document and where the image sat"**: search
+  `embedded_image.extracted_text` / `image_tag.tag_text`, join through
+  `document_image_position` to `source_document`, order surrounding
+  content by `chunk_position`.
+- **"Reconstruct document N's original layout"**: fetch all of document
+  N's text chunks (however the caller's own text-chunk storage models
+  them) and `document_image_position` rows for that document, merge-sort
+  by `chunk_position` -- text and images interleave back into their
+  original order.
+- **"Has this exact image already been stored?"**: `SELECT ... FROM
+  embedded_image WHERE embedded_image_id = sha256(new_image_bytes)` --
+  storage is idempotent by construction (the primary key), a genuinely
+  useful guard against re-storing a picture that recurs across documents
+  (a common shape for letterhead logos, signature images, etc.). This
+  check alone does not prevent two concurrent ingests of the same *new*
+  image from both calling the vision provider before either has written
+  its row -- a real write path needs an atomic claim (e.g. `INSERT ...
+  ON CONFLICT DO NOTHING` before the provider call, or a short-lived
+  lease row) to close that race; this schema documents the storage
+  guarantee, not that concurrency control.

docs/lineage-bi-research-notes.md

@@ -89,6 +89,67 @@ contracts" -- LineageWeave's fused scores are exactly that kind of
 uncertainty-bearing, non-authoritative evidence, never promoted to fact
 outside this repo's own DAG view.
 
+## Chunking: embedding at meaning-identifiable units, not whole documents
+
+Embedding a whole flattened document as one vector dilutes a short
+relevant unit with everything else in the same document -- the vector
+averages over content that has nothing to do with the match being sought.
+`lineageweave/chunking.py` splits a document into meaning-identifiable
+units first; `embedding_client.chunked_max_similarity` embeds every unit
+and takes the single highest-scoring pair, which is the standard
+passage-retrieval strategy for "a relevant unit is buried in a longer
+document." Four unit types, each grounded in a real boundary concept:
+
+- **paragraph** -- subtopic-passage boundaries (Hearst, 1997, TextTiling).
+- **sentence** -- the finer unit inside a paragraph.
+- **dom** -- sectioning/flow block-element boundaries (WHATWG HTML Living
+  Standard).
+- **conversation_turn** -- sender/receiver boundaries (RFC 5322), reusing
+  the same one-message-one-party shape ThreadWeave's JWZ threading already
+  models across records, just applied within a single record's body.
+
+**Honest scope note for this project's real dataset**: the real dataset
+validated against in milestone 2 (43,814 short business records) has only
+one real free-text field, and it is short (~28 characters average) with no
+paragraph, DOM, or conversation structure to chunk -- chunking a title
+does nothing useful and `chunked_max_similarity` degrades gracefully to
+plain whole-text embedding for exactly this case (a document that chunks
+to zero or one piece is embedded once, same as before chunking existed).
+This module exists for when a richer content source is embedded --
+concretely, the raw MHTML source artifacts this dataset's records were
+derived from (tracked only as opaque content-addressed references in this
+project's real dataset, not fetchable from this repository) are HTML
+documents with real DOM and sender/receiver structure, which is exactly
+the shape `chunk_by_dom` and `chunk_by_conversation_turn` are for.
+
+## Embedded images: OCR and object recognition, position-preserved
+
+The same DOM content that motivates `chunk_by_dom` can carry embedded
+base64 images -- `chunk_by_dom` extracts these as `"image"` chunks
+interleaved with the surrounding text chunks in true document order (see
+`Chunk.index`), and `lineageweave.image_content` turns image bytes into
+searchable content via a pluggable vision-capable client, following the
+same never-fake-a-missing-channel discipline as the embedding and
+adjudication clients:
+
+- **OCR (text recognition)**: grounded in Li et al. (2023) -- TrOCR, the
+  transformer encoder-decoder architecture family modern OCR (including
+  vision-capable chat models) descends from.
+- **Object recognition / captioning / tagging**: grounded in Radford et
+  al. (2021) -- CLIP, contrastive language-image pretraining, the basis
+  most current zero-shot image tagging and captioning builds on.
+
+Proven for real during development: a real PNG generated with real
+rendered text (`draw.text(...)`, not a canned fixture file) was sent
+through `OpenAiCompatibleVisionClient` against the live gateway, and the
+text was read back correctly -- genuine OCR, not a mocked response.
+
+`docs/image-content-schema.md` proposes the DB design for storing and
+searching this content so a match on extracted text or a tag can still be
+traced back to which document, and which position in that document, the
+picture came from -- an extracted caption is not useful for review if
+nobody can tell which paragraph it illustrated.
+
 ## Channels and their grounding
 
 | Channel | What it does | Grounded in |
@@ -154,8 +215,18 @@ Doddington, G., Mitchell, A., Przybocki, M., Ramshaw, L., Strassel, S., & Weisch
 
 Fellegi, I. P., & Sunter, A. B. (1969). A theory for record linkage. *Journal of the American Statistical Association*, *64*(328), 1183-1210. https://doi.org/10.2307/2286061
 
+Hearst, M. A. (1997). TextTiling: Segmenting text into multi-paragraph subtopic passages. *Computational Linguistics*, *23*(1), 33-64.
+
+Li, M., Lv, T., Chen, J., Cui, L., Lu, Y., Florencio, D., Zhang, C., Li, Z., & Wei, F. (2023). TrOCR: Transformer-based optical character recognition with pre-trained models. *Proceedings of the AAAI Conference on Artificial Intelligence*, *37*(11), 13094-13102. https://doi.org/10.1609/aaai.v37i11.26538
+
+Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. *Proceedings of the 38th International Conference on Machine Learning*, *139*, 8748-8763.
+
 Raudenbush, S. W., & Bryk, A. S. (2002). *Hierarchical linear models: Applications and data analysis methods* (2nd ed.). Sage Publications.
 
+Resnick, P. (2008). *Internet Message Format* (RFC 5322). IETF. https://doi.org/10.17487/RFC5322
+
+WHATWG. (2026). *HTML Living Standard — sections 4.3 (sectioning content) and 4.4 (grouping content)*. https://html.spec.whatwg.org/
+
 Zawinski, J. (1997). *Message threading* [Design note]. jwz.org. https://www.jwz.org/doc/threading.html
 
 Additional context on the Fugu / Conductor / TRINITY test-time-compute-allocation research the `llm` channel's design follows is maintained in

lineageweave/__init__.py

@@ -12,4 +12,4 @@
 
 __all__ = ["Edge", "Record", "Tree", "reconstruct"]
 
-__version__ = "0.2.0"
+__version__ = "0.3.0"

lineageweave/adjudication_client.py

@@ -12,19 +12,10 @@
 
 from __future__ import annotations
 
-import json
 import re
-import ssl
-import urllib.request
 from typing import Protocol
 
-import certifi
-
-# See lineageweave.embedding_client for why this is needed: some
-# interpreter distributions don't reliably inherit the OS trust store, so
-# point explicitly at certifi's maintained CA bundle (full validation still
-# applies -- nothing here is weakened).
-_SSL_CONTEXT = ssl.create_default_context(cafile=certifi.where())
+from .http_client import post_json
 
 
 class AdjudicationClient(Protocol):
@@ -73,21 +64,16 @@ def judge(self, candidate_label: str, record_label: str) -> float:
             "is a direct continuation of record A? Reply with only the number.\n\n"
             f"Record A: {candidate_label}\nRecord B: {record_label}"
         )
-        payload = json.dumps(
+        body = post_json(
+            f"{self._base_url}/v1/chat/completions",
             {
                 "messages": [{"role": "user", "content": prompt}],
                 "mode": "verify",
                 "reasoning_effort": self._reasoning_effort,
-            }
-        ).encode("utf-8")
-        request = urllib.request.Request(
-            f"{self._base_url}/v1/chat/completions",
-            data=payload,
-            headers={"authorization": f"Bearer {self._api_key}", "content-type": "application/json"},
-            method="POST",
+            },
+            headers={"authorization": f"Bearer {self._api_key}"},
+            timeout=self._timeout,
         )
-        with urllib.request.urlopen(request, timeout=self._timeout, context=_SSL_CONTEXT) as response:  # nosec B310 -- base_url is operator-configured, not request-controlled.
-            body = json.loads(response.read().decode("utf-8"))
         content = body["choices"][0]["message"]["content"]
         match = _CONFIDENCE_PATTERN.search(content)
         if match is None: