fix(ui): parse invoice HTML images with an HTML parser (v2.10.2)

ARCHITECTURE.md

@@ -64,7 +64,7 @@ flowchart LR
 | `chunking.py` | Splits a document into meaning-identifiable units (paragraph, sentence, DOM, conversation-turn) plus embedded-image extraction, in document order |
 | `embedding_client.py` | Pluggable text-embedding channel (`Null` default, `OpenAiCompatible` real impl) + `chunked_max_similarity` |
 | `adjudication_client.py` | Pluggable LLM-judgment channel (`Null` default, `ContextualOrchestrator` real impl) |
-| `image_content.py` | Pluggable vision channel: OCR + object recognition/tagging for embedded images (`Null` default, `OpenAiCompatibleVisionClient` real impl). The product popup (`frontend/src/PostBody.tsx`) renders each `data:image` payload in document order so the buyer sees the picture, not the base64 string; GET does not call the vision client. |
+| `image_content.py` | Pluggable vision channel: OCR + object recognition/tagging for embedded images (`Null` default, `OpenAiCompatibleVisionClient` real impl). The product popup (`frontend/src/PostBody.tsx`) and `extract_base64_images` parse with the same HTML rules as `chunk_by_dom` (ADR 0031) so invoice-like `alt` values still show the picture; GET does not call the vision client. |
 | `tepp_client.py` | TEPP's published `AnalysisRunRequest` wire contract, pluggable transport |
 | `rankweave_client.py` | Fail-closed RankWeave ranking port (`weighted_reciprocal_rank_fuse` in-process; never invent a fused score or a theta) |
 | `reconstruct.py` | The pipeline: group → candidate window → score → fuse → thread |

CHANGELOG.d/2.10.2-embedded-image-html-parser.md

@@ -0,0 +1,5 @@
+# 2.10.2 Parse invoice HTML images with an HTML parser
+
+Opening a post whose embedded picture uses invoice-like HTML
+(`alt="Invoice > 1000"`) shows the picture between the surrounding
+sentences. The raw base64 string is gone (ADR 0031).

CHANGELOG.md

@@ -4,6 +4,17 @@ All notable changes to this project are documented here. Format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versioning follows
 [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.10.2] - 2026-08-17
+
+### Fixed
+
+- Opening a post whose embedded picture uses invoice-like HTML
+  (`alt="Invoice > 1000"`, unquoted `width`, newlines in the base64)
+  now shows the picture. The raw payload no longer returns when a
+  remote-only or SVG tag is the whole body. Re-export as PNG or JPEG
+  if the type is rejected. The popup, `extract_base64_images`, and
+  `chunk_by_dom` share one raster allowlist (ADR 0031).
+
 ## [2.10.1] - 2026-08-17
 
 ### Fixed

docs/adr/0031-embedded-image-html-parser.md

@@ -0,0 +1,78 @@
+# ADR 0031 — Embedded images use an HTML parser and a raster allowlist
+
+**Decision status:** Accepted
+**Date:** 2026-08-17
+
+## Context
+
+The product popup stopped dumping a well-formed
+`data:image/png;base64,...` invoice as a base64 wall. The splitter and
+`extract_base64_images` still used a `[^>]*` regex. Real invoice HTML
+puts `>` inside `alt` or `title` *before* `src`. That shape is legal
+HTML (WHATWG, n.d.) and is what `chunk_by_dom` already parses. The regex
+missed the picture and put the payload back into the text node.
+
+The same open MIME class `image/[a-zA-Z0-9.+-]+` accepted
+`image/svg+xml`. SVG-as-`<img>` does not run script in current browsers,
+but the regex also fed the vision channel. `atob` and
+`b64decode(validate=True)` already disagreed on padding.
+
+ADR 0019 is the R&R catalog-identity decision. This decision is the
+viewer/extractor parse contract. Layout clues stay as character offsets
+and `chunk_position` rows — never raw HTML in the knowledge graph or in
+a persisted post body.
+
+Persistence of OCR under the figure (Li et al., 2023; Radford et al.,
+2021) is still the next buyer slice. It must not land on a splitter that
+fails the HTML the buyer actually opens.
+
+## Decision
+
+The popup (`splitPostBody`), `extract_base64_images`, and `chunk_by_dom`
+share one decode helper (`lineageweave.embedded_image_payload`):
+
+1. Parse with an HTML parser (`DOMParser` in the browser, `html.parser`
+   in Python). Comments, `<style>`, and `<script>` do not yield images.
+2. Accept only raster MIME types: `png`, `jpeg`/`jpg`, `gif`, `webp`,
+   `avif`. Reject SVG and non-image labels.
+3. Require strict base64 padding and a matching magic-byte signature
+   (Boutell & Randers-Pehrson, 2003, for PNG).
+4. Never fall back to the raw `post_body` when every tag strips away.
+   Tell the operator to re-export.
+5. Lock the contract with
+   `tests/fixtures/synthetic_invoice_embedded_image.html` — the same
+   file the TypeScript and Python tests read.
+
+`GET /api/posts/{id}` still returns raw `post_body`. Vision captions
+stay on Extract Keyman / Ask until a later persistence PR.
+
+## Consequences
+
+- Open a post whose invoice HTML has `alt="Invoice > 1000"`. You see
+  the picture and the surrounding sentences. The raw base64 string is gone.
+- An SVG or a remote `https://` image is not loaded. Re-export as an
+  embedded PNG or JPEG and open the post again.
+- A commented-out `<img>` or a CSS `background:url(data:image/...)`
+  does not appear as a picture and does not leak into the text.
+
+## References
+
+Boutell, T., & Randers-Pehrson, G. (Eds.). (2003). *Portable Network
+Graphics (PNG) specification (second edition)*. World Wide Web
+Consortium. https://www.w3.org/TR/2003/REC-PNG-20031110/
+
+Li, M., Lv, T., Chen, J., Cui, L., Lu, Y., Florencio, D., Zhang, C.,
+Li, Z., & Wei, F. (2023). TrOCR: Transformer-based optical character
+recognition with pre-trained models. *Proceedings of the AAAI
+Conference on Artificial Intelligence, 37*(11), 13094–13102.
+https://doi.org/10.1609/aaai.v37i11.26538
+
+Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S.,
+Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., &
+Sutskever, I. (2021). Learning transferable visual models from natural
+language supervision. In M. Meila & T. Zhang (Eds.), *Proceedings of
+the 38th International Conference on Machine Learning* (pp. 8748–8763).
+PMLR. https://proceedings.mlr.press/v139/radford21a.html
+
+WHATWG. (n.d.). *HTML living standard*.
+https://html.spec.whatwg.org/multipage/

docs/image-content-schema.md

@@ -85,14 +85,16 @@ picture sat relative to the surrounding paragraphs."
 ## Viewer contract (before persistence exists)
 
 The demo popup does not yet read these tables. It splits the live
-`post_body` the same way `extract_base64_images` does: each
-`data:image/...;base64,...` payload becomes an `<img>` at its original
-character offset, and the surrounding HTML is shown as text. A buyer who
-opens the post sees the picture that sat between the paragraphs, not the
-base64 wall. Remote `src="https://..."` tags are stripped, never fetched.
-OCR, caption, and tag search still require the vision client on extract /
-Ask (Li et al., 2023; Radford et al., 2021) and, in a real deployment,
-the tables below.
+`post_body` the same way `extract_base64_images` and `chunk_by_dom` do:
+an HTML parser walks the document, each raster `data:image/...;base64,...`
+payload becomes an `<img>` at its original character offset, and the
+surrounding HTML is shown as decoded text (ADR 0031). A buyer who opens
+an invoice-like post — including `alt="Invoice > 1000"` — sees the
+picture that sat between the paragraphs, not the base64 wall. Remote
+`src="https://..."` tags, `image/svg+xml`, HTML comments, and CSS
+`background:url` payloads are not loaded. OCR, caption, and tag search
+still require the vision client on extract / Ask (Li et al., 2023;
+Radford et al., 2021) and, in a real deployment, the tables below.
 
 ## Query shapes this supports
 
@@ -120,6 +122,10 @@ the tables below.
 
 ## References
 
+Boutell, T., & Randers-Pehrson, G. (Eds.). (2003). *Portable Network
+Graphics (PNG) specification (second edition)*. World Wide Web Consortium.
+https://www.w3.org/TR/2003/REC-PNG-20031110/
+
 Li, M., Lv, T., Chen, J., Cui, L., Lu, Y., Florencio, D., Zhang, C., Li, Z.,
 & Wei, F. (2023). TrOCR: Transformer-based optical character recognition
 with pre-trained models. *Proceedings of the AAAI Conference on Artificial
@@ -131,3 +137,6 @@ Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I.
 supervision. In M. Meila & T. Zhang (Eds.), *Proceedings of the 38th
 International Conference on Machine Learning* (pp. 8748–8763). PMLR.
 https://proceedings.mlr.press/v139/radford21a.html
+
+WHATWG. (n.d.). *HTML living standard*.
+https://html.spec.whatwg.org/multipage/

frontend/package.json

@@ -1,7 +1,7 @@
 {
   "name": "frontend",
   "private": true,
-  "version": "2.10.1",
+  "version": "2.10.2",
   "type": "module",
   "scripts": {
     "dev": "vite",

frontend/src/PostBody.tsx

@@ -13,7 +13,10 @@ function renderSegment(segment: PostBodySegment, index: number) {
         <figure key={`post-body-image-${index}`} className="post-embedded-image">
           <img
             src={segment.src}
-            alt={`Embedded image at character offset ${segment.position}`}
+            alt={
+              segment.alt ||
+              `Embedded image at character offset ${segment.position}`
+            }
           />
           <figcaption>
             Image from this post. Extract Keyman or ask a question to read text

frontend/src/postBodyDisplay.test.ts

@@ -1,10 +1,18 @@
+import { readFileSync } from "node:fs";
+import { dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
 import { describe, expect, it } from "vitest";
 import { splitPostBody } from "./postBodyDisplay";
 
 /** 1x1 transparent PNG — the same synthetic fixture the Python vision tests use. */
 const TINY_PNG_B64 =
   "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=";
 
+const INVOICE_HTML = readFileSync(
+  join(dirname(fileURLToPath(import.meta.url)), "../../tests/fixtures/synthetic_invoice_embedded_image.html"),
+  "utf8",
+);
+
 describe("splitPostBody", () => {
   it("leaves a plain-text post unchanged so existing popups keep their wording", () => {
     expect(splitPostBody("The full body text.")).toEqual([
@@ -30,6 +38,7 @@ describe("splitPostBody", () => {
         src: `data:image/png;base64,${TINY_PNG_B64}`,
         mimeType: "image/png",
         position: html.indexOf("<img"),
+        alt: "",
       },
       { kind: "text", text: "Please confirm." },
     ]);
@@ -62,6 +71,15 @@ describe("splitPostBody", () => {
     ]);
   });
 
+  it("rejects unpadded base64 the same way Python validate=True does", () => {
+    expect(splitPostBody('<img src="data:image/png;base64,YQ">')).toEqual([
+      {
+        kind: "text",
+        text: "Embedded image could not be decoded. Re-export the source post and open it again.",
+      },
+    ]);
+  });
+
   it("does not turn a remote http img into a loaded image", () => {
     const html = '<p>See</p><img src="https://example.test/invoice.png"><p>end</p>';
     const segments = splitPostBody(html);
@@ -71,4 +89,118 @@ describe("splitPostBody", () => {
     );
     expect(JSON.stringify(segments)).not.toContain("https://example.test");
   });
+
+  it("does not return the raw tag when the post is only a remote image", () => {
+    const segments = splitPostBody('<img src="https://example.test/invoice.png">');
+    expect(segments.every((segment) => segment.kind === "text")).toBe(true);
+    expect(JSON.stringify(segments)).not.toContain("https://example.test");
+    expect(segments[0]?.kind === "text" && segments[0].text).toMatch(/Re-export the source with embedded pictures/);
+  });
+
+  it("does not render image/svg+xml as a picture", () => {
+    const html =
+      '<img src="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==">';
+    const segments = splitPostBody(html);
+    expect(segments.every((segment) => segment.kind === "text")).toBe(true);
+    expect(JSON.stringify(segments)).not.toContain("image/svg+xml");
+    expect(segments[0]?.kind === "text" && segments[0].text).toMatch(/Re-export as PNG or JPEG/);
+  });
+
+  it("renders jpeg gif webp and avif when the magic bytes match", () => {
+    const jpeg = btoa(String.fromCharCode(0xff, 0xd8, 0xff, 0x00));
+    const gif87 = btoa("GIF87a\0\0");
+    const gif89 = btoa("GIF89a\0\0");
+    const webp = btoa("RIFF\0\0\0\0WEBP");
+    const avif = btoa("\0\0\0\0ftypavif\0\0\0\0");
+    const avis = btoa("\0\0\0\0ftypavis\0\0\0\0");
+    const mif1 = btoa("\0\0\0\0ftypmif1\0\0\0\0");
+    expect(splitPostBody(`<img src="data:image/jpeg;base64,${jpeg}">`)[0]).toMatchObject({
+      kind: "image",
+      mimeType: "image/jpeg",
+    });
+    expect(splitPostBody(`<img src="data:image/jpg;base64,${jpeg}">`)[0]).toMatchObject({
+      kind: "image",
+      mimeType: "image/jpg",
+    });
+    expect(splitPostBody(`<img src="data:image/gif;base64,${gif87}">`)[0]).toMatchObject({
+      kind: "image",
+      mimeType: "image/gif",
+    });
+    expect(splitPostBody(`<img src="data:image/gif;base64,${gif89}">`)[0]).toMatchObject({
+      kind: "image",
+      mimeType: "image/gif",
+    });
+    expect(splitPostBody(`<img src="data:image/webp;base64,${webp}">`)[0]).toMatchObject({
+      kind: "image",
+      mimeType: "image/webp",
+    });
+    expect(splitPostBody(`<img src="data:image/avif;base64,${avif}">`)[0]).toMatchObject({
+      kind: "image",
+      mimeType: "image/avif",
+    });
+    expect(splitPostBody(`<img src="data:image/avif;base64,${avis}">`)[0]).toMatchObject({
+      kind: "image",
+      mimeType: "image/avif",
+    });
+    expect(splitPostBody(`<img src="data:image/avif;base64,${mif1}">`)[0]).toMatchObject({
+      kind: "image",
+      mimeType: "image/avif",
+    });
+  });
+
+  it("does not load a protocol-relative remote image", () => {
+    const segments = splitPostBody('<img src="//example.test/invoice.png">');
+    expect(segments[0]?.kind === "text" && segments[0].text).toMatch(/Remote images were not loaded/);
+  });
+
+  it("returns no segments for empty markup or an image tag without a src", () => {
+    expect(splitPostBody("<div></div>")).toEqual([]);
+    expect(splitPostBody("<img>")).toEqual([]);
+  });
+
+  it("rejects a padded payload whose bytes are not a raster picture", () => {
+    expect(splitPostBody('<img src="data:image/png;base64,AAAA">')).toEqual([
+      {
+        kind: "text",
+        text: "Embedded image could not be decoded. Re-export the source post and open it again.",
+      },
+    ]);
+    expect(splitPostBody('<img src="data:image/jpeg;base64,AAAA">')[0]).toMatchObject({
+      kind: "text",
+    });
+    expect(splitPostBody('<img src="data:image/gif;base64,AAAA">')[0]).toMatchObject({
+      kind: "text",
+    });
+    expect(splitPostBody('<img src="data:image/webp;base64,AAAA">')[0]).toMatchObject({
+      kind: "text",
+    });
+    expect(splitPostBody('<img src="data:image/avif;base64,AAAA">')[0]).toMatchObject({
+      kind: "text",
+    });
+  });
+
+  it("keeps the picture when alt contains > and does not leak the invoice fixture", () => {
+    const segments = splitPostBody(INVOICE_HTML);
+    const images = segments.filter((segment) => segment.kind === "image");
+    const text = segments
+      .filter((segment) => segment.kind === "text")
+      .map((segment) => (segment.kind === "text" ? segment.text : ""))
+      .join(" ");
+
+    expect(images).toHaveLength(1);
+    expect(images[0]).toMatchObject({
+      kind: "image",
+      mimeType: "image/png",
+      alt: "Invoice > 1000",
+      src: `data:image/png;base64,${TINY_PNG_B64}`,
+    });
+    expect(text).toContain("Quote attached.");
+    expect(text).toContain("Terms & conditions.");
+    expect(text).toContain("Qty");
+    expect(text).toContain("Please confirm.");
+    expect(text).not.toContain(TINY_PNG_B64);
+    expect(text).not.toContain("data:image");
+    expect(text).not.toContain("example.test");
+    expect(text).not.toContain("background:url");
+  });
 });