Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,10 @@ contextual-orchestrator owns model discovery and selection.
from derived semantic text, while retaining the source body and meaningful
list/heading nesting. A buyer-facing post view must render semantic
paragraphs, not the authoring application's spacing workaround.
- Quantity HTML `<sup>`/`<sub>` and caret exponents such as `m^3` become
Unicode in derived units and React `<sup>`/`<sub>` in the post view
(ADR 0165). Never assign the body to `innerHTML`. Do not treat
`qty < 50` or a leading footnote `^1` as an exponent.
- Image descriptions, OCR text, and region evidence are analysis artifacts,
not buyer-facing prompt instructions. Buyer UI shows the source content and
useful captions/evidence only, with provenance where appropriate.
Expand Down
8 changes: 8 additions & 0 deletions CHANGELOG.d/2.12.18-quantity-superscripts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# 2.12.18 — Quantity superscript display

## Fixed

- Post popups now show cubic metres and similar quantities as superscripts
and subscripts (`12 m³`, `H₂O`) instead of flattened `m^3` or `m3`.
Semantic units store Unicode so embeddings keep the exponent. Comparison
operators and leading footnote carets stay literal (ADR 0165).
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,14 @@ All notable changes to this project are documented here. Format follows
environment, so local OIDC and synthetic-data workflows resolve the same
pinned dependencies as CI.

## [2.12.18] - 2026-08-24

### Fixed

- Quantity superscripts and subscripts (`m³`, `H₂O`) now render as
text-level runs in the post popup and persist as Unicode in semantic
units. Comparison operators and leading footnote carets stay literal.

## [2.12.17] - 2026-08-24

### Added
Expand Down
55 changes: 55 additions & 0 deletions docs/adr/0165-quantity-script-display.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# ADR 0165: Render quantity superscripts as text runs, Unicode in units

- Status: Accepted
- Date: 2026-08-22
- Depends on: [0061](0061-post-body-character-reference-decoding.md), [0062](0062-semantic-unit-embedding.md), [0102](0102-semantic-source-unit-boundaries.md)

## Context

Imported posts write cubic metres and similar quantities as HTML
`<sup>`/`<sub>` or as caret exponents (`m^3`). The display path converted
`<sup>` into a caret and then rendered the paragraph as a React text node,
so buyers saw `m^3` instead of a superscript. The DOM chunker dropped the
tags entirely, so embeddings received concatenated `m3`, which is a
different quantity. Comparison operators such as `qty < 50` must remain
plain text, and a leading footnote caret (`^1 …`) is not a unit exponent.

## Decision

- Preserve the source body. Derived semantic text maps HTML `<sup>`/`<sub>`
and quantity caret exponents onto Unicode Super/Subscript characters
(The Unicode Consortium, 2024, §22.4) so search and embeddings can tell
`m³` from `m3` without keeping markup in the unit (Cai, Yu, Wen, & Ma,
2003).
- The buyer post view splits those Unicode (or leftover caret) runs and
renders them as React `<sup>`/`<sub>` elements. The body is never
assigned to `innerHTML` (ADR 0061).
- Only a letter, digit, or closing `)` immediately followed by `^` and a
short numeric/`n` exponent is treated as a quantity. A leading `^1`
footnote marker and comparison operators stay literal.
- Unmapped script runs keep a caret or underscore so they remain visible
rather than silently concatenating. Full formula ontology remains out of
scope; this decision covers quantity display and unit-level text.

## Consequences

- The post popup shows `12 m³` and `H₂O` as superscripts and subscripts.
- Newly persisted `post_content_unit` text stores Unicode quantities, so
later retrieval does not depend on HTML surviving the chunker.
- Existing concatenated `m3` units stay until re-ingestion; caret-form
units still render through the display splitter.

## References

Cai, D., Yu, S., Wen, J.-R., & Ma, W.-Y. (2003). *VIPS: A vision-based
page segmentation algorithm* (Microsoft Research Technical Report
MSR-TR-2003-79). Microsoft Research.

International Organization for Standardization. (2022). *Quantities and
units — Part 1: General* (ISO 80000-1:2022).

The Unicode Consortium. (2024). *The Unicode Standard* (Version 16.0.0).
https://www.unicode.org/versions/Unicode16.0.0/

WHATWG. (n.d.). *HTML living standard: The `sub` and `sup` elements*.
https://html.spec.whatwg.org/multipage/text-level-semantics.html#the-sub-and-sup-elements
2 changes: 1 addition & 1 deletion docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ decision from them.
| Supporting document | Normative ADR |
|---|---|
| [`product-technical-gap-baseline.md`](../product-technical-gap-baseline.md) | Product/technical traceability projection across the ADR set; ADRs remain normative |
| [`lineage-bi-research-notes.md`](../lineage-bi-research-notes.md) | [0084](0084-lineage-research-grounding.md), [0062](0062-semantic-unit-embedding.md), [0064](0064-lineage-evidence-and-tree-assembly.md), [0024](0024-rankweave-fusion-fail-closed.md), [0167](0167-rankweave-ranking-channel-evidence.md) |
| [`lineage-bi-research-notes.md`](../lineage-bi-research-notes.md) | [0084](0084-lineage-research-grounding.md), [0062](0062-semantic-unit-embedding.md), [0064](0064-lineage-evidence-and-tree-assembly.md), [0024](0024-rankweave-fusion-fail-closed.md), [0165](0165-quantity-script-display.md), [0167](0167-rankweave-ranking-channel-evidence.md) |
| [`PROV_O_IMPLEMENTATION.md`](../PROV_O_IMPLEMENTATION.md) | [0065](0065-prov-o-provenance-boundary.md) |
| [`PROV_O_IMPLEMENTATION_MATRIX.md`](../PROV_O_IMPLEMENTATION_MATRIX.md) | [0065](0065-prov-o-provenance-boundary.md) |
| [`ONTOLOGY_NAMESPACE_INVENTORY.md`](../doctoring/ONTOLOGY_NAMESPACE_INVENTORY.md) | [0157](0157-public-ontology-namespace-identity.md) |
Expand Down
18 changes: 18 additions & 0 deletions docs/lineage-bi-research-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,8 @@ Anagnostopoulos, E., Batsakis, S., & Petrakis, E. G. M. (2013). CHRONOS: A reaso

Browne, W. J., Goldstein, H., & Rasbash, J. (2001). Multiple membership multiple classification (MMMC) models. *Statistical Modelling*, *1*(2), 103-124. https://doi.org/10.1177/1471082X0100100202

Cai, D., Yu, S., Wen, J.-R., & Ma, W.-Y. (2003). *VIPS: A vision-based page segmentation algorithm* (Microsoft Research Technical Report MSR-TR-2003-79). Microsoft Research.

Chang, J., & Blei, D. M. (2009). Relational topic models for document networks. In D. van Dyk & M. Welling (Eds.), *Proceedings of the 12th International Conference on Artificial Intelligence and Statistics* (pp. 81-88). PMLR.

Christen, P. (2012). *Data matching: Concepts and techniques for record linkage, entity resolution, and duplicate detection*. Springer. https://doi.org/10.1007/978-3-642-31164-2
Expand All @@ -225,6 +227,8 @@ Gildea, D., & Jurafsky, D. (2002). Automatic labeling of semantic roles. *Comput

Hearst, M. A. (1997). TextTiling: Segmenting text into multi-paragraph subtopic passages. *Computational Linguistics*, *23*(1), 33-64.

International Organization for Standardization. (2022). *Quantities and units — Part 1: General* (ISO 80000-1:2022).

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, & H. Lin (Eds.), *Advances in Neural Information Processing Systems* (Vol. 33, pp. 9459-9474). Curran Associates.

Li, M., Lv, T., Chen, J., Cui, L., Lu, Y., Florencio, D., Zhang, C., Li, Z., & Wei, F. (2023). TrOCR: Transformer-based optical character recognition with pre-trained models. *Proceedings of the AAAI Conference on Artificial Intelligence*, *37*(11), 13094-13102. https://doi.org/10.1609/aaai.v37i11.26538
Expand All @@ -241,6 +245,8 @@ Sun, Q., Yuan, J., He, S., Guan, X., Yuan, H., Fu, X., Li, J., & Yu, P. S. (2025

Tong, H., Faloutsos, C., & Pan, J.-Y. (2006). Fast random walk with restart and its applications. *Proceedings of the Sixth International Conference on Data Mining (ICDM'06)*, 613-622. https://doi.org/10.1109/ICDM.2006.70

The Unicode Consortium. (2024). *The Unicode Standard* (Version 16.0.0). https://www.unicode.org/versions/Unicode16.0.0/

Wang, Q., Fu, Y., Cao, Y., Wang, S., Tian, Z., & Ding, L. (2023). *Recursively summarizing enables long-term dialogue memory in large language models*. arXiv. https://arxiv.org/abs/2308.15022

WHATWG. (2026). *HTML Living Standard — sections 4.3 (sectioning content) and 4.4 (grouping content)*. https://html.spec.whatwg.org/
Expand Down Expand Up @@ -379,3 +385,15 @@ summarize-and-replace older turns instead of an unbounded transcript or a
hard truncation that silently drops earlier decisions. This is recorded
here as the citation this feature would build on, not as a claim that
conversation-level compression is implemented today.

## Quantity scripts in source units (ADR 0165)

Board exports write cubic metres as HTML `<sup>` or as `m^3`. Flattening
those tags concatenates `m3`, which is a different quantity, and leaving
the caret in the buyer view hides the exponent. Derived units map a short
HTML/caret exponent onto Unicode Super/Subscript characters (The Unicode
Consortium, 2024, §22.4) so embeddings keep the unit (Cai, Yu, Wen, & Ma,
2003) while the post view renders React `<sup>`/`<sub>` instead of
`innerHTML`. ISO 80000-1 treats the exponent on a unit symbol as part of
the quantity, not decoration. Comparison operators and a leading footnote
caret stay literal. Full formula ontology is still open.
2 changes: 1 addition & 1 deletion frontend/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "frontend",
"private": true,
"version": "2.12.17",
"version": "2.12.18",
"type": "module",
"scripts": {
"dev": "vite",
Expand Down
19 changes: 19 additions & 0 deletions frontend/src/App.css
Original file line number Diff line number Diff line change
Expand Up @@ -350,6 +350,25 @@
white-space: pre-wrap;
}

.post-body-text sup,
.post-body-text sub,
.post-body-table sup,
.post-body-table sub {
font-size: 0.75em;
line-height: 0;
font-weight: 600;
}

.post-body-text sup,
.post-body-table sup {
vertical-align: super;
}

.post-body-text sub,
.post-body-table sub {
vertical-align: sub;
}

.post-embedded-image {
margin: 0;
padding: var(--post-image-padding);
Expand Down
118 changes: 118 additions & 0 deletions frontend/src/PostBody.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,101 @@ describe("PostBody", () => {
expect(screen.getByText("Embedded image")).toBeInTheDocument();
});

it("renders raw and persisted encoded non-script markup as the same inert text", () => {
const encoded =
"Keep &lt;b&gt;bold&lt;/b&gt;, &lt;sup-note&gt;2&lt;/sup-note&gt;, " +
"&lt;sub:item&gt;3&lt;/sub:item&gt;, and &lt;script&gt;alert(1)&lt;/script&gt; literal.";
const visible =
"Keep <b>bold</b>, <sup-note>2</sup-note>, <sub:item>3</sub:item>, and <script>alert(1)</script> literal.";
const { container, rerender } = render(<PostBody body={`<p>${encoded}</p>`} />);

expect(screen.getByText(visible)).toBeInTheDocument();
expect(container.querySelector("b")).toBeNull();
expect(container.querySelector("script")).toBeNull();
expect(container.querySelector("sup-note")).toBeNull();

rerender(
<PostBody
body={`<p>${encoded}</p>`}
structureUnits={[
{
unit_index: 0,
unit_kind_code: "plain_text",
unit_text: encoded,
indent_level: 0,
indent_source_code: "explicit",
indent_confidence: 1,
indent_evidence: "Synthetic encoded source",
},
]}
/>,
);

expect(screen.getByText(visible)).toBeInTheDocument();
expect(container.querySelector("b")).toBeNull();
expect(container.querySelector("script")).toBeNull();
expect(container.querySelector("sup-note")).toBeNull();
});

it("renders raw and legacy persisted encoded scripts with the same semantics", () => {
const encoded =
"Volume x&lt;sup&gt;2&lt;/sup&gt;, coolant H&lt;sub&gt;2&lt;/sub&gt;O, and area m&amp;#94;3.";
const { container, rerender } = render(<PostBody body={`<p>${encoded}</p>`} />);

expect([...container.querySelectorAll("sup")].map((node) => node.textContent)).toEqual([
"2",
"3",
]);
expect(container.querySelector("sub")?.textContent).toBe("2");

rerender(
<PostBody
body={`<p>${encoded}</p>`}
structureUnits={[
{
unit_index: 0,
unit_kind_code: "plain_text",
unit_text: encoded,
indent_level: 0,
indent_source_code: "explicit",
indent_confidence: 1,
indent_evidence: "Synthetic legacy persisted unit",
},
]}
/>,
);

expect([...container.querySelectorAll("sup")].map((node) => node.textContent)).toEqual([
"2",
"3",
]);
expect(container.querySelector("sub")?.textContent).toBe("2");
});

it("normalizes legacy encoded scripts in persisted table cells", () => {
const { container } = render(
<PostBody
body="<table><tr><td>Measure</td><td>Volume</td></tr></table>"
structureUnits={[
{
unit_index: 0,
unit_kind_code: "table_row",
unit_label: "tr",
unit_text: "Measure | 12 m&lt;sup&gt;3&lt;/sup&gt;",
indent_level: 0,
indent_source_code: "explicit",
indent_confidence: 1,
indent_evidence: "Synthetic legacy persisted table row",
},
]}
/>,
);

const superscript = container.querySelector("td sup");
expect(superscript?.textContent).toBe("3");
expect(superscript?.closest("td")?.textContent).toBe("12 m3");
});

it("renders authoritative LLM structure levels for semantic list units", () => {
render(
<PostBody
Expand Down Expand Up @@ -318,4 +413,27 @@ describe("PostBody", () => {
expect(screen.getByText("Before").compareDocumentPosition(screen.getByAltText("Source diagram")) & Node.DOCUMENT_POSITION_FOLLOWING).toBeTruthy();
expect(screen.getByAltText("Source diagram").compareDocumentPosition(screen.getByText("After")) & Node.DOCUMENT_POSITION_FOLLOWING).toBeTruthy();
});

it("renders quantity superscripts as text-level sup without using innerHTML", () => {
const { container } = render(<PostBody body="<p>Tank volume is 12 m<sup>3</sup>.</p>" />);

const paragraph = container.querySelector("p.post-body-text");
const superscript = paragraph?.querySelector("sup");
expect(superscript?.textContent).toBe("3");
expect(paragraph?.textContent).toBe("Tank volume is 12 m3.");
});

it("renders caret exponents and subscripts from mixed source text", () => {
const { container } = render(<PostBody body="Coolant is H<sub>2</sub>O at 12 m^3." />);

expect(container.querySelector("sub")?.textContent).toBe("2");
expect(container.querySelector("sup")?.textContent).toBe("3");
});

it("keeps comparison operators visible as ordinary text", () => {
render(<PostBody body="Need delivery if qty < 50 and price > 10." />);

expect(screen.getByText("Need delivery if qty < 50 and price > 10.")).toBeInTheDocument();
expect(document.querySelector("sup")).not.toBeInTheDocument();
});
});
40 changes: 32 additions & 8 deletions frontend/src/PostBody.tsx
Original file line number Diff line number Diff line change
@@ -1,7 +1,25 @@
import { splitPostBody, type PostBodySegment } from "./postBodyDisplay";
import {
decodeHtmlEntities,
splitPostBody,
splitScriptRuns,
normalizeScriptText,
type PostBodySegment,
} from "./postBodyDisplay";
import { t } from "./i18n";
import type { PostContentUnit, PostImageContent } from "./api";
import type { ReactNode } from "react";
import { Fragment, type ReactNode } from "react";

function renderStyledText(text: string): ReactNode {
return splitScriptRuns(text).map((run, index) => {
if (run.script === "super") {
return <sup key={`post-body-sup-${index}`}>{run.text}</sup>;
}
if (run.script === "sub") {
return <sub key={`post-body-sub-${index}`}>{run.text}</sub>;
}
return <Fragment key={`post-body-text-run-${index}`}>{run.text}</Fragment>;
});
}

function parsePipeDelimitedTable(text: string): string[][] | null {
const rows = text
Expand All @@ -21,14 +39,14 @@ function parsePipeDelimitedTable(text: string): string[][] | null {

function renderImageText(text: string) {
const rows = parsePipeDelimitedTable(text);
if (!rows) return <p>{text}</p>;
if (!rows) return <p>{renderStyledText(text)}</p>;
Comment thread
seonghobae marked this conversation as resolved.
return (
<table className="post-body-table post-image-text-table">
<tbody>
{rows.map((row, rowIndex) => (
<tr key={`post-image-text-row-${rowIndex}`}>
{row.map((cell, cellIndex) => (
<td key={`post-image-text-cell-${rowIndex}-${cellIndex}`}>{cell}</td>
<td key={`post-image-text-cell-${rowIndex}-${cellIndex}`}>{renderStyledText(cell)}</td>
))}
</tr>
))}
Comment thread
seonghobae marked this conversation as resolved.
Expand Down Expand Up @@ -102,7 +120,7 @@ function renderSegment(segment: PostBodySegment, index: number, imageContent?: P
: undefined
}
>
{segment.text}
{renderStyledText(segment.text)}
</p>
);
case "image":
Expand All @@ -128,8 +146,12 @@ function isStructuredTableRow(unit: PostContentUnit): boolean {
* its source display is still one text segment, so ordinal matching shifts
* indentation for every later unresolved unit.
*/
function displayUnitText(value: string): string {
return decodeHtmlEntities(normalizeScriptText(value));
}

function normalizedUnitText(value: string): string {
return value.replace(/\s+/g, " ").trim();
return displayUnitText(value).replace(/\s+/g, " ").trim();
}

/**
Expand Down Expand Up @@ -220,7 +242,9 @@ function renderStructuredUnits(
{rows.map((row, rowIndex) => (
<tr key={`post-body-table-row-${row.unit_index}-${rowIndex}`}>
{row.unit_text.split(/\s*\|\s*/).map((cell, cellIndex) => (
<td key={`post-body-table-cell-${row.unit_index}-${cellIndex}`}>{cell}</td>
<td key={`post-body-table-cell-${row.unit_index}-${cellIndex}`}>
{renderStyledText(displayUnitText(cell))}
</td>
))}
</tr>
))}
Expand All @@ -239,7 +263,7 @@ function renderStructuredUnits(
renderSegment(
{
kind: "text",
text: unit.unit_text,
text: displayUnitText(unit.unit_text),
...(unit.unit_label === "footnote" || sourceText?.role === "footnote"
? { role: "footnote" as const }
: {}),
Expand Down
Loading