feat: persist leftover observed Y and expected E (v2.12.20)

AGENTS.md

@@ -186,12 +186,14 @@ in the same spirit) -- never against real data, per the hard rule above.
 against a live local stack (`make up`) and self-skip without one -- see
 [README.md](README.md#local-product-stack-docker-compose).
 
-Period leftover pairs (ADR 0017 / 0018) are computed in
+Period leftover pairs (ADR 0048 / 0049 / 0168) are computed in
 `lineageweave/leftover_pairs.py` from the residual after a real
 GRM/GPCM score, never invented. Missing cells stay out of the
 Gabriel factorization. Closest and farthest post–criterion pairs
-persist to `report_leftover_pair` and sit above the member list so
-a click opens that post.
+persist `leftover_observed_score`, `leftover_expected_score`, and
+`leftover_residual` (`R = Y − E`) to `report_leftover_pair` and sit
+above the member list so a click opens that post. Do not invent a
+leftover score or a second theta.
 
 `frontend/` has its own toolchain (Node pinned via `frontend/mise.toml`,
 pnpm via Corepack -- do not add a second Node package manager or a

ARCHITECTURE.md

@@ -588,8 +588,9 @@ on those same fixed parameters (Kim, 2006 FIPC). After scoring,
 `information_polytomous` ranks the shared-bank items by Fisher
 information at the group's mean θ (Lord, 1980 max-info CAT). Rankings
 persist to `report_item_information`. After those IRT main effects,
-residual SVD leftover pairs (Jeon et al., 2021; ADR 0017) persist to
-`report_leftover_pair`. Results persist to
+residual SVD leftover pairs (Jeon et al., 2021; ADR 0048 / 0168) persist to
+`report_leftover_pair` with observed Y, expected E[Y|θ, item], and
+leftover residual R = Y − E. Results persist to
 `report_period_score` / `report_member_score`.
 `GET /api/reports/{grouping}` lists the trend;
 `GET /api/reports/{grouping}/{period}` is ABAC-filtered;
@@ -601,7 +602,8 @@ bank as the dummy high/low band rows, so comparison-strip click
 through opens those DAG posts. Report members include the earliest
 open ticket title, status lookup label, and due date when one exists. The home page renders
 the actual mean θ, the FIPC delta, the CAT-selected item, leftover
-closest/farthest pairs above the member list, and the
+closest/farthest pairs (observed, expected, leftover residual) above
+the member list, and the
 PU / corp / thread comparison -- never a placeholder. TEPP is unchanged.
 
 ## Phase 6b: Knowledge Graph as a real Ontology + Semantic Layer

CHANGELOG.d/2.12.20-leftover-observed-expected.md

@@ -0,0 +1,13 @@
+# 2.12.20 — Leftover observed Y and expected E
+
+## Added
+
+- Persist leftover observed score `Y` and expected score `E[Y|θ, item]`
+  on each closest/farthest leftover pair so leftover residual
+  `R = Y − E` is checkable (ADR 0168). Missing cells stay out of the
+  factorization. Existing leftover rows are derived; migration `0108`
+  deletes incomplete rows rather than inventing `Y` or `E`.
+- After `make seed`, leftover pair badges name observed, expected, and
+  leftover residual above the member list. Click still opens that post.
+  Copy tells the buyer to check leftover residual equals observed minus
+  expected. Do not invent a leftover score or a second theta.

CHANGELOG.md

@@ -12,6 +12,18 @@ All notable changes to this project are documented here. Format follows
   environment, so local OIDC and synthetic-data workflows resolve the same
   pinned dependencies as CI.
 
+## [2.12.20] - 2026-08-24
+
+### Added
+
+- Leftover closest/farthest pairs now persist observed score `Y` and
+  expected score `E[Y|θ, item]` beside leftover residual `R = Y − E`
+  (ADR 0168). After `make seed`, leftover pair badges name observed,
+  expected, and leftover residual above the member list; click still
+  opens that post. Check leftover residual equals observed minus
+  expected. Missing cells stay out of the factorization. Do not invent
+  a leftover score or a second theta.
+
 ## [2.12.6] - 2026-08-20
 
 ### Added

backend/app/report_ingestion.py

@@ -443,8 +443,9 @@ async def persist_period_report(
             """
             insert into report_leftover_pair (
                 grouping_kind, grouping_key, period_code, rubric_version,
-                pair_kind, post_id, criterion_code, leftover_distance, leftover_residual
-            ) values ($1,$2,$3,$4,$5,$6,$7,$8,$9)
+                pair_kind, post_id, criterion_code, leftover_distance, leftover_residual,
+                leftover_observed_score, leftover_expected_score
+            ) values ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11)
             """,
             grouping_kind,
             grouping_key,
@@ -455,6 +456,8 @@ async def persist_period_report(
             pair.criterion_code,
             pair.leftover_distance,
             pair.leftover_residual,
+            pair.leftover_observed_score,
+            pair.leftover_expected_score,
         )
 
 
@@ -601,7 +604,8 @@ async def fetch_period_reports(
     leftover = await conn.fetch(  # nosemgrep: python.lang.security.audit.sqli.asyncpg-sqli.asyncpg-sqli
         f"""
         select lp.grouping_key, lp.pair_kind, lp.post_id, lp.criterion_code,
-               lp.leftover_distance, lp.leftover_residual, p.post_title,
+               lp.leftover_distance, lp.leftover_residual,
+               lp.leftover_observed_score, lp.leftover_expected_score, p.post_title,
                p.visibility_code, p.corporate_entity_id,
                ({_SOURCE_CONTEXT_PRESENT_SQL}) as has_real_source_context
         from report_leftover_pair lp
@@ -698,6 +702,8 @@ async def fetch_period_reports(
                         "criterion_code": str(row["criterion_code"]),
                         "leftover_distance": float(row["leftover_distance"]),
                         "leftover_residual": float(row["leftover_residual"]),
+                        "leftover_observed_score": float(row["leftover_observed_score"]),
+                        "leftover_expected_score": float(row["leftover_expected_score"]),
                         "visibility_code": row["visibility_code"],
                         "corporate_entity_id": str(row["corporate_entity_id"]),
                         "has_real_source_context": bool(row["has_real_source_context"]),

backend/tests/test_api.py

@@ -4518,6 +4518,14 @@ def test_seed_period_report_surfaces_on_get_reports(client, demo_analyst_token,
     assert leftover_kinds <= {"closest", "farthest"}
     assert all(pair["post_title"] for pair in high_report.get("leftover_pairs", []))
     assert all(pair["leftover_distance"] >= 0 for pair in high_report.get("leftover_pairs", []))
+    assert all(
+        abs(
+            pair["leftover_residual"]
+            - (pair["leftover_observed_score"] - pair["leftover_expected_score"])
+        )
+        < 1e-6
+        for pair in high_report.get("leftover_pairs", [])
+    )
 
     week3 = client.get(
         "/api/reports/process_unit/2026-W03",

docker/postgres-init/migrate.sh

@@ -18,7 +18,7 @@ for migration in /opt/lineageweave/migrations/*.sql; do
     migration_name=${migration##*/}
     case "$migration_name" in
         0012_*|0013_*|0014_*|0015_*|0016_*|0017_*|0018_*|0019_*|0020_*|0021_*|0022_*|0023_*|0024_*|0025_*|0026_*|0027_*|0028_*|0029_*|0030_*|0031_*|0032_*|0033_*|0034_*|0035_*|0036_*|0037_*|0038_*|0039_*|0040_*|0041_*|0042_*|0043_*|0044_*|0045_*|0046_*|0047_*|0048_*|0049_*|0050_*) ;;
-        0060_*|0100_*|0101_*|0102_*) ;;
+        0060_*|0100_*|0101_*|0102_*|0108_*) ;;
         *) continue ;;
     esac
     printf 'Applying %s\n' "$migration_name"

docs/adr/0168-leftover-observed-expected.md

@@ -0,0 +1,66 @@
+# ADR 0168 — Persist leftover observed Y and expected E[Y|θ, item]
+
+**Decision status:** Accepted
+**Date:** 2026-08-24
+
+Amends [ADR 0048](0048-persist-lsirm-leftover-pairs.md) and
+[ADR 0049](0049-leftover-pair-report-ui.md). Independent of leftover-map
+coordinates, leftover criterion landing, leftover complete-case
+coverage, leftover-map axis share, leftover comparison-strip reuse,
+and two-axis leftover-map distance.
+
+## Context
+
+ADR 0048 already computes leftover residual `R = Y − E[Y|θ, item]`
+after IRT main effects (Jeon et al., 2021, eq. 3; Gabriel, 1971) and
+persists closest/farthest pairs with `leftover_distance` and
+`leftover_residual`. The buyer could see distance `d` but could not
+check that residual identity: observed `Y` and expected `E` were
+computed, then dropped.
+
+Backfilling `Y = R` and `E = 0` would invent an expected score.
+Persisting only `R` leaves the identity uncheckable.
+
+## Decision
+
+Each leftover pair persists `leftover_observed_score` (`Y`) and
+`leftover_expected_score` (`E[Y|θ, item]`) next to
+`leftover_residual`. The check
+`abs(R − (Y − E)) < 1e-9` is named
+`leftover_pair_residual_identity`. Missing or non-finite cells never
+become pairs. Existing leftover rows are derived: migration `0108`
+deletes incomplete rows so rebuild/seed rewrites honest `Y` and `E`
+instead of fabricating them.
+
+Do not invent a leftover score. Do not invent a second theta.
+
+After `make seed`, leftover pair badges name observed, expected, and
+leftover residual above the member list. Click still opens that post.
+Copy tells the buyer to check leftover residual equals observed minus
+expected.
+
+## Consequences
+
+`GET /api/reports/{grouping}/{period}` returns `leftover_observed_score`
+and `leftover_expected_score` on each pair. Hidden posts stay hidden
+through the same ABAC join as members. Migration `0108` upgrades
+volumes that already applied `0012`. Fresh `0001` schema includes the
+same columns and identity check.
+
+## Related
+
+Does not mix into leftover persist-map, leftover criterion landing,
+leftover complete-case coverage, leftover-map axis share, leftover
+comparison-strip reuse, or two-axis leftover-map distance. Issues #79
+and #87 stay open.
+
+## References
+
+Gabriel, K. R. (1971). The biplot graphic display of matrices with
+application to principal component analysis. *Biometrika, 58*(3),
+453–467. https://doi.org/10.1093/biomet/58.3.453
+
+Jeon, M., Jin, I. H., Schweinberger, M., & Baugh, S. (2021). Mapping
+unobserved item–respondent interactions: A latent space item response
+model with interaction map. *Psychometrika, 86*(2), 378–403.
+https://doi.org/10.1007/s11336-021-09762-5

frontend/package.json

@@ -1,7 +1,7 @@
 {
   "name": "frontend",
   "private": true,
-  "version": "2.12.6",
+  "version": "2.12.20",
   "type": "module",
   "scripts": {
     "dev": "vite",

frontend/src/App.test.tsx

@@ -932,6 +932,8 @@ describe("App, authenticated", () => {
                     criterion_code: "sales_lead_specificity",
                     leftover_distance: 0.12,
                     leftover_residual: 0.4,
+                    leftover_observed_score: 3.4,
+                    leftover_expected_score: 3.0,
                   },
                   {
                     pair_kind: "farthest",
@@ -940,6 +942,8 @@ describe("App, authenticated", () => {
                     criterion_code: "general_sentiment_negative",
                     leftover_distance: 1.84,
                     leftover_residual: -1.1,
+                    leftover_observed_score: 1.0,
+                    leftover_expected_score: 2.1,
                   },
                 ],
                 members: [
@@ -3357,13 +3361,19 @@ describe("App, authenticated", () => {
     });
     expect(closestPair).toHaveTextContent("Closest leftover: Public post · sales-lead");
     expect(closestPair).toHaveTextContent(
-      "Open this post to read the criterion it sat closest to after main effects.",
+      "Open this post to read the closest leftover criterion. Check leftover residual equals observed minus expected.",
     );
+    expect(closestPair).toHaveTextContent("observed 3.40");
+    expect(closestPair).toHaveTextContent("expected 3.00");
+    expect(closestPair).toHaveTextContent("leftover 0.40");
     expect(closestPair).toHaveTextContent("d 0.12");
     expect(farthestPair).toHaveTextContent("Farthest leftover: Specification revision requested · negative");
     expect(farthestPair).toHaveTextContent(
-      "Open this post to read the criterion it sat farthest from after main effects.",
+      "Open this post to read the farthest leftover criterion. Check leftover residual equals observed minus expected.",
     );
+    expect(farthestPair).toHaveTextContent("observed 1.00");
+    expect(farthestPair).toHaveTextContent("expected 2.10");
+    expect(farthestPair).toHaveTextContent("leftover -1.10");
     expect(farthestPair).toHaveTextContent("d 1.84");
     const memberButton = screen.getByRole("button", { name: /open report post: public post/i });
     expect(closestPair.compareDocumentPosition(memberButton) & Node.DOCUMENT_POSITION_FOLLOWING).toBeTruthy();

frontend/src/App.tsx

@@ -3396,8 +3396,8 @@ function ReportsPanel({
                     pair.pair_kind === "farthest" ? "Farthest leftover" : "Closest leftover";
                   const nextAction =
                     pair.pair_kind === "farthest"
-                      ? "Open this post to read the criterion it sat farthest from after main effects."
-                      : "Open this post to read the criterion it sat closest to after main effects.";
+                      ? "Open this post to read the farthest leftover criterion. Check leftover residual equals observed minus expected."
+                      : "Open this post to read the closest leftover criterion. Check leftover residual equals observed minus expected.";
                   const criterion = criterionShortLabel(pair.criterion_code);
                   return (
                     <li
@@ -3413,6 +3413,15 @@ function ReportsPanel({
                           {kindLabel}: {pair.post_title} · {criterion}
                         </span>
                         <span className="post-badge">{nextAction}</span>
+                        <span className="post-badge">
+                          observed {pair.leftover_observed_score.toFixed(2)}
+                        </span>
+                        <span className="post-badge">
+                          expected {pair.leftover_expected_score.toFixed(2)}
+                        </span>
+                        <span className="post-badge">
+                          leftover {pair.leftover_residual.toFixed(2)}
+                        </span>
                         <span className="post-badge">d {pair.leftover_distance.toFixed(2)}</span>
                       </button>
                     </li>

frontend/src/api.ts

@@ -759,6 +759,8 @@ export interface LeftoverPair {
   criterion_code: string;
   leftover_distance: number;
   leftover_residual: number;
+  leftover_observed_score: number;
+  leftover_expected_score: number;
 }
 
 export interface PeriodGroupReport {

lineageweave/leftover_pairs.py

@@ -1,9 +1,12 @@
-"""Jeon leftover post–criterion pairs after a main-effect IRT (ADR 0017).
+"""Jeon leftover post–criterion pairs after a main-effect IRT (ADR 0048 / 0168).
 
 Does not import ``fast_mlsirm`` or ``period_report``. A Gabriel biplot
 of the residual ``R = Y − E[Y|θ, item]`` supplies person and item
 positions. Missing response cells are excluded from the factorization;
-they are never treated as zero residuals.
+they are never treated as zero residuals. Closest and farthest pairs
+keep the observed score ``Y`` and expected score ``E`` so a buyer can
+check the leftover residual identity. This module does not invent a
+leftover score or a second theta.
 """
 
 from __future__ import annotations
@@ -26,6 +29,8 @@ class LeftoverPair:
     criterion_code: str
     leftover_distance: float
     leftover_residual: float
+    leftover_observed_score: float
+    leftover_expected_score: float
 
 
 def leftover_pairs_from_residual(
@@ -68,7 +73,7 @@ def leftover_pairs_from_residual(
     else:
         center = float(np.mean([residual[person, item] for person, item in observed]))
     person_pos, item_pos = _complete_case_positions(residual, center, keep_person, keep_item)
-    candidates: list[tuple[float, str, str, float]] = []
+    candidates: list[tuple[float, str, str, float, float, float]] = []
     if person_pos is not None and item_pos is not None:
         person_index = np.flatnonzero(keep_person)
         item_index = np.flatnonzero(keep_item)
@@ -82,42 +87,47 @@ def leftover_pairs_from_residual(
             )
             if not np.isfinite(distance):
                 continue
-            candidates.append(
-                (
-                    max(distance, 0.0),
-                    post_ids[person],
-                    item_codes[item],
-                    float(residual[person, item]),
-                )
-            )
+            candidates.append(_candidate(distance, post_ids, item_codes, matrix, expected, residual, person, item))
     if not candidates:
         for person, item in observed:
             distance = abs(float(residual[person, item]) - center)
-            candidates.append(
-                (
-                    max(distance, 0.0),
-                    post_ids[person],
-                    item_codes[item],
-                    float(residual[person, item]),
-                )
-            )
+            candidates.append(_candidate(distance, post_ids, item_codes, matrix, expected, residual, person, item))
     closest = min(candidates, key=lambda row: (row[0], row[1], row[2]))
     farthest = max(candidates, key=lambda row: (row[0], row[1], row[2]))
+    return (_pair(PAIR_KIND_CLOSEST, closest), _pair(PAIR_KIND_FARTHEST, farthest))
+
+
+def _candidate(
+    distance: float,
+    post_ids: list[str],
+    item_codes: tuple[str, ...],
+    matrix: np.ndarray,
+    expected: np.ndarray,
+    residual: np.ndarray,
+    person: int,
+    item: int,
+) -> tuple[float, str, str, float, float, float]:
+    """One observed leftover cell: distance, ids, R, Y, and E."""
     return (
-        LeftoverPair(
-            pair_kind=PAIR_KIND_CLOSEST,
-            post_id=closest[1],
-            criterion_code=closest[2],
-            leftover_distance=closest[0],
-            leftover_residual=closest[3],
-        ),
-        LeftoverPair(
-            pair_kind=PAIR_KIND_FARTHEST,
-            post_id=farthest[1],
-            criterion_code=farthest[2],
-            leftover_distance=farthest[0],
-            leftover_residual=farthest[3],
-        ),
+        max(distance, 0.0),
+        post_ids[person],
+        item_codes[item],
+        float(residual[person, item]),
+        float(matrix[person, item]),
+        float(expected[person, item]),
+    )
+
+
+def _pair(kind: str, row: tuple[float, str, str, float, float, float]) -> LeftoverPair:
+    """Map a leftover candidate tuple onto the persisted pair contract."""
+    return LeftoverPair(
+        pair_kind=kind,
+        post_id=row[1],
+        criterion_code=row[2],
+        leftover_distance=row[0],
+        leftover_residual=row[3],
+        leftover_observed_score=row[4],
+        leftover_expected_score=row[5],
     )