Skip to content
25 changes: 13 additions & 12 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -259,23 +259,24 @@ stops startup instead of leaving a healthy-looking partial schema, and
application code must not compensate for a missing table.

Period leftover pairs (ADR 0017 / 0018 / 0048 / 0049 / 0119 / 0158 / 0162 /
0163 / 0164 / 0182) are computed in `lineageweave/leftover_pairs.py` from the
0163 / 0164 / 0182 / 0186) are computed in `lineageweave/leftover_pairs.py` from the
residual after a real GRM/GPCM score, never invented. Distances are
Euclidean on the two-dimensional Gabriel leftover map; missing cells stay
out of the factorization. Closest and farthest post–criterion pairs
persist to `report_leftover_pair` with signed residual `R`, observed
`Y`, and expected `E[Y|θ, item]` so `R = Y − E` remains auditable,
plus leftover-map rank so rank 0 is not read as structure, and
unexplained leftover `U = R − R̂` next to leftover-map distance `d`
after two-axis Gabriel reconstruction. They sit above the member
list so a click opens that post with the leftover criterion current
in Post quality (ADR 0158). Two-axis reconstruction `R̂` is not
persisted. Leftover-map axis share (ADR 0148) is Gabriel inertia of
residual SVD axes 1 and 2 and persists to `report_leftover_map_axis`.
Rank-0 residuals emit two zero-share axes; the shares are report-level
and are not a leftover score. Complete-case coverage (ADR 0168) persists to
`Y`, and expected `E[Y|θ, item]` so `R = Y − E` remains auditable, plus
leftover-map rank so rank 0 is not read as structure, plus two-axis
leftover-map reconstruction `R̂` of centered leftover, plus unexplained
leftover `U = R − R̂` next to leftover-map distance `d`. They sit above
the member list so a click opens that post with the leftover criterion
current in Post quality (ADR 0158). Leftover-map axis share
(ADR 0148) is Gabriel inertia of residual SVD axes 1 and 2 and persists
to `report_leftover_map_axis`. Rank-0 residuals emit two zero-share
axes; the shares are report-level and are not a leftover score.
Complete-case coverage (ADR 0168) persists to
`report_leftover_map_coverage` and captions the pair list with how
many scored posts entered the map.
many scored posts entered the map; without a complete-case rectangle
there is no pair to caption, only coverage counts.

Period leftover pairs (ADR 0017 / 0018 / 0048 / 0049 / 0149) are computed in
`lineageweave/leftover_pairs.py` from the residual after a real
Expand Down
12 changes: 7 additions & 5 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -596,11 +596,12 @@ on those same fixed parameters (Kim, 2006 FIPC). After scoring,
information at the group's mean θ (Lord, 1980 max-info CAT). Rankings
persist to `report_item_information`. After those IRT main effects,
residual SVD leftover pairs on two Gabriel axes (Jeon et al., 2021;
ADR 0017 / 0048 / 0049 / 0119 / 0158 / 0162 / 0163 / 0164 / 0182) persist to
ADR 0017 / 0048 / 0049 / 0119 / 0158 / 0162 / 0163 / 0164 / 0182 / 0186) persist to
`report_leftover_pair` with signed residual `R`, observed `Y`, expected
`E[Y|θ, item]`, full leftover-map rank, and unexplained leftover
`U = R − R̂` named on the pair row. Leftover-map axis share (Gabriel
inertia of residual SVD axes 1 and 2; ADR 0148) persists to
`E[Y|θ, item]`, full leftover-map rank, two-axis leftover-map
reconstruction `R̂`, and unexplained leftover `U = R − R̂` named on the
pair row. Leftover-map axis share (Gabriel inertia of residual SVD
axes 1 and 2; ADR 0148) persists to
`report_leftover_map_axis`. Complete-case leftover-map coverage (ADR
0168) persists to `report_leftover_map_coverage` so readers see how
many scored posts entered the factorization. Results persist to
Expand All @@ -617,7 +618,8 @@ through opens those DAG posts. Report members include the earliest
open ticket title, status lookup label, and due date when one exists. The home page renders
the actual mean θ, the FIPC delta, the CAT-selected item, leftover
closest/farthest pairs (signed residual `R`, observed `Y`, expected
`E`, full rank, and two-axis leftover-map distance `d` after IRT main
`E`, full rank, two-axis leftover-map reconstruction `R̂`, and
two-axis leftover-map distance `d` after IRT main
effects) above the member list, leftover-map axis share for residual
SVD axes 1 and 2, and complete-case coverage captions (map used N of M
scored posts), plus the
Expand Down
8 changes: 8 additions & 0 deletions CHANGELOG.d/2.12.30-leftover-map-reconstruction.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
## 2.12.30 — Leftover-map reconstruction

- Persist two-axis leftover-map reconstruction `R̂_c = ξ_{1:2} · ζ_{1:2}`
of centered leftover on leftover post–criterion pairs (ADR 0186).
After `make seed`, closest and farthest leftover pairs sit above the
member list with `R̂ {value}` next to leftover-map distance `d`; click
opens that post. Fallback pairs omit reconstruction. Rank-0 origin
maps store `R̂ = 0`. Never invent a leftover score or a theta.
13 changes: 11 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,15 +107,24 @@ All notable changes to this project are documented here. Format follows
is unwired, with `이 범위의 일정을 아직 받을 수 없습니다`. Weekly VOC and
newspaper stay on the board.

## [2.12.30] - 2026-08-24

### Added

- Period leftover pair rows now name two-axis leftover-map reconstruction
`R̂` of centered leftover next to leftover-map distance `d`, then open
that post (Gabriel, 1971; Jeon et al., 2021, eq. 3; ADR 0186). Fallback
pairs omit the reconstruction rather than inventing one. Rank-0 origin
maps store `R̂ = 0`. Never invent a leftover score or a theta.

## [2.12.26] - 2026-08-24

### Added

- Period leftover pair rows now name unexplained leftover `U = R − R̂`
next to leftover-map distance `d`, then open that post (Gabriel, 1971;
Jeon et al., 2021, eq. 3; ADR 0182). A missing unexplained leftover
omits the badge rather than inventing a leftover score. Two-axis
reconstruction `R̂` stays internal and is not persisted.
omits the badge rather than inventing a leftover score.

## [2.12.19] - 2026-08-24

Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ cutoff.

Create/start endpoint rules (ADR 0017 / 0021), tie-vs-miss similarity
(ADR 0026), R&R catalog ids (ADR 0019 / 0027), leftover pairs
(ADR 0048–0164 / 0182), the text-channel embedding swap and cosine
(ADR 0048–0164 / 0182 / 0186), the text-channel embedding swap and cosine
clamp (ADR 0190), per-edge channel-score persistence (ADR 0195),
migration replay (ADR 0166), docstring coverage, and the measurement
boundary are all stated in [AGENTS.md](AGENTS.md) -- read it before
Expand Down
12 changes: 9 additions & 3 deletions backend/app/report_ingestion.py
Original file line number Diff line number Diff line change
Expand Up @@ -445,8 +445,8 @@ async def persist_period_report(
grouping_kind, grouping_key, period_code, rubric_version,
pair_kind, post_id, criterion_code, leftover_distance, leftover_residual,
observed_response, expected_response, leftover_map_rank,
leftover_map_unexplained
) values ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13)
leftover_map_reconstruction, leftover_map_unexplained
) values ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14)
""",
grouping_kind,
grouping_key,
Expand All @@ -460,6 +460,7 @@ async def persist_period_report(
pair.observed_response,
pair.expected_response,
pair.leftover_map_rank,
pair.leftover_map_reconstruction,
pair.leftover_map_unexplained,
)
for axis in report.leftover_map_axes:
Expand Down Expand Up @@ -646,7 +647,7 @@ async def fetch_period_reports(
select lp.grouping_key, lp.pair_kind, lp.post_id, lp.criterion_code,
lp.leftover_distance, lp.leftover_residual,
lp.observed_response, lp.expected_response, lp.leftover_map_rank,
lp.leftover_map_unexplained, p.post_title,
lp.leftover_map_reconstruction, lp.leftover_map_unexplained, p.post_title,
p.visibility_code, p.corporate_entity_id,
({_SOURCE_CONTEXT_PRESENT_SQL}) as has_real_source_context
from report_leftover_pair lp
Expand Down Expand Up @@ -785,6 +786,11 @@ async def fetch_period_reports(
if row["leftover_map_rank"] is None
else int(row["leftover_map_rank"])
),
"leftover_map_reconstruction": (
None
if row["leftover_map_reconstruction"] is None
else float(row["leftover_map_reconstruction"])
),
"leftover_map_unexplained": (
None
if row["leftover_map_unexplained"] is None
Expand Down
11 changes: 10 additions & 1 deletion backend/tests/test_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

from __future__ import annotations

import math
import os
import uuid
from contextlib import closing
Expand Down Expand Up @@ -162,6 +163,11 @@
/ "migrations"
/ "0182_report_leftover_map_unexplained.sql"
)
_LEFTOVER_MAP_RECONSTRUCTION_MIGRATION = (
Path(__file__).resolve().parents[2]
/ "migrations"
/ "0186_report_leftover_map_reconstruction.sql"
)


def _postgres_available() -> bool:
Expand Down Expand Up @@ -305,6 +311,7 @@ def seeded_db(demo_analyst_token):
cur.execute(_GLOBAL_ASK_JOB_MIGRATION.read_text())
cur.execute(_LEFTOVER_MAP_AXIS_MIGRATION.read_text())
cur.execute(_LEFTOVER_MAP_UNEXPLAINED_MIGRATION.read_text())
cur.execute(_LEFTOVER_MAP_RECONSTRUCTION_MIGRATION.read_text())
cur.execute(
"insert into common_lookup_value (lookup_category, lookup_code, lookup_label) values "
"('corporate_entity_level', 'group', 'Group'), "
Expand Down Expand Up @@ -5044,9 +5051,11 @@ def test_seed_period_report_surfaces_on_get_reports(client, demo_analyst_token,
assert all(pair["leftover_distance"] >= 0 for pair in high_report.get("leftover_pairs", []))
for pair in high_report.get("leftover_pairs", []):
assert pair["leftover_map_rank"] >= 0
reconstruction = pair.get("leftover_map_reconstruction")
if reconstruction is not None:
assert math.isfinite(reconstruction)
unexplained = pair.get("leftover_map_unexplained")
assert unexplained is None or isinstance(unexplained, (int, float))
assert "leftover_map_reconstruction" not in pair
observed = pair.get("observed_response")
expected = pair.get("expected_response")
if observed is None or expected is None:
Expand Down
16 changes: 8 additions & 8 deletions docs/adr/0003-fast-mlsirm-report-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,14 +100,14 @@ than one large PR:
`information_polytomous` (Lord, 1980 max-info). Persist the ranking
(`report_item_information`) and show the rank-1 item on the Period
reports panel. Do not reimplement an information function here.
7. **Leftover-pair slice** (shipped in 0.71.2; ADR 0017 / 0018 / 0048 / 0049 / 0182): after
IRT main effects, persist closest and farthest post–criterion pairs
from the residual leftover map, and name unexplained leftover
`U = R − R̂` when Gabriel coordinates exist so the leftover cell the
two-axis map does not reconstruct is not read as leftover residual
or leftover-map distance. Do not persist two-axis reconstruction
`R̂`. Do not fork LSIRM; do not invent a
leftover-pair API inside `fast-mlsirm` in this slice.
7. **Leftover-pair slice** (shipped in 0.71.2; ADR 0017 / 0018 / 0048 /
0049 / 0182, amended by ADR 0186): after IRT main effects, persist
closest and farthest post–criterion pairs from the residual leftover
map, including two-axis leftover-map reconstruction `R̂` and
unexplained leftover `U = R − R̂` when Gabriel coordinates exist so
the leftover cell the two-axis map does not reconstruct is not read
as leftover residual or leftover-map distance. Do not fork LSIRM; do
not invent a leftover-pair API inside `fast-mlsirm` in this slice.
8. **Leftover-map axis-share slice** (ADR 0148): persist Gabriel inertia
`σ_k² / Σ_j σ_j²` of leftover-map axes 1 and 2 on the same residual
SVD. Rank-0 residuals emit two zero-share axes. Do not invent a
Expand Down
13 changes: 7 additions & 6 deletions docs/adr/0048-persist-lsirm-leftover-pairs.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@
**Amended by:** [ADR 0119](0119-leftover-map-two-dimensional-distance.md) (two leftover-map axes);
[ADR 0163](0163-leftover-observed-expected.md) (observed Y and expected E);
[ADR 0164](0164-leftover-map-rank.md) (full map rank);
[ADR 0182](0182-leftover-map-unexplained.md) (unexplained leftover U)
[ADR 0182](0182-leftover-map-unexplained.md) (unexplained leftover U);
[ADR 0186](0186-leftover-map-reconstruction.md) (two-axis reconstruction R̂)

## Context

Expand Down Expand Up @@ -38,11 +39,11 @@ Euclidean on the two leftover-map axes (ADR 0119). Each leftover row
also names observed `Y` and expected `E[Y|θ, item]` so residual
reconciles to `Y − E` (ADR 0163), and names the full singular-value
rank while distance remains on the first two axes (ADR 0164). Each
leftover row also names unexplained leftover `U = R − R̂` when
Gabriel coordinates exist so the leftover cell the two-axis map does
not reconstruct is not read as leftover residual `R` or leftover-map
distance `d` (ADR 0182). Two-axis reconstruction `R̂` is computed
internally and is not persisted.
leftover row also names two-axis leftover-map reconstruction
`R̂_c = ξ_{1:2} · ζ_{1:2}` of centered leftover (ADR 0186) and
unexplained leftover `U = R − R̂_c` when Gabriel coordinates exist so
the leftover cell the two-axis map does not reconstruct is not read
as leftover residual `R` or leftover-map distance `d` (ADR 0182).

Cascade the rows with `report_period_score`. A leftover post must
also be a `report_member_score` row, and the leftover criterion
Expand Down
13 changes: 8 additions & 5 deletions docs/adr/0049-leftover-pair-report-ui.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@
[ADR 0163](0163-leftover-observed-expected.md) (observed Y and expected E);
[ADR 0164](0164-leftover-map-rank.md) (full map rank);
[ADR 0182](0182-leftover-map-unexplained.md) (unexplained leftover U);
[ADR 0158](0158-leftover-criterion-evaluation-landing.md) (criterion evaluation landing)
[ADR 0158](0158-leftover-criterion-evaluation-landing.md) (criterion evaluation landing);
[ADR 0186](0186-leftover-map-reconstruction.md) (two-axis reconstruction R̂)

## Context

Expand All @@ -23,8 +24,9 @@ second navigation surface.
On each period-report group, render leftover pairs **above** the
member list. Each pair is a button: closest or farthest label, post
title, criterion short label, signed residual `R`, two-axis leftover-map
distance, full map rank, observed `Y`, expected `E` when finite, and
unexplained leftover `U` when finite.
distance, full map rank, leftover-map reconstruction `R̂` when finite,
observed `Y`, expected `E` when finite, and unexplained leftover `U`
when finite.
The next action names every available measurement before opening the
post; no amendment hides another, rank 0 explicitly names no
leftover structure, and unexplained leftover names "leftover map leaves
Expand All @@ -35,8 +37,9 @@ Clicking the button opens that post with leftover focus so Post
quality marks the named criterion current (ADR 0158). Residual naming
is [ADR 0162](0162-leftover-residual-disclosure.md), observed/expected
naming is [ADR 0163](0163-leftover-observed-expected.md), rank naming
is [ADR 0164](0164-leftover-map-rank.md), unexplained leftover naming
is [ADR 0182](0182-leftover-map-unexplained.md).
is [ADR 0164](0164-leftover-map-rank.md), reconstruction naming is
[ADR 0186](0186-leftover-map-reconstruction.md), unexplained leftover
naming is [ADR 0182](0182-leftover-map-unexplained.md).

After `make seed`, closest and farthest leftover pairs sit above the
member list. Click a pair to open that post with the leftover
Expand Down
93 changes: 93 additions & 0 deletions docs/adr/0186-leftover-map-reconstruction.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# ADR 0186 — Name leftover-map reconstruction on leftover pairs

**Decision status:** Accepted
**Date:** 2026-08-24

Amends [ADR 0048](0048-persist-lsirm-leftover-pairs.md) and
[ADR 0049](0049-leftover-pair-report-ui.md).

## Context

ADR 0048 already persists leftover-map distance `d = ‖ξ_p − ζ_i‖` and
leftover residual `R = Y − E[Y|θ, item]` on `report_leftover_pair`.
ADR 0049 already renders closest and farthest pairs above the member
list and opens the named post. Distance is the Jeon et al. (2021,
eq. 3) map gap. Gabriel (1971) reconstructs a *centered* matrix from
the biplot as the inner product of person and item coordinates. The
leftover map buyers read is two-axis: unused axes pad with zero, and
hidden SVD axes after the second are dropped. Two-axis reconstruction
`R̂_c = ξ_{1:2} · ζ_{1:2}` therefore recovers centered leftover
`R̃ = R − center`, not raw residual `R` and not leftover-map distance
`d`. Hiding `R̂_c` lets a buyer read distance as if it named how much
leftover the two leftover-map axes reconstruct.

This increment does not persist leftover-map coordinates, does not
persist leftover-map unexplained leftover `U` or `U_c`, does not
persist leftover-map explained leftover share `e`, unexplained leftover
share `s`, or leftover-map cross share `x`, does not name leftover-map
cosine or length, does not name observed `Y` / expected `E`, does not
name leftover-map rank, does not split leftover-map distance onto two
axes, and does not land Post quality on the leftover criterion.
Leftover-map distance stays two-axis Euclidean (ADR 0119).

The unprotected-stack reconstructions for neighbouring leftover facts
use 0162–0185. This protected-main increment uses **0186** so it does
not collide with leftover-map cross share (0185), leftover-map
explained leftover share (0184), leftover-map unexplained leftover
share (0183), leftover-map unexplained leftover (0182), leftover-map
length (0181 on the length stack), leftover-map cosine (0180),
leftover-map inner product (0179), leftover residual disclosure
(0162 / 0178), leftover observed `Y` / expected `E` (0163 / 0170),
leftover-map rank (0164 / 0172), two-axis leftover-map distance
(0119 / 0166), leftover coverage (0165 / 0168), leftover-map axis
share (0148), or leftover interaction-map persistence (0121).

## Decision

Each leftover pair names `leftover_map_reconstruction` — two-axis
Gabriel reconstruction `R̂_c = ξ_{1:2} · ζ_{1:2}` of centered leftover
after IRT main effects. Migration `0186` is the single source of the
column on every install path, fresh or existing -- shipped migrations
(`0001` / `0012`) are never edited after the fact. The column is
nullable so older leftover rows keep distance and residual without
fabricating a reconstruction. Fallback pairs that have no
complete-case leftover map omit the value rather than inventing one.
A rank-0 origin map stores `0.0` (`ξ = 0`, `ζ = 0`), not a missing
value. A non-finite reconstruction stores null rather than inventing
a leftover score. A finite negative reconstruction is stored; do not
add a nonnegative CHECK. Do not persist
`leftover_map_explained_share`, `leftover_map_unexplained_share`,
`leftover_map_cross_share`, or leftover-map coordinates. The signed
`leftover_map_unexplained` value `U = R - R̂` remains persisted under ADR
0182; it is evidence, not one of the prohibited derived share aliases.

The pair button shows `R̂ {reconstruction}` next to leftover-map
distance `d` when the value is a finite number, including a signed
negative reconstruction. Next action: two leftover-map axes
reconstruct centered leftover `R̂` after IRT main effects; open this
post. A missing or non-finite reconstruction omits the badge and
keeps the existing closest/farthest next action. Do not invent a
leftover score or a theta.

## Consequences

`GET /api/reports/{grouping}/{period}` returns
`leftover_map_reconstruction`. After `make seed`, closest and farthest
leftover pairs sit above the member list with leftover-map
reconstruction; click opens that post. Hidden posts stay hidden.

Independent of leftover interaction-map persistence, leftover-criterion
evaluation landing, leftover residual UI extraction, two-axis leftover-map
distance, leftover observed `Y` / expected `E`, leftover-map rank, and
leftover-map share identities.

## References

Gabriel, K. R. (1971). The biplot graphic display of matrices with
application to principal component analysis. *Biometrika, 58*(3),
453–467. https://doi.org/10.1093/biomet/58.3.453

Jeon, M., Jin, I. H., Schweinberger, M., & Baugh, S. (2021). Mapping
unobserved item–respondent interactions: A latent space item response
model with interaction map. *Psychometrika, 86*(2), 378–403.
https://doi.org/10.1007/s11336-021-09762-5
Loading
Loading