Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,12 +186,14 @@ in the same spirit) -- never against real data, per the hard rule above.
against a live local stack (`make up`) and self-skip without one -- see
[README.md](README.md#local-product-stack-docker-compose).

Period leftover pairs (ADR 0017 / 0018) are computed in
Period leftover pairs (ADR 0048 / 0049 / 0183) are computed in
`lineageweave/leftover_pairs.py` from the residual after a real
GRM/GPCM score, never invented. Missing cells stay out of the
Gabriel factorization. Closest and farthest post–criterion pairs
persist to `report_leftover_pair` and sit above the member list so
a click opens that post.
persist to `report_leftover_pair` and sit above the member list with
unexplained leftover share `s = U_c² / R̃²` of centered leftover next
to leftover-map distance `d` so a click opens that post. Centered
leftover `U_c` and two-axis reconstruction `R̂_c` are not persisted.

`frontend/` has its own toolchain (Node pinned via `frontend/mise.toml`,
pnpm via Corepack -- do not add a second Node package manager or a
Expand Down
5 changes: 3 additions & 2 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -588,8 +588,9 @@ on those same fixed parameters (Kim, 2006 FIPC). After scoring,
`information_polytomous` ranks the shared-bank items by Fisher
information at the group's mean θ (Lord, 1980 max-info CAT). Rankings
persist to `report_item_information`. After those IRT main effects,
residual SVD leftover pairs (Jeon et al., 2021; ADR 0017) persist to
`report_leftover_pair`. Results persist to
residual SVD leftover pairs (Jeon et al., 2021; ADR 0048 / 0183) persist to
`report_leftover_pair` with unexplained leftover share `s = U_c² / R̃²` of
centered leftover named on the pair row. Results persist to
`report_period_score` / `report_member_score`.
`GET /api/reports/{grouping}` lists the trend;
`GET /api/reports/{grouping}/{period}` is ABAC-filtered;
Expand Down
9 changes: 9 additions & 0 deletions CHANGELOG.d/2.12.27-leftover-map-unexplained-share.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
## 2.12.27 — Leftover-map unexplained leftover share

- Persist unexplained leftover share `s = U_c² / R̃²` of centered
leftover on leftover post–criterion pairs (ADR 0183). After
`make seed`, closest and farthest leftover pairs sit above the
member list with `U²/R̃²` next to leftover-map distance `d`; click
opens that post. Omit the badge when the share is missing. Never
invent a leftover score. Do not persist leftover-map unexplained
leftover `U` or reconstruction `R̂`.
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,17 @@ All notable changes to this project are documented here. Format follows
environment, so local OIDC and synthetic-data workflows resolve the same
pinned dependencies as CI.

## [2.12.27] - 2026-08-24

### Added

- Period leftover pair rows now name unexplained leftover share
`s = U_c² / R̃²` of centered leftover next to leftover-map distance
`d`, then open that post (Gabriel, 1971; Jeon et al., 2021, eq. 3;
ADR 0183). A missing share omits the badge rather than inventing a
leftover score. Centered leftover `U_c` and two-axis reconstruction
`R̂_c` stay internal and are not persisted.

## [2.12.6] - 2026-08-20

### Added
Expand Down
4 changes: 3 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,9 @@ chip name contains `Corporate entity: Demo Corp` and the persisted
mean θ. The period-report panel says Demo Corp is the opened grouping
and to read its mean θ and member posts, then open a post. Those
members land immediately under that next action, ahead of Other Corp
and the week strip. Opening Public post names the next action: read
and the week strip. After `make seed`, leftover closest/farthest pairs
sit above the member list with unexplained leftover share `U²/R̃²`
next to leftover-map distance `d`. Opening Public post names the next action: read
Event Lineage, Keyman, and evaluation on that post. The popup Event
Lineage DAG marks that post current. After that current node, the
popup names Keyman and evaluation as the next read. After landed
Expand Down
14 changes: 11 additions & 3 deletions backend/app/report_ingestion.py
Original file line number Diff line number Diff line change
Expand Up @@ -443,8 +443,9 @@ async def persist_period_report(
"""
insert into report_leftover_pair (
grouping_kind, grouping_key, period_code, rubric_version,
pair_kind, post_id, criterion_code, leftover_distance, leftover_residual
) values ($1,$2,$3,$4,$5,$6,$7,$8,$9)
pair_kind, post_id, criterion_code, leftover_distance, leftover_residual,
leftover_map_unexplained_share
) values ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10)
""",
grouping_kind,
grouping_key,
Expand All @@ -455,6 +456,7 @@ async def persist_period_report(
pair.criterion_code,
pair.leftover_distance,
pair.leftover_residual,
pair.leftover_map_unexplained_share,
)


Expand Down Expand Up @@ -601,7 +603,8 @@ async def fetch_period_reports(
leftover = await conn.fetch( # nosemgrep: python.lang.security.audit.sqli.asyncpg-sqli.asyncpg-sqli
f"""
select lp.grouping_key, lp.pair_kind, lp.post_id, lp.criterion_code,
lp.leftover_distance, lp.leftover_residual, p.post_title,
lp.leftover_distance, lp.leftover_residual, lp.leftover_map_unexplained_share,
p.post_title,
p.visibility_code, p.corporate_entity_id,
({_SOURCE_CONTEXT_PRESENT_SQL}) as has_real_source_context
from report_leftover_pair lp
Expand Down Expand Up @@ -698,6 +701,11 @@ async def fetch_period_reports(
"criterion_code": str(row["criterion_code"]),
"leftover_distance": float(row["leftover_distance"]),
"leftover_residual": float(row["leftover_residual"]),
"leftover_map_unexplained_share": (
None
if row["leftover_map_unexplained_share"] is None
else float(row["leftover_map_unexplained_share"])
),
"visibility_code": row["visibility_code"],
"corporate_entity_id": str(row["corporate_entity_id"]),
"has_real_source_context": bool(row["has_real_source_context"]),
Expand Down
13 changes: 13 additions & 0 deletions backend/tests/test_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,11 @@
/ "migrations"
/ "0102_project_bound_summary_event.sql"
)
_LEFTOVER_MAP_UNEXPLAINED_SHARE_MIGRATION = (
Path(__file__).resolve().parents[2]
/ "migrations"
/ "0183_report_leftover_map_unexplained_share.sql"
)


def _postgres_available() -> bool:
Expand Down Expand Up @@ -226,6 +231,7 @@ def seeded_db(demo_analyst_token):
cur.execute(_MAJOR_EVENT_ACTION_MIGRATION.read_text())
cur.execute(_PROJECT_BOUND_ACTION_MIGRATION.read_text())
cur.execute(_PROJECT_BOUND_EVENT_MIGRATION.read_text())
cur.execute(_LEFTOVER_MAP_UNEXPLAINED_SHARE_MIGRATION.read_text())
cur.execute(
"insert into common_lookup_value (lookup_category, lookup_code, lookup_label) values "
"('corporate_entity_level', 'group', 'Group'), "
Expand Down Expand Up @@ -4518,6 +4524,13 @@ def test_seed_period_report_surfaces_on_get_reports(client, demo_analyst_token,
assert leftover_kinds <= {"closest", "farthest"}
assert all(pair["post_title"] for pair in high_report.get("leftover_pairs", []))
assert all(pair["leftover_distance"] >= 0 for pair in high_report.get("leftover_pairs", []))
for pair in high_report.get("leftover_pairs", []):
share = pair.get("leftover_map_unexplained_share")
assert share is None or isinstance(share, (int, float))
if share is not None:
assert share >= 0
assert "leftover_map_unexplained" not in pair
assert "leftover_map_reconstruction" not in pair

week3 = client.get(
"/api/reports/process_unit/2026-W03",
Expand Down
2 changes: 1 addition & 1 deletion docker/postgres-init/migrate.sh
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ for migration in /opt/lineageweave/migrations/*.sql; do
migration_name=${migration##*/}
case "$migration_name" in
0012_*|0013_*|0014_*|0015_*|0016_*|0017_*|0018_*|0019_*|0020_*|0021_*|0022_*|0023_*|0024_*|0025_*|0026_*|0027_*|0028_*|0029_*|0030_*|0031_*|0032_*|0033_*|0034_*|0035_*|0036_*|0037_*|0038_*|0039_*|0040_*|0041_*|0042_*|0043_*|0044_*|0045_*|0046_*|0047_*|0048_*|0049_*|0050_*) ;;
0060_*|0100_*|0101_*|0102_*) ;;
0060_*|0100_*|0101_*|0102_*|0183_*) ;;
*) continue ;;
esac
printf 'Applying %s\n' "$migration_name"
Expand Down
9 changes: 7 additions & 2 deletions docs/adr/0003-fast-mlsirm-report-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,9 +100,14 @@ than one large PR:
`information_polytomous` (Lord, 1980 max-info). Persist the ranking
(`report_item_information`) and show the rank-1 item on the Period
reports panel. Do not reimplement an information function here.
7. **Leftover-pair slice** (shipped in 0.71.2; ADR 0017 / 0018): after
7. **Leftover-pair slice** (shipped in 0.71.2; ADR 0048 / 0049 / 0183): after
IRT main effects, persist closest and farthest post–criterion pairs
from the residual leftover map. Do not fork LSIRM; do not invent a
from the residual leftover map, and name unexplained leftover share
`s = U_c² / R̃²` of centered leftover when Gabriel coordinates exist
so the leftover cell the two-axis map does not reconstruct is not
read as leftover residual or leftover-map distance. Do not persist
leftover-map unexplained leftover `U` or two-axis reconstruction
`R̂`. Do not fork LSIRM; do not invent a
leftover-pair API inside `fast-mlsirm` in this slice.

**TEPP boundary.** [ARCHITECTURE.md](../../ARCHITECTURE.md) already
Expand Down
8 changes: 7 additions & 1 deletion docs/adr/0048-persist-lsirm-leftover-pairs.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,13 @@ and one `farthest` observed cell per period report in
`report_leftover_pair` (3NF, two-or-more-word `snake_case`).

The biplot lives in `lineageweave/leftover_pairs.py` so leftover
tests do not import `period_report` or `fast_mlsirm`.
tests do not import `period_report` or `fast_mlsirm`. Each leftover
row also names unexplained leftover share `s = U_c² / R̃²` of
centered leftover when Gabriel coordinates exist so the leftover
cell the two-axis map does not reconstruct is not read as leftover
residual `R` or leftover-map distance `d` (ADR 0183). Centered leftover
`U_c` and two-axis reconstruction `R̂_c` are computed internally and
are not persisted.

Cascade the rows with `report_period_score`. A leftover post must
also be a `report_member_score` row, and the leftover criterion
Expand Down
9 changes: 7 additions & 2 deletions docs/adr/0049-leftover-pair-report-ui.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,13 @@ On each period-report group, render leftover pairs **above** the
member list. Each pair is a button: closest or farthest label, post
title, criterion short label, leftover-map distance, and the next
action (“Open this post to read the criterion it sat closest to /
farthest from after main effects.”). Clicking the button opens that
post with the same handler as a member row.
farthest from after main effects.”). Unexplained leftover share next
action: leftover map leaves unexplained share `s` of centered leftover
after IRT main effects; open this post to read the named criterion. A
missing share keeps the existing closest/farthest next action.
Clicking the button opens that post with the same handler as a member
row. Unexplained leftover share naming is
[ADR 0183](0183-leftover-map-unexplained-share.md).

After `make seed`, closest and farthest leftover pairs sit above the
member list. Click a pair to open that post.
Expand Down
100 changes: 100 additions & 0 deletions docs/adr/0183-leftover-map-unexplained-share.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# ADR 0183 — Name unexplained leftover share on period-report pair rows

**Decision status:** Accepted
**Date:** 2026-08-24

Amends [ADR 0048](0048-persist-lsirm-leftover-pairs.md) and
[ADR 0049](0049-leftover-pair-report-ui.md).

## Context

ADR 0048 already persists leftover-map distance `d = ‖ξ_p − ζ_i‖` and
leftover residual `R = Y − E[Y|θ, item]` on `report_leftover_pair`.
ADR 0049 already renders closest and farthest pairs above the member
list and opens the named post. Distance is the Jeon et al. (2021,
eq. 3) map gap. Gabriel (1971) reconstructs a *centered* matrix from
the biplot as the inner product of person and item coordinates. The
leftover map buyers read is two-axis: unused axes pad with zero, and
hidden SVD axes after the second are dropped. Two-axis reconstruction
`R̂_c = ξ_{1:2} · ζ_{1:2}` therefore recovers centered leftover
`R̃ = R − center`, not raw residual `R`. Hiding unexplained leftover
share `s = U_c² / R̃²` (`U_c = R̃ − R̂_c`) lets a buyer read leftover
residual `R` or leftover-map distance `d` as the leftover the two-axis
map does not reconstruct. Subtracting reconstruction from raw `R`
(or dividing by `R²`) leaves the grand mean inside the named leftover,
so a fully reconstructed rank-1 cell would look unexplained whenever
`center ≠ 0`.

This increment does not persist leftover-map reconstruction `R̂` or
`R̂_c`, does not persist leftover-map unexplained leftover `U` or
`U_c`, does not persist leftover-map coordinates, does not name
leftover-map inner product, cosine, or length, does not name observed
`Y` / expected `E`, does not name leftover-map rank, does not split
leftover-map distance onto two axes, and does not land Post quality on
the leftover criterion. Leftover-map distance stays full-rank
Euclidean. Centered leftover `U_c` and reconstruction `R̂_c` are
computed internally so `s` is honest, then discarded.

The unprotected-stack reconstructions for neighbouring leftover facts
use 0162–0182. This protected-main increment uses **0183** so it does
not collide with leftover-map unexplained leftover (0182), leftover-map
reconstruction (0181), leftover-map length (0181 on the length stack),
leftover-map cosine (0180), leftover-map inner product (0179), leftover
residual disclosure (0178), leftover observed `Y` / expected `E`
(0170), leftover-map rank (0172), two-axis leftover-map distance
(0166), leftover coverage (0165 / 0168), leftover-map axis share
(0148), or leftover interaction-map persistence (0121).

## Decision

Each leftover pair names `leftover_map_unexplained_share` — unexplained
leftover share `s = U_c² / R̃²` of centered leftover after two-axis
Gabriel reconstruction `R̂_c = ξ_{1:2} · ζ_{1:2}`. Migration `0183`
is the single source of the column on every install path, fresh or
existing -- shipped migrations (`0001` / `0012`) are never edited after
the fact. The column is nullable so older leftover rows keep distance
and residual without fabricating a share. Fallback pairs that have no
complete-case leftover map omit the value rather than inventing one.
A rank-0 origin map stores `0.0` (`R̃ = 0` and `U_c = 0`), not a
missing value. A fully reconstructed rank-1 cell stores `0.0` even when
the grand mean of `R` is nonzero. A non-finite or negative share stores
null rather than inventing a leftover score. Named check
`leftover_pair_unexplained_share_nonnegative_chk` rejects a stored
negative share. Do not persist `leftover_map_unexplained` or
`leftover_map_reconstruction`.

The pair button shows `U²/R̃² {share}` next to leftover-map distance
`d` when the value is a finite non-negative number. Next action:
leftover map leaves unexplained share `s` of centered leftover after
IRT main effects; open this post to read the named criterion. A
missing or non-finite share omits the badge and keeps the existing
closest/farthest next action. Do not invent a leftover score. Do not
invent a theta.

## Consequences

`GET /api/reports/{grouping}/{period}` returns
`leftover_map_unexplained_share`. After `make seed`, closest and
farthest leftover pairs sit above the member list with named
`U²/R̃²` next to `d`; click opens that post. Hidden posts stay hidden.

## Related

Independent of leftover interaction-map persistence, leftover-criterion
evaluation landing, leftover residual disclosure, leftover observed
`Y` / expected `E`, leftover-map complete-case coverage, leftover-map
axis share, leftover pairs on the grouping comparison strip, two-axis
leftover-map distance, leftover-map rank, leftover-map inner product,
leftover-map cosine, leftover-map length, leftover-map reconstruction,
and leftover-map unexplained leftover.

## References

Gabriel, K. R. (1971). The biplot graphic display of matrices with
application to principal component analysis. *Biometrika, 58*(3),
453–467. https://doi.org/10.1093/biomet/58.3.453

Jeon, M., Jin, I. H., Schweinberger, M., & Baugh, S. (2021). Mapping
unobserved item–respondent interactions: A latent space item response
model with interaction map. *Psychometrika, 86*(2), 378–403.
https://doi.org/10.1007/s11336-021-09762-5
2 changes: 1 addition & 1 deletion frontend/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "frontend",
"private": true,
"version": "2.12.6",
"version": "2.12.27",
"type": "module",
"scripts": {
"dev": "vite",
Expand Down
8 changes: 6 additions & 2 deletions frontend/src/App.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -932,6 +932,7 @@ describe("App, authenticated", () => {
criterion_code: "sales_lead_specificity",
leftover_distance: 0.12,
leftover_residual: 0.4,
leftover_map_unexplained_share: 0.12,
},
{
pair_kind: "farthest",
Expand All @@ -940,6 +941,7 @@ describe("App, authenticated", () => {
criterion_code: "general_sentiment_negative",
leftover_distance: 1.84,
leftover_residual: -1.1,
leftover_map_unexplained_share: 0.45,
},
],
members: [
Expand Down Expand Up @@ -3357,13 +3359,15 @@ describe("App, authenticated", () => {
});
expect(closestPair).toHaveTextContent("Closest leftover: Public post · sales-lead");
expect(closestPair).toHaveTextContent(
"Open this post to read the criterion it sat closest to after main effects.",
"Leftover map leaves unexplained share 0.12 of centered leftover after IRT main effects. Open this post to read sales-lead.",
);
expect(closestPair).toHaveTextContent("U²/R̃² 0.12");
expect(closestPair).toHaveTextContent("d 0.12");
expect(farthestPair).toHaveTextContent("Farthest leftover: Specification revision requested · negative");
expect(farthestPair).toHaveTextContent(
"Open this post to read the criterion it sat farthest from after main effects.",
"Leftover map leaves unexplained share 0.45 of centered leftover after IRT main effects. Open this post to read negative.",
);
expect(farthestPair).toHaveTextContent("U²/R̃² 0.45");
expect(farthestPair).toHaveTextContent("d 1.84");
const memberButton = screen.getByRole("button", { name: /open report post: public post/i });
expect(closestPair.compareDocumentPosition(memberButton) & Node.DOCUMENT_POSITION_FOLLOWING).toBeTruthy();
Expand Down
Loading
Loading