Skip to content
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ reimplementing them:
multi-channel score fusion (`weighted_convex_fuse` in
`reconstruct.py`) and the buyer-facing Rankings port
(`rankweave_client.py`) -- never invent a fused score or a theta.
Rankings disclose Cormack RRF channel contributions from owned
rank lists (ADR 0167).
- [TEPP](https://github.com/ContextualWisdomLab/TEPP)'s published wire
contract for calibrated measurement (`tepp_client.py`) -- never
reimplement TEPP's model here.
Expand Down
5 changes: 4 additions & 1 deletion ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,10 @@ flowchart LR
`rankweave_client.py`'s default transport raises
`RankWeaveNotAvailable`. `GET /api/rankings` then returns
`rankweave_not_available` and an empty ranking list. Hidden posts
are omitted from every channel. See ADR 0024.
are omitted from every channel. Accepted hits include
`channel_evidence` computed from owned temporal/lexical ranks
(Cormack weighted RRF contribution); RankWeave extra fields are
ignored and no theta is invented. See ADR 0024 and ADR 0167.

## Standards and citations

Expand Down
11 changes: 11 additions & 0 deletions CHANGELOG.d/2.12.17-rankweave-ranking-channel-evidence.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# 2.12.17 — RankWeave ranking channel evidence

Rankings now shows why a fused hit landed.

- `GET /api/rankings` attaches owned-channel evidence: 1-based
newest-first and title-overlap ranks plus Cormack RRF contribution
`weight / (η + rank)` with η = 60.
- A channel the post is missing from is omitted. RankWeave extra
fields are ignored. No fused score or theta is invented.
- The Rankings list discloses that evidence under each hit. Copy
states this is not a calibrated score. Click still opens the post.
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,15 @@ All notable changes to this project are documented here. Format follows
environment, so local OIDC and synthetic-data workflows resolve the same
pinned dependencies as CI.

## [2.12.17] - 2026-08-24

### Added

- Rankings now names the newest-first and title-overlap ranks that
RankWeave fused, with each channel's Cormack RRF contribution.
Missing channels stay omitted. This is ranking evidence, not a
calibrated score. Click still opens that post.

## [2.12.15] - 2026-08-24

### Added
Expand Down
7 changes: 4 additions & 3 deletions backend/app/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -3123,11 +3123,12 @@ async def read_rankings(
account: CurrentAccount = Depends(get_current_account),
pool: asyncpg.Pool = Depends(get_pool),
) -> dict[str, Any]:
"""RankWeave fusion of ABAC-visible posts (ADR 0024).
"""RankWeave fusion of ABAC-visible posts (ADR 0024 / ADR 0167).

Hidden posts are omitted from every channel. Never invents a fused
score or a theta. Fail-closed when RankWeave is disabled or the
library is missing.
score or a theta. Channel evidence is computed from owned rank
lists. Fail-closed when RankWeave is disabled or the library is
missing.
"""
_require_post_read(account)
async with pool.acquire() as conn:
Expand Down
5 changes: 5 additions & 0 deletions docs/adr/0024-rankweave-fusion-fail-closed.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,11 @@ tables, and does not bind the demo IdP to production Keyverse.
4. After login, Rankings sits above Calendar. Unavailable copy is
**Rankings · RankWeave not available**. An accepted hit lists the
title; click opens that `source_post`.
5. Accepted hits also disclose owned-channel evidence (ADR 0167):
1-based `channel_rank` and Cormack contribution
`weight / (η + rank)` for each channel the post actually appears
in. Missing channels are omitted. RankWeave extra fields are
ignored. Copy states this is not a calibrated score.

## Consequences

Expand Down
67 changes: 67 additions & 0 deletions docs/adr/0167-rankweave-ranking-channel-evidence.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# ADR 0167 — Disclose RankWeave ranking channel evidence

**Decision status:** Accepted
**Date:** 2026-08-23

## Context

ADR 0024 already fuses ABAC-visible posts through RankWeave weighted
reciprocal-rank fusion (Cormack et al., 2009, η = 60; Samuel et al.,
2025, unequal weights). `GET /api/rankings` returned only `post_id`,
`post_title`, and 1-based `fused_rank`. A reader could open the hit
but could not see which owned channel ranked it, or how much that
rank contributed.

Event Lineage channel evidence (ADR 0124 on #387) explains
reconstructed parent→child edges from persisted convex-fusion
scores. Rankings is a different surface: two rank-only channels
(`temporal`, `lexical`) fused in-process at GET time. There is no
persisted ranking table and no TEPP theta. RankWeave extra fields
must not be trusted; a missing channel stays missing.

This ADR does not replace reconstruction fusion, leftover pairs,
TEPP receipts, Allen interval labels, or Event Lineage isolation
reasons.

## Decision

1. Compute ranking channel evidence from LineageWeave-owned rank
lists, never from RankWeave payload extras. For each fused hit
and each channel with a positive weight, take the 1-based rank
of that `post_id` in the ordered id list. Skip a channel the
post is absent from. Do not invent a rank.
2. Contribution is Cormack weighted RRF:
`weight / (η + rank)` with η = 60. Sort by contribution
descending, then `signal_code`. `rank` on the evidence row is
that 1-based evidence order. `channel_rank` is the 1-based
position in that channel.
3. Labels: `temporal` = **Newest first**, `lexical` = **Title
overlap**. The payload never includes a fused score or a theta.
4. Rankings lists the evidence under each accepted hit as an
accessible sibling list, not hover-only. Copy states that
RankWeave fused newest-first and title-overlap ranks, and that
this is not a calibrated score. Click still opens that post.
5. Unavailable RankWeave stays empty (`rankweave_not_available`).
Hidden posts remain omitted from every channel.

## Consequences

Buyers can see why a Rankings hit landed without treating RRF as
measurement. A later channel or weight change recomputes evidence
on the next GET; nothing is persisted. Event Lineage evidence
remains the reconstruct/convex path.

## References

Cormack, G. V., Clarke, C. L. A., & Buettcher, S. (2009). Reciprocal
rank fusion outperforms Condorcet and individual rank learning
methods. In *Proceedings of the 32nd international ACM SIGIR
conference on Research and development in information retrieval*
(pp. 758–759). ACM. https://doi.org/10.1145/1571941.1572114

Samuel, D., MacAvaney, S., Yates, A., Zhang, E., Zhang, S.,
Macdonald, C., & Ounis, I. (2025). *Weighted reciprocal rank fusion
for multi-channel retrieval* [Preprint].

ADR 0024 (RankWeave fusion fail-closed)
ADR 0124 (Event Lineage channel evidence; separate surface)
2 changes: 1 addition & 1 deletion docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ decision from them.
| Supporting document | Normative ADR |
|---|---|
| [`product-technical-gap-baseline.md`](../product-technical-gap-baseline.md) | Product/technical traceability projection across the ADR set; ADRs remain normative |
| [`lineage-bi-research-notes.md`](../lineage-bi-research-notes.md) | [0084](0084-lineage-research-grounding.md), [0062](0062-semantic-unit-embedding.md), [0064](0064-lineage-evidence-and-tree-assembly.md) |
| [`lineage-bi-research-notes.md`](../lineage-bi-research-notes.md) | [0084](0084-lineage-research-grounding.md), [0062](0062-semantic-unit-embedding.md), [0064](0064-lineage-evidence-and-tree-assembly.md), [0024](0024-rankweave-fusion-fail-closed.md), [0167](0167-rankweave-ranking-channel-evidence.md) |
| [`PROV_O_IMPLEMENTATION.md`](../PROV_O_IMPLEMENTATION.md) | [0065](0065-prov-o-provenance-boundary.md) |
| [`PROV_O_IMPLEMENTATION_MATRIX.md`](../PROV_O_IMPLEMENTATION_MATRIX.md) | [0065](0065-prov-o-provenance-boundary.md) |
| [`ONTOLOGY_NAMESPACE_INVENTORY.md`](../doctoring/ONTOLOGY_NAMESPACE_INVENTORY.md) | [0157](0157-public-ontology-namespace-identity.md) |
Expand Down
2 changes: 1 addition & 1 deletion frontend/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "frontend",
"private": true,
"version": "2.12.15",
"version": "2.12.17",
"type": "module",
"scripts": {
"dev": "vite",
Expand Down
21 changes: 21 additions & 0 deletions frontend/src/App.css
Original file line number Diff line number Diff line change
Expand Up @@ -706,6 +706,27 @@
gap: 0.5rem;
}

.ranking-hit {
flex-direction: column;
align-items: stretch;
justify-content: flex-start;
}

.ranking-channel-evidence-copy {
margin: 0 0 0.75rem;
}

.ranking-channel-evidence {
list-style: none;
margin: 0;
padding: 0 0 0 0.5rem;
font-size: 0.85rem;
}

.ranking-channel-evidence li {
margin: 0.15rem 0;
}

.ticket-title {
flex: 1;
}
Expand Down
55 changes: 55 additions & 0 deletions frontend/src/App.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,14 @@ describe("App, authenticated", () => {
post_id: string;
post_title: string;
fused_rank: number;
channel_evidence?: {
signal_code: string;
signal_label: string;
channel_rank: number;
weight: number;
contribution: number;
rank: number;
}[];
}[];
};
chatUnavailable?: boolean;
Expand Down Expand Up @@ -2592,11 +2600,47 @@ describe("App, authenticated", () => {
post_id: "post-1",
post_title: "Public post",
fused_rank: 1,
channel_evidence: [
{
signal_code: "lexical",
signal_label: "Title overlap",
channel_rank: 2,
weight: 0.75,
contribution: 0.75 / 62,
rank: 1,
},
{
signal_code: "temporal",
signal_label: "Newest first",
channel_rank: 2,
weight: 0.25,
contribution: 0.25 / 62,
rank: 2,
},
],
},
{
post_id: "post-2",
post_title: "Pricing renegotiation: revised quote sent",
fused_rank: 2,
channel_evidence: [
{
signal_code: "lexical",
signal_label: "Title overlap",
channel_rank: 1,
weight: 0.75,
contribution: 0.75 / 61,
rank: 1,
},
{
signal_code: "temporal",
signal_label: "Newest first",
channel_rank: 1,
weight: 0.25,
contribution: 0.25 / 61,
rank: 2,
},
],
},
],
},
Expand All @@ -2609,6 +2653,17 @@ describe("App, authenticated", () => {
expect(rankingButton).toHaveTextContent("Public post");
expect(rankingButton).toHaveTextContent("Rankings · rankweave");
expect(rankingButton).toHaveTextContent("rank 1");
expect(
screen.getByText(
"RankWeave fused newest-first and title-overlap ranks. This is not a calibrated score.",
),
).toBeInTheDocument();
expect(
screen.getByRole("list", { name: "Ranking evidence for Public post" }),
).toHaveTextContent("Title overlap rank 2, contribution 0.012097");
expect(
screen.getByRole("list", { name: "Ranking evidence for Public post" }),
).toHaveTextContent("Newest first rank 2, contribution 0.004032");
expect(screen.queryByRole("button", { name: /open ranking: private parent/i })).not.toBeInTheDocument();

await userEvent.click(rankingButton);
Expand Down
69 changes: 48 additions & 21 deletions frontend/src/App.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -3059,6 +3059,10 @@ function AnalysisRunsPanel({
);
}

function formatRankingContribution(value: number): string {
return value.toFixed(6);
}
Comment thread
seonghobae marked this conversation as resolved.

function RankingsPanel({
accessToken,
onSelectPost,
Expand All @@ -3077,9 +3081,9 @@ function RankingsPanel({
}, [accessToken]);

return (
<section className="popup-section lineage-home" aria-label="Rankings">
<section className="popup-section lineage-home" aria-label={t("Rankings")}>
<div className="lineage-home-header">
<h2>Rankings</h2>
<h2>{t("Rankings")}</h2>
{ranking && (
<span className="post-badge">
{ranking.status === "accepted"
Expand All @@ -3089,30 +3093,53 @@ function RankingsPanel({
)}
</div>
{error && <p className="error">{error}</p>}
{ranking === null && !error && <p>Loading rankings...</p>}
{ranking === null && !error && <p>{t("Loading rankings...")}</p>}
{ranking && ranking.status === "unavailable" && (
<p className="popup-placeholder">Rankings · RankWeave not available</p>
<p className="popup-placeholder">{t("Rankings · RankWeave not available")}</p>
)}
{ranking && ranking.status === "accepted" && ranking.rankings.length === 0 && (
<p className="popup-placeholder">No fused rankings from RankWeave.</p>
<p className="popup-placeholder">{t("No fused rankings from RankWeave.")}</p>
)}
{ranking && ranking.rankings.length > 0 && (
<ul className="ticket-list" aria-label="Fused rankings">
{ranking.rankings.map((hit) => (
<li key={hit.post_id} className="ticket-list-item">
<button
className="post-list-item"
aria-label={`Open ranking: ${hit.post_title}`}
onClick={() => onSelectPost(hit.post_id)}
>
<span className="ticket-title">{hit.post_title}</span>
<span className="post-badge">Rankings · rankweave</span>
<span className="post-badge">rank {hit.fused_rank}</span>
</button>
</li>
))}
</ul>
)}
<>
<p className="ranking-channel-evidence-copy">
{t(
"RankWeave fused newest-first and title-overlap ranks. This is not a calibrated score.",
)}
</p>
<ul className="ticket-list" aria-label={t("Fused rankings")}>
{ranking.rankings.map((hit) => (
<li key={hit.post_id} className="ticket-list-item ranking-hit">
<button
className="post-list-item"
aria-label={tf("Open ranking: {title}", { title: hit.post_title })}
onClick={() => onSelectPost(hit.post_id)}
>
<span className="ticket-title">{hit.post_title}</span>
<span className="post-badge">{t("Rankings · rankweave")}</span>
<span className="post-badge">{tf("rank {rank}", { rank: String(hit.fused_rank) })}</span>
</button>
{(hit.channel_evidence ?? []).length > 0 ? (
<ul
className="ranking-channel-evidence"
aria-label={tf("Ranking evidence for {title}", { title: hit.post_title })}
>
{(hit.channel_evidence ?? []).map((item) => (
<li key={item.signal_code}>
{tf("{label} rank {rank}, contribution {contribution}", {
label: t(item.signal_label),
rank: String(item.channel_rank),
contribution: formatRankingContribution(item.contribution),
})}
Comment thread
seonghobae marked this conversation as resolved.
</li>
))}
</ul>
) : null}
</li>
))}
</ul>
</>
)}
</section>
);
}
Expand Down
10 changes: 10 additions & 0 deletions frontend/src/api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1037,10 +1037,20 @@ export function startAnalysisRun(
});
}

export interface RankingChannelEvidence {
signal_code: string;
signal_label: string;
channel_rank: number;
weight: number;
contribution: number;
rank: number;
}

export interface RankedPost {
post_id: string;
post_title: string;
fused_rank: number;
channel_evidence?: RankingChannelEvidence[];
}

export interface RankingList {
Expand Down
3 changes: 3 additions & 0 deletions frontend/src/i18n.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,9 @@ describe("i18n", () => {
"Read leftover map rank {rank}, observed Y {observed}, and expected E {expected} after IRT main effects, then open this post.",
"Leftover map rank 0 means no leftover structure after IRT main effects. Read observed Y {observed} and expected E {expected}, then open this post.",
"Showing the first {shown} of {total} posts known at this cutoff.",
"Rankings",
"Title overlap",
"RankWeave fused newest-first and title-overlap ranks. This is not a calibrated score.",
] as const;

it("supports the five product locales", () => {
Expand Down
Loading
Loading