feat(lineage): estimate channel-fusion weights psychometrically (ADR 0145)

[seonghobae]

Summary

The standing product requirement — repeated across many sessions — is that scoring weights be grounded in published measurement methodology via the org's own psychometric libraries, never hand-picked. reconstruct()'s DEFAULT_CHANNEL_WEIGHTS (0.15/0.15/0.30/0.40) had no citation; only a code comment. This PR replaces assertion with estimation:

• ADR 0145 (docs/adr/0145-psychometric-channel-weight-estimation.md): channels = items observing the latent "these posts are genuinely related" trait; scored candidate pairs = respondents; reconstruction groups = multilevel nesting. Weights = normalized natural-scale discriminations from fast-mlsirm's MLS2PLM (mmle estimator with per-group cluster random intercepts — Fox & Glas, 2001), the information-optimal weighting per Birnbaum (1968); multilevel to avoid the atomistic fallacy (Robinson, 1950). Full APA 7th references in the ADR.
• lineageweave/channel_weight_estimation.py: fail-closed at every step — fast_mlsirm unimportable, sample < 200 pairs, a degenerate channel (never/always clears the fusion floor), or any non-finite alpha → None, and rebuilds keep the fallback constants (now labeled as ungrounded fallback, not justified default). Dichotomization happens at DEFAULT_MIN_FUSED_SCORE so the measurement model observes the same binary event the fusion decision acts on.
• Migration 0135 lineage_channel_weight (+ rollback + migrate.sh whitelist + test-harness registration): persisted, provenance-bearing weights (method code, sample size, timestamp). rebuild_lineage uses them only on an exact active-channel-set match — no partial mixing of estimated and hand-picked weights; a DB without the migration is treated as "no estimate persisted", not an error.
• scripts/estimate_channel_weights.py: samples pairs with reconstruct's exact candidate geometry (same grouping fallback, same trailing window), fits, persists. llm channel joins in a follow-up per ADR 0145 §5.
• scripts/backfill_thread_group_keys.py: the estimation's prerequisite. The zcrht811 import put per-row-unique identifiers in BOTH grouping columns (thread_group_key = own record GUID on 43,811/43,839 rows; secondary_grouping_key = own document number), so _group_by saw ~43k singleton groups and no channel ever scored a real pair — Event Lineage silently non-functional corpus-wide. The backfill clears placeholder-signature rows to '' (migration 0002's designed no-signal value → process-unit/corporate-entity candidate pools) and routes source_project_code into secondary_grouping_key — the secondary-key channel's own documented signal ("e.g. project code") — as fused evidence, never a hard partition that would wall related posts apart (the earlier draft made exactly that mistake; caught in design review). Fails closed if an analysis_scope_thread_group run (TEPP/report scope keys resolve live against thread_group_key) would be orphaned.

Verification

• Parameter recovery (the org's measurement standard): planted discrimination ordering strong > mid > weak recovered by the real fast-mlsirm fit, with genuine per-cluster base-rate variance (the structure random intercepts model). Probed estimator configs first: jmle blows up (disc ≈ 654 — classic JMLE slope divergence), mmle+cluster on structureless clusters flattens slopes; both findings documented in test comments.
• Fail-closed live check: estimation script against the current (un-backfilled) corpus refuses cleanly — 105 pairs < 200 minimum, nothing written.
• pytest tests/test_channel_weight_estimation.py tests/test_estimate_channel_weights_script.py tests/test_backfill_thread_group_keys.py — 14 passed, 1 honest skip.
• backend/tests/test_api.py::test_rebuild_lineage_recovers_the_a100_fork + lineage-ingestion suite — green after harness migration registration.
• Full suite: 876 passed; remaining failures are pre-existing/environmental (orchestrator-unavailable ×2, resolved_date_text ×1), none in files this PR touches.

Operator sequence (not run here — mass rewrite of shared data needs explicit operator action)

1. python scripts/backfill_thread_group_keys.py --dry-run → review counts → run without --dry-run
2. python scripts/estimate_channel_weights.py --dry-run → review weights → run without --dry-run
3. POST /api/lineage/rebuild

🤖 Generated with Claude Code

https://claude.ai/code/session_015HgzNGsCxqiTaT4YuJEb5J

---

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 8ac8b9d3-5737-4287-b978-c819af3bd7c5

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[devin-ai-integration]

Devin Review found 3 potential issues.