docs(research): uncertainty-aware retrieval analysis (#84) - #137
Closed
jphein wants to merge 2 commits into
Closed
Conversation
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
jphein
force-pushed
the
docs/uncertainty-research-84
branch
from
May 23, 2026 01:05
9ea1483 to
90cb6d2
Compare
Analyses the MIT CSAIL paper "Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty" (Damani et al., arXiv:2507.16806, ICLR 2026) for applicability to mempalace's hybrid search pipeline. Conclusion: RLCR (the headline technique) does not transfer directly because it is an RL fine-tuning method for generative reasoning LMs and mempalace's retrieval pipeline is not generative-LM-mediated. Two transferable kernels identified for a follow-up spike: 1. Calibrated confidence field on search hits, fit via isotonic regression on existing probe sets. Storage ~1KB, query cost sub-microsecond, zero new dependencies. 2. Brier-score column in scripts/eval_multi_encoder_rrf.py so every future retrieval change is evaluated on calibration alongside MRR and Recall. The doc covers: paper summary with verbatim formulas + Qwen2.5-7B benchmark table; per-source calibration treatment for vector/BM25/AGE candidates; why a confidence threshold should not become a suppression gate (closets-as-signal-not-gate philosophy); cost-vs-fidelity tradeoffs across three label sources; explicit non-transfer list; revisit conditions for the headline technique. Recommendation: defer the headline RLCR integration, spike the calibration kernel behind a config flag, ship the Brier-score eval column unconditionally. Closes #84.
Fills in the previously-TBD commit field on the uncertainty-aware-retrieval-research fork-changes entry and re-renders FORK_CHANGELOG.md / README.md so the changelog points at the actual commit instead of a TBD placeholder.
jphein
force-pushed
the
docs/uncertainty-research-84
branch
from
May 23, 2026 01:10
90cb6d2 to
83d4730
Compare
Collaborator
Author
|
Superseded by clean rebased PR. |
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #84.
Analysis of the MIT CSAIL paper Beyond Binary Rewards: Training LMs
to Reason About Their Uncertainty
(Damani et al., ICLR 2026) for applicability to mempalace's hybrid
search pipeline.
Conclusion: defer the headline technique, ship two transferable kernels.
RLCR (Reinforcement Learning with Calibration Rewards) is an RL
fine-tuning method for generative reasoning LMs. mempalace's
retrieval pipeline is not generative-LM-mediated, so the headline
technique does not transfer directly. The press-release framing
"teach AI to say I'm not sure" collapses two distinct ideas —
model-generation uncertainty (what the paper addresses) and
retrieval-ranking uncertainty (what issue #84 actually asks about).
Two transferable kernels worth a 1-week follow-up spike:
confidencefield on search hits. Apply isotonicregression on top of the existing
similarityscore using then=200 git-derived probe set. Storage ~1KB, query cost ~µs,
zero new dependencies, no model training. Surface as an optional
field (not a suppression gate, per the fork's
closets-as-signal-not-gate philosophy).
scripts/eval_multi_encoder_rrf.pyso every future retrieval change is evaluated on calibration
alongside MRR and Recall. ~20 LOC, permanent value-add even if
the calibration kernel itself never ships.
The doc covers paper summary with verbatim formulas + Qwen2.5-7B
benchmark table; per-source calibration treatment for
vector/BM25/AGE-graph candidates; cost-vs-fidelity tradeoffs across
three label sources; explicit non-transfer list (RL training of
embeddings, LLM-verbalised confidence per hit, confidence-weighted
majority vote); revisit conditions (RAG endpoint shipping,
drawer-read telemetry available, hybrid becoming the default
candidate strategy).
Files
docs/research/uncertainty-aware-retrieval.md(new, ~21KB)docs/fork-changes.yaml— new `uncertainty-aware-retrieval-research` entryFORK_CHANGELOG.md,README.md— re-rendered from yamlTest plan
_hybrid_rankbehaviourNo code changes in this PR — analysis only.