feat: per-drawer salience (dynamics extended to drawers) - #1950
Open
et1975 wants to merge 2 commits into
Open
Conversation
3 tasks
ATKabli
added a commit
to ATKabli/mempalace
that referenced
this pull request
Aug 8, 2026
Issue MemPalace#1629 asked why wake-up did not read like the essential story it promises. Ordering was half the answer and is fixed. The other half is what gets in: L1 has room for about 15 lines, and it filled them with whatever sorted highest, including harness injection blocks, raw tool output, markdown tables, bare timestamp lines, three near-identical chunks of one transcript, and snippets that opened mid-word. On a 149k-drawer palace the result read as fragment soup. Selection now happens between sorting and rendering. _l1_salience(text) scores a candidate. Negative means junk and drops it. Exclusion is deliberately narrow and structural, never lexical: - a harness wrapper *opening* the drawer (<system-reminder>, <command-message>, <local-command-caveat>, <task-notification>, [SYSTEM NOTIFICATION), matched with startswith after normalization - a timestamp-only or rule-only drawer - text below a 20-character floor - pipe/dash density above 0.18, which is a table, a diff or an ASCII rule - a prose ratio below 0.78, the fraction of characters that are letters or spaces The prose ratio is the load-bearing rule and the general form of a marker list: JSON, ls output, env dumps, URL lists, log lines and bare code fences all fail it without any marker being named, while an engineering sentence quoting a path, a version pin or a fence passes. Measured over the 21-item corpus in test_l1_salience_drops_soup_without_markers, soup lands in 0.40 to 0.82 and prose in 0.84 to 0.99. The floor sits at 0.78, deliberately below the gap: a false drop loses a real memory forever, a false keep costs one wake-up line. The residue that clears it, word-heavy tracebacks and diffs, is demoted rather than excluded. Nothing is excluded for merely mentioning tooling. tool_use, tool_result, fences, "Exit code:" and "cwd was reset" cost a ranking point and nothing more, because engineering writing quotes those constantly. Positive scoring is outcome wording (2 points), a clean sentence or heading start (1), and freedom from tool noise (1). Outcome is worth strictly more than both presentation signals, otherwise a drawer that is both an outcome and a little noisy ties with background chatter that merely reads cleanly: outcome + clean 4 chatter + clean 2 outcome + noisy 3 chatter + noisy 1 Anchoring rather than substring matching is the whole correctness argument here. An earlier revision also dropped any body that *repeated* a wrapper, reasoning that a repeat means concatenated injections. That rule was unanchored and re-created the same defect: an engineer writing about why a reminder fired twice names the tag twice. It is gone. Genuine concatenated soup is already caught by density and the prose floor, neither of which needs a marker named. _l1_select applies the caps. At most L1_MAX_PER_SOURCE (2) lines from one source file, so a single long transcript cannot own the story. Drawers with no source_file are uncapped, because the cap exists to stop one known file dominating and lumping unattributed drawers under one empty key would do the opposite. Near-duplicates collapse on whole-body word overlap at 0.8, not on a leading slice: mined session summaries routinely share a templated opening, and matching the first 80 characters collapsed three summaries reporting a migration, a revert and an RTL fix into one and lost two outcomes silently. Whole-body overlap scores that case at 0.27 and keeps all three, while genuine restatements stay above 0.84. The comparison list never exceeds max_drawers, so selection stays linear: measured on this machine at 8.7 ms for 500 candidates, 16.9 ms for 1,000 and 34.2 ms for 2,000, which is the MAX_SCAN ceiling, against the repo's 100 ms startup-injection budget. Rendering groups rooms in the order the ranked list arrives, not alphabetically. Grouping by room and then sorting room names before the MAX_CHARS truncation undid the ranking at the last step: a salience-4 outcome in a room named zzz_outcomes was cut while salience-2 chatter in aaa_trivia rendered. _l1_snippet composes the line: start after the next sentence boundary when a chunk opens mid-sentence, cut on a word boundary. The result is always a contiguous substring of the drawer, never a paraphrase, so the verbatim promise holds. Filtered drawers are untouched in the palace and still returned by L2 and L3. When every candidate scores as junk, L1 renders exactly what it rendered before the filter. An empty wake-up is the worse failure. No LLM call, no embedding, no I/O: a few passes over text already in memory, because L1 runs inside the wake-up hook's latency budget. Two details for non-English palaces. The clean-start point tests "not lowercase" rather than "is uppercase", so Arabic, Hebrew and CJK are not permanently denied a point that every English drawer collects. And L1_OUTCOME_KEYWORDS is English, which its docstring says along with the real two-step for extending it: the regex is compiled once at import, so appending to the tuple alone does nothing, and _l1_compile_outcome_re is exposed for that reason. The junk, cap, duplicate and boundary rules are language-neutral. Known gap, left for a follow-up to keep this in scope: the outcome keyword list has no failure vocabulary. "died", "crashed", "OOM", "timed out" and "regressed" are absent, so a drawer reporting a failure ranks as chatter unless the sentence happens to also say something passed. Orthogonal to PR MemPalace#1950 (per-drawer salience from the dynamics model plus retrieval potentiation). That work scores drawers at retrieval time in dynamics.py/searcher.py/service.py/mcp_server.py; this is snippet-level composition inside layers.py and touches none of those files. If MemPalace#1950 lands, its per-drawer signal can feed the importance key L1 already sorts on, and this filter keeps doing the part it does. Tests: salience edges (anchored wrapper, prose naming a wrapper twice, date-only, table soup, short text, outcome boost, fragment vs whole thought, the ISO-designator false positive, non-Latin prose scoring like Latin), a marker-free soup corpus, noise demotion ordering, snippet boundaries (sentence start, no-boundary passthrough, word-boundary cut, newline collapse), selection (per-source cap, unattributed drawers uncapped, templated-lead summaries kept, near-duplicate suppression, tier order, all-junk empty), render order under the character cap, and end to end through Layer1.generate including the fallback and a verbatim-substring assertion.
ATKabli
added a commit
to ATKabli/mempalace
that referenced
this pull request
Aug 14, 2026
Issue MemPalace#1629 asked why wake-up did not read like the essential story it promises. Ordering was half the answer and is fixed. The other half is what gets in: L1 has room for about 15 lines, and it filled them with whatever sorted highest, including harness injection blocks, raw tool output, markdown tables, bare timestamp lines, three near-identical chunks of one transcript, and snippets that opened mid-word. On a 149k-drawer palace the result read as fragment soup. Selection now happens between sorting and rendering. _l1_salience(text) scores a candidate. Negative means junk and drops it. Exclusion is deliberately narrow and structural, never lexical: - a harness wrapper *opening* the drawer (<system-reminder>, <command-message>, <local-command-caveat>, <task-notification>, [SYSTEM NOTIFICATION), matched with startswith after normalization - a timestamp-only or rule-only drawer - text below a 20-character floor - pipe/dash density above 0.18, which is a table, a diff or an ASCII rule - a prose ratio below 0.78, the fraction of characters that are letters or spaces The prose ratio is the load-bearing rule and the general form of a marker list: JSON, ls output, env dumps, URL lists, log lines and bare code fences all fail it without any marker being named, while an engineering sentence quoting a path, a version pin or a fence passes. Measured over the 21-item corpus in test_l1_salience_drops_soup_without_markers, soup lands in 0.40 to 0.82 and prose in 0.84 to 0.99. The floor sits at 0.78, deliberately below the gap: a false drop loses a real memory forever, a false keep costs one wake-up line. The residue that clears it, word-heavy tracebacks and diffs, is demoted rather than excluded. Nothing is excluded for merely mentioning tooling. tool_use, tool_result, fences, "Exit code:" and "cwd was reset" cost a ranking point and nothing more, because engineering writing quotes those constantly. Positive scoring is outcome wording (2 points), a clean sentence or heading start (1), and freedom from tool noise (1). Outcome is worth strictly more than both presentation signals, otherwise a drawer that is both an outcome and a little noisy ties with background chatter that merely reads cleanly: outcome + clean 4 chatter + clean 2 outcome + noisy 3 chatter + noisy 1 Anchoring rather than substring matching is the whole correctness argument here. An earlier revision also dropped any body that *repeated* a wrapper, reasoning that a repeat means concatenated injections. That rule was unanchored and re-created the same defect: an engineer writing about why a reminder fired twice names the tag twice. It is gone. Genuine concatenated soup is already caught by density and the prose floor, neither of which needs a marker named. _l1_select applies the caps. At most L1_MAX_PER_SOURCE (2) lines from one source file, so a single long transcript cannot own the story. Drawers with no source_file are uncapped, because the cap exists to stop one known file dominating and lumping unattributed drawers under one empty key would do the opposite. Near-duplicates collapse on whole-body word overlap at 0.8, not on a leading slice: mined session summaries routinely share a templated opening, and matching the first 80 characters collapsed three summaries reporting a migration, a revert and an RTL fix into one and lost two outcomes silently. Whole-body overlap scores that case at 0.27 and keeps all three, while genuine restatements stay above 0.84. The comparison list never exceeds max_drawers, so selection stays linear: measured on this machine at 8.7 ms for 500 candidates, 16.9 ms for 1,000 and 34.2 ms for 2,000, which is the MAX_SCAN ceiling, against the repo's 100 ms startup-injection budget. Rendering groups rooms in the order the ranked list arrives, not alphabetically. Grouping by room and then sorting room names before the MAX_CHARS truncation undid the ranking at the last step: a salience-4 outcome in a room named zzz_outcomes was cut while salience-2 chatter in aaa_trivia rendered. _l1_snippet composes the line: start after the next sentence boundary when a chunk opens mid-sentence, cut on a word boundary. The result is always a contiguous substring of the drawer, never a paraphrase, so the verbatim promise holds. Filtered drawers are untouched in the palace and still returned by L2 and L3. When every candidate scores as junk, L1 renders exactly what it rendered before the filter. An empty wake-up is the worse failure. No LLM call, no embedding, no I/O: a few passes over text already in memory, because L1 runs inside the wake-up hook's latency budget. Two details for non-English palaces. The clean-start point tests "not lowercase" rather than "is uppercase", so Arabic, Hebrew and CJK are not permanently denied a point that every English drawer collects. And L1_OUTCOME_KEYWORDS is English, which its docstring says along with the real two-step for extending it: the regex is compiled once at import, so appending to the tuple alone does nothing, and _l1_compile_outcome_re is exposed for that reason. The junk, cap, duplicate and boundary rules are language-neutral. Known gap, left for a follow-up to keep this in scope: the outcome keyword list has no failure vocabulary. "died", "crashed", "OOM", "timed out" and "regressed" are absent, so a drawer reporting a failure ranks as chatter unless the sentence happens to also say something passed. Orthogonal to PR MemPalace#1950 (per-drawer salience from the dynamics model plus retrieval potentiation). That work scores drawers at retrieval time in dynamics.py/searcher.py/service.py/mcp_server.py; this is snippet-level composition inside layers.py and touches none of those files. If MemPalace#1950 lands, its per-drawer signal can feed the importance key L1 already sorts on, and this filter keeps doing the part it does. Tests: salience edges (anchored wrapper, prose naming a wrapper twice, date-only, table soup, short text, outcome boost, fragment vs whole thought, the ISO-designator false positive, non-Latin prose scoring like Latin), a marker-free soup corpus, noise demotion ordering, snippet boundaries (sentence start, no-boundary passthrough, word-boundary cut, newline collapse), selection (per-source cap, unattributed drawers uncapped, templated-lead summaries kept, near-duplicate suppression, tier order, all-junk empty), render order under the character cap, and end to end through Layer1.generate including the fallback and a verbatim-substring assertion.
Member
|
Thanks for this contribution, and apologies for the slow turnaround.
If you'd rather not pick it back up, no problem at all — just say so and I'll close it out, and thanks either way for taking the time to send it. |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…r_id search_memories leaked the internal _parent_drawer_id key to direct callers; salience potentiation now reads the public logical drawer_id field (parent id for chunked drawers) instead, and _finalize_candidate_hits scrubs _parent_drawer_id again. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
et1975
force-pushed
the
feat/1921-drawer-salience
branch
from
August 15, 2026 21:03
e6f2e27 to
b558393
Compare
Contributor
Author
|
@igorls rebased on latest |
ATKabli
added a commit
to ATKabli/mempalace
that referenced
this pull request
Aug 18, 2026
Issue MemPalace#1629 asked why wake-up did not read like the essential story it promises. Ordering was half the answer and is fixed. The other half is what gets in: L1 has room for about 15 lines, and it filled them with whatever sorted highest, including harness injection blocks, raw tool output, markdown tables, bare timestamp lines, three near-identical chunks of one transcript, and snippets that opened mid-word. On a 149k-drawer palace the result read as fragment soup. Selection now happens between sorting and rendering. _l1_salience(text) scores a candidate. Negative means junk and drops it. Exclusion is deliberately narrow and structural, never lexical: - a harness wrapper *opening* the drawer (<system-reminder>, <command-message>, <local-command-caveat>, <task-notification>, [SYSTEM NOTIFICATION), matched with startswith after normalization - a timestamp-only or rule-only drawer - text below a 20-character floor - pipe/dash density above 0.18, which is a table, a diff or an ASCII rule - a prose ratio below 0.78, the fraction of characters that are letters or spaces The prose ratio is the load-bearing rule and the general form of a marker list: JSON, ls output, env dumps, URL lists, log lines and bare code fences all fail it without any marker being named, while an engineering sentence quoting a path, a version pin or a fence passes. Measured over the 21-item corpus in test_l1_salience_drops_soup_without_markers, soup lands in 0.40 to 0.82 and prose in 0.84 to 0.99. The floor sits at 0.78, deliberately below the gap: a false drop loses a real memory forever, a false keep costs one wake-up line. The residue that clears it, word-heavy tracebacks and diffs, is demoted rather than excluded. Nothing is excluded for merely mentioning tooling. tool_use, tool_result, fences, "Exit code:" and "cwd was reset" cost a ranking point and nothing more, because engineering writing quotes those constantly. Positive scoring is outcome wording (2 points), a clean sentence or heading start (1), and freedom from tool noise (1). Outcome is worth strictly more than both presentation signals, otherwise a drawer that is both an outcome and a little noisy ties with background chatter that merely reads cleanly: outcome + clean 4 chatter + clean 2 outcome + noisy 3 chatter + noisy 1 Anchoring rather than substring matching is the whole correctness argument here. An earlier revision also dropped any body that *repeated* a wrapper, reasoning that a repeat means concatenated injections. That rule was unanchored and re-created the same defect: an engineer writing about why a reminder fired twice names the tag twice. It is gone. Genuine concatenated soup is already caught by density and the prose floor, neither of which needs a marker named. _l1_select applies the caps. At most L1_MAX_PER_SOURCE (2) lines from one source file, so a single long transcript cannot own the story. Drawers with no source_file are uncapped, because the cap exists to stop one known file dominating and lumping unattributed drawers under one empty key would do the opposite. Near-duplicates collapse on whole-body word overlap at 0.8, not on a leading slice: mined session summaries routinely share a templated opening, and matching the first 80 characters collapsed three summaries reporting a migration, a revert and an RTL fix into one and lost two outcomes silently. Whole-body overlap scores that case at 0.27 and keeps all three, while genuine restatements stay above 0.84. The comparison list never exceeds max_drawers, so selection stays linear: measured on this machine at 8.7 ms for 500 candidates, 16.9 ms for 1,000 and 34.2 ms for 2,000, which is the MAX_SCAN ceiling, against the repo's 100 ms startup-injection budget. Rendering groups rooms in the order the ranked list arrives, not alphabetically. Grouping by room and then sorting room names before the MAX_CHARS truncation undid the ranking at the last step: a salience-4 outcome in a room named zzz_outcomes was cut while salience-2 chatter in aaa_trivia rendered. _l1_snippet composes the line: start after the next sentence boundary when a chunk opens mid-sentence, cut on a word boundary. The result is always a contiguous substring of the drawer, never a paraphrase, so the verbatim promise holds. Filtered drawers are untouched in the palace and still returned by L2 and L3. When every candidate scores as junk, L1 renders exactly what it rendered before the filter. An empty wake-up is the worse failure. No LLM call, no embedding, no I/O: a few passes over text already in memory, because L1 runs inside the wake-up hook's latency budget. Two details for non-English palaces. The clean-start point tests "not lowercase" rather than "is uppercase", so Arabic, Hebrew and CJK are not permanently denied a point that every English drawer collects. And L1_OUTCOME_KEYWORDS is English, which its docstring says along with the real two-step for extending it: the regex is compiled once at import, so appending to the tuple alone does nothing, and _l1_compile_outcome_re is exposed for that reason. The junk, cap, duplicate and boundary rules are language-neutral. Known gap, left for a follow-up to keep this in scope: the outcome keyword list has no failure vocabulary. "died", "crashed", "OOM", "timed out" and "regressed" are absent, so a drawer reporting a failure ranks as chatter unless the sentence happens to also say something passed. Orthogonal to PR MemPalace#1950 (per-drawer salience from the dynamics model plus retrieval potentiation). That work scores drawers at retrieval time in dynamics.py/searcher.py/service.py/mcp_server.py; this is snippet-level composition inside layers.py and touches none of those files. If MemPalace#1950 lands, its per-drawer signal can feed the importance key L1 already sorts on, and this filter keeps doing the part it does. Tests: salience edges (anchored wrapper, prose naming a wrapper twice, date-only, table soup, short text, outcome boost, fragment vs whole thought, the ISO-designator false positive, non-Latin prose scoring like Latin), a marker-free soup corpus, noise demotion ordering, snippet boundaries (sentence start, no-boundary passthrough, word-boundary cut, newline collapse), selection (per-source cap, unattributed drawers uncapped, templated-lead summaries kept, near-duplicate suppression, tier order, all-junk empty), render order under the character cap, and end to end through Layer1.generate including the fallback and a verbatim-substring assertion.
igorls
added
enhancement
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds per-drawer salience by reusing the existing dynamics model (Hebbian potentiation, Ebbinghaus decay, and Cepeda spacing) for drawer metadata as well as graph connections. Drawer reads now expose a lazy-decayed
salienceblock from copied metadata, withfiled_atused as the drawer fallback forlast_activatedinitialization.This is backward-compatible and default-safe: metadata fields are additive, search/get read paths remain read-only by default, and retrieval potentiation is opt-in only via
MEMPALACE_SALIENCE_POTENTIATE=true. When enabled, search updates drawer metadata only (no re-embedding), skips writes on read-only servers or when the MCP writer lock is unavailable, and keeps chunked logical drawers consistent across all physical chunks.Also adds the read-only
mempalace_drawer_saliencetool and wires it throughTOOLSandservice.READ_TOOLS.Closes #1921
How to test
uv run pytest tests/test_drawer_salience.py -v→ 10 passeduv run pytest tests/test_dynamics.py -v→ 25 passeduv run pytest tests/test_searcher.py -v→ 45 passeduv run pytest tests/test_closets.py -v→ green after the_parent_drawer_idregression fixuv run pytest tests/ --ignore=tests/benchmarks→ 3268 passed; 3 unrelated pre-existing develop failures (2 tracked in test_repair.py: 2 tests fail on SQLite 3.53.2 — FTS5 corruption message wording mismatch in _errors_are_isolated_fts5 #1925, 1 antigravity install)uv run ruff check .→ cleanuv run ruff format --check .→ cleanNote: the second commit restores
_parent_drawer_idscrubbing from public search hits, exposes the logical drawer id asdrawer_id, and uses that field for potentiation;tests/test_closets.pycovers the regression.Checklist
python -m pytest tests/ -v)ruff check .)