feat(layers): select L1 wake-up drawers by salience

[ATKabli]

What does this PR do?

#1629 asked why wake-up did not read like the "essential story" it promises. Ordering was half the answer and is fixed. The other half is what gets in.

L1 has room for about 15 lines and fills them with whatever sorted highest. On a real mined palace that means harness injection blocks, raw tool output, markdown tables, bare timestamp lines, three near-identical chunks of the same transcript, and snippets that open in the middle of a word. The layer that is supposed to be the first thing an agent reads was reading as fragment soup, and no amount of re-sorting fixes it because the junk sorts exactly like the story.

This adds a deterministic selection pass between the existing sort and the render. No LLM call, no embedding, no I/O: a few passes over text already in memory, because L1 runs inside the wake-up hook's latency budget.

What gets excluded, and why it is not a marker list

Exclusion is structural, never lexical. _l1_salience(text) returns negative, and the drawer is dropped, only for:

• a harness wrapper opening the drawer (<system-reminder>, <command-message>, <local-command-caveat>, <task-notification>, [SYSTEM NOTIFICATION), matched with startswith after normalization
• a timestamp-only or rule-only drawer
• text below a 20-character floor
• pipe/dash density above 0.18, which is a table, a diff or an ASCII rule
• a prose ratio below 0.78: the fraction of characters that are letters or spaces

The prose ratio is the load-bearing rule and the reason this is not a longer ban list. It is the general form of what a marker list reaches for: JSON, ls output, env dumps, URL lists, log lines and bare code fences all fail it without any marker being named, while an engineering sentence quoting a path, a version pin or a fence passes.

Calibrated over the 21-item corpus in test_l1_salience_drops_soup_without_markers:

char_ratio   soup [0.404 .. 0.822]    story [0.838 .. 0.986]

The populations separate by 0.016. The floor sits at 0.78, deliberately below the gap: a false drop loses a real memory forever, a false keep costs one wake-up line. The two things that clear the floor and should not have (a word-heavy Python traceback at 0.800, a git diff at 0.822) are demoted, not excluded.

Nothing is excluded for merely mentioning tooling. tool_use, tool_result, fences, Exit code: and cwd was reset cost a ranking point and nothing more, because engineering writing quotes those constantly.

Ranking

Outcome wording is worth 2 points, a clean sentence or heading start 1, and freedom from tool noise 1. Outcome is worth strictly more than both presentation signals, otherwise a drawer that is both an outcome and a little noisy ties with background chatter that merely reads cleanly:

outcome + clean   4        chatter + clean   2
outcome + noisy   3        chatter + noisy   1

Outcome always beats chatter regardless of noise.

Caps and composition

• _l1_select(...): at most L1_MAX_PER_SOURCE (2) lines from any one source file, so a single long transcript cannot own the story. Drawers with no source_file are uncapped, because the cap exists to stop one known file dominating and lumping unattributed drawers under one empty key would do the opposite. Worth knowing: a palace whose metadata lacks source_file entirely gets no cap at all.
• Near-duplicates collapse on whole-body word overlap at 0.8, not on a leading slice. Matching the first 80 characters silently collapsed distinct drawers sharing a templated opening, which mined session summaries routinely have: three summaries reporting a migration, a revert and an RTL fix scored as one drawer and two outcomes were lost. Whole-body overlap scores that case at 0.27 and keeps all three, while genuine restatements of one drawer stay above 0.84.
• Rooms render in ranked order, not alphabetically. Grouping by room and then sorting room names before the MAX_CHARS truncation undid the ranking at the last step: a salience-4 outcome in a room named zzz_outcomes was cut while salience-2 chatter in aaa_trivia rendered.
• _l1_snippet(...) composes the line: start after the next sentence boundary when a chunk opens mid-sentence, cut on a word boundary. The result is always a contiguous substring of the drawer, never a paraphrase, so the verbatim promise holds. Filtered drawers are untouched in the palace and still returned by L2 and L3.
• When every candidate scores as junk, L1 renders exactly what it rendered before this PR. An empty wake-up is the worse failure mode.

Non-English palaces

Two details. The clean-start point tests "not lowercase" rather than "is uppercase", so Arabic, Hebrew and CJK are not permanently denied a point that every English drawer collects. And L1_OUTCOME_KEYWORDS is English, which its docstring says along with the real two-step for extending it: the regex is compiled once at import, so appending to the tuple alone does nothing, and _l1_compile_outcome_re() is exposed for that reason. The junk, cap, duplicate and boundary rules are language-neutral. The prose ratio is character-based rather than token-based for the same reason: Chinese has no spaces, so a token-based ratio scored space-free CJK at 0.00.

Known gap, deliberately left for a follow-up

The outcome keyword list has no failure vocabulary. "died", "crashed", "OOM", "timed out" and "regressed" are absent, so a drawer reporting a failure ranks as chatter unless the sentence happens to also say something passed. Kept out of this PR to keep it in scope; worth its own change.

Orthogonal to #1950

That PR adds per-drawer salience from the dynamics model plus retrieval potentiation, in dynamics.py, searcher.py, service.py and mcp_server.py. This is snippet-level composition inside layers.py and touches none of those files. If #1950 lands, its per-drawer signal can populate the importance key that L1 already sorts on, and this filter keeps doing the part it does. The two do not overlap in files or in intent.

Measured on the 149k-drawer palace this was written for: wake-up reads as a 7-source outcome story in 0.72s, against the oldest-backfill fragment soup it used to print.

How to test

pytest tests/ -q --ignore=tests/benchmarks     4323 passed, 31 skipped
ruff format --check .                          212 files already formatted
ruff check .                                   All checks passed!

Baseline origin/develop@906b918 (develop after the 3.7.1 sync) in the same environment: 4295 passed, 31 skipped, so +28 tests and no existing test changed or deleted. ruff 0.16.1 (the repo pin), Python 3.12.13.

@mvalentsev's eight-drawer repro is a test. Against the earlier revision of this filter, which matched markers as substrings anywhere in the body:

deadlock-root-cause      DROPPED     migration-that-worked    DROPPED
retry-fix                DROPPED     build-oom                DROPPED
agent-loop               DROPPED
prose-1/2/3              kept (score=1)
legit outcome drawers dropped: 5/5
fallback fires (all junk)? no

All five outcome drawers dropped, and because the three prose drawers survive the everything-junk fallback never fires, so the loss was silent. Worse than reported: the three survivors scored 1 and the five dropped would each have scored 2. The filter was not merely losing content, it was inverting its own ranking and keeping the chatter. All five now survive and all five outrank the background prose.

The render-order defect, demonstrated:

salience: chatter=2  outcome=4        MAX_CHARS=120
[aaa_trivia]   - Background chatter about the office coffee machine ...
[zzz_outcomes] ... (more in L3 search)
outcome rendered? False    chatter rendered? True

The near-duplicate defect, across three keying strategies:

case	prefix-80	Jaccard	correct
the PR's own near-dup test	collapse	0.842 collapse	collapse
realistic summaries, templated lead	collapse (wrong)	0.266 keep	keep
toy boilerplate, tiny tails	collapse	0.700 keep	keep

Cost of the whole-body comparison. The comparison list never exceeds max_drawers, so selection stays linear in candidates. Measured on this machine, median of 15 runs:

n=  500   8.7 ms
n= 1000  16.9 ms
n= 2000  34.2 ms      <- MAX_SCAN ceiling

against the repo's stated 100 ms startup-injection budget.

Test coverage: salience edges (anchored wrapper, prose naming a wrapper twice, date-only and rule-only text, table soup, sub-floor text, the outcome boost, a mid-sentence fragment ranking below the same sentence whole, the ISO-designator false-positive guard, non-Latin prose scoring like Latin); a marker-free soup corpus (JSON, ls output, log lines, env dump, URL list, pytest tail, traceback, git diff), which is the adversarial case for the new design since none of it would have been caught by any ban list; noise demotion ordering and fenced code ranking below a clean outcome; snippet boundaries (sentence start, no-boundary passthrough, word-boundary cut, newline collapse); selection (per-source cap, unattributed drawers uncapped, templated-lead summaries kept, near-duplicate suppression, tier order, all-junk empty); render order under the character cap; and end to end through Layer1.generate including the fallback and a verbatim-substring assertion.

Landing order with #2168. The two branches were merged in a scratch worktree. mempalace/layers.py auto-merges with no conflict. The only textual conflicts are additive and both resolve by union: the shared import block at the top of tests/test_layers.py, and the ## [Unreleased] anchor in CHANGELOG.md. The composed tree was verified green at the base where the compose was run (3903 passed, 31 skipped, exactly 3850 + 25 + 28, ruff clean), so nothing is lost or duplicated by combining them; both branches now sit rebased on the same post-3.7.1 develop (906b918) and the conflict shape is unchanged. Whichever lands second needs that two-hunk rebase and nothing more.

One thing this PR does not promise. The per-source cap and the near-duplicate collapse can leave L1 with fewer than MAX_DRAWERS lines on a palace whose candidate window is dominated by one file or by templated summaries. That is deliberate: underfilling the wake-up with distinct content beats filling it with two lines of the same thing.

Checklist

• Tests pass (python -m pytest tests/ -v) — 4323 passed, 31 skipped, against 4295 on develop in the same env
• No hardcoded paths
• Linter passes (ruff check .) — clean on ruff 0.16.1, along with ruff format --check .

Docs: a "What Layer 1 leaves out" section in website/concepts/memory-stack.md, and a CHANGELOG entry under Unreleased. No new dependencies.

[mvalentsev]

_l1_salience matches _L1_SCAFFOLDING_MARKERS as substrings, so a drawer scores -1 for containing a fenced block, or the literal tool_use, or Exit code:, whatever else it says.

Eight synthetic drawers, one wing, same palace, Layer1.generate() on develop and on this branch:

develop      dropped : []
this branch  dropped : ['code1', 'code2', 'code3', 'exit1', 'tool1']

Those five are a deadlock root cause, the migration that finally worked, a retry fix, a build that died on Exit code: 137, and a note about an agent looping on tool_use. The fallback does not catch this: it fires only when every candidate scores as junk, and here the three prose drawers survive and fill L1 on their own.

[ATKabli]

You are right, and the repro is worse than it looks. I reproduced your eight drawers: all five outcome drawers are dropped, and because the three prose drawers survive, the everything-junk fallback never fires, so the loss is silent. The part I had missed is that the five dropped drawers each scored 2 and the three survivors scored 1. The filter was not just losing content, it was inverting the ranking and keeping the chatter.

Fixed by separating two things I had wrongly merged.

Structural harness wrappers (<system-reminder>, <command-message>, <local-command-caveat>, <task-notification>, [SYSTEM NOTIFICATION) still exclude, but only when one opens the drawer. A drawer that quotes one part way through is someone writing about the tooling, which is exactly your point.

I want to flag one thing I tried and threw away, because it was the same bug wearing a different hat. My first pass also dropped any body that repeated a wrapper, on the theory that a repeat means concatenated injections. That rule was unanchored, so it re-created your complaint: an engineer explaining why a reminder fired twice writes the tag twice and loses the drawer. It is gone. Genuine concatenated soup is already caught by table density and the prose floor, neither of which needs a marker named, and there is now a regression test for prose that names the same wrapper twice mid-sentence.

tool_use, tool_result, fences, Exit code: and cwd was reset no longer exclude anything. They cost a ranking point, so a noisy drawer sits below an equally outcome-shaped clean one and far above nothing. Outcome is weighted above the presentation signals, otherwise a drawer that was both an outcome and slightly noisy tied with background chatter.

To stop that from just readmitting log soup, exclusion is now based on shape rather than on naming markers: a prose ratio over the whole body, characters that are letters or spaces. That drops JSON, log lines, env dumps, URL lists, ls output and bare code fences, none of which carry a marker and all of which used to pass the old check. I set the floor below the measured gap between the two populations on purpose, since a false drop loses a memory and a false keep costs one line. The residue that clears it, word-heavy tracebacks and diffs, is demoted rather than excluded.

Your five drawers all survive and all outrank the background prose. Both your repro and a soup corpus that names no markers are now tests.

Three other things came out of chasing this, all folded into the one commit since this never merged:

• Rendering grouped by room and sorted rooms by name before the character cap, so an outcome in a room called zzz_* was cut while chatter in aaa_* rendered. Ranking was being undone at render time.
• Near-duplicate detection keyed on the first 80 characters, so three session summaries sharing a templated opening collapsed into one and lost two outcomes. Now compares whole-body word overlap. Selection stays linear; measured at 34 ms for 2,000 candidates against the 100 ms startup budget.
• L1_OUTCOME_KEYWORDS documented an extension path that did not work, since the regex is compiled once at import. Extracted a compile helper and documented the real two-step.

Still open and worth a follow-up: the keyword list has no failure vocabulary ("died", "crashed", "timed out"), so your build drawer only ranks as an outcome because the sentence ends "and it passed".

Rebased on current develop. Full suite 3878 passed / 31 skipped against 3850 on develop in the same environment; ruff 0.16.1 clean.

[ATKabli]

The rework is pushed. Exclusion is now structural and anchored only: a drawer is dropped for a harness wrapper opening it (startswith after normalization), a timestamp-only body, a sub-20-char body, table/rule density, or a prose ratio below 0.78 — never for containing a marker as a substring. Your markers (```, tool_use, Exit code:) moved to a demotion-only list that can cost a ranking point but can never exclude, so the failure mode you proved is structurally impossible in this shape.

Your eight-drawer scenario is now a checked-in regression test (test_l1_select_keeps_the_reviewers_eight_drawer_repro): all five outcome drawers survive and outrank all three background-prose drawers. The room-ordering bug this uncovered (alphabetical room sort discarding the highest-ranked drawer) is also fixed.

Rebased onto 906b918 (post-3.7.1 develop): 4323 passed, 31 skipped, ruff clean. PR body updated to match.