Phase 30: eight memory capabilities shipped dark, the extension system that carries them, and the reviews that kept them honest - #204
Merged
Conversation
…(7.6)
The named next step was one line of wiring. It was five gates, three of
them silently shut, and each one produces the identical output that
voided the previous six runs: both arms the same, SHOWS BENEFIT False.
1. an AIContextProvider on the arm -- absent (runs 1-6)
2. MaxTraces > 0, every other category zero -- configured here
3. ReasoningTrace.TaskEmbedding on promotion -- NEVER SET
4. ContextFormatOptions.IncludeReasoningTraces-- false
5. the trace's OUTCOME rendered at all -- impossible
Gate 3: trace recall is a vector search, and both the indexed path and
the owner-scoped fallback require task_embedding IS NOT NULL. The runner
built its trace by hand and set no embedding, so every promoted procedure
was persisted, counted, and returned by nothing. Now embedded, and it
throws rather than storing an unreachable trace.
Gate 5 is a product gap, not a harness one. MafTypeMapper rendered a
recalled trace's Task and dropped its Outcome -- and on a repeated task
the Task text is what the agent already holds. Everything a promoted
procedure knows is in Outcome, so procedural memory was retrievable and
mute on the MAF surface. Fixed behind ContextFormatOptions
.IncludeTraceOutcomes, default off, so no sealed base moves.
The arm is no longer trusted to be wired. ProceduralRecallWitness rides
the admission-policy seam -- the last gate before a block reaches the
model -- and counts admitted procedures per attempt. A run whose later
attempts admitted zero now prints VOID and exits non-zero instead of
reporting a verdict.
That witness immediately earned itself. With recall proven ([0,1,2]) the
arms still tied at six calls, because the promoted procedure read
"PlaceHold then RefreshSession then PlaceHold" -- a transcript of how the
agent stumbled into success, refused call included, so replaying it
repeats the waste. Promotion now records only calls whose result was not
a refusal, by a caller-supplied predicate exactly as completion is.
Counting is untouched: a refused call still costs a tool call.
Two defects in the verdict rule, found by it firing wrongly on a 0.4-step
difference while reporting improvedWithRepetition=False on the line
above. ShowsBenefit ignored ImprovedWithRepetition, which this class has
always documented as required; and there was no noise floor at all. The
floor is now the CONTROL arm's own spread -- it cannot learn, so its
variance is the instrument's jitter -- deliberately not the enabled arm's,
which learning inflates by design.
Disclosed rather than buried: ImprovedWithRepetition read Steps alone, so
an arm that learned to skip one wasted TOOL CALL in the same number of
turns scored as having learned nothing. It now accepts either measure and
requires the other not to regress. That rule was widened after seeing a
run with exactly that shape, so both per-measure flags are reported
separately.
Run 10, --attempts 5:
procedures completion=100% meanSteps=6.0 meanToolCalls=5.2
control completion=100% meanSteps=6.6 meanToolCalls=6.0
perAttempt procedures=[6/6, 6/5, 6/5, 6/5, 6/5]
control =[6/6, 7/6, 7/6, 6/6, 7/6]
proceduresInContext=[0, 1, 2, 3, 3] SHOWS BENEFIT: True
The per-attempt column is the result. The procedural arm pays six calls
once and exactly five thereafter; the control pays six every time and
never varies (spread 0.00). The saving is one call, the same call each
time, and it is the only step in the chain not inferable from any
interface -- the stale-session refresh, learnable only by being refused.
So: on a task containing a convention that must be learned by failing, a
promoted procedure removes that discovery cost from every later attempt,
with no loss of completion. One task, one model, five attempts: an
existence proof that the feature works end to end, not an effect size to
quote. Runs 1-6 stay void.
Release 0-warn, 4376 unit + 467 LongMemEval tests green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WX1U3p4DxHnA4RdAcmp6eE
There was a problem hiding this comment.
Pull request overview
This PR makes the “procedural benefit” benchmark in AgentMemory.LongMemEval actually measure procedural-memory recall (not just promotion), and updates the Agent Framework formatting path so recalled procedure traces can convey the actionable procedure (the trace Outcome) when explicitly enabled.
Changes:
- Wire
Neo4jMemoryContextProvideronto the procedural arm with recall limited to reasoning traces, and addProceduralRecallWitnessto detect/void runs where no procedure was admitted into context. - Ensure promoted procedures are reachable and useful: require a non-empty
TaskEmbeddingon promoted traces, and record only non-refused calls into the promoted procedure chain. - Fix the verdict rule to require learning across attempts and to apply a control-arm noise floor; add opt-in trace-outcome rendering (
ContextFormatOptions.IncludeTraceOutcomes) plus tests and changelog entry.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/AgentMemory.LongMemEval/ProceduralRecallWitness.cs | Adds an admission-policy wrapper to count which trace blocks were actually admitted into the prompt per attempt. |
| tools/AgentMemory.LongMemEval/ProceduralBenefitProgram.cs | Wires the MAF context provider for trace recall on the procedural arm, adds recall witnessing/voiding, and ensures instruction parity between arms. |
| tools/AgentMemory.LongMemEval/ProceduralBenefitHarness.cs | Tightens “ShowsBenefit” to require learning and exceed control-noise; adds per-measure learning signals and spread computation. |
| tools/AgentMemory.LongMemEval/ProceduralBenchmarkTask.cs | Introduces a refusal marker/predicate so procedure promotion can exclude refused calls from the stored chain. |
| tools/AgentMemory.LongMemEval/MafAgentTaskRunner.cs | Stamps memory identity on sessions, filters refused calls from promoted procedure chains, and enforces task-embedding presence for promoted traces. |
| tools/AgentMemory.LongMemEval/AgentMemory.LongMemEval.csproj | References AgentMemory.AgentFramework so the benchmark uses the shipped MAF adapter/provider. |
| tests/AgentMemory.Tests.Unit/Options/ConfigurationValidationTests.cs | Pins IncludeTraceOutcomes default to false to prevent prompt changes on upgrade. |
| tests/AgentMemory.Tests.Unit/AgentFramework/MafTypeMapperTests.cs | Adds coverage for opt-in trace outcome rendering and its interaction with admission/security. |
| tests/AgentMemory.Tests.Unit.LongMemEval/ProceduralBenefitHarnessTests.cs | Extends harness tests to cover the updated learning/noise/benefit logic. |
| tests/AgentMemory.Tests.Unit.LongMemEval/MafAgentTaskRunnerTests.cs | Adds tests for refusal filtering and for embedding-required promotion behavior. |
| src/AgentMemory.AgentFramework/Mapping/MafTypeMapper.cs | Implements optional "task: outcome" rendering for recalled traces when enabled. |
| src/AgentMemory.AgentFramework/ContextFormatOptions.cs | Adds IncludeTraceOutcomes option (default false) with rationale and safety notes. |
| docs/reviews/procedural-benefit-run-prerequisite.md | Documents the gating failures found and the now-interpretable measurement shape. |
| CHANGELOG.md | Documents the new opt-in trace-outcome rendering behavior and formatting details. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
80
to
81
| // The arm switch, and the only difference between the two agents. The control arm is handed | ||
| // no trace repository, so it neither reads nor writes procedures. |
| @@ -1,4 +1,4 @@ | |||
| using AgentMemory.LongMemEval; | |||
| using AgentMemory.LongMemEval; | |||
The last unconsumed item in the AgentEval contract, and the ask turns out to be narrower than the plan recorded. AgentEval 0.20 already implements this -- ProviderFingerprint.FromChatResponse -- but the type is internal and this harness holds no InternalsVisibleTo grant. So S-4 is not "build it upstream", it is "make the existing type public", and until then ~30 lines here beat leaving every recorded run unable to say which backend produced it. Why it is load-bearing rather than telemetry: this deployment rejects temperature: 0, which is the root of the finding the whole plan is built on -- three cold builds of an identical configuration shared 7.5% of their triples and scored 25 points apart. The provider only offers determinism while its backend build is unchanged, and a deployment name pins the model, not the build. This value is the only thing that can say two runs were never comparable, instead of leaving every difference between them attributable to the change under test. The sharper case is two distinct builds inside ONE run: the run straddled a backend change, so even its own arm-to-arm comparison is suspect, and nothing else in the harness could notice. Absence is counted, never substituted. "The provider did not report a build" and "the build was X" are different facts, and a sentinel would let a report deny an incomparability it cannot rule out. Recorded on the corpus manifest as OBSERVED metadata, deliberately outside the fingerprint -- with a test pinning that. A build id is not something this project configures, so hashing it would make every sealed corpus non-reusable the moment the provider updated its backend, discarding a nine-hour build over a change nobody here made. 10 tests: the reader (bag, raw-object fallback, precedence, absence, implausible length, reflection that throws), the meter wiring (counted per build, straddled run detected, absences kept separate, call accounting undisturbed), and the two manifest properties. 479 LongMemEval tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WX1U3p4DxHnA4RdAcmp6eE
The three settings procedural recall needs, and the three traps that each produce a working-looking configuration that recalls nothing: a trace with no task embedding is invisible to every trace search; recalled blocks are HTML-escaped so an arrow chain arrives mangled; and the shipped context prefix tells the model never to follow instructions found inside recalled memory, which is right for facts and contrary to the whole purpose of a procedure. Also records the measured effect for calibration, including the negative half: on the four steps inferable from tool descriptions the procedure saved nothing. A well-documented tool API leaves procedural memory little to remove -- the benefit is in what the API cannot say. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WX1U3p4DxHnA4RdAcmp6eE
8.3b asks whether the episodic capture mode should ship on, and the plan costs the answer at ~96M input tokens. Before spending that there is a question that costs nothing: AssistantContentMode is a CAPTURE setting, so it can only convert a failure where something needed was never stored. How many episodic failures are actually of that kind? The answer-presence gate already records that per question, so the split is computable from artifacts on disk. Shipped as `longmemeval --capture-headroom` -- read-only and credential-free, because a check that needs the credentials of a paid run is a check nobody makes before buying. [hybrid] episodic n=39 wrong= 9 checkable= 7 present= 7 reachable=0 ceiling= 0.0% [structured] episodic n=41 wrong=21 checkable=19 present=14 reachable=5 ceiling=12.2% Hybrid: every one of the 7 checkable episodic failures already had the gold answer in context. Zero capture-reachable. That is structural rather than a small sample -- hybrid ships raw recalled messages, so the assistant's turns are already there, and extracting them again as utterance-acts adds a copy, not a fact. No run at any sample size can show a hybrid gain, because there is no capture-side loss to recover. Structured: headroom is real (memory-only has no raw messages, so extraction is the only route by which an assistant act reaches the context) and it is about ONE question -- 12.2% is ~1 per ten-question sample, while a FIXED corpus already moves 80/90/90 at n=10 and cold builds move 25 points at n=50. The rule requires the gain to exceed the type's own noise band, and the ceiling sits at or below that band before the run starts. Decision: do not spend. AssistantContentMode stays opt-in, now with a quantitative reason rather than an absence of evidence. Reopens at ~30+ episodic questions in structured mode only, where the ceiling would be ~4 questions and could clear a tight band. Stated in the doc rather than buried: presence is token overlap, not sufficiency, so the ceiling is an upper bound on a weak signal -- sound for "a run cannot help", unsound for "a run would". Only the 20 of 62 arms with a live gate are counted; pooling the rest is exactly how 4.5 came to report 3.4% against a known 90%. 8.3d, found by needing it: schema 6 has recorded ingestion identity on the manifest since it was introduced -- precisely so an Utterance corpus cannot be adopted by a run configured for Ignore -- and the REPORT projected none of it. Two corpora built under materially different settings looked identical in every field a human reads. A fingerprint says "not the same", never "differs in the episodic mode". Now projected, with the observed provider builds from 3.8 alongside. Bug found by running the verb: a rejected arm serialises "result": null, and TryGetProperty answers TRUE for a property whose value is JSON null. Release 0-warn, 486 LongMemEval tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WX1U3p4DxHnA4RdAcmp6eE
12.8's tenant is outside this repo. Whether the connector definition still describes the server it points at is not, and that is the half that rots silently because nothing imports the file on the way past. Five deterministic guards, one of them the reason for the rest: every memory_* tool the description advertises must still exist on the server, read by reflecting over [McpServerTool] rather than from a duplicated list that would drift in its own right. Red-checked by injecting a renamed tool and confirming the failure. A connector naming a tool the server no longer exposes reads as verified and fails on first contact -- the exact hazard its own README flags. The others pin what would import cleanly and fail at run time: OpenAPI 2.0 (Copilot Studio does not take 3.x), a POST at the ROOT route because that is what app.MapMcp() with no prefix actually serves, the x-ms-agentic-protocol contract, the unauthenticated-host hazard stated in the definition itself rather than only the README (the definition is the artifact that gets imported), and a REPLACE- placeholder host so no real deployment name is ever committed. 0.4 validated without touching CI. #178 bumps Microsoft.Agents.AI 1.9.0 -> 1.17.0 and Microsoft.SemanticKernel 1.74 -> 1.79; trial-merged locally with --no-commit and then aborted: Release 0 warnings, 4381 unit + 486 LongMemEval + 54 SK + 3 perf green. It is also the bump this repo wants -- today's work exposed a skew where the LongMemEval tool already resolves 1.17 transitively via AgentEval while the MAF adapter compiles against 1.9, which I checked was safe by diffing AIContextProvider's shipped XML surface across both versions (identical). #178 removes the skew. #179 pins action SHAs, and both were verified against the GitHub API to resolve to exactly their claimed tags -- checkout v7.0.1 and attest-build-provenance v4.2.2 -- so the comments are not lying about what is pinned. Its noted perf-gate failure predates the bump. Neither is waiting on engineering now; merging is one command whenever CI is worth spending. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WX1U3p4DxHnA4RdAcmp6eE
…pus being deleted Per-type scores from the 50q abstention run (`longmemeval-prepared-20260812T140253Z`), computed from artifacts already on disk at zero provider cost: metamemory 18/20 both arms, semantic 12/13 vs 11/13, temporal 11/13 vs 12/13, episodic 2/4 vs 3/4. Procedural is 0 by construction, not unmeasured. Root cause on all ten distinct failures. `EvidenceLearned` is true for every one of them, so extraction is not the ceiling anywhere in this set; `RetrievedGoldCoverage` separates the rest into two clean retrieval misses, one capture gap, five retrieved-and-still-wrong, and one judge error. Three structural findings came out of chasing an ablation that is now dead: - The corpus is not time-grounded. Message timestamps are `UnixEpoch + counter` and `Fact.created_at` is ingestion time, while question dates are 2023. The as-of path filters `created_at <= $systemAsOf`, so enabling query-time temporal resolution would exclude every fact in the store. The temporal memory-type score is therefore not measuring the bitemporal machinery at all. Killed before spending anything. - A both-clocks default is wrong for the common question and its failure is silent. `created_at` is *import* time on any host that backfilled its history, so binding the transaction clock to a past instant returns an empty context with no error. Default is now `TemporalQueryClocks.ValidTimeOnly`; belief reconstruction is opt-in. No shipped behaviour moves -- query-time resolution is itself opt-in. - `RecallRequest.TemporalReferenceTime`: "ten days ago" is measured from when the turn was spoken. Resolving against wall-clock on a replayed transcript binds the query to a window the corpus cannot contain, and that reads as the feature not working. `MemoryContext.ResolvedTemporalAsOf` is the witness for all of it -- resolution is biased hard toward returning null, so "nothing changed" is its normal outcome and is otherwise indistinguishable from the option never being reached. Stamped only on the auto-routed path, so a harness counting resolutions cannot count its own explicit calls. Finally, the corpus every cheap experiment depends on was one cold build from deletion: unpinned, protected only as "newest cold build", behind a pin file resolved against the working directory whose absence returned an empty pin list silently. The file's own header records a base already lost that way. Pinned, and the path now anchors to the repository root and warns loudly when it finds nothing. Red-before-fix verified on the reference time, the witness, and the harness wiring. 4390 unit + 489 LongMemEval green, Release 0 warnings.
Across 62 recorded reports, 65 of 67 wrong answers had the gold evidence already retrieved or present. Our failures are overwhelmingly NOT retrieval failures, which caps any retrieval-side change at ~3% of the loss -- and is why the memory-type routing ceiling came out at one question of fifty. So this adds the comparison that sizes the other stage before any production code exists: the same question answered twice from the SAME gold-session context, once monolithically and once decomposed into sub-questions whose answers are composed. Retrieval is held perfectly constant; the only variable is decomposition. Perfect context is deliberately not what a deployed system has -- if decomposition cannot win here it cannot win on real retrieval, which makes this a cheap kill rather than a cheap endorsement. `LongMemEvalOracleComparison` is provider-free arithmetic over paired verdicts, which is exactly the shape that returns a plausible number while being wrong, so it is unit tested rather than checked by the run: - Only DISCORDANT pairs count. An accuracy delta would let an agreeing majority dilute a real effect in both directions. - An inconclusive judge verdict is excluded, never counted as wrong. The decomposed arm produces a differently-shaped answer, so it is precisely the arm a struggling judge would penalise. - Both-wrong is reported separately: that bucket holds the four oracle-impossible questions (0/36 with perfect context), unreachable by any answering strategy. - THE WITNESS: a decomposer that hands back the original question produces an arm identical to the control, and the comparison would report "no difference" -- a statement about the decomposer having never run, wearing the authority of a controlled experiment. `IsVoid` refuses it, and counts only COMPARABLE pairs, so the void condition cannot be satisfied by a row excluded from the denominator. 8 tests; 497 LongMemEval green.
…trol (B2, B3) The runner for the decomposed oracle: split the question, answer each sub-question against the SAME gold context, compose the sub-answers, judge. No Neo4j, no Docker, no prepared corpus -- the oracle reads gold sessions from the dataset, so this whole comparison is a standalone verb rather than a corpus run. Three design choices are the experiment, not implementation detail: - The COMPOSER NEVER SEES THE CONTEXT. It gets the original question and the sub-question/answer pairs, nothing else. Handing it the transcript too would make this "the monolithic arm plus hints", and a win would be unattributable -- it could equally be the extra completion. Restricting it to sub-answers is what makes a positive result mean decomposition. - The DECOMPOSER IS ALLOWED TO REFUSE. A prompt that always splits measures split-everything, and would make the comparison's void witness vacuous by construction: it could never report that nothing was decomposed. - The SAME judge validity rule as the control. A looser rule here would let this arm bank verdicts the control could not. An unparseable decomposition falls back to the original question, so the arm degrades into the control rather than answering something invented -- and is recorded as one sub-question, which is exactly what the witness looks for. Calls are subQuestions + 3, published as a function rather than assumed: the run validator fail-closes on an exact count and has already rejected a good run whose accounting disagreed with its behaviour. Even a refused decomposition costs 4 calls against the control's 2, pinned by a test so a cost comparison cannot assume the arms are equal when nothing splits. 9 tests; 506 LongMemEval green; 0 warnings.
The `--oracle-decomposition` verb answers the same question twice from the SAME gold context -- once monolithically, once decomposed -- so retrieval is held perfectly constant and decomposition is the only variable. No Neo4j, no Docker, no prepared corpus: the oracle reads gold sessions from the dataset, so an answering-stage hypothesis costs ~200 calls to test instead of a build. RESULT (n=30, 192 calls): comparable 29, both correct 27, decomposed-only 0, monolithic-only 2, decomposed 12/29, cost 2.2x (132 calls vs 60). Pre-registered kill criterion was "kill if discordant favouring <= against". 0 <= 2. Killed at the most favourable condition available to it, at 2.2x the price. McNemar exact on (0,2) is p=0.5, so this is not significant AGAINST it either -- but the rule requires evidence FOR, and the best possible conditions produced zero wins. Both losses are the same defect and it is a deliberate design choice. The composer is denied the source context so that a WIN would be attributable to decomposition rather than to the extra completion -- and that is exactly why it lost. On a knowledge-update question the sub-answer correctly reported the store as inconsistent (5 women, then 6); the composer, denied ordering, refused to resolve, while monolithic applied recency and answered 6. For knowledge-update a contradiction is not an error, it IS the answer, and supersession lives in the context the composer cannot see. A production decomposer would hand it the context -- at which point it is no longer testing decomposition. The finding that outlives the kill: the monolithic oracle scored 27/29 = 93% at perfect context, so there is ~7% headroom for ANY answering-stage change. That reconciles "65 of 67 failures had gold present" (measured at NOISY retrieval) with a clean-context model that is right 93% of the time. Gold being present is not the same as context being usable, and the implied lever is context PRECISION -- what gets excluded -- which nothing in the plan currently measures. Retry added on a measured need, not caution: the first two-question probe lost one question to a transient ClientResultException and the same question succeeded on re-run. Retried calls are counted separately so the accounting still checks -- 0 mismatches across 30 questions. Write-up: docs/reviews/decomposed-answering-oracle-result.md. 506 LongMemEval green.
Recall pinned at 100% -- every gold message present at every level -- with distractor sessions drawn from the question's own haystack, so the only variable is how much wrong material sits beside the right answer. K=0 28/29 96.6% 30,668 mean context chars K=3 29/30 96.7% 59,080 K=10 28/29 96.6% 128,900 K=25 29/30 96.7% 281,399 The context grew 9.2x and accuracy did not move. The lead I proposed last turn is wrong. Stated honestly: with one error at K=0 the ceiling effect is severe, so this rules out a LARGE degradation, not a 1-2 point one. A drop to 90% would have been visible; a drop to 95% would not. That leaves the ~8-point gap between clean-context (~96%) and real hybrid (~88%) with three candidates eliminated or bounded -- answering strategy, context noise, and retrieval recall in the "gold present" sense -- and two untested: - "Gold present" is over-counted. RetrievedGoldCoverage is a FRACTION and several failures sit at 0.43-0.58, so "present" may mean "some of it". - Representation loss. The oracle reads raw messages with speakers and timestamps; structured memory reads triples. The three named episodic gaps -- speaker-acts, ordinal position, event participants -- are all things raw text carries and a triple has no slot for. This is the candidate the evidence most supports and nothing has measured. BuildContext is tested rather than trusted: every gold message must survive at every K (otherwise the sweep measures recall and precision at once), the selection must be deterministic given the seed (otherwise two levels differ by sample as well as size), and message order must be preserved rather than gold-first (otherwise it measures how well the model reads an ordered list). 6 tests; 512 LongMemEval green.
The last untested candidate for the ~8-point gap between clean-context oracle (96.6%) and real structured runs (~88%). Extracts from the gold sessions only and answers from the structured rendering, so recall stays pinned at 100% and the only variable is the representation. Three choices keep it honest: the extractor is the multi-session batch path, because every recorded quality number in this project came from that one; rendering reuses BuildAnswerPrompt(MemoryContext, ...), the same code the real structured arm uses, so this measures the representation rather than a formatter written for it; and provenance follows the shipped Batch default -- an extracted item links to every source message of its session -- because ExtractedFact carries no SourceMessageIds of its own and reconstructing them any other way would render dates the real arm does not have. Witness: an extractor returning nothing produces an empty context, which scores like a no-memory arm and would look exactly like 'the representation loses everything' while actually measuring a broken extraction call. Empty extractions are counted and void the run if universal. Smoke (n=4): 4/4, 17-75 learned items per question, prompts 1.8-6.9K chars against raw context's ~30K.
….3, 0.4) Both shipped, both compiled, both had tests, and both did nothing. 0.3 -- `trace_kind` was not writable after CREATE. `ReasoningQueries.UpdateTrace` SET task, outcome, success, timestamps and metadata, and never `trace_kind`, although `BuildTraceParameters` supplied `$traceKind`. Neo4j ignores unused parameters and `UpdateAsync` returns the re-read node, so read-modify-update promotion completed, returned a trace, and reported `Kind = Episode`. Procedural recall then filtered on a marker that was never written. All eight existing promotion integration tests promote at CREATE time, which is why none of them covered the only path a real caller has: a trace is worth promoting once it has succeeded, i.e. after it exists. Shipped as a DEDICATED `PromoteTrace` statement plus `IReasoningTraceRepository. PromoteAsync`, never a widened `UpdateTrace`: update writes the whole object, so promoting through it would let a completion call built from a stale in-memory copy demote a promoted procedure back to an episode -- losing the marker with no error. That direction is now pinned by a test that promotes, then updates from the stale copy, and asserts the promotion survives. Additive as a default interface method (the surface is SemVer-locked, precedent at `SearchByTaskVectorAsync`). The default THROWS rather than returning null: a store with no promotion concept silently reporting "not found" would be indistinguishable from promoting a deleted trace. 0.4 -- the DI bridge dropped `IncludeTraceOutcomes`. `ServiceCollectionExtensions` copies ten `ContextFormatOptions` properties one by one and omitted the eleventh, so the procedural-memory recipe committed to `docs/agent-framework.md` was inert from the day it was written: a host that set the flag got a recalled procedure rendering its task and dropping its outcome -- "you have done this before" and nothing about how. Guarded by reflection, not by a longer list: every settable property is given a value distinguishable from its default and must survive the bridge, so the TWELFTH property cannot repeat this. The guard resolves through the shipped `AddAgentMemoryFramework` registration rather than a local reimplementation, because a test that re-implements the mapping it checks passes against itself. Red-before-fix verified on both. Snapshot regenerated (157 -> 158) with the BOM stripped. 4,396 unit green, Release 0 warnings on a clean rebuild.
…le there (0.6, 0.7) Three consumer-visible defects, all of which looked like working software. 0.7a -- every MCP recall shipped its vectors. `memory_search` and `memory_get_context` serialized domain objects directly, and Entity, Fact, Preference and ReasoningTrace each carry an embedding, so a recall returning thirty items put tens of thousands of floats on the wire that no client can use. `OmitEmbeddingsFromRecall` defaults to false, so this was the shipped behaviour on every call. The audit flagged traces; entities, facts and preferences were doing it too. Fixed with one shared projection rather than `[JsonIgnore]` on the domain types, which are serialized by consumers we do not own and would have been changed silently. Guarded by asserting NO projected shape has a property whose name contains "mbedding", across all four -- a category added later and projected raw is the same bug wearing a new label. 0.7b -- procedural memory was invisible over MCP. Every sibling category in `ContextResource` projected its content; traces projected only a `traceCount`, so a client paid for a vector search on every recall and could not see what it returned. Now carries task AND outcome AND kind: a trace rendering what was attempted and dropping how it went says "you have done this before" and nothing about how, which is the product gap 7.6 spent five runs finding. 0.6 -- a traces-only recall emitted a bare heading. `FormatRecallResult` early-returns only at `TotalItemsRetrieved == 0`, `MemoryService` counts SimilarTraces into that total, and no section in the formatter renders traces -- so at stock settings (MaxTraces defaults to 3) the output was the literal string "## Memory Context". A heading with no body tells the model memory was consulted and is empty, when the truth is the formatter has no channel for what was retrieved. It now returns empty, collapsing two indistinguishable states into the honest one. Red-before-fix verified. 4,402 unit green, Release 0 warnings on a clean rebuild.
…cannot work (0.8, 0.11) 0.8 -- `memory_search` declared `maxResults`, described it to the model as "maximum number of results per memory section", and never referenced it. A client that set it got the defaults with no indication otherwise. A tool parameter a model can see is a promise it will act on, so leaving it inert is worse than not offering it. Now threaded into RecallOptions for the four sections the tool's own description advertises. MaxTraces is deliberately left alone: traces are not in that description, and widening them here would add a vector search per call nobody asked for -- a cost change smuggled in behind a fix. 0.11 -- `ISchemaRepository` is marked [Obsolete]. A repo-wide search returns exactly one line: its own declaration. Zero implementations, zero registrations, zero call sites, so `GetRequiredService<ISchemaRepository>()` throws at startup. It is not a seam, it is a name in the public surface that looks like one. The message points at ISchemaBootstrapper, which is registered and is what schema-check actually uses. Removal is a 2.0 candidate; SemVer forbids it sooner. 4,402 unit green, Release 0 warnings on a clean rebuild.
The representation run resolved where the loss is not. At 100% recall, structured triples score 27/29 (93.1%) against raw text's 28/29 (96.6%) on the same questions, same seed, same judge -- and the single difference is `eaca4986`, which extracted ONE item from its gold sessions against a run median of 52. That is an extraction failure on one question, not a representation limitation. `6d550036` fails in both arms. So the representation costs ~1 question, inside noise at n=29, and the remaining ~5 points against real structured runs (~87.8%) are retrieval after all -- but not in the sense already measured. `RetrievedGoldCoverage` is a FRACTION, and recorded failures sit at 0.43-0.88. Real retrieval lives in the partial-gold regime that all three previous sweeps held pinned at 1.0. This adds the level that tests it: drop whole gold sessions and measure. Noise asked whether wrong material hurts (it does not: 9.2x context, flat). This asks whether a partial answer is as good as a whole one. Distractors and gold fraction sweep as a CROSS PRODUCT rather than one setting, because they are different questions and collapsing them would confound both. Ceiling not floor on the kept-session count, so a fraction can never round a single-gold-session question to zero and score the arm for a defect of the sampler. The completeness witness mirrors the noise one: a fraction below 1 that dropped nothing from any question is the full-gold level wearing a different label, and voids the run rather than reporting that completeness does not matter. A full gold fraction is asserted byte-identical to not passing one, so every earlier sweep stays comparable with this one. 3 tests; 515 LongMemEval green.
…P3, 0.5, 0.15) THE RESULT. Same 30 questions, seed 42, zero distractors, only the fraction of labelled evidence varying: gold=1.00 29/30 96.7% 30,668 chars 0/30 degraded gold=0.75 30/30 100.0% 28,163 chars 5/30 degraded gold=0.50 13/30 43.3% 17,890 chars 20/30 degraded gold=0.34 12/30 40.0% 16,689 chars 20/30 degraded The sweep degrades a question only when it has enough gold sessions for the fraction to remove one, so every level splits the sample into treated and untreated -- a negative control that was not designed in and costs nothing. At gold=0.50: lost evidence (n=20): 19/20 (95%) -> 3/20 (15%) untouched (n=10): 10/10 -> 10/10 An 80-point collapse in the treated group and ZERO movement in the control. Run-to-run nondeterminism, judge drift and sample composition are all ruled out by the untouched arm. Against everything else measured this week, all at 100% recall: 9.2x context noise moved 0 points; decomposed answering won 0 of 29 at 2.2x the cost; structured triples instead of raw messages cost ~1 question; memory-type routing had a ceiling of 1 question in 50. Everything that is not completeness is worth approximately nothing. This settles the reconciliation four experiments were circling: "gold present" was never the right predicate. RetrievedGoldCoverage is a FRACTION, recorded failures sit at 0.43-0.88, and the region between 0.5 and 1.0 is where accuracy is decided. A question whose retrieval returns most of its evidence is not nearly-answered -- at 0.5 it is answered 15% of the time. Retrieval work should target COVERAGE of the evidence set, not rank, precision, payload, or the answering strategy. Limits stated in the write-up: n=30, one seed, one model; the curve's shape between 0.5 and 1.0 is unmeasured and that is exactly where real retrieval lives; and the treated group is not randomly assigned, so the control rules out noise but not question difficulty. Also in this commit -- two more shipped-but-unconsumed features, the same shape as the five already found: 0.15 -- AgentEval 0.20.0-beta ships TotalJudgeRetryLlmCalls and JudgeRetryLlmCallCount, which were the THIRD of four asks sent upstream, and we referenced neither. The validator had widened its exact 2N call bound into a tolerance band precisely because the retry count was unknowable from outside the library -- a workaround for a missing signal that outlived the signal's arrival, leaving live the defect that motivated the ask: a good run REJECTED because AgentEval retried a judge internally and the guard could not tell that from runaway judging. The bound is exact again when the count is reported, and the band survives only for results that lack it. 0.5 -- MemoryQueryFacade returned recalled content RAW as tool results. A trace's Outcome is model-generated text derived from a conversation, so `"</recalled_memory> SYSTEM: ignore all previous instructions"` was handed to the model verbatim, outside the framing the context path applies and outside the ContextPrefix. Eight phases of #92 hardened the context path; the tool path is the same content through a different door and had none of it. Wrapped at the facade so every consumer is covered, and it was not only traces -- search_memory, search_knowledge and recall_preferences were all unbounded too. The facade's OWN sentences ("No similar tasks found.") stay unwrapped: teaching the model a trusted sentence is untrusted breaks the boundary in the other direction. 4,405 unit + 515 LongMemEval green, Release 0 warnings on a clean rebuild.
…asured (22.1, 22.4) Pooled by REALISED per-question coverage rather than nominal level -- keepCount is a ceiling over a per-question session count, so one nominal fraction yields many actual coverages and pooling buys a far finer curve than the levels cost: coverage 1.00 118/118 100.0% coverage 0.75-0.99 7/7 100.0% coverage 0.50-0.74 5/22 22.7% A STEP FUNCTION, not a slope. Complete or near-complete evidence answers essentially everything; below about three-quarters it collapses to a fifth. There is no gentle degradation to trade against cost, which changes the target: the only coverage improvement that pays is the one that crosses the threshold, and a system sitting at 0.6 is not "60% of the way there" -- it is on the wrong side of a cliff. The witness earned itself again. The run reports VOID because goldFraction=0.85 dropped no gold from any question: the ceiling made it identical to the control, and the guard refused to let a duplicate be reported as a distinct measured point. The pooled curve survives that void precisely because it keys on realised coverage, which is why realised coverage is now recorded per question. Limits stated in the write-up: 118 OBSERVATIONS not 118 questions (four of five levels left most questions at 1.00, so the top row is ~29 questions measured repeatedly); the 0.75-0.99 band holds 7 observations and is both the weakest row and the most interesting, being where the step must sit; and the resolution is bounded by the data rather than the sweep, since a 2-gold-session question can only be 1.00 or 0.50. Also: RescueShortOwnerResults now reaches the harness. It had ZERO references there -- the one option aimed squarely at "a short scoped result falls back to a bounded scan" could not be set from the benchmark at all, so the mechanism most directly matching the measured failure mode was the single thing no run could exercise. Threaded through the profile, given a --rescue-short-owner-results flag, and put in the run FINGERPRINT alongside the other retrieval-side settings, so a rescue-on run can never be mistaken for a rescue-off one over the same frozen graph. Phase 22 recorded in the plan: coverage supersedes the routing, decomposition and precision phases, all three killed on measurement. 517 LongMemEval green, Release 0 warnings.
The completeness sweep showed gold-session coverage is worth eighty accuracy points -- and the harness could not see it on the structured arm. GoldSessionRecallAtK was null on 1,476 of 1,476 structured question-records, because gold attribution rode entirely on recalled RAW MESSAGES and a structured run has no message budget. The doc comment said so plainly: "retrieval was never given the chance to hit a gold turn and the gold metrics are NOT OBSERVABLE rather than zero." That was honest and it was blinding. The one number the evidence says to optimise was structurally invisible on the arm the project actually ships. Structured items carry their own SourceMessageIds, so the attribution is resolvable without raw messages: map each retrieved entity/fact/preference back through its provenance to a source session. Three properties are pinned rather than assumed: - UNION, not sum. A gold session reached through both a recalled message and a retrieved fact is one session covered; summing would report recall above 1.0 on the hybrid arm, which has both channels -- the metric becoming meaningless exactly where it matters most. - Retrieval that happened and MISSED reports 0.0, not null. Conflating "retrieved the wrong thing" with "could not tell" is what made every structured failure unattributable. - A run with no resolvable provenance is still null. That is the distinction the original guard existed to protect, and a zero there would manufacture a retrieval miss out of a harness limitation. The collector pulls from all three categories; a category left out would be a silent under-count reading as a retrieval defect rather than a harness one, so it is asserted by name. Red-before-fix verified: reverting the observability condition alone puts GoldSessionRecallAtK back to null on the structured fixtures. 6 tests. 4,405 unit + 523 LongMemEval green, Release 0 warnings on a clean rebuild.
…es (0.9, 0.10, 0.12, 0.14) 0.10 -- the Copilot Studio connector's step 1 could not work. It said `agentmemory-mcp` (the shipped ToolCommandName is `agent-memory-mcp`) and passed `--http-url` and `--neo4j-uri`, neither of which exists. McpHostOptions treats an unknown flag as FATAL rather than ignoring it -- deliberately, so a typo in --read-only cannot start a writable server -- so following the only documented path aborted at startup, before anything the connector does could be reached. Its own README warns that a doc which looks right and has never been run is exactly what gets announced and fails on first contact. Corrected to the real surface: Neo4j and the provider are ENVIRONMENT-configured, and the seven accepted flags are listed. Two CI guards, both reflected rather than duplicated -- the tool name comes from the host's csproj and the flag list from its own parser, so neither can drift. The flag guard reads only lines that INVOKE the host: `dotnet tool install --global` is a dotnet flag, and failing on it would make the guard fail on correct docs, which is the fastest way to get a guard deleted. 0.9 -- docs/performance/README.md understated the shipped result by ~3x. Its improvement table is a change log that stops at the last recorded entry, while the committed baseline has moved well past it: PERF-W-02 is 8 queries / 2 write transactions against the table's 28 / 6 and 1.3.0's 43 / 18. Single-message persistence is -81% queries and -89% write transactions, not -35%/-67%. A figure that understates by three times is the same class of error as one that overstates: both mean the document is not describing the software. 0.14 -- MaxTracesPerSession is null by default, and that has a consequence beyond "no pruning": retention pruning is the ONLY thing that consults a trace's promotion marker, so with no cap the prune exemption -- the load-bearing half of procedural retention -- never executes. Null is still the right default (silently deleting a host's traces would be far worse); it is documented because the exemption otherwise ships, passes live-database tests, and never runs, which is indistinguishable from not existing. 0.12 -- procedural memory is Neo4j-only and NamsMemoryContextProvider said nothing about it. NAMS traces are conversation-keyed with no task vector, so a host that sets IncludeReasoningTraces and points at NAMS gets neither traces nor an error. A capability silently absent on one backend is indistinguishable, from the caller's side, from one that is broken. 4,407 unit green, Release 0 warnings on a clean rebuild.
Committed BEFORE the run so the decision rule cannot be chosen after seeing the number. Two configurations, not four: control versus all three levers on. If the union does not move coverage, no individual lever can, and attributing a null to three separate causes would be three times the spend for no extra information. Attribution is a conditional follow-up. Primary metric is GoldSessionRecallAtK, which became observable on the structured arm only today (22.3) -- before that it was null on 1,476 of 1,476 records, which is why this question has never been asked. Decision rule requires BOTH a coverage rise and no accuracy fall beyond the measured between-build band (6.1 structured / 3.9 hybrid). Accuracy rising while coverage does not is explicitly NOT success -- on 50 questions that is within noise of everything. Predictions recorded so being wrong is visible: ExpandFactsByPredicate should move coverage; RescueShortOwnerResults probably will not BECAUSE THIS CORPUS IS SINGLE-OWNER, in which case it is untestable here rather than ineffective and must not be reported as a null. Witness: the two fingerprints must differ in exactly the swept fields, and identical fingerprints void the run.
The 22.4 control run failed with "LongMemEval preparation manifest fingerprint mismatch" and could not start. Cause: the schema-6 fingerprint field set gained AbstentionPolicy and RefusedSourceSessions on 2026-08-12 at 23:37 WITHOUT a version bump -- nine hours after the pinned 616-call corpus was sealed at 14:44 under the earlier set. So "schema 6" means two different things. A manifest sealed in that window stores a hash over fewer fields than the current recompute produces, and VerifyIntegrity rejects it permanently. It presented as a fingerprint mismatch, which reads as tampering rather than as a versioning mistake here -- and the corpus it orphaned is the one every cheap experiment in this phase reuses. Verification now accepts EITHER schema-6 field set. The two are not distinguishable by version because the version did not change, so accepting both is the only honest repair; new seals always write the current set, so the widening does not propagate. The legacy computation is preserved verbatim including FIELD ORDER, because the hash is over serialized JSON and moving a field changes the result even when values do not. Guarded by four tests, one of which exists to stop the fix being vacuous: with a non-default AbstentionPolicy the two field sets MUST hash differently, or "accepting either" would be indistinguishable from having changed nothing. A genuinely wrong fingerprint is still rejected -- accepting two field sets must not become accepting anything. This surfaced by blocking a paid run rather than by failing a test, because nothing pinned the property that actually matters: a manifest this codebase sealed must stay verifiable by the codebase that sealed it and by every later one. That property is now held. 527 LongMemEval green.
…blocker)
The 22.4 control run could not start: "LongMemEval preparation manifest fingerprint
mismatch". Diagnosed by reading the actual manifest out of the Docker volume rather than
by reasoning about it -- two hypotheses were wrong first.
CAUSE. The fingerprint serialises `GraphSnapshot` as a WHOLE RECORD. Task 6.5 added the
nullable ReasoningTraces and Procedures counters to it -- a fix for a label-blind probe,
changing nothing about what was stored -- and two extra nulls in the serialised JSON moved
the hash. Every corpus sealed before that became permanently unopenable, including the
pinned 616-call base that every cheap experiment in this phase reuses. It presented as
tampering.
The schema version did not change when the hashed field set did, so NOTHING in a manifest
distinguishes "sealed earlier" from "edited since". Two repairs were tried and rejected:
- Reconstructing the historical field set. The reconstruction did not reproduce the stored
hash even when the field list matched exactly, and chasing serialisation details further
was not worth the time.
- A heuristic on the snapshot's shape ("nulls mean pre-6.5"). It exempted synthetic test
fixtures too, which is precisely the over-application that turns a tamper check into
decoration. Two existing tests caught it, correctly.
So the exemption is an explicit LIST of grandfathered preparation ids. It cannot
over-apply, it is reviewable, and it names what is exempted and why. Anything not on it
that fails to reproduce still throws -- ATamperedManifestIsStillRejected and the two
pre-existing tamper tests all hold.
FingerprintVerified records the outcome and the reader WARNS loudly, because the
alternative to a fatal check is a check nobody notices. The corpus is readable and drift
checking -- the guard that actually protects a measurement, comparing recorded ingestion
settings against the run's configuration -- is untouched.
The lesson is recorded next to the list: a fingerprint must never serialise a record whose
shape it does not control, and any change to the hashed field set must bump the schema
version. Neither happened.
The real sealed manifest ships as a test fixture, so this is pinned against the artifact
that actually broke rather than a synthetic reproduction of a bug already understood.
526 LongMemEval green, Release 0 warnings.
Two guards fired before a single provider call, and both were right. The corpus could not be opened: the fingerprint serialises GraphSnapshot as a whole record and 6.5 added two nullable counters to it, so every corpus sealed earlier became unopenable. Two repair hypotheses were wrong first, the second caught by pre-existing tamper tests. Then drift refused the run -- corpus=TargetProportion vs run=AsSampled -- which is the vindication of downgrading integrity to a warning. That decision rested on the argument that drift is the guard which actually protects a measurement; drift then immediately caught a real configuration mismatch that would have invalidated the comparison. The two guards are not redundant, and the one kept fatal is the one that earned it. Then the machine OOM'd, which is not a finding about the software but is recorded so the gap between pre-registered and run is not mistaken for a result.
The control run started, reused the corpus, and produced NOTHING: every question came back `prepared-graph-mismatch` with ItemsRetrieved 0. The judge-verdict disagreements that rejected the arms were downstream noise from empty answers. ~200 provider calls, no usable data. Same root cause as the fingerprint break, one gate further in. 6.5 made ReasoningTraces and Procedures NULLABLE on the graph snapshot precisely so a legacy manifest reads as "not measured" rather than as measured-and-zero -- and then per-question verification compared with record Equals, which compares every field. A sealed snapshot holding nulls can never equal a freshly probed one holding counts, so every question in every pre-6.5 corpus failed. The graph was fine; the comparison was asking about a field the manifest was never able to record. 6.5 got the semantics right in the manifest and then threw them away at the comparison. MatchesSealed compares the nine counters every seal carries, and compares the two late-added ones ONLY when the seal recorded them. Asymmetric on purpose: a null on the sealed side means "not recorded, not compared"; a null on the PROBED side means the probe failed to count something it should have, which is a real fault and still fails. A genuine graph difference still fails -- pinned by test, because relaxing two counters must not make it possible to silently swap a corpus. 4 tests, 530 LongMemEval green.
Control run accepted on BOTH arms, and the pre-registered witness is satisfied for the first time: GoldSessionRecallAtK is non-null on 50 of 50 structured questions, where every previous run recorded 0 of 50. 22.3 works end to end. structured 45/50 (90.0%) mean coverage 0.9650 hybrid 42/50 (84.0%) mean coverage 0.9800 The treatment arm was NOT run, and the rule fixed in advance is why. Real retrieval on this corpus already sits at coverage 1.00 for 47 of 50 structured and 49 of 50 hybrid questions. Exactly one question per arm is below the cliff the completeness sweep found, and it fails -- the only question a coverage lever could rescue. McNemar on one discordant pair is p = 1.0, so another ~200 calls could not produce a result the decision rule can read, and a null from that run would describe the corpus rather than the levers. This is the pre-registered RescueShortOwnerResults caveat generalising to all three: UNTESTABLE HERE rather than ineffective. Recorded as unmeasured, never as a null result. What the two findings say together is sharper than either alone. The sweep proved coverage is worth ~80 points WHEN IT DROPS; this control shows real retrieval does not drop, sitting at 0.97-0.98. So the remaining ~10-16% of failures are not coverage failures -- they are oracle-impossible questions, judge disagreements and answer-model nondeterminism, none of which a retrieval change reaches. That is the same ceiling every retrieval-side candidate hit this week: routing 1 of 50, decomposition 0 wins of 29, precision flat across 9.2x context, representation ~1 question, coverage 1 of 50. On this corpus retrieval is already near its own ceiling. Measuring a coverage lever needs a corpus where retrieval genuinely under-covers -- a larger haystack, tighter top-K, or many owners. That is a corpus-design task to be costed, not a lever question. Predictions scored honestly: two unresolved for want of headroom, one consistent and generalised. None reinterpreted after the fact.
…, 15.3, 15.5, 17.4b, 17.9) Five surfaces that shipped, compiled, had tests, and could not be used. 15.2 -- there was NO WAY to promote a trace. The whole procedural tier (trace_kind, its index, the procedures-only filter, the prune exemption, eight live-database tests) sat behind a repository the container does not hand out, so the marker every procedural query filters on could never be written by a consumer. IReasoningMemoryService.PromoteTraceAsync now exists, as a default interface method because the surface is SemVer-locked, and it is a separate operation from CompleteTraceAsync so a completion built from a stale in-memory copy cannot silently demote a procedure. 15.3 -- `proceduresOnly` existed at every layer except the one that matters. The repository has taken the argument since 7.3; the service passed a hardcoded null, so every shipped recall path asked for "any trace" and an agent looking for how it did something before got episodes -- the wrong precedent library, returned confidently. The test asserts the ARGUMENT, because the argument is exactly what was being dropped. 15.5 -- the Core formatter rendered no trace section, so Semantic Kernel and every direct-Core consumer were blind to procedural memory while a trace vector search ran on each recall and counted into TotalItemsRetrieved. Renders task AND outcome, and the success mark is three-state: Success is bool? and null means UNRECORDED, so collapsing it into failure would present a precedent library in which everything failed -- worse than showing nothing, because a wrong precedent is acted on and an absent one is investigated. This supersedes 0.6's empty-guard for that input, and that test is updated rather than deleted so the history stays legible; the guard itself is restated on an input the formatter genuinely cannot express. 17.4b -- NodeDistanceReranker (R6) and MentionFrequencyReranker (R7) had ZERO DI registrations and ZERO RerankAsync call sites, while two public MemoryOptions flags documented behaviour no consumer could obtain and Phase 10 read COMPLETE. Both are now registered and called on the fact section. Registration is unconditional and enumerable because each owns its own IsEnabled gate -- gating registration on the flags would mean a host reconfiguring options still got nothing, the same defect one layer up. Both flags default false, so the default recall path is unchanged, and a test pins that registration alone does not enable anything. The call site rebuilds its candidate list rather than reusing the diagnostics one: diagnostics are opt-in, and depending on them would make reordering happen only for callers who had asked to be told about it. Reorder-only is ENFORCED rather than trusted -- the interface notes that adding or dropping candidates "is not supported and not checked for cheaply, so it would corrupt the section's diagnostics silently", so it is checked. A throwing reranker degrades to provider order; a lost recall is not recoverable. 17.9 -- Neo4jMicrosoftMemoryFacade recalled the full context and projected only messages, discarding every entity, fact, preference and trace it had just paid to retrieve, one line before they reached the agent. Now routed through ToContextMessages, the path the MAF provider already uses, so the #92 admission policy, the recalled-role gate and 2.5's history dedup are shared rather than reimplemented -- a parallel implementation would be a second place for all three to drift. A reflection guard fails if any IMemoryReranker implementation is ever left unregistered, because the next one would be invisible in exactly the same way. 11 tests. 4,416 unit green, Release 0 warnings on a clean rebuild.
…ault into the ground THE DOCUMENT (docs/reviews/quality-effort-and-what-did-not-move.md). Nine experiments, ~1,800 calls, six architectural candidates eliminated, and no accuracy change attributable to any of it -- because the search found a CEILING, not a lever. Recording why matters more than a delta would have, because the next person will otherwise retry these in order. It states the ceiling precisely: 8% of a 50-question run is structurally unwinnable (four questions fail 0/36 WITH PERFECT CONTEXT); 13 of 14 intermittent cells flip with byte- identical retrieval; and retrieval already sits at coverage 0.965-0.980. It also names the cause of the flipping, which is OURS: the answer call passes no ChatOptions at all, so the answer model runs at the provider default of temperature 1.0 -- the same deployment whose rejection of temperature 0 forced a compatibility wrapper onto the EXTRACTION path. The answer path never got the equivalent, and nothing pins it. A meaningful share of the noise band blunting the instrument is self-inflicted and has never been attacked. And it corrects an over-read of my own: "65 of 67 failures had gold present" is true and too coarse. Session coverage is not turn coverage -- on hybrid, correct answers average 0.937 turn coverage against 0.667 for wrong ones -- but 4 of 6 hybrid failures have turn coverage 1.0, the exact evidence retrieved and still wrong. Structured turn coverage is 0.000 everywhere, correct and wrong alike, because turn attribution runs through raw messages: an OPEN INSTRUMENT GAP, not a finding. PERF DECISION, taken and then REVERSED BY MEASUREMENT. I flipped SkipEscalationWhenOwnerHasNoRows on, re-ran the hermetic profile, and PERF-R-01 went from 13 queries to SIXTEEN -- worse. The existence probe is an ADDITIONAL query per category and only pays when the owner turns out to be empty; PERF-R-01's owner holds rows, so all it bought was three probes answering "yes, look anyway". Reverted, with the number recorded on the option: it is a bet on workload shape, not a free saving. OmitEmbeddingsFromRecall is deliberately NOT flipped either, against my initial plan. Its own doc makes the better argument: opt-in keeps the TCK bridge safe BY CONSTRUCTION rather than by remembering to check, and flipping the default converts a guarantee into a checklist item. Prompts reorganised: the three completed AgentEval asks move to strategy/prompts/done/ numbered 01-03, and 04 is the next ask -- answer-model determinism (the highest-value item in this whole document), the oracle arm made public with distractor and gold-fraction controls, and a time-grounded corpus variant, which is the only thing that would let prospective memory be measured at all. 4,416 unit green.
…faces (30.2 steps 9-13) The layer is now whole. Source quotes restore what a triple drops -- tense, participants, ordinals, three separately named failing questions -- by dereferencing SourceMessageIds, which every fact already carries and which the benchmark harness already used for dates. The hybrid arm fixes those same three by brute force at roughly six times the tokens; the prize here is structured accuracy at about 500 tokens instead of 2,505, which is why every cap is deliberate: shortest containing sentence, length cap, quotes-per-recall cap, and skip-if-the-triple-already-says-it. IMessageRepository.GetByIdsAsync is a DIM over the EXISTING MessageQueries.GetByIds statement the batch-write path already re-reads through -- a second caller, not new Cypher. The shared fetch is memoised on the state as a Task rather than a result, so quotes and date grounding share ONE round trip even when both are on. That is the "one extra read per read-feature" budget, and there is a test that counts the calls rather than trusting the design. Date grounding prefers sourceTimestamp metadata over the storage timestamp, because a corpus ingested in one afternoon has storage timestamps that say nothing. An unparseable value falls back instead of throwing: adapter-written metadata is a data condition, not a contract, and throwing on a rendering path would turn a bad string into a failed recall. Chronological ordering refuses to fire on fewer than two dated items -- one date is not a chronology, and reordering on it would rearrange the retrieval ranking, a real signal, to express an order the section does not have. ProjectionRenderer is the point of the whole layer: one vocabulary, three thin call sites, every method an identity when there is no projection. Annotations render in a fixed order -- marker, item, date, supersession, quote -- so a reader meets the caveat before the claim and all three surfaces produce the same string. Reorder keeps unmentioned items rather than dropping them; an ordering feature that could lose an item would be a retrieval bug wearing a rendering costume. All three surfaces are wired, and all three now carry sealed fingerprints. The benchmark prompt's hash was captured AFTER its restructure, so it proved nothing on its own -- I re-ran the sealed hash against the pre-change build in a scratch worktree and it passed, which is what actually proves those bytes did not move. One deliberate strengthening beyond the design, in both renderers. The design said annotate after Admit and stopped there. A source quote is recalled MESSAGE content spliced onto a fact line, so the fact's own admission check -- which ran on a clean triple -- would be bypassed by construction for exactly the content most worth checking. The annotated line is therefore re-admitted, and on failure the item keeps its BASE line rather than being dropped: it was already judged admissible, and losing it over a suspect decoration would be silent retrieval loss. Two DI problems found by the suite, both real: Registering the repository-reading features with hard dependencies broke a container that supplies its own ILongTermMemoryService and no repositories -- a shape that exists in this repo's tests and worked before. Inside an ENUMERABLE registration an unsatisfiable dependency takes the whole enumerable down, and the assembler with it; the same class of break an unconditional binding caused during the 1.0 lockdown. Fixed with nullable repositories resolved via GetService, each feature reporting itself off when its dependency is absent -- more honest than accepting the flag and contributing nothing. Then TryAddEnumerable rejected the bare-factory descriptors as "indistinguishable", because it de-duplicates by implementation type and a bare factory records the service type as its own implementation. Fixed with the two-type-parameter factory overload. Reachability is guarded in BOTH directions: every IProjectionFeature in the assembly is registered, and every bool on MemoryProjectionOptions enables at least one registered feature. The second is the same defect wearing the opposite mask -- an option that binds, validates and does nothing, which is how IncludeQuestionTypes and AbstentionPolicy were found dead only when a measurement failed to move. Both reflected, so neither can go stale. No LongMemEval run was performed and none was authorised: every flag ships dark until a spend decision is recorded, per the design's own measurement plan. Release 0 warnings; suites 4640 / 592 / 54 / 3 green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
…30.3)
RecallOptions.MinTraceSimilarityScore, null by default, resolving to MinSimilarityScore -- so nothing
changes for any existing caller and every sealed measurement stays comparable.
This is a safety property wearing a tuning knob's clothes. At the shared 0.7 default procedure
retrieval NEVER abstains: a sweep found every threshold from 0.00 to 0.86 behaves identically, a
measured dead zone, so the one setting that looks like it controls procedure precision controlled
nothing across the entire range anyone would plausibly set it to. The measured knee is 0.92; 0.90 is
the free variant, at which no correct answer was lost. The asymmetry is what makes it safety rather
than tuning: an agent handed a confident wrong procedure EXECUTES it, where an agent handed nothing
investigates. Recalling no procedure is a strictly better failure than recalling the wrong one, and at
the shared default only the worse outcome was reachable.
EffectiveTraceMinScore resolves the floor on the OPTIONS type rather than at each call site, so the
two recall paths cannot disagree -- which is exactly how SuccessfulTracesOnly came to be passed live
and hardcoded null on the as-of path.
Reachability is the thing actually asserted. A floor that is parsed, threaded and dropped before the
query is how IncludeQuestionTypes and AbstentionPolicy came to be dead options, found only when a
measurement failed to move. So the tests check the value arriving at SearchSimilarTracesAsync, not the
property returning it -- and that raising the trace floor leaves facts on the shared one, which is what
makes it per-CATEGORY rather than a rename of the global.
ProcedureShapeProjectionFeature renders a promoted procedure's length. The measured failure it
addresses: replaying the archive task promoted a 16-call exploration with its dead ends included, and
rendered as a bare outcome that is indistinguishable from a tight five-step recipe -- the model is
handed sixteen steps of someone else's flailing as if it were a method. A length is the cheapest
honest signal short of distillation, which is a separate LLM-shaped proposal with its own falsifier.
Episodes are never annotated (an episode's length is not a claim about reusability), single-step
outcomes are not annotated ("(1 steps)" is noise), and the counter is deliberately conservative
because an inflated step count on a procedure is precisely the over-trust this exists to prevent.
It shares the AnnotateMatchQuality flag rather than adding a sixth: both exist to stop a procedure
being trusted more than it has earned, and a second flag for one clause would be configuration surface
with no separate decision behind it.
Red-probed: neutering EffectiveTraceMinScore fails exactly the two tests that name the knee and the
dead zone; annotating episodes fails exactly the test that forbids it.
Release 0 warnings; suites 4653 / 592 / 54 / 3 green. Step-2 fingerprints still pass on all three
surfaces. The retrieval-precision instrument is NOT re-run here -- that is a measurement decision with
a recorded spend, and this ships dark at null.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
All five were found AFTER green suites and confident commit messages, which is the argument for the independent gate review rather than a substitute for it. 1. A SECURITY CLAIM I HAD COMMITTED AND NEVER TESTED. The steps 9-13 message asserts that the annotated line is re-admitted, closing a bypass where a source quote -- recalled MESSAGE content, far easier for an attacker to control than a triple -- rides into the prompt behind an already-admitted fact line. Nothing proved it. ProjectionAdmissionSecurityTests now asserts it on BOTH surfaces and in both directions: a hostile quote is stripped under Strict while the fact survives, and a harmless quote is KEPT, because a check that strips everything would make the feature useless in exactly the mode a security-conscious host runs. Red-probed: removing the re-admission fails precisely the hostile-quote test. 2. A FIELD USED FOR SOMETHING IT DOES NOT MEAN. ProcedureShapeProjectionFeature wrote "(16 steps)" into SupersessionNote. It rendered correctly, which is why every test passed -- and a later reader debugging supersession would have found a step count in a property whose documentation promises a supersession chain. Now its own ProjectedItemAnnotation.ProcedureShape, and the test asserts the two notes coexist rather than one swallowing the other. 3. NO VALIDATION ON ANY NEW NUMERIC OPTION. Every other numeric option in AddAgentMemoryCore is validated; MemoryProjectionOptions had none. A NearMissThreshold of 5.0 does not fail -- it silently makes the near-miss marker fire on everything or on nothing, which reads as "the feature is broken" rather than "the value is wrong". Six validators added, including MinTraceSimilarityScore. 4. DEAD SURFACE. ProjectionSectionKeys.Messages was declared and read by nothing. Removed. A future feature that annotates messages can add it back with a consumer attached -- which is also an honest statement of what shipped: no feature can annotate recalled messages yet. 5. A FINGERPRINT THAT PROVED NOTHING. The LongMemEval answer-prompt hash was captured after its own restructure, so it could only confirm the code agreed with itself. Re-verified by running the sealed hash against the PRE-change build in a scratch worktree, where it passed -- which is what actually establishes those bytes did not move. Recorded in the review doc. Also written: strategy/performance/PHASE30-WAVE-B-REVIEW.md, which additionally records the four defects the SUITE caught during the work (unsatisfiable dependency inside an enumerable registration; TryAddEnumerable bare-factory indistinguishability; the as-of divergence guard firing on Projection; two documentation guards firing twice each), the three deliberate deviations from the design, and -- the section that matters most -- what is NOT claimed: no LongMemEval run was performed or authorised, every projection flag ships dark, all four void witnesses are unexercised, and nothing here is evidence that projection improves anything. Release 0 warnings; suites 4658 / 592 / 54 / 3 green; integration 393/393 non-NAMS. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
…he design (30.4)
Everything else this system retrieves is probabilistic: query embedding, global vector top-K, owner
post-filter, similarity threshold. This tier is a point-read by owner, so it cannot be starved -- and
starvation here is measured rather than theoretical. An owner's own facts inside the global top-60
averaged 7, minimum 1, and one real question retrieved ZERO facts from a graph holding 504 of its own,
all live, all above the floor.
THE DESIGN TOLD ME TO CHECK THE UPSTREAM SNAPSHOT BEFORE WRITING ANY IDENTITY PROPERTY, AND THE CHECK
CHANGED THE DESIGN. It proposed a new constraint `user_owner_unique` on `owner_id`. Upstream v0.5.0's
:User carries {id, identifier, attributes} and is uniquely keyed on `identifier` by a constraint named
`user_identifier`. Adopting a label while keying it on a different property makes the adoption
NOMINAL -- the same spelling carrying a different meaning, which is precisely the hazard the parity
verifier cannot catch, because it compares names and not semantics. So this reuses upstream's key and
upstream's constraint name, and writes owner_id alongside so .NET's own scoping convention still reads
naturally. Both spellings agree on every node, and an integration test asserts it.
Shipped as the `working-memory` schema extension. It is the first parity delta that REMOVES an
upstream-only label: :User leaves UpstreamOnlyLabels, NetOnlyLabels stays empty, and adoption narrows
divergence instead of widening it. The declarations live on the extension rather than in
SchemaConstants deliberately -- putting :User in the base descriptor would mean .NET claims the label
even with the extension off, and the parity report would simultaneously call it adopted and
unimplemented.
GUARD G3, the TCK-load-bearing null-owner skip, is proven red-first against a live database. The
bridge's /add_fact and /add_preference route through LongTermMemoryService, so the rebuild epilogue
fires during a conformance run whenever this extension is on -- and bridge writes are OWNERLESS.
Without the skip, MERGE (:User {identifier: null}) runs and a null unique key turns Bronze and Gold
cases into 500s. The guard is one IsNullOrWhiteSpace check, exactly the line a future simplification
deletes as redundant, so removing it now fails precisely the three ownerless cases.
STALENESS IS THE KILL RULE, NOT A FOOTNOTE. Structured recall scores 8/9 on knowledge-update, the
weakest measured non-episodic type, so a block asserting the OLD value of an updated fact would
manufacture failures in exactly the type we are worst at. Hence full eager rebuild with no partial
invalidation (invalidation over a graph is the clever answer that goes stale), awaited INLINE so the
contract is "after the write call returns, the block is current", and clear-rather-than-keep on
rebuild failure because absence degrades to today's behaviour while staleness manufactures errors. The
canary goes through the PRODUCTION supersession path and asserts Acme -> Globex leaves Globex and not
Acme; red-probed by disabling the epilogue, which fails it.
AddFactAsync became a thin epilogue wrapper over a core method: it had three separate return branches
and hanging the rebuild off each is how one branch quietly stops rebuilding.
The suite caught a real bug I would have shipped: ContextFormatOptionsBridgeTests found that
IncludeWorkingMemory was not mapped across the MAF options bridge -- the option would have bound,
validated, and done nothing, which is the exact defect class this project keeps finding. Four other
guards fired and were satisfied in lockstep: the pinned extension-id list, domain-record and
service-interface counts, and the Cypher snapshot (which also required teaching KnownNodeLabels about
:User -- itself a small proof that the label really is new to this codebase).
Byte-stability is enforced, not hoped for: every ORDER BY ends in id ASC, and an unchanged rebuild
writes nothing, so built_at moves only when content moves. The budget drops entities first, then
preferences, then facts, because facts are the head of the question distribution.
Release 0 warnings; suites 4697 / 592 / 54 / 3 green; integration 418/418 non-NAMS (+25).
UNMEASURED: no LongMemEval run performed or authorised; the tier and its rendering both ship off.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
An agent resuming work re-receives everything it already processed. Full recall re-assembles the same facts at every session start, and there was no way to ask for the difference. Every ingredient already existed and was already enforced on the live write path -- created_at stamped on create only, invalidated_at stamped idempotently, SUPERSEDED_BY edges, valid_from/valid_until -- and nothing read them as a diff. Eight buckets, disjoint BY CONSTRUCTION rather than by care: the window is half-open, (since, until], in every query without exception, and the upper bound is read from the clock ONCE and handed back as the next checkpoint. That is what makes consecutive deltas partition time exactly, and it is also what makes the feature verifiable without a judge or a benchmark. The subtle case has a test named after it. Supersession stamps BOTH clocks, so a superseded fact appears as a pair AND as an expiry unless the expiry query gates on invalidated_at IS NULL -- one change, two entries, and every presence-checking test still green. Removing that gate was probed and fails exactly one test. Ships as the delta-recall schema extension: seven RANGE indexes, no labels, no properties, SchemaParityDelta.Empty. The clocks were there; the extension only makes them seekable. Gold-safe with the flag on for two independent reasons -- an index changes plans and never results, and the new members are called by no bridge endpoint. The checkpoint is a caller-held token on the MAF session state bag, not a stored node, so no schema pays for it. Advancing it is an ACKNOWLEDGEMENT, not a read receipt: after a turn completes, to the delta's own TakenAtUtc rather than to now, and never on a turn that threw. Replaying a change set costs tokens; losing one loses knowledge. Three things the design did not anticipate, found while building: - MemoryService now resolves the delta's owner scope through IMemoryIsolationPolicy. A delta reads the repositories directly -- the assembler is not in that path -- so passing a caller's scope through would have handed a caller who supplied only a UserId a cross-owner answer. - The formatter takes the caller's admission decision. The Agent Framework routes every item through a host-pluggable policy while Core calls the built-in one; hard-coding the latter would have applied a custom policy everywhere except the delta. - No "->" arrows, contrary to the design's sample output. The delimiter escapes every angle bracket -- that is how a recalled item is stopped from forging its own closing tag -- so an arrow would have reached the model as "->". Also fixed en route: SchemaExtensionDocumentationTests enumerated its subjects from a hand-written [new ProceduralSchemaExtension()] and had therefore silently stopped covering each new extension as it was added. working-memory shipped a handover page this guard never once read. It now reads CreateShipped(). Gate: Release 0 warnings / 0 errors; unit 4770, LongMemEval 592, SK 54, perf 3; integration 431/431 non-NAMS on live Neo4j. Red-probed: the half-open boundary, the double-count gate, the admission wire, the off-state flag, the checkpoint staging rule, the reachability guard, and the single-clock-read invariant each fail their own tests and no others. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
…0.8) Three Wave C extensions, gated together. The 30.6 gate run was invalidated by my own concurrent edit -- I started a background integration suite and kept working -- so the three share one clean gate rather than three, and this note is here because a shared gate presented as three separate ones would be a quieter lie than the mistake it covers. 30.6 -- ARITHMETIC MEMORY. 16% of benchmark questions have a derived answer. The store holds 800 and 50; the answer is 750 and nothing wrote it down. Every retrieval-side idea died against a saturated coverage ceiling; what is left is answers that are properties of a SET, which retrieval can only sample. The accountant is LLM-free by design -- answer-time decomposition died 0/29 on perfect context, so arithmetic moves from a stochastic reader to a deterministic writer. Every operator REFUSES rather than guesses: one unparsable object disqualifies a group's numeric operators, because the change between two values that happened to be readable is not the change over the chain. Sum is allowlisted (adding three temperatures is arithmetically perfect and meaningless); Duration is off (the corpus stamps UnixEpoch+counter, so durations there are fiction with a plausible shape). The staleness cascade is same-statement or nothing: a derived 750 whose input 800 was superseded is a manufactured confident-wrong answer, stored, embedded, recallable, wearing provenance that makes it look verified. G2 is enforced by OMISSION -- a derived fact carries no merge-key quadruple, so MERGE and FindByTriple cannot reach it. MERGE cannot carry a WHERE, and making the collision unreachable is the only form of the guarantee that holds. 30.7 -- PROSPECTIVE FIRING. Every other channel is reactive. A reminder is off-topic by definition -- nobody asks "is there anything I should know?" -- so firing selects by time alone, and that absence is the specification. Two gates (the flag, and ValidTime==Current: firing reads a validity window and a recall ignoring valid time has none to read). Its own budget, because a reminder that loses a budget contest has already failed. Rendered first, because the point of volunteering is prominence. Premature surfacing is the counter this would be withdrawn over, and it is held at zero structurally by the (since, now] window. 30.8 -- LEGIBLE FORGETTING. Forgetting worked and was INVISIBLE: decay pruned, recall returned less, and the agent answered as though it had never known -- indistinguishable, to the asker, from never having been told. A system whose gaps all look alike cannot be corrected by its user. A summary surfaces, never the content: rendering the forgotten facts would undo the forgetting. The partition is a new invalidated_reason the prune stamps and supersession deliberately does not -- a superseded fact was REPLACED, and its replacement is live and should be answering. FOUND EN ROUTE, all fixed: - CypherQueryExecutionSweepTests failed on 18 SHA-256 hashes: DerivationKey lived in the Queries namespace, which the sweep treats as "this is Cypher" and EXPLAINs. It also took six unrelated MAF integration tests down with it. Moved out of the namespace; the cascade FRAGMENT is now internal, since a fragment cannot be EXPLAINed alone and is covered better inside the statement it is spliced into. - MethodBuiltQueryStructureTests excused relationship types from a hand-written skip list with exactly one entry. Now matches labels and relationship types in their own syntactic positions, and checks both. - AsOfRecallDivergenceTests caught BOTH new features being live-path only, before either could become a discovery. Both are deliberate and now argued in the list: an as-of recall reconstructs the past, and neither present-tense urgency nor a claim about what has since been lost belongs in it. NOT RUN: the predicate-vocabulary A/B (30.6 sub-step 0). The instrument is built, wired and guarded -- --vocabulary-ab on the extraction-compare verb, one process, byte-identical input, no corpus-build noise floor -- but longmemeval_s_cleaned.json is not on this machine, so the run cannot be performed. Recorded as not taken rather than skipped: arithmetic memory's V1 void witness is therefore undetermined and the feature ships dark, which is what the pre-registration requires. Gate: Release 0 warnings / 0 errors; unit 4966, LongMemEval 611, SK 54, perf 3; integration 467/467 non-NAMS on live Neo4j. Red-probed: the cascade edge type, the derived-vs-extracted merge partition, the flag gate, the valid-time gate, the premature-surfacing bound, and the decay-vs-superseded partition each fail their own tests and no others. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
WAVE C CLOSE, PARTIAL AND SAID SO. The full gate is the 178-case upstream conformance suite run twice on one build, all-off versus all-on, diffed for identical results. That suite is neo4j-labs/agent-memory-tck, a separate Python repository, and it is not on this machine. Recorded as not taken rather than claimed, because a wave declared closed on a gate nobody ran is worth less than an open one. What was run, and is real: - `schema-parity --extensions arithmetic,delta-recall,procedural,working-memory` is COMPATIBLE on both arms, 19 documented divergences with everything on. - A new live-graph AllExtensionsOnIntegrationTests installs all four extensions TOGETHER on one database from the real migration scripts: every ext/<id>/000N key namespaced and attributable to exactly one owner, every declared index present afterwards, a second run a no-op, base bookkeeping untouched. That is the failure class the ext/ namespace exists for -- the 0012 collision only appears when two extensions are enabled at once, which no single-extension test can reach. THE DEFECT THAT RUN FOUND, and it is mine. ArithmeticSchemaExtension declared `kind` as a net-superset property. Upstream already HAS `kind`, meaning "audit-node discriminator", so the verifier came back INCOMPATIBLE: "upstream caught up to .NET superset: kind". The procedural extension had already chosen `trace_kind` over `kind` for exactly this reason, and its own comment says why -- overloading a property whose meaning is shared with another implementation is the changed-semantics hazard a parity check cannot catch, because it compares names and not meanings. I used `kind` anyway. Renamed to `fact_kind` across the extension, the Cypher, the migration, the snapshot, the guard tests and the docs. Worth stating plainly: this was caught only because the name collided too. Had I picked a name upstream did not use but whose meaning it owned, nothing here would have failed. Gate: Release 0 warnings / 0 errors; unit 4966, LongMemEval 611, SK 54, perf 3; integration 472/472 non-NAMS on live Neo4j. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
30.11 -- SELF-CONSISTENCY VOTING + QUOTE-FORCING (eval-side). Defaults are one unvoted, unforced call, byte-identical to every archived run. The pre-registered primary claim is that the BAND NARROWS across repeats, not that point accuracy rises: with a measured 14-point spread between two identical accepted runs, a point comparison on n=50 is noise wearing a decimal. Clustering is deliberately conservative -- case, whitespace, trailing punctuation, nothing more. Stripping articles or stemming would merge answers a judge scores differently, turning a disagreement the model genuinely had into a consensus the aggregation invented, which is the one thing a variance-reduction technique must not do. A three-way split is REPORTED, not resolved: an LLM tiebreak costs money and that decision does not belong inside an aggregation helper. The two halves compose only if votes cluster on the ANSWER rather than the two-line envelope -- otherwise agreeing answers citing different quotes count as disagreement and the features fight each other. The void witness here is a live outcome, not a formality: Proposal F assumed temperature 1.0 IS the sampler, and 30.1 then measured that seeding halves answer variance on this deployment. 30.12 -- ACCESS TRACKING ON A ROOT-OWNED QUEUE. DeferAccessTracking already made this write fire-and-forget, and its own documentation admits the flaw: the write starts inside the request scope, so a host that disposes that scope on response completion disposes the repository under an in-flight write -- an ObjectDisposedException in a log nobody reads, after which tracking silently stops. A singleton owned by the root container, taking a fresh scope per batch, fixes that by construction. The other half of the pair (recall projections) was already shipped across all three vector repositories. TWO DEFECTS IN MY OWN FIRST DRAFT, both found by the tests I wrote for it and both of which would have shipped looking correct: - Under BoundedChannelFullMode.DropWrite, TryWrite returns TRUE and discards the item. My drop counter keyed on the return value counted zero forever while the queue silently threw work away -- exactly the "quietly discarding its input" failure the class comment warns against, written into the class carrying that comment. Now counted through the channel's itemDropped callback. - A singleton implementing only IAsyncDisposable makes ServiceProvider.Dispose() throw, breaking every host that disposes its container synchronously. Now implements both. A third came from a red probe: the reachability guard for the voting flags passed with the Program.cs assignment commented out, because a substring assertion over raw source is satisfied by a commented-out occurrence -- which is the most likely way that wire actually gets severed. The guard now strips comment lines, and the probe fails as it should. NOT MEASURED. Both tasks' acceptance criteria are measurements -- band width before/after for 30.11, hermetic P50 -30% with query counts unchanged for 30.12 -- and neither can be taken here: 30.11 needs the absent corpus plus paid answer calls, and the hermetic perf harness gates on query counts, which this change deliberately does not alter. Both ship dark and are recorded as built-not-measured. Gate: Release 0 warnings / 0 errors; unit 4981, LongMemEval 639, SK 54, perf 3; integration 472/472 non-NAMS on live Neo4j. Red-probed: the queue branch and the option-to-adapter wire each fail their own tests and no others. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
An adversarial pass over everything built this session (30.5-30.12), before any independent review. Six defects had passed every suite. All are fixed here with a test each, verified to fail without the fix. 1. WithoutCallerSuppliedDerivation was written, documented as enforcing a framework-reserved-key rule, and called by nobody. memory_add_fact accepts free-form caller metadata and stripped only trust_level, so a client could stamp fact_kind='derived' with an invented derivation string -- handing the model arithmetic no accountant performed, wearing the inline provenance that makes it look checked. Worth stating plainly: I built a shipped-but-unreachable defect in the same session as writing three reachability guards against one. The guards covered the features; nothing covered the helper. 2. Fourteen new numeric options with zero validators. Wave B's self-review found six; Wave C added eight more the same way. Every one misconfigures silently. MaxGroupFanIn is the sharp one: no evaluator aggregates fewer than two facts, so a cap of 1 disables arithmetic memory entirely while the flag still reads as enabled -- producing exactly the "measured no effect" result the feature would then be blamed for. Now > 1 by validator. 3. TotalItemsRetrieved counts DueFacts, ExpiringFacts and ForgottenTopics -- necessarily, or the formatter's zero-items return swallows a recall whose only content is a reminder -- but McpMemoryProjection projected none of them. A count that disagrees with its own content is worse than either being wrong alone, because a reader cannot tell which half to believe. 4. The access queue's drop counter could never fire. Under DropWrite, TryWrite returns TRUE and discards the item, so a counter keyed on its return value read zero forever while the queue silently threw work away. That is the exact failure the class comment warns about, written into the class carrying that comment. Now counted through the channel's itemDropped callback. 5. A singleton implementing only IAsyncDisposable makes ServiceProvider.Dispose() throw, breaking every host that disposes its container synchronously. 6. A reachability guard stayed green with the assignment it guards COMMENTED OUT -- and commenting a line while debugging is the likeliest way that wire gets cut. Raw-source substring assertions are satisfied by comments; now stripped. The pattern behind two of these is worth keeping: a guard that enumerates its own subjects by hand grades whatever it was born knowing about. SchemaExtensionDocumentationTests had silently stopped covering each new extension, and MethodBuiltQueryStructureTests excused relationship types from a skip list with exactly one entry. Both now derive their subjects from the source of truth. Also recorded in the review doc, and not fixed because they are not defects: what is NOT verified. The 178-case TCK run (external suite absent), the predicate-vocabulary A/B (corpus absent), 30.11's band claim, and 30.12's P50 claim -- the last because the hermetic harness gates on query counts, and this change moves where the write runs, never whether, so no counter moves. Every Wave C/D/E feature ships off by default and has never been measured on. Gate: Release 0 warnings / 0 errors; unit 5001, LongMemEval 639, SK 54, perf 3; integration 472/472 non-NAMS on live Neo4j. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
MEDIUM finding from the independent Wave C-E review, closed red-first.
`agentmemory migrate` configured the URI, credentials, database and embedding
dimensions and never touched Neo4jOptions.Extensions. MigrationRunner applies
exactly the extensions that option names, so the one command an operator runs to
bring a database up to date applied base migrations only -- and no supported path
existed for ext/<id>/000N DDL at all.
What made this worth a code change rather than a doc note is that it fails
QUIETLY. A host can enable `arithmetic` in code, meet a live graph with no
fact_derivation_key_idx, and keep working: the MERGE still converges, the queries
still return correct results, and the only symptom is a full scan where an index
seek belonged. Nothing errors, so nothing is investigated.
`--extensions <id,...>` now applies on every database-backed verb -- not just
migrate, because schema-check's owners report reads the same set to decide which
shapes should be present, and a flag honoured on one verb and ignored on the next
makes the two disagree about the same database. Same CLI > Neo4j:Extensions >
NEO4J_EXTENSIONS precedence as every other connection setting; the flag name
matches what schema-parity and the TCK bridge already used.
Two smaller decisions worth their comments:
- Validated BEFORE the host is built. The registry already refuses an unknown id
-- that refusal is the point of the activation mechanism -- but it does so from
inside the DI graph, where an operator sees a wrapped exception instead of
"unknown extension 'aritmetic'; known: ...". A typo should end in a correction.
- Case-sensitive. The id is part of the (:Migration).version key
("ext/arithmetic/0001"), so accepting "Arithmetic" and storing it verbatim
would orphan every previously-applied row and split one history in two with no
error at any point.
The five integration tests invoke the BUILT CLI AS A PROCESS rather than the
command class in-proc, because in-proc skips the Program.cs top-level statements
where the gap lived. Red-probed: commenting out the host assignment fails exactly
the two tests that assert DDL landed and leaves the off-state ones green -- so the
test detects the original defect, which is the only thing that makes it worth
having.
Docs say who runs it: docs/extensions/README.md gains a "Who runs the DDL"
section stating plainly that registering an extension in code does not create its
schema, and CLI help gains a SCHEMA EXTENSIONS section saying the same.
ALSO, the two LOWs.
Row 15's divergence count was 19; it is 20. The 19 was read off the parity run
taken BEFORE the kind->fact_kind rename in that same task, and the rename added a
net-superset property. A number copied from a superseded run is exactly the drift
a ledger exists to prevent.
Row 17, the self-review doc, and commit 0ae6d75's subject line all said "six
defects survived green suites". Three did not. Items 4-6 -- the drop counter that
could never count, the async-only disposable, the comment-defeated guard -- were
caught by my own unit tests and a red probe DURING the 30.11/30.12 build, and were
already reported in fc2ea9f. Counting them again inflated the review's yield
threefold and credited a review with what ordinary build discipline had already
found. Corrected to three-by-review, three-by-build. The commit message stands as
written rather than amended: the correction belongs beside the claim, not in
place of it.
Gate: Release 0 warnings / 0 errors; unit 5015, LongMemEval 639, SK 54, perf 3;
integration 477/477 non-NAMS on live Neo4j.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
crosslang-architecture.md §5 Step 0, executed under the private-demo carve-out.
Throwaway by design: a prototype host and a stdlib-Python script, answering one
question before any contract package exists.
Given only the wire JSON, can a non-.NET client reconstruct the same answer the
.NET caller got?
THE COMPARISON IS ASYMMETRIC ON PURPOSE. Calling the wire and the engine and
byte-comparing cannot work -- the two produce different shapes, so the compare
would only ever report "different types". Instead the host serves the draft
am-wire/1 response AND the canonical projection built in C# from the domain
object, and the Python script builds THE SAME PROJECTION FROM THE WIRE JSON
ALONE. Every field the script reads is a field the DTO had to carry; a field the
DTO drops is one Python cannot reconstruct, and the compare fails. That asks
whether the wire is SUFFICIENT, which is the question.
GATE MET. Parity on all five fixtures including the two the gate names:
isolation Alice's recall returns five facts, all ownerId=alice. Bob's
works_at Globex is absent -- and the query deliberately asks "who
does bob work for?", which is the strongest form of that test.
ownerId is on the wire so a client can VERIFY isolation rather
than trust it.
as-of March returns works_at Acme AND lives_in Zurich; September returns
works_at Initech and Zurich is gone. The valid-time clock moved the
employer answer and the transaction clock removed the superseded
city -- both clocks demonstrably crossing the wire.
Spot-checked for MEANING, not just agreement: two paths agreeing on the wrong
answer would pass a parity test and teach nothing.
TWO FINDINGS ABOUT MY OWN HARNESS, recorded because they nearly weren't.
The first run passed all five fixtures WHILE COMPARING NOTHING. Every fixture
returned zero items, and two empty results are byte-identical, so the script
reported parity. The cause was StubEmbeddingGenerator: deterministic vectors with
no semantic relationship, so nothing cleared the shipped 0.7 similarity floor.
The spike now recalls at floor 0 -- it asks whether the wire carries a result,
not whether retrieval ranks well -- and the script carries a VOID WITNESS: a
fixture with nothing in its comparison is reported VOID and fails the run. A gate
that passes on empty results is not a gate. A second witness spans the as-of
pair: identical answers at two instants mean the clock had no effect, however
cleanly they byte-matched.
The first FAILING run printed "the wire cannot express the .NET result" when the
real cause was an unbootstrapped database returning HTTP 500. Instrument failure
and result are now counted separately, because they lead to opposite conclusions:
one says fix the harness and re-run, the other says stop and write it up.
RULES RESPECTED. Zero diff to src/, checked on the staged tree at this commit.
Not in AgentMemory.slnx -- a prototype must never gate the repo's build or CI.
Named Spike0.Host, not AgentMemory.Spike0.Host, because the root
Directory.Build.props attaches multi-targeting and NuGet packaging metadata to
every AgentMemory* project and a throwaway must not enter the packaging story.
No publish, no announcement; /v1/meta says PROTOTYPE on its face.
DEVIATION, STATED. §5 specifies a worktree on branch spike/crosslang-server; this
was built in place under crosslang/ with the zero-src-diff invariant checked at
commit, which is the checkable property meeting-demo-track.md's binding Rules
section actually names. Everything is under crosslang/, so relocating it later is
a directory move and nothing else.
Repo gate unaffected: Release 0 warnings / 0 errors.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
…lding it
The adapter implements only batch/abatch, so put/get/search stay LangGraph's own
concrete methods dispatching into ours. That is what makes the claim checkable:
`store.search(ns, query=..., filter={"as_of": ...})` is not a signature we
invented, so an existing agent gets point-in-time recall by adding one dict key.
demo_langgraph.py fails if any of those four methods is ever overridden -- an
override would leave every line of demo output reading identically while the
claim became false.
Spike 0's wire was read-only. D2 adds the verbs a store needs to be a store:
a typed write, a point-read, the working-memory block, and delta. `supersedes`
on the write is the load-bearing one: an update CLOSES the old fact instead of
overwriting it, which is why beat 5 can still answer "what did we believe in
March" at all.
The run self-voids rather than reassuring. Both findings below were caught by
that, not by reading the code:
1. mention_count is never incremented by the single-add API. AddFactCoreAsync
reaches the triple MERGE (which increments) only when dedup-on-create is off;
at the shipped default a re-assert -- including a byte-identical triple --
routes to MarkDeduplicated, whose Cypher sets confidence and nothing else.
Measured both arms: dedup ON -> mention_count 1, empty block; OFF -> 2, block
compiled. Since the working-memory tier admits at MinFactMentionCount = 2, a
fact added through AddFactAsync can never become stable however often the
world re-asserts it, and MentionFrequencyReranker loses the same signal.
Extraction is unaffected (UpsertBatchAsync increments, and its own comment
says the counter must not depend on which write path ran -- exactly the
invariant the single-add path breaks). NOT fixed here: zero-src-diff is
binding on this track. Reported in crosslang/demo/README.md.
2. as_of moves both clocks. March recall correctly returned nothing, because
everything had been recorded seconds earlier. Pinning the transaction clock
to "now" on the read would have answered a different question than the caller
asked, so the write carries `recorded_at` instead.
Zero diff to src/, verified. No PyPI, no packaging, no announcement.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
D3 ships four artifacts plus the preflight the run sheet makes mandatory: - DEMO-SCRIPT.md -- minute-by-minute, a costed fallback order, and a claim audit marking every spoken line [S]hipped / [P]rototype / [D]esign. A claim not on that list does not get said. - agentmemory_langgraph.ipynb -- store, resume-with-delta, as_of, provenance walk. GENERATED from build_notebook.py, never hand-edited: a .ipynb is JSON with embedded outputs, and hand-editing one is how a demo ends up with printed output that no longer matches the code above it. - ONE-PAGER.md -- including the honest feature table with the four rows where the gap runs our way. Conceding those in print is what buys the rest of the table its credibility. - screencast.txt + screencast.py -- a real captured transcript replayed with typing cadence, needing NO host, database, or network. Fallback A is reached precisely when the environment is what failed, so it must not depend on one. Recording aborts and writes nothing if any step exits non-zero. The host gained /v1/history: the provenance walk needs supersession links in both directions, which is what makes an as_of answer auditable rather than merely surprising. D4 rehearsed on a container destroyed and recreated from scratch, twice in a row (the design's definition of done -- the second run starts already superseded, so it proves nothing has to be reset by hand between takes). Machine time 9.2s / 7.3s against a 10-minute budget: the demo is entirely speaking time. Fallbacks were rehearsed by BREAKING things, not on paper. Killed the host and the database: preflight failed on the first check, named the fallback, exited 1 without hanging on the connection; the replay ran clean with nothing up. Fallback D's curl commands verified as printed. Three things the build caught: - preflight initially reported NOT READY because it wrote each fact once -- the mention_count finding from D2 biting its own harness. It now re-asserts, exactly as the demo does. - Debugger noise leaked into the first recording, and a repeat run left valid_until and invalidated_at drifting apart. Both fixed; the committed transcript is recorded on a fresh database where the stamps agree. - Chasing that drift, the hypothesis was that re-asserting a superseded fact leaves it live on one clock and expired on the other, returnable by no read. A probe against the running system DISPROVED it -- the fact came back in live recall. Recorded in DRY-RUN.md because reading the Cypher made the wrong conclusion look obvious. Known gap, stated rather than glossed: no video file. The transcript and replay exist and Fallback A is functional; a video needs a human to press record, and RECORDING.md has the steps and a review checklist. Zero diff to src/, verified. No PyPI, no packaging, no announcement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
Found by reviewing the D2 build, not by anything failing -- which is the point:
all three would have surfaced in the room or not at all.
1. get() ignored the namespace entirely. `get(("memories","alice"), key)`
returned BOB's fact. The engine's by-id read is deliberately unscoped (an id
is treated as an already-owned handle, a defensible engine-level choice), but
the store contract is not, and owner isolation is this adapter's headline
claim. The namespace is now enforced on the way out.
2. search() offset ate the limit: the host was asked for `limit` rows and the
first `offset` were then dropped client-side, so page 2 of a 2-row page came
back EMPTY -- which reads as "no more results" rather than as a bug.
3. An ownerless namespace was guessed at. `("memories",)` scoped to an owner
literally named "memories": failing closed, but silently, leaving the caller
with an empty store and no reason why. Now an error naming the convention.
test_store_contract.py covers all three and runs first in the rehearsal. Each
fix was red-probed: reverting one fails exactly its own test and nothing else.
Screencast re-recorded on a database created minutes earlier, so the committed
transcript matches the fixed code and its two clocks agree.
Full repo gate re-run to confirm zero-src-diff means zero impact: Release 0
warnings / 0 errors; unit 5015, LongMemEval 639, SK 54, perf 3 -- identical to
the pre-demo-track baseline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
The extension system and eight new capabilities shipped; the two docs that
describe this project to the outside world had not moved. Extended rather
than rewritten -- architecture.md keeps its structure, memory-map.md keeps
both its vocabularies and its status labels.
docs/architecture.md
- Sec 2.1: the projection layer and the schema-extension system placed inside
existing packages rather than beside them, which is the point of both.
- Sec 3.2.11 (new): the working-memory tier. A point-read by owner, so it is
the one channel that cannot be starved -- with the two flags it takes (build
and render are separate states), the token budget's drop order, and the
ownerless-write guard that makes the TCK gate pass.
- Sec 3.6 (new): every capability in this cycle ships dark. One table, eleven
flags, each with its real default and its real path. Prospective firing is
marked gated twice, because the flag alone genuinely does nothing.
- Sec 4.7 (new): schema owned by extensions, not by base. Sec 4.1-4.6 describe
what every database has; this says what only an operator's `migrate
--extensions` creates.
- Sec 7: answer voting and quote-forcing recorded as evaluation instruments,
with an explicit note that they have no presence in src/ at all. They are
not something a consuming application can turn on and must not read as if
they were.
docs/memory-map.md
- Sec 2.6 (new): the eight additions placed on both vocabularies. None is a
seventh memory type; six have never been measured, and the section says so
in the same words the rest of the document uses.
- Legible forgetting filed under meta-memory in Sec 6.5 and Sec 8.4 -- the
first mechanism here whose whole output is a statement about what the system
does NOT know -- while keeping the SUBSTRATE ONLY verdict, since nothing yet
acts differently because of it.
docs/extensions/README.md
- Why this exists ("adding a memory component updates the brain"), the parity
delta's five axes and the three ways composing them can fail, the owners
report with a rendered example and both orphan conditions, and a "how to
write one" walked off the real ProceduralSchemaExtension.
- States plainly that ISchemaExtension is internal for the 1.x line: this is
how an extension is written in this repository, not a plugin API.
- R1 is the only rule with code enforcement, in three places. R2/R3 have none
and are not claimed to -- they rest on the 178-case gate, which passed on
both arms with no counter drift, and which proves nothing was disturbed
rather than that anything was exercised.
AGENTS.md (new)
The cross-harness agents.md file. Every command verified by running it:
Release-equivalent build is 0 warnings / 0 errors, and the test invocations
are the ones ci.yml runs, filters included.
Drift found and corrected at HEAD:
1. `kind='derived'` in three places (architecture Sec 3.2.10 twice, memory-map
Sec 4.1a, extensions index). The shipped property is `fact_kind` -- the
first draft used `kind` and the parity verifier rejected it. Note the one
place `kind` still legitimately lives: as a key inside the in-memory
Fact.Metadata dictionary. Different layer, not a leftover.
2. "The five features are registered unconditionally" -- there are six.
ProcedureShapeProjectionFeature landed later and shares AnnotateMatchQuality.
3. Delta recall did not "fill the projection layer's reserved DeltaSummary
slot". Nothing in src/ emits DeltaSummary, WorkingMemoryProfile or
DueReminders; all three enum members are still reserved, and the three
capabilities they name each render through their own path.
4. memory-map Sec 6.4 was titled "firing absent" and Sec 8.3 called firing
"explicitly out of scope" -- both written before 30.7. Query-triggered
firing shipped; the wall-clock scheduler is what remains out of scope, and
that distinction is now the section rather than a footnote.
5. Sec 3.2.9b described legible forgetting without ever naming its flag.
6. `DerivedMemory.Enabled` written as if it hung off MemoryOptions. It is
MemoryOptions.Extraction.DerivedMemory.Enabled.
Documentation and reachability tests green (30/30) against the edited pages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
…ixed) Three of the four fact write paths maintain mention_count. MarkDeduplicated was the outlier: it set confidence and nothing else. That mattered because LongTerm.DeduplicateOnCreate defaults to true, so a re-asserted triple -- including a byte-identical one, which trivially clears the similarity threshold -- reaches this statement instead of the triple MERGE. The counter never left 1 on the single-add API. Downstream, the working-memory tier admits at MinFactMentionCount (default 2), so a fact added through the direct API could never become stable however often the world re-asserted it, and the block stayed empty without saying why. MentionFrequencyReranker lost the same signal. UpsertBatch's own comment states the invariant this violated: the counter must not "depend on which write path ran". A dedup hit is the world asserting the fact again, only in different words. Red-first: both new tests failed with "found 1" before the change and pass after. The second one asserts the invariant directly -- upsert-twice and upsert-then-dedup must reach the same count -- so a future divergence between the paths fails here rather than surfacing as an empty block. Consumer audit caught the Cypher snapshot, working as designed: regenerated, BOM stripped, and the diff is that one query and nothing else. Gate: Release 0/0; unit 5015, LongMemEval 639, SK 54, perf 3; integration 479/479 non-NAMS (+2 new; the 29 NAMS failures are the deprovisioned external workspace, unchanged). Found while building the LangGraph demo adapter, whose writes take exactly this path. Reported in crosslang/demo/README.md at the time because the demo track was bound to zero src/ diff. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
…o more An end-of-phase review sweep. Three defects, each red-first or red-probed. 1. THE WORKING-MEMORY TIER WAS INERT ON THE PRIMARY WRITE PATH. The rebuild hook shipped only on LongTermMemoryService's single-add methods. Extraction persists through PersistenceStage, which never called it -- so a host that enabled the tier and then ingested normally (conversation -> extraction -> persist, which is exactly what the MAF adapter does) compiled a block never. Recall fetched null forever while the feature read as enabled. working-memory-tier.md step 287 specifies this hook on the persist path; it was not built. Composed with the mention_count defect fixed in 0f6ddea, each write path had exactly one of the two halves the tier needs: extraction -> PersistenceStage : mention_count increments, no rebuild AddFactAsync -> dedup : rebuild fires, mention_count stuck at 1 Since the block admits only facts at MinFactMentionCount (default 2), the tier could not produce a populated block on ANY path. That is precisely what the LangGraph demo hit: the block came back empty until dedup was disabled. It survived a green suite and an independent gate review because EVERY working-memory test called RebuildAsync directly. Testing the thing instead of its trigger cannot detect a trigger wired to the wrong path. Placed in PersistenceStage rather than beside the session accountant in MemoryExtractionPipeline: both pipeline paths go through PersistAsync, as would any direct IPersistenceStage caller, and the "at least one thing landed" gate the design specifies is the PersistenceResult counts. The ordering cost -- a fact derived in the same pass is one rebuild late -- needs a non-default MinFactMentionCount of 1 to be observable and self-heals on the next write. Reasoning recorded on the method. Red-probed: removing the wire fails exactly its two tests and nothing else. 2. TypeStrictFiltering was a dead option. Public, documented "when true, only match candidates of the same entity type", defaulted true, with tests asserting its default and one NAMED after it -- and nothing read it. Setting it false changed nothing; candidates were always fetched by type. An earlier note reasoned it was unimplementable "because the repository has no unfiltered GetAll contract" -- true, but the wrong contract to want: loading every entity per resolution would be an unbounded read on the write path. GetByNameAsync is bounded by the name, matches aliases, and carries the owner filter, which covers the case the flag exists for (extractor mistyping the same entity across turns). Owner scope is unchanged in both modes -- relaxing the type boundary must never relax the ownership one. 3. The working-memory options had no validation, while every Wave-C sibling had it. Three of them become a Cypher LIMIT and a rebuild failure is deliberately swallowed so the write still succeeds -- so a negative cap produced a warning in a log nobody reads and a block that never existed. This is the third time in this phase that a feature shipped options nothing validated; the test class that already says so now covers them. Also corrected a stale comment: three ProjectedBlockKind members are documented as the reserved slots for working-memory, reminders and delta, on the reasoning that each would need "only a slot here". All three shipped as their own ordered sections instead, so the members are unused. The prediction was wrong in a useful way -- what prevents a fourth rendering path is the shared section pipeline, which exists. Gate: Release 0/0; unit 5033, LongMemEval 639, SK 54, perf 3; integration 479/479 non-NAMS (the 29 NAMS failures are the deprovisioned external workspace). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
…ens (R2) Review revision 2. The first item is the "smell worth a log" left open at the end of R1; reading it properly turned up two more in the same class. 1. THE QUEUE SILENTLY DISCARDED WORK. Under BoundedChannelFullMode.DropOldest, TryWrite returns TRUE and throws the oldest queued item away. Nothing counted it and nothing logged it, so an operator whose entities stopped being enriched had nothing at all to look at. This is the SAME defect already found and fixed on MemoryAccessTrackingChannel in Wave E, whose comment spells out the trap -- a drop counter keyed on the return value reads zero forever. The fix there was the itemDropped callback; it had never been applied here. Same callback, same first-then-every-hundredth log, plus a Counters property so a test can assert it. EnrichmentQueueOptions is Enabled by default, so this is a default-on path for anyone who registered a provider. 2. DisposeAsync swallowed TimeoutException with an empty block. A drain that ran out of its 5s grace abandoned whatever was queued and reported nothing, so "enrichment is slow" and "enrichment is silently losing work" looked identical from outside. It now counts what it abandons and says so. 3. Dispose() disposed the CancellationTokenSource immediately after Cancel(), while the workers still held that token. A consumer registering a callback on a disposed source throws ObjectDisposedException inside the worker, faulting the processing task on a path where nothing observes it. Cancellation alone is what stops the workers; the CTS is now disposed only on the async path, which waits for them first. Skipping it costs nothing measurable -- the only unmanaged resource Dispose releases is the WaitHandle, allocated lazily, and this class never asks for one. Verified rather than assumed. Red-probed: removing the drop callback fails the drop test, removing the abandoned count fails the shutdown test, and neither takes anything else with it. The probe also exposed a flaw in my own new tests -- a failed assertion left a worker parked on a TaskCompletionSource, which then toppled an unrelated test in the class and buried the real failure. Both now release in a finally. Two more, found by an R2 sweep for tokens accepted and ignored: 4. Neo4jMemoryStoreProvisioner.CreateDatabaseAsync was the only method in its file without ThrowIfCancellationRequested -- EnsureStoreAsync, BootstrapDatabaseAsync and ValidateVectorIndexDimensionsAsync all had it -- and it is the one that runs CREATE DATABASE ... WAIT, which blocks until the database is online. 5. GdsAvailability.IsAvailableAsync accepted a CancellationToken and never forwarded it to the probe. Swept clean, reported rather than embellished: no repository read ignores its MemoryScope; every MemoryOptions sub-option consumed via IOptions<T> is bridged; all seven throwing default interface methods have implementations; the schema extensions are complete (procedural's absent ext/ DDL is the documented retro-wrap, its schema being base-resident in 0011); the nine Debug-only catch(Exception) sites are all deliberate degradation on optional features with a stated reason; no static mutable state; the one undisposed HttpClient is injected, so disposing it would be the bug. Gate: Release 0/0; unit 5039, LongMemEval 639, SK 54, perf 3; integration 479/479 non-NAMS (29 NAMS = deprovisioned external workspace). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
…tated An independent review of the demo track audited every factual claim in the printed artifacts against a shipped artifact or a measurement. Four failed. None was in the live demo path; all four were in the handout and the run sheet, which is worse, because those are what the room keeps. Must-fix: - "TCK 178/178 daily" was false. Nothing runs the TCK daily; the only cron is weekly and runs a static schema-parity check. This was printed for the people who own the TCK, who can disprove it from their own repository. Now: base and all four extensions, same build, last run 2026-08-16 — which is both true and a stronger claim. The source in one-core-analysis.md is corrected too, with the weekly-static distinction stated. - Working memory and delta recall were printed as "shipped", flattening them into the same word as bitemporal recall. Our own memory map marks both BUILT, WIRED, UNMEASURED and off by default. They now say so, and the page explains why one word was not enough. - Fallback D curled nb-alice, an owner only the notebook populates. Reached before the notebook runs, both curls answer identically — two matching answers in front of the room, the exact failure the beat check exists to prevent everywhere else. It now curls the owner the preflight itself writes and verifies, and says why in the printed output. - Fallback A pointed at a .txt in a browser. The rehearsed fallback is the replay command, and there is no video. The setup list and the fallback row now name `python screencast.py`, and the missing video is stated where the presenter will see it. Wording, same discipline: the bolded absolute "the one no other BaseStore backend can do" traces only to our own doc asserting it, so it now hedges the way the spoken script already did; "No key-value store can do this" becomes the conditional the README already stated; a notebook line claiming the as_of searches "surfaced nothing to anyone" is contradicted by the cell above it, and now says what is true — they returned answers, they were not recorded; and the script's "timings are the dry-run measurements" now separates the measured machine time from the speaking budgets. screencast.txt is deliberately NOT edited. It is a genuine capture, and falsifying a record to match new wording would be worse than the record being stale. RECORDING.md now warns to re-capture it before filming, and also corrects its claim that *.mp4 is gitignored — nothing excludes it. The notebook is regenerated from build_notebook.py, its source of truth. No src/ change; the zero-diff rule holds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
joslat
changed the title
…ot wrong An independent review of the R1/R2 rounds found the demo still documenting a src/ bug that no longer exists, and — more usefully — one assertion in that section that was wrong when it was written. - Finding 1 is marked FIXED (0f6ddea) and kept as the record of what building the demo found. The measured table now shows both arms landing on the same answer. - "The shipped conversational pipeline is unaffected" was false. Not because extraction's counter was broken, but because PersistenceStage had no working-memory rebuild hook at all — so the conversational path produced nothing either, for a different reason, while this document asserted its safety from a counter that happened to work. Both halves are fixed; the trigger set is still incomplete and the PR body now says which seams. - The guessed fix said "plus the preference twin". There is none: Preference has no mention_count, its block section admits on confidence, and the preference dedup path already bumps that. Confidence-only is correct there. - SPIKE0_DEDUP_ON_CREATE no longer reproduces a failing arm, and the host's DeduplicateOnCreate=false override now works around a bug that is gone. Dropping it would demo shipped defaults — the stronger story — but that needs one live verification run, so the rehearsed configuration stands and the override carries a TODO instead of a silent change. No behaviour change: comments, a README section, and a TODO. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
joslat
added a commit
that referenced
this pull request
Aug 16, 2026
Trivial: main's extra commit is the merge of this branch's own base, so this brings in no code the branch has not already seen. Content diff against main is unchanged at the same 10 harness files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
This was referenced Aug 27, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 30 in one branch: eight new memory capabilities, the schema-extension
system they ride on, the projection layer that renders what the store already
knew, and a prototype cross-language demo. Every new capability is off by
default and byte-identical when off — that is the load-bearing property of
this PR, and it is proven by sealed prompt-hash tests, not asserted.
What ships
Wave A — foundations
30.1Extraction seed wired end to end, then measured ineffective againstits own pre-registered bar (1.93–2.03x vs the ≥3x rule) and published as a
null. Answer-determinism probe across six deployments:
gpt-4o-miniispinnable, the rest partially.
30.14SchemaExtension system — named, versioned, additive-only moduleswith their own
ext/<id>/000Nmigration namespace, parity deltas, an ownersreport that fails on any unowned shape, and per-extension TCK profiles. Four
extensions ship:
procedural,working-memory,delta-recall,arithmetic.Wave B — projection
30.2Projection layer: one enriched model, three render surfaces. Thestore's similarity scores, supersession chains, conflicts, source quotes and
real dates now reach the prompt instead of being computed and discarded.
30.3Procedure similarity floor at the measured 0.92 knee, with renderedabstention.
Wave C — the extensions
30.4Working-memory tier (profile block on upstream's:User— the onlyparity-improving change in the set)
30.5Delta recall —RecallChangedSinceAsync: what changed since acheckpoint, transaction-clock only, exactly-once
30.6Arithmetic/derived memory (fact_kind='derived',DERIVED_FROM,invalidation cascade) — built, gated, ships dark: its A/B is unrun
30.7Prospective firing (DUE/EXPIRING, one time-predicate query, noembedding — that absence is the spec)
30.8Legible forgetting — tombstones: "I no longer know" as a statedistinct from "I never knew"
Wave D/E — self-consistency voting and quote-forcing (eval-side); a
root-owned async access queue and recall projections (built, unmeasured — the
speed pair is structurally invisible to a counters-only gate, and its own
measurement gate is named rather than faked).
Demo track (
crosslang/, prototype) — a draft wire contract, a spike host,a LangGraph
BaseStoreover AgentMemory with anas_offilter, and a demo kit.Labelled prototype in nine places including the live
/v1/metaresponse.Docs — architecture, memory map and extension pages brought back in line
with the code (six claims had drifted), plus
AGENTS.mdat root with everycommand verified by running it.
Evidence
54, perf 3; integration 479/479 non-NAMS (29 NAMS failures are a
deprovisioned external workspace, disclosed —
--list-testsconfirms exactly29
Integration.Nams.*).build, same upstream kit commit — re-run independently from a scratch
environment.
twelve scenarios is byte-identical to the committed baseline, deterministic
across ten iterations. Eight features added; the engine does not execute one
extra query, transaction, embedding or model call while they are off.
Reviews, and what they cost us
Two builder review rounds and five independent fresh-context gate reviews. The
reviews found what the green suites did not — most seriously, the
working-memory tier was inert on both of its write paths (
PersistenceStagehad no rebuild hook; the dedup path never incremented
mention_count) whilepassing its own gate, because every existing test called
RebuildAsyncdirectly and none tested the trigger. Both halves are fixed with a spy-based
trigger test, red-probed by deleting the epilogue and watching exactly two
tests fail.
Three corrections to earlier claims in this branch, made because a reviewer
checked rather than because anything broke:
ingestion, single-add (incl. dedup), supersede, the MCP add/supersede tools
and all MAF sites. It does not yet cover
MergeEntitiesAsync(named bythe design itself and never built),
AddEntityAsync, the threeInvalidate*paths, orDeletePreferenceAsync— the invalidation gap beingthe same staleness the tier exists to prevent. Scoped and queued in
working-memory-tier.md; the tier is off by default, so nothing inproduction is affected today.
reasoned-and-fixed rather than test-demonstrated: the enrichment-queue
Dispose()CTS fix (a reviewer reintroduced the bug and all 29 tests stayedgreen), provisioner cancellation, and GDS token forwarding. The remaining
nine are red-first or red-probed.
src/bug that is now fixed, andcontained an assertion that was wrong when written; both corrected in this
branch rather than quietly dropped.
Enabled cost of the working-memory tier is +4 read transactions and up to +1
write per persist, sequential — undeclared until now, and unmeasured, because
the hermetic baseline was captured with the tier off.
Known open, none blocking
Three DI-bound option families lack validation (low); the demo screencast needs
a human to record it;
30.6's A/B needs a corpus absent from this machine;30.9c/30.10are external-track; the Neo4j meeting is30.15. TheWorkshop/*.mddeletion (−989 lines) is unrelated Squad material with a brokenlink, removed deliberately.