Phases 4, 7-9, 11-12: procedural falsifiers, per-message trust, per-item provenance, write-time resolution, the sufficiency AUC, and a runnable MCP server - #182
Merged
Conversation
PLAN 5.5. A per-type subset is small -- episodic is 6 of 50 questions, i.e. 16.7 accuracy points per question -- and the whole-run band (~±9 points at n=50) does not transfer to it. Quoting a per-type figure without its own band is how a table invites exactly the comparison it cannot support. It also cannot be inferred from question count: extraction on this deployment is non-deterministic, so two builds of one configuration differ in what they stored, and the spread that matters is the measured one rather than a binomial estimate. Separates() is deliberately crude and conservative -- not a significance test. With a handful of runs the honest question is only "is this bigger than what the same configuration already varied by?", and a difference that fails it has no business being reported. A SINGLE run separates nothing, which is the correct answer rather than a missing feature: a first-ever per-type table cannot carry a comparative claim. The zero-spread case is covered on purpose, because it is what a deterministic extractor looks like and therefore the whole prize of the gpt-4o finding: a zero band means one build per arm suffices instead of three. 307 -> 313 LongMemEval tests green, Release 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
PLAN 5.6 + 5.7.
ABLATION compares a run with a capability on against the same run with it off,
PER QUESTION rather than in aggregate. An aggregate hides the two things worth
knowing: which questions a capability rescued, and whether it broke any that
previously worked. A capability that gains three and loses three reports as
neutral and is not -- it is churn, and churn is the signature of a change moving
noise rather than adding signal. The renderer calls churn out by name.
A NEGATIVE result is representable and printed, not clamped. Finding a component
that does not earn its tokens in-house is the whole point; the alternative is a
competitor finding it.
A question judged in only ONE arm is excluded from the comparison, because
including it would let an arm that simply answered fewer questions look like a
capability effect -- indistinguishable from a real gain in the aggregate.
THE REPORT enforces the integrity rules in code rather than leaving them to
whoever writes the surrounding prose, because the failure mode is a number that
travels without its context:
- every figure carries its question count AND what one question is worth
(episodic is 6 questions -- 16.7 accuracy points each)
- a figure with no measured band is labelled as having none, never quoted bare
- an ablation inside the noise floor prints as "no result", not a small win
- types the dataset CANNOT reach are stated, so a missing row never reads as a
zero -- procedural is unreachable here at any sample size
- the mapping revision is printed, because the grouping is an opinion
Two of my own test assertions were convoluted enough to be misleading and were
rewritten to say what they mean. The direct version then FAILED -- correctly:
"not measured" also appears in legitimate closing prose, so the assertion now
targets the band cell's actual marker. The test was wrong, not the code.
326 LongMemEval tests green, Release 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
PLAN 7.1-7.4. A trace and a procedure are the same record read two ways: an
episode says what happened ONCE and is retrieved by when; a procedure says what
to do NEXT TIME and is retrieved by similarity of the TASK. Different retrieval
key, so different kind -- not one kind named twice.
TraceKind {Episode, Procedure}, Episode by default
trace_kind on the node + trace_kind_idx + migration 0011
proceduresOnly filter on task-similarity search, null by default
RETENTION EXEMPTION for promoted procedures
Named trace_kind and NOT kind: `kind` already means "audit-node discriminator"
here AND upstream, and overloading a property whose meaning is shared with
another implementation is the changed-semantics hazard schema parity exists to
catch. A new PROPERTY is ungated by that verifier; a label or edge is not -- which
is why promotion is a property rather than a :Procedure label, strictly more risk
for zero more function.
THE LOAD-BEARING PART is the prune exemption. PruneSessionTraces orders by
started_at with age as its ONLY criterion and fires on every trace creation once
MaxTracesPerSession is set, so without it promotion is undone by recency and the
capability does not exist at all.
NULL-safety in both directions, and both are deliberate:
- the prune says "is NOT a procedure" via coalesce, so a trace written before
trace_kind existed is still PRUNABLE. A NULL-unsafe form would exempt every
legacy trace, quietly turning a bounded store into an unbounded one -- a
retention cap that silently stops capping.
- the search coalesces too, so an episode filter still sees legacy traces rather
than silently emptying a corpus that is entirely legacy, which is exactly what
a pre-existing store is.
TCK guard: the filter defaults to null so /get_similar_traces emits byte-identical
Cypher. A non-null default would break Gold 18/18 immediately -- by filtering a
corpus holding no promoted traces at all, i.e. to zero.
Threaded through as a default interface method, and the stored value is a
lowercase STRING rather than an enum ordinal: an ordinal would silently re-point
every existing node at a different meaning if the enum were ever reordered.
Four guards fired and were updated deliberately, not silenced: Cypher snapshot
(regenerated, BOM stripped per the documented gotcha), query-constant count,
bootstrap statement count, and property-index count -- the last now asserts
trace_kind_idx BY NAME as well as by count, so a future edit cannot swap one index
for another and keep the total looking right.
4,066 unit tests green, Release 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
…th episodes Two properties decide whether procedural memory exists at all: a promoted trace can be retrieved AS a procedure, and it survives retention. Both live entirely in Cypher, so both are asserted against a live database -- a substituted repository would only prove the arguments were passed, which was never in doubt. The first run found a real bug. The owner-starvation rescue drops to a scoped similarity scan when the indexed pass and its widening both return nothing, and that scan preserved the success filter but NOT proceduresOnly. So "find me a procedure for this task" fell through to a scan that happily returned an ordinary episode. That is not an edge case: a corpus with no promoted procedures makes the indexed pass return zero by construction, which is exactly the condition that triggers the rescue -- so the wrong answer was the common path, and no caller could have detected it. The filter is now carried through, null-safe, for the same reason the success filter always was: a search that legitimately matches nothing must return nothing rather than be rescued into the wrong answer. Also covered: legacy traces with no stored trace_kind read as episodes on both sides of the filter (the pre-migration corpus must not vanish), procedures do not consume the retention budget (exempting from deletion while still counting toward maxToKeep would evict live episodes to make room for traces never at risk), and the exemption clause did not weaken owner confinement -- it was added to the WHERE of a destructive session-keyed delete, which is precisely where a guard gets loosened by accident. 4066 unit + 8 new integration green. The 29 failing Nams.* integration tests are the deprovisioned external workspace, unrelated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
8.1/8.2. Trust was stamped once per extraction request and applied to the whole batch. That is invisible while AssistantContentMode.Ignore ships -- nothing assistant-derived is extracted at all -- and becomes a defect the instant it is switched on, because every claim the model made about the world is then stored with the same label as the ones the user typed, permanently and silently. Facts and preferences now carry SourceRole, requested from the model only when assistant content is actually being extracted. At defaults the prompt is byte-identical, SourceRole is null everywhere, and refinement is the identity. Only "assistant" is interpreted: MemoryTrustLevel is ordered so >= means "at least this trusted" and the default request trust is Untrusted, so mapping "user" or "tool" would RAISE trust on hosts that never asked for any, on the strength of a label the model wrote about itself -- and admission bypass and the system-role gate both compare with >=. Composition is max, so a host's own ApplicationTrusted declaration is never demoted by a self-report. The falsifier asserts the distribution over a mixed batch, not the mapping function -- a correct mapping that is never reached would satisfy the latter -- and it found a second defect on the way: the multi-session rung asked for valid_from/valid_until whenever TemporalValidityMode.Extract was set, then dropped both when projecting. The setting was inert under batched extraction and the prompt-level conformance test passed the whole time. Fixed, with the rule now asserted at the layer where it broke (verified red first). 8.3a. The episodic decision was not decidable as written: episodic is 6 of 50 questions, so one item is 16.7 points, while two runs of an identical config measured 25 points apart on 50. --memory-types samples the type instead of slicing a mixed run. IncludeQuestionTypes had shipped in AgentEval 0.20 with no caller -- a dead option. It is validated before any container starts, derived from the same embedded taxonomy the reports use, stamped into the fingerprint with the map revision, and refuses "procedural" outright rather than widening back to the full sample. 4087 unit + 338 LongMemEval green, Release 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
EXTRACTED_FROM is written per ingestion batch -- every item linked to every message the call saw. On the evaluation corpus a fact links to a mean of 12 source messages and as many as 30. The problem is not imprecision: any attribution metric derived from that edge is satisfied by construction. Ask "is the true source among this fact's linked messages?" and the answer is yes across thirty of them, so the metric cannot fail and therefore measures nothing. ExtractionProvenanceMode.PerItem numbers the transcript turns, asks which single turn states each fact and preference, and resolves that to one edge. Opt-in and default Batch, because it changes the rendered conversation as well as the instruction and prompt bytes are fingerprinted into every measured run. Facts and preferences only, deliberately. A fact asserts one claim made in one statement; an Entity node is a merged identity that legitimately appears across many turns, so narrowing it would be wrong rather than precise. An unusable turn -- absent, zero, negative, past the end -- keeps the batch links. Clamping to the nearest valid index is the tempting bug: it always produces an answer and the answer is fabricated. A resolved turn REPLACES the batch links, so a guessed number does not add noise, it discards the true source and substitutes a wrong one, and afterwards that is indistinguishable from precise attribution. Edges are written from the input item, never the merged repository result. A MERGE returns the stored node, whose source ids are the union accumulated over earlier ingestions; writing edges from that would re-link the fact to messages it was not extracted from now, restoring exactly the breadth this removes while the stored property still looked correct. The test's fact repository returns such a node specifically to catch that. The batch rung numbers turns WITHIN each source session, restarting at 1: results are demultiplexed per session, and each session's own message ids are what a turn indexes into. All three rungs carry the instruction, asserted at both the prompt and the projection layer. 4106 unit + 338 LongMemEval + 318 non-NAMS integration green, Release 0 warnings. Enum guard and docs/architecture.md updated deliberately for the new enum. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
…l call
The write path appends. "I live in Zurich" three months after "I live in Basel"
leaves both live, both retrievable and both equally confident, and recall is then
asked to choose between two assertions with nothing to choose on.
The field's answer is a model call classifying each candidate write as
ADD/UPDATE/DELETE/NOOP. Three of those four are decidable from the store: the
exact triple already MERGEs, so NOOP is free; a new object for a functional
relation is UPDATE; everything else is ADD. Deciding them from the store costs
zero completions -- stronger than the "fold it into the unified call" design --
cannot vary between runs, and can be falsified in a unit test rather than
measured against a provider. DELETE genuinely needs the conversation ("forget I
ever lived in Basel" is not derivable from a triple) and is left out rather than
approximated, because it is the one direction that loses data.
The entire safety property is cardinality. Superseding on any repeated
subject+predicate would close "likes coffee" the moment "likes tea" arrived: a
true fact gone from live recall, with the graph still looking correct. So
functional relations are declared per-relation in relation-vocabulary.json, with
the argument for each recorded beside it, and everything undeclared -- every
event relation, most state relations, and every predicate the extractor invents
-- is multi-valued. Six relations qualify. The annotation is not part of the
vocabulary fingerprint, which hashes canonical names against surface forms, so
every sealed base stays comparable.
Candidate selection is Cypher, including the liveness filter, because
invalidated_at is not on the domain record: an in-memory filter cannot tell a
closed fact from a live one, and re-selecting closed facts would add a
:SUPERSEDED_BY edge per subsequent arrival and fan a chain into a star.
Off by default -- it changes what live recall returns and every recorded
measurement was taken append-only. Owner-confined, since supersession closes a
fact and reaching across owners would close one belonging to somebody who was not
in the conversation. Best-effort after the write: losing a supersession costs
precision, failing the ingestion would lose the memory. A store that has not
implemented the new repository method simply keeps appending.
The falsifier covers both halves, because each is trivially satisfiable alone.
4118 unit + 338 LongMemEval + 326 non-NAMS integration green, Release 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
Per-section diagnostics now say why a section came back thin. Everything
downstream -- calibration, "I don't know", any confidence surface -- assumes
those numbers mean something: that a low top score indicates the answer was not
retrievable. Nothing has ever tested that, and if the scores are noise then all
of it is decoration.
AUC is the right summary because it needs no threshold. "Does topScore < 0.7
predict absence?" answers a question about 0.7; AUC asks whether the signal
orders answerable above unanswerable at all, over every threshold at once. 0.5
is the kill line.
Which makes this instrument's own correctness unusually load-bearing -- a
flattering implementation keeps a dead track alive for a quarter -- so it is
built against the ways an AUC goes wrong while looking fine:
- Mann-Whitney over mid-ranks, not trapezoids over a swept threshold. Ties are
the failure mode: a signal returning the same value everywhere is one large tie
block, and a curve-based version scores those as if ordered, reporting ~0.8 for
a coin. The rank form gives ties their 0.5 and reports the coin flip.
- One empty class returns null, never 0.5. A run where every answer was present
never put the question; 0.5 is the kill line, so that default would kill a
track on absent evidence.
- Inverted separation reports 0.0, not max(auc, 1-auc). An AUC of 0 is a wiring
bug -- a perfect predictor read backwards -- and the tempting symmetric "fix"
hides exactly that.
- Questions with no collected signal or an uncheckable answer are excluded and
COUNTED. The denominator decides whether the number means anything, and 0.83
over 47 present and 3 absent is three questions' worth of evidence.
- The 0.6 justification line is in code, fixed before any number was seen.
The signal itself is the strongest similarity any searched section achieved, max
across tiers rather than facts alone, with a searched-but-empty section
contributing its threshold floor -- that is evidence ("we looked at this
threshold and found nothing"), and dropping it would measure the AUC only over
questions where retrieval already succeeded. Null when diagnostics were off,
never 0.
Emitted on every run because it costs nothing: no extra call, no rebuild, it
reads two fields already recorded. A number that appears only when someone
remembers to ask for it never gets asked for.
Not yet run: 4.1's diagnostics postdate every frozen base, so the AUC needs one
fresh run to exist. Phase 10 stays unstarted -- its gate is instrumented, not
fired.
4118 unit + 348 LongMemEval green, Release 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
They shipped with no host, so the only people who could try them were people who would have written one anyway. tools/AgentMemory.McpHost is a DotnetTool package -- `dotnet tool install -g AgentMemory.McpHost`, then `agent-memory-mcp` -- speaking stdio or HTTP, configured entirely by environment, with a Dockerfile and a compose file that stand up Neo4j and the server together from nothing. --read-only removes every write tool from the server's tool list rather than refusing them when called. A tool a client can see is a tool a model will try, and an error return teaches it nothing about what the server is for. The classification enumerates the WRITES, not the reads. Listing the reads would be the shorter list -- 9 against 16 -- and it is the wrong one: a tool added later and forgotten would default to "not a write" and stay exposed. Enumerating writes makes the same forgetfulness fail the other way, and a guard test reflects over the tool attributes and fails if any tool escapes classification entirely. graph_query is withheld even with EnableGraphQuery set: it takes arbitrary Cypher, and a mode whose whole value is "this cannot change anything" must not rest on a second component's parser. The registration filter was a silent no-op in its first version -- it read ImplementationInstance, but the SDK registers tools as singleton factories, so every name came back null and nothing was ever removed. The test that asserts the filter on a real registration, rather than on the classification lists, is what caught it; the lists were perfect the whole time. Reading the option also had to avoid resolving IOptions, which would have run ValidateOnStart's validators during registration and moved an unrelated DefaultConfidence error from host startup to AddAgentMemoryMcpTools. An existing test caught that one. Bootstrap-on-start is the other load-bearing piece: a missing vector index returns no rows rather than an error, so a server started without the schema looks healthy and answers nothing. It exits non-zero if it cannot bootstrap, because an operator who asked for it and did not get it must not be left with a server that looks fine. Also: an unknown flag is an error rather than being ignored, since a typo in --read-only would otherwise start a fully writable server the operator believes is read-only; --read-only is the UNION of flag and environment rather than an override, because for a safety switch neither absence should cancel the other; NEO4J_PASSWORD has no default, because a blank password becomes an auth failure at the first query, which from an MCP client is indistinguishable from an empty database. 4157 unit + 348 LongMemEval + 54 SK green, Release 0 warnings. Packs as a valid tool package and is listed in eng/release-packages.txt. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
The task's stated defect was already fixed. "SchemaBootstrapper only polls VECTOR indexes for ONLINE, so a failed range index degrades quietly" was true when it was written and is not now: both the bootstrapper and schema-check check every index for FAILED, scope it to the ones we create, and treat optimization-only failures as warnings rather than startup errors. The gap that remains is the state neither of them could describe, and the code dismisses it in a comment: "POPULATING is a normal transient state". It is transient, and it is also the state most likely to be mistaken for a correctness problem. A vector search against a still-building index SUCCEEDS and returns a subset of the corpus -- no error, no warning -- so recall is quietly partial and the symptom is "memory seems to have forgotten things". That is the same failure signature as a missing index, and an operator can lose an afternoon to it. schema-check now reports POPULATING indexes with their build percentage and says what it means. Exit stays 0: it is legitimate right after bootstrap, and failing on it would make the command unusable at exactly the moment it is most likely to be run. The "fragmentation degrades vector search 40-60% silently" claim behind the original task is deliberately not adopted. It is an unsourced number about a different engine, nothing here measures it, and acting on it would mean building a remedy for a problem we cannot observe. The query is a const, so the Cypher snapshot was regenerated (BOM stripped). Two CLI tests stubbed the transaction runner by return type and needed updating for the new shape -- done deliberately, with the reason recorded, not by widening the stub. 4158 unit tests green, Release 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
There was a problem hiding this comment.
Pull request overview
This PR advances AgentMemory’s evaluation harness and runtime capabilities across multiple phases by adding a runnable MCP host distribution, strengthening extraction provenance/trust semantics, introducing procedural-memory support in Neo4j, and adding deterministic write-time fact supersession and sufficiency/AUC instrumentation for abstention work.
Changes:
- Add a turnkey
AgentMemory.McpHost.NET global tool (and container stack) to run the MCP server over stdio/HTTP, including a true read-only mode that removes write tools from the advertised tool list. - Extend extraction to support per-message trust (
source_role) and per-item provenance (source_turn) end-to-end (prompt → projection → persistence), plus add typed LongMemEval sampling/reporting and a rank-based sufficiency AUC metric. - Implement procedural trace promotion primitives (trace kind, indexing, retention exemption) and write-time fact supersession for declared single-valued relations, backed by unit + integration falsifiers.
Reviewed changes
Copilot reviewed 76 out of 76 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/AgentMemory.McpHost/README.md | Documents installing/running the new MCP host tool, read-only behavior, and Docker usage. |
| tools/AgentMemory.McpHost/Program.cs | Entrypoint delegating to a testable host runner. |
| tools/AgentMemory.McpHost/McpHostProgram.cs | Implements host wiring (Neo4j, embeddings, MCP transports) and schema bootstrap behavior. |
| tools/AgentMemory.McpHost/McpHostOptions.cs | Adds env-first option parsing, safety checks, and help text for the MCP host tool. |
| tools/AgentMemory.McpHost/Dockerfile | Builds/publishes the MCP host into an ASP.NET runtime container image. |
| tools/AgentMemory.McpHost/docker-compose.yml | Provides a runnable Neo4j + MCP host stack with healthcheck ordering. |
| tools/AgentMemory.McpHost/AgentMemory.McpHost.csproj | Defines the MCP host as a packable .NET global tool with required deps. |
| tools/AgentMemory.LongMemEval/Program.cs | Wires typed sampling and emits sufficiency AUC into run output. |
| tools/AgentMemory.LongMemEval/LongMemEvalTypedReport.cs | Renders per-type accuracy tables with required caveats/bands/ablations. |
| tools/AgentMemory.LongMemEval/LongMemEvalTypedNoiseFloor.cs | Computes per-type repeat-variance “band” to prevent overclaiming. |
| tools/AgentMemory.LongMemEval/LongMemEvalSufficiencyReport.cs | Projects telemetry into an auditable sufficiency AUC report payload. |
| tools/AgentMemory.LongMemEval/LongMemEvalSufficiencyAuc.cs | Implements rank-based AUC computation and diagnostics for sufficiency signals. |
| tools/AgentMemory.LongMemEval/LongMemEvalMemoryTypeSelection.cs | Converts requested “memory types” into dataset task-label filters with refusals. |
| tools/AgentMemory.LongMemEval/LongMemEvalMemoryTypeMap.cs | Exposes taxonomy (task label → types) plus known type inventory and revisioning. |
| tools/AgentMemory.LongMemEval/LongMemEvalAblation.cs | Adds per-question ablation comparison utilities and noise-floor gating. |
| tools/AgentMemory.LongMemEval/AgentMemoryLongMemEvalAdapter.cs | Records sufficiency signals into question telemetry from retrieval diagnostics. |
| tools/AgentMemory.Cli/Commands/MemoryCommands.cs | Enhances schema-check to report POPULATING indexes and population percent. |
| tests/AgentMemory.Tests.Unit/Queries/ProcedureTraceQueryTests.cs | Unit guards for procedure filtering and retention exemption Cypher safety. |
| tests/AgentMemory.Tests.Unit/Queries/CypherQuerySnapshotTests.cs | Updates expected Cypher query inventory count for new schema/query changes. |
| tests/AgentMemory.Tests.Unit/Queries/CypherQuerySnapshot.snap | Updates cypher snapshot for trace_kind write, index state query, and new index. |
| tests/AgentMemory.Tests.Unit/McpServer/ReadOnlyModeTests.cs | Verifies no tool escapes read/write classification and DI filtering works. |
| tests/AgentMemory.Tests.Unit/McpServer/McpHostOptionsTests.cs | Unit tests for MCP host option parsing defaults/precedence/refusals. |
| tests/AgentMemory.Tests.Unit/Infrastructure/SchemaBootstrapperTests.cs | Updates bootstrap statement counts and asserts the new trace_kind_idx. |
| tests/AgentMemory.Tests.Unit/Infrastructure/AbstractionsContractGuardTests.cs | Updates documented enum count guard to reflect newly added enums. |
| tests/AgentMemory.Tests.Unit/Extraction/WriteTimeSupersessionTests.cs | Unit falsifier for write-time supersession gating and safety invariants. |
| tests/AgentMemory.Tests.Unit/Extraction/PerMessageTrustTests.cs | Unit falsifier ensuring mixed batches persist distinct trust levels per item. |
| tests/AgentMemory.Tests.Unit/Extraction/ExtractorProjectionConformanceTests.cs | Ensures requested fields survive projection across all extractors/rungs. |
| tests/AgentMemory.Tests.Unit/Cli/CliCommandsTests.cs | Updates CLI tests for new IndexState shape and POPULATING reporting. |
| tests/AgentMemory.Tests.Unit/AgentMemory.Tests.Unit.csproj | Adds reference to McpHost project so option parsing can be unit-tested. |
| tests/AgentMemory.Tests.Unit.LongMemEval/TypedReportTests.cs | Tests typed report caveats (band, per-question points, unreachable types). |
| tests/AgentMemory.Tests.Unit.LongMemEval/TypedNoiseFloorTests.cs | Tests typed noise floor computation and separation behavior. |
| tests/AgentMemory.Tests.Unit.LongMemEval/SufficiencyAucTests.cs | Tests AUC correctness for ties, inversion, empty classes, and invariants. |
| tests/AgentMemory.Tests.Unit.LongMemEval/MemoryTypeSelectionTests.cs | Tests typed sampling selection and refusal semantics. |
| tests/AgentMemory.Tests.Unit.LongMemEval/AblationTests.cs | Tests ablation flip accounting and noise-floor survival logic. |
| tests/AgentMemory.Tests.Integration/Repositories/SupersededCandidateIntegrationTests.cs | Integration tests for Cypher-based supersession candidate selection. |
| tests/AgentMemory.Tests.Integration/Repositories/ProcedurePromotionIntegrationTests.cs | Integration falsifiers for procedural recall and retention survival. |
| src/AgentMemory.Neo4j/Schema/Migrations/0011_trace_kind.cypher | Adds migration for trace_kind_idx to support procedural trace filtering. |
| src/AgentMemory.Neo4j/Repositories/Neo4jReasoningTraceRepository.cs | Adds procedures-only filtering, preserves it in fallbacks, and maps trace_kind. |
| src/AgentMemory.Neo4j/Repositories/Neo4jFactRepository.cs | Implements store-side superseded-candidate selection API. |
| src/AgentMemory.Neo4j/Queries/SchemaQueries.cs | Adds TraceKindIndex and extends index-state query to include population percent. |
| src/AgentMemory.Neo4j/Queries/ReasoningQueries.cs | Adds trace_kind write, filter support, and prune exemption for procedures. |
| src/AgentMemory.Neo4j/Queries/FactQueries.cs | Adds Cypher for selecting live superseded candidate facts. |
| src/AgentMemory.McpServer/ServiceCollectionExtensions.cs | Applies DI-level read-only filtering to remove write tools from registration. |
| src/AgentMemory.McpServer/McpToolAccess.cs | Centralizes read/write tool classification with fail-closed semantics. |
| src/AgentMemory.McpServer/AgentMemoryMcpOptions.cs | Adds read-only option documentation and semantics at the MCP server layer. |
| src/AgentMemory.Extraction.Llm/LlmUnifiedMemoryExtractor.cs | Adds provenance mode wiring and projects source role/turn into domain output. |
| src/AgentMemory.Extraction.Llm/LlmPreferenceExtractor.cs | Numbers transcript in per-item mode and projects role/turn fields. |
| src/AgentMemory.Extraction.Llm/LlmMultiSessionUnifiedMemoryExtractor.cs | Wires provenance instruction, numbered batch transcript, and projection of new fields. |
| src/AgentMemory.Extraction.Llm/LlmFactExtractor.cs | Numbers transcript in per-item mode and extends prompt/projection to include role/turn. |
| src/AgentMemory.Extraction.Llm/LlmExtractionOptions.cs | Adds Provenance option with default Batch to preserve existing prompt bytes. |
| src/AgentMemory.Extraction.Llm/Internal/LlmResponseModels.cs | Adds DTO fields for source_role and source_turn. |
| src/AgentMemory.Extraction.Llm/ExtractionPromptSemantics.cs | Adds source_role and provenance (source_turn) prompt semantics. |
| src/AgentMemory.Core/Memory/RelationVocabularyDocument.cs | Adds cardinality metadata to the vocabulary entry model. |
| src/AgentMemory.Core/Memory/relation-vocabulary.json | Declares single-valued relations (functional cardinality) with rationale. |
| src/AgentMemory.Core/Memory/MemoryRelationCardinality.cs | Loads single-valued predicate set from vocabulary and resolves surface forms. |
| src/AgentMemory.Core/Extraction/WriteTimeFactResolution.cs | Adds store-free gate for whether a fact is eligible to supersede. |
| src/AgentMemory.Core/Extraction/SourceTurnProvenance.cs | Resolves source_turn to message ids with safe fallback to batch links. |
| src/AgentMemory.Core/Extraction/SourceRoleTrust.cs | Refines per-item trust based on source_role while preserving monotonicity. |
| src/AgentMemory.Core/Extraction/PersistenceStage.cs | Applies per-item trust/provenance and performs best-effort write-time supersession. |
| src/AgentMemory.Core/Extraction/ConversationTextBuilder.cs | Adds numbered transcript rendering for per-item provenance extraction. |
| src/AgentMemory.Abstractions/Repositories/IReasoningTraceRepository.cs | Adds a default-interface overload for procedures-only trace vector search. |
| src/AgentMemory.Abstractions/Repositories/IFactRepository.cs | Adds a default-interface method for superseded-candidate selection (safe default empty). |
| src/AgentMemory.Abstractions/Options/ExtractionProvenanceMode.cs | Introduces per-item vs batch provenance mode enum. |
| src/AgentMemory.Abstractions/Options/ExtractionOptions.cs | Adds SupersedeReplacedFacts option (off by default) for write-time supersession. |
| src/AgentMemory.Abstractions/Domain/Reasoning/TraceKind.cs | Introduces TraceKind enum (Episode/Procedure) with schema rationale. |
| src/AgentMemory.Abstractions/Domain/Reasoning/ReasoningTrace.cs | Adds Kind to the reasoning trace domain model. |
| src/AgentMemory.Abstractions/Domain/Extraction/ExtractedPreference.cs | Adds source role/turn provenance fields to extracted preferences. |
| src/AgentMemory.Abstractions/Domain/Extraction/ExtractedFact.cs | Adds source role/turn provenance fields to extracted facts. |
| eng/release-packages.txt | Adds McpHost tool project to release package list. |
| docs/architecture.md | Updates architecture doc to mention new enums (but count needs correction). |
| Directory.Build.props | Excludes McpHost from multi-targeting rule (tool stays single-targeted). |
| CHANGELOG.md | Documents procedural trace promotion and retention behavior changes. |
| AgentMemory.slnx | Adds McpHost tool project to the solution structure. |
| .github/workflows/release.yml | Ensures McpHost tool project is built in release workflow. |
| .github/workflows/ci.yml | Ensures McpHost tool project is built in CI workflow. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+69
to
+71
| var distinct = observations.Select(o => o.Signal).Distinct().Count(); | ||
| if (distinct < observations.Count) ties = observations.Count - distinct; | ||
|
|
| | **Dependencies** | **Microsoft.Extensions.AI.Abstractions** 10.8.0 (approved, D-AR2-1) — .NET BCL otherwise (multi-targets net8.0/net9.0/net10.0) | | ||
| | **MUST NOT reference** | Neo4j.Driver, Microsoft.Agents.*, any GraphRAG SDK, any MCP SDK, any NuGet package **except** Microsoft.Extensions.AI.Abstractions | | ||
| | **Key types** | 52 domain records (Conversation, Message, Entity, Fact, Preference, Relationship, MemoryHistoryQuery, MemoryHistoryRecord, ReasoningTrace, ReasoningStep, ToolCall, ToolCallStats, IngestionItemOutcome, MemoryContextRankedItem, MemoryContextSectionDiagnostics, UnifiedExtractionResult, etc.), 41 service interfaces (incl. `IMemoryIsolationPolicy`, `IUnifiedMemoryExtractor`, and `IMultiSessionUnifiedMemoryExtractor`), 11 repository interfaces, 16 configuration types (incl. `MemoryRankingOptions`, `MemoryIsolationOptions`), 26 enums (incl. `MemoryProfile`, `RankingIntent`, `DuplicateStatus`, `EntityMatchType`, `MemoryNodeKind`, `MemoryOperationAccess`, `MemoryIsolationMode`, `IngestionStatus`, `IngestionStage`, `IngestionItemStatus`, `MemoryItemKind`, `IngestionFailureMode`, `MemoryTrustLevel`, `AssistantContentMode`, `TemporalValidityMode`) | | ||
| | **Key types** | 52 domain records (Conversation, Message, Entity, Fact, Preference, Relationship, MemoryHistoryQuery, MemoryHistoryRecord, ReasoningTrace, ReasoningStep, ToolCall, ToolCallStats, IngestionItemOutcome, MemoryContextRankedItem, MemoryContextSectionDiagnostics, UnifiedExtractionResult, etc.), 41 service interfaces (incl. `IMemoryIsolationPolicy`, `IUnifiedMemoryExtractor`, and `IMultiSessionUnifiedMemoryExtractor`), 11 repository interfaces, 16 configuration types (incl. `MemoryRankingOptions`, `MemoryIsolationOptions`), 26 enums (incl. `MemoryProfile`, `RankingIntent`, `DuplicateStatus`, `EntityMatchType`, `MemoryNodeKind`, `MemoryOperationAccess`, `MemoryIsolationMode`, `IngestionStatus`, `IngestionStage`, `IngestionItemStatus`, `MemoryItemKind`, `IngestionFailureMode`, `MemoryTrustLevel`, `AssistantContentMode`, `TemporalValidityMode`, `TraceKind`, `ExtractionProvenanceMode`) | |
Comment on lines
+66
to
+68
| foreach (var argument in args.Where(a => a.StartsWith("--", StringComparison.Ordinal))) | ||
| if (!Known.Contains(argument, StringComparer.Ordinal)) | ||
| throw new ArgumentException($"unknown option '{argument}'."); |
The suite failed roughly 1 run in 4 on FactVectorYieldTelemetryTests.NothingIsMeasuredWhenNoListenerWantsTheData. That test asserts zero overhead when nothing is listening -- with sampling declined the Activity is never created and the search behaves exactly as it does with no diagnostics at all. ActivityListener is process-global and sampling is a UNION across listeners, so one listener sampling everything from the source forces creation of every AgentMemory span in every test class running concurrently. Two listeners in Neo4jTransactionRunnerTests did exactly that. The telemetry test's own harness already carries a comment describing this hazard and scoping itself to avoid being that neighbour; these two were the neighbour. Both are now scoped by span name. The query-span listener needs BOTH memory.db.query and memory.db.tx: the query-span wrapper is only installed when the enclosing tx activity exists, so scoping to the asserted name alone yielded zero query spans -- which the first attempt did, and six runs caught. Also hardened ReadOnlyRequested: it invokes registered IConfigureOptions factories to read the setting without running the options validators, and was passing null as the provider. A configuration source bound to services that only exist in the real provider would have NRE'd during registration for every host. It now passes an empty provider and skips a source it cannot evaluate, since read-only might not be seen from that source but the alternative is a host that cannot start at all. 4158 unit tests green across eight consecutive runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
This was referenced Aug 27, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ten items across phases 4, 7, 8, 9, 11 and 12, each with its own falsifier, each committed separately.
What shipped
7.5 — procedural falsifiers. Two properties decide whether procedural memory exists: a promoted trace can be retrieved as a procedure, and it survives retention. Both live entirely in Cypher, so both are asserted against a live database. The first run found a real bug: the owner-starvation rescue fallback dropped
proceduresOnly, so a procedure lookup that found nothing in the index was rescued into returning an episode. Not an edge case — a corpus with no promoted procedures makes the indexed pass return zero by construction, so the wrong answer was the common path.8.1 / 8.2 — per-message trust. Trust was stamped once per extraction request. Invisible while
AssistantContentMode.Ignoreships; a defect the instant it is switched on, because every claim the model made is then stored with the same label as the ones the user typed. Facts and preferences now carrySourceRole, requested only when assistant content is actually extracted. Onlyassistantis interpreted —MemoryTrustLevelis ordered so>=means "at least this trusted" and the default isUntrusted, so mappinguser/toolwould raise trust on hosts that never asked for any, and admission bypass compares with>=. The falsifier found a second defect: the multi-session rung asked forvalid_from/valid_untiland then dropped both when projecting, soTemporalValidityModewas silently inert under batched extraction while the prompt-level conformance test passed throughout.8.3a — typed sampling. The episodic decision was not decidable as written: episodic is 6 of 50 questions, so one item is 16.7 points, while two runs of an identical config measured 25 points apart on 50.
--memory-typessamples the type instead of slicing a mixed run.IncludeQuestionTypeshad shipped in AgentEval 0.20 with no caller — a dead option. Refusesproceduraloutright rather than widening back to the full sample.9.3 — per-item provenance.
EXTRACTED_FROMis batch-level; a fact links to a mean of 12 source messages. The problem is not imprecision — any attribution metric derived from that edge is satisfied by construction.ExtractionProvenanceMode.PerItemnumbers the transcript and resolves one edge. An unusable turn keeps the batch links: clamping to the nearest valid index is the tempting bug, because it always produces an answer and the answer is fabricated.9.1 / 9.2 — write-time resolution, with no model call. Three of ADD/UPDATE/DELETE/NOOP are decidable from the store. Deciding them there costs zero completions — stronger than the "fold it into the unified call" design — and is falsifiable in a unit test. DELETE is left out rather than approximated. The whole safety property is cardinality: superseding on any repeated subject+predicate would close "likes coffee" when "likes tea" arrives, so functional relations are declared per-relation in the vocabulary artifact and everything undeclared is multi-valued.
4.2 — the AUC that decides whether the abstention track lives. Mann–Whitney over mid-ranks, not trapezoids over a swept threshold: a constant signal is one large tie block, and a curve-based version scores those as if ordered, reporting ~0.8 for a coin. One empty class returns null, never 0.5 — 0.5 is the kill line. The 0.6 justification threshold is in code, fixed before any number was seen. Not yet run: the diagnostics postdate every frozen base, so the number needs one fresh run.
11.1 — turnkey MCP distribution. The 25 tools shipped with no way to run them without writing a .NET host first.
AgentMemory.McpHostis aDotnetTool— stdio or HTTP, env-var config, schema bootstrap on start, Dockerfile and compose.--read-onlyremoves write tools rather than refusing them. The filter was a silent no-op in its first version (it readImplementationInstance; the SDK registers factories) — caught by the test that asserts the filter on a real registration rather than on the classification lists.12.2 — index hygiene. The stated defect was already fixed. The real gap was the state nothing could describe: a vector search against a POPULATING index succeeds and returns a subset of the corpus, which reads as memory having forgotten things.
Deliberately not done
Verification
4,158 unit + 348 LongMemEval + 54 SK + 326 non-NAMS integration, Release 0 warnings. The 29 failing
Nams.*integration tests are the deprovisioned external workspace, unrelated to this branch.Guards updated deliberately, never silenced: enum count (+
ExtractionProvenanceMode), Cypher snapshot (BOM stripped), and two CLI tests whose runner stubs matched by return type.🤖 Generated with Claude Code
https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE