Phase 10 complete, Phase 2 tail, and the AUC that opened the gate - #184
Merged
Conversation
The preflight pins the stratified ten to a recorded plan -- exactly 474 source sessions -- so a change in sampling or batching cannot pass unnoticed. That guard fired on the first typed preflight, and correctly: --memory-types episodic selects different questions, so it planned 473. A typed sample is a different plan by construction, so the canonical number does not describe it. The guard is now skipped for typed samples and says so on stdout, rather than being adjusted or silently dropped: a guard that exists to catch sampling drift is worth more intact than widened. Verified both directions with zero provider calls: canonical stratified 10 -> 121 calls, 474 sessions (guard passes) episodic 10 -> 122 calls, 473 sessions (guard skipped, announced) episodic 50 -> 624 calls, 2418 sessions, 26.1M est. input tokens That last line is the price of one 8.3b arm-build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
There was a problem hiding this comment.
Pull request overview
This PR scopes the “canonical fixed-ten” preflight guard to the untyped (default/stratified) sample so typed samples (via --memory-types ...) don’t fail the canonical source-session expectation and instead clearly announce that the guard is not applicable.
Changes:
- Introduces a
canonicalPlanpredicate that includesoptions.MemoryTypes.Count == 0before enforcing the fixed expected source-session count. - Adds a stdout message when a typed sample is used to explain why the canonical guard is skipped.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+321
to
+325
| // The canonical guard pins the STRATIFIED ten to a recorded plan, so a change in | ||
| // sampling or batching cannot pass unnoticed. A typed sample is a different plan by | ||
| // construction -- --memory-types episodic selects different questions, with a | ||
| // different number of source sessions -- so the canonical number does not describe | ||
| // it. Skipped rather than adjusted, and said out loud: silently dropping a guard that |
…s exposed Three fixes, all found by running the pipeline against real data rather than by reading it. 1. The dataset is untracked everywhere. It is gitignored in this repo AND in AgentEval, tracked by neither, and exists on exactly one disk -- its path had only ever lived in shell history, which is how a session went hunting for it. The sha256 is now pinned in git (64 characters, versioned forever) and checked just before a 7-9 hour build: the last cheap moment to learn that a recovered or re-downloaded file is not the one every sealed corpus used. Warns rather than fails, because evaluating a different variant is legitimate work and the actual sha already travels in the fingerprint -- but it can never pass silently. LONGMEMEVAL_DATASET resolves the path, with the AgentEval checkout as fallback. Note for whoever keeps a backup copy: a gitignored copy INSIDE either repo is less safe than one outside, because `git clean -xdf` deletes ignored files. 2. Bumping the manifest schema to 6 orphaned every corpus sealed at schema 5 -- each one a 7-9 hour build -- because VerifyIntegrity demanded an exact match. That is the precise opposite of what recording ingestion identity is for. Older schemas are now read using the field set they were written under; only NEWER ones are refused, since they were written by a build that knew things this one does not. Caught by trying to reuse a real 2026-08-09 corpus, not by a test. 3. And the flaw that survived the first version of the drift check. Schema 5 carried none of the ingestion-identity fields, so deserialising one fills them from the record's DEFAULTS -- "Ignore", "Batch", false. Comparing against those reported a corpus built with Utterance as MATCHING a run configured for Ignore: unknown silently reading as agreement, which is the one thing the check exists to prevent. It survived because the two values coincided. Pre-schema-6 manifests now report every such field as unrecorded. memoryTypes is the deliberate exception: empty means "every type", and a corpus built before typed sampling existed genuinely is an all-types corpus, so empty-against-empty is a real match -- while an episodic run against that same corpus still drifts, because it never ingested those questions. 4158 unit + 368 LongMemEval green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
…against
4.2's AUC came back below 0.5 twice, which read like an inverted signal. It is
not. Two runs against real corpora say what it actually is:
10q episodic: signal spread 0.0256 across all questions (0.904..0.933),
present mean 0.919 vs absent mean 0.921, AUC decided by 14
pairwise comparisons -- one flip moves it 0.071.
50q stratified: 32 present, ONE absent. The AUC is that single question's rank,
which is why structured reported 0.969 and hybrid 0.094 from the
same corpus.
The answer-presence gate finds the answer present in 97% of questions. AUC needs
an unanswerable class to order against, and this sample barely has one, so the
metric is not noisy -- it is undefined.
LongMemEval's _abs questions ARE that class, and AbstentionPolicy /
AbstentionTargetProportion have been available since AgentEval 0.20 with no
caller. The third dead option this branch has found. Our own plan already
recorded that across 52 runs not one _abs question ever ran; this is why.
--abstention as-sampled|exclude|only|target and --abstention-proportion now reach
sampling. Default as-sampled, which is what every recorded run used: the default
is preserved rather than improved, because changing what a sample contains
changes every number computed from it. --abstention-proportion is rejected
outside (0,1), since 0 silently means exclude and 1 means only.
Abstention is ingestion-affecting, not merely evaluation-affecting: an _abs
question still carries a conversation history, so including one changes which
histories were ingested. It is therefore sealed into the manifest, part of the
fingerprint, and compared on reuse -- and a schema-5 manifest reports it as
unrecorded rather than as the "AsSampled" default, or unknown would read as
agreement through the newest field.
The canonical fixed-ten guard now keys on canonical SAMPLING rather than on the
type filter alone, enumerated so a third sampling knob fails loudly instead of
inheriting the recorded count. Verified: canonical still plans exactly 474
sessions and passes; abstention plans 481 and is announced.
4158 unit + 370 LongMemEval green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
… heuristic
Wiring abstention sampling did not fix the AUC, and the reason is the finding.
Abstention questions turned out to be perfectly checkable -- and the presence gate
reported 3 of 4 of them as PRESENT. Their topic is discussed in the conversation
even though the specific fact is not, so the gold answer's distinctive tokens are
there to find. The gate is a deliberately cheap floor for spotting extraction
failure; it was never a ground truth for "was this question answerable", and
using it as one put most of the unanswerable class on the wrong side of every
pairwise comparison.
The dataset already knows. An _abs question is unanswerable by definition, and
AgentEval surfaces IsAbstention on every entry. That label is free, exact, and
needs no gate -- so it now takes precedence, and the gate keeps its real job:
deciding answerability for ORDINARY questions, where the dataset asserts nothing
about whether our extraction happened to store the answer.
An abstention question is also no longer required to have a checkable gold
answer, since an abstention answer is often phrased in common words with no
distinctive tokens; requiring the gate to confirm a dataset fact would drop
exactly the questions that make the metric computable.
Measured, on the same corpus, free (a reuse, no extraction):
before structured AUC 0.969 / hybrid 0.094 over 32 present, 1 absent
-- one observation, so each arm reported that question's luck
after structured 0.600 / hybrid 0.600 over 5 present, 4 absent
Both arms agreeing is the signal that it is now measuring the signal rather than
a single draw. 0.600 sits exactly on the justification line fixed in code before
any number existed -- at n=9, where one flip moves it 0.05. Weakly above a coin,
and not yet worth building on.
abstentionQuestions is reported alongside the denominators, so a zero -- meaning
the unanswerable class came entirely from accidental extraction misses, as in the
50-question run -- is visible without recounting.
4158 unit + 376 LongMemEval green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
The two abstention-enriched corpora were indistinguishable from an ordinary one in the listing, which defeats the point of having a catalog to pick from: it is the difference between a corpus the sufficiency AUC can be computed on and one it cannot. Sealed in the manifest already; only the display was missing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
…more widening The vector index is global, so an owner filter is a POST-filter on a top-K drawn from every tenant: measured, a mean of 7 of 60 candidates reached the querying owner. Only a totally empty result triggered a rescue, on the argument that a short-but-non-empty result still answers the question. That argument has a measured counter-example. Question 5d3d2817 returned 2 facts from a 710-fact graph with the gold answer present at coverage 1.00, and both arms answered it wrongly. The claim is true for a small tenant and false for a crowded one, and the returned count cannot separate them. The plan asked for a ratio gate. That is the wrong shape: a fraction of the limit needs a cutoff nobody can justify, and any shortfall might be crowding. The gate is simply "short". The rescue is the SCAN, not another widening -- and this is what dissolves the original objection rather than trading against it. Widening is a second draw on the same global index, and a tenant losing to 50 neighbours at top-60 usually loses again at top-480. The scan is bounded by one owner rows, so the small tenant the "do not tax them" argument protected pays LESS for it than for a wider index query. It takes whichever result is larger, so a genuinely sparse owner never loses indexed rows to it. Off by default; it trades latency for recall on every short result and every recorded measurement was taken without it. Fact path only for now -- that is where the counter-example is -- with the other three tracked as 2.12, because "a setting only some components respect" is the exact defect this session already found twice. The scan is now one extracted helper reached by both the empty and short paths; two copies of a fallback drift. 4166 unit tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
Traces were the one recall category with no trust signal. StartTraceAsync strips any caller-supplied trust_level -- correctly, since a self-assigned ApplicationTrusted would bypass the admission policy instruction-detection for the trace own Task text on recall -- and then nothing stamped one. Every trace read back as Untrusted, making "the agent generated this" and "no signal was recorded" the same value, which is what the enum exists to prevent. ModelGenerated is the accurate label: a trace is the agent own record of what it did, not something a user asserted. Stamped from configuration, after the strip, never from the caller. Safe at shipped defaults, and asserted rather than claimed: MinimumTrustForAdmissionBypass defaults to ApplicationTrusted, which ModelGenerated does not reach, so no trace gains a bypass; MinimumTrustForSystemRole defaults to Untrusted, which everything already met. A host that lowered the bypass threshold is the one case this changes, and it can set the level back. An existing test proved "stripped" by asserting the result was Untrusted. That was the same thing until now, so the proxy stopped tracking the property; it now asserts what it always meant -- the caller ApplicationTrusted is not what got stored. 4171 unit tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
1.5 asks whether turning the gate on changes results over a corpus nobody set validity bounds on -- if it does, an unaudited writer exists. The inventory is the answer, so the tests pin it rather than asserting a vague "nothing changed". Two writers, both deliberate: temporal extraction when TemporalValidityMode .Extract asks the model for a window, and supersession, which stamps valid_until as it closes a fact. The second matters because 9.1 made it reachable on the ingestion path rather than only from an offline hygiene pass. And it changes nothing. Supersession stamps invalidated_at too, and the transaction-clock filter already removes the fact from live recall whatever this option says. So the valid-time gate is REDUNDANT for superseded facts and load-bearing only for windows the conversation actually stated -- worth recording before someone reaches for it to hide superseded data. Six live-Neo4j tests: both modes identical with no bounds; each bound filtering when set (including valid_from in the future, which a valid_until-only implementation would return as current); null meaning unbounded rather than "not yet valid"; and the supersession interaction. 1.7 documents all three of this session opt-in behaviours in CHANGELOG with the measurement that motivated each. TCK exposure confirmed zero. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
…uild The 50-question abstention build died after 270 of 616 extraction calls. Two sessions of a public research dataset tripped an Azure content policy; the batch-split path treated the 400 as a shape failure, halved the batch, re-sent the same text to the same filter, and propagated once the batch reached size 1. An hour of work lost to content that was never going to be accepted. A refusal is terminal. Neither a retry nor a split changes the text or the policy, so the only useful responses are to skip the affected sessions or abandon the run -- and skipping is overwhelmingly better PROVIDED the loss is recorded. A corpus with gaps that looks complete would have those gaps attributed to recall, which is the failure this whole track exists to prevent. So: content refusals are classified separately from shape failures (matched on the provider own vocabulary, since 400 covers both "your request is malformed", which splitting legitimately diagnoses, and "I will not process this", which it cannot); the refused sessions are skipped with an empty result; the count is recorded in diagnostics, warned loudly, sealed into the manifest and folded into its fingerprint. And refused above a 2hare of planned sessions, because the failure changes character with scale: a handful out of 2,418 is noise, hundreds is a different corpus wearing the same name. The observed rate here is a fraction of one percent, so a run approaching the limit has something new happening. Transient failures (408/429/5xx) are neither refusals nor shape failures and keep their existing handling -- the same text may well succeed later. 4185 unit tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
Two gaps in yesterday hour-old fix, both exposed by asking the obvious operator questions: which model, and which prompt. 1. The fix skipped the WHOLE batch. Measured, the batch that killed the 616-call preparation went 4 -> 2 -> 1 and still failed at 1, so one session content was responsible and its three batch-mates were innocent. A multi-session refusal is now split first -- not to retry, since the policy is deterministic for the same text, but to ISOLATE -- and only the individual session that is still refused at size 1 is skipped. 2. Nothing recorded WHICH session. The batch diagnostics are deliberately content-free, which is right, but they recorded only the exception type and batch size -- so a refusal that will recur left no trace of what caused it, and could not be investigated or raised with the provider. The session ID is now recorded and printed. The identifier, never the text: the id is the handle and the dataset is the lookup, which keeps the diagnostics free of dataset content while still being actionable. On retry specifically: there is nothing to retry. An Azure content filter is deterministic for the same input, so re-sending cannot succeed. Isolation is the only recovery that gains anything, and it gains the batch-mates. 4187 unit tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
The fact path got the rescue first because that is where the measured counter-example lives (2 facts of a 710-fact graph, answer present, answered wrongly). Leaving the other three would have meant a host enabling RescueShortOwnerResults gets it on facts and silently not on entities, preferences or traces -- the "a setting only some components respect" shape this project has already found twice, in TemporalValidityMode inert on the batch rung and in a read-only filter that removed nothing. Each path now has its owner-scoped scan extracted into one helper reached by both the empty and the short condition; two copies of a fallback drift. The trace path needed its own signature because successFilter and proceduresOnly travel with it -- and proceduresOnly reaching the scan is the fix from 7.5, preserved. Off by default everywhere, as on the fact path. 4187 unit + 332 non-NAMS integration green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
4.6. The 50-question abstention-enriched run: structured AUC 0.709, hybrid 0.768, over 22 present and 20 absent. That is a real class balance -- 440 pairwise comparisons instead of the 32 the stratified run managed with its single absent observation -- and both arms agree while both clear the 0.6 line that was fixed in code before any number existed. The progression is the story: 32/1 gave 0.969 and 0.094 from the same corpus (one question rank); 5/4 gave 0.600 twice (real but n=9); 22/20 gives 0.709/0.768. Phase 10 gate is open -- reranking is no longer reordering a candidate set whose starvation is unmeasured. Accuracy 87.8tructured, 88.0
…first time Two different things share the word "absent", and conflating them is easy enough that it happened in conversation. The sufficiency AUC's absentCount is a ground-truth INPUT -- the label retrieval confidence is ordered against -- and it is identical across arms by construction. Abstention accuracy is an OUTCOME: whether the agent actually behaved correctly on those questions. Reporting only the first invites reading a class balance as a result. On the 50q abstention-enriched run: 18 of 20 correct in BOTH arms (90%), against 25/30 structured and 26/30 hybrid on ordinary questions. The system declines to answer what memory does not hold more reliably than it retrieves what it does. That is also an independent cross-check on the AUC. A system scoring 90% on abstention is genuinely separating answerable from unanswerable, which is what an AUC of 0.709-0.768 claims; two instruments built on different evidence agreeing is worth more than either alone. Worth its own line because the project's own taxonomy names abstention questions as the only place this dataset scores meta-memory at all -- and across 52 recorded runs before typed sampling shipped, not one had ever been drawn. This is the first meta-memory measurement the project has taken. The failure is named for the behaviour rather than as "incorrect": answeredWhenItShouldHaveAbstained is the agent asserting something memory did not hold, which is the failure a memory system is least able to detect in itself. Two of twenty. No abstention questions reports null, not zero -- a rate of 0 would read as "it never abstains correctly" when the truth is "it was never asked to". An unjudged question leaves both denominators alone rather than counting wrong, since judge-parse failures are a known recurring class here. 4187 unit + 391 LongMemEval green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
10.1's premise was killed by measurement and is not being resurrected. R2 wanted a reranker in order to fuse GraphRAG as a distinct channel; of 132 GraphRAG items 132 were duplicates of what the structured surface already returned and 0 verdicts changed. Fusing a channel that returns the same rows in a different order cannot help, and its revival trigger -- a corpus with a genuinely separate knowledge graph -- has not fired. But 10.2 and 10.3 need the extension point, which is a different thing from the fusion. So the seam ships and the use case stays dead: the mechanism was worth building, the application was not. The contract states what actually matters. Reorder-only, because a filter belongs in retrieval where its effect on recall is measurable and a reranker that changes the set corrupts the section's diagnostics silently. One bounded query rather than one per candidate. Owner scope carried explicitly, because a reordering pass that reads another owner's graph is an isolation breach wearing a ranking hat. And a thrown reranker degrades to the provider's order rather than failing the recall, since a degraded order is recoverable and a failed recall is not. 10.2 is the first user of it: find the entity the query is about by the SAME embedding the retrieval used -- so "near the centroid" and "similar to the query" are commensurable rather than two unrelated notions of relevance -- then boost by gamma^hops over ABOUT/RELATED_TO. One decay constant, not two: it reuses EffectiveStructuralDecayGamma, the same one GraphRAG hop-decay uses, and is inert when that decay is off. A second independently-tuned constant for the same idea would drift and nobody would know which result came from which. The boost is multiplicative, so graph closeness is evidence about relevance rather than a replacement for it -- a candidate retrieval ranked badly is not promoted past a strong one by adjacency alone. EXTRACTED_FROM is excluded from the traversal on purpose: it links to source messages, so following it would make every fact from a shared conversation look adjacent regardless of subject. Off by default, and gated hard: non-fact sections, missing embeddings and single-candidate sets all return before any query is issued. Shipped behind the sufficiency work deliberately -- reranking reorders survivors, so over a starved candidate set it reorders seven of an owner's 504 facts and reports success. The AUC measured 0.709-0.768 first. 4198 unit tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
A fact the world asserted five times is usually more central than one mentioned once, and similarity cannot see that: two facts phrased alike score alike however often either was actually said. The signal's provenance IS the task. mention_count is incremented by the ON MATCH of the fact triple MERGE -- once per ingestion that re-states an existing triple -- so it measures how often the world said something. The tempting substitute, :MemoryReadAudit, counts how often WE surfaced it, and ranking on that closes a loop: high rank causes retrieval, retrieval raises the count, the count raises the rank. It would look like learning and be self-reinforcement. Refused, and tested for by name. Logarithmic rather than linear, because the gap between one mention and three is real while the gap between thirty and thirty-two is noise -- and under a linear boost a chatty topic would bury a precise answer. The weight is small enough that salience breaks near-ties among plausible answers rather than substituting for matching the question: a 50-mention weak match still loses to a strong one. Incremented on all three upsert paths -- single, batch and fused -- because a counter that depends on which write path ran is the "setting only some components respect" shape in a different costume. Pre-existing facts coalesce to 1, the honest reading of an absent counter, which leaves their ranking unchanged since log(1) is 0. Six live-database tests pin what the Cypher text cannot: that re-assertion increments, that a different fact about the same subject does not, that a different object is a different fact, that another owner's assertion counts separately -- and that reading a fact five times leaves the counter at 1. Phase 10 is complete. Its gate was 4.6, which reported AUC 0.709/0.768. 4212 unit + 338 non-NAMS integration green. Cypher snapshot regenerated (BOM stripped) and the query-count constant updated for the one new const. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
Access tracking is bookkeeping -- it feeds decay and retention, and nothing in the recall the caller is waiting for depends on it. It was already batched down from 25 write transactions to one, but still awaited before the model is invoked. MemoryOptions.DeferAccessTracking starts it without waiting. Two details carry the whole change: The deferred work runs on CancellationToken.None, not the request's token. That token is cancelled as soon as the response completes, so passing it through would cancel the very write being deferred -- an option that reads as enabled and does nothing, with no error anywhere. The test captures the token rather than trusting the code. And it is off by default for a hazard rather than out of caution. The write runs on SCOPED services -- a driver session, a repository -- so a host that disposes its DI scope when the response returns disposes them out from under the deferred write. That surfaces as an ObjectDisposedException in a log nobody reads, and access tracking silently stops working. A long-lived agent or hosted service can enable it; a request-scoped host should not, and the log message says so by name so the failure diagnoses itself. Failures go through a fault continuation rather than being discarded, so a deferred failure is logged instead of becoming an unobserved task exception. 4216 unit tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
The MAF provider receives the full live thread and discarded it, so recall's RecentMessages re-sent turns the model was already being given and the host paid for both copies. The fingerprint is content-only, and the plan's warning about why was verified rather than taken on trust: RecalledMessageRoleGate.EffectiveRole rewrites a privileged role down to "user" when trust is below the threshold, leaving content identical. A role-keyed match would therefore find nothing on exactly the hosts that hardened MinimumTrustForSystemRole -- the feature would work everywhere except the security-conscious configuration, and look correct in every test that did not set that option. There is now a test that does. Filtering happens BEFORE MaxChatHistoryMessages, which is the quality half rather than only the cost half: with a budget of 2 and the two newest turns already in the thread, the budget now carries two OLDER messages the model has not seen. Filtering afterwards would have spent both slots on duplicates and delivered nothing. Normalisation collapses whitespace and folds with ToUpperInvariant -- the live thread and the stored copy travel different paths, one through the host's formatting and one through persistence, so a trailing newline is not a different message; and a culture-sensitive fold would make dedup depend on the host's locale, since a Turkish locale maps i to a different capital. The live thread is materialised once. It arrives as IEnumerable, and it is now enumerated both for the user-message query and for dedup -- enumerating a lazily built source twice would be a silent correctness bug. On by default: sending the model two copies of the same turn has no upside, and the comparison is a hash set over a thread the provider already holds. Passing null reproduces the pre-2.5 output exactly, which is what every recorded measurement was taken under. 4226 unit tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
joslat
changed the title
Confidence was set once by extraction and never moved, so a fact stated five times and one stated in passing carried whatever number the extractor happened to report. The two events S2 needs already existed after this session's other work: mention_count (10.3) marks the moment the world re-asserts a triple, and write-time supersession (9.1) marks the moment one fact replaces another. So this is arithmetic on moments already identified rather than new machinery. Corroboration adds alpha on the MERGE's ON MATCH. Contradiction subtracts 2*alpha from the superseded loser, and the asymmetry is deliberate: being contradicted is stronger evidence against a fact than one more restatement is for it -- a repeated claim may be a habit of phrasing, a replaced one is a claim the world stopped believing. Only the loser moves; demoting both would make every contradiction erode the graph. Clamped to [0,1] at both ends, and checked in a live database rather than in the Cypher text, because that arithmetic is where an escape would happen and confidence is read by ranking, dedup and decay -- a negative value would be consumed as a number rather than rejected as an error. The gate lives in the Cypher, not in C#: at alpha 0 the assignment is byte-for-byte the original, so no sealed measurement moves and there is no second code path to keep in step. Applied on all three write paths -- single, batch and fused -- because reinforcement that depended on which path ran would be a property of batching rather than of the conversation. Weak conflict (-alpha) is deliberately not built. For a multi-valued predicate a second object is not a conflict at all -- "likes tea" and "likes coffee" are both true -- and detecting the genuinely ambiguous case would cost a lookup on every write to penalise something that is usually correct. 4226 unit + 345 non-NAMS integration green. Cypher snapshot regenerated (BOM stripped); no query-count change, since these are edits rather than additions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
The supersede query has two callers, not one: the write-time path added in 9.1 and the offline conflict-resolution hygiene pass. Only the first passed reinforceAlpha, so ResolveFactContradictionsAsync failed with "Expected parameter(s): reinforceAlpha" the moment it ran. Two things worth recording. First, a contradiction resolved by the hygiene pass is the same event as one resolved at write time, so it should move confidence the same way -- the omission was a wiring gap, not a design choice. Second, only the integration suite could catch it: a missing Cypher parameter is rejected by the server, so no unit test with a substituted transaction runner would ever see it. This is the third time this session that "wire it up everywhere" has been the actual work rather than the feature: TemporalValidityMode inert on one extractor rung, RescueShortOwnerResults on one of four vector paths, and now reinforcement on one of two supersede callers. 4226 unit + 345 non-NAMS integration green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
RecallAsOfAsync(validAsOf, systemAsOf) has existed since the bitemporal work and nothing in an ordinary conversation could reach it: "what did I think back in March?" recalled against now, exactly like every other question. Phase 1 shipped a capability nothing could ask for. TemporalQueryParser resolves the instant a question asks about -- deterministic regex, no model call, resolving against IClock rather than the wall clock. MemoryService.RecallAsync routes a resolved query to RecallAsOfCoreAsync with BOTH clocks set: "what did I think then" means what was true then as known then, and moving only the valid clock would answer with today's corrections applied to the past. Precision over recall, deliberately. The failure modes are asymmetric: a missed expression costs nothing (the turn recalls against now, today's behaviour), while a false positive silently narrows recall to a window the user never asked about and returns an answer that looks entirely ordinary. So "in March" resolves and a bare "March" -- a surname -- does not; "last week" resolves and "the last item" does not; "in 2024" resolves and "in 2024 units of stock" does not. Off by default, since it changes which memories a temporal question sees. Wiring is tested at the seam, not just the parser: 3 of the 8 routing tests fail when the branch is disabled, verified by disabling it. A parser nothing calls passes all of its own tests while the bitemporal path stays exactly as unreachable as before -- the shape this track has now hit four times. The memory://context MCP resource is the one read surface that does not honour the option; it composes the assembler directly and the MCP server references only Abstractions by design. Documented on the option rather than papered over. 4,263 unit tests green (+37). Release build 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
The extraction call is where the money is: the write is one round trip to Neo4j, while extraction is a model call over the rendered transcript and on a corpus build it is essentially the whole bill. An "ok, thanks!" turn pays it for nothing. The gate sits before the completion, not before the write, and that placement is the finding rather than a detail. A write-side gate -- skip persisting a triple already in the store -- is the obvious reading of "novelty gating" and would have silently disabled two features shipped this same week: 12.4's confidence reinforcement, where a re-asserted fact earns alpha on the MERGE, and R7's mention_count, which the salience reranker reads. Corroboration IS the repeated write. Both would have stayed enabled and gone inert. Precision over recall, and the asymmetry is worse here than anywhere else in the system. Declining to gate costs one call. Gating a turn that did carry a fact means the memory is never formed, nothing downstream can recover it, and nothing can even report it missing. So the vocabulary is closed and deliberately excludes "yes", "no" and "sure": each is a complete answer to a question, and the question may have sat in the previous batch. "no problem" therefore does not gate either -- a wasted call is the cheaper mistake by a wide margin, and that trade is pinned by its own test rather than left as an accident. The pre-extracted path is explicitly not gated: the caller has already paid for that completion, so gating there discards work instead of avoiding cost -- the gate's exact inverse. Off by default; transcript bytes are fingerprinted into every measured run. Wiring verified red-before-fix on both dispatch paths, per-type and unified. 4,299 unit tests green (+36). Release build 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
One batch is often not enough to understand itself: "I moved there last year" needs the turn that named the place, and "she recommended it" needs the turn that named her. Without context those extract nothing, or extract an unresolved pronoun as an entity. Widening the batch fixes the reference and breaks something worse, and the plan's own note for this task had gone stale on exactly that point. It reasoned that re-extracting the preceding turns was "harmless for an exact triple, which MERGEs" -- true when it was written, false as of this week. S2 confidence reinforcement and R7's mention_count both key off the re-assertion, so extracting context would make a fact earn confidence and mentions every time it happened to sit inside a sliding window. Two signals meaning "the world keeps asserting this" would quietly come to mean "this was said recently", and nothing would look broken. The context/target split is therefore a correctness property now, not a question of token efficiency. ExtractionWindow carries Targets and Context separately. Extractors read the context, extract only from the targets, and provenance is attributed only to the targets -- an EXTRACTED_FROM edge to a context turn would assert the memory was stated in the very turn the prompt forbade extracting from. Extension uses new method names (ExtractWithContextAsync) rather than overloads. An overload made ExtractAsync(null, ct) ambiguous at existing call sites, which is a source break on a SemVer-locked public surface. The off state is call-identical, not just byte-identical: with no context the original method is invoked, so an extractor that never implemented the window overload -- including a third party's -- takes the exact path it always did. That also proved the point when the first version routed everything through the new method and broke ten existing mocks. Context renders in an unnumbered fence, because 9.3 resolves per-item provenance positionally: numbering context into the same sequence would shift every target index, and the result is not a crash but each fact attributed a few turns early, indistinguishable afterwards from precise attribution. The window is filled once in ExtractAndPersistAsync, the chokepoint the Agent Framework provider, the memory facade and the MCP ingest tool all pass through -- a window resolved per caller would depend on the host. Off by default (0 turns), and at 0 no history query is issued at all. 4,312 unit tests green (+13). Release build 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
LongMemEvalGraphProbe counted Entity, Fact, Preference and RELATED_TO, and was label-blind to :ReasoningTrace. So "the corpus contains no traces" and "the probe cannot see traces" produced byte-identical output, and Phase 7's procedural work was about to be measured against a graph nobody had confirmed held anything to measure. The probe now counts traces, and procedures separately from traces in general -- Phase 7 promotes procedures specifically, and a lumped count would report a corpus as procedure-bearing when it holds only episodes. TotalLearned deliberately still excludes traces. It appears in every measurement sealed before the probe could see them, and quietly widening its definition would make every prior build appear to have grown for a reason unrelated to what was extracted. The counts are nullable rather than defaulting to zero. A manifest written before this change has no such field, and a non-nullable int would deserialize it to 0 -- reproducing in the recorded data the exact ambiguity this task exists to remove. TracesMeasured separates "looked and found none" from "never looked", the same unknown-reading-as- agreement trap caught earlier in the preparation manifests. 396 LongMemEval tests green (+5). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
An agent with no procedural memory investigates: slower, and safe. An agent with the wrong procedure executes -- confidently, on a plan built for a different task. Every efficiency measure a promotion feature has (hit rate, steps, tool calls, latency) improves when the retriever becomes more willing to answer, including when it becomes more willing to answer wrongly. Latency alone cannot tell those apart. ProcedureRetrievalPrecision therefore reports three outcomes rather than an accuracy: correct, wrong, and abstained. Abstention is deliberately not a failure -- it is the safe outcome, and folding it in with wrong answers makes a cautious retriever indistinguishable from a reckless one. That is the distinction Phase 5 drew for meta-memory, and it counts for more here because acting on a wrong procedure is paid in tool calls rather than in a sentence. WrongProcedureRate is the figure a promotion change has to be judged against. PrecisionWhenAnswering is reported beside PrecisionAtOne because the two move in opposite directions as a retriever gets more cautious, and either alone hides the trade. Retrieving anything for a task no stored procedure fits counts as wrong, not as a near miss: that is exactly the case where an agent executes a plan for a different problem. Deterministic and provider-free -- it scores id lists, so it is asserted in unit tests rather than measured against a model. Its consumer is 7.6's harness, still to be built. 403 LongMemEval tests green (+7). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
…3.7) AgentEval's free-text verdict parser vetoes a leading "yes" when the word "no" appears later, so a judge answering "yes -- there is no discrepancy" is scored as a failure. That is a systematic mis-scoring and StructuredJson is the actual fix. The plan recorded this as "wired and reachable". It was not: the protocol parameter existed on CreateOptions and every call site took the default, so no run could ever select it -- the dead-option shape, found for the fourth time in this track. Now selectable via --judge-protocol, defaulting to free-text. The default deliberately does not move: AgentEval's own docs say results under StructuredJson are not comparable with a free-text base, and every sealed base here is free-text. Flipping it would not produce a wrong number, it would produce a better one that silently invalidates every comparison anybody draws against the existing runs. An unrecognised value throws instead of falling back. A silent fallback would mean a run the operator believed was StructuredJson produced a free-text score under a StructuredJson label -- worse than either, and invisible afterwards. The choice is emitted unconditionally into the report's protocol block, beside the extraction vocabulary and assistant-content fingerprints, so a StructuredJson score cannot sit next to a FreeText one without the comparability break being visible in the artifact itself. The remaining half -- running it on a fresh base -- is provider spend, the same gate as 4.2 and 8.3b. 4,312 unit + 410 LongMemEval tests green (+7). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
Recall about a well-known entity returns twenty facts that each spend context to say one thing. A summary says the same in one item. But a summary is derived memory, and derived memory is where a store quietly starts lying: the sources change, the summary does not, and afterwards nothing about it looks any different. So staleness is proved rather than assumed. Each summary carries a fingerprint over the exact facts it was written from -- id, confidence and invalidation, order-independent so a query-plan change cannot invalidate the whole store without a fact having moved. Before a summary is used the fingerprint is recomputed and compared; a mismatch means it is not returned at all. Not returned with an IsStale flag: a flag puts the decision in every caller's hands, and one caller forgetting to check is indistinguishable from correct memory. Detection on read rather than invalidation on write, deliberately. A write-side sweep has to find every affected summary, has to run in the same transaction, and leaves a summary looking current whenever any path writes a fact without going through it. Recomputing on read cannot be bypassed by a writer that did not know summaries existed. Confidence is in the fingerprint because S2 reinforcement moves it, and a summary stating flatly what the store has since grown doubtful about is precisely the stale shadow this design exists to prevent. Found on the way: invalidated_at has always been recorded in the store -- supersession is non-destructive so as-of recall can reach it -- but was never projected onto the Fact record, so no caller could tell a superseded fact from a live one. Now surfaced as InvalidatedAtUtc. The default synthesizer is deterministic and makes no model call. Summaries are regenerated whenever any source moves, so a completion per entity per change would scale cost with exactly how much the conversation talks about its subjects. Reproducible output also means a change in the text always signals a change in the facts. Summary nodes keep their own EXTRACTED_FROM edges, rebuilt wholesale on regeneration -- a summary rebuilt from fewer sources must not keep claiming provenance its content can no longer support. 4,329 unit (+17) and 352 integration (+7) green. Release 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
The honest assessment of this store was "capable but opaque": memory could be queried but not seen. `agentmemory block` renders what an owner holds -- entities, facts, preferences -- as one readable page. The design decision that matters is what was left out. Block-memory systems elsewhere let the agent edit its own block and hand it back, and at that point the block is the store: every provenance edge, trust level and supersession record in the graph describes a shadow of what the system actually believes. So there is no `block --write` and no parser turning a block back into memories. The absence is the feature. Legibility stays actionable another way: every line prints its memory id, so a human who spots something wrong acts on that exact memory through invalidate or supersede -- audited, attributable, reversible -- instead of rewriting prose and hoping something parses it. Two failure modes get explicit guards, because both look exactly like a correct block to whoever is reading it. Superseded memories are dropped twice, at the query and again in the renderer, since a caller forgetting IncludeInvalidated would otherwise get retracted claims presented as current. And truncation is counted into OmittedCount *and* stated in the rendered text -- a block that quietly stops short reads as "this is everything", which invites the conclusion that a missing memory was never stored. Built on the existing audited IMemoryHistoryService read path rather than new queries, which is what makes calling it a projection true rather than aspirational. Rendering is deterministic, so diffing two blocks shows what changed in memory rather than in the renderer. 4,343 unit tests green (+14). Release build 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
Loading a backlog was always possible, and both obvious ways are wrong in opposite directions. A serial loop over ExtractAndPersistAsync is correct and takes hours. An unbounded Parallel.ForEachAsync saturates the provider quota and the connection pool, and the damage lands on everything else sharing the process: median latency barely moves, which is what makes it hard to notice, while p99 degrades 20-70% under saturation. IngestBulkAsync is a default interface method on IMemoryIngestion that paces calls the host could have made itself. Deliberately composed rather than new machinery: a separate bulk pipeline would be a second place for trust stamping, provenance and owner scoping to drift out of agreement with the per-request path, and that drift would only surface in the corpus months later. Failures come back per request with their index into the submitted list. A bulk API returning "8,412 of 10,000 succeeded" tells the caller they have a problem and nothing about which inputs to retry, so the realistic response is to re-run all ten thousand -- more expensive than the failure was. NotAttemptedCount is kept separate from FailedCount for the same reason: re-running a request that was never attempted is always safe, re-running one that failed part-way may not be. A stop-on-error run completes normally and returns its report; a cancellation the caller requested still throws. Discarding the record of what did succeed because something failed is the opposite of useful. Its own test found a real bug: the semaphore wait sat outside the try block, so tasks queued behind the gate when a stop-on-error run halted threw OperationCanceledException straight through Task.WhenAll instead of being recorded as unattempted. Scoped honestly, per the plan: this is a public surface plus documentation. It is not the 10.70x throughput figure, which is ten-owner concurrency and has always been available to any caller willing to parallelise -- what is documented is the safe way to do that. 4,352 unit tests green (+9). Release build 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
RecallOptions.LatencyBudget bounds how long context assembly may take. Sections still running when it expires are dropped and the context comes back without them, rather than the caller waiting on the slowest one. The marking is the feature, not the timing. A section cut short reports Searched = true and Returned = 0, which is byte-identical to a section that ran to completion and genuinely found nothing -- so the obvious implementation degrades completely invisibly, and the caller answers confidently from memory that was never consulted. Every dropped section now carries TimedOut in its diagnostics, and the context carries LatencyBudgetExceeded so a caller can notice without inspecting each section. LatencyBudgetExceeded is deliberately a separate flag from Truncated. Truncation is the context budget trimming memories that were retrieved; this is a retrieval that never came back. A caller told only "truncated" would reasonably conclude the memories exist and were dropped to fit, when they may never have been looked at. Abandoned queries are not cancelled. They are already in the driver and tearing them down buys nothing the caller is waiting for -- it turns a wasted result into a wasted result plus a cancellation storm. Their faults are observed so a slow section that also fails does not surface later as an unobserved task exception. Off by default, and with no budget the wait is the original unconditional Task.WhenAll, byte for byte. Wiring verified red-before-fix: 2 of 7 tests fail with the branch disabled, and the red run takes 1m30s against 0.8s green -- the runtime difference is itself the proof that the budget short-circuits rather than merely relabelling a completed wait. 4,359 unit (+7) and 352 integration green. Release 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
"1.6k context tokens instead of 115k" is the most persuasive number in this category precisely because it is a claim about architecture. It cannot be inflated by a better answer model, cannot be tuned with prompt engineering, and does not move when the judge changes its mind -- it is either true of the assembled context or it is not. Which is exactly why it had to wait for C4. Until real tokenization landed, this codebase converted tokens to characters by multiplying by four, and a compression ratio derived from that would have been a claim about arithmetic wearing the clothes of a claim about design. ContextTokenBreakdown reports per-section token cost of an assembled context against the full transcript. The baseline is deliberately the whole conversation -- what a memoryless agent would have to send to answer the same question -- rather than a truncated window, which would flatter the result by measuring against a system that has already given up on remembering. Three choices worth naming. The counting method travels with the number, so a fallback estimate can never be read as a measurement. Empty sections are still listed, because an omitted section reads as "this kind of memory does not exist" rather than "it contributed nothing here". And an empty history yields a null ratio rather than infinity: an absent denominator is an absent measurement, and reporting a number there would be the single most flattering way to be wrong. Provider-free, so the instrument is asserted in unit tests rather than bought. Per-stage latency already existed via LongMemEvalStage; what remains for publication is one measured run. 416 LongMemEval tests green (+6). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
LongMemEval cannot answer whether procedural memory helps. It scores answers about a transcript; a promoted procedure changes how an agent works across repeated attempts at a multi-step task. With no trace nodes and -- until 6.5 -- a probe blind to them, a promoted procedure was invisible to every instrument this project had. The feature could not fail. That is the defect shape this harness exists to prevent, not merely a gap in coverage. Two comparisons, and both are needed. On versus off answers "does it help at all". First run versus last answers "did it help because it learned" -- an arm that is already cheap on attempt one learned nothing during the run, and whatever separates the arms is not a procedure. Every efficiency number is gated on completion rate, without tolerance. An agent that abandons a task sooner takes fewer steps and makes fewer tool calls, and both read as improvements in every measure a harness would naturally report; a system that scored giving up as its biggest win would be worse than no measurement. Any drop in completion disqualifies the efficiency claim outright, because an agent with the wrong procedure executes confidently and the steps it saved are not a saving if the task is not done. Step and tool-call means are taken over completed runs only. Mixing "solved it in three steps" with "gave up after three" treats abandonment as a cheap success. Arms run sequentially. The hypothesis is that attempt N benefits from what attempt N-1 stored; running them concurrently would race the write that the next read depends on, and measure a feature that had not happened yet. Driven through an IAgentTaskRunner seam, so the part that can be quietly wrong is unit-tested against a scripted agent while the part that costs money is deferred to a measured run. 4,359 unit + 425 LongMemEval tests green (+9). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
This was referenced Aug 27, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
17 commits. The measurement track reported, Phase 10 shipped on the back of it, and Phase 2's buildable tail closed.
The number that unblocked everything
4.6 — sufficiency AUC: 0.709 (structured) / 0.768 (hybrid) over 22 present / 20 absent.
It took three attempts to become a number at all, and the journey is the finding:
The blocker was never the signal. The answer-presence gate reports the answer present in ~97% of questions, so the "unanswerable" class came only from accidental extraction misses. Abstention questions are the designed supply — and
AbstentionPolicyhad shipped in AgentEval 0.20 with no caller.Then wiring abstention still wasn't enough: the gate called 3 of 4 abstention questions "present", because their topic is discussed even though the fact is not. A cheap floor for extraction failure is not a ground truth for answerability. The dataset's own
IsAbstentionis.Also measured for the first time: meta-memory. 18/20 (90%) on abstention questions in both arms, against 25/30 and 26/30 ordinary — the system declines to answer what memory does not hold more reliably than it retrieves what it does. Across 52 prior runs, no abstention question had ever been drawn.
Phase 10 — complete
EXTRACTED_FROMexcluded — it links to source messages, so following it would make every fact from a shared conversation look adjacent.:MemoryReadAuditsubstitution that would close a rich-get-richer loop, and live-DB tests pin the direction: reading a fact five times leaves the counter at 1.Resilience found by running it
A 50q build died after 270 of 616 calls because two dataset sessions tripped an Azure content filter, and the split path re-sent the same text to the same policy until the batch reached size 1. Refusals are now terminal-by-classification, isolated by splitting (the batch that killed it went 4→2→1: one session was responsible, three batch-mates were innocent), skipped by session id, recorded in the manifest and fingerprint, and refused above 2%.
And
LongMemEvalRefusedEvidenceanswers the question tolerance cannot: did a refusal cost a question its gold evidence? On the successful run, all three held context only — checked, not assumed.Phase 2 tail
Verification
4,226 unit + 391 LongMemEval + 338 non-NAMS integration green. Guards updated deliberately, never silenced: Cypher snapshot (BOM stripped), query count 155→156, service interfaces 41→42, enum count.
🤖 Generated with Claude Code
https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE