feat: LongMemEval measurement track, quality instrumentation, and the product fixes it found - #177
Merged
Merged
Conversation
J1.5c. J1.5b found seven high-frequency predicates the 101-entry vocabulary did not know. Four canonical relations added -- helped, provided, noticed, heard, with inflections and non-colliding synonyms -- taking the table to 106. `practiced` is deliberately excluded: it is a held-out miss, and adding it would fit the vocabulary to the slice that exists to test it. Predicted that adding the six build-slice misses would take build coverage to 100% while held-out stayed at 93.8%, failing the gate by 6.25 points as it detected fitting-to-observed. Confirmed exactly. That failure exposed something worse than the vocabulary gap. The bound band holds ~16 held-out predicates, so one miss is 6.25 points -- already past the 5-point tolerance. At this slice size the gate cannot express "slightly worse"; it is binary and hostage to which predicates the split drew. Same class of defect as the original 15.4-point skew failure: a threshold finer than the statistic's resolution. Fixed by measuring the distribution rather than one draw. Across five seeds the gate passes 4/5 with held-out coverage at 100% on every passing split. The single failure is seed 42 -- the split whose build slice authored the edit, and therefore not a held-out evaluation of it. Multi-seed support lives in the tool, not in a planning document, because a protocol recorded only in prose is one nobody runs. The tempting wrong move is recorded because it was tempting: widening the tolerance to max(5 points, one predicate) lands at exactly 6.25 and flips the seed-42 FAIL to a PASS at the boundary. That is a threshold chosen to fit the result, and it was rejected. These entries are coverage-verified but extraction-unmeasured. J1.6 is the standing proof that a vocabulary edit moves extraction unpredictably, so J1.5d holds a cold rebuild to measure it. 3805 core + 179 harness unit tests green; all 93 vocabulary gates pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
The hybrid arm has been rejected three times on one question's judge verdict (gpt4_7fce9456), and each rejection took a complete 30-question run's results with it -- `result` was written only when the arm was accepted, so one unjudgeable question destroyed the evidence for the other twenty-nine. The acceptance guard is unchanged: `accepted` is still false and `result` is still null. The measurements are now also written under `unacceptedResult`, a name no reader can mistake for an accepted result. Rejecting an arm should mean "do not treat this as a verified score", not "delete the data". The rejection was also undiagnosable. A bare catch around the judge call made a provider failure indistinguishable from an answer whose shape the parser does not accept, and the parser keys on the explanation's first letter-token being exactly yes or no. Both are now recorded: FailureKind distinguishes threw:<ExceptionType> from unparseable, and RejectedToken carries the leading token the parser refused -- the judge's own verdict word, never the explanation body. No provider detail and no user content enters the artifact, which was the reason the catch was silent in the first place. 179 harness unit tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
An n=50 preparation ran 37 minutes and aborted at question 20 with "observed 14 calls, 1 failures ... expected exactly 12 unified-batch calls", caused by a single System.ClientModel.ClientResultException. The accounting guard was right to fail closed; the reason it had something to reject was a defect one layer down. LlmMultiSessionUnifiedMemoryExtractor caught `Exception` and split the batch whenever it held more than one session. Splitting is a genuine remedy when the batch itself is the problem -- too many input tokens, an incomplete acknowledgement, an unusable source-session key, an unparseable response -- and every one of those arrives as a FormatException, BatchValidationException deriving from it. A provider transport failure is not that. Halving the batch re-sends the same request shape to the same endpoint that just failed, so the split neither diagnoses nor fixes anything, and it doubles the call count for that question. Transport failures belong to the configured retry policy. The catch is now FormatException, which is exactly the set of shape-caused failures. Also surfaces the provider status code in the accounting-mismatch message. The old message named the exception type but not the status, which is the difference between "we are being rate limited", "the request was malformed" and "the service failed" -- three problems with opposite responses. Status codes carry no content, which was the reason the detail was omitted. Tests written red-first: both failed against the old behaviour on call counts, proving they detect the split. The control test asserts that unparseable responses still split, so the fix cannot have removed the useful half. The exact split call count is deliberately not asserted -- that is recursion depth, not the property under test. 3807 core + 179 harness unit tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
The comment committed with the batch-split fix said transport failures "belong to the configured retry policy". Checking it found no such policy on that path. LlmExtractionOptions.MaxRetries is set by the harness, validated at registration, and honoured by LlmExtractionRunner -- but only by re-prompting on unparseable JSON. The runner's GetResponseAsync call sits outside any try/catch, so a transport exception propagates on the first attempt. Nothing else retries; the call meter counts retries without performing any. So a 614-call preparation had no protection against a single transient, which is how the first n=50 attempt died at question 20 after 37 minutes. The splitter was the only thing resembling a retry for transports, and a bad one: it re-sent to the endpoint that had just failed. Removing it removed a misleading recovery, not a working one -- which is what the corrected comment now says. A real transport retry is not a one-line addition: the prepared-pair guard requires exactly BatchCount calls and zero failures, so any retry trips the same guard that the split tripped. The refinement is recorded in the plan -- the invariant should be "each batch produced exactly one SUCCESSFUL call and every retry is recorded", not "no provider error ever occurred", which no 614-call network run can promise. Deferred while the n=50 run is in flight so its result is produced under the contract that was recorded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
Two n=50 preparations died mid-run on a single transient provider failure -- 614 calls each, at 37 and 26 minutes. There was no transport retry anywhere in the extraction path: LlmExtractionRunner honoured MaxRetries only by re-prompting on unparseable JSON, and its provider call sat outside any catch. At that call volume the absence was not a robustness gap, it was a hard blocker on the measurement. Three changes, together, because any one alone just moves the failure: 1. LlmExtractionRunner retries transport failures with backoff, bounded by MaxRetries. FormatException is not retried -- it is caused by the request's own shape, so re-sending it unchanged cannot help, and the parse loop and batch splitter already handle that family. Cancellation is never retried: doing so would make the preparation watchdog's timeout unenforceable. 2. The prepared-pair accounting guard now checks that every batch produced exactly one SUCCESSFUL unified-batch call, rather than that no provider error ever occurred -- which no 614-call network run can promise. This is simpler than the design anticipated: a recovered retry is exactly one extra call plus one failure, so calls-minus-failures recovers the successful count with no new declaration mechanism. Nothing is loosened; an unrecovered failure still throws before the check and a spurious extra call still trips it. 3. The top-level handler printed only exception.Message, so a stage wrapper reported "LongMemEval batched extraction stage failed." and a 26-minute run died with no cause attached. It now prints the whole inner-exception chain, types and messages, no stack traces. Tests red-first: the retry tests failed on call counts against the old behaviour, and the persistent-failure and cancellation cases pin that the retry is bounded and never swallows a cancellation. 3810 core + 179 harness unit tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
AddAgentMemoryCore(MemoryOptions) replaces the registration of IOptions<MemoryOptions> only. A host resolving IOptionsMonitor<MemoryOptions> or IOptionsSnapshot<MemoryOptions> would still go through the options factory and receive defaults, so the same options type would report two different values. Nothing in this product resolves either -- verified, every consumer takes IOptions<MemoryOptions> -- so this is latent rather than active. Recording it on the overload is cheaper than someone finding it in a host. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
The third n=50 attempt died at 15 minutes on a genuine parse-or-format batch split doing exactly what it is designed to do: 16 successful calls against 12 planned, zero failures. Three consecutive 15-40 minute preparations have now been ended by this guard, none of them for a real defect. The guard demanded exactly the planned number of provider calls and zero failures -- that is, it demanded that nothing ever went wrong. That is incompatible with the recovery paths the extractor ships: a parse retry re-prompts, a split re-sends the halves, and both legitimately add calls. Correctness was never this guard's job and still is not. The session-set comparison beside it proves every planned source session was persisted, in chronological order, all succeeded, and it is untouched. This is a cost guard, so the invariant it should express is "no unaccounted work": at least the planned calls happened, nothing of an unexpected purpose ran, and any excess coincides with a recorded split or retry. Refining a guard is only defensible if it still catches what it was for, so the decision is extracted as a pure function and tested at its boundaries. The load-bearing case -- excess calls with no recorded split and no recorded retry -- is still rejected, as is under-running, an unexpected call purpose even when recovery is recorded, and missing split diagnostics, which fail closed rather than reading as innocent. The pre-existing FailsClosedOnAnUnplannedProviderCall test still passes on behaviour; only its message assertion moved, and its comment now records why. 3810 core + 186 harness unit tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
The fourth n=50 attempt burned its full 60-minute watchdog budget and stopped at 544 of 614 calls with 7 failures. The dump named the cause precisely: first_failure_status=400, slowest_input=59043 tokens. HTTP 400 is Bad Request -- the deployment was rejecting oversized batches outright. This indicts the previous two commits, together. The transport retry treated everything that was not a FormatException as transient, so it re-sent requests the provider had already rejected as malformed, three times each. And narrowing the splitter to FormatException meant an oversized-input 400 no longer split either. Between them, oversized batches became unrecoverable and expensive: the one path that could have fixed them declined, and the other paid for the same rejection repeatedly. Both classifications are now explicit and tested: - Retry 408, 429 and 5xx, plus failures carrying no status at all, which never reached the service. Everything else is permanent; re-sending an identical request cannot change a 400. - Split on FormatException OR a permanent 4xx. An oversized request IS a batch-shape problem, which is exactly what splitting is for. 408 and 429 are excluded deliberately -- answering a rate limit by sending more requests is the wrong direction. The status is read reflectively, covering ClientResultException.Status and HttpRequestException.StatusCode, so this library needs no package dependency to classify an error. Also lowers the harness to --max-input-tokens 40000 from the 100,000 default, which was never a real limit for this deployment. That removes the trigger; the two fixes above handle it if it recurs. 3818 core + 186 harness unit tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
The n=50 comparison localised Structured's deficit precisely: multi-session questions, 53.8% against Hybrid's 84.6%, carrying 5 of the 8 discordant pairs. Every other question type is within one question, and knowledge-update and single-session-user are identical. It could not say why. The losses and wins are indistinguishable by volume -- 61.8 mean facts on the losses against 59.0 on the wins, 1,841 tokens against 1,768 -- so Structured is not retrieving less. And GoldEvidenceCoverage is null in every prepared-pair report, because the existing probe sits inside `if (!PreparedMemory)` and therefore runs only during preparation, never on the path that produces the measurements. That probe also answers a different question: it asks whether anything was LEARNED from the gold sessions. This adds the retrieval-side half -- whether the gold evidence reached the context -- by intersecting the retrieved facts' SourceMessageIds with the question's gold source messages. Facts carry that provenance and the Neo4j repository populates it on read, so no new query is needed. Together the two separate three failures that look identical in a score: the evidence was never extracted, it was extracted but not retrieved, or it was retrieved and the reader still got it wrong. Only the middle one is a retrieval problem, and only the first is an extraction problem, so this is what decides where the next effort goes. Returns null rather than zero when a question has no gold messages: zero coverage of nothing is not a miss and must not be averaged in as one. 3818 core + 191 harness unit tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
Predicate-expansion resolution was computed inline inside the fact-search call and discarded, so no report could distinguish "expansion had nothing to expand" from "expansion ran and did not help". Those need opposite responses: a missing vocabulary entry, versus a retrieval or reading problem. The distinction is not theoretical. Checking the four multi-session questions Structured loses at n=50 against the shipped lexicon by hand found two blocked at the query side before any retrieval happens, for opposite reasons: service/serviced/servicing is absent from the table entirely, and `has` is a deliberate query stop form that must not expand because it would pull in the whole graph. Both failed with expansion enabled and nothing to expand, and both looked exactly like an ordinary retrieval miss in the report. MemoryContext now carries ResolvedQueryRelations, and the harness reports it per question, so that state is visible in every future run instead of requiring the lexicon to be consulted by hand afterwards. Also retracts, in the plan, a conclusion drawn earlier in this session. The retrieved-gold-coverage metric added a few commits ago saturates: 30 of 50 questions land on exactly 6/7 because one gold message per question systematically yields no fact. Two group means were compared without checking the distribution behind them, and a saturated metric produces near-equal means for any two groups. The claim that the multi-session deficit is not a retrieval failure does not follow and is withdrawn; the metric cannot answer that question. Relation completeness, not message coverage, is what could. 3821 core + 191 harness unit tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
J1.7e, now with a measured target instead of a guess. Reporting ResolvedQueryRelations localised the whole Structured/Hybrid gap: on the 40 questions where predicate expansion resolves at least one relation Structured scores 87.5% against Hybrid's 90.0%, at 2.5x less context. On the 10 where it resolves nothing, Structured collapses to 50.0% while Hybrid holds at 80.0%. Those ten questions carry the arm gap. Reading the ten showed what they needed. Four relations were absent from the table entirely -- tried, wakes, led, commutes -- and each maps to a question in the bucket: how many Korean restaurants have I tried, what time do I wake up, how many women on the team led by my former manager, how long is my daily commute. Two more were surface-form gaps on relations already present, and the ambiguity gate caught both attempts to add them as new canonicals: `participate in` already belongs to `attended` and only the bare `participated` was missing, and `stay` already belongs to `stayed at`. Failing closed there is exactly what that gate is for. Not every blocked question is fixable this way, and conflating them would be a mistake. `has` is a deliberate query stop form -- expanding it would pull in the whole graph on any question containing "is" -- so aggregating over a stop-formed relation is a missing capability, not a missing word. One question is a conversational back-reference with no relation at all. This is a query-side change only: the graph is untouched, so it is measurable by reuse on the frozen n=50 base with no rebuild and no extraction spend, and the build-variance caveat does not apply. Prediction locked before running: the bucket shrinks from 10 to about 4-5, and Structured moves +2 to +4 questions. 3821 core + 191 harness unit tests green; all 96 vocabulary gates pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
Question dad224aa was rejected in 2 of 5 identical n=50 repeats with "AgentEval judge returned no valid yes/no verdict", each rejection discarding a whole arm. The diagnostic added earlier captured the cause exactly: FailureKind=unparseable, RejectedToken="Judge". So this was never the judge being wrong. The judge produced a verdict; the parser could not read it. It stripped exactly two hardcoded prefixes -- "Judge said:" and "Judge outcome:" -- and required the next letter-token to be yes or no. A third "Judge...:" shape falls straight through. On one of the two runs the retry recovered the same question with a valid verdict, which is the clearest evidence available that the judgement was fine and the parsing was not. The fix is deliberately one-way: a leading label is only stripped when what follows is ACTUALLY a yes or no, so tolerance can never manufacture a verdict from a hedge. The label must also begin "judg" and sit within 32 characters, and only the first colon is considered, so a sentence that merely contains a colon cannot be mined. The first attempt was too permissive -- any short label -- and the guard test caught it: "maybe: yes" parsed as a verdict. That case is now pinned alongside "Judge verdict: partially correct", "Judge could not determine" and "The answer is correct.", all of which must stay invalid. An unreadable judgement has to stay unreadable, because inventing one silently scores a question nobody judged. 3821 core + 208 harness unit tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
…d on For a question naming a relation, computes two numbers that do different jobs and must never be collapsed into one ratio: D = live facts in the OWNER'S GRAPH under that relation N = distinct facts in the CONTEXT under that relation D is the point. It is a deterministic Cypher count(), immune to answer-model and judge non-determinism, so if a change stops learning a relation the questions need, D drops and says so. Nothing else in this repository can see that: the deterministic fixture sits at 1.000 by construction with no headroom, and the LongMemEval channel carries sd 9.3 cold-build. That gap is why 44 register items that touch extraction cannot currently be accepted or rejected on evidence. Keeping D and N apart is what makes the metric diagnostic rather than merely another score. D = 0 with a non-empty key set means the relation was never extracted -- an extraction or vocabulary miss. N < D means it was extracted and retrieval left some behind -- a retrieval defect. A single ratio renders those identical, which is precisely how the saturated message-coverage metric misled this track once already. Also reports LimitBinding (D > MaxExpandedFacts), because completeness is arithmetically impossible when the graph holds more than the single shared LIMIT can return, and that is a budget fact rather than a retrieval defect. The probe mirrors FactQueries.SearchByCanonicalPredicates' WHERE clause exactly, minus ORDER BY/LIMIT, so the denominator counts precisely the rows expansion could have returned. It deliberately does NOT use coalesce(predicate_key, toLower(predicate)): a fact with a null predicate_key is invisible to expansion, so counting it would report an unreachable fact as a retrieval miss. The interface method is default-bodied, so none of the seven existing test stubs changed. Not-measured is an ABSENT field rather than an all-null object, matching ReadGoldCoverageAsync -- caught by an existing equivalence test that the first wiring broke. 3821 core + 215 harness unit tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
An 18-agent adversarial review of this branch against main produced four
blocker claims and three should-fix claims; a verify pass refuted or
downgraded five of twelve. These are the three that survived.
1. CI would have been permanently red. AgentMemory.slnx lists the LongMemEval
harness and its test project, and the harness carried an unconditional
ProjectReference to a hardcoded absolute path in a separate repository. A
verifier reproduced the exact failure: restore SUCCEEDS with only a warning,
then Release build fails with 56 CS0246 errors, so the break lands after the
step that would have made it obvious. Both ci.yml and release.yml run
`dotnet build AgentMemory.slnx -c Release`.
Fixed by consuming AgentEval as a package instead. AgentEval.Memory is
IsPackable=false but ships bundled inside the AgentEval umbrella via
ProjectReference PrivateAssets="all", and this harness uses only public API --
no InternalsVisibleTo grant exists -- so 0.18.0-beta from nuget.org satisfies
it with no change to AgentEval and no new release. It also version-pins the
evaluator, so a benchmark run stops depending on whatever sits in a sibling
working tree.
2. Existing databases would have silently duplicated facts. Facts now MERGE on
{subject_key, predicate_key, object_key, owner_key}; main has no *_key property
anywhere, so on a pre-1.4 store those match nothing and every upsert of an
existing fact creates a duplicate. BackfillCanonicalFactKeysAsync fixes it but
runs only inside BootstrapAsync, and ISchemaBootstrapper is TryAddTransient --
the library never invokes it.
Deliberately NOT fixed with a write-path scan or an auto-running hosted
service: BootstrapAsync is the documented startup step, present in README,
getting-started, agent-framework and every sample, so the backfill does run for
anyone following the docs. Adding a per-write legacy scan would trade a
contract violation for a permanent hot-path cost -- a new defect to fix an old
one. Instead the state is made visible where an operator looks for it:
schema-check now reports pending canonical keys and exits non-zero, and the
CHANGELOG carries an explicit upgrade note.
3. A documented extension point crashed on upgrade.
Neo4jMemoryPersistenceTransaction hard-cast the public INeo4jTransactionRunner
to the later-added INeo4jAtomicTransactionRunner and threw. That runner is
registered with TryAddSingleton precisely so hosts can substitute their own,
and this type is registered with Replace, so it receives whatever they
supplied: a substitution that was legal when written became a startup crash,
with no compile-time signal.
Atomicity is optional by design -- IMemoryPersistenceTransaction carries
SupportsAtomicRollback and PersistenceStage already branches on it -- so a
non-atomic runner now degrades to pass-through and reports false, rather than
refusing to start or claiming a rollback guarantee it cannot keep. Pinned by
four tests including the two failure modes that matter: it must not throw, and
it must not report atomicity it does not have.
Release build of the full solution: 0 warnings, 0 errors.
3825 core + 215 harness unit tests green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
joslat
added
perf-counter-change
The perf gate failed on PERF-W-05 bytes_est +23% (43218 -> 53274). It is not a regression, and the override was the wrong tool for it. The baseline predates the batched UNWIND upserts and batched embeddings that are on this branch. Re-recording shows what actually moved: PERF-W-05 queries 278 -> 8, tx.write 257 -> 1, embed.requests 7 -> 3 PERF-W-03 queries 88 -> 13, tx.write 48 -> 7, bytes_est -5,108 PERF-W-02 queries 43 -> 8, tx.write 18 -> 2, bytes_est -3,828 Two of the three write scenarios drop bytes as well as round trips. PERF-W-05 rises because 257 tiny writes become one batch: the payload concentrates into a single round trip rather than shrinking, which is the trade batching makes. Deliberately NOT resolved with the perf-counter-change label. That override exists for structural counters and the gate excludes bytes_est from it by an explicit `continue` before the override branch is reached -- payload is held to a harder standard on purpose. Forcing it would have meant defeating a guard rather than updating a stale baseline. The label has been removed from the PR. Only four counters increase anywhere: PERF-W-05 bytes_est as above, and PERF-R-04/07/08 bytes_est by 1,034 each (+0.7%, inside the 5% tolerance, with recall.chars +5 from the added ResolvedQueryRelations field). Four scenarios that exist on the branch but had no baseline entry are now recorded: PERF-W-06, W-07, W-08, W-09. Quality is unchanged -- every numeric metric identical, tolerance still 0. `perf baseline --update` dropped two non-numeric annotations, retrievalMeasurement "deterministic-plumbing" and semanticQualityClaim false, which exist precisely so that a 1.000 is not read as a quality claim. Both restored by hand; losing them silently would have been the more damaging half of this commit. Gate verified locally against the fresh run: perf gate: PASS. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
joslat
removed
the
perf-counter-change
This was referenced Aug 13, 2026
This was referenced Aug 27, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
The retrieval-quality measurement track: a LongMemEval benchmark harness, the instrumentation to make
its results trustworthy, and the product fixes that work uncovered.
41 commits · 276 files · ~35k insertions. Release build 0 warnings; 4,100 unit tests and 310
Neo4j-backed integration tests green.
The headline result
The first statistically significant arm comparison this project has produced, at n=50, paired:
41 of 50 questions are concordant and carry no information; the entire signal is 9 discordant pairs,
8 favouring Hybrid. McNemar exact p = 0.0391. The measured price of +14 points is 2.5x tokens.
Why the instrumentation was necessary
Cold-build extraction variance was never measured. It is sd 9.3 points, implying +/-14.9 points
on a cold-vs-cold difference at n=30 — wider than most of the effects previously claimed. Several
earlier conclusions are inside that band and are recorded as such rather than carried forward.
Two instruments now exist that do not depend on the judge:
it moves more than 4 questions at n=50.
count()per question. It separates "the relationwas never extracted" from "it was extracted and retrieval left some behind", which an accuracy
score cannot.
Product fixes included
AddAgentMemoryCore(MemoryOptions)/AddNeo4jAgentMemory(MemoryOptions, ...)—MemoryOptionsis an init-only record, so the documented
Action<MemoryOptions>could not set any property. Theshipped BlendedAgent sample was running with GraphRAG off. Purely additive; sample and README fixed.
transient. Permanent 4xx are not retried; oversized requests split instead.
oversized batch and re-sent to the endpoint that had just failed.
four sites.
INeo4jTransactionRunnersubstitution no longer throws — a public,TryAdd-registered seamwas hard-cast to a later-added interface. It now degrades to pass-through and reports
SupportsAtomicRollback = falsehonestly.changed. Kept, but not to be pointed at a memory-native index.
Upgrade note
Facts are now identified by canonical keys and MERGE on them. Call
ISchemaBootstrapper.BootstrapAsync()before writing — as getting-started and every sample alreadydo — or an upsert of an existing fact will not match it and will create a duplicate.
schema-checknow reports this state and exits non-zero. See CHANGELOG.
Review
An 18-agent adversarial pre-merge review raised 12 claims; a verify pass refuted or downgraded 5.
The three that survived are fixed in
193f823: CI would have been permanently red (the harnesscarried a ProjectReference to an absolute path on one machine — now a version-pinned
AgentEvalpackage), the fact-key migration gap above, and the transaction-runner cast.
AgentMemory.Tests.Integration.Nams.*skip withoutNAMS_API_KEY/NAMS_DEV_WORKSPACE_ID; theyfail locally only because those credentials point at a deprovisioned workspace. This branch touches
no NAMS code.
🤖 Generated with Claude Code
https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
Performance counter change
Perf counter change justification: PERF-W-05 neo4j.bytes_est rises 43218 -> 53274 (+23%) because batched UNWIND upserts and batched embeddings replace per-item writes -- the same scenario drops neo4j.queries 278 -> 8, neo4j.tx.write 257 -> 1 and embed.requests 7 -> 3, so payload concentrates into one round trip instead of being spread across 257; the baseline predates that batching work.
The gate compares one counter at a time, so it sees the payload rise without the round-trip collapse
that causes it. The full picture for PERF-W-05 on this branch:
neo4j.queriesneo4j.tx.writeembed.requestsneo4j.recordsneo4j.bytes_estThe baseline is not being re-recorded here: the labelled-and-justified override is the mechanism this
repository already provides for a deliberate structural counter change, and it keeps the old number
visible for the next comparison rather than quietly replacing it.