TypedMemEval prerequisites: a timestamped history channel, an embedded-corpus evidence index, and a verb to run them - #205
Merged
Merged
Conversation
…0.9c) 0.22.0-beta is the TypedMemEval release: five embedded corpora, typed outcomes, ITimestampedHistoryInjectableAgent. The pin compiled with zero source changes, matching the free-checks finding. 0.23.0-beta will supersede it soon with revised corpora and new ids/hashes, so the rule recorded in the csproj comment binds everything that follows: no corpus SHA-256 or question-count literal anywhere in this harness -- counts and identifiers are read from TypedMemEvalCorpus and TypedMemEvalVerticals at run time, and the version transition is one line here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
…me (30.9c prereq B)
The envelope pipeline only knew how to build its evidence index from a
dataset FILE (LongMemEvalEvidenceIndex.Load(datasetPath, ...)).
TypedMemEval corpora are embedded resources, so every typed run
necessarily ran without an index, attached no envelope, and reported
attribution Unobserved -- the exact gap the free-checks pass named.
CreateTypedMemEval(vertical, facade) closes it: entries come from
TypedMemEvalCorpus.Load under an ExternalBenchmarkOptions mapping that
replicates AgentEval's INTERNAL TypedMemEvalOptions.ToExternalOptions
(stratified sampling on, session boundaries preserved, corpus id as
dataset mode with the -control suffix, the vertical's required grounding,
TextBlob forced up to structured under grounding). A replica without a
drift guard is a lie waiting for the next release, so a test invokes the
real internal mapper by reflection and holds the replica to
property-for-property equality across all five verticals and both arms.
Grounded verticals need a second construction path, because the runner
injects them through the TIMESTAMPED channel: the formatted content
comes from FormatTimestamped (the boundary marker drops its date under
TimestampsOnly, so an index built from the dated Format output would
never align), the invocation prompt is grounding-aware (bare question
under TimestampsOnly), and the fingerprint is salted with the history's
QueryTime. The salt is identity, not metadata: Prospective pair arms are
one haystack and one question text asked at two instants -- in the v1
corpus the arms happen to differ in one padding turn, but the question
texts are already identical, so turns-only fingerprints are one padding
shuffle away from making the two arms unresolvable.
Proven red first, each by breaking its seam and watching exactly the
right tests fail:
- envelope attachment disabled in the adapter (the original defect
shape) -> the offline stub-agent + stub-judge Forgetting run reported
attribution Unobserved and failed its assertions;
- QueryTime dropped from the fingerprint -> the identical-haystack
synthetic pair collided ("found 2") while the real-corpus alignment
test kept passing, confirming the synthetic pair is the discriminating
case;
- replica drifted (StratifiedSampling=false) -> the reflection guard
failed AND the offline run broke mid-pipeline on mis-selection;
- Current Date prefix kept under TimestampsOnly -> both Prospective
resolve tests failed.
The offline test is the free-checks probe made permanent: Forgetting,
MaxQuestions=4, stub judge answering {"outcome":"abstained"}, no
provider, no container -- and with the index wired, not one question
reports the attribution channel as unobserved.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
…Time as the recall anchor (30.9c prereq A) The free-checks pass found ITimestampedHistoryInjectableAgent implemented NOWHERE -- the tg work delivered dates as text, and AgentEval's Prospective vertical refuses to run without the typed channel, before its first provider call, because the in-text dates are precisely what TimestampsOnly grounding takes away. The adapter now implements it, with three semantics: 1. Each turn's Timestamp becomes the stored messages' own clock (Message.TimestampUtc -- the product's valid time) for both halves of the pair, threading through the same BuildMessages path the plain channel uses. The untimestamped channel keeps its epoch+ordinal clock so every sealed measurement stays comparable with itself. 2. No date text is EVER added to message content. Dates are structural here; a harness that re-printed them into the text would hand back the crutch the mode exists to remove. 3. The history's QueryTime is the answer-time anchor: recall runs through RecallAsOfAsync with QueryTime as the VALID-time clock (a fact's validity window is judged against the question's now -- the prospective-memory semantics under measurement) and the machine's now as the TRANSACTION clock, because the corpus was ingested moments ago and a 2023 transaction bound would erase everything just stored. The answer prompt's "Current date" line renders from the same QueryTime -- data the system under test legitimately holds -- not from the evaluator-side index. Timestamped injection REFUSES options the point-in-time recall path does not implement (predicate expansion, query-relation resolution, GraphRAG): accepting them would run a question whose configuration silently did nothing -- the dead-option shape. Evidence resolution goes through the QueryTime-salted Resolve so pair arms stay attributable. Every test proven red first by breaking its seam: - epoch clock kept in BuildMessages -> the valid-time test failed; - recall forced through RecallAsync -> the anchor test failed; - injection prefixed [yyyy/MM/dd] onto content -> the no-date-text assertion failed; - reset left the timestamped state behind -> the reset test failed, but ONLY after being strengthened: its first version invoked before resetting, and InvokeAsync consumes the pending state itself, so that sequence could not observe the break. The committed test uses the injected-but-never-invoked sequence -- the runner's shape when a question dies between injection and the agent call -- where reset is load-bearing; - the dead-option guard and the double-injection guard each removed -> exactly their tests failed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
…bands (30.9c prereq C) Dispatched like every other verb -- before the default parser, so the switch can never fall through and silently measure something else -- and listed in the main verb's KnownOptions for the same reason. --typedmemeval <vertical|all> runs the structured memory arm (fresh adapter, fresh embedded-corpus evidence index, and a distinct runId per run over one shared container; isolation is by owner scope, not teardown). --oracle routes to RunOracleAsync under LongMemEvalOracleOptions.GoldOnly and constructs no memory stack at all. --control is the Prospective pair's control arm: same corpus, dates re-added to the text, the run labelled -control so it can never be banded with the probe by accident. Each run's native ExternalBenchmarkResult JSON lands under artifacts/evaluation/, named by vertical, arm, seed, run index, and UTC stamp; past one run the TypedMemEvalRunSet.Summarize band prints with QuestionsWithFlips -- the number that says whether a narrow band is evidence. Two refusals happen at parse time, before any container or credential is touched: --runs above 1 without --random-seed (unseeded runs draw different questions; Summarize would refuse AFTER paying for every run), and --control outside prospective (a control with no pair to control). The memory arm is structured-only by design: the timestamped channel's point-in-time recall performs no semantic message search, so a raw or hybrid arm under it could never fill a message budget honestly. The KnownOptions/validator discipline is guarded the way ExtractionSeedCommandLineTests does it: a closed map from every advertised option to the record property that honours it, so an option added to the list without being carried is the validator lying and fails by name. Proven red first, seam by seam: the unseeded-band check and the control check each disabled -> exactly their tests failed; seeds routed through the positive-integer parser -> the zero/negative theory failed; the verb switch delisted from the main KnownOptions -> the advertising test failed; a bogus --output advertised -> the drift guard failed. Recorded, not hidden: a smoke of the verb's error paths assumed the AZURE_OPENAI_* variables were unset on this machine; they were set, and "--typedmemeval forgetting --max-questions 4 --random-seed 42" ran for real until killed at extraction unit 8/24 of question 1 -- one embedding probe plus ~32 extraction calls of held spend, no report produced, the container reaped. The run phase must treat any verb smoke on this machine as spend unless the variables are cleared first. Side-effect worth keeping: the verb demonstrably boots end to end (env -> corpus -> index -> container -> extraction). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
… before any spend (30.9c) The one-line bump the harness was designed for: zero API breaks, drift guards 2/2, suite 488/488. v3 SHAs match the maintainer's published prefixes for all five verticals, each meta is self-consistent, and an independent offline separability probe (pooled session-shape rank AUC) found no trivially separable gold - replicating the corpus's own V7 table where features overlap. v1 numbers stay recorded as wiring-proof only; nothing v1 is comparable with v3. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
…ts (30.9c) TypedMemEvalRunSet.DetectSeedOverlap is upstream's authority on the twelve tg-probe questions carried into Prospective; this wires it at the one place this verb aggregates results across verticals, and prints its warning verbatim. Red-first: the reachability test (comment-stripped, per the Wave C-E lesson about substring guards satisfied by comments) failed with the method defined but uncalled, and only then was the RunAsync wire added. Suite 488 -> 492. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
HEAD moved ~80 commits and added ~6,090 lines to tools/AgentMemory.LongMemEval while the six prereq commits sat on a worktree branch. Merged rather than rebased so the same three collisions are resolved once instead of six times. Every conflict was additive on both sides, and ours was adapted to HEAD's shape, never the reverse. - AgentMemory.LongMemEval.csproj: HEAD split the package group in two to pin Microsoft.Bcl.Memory 10.0.11 (29.1, GHSA-73j8-2gch-69rq) and had moved AgentEval to 0.21.0-beta. Kept HEAD's two-group layout and the security pin, and kept our 0.23.0-beta AgentEval pin on top of it: 0.23.0-beta is the v3-corpora release, and v1's corpora carry gold-separability tells that must not be cited. It is unlisted on nuget.org, but unlisting is not deletion and the explicit pin still restores -- verified here, project.assets.json resolves AgentEval/0.23.0-beta. - Program.cs, KnownOptions: both sides appended to the same list. Kept every entry from both; --typedmemeval keeps its place beside the other verb switches. The dispatch block itself auto-merged and sits after --prepared-pair, still ahead of the parser, which is what keeps an unrecognised verb from falling through to the default arm. - AgentMemoryLongMemEvalAdapter.cs, the retrieval call: a genuine overlap rather than two appends. We had hoisted the RecallRequest into a local so the call site could choose between RecallAsync and RecallAsOfAsync; HEAD had introduced 27.4's retrievalQuery and rewritten the same RecallRequest inline with Query = retrievalQuery. Kept the hoisted local, moved the formulator above it so retrievalQuery is what the local carries, and left the QueryTime-anchored call site as ours. Both features are live: a formulated query, recalled as of the question's own instant. LongMemEvalEvidenceIndex.cs merged without conflict despite being listed as one; only the csproj, Program.cs and the adapter collided. Release build of AgentMemory.slnx is 0 warnings / 0 errors. The harness suite is 676 passing: HEAD's 639 plus our 37, with nothing lost -- the 37 was measured by filter against the four new classes, not inferred from the difference. Both drift guards pass by name: OptionMappingMatchesAgentEvalsInternalMapper_ForEveryVerticalAndArm, which is also the check that the 0.23.0-beta mapping still matches AgentEval's own, and CreateTypedMemEval_AlignsWithTheRunnersTimestampedInjection_ForProspective. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
There was a problem hiding this comment.
Pull request overview
Adds the missing harness-side prerequisites to run AgentEval’s TypedMemEval benchmark family inside this repo’s LongMemEval tooling, without changing src/ (adapter support for timestamped injection, embedded-corpus evidence indexing, and a dedicated --typedmemeval verb), plus a wired guard to warn on overlapping seed sets.
Changes:
- Introduces
--typedmemevalverb implementation with option parsing, multi-run banding, report persistence, and AgentEval’s seed-overlap warning. - Adds evidence-index construction for TypedMemEval’s embedded corpora, including timestamped-history alignment and QueryTime-salted fingerprints for Prospective pair attribution.
- Extends the LongMemEval adapter to implement
ITimestampedHistoryInjectableAgentand route timestamped runs through point-in-time recall (RecallAsOfAsync) with the TypedMemEval clocks.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tools/AgentMemory.LongMemEval/TypedMemEvalProgram.cs | Implements the --typedmemeval verb, vertical orchestration, run persistence, and seed-overlap warning wiring. |
| tools/AgentMemory.LongMemEval/TypedMemEvalOptionMapping.cs | Replicates AgentEval’s internal facade→external options mapping for evidence-index alignment, with drift guarded by tests. |
| tools/AgentMemory.LongMemEval/Program.cs | Adds verb dispatch, KnownOptions entry, and help text for --typedmemeval. |
| tools/AgentMemory.LongMemEval/LongMemEvalEvidenceIndex.cs | Adds timestamped and TypedMemEval embedded-corpus evidence index creation, including QueryTime-salted fingerprints and prompt alignment. |
| tools/AgentMemory.LongMemEval/AgentMemoryLongMemEvalAdapter.cs | Implements timestamped history injection and anchors retrieval/prompt “now” to QueryTime via RecallAsOfAsync. |
| tools/AgentMemory.LongMemEval/AgentMemory.LongMemEval.csproj | Pins AgentEval to 0.23.0-beta (TypedMemEval v3 corpora). |
| tests/AgentMemory.Tests.Unit.LongMemEval/TypedMemEvalSeedOverlapGuardTests.cs | Tests the seed-overlap warning is both correct and actually wired into result assembly. |
| tests/AgentMemory.Tests.Unit.LongMemEval/TypedMemEvalEvidenceWiringTests.cs | Guards option-mapping drift and verifies embedded-corpus evidence wiring yields observed attribution offline. |
| tests/AgentMemory.Tests.Unit.LongMemEval/TypedMemEvalCommandLineTests.cs | Verifies CLI parsing, defaults, and that every advertised option is carried onto the options record. |
| tests/AgentMemory.Tests.Unit.LongMemEval/TimestampedHistoryInjectionTests.cs | Pins timestamped-channel semantics: valid-time timestamps, QueryTime-anchored recall, and state reset behavior. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+259
to
+262
| var turn = history.Turns[index]; | ||
| pairs[index] = (turn.UserMessage, turn.AssistantResponse); | ||
| timestamps[index] = turn.Timestamp; | ||
| } |
Trivial: main's extra commit is the merge of this branch's own base, so this brings in no code the branch has not already seen. Content diff against main is unchanged at the same 10 harness files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
This was referenced Aug 27, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The three things the TypedMemEval benchmark family needs from our side before a
single number can be produced, plus the guard AgentEval asked us to wire. All of
it lives in
tools/AgentMemory.LongMemEval/; nosrc/change.Consumed design:
AgentEvalADR-026 (TypedMemEval), our prompt 06 v2 hand-off,and the free-checks pass that established these three were missing.
What this adds
A —
ITimestampedHistoryInjectableAgenton the adapter. Prospective refusesto run without it, and nothing in the repo implemented it: the earlier
time-grounded work delivered dates as text, which is exactly what the vertical
exists to take away. Each turn's timestamp becomes valid time on the ingested
message; the history's
QueryTimeanchors the read throughRecallAsOfAsync(asOf: QueryTime, systemAsOf: UtcNow)— the question's own "now"on the valid clock, the machine's now on the transaction clock. No date text is
ever appended to message content. Injection refuses predicate expansion, query
relations and GraphRAG rather than silently ignoring them, since the as-of path
does not implement them.
B — an evidence index built from embedded corpora. The envelope path
previously required a dataset file; TypedMemEval corpora are embedded
resources, so every question reported attribution
Unobserved.CreateTypedMemEval(vertical, facade)builds entries fromTypedMemEvalCorpus.Loadunder a faithful replica of AgentEval's internal optionmapping, with a QueryTime-salted fingerprint — without the salt, a
before/after pair sharing one haystack collides.
C — the
--typedmemevalverb.<slug|all>,--max-questions,--random-seed,--answer-seed,--runs N(withTypedMemEvalRunSetbandreporting incl.
QuestionsWithFlips),--oracle(gold-only ceiling),--control(the Prospective control arm). Parse-time refusals for--runs>1without a seed and
--controloff Prospective; a KnownOptions-vs-record driftguard in the established shape.
D — AgentEval's
DetectSeedOverlapguard, wired where the verb assemblesresults, so a report that contains both the time-grounded corpus and
TypedMemEval-Prospective warns about the twelve shared questions instead of
double-counting them.
Verification
AgentMemory.slnx: 0 warnings, 0 errorswere measured with a filter over the four new test classes, not inferred from
the difference, so a test that vanished in the merge could not hide in the
arithmetic
OptionMappingMatchesAgentEvalsInternalMapper_ForEveryVerticalAndArm(whichalso corroborates the pin, since it compares against AgentEval's own internal
mapper at 0.23.0-beta) and
CreateTypedMemEval_AlignsWithTheRunnersTimestampedInjection_ForProspectiveMerge notes
Three real conflicts against main, one of which was not the append-vs-append the
dry run predicted. In
AgentMemoryLongMemEvalAdapter.cs, this branch hoisted theRecallRequestinto a local so the call site could choose betweenRecallAsyncand
RecallAsOfAsync, while main's query-formulation work had rewritten thatsame request inline with
Query = retrievalQuery. Keeping "both sides" wouldhave produced either a dead local or a silently dropped formulator; resolved by
computing
retrievalQueryabove the hoisted local and feeding it in, so aformulated query is recalled as of the question's own instant and both features
are live. The
.csprojalso carried more than expected — main had split thepackage group for a
Microsoft.Bcl.Memorysecurity pin, which is preserved.The pin
AgentEval 0.23.0-betais unlisted on NuGet (its v3 corpora were withdrawnafter a separability defect that our own consumer probe helped surface).
Unlisting is not deletion: an explicit pin restores,
project.assets.jsonresolves it, and the drift guard passes against it. When v4 publishes this
becomes the one-line bump the design promised — and only then can a citable
number be produced.