Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
acfdc1c
Scope the canonical preflight guard to the untyped sample
joslat Aug 12, 2026
2424709
Dataset locator, schema-tolerant manifests, and the flaw a real corpu…
joslat Aug 12, 2026
1016432
Wire abstention sampling: the class the sufficiency AUC has to order …
joslat Aug 12, 2026
44203f7
Label answerability from the dataset, not from the extraction-failure…
joslat Aug 12, 2026
ac7e7c6
Show the abstention policy in the corpus listing
joslat Aug 12, 2026
5028859
2.11: rescue the short-but-non-empty owner result with the scan, not …
joslat Aug 12, 2026
9c9b005
6.3: give reasoning traces a trust level
joslat Aug 12, 2026
82f46f5
1.5/1.7: the valid-time falsifier, and what it found
joslat Aug 12, 2026
fe54e76
4.6a: survive provider content refusals instead of losing the whole b…
joslat Aug 12, 2026
cc34824
Isolate a refused batch instead of skipping it, and record which session
joslat Aug 12, 2026
e9ef85d
2.12: the short-owner rescue on all four scoped vector paths
joslat Aug 12, 2026
e563b6b
4.6/4.6b: the AUC reports, and whether the refusals cost anything
joslat Aug 12, 2026
0a43bba
Report abstention accuracy: the meta-memory number, measured for the …
joslat Aug 12, 2026
30bc647
10.1/10.2: the reranker seam, and node-distance reranking on it
joslat Aug 12, 2026
176a65b
10.3: mention-frequency reranking, on the ingestion-side signal
joslat Aug 12, 2026
a3f25ea
2.4: take the access-tracking write off the blocking recall path
joslat Aug 12, 2026
7a65974
2.5: stop re-sending the turns the host is already sending
joslat Aug 12, 2026
43d7d45
12.4: confidence that earns and loses
joslat Aug 12, 2026
bbe7749
Wire the second supersede caller for S2
joslat Aug 12, 2026
f2050b1
Make bitemporal recall reachable from a natural-language turn (R4)
joslat Aug 12, 2026
133be90
Stop paying for extraction on turns that cannot carry a fact (E4)
joslat Aug 12, 2026
b77a1fb
Extract with preceding turns as context, never as targets (E2)
joslat Aug 12, 2026
55bbcd0
Let the graph probe see procedural memory (PLAN 6.5)
joslat Aug 12, 2026
6b6c995
Measure whether the retrieved procedure is the right one (PLAN 7.7)
joslat Aug 12, 2026
9ff97c4
Make the StructuredJson judge protocol reachable and self-declaring (…
joslat Aug 12, 2026
1cfea43
Synthesize entity summaries that cannot go stale unnoticed (S1)
joslat Aug 12, 2026
d034d73
Make memory legible without making the block the store (S4)
joslat Aug 12, 2026
72c29cf
Document the bulk-ingestion path and bound its concurrency (rank 27)
joslat Aug 12, 2026
5802411
Degrade recall under a latency budget, visibly (rank 13)
joslat Aug 12, 2026
369fb97
Measure what memory saves in tokens, honestly (B3)
joslat Aug 12, 2026
acdc8c3
Build the harness that can tell procedural memory it failed (7.6)
joslat Aug 12, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 18 additions & 3 deletions tools/AgentMemory.LongMemEval/LongMemEvalPreparedPairProgram.cs
Original file line number Diff line number Diff line change
Expand Up @@ -318,14 +318,29 @@ await LongMemEvalOrphanSweep
plans.Sum(plan => plan.SourceSessionCount);
var plannedInputTokens =
plans.Sum(plan => plan.TotalEstimatedInputTokens);
if (options.Questions == DefaultQuestions &&
options.Seed == DefaultSeed &&
plannedSourceSessions != FixedTenExpectedSourceSessions)
// The canonical guard pins the STRATIFIED ten to a recorded plan, so a change in
// sampling or batching cannot pass unnoticed. A typed sample is a different plan by
// construction -- --memory-types episodic selects different questions, with a
// different number of source sessions -- so the canonical number does not describe
// it. Skipped rather than adjusted, and said out loud: silently dropping a guard that
Comment on lines +348 to +352
// exists to catch sampling drift would be worse than not having it.
var canonicalPlan = options.Questions == DefaultQuestions
&& options.Seed == DefaultSeed
&& options.MemoryTypes.Count == 0;
if (canonicalPlan && plannedSourceSessions != FixedTenExpectedSourceSessions)
{
throw new InvalidOperationException(
$"Canonical fixed-ten preflight produced {plannedSourceSessions} " +
$"source sessions; expected exactly {FixedTenExpectedSourceSessions}.");
}
if (!canonicalPlan && options.MemoryTypes.Count > 0)
{
Console.WriteLine(
"longmemeval: typed sample "
+ $"({string.Join("+", options.MemoryTypes)}) -- the canonical fixed-ten "
+ $"source-session guard ({FixedTenExpectedSourceSessions}) does not apply, "
+ "because a typed sample selects different questions by construction.");
}
Console.WriteLine(
$"longmemeval: frozen preparation preflight {plannedCalls} calls for " +
$"{plannedSourceSessions} source sessions and " +
Expand Down