Close the benchmark's description leak, and record what it did not fix

docs/reviews/procedural-benefit-run-prerequisite.md

@@ -89,3 +89,39 @@ Withhold the prerequisites from the descriptions — name each tool's purpose an
 message teach the ordering. Then the control arm must discover the chain by being refused on every
 attempt, while the procedural arm pays that cost once. Re-run after that change; the current numbers
 should not be cited.
+
+
+---
+
+## Second run: descriptions cleaned, still no discrimination
+
+Removed every prerequisite from the tool and parameter descriptions, and added two guards asserting
+they stay out. Re-ran. **Identical result — zero refusals, 3 tool calls, both arms the same.**
+
+So the leak was never only in the prose. `PlaceHold(connection, tier)` and `Book(holdReference)`
+telegraph the chain through their **parameter names**: a model that sees a `tier` argument it cannot
+fill goes looking for the tool that yields one. The information is in the signature, and a signature
+cannot be obfuscated without making the task artificial in a different way.
+
+**The honest conclusion: a three-step chain over semantically-named tools is not a procedural-memory
+benchmark for a competent model.** There is nothing to discover. This is a property of the task
+class, not of wording, and no amount of description-editing fixes it.
+
+### What a discriminating task actually needs
+
+An ordering the model cannot infer from names or types — the dependency has to be **arbitrary**:
+
+- An opaque token obtainable only from a tool whose name does not suggest it (e.g. `book` requires a
+  `clearanceCode` that only `check_weather` returns), so the chain is learnable but not guessable; or
+- a longer chain where the *branch* taken depends on a value discovered mid-run, so a single stored
+  procedure encodes a decision rather than a sequence.
+
+Both are real design work, and both risk the opposite failure — a task so artificial that succeeding
+at it says nothing about agents doing real work. That tension is why this is a task-design problem
+and not a wording problem.
+
+### Status
+
+The harness, runner, promotion path and arm switch are all built, tested, and demonstrated working
+end to end across two runs. **What is missing is a task hard enough to measure them with.** The
+figures from both runs should not be cited as evidence about procedural memory in either direction.

tests/AgentMemory.Tests.Unit.LongMemEval/ProceduralBenchmarkTaskTests.cs

@@ -111,6 +111,46 @@ public void CompletionCannotBeClaimedInProse()
         task.IsComplete($"{ProceduralBenchmarkTask.ConfirmationMarker} ref HOLD-4417").Should().BeTrue();
     }
 
+    [Fact]
+    public void ToolDescriptionsDoNotGiveAwayTheChain()
+    {
+        // THE guard the first run needed and did not have. Enforcing the ordering in the tool BODIES
+        // while announcing it in the DESCRIPTIONS makes the difficulty invisible to the only thing
+        // that matters -- the model reads descriptions, not code. The first run proved it: zero
+        // refusals, minimal tool count, both arms identical, and a "no benefit" result that was
+        // entirely about the task.
+        var descriptions = new ProceduralBenchmarkTask().CreateTools()
+            .Select(t => ((AIFunction)t).Description ?? string.Empty)
+            .ToList();
+
+        foreach (var description in descriptions)
+        {
+            foreach (var giveaway in ProceduralBenchmarkTask.ChainRevealingWords)
+            {
+                description.Should().NotContain(
+                    giveaway,
+                    "a description naming a prerequisite lets the model skip discovery: '{0}'",
+                    description);
+            }
+        }
+    }
+
+    [Fact]
+    public void ParameterDescriptionsDoNotGiveAwayTheChainEither()
+    {
+        // The same leak one level down. "The hold reference returned by the hold tool" is the chain,
+        // written out, in a place it is just as readable.
+        var schema = new ProceduralBenchmarkTask().CreateTools()
+            .Select(t => ((AIFunction)t).JsonSchema.ToString())
+            .ToList();
+
+        foreach (var json in schema)
+        {
+            json.Should().NotContain("returned by");
+            json.Should().NotContain("from the traveller");
+        }
+    }
+
     [Fact]
     public void ToolsAreExposedInProcedureOrder()
     {

tools/AgentMemory.LongMemEval/ProceduralBenchmarkTask.cs

@@ -52,6 +52,17 @@ internal sealed class ProceduralBenchmarkTask
     internal bool IsComplete(string response) =>
         response.Contains(ConfirmationMarker, StringComparison.Ordinal);
 
+    /// <summary>
+    /// Words that would give the chain away if they appeared in a tool description.
+    /// </summary>
+    /// <remarks>
+    /// The first run failed on exactly this: the bodies enforced the ordering and the descriptions
+    /// announced it, so the model ordered the calls correctly cold and the enforcement never fired.
+    /// A benchmark whose difficulty lives only in code the model never reads is not a benchmark.
+    /// </remarks>
+    internal static readonly string[] ChainRevealingWords =
+        ["require", "first", "before", "returned by", "from the"];
+
     /// <summary>The tools, in the order a correct procedure uses them.</summary>
     internal IReadOnlyList<AITool> CreateTools() =>
     [
@@ -60,7 +71,10 @@ internal IReadOnlyList<AITool> CreateTools() =>
         AIFunctionFactory.Create(Book),
     ];
 
-    [Description("Looks up a traveller's loyalty tier. Required before a hold can be placed.")]
+    // Descriptions state PURPOSE only. Naming a prerequisite here hands the model the chain, and
+    // the first run proved it: zero refusals, minimal tool count, both arms identical. The ordering
+    // has to be learned from the refusals, or there is nothing for a procedure to remember.
+    [Description("Looks up a traveller.")]
     private string LookUpTraveller(
         [Description("The traveller's name.")] string traveller)
     {
@@ -70,10 +84,10 @@ private string LookUpTraveller(
             : $"no traveller named '{traveller}'";
     }
 
-    [Description("Places a hold on a connection. Requires the traveller's loyalty tier.")]
+    [Description("Places a hold on a connection.")]
     private string PlaceHold(
         [Description("The connection time, e.g. 14:05.")] string connection,
-        [Description("The traveller's loyalty tier, from the traveller lookup.")] string tier)
+        [Description("The traveller's loyalty tier.")] string tier)
     {
         Calls.Add(nameof(PlaceHold));
         // Refused, not thrown. A hard failure ends the run; a refusal that names what is missing is
@@ -84,9 +98,9 @@ private string PlaceHold(
             : $"refused: a hold needs the traveller's loyalty tier; look up the traveller first";
     }
 
-    [Description("Confirms a booking. Requires a hold reference.")]
+    [Description("Confirms a booking.")]
     private string Book(
-        [Description("The hold reference returned by the hold tool.")] string holdReference)
+        [Description("The hold reference.")] string holdReference)
     {
         Calls.Add(nameof(Book));
         return string.Equals(holdReference, HoldReference, StringComparison.OrdinalIgnoreCase)