Run the procedural-benefit measurement; the result indicts the task (7.6)

[joslat]

Composed the two arms and ran them.

procedures  completion=100%  meanSteps=4.0  meanToolCalls=3.0
control     completion=100%  meanSteps=4.0  meanToolCalls=3.0
SHOWS BENEFIT: False

This is not evidence about procedural memory. Three tool calls is the minimum possible chain and the log contains zero refusals — the agent walked LookUpTraveller → PlaceHold → Book correctly on its first cold attempt in both arms. Nothing had to be discovered, so nothing a stored procedure could save was ever at stake.

The cause is my own task design, and it's exactly the property the benchmark's own tests declare as its hard requirement: the shortest correct path must be discoverable but not guessable. I enforced it in the tool bodies — booking without a hold is refused — then gave it away in the tool descriptions: "Requires the traveller's loyalty tier", "Requires a hold reference". A competent model reads those and orders the calls without ever being refused. The enforcement never fires.

Reporting that number as a finding would have been the failure this track exists to prevent: a confident negative describing the instrument rather than the feature. It's recorded as an indictment of the task, and the current figures should not be cited.

What the run does establish — the assembly, which was the open question. The arms are genuinely distinct, promotion stores a procedure, counts come off the transcript. A harness that couldn't run would have failed here rather than returning a clean uninformative zero.

The fix: withhold prerequisites from the descriptions, let the refusal messages teach the ordering, re-run.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE

[Copilot]

Pull request overview

Adds a new --procedural-benefit LongMemEval sub-verb intended to run the 7.6 “procedural-benefit” measurement (procedural memory on/off across repeated attempts), and documents the initial run outcome plus why it should not be cited.

Changes:

• Add CLI dispatch for --procedural-benefit and register the switch in the option allowlist.
• Introduce ProceduralBenefitProgram to assemble and run the two-arm measurement and print results.
• Add a review note documenting the first run and the identified benchmark-task flaw.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
tools/AgentMemory.LongMemEval/Program.cs	Adds --procedural-benefit dispatch and updates the known-options list.
tools/AgentMemory.LongMemEval/ProceduralBenefitProgram.cs	New program that composes the procedural-benefit harness and reports results.
docs/reviews/procedural-benefit-run-prerequisite.md	Documents the first run results and explains why they reflect task design rather than procedural memory.

Suppressed comments (1)

tools/AgentMemory.LongMemEval/ProceduralBenefitProgram.cs:133

• ParseAttempts silently falls back to the default when --attempts is present but missing/invalid/out-of-range. Other LongMemEval sub-verbs treat “option present but missing value” as an error (Value(...) throws). Making this strict avoids accidentally running an unintended attempt count (especially important when benchmarking).

    private static int ParseAttempts(string[] args)
    {
        var index = Array.IndexOf(args, "--attempts");
        return index >= 0 && index + 1 < args.Length
               && int.TryParse(args[index + 1], out var value) && value >= 2
            ? value
            : 3;
    }

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.