Build the task procedural memory is measured against (7.6)

[joslat]

A benchmark for procedural memory has one hard requirement: the shortest correct path must be discoverable but not guessable. If an agent finishes on attempt one by calling the obvious tool, there is nothing to learn, both arms tie, and the harness reports "procedural memory does not help" for a reason entirely about the task rather than the feature.

So the chain is enforced: booking requires a hold, a hold requires the traveller's loyalty tier, and the tier lives behind a lookup the prompt never mentions. An agent meeting this cold discovers it by being refused; an agent with the procedure stored walks it. That gap is the whole signal.

Tools refuse politely and name the missing step rather than throwing — a hard failure ends the run instead of teaching it, and a control arm that cannot learn the chain cold measures the harness instead of the agent.

Deterministic environment. The agent is the only nondeterministic part; at three attempts per arm there's nowhere near the sample to separate a varying world from a memory effect — the variance would just be reported as one.

Completion is the marker the booking tool emits, never the agent claiming it finished. A wrong procedure is reported fluently — invisible to every efficiency number, which is why 7.7 scores wrong-procedure rate separately.

Seven provider-free tests: booking directly is refused, a hold without the tier is refused, completion cannot be claimed in prose. What remains for 7.6 is only the measured run.

4,370 unit and 443 LongMemEval green (+7).

🤖 Generated with Claude Code

https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE

[Copilot]

Pull request overview

Adds a deterministic, multi-step “procedural benchmark” task plus unit tests intended to validate the enforced tool-call chain used by the 7.6 procedural-memory benefit harness.

Changes:

• Introduces ProceduralBenchmarkTask with three tools (lookup → hold → book), refusal messaging, and a completion marker.
• Adds unit tests asserting refusals for shortcuts and basic determinism of the task environment.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
tools/AgentMemory.LongMemEval/ProceduralBenchmarkTask.cs	New procedural benchmark task with tool chain, refusals, and completion predicate.
tests/AgentMemory.Tests.Unit.LongMemEval/ProceduralBenchmarkTaskTests.cs	New unit tests validating the chain behavior and determinism.

Suppressed comments (6)

tools/AgentMemory.LongMemEval/ProceduralBenchmarkTask.cs:53

• IsComplete currently checks only for the marker substring. Even with a hidden marker, this makes it easy to accidentally mark runs complete when the agent includes extra text, and it doesn’t bind completion to the specific booking reference. Tightening the predicate to the exact booking-tool output reduces false positives and better matches the docstring (“only emit by completing the real chain”).

    internal bool IsComplete(string response) =>
        response.Contains(ConfirmationMarker, StringComparison.Ordinal);

tools/AgentMemory.LongMemEval/ProceduralBenchmarkTask.cs:76

• To ensure “refused, not thrown” behavior when a model tries place_hold without the loyalty tier (or book without a hold reference), the parameters should be optional in the tool schema. As written, tier / holdReference are required, so the framework may reject the call before your refusal message can teach the missing step.

    [Description("Places a hold on a connection. Requires the traveller's loyalty tier.")]
    private string PlaceHold(
        [Description("The connection time, e.g. 14:05.")] string connection,
        [Description("The traveller's loyalty tier, from the traveller lookup.")] string tier)

tools/AgentMemory.LongMemEval/ProceduralBenchmarkTask.cs:90

• Same as PlaceHold: if the agent calls book without a hold reference, a required parameter can cause a schema/argument error before your refusal message executes. Making holdReference optional keeps the “polite refusal that names the missing step” property.

    [Description("Confirms a booking. Requires a hold reference.")]
    private string Book(
        [Description("The hold reference returned by the hold tool.")] string holdReference)
    {

tools/AgentMemory.LongMemEval/ProceduralBenchmarkTask.cs:63

• The tool descriptions currently spell out the dependency chain (“Required before…”, “Requires…”), which makes the shortest correct path guessable from the tool schema alone (no refusal/discovery needed). If the benchmark’s goal is to force discovery via refusals, consider making the descriptions neutral and relying on refusal messages to reveal prerequisites.

    [Description("Looks up a traveller's loyalty tier. Required before a hold can be placed.")]

tools/AgentMemory.LongMemEval/ProceduralBenchmarkTask.cs:73

• Same concern as above: this description advertises the missing prerequisite up front, which may let an agent succeed attempt 1 without ever being refused (weakening the intended “discoverable but not guessable” property).

    [Description("Places a hold on a connection. Requires the traveller's loyalty tier.")]

tests/AgentMemory.Tests.Unit.LongMemEval/ProceduralBenchmarkTaskTests.cs:112

• After tightening IsComplete to bind completion to the booking tool output (marker + reference), this assertion will become brittle because it hardcodes a specific hold reference. It’s enough here to assert that prose without the exact booking output does not complete the task; the positive completion case is already covered by TheChainCompletesWhenWalkedInOrder.

        task.IsComplete("I have successfully completed the booking for you!").Should().BeFalse();
        task.IsComplete($"{ProceduralBenchmarkTask.ConfirmationMarker} ref HOLD-4417").Should().BeTrue();
    }

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.