Close the benchmark's description leak, and record what it did not fix

[joslat]

Removed every prerequisite from the benchmark's tool and parameter descriptions, and added two guards keeping them out — the check the first run needed and didn't have, since enforcing an ordering in code the model never reads isn't enforcement.

Re-ran. Identical result: zero refusals, three tool calls, both arms the same.

So the leak was never only in the prose. PlaceHold(connection, tier) and Book(holdReference) telegraph the chain through their parameter names — a model that sees a tier argument it can't fill goes looking for the tool that yields one. The information is in the signature, and a signature can't be obfuscated without making the task artificial in a different way.

The honest conclusion is about the task class, not the wording: a three-step chain over semantically-named tools is not a procedural-memory benchmark for a competent model, because there's nothing to discover.

A discriminating task needs an ordering the model can't infer from names or types — an opaque token from a tool whose name doesn't suggest it, or a chain whose branch depends on a value found mid-run, so a stored procedure encodes a decision rather than a sequence. Both are real design work, and both risk the opposite failure: a task so artificial that succeeding says nothing about agents doing real work.

Harness, runner, promotion path and arm switch are built, tested, and demonstrated working across two runs. What's missing is a task hard enough to measure them with. Neither run's figures should be cited as evidence in either direction.

451 LongMemEval green (+2 guards).

🤖 Generated with Claude Code

https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE

[Copilot]

Pull request overview

This PR tightens the procedural-memory benchmark against “description leaks” by removing prerequisite-hinting language from tool/parameter descriptions and adding unit-test guards intended to prevent those hints from being reintroduced.

Changes:

• Removed prerequisite language from ProceduralBenchmarkTask tool and parameter descriptions to avoid advertising the intended call chain.
• Added a shared list (ChainRevealingWords) of chain-revealing phrases and a unit test that asserts tool descriptions do not contain them.
• Added a unit test asserting generated JSON schema text does not include specific chain-revealing phrases in parameter descriptions, plus documentation capturing the (unchanged) second-run outcome.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
tools/AgentMemory.LongMemEval/ProceduralBenchmarkTask.cs	Removes prerequisite-hinting language from tool and parameter descriptions; introduces a shared “chain revealing words” list.
tests/AgentMemory.Tests.Unit.LongMemEval/ProceduralBenchmarkTaskTests.cs	Adds tests intended to guard against chain-leaking wording in tool descriptions and parameter schema.
docs/reviews/procedural-benefit-run-prerequisite.md	Records the second-run conclusion that the benchmark still doesn’t discriminate even after description cleanup.

Suppressed comments (1)

tests/AgentMemory.Tests.Unit.LongMemEval/ProceduralBenchmarkTaskTests.cs:151

• The parameter-schema leak guard is also case-sensitive (NotContain("returned by"), NotContain("from the traveller")). Capitalization changes in descriptions (e.g., "Returned by") would bypass this test while still leaking the chain.

        var schema = new ProceduralBenchmarkTask().CreateTools()
            .Select(t => ((AIFunction)t).JsonSchema.ToString())
            .ToList();

        foreach (var json in schema)
        {
            json.Should().NotContain("returned by");
            json.Should().NotContain("from the traveller");
        }

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.