Expensive action space; the obstacle turns out to be the task class (7.6) - #200
Merged
Conversation
Fourth attempt at making the procedural benchmark discriminate, implementing what the third run's finding called for: twelve plausible decoy tools beside the four real ones, so calling everything costs sixteen invocations rather than four. Both arms still called exactly the four right tools. The model selected them out of sixteen without exploring, so the decoys cost nothing and created no discovery to save. The procedural arm came out slightly WORSE -- 4.7 steps against 4.0 -- the recalled procedure adding context the agent then had to read past. Four attempts now converge on a conclusion that is about the feature rather than my benchmark. The obstacle was never the wording, the parameter names, the arbitrariness of the dependency, or the size of the action space. It is that a competent model does not explore on this class of task at all: it reads tool descriptions and selects correctly on the first attempt. Procedural memory has no exploration cost to remove because there is none. Stated narrowly, because the broad version would be wrong: for tasks where correct tool selection is inferable from tool descriptions -- which is what a well-designed tool API is -- a stored procedure saves nothing and carries a small context cost. The place to look for benefit is tasks where the right action is not inferable from the interface: undocumented sequencing, environment quirks, conventions learned from failure rather than from a schema. Those are what a human writes a runbook for, and a runbook is what a procedure is. The instrument is sound and demonstrated working across four runs. What it keeps reporting is that this task class has nothing for the feature to do -- which is worth knowing before anyone builds a larger benchmark, and is the finding to carry forward rather than the numbers. 453 LongMemEval tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
There was a problem hiding this comment.
Pull request overview
Expands the procedural-memory benchmark task’s tool action space by adding decoy tools, aiming to make “call everything” exploration meaningfully costly and to document the results of a fourth evaluation run.
Changes:
- Added 12 decoy tools to
ProceduralBenchmarkTask.CreateTools()via a newDecoys()helper. - Updated the unit test to expect 16 exposed tools.
- Extended the review write-up documenting the fourth run and its conclusions.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| tools/AgentMemory.LongMemEval/ProceduralBenchmarkTask.cs | Adds decoy tools to expand the benchmark toolset and record decoy invocations. |
| tests/AgentMemory.Tests.Unit.LongMemEval/ProceduralBenchmarkTaskTests.cs | Updates expectations for the expanded tool list. |
| docs/reviews/procedural-benefit-run-prerequisite.md | Documents the fourth run’s setup and findings. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
82
to
86
| AIFunctionFactory.Create(PlaceHold), | ||
| AIFunctionFactory.Create(CheckServiceBulletin), | ||
| AIFunctionFactory.Create(Book), | ||
| .. Decoys(), | ||
| ]; |
Comment on lines
+102
to
+116
| new (string Name, string Description)[] | ||
| { | ||
| ("check_seat_map", "Returns the seat map for a connection."), | ||
| ("list_fare_classes", "Lists fare classes available on a connection."), | ||
| ("get_station_facilities", "Returns facilities at a station."), | ||
| ("check_loyalty_balance", "Returns a traveller's loyalty point balance."), | ||
| ("list_connections", "Lists connections between two stations."), | ||
| ("get_refund_policy", "Returns the refund policy for a fare class."), | ||
| ("check_platform", "Returns the departure platform for a connection."), | ||
| ("get_catering_menu", "Returns the catering menu for a connection."), | ||
| ("list_baggage_rules", "Returns baggage allowance rules."), | ||
| ("check_delay_history", "Returns recent punctuality for a connection."), | ||
| ("get_carriage_layout", "Returns the carriage layout for a connection."), | ||
| ("list_partner_operators", "Lists partner operators for a route."), | ||
| } |
Comment on lines
187
to
193
| public void ToolsAreExposedInProcedureOrder() | ||
| { | ||
| new ProceduralBenchmarkTask().CreateTools().Should().HaveCount(4); | ||
| // Four real tools plus twelve plausible decoys. The decoys are the third run's finding made | ||
| // concrete: with a small tool set an agent calls everything and skips discovery, so a stored | ||
| // procedure saves nothing and both arms tie. | ||
| new ProceduralBenchmarkTask().CreateTools().Should().HaveCount(16); | ||
| } |
This was referenced Aug 27, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fourth attempt, implementing what the third run's finding called for: twelve plausible decoy tools beside the four real ones, so calling everything costs sixteen invocations rather than four.
Both arms still called exactly the four right tools — selected out of sixteen without exploring, so the decoys cost nothing. And the procedural arm came out slightly worse (4.7 vs 4.0 steps), the recalled procedure adding context the agent had to read past.
Four attempts converge on a conclusion about the feature, not the benchmark. The obstacle was never the wording, the parameter names, the arbitrariness of the dependency, or the action-space size. It's that a competent model doesn't explore on this task class at all — it reads tool descriptions and selects correctly first time. Procedural memory has no exploration cost to remove because there is none.
Stated narrowly, because the broad version would be wrong:
The place to look for benefit is tasks where the right action isn't inferable from the interface: undocumented sequencing, environment quirks, conventions learned from failure. Those are what a human writes a runbook for — and a runbook is what a procedure is.
The instrument is sound and demonstrated across four runs. What it keeps reporting is that this task class has nothing for the feature to do — worth knowing before building a larger benchmark.
🤖 Generated with Claude Code
https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE