Add the arbitrary dependency; find the real obstacle is the action space (7.6) - #199
Merged
Merged
Conversation
Third attempt at making the procedural benchmark discriminate. Booking now needs a clearance code whose only source is a service-bulletin lookup -- a tool whose name gives no hint -- and the refusal names the missing code without saying where to find it. Both properties are pinned by tests. Result: four tool calls, both arms, zero refusals. The model called all four tools before booking. That is the structural finding, and it survives any amount of dependency-hiding. With a small tool set, exhaustive calling is a cheap and correct strategy: the agent never has to discover an ordering because it never has to choose. A stored procedure cannot pay for itself when the unguided policy is already near-optimal. So the requirement was never "hide the dependency". It is to make exploration expensive -- a large action space where calling everything costs more than the task is worth, wrong steps that are irreversible or penalised so exploring has a visible price, or a chain long enough that the number of orderings defeats guessing. All three make the benchmark meaningfully bigger, and that is the honest scope of what remains. Three designs have now converged on the same conclusion, recorded as a table in the review note: prerequisites in descriptions, prerequisites in parameter names, and an arbitrary dependency all produced identical arms. None of those figures is evidence about procedural memory in either direction, and the note says so. The harness, runner, promotion path and arm switch are built, tested and demonstrated working end to end across all three runs. What is missing is a task whose action space makes exploration cost something. 451 LongMemEval tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
There was a problem hiding this comment.
Pull request overview
Updates the LongMemEval procedural benchmark task to introduce an additional, intentionally arbitrary prerequisite (a clearanceCode) sourced only from a newly added service-bulletin tool, and refreshes tests/docs to reflect the third-run findings about action-space size and exhaustive tool calling.
Changes:
- Add
CheckServiceBulletintool and aclearanceCoderequirement toBook, with refusal messaging that names the missing code without revealing its source. - Extend unit tests to cover the new tool step, refusal behavior, and tool exposure count.
- Document the third run’s outcome and the conclusion that discrimination requires making exploration expensive (larger action space / penalties / longer chains).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| tools/AgentMemory.LongMemEval/ProceduralBenchmarkTask.cs | Adds the service-bulletin tool and enforces the new clearance-code prerequisite for booking. |
| tests/AgentMemory.Tests.Unit.LongMemEval/ProceduralBenchmarkTaskTests.cs | Updates and adds tests to validate the new dependency and refusal constraints. |
| docs/reviews/procedural-benefit-run-prerequisite.md | Records the third-run design and the structural conclusion about small action spaces. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| var booking = await InvokeAsync( | ||
| task, "Book", new { holdReference = "HOLD-4417", clearanceCode = "guessed" }); | ||
|
|
||
| booking.Should().NotContain("bulletin"); |
Comment on lines
187
to
190
| public void ToolsAreExposedInProcedureOrder() | ||
| { | ||
| new ProceduralBenchmarkTask().CreateTools().Should().HaveCount(3); | ||
| new ProceduralBenchmarkTask().CreateTools().Should().HaveCount(4); | ||
| } |
This was referenced Aug 27, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third attempt at making the benchmark discriminate. Booking now needs a
clearanceCodewhose only source is a service-bulletin lookup — a tool whose name gives no hint — and the refusal names the missing code without saying where to find it. Both pinned by tests.Result: 4 tool calls, both arms, zero refusals. The model called all four tools before booking.
That's the structural finding, and it survives any amount of dependency-hiding. With a small tool set, exhaustive calling is a cheap and correct strategy: the agent never has to discover an ordering because it never has to choose. A stored procedure can't pay for itself when the unguided policy is already near-optimal.
So the requirement was never "hide the dependency" — it's to make exploration expensive: a large action space where calling everything costs more than the task is worth; wrong steps that are irreversible or penalised so exploring has a visible price; or a chain long enough that orderings defeat guessing. All three make the benchmark meaningfully bigger, and that's the honest scope of what remains.
Harness, runner, promotion path and arm switch are built, tested, and demonstrated working end to end across all three runs. None of these figures is evidence about procedural memory in either direction, and the note says so.
453 LongMemEval green.
🤖 Generated with Claude Code
https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE