Build the MAF runner the procedural harness needs (7.6)

[joslat]

The harness decides what a benefit is and is unit-tested against a scripted runner. This is the half that couldn't be scripted — the part that actually calls an agent, and therefore where an honest step and tool-call count has to come from.

Both figures are read off the returned messages, never from the agent's own account of itself. Asking a model how many tools it used produces a number that tracks how talkative it is, and a procedure that merely made the agent more confident would show up as an efficiency win.

Tool results are excluded from steps. The environment answering is not the agent acting, and counting it would make a procedure that batches its tool calls look like it took MORE steps rather than fewer — inverting the exact signal the harness exists to detect.

A fresh session per attempt. Procedural memory must carry across attempts through the store; a shared session carries it through the context window instead, and the measurement would credit memory for what the transcript did.

Completion is a caller-supplied predicate, not the agent declaring itself finished — a wrong procedure is reported fluently, which is why 7.7 measures wrong-procedure rate separately.

Six provider-free tests cover the counting rules — the part that can be quietly wrong. What remains is the concrete repeated task with its tools, then the measured run.

4,370 unit and 436 LongMemEval green (+6).

🤖 Generated with Claude Code

https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE

[Copilot]

Pull request overview

Adds the “real agent” implementation of the procedural-benefit harness runner (7.6) for LongMemEval, plus provider-free unit tests to validate step/tool-call counting rules based on returned messages rather than agent self-reporting.

Changes:

• Introduces MafAgentTaskRunner to execute repeated tasks through a live Microsoft Agent Framework AIAgent, returning completion/step/tool-call metrics.
• Implements step and tool-call counting based on ChatMessage roles/contents, explicitly excluding tool-result messages from step counts.
• Adds unit tests that construct transcripts and assert the counting rules and caller-supplied completion predicate behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
tools/AgentMemory.LongMemEval/MafAgentTaskRunner.cs	New real-agent task runner that measures completion, assistant steps, and tool calls from the returned message stream.
tests/AgentMemory.Tests.Unit.LongMemEval/MafAgentTaskRunnerTests.cs	Provider-free tests validating step/tool-call counting semantics and completion predicate intent.

Suppressed comments (1)

tools/AgentMemory.LongMemEval/MafAgentTaskRunner.cs:74

• isComplete is documented as deciding completion from the final transcript, but the runner passes only response.Text (typically just the last assistant message). That prevents completion checks that need to look at tool results / earlier turns and makes the measurement easier to fool with a self-reported final claim. Consider building a transcript from response.Messages (including tool calls/results) and pass that to _isComplete.

        var messages = response.Messages ?? [];
        return new AgentTaskRun(
            Completed: _isComplete(response.Text ?? string.Empty),
            Steps: CountSteps(messages),
            ToolCalls: CountToolCalls(messages));

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.