Skip to content

feat(mcp): a tool response carries errors recorded since the last one - #288

Merged
paddymul merged 4 commits into
mainfrom
feat/mcp-surface-new-errors
Sep 29, 2026
Merged

paddymul merged 4 commits into
mainfrom
feat/mcp-surface-new-errors

Conversation

@paddymul

@paddymul paddymul commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Problem

errors.jsonl is written by more than the MCP tools. The heal path in result_cache.py records unfaithful_heal, the companion records browser chart failures (chart_render) and failed code edits from the UI (api_code). No MCP tool returns those records, so a model driving tallyman over MCP does not see them.

In one session the log collected six unfaithful_heal records, one for each version (v1 to v6) of the derived entry qb_epa. The model saw only the first, and only because it tailed errors.jsonl by hand. It guessed a wrong cause from that one record and never learned about the other five.

Fix

_tag_project, which already adds project and the project-switch warning to every tool reply, now adds new_errors when the active project's log has records this MCP process has not reported yet. With none, the field is absent.

{
  "active": "demo",
  "available": ["demo"],
  "project": "demo",
  "new_errors": [
    {
      "id": "57b476ad2161",
      "created_at": "2026-09-28T21:53:26.210465+00:00",
      "code": "unfaithful_heal",
      "hash": "abc123def456",
      "message": "self-heal produced different rows than were built [structural (#88)]: digest 1 != recorded 2"
    }
  ]
}
  • Items carry id, created_at, code, hash, alias and version (from version_of_hash, when the hash is in an alias history), tool, and message cut at 300 characters. code is echoed only when it is a short kind like unfaithful_heal: on a build failure the field holds the whole recipe, so it is left out. No traceback, no prompt, and keys whose value is null are dropped.
  • Newest first, at most 10, with new_errors_omitted giving the count left out.
  • The cursor is in-process and kept per log file (so per project), as the file's inode and the byte offset read up to. It starts at this process's start: a record written before the process started is not reported, one written after it started but before the session's first call is. Switching projects uses the other project's cursor, and its pending records come with the switch reply.
  • An unchanged log costs one stat. A changed one is read from the offset. A line with no newline yet (an append in progress) is left for the next call, a line that is not a JSON object is skipped, and a log that was cleared and written again (new inode, or smaller than the offset) is read from the start.
  • A record the same call already answers with is not repeated. That is a record this process wrote with tool equal to the tool being answered (its own build error, the recalc failures and skips of catalog_revise's cascade), or one whose id is anywhere in the reply as id or error_id (catalog_chart_errors returns the records themselves). To know which records this process wrote, record_error now calls the hooks in a new errors.RECORD_HOOKS list, and the server registers one that keeps the ids. A heal recorded in this process while a tool reads an entry has tool None, so it is still reported.
  • A failure while reading the log is logged and the reply goes out without the field.

catalog_run's docstring has two lines on new_errors, since the model never sees _tag_project's.

#248 makes the SPA refetch on the unfaithful_heal SSE event. This PR does not touch the SPA or SSE, so the two do not overlap.

Tests

tests/test_mcp_new_errors.py, 11 tests. record_error called directly stands in for another process. They cover: a record reported once on the next reply and absent after; a record from before the process started skipped while one from before the first call is reported; the compact item with alias and version and a 2000-character message; recipe code not echoed; catalog_run's own build error not repeated; catalog_revise's cascade failure not repeated; a heal recorded in this process during a call still reported; the cap of 10 with new_errors_omitted == 3; a torn line and a half-written line; an unchanged log not opened; separate cursors across project_switch.

  • Red: 6be6214. CI failed the 11 new tests and nothing else (11 failed, 1066 passed). Each failed on the missing new_errors field.
  • Fix: e772e53. CI green: fast suite 1077 passed, integration 7 passed, ruff clean.
  • Local: fast suite 1077 passed, 1 skipped; integration suite 7 passed; uvx ruff check clean.

End to end, I ran tallyman mcp over stdio with fastmcp's Client against a temporary TALLYMAN_HOME. A record written before the server started was not reported. I called project_list, then recorded an unfaithful_heal from a separate uv run python process. The next project_list reply is the JSON above, and the call after it had no new_errors.

Not in this PR

  • Why qb_epa healed unfaithfully at every version. This PR only makes the records reach the model.
  • The cursor lives in memory, so a restarted MCP process does not report records written between the old process's last reply and the new one's start.
  • If a cleared log is written again and gets back the same inode, and grows past the old offset before the next call, the records before the offset are missed. clear_errors is called only by the companion's clear-errors route (the error banner's dismiss).

🤖 Generated with Claude Code

paddymul and others added 2 commits September 28, 2026 17:49
…ponse once

errors.jsonl gets records no MCP tool returns: the companion's heal hook
(unfaithful_heal), browser chart render failures, code edits in the UI.
These tests pin a `new_errors` field on every tool response: each record
reported once, from this process's start, per project, capped, without
repeating a tool's own reported failure, and tolerant of a torn line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_tag_project now adds `new_errors` to a reply when the active project's
errors.jsonl has records this MCP process has not reported: the
companion's unfaithful_heal, a browser chart failure, a UI code edit.
A per-log cursor (inode, byte offset) starts at this process's start,
so an unchanged log costs one stat and old sessions' history is not
replayed. A record the call already answers with is left out: one
this process wrote for the same tool (tracked through a new
errors.RECORD_HOOKS), or one whose id is in the reply. Items are
compact (no traceback or recipe code), newest first, capped at 10
with `new_errors_omitted`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@paddymul
paddymul marked this pull request as ready for review September 28, 2026 22:08
paddymul and others added 2 commits September 29, 2026 10:53
FastMCP runs tool calls on a thread pool. A project_list that returns while catalog_run is between recording its build
error and replying reads that record from the log and reports it as new_errors, and catalog_run's own reply carries
it again in `error`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…any reply

A record is left out of new_errors when this process wrote it with `tool` set to the tool whose call was running at
the time (a contextvar set by the tool wrapper). Such a record is in that call's own reply, as `error` or in `recalc`.
The check compared the record's tool with the tool being answered, so a concurrent call of another tool reported it.
A record written outside a tool call, or during one without that tool's name (a heal while a tool reads an entry),
is still reported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@paddymul

Copy link
Copy Markdown
Contributor Author

Review pass over this PR. Findings and how each was addressed:

  1. A concurrent tool call reported another call's own error. The new_errors filter compared a record's tool with the tool being answered. FastMCP runs tool calls on a thread pool, so a project_list returning while catalog_run was between recording its build error and replying reported that error, and catalog_run's reply carried it again. Fixed in 38b8d45: the tool wrapper sets a contextvar, and the record hook notes which tool was running when each record was written. A record whose tool matches that tool is left out of every reply. Test in f9d3695 (failed on CI, then passed with the fix).
  2. Cost of the first log read. Declined. The largest errors.jsonl on disk (208 KB, 13 records) reads in 0.3 ms, an unchanged log costs a 20 µs stat, and 10 version_of_hash lookups take 0.7 ms.

Noted, not changed: _recorded_by keeps one id for each error recorded during a tool call for the life of the process. It could be pruned as the cursor passes each record.

🤖 Generated with Claude Code

@paddymul
paddymul merged commit b900e60 into main Sep 29, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant