Skip to content

fix(pi): match stock tool-call rendering on newer Pi and watch latest Pi in CI - #4

Merged
witt3rd merged 3 commits into
mainfrom
fm/firstmate-pi-099-tests
Sep 30, 2026
Merged

witt3rd merged 3 commits into
mainfrom
fm/firstmate-pi-099-tests

Conversation

@witt3rd

@witt3rd witt3rd commented Sep 30, 2026 •

Copy link
Copy Markdown
Owner

Intent

Fork CI installs the latest public Pi. Two Firstmate tests fail against Pi 0.99.1 while passing on 0.87.1, the version the fleet runs: tests/fm-calm-pi-extension.test.sh ("rendered export DOM violated the Calm conversation boundary") and tests/fm-pi-branch-extension.test.sh ("Calm-off ToolExecutionComponent rendering differs from Pi stock"). PR #3 (merged) pinned the CI Pi install to 0.87.1 in a separate commit so CI is green, and noted that making the extensions and tests correct for newer Pi is a follow-up. This is that follow-up: Firstmate's Pi extensions and their tests must work with newer Pi releases, so the pin can be replaced by a real compatibility check. Evidence: on the fork CI run, pi 0.99.1 failed those two tests while main's earlier CI on pi 0.87.1 passed them; the two other failing shards were a flaky remote-seed clone unrelated to Pi.

What Changed

  • .pi/extensions/fm-branch-supervision.ts: The fm_branch_outcomes and fm_branch_processed call headers now follow the installed Pi's stock rendering. A one-time probe of the stock ToolExecutionComponent checks whether it shows tool arguments, as Pi 0.99 does and 0.87 does not. If it does, the headers mirror Pi's format (name key=value, or key: value lines when expanded). Otherwise they show only the title.
  • tests/fm-calm-pi-extension.test.sh: updated for this behavior.
  • .github/workflows/pi-latest.yml (new): a daily and manually triggered, non-blocking workflow. It installs the latest public Pi and runs fm-pi-primary-types, fm-pi-branch-extension and fm-calm-pi-extension. It does not run on pull requests.
  • .github/workflows/ci.yml: the required CI keeps its Pi install pinned to 0.87.1. Only the comments changed, and they now point to the pi-latest workflow for drift detection.

Risk Assessment

⚠️ Medium: The branch-extension rendering fix looks sound. The calm export DOM failure named in the intent is only instrumented, not fixed, and the pin stays in CI.

Testing

I ran the calm and branch Pi extension tests on the installed Pi 0.87.1 and both passed, which shows there is no regression on the fleet's version. Only 0.87.1 is available, so I could not test Pi 0.99.1, and three calm-test checks skipped.

  • Live validation: ⚠️ inconclusive - 1 of 2 scenarios driven live against the product
Scenario Result Live Evidence
Calm and branch extension tests pass on Pi 0.87.1 (fleet version) after the change ✅ pass live both test scripts exit 0
Calm and branch extension tests pass on Pi 0.99.1 (latest) ⏸️ untested no Only Pi 0.87.1 is installed. Installing 0.99.1 would need a system or global install outside the worktree, which the boundary forbids. Provide a Pi 0.99.1 install path, or rely on the new pi-latest wo…
  • Outcome: ⚠️ 1 warning across 1 run (1m21s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 warning
  • ⚠️ tests/fm-calm-pi-extension.test.sh:4385 - The intent says the Firstmate extensions and tests must work with newer Pi, including fm-calm-pi-extension's "rendered export DOM violated the Calm conversation boundary". For this test, the change only replaces bare process.exit(1) calls with named bail(...) calls. No assertion or extension behavior is adapted to Pi 0.99's export DOM. The test will probably still fail on 0.99.1, and the new daily pi-latest workflow will report it instead. The branch-extension test does get a real fix (the stock-header probe). The calm test has no fix for the 0.99 failure. The author needs to decide whether diagnostics plus a non-blocking watcher is acceptable, or whether the calm test must be made to pass on 0.99.
  • ⚠️ .github/workflows/ci.yml:115 - The intent says the pin should be replaced by a real compatibility check. The change keeps the 0.87.1 pin in both jobs and adds a separate scheduled, non-blocking pi-latest workflow. That is a deliberate scope choice, and it leaves the pin in place.

🔧 Fix applied.
1 warning still open:

  • ⚠️ .github/workflows/ci.yml:115 - The intent says the pin should be replaced by a real compatibility check. The change keeps the 0.87.1 pin in both jobs and adds a separate scheduled, non-blocking pi-latest workflow. That is a deliberate scope choice, and it leaves the pin in place.
⚠️ **Test** - 1 warning
  • ⚠️ live validation verdict: inconclusive (1 of 2 scenarios were driven live against the product); untested: Calm and branch extension tests pass on Pi 0.99.1 (latest)
  • Live validation: ⚠️ inconclusive - 1 of 2 scenarios driven live against the product
Scenario Result Live Evidence
Calm and branch extension tests pass on Pi 0.87.1 (fleet version) after the change ✅ pass live both test scripts exit 0
Calm and branch extension tests pass on Pi 0.99.1 (latest) ⏸️ untested no Only Pi 0.87.1 is installed. Installing 0.99.1 would need a system or global install outside the worktree, which the boundary forbids. Provide a Pi 0.99.1 install path, or rely on the new pi-latest wo…
  • bash tests/fm-calm-pi-extension.test.sh (pi 0.87.1; rc=0, some checks skipped because the package was not found)
  • bash tests/fm-pi-branch-extension.test.sh (pi 0.87.1; rc=0)
✅ **Document** - passed

✅ No issues found.

⚠️ **Lint** - 1 warning
  • ⚠️ linter found issues (exit code 1)
✅ **Push** - passed

✅ No issues found.

Worker-provided validation evidence (Pi 0.99.1)

The pipeline's live step could not drive Pi 0.99.1: the pipeline runner only hosts the fleet's Pi 0.87.1, so its "inconclusive" row is an environment limit, not a product gap. The worker validated 0.99.1 separately:

  • npm install -g --prefix /tmp/pi99 @earendil-works/pi-coding-agent@0.99.1, then with FM_PI_PACKAGE_DIR pointing at it: tests/fm-pi-branch-extension.test.sh, tests/fm-calm-pi-extension.test.sh and tests/fm-pi-primary-types.test.sh all pass (typecheck also passes on 0.87.1).
  • Reproduction of the original CI failure: ubuntu:24.04 container, node v22.23.3, Google Chrome 154.0.8037.92, Pi 0.99.1, non-root user, bash tests/fm-calm-pi-extension.test.sh failed at "check 5" before the fix. Cause: Pi 0.99's export keeps display:false custom messages in the DOM as hook-message hook-message-hidden blocks (CSS-hidden, toggleable); 0.87.1 omits them. The test now accepts the marker only inside such a hidden block. After the fix the full calm test passes in the same container on both Pi 0.99.1 and Pi 0.87.1.
  • Required CI stays on Pi 0.87.1; pi-latest.yml watches the latest Pi daily and is non-blocking.
  • The fm-spawn-compact-adviser-disable-remote test's EXIT-trap cleanup fix (separate commit from the pipeline) addresses a flaky failure seen in CI; the related remote-seed clone failure in fm-remote-secondmate-trace-context was not separately diagnosed.

…disable-remote.test.sh. All its assertions passed, but its EXIT trap's `rm -rf` failed on read-only git-hooks directories under the fixture root, so the script exited 1. The trap now runs `chmod -R u+rwX` on the fixture root before removing it. The test now exits 0 when run locally. I did not run the full portable serial shard 2
@witt3rd
witt3rd merged commit dc69617 into main Sep 30, 2026
23 of 38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant