release: v0.1.1212 — ai-agent starter passes its own eval out of the box

[kwakayama]

Cuts v0.1.1212 to ship one fix: the ai-agent starter's own eval now passes out of the box.

v0.1.1211 (#3459) removed two rubric lines the assistant could not reliably meet. That helped, but measuring properly showed the starter still failed about one run in four, for two causes that had nothing to do with the rubric. #3459 merged while that measurement was still running, so the fix missed the release.

Both causes were found by reading eval traces, not by reading the source.

1. b meant two different things

operation: v.enum(["add", "subtract", "multiply", "divide", "round"]),
a: v.number(),
b: v.number(),   // second operand -- except for `round`, where it is decimal places

The model could not hold both meanings at once. From a failing trace:

{"operation":"subtract","a":99.71,"b":66.48,"round":0}   <- invented a key the schema never declared
{"operation":"divide","a":9971,"b":0}
  -> {"error":"Cannot divide by zero"}

It wanted to round, the schema gave it no way to say so on a divide, and it encoded the request as b: 0. The tool throws, and agent.noFailedTools fails on a single thrown call.

2. Rounding each share cannot preserve a total

No prompt wording fixes this one. Asked to divide $99.71 three ways, the model rounded each share on its own and answered:

Split among 3 people: $99.71 / 3 = $33.24 each.

That is $99.72.

One operation fixes both

split divides a money amount in integer cents and returns shares that add up to it exactly:

{"operation":"split","a":99.71,"b":3}  ->  {"result":[33.24,33.24,33.23]}

Because the model never rounds money, it never needs a decimal-places argument either. So round retires, and the float-rounding helper it required (Number.EPSILON nudge into toFixed) retires with it. Money is counted here, not rounded. The system prompt shrinks too, since it no longer has to explain how to distribute a leftover cent.

The tool is 38 lines: four operations that do the obvious thing, plus split.

b becomes an array length, so an unbounded value from the model would throw RangeError and fail the same gate this PR exists to fix. Splits above 1000 shares are refused rather than silently capped, since returning 1000 shares when 5000 were asked for is quietly wrong.

Review follow-ups

• Unbounded split count (#discussion_r3735861613) — valid, fixed. I refuse above 1000 rather than Math.min capping, since a silently wrong split defeats the point of an operation that promises the shares add up. CodeRabbit confirmed the approach.

Evidence

Measured the way a user meets it: veryfront init --template ai-agent, then npm run eval -- assistant against the gateway.

	before	after
runs green	6 / 8	8 / 8
agent.noFailedTools	7 / 8	8 / 8
judge.rubric	6 / 8	8 / 8
called split	n/a	8 / 8

All 8 called split and read the shares straight back, so this is the mechanism working rather than the sample being kind. Dropping the rounding helper costs nothing: the model receives 15.209999999999999 for the tip and writes $15.21 unaided, with no float noise in any of the 8 answers.

18% tip on $84.50 = $15.21
Total = $84.50 x 1.18 = $99.71

Split $99.71 among three people:
- Person 1: $33.24
- Person 2: $33.24
- Person 3: $33.23

What this does not claim. Eight consecutive green runs is not proof of determinism. A live model graded by a live model can still surprise you, and the judge resolves to the auto model, so the rate will drift as that moves. What changed is that both systematic failures are gone: one was a schema the model could not use correctly, the other was arithmetic it was being asked to do in its head. Nothing in CI runs this eval, so nothing catches a regression here except someone scaffolding.

Testing

• cli/templates/, cli/commands/init/: 15 passed, 208 steps, 0 failed
• deno check, deno lint (36 files), deno fmt --check: clean
• Split invariant tested across [99.71, 3], [0.01, 3], [10, 4], [-99.71, 3], asserting the shares always add back to the total, plus the 1000-share ceiling and the zero guard

cli/templates/manifest.json embeds template contents verbatim and is regenerated, so veryfront init scaffolds the fixed files.

Release bump is the standard three files. No tag pushed.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

[coderabbitai]

📝 Walkthrough

Walkthrough

The calculator template replaces round with split and optional decimals rounding. The assistant instructions now require direct calculator use for currency shares. Template tests cover the new behavior. Project versions increase to 0.1.1212.

Changes

Calculator and assistant templates

Layer / File(s)	Summary
Calculator contract and runtime cli/templates/files/ai-agent/tools/calculator.ts, cli/templates/manifest.json, cli/templates/index.test.ts	The calculator accepts split and optional decimals. It rejects zero divisors, returns cent-accurate shares, and rounds arithmetic results. Tests cover these behaviors.
Assistant instruction wiring cli/templates/files/ai-agent/agents/assistant.ts, cli/templates/manifest.json, cli/templates/index.test.ts	Assistant instructions require immediate calculator use and explicit split usage for currency shares. Prompt assertions validate the updated instructions.
Release version alignment deno.json, src/utils/version-constant.ts	The project version changes from 0.1.1211 to 0.1.1212.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant User
  participant Assistant
  participant Calculator
  User->>Assistant: Request currency shares
  Assistant->>Calculator: Execute split operation
  Calculator-->>Assistant: Return cent-accurate shares
  Assistant-->>User: State every returned share

Loading

Possibly related PRs

• veryfront/veryfront-code#3391: Updates the AI-agent prompt and calculator-related tests in the same template area.

Suggested reviewers: kojiwakayama

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the release version and the primary ai-agent starter evaluation improvement.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch release/v0.1.1212

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cli/templates/files/ai-agent/tools/calculator.ts`:
- Around line 21-32: Bound the caller-controlled split count in calculator.ts by
capping the computed parts value at 1,000 while preserving the minimum of one
and existing truncation/absolute-value behavior before Array.from. Regenerate
the embedded copy in cli/templates/manifest.json at line 34 from calculator.ts;
do not edit the manifest copy manually.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 55a8c0ea-377a-49fd-adb3-7bc9fdc20d51

📥 Commits

Reviewing files that changed from the base of the PR and between 9acfc01 and cf04c23.

⛔ Files ignored due to path filters (1)

• src/html/hydration-script-builder/hydration-runtime.generated.ts is excluded by !**/*.generated.*

📒 Files selected for processing (6)

• cli/templates/files/ai-agent/agents/assistant.ts
• cli/templates/files/ai-agent/tools/calculator.ts
• cli/templates/index.test.ts
• cli/templates/manifest.json
• deno.json
• src/utils/version-constant.ts