Make starter arithmetic verifiable by default

[kojiwakayama]

Summary

• require the AI Agent starter to use its calculator for arithmetic
• evaluate the exact user-facing tip prompt with deterministic amount, tool-use, and tool-failure gates
• add judges.llm.rubric() as a reusable built-in semantic grader
• make both built-in LLM judges fail cleanly on provider/runtime errors
• isolate trusted rubric-judge instructions from untrusted evaluation data
• document the judge API and bump the release to 0.1.1173

Evidence

• published starter baseline: calculator called in 4/5 runs (80%)
• updated starter policy: calculator called in 5/5 runs (100%)
• built-in rubric judge: correct answer scored 1/pass; incorrect answer scored 0/fail
• adversarial prompt-data and judge-provider-failure regression tests
• deno task fmt:check
• deno task lint
• deno task typecheck
• deno task test:unit (2716 tests, 22089 steps, 0 failures)
• public guide validation (65 guides, 109 public-doc files)
• guide tests (45 tests, 87 steps, 0 failures)

Known validation gaps

• typecheck:consumer requires the Storybook-local TypeScript toolchain, which is not installed in this worktree
• full docs:validate is blocked on current main by the unrelated missing release-assets API reference/JSDoc entry; guide and public-doc validation pass

[Copilot]

Pull request overview

This PR strengthens Veryfront’s eval tooling and starter template by adding a reusable built-in LLM rubric judge, then using it to make the AI Agent starter’s arithmetic behavior verifiable by default (calculator tool use, deterministic answer checks, and judge-based quality gating). It also updates public docs and bumps the release version.

Changes:

• Added judges.llm.rubric() as a built-in general-purpose semantic grader for metrics.judge.rubric.
• Updated the AI Agent CLI template to instruct calculator usage for arithmetic and to ship a stricter smoke eval (tool-use, tool-failure, numeric contains, and rubric judge gates).
• Updated eval documentation/API reference exports and bumped version to 0.1.1173.

Verification noted in PR description:

• deno task fmt:check, deno task lint, deno task typecheck
• deno task test:unit (2716 tests, 0 failures)
• Guide/public-doc validation passing; typecheck:consumer and full docs:validate have known external blockers (as described)

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/utils/version-constant.ts	Bumps shared VERSION constant to 0.1.1173.
deno.json	Bumps package version to 0.1.1173.
src/eval/judges.ts	Adds EvalLlmRubricJudgeOptions, rubric prompt builder, and judges.llm.rubric() factory.
src/eval/judges.test.ts	Adds unit coverage ensuring rubric judge prompt construction and JSON parsing behavior.
src/eval/index.ts	Exports EvalLlmRubricJudgeOptions from the public eval entrypoint.
docs/guides/evals.md	Documents rubric judges using the new built-in judges.llm.rubric() helper.
docs/api-reference/veryfront/eval.md	Adds API reference entries/links for EvalLlmRubricJudgeOptions and updates source anchors.
cli/templates/manifest.json	Updates AI Agent template strings: system prompt requires calculator for arithmetic; eval uses rubric judge + stricter gates.
cli/templates/index.test.ts	Updates template tests to assert new system prompt text and rubric-judge eval wiring.
cli/templates/files/ai-agent/evals/assistant.eval.ts	Updates shipped template eval to use the tip/split prompt and judges.llm.rubric() gate.
cli/templates/files/ai-agent/agents/assistant.ts	Updates shipped template agent system prompt to require calculator tool for arithmetic.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

docs/api-reference/veryfront/eval.md:188

• The API reference "Source" link for judges points at the wrong line in src/eval/judges.ts (currently the export starts at line 358). This makes the generated documentation link land in the middle of createLlmGroundednessJudge instead of the constant definition.

| `judges` | Built-in judge factories for semantic eval metrics. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/eval/judges.ts#L345) |

src/eval/judges.ts:336

• createLlmRubricJudge fails closed on provider/runtime errors, but createLlmGroundednessJudge still allows generateText() errors to throw. Because runRecord() does not catch metric evaluation errors, a judge outage can abort the entire eval run instead of producing a failed metric result. Consider adding the same try/catch fail-closed behavior to the groundedness judge.

function createLlmGroundednessJudge(
  options: EvalLlmGroundednessJudgeOptions = {},
): GroundednessJudge {

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

src/eval/judges.ts:276

• parseJudgeResponse() currently makes the final pass depend on the model-provided pass boolean (modelPass && score >= threshold). Because metric gating already applies score thresholds, a judge response with a high score but pass: false will be forced to fail (false negative) even though it meets the threshold. Consider treating the numeric score as authoritative and deriving pass from score >= threshold so eval outcomes are stable even if the judge model returns an inconsistent pass field.

    const details = [
      typeof parsed.explanation === "string" && parsed.explanation.trim()
        ? parsed.explanation.trim()
        : "LLM judge returned a structured score.",
      ...(unsupportedClaims.length > 0