docs: require eval coverage for skill changes

[pollychen-lab]

Summary

• add AGENTS.md guidance requiring eval coverage before skill behavior issues are marked ready-to-code
• document that missing eval coverage should block implementation until coverage lands
• require behavior-changing implementations to add/update eval cases or explain existing coverage

Closes #116.

Validation

• git diff --check
• PATH="/tmp/fullsend-tools:$PATH" ./eval/lint-cases.sh triage
• uvx pre-commit run --files AGENTS.md

Note: make script-test currently fails in scripts/post-triage-test.sh at prerequisites-creates-allowed-issue; that failure is unrelated to this AGENTS.md-only docs change.

[github-actions]

Thank you for your interest in contributing, @pollychen-lab.

This project uses a vouch system for first-time contributors. Before submitting a pull request, you need to be vouched by a maintainer.

To get vouched:

1. Open a Vouch Request discussion on the main fullsend repo.
2. Describe what you want to change and why.
3. Write in your own words — do not have an AI generate the request.
4. A maintainer will comment /vouch if approved.
5. Once vouched, open a new PR (preferred) or reopen this one.

See CONTRIBUTING.md for details.

[github-actions]

Functional tests did not run

Functional tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

[qodo-code-review]

PR Summary by Qodo

Docs: require eval coverage before skill behavior changes

📝 Documentation 🕐 Less than 10 minutes

AI Description

• Add triage guidance to verify functional eval coverage before marking skill behavior issues
 ready-to-code.
• Treat missing evals as a prerequisite that must land before behavior-changing implementation.
• Require behavior changes to add/update eval cases or justify existing coverage.

Diagram

graph TD
  A["Issue triage"] --> B{“Eval coverage exists?”}
  B -->|"Yes"| C["Mark ready-to-code"] --> D["Implement behavior change"] --> E["Update/add eval cases
(or justify coverage)"]
  B -->|"No"| F["Create eval first
(or split work)"] --> C

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Automated CI/PR gate for eval coverage

• ➕ Enforces the policy consistently; reduces reviewer/triage burden.
• ➕ Can be extended to require updated eval cases when skill behavior files change.
• ➖ Non-trivial to implement robustly (skill↔eval mapping, false positives).
• ➖ May block legitimate changes when mapping/coverage is intentionally indirect.

2. Issue template checklist (ready-to-code criteria)

• ➕ Low-effort reinforcement at the point of issue creation/triage.
• ➕ Makes the policy visible without adding automation complexity.
• ➖ Still relies on humans; can be skipped or misapplied.
• ➖ Does not validate that eval coverage truly exists/executes.

Recommendation: The documentation-first approach in this PR is the right minimal, immediately actionable step. Consider a follow-up to add an issue template checklist and/or a lightweight CI check once the skill↔eval mapping rules are stable enough to avoid frequent false positives.

Files changed (1) +16 / -0

Documentation (1) +16 / -0

AGENTS.mdAdd triage requirement for eval coverage on skill behavior changes +16/-0

Add triage requirement for eval coverage on skill behavior changes

• Adds a new section documenting that issues changing skill behavior must verify functional eval coverage before being marked ready-to-code. Specifies where to find the expected eval.yaml and instructs treating missing coverage as a prerequisite, plus requiring behavior-change implementations to update/add eval cases or justify existing coverage.

AGENTS.md

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (1) 📜 Skill insights (1)

Context used

✅ Tickets: 🎫 Triage agent should treat absent eval coverage as a prerequisite for issues that modify agent skill behavior

✅ Compliance rules (platform): 55 rules

✅ Skills: 4 invoked

  code-review

  code-implementation

  pr-review

  docs-review

1. eval/<agent-or-skill-name> path mismatch 📎 Requirement gap ⚙ Maintainability

Description

AGENTS.md documents eval coverage using eval/<agent-or-skill-name>/eval.yaml and references an
“established mapping” when names differ, but it does not specify the required canonical path
eval/<skill-name>/eval.yaml, the current coverage state, or how to determine/apply the mapping.
This ambiguity can lead to inconsistent triage and readiness-to-code gating for skill behavior
changes, contrary to the documented prerequisite requirement.

Code

AGENTS.md[R94-97]

+verify that the affected skill has functional eval coverage. For local skill
+definitions under `skills/<skill-name>/SKILL.md`, check for the corresponding
+eval config under `eval/<agent-or-skill-name>/eval.yaml`, using the repo's
+established mapping if the skill name and eval directory differ.

Relevance

⭐⭐ Medium

No historical evidence on canonical eval path/mapping doc requirement in AGENTS.md; only general doc
updates in PR #19/#25.

PR-#19
PR-#25

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

PR Compliance ID 1540707 requires AGENTS.md to explicitly state the expected eval location as
eval/{skill-name}/eval.yaml, note the current coverage state, and treat missing evals as a blocker
before marking skill behavior changes as ready-to-code. The newly added guidance instead points to
eval/<agent-or-skill-name>/eval.yaml and refers to an unspecified “established mapping” without
explaining where it is defined or how to use it; since the repo’s eval tooling is keyed by agent
name passed to fullsend run <agent> (constructing eval/<agent>/eval.yaml) and agents (e.g.,
review) can include skills whose directories differ (e.g., skills/pr-review), readers may
inconsistently look for different eval paths (e.g., eval/pr-review/eval.yaml vs
eval/review/eval.yaml).

Document eval coverage prerequisite for skill behavior changes in AGENTS.md
AGENTS.md[91-105]
AGENTS.md[93-97]
eval/run-functional.sh[24-31]
eval/lint-cases.sh[14-18]
eval/scripts/run-fullsend.sh[20-26]
eval/scripts/run-fullsend.sh[70-76]
harness/review.yaml[8-16]
config.yaml[5-10]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Update `AGENTS.md` so the eval-coverage prerequisite for skill behavior changes is unambiguous and compliant: explicitly require eval coverage at the canonical path `eval/<skill-name>/eval.yaml` for behavior changes under `skills/<skill-name>/SKILL.md`, clearly state the current eval coverage state, and remove/replace the vague “established mapping” guidance by documenting exactly where the mapping is defined and how to apply it (including clarifying the relationship to agent-name-based tooling like `fullsend run <agent>`).

## Issue Context
PR Compliance ID 1540707 requires AGENTS.md to (a) specify the expected eval path `eval/{skill-name}/eval.yaml`, (b) state that missing evals should block readiness-to-code, and (c) note the current coverage state so triage is consistent. The current text instead uses `eval/<agent-or-skill-name>/eval.yaml` and refers to an “established mapping” without explaining how to determine it, even though eval execution is driven by the agent name argument to `fullsend run` (which resolves to `eval/<agent>/eval.yaml`) and agents can include skills whose directory names don’t match the agent name (e.g., `review` including `skills/pr-review`), making the mapping non-obvious.

## Fix Focus Areas
- AGENTS.md[91-105]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Protected file AGENTS.md modified 📜 Skill insight § Compliance

Description

This PR modifies AGENTS.md, which is a protected governance/infrastructure path requiring explicit
human review and must not be auto-approved. A compliance finding is required whenever protected
paths are touched.

Code

AGENTS.md[R91-105]

+## 8. Eval coverage for skill changes
+
+Before marking an issue `ready-to-code` for changes to skill behavior,
+verify that the affected skill has functional eval coverage. For local skill
+definitions under `skills/<skill-name>/SKILL.md`, check for the corresponding
+eval config under `eval/<agent-or-skill-name>/eval.yaml`, using the repo's
+established mapping if the skill name and eval directory differ.
+
+If no matching eval coverage exists on the base branch, treat eval creation
+as a prerequisite and do not promote the issue to implementation. Ask for an
+eval task first, or split the work so coverage lands before behavior changes.
+
+If coverage exists, require the implementation plan to update or add eval
+cases for the changed behavior, or explicitly explain why the existing cases
+already cover the change.

Relevance

⭐⭐ Medium

No prior accepted/rejected “AGENTS.md is protected” precedent; AGENTS.md edits merged in PR #19/#25.

PR-#19
PR-#25

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The compliance rule flags any PR that modifies protected governance/infrastructure paths, explicitly
including AGENTS.md. The diff shows new content added to AGENTS.md in this PR.

AGENTS.md[91-105]
Skill: pr-review

---

[qodo-code-review]

1. eval/ path mismatch 📎 Requirement gap ⚙ Maintainability

AGENTS.md documents eval coverage using eval/<agent-or-skill-name>/eval.yaml and references an
“established mapping” when names differ, but it does not specify the required canonical path
eval/<skill-name>/eval.yaml, the current coverage state, or how to determine/apply the mapping.
This ambiguity can lead to inconsistent triage and readiness-to-code gating for skill behavior
changes, contrary to the documented prerequisite requirement.

Agent Prompt

## Issue description
Update `AGENTS.md` so the eval-coverage prerequisite for skill behavior changes is unambiguous and compliant: explicitly require eval coverage at the canonical path `eval/<skill-name>/eval.yaml` for behavior changes under `skills/<skill-name>/SKILL.md`, clearly state the current eval coverage state, and remove/replace the vague “established mapping” guidance by documenting exactly where the mapping is defined and how to apply it (including clarifying the relationship to agent-name-based tooling like `fullsend run <agent>`).

## Issue Context
PR Compliance ID 1540707 requires AGENTS.md to (a) specify the expected eval path `eval/{skill-name}/eval.yaml`, (b) state that missing evals should block readiness-to-code, and (c) note the current coverage state so triage is consistent. The current text instead uses `eval/<agent-or-skill-name>/eval.yaml` and refers to an “established mapping” without explaining how to determine it, even though eval execution is driven by the agent name argument to `fullsend run` (which resolves to `eval/<agent>/eval.yaml`) and agents can include skills whose directory names don’t match the agent name (e.g., `review` including `skills/pr-review`), making the mapping non-obvious.

## Fix Focus Areas
- AGENTS.md[91-105]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

[qodo-code-review]

2. Protected file agents.md modified 📜 Skill insight § Compliance

This PR modifies AGENTS.md, which is a protected governance/infrastructure path requiring explicit
human review and must not be auto-approved. A compliance finding is required whenever protected
paths are touched.