docs(glossary): define eval scenario vs eval measurement

[ascerra]

Summary

• Add Eval Scenario and Eval Measurement to docs/glossary.md to replace the overloaded bare term "evals"
• Scenarios = fixed reproducible test cases; measurements = scores/judges/metrics (including on wild production traffic)
• Cross-link each other and point at testing-agents.md / Observability

Source: Fullsend team sync 2026-07-14 terminology decision (Ralph / Adam / Marta).

Test plan

• Confirm glossary anchors #eval-scenario and #eval-measurement resolve in rendered docs
• Spot-check entry style matches neighboring glossary terms

Made with Cursor

[qodo-code-review]

PR Summary by Qodo

Define Eval Scenario vs Eval Measurement in glossary

📝 Documentation 🕐 Less than 10 minutes

AI Description

• Add glossary definitions for **Eval Scenario** and **Eval Measurement** to disambiguate “evals”.
• Cross-link the two terms and point readers to testing-agents guidance and Observability.

Diagram

graph TD
  U([Reader]) --> G["docs/glossary.md"] --> EM["Eval Measurement"] --> TA["problems/testing-agents.md"]
  G --> ES["Eval Scenario"] --> O["Observability (glossary section)"]
  EM --> ES
  ES --> EM

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Single “Evals” glossary entry with two subsections

• ➕ Keeps search/mental model centered on the commonly used term “evals”
• ➕ Avoids adding two separate headings to the glossary index
• ➖ Continues to legitimize an overloaded term instead of deprecating it
• ➖ Makes anchors less explicit (#evals vs #eval-scenario/#eval-measurement)

2. Add a dedicated terminology/ADR page and keep glossary brief

• ➕ Provides room for richer examples and decision rationale
• ➕ Can be a canonical reference linked from many docs
• ➖ Higher overhead for readers; glossary no longer self-contained for quick definitions
• ➖ Requires more cross-doc maintenance than a concise glossary entry

Recommendation: The PR’s approach (two explicit glossary terms with mutual cross-links and references) is the best tradeoff: it directly resolves the ambiguity of “evals” while keeping the definitions discoverable and anchor-linkable. Consider a follow-up sweep to replace existing uses of the bare term “evals” elsewhere in docs with the specific term intended.

Files changed (1) +10 / -0

Documentation (1) +10 / -0

glossary.mdAdd glossary definitions for Eval Measurement and Eval Scenario +10/-0

Add glossary definitions for Eval Measurement and Eval Scenario

• Introduces two new glossary entries to disambiguate the overloaded term “evals.” Each definition clarifies scope (curated scenarios vs scoring/metrics), cross-links to the other term, and references testing-agents guidance and Observability.

docs/glossary.md

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 7:50 PM UTC · Ended 7:54 PM UTC
Commit: 5cd495a · View workflow run →

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Context used

✅ Compliance rules (platform): 54 rules

✅ Skills: writing-user-docs, writing-adrs

1. E section out of order ✓ Resolved 🐞 Bug ⚙ Maintainability

Description

The new Eval Measurement/Scenario entries are placed before existing E terms (Entry Point,
Escalation, Evergreen), breaking the alphabetical ordering pattern used throughout the glossary and
making terms harder to scan and maintain.

Code

docs/glossary.md[R69-79]

+### Eval Measurement
+
+A score, judge, or metric applied to agent (or agent-chain) behavior — for example cost per run, whether the code agent later passes review, or whether a review agent recommends merge and a human still intervenes. Measurements are not the inputs under test; they are what you score. The same measurement can be applied to curated [eval scenarios](#eval-scenario) or to live ("wild") production traffic, at agent scope or across the platform chain. Prefer this term (or synonyms *eval score* / *eval judge*) over the bare word "evals," which is ambiguous with [eval scenarios](#eval-scenario).
+See [testing-agents.md](problems/testing-agents.md) and [Observability](#observability).
+
+### Eval Scenario
+
+A fixed, reproducible test case — a concrete input with an expected outcome that you re-run when an agent changes. Example: triage is presented with an issue asking to add a cheeseburger to the README and is expected to reject and close it. Scenarios are maintained like tests: if intentional agent behavior changes, update the scenario expectations. They answer "did this change make the agent better or worse on known cases?" and can later grow by promoting interesting production cases from telemetry into the curated set. Distinct from [eval measurements](#eval-measurement) (the scores/judges applied to a scenario or to wild traffic). Prefer this term over the bare word "evals."
+See [testing-agents.md](problems/testing-agents.md) (golden-set evaluation).
+
### Entry Point

Relevance

⭐⭐⭐ High

Glossary introduced as alphabetically organized for scannability; keeping E-section sorted matches
stated convention.

PR-#136
PR-#194

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Other letter sections demonstrate alphabetical ordering by term name (e.g., A: Agent
Infrastructure/Registry/Runtime/Automerge; D: Debouncing/Default Agent/Derived Agent). In contrast,
the E section now lists Eval ... before Entry Point/Escalation, indicating the new entries
disrupted the established ordering convention.

docs/glossary.md[9-29]
docs/glossary.md[50-65]
docs/glossary.md[67-92]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`docs/glossary.md` appears to keep terms alphabetically ordered within each letter section (A, C, D, etc.). The PR inserts `Eval Measurement` and `Eval Scenario` at the top of the `## E` section, but alphabetically `Entry Point` (En) and `Escalation` (Es) should come before `Eval ...` (Ev). This breaks the established glossary convention and makes the section harder to scan.

## Issue Context
Current order in `## E` is:
- Eval Measurement
- Eval Scenario
- Entry Point
- Escalation
- Evergreen

Expected alphabetical order is:
- Entry Point
- Escalation
- Eval Measurement
- Eval Scenario
- Evergreen

## Fix Focus Areas
- docs/glossary.md[67-92]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[ascerra]

Addressed Qodo: reordered E section alphabetically — Entry Point → Escalation → Eval Measurement → Eval Scenario → Evergreen.

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 7:55 PM UTC · Completed 8:02 PM UTC
Commit: 8a12345 · View workflow run →

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[fullsend-ai-review]

Looks good to me

---

Labels: PR modifies glossary documentation under docs/.

[ralphbean]

ty!

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 8:52 PM UTC · Completed 8:59 PM UTC
Commit: 8a12345 · View workflow run →

[fullsend-ai-retro]

Retro: PR #4874 — docs(glossary): define eval scenario vs eval measurement

Verdict: Clean workflow, no new proposals. All improvement areas identified are already tracked by existing open issues.

Timeline

Time (UTC)	Event
2026-07-14 19:49	PR opened — 1 file, +10 lines to docs/glossary.md (ascerra, co-authored with Cursor)
2026-07-14 19:50	Review agent dispatched (run 29363336672)
2026-07-14 19:52	Qodo correctly flags alphabetical ordering issue in E section
2026-07-14 19:53	Author pushes fix commit; first review run cancelled via concurrency group
2026-07-14 19:54	Second review agent dispatched (run 29363600581)
2026-07-14 20:02	Review agent approves — "Looks good to me"
2026-07-14 20:18	ralphbean approves — "ty!"
2026-07-15 06:19	rh-hemartin approves (no body)
2026-07-20 20:50	PR merged (6 days after full approval)

What went well

• Qodo caught a real issue (alphabetical ordering) within 2 minutes of PR creation, and the author fixed it within 2 minutes.
• Review agent correctly approved a clean docs-only change on the first iteration, matching both human reviewers' judgment.
• Concurrency group worked as designed — the stale first review run was cancelled when a new commit was pushed, and the second run completed successfully.
• No false positives — neither the review agent nor Qodo raised spurious issues on the final commit.

Observations mapped to existing issues

• Cancelled first review run (~4 min wasted): The first run consumed ~4 minutes and 26 tool calls before cancellation. Debouncing proposals in #1014 and #4960 would prevent dispatching a review before rapid follow-up commits settle.
• Review overhead for trivial docs changes: The successful run took ~8 minutes with 58 tool calls (including 4 sub-agent dispatches) for a 10-line glossary addition. #1900 proposes a fast path for trivial docs/prompt-only changes that would reduce this overhead.
• 6-day merge delay after full approval: The PR was fully approved (1 bot + 2 humans) by 2026-07-15 but not merged until 2026-07-20. #966 and #1974 cover surfacing and escalating agent-approved PRs awaiting human action.
• Autonomy readiness evidence: The review agent's approval fully matched both human reviewers on this docs-only change. This corroborates the pattern tracked in #4751, #4765, and #5289 — review agent verdicts on docs-only PRs consistently align with human approvals.