docs: add applied agent-eval-tools doc

[Benkapner]

Summary

Adds docs/problems/applied/agent-eval-tools/README.md covering how fullsend's problem areas manifest in agent-driven tools that evaluate agent configurations (the "who watches the watchmen" angle). Follows the existing applied-problems pattern established by applied/konflux-ci/.

Names and links harness-eval as the concrete example, includes a Technology landscape section, and maps four problem areas (testing-agents, MCP config drift, tool call risk assessment, trustworthiness evidence) to the evaluation tooling context.

Testing

• Docs-only change, no code/lint/test impact

Checklist

• PR title follows Conventional Commits
• Commits are signed off (DCO)

[github-actions]

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

[github-actions]

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

[qodo-code-review]

PR Summary by Qodo

Docs: cross-reference problem docs in roadmap and add applied harness-eval note

📝 Documentation 🕐 10-20 Minutes

AI Description

• Add roadmap cross-references to relevant problem documents for Testing and Security hardening.
• Add an applied problem doc on agent setup evaluation tools (“who watches the watchmen”).
• Improve discoverability of existing problem areas from the roadmap.

Diagram

graph TD
  A["docs/roadmap.md"] --> B["Problem docs (existing)"]
  C["docs/problems/applied/harness-eval/README.md"] --> B

Loading

High-Level Assessment

The chosen approach (explicit cross-links in roadmap sections plus an applied-problems writeup) is the most direct way to improve discoverability without adding new navigation structure. Alternatives like a centralized index or autogenerated link sections would add maintenance overhead for limited benefit at this scale.

Files changed (2) +35 / -0

Documentation (2) +35 / -0

README.mdAdd applied problem doc for agent setup evaluation tooling +31/-0

Add applied problem doc for agent setup evaluation tooling

• Introduces a new applied-problems README explaining how core fullsend problem areas manifest in agent configuration evaluation tools (linters/scanners). Cross-references relevant existing problem docs and calls out tooling-specific considerations like recursive evaluation and multi-assistant format support.

docs/problems/applied/harness-eval/README.md

roadmap.mdAdd “Related problem docs” cross-links in Testing and Security hardening sections +4/-0

Add “Related problem docs” cross-links in Testing and Security hardening sections

• Adds inline “Related problem docs” link lists to the Testing and Security hardening roadmap sections to connect roadmap initiatives to previously-merged problem documents. This improves navigation between strategic roadmap items and deeper problem-area writeups.

docs/roadmap.md

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

1. Audit log link unanchored ✓ Resolved 🐞 Bug ≡ Correctness

Description

In docs/roadmap.md, the “audit-log-integrity” related-doc item links to
problems/security-threat-model.md without the section anchor, so readers don’t land on the audit log
integrity section the label implies. This makes the roadmap cross-reference harder to use and easy
to misinterpret.

Code

docs/roadmap.md[204]

+Related problem docs: [audit-log-integrity](problems/security-threat-model.md), [MCP config drift](problems/mcp-config-drift.md), [tool call risk assessment](problems/tool-call-risk-assessment.md)

Relevance

●●● Strong

They frequently accept fixes to docs cross-references/anchors so link targets match the referenced
section.

PR-#226
PR-#770

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The roadmap’s new related-docs line uses link text “audit-log-integrity” but points only to
security-threat-model.md (no fragment). The threat model contains a specific “audit log integrity”
section with a stable GitHub heading anchor that can be linked directly.

docs/roadmap.md[193-205]
docs/problems/security-threat-model.md[424-426]
PR-#226

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`docs/roadmap.md` adds a related-doc link labeled `audit-log-integrity`, but the URL points to the top of `problems/security-threat-model.md` rather than the specific “audit log integrity” section.

### Issue Context
The security threat model contains a dedicated heading `## Cross-cutting concern: audit log integrity`, so the roadmap link can deep-link directly to that section.

### Fix Focus Areas
- docs/roadmap.md[193-205]
- docs/problems/security-threat-model.md[424-426]

### Suggested change
Update the roadmap link to include the heading fragment, e.g.:
- `[audit log integrity](problems/security-threat-model.md#cross-cutting-concern-audit-log-integrity)`

(Optionally align the link text casing with other items.)

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Applied index omits doc ✓ Resolved 🐞 Bug ⚙ Maintainability

Description

This PR adds docs/problems/applied/harness-eval/README.md but doesn’t add it to
docs/problems/applied/README.md, leaving the new applied doc undiscoverable from the applied-doc
index referenced elsewhere in the docs. Readers following the applied-docs entry point won’t see the
new document.

Code

docs/problems/applied/harness-eval/README.md[R1-3]

+# Applied: Agent Setup Evaluation Tools
+
+How fullsend's problem areas manifest in tools that evaluate agent configurations.

Relevance

●●● Strong

Repo has accepted adding new docs to an index/README to keep them discoverable.

PR-#2549

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The applied index currently lists only konflux-ci, while the PR introduces a new applied doc
directory; vision.md explicitly directs readers to the applied docs entry point, so omitting the new
item makes it harder to find.

docs/problems/applied/README.md[5-8]
docs/problems/applied/harness-eval/README.md[1-3]
docs/vision.md[52-52]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
A new applied doc directory `applied/harness-eval/` is added, but `docs/problems/applied/README.md` is not updated to list it.

### Issue Context
`docs/vision.md` points readers to `problems/applied/` for applied docs, so the applied index should enumerate new entries.

### Fix Focus Areas
- docs/problems/applied/README.md[5-8]
- docs/vision.md[52-52]

### Suggested change
Add a new bullet under “Current consumers” (or rename that section to something more general) such as:
- `- **[harness-eval](harness-eval/)** — Applied considerations for agent setup evaluation tools.`

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Plaintext problem references ✓ Resolved 🐞 Bug ⚙ Maintainability

Description

docs/problems/applied/harness-eval/README.md references related problem docs as plain filenames
(e.g., “(testing-agents.md)”), which aren’t clickable and are inconsistent with the established
applied-doc pattern of using relative Markdown links. This reduces navigability and makes it harder
to jump from the applied doc to the underlying problem docs.

Code

docs/problems/applied/harness-eval/README.md[R11-25]

+### Testing agents (testing-agents.md)
+
+An evaluation tool's own agent setup (skills, commands, hooks) needs the same static analysis it provides to others. This is the "who watches the watchmen" problem. The tool must dogfood its own checks, and its CI must gate on its own lint and security rules. Failure to do this means the tool's own configuration can drift into the patterns it flags for others.
+
+### MCP configuration drift (mcp-config-drift.md)
+
+Evaluation tools that integrate with MCP servers (for LLM-based review) face config drift when new MCP servers are added or removed. The tool's own cross-component analysis (phantom MCP detection) directly addresses this problem for downstream users, but the tool itself must also keep its own MCP config current.
+
+### Tool call risk assessment (tool-call-risk-assessment.md)
+
+Evaluation tools that run security scans must not themselves become attack vectors. A malicious skill under evaluation could contain patterns designed to influence the evaluator's behavior (anti-jailbreak patterns, evaluator-targeted prompt injection). The tool needs its own defense against adversarial inputs, which is distinct from the defenses it provides to users.
+
+### Trustworthiness evidence (trustworthiness-evidence.md)
+
+For an evaluation tool, trustworthiness evidence takes a specific form: false positive rate, false negative rate, and rule accuracy over time. If the tool flags too many false positives, teams disable it. If it misses real issues, teams lose trust. Tracking these metrics is how the tool earns continued adoption.

Relevance

●●● Strong

Precedent: convert plain-text doc references into explicit relative Markdown links for
consistency/navigation.

PR-#770

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The new harness-eval applied doc uses filename-only references in the headings, while an existing
applied doc demonstrates the repo’s preferred pattern: real Markdown links with correct relative
paths to docs/problems/*.md.

docs/problems/applied/harness-eval/README.md[11-25]
docs/problems/applied/konflux-ci/README.md[32-35]
PR-#226

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The new applied doc names related problem docs as bare filenames in parentheses, rather than actual Markdown links.

### Issue Context
Existing applied docs link to the general problem docs using correct relative paths (e.g. `../../architectural-invariants.md`). From `docs/problems/applied/harness-eval/README.md`, the general problem docs live at `../../<problem>.md`.

### Fix Focus Areas
- docs/problems/applied/harness-eval/README.md[11-25]
- docs/problems/applied/konflux-ci/README.md[32-35]

### Suggested change
Convert headings to links, e.g.:
- `### [Testing agents](../../testing-agents.md)`
- `### [MCP configuration drift](../../mcp-config-drift.md)`
- `### [Tool call risk assessment](../../tool-call-risk-assessment.md)`
- `### [Trustworthiness evidence](../../trustworthiness-evidence.md)`

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

4. harness-eval lacks solution options ✓ Resolved 📘 Rule violation ⌂ Architecture

Description

The new problem document under docs/problems/ does not present at least two solution options with
clearly labeled trade-offs, which makes it read as descriptive background rather than an
options-and-tradeoffs problem doc.

Code

docs/problems/applied/harness-eval/README.md[R1-31]

+# Applied: Agent Setup Evaluation Tools
+
+How fullsend's problem areas manifest in tools that evaluate agent configurations.
+
+## Context
+
+Agent setup evaluation tools (linters, security scanners for skills/commands/agents/hooks) are themselves agent-driven systems. They face a subset of the same challenges fullsend faces, viewed from the tooling side rather than the platform side.
+
+## Relevant problem areas
+
+### Testing agents (testing-agents.md)
+
+An evaluation tool's own agent setup (skills, commands, hooks) needs the same static analysis it provides to others. This is the "who watches the watchmen" problem. The tool must dogfood its own checks, and its CI must gate on its own lint and security rules. Failure to do this means the tool's own configuration can drift into the patterns it flags for others.
+
+### MCP configuration drift (mcp-config-drift.md)
+
+Evaluation tools that integrate with MCP servers (for LLM-based review) face config drift when new MCP servers are added or removed. The tool's own cross-component analysis (phantom MCP detection) directly addresses this problem for downstream users, but the tool itself must also keep its own MCP config current.
+
+### Tool call risk assessment (tool-call-risk-assessment.md)
+
+Evaluation tools that run security scans must not themselves become attack vectors. A malicious skill under evaluation could contain patterns designed to influence the evaluator's behavior (anti-jailbreak patterns, evaluator-targeted prompt injection). The tool needs its own defense against adversarial inputs, which is distinct from the defenses it provides to users.
+
+### Trustworthiness evidence (trustworthiness-evidence.md)
+
+For an evaluation tool, trustworthiness evidence takes a specific form: false positive rate, false negative rate, and rule accuracy over time. If the tool flags too many false positives, teams disable it. If it misses real issues, teams lose trust. Tracking these metrics is how the tool earns continued adoption.
+
+## Unique considerations
+
+- **Recursive evaluation:** the tool must be able to evaluate its own setup without circular dependency issues
+- **Rule accuracy feedback loop:** users who suppress findings or override verdicts generate signal about rule quality
+- **Multi-tool support:** unlike fullsend (which targets a specific platform), evaluation tools must handle multiple AI assistants (Claude Code, Cursor, Copilot, Gemini, OpenCode) with different configuration formats

Relevance

● Weak

Team has rejected “must add options/trade-offs” structure enforcement for problem docs in similar
cases.

PR-#3063
PR-#2549

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

PR Compliance ID 1062035 requires each new/modified file under docs/problems/ to include at least
two distinct options with explicit trade-offs. The added
docs/problems/applied/harness-eval/README.md contains context and relevant-problem-area
descriptions but no Option 1/Option 2 (or equivalent) sections with pros/cons or risks.

Rule 1062035: Problem docs must present multiple options with trade-offs, not a single prescribed solution
docs/problems/applied/harness-eval/README.md[1-31]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new problem doc does not describe at least two distinct solution options/approaches and their trade-offs.

## Issue Context
Compliance requires problem docs under `docs/problems/` (including `applied/`) to document multiple options with clearly labeled trade-offs (pros/cons, risks), even if one option is recommended.

## Fix Focus Areas
- docs/problems/applied/harness-eval/README.md[1-31]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

✅ Compliance rules (platform): 54 rules

✅ Skills: writing-user-docs, writing-adrs

Tip of the day

💡 Did you know, you can start a comment with 'qodo' or '@qodo' to chat about any finding

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[github-actions]

Site preview

Preview: https://6abbaaaa-site.fullsend-ai.workers.dev

Commit: b9b43238beaf100b9eb792a387e02edbfacfa009

[rh-hemartin]

Remove the "related problem docs" from the roadmap, the convention does not exist as far as I can see and I don't want to start that convention.

[rh-hemartin]

I agree with the document, but I do think that you need to refer to them more specfically as "agent-driven tools", for example here:

# Agent Setup Evaluation Tools

How fullsend's problem areas manifest in tools that evaluate agent configurations.

I would go with

# Agent-drive Evaluation Tools

How Fullsend's own problem areas appear in agent-driven tools  that evaluate agent configurations (or "who watches the watchmen" problem).

[Benkapner]

addressed all notes @rh-hemartin :

• removed the "Related problem docs" lines from the roadmap sections, won't start that convention
• renamed the applied doc to "Agent-Driven Evaluation Tools" per your suggestion
• qodo findings were already fixed in the previous push (section anchor on audit-log link, relative markdown links, applied index updated)

[waynesun09]

[MEDIUM] PR title/description still describe the reverted roadmap cross-reference work

The PR title ("docs: cross-reference problem docs in roadmap, add applied harness-eval doc") and the body's "Roadmap cross-references" section describe adding "Related problem docs" links to two docs/roadmap.md sections. That work was added in commit 4b51271 but fully reverted in the final commit d030fa8 after reviewer feedback ("the convention does not exist... I don't want to start that convention"), confirmed by the author's own follow-up comment that the "Related problem docs" lines were removed from the roadmap. git diff main...pr for docs/roadmap.md is now empty — the final diff touches only docs/problems/applied/README.md and docs/problems/applied/harness-eval/README.md. Nobody updated the PR title/body to match.

Concrete downstream impact: .goreleaser.yml's changelog exclude filter drops ^docs(\(.*\))?: commits, so this docs-only title won't leak into the public release-notes changelog. However, this repo's squash-merge title default is "commit or PR title" — since this PR has 3 commits, GitHub will default the squash commit's title to this (stale) PR title, permanently misdescribing the change in git log, even though the squash body would default to the accurate concatenated individual commit messages.

Suggestion: Edit the PR title (e.g., "docs: add applied harness-eval doc") and delete the body's "Roadmap cross-references" section before merging, so the merged commit accurately reflects that only the applied/harness-eval addition shipped.

---

Review-only pass; not self-assigning or requesting changes.

[Benkapner]

fixed many things in the latest push. PR title updated to "docs: add applied agent-eval-tools doc" and body rewritten to match. @waynesun09 thank you for the review

[waynesun09]

Review-only pass (follow-up); not self-assigning or requesting changes.

One additional candidate finding (cross-linking testing-agents.md's harness-eval-lab reference) was skipped as already covered by the earlier review thread's suggestions.

[waynesun09]

[HIGH] DCO check fails on final commit — missing required Signed-off-by trailer

The branch tip, commit 5194d42f ("fix(docs): address review feedback on agent-eval-tools doc"), is failing the DCO check (gh pr checks 5518 shows DCO as fail; the check run's conclusion is action_required). The commit's author/committer is a human account, not a bot, and its message ends only with an AI-assistant attribution trailer — no Signed-off-by: line. All four earlier commits on this branch end with a proper Signed-off-by: trailer.

CONTRIBUTING.md and AGENTS.md both state that human-driven agent sessions should sign off — the human directing the session is the one certifying the DCO — with the bot-identity exemption applying only to fully autonomous agent commits (author type Bot). This is a human-authored, assistant-assisted commit, so per the repo's own written policy it needs sign-off, and currently lacks it. This also currently falsifies this PR's own checklist line "Commits are signed off (DCO)".

Caveat: the repo's active branch ruleset's required_status_checks list doesn't explicitly name DCO, so I can't confirm with certainty this technically blocks the GitHub merge button. Regardless, it's a real, currently-failing check that contradicts repo policy.

Suggestion: Since this commit is the branch tip, git commit --amend -s --no-edit and force-push (no rebase needed) so it carries a Signed-off-by: trailer like every other commit on the branch.

---

Review-only pass; not self-assigning or requesting changes.

[Benkapner]

Pushing back on the directory rename ("agent-eval-tools" to something else): the disambiguation parenthetical on first mention already distinguishes harness-eval from agent-eval-harness in prose. Renaming the directory would break the existing index entry in applied/README.md and any external links. The directory name describes the doc's topic, not the tool itself. @waynesun09

except for that i fixed all findings:

• Fixed DCO (commit now has Signed-off-by)
• Added harness-eval as a concrete fullsend consumer (citing .fullsend/config.yaml and bot-authored PR docs: add pipeline-execution-feedback problem doc #8)
• Renamed title to "Agent Setup Evaluation Tools", scoped agent-driven language to optional LLM modes
• Replaced "pre-commit hooks" with "Tekton tasks" per actual install docs
• Converted ADR-0051 and runtimes.md to proper relative markdown links

[waynesun09]

Review-only pass; not self-assigning or requesting changes.

[Benkapner]

On "Technology landscape generalizes from a single tool" - the section describes how this category of tools works, with harness-eval as the named example. The purpose of the applied doc is to map fullsend problems to this tooling context, not to survey the landscape. The specifics (CLI delivery, YAML/JSON parsing, AST analysis) are properties of the problem space, not just harness-eval. Adding a second tool name would be arbitrary since the tools i know that does something close to it are not relevant.. dont you think we can say harness-eval? @waynesun09

On "Doc structure doesn't satisfy applied/README.md checklist": acknowledged. The doc maps 4 of ~30 problem areas because those are the ones that meaningfully manifest differently in evaluation tooling. Adding coverage-for-coverage-sake would dilute the doc. Happy to add a "Why this is interesting" sentence if that resolves it, but restructuring to mirror konflux-ci's breadth doesn't match the scope of this contribution.

[rh-hemartin]

I'm going to resolve everything and merge.