docs(problems): add static analysis layer to testing-agents

[Benkapner]

Summary

Adds a new section to docs/problems/testing-agents.md covering static analysis for agent configurations, a complementary testing layer to the behavioral approaches already documented.

The document currently focuses on testing agent behavior (golden-set evaluation, behavioral contracts, canary deployments, mutation testing) and surveys tools for that purpose. This contribution adds a missing layer: testing the agent configuration itself without executing anything. It covers two levels of analysis:

• Component-level - checking each skill, command, hook, and context file individually for structural integrity, security patterns, and token budget
• Setup-level - checking how components relate as a system: redundancy detection, dependency validation, token budget distribution, trigger overlap, and dimension scoring

This work is similar to an evaluation framework i developed that implements these static analysis and setup-level checks for Claude Code configurations. My tool runs 24 lint rules across 7 categories, performs system-level analysis (budget, triggers, dependencies), and scores configurations across dimensions like soundness, safety, coherence, and efficiency.

Follow-up

i would like to open a follow-up issue to explore deeper integration between this evaluation framework and fullsend, including contributing the tool itself or adapting its approach for fullsend's agent configuration testing pipeline (Step 1 of the CI pipeline described in the doc).

[fullsend-ai-review]

Review

Findings

Medium

• [vendor-attribution-risk] docs/problems/testing-agents.md — The inline reference to harness-eval-lab with the claim it "has been applied to production setups" creates an asymmetry with how other tools are introduced in this document. Every other framework (promptfoo, deepeval, Inspect AI, METR Vivaria, DeepTeam, lightspeed-evaluation) receives a dedicated subsection with trade-off analysis. The harness-eval-lab reference is presented inline as a validating example without comparable critical treatment, and the PR author owns the linked repository. Consider either: (1) removing the specific framework reference and keeping the discussion tool-agnostic, or (2) adding harness-eval-lab to the Eval frameworks section with the same trade-off analysis applied to other tools, with an explicit note about the author's relationship to the tool.

Low

• [heading-hierarchy-and-style] docs/problems/testing-agents.md:307 — The heading "Elaborating on Step 1: static analysis for agent configurations" uses gerund/meta framing ("Elaborating on") which is inconsistent with other level-3 headings in the document. All other headings use declarative or noun-phrase style (e.g., "Non-determinism", "Prompt evaluation (unit-test layer)"). Consider: "Static analysis for agent configurations".

• [trade-offs-framing] docs/problems/testing-agents.md:338 — Trade-offs are listed under a "Trade-offs:" bold label. The document's established pattern uses ### Trade-offs as a heading (lines 80, 109, 133, 157). Given the current nesting under a ### heading, #### Trade-offs may be the appropriate level to maintain both consistency and hierarchy.

• [structural-imbalance] docs/problems/testing-agents.md — The new subsection adds 47 lines elaborating on Step 1 of the CI pipeline, while Steps 2–5 receive no comparable elaboration. The CI pipeline section itself is 7 lines. The Approaches 1–4 sections are each 20–35 lines as peer sections. Consider restructuring as a peer approach ("Approach 5: Static configuration analysis") or condensing to match the CI pipeline section's summary style.

Info

• [subsection-structure] docs/problems/testing-agents.md:317 — Level-4 headings (####) are used for "Component-level analysis" and "Setup-level analysis." Existing #### headings in the Eval frameworks section are all tool names, though the use of #### for conceptual subsections under a ### heading is standard Markdown hierarchy.

• [terminology-consistency] docs/problems/testing-agents.md:310 — The section introduces "agent configurations" as a formal term distinct from "agent instructions." The distinction is explicitly defined and conceptually useful for static analysis, which operates on the broader configuration. The existing CI pipeline heading (line 295) already uses "agent configurations," so the term is not entirely new to the document.

• [missing-authorization] docs/problems/testing-agents.md — This PR adds 47 lines without a linked issue. The PR body mentions intent to open a follow-up issue. The project welcomes contributions to exploratory problem documents, and the requires-manual-review label is applied.

• [scope-coherence] docs/problems/testing-agents.md — The added content expands the document's scope from behavioral testing to structural validation of agent configurations. The new section explicitly states it "does not test behavior at all." This may signal the problem space is large enough to warrant a separate document in the future.

---

Labels: PR adds content to an exploratory problem document by an external contributor.

Previous run

Review

Findings

Medium

• [citation-and-reference-style] docs/problems/testing-agents.md:313 — The inline citation of harness-eval-lab with the claim "has been applied to production setups" is inconsistent with how other tools are introduced in this document. All other evaluation frameworks (promptfoo, deepeval, lightspeed-evaluation, Inspect AI, METR Vivaria, DeepEval Synthesizer, DeepTeam) are introduced in dedicated subsections with objective capability descriptions. The inline self-referential citation with an unsubstantiated maturity claim breaks this established pattern. Consider either: (1) removing the production claim and describing capabilities objectively (e.g., "An open-source evaluation framework implements these checks for Claude Code configurations"), or (2) moving the reference to the Eval frameworks section with a dedicated subsection matching the pattern used for other tools. See also: [external-reference] info finding.

Low

• [heading-hierarchy-and-style] docs/problems/testing-agents.md:307 — The heading "Elaborating on Step 1: static analysis for agent configurations" uses gerund/meta framing. Other headings in this document use declarative or noun-phrase style ("Non-determinism", "Behavioral surface area", "Practical architecture", "Measuring agent capability drift"). Consider: "Static analysis for agent configurations" or "Static analysis layer (Step 1)".

• [terminology-consistency] docs/problems/testing-agents.md:309 — The section introduces "agent configurations" as a term distinct from "agent instructions," but the rest of the document uses "agent instructions" inclusively (~15 occurrences). The distinction is conceptually useful — static analysis operates on the broader configuration, not just instruction text — but introducing it mid-document without establishing it in the framing sections creates a vocabulary split. If the distinction is load-bearing, consider introducing it earlier (e.g., in "What needs testing") or adding it to docs/glossary.md. If not, the existing "agent instructions" term suffices with a note that static analysis covers supporting components as well.

• [subsection-structure] docs/problems/testing-agents.md:315 — Level-4 headings (####) are used for Component-level and Setup-level sections. The rest of this document uses level-4 headings exclusively for specific tool names within the Eval frameworks section. All other conceptual subsections use level-3 (###) headings. Consider promoting these to level-3 to match the established pattern.

• [missing-authorization] docs/problems/testing-agents.md — This PR adds 47 lines introducing a new conceptual layer without a linked issue. The project welcomes contributions to exploratory problem documents and CLAUDE.md explicitly encourages expanding docs/problems/, so this is not blocking. Filing an issue retrospectively would provide traceability and a discussion space for the static analysis layer's relationship to existing approaches — the PR author mentions wanting to open a follow-up issue.

• [structural-coherence] docs/problems/testing-agents.md:307 — At 47 added lines, this subsection of one CI pipeline step is denser than comparable top-level approaches (Golden-set evaluation: ~40 lines, Behavioral contracts: ~20 lines). The document's Approaches 1–4 are parallel top-level sections; this elaboration of Step 1 creates structural imbalance where one pipeline step gets deep treatment while Steps 2–4 get one sentence each. Consider whether this warrants its own top-level approach section (e.g., "Approach 5: Static configuration analysis").

Info

• [trade-offs-framing] docs/problems/testing-agents.md:338 — Trade-offs are listed under a "Trade-offs:" bold label, whereas existing approach sections use "### Trade-offs" headings with "Pros:" / "Cons:" subsections. Minor stylistic variation — either works, but consistency with the established pattern would be cleaner.

• [external-reference] docs/problems/testing-agents.md:313 — The reference to the author's own framework appears inline in a conceptual section rather than in the Eval frameworks section where other tools are surveyed with dedicated subsections. The inline treatment is defensible since static analysis tooling doesn't fit cleanly into the behavioral evaluation categories, but the appearance differs from how other tools are introduced. See also: [citation-and-reference-style] medium finding.

Previous run (2)

Review

Findings

Medium

• [citation-and-reference-style] docs/problems/testing-agents.md:313 — The reference to the author's personal project (harness-eval-lab) is embedded inline with an unsubstantiated "has been applied to production setups" claim. Other tools in this document (promptfoo, deepeval, Inspect AI, DeepTeam) are introduced by name in dedicated subsections with capabilities described objectively. The inline self-referential citation with maturity claims is inconsistent with that pattern. Consider either: (1) citing the tool more neutrally without production claims (e.g., "An open-source implementation of these checks for Claude Code configurations exists"), or (2) giving it a dedicated subsection like the other surveyed tools if it warrants detailed treatment.

Low

• [heading-hierarchy-and-style] docs/problems/testing-agents.md:307 — The heading "Elaborating on Step 1: static analysis for agent configurations" uses gerund/meta framing. Existing headings in this document use declarative or noun-phrase style ("Why this is a distinct problem", "What makes agent testing hard", "CI pipeline for agent configurations"). Consider: "Static analysis for agent configurations" or "Static analysis layer (Step 1)".

• [terminology-consistency] docs/problems/testing-agents.md:309 — The section introduces "agent configurations" as a new term distinct from "agent instructions," but the rest of the document (and the project glossary) uses "agent instructions" inclusively (~15 occurrences). If the distinction is load-bearing, it should be adopted more broadly or added to docs/glossary.md. If not, the existing "agent instructions" term suffices with a note that static analysis covers supporting components (skills, hooks, context files) as well.

• [scope-granularity] docs/problems/testing-agents.md:307 — At 53 added lines, this subsection of one CI pipeline step is denser than comparable top-level approaches (Golden-set evaluation: 40 lines, Behavioral contracts: 20 lines). The document does have variable depth elsewhere (Eval frameworks: 115 lines), so this isn't strictly inconsistent, but consider whether component-level, setup-level, and LLM-based scoring warrant their own top-level approach section (e.g., "Approach 5: Static configuration analysis") rather than nesting under the CI pipeline.

• [subsection-structure] docs/problems/testing-agents.md:315 — Level-4 headings (####) are used for Component-level, Setup-level, and Optional sections. The rest of this document uses only level-2 and level-3 headings. If promoted to a top-level approach section, these would naturally become level-3 headings matching the existing pattern.

Info

• [trade-offs-framing] docs/problems/testing-agents.md:338 — Trade-offs are listed under a "Trade-offs:" bold label, whereas existing approach sections use "Pros/Cons" subsections. Minor stylistic variation — either works, but consistency with established pattern would be cleaner.

Previous run (3)

Review

Findings

Low

• [list-formatting-pattern] docs/problems/testing-agents.md — The Component-level and Setup-level subsections use bold-label-with-period format (**Structural integrity.** Every component...) while the document's established pattern uses bullet lists with em-dash (- **Label** — ...). See the eval framework subsections (promptfoo, deepeval, Inspect AI) for the established convention. Converting to bulleted format would improve visual consistency.

• [terminology-consistency] docs/problems/testing-agents.md — The new section introduces "agent configurations" as a term distinct from "agent instructions" (with clear definitions in the opening paragraphs), but the rest of the document uses only "agent instructions" throughout (26+ occurrences). The distinction is conceptually useful — static analysis operates on the broader configuration, not just instruction text — but introducing it mid-document without updating framing sections creates a vocabulary split. Consider adding a brief note acknowledging the distinction or cross-referencing it earlier.

• [architectural-coherence] docs/problems/testing-agents.md — Component-level analysis lacks a dedicated trade-offs subsection. The document's Approaches 1–4 each have structured Trade-offs sections, and the Setup-level analysis subsection includes an inline Trade-offs list, but Component-level analysis has no equivalent. A brief note covering component-level limitations (e.g., metadata checks cannot validate instruction quality, security pattern scanning has false-positive risk from legitimate examples) would complete the pattern.

• [external-reference-pattern] docs/problems/testing-agents.md — The harness-eval-lab reference appears inline as a parenthetical GitHub link while other tools in the document (promptfoo, deepeval, Inspect AI, etc.) are introduced with dedicated subsections including name, link, description, and feature list. The inline treatment is defensible since static analysis tooling does not fit cleanly into the behavioral evaluation categories, but the author may want to add a brief disclosure that they created the referenced framework, for transparency.

• [scope-authorization] docs/problems/testing-agents.md — This PR adds a new conceptual layer without a linked issue. While the project welcomes contributions to exploratory problem documents and CLAUDE.md encourages expanding docs/problems/, filing an issue retrospectively would provide traceability and a discussion space for the static analysis layer's relationship to existing approaches. The PR author mentions wanting to open a follow-up issue.

Info

• [tonal-consistency] docs/problems/testing-agents.md — The author uses appropriately conditional/exploratory language throughout ("could identify", "may flag", "can measure", "would need tuning"). This aligns well with the document's established exploratory tone and CLAUDE.md's guidance that documents should present options with trade-offs rather than prescribe solutions.

Previous run (4)

Review

Findings

Low

• [list-formatting-pattern] docs/problems/testing-agents.md — The Component-level and Setup-level subsections use bold-label-with-period format (**Structural integrity.** Every component...) while the document's established pattern uses bullet lists with em-dash (- **Label** — ...). See the eval framework subsections (promptfoo, deepeval, Inspect AI) for the established convention. Converting to bulleted format would improve visual consistency.

• [terminology-consistency] docs/problems/testing-agents.md — The new section introduces "agent configurations" as a term distinct from "agent instructions" (with clear definitions in the opening paragraphs), but the rest of the document uses only "agent instructions" throughout (26+ occurrences). The distinction is conceptually useful — static analysis operates on the broader configuration, not just instruction text — but introducing it mid-document without updating framing sections creates a vocabulary split. Consider adding a brief note acknowledging the distinction or cross-referencing it earlier.

• [architectural-coherence] docs/problems/testing-agents.md — Component-level analysis lacks a dedicated trade-offs subsection. The document's Approaches 1–4 each have structured Trade-offs sections, and the Setup-level analysis subsection includes an inline Trade-offs list, but Component-level analysis has no equivalent. A brief note covering component-level limitations (e.g., metadata checks cannot validate instruction quality, security pattern scanning has false-positive risk from legitimate examples) would complete the pattern.

• [scope-authorization] docs/problems/testing-agents.md — This PR adds a new conceptual layer without a linked issue. While the project welcomes contributions to exploratory problem documents and CLAUDE.md encourages expanding docs/problems/, filing an issue would provide traceability and a discussion space for the static analysis layer's relationship to existing approaches. The PR author mentions wanting to open a follow-up issue.

• [external-reference-pattern] docs/problems/testing-agents.md — The harness-eval-lab reference appears inline as a parenthetical GitHub link while other tools in the document (promptfoo, deepeval, Inspect AI, etc.) are introduced with dedicated subsections including name, link, description, and feature list. If the tool is meant as a surveyed framework, it would benefit from the established introduction pattern; if it is a reference implementation example, the inline mention is adequate. See also: [scope-coherence] note — static analysis tooling does not fit cleanly into the behavioral evaluation categories, so the inline treatment in a separate section is defensible.

Info

• [tonal-consistency] docs/problems/testing-agents.md — The author uses appropriately conditional/exploratory language throughout ("could identify", "may flag", "can measure", "would need tuning"). This aligns well with the document's established exploratory tone and CLAUDE.md's guidance that documents should present options with trade-offs rather than prescribe solutions.

Previous run (5)

Review

Findings

Low

• [list-formatting-pattern] docs/problems/testing-agents.md — The Component-level and Setup-level subsections use bold-label-with-period format (**Structural integrity.** Every component...) while the document's established pattern uses bullet lists with em-dash (- **Label** — ...). See Approaches 1–4 Pros/Cons sections for the established convention.

• [terminology-consistency] docs/problems/testing-agents.md — The new section introduces "agent configurations" as a term distinct from "agent instructions" (with clear definitions), but the rest of the document uses only "agent instructions" throughout (26+ occurrences). Introducing it in a subsection without updating the document's framing sections creates a vocabulary split. Consider introducing the distinction earlier or cross-referencing the Agent Harness component in docs/architecture.md.

• [architectural-coherence] docs/problems/testing-agents.md — The author has added trade-offs for Setup-level analysis and Optional LLM scoring (addressing prior review feedback), but Component-level analysis still lacks a dedicated trade-offs subsection. The general trade-offs (false confidence, lint maintenance) apply to component-level too, but the document's Approaches 1–4 each have structured Trade-offs sections. A brief note covering component-level limitations (e.g., metadata checks can't validate instruction quality, security pattern scanning has false-positive risk from legitimate examples) would complete the pattern.

• [scope-authorization] docs/problems/testing-agents.md — This PR adds a new conceptual layer without a linked issue. While the project welcomes contributions to exploratory problem documents, filing an issue would provide traceability and a discussion space for the static analysis layer's relationship to existing approaches.

• [external-reference-pattern] docs/problems/testing-agents.md — The harness-eval-lab reference appears inline while other tools in the document (promptfoo, deepeval, Inspect AI) are introduced with dedicated subsections. If the tool is meant as a surveyed framework, it would benefit from the established introduction pattern; if it is an implementation example, the inline mention is adequate.

Info

• [tonal-consistency] docs/problems/testing-agents.md — The author has improved the tone since the prior review, shifting from declarative to conditional/exploratory language ("could identify", "may flag", "can measure"). Remaining declarative statements are contextually appropriate as definitional rather than prescriptive.

Previous run (6)

Review

Findings

Medium

• [scope-classification-mismatch] docs/problems/testing-agents.md — The Component-level and Setup-level analysis subsections present techniques (TF-IDF similarity, token budget distribution, dependency validation, trigger overlap, dimension scoring) without trade-offs subsections. The author has improved the tone since the prior review — language is now appropriately conditional ("could identify", "may flag", "can measure") — and the section is correctly positioned as a subsection of the CI pipeline rather than a standalone top-level section. However, CLAUDE.md mandates exploratory framing with trade-offs, and Approaches 1–4 in this document each include explicit Trade-offs subsections. The Optional LLM-based rubric scoring subsection has trade-offs, but the core static analysis subsections do not. For example: redundancy detection via TF-IDF has real limitations (misses semantic similarity, thresholds are empirical, false positives from intentional overlap); dependency validation catches structural breaks but not semantic drift; token budget analysis requires defining "recommended" thresholds that may vary by use case.
  Remediation: Add a brief trade-offs subsection for the static analysis techniques, covering limitations like false positives, threshold tuning challenges, and the gap between structural validity and behavioral correctness. This would align with the pattern established by Approaches 1–4.

Low

• [terminology-consistency] docs/problems/testing-agents.md — The new section introduces "agent configurations" as a term distinct from "agent instructions" (with clear definitions), but the rest of the document uses only "agent instructions" throughout (26+ occurrences). While the distinction is conceptually useful, introducing it in a subsection without updating the document's framing sections (title, intro, "What needs testing") creates a vocabulary split.

• [list-formatting-pattern] docs/problems/testing-agents.md — The Component-level and Setup-level subsections use bold-label-with-period format ("Structural integrity. Every component...") while the document's established pattern uses bullet lists with em-dash ("- Label — ..."). See Approaches 1–4 Pros/Cons sections for the established convention.

• [missing-authorization] docs/problems/testing-agents.md — This PR adds 45 lines introducing a new conceptual layer without a linked issue. While the project welcomes contributions to exploratory problem documents, traceability to an issue helps reviewers understand the rationale.

Info

• [tonal-consistency] docs/problems/testing-agents.md — The author has significantly improved the tone since the prior review, shifting from declarative to conditional language. Some residual declarative statements remain ("Static analysis operates on...", "Both matter.") but these are reasonable for establishing the relationship between behavioral and structural testing.

• [cross-reference-pattern] docs/problems/testing-agents.md — The external framework reference (harness-eval-lab) appears inline without following the document's established tool-introduction pattern. Other tools (promptfoo, deepeval, Inspect AI) are introduced with dedicated subsections including name, description, and detailed context before discussion.

Previous run (7)

Review

Findings

Medium

• [scope-classification-mismatch] docs/problems/testing-agents.md — The Component-level and Setup-level analysis subsections present techniques (TF-IDF similarity, token budget distribution, dependency validation, dimension scoring) without trade-offs subsections. The author has improved the tone since the prior review — language is now appropriately conditional ("could identify", "may flag", "can measure") — and the section is correctly positioned as a subsection of the CI pipeline rather than a standalone top-level section. However, CLAUDE.md mandates exploratory framing with trade-offs, and Approaches 1–4 in this document each include explicit Trade-offs subsections. The Optional LLM-based rubric scoring subsection has trade-offs, but the core static analysis subsections do not. For example: redundancy detection via TF-IDF has real limitations (misses semantic similarity, thresholds are empirical, false positives from intentional overlap); dependency validation catches structural breaks but not semantic drift; token budget analysis requires defining "recommended" thresholds that may vary by use case.
  Remediation: Add a brief trade-offs subsection for the static analysis techniques, covering limitations like false positives, threshold tuning challenges, and the gap between structural validity and behavioral correctness. This would align with the pattern established by Approaches 1–4.

Low

• [terminology-consistency] docs/problems/testing-agents.md — The new section introduces "agent configurations" as a term distinct from "agent instructions" (with clear definitions), but the rest of the document uses only "agent instructions" throughout (26+ occurrences). While the distinction is conceptually useful, introducing it in a subsection without updating the document's framing sections (title, intro, "What needs testing") creates a vocabulary split.

• [list-formatting-pattern] docs/problems/testing-agents.md — The Component-level and Setup-level subsections use bold-label-with-period format ("Structural integrity. Every component...") while the document's established pattern uses bullet lists with em-dash ("- Label — ..."). See Approaches 1–4 Pros/Cons sections for the established convention.

• [missing-authorization] docs/problems/testing-agents.md — This PR adds 45 lines introducing a new conceptual layer without a linked issue. While the project welcomes contributions to exploratory problem documents, traceability to an issue helps reviewers understand the rationale.

Info

• [tonal-consistency] docs/problems/testing-agents.md — The author has significantly improved the tone since the prior review, shifting from declarative to conditional language. Some residual declarative statements remain ("Static analysis operates on...", "Both matter.") but these are reasonable for establishing the relationship between behavioral and structural testing.

• [cross-reference-pattern] docs/problems/testing-agents.md — The external framework reference (harness-eval-lab) appears inline without following the document's established tool-introduction pattern. Other tools (promptfoo, deepeval, Inspect AI) are introduced with dedicated subsections including name, description, and detailed context before discussion.

Previous run (8)

Review

Findings

Medium

• [scope-classification-mismatch] docs/problems/testing-agents.md — The new section presents static analysis techniques (component-level integrity, security patterns, token budgets, redundancy detection, dependency validation, trigger overlap, dimension scoring) in a prescriptive, declarative tone rather than the exploratory framing used elsewhere in the document. The existing approaches (1–4) are explicitly numbered with trade-off sections; the new content reads more like documentation of an established system. CLAUDE.md states: "This is a design exploration, not a spec. Documents should present multiple options with trade-offs, not prescribe single solutions." Only the optional LLM rubric scoring subsection includes a trade-offs section — the core static analysis subsections (component-level, setup-level) do not.
  Remediation: Reframe the content to align with the exploration mandate. Consider presenting static analysis as "Approach 5" with an explicit trade-offs section covering limitations (e.g., static analysis cannot catch semantic drift, threshold tuning for similarity scores is empirical, configuration linting does not validate behavioral correctness). Alternatively, add a brief trade-offs subsection for both component-level and setup-level analysis paralleling the pattern used in Approaches 1–4.

Low

• [missing-authorization] docs/problems/testing-agents.md — This PR adds 43 lines introducing a new conceptual layer (static analysis as distinct from behavioral testing) without a linked issue. While the project does not strictly require linked issues for section additions to existing problem documents, traceability to a discussion or issue helps reviewers understand the rationale and prevents undiscussed architectural concepts from entering the document without prior alignment.

• [terminology-consistency] docs/problems/testing-agents.md:317 — The new section introduces the term "agent configurations" while the existing document consistently uses "agent instructions" (26+ occurrences) and "instruction set." The CI pipeline section heading already uses "agent configurations" (line 295), creating ambiguity about whether these are the same concept. Consider aligning with "agent instructions" or adding a note clarifying the distinction.

• [heading-hierarchy] docs/problems/testing-agents.md:315 — The new section is placed between "Measuring agent capability drift" and "Relationship to other problem areas," but it thematically belongs near the "CI pipeline for agent configurations" section (line 295) which it explicitly references as Step 1. Moving it adjacent to the CI pipeline section would improve document flow.

• [architectural-coherence] docs/problems/testing-agents.md — The new section introduces analysis techniques (TF-IDF cosine similarity, token budget distribution) without explaining how they relate to the eval frameworks already surveyed or the broader CI pipeline architecture. The connection to CI pipeline Step 1 is stated but could be more explicit about whether this elaborates, replaces, or extends the existing Step 1 description.

• [external-reference-gap] docs/problems/testing-agents.md — The PR body mentions "an evaluation framework i developed that implements these static analysis and setup-level checks" but the added content provides no reference to this work. The document establishes a pattern of referencing external validation (e.g., Experiment 004 at line 173). Adding a reference would provide traceability.

• [list-formatting-pattern] docs/problems/testing-agents.md:325 — The component-level analysis subsection uses bold-label-with-period format ("Structural integrity. ...") while the document's established pattern uses bullet lists with em-dash ("- Label — ..."). Consider aligning with the existing convention.

Info

• [consistency] docs/problems/testing-agents.md — The new section uses present-tense declarative language ("Static checks can scan for...") while existing approach sections use more conditional, exploratory language. This tonal shift subtly suggests the static analysis layer is more established than the other approaches.

• [cross-reference-completeness] docs/problems/testing-agents.md — The "Optional: LLM-based rubric scoring" subsection describes a configuration evaluation approach that does not fit into the existing eval frameworks taxonomy (prompt evaluation, agent evaluation, input mutation). A brief note distinguishing it from those categories would aid reader comprehension.

Previous run (9)

Review

Findings

Medium

• [scope-clarity] docs/problems/testing-agents.md:323-354 — The new section presents static analysis techniques (TF-IDF cosine similarity at specific thresholds, dimension scoring, dependency validation) as determinate solutions rather than exploring trade-offs. CLAUDE.md states that problem documents should "present multiple options with trade-offs, not prescribe single solutions." The existing Approaches 1–4 in this document each include explicit Trade-offs subsections with Pros/Cons; the new section has none. For example, static analysis has real limitations worth discussing: false positives from similarity thresholds, maintenance burden of lint rules, inability to catch semantic issues, and the risk that passing static checks creates false confidence.
  Remediation: Add a trade-offs subsection discussing limitations of static analysis for agent configurations (false positives, maintenance cost, semantic blind spots). Consider reframing prescriptive statements (e.g., "A similarity threshold of around 0.85 flags likely duplicates") as options to evaluate rather than settled decisions.

Low

• [section-ordering] docs/problems/testing-agents.md:323 — The new section is placed between "Relationship to other problem areas" (a cross-reference summary) and "Open questions" (the document closer). Substantive content in this document consistently appears before both of these sections. Additionally, the CI pipeline section (line 299) already lists "Static analysis" as Step 1 — the new section expands on that concept without cross-referencing it, creating structural redundancy.
  Remediation: Move the section before "Relationship to other problem areas" and add a cross-reference to/from the CI pipeline's Step 1.

• [unclear-reference] docs/problems/testing-agents.md:338 — The phrase "agent configurations can be analyzed as systems with LLM (as skill.md)" is grammatically unclear and appears to be an incomplete or malformed reference. It is not clear what "with LLM (as skill.md)" is intended to convey.
  Remediation: Clarify or remove the parenthetical. If referring to a specific file, use a proper markdown link.

Info

• [external-contribution] docs/problems/testing-agents.md — External contribution with no linked issue. The author mentions a personal evaluation framework and intent to explore deeper integration in a follow-up. The contribution is self-contained as a docs addition to an exploratory problem document, which CLAUDE.md explicitly welcomes.

[Benkapner]

Thanks for the review. Addressed all three findings:

scope-clarity - Added an "Optional: LLM-based rubric scoring" subsection with explicit trade-offs (cost scaling, non-determinism, judge bias, dimension fit). The deterministic static analysis layer is intentionally presented without trade-offs since it's fast, free, and CI-friendly by nature. The trade-offs live where the LLM dependency is introduced.

section-ordering - Moved the section before "Relationship to other problem areas" and added a cross-reference to the CI pipeline's Step 1.

unclear-reference - Removed the "with LLM (as skill.md)" phrase.

[Benkapner]

Addressed the remaining findings:

scope-classification-mismatch - Added a trade-offs block for the static analysis subsections covering: empirical threshold tuning, TF-IDF missing semantic similarity, dependency validation not catching semantic drift, false confidence from passing structural checks, and lint rule maintenance burden.

terminology-consistency - The clarifying paragraph distinguishing "agent configurations" from "agent instructions" is intentional. The section operates on configurations (the broader structure, not just the text), and the distinction is defined in the intro. Happy to adjust if a maintainer prefers different framing.

list-formatting-pattern - Noted. Happy to reformat if a maintainer prefers the em-dash bullet convention for this section.

missing-authorization - Will open a follow-up issue for deeper integration work.

cross-reference-pattern - The inline reference to harness-eval-lab is intentional. A dedicated subsection would be disproportionate for what is currently a single reference, not a tool recommendation.

[Benkapner]

Squashed into a single signed-off commit to fix the DCO check. No content changes from the previous version.

@ifireball

[ralphbean]

Good addition — static analysis of agent configs is a real gap in this doc. A couple things to sort out inline.

[ralphbean]

[moderate] A few passages in the setup-level section read more like documentation for a specific tool than exploration of the design space. "TF-IDF cosine similarity" is presented as the technique for redundancy detection, and "a similarity threshold of around 0.85" / "around 0.50" read as calibration values from a particular implementation. The dimension scoring paragraph (soundness, safety, coherence, efficiency) has a similar feel.

I think we should present these as one option among several — embedding-based similarity, LLM-based comparison, etc. — and frame the thresholds as illustrative rather than recommended. Something like "one possible scoring taxonomy" for the dimensions would fit the doc's tone better.

The doc's convention (per CLAUDE.md: "present multiple options with trade-offs, not prescribe single solutions") is well-followed elsewhere — e.g., Approach 1 describes what golden-set evaluation is without prescribing a specific implementation. Bringing this section in line with that pattern would strengthen it.

[Benkapner]

i agree @ralphbean , TF-IDF was the cheapest technique i had from my implementation so i used it as an example, but you're right it reads like it's the only option so i changed it. The best approach would actually be LLM-based semantic matching if cost allows, but for CI gating you'd want something cheaper.
i reframed it to similarity detection is now presented as a range of techniques (TF-IDF for cheap/fast, embeddings for better accuracy, LLM-based for best semantic matching) with cost/accuracy trade-offs between them. Thresholds are framed as illustrative values from one implementation, not recommendations. Dimension scoring is now "one possible scoring taxonomy.

also agreed on matching the approach 1 pattern. the updated version describes what static analysis checks are possible without prescribing how to implement them

[ralphbean]

[minor] Two small notes, neither blocking:

1. This section is a bit long relative to comparable ones (canary deployments ~24 lines, mutation testing ~22 lines). The LLM-based rubric scoring subsection drifts into behavioral testing territory already covered by the eval frameworks section. Could be trimmed or folded into a sentence or two.

2. The new content raises some good open questions — what similarity thresholds work across different setups? should lint rules be configuration-specific or universal? — that the Open Questions section at the bottom would be a natural home for.

[Benkapner]

1. i trimmed the LLM rubric scoring subsection to two sentences, pointing to the existing eval frameworks discussion for LLM-as-judge trade-offs.

2. i added three open questions: similarity thresholds across setups, whether lint rules should be universal or per-architecture, and what token budget thresholds are appropriate for different component types across model context window sizes.

[ralphbean]

LGTM. Previous feedback addressed in 92e2d17.

[Benkapner]

thanks @ralphbean, is there anything i can do to help with the new PR's? or do changes here? its just a doc but i can elaborate the work

[ralphbean]

@Benkapner sorry, this fell through the cracks!

[github-actions]

Site preview

Preview: https://c01e6684-site.fullsend-ai.workers.dev

Commit: f112cad1841f9eac3da8d08e521a8ef30d3f4445

[github-actions]

E2E tests are running

Authorization passed for this commit. See the E2E Tests workflow for results.

[fullsend-ai-review]

🤖 Review · ❌ Terminated · Started 5:50 PM UTC · Ended 6:04 PM UTC
Commit: 4e21a60 · View workflow run →

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[fullsend-ai-review]

See the review comment for full details.

[fullsend-ai-review]

[low] heading-hierarchy-and-style

The heading 'Elaborating on Step 1: static analysis for agent configurations' uses gerund/meta framing inconsistent with other level-3 headings in the document, which all use declarative or noun-phrase style.

Suggested fix: Change to a declarative heading like 'Static analysis for agent configurations'.

[fullsend-ai-review]

[low] trade-offs-framing

Trade-offs are listed under a bold label rather than the document's established heading pattern (### Trade-offs or #### Trade-offs).

Suggested fix: Change 'Trade-offs:' to '#### Trade-offs' to match the established section structure while maintaining correct heading hierarchy.

[fullsend-ai-review]

[info] subsection-structure

Level-4 headings (####) are used for conceptual subsections ('Component-level analysis', 'Setup-level analysis'). Existing #### headings in the Eval frameworks section are all tool names, though using #### for sub-topics under ### is standard Markdown hierarchy.

[fullsend-ai-review]

[info] terminology-consistency

The section introduces 'agent configurations' as distinct from 'agent instructions.' The distinction is explicitly defined and the existing CI pipeline heading (line 295) already uses 'agent configurations.'

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 5:50 PM UTC · Completed 6:04 PM UTC
Commit: f112cad · View workflow run →

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 6:17 PM UTC · Completed 6:23 PM UTC
Commit: f112cad · View workflow run →

[fullsend-ai-retro]

Retro: PR #1826 — docs(problems): add static analysis layer to testing-agents

Timeline

This was an external community contribution by Benkapner adding 47 lines to docs/problems/testing-agents.md, covering static analysis for agent configurations. The PR was created on Jun 3, received human approval on Jun 7, and was merged on Jun 18 after an 11-day delay (acknowledged by the maintainer as having "fallen through the cracks").

Findings

Review iteration count was high for a docs-only PR. The review bot ran 10 iterations on this 47-line, single-file documentation change. Of those, approximately 7 were triggered by legitimate events (PR open, author comments, force pushes, final pre-merge check), but ~3 were redundant — re-reviewing unchanged code with no visible trigger event. This pattern of excessive review cycles on trivial changes wastes tokens without adding value.

Human review quality was good. ralphbean's review caught a real issue — prescriptive tone presenting specific thresholds and techniques as definitive rather than illustrative — and the author addressed feedback well in a single iteration.

Bot review findings were appropriate. The bot's final review produced only low/info-level findings (heading style, formatting patterns), which is the right calibration for a docs PR that had already been through human review.

Proposals: None (existing coverage)

All improvement opportunities identified are already covered by open issues:

• Redundant review dispatches → #893, #963, #1271, #1282, #2399
• Fast path for trivial docs changes → #1900
• Redundant internal iterations on clean diffs → #1370
• Stabilize approval on trivial changes → #947
• Defer ready-for-merge on docs PRs until human review → #1772

This PR provides additional evidence for prioritizing these existing issues, particularly #1900 (docs fast path) and #893 (redundant dispatch filtering).