Repository navigation
docs(problems): add static analysis layer to testing-agents #1826
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -304,6 +304,50 @@ A practical CI pipeline for agent instruction changes might look like: | |
|
|
||
| Steps 2-4 are expensive (they invoke the LLM), so they may need dedicated pipeline infrastructure separate from normal build pipelines. Cost management is a real constraint — see [agent-infrastructure.md](agent-infrastructure.md). | ||
|
|
||
| ### Elaborating on Step 1: static analysis for agent configurations | ||
|
|
||
| Step 1 above summarizes static analysis as linting for "obvious issues." This section expands on what that layer looks like in practice and what classes of problems it can catch. | ||
|
|
||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [info] terminology-consistency The section introduces 'agent configurations' as distinct from 'agent instructions.' The distinction is explicitly defined and the existing CI pipeline heading (line 295) already uses 'agent configurations.' |
||
| The rest of this document uses "agent instructions" to refer to the natural-language text that governs agent behavior (system prompts, CLAUDE.md files, review criteria). "Agent configurations" refers to the broader structure: instructions plus the skills, commands, hooks, sub-agent definitions, and context files that together define an agent's setup. Static analysis operates on the configuration as a whole, not just the instruction text, because structural problems (broken references between components, redundant skills, unbalanced token budgets) live at the configuration level. | ||
|
|
||
| The evaluation frameworks surveyed above test agent *behavior*: they run agents or prompts, evaluate outputs, and score results. Static analysis operates on the agent *configuration itself* without executing anything. Behavioral testing answers whether an agent *does the right thing*. Static analysis answers whether the configuration is *well-formed, secure, non-redundant, and internally consistent*. Both matter. A configuration can produce correct agent behavior while carrying structural defects (broken references, credential exposure, duplicate skills consuming context budget), and a perfectly structured configuration can still give bad guidance. These are different failure classes, and catching one does not catch the other. This layer is distinct from the prompt evaluation, agent evaluation, and input mutation categories surveyed above; it does not test behavior at all, but rather validates the structural and security properties of the configuration that behavioral testing takes as a given. | ||
|
|
||
| Application code has linters that catch structural problems, security anti-patterns, and style violations without executing the code. Agent configurations are similarly lintable. This layer is deterministic, fast, and CI-friendly. It requires no LLM calls, runs in seconds, and can gate every instruction change at zero marginal cost: if an instruction change breaks structure or introduces a security pattern, there is no reason to spend LLM budget on behavioral evaluation. An [open-source evaluation framework](https://github.com/Benkapner/harness-eval-lab) implements these checks for Claude Code configurations and has been applied to production setups. | ||
|
|
||
| #### Component-level analysis | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [info] subsection-structure Level-4 headings (####) are used for conceptual subsections ('Component-level analysis', 'Setup-level analysis'). Existing #### headings in the Eval frameworks section are all tool names, though using #### for sub-topics under ### is standard Markdown hierarchy. |
||
|
|
||
| Each component in an agent configuration can be checked individually (skills, commands, md files, hooks etc.): | ||
|
|
||
| **Structural integrity.** Every component has metadata requirements: skills need descriptions, frontmatter must parse as valid YAML, referenced scripts must exist. These are the equivalent of syntax checks for code. A skill without a description may not trigger correctly; a command referencing a missing script would fail at runtime. Static analysis can catch these before deployment. | ||
|
|
||
| **Security patterns.** Agent instructions can inadvertently introduce security vulnerabilities. Static checks can scan for credential exposure (API keys, tokens, or secrets embedded in instruction text) and for prompt injection patterns baked into the instructions themselves (jailbreak phrases, role override attempts, instruction-ignoring directives). This is distinct from adversarial *input* testing (Step 4 above): it catches vulnerabilities in the *instructions*, not in the inputs the agent will receive. | ||
|
|
||
| **Token budget per component.** Individual components that exceed recommended token budgets can be flagged, identifying instructions that could be condensed. A single overweight skill may not break anything on its own, but it consumes context window space that other skills and task context need. | ||
|
|
||
| #### Setup-level analysis | ||
|
|
||
| Beyond per-component checks, agent configurations can be analyzed as systems. Individual components may each pass their own checks while the configuration as a whole has problems: an unbalanced token budget, clusters of overlapping triggers, duplicate content across skills, or broken references between components. | ||
|
|
||
| **Redundancy detection.** When an agent configuration grows organically, skills and instructions accumulate. Similarity detection across instruction texts could identify near-duplicate components (two skills that give substantially the same guidance with different names). Approaches range from lightweight (TF-IDF cosine similarity, fast and free but limited to lexical overlap) to more accurate (embedding-based comparison, LLM-based semantic matching) at increasing cost. For CI gating, cheaper techniques may be preferable; for periodic audits, more expensive approaches could catch subtler duplicates. Illustrative thresholds from one implementation: around 0.85 for likely duplicates, around 0.50 for trigger overlap, though the right values are configuration-dependent. | ||
|
|
||
| **Dependency validation.** Agent configurations have internal references: agents reference skills, commands reference scripts, instructions reference other components by name. Static analysis can map these dependencies and flag two classes of problems: broken references (an agent that references a skill that does not exist) and orphaned components (a skill that nothing references, suggesting it may be dead weight or a misconfiguration). This provides a partial, deterministic answer to the absence detection problem identified earlier in this document. If someone deletes a skill that an agent depends on, dependency validation catches the broken reference. It does not catch capabilities that silently vanish because an instruction was reworded, but it catches the structural case where a component is removed entirely. | ||
|
|
||
| **Token budget distribution.** Agent configurations have a token economy: some instructions are always loaded (system prompts, CLAUDE.md), while others load on demand (skills triggered by specific situations). Setup-level analysis can measure this distribution and flag inversions, for example a setup where always-loaded content consumes the majority of the context window, leaving little room for on-demand skills or actual task context. | ||
|
|
||
| **Trigger overlap.** Skills that activate based on natural-language trigger descriptions can overlap: two skills with similar "when to use" descriptions may both load for the same user request, consuming context budget without adding distinct value. The same similarity detection techniques used for redundancy detection could surface these overlaps. | ||
|
|
||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [low] trade-offs-framing Trade-offs are listed under a bold label rather than the document's established heading pattern (### Trade-offs or #### Trade-offs). Suggested fix: Change 'Trade-offs:' to '#### Trade-offs' to match the established section structure while maintaining correct heading hierarchy. |
||
| **Dimension scoring.** Setup-level analysis can aggregate per-component findings into configuration-wide scores. One possible scoring taxonomy: structural soundness (percentage of components without errors), safety (absence of credential or injection patterns), coherence (no duplicates, broken dependencies, or trigger overlaps), and efficiency (balanced token budget, minimal redundancy). Whatever dimensions are chosen, the scores could provide a baseline that is tracked over time: if an instruction change drops a score, it likely introduced a problem. | ||
|
|
||
| **Trade-offs:** | ||
|
|
||
| - Similarity thresholds are empirical. What counts as "near-duplicate" depends on the configuration; thresholds that work for one setup may produce false positives or miss real duplicates in another. Intentionally similar skills (e.g., a Python review skill and a Go review skill) may be flagged as redundant when they serve distinct purposes. | ||
| - Lightweight similarity techniques (e.g., TF-IDF) catch lexical overlap but miss semantic similarity. Two skills that express the same guidance in different wording will not be flagged. More expensive techniques (embeddings, LLM-based matching) close this gap at higher cost. | ||
| - Dependency validation catches structural breaks (deleted skills, missing scripts) but not semantic drift. If a skill's content is reworded to remove a capability without changing its name or references, dependency analysis will not notice. | ||
| - Passing static checks can create false confidence. A configuration that is structurally sound, non-redundant, and security-clean can still give the agent bad guidance. Static analysis validates form, not function. | ||
| - Lint rules require maintenance as agent tooling evolves and new anti-patterns emerge. | ||
|
|
||
| An optional deeper layer could use an LLM to score each component against qualitative rubrics and produce a keep/review/remove verdict, catching problems that static analysis cannot (e.g., structurally valid but vague guidance). This introduces the cost, non-determinism, and judge bias trade-offs common to all LLM-as-judge approaches discussed in the eval frameworks section above. | ||
|
|
||
| ## Measuring agent capability drift | ||
|
|
||
| Beyond testing individual instruction changes, there's a need for ongoing monitoring: | ||
|
|
@@ -332,3 +376,6 @@ Beyond testing individual instruction changes, there's a need for ongoing monito | |
| - Can agents test other agents, or does that create circular trust dependencies? (Agent A tests Agent B, but who tests Agent A?) | ||
| - How do we test cross-agent composition without combinatorial explosion of test scenarios? | ||
| - Is there a meaningful equivalent of "code coverage" for natural-language instructions, or is that a false analogy? | ||
| - What similarity thresholds work across different agent setups, or should thresholds be tuned per configuration? | ||
| - Should lint rules for agent configurations be universal or adapted per agent architecture? | ||
| - What token budget thresholds are appropriate for different component types (skills, commands, CLAUDE.md), and how should those thresholds account for variation in context window sizes across models? | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[low] heading-hierarchy-and-style
The heading 'Elaborating on Step 1: static analysis for agent configurations' uses gerund/meta framing inconsistent with other level-3 headings in the document, which all use declarative or noun-phrase style.
Suggested fix: Change to a declarative heading like 'Static analysis for agent configurations'.