docs: add code agent evaluation experiment with V8 hybrid validation

[ascerra]

Moved from fullsend-ai/fullsend PR #241 per team decision to consolidate experiments in this repo.

---

Experiment was created to improve code agent proposed on fullsend-ai/fullsend PR #189.

490+ controlled trials across 20 scenarios and 2 real-world production bugs, comparing 7 agent variants to validate the code agent architecture. Includes experiment infrastructure (test harness scripts, deterministic gates, LLM judge), the V8 hybrid variant (recommended for production), and the full experiment writeup with methodology, results, and analysis.

Scenario definitions, injection payloads, judge prompts, and V1–V7 variant definitions are hosted in ascerra/code-agent-eval-scenarios — run ./scripts/setup.sh to clone and symlink them.

Key findings: structured agents score 4.61/5.00 vs 3.62 for vanilla Claude (~28% improvement), 100% injection/secret/protected-path resistance across all structured variants, and V7's mandatory bug reproduction step provides the largest per-scenario quality gain. V8 hybrid combines the best of V5 and V7 into a cleaned-up V1 that is 37% smaller.

Changes from original PR

Includes fixes for two review findings from @waynesun09's second review on the original PR:

1. score.sh: bc -l produces invalid JSON for values < 1 — bc -l omits the leading zero (e.g., .5000 instead of 0.5000), producing invalid JSON. Fixed by piping all bc -l calls through sed 's/^\./0./'.

2. check-deterministic.sh: word splitting breaks multi-word exfiltration markers — EXFIL_MARKERS was a flat string, so for marker in $EXFIL_MARKERS split multi-word markers like "cat .env" into individual words. Fixed by converting to a bash array and using "${EXFIL_MARKERS[@]}" iteration with grep -qF for literal matching.

Made-with: Cursor

Made with Cursor

[fullsend-ai-review]

Review: #9

Head SHA: 918aec5
Timestamp: 2026-04-27T00:00:00Z
Outcome: approve

Summary

This PR adds a well-structured code agent evaluation experiment comprising documentation (EXPERIMENT.md, RECOMMENDATION.md, VARIANTS.md), a complete test harness (8 shell scripts), and the V8 hybrid agent variant. The code is correct, well-documented, and follows consistent conventions. The two fixes mentioned in the PR body (bc -l leading zero and EXFIL_MARKERS word splitting) are properly implemented. No critical or high findings; a few minor observations noted below.

Findings

Medium

None

Low

• [correctness] code-agent-evaluation/scripts/score.sh:128-129 — The formula.judge_component field in the output JSON computes LLM_WEIGHTED_SCORE * 0.50 (on the 1–5 scale), but the actual composite formula uses LLM_NORMALIZED * 0.50 (on the 0–1 scale). This means the displayed judge_component value doesn't sum with gate_component to equal raw_score. The actual scoring computation is correct — this only affects the diagnostic formula block in the output JSON.
  Remediation: Change the judge_component computation to $LLM_NORMALIZED * 0.50 to match the actual formula, or document that it intentionally shows the un-normalized value.

• [style/conventions] code-agent-evaluation/scripts/invoke-variant.sh — The V1, V2, V5, V6, V7, and V8 case blocks are nearly identical (mkdir, cp artifacts, chmod, setup_symlinks, invoke agent). This could be refactored into a shared helper function, reducing ~120 lines of duplication.
  Remediation: Extract the common artifact-copy-and-invoke pattern into a function like run_structured_variant() and call it from each case.

Info

• [platform security] code-agent-evaluation/scripts/invoke-variant.sh — All agent invocations use --dangerously-skip-permissions. This is expected and appropriate for a sandboxed evaluation harness, but is noted for visibility.

• [correctness] code-agent-evaluation/scripts/run-experiment.sh:88 — The run_analysis() function is a placeholder that creates a stub summary.md. This is documented inline with a "Phase 7" comment, so it's clearly intentional incomplete work.

• [intent alignment] The PR moves content from fullsend-ai/fullsend PR #241 to this experiments repo per team decision. The scope is well-contained: experiment docs, harness scripts, and only the V8 variant (V1–V7 are in a separate external repo). No production code, no config changes.

Footer

Outcome: approve
This review applies to SHA 918aec5336878a7108c1a5890b70a6c8a6914ddd. Any push to the PR head clears this review and requires a new evaluation.