forge-v2: Generalize evals infra to support more types of evals, organization and queuing of named suites

[kimjune01]

Forge-v2 refactored version of google-gemini/gemini-cli#24941

Original PR

Generalize evals infra to support more types of evals, organization and queuing of named suites

Generalizes the evals infrastructure to:

• Support grouping of evals into named 'suites' which can be run by name.
• Support categorization of evals into behavioral (what we have today), hero (small number of larger and more representative evals), and component level (benchmarks for a specific component's effectiveness).
• Helpers for component level evals.

This is a precursor to having component level evaluation suites, such as one for compression, which measures the effectiveness of that component in isolation.

It also paves the way for having more and varied tests.

Only the behavioral evals tests are run by default. Other suites can be run on demand by feature maintainers or autonomously at PR time via google-gemini#24877

Here's what the run queue UI looks like:

---

Forge-v2 pipeline results

Metric	Value
Blind-blind winner	opus (Opus 4.6 vs Codex GPT-5.4)
Build + tests	PASS
Complexity gate	Δ=0 (PASS)
Gemini reviewer	✅ Approved ("No comments")

Refactoring claims applied

• C1 — Correct the component helper description
• C2 — Return the initialized component config
• C3 — Make the component Config import type-only
• C4 — Make the app helper BaseEvalCase import type-only

What is this?

This diff shows the output of a forge-wrapped LLM refactoring pipeline applied to PR google-gemini#24941's code at the point where tests first passed (C_test). The question: can an autonomous pipeline improve the implementation before human review?

Pipeline: goal-anchored volley → adversarial hunt-spec → blind-blind implementation (Opus 4.6 + Codex GPT-5.4, smaller-churn wins) → hunt-code with full build+tests → Gemini 3.1 Pro reviewer-loop → complexity gate (δ=0.05).

Experiment: refactor-equivalence v2 — Does an LLM refactoring pass help or hurt brownfield PRs?

We'd love your take: would you approve this diff? 🙏