Skip to content

CNTRLPLANE-3339: add agent and convention eval framework #8382

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
enxebre wants to merge 2 commits into from
Closed

CNTRLPLANE-3339: add agent and convention eval framework #8382

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
335d6b2
docs: update api-sme agent and api/AGENTS.md conventions
enxebre Apr 30, 2026
6f26580
test(evals): add Go test harness for agent and convention evals
enxebre Apr 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .claude/agents/api-sme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@ You are an API subject matter expert system architect specializing in HCP.
- Basic security patterns (auth, rate limiting)

## Approach
1. Follow OpenShift dev guides from https://github.com/openshift/enhancements/tree/master/dev-guide
2. Apply best practices from https://github.com/openshift/enhancements/blob/master/dev-guide/api-conventions.md
3. Consider any API stable, running in production and ensure any API change is backward compatible
4. Keep it simple - avoid premature optimization

**MANDATORY**: Before writing any review, you MUST run `make api-lint-fix` and include its output in your review. Do not skip this step. The linter is the authoritative source for convention violations. Your review must start with the linter findings, then add your own analysis on top.

Stick to ../api/AGENTS.md

@coderabbitai coderabbitai Bot Apr 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Fix the API guide path.

../api/AGENTS.md does not resolve to the authoritative file from this location, so the agent can miss the repo’s primary API guidance entirely. Point this at an unambiguous repo path like api/AGENTS.md instead.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.claude/agents/api-sme.md at line 19, Update the broken relative link in
.claude/agents/api-sme.md: replace the reference string '../api/AGENTS.md' with
the unambiguous repo path 'api/AGENTS.md' so the agent points to the
authoritative API guide; ensure the changed symbol is the link text or path
occurrence in the file (search for '../api/AGENTS.md' and change it to
'api/AGENTS.md').


## Output
- API definitions that align with OpenShift and Kubernetes best practices
Expand Down
22 changes: 22 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -375,6 +375,28 @@ test-shard: generate
@echo "Running shard tests for packages: $(TEST_PACKAGES)"
$(GO) test -race -parallel=$(NUM_CORES) -count=1 -timeout=30m $(TEST_PACKAGES) -coverprofile $(COVER_PROFILE)

EVAL_MODEL ?= claude-opus-4-6
EVAL_JUDGE_MODEL ?= claude-opus-4-6
EVAL_RUNS ?= 1
EVAL_THRESHOLD ?= 0.8
EVAL_FOCUS ?=
EVAL_VERBOSE ?=

EVAL_GO_TEST = cd test/eval && EVAL_MODEL=$(EVAL_MODEL) EVAL_JUDGE_MODEL=$(EVAL_JUDGE_MODEL) EVAL_RUNS=$(EVAL_RUNS) EVAL_THRESHOLD=$(EVAL_THRESHOLD) \
$(GO) test -v -tags eval -count=1 -timeout=30m ./... $(if $(EVAL_VERBOSE),-ginkgo.v)

# Discover eval categories: top-level dirs (conventions, etc.) + subdirs under sme-agents/
EVAL_CATEGORIES := $(filter-out sme-agents,$(notdir $(wildcard test/eval/testdata/*))) $(notdir $(wildcard test/eval/testdata/sme-agents/*))
EVAL_TARGETS := $(addprefix eval-,$(EVAL_CATEGORIES))

.PHONY: eval-agents
eval-agents: ## Run all agent eval tests (use -j for parallel). Requires claude CLI and API key.
$(if $(EVAL_FOCUS),$(EVAL_GO_TEST) -ginkgo.focus="$(EVAL_FOCUS)",$(MAKE) -j $(EVAL_TARGETS))

.PHONY: $(EVAL_TARGETS)
$(EVAL_TARGETS): eval-%:
$(EVAL_GO_TEST) -ginkgo.focus="$*"
Comment thread
enxebre marked this conversation as resolved.

# OCP envtest index for downstream kubebuilder assets
ENVTEST_OCP_INDEX := https://raw.githubusercontent.com/openshift/api/master/envtest-releases.yaml
# OCP version to Kubernetes version mapping (OCP 4.x -> K8s 1.(x+13))
Expand Down
33 changes: 22 additions & 11 deletions api/AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,17 +15,6 @@ For conventions, always trust the kube-api-linter (`make api-lint-fix`). Do not
- Use feature gates for experimental functionality
- CRD generation via controller-gen with OpenShift-specific tooling

Key make targets for API work:

```bash
make api # Regenerate all CRDs, deepcopy, clients
make api-lint-fix # Run API linter and auto-fix violations
make verify-api-deps # Verify API dependencies
make verify # Full verification (includes api, fmt, vet, lint)
make update # Full update (api-deps, workspace-sync, deps, api, api-docs, clients)
ENVTEST_OCP_K8S_VERSIONS=1.35.0 make test-envtest-ocp # Run envtest for CEL validations
```

### API Dependencies

It is imperative that the imported dependencies are kept minimal. Use `make verify-api-deps` to verify that the dependencies are allowed.
Expand Down Expand Up @@ -57,6 +46,16 @@ To avoid introducing new dependencies, do not add utils or methods to the API ty

## API Type Change Guidelines

### Best Practices and Patterns

Use api/karpenter/v1beta1/karpenter_types.go and api/hypershift/v1beta1/etcdbackup_types.go as examples of best practices and patterns.

Don't use the other existing APIs as examples as they might have many legacy constraints.

### Field Grouping

**When multiple fields on a spec share a common prefix or relate to the same feature, they MUST be grouped into a dedicated struct.** Top-level specs like HostedClusterSpec and NodePoolSpec should only contain fields that are independently meaningful. If removing one field would make another field meaningless, they belong together in a sub-struct. A common signal is fields that share a name prefix (e.g., `BarEndpoint`, `BarConfig`, `BarID` all relate to "Bar" and should be a single `Bar` field with a `BarSpec` struct).

### N-1 and N+1 Compatibility

Every change to an API type must be safe for both:
Expand Down Expand Up @@ -85,3 +84,15 @@ See `api/hypershift/v1beta1/nodepool_types_test.go` for an example of this patte

All API CEL validations must be covered with envtests, see test/envtest/README.md for details

#### Key make targets for API work:

```bash
make api # Regenerate all CRDs, deepcopy, clients
make api-lint-fix # Run API linter and auto-fix violations
make verify-api-deps # Verify API dependencies
make verify # Full verification (includes api, fmt, vet, lint)
make update # Full update (api-deps, workspace-sync, deps, api, api-docs, clients)
ENVTEST_OCP_K8S_VERSIONS=1.35.0 make test-envtest-ocp # Run envtest for CEL validations
```

All these must pass for any change before creating a PR
100 changes: 100 additions & 0 deletions test/eval/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# Agent & Convention Evals

Evaluation framework for testing Claude Code agent definitions and
AGENTS.md conventions. Each scenario sends a prompt to an agent (or
base Claude), then uses an LLM judge to check the output against
expected issues.

## Prerequisites

- `claude` CLI installed and authenticated
- Go 1.25+

## Quick Start

```bash
# Run all scenarios in parallel
make eval-agents

# Run a single agent
make eval-api-sme

# Run with verbose output
make eval-agents EVAL_FOCUS=api-sme EVAL_VERBOSE=1

# Multiple runs with pass-rate threshold
make eval-agents EVAL_RUNS=5 EVAL_THRESHOLD=0.6
```

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `EVAL_MODEL` | `claude-opus-4-6` | Model for agent invocation |
| `EVAL_JUDGE_MODEL` | `claude-opus-4-6` | Model for judging |
| `EVAL_RUNS` | `1` | Number of trials per scenario |
| `EVAL_THRESHOLD` | `0.8` | Minimum pass rate (0.0-1.0) |
| `EVAL_FOCUS` | | Ginkgo focus filter (substring match) |
| `EVAL_VERBOSE` | | Set to `1` for verbose agent output |

## Directory Structure

```
test/eval/
eval_test.go # Test harness
testdata/
sme-agents/ # Agent scenarios (uses --agent flag)
<agent-name>/
<scenario>/
prompt.txt # Input prompt
expected.txt # Expected issues, one per line
patch.diff # Optional: applied before run
conventions/ # Convention tests (no agent)
<scenario>/
prompt.txt
expected.txt
```
Comment on lines +42 to +56

@coderabbitai coderabbitai Bot Apr 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Add a language to this fenced block.

markdownlint is already flagging this with MD040. text is enough for the directory tree.

🧰 Tools
🪛 markdownlint-cli2 (0.22.1)

[warning] 42-42: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@test/eval/README.md` around lines 42 - 56, The fenced code block showing the
directory tree in the README should include a language specifier to satisfy
markdownlint MD040; update the opening triple backticks for the block containing
the directory listing to ```text (i.e., change ``` to ```text) so the directory
tree is treated as plain text and the lint warning is resolved.


## Adding a Scenario

1. Create a directory under `sme-agents/<agent>/` or `conventions/`
2. Add `prompt.txt` with the input prompt
3. Add `expected.txt` with expected issues, one per line
4. Optionally add `patch.diff` to apply code changes before the run
5. Run it: `make eval-agents EVAL_FOCUS=<scenario-name> EVAL_VERBOSE=1`
6. Iterate on `expected.txt` until the pass rate is stable

The make target is auto-discovered — no Makefile changes needed.

## How It Works

1. **Discovery**: scans `testdata/` for scenarios with `prompt.txt`
and `expected.txt`
2. **Patch** (optional): applies `patch.diff` to the repo so agents
can run tools against real code (e.g., `make api-lint-fix`)
3. **Agent invocation**: runs `claude --agent <name> -p <prompt>`
with tools enabled if a patch is present, disabled otherwise
Comment thread
coderabbitai[bot] marked this conversation as resolved.
4. **Judge**: a separate Claude call checks the agent output against
expected issues using semantic matching
5. **Pass rate**: runs N trials (`EVAL_RUNS`), asserts the pass rate
meets the threshold (`EVAL_THRESHOLD`)
6. **Cleanup**: reverts any patches applied

## Scenario Types

### SME Agent Scenarios (`sme-agents/`)

Test specific agent definitions (`.claude/agents/<name>.md`). The
agent is invoked with `--agent <name>`. Use `patch.diff` to place
code in the repo for the agent to review with its tools.

### Convention Scenarios (`conventions/`)

Test that base Claude (no agent) follows AGENTS.md conventions.
Useful for validating that code style rules, naming conventions,
and other repo-wide policies are applied.

## Cost

Each scenario costs ~$0.50-2.00 (agent + judge). A full run of all
6 scenarios costs ~$5-15 depending on how much the agent reads.
Loading
Loading