Skip to content

docs(#5670): add fixture authoring section to behaviour-testing guide #5673

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rh-hemartin merged 1 commit into from
Jul 30, 2026
Merged

docs(#5670): add fixture authoring section to behaviour-testing guide #5673

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions docs/guides/dev/behaviour-testing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,62 @@ And the agent will output issues.out with:

Use tags only for **exceptions** when a backend cannot run a scenario yet: `@skip:gitlab`, `@skip:per-org`, `@requires:per-repo`. Untagged scenarios run everywhere applicable.

## Fixture authoring

Every scenario that dispatches an agent stage must include a `write_fixture` row emitting `output/agent-result.json` with content that conforms to the stage's result schema. The harness post-script validates this file before performing any post-processing (labelling, commenting, PR creation). If the fixture is missing or invalid, the harness fails with `Validation failed: FAIL: output/agent-result.json not found`.

### Checklist for new scenarios

Before merging a scenario that dispatches an agent stage:

1. Identify the agent **role** (triage, code, review, fix, retro, prioritize). The role determines which result schema applies.
2. Create a fixture JSON file under `e2e/behaviour/fixtures/<stage>/` that satisfies the corresponding schema at `schemas/<stage>-result.schema.json` (scaffolded into target repos under `internal/scaffold/fullsend-repo/schemas/`).
3. Add a `write_fixture` row to the dummy agent table:
```
| Emit <stage> JSON | write_fixture | output/agent-result.json, fixtures/<stage>/<name>.json |
```
4. Add an assertion step to verify the fixture was written:
```gherkin
And the agent will succeed to Emit <stage> JSON
```
5. Verify that fixture field values satisfy downstream CLI validation, not just the JSON schema. For example, `head_sha` in `review-result` must be a full-length 40-character hex SHA (`abcdef0123456789abcdef0123456789abcdef01`), not a short prefix.
Comment thread
rh-hemartin marked this conversation as resolved.
Resolved

### Fixture inventory

Existing fixtures under `e2e/behaviour/fixtures/`:

| Fixture | Target schema | Purpose |
|---------|---------------|---------|
| `triage/sufficient.json` | `triage-result.schema.json` | Triage stage result with `action: "sufficient"` |
| `dispatch/ok.json` | _(none — dispatch proof)_ | Lightweight proof-of-execution marker for dispatch scenarios |
| `review/comment.json` | `review-result.schema.json` | Review stage result with `action: "comment"` |

The `dispatch/ok.json` fixture is not emitted as `output/agent-result.json` — it is used for auxiliary proof-of-execution files (e.g., `output/bash-routing-ok.json`). Scenarios that dispatch a **real agent stage** (triage, review, code, fix) must emit a schema-valid fixture to `output/agent-result.json`.

### Adding a fixture for a new stage

To add a fixture for a stage that does not yet have one (e.g., code or fix):

1. Read the stage's result schema under `internal/scaffold/fullsend-repo/schemas/<stage>-result.schema.json`.
2. Copy the closest existing fixture and adapt it to satisfy the new schema's `required` fields and `additionalProperties: false` constraint.
3. Populate field values with plausible test data. Pay attention to:
- **String patterns** — schemas may enforce regex patterns (e.g., `repo` must match `^[^/]+/[^/]+$`).
- **Conditional requirements** — some schemas use `allOf`/`if`/`then` to require extra fields depending on the `action` value (e.g., `review-result` requires `head_sha` and `body` when `action` is `"comment"`).
- **Downstream validation** — the harness post-script may apply stricter checks than the schema. For example, SHAs must be full-length hex, not truncated.
4. Place the fixture in `e2e/behaviour/fixtures/<stage>/<name>.json`.
5. Reference it in your scenario's dummy agent table with a `write_fixture` row targeting `output/agent-result.json`.

**Example:** The fork-bash-routing scenario dispatches a review agent, so it emits `review/comment.json` as the agent result:

```gherkin
Given a dummy agent that would:
| description | op | args |
| Prove bash routing | write_fixture | output/bash-routing-ok.json, fixtures/dispatch/ok.json |
| Emit review JSON | write_fixture | output/agent-result.json, fixtures/review/comment.json |
```

The first row proves execution via an auxiliary file; the second row emits the schema-valid `agent-result.json` that the harness post-script requires.

## Running locally

```bash
Expand Down
Loading