ci(e2e): add Gherkin behaviour tests with dummy runtime

.codecov.yml

@@ -18,6 +18,7 @@ coverage:
 ignore:
   - "**/*_test.go"
   - "**/testdata/**"
+  - "e2e/behaviour/**"
   - "docs/**"
   - "hack/**"
   - "experiments/**"

Makefile

@@ -28,7 +28,7 @@ help:
 	@echo "  script-test          - Run shell script tests (post-triage, post-code, post-review, pre-fetch-prior-review, reconcile-repos, validate-output-schema)"
 	@echo "  test                 - Run all checks: lint-all, go-test, script-test, lint-eval-cases"
 	@echo "  e2e-test             - Run admin e2e tests (CI: OIDC mint; local: gh auth login or GH_TOKEN)"
-	@echo "  behaviour-test       - Run Gherkin behaviour tests (CI: OIDC mint; local: gh auth login or GH_TOKEN)"
+	@echo "  behaviour-test       - Run Gherkin behaviour tests (installs fullsend per-repo; CI: OIDC mint)"
 	@echo "  lint-eval-cases      - Lint eval case definitions (annotations.yaml completeness)"
 	@echo "  functional-tests     - Run functional agent tests (requires EVAL_ORG, FULLSEND_DIR, GH_TOKEN, GCP creds)"
 

docs/ADRs/0003-org-config-repo-convention.md

@@ -228,6 +228,8 @@ sandbox:
   filesystem: ephemeral
 ```
 
+Backend selection for behaviour tests (`defaults.runtime` / per-repo `runtime: dummy`) is documented in [ADR 0066](0066-behaviour-tests-with-gherkin-and-drivers.md), not in the org config example above.
+
 Per-repo overrides in `repos/<repo-name>.yaml` can override org defaults
 (within the bounds of org-wide guardrails that cannot be weakened).
 

docs/ADRs/0066-behaviour-tests-with-gherkin-and-drivers.md

@@ -0,0 +1,42 @@
+---
+title: "66. Behaviour tests with Gherkin and pluggable drivers"
+status: Accepted
+relates_to:
+  - agent-infrastructure
+  - agent-architecture
+topics:
+  - e2e
+  - testing
+  - behaviour-tests
+---
+
+# 66. Behaviour tests with Gherkin and pluggable drivers
+
+Date: 2026-06-07
+
+## Status
+
+Accepted
+
+## Context
+
+Fullsend needs end-to-end tests that validate **deterministic platform behaviour** — dispatch routing, harness loading, schema validation, post-scripts, token scoping, sandbox policy, and SCM mutations — without depending on LLM output. This is distinct from admin install e2e ([ADR 0040](0040-org-pool-for-parallel-e2e-tests.md)) and from LLM/instruction testing ([testing-agents.md](../problems/testing-agents.md)).
+
+Runtime selection is shared with production via `defaults.runtime` in org `config.yaml` ([runtimes.md](../runtimes.md)). Harness definitions remain as in [ADR 0024](0024-harness-definitions.md). Per-repo install mode ([ADR 0033](0033-per-repo-installation-mode.md)) is the behaviour v1 default; per-org install is deferred.
+
+## Decision
+
+- Add **behaviour tests** under `e2e/behaviour/` using **godog** and portable Gherkin feature files.
+- Exercise **real SCM + real CI** through **driver interfaces** (`scm.Driver`, `ci.Driver`, `install.Driver`); v1 implementations target GitHub and GitHub Actions.
+- Substitute inference with a **dummy runtime** (`runtime: dummy` in per-repo config, or `defaults.runtime: dummy` for per-org) that executes scripted operations in the real OpenShell sandbox and emits `behaviour-results.json`.
+- Select backends via **runner env** (`BEHAVIOUR_SCM`, `BEHAVIOUR_CI`, `BEHAVIOUR_INSTALL_MODE`); feature files stay install-mode agnostic. v1 runs **per-repo** against the halfsend org pool; the suite provisions fullsend via `fullsend github setup` rather than requiring pre-installed orgs.
+- Use **compatibility tags** (`@skip:*`, `@requires:*`) to filter scenarios for future backends; tags do not select configuration.
+
+## Consequences
+
+- Behaviour tests can pass while prompt quality regresses; LLM evals remain necessary for instruction coverage.
+- Behaviour orgs are provisioned at suite start with `--runtime dummy`; production orgs must not use dummy unintentionally.
+- Adding GitLab or Tekton requires new drivers and runner env values, not feature file rewrites.
+- Dummy runtime op vocabulary stays minimal; new ops require runtime + docs updates when scenarios need them.
+- Behaviour tests depend on live external infrastructure: GitHub API, GitHub Actions runners, GCP WIF/mint, and the shared halfsend org pool. Transient outages, API rate limits, or pool org state corruption can fail the suite; CI distinguishes infrastructure failures from regressions via workflow logs and artifact inspection, but there is no offline fallback.
+- Behaviour tests share the halfsend org pool and lock mechanism with admin e2e tests (`e2e.yml` runs both jobs). Lock hold time scales with scenario count; pool size was doubled to absorb the additional load and can be increased again if contention appears.

docs/architecture.md

@@ -128,7 +128,11 @@ This is the thing that actually reasons and acts. Everything else in this docume
 
 **Decided (implementation):**
 
-- The `fullsend run` runner delegates in-sandbox agent execution to a `runtime.Runtime` interface; the MVP registers Claude Code only. Bootstrap uses a portable `BootstrapInput` interface with optional extensions such as `ClaudeHooksBootstrap` for sandbox tool hooks. Transcript and debug artifact handling use a separate `TranscriptHandler` interface. See [runtimes.md](runtimes.md) for the per-runtime security feature matrix required when adding a new backend.
+- The `fullsend run` runner delegates in-sandbox agent execution to a `runtime.Runtime` interface; production orgs default to Claude Code. Runtime selection is configured in `defaults.runtime` on the org `config.yaml` and resolved via `runtime.ResolveFromConfig()`. A **dummy** runtime executes scripted operations in the real OpenShell sandbox for behaviour tests (inference removed). Bootstrap uses a portable `BootstrapInput` interface with optional extensions such as `ClaudeHooksBootstrap` for sandbox tool hooks. Transcript and debug artifact handling use a separate `TranscriptHandler` interface. See [runtimes.md](runtimes.md) for the per-runtime security feature matrix required when adding a new backend.
+
+### Behaviour testing
+
+End-to-end **behaviour tests** under `e2e/behaviour/` validate deterministic platform code — dispatch routing, harness loading, sandbox policy, SCM mutations — with the LLM layer removed via the dummy runtime. Tests exercise real GitHub and GitHub Actions through pluggable SCM and CI drivers; Gherkin scenarios stay install-mode agnostic while runner env vars select backends. This coverage is **orthogonal** to LLM and instruction testing in [testing-agents.md](problems/testing-agents.md). See [ADR 0066](ADRs/0066-behaviour-tests-with-gherkin-and-drivers.md).
 
 **Open questions:**
 

docs/cli/github.md

@@ -52,6 +52,7 @@ fullsend github setup <owner/repo> \
 | `--app-set` | `fullsend-ai` | App set name prefix for GitHub Apps |
 | `--agents` | `fullsend,triage,coder,review,retro,prioritize` | Agent roles to provision |
 | `--direct` | `false` | Push scaffold directly instead of creating a PR |
+| `--runtime` | `claude` | Agent runtime backend (`claude` or `dummy`; `dummy` is for behaviour test orgs only) |
 
 ### Required OAuth scopes
 

docs/guides/README.md

@@ -45,4 +45,6 @@ Guides for contributors developing and testing fullsend itself.
 
 - [E2E testing](dev/e2e-testing.md) — Local and CI e2e runs, including PR authorization and `ok-to-test`
 - [CLI internals](dev/cli-internals.md) — Command structure, installation pipeline, and sandbox runtime
+- [Behaviour testing](dev/behaviour-testing.md) — Write Gherkin scenarios for end-to-end agent behaviour
+- [Behaviour test drivers](dev/behaviour-drivers.md) — Implement SCM and CI drivers for behaviour tests
 - [Testing workflow changes](dev/testing-workflows.md) — Point a live GitHub org at a branch to test workflow, action, and agent changes before release

docs/guides/dev/behaviour-drivers.md

@@ -0,0 +1,68 @@
+# Behaviour test drivers
+
+Behaviour tests isolate forge-specific code behind drivers so Gherkin scenarios stay portable.
+
+## Interfaces
+
+| Interface | Package | Responsibility |
+|-----------|---------|----------------|
+| `scm.Driver` | `e2e/behaviour/drivers/scm` | Issues, comments, labels (via GetIssue), file commits |
+| `ci.Driver` | `e2e/behaviour/drivers/ci` | Workflow polling, logs, artifact download |
+| `install.Driver` | `e2e/behaviour/drivers/install` | Provision and tear down fullsend in the acquired pool org |
+| `install.State` | `e2e/behaviour/drivers/install` | Post-install config paths (script commits, workflow polling) |
+
+v1 reference implementations:
+
+- `e2e/behaviour/drivers/scm/github/`
+- `e2e/behaviour/drivers/ci/githubactions/`
+- `e2e/behaviour/drivers/install/perrepo_github.go` (`BEHAVIOUR_INSTALL_MODE=per-repo`)
+
+## Runner configuration
+
+Set when starting the suite (not in feature files):
+
+```
+BEHAVIOUR_SCM=github              # future: gitlab, forgejo
+BEHAVIOUR_CI=githubactions        # future: tekton, gitlabci
+BEHAVIOUR_INSTALL_MODE=per-repo   # v1 default and only supported value
+```
+
+The suite in `e2e/behaviour/suite_test.go` acquires a pool org, runs pre-install cleanup, calls `install.Driver.Install`, then constructs SCM and CI drivers. Unsupported `BEHAVIOUR_INSTALL_MODE` values fail at suite startup.
+
+### Install driver (v1 per-repo)
+
+Uses `fullsend inference provision <org>/test-repo` then `fullsend github setup <org>/test-repo --vendor --direct --skip-app-setup --runtime dummy` with the repo-scoped WIF provider from provision (`E2E_GCP_PROJECT_ID`). Pool orgs must already have shared GitHub Apps, org-level mint enrollment, and per-repo mint enrollment for `test-repo` (one-time GCP admin step on the hosted mint project). The driver does not run `fullsend admin install` or `fullsend mint enroll`. See [e2e-testing.md](e2e-testing.md#behaviour-tests-and-per-repo-mint-enrollment).
+
+Teardown removes shim workflows, stale branches, and open fullsend PRs on `test-repo` via shared helpers in `e2e/admin/cleanup.go`.
+
+## Adding an SCM driver
+
+1. Implement `scm.Driver` in `e2e/behaviour/drivers/scm/<vendor>/`.
+2. Register the driver in `suite_test.go` when `BEHAVIOUR_SCM=<vendor>`.
+3. Document the env var value here.
+4. Add `@skip:<vendor>` tags on scenarios that cannot run until the driver is complete.
+
+Use `forge.Client` for operations it already exposes; add REST helpers inside the driver package only when necessary (e.g. `GetIssue` with labels).
+
+## Adding a CI driver
+
+1. Implement `ci.Driver` — `WaitForWorkflow`, `AssertNoWorkflow`, `GetRunLogs`, `DownloadArtifacts`.
+2. Map forge `WorkflowRun` types to portable polling logic; reuse patterns from `e2e/admin/admin_test.go`.
+3. Register in suite init for the matching `BEHAVIOUR_CI` value.
+
+## Step definitions
+
+Steps must **not** import `internal/forge/github` directly — only drivers. This keeps scenarios vendor-agnostic.
+
+Steps use `world.Install` for config repo paths (`ConfigOwner`, `ConfigRepo`, `ConfigPathPrefix`) instead of hardcoding the per-org `.fullsend` config repo.
+
+## Testing drivers
+
+Prefer unit tests with `httptest` for REST helpers. Optional smoke scenarios against live backends mirror admin e2e credentials (`GITHUB_TOKEN`, halfsend org pool).
+
+## Future backends checklist
+
+- [ ] GitLab SCM driver + `@skip:gitlab` tag removal
+- [ ] Tekton or GitLab CI driver
+- [ ] Per-org install driver (`BEHAVIOUR_INSTALL_MODE=per-org`)
+- [ ] Non-GitHub install backends

docs/guides/dev/behaviour-testing.md

@@ -0,0 +1,85 @@
+# Behaviour testing
+
+End-to-end Gherkin tests under `e2e/behaviour/` validate **deterministic platform code** with inference removed. They are **orthogonal** to LLM and instruction testing in [testing-agents.md](../../problems/testing-agents.md) and to admin install e2e in `e2e/admin/`.
+
+| | Behaviour tests | LLM evals | Admin e2e | Unit tests |
+|---|-----------------|-----------|-----------|------------|
+| **Target** | Platform workflows, sandbox, SCM | Prompts, models | Install/uninstall | Go functions |
+| **Inference** | Dummy runtime | Real LLM | Real LLM | N/A |
+| **Infrastructure** | Live GitHub + GHA | Varies | Live GitHub + GHA | None |
+
+## When to add a behaviour test
+
+Add one when a **user-visible workflow** must be verified end-to-end (dispatch → workflow → post-script → SCM state) and the assertion is **binary**. Prefer unit tests for pure Go logic and admin e2e for install provisioning.
+
+## Layout
+
+```
+e2e/behaviour/
+  features/          # Portable Gherkin scenarios
+  fixtures/          # Static content for write_fixture ops
+  steps/             # Step definitions
+  world/             # Scenario state
+  drivers/           # SCM, CI, env interfaces + v1 impls
+  suite_test.go      # godog entry (build tag: behaviour)
+```
+
+## Writing scenarios
+
+Describe **user-visible behaviour** only. Do not encode SCM vendor, CI platform, or install mode in feature files.
+
+### Dummy agent tables
+
+```gherkin
+Given a dummy agent that would:
+  | description      | op            | args                                                      |
+  | Emit triage JSON | write_fixture | output/agent-result.json, fixtures/triage/sufficient.json |
+```
+
+| Column | Meaning |
+|--------|---------|
+| `description` | Human label matched by assertion steps |
+| `op` | `read_file`, `url_get`, `write_fixture` |
+| `args` | Op-specific; see below |
+
+**`write_fixture`:** `dest_path, fixtures/...` — content lives in `e2e/behaviour/fixtures/`, embedded in the committed scenario script at `.fullsend/behaviour/current-scenario.yaml`.
+
+### Assertion steps
+
+Each assertion verifies immediately against workflow artifacts. If the triage workflow has not been waited on yet, the step waits for completion and downloads artifacts first (same as `Then the triage workflow completes successfully`).
+
+```gherkin
+Then the agent will succeed to Emit triage JSON
+And the agent will fail to Search for foo
+And the agent will output issues.out with:
+  """
+  expected content
+  """
+```
+
+### Compatibility tags
+
+Use tags only for **exceptions** when a backend cannot run a scenario yet: `@skip:gitlab`, `@skip:per-org`, `@requires:per-repo`. Untagged scenarios run everywhere applicable.
+
+## Running locally
+
+```bash
+# Local: gh auth login or export GH_TOKEN/GITHUB_TOKEN with access to halfsend org pool
+make behaviour-test
+```
+
+In CI, the test runner mints cross-org `e2e` installation tokens via OIDC (same as admin e2e) for GitHub API operations. Triage workflows on the pool org's `test-repo` mint same-org `triage` tokens from vendored reusable workflows; those require per-repo mint enrollment (`PER_REPO_WIF_REPOS`) on the hosted mint project. Pool `test-repo` repos are enrolled once by a GCP admin — not during CI install. The install driver provisions repo-scoped inference WIF via `fullsend inference provision` before `github setup`. See [e2e-testing.md](e2e-testing.md#behaviour-tests-and-per-repo-mint-enrollment).
+
+Runner env (defaults shown):
+
+```
+BEHAVIOUR_SCM=github
+BEHAVIOUR_CI=githubactions
+BEHAVIOUR_INSTALL_MODE=per-repo
+E2E_GCP_PROJECT_ID=...        # inference project; install runs inference provision per pool repo
+E2E_GCP_WIF_PROVIDER=...      # CI job GCP auth (not written to pool test-repo secrets)
+```
+
+Triage scenarios apply the `ready-for-triage` label (not `/fs-triage` comments) because the per-repo shim ignores `issue_comment` events from bot users and CI uses minted e2e installation tokens.
+
+See [behaviour-drivers.md](behaviour-drivers.md) for driver configuration and [ADR 0066](../../ADRs/0066-behaviour-tests-with-gherkin-and-drivers.md) for the decision record.

docs/guides/dev/e2e-testing.md

@@ -15,7 +15,7 @@ Playwright or stored sessions.
 
 Before running e2e locally or in CI:
 
-1. **Pool orgs** (`halfsend-01` … `halfsend-06`) provisioned per [Pool org provisioning](#pool-org-provisioning) below
+1. **Pool orgs** (`halfsend-01` … `halfsend-12`) provisioned per [Pool org provisioning](#pool-org-provisioning) below
 2. **Mint** deployed with `e2e` role enrolled and `ALLOWED_ORGS` including `fullsend-ai`
 3. **CI only:** pool orgs with `FULLSEND_FOREIGN_E2E_REPOS` authorizing `fullsend-ai/fullsend`
 4. **Local only:** `gh auth login` (or `GH_TOKEN` / `GITHUB_TOKEN`) with admin access on pool orgs
@@ -36,10 +36,12 @@ Optional environment variables:
 | `GH_TOKEN` / `GITHUB_TOKEN` | Override token source for local runs |
 | `FULLSEND_MINT_URL` | Override mint endpoint (default: hosted public mint, same as `fullsend admin --mint-url`) |
 | `E2E_LOCK_TIMEOUT` | Max wait for a free pool org (default 10m) |
-| `E2E_GCP_PROJECT_ID` | GCP project for inference-related setup (if needed) |
+| `E2E_GCP_PROJECT_ID` | GCP project for inference setup (`github setup --inference-project`) |
+
+Behaviour tests use the same pool orgs but install via `fullsend github setup` (per-repo) instead of `fullsend admin install`. See [behaviour-testing.md](behaviour-testing.md) and [behaviour-drivers.md](behaviour-drivers.md).
 
 Tests acquire an exclusive lock on one org from the pool (`halfsend-01` …
-`halfsend-06`) — see [ADR 0040](../../ADRs/0040-org-pool-for-parallel-e2e-tests.md).
+`halfsend-12`) — see [ADR 0040](../../ADRs/0040-org-pool-for-parallel-e2e-tests.md).
 
 ## CI runs
 
@@ -55,10 +57,34 @@ Required repository secrets:
 |--------|---------|
 | `E2E_GCP_WIF_PROVIDER` | GCP WIF provider (inference / auxiliary GCP access) |
 | `E2E_GCP_SERVICE_ACCOUNT` | GCP service account for WIF |
-| `E2E_GCP_PROJECT_ID` | GCP project ID |
+| `E2E_GCP_PROJECT_ID` | GCP project ID for inference secrets (`github setup --inference-project`) |
 
 Mint URL uses the hosted public endpoint by default (same as `fullsend admin --mint-url`). Override with org/repo variable `FULLSEND_MINT_URL` if needed; no separate e2e secret.
 
+### Behaviour tests and per-repo mint enrollment
+
+Behaviour tests install fullsend in **per-repo** mode (`fullsend github setup`). Triage workflows on the pool org's `test-repo` mint same-org `triage` tokens from vendored reusable workflows; that requires per-repo mint enrollment (`PER_REPO_WIF_REPOS`). The install driver does **not** run `mint enroll` — pool org `test-repo` repos must be enrolled once by a GCP admin on the hosted mint project.
+
+Inference (`E2E_GCP_PROJECT_ID`) and mint (`it-gcp-konflux-dev-fullsend` for the hosted mint) may be different GCP projects. The behaviour install driver runs `fullsend inference provision <org>/test-repo` using CI credentials on the inference project (same access model as admin e2e), then passes the repo-scoped WIF provider to `github setup`. `E2E_GCP_WIF_PROVIDER` authenticates the CI job itself; it is not written to pool org repos.
+
+The CI service account needs inference-provision IAM on `E2E_GCP_PROJECT_ID`:
+
+| IAM role | Purpose |
+|----------|---------|
+| `roles/iam.workloadIdentityPoolAdmin` | Create/update repo-scoped inference WIF providers |
+| `roles/resourcemanager.projectIamAdmin` | Grant `roles/aiplatform.user` to repo WIF principals |
+
+One-time enrollment for all pool orgs (idempotent):
+
+```bash
+export GCP_PROJECT=it-gcp-konflux-dev-fullsend
+for i in $(seq -w 1 12); do
+  fullsend mint enroll "halfsend-${i}/test-repo" --project="$GCP_PROJECT" --region=us-central1
+done
+```
+
+Re-run enrollment when adding a new pool org or after mint infrastructure changes that drop `PER_REPO_WIF_REPOS` entries. See [mint-administration.md](../infrastructure/mint-administration.md) for required operator IAM.
+
 ## Pool org provisioning
 
 Each pool org must be provisioned before e2e can use it:
@@ -82,7 +108,7 @@ fullsend admin foreign list --org halfsend-01
 # expect e2e → fullsend-ai/fullsend
 ```
 
-Existing pool orgs (`halfsend-01` … `halfsend-06`) need a one-time operator pass: install the e2e app (if missing) and run:
+Existing pool orgs (`halfsend-01` … `halfsend-12`) need a one-time operator pass: install the e2e app (if missing) and run:
 
 ```bash
 fullsend admin foreign allow --org halfsend-NN --role e2e --caller fullsend-ai/fullsend

docs/guides/infrastructure/mint-administration.md

@@ -72,6 +72,8 @@ Pass this URL as `--mint-url` when running `fullsend github setup`, or set the `
 
   `roles/owner` covers all of the above for users with broad access.
 
+  **Behaviour / e2e pool orgs:** Per-repo behaviour tests need `halfsend-NN/test-repo` enrolled on the hosted mint (`PER_REPO_WIF_REPOS`). Run `fullsend mint enroll owner/repo` once per pool org — not from CI. See [e2e-testing.md](../dev/e2e-testing.md#behaviour-tests-and-per-repo-mint-enrollment).
+
   An administrator can grant all required roles with a single script:
 
   ```bash