Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,7 @@ e2e-test:
go test -tags e2e -v -count=1 -timeout 30m ./e2e/admin/

behaviour-test:
go test -tags behaviour -v -count=1 -timeout 45m ./e2e/behaviour/
go test -tags behaviour -race -v -count=1 -timeout 45m ./e2e/behaviour/

# Functional agent evals — run agents against ephemeral GitHub repos and judge results.
# Required env: EVAL_ORG (GitHub org for ephemeral repos), plus GCP creds for Vertex AI.
Expand Down
24 changes: 24 additions & 0 deletions docs/guides/dev/behaviour-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,30 @@ Use tags only for **exceptions** when a backend cannot run a scenario yet: `@ski
make behaviour-test
```

### Parallel execution

The suite runs scenarios in parallel by default (`GODOG_CONCURRENCY=12`,
matching the repo pool size). Each scenario gets its own `World` clone and
leases a unique `test-repo-NN` from the pool, so no cross-scenario state
is shared. The `behaviour-test` Make target includes `-race` to catch
data races under concurrent execution.

To adjust concurrency:

```bash
# Run at default concurrency (12)
make behaviour-test

# Run with explicit concurrency
GODOG_CONCURRENCY=4 make behaviour-test

# Serial mode for debugging
GODOG_CONCURRENCY=1 make behaviour-test
```

Serial mode (`GODOG_CONCURRENCY=1`) is useful when debugging a single
scenario or when `-v` output from multiple scenarios would interleave.

In CI, the test runner mints cross-org `e2e` installation tokens via OIDC (same as admin e2e) for GitHub API operations. Triage workflows on the pool org's `test-repo` mint same-org `triage` tokens from vendored reusable workflows; those require per-repo mint enrollment (`PER_REPO_WIF_REPOS`) on the hosted mint project. Pool `test-repo` repos are enrolled once by a GCP admin — not during CI install. The install driver provisions repo-scoped inference WIF via `fullsend inference provision` before `github setup`. See [e2e-testing.md](e2e-testing.md#behaviour-tests-and-per-repo-mint-enrollment).

### Lazy create+install (`RepoEnsurer`)
Expand Down
13 changes: 10 additions & 3 deletions e2e/behaviour/suite_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ package behaviour_test
import (
"context"
"os"
"strconv"
"testing"

"github.com/cucumber/godog"
Expand All @@ -23,8 +24,14 @@ func TestBehaviourSuite(t *testing.T) {
if testing.Short() {
t.Skip("skipping behaviour tests in short mode")
}
if c := os.Getenv("GODOG_CONCURRENCY"); c != "" && c != "1" {
t.Fatalf("behaviour suite does not support GODOG_CONCURRENCY=%q: per-scenario World isolation is in place but drivers (SCM, CI, Install) are shared by reference and have not been validated under -race with concurrent scenarios yet; see #5441 for parallel support", c)

concurrency := 12
if c := os.Getenv("GODOG_CONCURRENCY"); c != "" {
n, err := strconv.Atoi(c)
if err != nil || n < 1 {
t.Fatalf("GODOG_CONCURRENCY must be a positive integer, got %q", c)
}
concurrency = n
}

cfg := env.LoadRunnerConfig()
Expand Down Expand Up @@ -96,7 +103,7 @@ func TestBehaviourSuite(t *testing.T) {
Paths: []string{"features"},
TestingT: t,
Tags: os.Getenv("GODOG_TAGS"),
Concurrency: 1,
Concurrency: concurrency,
},
}
if st := suiteRunner.Run(); st != 0 {
Expand Down
Loading