Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions docs/guides/dev/behaviour-drivers.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,12 @@ Behaviour tests isolate forge-specific code behind drivers so Gherkin scenarios
|-----------|---------|----------------|
| `scm.Driver` | `pkg/behaviourtest/drivers/scm` | Issues, comments, labels (via GetIssue), file commits |
| `ci.Driver` | `pkg/behaviourtest/drivers/ci` | Workflow polling, logs, artifact download |
| `install.Driver` | `pkg/behaviourtest/drivers/install` | Provision and tear down fullsend in the acquired pool org |
| `install.Driver` | `pkg/behaviourtest/drivers/install` | Unified repo allocation: leases pool slots, lazily creates/installs repos, and manages mint lifecycle via `AllocateRepo`/`DeallocateRepo`/`Finalize`/`Capacity` |
| `install.Factory` | `pkg/behaviourtest/drivers/install` | Constructs a unified `Driver` from suite-level parameters; suite code selects a Factory and does not compose `MintDriver` + ensurer + pool itself (#6135) |
| `install.MintDriver` | `pkg/behaviourtest/drivers/install` | Provision and tear down fullsend mint in the acquired pool org (internal to composed `Driver`) |
| `install.State` | `pkg/behaviourtest/drivers/install` | Post-install config paths (script commits, workflow polling) |
| `install.RepoEnsurer` | `pkg/behaviourtest/drivers/install` | Lazily create and install numbered pool repos on demand; caches by org/repo key; concurrent-safe via singleflight |
| `install.StateProvider` | `pkg/behaviourtest/drivers/install` | Optional interface for `Driver` implementations that carry suite-level install state (`InstallState() State`) |
| `install.RepoEnsurer` | `pkg/behaviourtest/drivers/install` | Lazily create and install numbered pool repos on demand; caches by org/repo key; concurrent-safe via singleflight (internal to composed `Driver`) |

v1 reference implementations:

Expand All @@ -28,11 +31,11 @@ BEHAVIOUR_CI=githubactions # future: tekton, gitlabci
BEHAVIOUR_INSTALL_MODE=per-repo # v1 default and only supported value
```

The suite in `e2e/behaviour/suite_test.go` (or an external runner) acquires a pool org via `pkg/e2etest`, runs pre-install cleanup, calls `install.Driver.Install` (which deploys the mint), constructs SCM and CI drivers, creates a `world.RepoPool` (a buffered-channel lease pool of logical repo names), then runs godog with `pkg/behaviourtest/suite.InitScenario`. `InitScenario` clones a template `*world.World` per scenario and leases a unique repo name from the pool for the scenario's duration. Unsupported `BEHAVIOUR_INSTALL_MODE` values fail at suite startup.
The suite in `e2e/behaviour/suite_test.go` (or an external runner) acquires a pool org via `pkg/e2etest`, runs pre-install cleanup, selects an `install.Factory` (e.g. `cfmint.NewFactory`), and calls it to obtain a unified `install.Driver`. The Factory internally creates a `MintDriver`, performs suite setup (mint deploy via `NewComposedDriver`), and wires the internal repo-name pool and `RepoEnsurer` behind the Driver — the suite does not compose these itself. The suite then constructs SCM and CI drivers and runs godog with `pkg/behaviourtest/suite.InitScenario`. `InitScenario` clones a template `*world.World` per scenario; repo allocation (slot lease + lazy create/install) is handled by the unified driver's `AllocateRepo` when the `Given the enrolled test repository` step runs. Unsupported `BEHAVIOUR_INSTALL_MODE` values fail at suite startup.

### Install driver (v1 per-repo)

The install driver only manages the **mint lifecycle**: the cfmint driver deploys a Cloudflare Worker preview mint and tears it down; the legacy driver holds a pre-configured mint URL. Neither driver runs `github setup`, post-install validation, or `TeardownPerRepoInstall` on any target repository — that responsibility belongs to the `RepoEnsurer`, which lazily creates and installs numbered pool repos (`test-repo-01` … `test-repo-12`) on demand via `EnsureRepo`.
The unified `install.Driver` (created by `install.NewComposedDriver`) wraps a `MintDriver` with an internal pool and `RepoEnsurer`. The `MintDriver` only manages the **mint lifecycle**: the cfmint driver deploys a Cloudflare Worker preview mint and tears it down; the legacy driver holds a pre-configured mint URL. Neither mint driver runs `github setup`, post-install validation, or `TeardownPerRepoInstall` on any target repository — that responsibility belongs to the internal `RepoEnsurer`, which lazily creates and installs numbered pool repos (`test-repo-01` … `test-repo-12`) on demand via `AllocateRepo`.

Pool orgs must already have shared GitHub Apps, org-level mint enrollment, and per-repo mint enrollment for each numbered repo (one-time GCP admin step on the hosted mint project). The driver does not run `fullsend admin install` or `fullsend mint enroll`. See [e2e-testing.md](e2e-testing.md#behaviour-tests-and-per-repo-mint-enrollment).

Expand Down
37 changes: 30 additions & 7 deletions docs/guides/dev/behaviour-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,9 +208,9 @@ scenario or when `-v` output from multiple scenarios would interleave.

In CI, the test runner mints cross-org `e2e` installation tokens via OIDC (same as admin e2e) for GitHub API operations. Triage workflows on the pool org's `test-repo` mint same-org `triage` tokens from vendored reusable workflows; those require per-repo mint enrollment (`PER_REPO_WIF_REPOS`) on the hosted mint project. Pool `test-repo` repos are enrolled once by a GCP admin — not during CI install. The install driver provisions repo-scoped inference WIF via `fullsend inference provision` before `github setup`. See [e2e-testing.md](e2e-testing.md#behaviour-tests-and-per-repo-mint-enrollment).

### Lazy create+install (`RepoEnsurer`)
### Lazy create+install (`AllocateRepo`)

The `Given the enrolled test repository` step lazily creates and installs numbered pool repos (`test-repo-NN`) on demand via `RepoEnsurer`. When a scenario leases a repo name from the pool and an ensurer is configured, the step calls `EnsureRepo(ctx, org, repoName)` which:
The `Given the enrolled test repository` step lazily creates and installs numbered pool repos (`test-repo-NN`) on demand via the unified `install.Driver.AllocateRepo`. When a scenario calls `AllocateRepo`, the composed driver leases a slot from its internal pool and delegates to the internal `RepoEnsurer`, which calls `EnsureRepo(ctx, org, repoName)` to:

1. Creates the repo if it does not exist (the forge's `auto_init` provides the initial commit).
2. Validates post-install files; if validation fails, runs `fullsend github setup` (and inference provision when configured).
Expand Down Expand Up @@ -395,24 +395,47 @@ require github.com/fullsend-ai/fullsend v0.x.y // released tag, not @main

### API changes

**`suite.InitScenario` signature change:** The function signature changed from `InitScenario(sc, w)` to `InitScenario(sc, template, pool)` starting in the release that includes this change. Instead of passing a single `*world.World`, callers now pass a template `*world.World` (cloned per scenario) and a `*world.RepoPool` (used to lease unique repo names). Update your `suite_test.go` accordingly:
**`suite.InitScenario` signature change:** The function signature changed from `InitScenario(sc, template, pool)` to `InitScenario(sc, template)` starting in the release that includes this change. The `*world.RepoPool` parameter was removed because repo allocation is now handled by the unified `install.Driver` on the World template — `AllocateRepo` is called by the `Given the enrolled test repository` step, and `DeallocateRepo` is called in the After hook. Select an `install.Factory` and call it to obtain a Driver, then set `World.RepoDriver` on the template:

```go
pool, err := world.NewRepoPool(12)
// Select an install.Factory — suite code does not compose
// MintDriver + ensurer + pool itself.
factory := cfmint.NewFactory(ctx, e2eCfg, 12, cfmintCfg)

driver, err := factory(org, client, token, binary, gcpProjectID, t.Logf)
if err != nil {
t.Fatalf("creating repo pool: %v", err)
t.Fatalf("creating install driver: %v", err)
}
t.Cleanup(func() {
if err := driver.Finalize(context.Background()); err != nil {
t.Logf("driver finalize: %v", err)
}
})

// Extract install state from the driver.
var mintState install.State
if sp, ok := driver.(install.StateProvider); ok {
mintState = sp.InstallState()
}

template := &world.World{ /* ... driver fields ... */ }
template := &world.World{
Install: mintState,
RepoDriver: driver,
// ... other driver fields ...
}

suiteRunner := godog.TestSuite{
ScenarioInitializer: func(sc *godog.ScenarioContext) {
suite.InitScenario(sc, template, pool)
suite.InitScenario(sc, template)
},
// ...
}
```

**`World.Ensurer` → `World.RepoDriver` field change:** The `Ensurer install.RepoEnsurer` field on `world.World` has been replaced by `RepoDriver install.Driver`. External runners that set `World.Ensurer` must switch to `World.RepoDriver` and pass a unified `install.Driver` (created via an `install.Factory`).

**`install.Driver` → `install.MintDriver` rename:** The former `install.Driver` interface (with `Install`/`Teardown` methods) has been renamed to `install.MintDriver`. The `install.Driver` name now refers to the unified repo-allocation interface (`AllocateRepo`/`DeallocateRepo`/`Finalize`/`Capacity`). Update any code that references the old `install.Driver` type. The `cfmint.NewDriver` and `legacy.NewDriver` functions now return `install.MintDriver`.

**`steps.Register` signature change:** The function signature changed from `Register(ctx, w)` (where `ctx` was a `*godog.ScenarioContext` and `w` was a `*world.World`) to `Register(sc)` starting in the same release. Step definitions no longer receive `*world.World` as a parameter. Instead, they accept `context.Context` and extract the per-scenario World via `world.FromContext(ctx)`.

**`scm.Driver.DeleteRepo` addition:** The `scm.Driver` interface now includes a `DeleteRepo(ctx context.Context, owner, repo string) error` method. `CleanupScenario` calls it to delete ephemeral fork repos after each scenario. External `scm.Driver` implementations must add this method — return `forge.ErrNotFound` when the repository does not exist.
Expand Down
2 changes: 1 addition & 1 deletion docs/guides/dev/e2e-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ Prefer **`wrangler versions upload --name=mint-test --preview-alias=…`** so ru

Behaviour tests install fullsend in **per-repo** mode (`fullsend github setup`). Triage workflows mint same-org `triage` tokens from vendored reusable workflows; that requires per-repo mint enrollment (`PER_REPO_WIF_REPOS`). The install driver does **not** run `mint enroll` — pool org behaviour repos must be enrolled once by a GCP admin on the hosted mint project.

Admin e2e uses the singular `halfsend-NN/test-repo` name. Behaviour tests lease numbered `halfsend-NN/test-repo-01` … `test-repo-12` names from a `RepoPool`; these repos are **lazily created and installed** on demand by `RepoEnsurer` (see [behaviour-testing.md](behaviour-testing.md#lazy-createinstall-repoensurer)). Pre-provisioning numbered repos in the pool org is no longer required — mint enrollment for those names is still pre-provisioned so it is not on the critical path. Enroll base names only — do **not** enroll `*-fork` names (forks are ephemeral PR sources and mint against the enrolled base repo). GitHub repositories need not exist yet — enroll is a mint allowlist / WIF-provider update only.
Admin e2e uses the singular `halfsend-NN/test-repo` name. Behaviour tests lease numbered `halfsend-NN/test-repo-01` … `test-repo-12` slots via the unified `install.Driver.AllocateRepo`; these repos are **lazily created and installed** on demand by the composed driver (see [behaviour-testing.md](behaviour-testing.md#lazy-createinstall-allocaterepo)). Pre-provisioning numbered repos in the pool org is no longer required — mint enrollment for those names is still pre-provisioned so it is not on the critical path. Enroll base names only — do **not** enroll `*-fork` names (forks are ephemeral PR sources and mint against the enrolled base repo). GitHub repositories need not exist yet — enroll is a mint allowlist / WIF-provider update only.

Inference (`E2E_GCP_PROJECT_ID`) and mint (`it-gcp-konflux-dev-fullsend` for the hosted mint) may be different GCP projects. The behaviour install driver runs `fullsend inference provision <org>/test-repo` using CI credentials on the inference project (same access model as admin e2e), then passes the repo-scoped WIF provider to `github setup`. `E2E_GCP_WIF_PROVIDER` authenticates the CI job itself; it is not written to pool org repos.

Expand Down
2 changes: 1 addition & 1 deletion docs/guides/infrastructure/mint-administration.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ Pass this URL as `--mint-url` when running `fullsend github setup`, or set the `

`roles/owner` covers all of the above for users with broad access.

**Behaviour / e2e pool orgs:** Enroll `halfsend-NN/test-repo` (admin e2e) and `halfsend-NN/test-repo-01` … `test-repo-12` (lazily created and installed on demand by `RepoEnsurer` — see [behaviour-testing.md](../dev/behaviour-testing.md#lazy-createinstall-repoensurer)) on the hosted mint (`PER_REPO_WIF_REPOS`). Run `fullsend mint enroll owner/repo` once per name — not from CI; do not enroll `*-fork` names. Repos need not exist at enrollment time — enroll is a mint allowlist / WIF-provider update only; `RepoEnsurer` creates the repos when a behaviour scenario first leases them. See [e2e-testing.md](../dev/e2e-testing.md#behaviour-tests-and-per-repo-mint-enrollment).
**Behaviour / e2e pool orgs:** Enroll `halfsend-NN/test-repo` (admin e2e) and `halfsend-NN/test-repo-01` … `test-repo-12` (lazily created and installed on demand by the composed `install.Driver` — see [behaviour-testing.md](../dev/behaviour-testing.md#lazy-createinstall-allocaterepo)) on the hosted mint (`PER_REPO_WIF_REPOS`). Run `fullsend mint enroll owner/repo` once per name — not from CI; do not enroll `*-fork` names. Repos need not exist at enrollment time — enroll is a mint allowlist / WIF-provider update only; `AllocateRepo` creates the repos when a behaviour scenario first leases them. See [e2e-testing.md](../dev/e2e-testing.md#behaviour-tests-and-per-repo-mint-enrollment).

An administrator can grant all required roles with a single script:

Expand Down
79 changes: 35 additions & 44 deletions e2e/behaviour/suite_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,9 @@ import (
)

// poolSize is the number of enrolled test-repo-NN repos in the pool org.
// GODOG_CONCURRENCY must not exceed this — extra workers would block in
// pool.Acquire with no warning because the pool org only has test-repo-01
// through test-repo-12 with per-repo mint enrollment.
// This determines the driver's Capacity — the maximum number of
// concurrently allocated repos. Scenarios that exceed this block in
// AllocateRepo until a slot frees.
const poolSize = 12

// suiteName identifies this test suite. It is used to derive the CF Worker
Expand All @@ -46,9 +46,6 @@ func TestBehaviourSuite(t *testing.T) {
}
concurrency = n
}
if concurrency > poolSize {
t.Fatalf("GODOG_CONCURRENCY=%d exceeds repo pool size %d", concurrency, poolSize)
}

cfg := env.LoadRunnerConfig()
if err := cfg.Validate(); err != nil {
Expand All @@ -70,10 +67,10 @@ func TestBehaviourSuite(t *testing.T) {

binary := e2etest.BuildCLIBinary(t)

// Construct the CF mint driver with caller-provided parameters.
// The driver does not hardcode pool size or test-repo-NN assumptions;
// the calling code passes allowed orgs, per-repo WIF repos, and
// workflow host repos.
// Build a Factory that captures the cfmint configuration.
// The Factory is the #6135 consolidation seam — suite code
// selects a Factory and does not compose MintDriver + ensurer +
// pool itself.
//
// AllowedOrgs is explicitly empty ("") — this is per-repo mode, so
// we must not dual-enroll the pool org as an allowed org. The CLI's
Expand All @@ -82,63 +79,57 @@ func TestBehaviourSuite(t *testing.T) {
// WorkflowHostRepos registers the pool repos whose vendored workflows
// need to mint tokens. Without this, the mint rejects
// job_workflow_ref values from pool repos → 401.
installDriver, err := cfmint.NewDriver(client, token, binary, e2eCfg.GCPProjectID, t.Logf, cfmint.Config{
factory := cfmint.NewFactory(ctx, e2eCfg, poolSize, cfmint.Config{
PEMDir: e2eCfg.CFMintPEMDir,
SuiteName: suiteName,
AllowedOrgs: "",
PerRepoWIFRepos: buildPerRepoWIFRepos(org),
WorkflowHostRepos: buildWorkflowHostRepos(org),
AppSet: "fullsend-test",
})

e2etest.CleanupStaleResources(ctx, client, token, org, t)

// Call the Factory to construct the unified Driver. The Factory
// creates the MintDriver, calls mint.Install, and wires the
// internal pool + RepoEnsurer behind the Driver interface.
driver, err := factory(org, client, token, binary, e2eCfg.GCPProjectID, t.Logf)
if err != nil {
t.Fatalf("creating install driver: %v", err)
}

e2etest.CleanupStaleResources(ctx, client, token, org, t)

// Register teardown before Install so that a partially deployed
// preview mint is cleaned up even if Install fails partway through.
var installState install.State
// Register Finalize for teardown — it reclaims any outstanding
// leases, logs them, and tears down the mint (e.g. CF preview).
t.Cleanup(func() {
if installState == nil {
return
}
teardownCtx := context.Background()
if teardownErr := installDriver.Teardown(teardownCtx, org, installState); teardownErr != nil {
t.Logf("install teardown: %v", teardownErr)
if finalizeErr := driver.Finalize(context.Background()); finalizeErr != nil {
Comment thread
ifireball marked this conversation as resolved.
Resolved
Comment thread
ifireball marked this conversation as resolved.
Resolved
t.Logf("driver finalize: %v", finalizeErr)
}
})

installState, err = installDriver.Install(ctx, org)
if err != nil {
t.Fatalf("installing fullsend on %s: %v", org, err)
// Advisory: excess workers block in AllocateRepo (#6135).
if concurrency > driver.Capacity() {
t.Logf("WARNING: GODOG_CONCURRENCY=%d exceeds driver capacity %d; excess workers may block in AllocateRepo", concurrency, driver.Capacity())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] error-handling

The guard for GODOG_CONCURRENCY exceeding driver capacity was downgraded from t.Fatalf (hard failure) to t.Logf (advisory warning). While this is intentional per the #6135 contract (AllocateRepo blocks until a slot frees), excess goroutines will block until the go test timeout fires, producing a confusing timeout failure. The log warning appears early in output and may be missed.

Suggested fix: Consider using t.Errorf instead of t.Logf so the misconfiguration is visible in test results as a failure, even though the run continues.

}

// The install state carries the mint URL from the selected driver.
// Thread it to the ensurer so additional pool repos use the same
// mint endpoint.
if m, ok := installState.(install.MintURLProvider); ok && m.MintURL() != "" {
t.Logf("using mint URL for ensurer: %s", m.MintURL())
e2eCfg.MintURL = m.MintURL()
}

pool, err := world.NewRepoPool(poolSize)
if err != nil {
t.Fatalf("creating repo pool: %v", err)
}

ensurer := install.NewRepoEnsurer(e2eCfg, client, token, binary, t.Logf)

// The install driver only manages the mint lifecycle — it does not
// install on any specific repo. RepoName and RepoFull are set per-
// scenario by the ensurer when "Given the enrolled test repository"
// scenario by AllocateRepo when "Given the enrolled test repository"
// acquires a leased pool repo.
//
// The Driver carries the mint install state internally; extract it
// via the StateProvider interface so the suite does not thread State
// separately.
var mintState install.State
if sp, ok := driver.(install.StateProvider); ok {
mintState = sp.InstallState()
}

template := &world.World{
Config: cfg,
SCM: scmgh.New(client),
CI: gaci.New(client, token),
Install: installState,
Ensurer: ensurer,
Install: mintState,
RepoDriver: driver,
Org: org,
Token: token,
Logf: t.Logf,
Expand All @@ -148,7 +139,7 @@ func TestBehaviourSuite(t *testing.T) {

suiteRunner := godog.TestSuite{
Name: "behaviour",
ScenarioInitializer: func(sc *godog.ScenarioContext) { suite.InitScenario(sc, template, pool) },
ScenarioInitializer: func(sc *godog.ScenarioContext) { suite.InitScenario(sc, template) },
Options: &godog.Options{
Format: "pretty",
Paths: []string{"features"},
Expand Down
Loading
Loading