ci(e2e): add Gherkin behaviour tests with dummy runtime - #1982
Conversation
Site previewPreview: https://a549be94-site.fullsend-ai.workers.dev Commit: |
ifireball
changed the title
Review5 findings (0 critical, 0 high, 1 medium, 4 low) Re-review of files changed since prior review at FindingsMedium
Low
Previous runReview1 finding (0 critical, 0 high, 1 medium) | Verdict: comment Re-review of files changed since prior review at FindingsMedium
Low
Previous run (2)Review1 finding (0 critical, 0 high, 1 medium) | Verdict: comment Re-review with no file changes since prior review at FindingsMedium
Low
Previous run (3)Review1 finding (0 critical, 0 high, 1 medium) | Verdict: comment Re-review of files changed since prior review at FindingsMedium
Low
Previous run (4)Review1 finding (0 critical, 0 high, 1 medium) | Verdict: comment Re-review of files changed since prior review at FindingsMedium
Low
Previous run (5)ReviewFindingsMedium
Low
Previous run (6)ReviewReason: stale-head The review agent reviewed commit Previous run (7)Review3 findings (0 critical, 0 high, 3 medium) | Verdict: comment Re-review of files changed since prior review at FindingsMedium
Low
Labels: PR adds Gherkin behaviour test framework with dummy runtime, touching e2e tests, runner infrastructure, per-repo install mode, and ADR documentation. Previous run (8)Review1 finding (0 critical, 0 high, 1 medium) | Verdict: comment Re-review of 19 files changed since prior review at FindingsMedium
Previous run (9)Review — Comment4 findings (0 critical, 0 high, 4 medium) | Verdict: comment Re-review of 4 files changed since prior review at FindingsMedium
Previous run (10)Review — Comment2 findings (0 critical, 0 high, 2 medium) | Verdict: comment This PR adds Gherkin behaviour tests with a dummy runtime, touching 63 files. The change is well-structured and aligns with the scope authorized by issue #2276. Two medium-severity findings are noted below for consideration. Findings1.
|
![fullsend-ai-review[bot]](https://avatars.githubusercontent.com/in/3478501?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
About the @fullsend-ai-review "[misconfiguration-guard]" comment - I reject adding extra layers, users having to explicitly commit the "dummy" value into their configuration should be enough to convince us they know what they are doing if they do it. |
ralphbean
left a comment
ralphbean
left a comment
There was a problem hiding this comment.
I think this needs a few changes before we can merge. See inline comments.
waynesun09
left a comment
waynesun09
left a comment
There was a problem hiding this comment.
Review squad findings (6 agents, deduplicated against 13 existing review threads). 1 original CRITICAL finding dropped as false positive (claude.go not modified in this PR). 5 new findings posted — 1 HIGH, 4 MEDIUM.
|
Review follow-up in
Remaining coordination: ADR 0043 renumber when #1816 merges second. |
|
🤖 Finished Review · ✅ Success · Started 5:28 PM UTC · Completed 5:42 PM UTC |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
waynesun09
left a comment
waynesun09
left a comment
There was a problem hiding this comment.
Review Squad Report (8 agents dispatched, 14 findings after dedup, 4 false positives removed)
3 HIGH, 5 MEDIUM findings posted inline. The one CRITICAL finding reported by multiple agents (DummyRuntime.Run missing context.Context) was verified as a false positive — the Runtime interface does not require ctx in Run().
|
🤖 Finished Review · ❌ Failure · Started 10:35 AM UTC · Completed 11:00 AM UTC |
54c3af9 to
a59ab53
Compare
|
🤖 Review · |
a59ab53 to
a341fab
Compare
|
🤖 Review · |
|
🤖 Review · |
|
🤖 Review · |
Main already has ADR 0065 (provider-backed policy composition); renumber the behaviour-tests ADR and update cross-references. Signed-off-by: Barak Korren <bkorren@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
🤖 Finished Review · ✅ Success · Started 7:07 PM UTC · Completed 7:20 PM UTC |
| r := ClaudeRuntime{} | ||
| return Backend{Runtime: r, Transcripts: r}, nil | ||
| case "dummy": | ||
| // Selected only via explicit per-repo/org config (behaviour test orgs). |
There was a problem hiding this comment.
[low] fail-open
The dummy runtime is unconditionally available to any org that sets defaults.runtime: dummy. ADR 0066 documents this as intentional; activation requires deliberate admin configuration.
| return false, ctx.Err() | ||
| case <-time.After(jitter): | ||
| } | ||
| log.Printf("retrying commit to %s/%s@%s (attempt %d/%d): %v", owner, repo, branch, attempt+1, maxAttempts, err) |
There was a problem hiding this comment.
[low] pattern-inconsistency
commitFilesWithRetry adds log.Printf for retry logging. This package has zero log.Printf calls on main and does not import log — it relies on error returns for observability. The new import breaks the zero-logging pattern.
Suggested fix: Remove the log.Printf call. The error is already returned to the caller.
| fn, err := p.gcpAPI.GetFunction(ctx, p.cfg.ProjectID, p.cfg.Region, functionName) | ||
| mintURI, err := p.resolveMintURI(ctx) | ||
| if err != nil { | ||
| return fmt.Errorf("getting mint function: %w", err) |
There was a problem hiding this comment.
[low] dead-code
In EnsureOrgInMint, the mintURI == empty check after resolveMintURI is unreachable. resolveMintURI calls DiscoverMint, which always returns an error when the URI is empty — the function never returns (empty, nil).
Suggested fix: Remove the unreachable mintURI == empty check and its error return.
|
🤖 Finished Retro · ✅ Success · Started 7:27 PM UTC · Completed 7:37 PM UTC |
|
PR #1982 (Gherkin behaviour tests with dummy runtime) was a large human-authored PR (66 files, +4492/-116) that took 31 days to merge. The review process was thorough but extremely expensive: 60 review bot events (33 cancelled, 20 successful), consuming an estimated 9+ hours of review compute. Key issues: (1) 33 cancelled reviews from rapid-push dispatch storms, (2) the same dismissed finding re-raised 4 times, (3) 15 post-approval review runs adding no critical value, and (4) the most critical bug was found by the review squad, not the automated agent. Three proposals filed as evidence for existing open issues. Proposals filed
|
6 participants
Summary
defaults.runtimeorg config and--runtimeinstall flag with sharedruntime.ResolveFromConfig()selection infullsend run.behaviour-results.jsonfor deterministic assertions.e2e/behaviour/with pluggable GitHub/GitHub Actions drivers, triage scenarios, CI job, and ADR/docs.BEHAVIOUR_INSTALL_MODE=per-repo): suite runsfullsend inference provision(whenE2E_GCP_PROJECT_IDis set) andfullsend github setup --runtime dummyon pooltest-repo. Per-repo mint enrollment is a one-time admin step on the hosted mint project — the install driver does not runmint enrollat test time.Ops / merge prerequisites
Behaviour CI uses the same pool org credentials as admin e2e. Per-repo triage workflows need pool org
test-reporepos enrolled once on the hosted mint project (PER_REPO_WIF_REPOS). See e2e-testing.md — Behaviour tests and per-repo mint enrollment.When
E2E_GCP_PROJECT_IDis set, the behaviour install driver runsfullsend inference provision <org>/test-repousing CI credentials on the inference project (same access model as admin e2e). No additional mint-project IAM roles are required onE2E_GCP_SERVICE_ACCOUNTbeyond what admin e2e already needs.Pool recovery:
CleanupStaleResourcesruns before each install (teardown-first). Per-repo teardown removes shim workflow, enrollment branches, and fullsend PRs; a freshgithub setupis the canonical reset if.fullsend/artifacts remain from a partial run.Test plan
go test ./...go test -tags behaviour -c ./e2e/behaviour/...go test -tags e2e -c ./e2e/admin/...make behaviour-testagainst halfsend org pool withGITHUB_TOKENand per-repo dummy runtimeCloses #2276