ci(e2e): run the end-to-end suite on every code PR (#1420 §1)

[robotrocketscience]

Refs #1420 §1. Does not close the issue — §2 (perf), §3 (bench-gate corpus)
and §4 (mutation) remain open, so there is no Closes link.

The gap

e2e.yml:26 was:

if: github.event_name == 'push' || contains(github.event.pull_request.labels.*.name, 'e2e')

End-to-end coverage therefore ran on main after a change merged, or on a PR
only if someone remembered to add an e2e label.

This is not redundant with the unit suite. pytest tests/ imports from the
source tree; e2e exercises the three installed paths — uv-tool, pipx,
venv-pip. A regression in packaging, entry points, or the installed CLI is
invisible to the unit suite by construction and had no pre-merge gate at all.

A second, smaller defect fell out of reading it: the trigger was
types: [labeled, synchronize, reopened], which does not include opened. So
even the label escape hatch did not work on a fresh PR without a second push.

What ships

• Label opt-in removed; opened added to the event types.
• Trigger path-filtered to the same globs as ci.yml's code filter, so
  docs-only PRs stay off it.

Why trigger-level paths is safe here and must not be copied. e2e is not
one of the branch's required contexts (those are history-scan, pattern-scan,
pytest (3.12), pytest (3.13), secrets-scan). A path-filtered required
check never reports on a PR it skips and leaves that PR permanently pending —
which is exactly why ci.yml has no paths and filters inside the job with
dorny/paths-filter instead. The workflow comment says this and
test_e2e_is_path_filtered_at_the_trigger_not_inside_the_job pins it, so the
pattern cannot be copied into a required workflow by someone reading only the diff.

Tests

Appended to the existing tests/test_ci_path_filter.py rather than a new module
— that file already derives its expectations by scanning, instead of comparing a
hand-maintained list against the constants it was copied from, and these follow
the same shape.

test	pins
test_e2e_runs_on_code_prs_without_an_opt_in_label	the label gate is gone and opened is present
test_e2e_paths_cover_the_ci_code_filter	the two path lists cannot drift apart
test_e2e_is_path_filtered_at_the_trigger_not_inside_the_job	the required-vs-advisory distinction above

The second one is the load-bearing one: e2e.yml carries a "keep these in sync"
comment, and a comment cannot enforce that. A path added to ci.yml and
forgotten in e2e.yml means a PR that changes installed behaviour runs the unit
suite, skips e2e, and shows nothing red.

Verification

• Mutation-checked, cleared __pycache__, one per gate:

  mutation	result
  restore the label if:	test_e2e_runs_on_code_prs_without_an_opt_in_label fails
  drop benchmarks/** from e2e's paths	test_e2e_paths_cover_the_ci_code_filter fails

  Each turns exactly one test red. Restored: 7 passed.

• YAML validated; the e2e job now carries no if:, and surface-failure's
  github.event_name == 'pull_request' branch is unchanged and still labels a
  failing PR attn:e2e-failure.

Cost

e2e is timeout-minutes: 8 across a 3-way install matrix, running in parallel
with the rest. Median PR settle time before this change was ~4 min (measured on
runs created before the 2026-08-06 Actions incident — runs inside that window
report retry and queue time as duration and are not a usable baseline). The
post-merge number should be re-measured once Actions is healthy and quoted on
#1420 §1's AC3 rather than estimated here.

Out of scope

§2 perf gating, §3 the 36 corpus-dependent bench-gate tests, §4 mutation on PRs.
Each is a separate change with its own constraints — §3 in particular is blocked
on a security decision about self-hosted runners on a public repo, not on effort.

Summary by Sourcery

Ensure the end-to-end workflow runs on every code pull request and keep its path filter aligned with the main CI configuration.

Bug Fixes:

• Run the end-to-end suite on all code pull requests instead of only on pushes or label-gated PRs, and include newly opened PRs in the trigger types.

Enhancements:

• Path-filter the e2e pull_request trigger to match the CI workflow's code paths while explicitly documenting why this is safe only for non-required checks.
• Document the e2e triggering and path-filtering behaviour and its rationale in the changelog.

Documentation:

• Add changelog notes describing that e2e coverage now runs on every code PR and clarifying its relationship to the unit test suite.

Tests:

• Extend the CI path filter test suite with checks that e2e is no longer label-gated, that its paths stay in sync with the CI code filter, and that path filtering remains at the trigger for e2e but inside the job for required checks.

Summary by CodeRabbit

• Bug Fixes

  • End-to-end tests now run automatically for newly opened, updated, and reopened code-related pull requests.
  • Removed the requirement to add an opt-in label before running end-to-end checks.
  • Improved path filtering so relevant code changes consistently trigger the appropriate checks.

• Documentation

  • Added changelog notes describing the updated end-to-end test behavior.

[sourcery-ai]

Sorry @robotrocketscience, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[coderabbitai]

Warning

Review limit reached

@robotrocketscience, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 15 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 9962ee52-d6af-46b5-ad4b-75ecd00e00b1

📥 Commits

Reviewing files that changed from the base of the PR and between fd94c27 and 2d1e4ae.

📒 Files selected for processing (3)

• CHANGELOG/v4.md
• docs/design/testing-strategy.md
• tests/test_ci_path_filter.py

📝 Walkthrough

Walkthrough

The E2E workflow now runs for code-related pull requests without an e2e label. It includes newly opened, synchronized, and reopened pull requests. Tests verify path and trigger alignment with CI behavior.

Changes

E2E trigger alignment

Layer / File(s)	Summary
Update E2E workflow triggers .github/workflows/e2e.yml	The workflow adds code-related pull-request path filters, includes opened, synchronize, and reopened events, and removes the e2e label condition.
Validate trigger behavior tests/test_ci_path_filter.py, CHANGELOG/v4.md	Tests verify path coverage, event coverage, label-gate removal, and trigger-level filtering. The changelog records the changes.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• robotrocketscience/aelfrice#354: Introduced the E2E workflow that this change updates.
• robotrocketscience/aelfrice#371: Modified the same E2E workflow triggers and label-gated behavior.
• robotrocketscience/aelfrice#1195: Updated pull-request path filters and added workflow validation for another CI workflow.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the primary change: running the end-to-end suite on every code pull request.
Description check	✅ Passed	The description explains the problem, scope, implementation, tests, verification, costs, and out-of-scope work; it is mostly complete despite omitting the template headings.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch ci/issue-1420-e2e-on-code-prs

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sourcery-ai]

Reviewer's Guide

Ensures the GitHub Actions end-to-end workflow runs on every code-touching pull request (not just main or label-gated PRs), keeps its path filter aligned with the main CI code filter, and adds tests that pin this behavior and document why trigger-level path filtering is safe only for non-required checks, with a brief changelog entry describing the fix.

File-Level Changes

Change	Details	Files
Update e2e GitHub Actions workflow to always run on code PRs using a trigger-level paths filter and no label-based gate.	Change pull_request trigger types to include opened and remove labeled, matching how PRs should initially run e2e. Add a paths list under the pull_request trigger that mirrors ci.yml’s code filter, including src, tests, benchmarks, scripts, key project files, and the workflow itself. Remove the if condition that previously restricted the e2e job to push events or PRs with an e2e label. Expand and clarify workflow comments to explain why trigger-level paths is safe only because e2e is a non-required context, and to instruct keeping the path list in sync with ci.yml’s code filter.	.github/workflows/e2e.yml
Add tests that parse e2e.yml and ci.yml to enforce trigger behavior and path-filter consistency between e2e and CI.	Introduce a helper that reads on.pull_request.paths from e2e.yml and parses the glob list robustly, asserting exactly one paths key and non-empty results. Add a test that asserts the e2e workflow no longer contains the label-based if condition and that pull_request.types includes opened, ensuring e2e runs on new code PRs without an opt-in label. Add a test that compares ci.yml’s code filter globs with e2e.yml’s paths (excluding workflow-file entries) and fails if any CI code globs are missing from e2e, preventing silent under-triggering of e2e. Add a test that asserts ci.yml continues to use dorny/paths-filter inside the job and that e2e.yml does not, pinning the required-vs-advisory distinction and preventing incorrect copying of the trigger-level paths pattern into required workflows.	tests/test_ci_path_filter.py
Document the behavior change in the v4 changelog.	Add a Fixed entry explaining that end-to-end coverage now runs on every code PR, not just after merge or via label, and that the trigger is path-filtered to the same globs as ci.yml’s code filter with opened added as a pull_request type. Describe the rationale for not using trigger-level paths on required workflows and reference the new tests that enforce the absence of the label gate and path-list agreement, noting that the change was mutation-tested.	CHANGELOG/v4.md

Possibly linked issues

• ci: the quality tier does not gate PRs — e2e, perf, bench-gate and mutation are all skipped, label-gated or nightly #1420: Link: the PR fulfills ci: the quality tier does not gate PRs — e2e, perf, bench-gate and mutation are all skipped, label-gated or nightly #1420 §1 by ungating e2e on PRs and aligning its path filters with ci.yml.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/test_ci_path_filter.py`:
- Around line 149-166: Update the path-list parsing in the test to locate the
`paths:` key specifically under `on.pull_request`, rather than accepting any
unique `paths:` key in the workflow. Preserve the existing indentation-based
glob collection and assertions after scoping the scan to the pull-request
configuration.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: eb4717c6-b685-4f3d-8cfd-c625303297ae

📥 Commits

Reviewing files that changed from the base of the PR and between 9848be0 and fd94c27.

📒 Files selected for processing (3)

• .github/workflows/e2e.yml
• CHANGELOG/v4.md
• tests/test_ci_path_filter.py

[robotrocketscience]

[claim:review:Kulili:2026-08-06T20:25:22Z]

[robotrocketscience]

Review — the change is right; 6 findings fixed on this branch (fd94c27d..2d1e4aea)

Four lenses (AC compliance, trigger semantics, test adequacy, blast radius), each
finding then handed to an independent verifier told to refute it. 27 raised, 21
refuted, 6 survived. I re-derived the load-bearing numbers myself rather than
relaying them.

Fixed here

1. AC2 had no guard. test_e2e_paths_cover_the_ci_code_filter asserts
ci - e2e == {} — e2e's list is a superset of ci's — and a superset assertion
is blind to additions. Appending docs/** and CHANGELOG/** to the trigger kept
all seven tests green while putting the 3-leg install matrix on every docs-only
PR. That is AC2 reversed with nothing red to show for it, and it is the same
class as #1160 pointed the other way. test_e2e_still_skips_docs_only_prs mirrors
the guard ci.yml already has.

2. AC1 was pinned as the absence of one literal. Any re-gate with a different
spelling survives — if: github.event_name == 'push' on the job negates AC1
entirely and keeps labels.*.name, 'e2e' absent. Now asserted as the invariant:
the e2e job carries no job-level if: at all.

3. The paths parser was not bound to the trigger it claims. It grepped the
whole file for paths: and asserted exactly one, so it could not distinguish the
pull_request list from a push list, and a legitimate push: paths: would have
failed an unrelated test with a message about parser trust. It now walks
on: → pull_request: → paths: by indentation. Block extraction returns index
ranges, not line contents, because sibling blocks hold byte-identical lines
(branches: [main] appears under both triggers) and content matching binds to the
wrong one.

Mutations: docs/** added → red; job re-gated with a different expression → red;
benchmarks/** dropped → red; legitimate push: paths: added → still green.

4. CHANGELOG opened a second ### Fixed inside [Unreleased], above
### Added, while [Unreleased] already had one further down. Moved by hand
(not by a resolver); the diff against main is a pure insertion, zero removed lines.

5-6. docs/design/testing-strategy.md:43 still documented the e2e label
opt-in. Updated, and it now says plainly that the label no longer triggers
anything — a label that silently stopped working is worse than one that was
deleted.

AC3 — here is the number, and the honest caveat

The PR body quotes no wall-clock delta. Measured myself from the API, excluding
everything after the Actions incident began at 2026-08-06T15:22Z (per #1420's own
warning against incident-window data):

	n	median	p90	max
e2e full 3-leg matrix (push runs)	100	47s	—	81s
ci.yml on pull_request	83	220s	255s	321s

The distributions are disjoint: e2e's slowest observed run is 81s against a median
CI settle of 220s. The three legs run in parallel with everything else, so the
expected delta to median PR settle time is ~0, and e2e cannot become the
critical path unless it regresses by 3x.

The caveat that makes this an estimate rather than a measurement: there are zero
completed e2e pull_request runs in the workflow's entire history — 2,668 runs,
all skipped bar three that were cancelled. The figures above are the push regime
substituted for the PR regime, and the PR regime has more contention (a code PR
already fires ~25 non-skipped check-runs). AC3 as written is not fully answerable
until this change lands and produces the first real sample. Worth saying so in the
body rather than leaving the AC silently unaddressed.

Two things the review turned up that are not defects in this PR

Both reproduce, both are pre-existing, and neither is claimed here — but they
change how this PR should be read, so they should not stay buried in a review.

e2e has not executed against any change in 541 commits. The newest
push-event run is 2026-07-21T01:30:43Z at 3421cd73, pushed by a human.
git rev-list --count 3421cd73..github/main = 541. The reason is that
merge-train FF-pushes with secrets.GITHUB_TOKEN, and GitHub does not trigger
workflows on pushes made with that token — so the "post-merge regression catch"
half of e2e's trigger has been inert since the merge train took over. Combined
with the label opt-in nobody used, the installed-package path has had no gate at
all, on either side.

This makes #1420 §1's premise understated: a regression there is not "caught on
main instead of on the PR" — it is caught nowhere. It also means this PR will be
the first time e2e has actually run in 541 commits, which is worth knowing before
merging: if it comes back red, that is 16 days of accumulated packaging drift
surfacing at once, and under merge-train's all-checks gate it would block every
labelled PR simultaneously. The good news is that this PR triggers e2e on itself
(.github/workflows/e2e.yml is in its own paths list), so its own run is the
validation — hold the merge until it is observed green rather than merging on
the unit suite alone. I'd file the push-side deadness separately; say the word and
I will.

A failing e2e will block merge-train — but that is not new. I verified
merge-train.yml gates on the full check-runs rollup minus two names, so every
check-run gates, required or not. The PR's "e2e is not a required context" argument
is therefore about branch protection only. It is still correct that nothing new
enters the rollup: e2e already emits check-runs on unlabelled PRs today (verified
on merged head aae8126a: two e2e check-runs, both skipped), because
synchronize fires and the job-level if skips. This PR changes an existing
member's conclusion from skipped to success/failure. Worth one sentence in
the body, since "not required" reads as "cannot block" and that is not the case here.

Verification

• FF on github/main, 4 commits, all signed (G)
• uv run --extra archive pytest -q -p no:randomly: 7389 passed, 70 skipped,
  71 xfailed, 0 failed
• Note for anyone re-running: without --extra archive six uninstall tests fail
  locally. That is the documented false negative from a venv missing CI's deps,
  not a regression — they pass on github/main in isolation and the full suite is
  clean with the extra.
• tests/test_ci_path_filter.py: 8 passed
• discretion grep on added lines: clean

AC1 ✅ AC2 ✅ (now guarded) AC3 ⚠️ estimated above, unobservable until this lands.
Approving on the merged state, with the one hold: watch this PR's own e2e run
before labelling ready-to-merge. Actions is in a major outage (started
15:22Z), so nothing can run yet.

[github-actions]

PR-size soft cap

This PR is over the advisory size threshold:

• 216 changed lines (limit: 200)
• 4 changed files (limit: 3)

Bigger PRs collide with more open work, which under the parallel-session workflow tends to produce repeated attn:merge-conflict cycles (see #602). When practical, split into smaller PRs that each touch a focused surface.

This is advisory only — nothing is blocked. If the size is intentional (large refactor, module removal, generated code), apply the size:override label and this comment will be removed on the next push.

[robotrocketscience]

[claim:review:Garsecg:2026-08-06T22:55:07Z]

[robotrocketscience]

[release:review:Garsecg:2026-08-06T22:55:13Z]

[robotrocketscience]

[claim:review:Gylf:2026-08-06T22:59:53Z]

[robotrocketscience]

[release:review:Gylf:2026-08-06T22:59:58Z]

[github-actions]

merge-train: merged 2d1e4ae → main via FF push.

[github-actions]

merge-train: merged 2d1e4ae → main via FF push.