Skip to content

ci: split tests/test-execution.toml into one registry file per area - #36348

Merged
teamleaderleo merged 6 commits into
feat-cmux-nextfrom
lane/split-test-registry
Nov 5, 2026
Merged

teamleaderleo merged 6 commits into
feat-cmux-nextfrom
lane/split-test-registry

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

tests/test-execution.toml had 186 commits in 14 days, and nearly every pull request that added a test conflicted with the next one, because every new entry went in at the top of the same file. The registry is now tests/test-execution.d/, one file per area. A new test's entry goes only in its area's file, at its sorted place, so pull requests in different areas never touch the same file, and two in the same area insert at different lines.

Leo, 2026-11-05, on the hot-files list (schemas/defaults/cmux-next.json and tests/test-execution.toml): "i meant can you do all of htem"

How it's split

  • The entry for tests/test_<word>_….py goes in <word>.toml when that file exists, otherwise in other.toml. Nothing lists the areas, so there's no shared index everyone appends to. The split made an area file for each first word with at least 5 tests: ci, cmux, nightly, ios, issue, shell, pr, web, release, swift, check, tui, gh, cloud, app, dev, ghostty, lint, plus other.
  • A fragment holds only [[test]] entries. The registry's version = 1 is implied.
  • Every reader loads one merged view: the old single file if it's still there (an old base commit, a test fixture), then each fragment by file name. That logic lives in scripts/ci/test_execution_registry.py (load_tree, load_tree_at for a commit). detect_ci_change_areas.py and focused_tests.py read the fragments inline, because their trusted copies run without that module.
  • If an entry is in the wrong file or a fragment isn't sorted, the validator prints a warning and never fails (no new CI gate). python3 scripts/ci/test_execution_registry.py fix repairs both. validate_test_execution_registry.py --write and the pre-commit hook already put new entries in the right place.

Readers changed

run_python_test_lane.py, focused_tests.py (a fragment maps to the tests that read the registry, so the car guards don't list fragments as unmapped), validate_test_execution_registry.py (reads, base reads through git, --write, messages), cmux_next_route.py, detect_ci_change_areas.py (head registry, --test-registry-base now also takes a directory, native entries compared by path, not by order; referrer checks), ci.yml (extracts the base's registry files for that flag), delta_since_green.py, workflow_guard_groups.py, post_edit_checks.py, guard_attribution.py (message), scripts/git-hooks/pre-commit, plus the docs that name the file (CLAUDE.md, the dev-workflow skill, docs/ci/derived-not-declared.md). No Swift reads the registry.

Rebasing an open PR that edits the old file

Merging or rebasing onto this leaves a modify/delete conflict on tests/test-execution.toml. Run:

python3 scripts/ci/test_execution_registry.py absorb

It reads what your branch changed in the old file (against the commit your branch started from), puts added and changed entries in their area files, removes deleted ones, deletes the old file and stages the result. Then git merge --continue / git rebase --continue.

This PR's own merges of the base aren't hand-resolved either. Each one is rebuilt: take the base's tests/test-execution.toml, delete tests/test-execution.d/, run python3 scripts/ci/test_execution_registry.py split, commit. The split is deterministic, so the result is the base's registry, entry for entry. 4 open teammate PRs edit the old file (#27910, #23811, #22857, #18426), out of 203 open teammate PRs into feat-cmux-next.

The defaults JSON stays one file

schemas/defaults/cmux-next.json is a generated snapshot (defaults_guard.py snapshot). The base autofix writes it after each merge, and PRs aren't supposed to commit it (#21946): 164 of its 205 commits in 14 days are the bot's. It's generated from the Swift sources and the settings and action exports, and those are already spread across per-area files. Splitting the output wouldn't take conflicts off any pull request, so this PR leaves it alone.

Verification

  • The merged view equals the single file: at the pre-split commit, 639 entries, compared field by field. After the rebuild on the base at 42abbd2, it equals the base's 641. RegistrySplitTests.test_the_real_pre_split_registry_splits_into_the_same_entries re-runs this from history.
  • Every reader gives the same answer at the base and at this head: run_python_test_lane.py --list for each lane, and each --shard K/3, the validator, focused_tests.lanes, cmux_next_route.linux_guard_tests and the detector's inert-lane answer for every test. Order within a lane is now by area and path instead of insertion order. The lane members and shards are the same.
  • New tests in tests/test_ci_test_execution_registry.py (split round-trip, tomllib merged text, placement and sort warnings plus fix, --write placement, reading from git in both layouts, absorb on a real merge and a real rebase, fix refusing a duplicate instead of dropping it), tests/test_ci_change_areas.py (a fragment edit routes through the normal and trusted router, and re-ordered entries are no native change), tests/test_install_git_hooks.py (the hook registers into a fragment), tests/test_car_guards.py (a fragment is mapped) and tests/test_cmux_next_route.py.
  • Review follow-ups from the SHIP at adca52a are fixed here: F1 (compare native entries by path) and F2 (fix refuses a duplicate path instead of keeping one entry).

Observed / Not run

Ran on the lane host, at the base and at this head: test_ci_test_execution_registry, test_ci_change_areas, test_cmux_next_route, test_cmux_next_route_replay, test_install_git_hooks, test_ci_delta_since_green, test_feat_ci_guard_group, test_car_guards, test_ci_parallel_artifact_transport, test_ci_workflow_guards_are_wired, test_risk_tier, test_web_validation, test_cmux_next_prepush_gate, test_preflight_trust, test_post_edit_checks and test_pr_lane_coverage. All pass except test_validator_exit_code_tracks_added_tests_in_a_git_fixture. On this host it fails identically at the base (42abbd2) and at this head (67c0c52), before any registry code runs: the fixture's git push -u origin main to a local bare repo is refused by the host's brokered git wrapper. Base: Ran 40 tests ... FAILED (errors=1), head: Ran 52 tests ... FAILED (errors=1), both with only ERROR: test_validator_exit_code_tracks_added_tests_in_a_git_fixture ... CalledProcessError: Command '['git', ..., 'push', '-u', 'origin', 'main']' returned non-zero exit status 2. CI runs it. The car's linux guards (car_guards.py run --where linux) at 67c0c52 against 42abbd2: 40 pass, nothing unmapped, and two host-only base reds (that test, and test_web_lint_prepush.py, whose web build doesn't run here). Not run: the ci.yml routing step itself, which runs on this PR's CI.

Changelog

none

Agent session: agent session

🤖 Generated with Claude Code

teamleaderleo and others added 3 commits November 5, 2026 08:53
tests/test-execution.toml had 186 commits in 14 days, and every pull request
that touched it conflicted with the next. It is now tests/test-execution.d/,
one file per area (the test name's first word: ci.toml, nightly.toml, ...,
other.toml for the rest), each sorted by path. Every reader loads the same
merged view: the old single file while present, then each fragment by name.

Leo, 2026-11-05: "i meant can you do all of htem"

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…test-registry

# Conflicts:
#	tests/test-execution.toml
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 67c0c52: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr36348 -f ref=lane/split-test-registry -f platform=macos -f backend_mode=local

Shipped in:

@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Nov 5, 2026
teamleaderleo and others added 3 commits November 5, 2026 09:33
Review follow-ups at adca52a: the change-area detector compared the
base's and head's native registry entries as ordered lists, so the split's
re-ordering (by area, then path) read as a native change. It compares by path
now. `test_execution_registry.py fix` kept one of two entries for the same
path and dropped the other's fields; it refuses a duplicate now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-a345f4d82727e281eabe
…registry

Rebuilt, not hand-resolved: tests/test-execution.toml taken from origin/feat-cmux-next,
the fragments deleted, then python3 scripts/ci/test_execution_registry.py split.

Agent-Session: https://cmux.com/deeplink/agent?id=aid-a345f4d82727e281eabe
…the registry

The car guards listed every tests/test-execution.d/*.toml as unmapped: no test
names a fragment by its own path. focused_tests.py now maps a fragment to the
tests that read the registry (they name tests/test-execution.toml or
tests/test-execution.d), as the single file was mapped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-a345f4d82727e281eabe
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high): SHIP at 67c0c52

router-review of 42abbd2..67c0c52 (gpt-6.1-sol gave no reply in 1491 s; gpt-6-astra at high gave the verdict).

Visual verdict: no images supplied (range review); no UI in this diff.

  • [P2] [follow-up] [category:performance] [finding:F1] [outcome:pending] scripts/ci/detect_ci_change_areas.py:1672 rejects the existing serial field while deciding whether registry changes are Linux-only. Since the checked-in registry contains serial entries, every fragment edit fails this comparison and queues macOS plus Release jobs; include serial in the accepted fields or compare it explicitly.

Follow-ups:

  • F1: Fix registry native-change detection to handle serial entries. The detector rejected serial before this PR too (the single file had serial entries), so this doesn't block it. A follow-up PR fixes it.

@teamleaderleo
teamleaderleo merged commit ae92dc4 into feat-cmux-next Nov 5, 2026
250 of 270 checks passed
@teamleaderleo
teamleaderleo deleted the lane/split-test-registry branch November 5, 2026 10:49
teamleaderleo added a commit that referenced this pull request Nov 5, 2026
…gain (#36451)

test_registry_change_is_linux_only gave up on any registry holding a serial
entry or a linux-guard entry with requirements, and the registry has both, so
every registry edit (every new test) queued the macOS and Release jobs. It
accepts serial now (compared as part of a native entry) and counts a
linux-guard entry with requirements as native, instead of giving up on the
whole registry. Already so with the single tests/test-execution.toml before
#36348; the review of #36348 at 67c0c52 found it (F1).

Leo, 2026-11-05: "i meant can you do all of htem"


Agent-Session: https://cmux.com/deeplink/agent?id=aid-a345f4d82727e281eabe

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 67c0c528 Deployed Nov 5, 2026 by teamleaderleo via cmux-tui tree preflight #115440
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant