Skip to content

fix(OMN-14081): make pyproject.toml test-selector trigger content-aware - #1393

Merged
jonahgabriel merged 2 commits into
devfrom
jonah/omn-14081-pyproject-content-aware-test-selector
Jul 7, 2026
Merged

jonahgabriel merged 2 commits into
devfrom
jonah/omn-14081-pyproject-content-aware-test-selector

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Jul 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Closes OMN-14081 (parent epic OMN-13969, WS-T selective-testing revival).

omnibase_core's governed change-aware test selector escalated every PR to the full 40-shard suite (~22.5 min, 40 runner slots) with full_suite_reason=test_infrastructure, because pyproject.toml was a bare path-prefix trigger in test_selection_adjacency.yaml with zero content inspection. Since core is the foundational released package, nearly every PR touches pyproject.toml for a version bump or entry-point registration, so nearly every PR escalated. ENABLE_SMART_TESTS=true is live, so this was production CI behavior — the single biggest demand-reduction lever on core's queue.

Changes

  • Content-aware pyproject.toml classification (detect_test_paths.py): a base-vs-head TOML diff (classify_pyproject_dependency_relevant) narrows only when the change is confined to metadata-only [project] keys (version, entry-points, scripts, urls, description, classifiers, license, authors, …). It escalates on any change to the dependencies array, [project.optional-dependencies], [dependency-groups], [build-system], [tool.*] (incl. [tool.pytest.ini_options]/[tool.coverage]), requires-python, or any unrecognized key.
  • FAIL-CLOSED by construction: missing base/head content, a TOML parse failure, or no --base-ref supplied all escalate. This is an allow-list of SAFE keys, not a block-list of dep tables — an unknown/new pyproject key escalates by default. Shared-module / test-infra / threshold escalation is unchanged.
  • Remove .github/workflows/ from test_infrastructure_paths (align to omnibase_infra, which omits it). A workflow-only change carries zero test-code delta, is exercised on the PR's own CI run, and is backstopped by the unconditional merge_group → main full suite (root CLAUDE.md Rule TKT-003: Validate Import Resolution and Eliminate Unused Dependencies #4). scripts/ci/ is KEPT (conservative — it houses the selector itself). This addresses the merge_group-trim workflow-only PR case.
  • Wire --base-ref (from github.event.pull_request.base.sha) into ci.yml's detect-changes step so the classifier can read the base pyproject.toml.

Before / after proof (real version-bump PR #1388, 0.46.5 -> 0.46.6)

selector result
BEFORE (origin/dev) pyproject = blanket trigger is_full_suite=true, reason=test_infrastructure, 40 shards
AFTER (this PR) content-aware is_full_suite=false, narrowed

Dominant real-world case (1-module fix + version bump + lockfile): 40 shards → 1 shard (tests/unit/cli/).
Control — a real dependency add: still escalates (fail-closed preserved).

Tests

Adds 24 tests: classification unit tests (version bump, entry-point, metadata, real dep add, optional-deps, dependency-groups, build-system, uv.sources, requires-python, pytest-config, malformed/missing → fail-closed); compute_selection escalation paths (relevant True/False/None, shared-module non-suppression, ignored-when-absent); and CLI end-to-end tests against a real tmp git repo (--base-ref + git show chain).

Local gates green: ruff format + ruff check, mypy --strict, pre-commit (97 hooks), pytest tests/unit/scripts/ci/ (87 passed).

dod_evidence

Evidence-Source: OCC#3664
Evidence-Ticket: OMN-14081

The governed change-aware test selector escalated EVERY omnibase_core PR to
the full 40-shard suite (~22.5 min, 40 runner slots) with
full_suite_reason=test_infrastructure, because `pyproject.toml` was a bare
path-prefix trigger in test_selection_adjacency.yaml with zero content
inspection. Since core is the foundational released package, nearly every PR
touches pyproject.toml for a version bump or entry-point registration, so
nearly every PR escalated. ENABLE_SMART_TESTS=true is live, so this was
production CI behavior.

Changes (detect_test_paths.py + test_selection_adjacency.yaml + ci.yml):
- Content-aware pyproject.toml classification via a base-vs-head TOML diff
  (classify_pyproject_dependency_relevant). Narrow ONLY when the change is
  confined to metadata-only [project] keys (version, entry-points, scripts,
  urls, description, classifiers, license, authors, ...). Escalate on any
  change to the dependencies array, [project.optional-dependencies],
  [dependency-groups], [build-system], [tool.*] (incl. pytest/coverage
  config), requires-python, or any unrecognized key.
- FAIL-CLOSED by construction: missing base/head content, a TOML parse
  failure, or no --base-ref supplied all escalate. It is an allow-list of
  SAFE keys, not a block-list of dep tables — unknown keys escalate.
- Remove .github/workflows/ from test_infrastructure_paths (align to
  omnibase_infra). A workflow-only change carries zero test-code delta, is
  exercised on the PR's own CI run, and is backstopped by the unconditional
  merge_group -> main full suite (root CLAUDE.md Rule #4). scripts/ci/ is
  KEPT (conservative — it houses the selector itself).
- Wire --base-ref (from github.event.pull_request.base.sha) into ci.yml's
  detect-changes step so the classifier can read the base pyproject.toml.

Proof on the real version-bump PR #1388 (0.46.5 -> 0.46.6):
  BEFORE: is_full_suite=true, reason=test_infrastructure, 40 shards
  AFTER:  is_full_suite=false, narrowed
Typical case (1-module fix + version bump + lockfile): 40 shards -> 1 shard.
Control (real dependency add): still escalates (fail-closed preserved).

Adds 24 tests (classification unit tests, compute_selection escalation
paths, and CLI end-to-end tests against a real tmp git repo). Full local
gates green: ruff format+check, mypy --strict, pre-commit (97 hooks).

Parent epic OMN-13969 (WS-T selective-testing revival).
@jonahgabriel
jonahgabriel enabled auto-merge July 6, 2026 23:11
@coderabbitai

coderabbitai Bot commented Jul 6, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 45 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 1af097b8-7cf6-4aef-bb28-429109d75f76

📥 Commits

Reviewing files that changed from the base of the PR and between 20740dc and a6dec27.

📒 Files selected for processing (5)
  • .github/workflows/ci.yml
  • scripts/ci/detect_test_paths.py
  • scripts/ci/test_selection_adjacency.yaml
  • tests/unit/scripts/ci/test_detect_test_paths.py
  • tests/unit/scripts/ci/test_detect_test_paths_cli.py
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch jonah/omn-14081-pyproject-content-aware-test-selector

Comment @coderabbitai help to get the list of available commands.

@jonahgabriel
jonahgabriel added this pull request to the merge queue Jul 7, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Jul 7, 2026
@jonahgabriel
jonahgabriel merged commit d4ee265 into dev Jul 7, 2026
124 of 125 checks passed
@jonahgabriel
jonahgabriel deleted the jonah/omn-14081-pyproject-content-aware-test-selector branch July 7, 2026 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant