Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -276,10 +276,10 @@ lint-custom:
## LKG sidecar exists), and runs pytest on only that subset. Any
## sign of static-analysis fog (conftest / Makefile / pyproject /
## uv.lock / workflow / shared/tests / non-.py / gateway/*.py /
## dynamic-import / unresolvable-baseline / LKG-not-ancestor /
## canary every-10th invocation) widens to the full suite with an
## explicit trigger string on stderr. See docs/guides/testing.md
## and scripts/select_tests.py for the full design.
## dynamic-import / unresolvable-baseline / LKG-not-ancestor) widens
## to the full suite with an explicit trigger string on stderr. See
## docs/guides/testing.md and scripts/select_tests.py for the full
## design.
##
## CI runs `make test-all` (below) to keep the 80% coverage gate
## enforced; narrowing is a local-inner-loop optimisation only.
Expand Down
35 changes: 17 additions & 18 deletions docs/guides/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,12 @@ by the `make test` recipe. The algorithm is:
suite with trigger `unresolvable baseline`.
2. **Compute the changed-files set.** Union of `git diff --name-only
<baseline>...HEAD` and `git status --porcelain` (uncommitted
work). An empty diff falls back to the full suite — a fresh tree
should be honest about coverage.
work). An empty diff against a resolvable, current baseline
short-circuits to "selected 0 tests" and skips pytest entirely
— there is nothing reachable from a non-existent change. Empty
diff combined with an unresolvable baseline or a stale LKG still
widens to the full suite (those are real safety triggers, not
changeset reasoning).
3. **Evaluate fallback triggers.** Any trigger from §4 short-circuits
selection and runs the full suite, with the explicit trigger
string printed to stderr.
Expand Down Expand Up @@ -174,8 +178,6 @@ suite, with the explicit trigger string written to stderr (e.g.
| **Unresolvable changed path** | A changed path that cannot be mapped to an in-repo module (e.g. brand-new file not yet in grimp's graph, a `scripts/*.py` with no wheel binding). |
| **Unresolvable baseline** | LKG missing AND `origin/<base>` missing or `merge-base` failing. |
| **LKG not ancestor of HEAD** | The recorded LKG sha is not reachable from `HEAD` (force-push, reset, history rewrite). |
| **Empty diff** | `git diff` plus `git status` produces zero changed paths. Conservative — empty diff on a fresh tree should still be honest about coverage. |
| **Canary every 10th invocation** | A per-branch counter at `.egg-state/last-known-good/<branch>.canary` increments on every narrow `make test` invocation. When `count % 10 == 0`, the selector forces the full suite with trigger `canary (every-10th invocation)` and resets the counter. This catches silent drift that the static analysis missed. |

When a fallback fires, the stderr line uses the **explicit trigger
name** (e.g. `Makefile changed`, `gateway source change (importlib
Expand Down Expand Up @@ -221,7 +223,6 @@ Every invocation also persists a structured record. The schema:
| `compute_ms` | integer | Selector wall-clock in milliseconds. |
| `pytest_ms` | integer | Pytest wall-clock in milliseconds; written by the Makefile after pytest returns (`select_tests.py --patch-selection-json --head <sha> --pytest-ms <int>`). |
| `timestamp` | string | ISO-8601 UTC timestamp. |
| `canary_fired` | bool | Whether the canary trigger fired this invocation. |
| `changed_files` | list[string] | Paths from `git diff` + `git status`. |
| `changed_modules` | list[string] | Resolved module paths for the changed files. |
| `dynamic_import_seeds_hit` | list[string] | Module paths flagged as containing dynamic-import patterns and reachable from the changed set. |
Expand Down Expand Up @@ -305,19 +306,18 @@ Static reverse import graphs are powerful, but they cannot see:
(e.g. `subprocess.run(["python", "scripts/foo.py"])`) does not
have an import edge to that binary. Tests that exercise CLI
surfaces by subprocess will be missed by narrowing alone — this
is mitigated in practice because (a) such tests almost always
also touch a shared test fixture or a subprocess-helper module
that grimp does see, and (b) the canary every-10th-invocation
rule provides a periodic full-suite checkpoint.
is mitigated in practice because such tests almost always also
touch a shared test fixture or a subprocess-helper module that
grimp does see. Run `make test-all` before push for ground truth.
- **Data-file loads.** A test that does
`Path("fixtures/foo.json").read_text()` has no import edge to
the data file. Non-`.py` changes fall back to the full suite as
a blunt-but-stable mitigation.
- **Entry-point plugin registrations.** Plugins discovered via
`importlib.metadata.entry_points()` (e.g. some pytest plugins)
do not have static import edges from their consumers. The
dynamic-import scan picks up the consumer side; the canary
catches missed cases.
dynamic-import scan picks up the consumer side; missed cases
surface when CI runs `make test-all`.
- **Gateway's importlib test-loader.** This is the most
consequential blind spot in this repo. `gateway/tests/conftest.py`
defines `_load_module_with_replaced_imports`, which uses
Expand Down Expand Up @@ -369,7 +369,7 @@ Two directories accumulate on disk and are gitignored:
- `.egg-state/selection/` — one JSON record per `make test`
invocation, keyed by `HEAD` sha.
- `.egg-state/last-known-good/` — at most one `<branch>.sha` file
and one `<branch>.canary` counter per branch.
per branch.

**No automatic pruning is performed.** Per-invocation pruning
would add filesystem churn and a subtle GC dependency for marginal
Expand Down Expand Up @@ -443,8 +443,6 @@ The stderr trigger string is the diagnostic. Common cases:
sidecar.
- `unresolvable baseline` — `origin/main` is missing or
inaccessible. Check `git remote -v` and `git fetch origin`.
- `canary (every-10th invocation)` — expected once per ten narrow
runs.
- `gateway source change (importlib test-loader)` — known blind
spot, see §7. The full suite is the right answer here.

Expand All @@ -457,11 +455,12 @@ PYTEST_ARGS="path/to/specific/test.py" make test
An explicit path in `PYTEST_ARGS` bypasses narrowing — the
developer is asking for that specific selection.

**"I changed nothing but `make test` ran the full suite."**
**"I changed nothing and `make test` did nothing."**

Empty-diff handling intentionally falls back to the full suite — a
fresh tree should be honest about coverage. Make a change, or run
`make test-all` directly.
That is the correct behavior. With no changes against a resolvable
baseline, the selector emits `selected 0 tests (skipping pytest)` and
the Makefile prints `no tests selected` and exits 0. To run the full
suite anyway, use `make test-all`.

---

Expand Down
Loading
Loading