-
Notifications
You must be signed in to change notification settings - Fork 1
ci(coverage): give Ready the baseline file so it exercises the comparison path #542
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
c87c01a
ci(coverage): give Ready the baseline file so it exercises the compar…
stranske 615552c
docs(coverage): document coverage baseline fields and update procedure
stranske 983dd54
test(coverage): guard coverage-baseline.json against regressing its o…
stranske bfa7587
Claude (keepalive) automated update for PR #542
github-actions[bot] 389ec88
Claude (keepalive) automated update for PR #542
github-actions[bot] 79aadb7
test(coverage): validate active threshold configuration
stranske 033e86e
test(coverage): parse active configuration thresholds
stranske 3c6152f
chore(sync): apply Workflows warn_drop breach threshold in coverage_g…
stranske File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| { | ||
| "line": 80.0, | ||
| "warn_drop": 1.0, | ||
| "recovery_days": 3, | ||
| "updated": "2026-08-25", | ||
| "notes": "Ready exists to exercise every Workflows capability end-to-end, so it must exercise the coverage BASELINE path too, not just coverage measurement. Without this file tools/coverage_trend.py reports baseline_status=absent and computes no delta, and Maint Coverage Guard has nothing to compare against -- meaning the one repo whose job is to catch fleet CI defects was silently skipping the comparison. That mattered: the guard produced exactly one breach issue fleet-wide in ten months, and this is where that should have surfaced first. Keyed `line` deliberately: both tools/coverage_trend.py and tools/coverage_guard.py accept `line` or `coverage`, with `line` taking precedence, and exercising the precedence path is the point of a conformance repo. Set to 80 to match this repo's coverage-min and pyproject fail_under; measured coverage is 100% of the src/my_project scaffold, which is the only code Ready owns -- everything in scripts/ and tools/ is synced fleet tooling tested upstream in stranske/Workflows." | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| """Guard config/coverage-baseline.json against silently regressing the choices this repo made. | ||
|
|
||
| Ready's whole purpose is to exercise the fleet coverage pipeline end-to-end, so the baseline | ||
| file's fields aren't cosmetic: `line` (not `coverage`) exercises the precedence path both | ||
| tools/coverage_trend.py and tools/coverage_guard.py implement, and its value must agree with | ||
| coverage-min in pr-00-gate.yml and fail_under in pyproject.toml or the three drift apart. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import json | ||
| import math | ||
| import re | ||
| from pathlib import Path | ||
|
|
||
| REPO_ROOT = Path(__file__).resolve().parent.parent | ||
| BASELINE_PATH = REPO_ROOT / "config" / "coverage-baseline.json" | ||
| CI_WORKFLOW_PATH = REPO_ROOT / ".github" / "workflows" / "ci.yml" | ||
| PR_GATE_WORKFLOW_PATH = REPO_ROOT / ".github" / "workflows" / "pr-00-gate.yml" | ||
| PYPROJECT_PATH = REPO_ROOT / "pyproject.toml" | ||
|
|
||
|
|
||
| def _load_baseline() -> dict: | ||
| return json.loads(BASELINE_PATH.read_text()) | ||
|
|
||
|
|
||
| def test_baseline_file_exists() -> None: | ||
| assert BASELINE_PATH.is_file(), ( | ||
| "config/coverage-baseline.json must exist so Maint Coverage Guard has something to " | ||
| "compare against instead of reporting baseline_status=absent." | ||
| ) | ||
|
|
||
|
|
||
| def test_baseline_keys_line_not_coverage() -> None: | ||
| baseline = _load_baseline() | ||
| assert "line" in baseline, ( | ||
| "Baseline must key coverage as `line`: coverage_trend.py and coverage_guard.py both " | ||
| "accept `line` or `coverage`, with `line` taking precedence, and exercising that " | ||
| "precedence path is the point of a conformance repo." | ||
| ) | ||
| assert "coverage" not in baseline, ( | ||
| "Do not also set `coverage` -- that reintroduces the ambiguity `line` precedence is " | ||
| "meant to resolve." | ||
| ) | ||
|
|
||
|
|
||
| def _coverage_min(path: Path) -> float: | ||
| match = re.search( | ||
| r"(?m)^[ \t]*coverage-min[ \t]*:[ \t]*([\"']?)(\d+(?:\.\d+)?)\1[ \t]*(?:#.*)?$", | ||
| path.read_text(), | ||
| ) | ||
| assert match, f"{path.name} must set a numeric coverage-min." | ||
| return float(match.group(2)) | ||
|
|
||
|
|
||
| def test_baseline_matches_ci_gate_and_pyproject() -> None: | ||
| baseline = _load_baseline() | ||
| ci_min = _coverage_min(CI_WORKFLOW_PATH) | ||
| gate_min = _coverage_min(PR_GATE_WORKFLOW_PATH) | ||
|
|
||
| pyproject_text = PYPROJECT_PATH.read_text() | ||
| coverage_report = re.search( | ||
| r"(?ms)^\[tool\.coverage\.report\][ \t]*\n(.*?)(?=^\[|\Z)", pyproject_text | ||
| ) | ||
| assert coverage_report, "pyproject.toml must define [tool.coverage.report]." | ||
| fail_under_match = re.search( | ||
| r"(?m)^[ \t]*fail_under[ \t]*=[ \t]*(\d+(?:\.\d+)?)[ \t]*(?:#.*)?$", | ||
| coverage_report.group(1), | ||
| ) | ||
| assert fail_under_match, "pyproject.toml [tool.coverage.report] must set fail_under." | ||
| fail_under = float(fail_under_match.group(1)) | ||
|
|
||
| assert baseline["line"] == ci_min == gate_min == fail_under == 80.0, ( | ||
| "config/coverage-baseline.json `line`, ci.yml and pr-00-gate.yml `coverage-min`, " | ||
| "and pyproject.toml `fail_under` must all agree (80) so the four don't drift apart: " | ||
| f"line={baseline['line']!r}, ci={ci_min!r}, gate={gate_min!r}, " | ||
| f"fail_under={fail_under!r}" | ||
| ) | ||
|
|
||
|
|
||
| def test_baseline_has_warn_drop_and_recovery_days() -> None: | ||
| baseline = _load_baseline() | ||
| warn_drop = baseline.get("warn_drop") | ||
| assert isinstance(warn_drop, (int, float)) and not isinstance(warn_drop, bool) and math.isfinite(warn_drop), ( | ||
| "warn_drop must be a finite numeric value excluding booleans -- it's the coverage-point " | ||
| "drop that triggers a warning before a breach issue is opened." | ||
| ) | ||
| assert type(baseline.get("recovery_days")) is int, ( | ||
| "recovery_days must be an exact int excluding booleans -- consecutive passing days required before a breach " | ||
| "issue auto-closes." | ||
| ) | ||
| assert baseline["recovery_days"] > 0 | ||
|
stranske marked this conversation as resolved.
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.