fix(coverage): the trend reporter could not fail — an unread baseline scored 0.00% and always "improved" - #3251
Merged
Conversation
… scored 0.00% and always "improved"
Three defects in the coverage measurement, all of the same shape: one sentinel standing
for two different meanings, where only one of them is good news.
1. BASELINE KEY MISMATCH. `tools/coverage_trend.py` read only `payload.get("coverage")`.
`tools/coverage_guard.py`, reading the SAME config file, has always read
`payload.get("line", payload.get("coverage"))`. stranske/Trend_Model_Project ships
`{"line": 85.0}`, so the trend silently scored 0.0 while the guard scored 85.0 — two
scripts disagreeing about one file, and neither saying so.
2. ABSENT BASELINE RENDERED AS ZERO. A missing, unreadable or differently-keyed file all
collapsed to 0.0, so `delta = current - 0` printed as a large improvement on every run
and the comparison could never fail. Ten of the thirteen lane repos have no baseline
file at all. `_resolve_baseline` now returns None with a status naming WHICH cause
(unset / absent / unreadable / no_recognised_key), because each is a different fix; the
record carries `baseline: null`, `delta: null`, `baseline_status`, and the summary says
the status reflects only the `--minimum` floor.
This one was PINNED AS CORRECT: test_main_handles_missing_baseline_and_empty_hotspots
asserted `Baseline | 0.00%` and `baseline=0.00`. It was not unnoticed, it was ratified.
3. FOREIGN ROWS AVERAGED IN SILENTLY. A test that copies the source tree into a tmpdir
makes coverage.py record every module twice — once real, once as a barely-executed
duplicate that sorts to the top of the hotspot table. On Trend_Model_Project 95 rows
were `/tmp/pytest-of-runner/.../workspace/src/...`, paths that do not exist in the
repo, so the one actionable output pointed at files nobody could open. Now partitioned:
`foreign_file_count`, `foreign_files` and `current_project_only` sit BESIDE an unmodified
`current` (so the record still agrees with coverage.xml and the delta job), and the
summary names `[tool.coverage.run] omit` as the fix. "Every row is foreign" gets its own
message — that is a wrong `--project-root`, not contamination, and a different remedy.
Plus the baseline PATH: the workflow read `${GITHUB_WORKSPACE}/config/coverage-baseline.json`,
but PROJECT_ROOT is `workspace/<working-directory>` for a consumer that sets one — and
stranske/Travel-Plan-Permission does. Now PROJECT_ROOT first with workspace fallback, so
every `working-directory: "."` consumer resolves to exactly the same file as before.
MEASURED ON THE REAL ARTIFACT (Trend_Model_Project Gate run 32794743236):
before: Current 83.32% | Baseline 0.00% | Delta +83.32% | Status Pass
after: Current 83.33% | Baseline 85.00% | Delta -1.67% (a real regression)
95 foreign files reported; project-only coverage 90.79%
hotspots now name real files instead of 13 phantom /tmp paths
Deliberate break -> revert on all three, each failing only its own cases, revert
byte-identical: coverage-only key read (4 failures), 0.0-for-absent (2), no partitioning (2).
tests/tools/test_coverage_trend.py 17 -> 25. Full suite 1368 passed, 4 skipped (pre-existing).
black/ruff clean at line-length 100; workflow YAML parses.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Warning Review limit reachedNext included review available in 15 minutes. View limit detailsLimit details: You’ve used the included review currently available. Your 71 included PR review attempts over the past 7 days set your current allowance at 1 review per hour. Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab. Review configuration: ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (4)
Comment |
Contributor
Workflow source neededPR #3251 needs either a linked GitHub issue or one valid non-issue Workflow Source before PR metadata automation can manage it safely. Please do one of:
Once a valid source is present, this warning will not be reposted. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e3292f4731
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| --coverage-json coverage.json \ | ||
| --baseline "${GITHUB_WORKSPACE}/config/coverage-baseline.json" \ | ||
| --baseline "${BASELINE_PATH}" \ | ||
| --project-root "${PROJECT_ROOT}" \ |
There was a problem hiding this comment.
--project-root
For consumer repositories whose synced tools/coverage_trend.py still predates this commit, this step invokes that consumer-side script with an unsupported --project-root argument, so argparse exits with code 2 and the primary Python test job fails whenever the Gate enables coverage reporting. The reusable workflow takes effect immediately through @main, while the managed script reaches consumers only through a later sync, creating a fleet-wide failure window; either retain CLI compatibility until the script rollout completes or invoke a helper checked out from the same Workflows ref.
AGENTS.md reference: AGENTS.md:L31-L33
Useful? React with 👍 / 👎.
Contributor
Automated Status SummaryHead SHA: 1c7a273
Coverage Overview
Coverage Trend
Top Coverage Hotspots (lowest coverage)
Low Coverage Files (<50.0%)
Updated automatically; will refresh on subsequent CI/Docker completions. Keepalive checklistScopeNo scope information available Tasks
Acceptance criteria
|
stranske
added a commit
that referenced
this pull request
Aug 25, 2026
… have (#3253) REGRESSION I INTRODUCED IN #3251, caught before it hit the fleet. `reusable-10-ci-python.yml` ships to consumers INSTANTLY via @main. The script it runs, `${GITHUB_WORKSPACE}/tools/coverage_trend.py`, is the CONSUMER'S copy — a sync-managed file that only arrives on the next maint-68 run (daily, 05:00 UTC). maint-68's own header says this: "Runtime behavior already reaches consumers instantly via @main; only the synced file COPIES wait for this run." So for one sync window the workflow is newer than the script it calls, and #3251 added `--project-root` to the invocation. argparse exits 2 on an unrecognized argument, which would have failed the coverage step in all eight soft-gate consumers at once — verified against a real stale copy (stranske/Counter_Risk's): "error: unrecognized arguments: --project-root .". Probe rather than assume. The flag is optional by construction: the script defaults to cwd, which is already correct for every `working-directory: "."` consumer, so a stale copy loses only the monorepo fix and keeps working. Verified both directions under bash (the shell Actions actually uses): stale copy -> flag omitted, exit 0; current copy -> flag passed, exit 0. Same unquoted-expansion idiom as the existing $MIN_ARG two lines below. Damage: none. One Gate run started in the window (stranske/Orchestrator) and no consumer run had reached the coverage step. The shim is dated and self-removing by instruction: delete it once the fleet has synced past 2026-08-25. Co-authored-by: Codex Automation <codex-automation@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
that referenced
this pull request
Aug 25, 2026
…eaner one (#3254) A defect in my own #3251, found while using it to measure the fleet. `current_project_only` is meant to answer "what would coverage be without the rows measured outside the project" — the same number as `current`, minus contamination. It summed LINES only, while `current` is coverage.py's `totals.percent_covered`, which with branch coverage enabled is (covered_lines + covered_branches) over (statements + branches). coverage.py reports the statements-only figure separately, as `percent_statements_covered`. So on any repo with branch coverage on, the two numbers were on different bases and the gap between them read as contamination. It did not look like a bug: it produced a plausible number about three points away. Measured on the real Gate payloads: repo current project_only(before) foreign Pension-Data 87.78% 90.96% 0 Inv-Man-Intake 91.02% 93.71% 0 Travel-Plan-Permission 87.93% 90.93% 0 Manager-Database 78.01% 81.99% 0 Zero foreign rows in all four, and still a three-point "improvement" on offer. THE INVARIANT THAT CATCHES IT is cheap and is now a test: with no foreign rows, project-only MUST equal current. Verified against the real stranske/Pension-Data payload — 87.78% vs 87.78%, exact. Not affected: the Trend_Model_Project figures quoted in #3251 (83.33% contaminated -> 90.79% project-only, 95 foreign rows). That repo has branch coverage OFF, so both formulas coincide there — which is exactly why the defect survived the change that introduced it. One repo's data cannot exercise a branch-vs-statement distinction it does not have. Break -> revert: restoring the lines-only sum fails both the invariant test and the basis test; revert byte-identical, 28 pass. tests/tools/test_coverage_trend.py 25 -> 28. Co-authored-by: Codex Automation <codex-automation@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
6 tasks
Merged
8 tasks
stranske
added a commit
to stranske/Manager-Database
that referenced
this pull request
Aug 26, 2026
…or (#1598) Coverage was already MEASURED on this Gate (`coverage: true`); only the reporting leg was off, because `enable-soft-gate` defaults to false in reusable-10-ci-python.yml. So every Gate run computed a coverage number and then threw it away: no coverage-trend artifact, no hotspot table, and nothing for Maint Coverage Guard to read. Report-only by construction. The soft gate runs coverage_trend.py with `--soft`, which always exits 0, so this cannot fail a PR or block a merge. Measured 78.01% from this repo's own Gate payload on 2026-08-25, against the floor of 75 already configured here, so the first reports pass and this is not a latent red. Sequenced deliberately AFTER stranske/Workflows#3251/#3253/#3254, which repaired the reporter itself: before those, an absent baseline rendered as 0.00% and every run showed a large fake improvement. Turning reporting on beforehand would have manufactured exactly that here. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
to stranske/Fine-Art-Archive
that referenced
this pull request
Aug 26, 2026
The Gate carried `coverage: false` and `enable-soft-gate: false` while ci.yml has had `coverage: true` all along. `git log -S` traces the Gate's `false` to the initial library-port commit that created the workflow, so it is scaffolding default rather than a considered trade-off -- unlike stranske/Counter_Risk's, which carries an explicit "PR Gate is optimized for fast feedback. Full coverage enforcement runs on main" rationale and is being left alone. The capability is already proven here: this repo's ci.yml payload measures 81.45% over 8,639 statements, so turning it on for the Gate exposes a number that already exists. Report-only: coverage_trend.py runs with --soft and always exits 0, so this cannot fail a PR or block a merge. NO FLOOR IS SET OR CORRECTED HERE, deliberately. The Gate has no `coverage-min`, so it uses the reusable workflow's default of 70; ci.yml carries 25, which against a measured 81.45% sits 56 points below reality and cannot fail. That is drift rather than a judgement about this repo, and it is exactly the shape this programme keeps finding -- but setting a floor is a GATE change, and shipping one in the same PR that first turns reporting on would create a red before anyone has seen a single green report. Both belong with the baseline work, set at measured current and ratcheting up. Sequenced AFTER the repaired reporter reached this repo -- verified main's tools/coverage_trend.py carries BASELINE_KEYS and _resolve_baseline (stranske/Workflows#3251, #3254). Before that an absent baseline rendered as 0.00% and every run showed a large fake improvement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
to stranske/Fine-Art-Archive
that referenced
this pull request
Aug 26, 2026
#609) The Gate carried `coverage: false` and `enable-soft-gate: false` while ci.yml has had `coverage: true` all along. `git log -S` traces the Gate's `false` to the initial library-port commit that created the workflow, so it is scaffolding default rather than a considered trade-off -- unlike stranske/Counter_Risk's, which carries an explicit "PR Gate is optimized for fast feedback. Full coverage enforcement runs on main" rationale and is being left alone. The capability is already proven here: this repo's ci.yml payload measures 81.45% over 8,639 statements, so turning it on for the Gate exposes a number that already exists. Report-only: coverage_trend.py runs with --soft and always exits 0, so this cannot fail a PR or block a merge. NO FLOOR IS SET OR CORRECTED HERE, deliberately. The Gate has no `coverage-min`, so it uses the reusable workflow's default of 70; ci.yml carries 25, which against a measured 81.45% sits 56 points below reality and cannot fail. That is drift rather than a judgement about this repo, and it is exactly the shape this programme keeps finding -- but setting a floor is a GATE change, and shipping one in the same PR that first turns reporting on would create a red before anyone has seen a single green report. Both belong with the baseline work, set at measured current and ratcheting up. Sequenced AFTER the repaired reporter reached this repo -- verified main's tools/coverage_trend.py carries BASELINE_KEYS and _resolve_baseline (stranske/Workflows#3251, #3254). Before that an absent baseline rendered as 0.00% and every run showed a large fake improvement. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
to stranske/learning-management-system
that referenced
this pull request
Aug 26, 2026
#556) The Gate carried `coverage: false` and `enable-soft-gate: false` while ci.yml has had `coverage: true` all along. That split was never a decision: `git log -S` traces the Gate's `false` to the M0-001 scaffolding commit that created the workflow, so it is a template default that nobody revisited -- not a considered trade-off like stranske/Counter_Risk's, which carries an explicit "PR Gate is optimized for fast feedback" rationale. The capability is already proven here: this repo's ci.yml payload measures 87.48% over 10,805 statements, so turning it on for the Gate exposes a number that already exists rather than asking for new work. Report-only. coverage_trend.py runs with --soft and always exits 0, so this cannot fail a PR or block a merge; it adds the trend artifact, the hotspot table, and something for Maint Coverage Guard to read. coverage-min set to 80 to match ci.yml. Without it the Gate silently falls back to the reusable workflow's default of 70, so the two surfaces would report different verdicts about the same repository. At 87.48% this reports a pass rather than arriving red. Sequenced AFTER the repaired reporter reached this repo -- verified that main's tools/coverage_trend.py carries BASELINE_KEYS and _resolve_baseline (stranske/Workflows#3251, #3254). Before that, an absent baseline rendered as 0.00% and every run displayed a large fake improvement. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
to stranske/Portable-Alpha-Extension-Model
that referenced
this pull request
Aug 26, 2026
Without config/coverage-baseline.json, tools/coverage_trend.py reports baseline_status=absent and computes NO delta, and Maint Coverage Guard has nothing to compare against -- so a coverage regression here is currently undetectable. Set at MEASURED CURRENT (86.25%, Gate payload; reporting enabled by the held PR #2243), floored to 86, with warn_drop 1.0 absorbing run-to-run jitter. Deliberately not an aspiration: a baseline above reality is red on arrival and gets switched off, and one far below it can never fail. Both failure directions are live in this fleet right now -- Fine-Art-Archive carries 25 against a real 81.45%, Workflows carries 85 against a real 76.72%. Keyed `line`: coverage_trend.py and coverage_guard.py both accept `line` or `coverage`, with `line` taking precedence. That mismatch was itself a live defect until stranske/Workflows#3251. This is a ratchet, not a target: raise it as coverage rises. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
to stranske/Manager-Database
that referenced
this pull request
Aug 26, 2026
Without config/coverage-baseline.json, tools/coverage_trend.py reports baseline_status=absent and computes NO delta, and Maint Coverage Guard has nothing to compare against -- so a coverage regression here is currently undetectable. Set at MEASURED CURRENT (78.01%, Gate payload; reporting enabled in #1598), floored to 78, with warn_drop 1.0 absorbing run-to-run jitter. Deliberately not an aspiration: a baseline above reality is red on arrival and gets switched off, and one far below it can never fail. Both failure directions are live in this fleet right now -- Fine-Art-Archive carries 25 against a real 81.45%, Workflows carries 85 against a real 76.72%. Keyed `line`: coverage_trend.py and coverage_guard.py both accept `line` or `coverage`, with `line` taking precedence. That mismatch was itself a live defect until stranske/Workflows#3251. This is a ratchet, not a target: raise it as coverage rises. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
to stranske/Inv-Man-Intake
that referenced
this pull request
Aug 26, 2026
Without config/coverage-baseline.json, tools/coverage_trend.py reports baseline_status=absent and computes NO delta, and Maint Coverage Guard has nothing to compare against -- so a coverage regression here is currently undetectable. Set at MEASURED CURRENT (91.02%, Gate payload), floored to 91, with warn_drop 1.0 absorbing run-to-run jitter. Deliberately not an aspiration: a baseline above reality is red on arrival and gets switched off, and one far below it can never fail. Both failure directions are live in this fleet right now -- Fine-Art-Archive carries 25 against a real 81.45%, Workflows carries 85 against a real 76.72%. Keyed `line`: coverage_trend.py and coverage_guard.py both accept `line` or `coverage`, with `line` taking precedence. That mismatch was itself a live defect until stranske/Workflows#3251. This is a ratchet, not a target: raise it as coverage rises. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
to stranske/Pension-Data
that referenced
this pull request
Aug 26, 2026
Without config/coverage-baseline.json, tools/coverage_trend.py reports baseline_status=absent and computes NO delta, and Maint Coverage Guard has nothing to compare against -- so a coverage regression here is currently undetectable. Set at MEASURED CURRENT (87.78%, Gate payload), floored to 87, with warn_drop 1.0 absorbing run-to-run jitter. Deliberately not an aspiration: a baseline above reality is red on arrival and gets switched off, and one far below it can never fail. Both failure directions are live in this fleet right now -- Fine-Art-Archive carries 25 against a real 81.45%, Workflows carries 85 against a real 76.72%. Keyed `line`: coverage_trend.py and coverage_guard.py both accept `line` or `coverage`, with `line` taking precedence. That mismatch was itself a live defect until stranske/Workflows#3251. This is a ratchet, not a target: raise it as coverage rises. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
to stranske/trip-planner
that referenced
this pull request
Aug 26, 2026
Without config/coverage-baseline.json, tools/coverage_trend.py reports baseline_status=absent and computes NO delta, and Maint Coverage Guard has nothing to compare against -- so a coverage regression here is currently undetectable. Set at MEASURED CURRENT (90.40%, Gate payload), floored to 90, with warn_drop 1.0 absorbing run-to-run jitter. Deliberately not an aspiration: a baseline above reality is red on arrival and gets switched off, and one far below it can never fail. Both failure directions are live in this fleet right now -- Fine-Art-Archive carries 25 against a real 81.45%, Workflows carries 85 against a real 76.72%. Keyed `line`: coverage_trend.py and coverage_guard.py both accept `line` or `coverage`, with `line` taking precedence. That mismatch was itself a live defect until stranske/Workflows#3251. This is a ratchet, not a target: raise it as coverage rises. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
to stranske/Fine-Art-Archive
that referenced
this pull request
Aug 26, 2026
Without config/coverage-baseline.json, tools/coverage_trend.py reports baseline_status=absent and computes NO delta, and Maint Coverage Guard has nothing to compare against -- so a coverage regression here is currently undetectable. Set at MEASURED CURRENT (81.45%, ci.yml payload; Gate reporting enabled in #609), floored to 81, with warn_drop 1.0 absorbing run-to-run jitter. Deliberately not an aspiration: a baseline above reality is red on arrival and gets switched off, and one far below it can never fail. Both failure directions are live in this fleet right now -- Fine-Art-Archive carries 25 against a real 81.45%, Workflows carries 85 against a real 76.72%. Keyed `line`: coverage_trend.py and coverage_guard.py both accept `line` or `coverage`, with `line` taking precedence. That mismatch was itself a live defect until stranske/Workflows#3251. This is a ratchet, not a target: raise it as coverage rises. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
to stranske/Travel-Plan-Permission
that referenced
this pull request
Aug 26, 2026
Without config/coverage-baseline.json, tools/coverage_trend.py reports baseline_status=absent and computes NO delta, and Maint Coverage Guard has nothing to compare against -- so a coverage regression here is currently undetectable. Set at MEASURED CURRENT (87.93%, Gate payload; monorepo scope covers src/ only — see notes), floored to 87, with warn_drop 1.0 absorbing run-to-run jitter. Deliberately not an aspiration: a baseline above reality is red on arrival and gets switched off, and one far below it can never fail. Both failure directions are live in this fleet right now -- Fine-Art-Archive carries 25 against a real 81.45%, Workflows carries 85 against a real 76.72%. Keyed `line`: coverage_trend.py and coverage_guard.py both accept `line` or `coverage`, with `line` taking precedence. That mismatch was itself a live defect until stranske/Workflows#3251. This is a ratchet, not a target: raise it as coverage rises. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 26, 2026
Merged
Merged
Merged
stranske
added a commit
to stranske/Manager-Database
that referenced
this pull request
Aug 26, 2026
) Without config/coverage-baseline.json, tools/coverage_trend.py reports baseline_status=absent and computes NO delta, and Maint Coverage Guard has nothing to compare against -- so a coverage regression here is currently undetectable. Set at MEASURED CURRENT (78.01%, Gate payload; reporting enabled in #1598), floored to 78, with warn_drop 1.0 absorbing run-to-run jitter. Deliberately not an aspiration: a baseline above reality is red on arrival and gets switched off, and one far below it can never fail. Both failure directions are live in this fleet right now -- Fine-Art-Archive carries 25 against a real 81.45%, Workflows carries 85 against a real 76.72%. Keyed `line`: coverage_trend.py and coverage_guard.py both accept `line` or `coverage`, with `line` taking precedence. That mismatch was itself a live defect until stranske/Workflows#3251. This is a ratchet, not a target: raise it as coverage rises. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
to stranske/Inv-Man-Intake
that referenced
this pull request
Aug 26, 2026
Without config/coverage-baseline.json, tools/coverage_trend.py reports baseline_status=absent and computes NO delta, and Maint Coverage Guard has nothing to compare against -- so a coverage regression here is currently undetectable. Set at MEASURED CURRENT (91.02%, Gate payload), floored to 91, with warn_drop 1.0 absorbing run-to-run jitter. Deliberately not an aspiration: a baseline above reality is red on arrival and gets switched off, and one far below it can never fail. Both failure directions are live in this fleet right now -- Fine-Art-Archive carries 25 against a real 81.45%, Workflows carries 85 against a real 76.72%. Keyed `line`: coverage_trend.py and coverage_guard.py both accept `line` or `coverage`, with `line` taking precedence. That mismatch was itself a live defect until stranske/Workflows#3251. This is a ratchet, not a target: raise it as coverage rises. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
to stranske/Pension-Data
that referenced
this pull request
Aug 26, 2026
Without config/coverage-baseline.json, tools/coverage_trend.py reports baseline_status=absent and computes NO delta, and Maint Coverage Guard has nothing to compare against -- so a coverage regression here is currently undetectable. Set at MEASURED CURRENT (87.78%, Gate payload), floored to 87, with warn_drop 1.0 absorbing run-to-run jitter. Deliberately not an aspiration: a baseline above reality is red on arrival and gets switched off, and one far below it can never fail. Both failure directions are live in this fleet right now -- Fine-Art-Archive carries 25 against a real 81.45%, Workflows carries 85 against a real 76.72%. Keyed `line`: coverage_trend.py and coverage_guard.py both accept `line` or `coverage`, with `line` taking precedence. That mismatch was itself a live defect until stranske/Workflows#3251. This is a ratchet, not a target: raise it as coverage rises. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
to stranske/Fine-Art-Archive
that referenced
this pull request
Aug 26, 2026
Without config/coverage-baseline.json, tools/coverage_trend.py reports baseline_status=absent and computes NO delta, and Maint Coverage Guard has nothing to compare against -- so a coverage regression here is currently undetectable. Set at MEASURED CURRENT (81.45%, ci.yml payload; Gate reporting enabled in #609), floored to 81, with warn_drop 1.0 absorbing run-to-run jitter. Deliberately not an aspiration: a baseline above reality is red on arrival and gets switched off, and one far below it can never fail. Both failure directions are live in this fleet right now -- Fine-Art-Archive carries 25 against a real 81.45%, Workflows carries 85 against a real 76.72%. Keyed `line`: coverage_trend.py and coverage_guard.py both accept `line` or `coverage`, with `line` taking precedence. That mismatch was itself a live defect until stranske/Workflows#3251. This is a ratchet, not a target: raise it as coverage rises. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
to stranske/Travel-Plan-Permission
that referenced
this pull request
Aug 26, 2026
) Without config/coverage-baseline.json, tools/coverage_trend.py reports baseline_status=absent and computes NO delta, and Maint Coverage Guard has nothing to compare against -- so a coverage regression here is currently undetectable. Set at MEASURED CURRENT (87.93%, Gate payload; monorepo scope covers src/ only — see notes), floored to 87, with warn_drop 1.0 absorbing run-to-run jitter. Deliberately not an aspiration: a baseline above reality is red on arrival and gets switched off, and one far below it can never fail. Both failure directions are live in this fleet right now -- Fine-Art-Archive carries 25 against a real 81.45%, Workflows carries 85 against a real 76.72%. Keyed `line`: coverage_trend.py and coverage_guard.py both accept `line` or `coverage`, with `line` taking precedence. That mismatch was itself a live defect until stranske/Workflows#3251. This is a ratchet, not a target: raise it as coverage rises. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
to stranske/trip-planner
that referenced
this pull request
Aug 26, 2026
) Without config/coverage-baseline.json, tools/coverage_trend.py reports baseline_status=absent and computes NO delta, and Maint Coverage Guard has nothing to compare against -- so a coverage regression here is currently undetectable. Set at MEASURED CURRENT (90.40%, Gate payload), floored to 90, with warn_drop 1.0 absorbing run-to-run jitter. Deliberately not an aspiration: a baseline above reality is red on arrival and gets switched off, and one far below it can never fail. Both failure directions are live in this fleet right now -- Fine-Art-Archive carries 25 against a real 81.45%, Workflows carries 85 against a real 76.72%. Keyed `line`: coverage_trend.py and coverage_guard.py both accept `line` or `coverage`, with `line` taking precedence. That mismatch was itself a live defect until stranske/Workflows#3251. This is a ratchet, not a target: raise it as coverage rises. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
to stranske/Portable-Alpha-Extension-Model
that referenced
this pull request
Aug 26, 2026
) Without config/coverage-baseline.json, tools/coverage_trend.py reports baseline_status=absent and computes NO delta, and Maint Coverage Guard has nothing to compare against -- so a coverage regression here is currently undetectable. Set at MEASURED CURRENT (86.25%, Gate payload; reporting enabled by the held PR #2243), floored to 86, with warn_drop 1.0 absorbing run-to-run jitter. Deliberately not an aspiration: a baseline above reality is red on arrival and gets switched off, and one far below it can never fail. Both failure directions are live in this fleet right now -- Fine-Art-Archive carries 25 against a real 81.45%, Workflows carries 85 against a real 76.72%. Keyed `line`: coverage_trend.py and coverage_guard.py both accept `line` or `coverage`, with `line` taking precedence. That mismatch was itself a live defect until stranske/Workflows#3251. This is a ratchet, not a target: raise it as coverage rises. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
to stranske/Portable-Alpha-Extension-Model
that referenced
this pull request
Aug 26, 2026
…et is already met Only the REPORTING leg was off here: `coverage` defaults to true upstream, but `enable-soft-gate` defaults to false, so the Gate computed a coverage number on every run and discarded it -- no trend artifact, no hotspot table, and nothing for Maint Coverage Guard to read. Report-only by construction: coverage_trend.py runs with --soft and always exits 0, so this cannot fail a PR or block a merge. Measured 86.25% from this repo's own coverage payload on 2026-08-25. The floor here is 60 with the comment "Target: 85% - tracking issue to be created" -- that target is ALREADY MET, and has been, invisibly, for as long as reporting was off. A number nobody publishes cannot show you that you already succeeded. The floor stays at 60 in this change. Raising it is a gate change and belongs with the baseline work, not with turning reporting on; doing both at once would mean shipping a new red without ever having seen a green report. Sequenced deliberately AFTER stranske/Workflows#3251/#3253/#3254, which repaired the reporter: before those an absent baseline rendered as 0.00% and every run showed a large fake improvement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
to stranske/Portable-Alpha-Extension-Model
that referenced
this pull request
Aug 26, 2026
…et is already met Only the REPORTING leg was off here: `coverage` defaults to true upstream, but `enable-soft-gate` defaults to false, so the Gate computed a coverage number on every run and discarded it -- no trend artifact, no hotspot table, and nothing for Maint Coverage Guard to read. Report-only by construction: coverage_trend.py runs with --soft and always exits 0, so this cannot fail a PR or block a merge. Measured 86.25% from this repo's own coverage payload on 2026-08-25. The floor here is 60 with the comment "Target: 85% - tracking issue to be created" -- that target is ALREADY MET, and has been, invisibly, for as long as reporting was off. A number nobody publishes cannot show you that you already succeeded. The floor stays at 60 in this change. Raising it is a gate change and belongs with the baseline work, not with turning reporting on; doing both at once would mean shipping a new red without ever having seen a green report. Sequenced deliberately AFTER stranske/Workflows#3251/#3253/#3254, which repaired the reporter: before those an absent baseline rendered as 0.00% and every run showed a large fake improvement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
to stranske/Portable-Alpha-Extension-Model
that referenced
this pull request
Aug 26, 2026
…2243) * ci(coverage): turn on the reporting leg, and record that the 85% target is already met Only the REPORTING leg was off here: `coverage` defaults to true upstream, but `enable-soft-gate` defaults to false, so the Gate computed a coverage number on every run and discarded it -- no trend artifact, no hotspot table, and nothing for Maint Coverage Guard to read. Report-only by construction: coverage_trend.py runs with --soft and always exits 0, so this cannot fail a PR or block a merge. Measured 86.25% from this repo's own coverage payload on 2026-08-25. The floor here is 60 with the comment "Target: 85% - tracking issue to be created" -- that target is ALREADY MET, and has been, invisibly, for as long as reporting was off. A number nobody publishes cannot show you that you already succeeded. The floor stays at 60 in this change. Raising it is a gate change and belongs with the baseline work, not with turning reporting on; doing both at once would mean shipping a new red without ever having seen a green report. Sequenced deliberately AFTER stranske/Workflows#3251/#3253/#3254, which repaired the reporter: before those an absent baseline rendered as 0.00% and every run showed a large fake improvement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * ci(coverage): raise the floor 60 -> 86, now that the baseline is set Folded into this PR rather than opened separately, because a second PR would touch the same file and conflict with this one. The Gate carried `coverage-min: "60" # Target: 85% - tracking issue to be created` while measured coverage is 86.25%. Two things wrong with that line at once: the target it calls untracked was ALREADY MET, and the floor sat 26 points below reality, so it reported a pass on every input and could not have failed if coverage fell by a quarter. A number nobody publishes cannot tell you that you already succeeded. Sequenced deliberately, and the precondition is now met. The first commit on this branch said the floor was being left at 60 because raising a floor is a GATE change that belongs with the baseline work, not with the PR that first turns reporting on. #2245 has since set config/coverage-baseline.json at measured current, so that condition is satisfied and the floor moves here. 86 rather than 86.25 leaves the jitter tolerance where it belongs -- in the guard's warn_drop of 1.0 -- and matches the baseline file, so the soft gate's Status line and Maint Coverage Guard's comparison agree about this repo. The stale comment is removed with it. Leaving "tracking issue to be created" beside a met target would keep pointing a reader at work that does not exist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
to stranske/Orchestrator
that referenced
this pull request
Aug 30, 2026
…ported as 34.11% (#159) * ci: measure this repo's REAL coverage, which is 82.1% and has been reported as 34.11% pr-00-gate.yml measures coverage with pytest alone. Measured 2026-08-29 by running the command this job adds: 82.1%, over 44,740 statements with 8,026 missed, combined from 94 instrumented processes. The Gate reports 34.11%. Forty-eight points. WHY. 79 of 102 modules are exercised by a `--selftest` entry point rather than a pytest file -- the original design, from the initial commit -- and a selftest runs as a SUBPROCESS that pytest-cov cannot see. So the Gate has been measuring the instrument's blind spot and calling it a score: the same defect stranske/Workflows#3251 drained out of the fleet reporter, where a number nobody could measure rendered as a number that was measured. It mattered beyond cosmetics. Every other repo in the fleet now carries a config/coverage-baseline.json set at measured current; this one was deliberately SKIPPED, because a baseline against the wrong instrument would bake the blind spot in permanently. A SEPARATE JOB, NOT A FLAG ON THE VERDICT: * `--coverage` wraps ~94 child processes in `coverage run --parallel-mode`, so it is materially slower than the verdict, which is on the critical path for every PR. Measured in CI, the verify job is ~1m30s; this runs beside it rather than in front of it. (My first instinct was that this was too expensive to run at all -- from timing it locally at 7 minutes. That was the Dropbox tax CLAUDE.md warns about, not the real cost. CI is the number that decides.) * `--coverage` deliberately never touches the exit code, and verify.py pins that in a test, precisely so measuring cannot change a verdict. Its own job keeps that true by construction. continue-on-error, because a coverage MEASUREMENT must never redden a green verdict -- but the failure stays visible in the job list rather than being swallowed. Both summary branches self-describe: verify.py emits `coverage: NO DATA` and `coverage: FAILED` as well as the combined report, so an absent measurement prints as absent rather than as a low number. NOT YET WIRED TO THE GUARD, and the summary says so rather than implying enforcement. Maint Coverage Guard reads pr-00-gate.yml by a hardcoded workflowId and cannot see this job, so this repo still has no baseline. Publishing the honest figure is step one; teaching the guard to read it is a separate change -- the same hardcoded-workflowId limitation that makes stranske/Counter_Risk invisible to it. Verified: verify.py --coverage exits 0 and produces the report above. `.coverage` remains gitignored (#119), so uploading it as an artifact cannot reintroduce the tracked binary. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * ci: cite the CI figure, not the local one — they differ by 1.8 points and CI is authoritative The first version of this job's comment quoted 82.1%, measured on the owner's machine. This runner reports 80.3%: 44,741 statements, 8,828 missed, same 94 instrumented processes. The difference is real, not noise. 802 statements execute only where this instance's prerequisites exist -- agent CLIs, ~/.codex, a populated ledger -- and are missed on a bare runner. They are exactly the paths env_prereq guards, so the gap is the skip surface showing up in coverage rather than a measurement artefact. CLAUDE.md already says to take coverage from CI rather than a local run. Quoting the local number in the job that exists to establish the honest one would have overstated the repo by nearly two points, in a comment future readers would treat as the record. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(ci): address CodeRabbit review on combined-coverage job Pin checkout/setup-python/upload-artifact to verified SHAs, use shell: bash on the coverage pipeline so verify.py exit status survives tee, and publish the TOTAL row in the step summary. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Tim Stranske <tim@stranskemo.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Four defects in the fleet coverage measurement. Three share one shape — a single sentinel standing for two different meanings, where only one of them is good news.
The visible symptom, on every Gate run in
stranske/Trend_Model_Project:That repo's own
config/coverage-baseline.jsonsays85.0. Its real coverage of 83.32% is a breach, reported as an 83-point improvement.The four
1 — Baseline key mismatch.
tools/coverage_trend.pyread onlypayload.get("coverage").tools/coverage_guard.py, reading the same file, has always readpayload.get("line", payload.get("coverage")). Trend_Model_Project writesline; Workflows and LMS writecoverage. Two scripts disagreed about one config and neither said so.2 — An absent baseline rendered as zero. Missing, unreadable and differently-keyed all collapsed to
0.0, sodelta = current − 0printed as a large improvement every run and the comparison could never fail. Ten of thirteen lane repos have no baseline file at all._resolve_baselinenow returnsNoneplus a status naming which cause —unset/absent/unreadable/no_recognised_key— because each is a different fix.3 — Foreign rows silently averaged in. A test copying the source tree into a tmpdir makes coverage.py record each module twice: once real, once as a barely-executed duplicate that sorts to the top of every hotspot table. 95 rows on Trend_Model_Project were
/tmp/pytest-of-runner/.../workspace/src/…— paths that don't exist in the repo, so the only actionable output pointed at files nobody could open. Now partitioned and reported.currentis left exactly as coverage.py computed it (so the record still agrees withcoverage.xmland the delta job);current_project_onlysits beside it and the gap is the size of the problem.4 — Wrong baseline path for monorepo consumers. The workflow read
${GITHUB_WORKSPACE}/config/coverage-baseline.json, butPROJECT_ROOTisworkspace/<working-directory>when a consumer sets one — andstranske/Travel-Plan-Permissiondoes. Now PROJECT_ROOT first with workspace fallback, so everyworking-directory: "."consumer resolves to the identical file as before.Measured on the real artifact
Trend_Model_Project Gate run
32794743236, actualcoverage.json, actual baseline config:0.00%85.00%+83.32%✅−1.67%— a real regression/tmp/…Deliberate break → revert
coverage-only key read0.0for an absent baselineEach failed only its own cases; the revert was byte-identical and all 25 pass.
"Every row is foreign" gets its own message — that's a wrong
--project-root, not contamination, and pointing the reader atomitwould send them to fix the wrong file.Scope / compatibility
--project-rootdefaults to cwd and filtering is opt-in, so no existing caller changes behaviour._get_hotspotsreturns a 3-tuple now; it has no callers outside this module (coverage_guard._get_hotspotsis a separate function, untouched).coverage-trend.envgainsbaseline_statusandforeign_file_count;baseline=/delta=are now empty rather than0.00when unknown. No workflow step reads that file today.coverage_guard.pyloads its own baseline from config, so anullin the trend record doesn't affect it.docs/ci/WORKFLOWS.mdCoverage Guardrails updated per the repo's contract rule.tools/coverage_trend.pyis sync-managed (.github/sync-manifest.yml), so this reaches consumers on the next sync. There is notemplates/consumer-repo/tools/coverage_trend.pysecond copy.Validation
tests/tools/test_coverage_trend.py17 → 25pytest tests/tools/ tests/workflows/→ 1368 passed, 4 skipped (pre-existing skips)black --line-length 100 --target-version py312andruff checkclean; workflow YAML parses🤖 Generated with Claude Code