Skip to content

fix(ci): let coverage guard use payload-only Gate runs - #115

Closed
stranske wants to merge 4 commits into
mainfrom
codex/repair-coverage-guard-artifact-contract
Closed

fix(ci): let coverage guard use payload-only Gate runs#115
stranske wants to merge 4 commits into
mainfrom
codex/repair-coverage-guard-artifact-contract

Conversation

@stranske

@stranske stranske commented Aug 25, 2026

Copy link
Copy Markdown
Owner

Summary

The coverage guard incorrectly required optional trend artifacts before it could inspect a Gate payload. Successful Gate runs publish gate-coverage; trend/history downloads are already best-effort and the guard already handles their absence.

  • require only gate-coverage during candidate discovery
  • retain optional trend/history downloads
  • add a regression test for the required-versus-optional artifact contract

Validation

  • python3 -m pytest tests/test_coverage_guard_config.py
  • parsed .github/workflows/maint-coverage-guard.yml and .github/workflows/pr-00-gate.yml with PyYAML
  • git diff --check

Fixes the failed verification run 32790552224 after #110.

@coderabbitai

coderabbitai Bot commented Aug 25, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 8 minutes.

View limit details

Limit details: You’ve used the included review currently available. Your 69 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: ca16ba4a-75fa-48be-820b-dbec68a02cb2

📥 Commits

Reviewing files that changed from the base of the PR and between 529bf00 and 9e729dd.

📒 Files selected for processing (4)
  • .github/workflows/maint-coverage-guard.yml
  • .gitignore
  • tests/test_coverage_guard_config.py
  • tests/test_repo_artifact_hygiene.py

Comment @coderabbitai help to get the list of available commands.

@stranske
stranske deployed to agent-high-privilege August 25, 2026 00:06 — with GitHub Actions Active
@agents-workflows-bot

agents-workflows-bot Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Workflow source detected

PR #115 now has valid workflow source context (origin=local_request).

No linked GitHub issue is required for this PR.

@stranske-keepalive

stranske-keepalive Bot commented Aug 25, 2026

Copy link
Copy Markdown

Automated Status Summary

Head SHA: cce8c0e
Latest Runs: ⏳ pending — Gate
Required: core tests (3.12): ⏳ pending, core tests (3.13): ⏳ pending, docker smoke: ⏳ pending, gate: ⏳ pending

Workflow / Job Result Logs
(no jobs reported) ⏳ pending

Coverage Overview

  • Coverage history entries: 1

Coverage Trend

Metric Value
Current 34.25%
Baseline 0.00%
Delta +34.25%
Minimum 70.00%
Status ❌ Below minimum

Top Coverage Hotspots (lowest coverage)

File Coverage Missing
src/capability_effectiveness.py 0.0% 154
src/capability_firing_monitor.py 0.0% 192
src/capability_matcher_proposals.py 0.0% 111
src/capability_opportunity.py 0.0% 143
src/capability_propensity.py 0.0% 1669
src/ccusage_reconcile.py 0.0% 286
src/codemod_lane.py 0.0% 351
src/evidence_acquisition.py 0.0% 103
src/exploration_collection.py 0.0% 331
src/feature_scan.py 0.0% 118
src/frontend_verify.py 0.0% 255
src/improvement_log.py 0.0% 248
src/issue_readiness.py 0.0% 507
src/keepalive_evidence.py 0.0% 378
src/keepalive_supervisor.py 0.0% 322

Low Coverage Files (<50.0%)

File Coverage Missing
src/capability_effectiveness.py 0.0% 154
src/capability_firing_monitor.py 0.0% 192
src/capability_matcher_proposals.py 0.0% 111
src/capability_opportunity.py 0.0% 143
src/capability_propensity.py 0.0% 1669
src/ccusage_reconcile.py 0.0% 286
src/codemod_lane.py 0.0% 351
src/evidence_acquisition.py 0.0% 103
src/exploration_collection.py 0.0% 331
src/feature_scan.py 0.0% 118
src/frontend_verify.py 0.0% 255
src/improvement_log.py 0.0% 248
src/issue_readiness.py 0.0% 507
src/keepalive_evidence.py 0.0% 378
src/keepalive_supervisor.py 0.0% 322

Updated automatically; will refresh on subsequent CI/Docker completions.


Keepalive checklist

Scope

No scope information available

Tasks

  • No tasks defined

Acceptance criteria

  • No acceptance criteria defined

@stranske

stranske commented Aug 25, 2026

Copy link
Copy Markdown
Owner Author

Runner dispatch state for autofix on PR #115. Do not edit.

@stranske

stranske commented Aug 25, 2026

Copy link
Copy Markdown
Owner Author

Runner dispatch state for codex on PR #115. Do not edit.

@stranske-keepalive
stranske-keepalive Bot deployed to agent-high-privilege August 25, 2026 00:13 Active
@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Workflow state fingerprint for Agents Gate Followups. Do not edit.

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Workflow state fingerprint for Keepalive Loop Reporter. Do not edit.

@agents-workflows-bot
agents-workflows-bot Bot deployed to agent-high-privilege August 25, 2026 00:15 Active
@stranske-keepalive
stranske-keepalive Bot deployed to agent-high-privilege August 25, 2026 00:18 Active
@github-actions github-actions Bot added autofix Let bots format/lint automatically autofix:patch Autofix patch available labels Aug 25, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Autofix updated these files:

  • tests/test_coverage_guard_config.py

@stranske
stranske force-pushed the codex/repair-coverage-guard-artifact-contract branch from bba1b31 to 9e729dd Compare August 25, 2026 00:27
@stranske
stranske deployed to agent-high-privilege August 25, 2026 00:27 — with GitHub Actions Active

Copy link
Copy Markdown
Owner Author

Steward handoff: the exact head 9e729dda6bfc43aeb629c9c9f7b8a322957313a7 has completed green Gate/CI checks and zero active non-outdated review threads, but GitHub now reports mergeable=CONFLICTING, mergeStateStatus=DIRTY.

This is autofix-routed issue-workloop scope, not orphan-steward implementation work. Receiving worker: imi-merge-verify-closer (ACTIVE hourly). Concrete next action: rebase/resolve against current main, rerun the required Gate evidence, then re-check the exact head and active review threads before any merge. Do not merge the conflicting head.

stranske added a commit to stranske/Workflows that referenced this pull request Aug 25, 2026
…pendent reasons, all silent (#3252)

* fix(coverage-guard): the monitor has never once executed — three independent reasons, all silent

The fleet's nightly coverage monitor has produced exactly ONE baseline-breach issue in its
history (stranske/Trend_Model_Project #2933, 2025-10-23). That is not because coverage has
been healthy. It is because the guard has never run to completion anywhere.

1. A PAIRED LITERAL THAT DISAGREED. pr-00-gate.yml uploads the payload under the ARTIFACT
   name `gate-coverage.json` — the FILE name, used as the artifact name — while this guard
   required `gate-coverage`. So it probed ten successful Gate runs, found two of three
   required artifacts every time, and reported no usable run at all. Verified against live
   artifact listings: no consumer publishes a bare `gate-coverage` except Orchestrator,
   which renamed its own producer locally in its #110.

   Fixed HERE rather than at the producer, deliberately: pr-00-gate.yml is distributed
   create-only, so renaming it reaches NEW repos only, while this file is sync-managed and
   reaches all thirteen. Discovery now accepts either name and EMITS the one it found, so
   the downloader learns the name from the producer's actual artifact list. A discovered
   name cannot drift from the name that exists.

2. OPTIONAL ARTIFACTS WERE REQUIRED. Trend and history were in the required set, but their
   download steps already tolerate absence and the guard already omits their CLI arguments
   when missing. Requiring them made any repo that has not enabled the soft gate
   permanently unusable — a gate demanding an artifact its own subject need not emit. They
   are now optional, gated on discovery having resolved a name, and the artifact-cache
   downloads for the two of them stop failing fast.

3. A SKIP LOOKED EXACTLY LIKE A PASS. `guard` is gated on the rate-limit job, so when quota
   is low the job is skipped and THE WORKFLOW IS GREEN with nothing checked. stranske/Workflows'
   three most recent runs all read `success` at the workflow level and
   `coverage baseline monitor: skipped` at the job level. That is how a monitor that has
   never executed came to look healthy for months, and it is why "only Workflows succeeds"
   was the wrong reading of the fleet.

   A new always-run `report` job states which happened, in the run's own summary and as a
   notice. It does NOT fail on a deferral: the quota gate is legitimate, and turning a
   deferral into a red would train everyone to ignore the red. It makes the state legible,
   which is the actual defect.

Both copies patched — `.github/workflows/` (floating refs, artifact-cache) and
`templates/consumer-repo/.github/workflows/` (pinned SHAs, download-artifact). They are
deliberate variants, not mirrors, so the same logic is applied to each.

This SUPERSEDES stranske/Orchestrator#115, which relaxes the same required-artifact set but
in Orchestrator's local copy of a sync-managed file — the next template sync overwrites it,
and requiring a bare `gate-coverage` would still fail in the other twelve repos.

Validated: actionlint clean on both (SC2129 fixed rather than allowlisted); YAML parses;
1940 passed, 5 skipped (pre-existing) across the coverage/guard/workflow/template/sync suites.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(drift): refresh the coverage-guard fingerprint for the guard repair

Health 74 flagged pair.18 because both surfaces changed. The change is applied IDENTICALLY
to root and consumer, so it introduces no NEW divergence — the underlying divergence (root
uses the local artifact-cache action, consumers mint an App token and use pinned public
download actions) is unchanged, and divergence_reviewed deliberately keeps its 2026-08-23
date per this file's own convention.

Hashes taken from `check_template_drift.py --print-allowlist-template`, not computed by
hand: the checker normalizes before hashing, so a plain sha256 of the file does NOT match
the recorded value — verified against the untouched pr-00-gate pair before trusting it.

Break -> revert: restoring the old main_sha256 fails the check (exit 1); restoring the new
one passes (exit 0). The fingerprint is load-bearing, not decorative.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Codex Automation <codex-automation@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@stranske

Copy link
Copy Markdown
Owner Author

Superseded by stranske/Workflows#3252, now merged.

Two reasons this could not have worked as scoped:

  1. It edits a sync-managed file. maint-coverage-guard.yml is declared in .github/sync-manifest.yml, so the next template sync overwrites this change. Per Workflows CLAUDE.md: "Changes to consumer-facing workflows, prompts, scripts, or docs must be made in the Workflows source, not patched ad hoc in a consumer repo."

  2. Requiring a bare gate-coverage still fails in the other twelve repos. Verified against live artifact listings — no consumer publishes an artifact by that name. Orchestrator has one only because its own fix(ci): the Gate's coverage artifact had the name its only consumer wanted, plus .json #110 renamed the producer locally. So this change would have turned Orchestrator green while leaving the fleet dark, which reads as fixed and is the worst of the available outcomes.

#3252 fixes it upstream in both workflow copies: discovery accepts either gate-coverage or gate-coverage.json and emits the name it found, so the downloader learns it from the producer's actual artifact list rather than repeating a literal. It also makes trend/history genuinely optional (your diagnosis — correct, and kept), and adds an always-run job so a rate-limit skip stops reading as a pass.

That last one is the part neither of us had: the guard job is skipped when quota is low, and a skipped job leaves the workflow green. Workflows' three most recent runs all read success at the workflow level and coverage baseline monitor: skipped at the job level — the guard has never actually executed anywhere in the fleet. One breach issue in ten months was the symptom.

The diagnosis in this PR was right; it just needed to land one level up.

@stranske stranske closed this Aug 25, 2026
@stranske
stranske deleted the codex/repair-coverage-guard-artifact-contract branch August 30, 2026 01:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

autofix:escalated autofix:patch Autofix patch available autofix Let bots format/lint automatically

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant