Skip to content

fix(verify): -rs replaced pytest's -rfE and stopped listing failures - #4

Merged
stranske merged 1 commit into
mainfrom
claude/verify-r-flag
Aug 22, 2026
Merged

fix(verify): -rs replaced pytest's -rfE and stopped listing failures#4
stranske merged 1 commit into
mainfrom
claude/verify-r-flag

Conversation

@stranske

Copy link
Copy Markdown
Owner

Follow-up to #3, found by pointing the new instrument at a real red.

#3 added "print EVERY pytest failure, not the last 12 lines" — and in the same breath passed -rs to get skip reasons. pytest's default is -rfE, and passing -r replaces that set rather than adding to it. So FAILED/ERROR lines left the short summary, and run_pytest — which harvests failures from exactly those lines — found none.

Observed on a tree with one genuine failure:

    - 1 pytest failure(s)/error(s)
    pytest failures (0):

An instrument reporting less, introduced by the change that existed to make it report more. Same defect class this repo is named for, shipped inside its own countermeasure.

-rfEs restores failures and errors alongside skip reasons.

The selftest now asserts on the flag itself rather than on behaviour, because the symptom only appears on a red run: a green suite has no FAILED lines to lose, so no amount of local verification would have caught it. Deliberate break → revert: restoring -rs fails with -rs omits 'f': failures would not be listed; reverted byte-identical.

Verified: 330 passed / 0 failed / 0 of 24 skipped / 81 of 81 selftests / 5 of 5 gates.

🤖 Generated with Claude Code

Found by running the new instrument on a real red, which is the only way this
was ever going to show up.

The previous commit added "print EVERY pytest failure, not the last 12 lines",
and in the same breath passed `-rs` to get skip reasons. pytest's default is
`-rfE`; passing `-r` REPLACES that set rather than adding to it. So FAILED and
ERROR lines left the short summary, and `run_pytest` — which harvests failures
from exactly those lines — found none.

Observed, on a tree with one genuine failure:

    - 1 pytest failure(s)/error(s)
    pytest failures (0):

An instrument reporting less, introduced by the change that existed to make it
report more. Same defect class as the one this repo is named for, shipped inside
its own countermeasure.

`-rfEs` restores failures and errors alongside skip reasons. The selftest now
asserts on the flag itself, because the symptom appears only on a red run: a
green suite has no FAILED lines to lose, so nothing local would ever catch it.
Deliberate break -> revert: putting `-rs` back fails the selftest with "-rs omits
'f': failures would not be listed"; reverted byte-identical (`cmp`).

Verified: 330 passed / 0 failed / 0 of 24 skipped / 81 of 81 selftests / 5 of 5
gates.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Aug 22, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your current included review allowance is based on your included PR review attempts over the past 7 days.

Next review available in: 24 minutes

Limit details: You’ve used the included review currently available. Your 64 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

How can I continue?

Wait for the limit to reset, then comment @coderabbitai review or push new commits to the PR.

An organization admin can change what happens after included review limits in Billing.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 2ef323e7-a02d-4a11-a43f-5176e95413f4

📥 Commits

Reviewing files that changed from the base of the PR and between 10b5ccf and 37699f1.

📒 Files selected for processing (1)
  • verify.py

Comment @coderabbitai help to get the list of available commands.

@stranske
stranske merged commit 5dacf19 into main Aug 22, 2026
2 checks passed
@stranske
stranske deleted the claude/verify-r-flag branch August 22, 2026 02:29
stranske added a commit that referenced this pull request Aug 24, 2026
…wanted, plus .json (#110)

Maint Coverage Guard failed the moment it was unheld, on both runs (#3 scheduled,
#4 manual), with:

  error: Coverage verification could not find required coverage artifacts on a
         successful Gate run.
  warn:  Unable to locate a recent successful Gate workflow run with required
         coverage artifacts: gate-coverage-trend, gate-coverage-trend-history,
         gate-coverage.

maint-coverage-guard.yml downloads exactly three artifact NAMES. The Gate
produced the first two under those names and the third as "gate-coverage.json",
so the guard probed ten successful runs, found two of three every time, and
reported no usable run at all.

Measured rather than inferred: Gate run #241 (main, success) uploaded
gate-coverage-summary.md, gate-coverage.json, gate-summary.md,
gate-coverage-3.13-1, gate-coverage-trend-history, gate-coverage-trend,
gate-coverage-summary, gate-coverage-3.12-1 -- trend and trend-history present,
"gate-coverage" absent.

An artifact NAME and the FILE inside it are different things, and this step
conflated them. Renaming the artifact is safe and was checked, not assumed:
"gate-coverage.json" as an artifact name appears nowhere else in this repo, and
the only download-artifact steps in any workflow here are the guard's three. The
file inside is still gate-coverage.json, so anything reading contents is
unaffected.

A paired literal in two files that disagreed -- this repo's signature defect --
and it stayed invisible for as long as the guard was held. That is the second
thing the holds were concealing, after PR #90's six F821s.

pr-00-gate.yml is create_only in the Workflows sync manifest with Orchestrator
absent from overwrite_repos, so this local fix persists. The same mismatch
likely affects every consumer whose Gate emits coverage, which is worth raising
upstream separately.

448 collected, ruff and black clean, test_ci_gate_config 12 passed.

Co-authored-by: Tim Stranske <tim@stranskemo.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant