Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 29 additions & 9 deletions .github/workflows/pr-00-gate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -116,9 +116,9 @@ jobs:
# so there is exactly one literal per toggle and the call site cannot drift from the
# summary's idea of what ran. Do not repeat a `false` in the `with:` block.
#
# EVERY DISABLED CHECK STATES ITS BLOCKING **AND** DRAINABLE QUANTITY. A bare "604
# errors" reads as be-patient; "604 errors, drainable 0 per PR" reads as the deadlock it
# is. Full baseline, measurement commands and rule-by-rule drains:
# EVERY DISABLED CHECK STATES ITS BLOCKING **AND** DRAINABLE QUANTITY. A bare error
# count reads as be-patient; the same count with "drainable 0 per PR" beside it reads as
# the deadlock it is. Full baseline, measurement commands and rule-by-rule drains:
# docs/CI_LINT_BASELINE.md. Regenerate with `python3 scripts/ci_lint_baseline.py`.
#
# HISTORY, so nobody re-derives it: until 2026-08-23 all five python-ci jobs failed at
Expand All @@ -131,16 +131,20 @@ jobs:
pytest_markers = 'not quarantine and not slow'

# typecheck (and its deprecated `run-mypy` alias) -- OFF.
# blocking: 604 mypy errors across 90 of 189 files (measured 2026-08-23,
# `mypy --exclude .workflows-lib .`, mypy 2.3.1 as pinned).
# blocking: 608 mypy errors across 89 of 189 files (measured 2026-08-23 on 24cb115 with
# `python3 scripts/ci_lint_baseline.py`, which runs
# `mypy --exclude .workflows-lib .` under the pinned mypy 2.3.1).
# THIS NUMBER DRIFTS: it moved 601 -> 604 -> 607 -> 608 during
# 2026-08-23 as typed code landed, and nothing couples it to a
# measurement, so re-run that script rather than trusting it.
# drainable: 0 per PR. Every one is a real annotation or logic change in a distinct
# module; there is no formatter and no `--fix` for this.
# drains by: typed modules landing incrementally. Turn this on the same day the count
# reaches 0, not before -- a `disable_error_code` list covering 15 codes
# would make the job green while checking nothing, which is the defect
# verify.py exists to stop.
# reaches 0, not before -- the 15 commonest of the 19 codes cover 603
# of the 608, so a `disable_error_code` list would make the job green
# while checking essentially nothing, the defect verify.py exists to stop.
# note: mypy.ini is committed even though the check is off, because without it
# `mypy .` aborts on a duplicate-module setup error and the 604 above would
# `mypy .` aborts on a duplicate-module setup error and the count above would
# be unverifiable prose instead of a number anyone can regenerate.
typecheck = False

Expand Down Expand Up @@ -1104,6 +1108,22 @@ jobs:
github,
core,
task: 'gate-commit-status',
// `env: {}` PINS THIS CALL TO THE WORKFLOW TOKEN, and it is load-bearing.
// With the default `env: process.env` the token load balancer collects every App
// and PAT secret this job exposes and picks by remaining capacity, so which token
// posts the status varies run to run. `POST /statuses/{sha}` needs the `statuses`
// scope, which `GITHUB_TOKEN` has here (the job declares `statuses: write`) but the
// WORKFLOWS App installation does NOT. When the balancer picks the App the post
// 403s, `isIntegrationPermissionError` swallows it as best-effort, and the PREVIOUS
// status survives -- so a green run leaves a red `Gate / gate` that nothing clears.
// Measured on PR #54 (2026-08-23): `Selected token: WORKFLOWS_APP` then
// `POST /repos/stranske/Orchestrator/statuses/bfb6fb1... - 403`, with `STATE: success`
// computed one line earlier; run 32659615306 posted fine minutes before with the
// same declared permissions, because the balancer happened to pick another token.
// Declaring `capabilities: ['statuses:write']` would NOT fix it: token_load_balancer
// aliases that to the generic `write-repo`, which all three token types claim.
// Retries still apply -- only the token source is fixed.
env: {},
});
const owner = context.repo.owner;
const repo = context.repo.repo;
Expand Down
25 changes: 16 additions & 9 deletions docs/CI_LINT_BASELINE.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,12 +56,12 @@ Measured 2026-08-23 with the pinned versions, after the drain described below.
| --- | --- | --- | --- | --- |
| `lint-ruff` | `ruff check --extend-exclude .workflows-lib .` | **0 findings** | 0 | **ON, green** |
| `lint-format` | `black --check --line-length 100 --exclude '(\.venv\|\.workflows-lib\|node_modules)' .` | **0 files** | 0 | **ON, green** |
| `typecheck-mypy` | `mypy --exclude .workflows-lib .` | **604 errors in 89 of 189 files** | **0 per PR** | OFF |
| `typecheck-mypy` | `mypy --exclude .workflows-lib .` | **608 errors in 89 of 189 files** | **0 per PR** | OFF |
| `coverage` | `pytest --cov` | **0 startup errors** | 0 | **ON, green** |
| `python 3.12` / `python 3.13` | `pytest -m 'not quarantine and not slow' -n auto --dist=loadgroup` | **0 failures** (393 passed, 9 skipped) | 0 | **ON, green** |

One OFF row remains, and it states both quantities on purpose. `604 errors` alone reads as
*be patient*; `604 errors, drainable 0 per PR` reads as what it is. (`coverage` was the second such
One OFF row remains, and it states both quantities on purpose. An error count alone reads as
*be patient*; the same count with `drainable 0 per PR` beside it reads as what it is. (`coverage` was the second such
row until 2026-08-23; the entry below records how it drained rather than deleting the history.) The authoritative
copy of each of those annotations lives beside the toggle itself, in the `Compute Python CI toggles`
step of `.github/workflows/pr-00-gate.yml` — one place, so the call site and the `summary` job's
Expand Down Expand Up @@ -126,19 +126,26 @@ so hoisting it would break the module. The rule stays on for every other file.

## Deferred: mypy

* **Blocking:** 601 errors in 89 of 189 files. Top codes: `arg-type` 149, `index` 111, `assignment`
78, `union-attr` 43, `no-redef` 43, `attr-defined` 42, `operator` 37, `import-not-found` 25.
* **Blocking:** 608 errors in 89 of 189 files, across 19 distinct codes. Top codes: `arg-type` 151,
`index` 111, `assignment` 79, `attr-defined` 46, `no-redef` 43, `union-attr` 43, `operator` 37,
`import-not-found` 25.
* **This figure drifts, and nothing couples it to a measurement.** It read 601, then 604, then 607,
then 608 during 2026-08-23 alone, as typed code landed; the three places that recorded it
disagreed with each other and with reality at the same time. `test_ci_gate_config.py` asserts only
that a disabled toggle *states* a blocking and a drainable quantity, never that the number is
right — it cannot, without making the suite depend on an installed mypy. Re-run
`python3 scripts/ci_lint_baseline.py` before quoting it.
* **Drainable:** 0 per PR. There is no `mypy --fix`; every one is an annotation or logic change in a
distinct module.
* **Drains by:** typed modules landing incrementally. Flip `typecheck` on the day the count reaches
zero — **not** by adding a `disable_error_code` list. Fifteen codes cover 597 of the 601 and would
produce a green job that checks essentially nothing, which is precisely the defect `verify.py`
exists to stop.
zero — **not** by adding a `disable_error_code` list. The 15 commonest of the 19 codes cover 603 of
the 608, so such a list would produce a green job that checks essentially nothing, which is
precisely the defect `verify.py` exists to stop.
* **`mypy.ini` is committed even though the check is off**, and for one reason: without it `mypy .`
aborts with `Source file found twice under different module names: "_llm_client" and
"scripts.langchain._llm_client" ... (errors prevented further checking)`. That single setup error
is all a reader of the failed job would have seen; it masked the real number entirely. With
`explicit_package_bases = True` the 601 above is a number anyone can regenerate rather than prose.
`explicit_package_bases = True` the count above is one anyone can regenerate rather than prose.
It must be `mypy.ini` and not `pyproject.toml` or `setup.cfg` — see the next section.

## Drained 2026-08-23: coverage
Expand Down
16 changes: 9 additions & 7 deletions test_ci_gate_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,15 @@
the SAME rule set. Lose it and the Gate silently reverts to `--select E4,E7,E9,F` while Autofix
reverts to Ruff 0.16's much wider default — the disagreement that produced four 143-file
`chore(autofix)` commits on PR #42.
4. **A disabled check states its blocking AND drainable quantity.** "601 errors" reads as
be-patient; "601 errors, drainable 0 per PR" reads as a deadlock. The pair is the diagnosis.
4. **A disabled check states its blocking AND drainable quantity.** An error count alone reads
as be-patient; the same count with "drainable 0 per PR" beside it reads as a deadlock. The
pair is the diagnosis.
5. **One literal per toggle.** The `with:` block and the `summary` job's coverage branch both read
`needs.detect.outputs.*`, so a second hardcoded `false` would be a literal that can drift.
6. **The recorded baseline moves with the pins.** The counts are version-specific, so bumping a
pin without re-measuring must go red rather than quietly re-describing a toolchain nobody runs.
7. **`mypy.ini` never silences an error code.** Fifteen `disable_error_code` entries would cover
597 of the 601 findings and produce a green job that checks nothing.
603 of the 608 findings and produce a green job that checks nothing.
8. **A citation names a file that is there.** These configs are prose-heavy on purpose: each
tells the next reader where to re-measure before touching a pin. The pin file shipped citing
`docs/ci/LINT_BASELINE.md` while the real path was `docs/CI_LINT_BASELINE.md`, so the single
Expand Down Expand Up @@ -229,8 +230,9 @@ def test_every_disabled_toggle_states_blocking_and_drainable():
for field in ("blocking:", "drainable:", "drains by:"):
assert field in block, (
f"the `{name} = False` toggle does not state `{field}` in the comment above it. "
"Both quantities belong in the same place: '601 errors' reads as be-patient, "
"'601 errors, drainable 0 per PR' reads as the deadlock it is. See "
"Both quantities belong in the same place: an error count alone reads as "
"be-patient, the same count with 'drainable 0 per PR' beside it reads as the "
"deadlock it is. See "
f"{BASELINE_DOC.relative_to(HERE)}."
)

Expand Down Expand Up @@ -290,12 +292,12 @@ def test_mypy_config_silences_nothing():
require_checkout()
assert MYPY_INI.is_file(), (
"mypy.ini is missing. Without it `mypy .` aborts on 'Source file found twice under "
"different module names' and the recorded 601 becomes unverifiable prose."
"different module names' and the recorded count becomes unverifiable prose."
)
text = MYPY_INI.read_text(encoding="utf-8")
for forbidden in ("disable_error_code", "ignore_errors", "follow_imports = skip"):
assert forbidden not in text, (
f"mypy.ini sets `{forbidden}`. Fifteen disabled codes would cover 597 of the 601 "
f"mypy.ini sets `{forbidden}`. Fifteen disabled codes would cover 603 of the 608 "
"findings and make typecheck-mypy green while checking nothing — the exact defect "
"verify.py exists to stop. Leave the check OFF and drain the findings instead."
)
Expand Down
Loading