Check a driftNote on every run, and say what the audit does not evaluate

AUDIT.md

@@ -40,6 +40,8 @@ Which carried files and sections a repo is expected to have is decided by its sc
 
 For each applicable type in [`spec/project-types.json`][project-types] and every cross-cutting dimension, evaluate each check at its stated verdict tier:
 
+**Every check below is judged by hand, and none of them is mechanized.** [`spec/audit.py`][audit-runner] evaluates no check in `spec/project-types.json`, and reads that file for one purpose only, to resolve the id a registry `driftNote` names (section 8) against the catalog and against the repo's declared types. Resolving an id is not running the check it names, so a clean run is evidence about the deterministic subset in section 0 and evidence about **nothing** in this section. Read a clean run that way, because the two are easy to conflate: adding a check here changes what an auditor must judge and changes no tool's output, so the check reports nothing until someone evaluates it, and silence from a tool that was never looking reads exactly like a pass. Cite the `file:line` each check was judged against, since that citation is the only durable record that the judgment happened.
+
 - **letter** - the exact file, section, config, or construct is present.
 - **intent** - an equivalent outcome holds even if the form differs.
 
@@ -131,7 +133,7 @@ What a downstream repo does instead is unchanged and is where its context is wor
 
 **Re-running the audit needs a full hub clone with git history.** The verbatim stale-vs-modified classification walks the canonical's history (`git log` / `git show` from the hub root), so a shallow clone or a files-only checkout cannot answer "matches a past hub revision" and those findings are unreliable there. A downstream agent verifying one finding without the full history can instead compare against the current hub canonical on `main` (the whole file for a file-level unit, or the named `## heading` block for a verbatim section), which decides current-match but not stale-vs-modified. An agent picking up such an issue **re-runs the audit first and acts on the live result, not the pasted findings**: a repo moves between filing and pickup, so a stale block leads an agent to "fix" what is already fixed (re-requesting secrets that exist, attempting a no-op forward-sync). State the findings as evidence for *why* the issue was filed, never as the current state.
 
-**Reconcile `driftNotes` in the same pass.** A registry `driftNote` records a *current* deviation from the baseline. Once the deviation is resolved the note is deleted, not left describing finished work, since hand-maintained prose drifts silently otherwise. `spec/audit.py` flags this: when a repo audits clean but a note still asserts outstanding work ("pending", "not yet", "missing", "behind", ...), it raises a drift finding naming the note.
+**Reconcile `driftNotes` in the same pass.** A registry `driftNote` records a *current* deviation from the baseline. Once the deviation is resolved the note is deleted, not left describing finished work, since hand-maintained prose drifts silently otherwise. `spec/audit.py` flags two shapes of note, neither of them gated on the rest of the audit being clean. A note asserting outstanding work in prose ("pending", "not yet", "missing", "behind", ...) is contradicted outright by a clean audit, and where findings are open it is raised as a question of which one it means, because gating the check on a clean audit meant one standing finding a repo could not clear exempted its whole note list, and the repo carrying open findings is where a stale note is most likely. A note naming the check that would retire it, as a parenthesized id (`hugo.generator.pinned`), is the mechanically checkable shape and is surfaced on **every** run: the audit resolves the id against the catalog and confirms the repo declares its type, then hands the check itself to the auditor, since section 4 above is judged by hand. **So a note naming a check id is retired by a person, not by a run.** Write it that way anyway. The id says exactly what would close the note, and the surfaced finding puts that decision in front of whoever runs the audit rather than leaving the note to sit until someone rereads it.
 
 ## 9. Escalate
 

OPERATIONS.md

@@ -43,7 +43,7 @@ python3 spec/audit.py [RepoName]      # one repo
 python3 spec/audit.py --issue [RepoName]
 ```
 
-Findings are a point-in-time snapshot read live over the API. Re-run before acting on one, and quote the run stamp in any issue derived from it. The deterministic subset lives here, and the full letter-and-intent verdict is [AUDIT.md](./AUDIT.md).
+Findings are a point-in-time snapshot read live over the API. Re-run before acting on one, and quote the run stamp in any issue derived from it. The deterministic subset lives here, and the full letter-and-intent verdict is [AUDIT.md](./AUDIT.md). No per-type check in `spec/project-types.json` runs here, so a clean run is not evidence about any of them.
 
 ### Apply or verify repository configuration
 

registry/repos.json

@@ -319,7 +319,7 @@
             "consumerModel": "pull",
             "releaseTrigger": "dispatch-only",
             "configLayout": { "rulesetsDir": "repo-config", "pythonConfig": null },
-            "driftNotes": ["Hugo static site migrated off WordPress.com, stood up 2026-08-01; release model with a dispatch-only publisher that cuts the tag and a source archive.", "lineEndings lf on a release repo, where the rule grants the native-platform default to operational repos only: every consumer is Linux (the Hugo build, the Caddy container, and the rsync deploy).", "content/ is an imported WordPress archive, so the prose, spelling, and style gates are scoped to exclude it.", "The vendored theme under themes/ records no upstream repository, ref, or commit, is not a submodule, and no Dependabot ecosystem covers it, so it cannot be moved or diffed against upstream (hugo.vendored.provenance).", "The generator version and its checksum are pinned in both validate-task.yml and deploy-site-task.yml with nothing asserting the two copies agree, so a one-sided bump validates and deploys different generators (hugo.generator.pinned).", "Deploy credentials are per-environment GitHub Environment secrets and variables, which spec/secrets.json has no vocabulary for, so requiredSecrets stays empty and the names are declared in this repo's own secrets.json environments block; listing them here would make the audit demand them in the repository actions store."]
+            "driftNotes": ["Hugo static site migrated off WordPress.com, stood up 2026-08-01; release model with a dispatch-only publisher that cuts the tag and a source archive.", "lineEndings lf on a release repo, where the rule grants the native-platform default to operational repos only: every consumer is Linux (the Hugo build, the Caddy container, and the rsync deploy).", "content/ is an imported WordPress archive, so the prose, spelling, and style gates are scoped to exclude it.", "Deploy credentials are per-environment GitHub Environment secrets and variables, which spec/secrets.json has no vocabulary for, so requiredSecrets stays empty and the names are declared in this repo's own secrets.json environments block; listing them here would make the audit demand them in the repository actions store."]
         }
     ]
 }

spec/audit.py

@@ -51,6 +51,11 @@
 # Deliberately specific, so a note recording a permanent deviation must not match.
 # An example of one that must not match is a note reading that there is no get-version-task and validate-task is relied on instead.
 PENDING_MARKERS = ["pending", "not yet", "owed", "todo", "still", "behind", "missing", "absent"]
+# A driftNote may name the check that would retire it, as a parenthesized id: "(hugo.generator.pinned)".
+# Deliberately unanchored.
+# Both notes this form was introduced for end the sentence after the paren, so an end-anchored pattern matches neither of the two it was written to catch.
+# That is the shape a matcher fails at silently: it reports nothing, and a fleet with no such note in it reports exactly the same.
+CHECK_ID_RE = re.compile(r"\(([a-z][a-z0-9-]*(?:\.[a-z0-9-]+)+)\)")
 
 
 def load(rel):
@@ -113,6 +118,56 @@ def normalize_ruleset(payload):
     return json.dumps(sub, sort_keys=True)
 
 
+def check_id_owner(spec, cid):
+    """The project type owning a check id, and whether the catalog defines the id at all.
+
+    Returns (None, True) for a cross-cutting check, which every repo carries and no repo declares.
+    The catalog is a required input rather than an optional one: resolving against an absent
+    catalog would report every id as undefined, which is a louder failure than reporting none.
+    """
+    types = spec["types"]
+    for name, t in types.get("types", {}).items():
+        if any(c.get("id") == cid for c in t.get("checks", [])):
+            return name, True
+    for dim in types.get("crossCutting", {}).values():
+        if any(c.get("id") == cid for c in dim.get("checks", [])):
+            return None, True
+    return None, False
+
+
+def driftnote_findings(entry, spec, open_count):
+    """Freshness findings over one repo's registry driftNotes, given how many findings the audit already has.
+
+    Neither shape below is gated on the rest of the audit being clean, and the gate that used to wrap both
+    is the defect this replaces: one standing finding a repo cannot clear exempted its whole note list, so
+    the repo with open findings, where a stale note is most likely, was the one never checked.
+
+    A note naming a check id declares what would retire it, so it is surfaced on every run. What the audit
+    cannot do is decide it, since no per-type check is mechanized, so the id is resolved against the
+    catalog here and the check itself is left to the auditor. A pending-marker note is a prose claim that
+    work is outstanding, so a clean audit contradicts it outright while an unclean audit only asks which of
+    the open findings it means. Narrow markers keep a permanent-deviation note ("relies on validate-task")
+    from tripping.
+    """
+    out = []
+    for note in entry.get("driftNotes", []):
+        quoted = f"\"{note[:70]}{'...' if len(note) > 70 else ''}\""
+        for cid in CHECK_ID_RE.findall(note):
+            owner, known = check_id_owner(spec, cid)
+            if not known:
+                out.append(("DRIFT", f"registry: driftNote names check '{cid}', which spec/project-types.json does not define - fix the id or drop the note: {quoted}"))
+            elif owner and owner not in entry.get("types", []):
+                out.append(("DRIFT", f"registry: driftNote names check '{cid}', whose type '{owner}' this repo does not declare: {quoted}"))
+            else:
+                out.append(("DRIFT", f"registry: driftNote names check '{cid}', which this audit does not evaluate (no per-type check is mechanized, AUDIT.md section 4) - judge it by hand and delete the note once it passes: {quoted}"))
+        marker = next((w for w in PENDING_MARKERS if re.search(rf"\b{re.escape(w)}\b", note, re.I)), None)
+        if marker and not open_count:
+            out.append(("DRIFT", f"registry: driftNote says '{marker}' but the audit is clean - verify and reconcile: {quoted}"))
+        elif marker:
+            out.append(("DRIFT", f"registry: driftNote says '{marker}' while {open_count} finding(s) are open - confirm it describes one of them rather than closed work: {quoted}"))
+    return out
+
+
 def repo_slug(entry):
     # url is https://github.com/<owner>/<repo>
     return "/".join(entry["url"].rstrip("/").split("/")[-2:])
@@ -821,13 +876,7 @@ def audit_repo(entry, spec, branch=None):
                 findings.append(("LETTER", f"cspell: {ws_name} carries a cSpell word list while cspell.json is the single source of truth - delete the workspace copy (CODESTYLE.md Markdown and Spelling)"))
 
     # --- Registry driftNotes freshness ---
-    # Gated on everything else passing: a clean repo has no outstanding work for a pending-marker note to
-    # describe. Narrow markers keep a permanent-deviation note ("relies on validate-task") from tripping.
-    if not findings:
-        for note in entry.get("driftNotes", []):
-            marker = next((w for w in PENDING_MARKERS if re.search(rf"\b{re.escape(w)}\b", note, re.I)), None)
-            if marker:
-                findings.append(("DRIFT", f"registry: driftNote says '{marker}' but the audit is clean - verify and reconcile: \"{note[:70]}{'...' if len(note) > 70 else ''}\""))
+    findings.extend(driftnote_findings(entry, spec, len(findings)))
 
     # Stamp the commit actually read for the ground-truth branch. Never fall back to another branch:
     # a stamp naming develop while carrying main's sha would misattribute every finding.
@@ -1050,6 +1099,34 @@ def _selftest():
         else:
             print(f"  ok   ground branch: registry={entry.get('groundTruthBranch')} override={branch} -> {want}")
 
+    # The driftNote freshness rules, driven directly rather than through audit_repo.
+    # That exercises both the clean and the unclean audit without standing up a whole conformant repo to reach one branch.
+    note_spec = {"types": {"types": {"hugo": {"checks": [{"id": "hugo.generator.pinned"}]},
+                                     "docker": {"checks": [{"id": "docker.cache.registry"}]}},
+                           "crossCutting": {"setup": {"checks": [{"id": "setup.driftnotes.current"}]}}}}
+    note_cases = [
+        # The end-of-sentence period is the case an end-anchored pattern misses, and it is how both notes this form was introduced for are written.
+        ("check id followed by a period", ["hugo"], ["Pinned in two workflows (hugo.generator.pinned)."], 0, ["does not evaluate"]),
+        ("check id mid-sentence", ["hugo"], ["The pin (hugo.generator.pinned) is duplicated."], 0, ["does not evaluate"]),
+        ("cross-cutting id, which no repo declares", ["hugo"], ["Notes go stale (setup.driftnotes.current)."], 0, ["does not evaluate"]),
+        ("check id of an undeclared type", ["hugo"], ["Cache layer (docker.cache.registry)."], 0, ["this repo does not declare"]),
+        ("check id absent from the catalog", ["hugo"], ["Theme record (hugo.vendored.provenence)."], 0, ["does not define"]),
+        # A version string in parentheses is the false positive the id shape has to exclude.
+        ("parenthesized version, not a check id", ["hugo"], ["Generator held at (0.164.0) by the composite action."], 0, []),
+        ("permanent deviation, no marker and no id", ["hugo"], ["Relies on validate-task, having no get-version-task."], 0, []),
+        ("marker note on a clean audit", ["hugo"], ["The sibling doc is pending fleet ratification."], 0, ["but the audit is clean"]),
+        # The case the old gate suppressed outright: one unclearable finding, and the note never checked.
+        ("marker note with findings open", ["hugo"], ["The sibling doc is pending fleet ratification."], 1, ["while 1 finding(s) are open"]),
+    ]
+    for label, types, notes, open_count, wanted in note_cases:
+        got = driftnote_findings({"types": types, "driftNotes": notes}, note_spec, open_count)
+        texts = [t for _, t in got]
+        if len(texts) != len(wanted) or not all(w in t for w, t in zip(wanted, texts)):
+            ok = False
+            print(f"  FAIL driftNote {label} -> {texts}, want {wanted}")
+        else:
+            print(f"  ok   driftNote {label}: {len(texts)} finding(s)")
+
     # A ground-truth branch that does not resolve is one error, not a baseline's worth of letters.
     # Every `?ref=` read would 404 and report each carried file absent, describing the ref, not the repo.
     # The branch facts are already read at that point, so they are reported rather than dropped.
@@ -1094,7 +1171,8 @@ def render_issue(entry, findings, ground, audited_sha, run_utc, hub_sha):
     w(f"Generated from the hub audit of `{name}` ({types}). Run stamp `audit run {run_utc} | hub {hub_sha}`, "
       f"against `@ {stamp}` (the format AUDIT.md section 8 says a derived artifact quotes). Regenerate with "
       f"`spec/audit.py --issue {name}`. Findings are a point-in-time snapshot - re-run the audit before acting. "
-      f"This lists what the audit mechanically detects. The full letter and intent verdict lives in AUDIT.md.")
+      f"This lists what the audit mechanically detects. No per-type check in `spec/project-types.json` is run here, "
+      f"so nothing below is evidence about one; the full letter and intent verdict is judged by hand per AUDIT.md section 4.")
     w("")
     if not findings:
         w("The deterministic checks are clean - nothing to converge.")
@@ -1149,6 +1227,7 @@ def main(argv=None):
         "settings": load("repo-config/settings.json"),
         "secrets": load("spec/secrets.json"),
         "files": load("spec/files.json"),
+        "types": load("spec/project-types.json"),
     }
     issue_mode = a.issue
     wanted = {n.lower() for n in a.names}
@@ -1199,7 +1278,7 @@ def main(argv=None):
         stamp = f" @ {ground}@{audited_sha[:7]}" if audited_sha else ""
         print(f"== {entry['name']} ({', '.join(entry.get('types', []))}; {model}){stamp} ==")
         if not findings:
-            print("  clean (deterministic checks; the full letter+intent verdict is AUDIT.md's)")
+            print("  clean (deterministic checks only; no per-type check in spec/project-types.json is run here - AUDIT.md section 4 is judged by hand)")
         for kind, text in findings:
             print(f"  {kind:6} {text}")
             if kind in ("DEFECT", "LETTER", "ERROR"):