Track verbatim sections in the fleet burn-down report

reports/divergences.md

@@ -22,14 +22,16 @@ Generated by `python3 spec/fidelity_honesty.py --report` - do not hand-edit. Cur
 
 ## Untriaged - add a disposition to `spec/divergences.json`
 
-_None - every live divergence has a recorded disposition._
+- **AGENTS.md > Repository Boundaries and Write Safety** - hand-modified in AudioCleaner, DevKitCIoT, ESPHome-Config, ESPHome-NonRoot, Financial-Modeling, HolidayLights, HomeAssistant-Config, HomeAutomation-Config, KiCadLibrary, LanguageTags, MediaTools, NxWitness, PhotoCleaner, PlexCleaner, Utilities, VSCode-Server-DotNetCore, Vantage-Config, aiopurpleair, homeassistant-purpleair (verbatim canonical)
+- **AGENTS.md > Git and Commit Rules** - hand-modified in AudioCleaner, DevKitCIoT, ESPHome-Config, ESPHome-NonRoot, Financial-Modeling, HolidayLights, HomeAssistant-Config, HomeAutomation-Config, KiCadLibrary, LanguageTags, MediaTools, NxWitness, PhotoCleaner, PlexCleaner, Utilities, VSCode-Server-DotNetCore, Vantage-Config, aiopurpleair, homeassistant-purpleair (verbatim canonical)
+- **AGENTS.md > Verification Discipline** - hand-modified in AudioCleaner, DevKitCIoT, ESPHome-Config, ESPHome-NonRoot, Financial-Modeling, HolidayLights, HomeAssistant-Config, HomeAutomation-Config, KiCadLibrary, LanguageTags, MediaTools, NxWitness, PhotoCleaner, PlexCleaner, Utilities, VSCode-Server-DotNetCore, Vantage-Config, aiopurpleair, homeassistant-purpleair (verbatim canonical)
 
 ## Mechanical re-vendor (verbatim stale copies)
 
 A past hub revision, not the current canonical - the audit already flags these as DRIFT. Copy the current file down. No judgment needed.
 
 - **.markdownlint-cli2.jsonc** (15): DevKitCIoT, ESPHome-Config, ESPHome-NonRoot, HolidayLights, HomeAssistant-Config, HomeAutomation-Config, KiCadLibrary, LanguageTags, MediaTools, NxWitness, PlexCleaner, Utilities, VSCode-Server-DotNetCore, Vantage-Config, homeassistant-purpleair
-- **repo-config/configure.sh** (8): ESPHome-Config, Financial-Modeling, HomeAssistant-Config, HomeAutomation-Config, PlexCleaner, ProjectTemplate, Utilities, Vantage-Config
+- **repo-config/configure.sh** (7): ESPHome-Config, Financial-Modeling, HomeAssistant-Config, HomeAutomation-Config, PlexCleaner, Utilities, Vantage-Config
 
 ## Promote candidates (intent uniform -> verbatim)
 

spec/divergences.json

@@ -1,6 +1,6 @@
 {
     "$schema": "./divergences.schema.json",
-    "note": "Curated dispositions for known fleet divergences from the manifest canonicals - the burn-down ledger. spec/fidelity_honesty.py --report joins this against live fleet reality to write reports/divergences.md. A recorded divergence still present renders as a burn-down task with its disposition. A live divergence absent here renders as UNTRIAGED. A recorded divergence no longer live renders as resolved. Edit this file (not the generated report) and regenerate. dispositions cover per-repo file divergences from a verbatim or intent canonical. gaps cover files carried by the fleet but absent from spec/files.json. disposition vocabulary: re-vendor (drift-to-fix, copy the current canonical down), track (a gap to add to the manifest), accepted (a legitimate permanent divergence, no action), upstream-candidate (the downstream carries an improvement the hub should adopt, then re-vendor), investigate (recorded, decision pending).",
+    "note": "Curated dispositions for known fleet divergences from the manifest canonicals - the burn-down ledger. spec/fidelity_honesty.py --report joins this against live fleet reality to write reports/divergences.md. A recorded divergence still present renders as a burn-down task with its disposition. A live divergence absent here renders as UNTRIAGED. A recorded divergence no longer live renders as resolved. Edit this file (not the generated report) and regenerate. dispositions cover per-repo divergences from a verbatim or intent canonical - a whole file, or a verbatim section whose unit key is the file path, then a space-greater-space delimiter, then the section name (for example AGENTS.md > Git and Commit Rules). gaps cover files carried by the fleet but absent from spec/files.json. disposition vocabulary: re-vendor (drift-to-fix, copy the current canonical down), track (a gap to add to the manifest), accepted (a legitimate permanent divergence, no action), upstream-candidate (the downstream carries an improvement the hub should adopt, then re-vendor), investigate (recorded, decision pending).",
     "dispositions": [
         { "path": ".editorconfig-checker.json", "repos": ["HomeAutomation-Config", "HolidayLights"], "disposition": "accepted", "reason": "Both carry a legitimate repo-specific Exclude list (HomeAutomation-Config excludes a Vantage/ subtree, HolidayLights excludes .fseq sequence files). The uniform Disable block is carried intent-equivalent. Exclude is inherently repo-local, which is why the unit is intent, not verbatim.", "tracking": null },
         { "path": ".markdownlint-cli2.jsonc", "repos": ["aiopurpleair", "PhotoCleaner", "AudioCleaner"], "disposition": "re-vendor", "reason": "Verbatim config held as a hand-modified copy in these three (not a past hub revision). Restore the current canonical.", "tracking": null },

spec/fidelity_honesty.py

@@ -29,13 +29,7 @@
 REPORT_PATH = "reports/divergences.md"  # the generated, checked-in burn-down report (--report)
 
 
-def canonical_text(entry):
-    """The hub's canonical for a unit: its reference snippet, else its own root copy."""
-    ref = entry.get("reference") or entry["path"]
-    try:
-        return (audit.ROOT / ref).read_text(encoding="utf-8", errors="replace")
-    except OSError:
-        return None
+SECTION_SEP = " > "  # joins a file path and a verbatim section name into one unit label (path > section), ASCII
 
 
 def fetch(slug, path, ref):
@@ -46,46 +40,82 @@ def fetch(slug, path, ref):
     return base64.b64decode(content["content"]).decode("utf-8", "replace")
 
 
+def check_units(spec):
+    """Yield the fleet's check units - whole intent/verbatim files, plus each verbatim section of any file.
+
+    Each is (label, fidelity, entry_appliesTo, section_appliesTo, downstream_path, canonical_path, extract).
+    downstream_path is where the repo's copy lives (always the entry's `path`); canonical_path is the hub's
+    canonical (the `reference` snippet for a reference-backed unit, else the same path). extract is None for a
+    whole file, or a section extractor for a verbatim section (the region is compared, not the whole file).
+    """
+    for e in spec["files"]["baseline"]:
+        fid = e.get("fidelity")
+        if fid in ("intent", "verbatim"):
+            yield (e["path"], fid, e.get("appliesTo", "*"), "*", e["path"], e.get("reference") or e["path"], None)
+        sections = e.get("sections")
+        if not isinstance(sections, list):
+            continue
+        for elt in sections:
+            if isinstance(elt, dict) and elt.get("fidelity") == "verbatim" and isinstance(elt.get("name"), str):
+                name = elt["name"]
+                yield (f"{e['path']}{SECTION_SEP}{name}", "verbatim", e.get("appliesTo", "*"),
+                       elt.get("appliesTo", "*"), e["path"], e.get("reference") or e["path"],
+                       (lambda t, n=name: audit.extract_section(t, n)))
+
+
 def fidelity_pass(spec):
     defaults = spec["registry"].get("defaults", {})
     repos = [r for r in spec["registry"]["repos"] if r.get("status") == "cataloged"]
-    units = [e for e in spec["files"]["baseline"] if e.get("fidelity") in ("intent", "verbatim")]
 
-    spreads = []          # (entry, spread dict) for the full table
+    spreads = []          # (unit dict, spread dict) for the full table
     promote = []          # intent units that are uniform fleet-wide
     mislabel = []         # verbatim units that diverge non-stale
 
-    for e in units:
-        path, fid = e["path"], e["fidelity"]
-        canon = canonical_text(e)
+    for label, fid, ent_ap, sec_ap, down_path, canon_path, extract in check_units(spec):
+        unit = {"path": label, "fidelity": fid}
+        try:
+            canon_raw = (audit.ROOT / canon_path).read_text(encoding="utf-8", errors="replace")
+        except OSError:
+            canon_raw = None
+        canon = None if canon_raw is None else (canon_raw if extract is None else extract(canon_raw))
         if canon is None:
-            spreads.append((e, None))
+            spreads.append((unit, None))
             continue
         canon_hash = audit.content_hash(canon)
-        history = {audit.content_hash(t) for t in audit.git_file_history(e.get("reference") or path)}
+        # Hash each past revision's same region (a section may be absent in an old revision -> skip it).
+        history = set()
+        for t in audit.git_file_history(canon_path):
+            region = t if extract is None else extract(t)
+            if region is not None:
+                history.add(audit.content_hash(region))
         spread = {"match": [], "stale": [], "differs": [], "unavailable": []}
         for r in repos:
-            if not audit.applies(e.get("appliesTo", "*"), audit.repo_selectors(r, defaults)):
+            sel = audit.repo_selectors(r, defaults)
+            if not (audit.applies(ent_ap, sel) and audit.applies(sec_ap, sel)):
                 continue
-            text = fetch(audit.repo_slug(r), path, r.get("groundTruthBranch", "main"))
+            text = fetch(audit.repo_slug(r), down_path, r.get("groundTruthBranch", "main"))
             if text is None:  # missing (404) or present-but-non-inline (too large / encoding "none")
                 spread["unavailable"].append(r["name"])
                 continue
-            dh = audit.content_hash(text)
+            region = text if extract is None else extract(text)
+            if region is None:  # a verbatim section whose heading is absent downstream
+                spread["differs"].append(r["name"])
+                continue
+            dh = audit.content_hash(region)
             if dh == canon_hash:
                 spread["match"].append(r["name"])
             elif dh in history:
                 spread["stale"].append(r["name"])
             else:
                 spread["differs"].append(r["name"])
-        spreads.append((e, spread))
+        spreads.append((unit, spread))
         # A verbatim candidate has NO hand-modified copy ("differs") and at least one confirmed match with
         # the current canonical. Stale copies do not disqualify it - verbatim would flag them "stale ->
         # re-vendor", which is the point. A unit that is entirely stale/unavailable is not confirmed uniform.
         if fid == "intent" and spread["match"] and not spread["differs"]:
-            promote.append((e, spread))
+            promote.append((unit, spread))
         if fid == "verbatim" and spread["differs"]:
-            mislabel.append((e, spread))
+            mislabel.append((unit, spread))
     return spreads, promote, mislabel
 
 

spec/validate.py

@@ -285,6 +285,8 @@ def check_selector(where, applies_to):
             continue
         for elt in sections:
             if isinstance(elt, dict):
+                if not isinstance(elt.get("name"), str) or not elt.get("name"):
+                    errors.append(f"files.json: {path} section object missing a non-empty string 'name': {elt!r}")
                 check_selector(f"{path} section '{elt.get('name', '?')}'", elt.get("appliesTo", "*"))
                 # A section may carry its own fidelity (intent default, or verbatim for a universal rule block
                 # checked byte-for-byte). verbatim is meaningful only on a markdown file, where the heading
@@ -304,6 +306,15 @@ def check_selector(where, applies_to):
         div = load("spec/divergences.json")
         repo_names = {r.get("name") for r in repos["repos"] if isinstance(r, dict)}
         manifest_paths = {i.get("path") for i in baseline if isinstance(i, dict)}
+        # A verbatim section is an addressable unit too, labelled "path > section" (matches fidelity_honesty's
+        # SECTION_SEP), so a section-scoped divergence can carry its own disposition. Only well-formed section
+        # entries produce a label - a malformed one is already reported by the files.json checks above.
+        for i in baseline:
+            if not isinstance(i, dict) or not isinstance(i.get("path"), str) or not isinstance(i.get("sections"), list):
+                continue
+            for elt in i["sections"]:
+                if isinstance(elt, dict) and elt.get("fidelity") == "verbatim" and isinstance(elt.get("name"), str):
+                    manifest_paths.add(f"{i['path']} > {elt['name']}")
         # Guard the root type: a non-object root (a list from a bad edit) would crash the .get() calls below.
         if not isinstance(div, dict):
             errors.append("divergences.json: root must be an object")