From 455d0327a5cafe690ecd79fa6e9a90251ad24d49 Mon Sep 17 00:00:00 2001 From: PhysShell Date: Wed, 29 Jul 2026 09:41:19 +0000 Subject: [PATCH 1/4] docs(proposal): define human-confirmed song-id curation workflow MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Docs-only. Adds docs/proposals/song-id-curation-workflow.md (+ README index). Status: for discussion — binds nothing, not Accepted. No Rust/Python/corpus/ manifest/fixture/generated changes; no corpus labels written. Treats ADR-0031 and ADR-0032 as fixed law (10 invariants). Defines an offline, human-confirmed pipeline: inventory sources by sha256 -> non-authoritative metadata-only suggestions -> explicit curator decisions (no default = acceptance) -> immutable fingerprinted plan -> transactional apply to a fresh copy -> deterministic songs manifest -> validation via the existing core song_holdout_preflight. Curation unit is the sha256 source (all its chunks share one SongId, indivisible). Suggestions never write song_id; artist is a reported heuristic, not structured provenance. Makes the selections the brief required rather than deferring them: owner = standalone isolated song-curation/ tool (compared against griff curate / manifest); manifest strategy = tool owns song-aware generation first, extend griff manifest later; SongId = opaque ledger-issued song- (title-slug and membership-hash rejected); a domain-tagged sorted-line SHA-256 corpus fingerprint; an 11-category typed refusal taxonomy; and a four-slice RED->GREEN implementation sequence gated behind a new ADR (prov. ADR-0033) and a controlled pilot. Partial curation is holdout_ready:false and never a valid song-holdout corpus. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_012T7SRMiXmZe5v1imtrKMPi --- docs/proposals/README.md | 6 + docs/proposals/song-id-curation-workflow.md | 428 ++++++++++++++++++++ 2 files changed, 434 insertions(+) create mode 100644 docs/proposals/song-id-curation-workflow.md diff --git a/docs/proposals/README.md b/docs/proposals/README.md index 5b85e9da..a8c6ac06 100644 --- a/docs/proposals/README.md +++ b/docs/proposals/README.md @@ -51,3 +51,9 @@ live contract. — technique modules, a deterministic symbolic candidate terrain, and a transactional agent-editing contract for the cockpit. Status: for discussion. +- [`song-id-curation-workflow.md`](song-id-curation-workflow.md) — an offline, + human-confirmed workflow (inventory → suggest → confirm → plan → apply → + manifest → validate) for assigning `SongId` (ADR-0031) to corpus sources by + `sha256`, with suggestions held strictly non-authoritative and a typed + refusal taxonomy; recommends a standalone `song-curation/` tool and a + four-slice implementation gated behind a new ADR. Status: for discussion. diff --git a/docs/proposals/song-id-curation-workflow.md b/docs/proposals/song-id-curation-workflow.md new file mode 100644 index 00000000..50333f69 --- /dev/null +++ b/docs/proposals/song-id-curation-workflow.md @@ -0,0 +1,428 @@ +# Song-ID Curation Workflow (proposal) + +Status: for discussion (a proposal — it binds nothing and is **not** Accepted). + +An offline, **human-confirmed** workflow for assigning `SongId` (ADR-0031, +schema v10) to corpus sources, so that song-level holdout +([ADR-0032](../adr/0032-holdout-filtering-boundary.md); +[`../../reachability-lab`](../../reachability-lab)) becomes usable without +letting heuristics become provenance facts by clerical accident. This document +designs the workflow; it does not build it, and it writes no corpus labels. + +## 1. Fixed inputs — ADR-0031 and ADR-0032 are law here + +This proposal treats the following as **fixed**, not material to reinterpret: + +1. `SongId` identifies a composition at the **Work** level. +2. It is **opaque** and **curator-assigned**. +3. It is **never** derived automatically from musical or metadata similarity. +4. A tool may generate **suggestions**, but suggestions have **no authority**. +5. Every edition, transcription, arrangement, and cover of one composition uses + the **same** `SongId`. +6. `song_id` exists **only** for leakage-safe holdout and source-identity splits. +7. Per-`SourceRef.song_id` values are **authoritative** after application. +8. `CorpusManifest.songs` is a deterministic **convenience and cross-check**, not + the source of truth. +9. A holdout-ready corpus requires **complete, consistent** song coverage. +10. Unknown or inconsistent identity must **typed-refuse**, never be guessed. + +No arrangement-level exception is invented. Separating expressions/arrangements +is a *separate* future `arrangement_id` (a lower FRBR level), out of scope here +(ADR-0031). + +## 2. Goal + +A pipeline that can: + +``` +inventory sources + → generate non-authoritative grouping suggestions + → record explicit curator decisions + → build a deterministic application plan + → apply that plan transactionally to a fresh corpus copy + → generate the songs manifest + → validate consistency and holdout readiness +``` + +The system reduces curation **labour**; it never lets a heuristic promote itself +to a stored identity. + +## 3. Curation unit — the source file, by `sha256` + +Curation operates on **unique source files identified by exact `sha256`**, never +on individual chunks. All chunks sharing one `sha256` are **one indivisible +source unit**: + +- they must receive the **same** `SongId`; +- a decision may **not** label only some chunks of a source; +- conflicting existing labels on one source **typed-refuse** + (`ConflictingExistingSongIds`); +- a source without `sha256` is **not curatable** by this tool and typed-refuses + (`UnidentifiedSource`). + +`ChunkMeta.title`, filename, format, `bar_range`, `track_index`, and chunk ids +are **display / suggestion evidence** only; they do not define source identity. + +## 4. Ownership — selected: a standalone offline `song-curation/` tool + +**Selected:** a **standalone, isolated offline tool** (provisionally +`song-curation/`), following the established isolation posture of `fuzz` / `lab` +/ `census` / `migrate` / `reachability-lab` (ADR-0010): + +- not a production-generation dependency; +- **no** holdout or curation policy added to the cockpit; +- **no** in-place corpus mutation; +- **no** automatic execution in ordinary generation; +- `griff-core` supplies only reusable **schema and validation contracts** + (`ChunkMeta`, `SourceRef`, `CorpusManifest`, `SongId`, `source_sha256`, and the + existing `song_holdout_preflight`). + +### Comparison (required) + +| Owner | For | Against | Verdict | +|---|---|---|---| +| **Standalone `song-curation/`** (selected) | Matches the isolated-tool precedent; a multi-phase ledger/plan/apply workflow is too large for a subcommand; no production surface acquires curation policy; can carry its own artifact schemas. | One more isolated crate; not exercised by workspace CI (per the isolation policy, verified locally). | **Selected.** | +| Extend `griff curate` | Reuses an existing curation entry point. | `griff curate` is interactive per-chunk cockpit-adjacent curation; song identity is a *source-level, ledgered, transactional* operation with a fingerprint/plan/apply contract that does not fit an interactive per-chunk command, and it would pull curation policy into a production binary. | Rejected. | +| Extend `griff manifest` | Manifest generation already emits `CorpusManifest`. | `griff manifest` folds chunks into a manifest; it is not a curation ledger and must stay a pure projection. Owning issuance/decisions there overloads it and adds policy to production. | Rejected for *ownership* (but see §9 for its eventual manifest role). | + +Ownership is **selected now**, not deferred to implementation. + +## 5. Workflow phases + +### 5.1 Inventory (read-only) + +Read a corpus snapshot and **collapse chunks by exact source `sha256`** into +deterministic source records with at least: + +``` +source_sha256 +filenames # sorted, unique +titles # sorted, unique +formats # sorted, unique +chunk_ids # sorted +existing_song_ids # sorted, unique (from SourceRef.song_id) +existing_manifest_membership # SongIds naming this sha256 in CorpusManifest.songs +``` + +A source with **more than one** distinct existing `song_id` across its chunks → +`ConflictingExistingSongIds` (read-only refuses to summarize it as clean). A +chunk without `sha256` → `UnidentifiedSource`. Inventory writes nothing. + +### 5.2 Suggest (non-authoritative) + +Deterministic, **evidence-bearing** candidate groupings. **Version 1 uses +metadata evidence only:** + +- normalized `ChunkMeta.title`; +- normalized source filenames / stems; +- repeated source names with format/version suffixes removed by a **documented** + rule (reuse the census's `strip_version_suffix` convention: trailing `(...)` + removed only when the inner text starts with `ver`, contains ` by `, or is + all-digits — never for e.g. `(Reprise)`); +- already-confirmed identity relationships, when supplied. + +**No canonical artist field exists in the schema.** "Artist" parsed from a title +or filename is a **suggestion heuristic**, reported as such in `evidence`; it is +not structured provenance. If an artist signal is ever wanted as structured +input, it must arrive as a **separate explicit input artifact**, not inferred. + +**No** note-content similarity, embeddings, audio fingerprinting, cover +detection, or MIR classifier enters version 1. Every suggested group exposes its +**evidence** and **uncertainty**, refers to exact source hashes, and **never +writes `song_id`**. + +### 5.3 Confirm (explicit) + +A curator explicitly **accepts / rejects / splits / merges / manually defines** +source groupings. **No default action means acceptance.** A batch or unattended +run must **not** convert suggestions into decisions. Confirmation is captured in +the **decisions artifact** (§8) such that a later apply run can prove **every +stored label came from an explicit curator decision** (each decision names the +`source_sha256`s, the curator, and a timestamp; the apply report echoes the +decisions digest). + +### 5.4 Plan (immutable) + +Build an **immutable application plan** from: + +- an exact **corpus fingerprint** (§10); +- a **versioned decisions artifact**; +- the **tool-policy version**. + +The plan enumerates **every** intended source-level assignment and **every** +affected chunk. If the corpus fingerprint has changed since inventory/confirm, +planning or application **typed-refuses** +(`DecisionCorpusFingerprintMismatch` / `PlanCorpusFingerprintMismatch`) rather +than rebasing decisions onto a different snapshot. + +### 5.5 Apply (transactional) + +Apply **only** a previously validated plan. Required semantics: + +- write to a **fresh output directory** (`OutputAlreadyExists` if it exists; + `OutputWouldModifyInput` if it resolves inside/equal to an input — reuse the + `migrate-v9` preflight discipline); +- **all-or-nothing**; no partial output presented as successful; +- update **every** chunk sharing an assigned `sha256`; +- preserve every unrelated field byte-for-byte where serialization permits + (only `SourceRef.song_id` is added/changed); +- **deterministic** file order and JSON rendering; +- **idempotent** for already-correct labels; +- `ConflictingExistingSongIds` / `SourceAssignedToMultipleSongs` / + `UnknownDecisionSource` typed-refuse; +- **never clear** an existing label unless an explicit curator **correction** + authorizes it (`ExistingLabelReplacementNotAuthorized` otherwise). + +A **correction is a new curator decision**, not heuristic reconciliation. + +### 5.6 Generate manifest (deterministic) + +Generate `CorpusManifest.songs` deterministically from the applied per-source +labels: `SongId → sorted, unique [sha256]`. The per-source `SourceRef.song_id` +remains **authoritative**; the map is the cross-check (law 8). + +**Manifest ownership — selected:** for the **first implementation**, the +`song-curation/` tool **exclusively** owns song-aware manifest generation +(strategy 1), kept isolated until the workflow is proven. The **recommended +durable end state** is **strategy 2**: extend `griff manifest` to rebuild `songs` +from the authoritative per-source labels, as a **later, separately-accepted** +change. + +> **Interim hazard (open issue for review).** `griff manifest` today always emits +> `songs: None` (`ui-core/src/corpus.rs`). Until strategy 2 lands, running plain +> `griff manifest` over a curated corpus would **silently erase** the curated +> `songs` cross-check. The first implementation must therefore treat the curated +> corpus as tool-owned output and **document that ordinary `griff manifest` must +> not be run against it** until strategy 2 exists. Strategy 3 (curated manifest at +> a distinct path + ordinary generation *refuses to overwrite* it) is the +> alternative interim guard if reviewers prefer a hard stop over documentation. + +### 5.7 Validate + +Validation **uses the existing core `song_holdout_preflight`**, not a near-copy. +It also reports: + +``` +unique_source_count +labelled_source_count +unlabelled_source_count +song_count +conflicting_source_count +manifest_disagreement_count +holdout_ready: true | false +``` + +`holdout_ready: true` is permitted **only** when the complete corpus passes the +existing strict preflight (`song_holdout_preflight(&manifest) == Ok(())`). + +## 6. Partial curation + +Incremental curation is **supported** — requiring 100% completion before the +first confirmed decision would guarantee nobody ever curates. But: + +- partial output is explicitly marked **`holdout_ready: false`**; +- it is **not** described as a valid song-holdout corpus; +- uncurated sources remain `song_id: None`; +- a partial run **never** invents an "unknown" shared `SongId`; +- the complete gate remains `song_holdout_preflight == Ok(())`. + +Two states are **distinct** and must never be conflated: + +- a **valid partial curation snapshot** (`holdout_ready: false`, some sources + still `None`); +- a **complete holdout-ready corpus** (`holdout_ready: true`, strict preflight + passes). + +## 7. `SongId` issuance + +**Selected policy:** an **opaque, ledger-issued identifier** — +`song-` + a zero-padded monotonic counter maintained in the decisions ledger +(e.g. `song-000042`) — issued **once** at human confirmation and recorded in the +ledger. Required properties, all satisfied: + +- opaque and **non-semantic** (the number carries no meaning); +- issued **once** upon confirmation; **persisted** in the ledger; +- **never recomputed** from title, filename, membership, or source hashes; +- adding another manifestation to a song does **not** change its `SongId`; +- a rename or corrected title does **not** change it. + +A **title-derived slug** and a **hash of the current membership set** are both +**rejected** as canonical identities (both violate stability under +rename / added-manifestation). The encoding is filesystem-safe and JSON-stable. + +> The monotonic counter assumes a **single authoritative ledger**. If concurrent +> issuance across branches ever matters, a random **ULID/UUID** token is the +> drop-in alternative (still opaque, still ledger-recorded, never recomputed). +> Left as an open question for review; v1 assumes one ledger. + +## 8. Versioned artifacts (v1 schemas) + +All artifacts carry `schema`, `policy_id`, `policy_version` where applicable and +refer to sources by **exact `sha256`**. JSON, deterministically rendered. + +### 8.1 Suggestion artifact + +```json +{ + "schema": "song-curation.suggestions.v1", + "policy_id": "metadata-only", + "policy_version": "1", + "corpus_fingerprint": "", + "sources": [ { "source_sha256": "…", "filenames": ["…"], "titles": ["…"] } ], + "suggested_groups": [ + { "candidate_id": "g1", "source_sha256s": ["…","…"], "confidence": "low|medium|high" } + ], + "evidence": [ + { "candidate_id": "g1", "signals": ["normalized_title_match", "filename_stem_match"], "note": "…" } + ], + "warnings": ["artist parsed from title is heuristic, not structured provenance"] +} +``` + +Every suggested group references exact source hashes; the artifact **never** +contains a `song_id`. + +### 8.2 Decisions artifact (the ledger) + +```json +{ + "schema": "song-curation.decisions.v1", + "corpus_fingerprint": "", + "curator": "…", + "decided_at": "2026-07-29T00:00:00Z", + "next_song_seq": 43, + "decisions": [ + { + "song_id": "song-000042", + "source_sha256s": ["…","…"], + "action": "define | accept | merge | split | correct", + "note": "optional" + } + ] +} +``` + +- `next_song_seq` persists the issuance counter (§7). +- **Correction / merge semantics (no invisible rewrites):** a `correct` or + `merge` decision is a **new, appended** decision referencing the prior + `song_id`(s) it supersedes; earlier decisions are **never** edited in place. + Apply replays decisions in order; the latest decision for a `sha256` wins, and + a label change from a non-`None` value requires an explicit `correct` + (`ExistingLabelReplacementNotAuthorized` otherwise). The ledger is thus an + append-only audit trail. + +### 8.3 Application report (evidence, not authority) + +```json +{ + "schema": "song-curation.apply-report.v1", + "input_corpus_fingerprint": "", + "decisions_digest": "", + "output_corpus_fingerprint": "", + "assignments_applied": 0, + "assignments_unchanged": 0, + "coverage": { "unique_sources": 0, "labelled": 0, "unlabelled": 0, "songs": 0 }, + "refusals": [ { "kind": "…", "source_sha256": "…" } ], + "holdout_ready": false +} +``` + +The report is **evidence**, never a second authority for `song_id`. + +## 9. Determinism and the corpus fingerprint + +**Selected algorithm.** The corpus fingerprint is `source_sha256` (griff-core's +lowercase-hex SHA-256) of a canonical blob built as follows, so directory +timestamps and traversal order cannot affect it: + +1. For every chunk: the line `chunk\t\t\t\n`. +2. For every `CorpusManifest.songs` entry: the line + `song\t\t\n`. +3. Sort **all** lines as UTF-8 byte strings; concatenate; hash. + +This detects **added/removed chunks**, **changed source hashes**, **changed +existing labels**, and **changed manifest membership** — exactly the drifts that +must invalidate a plan. Suggestion and plan output are **byte-deterministic** for +the same inputs and policy version. + +## 10. Refusal taxonomy (typed, defined before implementation) + +- `UnidentifiedSource` — a chunk / source has no `sha256`. +- `ConflictingExistingSongIds` — one `sha256` carries more than one `song_id`. +- `UnknownDecisionSource` — a decision names a `sha256` absent from the corpus. +- `SourceAssignedToMultipleSongs` — one `sha256` assigned to two `SongId`s. +- `DecisionCorpusFingerprintMismatch` — decisions were made against a different + snapshot. +- `PlanCorpusFingerprintMismatch` — the corpus changed between plan and apply. +- `ExistingLabelReplacementNotAuthorized` — a non-`None` label would change + without an explicit `correct` decision. +- `ManifestDisagreement` — `CorpusManifest.songs` disagrees with the per-source + labels (propagated from the core preflight where applicable). +- `IncompleteCoverage` — not every participating source is labelled. +- `OutputWouldModifyInput` — the output path resolves inside/equal to an input. +- `OutputAlreadyExists` — the output path already exists. + +`IncompleteCoverage` is a **validation status** during partial curation, but +becomes a **refusal** when `--require-holdout-ready` (or equivalent) is set. + +## 11. Implementation sequence after acceptance (separate RED→GREEN slices) + +1. **Decision & validation core** — parse/validate a human-authored decisions + artifact; inventory sources by `sha256`; construct a deterministic **dry-run** + plan; **no** suggestions; **no** corpus writes. +2. **Transactional application** — apply a validated plan to a **fresh** output + tree; update every chunk of each source; generate the deterministic `songs` + map; produce the application report; prove **idempotence** and **no partial + writes**. +3. **Suggestion generator** — deterministic **metadata-only** suggestions; + evidence-rich output; **no** write path and **no** implicit acceptance. +4. **Controlled corpus pilot** — only after **independent acceptance** of slices + 1–3; operate on a small **copied subset**; human-confirm every assignment; + verify snapshot and manifest; run `song_holdout_preflight`; **no full-corpus + labeling** until the pilot is independently accepted. + +## 12. Explicit non-goals + +Automatic song assignment; musical similarity or cover detection; embeddings, +classifiers, LLM decisions, or audio analysis; production scoring; generation +changes; cockpit integration; silent repair of existing inconsistent labels; +in-place mutation; full-corpus labeling; track-index recovery; arrangement +identity; any change to ADR-0031 or ADR-0032. + +## 13. Governance path + +This proposal PR is **discussion only**. After independent review: + +1. accepted durable decisions move into a **new ADR (provisionally ADR-0033)**; +2. this proposal becomes historical context; +3. implementation starts **from the accepted ADR** as the separate RED→GREEN + slices above; +4. **corpus labeling remains prohibited** until the implementation and + controlled-pilot gates are explicitly opened. + +## 14. Selected choices and open questions (for the reviewer) + +**Selected in this draft (not left to implementation):** + +- **Owner:** a standalone isolated `song-curation/` tool (§4), over extending + `griff curate` / `griff manifest`. +- **Manifest strategy:** first implementation = tool exclusively owns song-aware + manifest generation (strategy 1); durable end state = extend `griff manifest` + (strategy 2, later, separately accepted). §5.6. +- **`SongId` encoding:** opaque ledger-issued `song-` (§7). +- **Fingerprint:** SHA-256 over sorted domain-tagged chunk + songs lines (§9). +- **Suggestion policy v1:** metadata-only, evidence-bearing, artist-as-heuristic + (§5.2). + +**Left open for review:** + +- Monotonic counter vs ULID/UUID for `SongId` under possible concurrent + issuance (§7). +- Interim manifest-erasure guard: documentation (strategy 1) vs a hard + refuse-to-overwrite at a distinct path (strategy 3) until strategy 2 lands + (§5.6). +- Whether the decisions ledger is one file or a JSONL append log (both satisfy + append-only; the schema in §8.2 is shown as a single document). +- The exact `strip_version_suffix` reuse vs a curation-specific normalization + rule for suggestions (§5.2). From d25a662096560b51953aae272d3bfa5ffb3bb1ca Mon Sep 17 00:00:00 2001 From: PhysShell Date: Wed, 29 Jul 2026 10:07:48 +0000 Subject: [PATCH 2/4] docs(proposal): close song-curation artifact and integrity contracts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address the five blocking findings; still docs-only, Status: for discussion. 1. Decisions ledger is now an append-only per-event JSON document following the CurationStoreV1 precedent: each event carries its own event_id, curator, occurred_at, and corpus_fingerprint (per-event fingerprints let a later event bind to its own post-apply snapshot honestly). `action` is a tagged union — accept / reject / manual_define / split / merge / correct — each with exact source hashes and supersedes_song_ids; a reject records the reviewed candidate + hashes so "reviewed and rejected" stays distinct from "never reviewed". 2. Added the serialized, digest-verified plan artifact (song-curation.plan.v1) as the only thing Apply consumes: assignments with expected_existing_song_id and affected_chunk_ids, generated_songs_map, and a plan_digest Apply recomputes (plan digest, corpus fingerprint, decisions digest, bindings, prior labels). 3. Fingerprint is now injective canonical compact-JSON records with a manifest_songs absent/present marker, so None vs Some({}) differ and tab/newline in ChunkId/SongId cannot collide; same scheme defines plan_digest / decisions_digest. Notes the existing core corpus_fingerprint and why a label-aware one is needed. 4. Manifest guard is now hard: the tool writes a distinct curated path and refuses ordinary /manifest.json; ordinary griff manifest cannot overwrite it; strategy 2 later. Corrected the builder citation to cli/src/main.rs:1998 (shared seam ui-core/src/corpus.rs:18). 5. Resolved the self-contradictions: §14 lists closed v1 choices only — census strip_version_suffix reused exactly, monotonic single-writer SongId (concurrent issuance out of scope, not open), JSON per-event ledger. Also tagged the untyped fenced blocks with `text`. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_012T7SRMiXmZe5v1imtrKMPi --- docs/proposals/song-id-curation-workflow.md | 344 ++++++++++++-------- 1 file changed, 214 insertions(+), 130 deletions(-) diff --git a/docs/proposals/song-id-curation-workflow.md b/docs/proposals/song-id-curation-workflow.md index 50333f69..e884a85a 100644 --- a/docs/proposals/song-id-curation-workflow.md +++ b/docs/proposals/song-id-curation-workflow.md @@ -34,7 +34,7 @@ is a *separate* future `arrangement_id` (a lower FRBR level), out of scope here A pipeline that can: -``` +```text inventory sources → generate non-authoritative grouping suggestions → record explicit curator decisions @@ -74,16 +74,17 @@ are **display / suggestion evidence** only; they do not define source identity. - **no** in-place corpus mutation; - **no** automatic execution in ordinary generation; - `griff-core` supplies only reusable **schema and validation contracts** - (`ChunkMeta`, `SourceRef`, `CorpusManifest`, `SongId`, `source_sha256`, and the - existing `song_holdout_preflight`). + (`ChunkMeta`, `SourceRef`, `CorpusManifest`, `SongId`, `source_sha256`, the + existing `song_holdout_preflight`, and the `CurationStoreV1` event/ledger + precedent in `core/src/curation_store.rs`). ### Comparison (required) | Owner | For | Against | Verdict | |---|---|---|---| -| **Standalone `song-curation/`** (selected) | Matches the isolated-tool precedent; a multi-phase ledger/plan/apply workflow is too large for a subcommand; no production surface acquires curation policy; can carry its own artifact schemas. | One more isolated crate; not exercised by workspace CI (per the isolation policy, verified locally). | **Selected.** | +| **Standalone `song-curation/`** (selected) | Matches the isolated-tool precedent; a multi-phase ledger/plan/apply workflow is too large for a subcommand; no production surface acquires curation policy; carries its own artifact schemas. | One more isolated crate; not exercised by workspace CI (per the isolation policy, verified locally). | **Selected.** | | Extend `griff curate` | Reuses an existing curation entry point. | `griff curate` is interactive per-chunk cockpit-adjacent curation; song identity is a *source-level, ledgered, transactional* operation with a fingerprint/plan/apply contract that does not fit an interactive per-chunk command, and it would pull curation policy into a production binary. | Rejected. | -| Extend `griff manifest` | Manifest generation already emits `CorpusManifest`. | `griff manifest` folds chunks into a manifest; it is not a curation ledger and must stay a pure projection. Owning issuance/decisions there overloads it and adds policy to production. | Rejected for *ownership* (but see §9 for its eventual manifest role). | +| Extend `griff manifest` | Manifest generation already emits `CorpusManifest`. | `griff manifest` folds chunks into a manifest; it is a pure projection, not a curation ledger. Owning issuance/decisions there overloads it and adds policy to production. | Rejected for *ownership* (but see §5.6 for its eventual manifest role). | Ownership is **selected now**, not deferred to implementation. @@ -94,19 +95,19 @@ Ownership is **selected now**, not deferred to implementation. Read a corpus snapshot and **collapse chunks by exact source `sha256`** into deterministic source records with at least: -``` +```text source_sha256 -filenames # sorted, unique -titles # sorted, unique -formats # sorted, unique -chunk_ids # sorted -existing_song_ids # sorted, unique (from SourceRef.song_id) +filenames # sorted, unique +titles # sorted, unique +formats # sorted, unique +chunk_ids # sorted +existing_song_ids # sorted, unique (from SourceRef.song_id) existing_manifest_membership # SongIds naming this sha256 in CorpusManifest.songs ``` A source with **more than one** distinct existing `song_id` across its chunks → -`ConflictingExistingSongIds` (read-only refuses to summarize it as clean). A -chunk without `sha256` → `UnidentifiedSource`. Inventory writes nothing. +`ConflictingExistingSongIds`. A chunk without `sha256` → `UnidentifiedSource`. +Inventory writes nothing. ### 5.2 Suggest (non-authoritative) @@ -115,16 +116,16 @@ metadata evidence only:** - normalized `ChunkMeta.title`; - normalized source filenames / stems; -- repeated source names with format/version suffixes removed by a **documented** - rule (reuse the census's `strip_version_suffix` convention: trailing `(...)` - removed only when the inner text starts with `ver`, contains ` by `, or is - all-digits — never for e.g. `(Reprise)`); +- repeated source names with format/version suffixes removed by the **census + `strip_version_suffix` rule, reused exactly** (trailing `(...)` removed only + when the inner text starts with `ver`, contains ` by `, or is all-digits — + never for e.g. `(Reprise)`); - already-confirmed identity relationships, when supplied. **No canonical artist field exists in the schema.** "Artist" parsed from a title or filename is a **suggestion heuristic**, reported as such in `evidence`; it is -not structured provenance. If an artist signal is ever wanted as structured -input, it must arrive as a **separate explicit input artifact**, not inferred. +not structured provenance. A structured artist signal, if ever wanted, must +arrive as a **separate explicit input artifact**, not inferred. **No** note-content similarity, embeddings, audio fingerprinting, cover detection, or MIR classifier enters version 1. Every suggested group exposes its @@ -135,29 +136,33 @@ writes `song_id`**. A curator explicitly **accepts / rejects / splits / merges / manually defines** source groupings. **No default action means acceptance.** A batch or unattended -run must **not** convert suggestions into decisions. Confirmation is captured in -the **decisions artifact** (§8) such that a later apply run can prove **every -stored label came from an explicit curator decision** (each decision names the -`source_sha256`s, the curator, and a timestamp; the apply report echoes the -decisions digest). +run must **not** convert suggestions into decisions. Confirmation is captured as +**append-only events** in the decisions ledger (§8.2), each naming its curator, +timestamp, corpus fingerprint, the exact `source_sha256`s, and — for a reviewed +suggestion — the `candidate_id`, so a later apply run can prove **every stored +label came from an explicit curator decision**, and "reviewed and rejected" +stays distinguishable from "never reviewed". ### 5.4 Plan (immutable) -Build an **immutable application plan** from: +Build an **immutable, serialized application plan** (§8.3) from: -- an exact **corpus fingerprint** (§10); -- a **versioned decisions artifact**; +- an exact **corpus fingerprint** (§9); +- a **versioned decisions ledger** (its digest); - the **tool-policy version**. The plan enumerates **every** intended source-level assignment and **every** -affected chunk. If the corpus fingerprint has changed since inventory/confirm, -planning or application **typed-refuses** +affected chunk, and carries a `plan_digest`. If the corpus fingerprint has +changed since inventory/confirm, planning or application **typed-refuses** (`DecisionCorpusFingerprintMismatch` / `PlanCorpusFingerprintMismatch`) rather than rebasing decisions onto a different snapshot. ### 5.5 Apply (transactional) -Apply **only** a previously validated plan. Required semantics: +Apply **only** a previously validated plan. Before writing, Apply **recomputes +and verifies**: (1) the `plan_digest`; (2) the current corpus fingerprint against +the plan's `input_corpus_fingerprint`; (3) the `decisions_digest`; (4) every +source→chunk binding; (5) every `expected_existing_song_id`. Then: - write to a **fresh output directory** (`OutputAlreadyExists` if it exists; `OutputWouldModifyInput` if it resolves inside/equal to an input — reuse the @@ -169,40 +174,45 @@ Apply **only** a previously validated plan. Required semantics: - **deterministic** file order and JSON rendering; - **idempotent** for already-correct labels; - `ConflictingExistingSongIds` / `SourceAssignedToMultipleSongs` / - `UnknownDecisionSource` typed-refuse; + `UnknownDecisionSource` / `PlanDigestMismatch` typed-refuse; - **never clear** an existing label unless an explicit curator **correction** - authorizes it (`ExistingLabelReplacementNotAuthorized` otherwise). + authorizes it (`ExistingLabelReplacementNotAuthorized` otherwise; the plan's + `expected_existing_song_id` is how the authorization is checked). A **correction is a new curator decision**, not heuristic reconciliation. -### 5.6 Generate manifest (deterministic) +### 5.6 Generate manifest (deterministic) — hard distinct-path guard Generate `CorpusManifest.songs` deterministically from the applied per-source labels: `SongId → sorted, unique [sha256]`. The per-source `SourceRef.song_id` remains **authoritative**; the map is the cross-check (law 8). -**Manifest ownership — selected:** for the **first implementation**, the -`song-curation/` tool **exclusively** owns song-aware manifest generation -(strategy 1), kept isolated until the workflow is proven. The **recommended -durable end state** is **strategy 2**: extend `griff manifest` to rebuild `songs` -from the authoritative per-source labels, as a **later, separately-accepted** -change. - -> **Interim hazard (open issue for review).** `griff manifest` today always emits -> `songs: None` (`ui-core/src/corpus.rs`). Until strategy 2 lands, running plain -> `griff manifest` over a curated corpus would **silently erase** the curated -> `songs` cross-check. The first implementation must therefore treat the curated -> corpus as tool-owned output and **document that ordinary `griff manifest` must -> not be run against it** until strategy 2 exists. Strategy 3 (curated manifest at -> a distinct path + ordinary generation *refuses to overwrite* it) is the -> alternative interim guard if reviewers prefer a hard stop over documentation. +**Selected interim contract (hard, not documentation).** Ordinary `griff +manifest` today always emits `songs: None` — the command builder in +`cli/src/main.rs:1998` constructs `CorpusManifest { …, songs: None }` (its shared +seam is `ui-core/src/corpus.rs:18`). A warning is not a data-integrity mechanism, +so the first implementation enforces a hard guard: + +- the `song-curation/` tool writes the curated manifest to a **distinct canonical + path** (e.g. `song-curation/manifest.json`); +- it **refuses** to target an ordinary `/manifest.json`; +- the **application report** records the curated manifest path and its digest; +- the controlled pilot (§11) consumes **that explicit path and digest**; +- ordinary `griff manifest` may create its normal manifest, but **cannot + overwrite** the curated artifact (they live at different paths); +- **later (strategy 2, separately accepted):** teach `griff manifest` to rebuild + `songs` from the authoritative per-source labels, at which point the distinct + path can be retired. + +This is fail-closed: no ordinary manifest rebuild can silently erase the curated +cross-check, because it never writes to the curated path. ### 5.7 Validate Validation **uses the existing core `song_holdout_preflight`**, not a near-copy. It also reports: -``` +```text unique_source_count labelled_source_count unlabelled_source_count @@ -226,19 +236,15 @@ first confirmed decision would guarantee nobody ever curates. But: - a partial run **never** invents an "unknown" shared `SongId`; - the complete gate remains `song_holdout_preflight == Ok(())`. -Two states are **distinct** and must never be conflated: - -- a **valid partial curation snapshot** (`holdout_ready: false`, some sources - still `None`); -- a **complete holdout-ready corpus** (`holdout_ready: true`, strict preflight - passes). +Two states are **distinct** and must never be conflated: a **valid partial +curation snapshot** (`holdout_ready: false`, some sources still `None`) and a +**complete holdout-ready corpus** (`holdout_ready: true`, strict preflight passes). -## 7. `SongId` issuance +## 7. `SongId` issuance (selected, closed for v1) -**Selected policy:** an **opaque, ledger-issued identifier** — -`song-` + a zero-padded monotonic counter maintained in the decisions ledger -(e.g. `song-000042`) — issued **once** at human confirmation and recorded in the -ledger. Required properties, all satisfied: +An **opaque, ledger-issued identifier** — `song-` + a zero-padded monotonic +counter maintained in the decisions ledger (e.g. `song-000042`) — issued **once** +at human confirmation and recorded in the ledger. Properties, all satisfied: - opaque and **non-semantic** (the number carries no meaning); - issued **once** upon confirmation; **persisted** in the ledger; @@ -247,18 +253,17 @@ ledger. Required properties, all satisfied: - a rename or corrected title does **not** change it. A **title-derived slug** and a **hash of the current membership set** are both -**rejected** as canonical identities (both violate stability under -rename / added-manifestation). The encoding is filesystem-safe and JSON-stable. - -> The monotonic counter assumes a **single authoritative ledger**. If concurrent -> issuance across branches ever matters, a random **ULID/UUID** token is the -> drop-in alternative (still opaque, still ledger-recorded, never recomputed). -> Left as an open question for review; v1 assumes one ledger. +**rejected** as canonical identities. The encoding is filesystem-safe and +JSON-stable. **v1 assumes a single authoritative single-writer ledger; concurrent +issuance across writers is explicitly out of scope for v1** (a random ULID/UUID +would be the drop-in encoding if that ever changes — a future policy-version bump, +not an open v1 alternative). ## 8. Versioned artifacts (v1 schemas) -All artifacts carry `schema`, `policy_id`, `policy_version` where applicable and -refer to sources by **exact `sha256`**. JSON, deterministically rendered. +All artifacts are deterministically-rendered JSON, refer to sources by exact +`sha256`, and carry `schema` (and `policy_id` / `policy_version` where a policy +is involved). ### 8.1 Suggestion artifact @@ -273,7 +278,7 @@ refer to sources by **exact `sha256`**. JSON, deterministically rendered. { "candidate_id": "g1", "source_sha256s": ["…","…"], "confidence": "low|medium|high" } ], "evidence": [ - { "candidate_id": "g1", "signals": ["normalized_title_match", "filename_stem_match"], "note": "…" } + { "candidate_id": "g1", "signals": ["normalized_title_match","filename_stem_match"], "note": "…" } ], "warnings": ["artist parsed from title is heuristic, not structured provenance"] } @@ -282,43 +287,102 @@ refer to sources by **exact `sha256`**. JSON, deterministically rendered. Every suggested group references exact source hashes; the artifact **never** contains a `song_id`. -### 8.2 Decisions artifact (the ledger) +### 8.2 Decisions ledger — append-only per-event, following `CurationStoreV1` + +One versioned JSON document (not JSONL) with an **append-only `events` array**, +matching the `core/src/curation_store.rs` `CurationStoreV1` / `CurationEvent` +precedent — where each event carries its **own** `event_id`, `curator`, +`occurred_at`, and `corpus_fingerprint`. Per-event fingerprints are load-bearing: +after a partial apply the corpus fingerprint changes, so a later event must bind +to **its own** snapshot; an envelope-only fingerprint could not be appended to +honestly (it would either misbind the new event or retroactively rebind old ones). ```json { "schema": "song-curation.decisions.v1", - "corpus_fingerprint": "", - "curator": "…", - "decided_at": "2026-07-29T00:00:00Z", + "created_corpus_fingerprint": "", "next_song_seq": 43, - "decisions": [ + "events": [ { - "song_id": "song-000042", - "source_sha256s": ["…","…"], - "action": "define | accept | merge | split | correct", - "note": "optional" + "event_id": "ev-000017", + "curator": "…", + "occurred_at": "2026-07-29T00:00:00Z", + "corpus_fingerprint": "", + "note": "optional", + "action": { + "kind": "accept_suggestion", + "candidate_id": "g1", + "source_sha256s": ["…","…"], + "assign_song_id": "song-000042", + "supersedes_song_ids": [] + } } ] } ``` -- `next_song_seq` persists the issuance counter (§7). -- **Correction / merge semantics (no invisible rewrites):** a `correct` or - `merge` decision is a **new, appended** decision referencing the prior - `song_id`(s) it supersedes; earlier decisions are **never** edited in place. - Apply replays decisions in order; the latest decision for a `sha256` wins, and - a label change from a non-`None` value requires an explicit `correct` - (`ExistingLabelReplacementNotAuthorized` otherwise). The ledger is thus an - append-only audit trail. +The envelope's `created_corpus_fingerprint` is informational (the store's first +snapshot); the **authoritative** fingerprint for each decision is the one on its +event. `next_song_seq` persists the issuance counter (§7). -### 8.3 Application report (evidence, not authority) +**`action` is a tagged union** (not one string plus fields meaningless for half +the variants). Every variant carries the resulting `source_sha256s` (exact), +`supersedes_song_ids` (possibly empty), and the assignment(s) it produces: + +- `accept_suggestion { candidate_id, source_sha256s, assign_song_id, supersedes_song_ids }` +- `reject_suggestion { candidate_id, reviewed_source_sha256s, reason? }` — records + a review that produced **no** assignment, so it stays distinct from "never + reviewed"; +- `manual_define { source_sha256s, assign_song_id }` +- `split { from_song_id, into: [ { assign_song_id, source_sha256s } … ], supersedes_song_ids: [from_song_id] }` +- `merge { from_song_ids, into_song_id, source_sha256s, supersedes_song_ids: from_song_ids }` +- `correct { source_sha256s, new_song_id, supersedes_song_ids }` — the only + variant permitted to change a non-`None` label. + +**No invisible rewrites:** events are immutable and append-only; a `correct` / +`merge` / `split` is a **new** event referencing the `song_id`(s) it supersedes. +Apply replays events in order; the latest event for a `sha256` wins; a change from +a non-`None` label requires a `correct` (`ExistingLabelReplacementNotAuthorized` +otherwise). + +### 8.3 Plan artifact (the only thing Apply consumes) + +```json +{ + "schema": "song-curation.plan.v1", + "policy_id": "…", + "policy_version": "1", + "input_corpus_fingerprint": "", + "decisions_digest": "", + "plan_digest": "", + "assignments": [ + { + "source_sha256": "…", + "song_id": "song-000042", + "expected_existing_song_id": null, + "affected_chunk_ids": ["…","…"] + } + ], + "generated_songs_map": { "song-000042": ["", "…"] } +} +``` + +`plan_digest` is computed over the canonical plan bytes **with the `plan_digest` +field omitted** (§9 record encoding). Apply verifies it (blocker: "apply only a +validated plan" is otherwise decorative). `expected_existing_song_id` is how a +label change is authorized: Apply refuses if the on-disk label differs from it. + +### 8.4 Application report (evidence, not authority) ```json { "schema": "song-curation.apply-report.v1", "input_corpus_fingerprint": "", "decisions_digest": "", + "plan_digest": "", "output_corpus_fingerprint": "", + "curated_manifest_path": "song-curation/manifest.json", + "curated_manifest_digest": "", "assignments_applied": 0, "assignments_unchanged": 0, "coverage": { "unique_sources": 0, "labelled": 0, "unlabelled": 0, "songs": 0 }, @@ -329,21 +393,35 @@ contains a `song_id`. The report is **evidence**, never a second authority for `song_id`. -## 9. Determinism and the corpus fingerprint +## 9. Determinism and the corpus fingerprint (injective) + +The core already has `corpus_fingerprint()` (`core/src/curation_store.rs:228`), +but it deliberately hashes each chunk's **material** identity and excludes mutable +curation fields — so it cannot detect a `song_id` or manifest-membership change, +which is exactly what a curation plan must be invalidated by. This workflow +therefore defines its **own** fingerprint over the label-bearing inputs. -**Selected algorithm.** The corpus fingerprint is `source_sha256` (griff-core's -lowercase-hex SHA-256) of a canonical blob built as follows, so directory -timestamps and traversal order cannot affect it: +**Encoding (injective, canonical).** Build these records, one per item, as +**compact UTF-8 JSON arrays** (JSON escaping makes the encoding injective — +`ChunkId` and `SongId` are unrestricted `String` and may contain tabs or +newlines, which an ad-hoc separator could not survive): -1. For every chunk: the line `chunk\t\t\t\n`. -2. For every `CorpusManifest.songs` entry: the line - `song\t\t\n`. -3. Sort **all** lines as UTF-8 byte strings; concatenate; hash. +```json +["manifest_songs", "absent"] // when CorpusManifest.songs is None +["manifest_songs", "present"] // when Some(...) — even empty {} +["chunk", "", "", ""] +["song", "", ["", "…"]] // one per songs-map entry +``` -This detects **added/removed chunks**, **changed source hashes**, **changed -existing labels**, and **changed manifest membership** — exactly the drifts that -must invalidate a plan. Suggestion and plan output are **byte-deterministic** for -the same inputs and policy version. +Sort the record **bytes** as UTF-8 strings, join with `\n`, and `source_sha256` +the result. This detects added/removed chunks, changed source hashes, changed +existing labels, and changed manifest membership; directory timestamps and +traversal order cannot affect it; and the `manifest_songs` presence record +distinguishes **absent** (`None`, skips cross-check) from **present-empty** +(`Some({})`, must account for every labelled source) — a distinction core makes +deliberately. Suggestion, plan, and fingerprint output are byte-deterministic for +the same inputs and policy version. The same compact-JSON-record + sort + hash +scheme defines `plan_digest` and `decisions_digest`. ## 10. Refusal taxonomy (typed, defined before implementation) @@ -352,10 +430,11 @@ the same inputs and policy version. - `UnknownDecisionSource` — a decision names a `sha256` absent from the corpus. - `SourceAssignedToMultipleSongs` — one `sha256` assigned to two `SongId`s. - `DecisionCorpusFingerprintMismatch` — decisions were made against a different - snapshot. + snapshot (per-event fingerprint mismatch). - `PlanCorpusFingerprintMismatch` — the corpus changed between plan and apply. +- `PlanDigestMismatch` — the plan bytes do not match `plan_digest`. - `ExistingLabelReplacementNotAuthorized` — a non-`None` label would change - without an explicit `correct` decision. + without an explicit `correct` (checked via `expected_existing_song_id`). - `ManifestDisagreement` — `CorpusManifest.songs` disagrees with the per-source labels (propagated from the core preflight where applicable). - `IncompleteCoverage` — not every participating source is labelled. @@ -368,18 +447,20 @@ becomes a **refusal** when `--require-holdout-ready` (or equivalent) is set. ## 11. Implementation sequence after acceptance (separate RED→GREEN slices) 1. **Decision & validation core** — parse/validate a human-authored decisions - artifact; inventory sources by `sha256`; construct a deterministic **dry-run** - plan; **no** suggestions; **no** corpus writes. + ledger; inventory sources by `sha256`; construct the deterministic serialized + **dry-run plan** (§8.3) and verify its digest; **no** suggestions; **no** + corpus writes. 2. **Transactional application** — apply a validated plan to a **fresh** output tree; update every chunk of each source; generate the deterministic `songs` - map; produce the application report; prove **idempotence** and **no partial - writes**. + map at the distinct curated path; produce the application report; prove + **idempotence** and **no partial writes**. 3. **Suggestion generator** — deterministic **metadata-only** suggestions; evidence-rich output; **no** write path and **no** implicit acceptance. 4. **Controlled corpus pilot** — only after **independent acceptance** of slices 1–3; operate on a small **copied subset**; human-confirm every assignment; - verify snapshot and manifest; run `song_holdout_preflight`; **no full-corpus - labeling** until the pilot is independently accepted. + verify snapshot and the curated manifest by digest; run + `song_holdout_preflight`; **no full-corpus labeling** until the pilot is + independently accepted. ## 12. Explicit non-goals @@ -400,29 +481,32 @@ This proposal PR is **discussion only**. After independent review: 4. **corpus labeling remains prohibited** until the implementation and controlled-pilot gates are explicitly opened. -## 14. Selected choices and open questions (for the reviewer) +## 14. Closed v1 choices (no architecture lottery for the implementer) -**Selected in this draft (not left to implementation):** +Every choice below is **selected for v1**, not left open: -- **Owner:** a standalone isolated `song-curation/` tool (§4), over extending - `griff curate` / `griff manifest`. -- **Manifest strategy:** first implementation = tool exclusively owns song-aware - manifest generation (strategy 1); durable end state = extend `griff manifest` - (strategy 2, later, separately accepted). §5.6. +- **Owner:** standalone isolated `song-curation/` tool (§4). +- **Ledger format:** one versioned JSON document with immutable, append-only + **per-event** records following `CurationStoreV1` (§8.2) — **not** JSONL, **not** + an envelope-only fingerprint. +- **Action model:** a **tagged `action` union** covering accept / reject / manual + define / split / merge / correct, each with exact hashes and + `supersedes_song_ids` (§8.2). +- **Plan:** a serialized, digest-verified `song-curation.plan.v1` artifact is the + **only** thing Apply consumes (§8.3, §5.5). - **`SongId` encoding:** opaque ledger-issued `song-` (§7). -- **Fingerprint:** SHA-256 over sorted domain-tagged chunk + songs lines (§9). -- **Suggestion policy v1:** metadata-only, evidence-bearing, artist-as-heuristic - (§5.2). - -**Left open for review:** - -- Monotonic counter vs ULID/UUID for `SongId` under possible concurrent - issuance (§7). -- Interim manifest-erasure guard: documentation (strategy 1) vs a hard - refuse-to-overwrite at a distinct path (strategy 3) until strategy 2 lands + counter>`, single-writer ledger; concurrent issuance **out of scope for v1** + (§7). +- **Fingerprint / digests:** SHA-256 over sorted **compact-JSON records** with a + `manifest_songs` absent/present marker (§9) — injective, presence-aware; the + same scheme defines `plan_digest` and `decisions_digest`. +- **Manifest guard:** the tool writes to a **distinct curated path** and refuses + ordinary `/manifest.json`; ordinary `griff manifest` cannot overwrite + it; strategy 2 (extend `griff manifest`) is a later, separately-accepted change (§5.6). -- Whether the decisions ledger is one file or a JSONL append log (both satisfy - append-only; the schema in §8.2 is shown as a single document). -- The exact `strip_version_suffix` reuse vs a curation-specific normalization - rule for suggestions (§5.2). +- **Suggestion normalization:** reuse the census `strip_version_suffix` rule + **exactly** for v1; any later divergence requires a policy-version bump (§5.2). + +Genuinely deferred (out of scope for v1, **not** unresolved alternatives): +multi-writer/concurrent `SongId` issuance; the strategy-2 `griff manifest` +extension; any non-metadata suggestion signal. From f8cd5c73070e276b126f7563a25f97973b900689 Mon Sep 17 00:00:00 2001 From: PhysShell Date: Wed, 29 Jul 2026 11:47:47 +0000 Subject: [PATCH 3/4] docs(proposal): bind ordered decision batches through plan and apply MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Close the three proof-chain contradictions exposed by the hardened schemas. Still docs-only, Status: for discussion. 1. The plan now EMBEDS the complete ordered decision batch (not just a digest), so Apply can recompute decisions_digest, replay the events, DERIVE the assignments itself, and compare them to the plan's — a digest cannot be recomputed from its own field, and plan_digest alone proves only internal integrity, not derivation from the decisions (DecisionProjectionMismatch on divergence). 2. Incremental curation now has an explicit batch/apply chain: the ledger is append-only BATCHES, each binding its events to one input_corpus_fingerprint; one plan embeds exactly one unapplied batch; historical batches are audit history and are never replayed; batch -> report -> next batch is linked by previous_application_report_digest with input==previous-output fingerprints; reuse of an applied batch refuses (DecisionBatchAlreadyApplied, ApplicationChainMismatch, DecisionBatchFingerprintMismatch). 3. The digests are now three distinct canonicalizations, not "the same scheme": order-insensitive corpus_fingerprint (set-like), order-SENSITIVE decisions_digest (events in exact append order, ordinal included, no sorting), and order-aware plan_digest (embedded batch order preserved, assignments/chunk-ids/map sorted). Updated §5.4/§5.5/§6/§8.2/§8.3/§8.4/§9/§10/§14 accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_012T7SRMiXmZe5v1imtrKMPi --- docs/proposals/song-id-curation-workflow.md | 418 +++++++++++--------- 1 file changed, 232 insertions(+), 186 deletions(-) diff --git a/docs/proposals/song-id-curation-workflow.md b/docs/proposals/song-id-curation-workflow.md index e884a85a..7b65d815 100644 --- a/docs/proposals/song-id-curation-workflow.md +++ b/docs/proposals/song-id-curation-workflow.md @@ -37,15 +37,16 @@ A pipeline that can: ```text inventory sources → generate non-authoritative grouping suggestions - → record explicit curator decisions - → build a deterministic application plan + → record explicit curator decisions (in append-only batches) + → build a deterministic application plan that embeds one unapplied batch → apply that plan transactionally to a fresh corpus copy → generate the songs manifest → validate consistency and holdout readiness ``` The system reduces curation **labour**; it never lets a heuristic promote itself -to a stored identity. +to a stored identity, and the ledger → plan → apply → report forms **one +verifiable chain**, not three documents trading meaningful-looking hashes. ## 3. Curation unit — the source file, by `sha256` @@ -136,48 +137,66 @@ writes `song_id`**. A curator explicitly **accepts / rejects / splits / merges / manually defines** source groupings. **No default action means acceptance.** A batch or unattended -run must **not** convert suggestions into decisions. Confirmation is captured as -**append-only events** in the decisions ledger (§8.2), each naming its curator, -timestamp, corpus fingerprint, the exact `source_sha256`s, and — for a reviewed -suggestion — the `candidate_id`, so a later apply run can prove **every stored -label came from an explicit curator decision**, and "reviewed and rejected" -stays distinguishable from "never reviewed". - -### 5.4 Plan (immutable) - -Build an **immutable, serialized application plan** (§8.3) from: - -- an exact **corpus fingerprint** (§9); -- a **versioned decisions ledger** (its digest); -- the **tool-policy version**. - -The plan enumerates **every** intended source-level assignment and **every** -affected chunk, and carries a `plan_digest`. If the corpus fingerprint has -changed since inventory/confirm, planning or application **typed-refuses** -(`DecisionCorpusFingerprintMismatch` / `PlanCorpusFingerprintMismatch`) rather -than rebasing decisions onto a different snapshot. - -### 5.5 Apply (transactional) - -Apply **only** a previously validated plan. Before writing, Apply **recomputes -and verifies**: (1) the `plan_digest`; (2) the current corpus fingerprint against -the plan's `input_corpus_fingerprint`; (3) the `decisions_digest`; (4) every -source→chunk binding; (5) every `expected_existing_song_id`. Then: - -- write to a **fresh output directory** (`OutputAlreadyExists` if it exists; - `OutputWouldModifyInput` if it resolves inside/equal to an input — reuse the - `migrate-v9` preflight discipline); +run must **not** convert suggestions into decisions. Confirmation appends +immutable **events** to the current **decision batch** (§8.2); every event names +its curator, timestamp, exact `source_sha256`s, and — for a reviewed suggestion — +the `candidate_id`, so a later apply run can prove **every stored label came from +an explicit curator decision**, and "reviewed and rejected" stays distinct from +"never reviewed". + +### 5.4 Plan (immutable, embeds one unapplied batch) + +Build an **immutable, serialized application plan** (§8.3) that **embeds exactly +one unapplied decision batch** — the complete ordered event objects, not just a +digest — so that Apply's decisions come entirely from the plan. The plan also +carries: + +- the batch's `input_corpus_fingerprint` (§9); +- `decisions_digest` — the digest of the embedded, **order-preserving** batch; +- the batch's `previous_application_report_digest` (chaining, §8.4); +- the **derived** `assignments` (every source-level `SongId` and every affected + chunk) and `generated_songs_map`; +- `plan_digest` over the whole plan. + +Planning refuses if the batch's `input_corpus_fingerprint` does not match the +current corpus (`DecisionBatchFingerprintMismatch`), or if the events within one +batch do not all share that fingerprint. + +### 5.5 Apply (transactional, chain-verified) + +Apply consumes the **plan** (which embeds its decision batch — the sole source of +the *decisions* it applies) and, for a **non-initial** batch, the **preceding +application report** (for **chain verification only**, never for decisions). +Before writing, Apply: + +1. recomputes `plan_digest` over the plan (`PlanDigestMismatch` on mismatch); +2. recomputes the embedded batch's `decisions_digest` from its ordered events; +3. verifies the batch `input_corpus_fingerprint` equals the **current** corpus + fingerprint (`PlanCorpusFingerprintMismatch`); +4. verifies the application chain — the batch's + `previous_application_report_digest` equals the digest of the supplied + preceding report, and that report's `output_corpus_fingerprint` equals this + batch's `input_corpus_fingerprint` (`ApplicationChainMismatch`); refuses a + batch already recorded as applied (`DecisionBatchAlreadyApplied`); +5. **replays the embedded events itself**, derives assignments + + `generated_songs_map`, and compares them to the plan's — any divergence is + `DecisionProjectionMismatch` (the plan cannot assert an assignment the + decisions do not produce); +6. checks every `expected_existing_song_id` against the on-disk label. + +Only then does it write: + +- to a **fresh output directory** (`OutputAlreadyExists` / `OutputWouldModifyInput`, + reusing the `migrate-v9` preflight discipline); - **all-or-nothing**; no partial output presented as successful; -- update **every** chunk sharing an assigned `sha256`; -- preserve every unrelated field byte-for-byte where serialization permits - (only `SourceRef.song_id` is added/changed); -- **deterministic** file order and JSON rendering; -- **idempotent** for already-correct labels; -- `ConflictingExistingSongIds` / `SourceAssignedToMultipleSongs` / - `UnknownDecisionSource` / `PlanDigestMismatch` typed-refuse; -- **never clear** an existing label unless an explicit curator **correction** - authorizes it (`ExistingLabelReplacementNotAuthorized` otherwise; the plan's - `expected_existing_song_id` is how the authorization is checked). +- updating **every** chunk sharing an assigned `sha256`; +- preserving every unrelated field byte-for-byte (only `SourceRef.song_id` + changes); +- **deterministic** file order and JSON rendering; **idempotent** for + already-correct labels; +- **never** clearing a non-`None` label unless a `correct` event authorizes it + (`ExistingLabelReplacementNotAuthorized`; the plan's `expected_existing_song_id` + is the check). A **correction is a new curator decision**, not heuristic reconciliation. @@ -199,10 +218,9 @@ so the first implementation enforces a hard guard: - the **application report** records the curated manifest path and its digest; - the controlled pilot (§11) consumes **that explicit path and digest**; - ordinary `griff manifest` may create its normal manifest, but **cannot - overwrite** the curated artifact (they live at different paths); + overwrite** the curated artifact (different paths); - **later (strategy 2, separately accepted):** teach `griff manifest` to rebuild - `songs` from the authoritative per-source labels, at which point the distinct - path can be retired. + `songs` from the authoritative per-source labels, retiring the distinct path. This is fail-closed: no ordinary manifest rebuild can silently erase the curated cross-check, because it never writes to the curated path. @@ -225,20 +243,28 @@ holdout_ready: true | false `holdout_ready: true` is permitted **only** when the complete corpus passes the existing strict preflight (`song_holdout_preflight(&manifest) == Ok(())`). -## 6. Partial curation - -Incremental curation is **supported** — requiring 100% completion before the -first confirmed decision would guarantee nobody ever curates. But: - -- partial output is explicitly marked **`holdout_ready: false`**; -- it is **not** described as a valid song-holdout corpus; -- uncurated sources remain `song_id: None`; -- a partial run **never** invents an "unknown" shared `SongId`; -- the complete gate remains `song_holdout_preflight == Ok(())`. - -Two states are **distinct** and must never be conflated: a **valid partial -curation snapshot** (`holdout_ready: false`, some sources still `None`) and a -**complete holdout-ready corpus** (`holdout_ready: true`, strict preflight passes). +## 6. Partial curation — the batch/apply chain + +Incremental curation is **first-class**: it proceeds as a **chain of batches**, +each applied to the corpus the previous one produced. After two partial passes a +ledger holds a batch against pristine fingerprint `A` (apply → corpus `B`, +report `R_A`) and a later batch against `B` (`previous_application_report_digest += digest(R_A)`). This resolves the "which events belong to this application" +question the flat-replay model could not: + +- every event in one batch binds to the **same** `input_corpus_fingerprint`; +- **one plan embeds exactly one unapplied batch**; +- **historical batches are audit history and are never replayed**; +- the next batch's `input_corpus_fingerprint` must equal the preceding accepted + report's `output_corpus_fingerprint` (`ApplicationChainMismatch` otherwise); +- reuse of an already-applied batch typed-refuses + (`DecisionBatchAlreadyApplied`). + +Partial output is `holdout_ready: false`, is **not** a valid song-holdout corpus, +leaves uncurated sources `song_id: None`, never invents an "unknown" shared +`SongId`, and the complete gate remains `song_holdout_preflight == Ok(())`. A +**valid partial snapshot** and a **complete holdout-ready corpus** are distinct +states, never conflated. ## 7. `SongId` issuance (selected, closed for v1) @@ -246,18 +272,17 @@ An **opaque, ledger-issued identifier** — `song-` + a zero-padded monotonic counter maintained in the decisions ledger (e.g. `song-000042`) — issued **once** at human confirmation and recorded in the ledger. Properties, all satisfied: -- opaque and **non-semantic** (the number carries no meaning); +- opaque and **non-semantic**; - issued **once** upon confirmation; **persisted** in the ledger; - **never recomputed** from title, filename, membership, or source hashes; -- adding another manifestation to a song does **not** change its `SongId`; -- a rename or corrected title does **not** change it. +- adding another manifestation does **not** change its `SongId`; a rename or + corrected title does **not** change it. A **title-derived slug** and a **hash of the current membership set** are both -**rejected** as canonical identities. The encoding is filesystem-safe and -JSON-stable. **v1 assumes a single authoritative single-writer ledger; concurrent -issuance across writers is explicitly out of scope for v1** (a random ULID/UUID -would be the drop-in encoding if that ever changes — a future policy-version bump, -not an open v1 alternative). +**rejected**. The encoding is filesystem-safe and JSON-stable. **v1 assumes a +single authoritative single-writer ledger; concurrent issuance is explicitly out +of scope for v1** (a random ULID/UUID would be the drop-in encoding via a future +policy-version bump — not an open v1 alternative). ## 8. Versioned artifacts (v1 schemas) @@ -287,65 +312,63 @@ is involved). Every suggested group references exact source hashes; the artifact **never** contains a `song_id`. -### 8.2 Decisions ledger — append-only per-event, following `CurationStoreV1` +### 8.2 Decisions ledger — append-only **batches** of immutable events -One versioned JSON document (not JSONL) with an **append-only `events` array**, -matching the `core/src/curation_store.rs` `CurationStoreV1` / `CurationEvent` -precedent — where each event carries its **own** `event_id`, `curator`, -`occurred_at`, and `corpus_fingerprint`. Per-event fingerprints are load-bearing: -after a partial apply the corpus fingerprint changes, so a later event must bind -to **its own** snapshot; an envelope-only fingerprint could not be appended to -honestly (it would either misbind the new event or retroactively rebind old ones). +One versioned JSON document (not JSONL) following the `CurationStoreV1` +event/ledger precedent (`core/src/curation_store.rs`). The ledger is an ordered +list of immutable **batches**; each batch is the unit of one application and +binds all its events to a single `input_corpus_fingerprint`: ```json { "schema": "song-curation.decisions.v1", - "created_corpus_fingerprint": "", "next_song_seq": 43, - "events": [ + "batches": [ { - "event_id": "ev-000017", - "curator": "…", - "occurred_at": "2026-07-29T00:00:00Z", - "corpus_fingerprint": "", - "note": "optional", - "action": { - "kind": "accept_suggestion", - "candidate_id": "g1", - "source_sha256s": ["…","…"], - "assign_song_id": "song-000042", - "supersedes_song_ids": [] - } + "batch_id": "batch-000004", + "input_corpus_fingerprint": "", + "previous_application_report_digest": null, + "events": [ + { + "event_id": "ev-000017", + "ordinal": 0, + "curator": "…", + "occurred_at": "2026-07-29T00:00:00Z", + "note": "optional", + "action": { + "kind": "accept_suggestion", + "candidate_id": "g1", + "source_sha256s": ["…","…"], + "assign_song_id": "song-000042", + "supersedes_song_ids": [] + } + } + ] } ] } ``` -The envelope's `created_corpus_fingerprint` is informational (the store's first -snapshot); the **authoritative** fingerprint for each decision is the one on its -event. `next_song_seq` persists the issuance counter (§7). - -**`action` is a tagged union** (not one string plus fields meaningless for half -the variants). Every variant carries the resulting `source_sha256s` (exact), -`supersedes_song_ids` (possibly empty), and the assignment(s) it produces: - -- `accept_suggestion { candidate_id, source_sha256s, assign_song_id, supersedes_song_ids }` -- `reject_suggestion { candidate_id, reviewed_source_sha256s, reason? }` — records - a review that produced **no** assignment, so it stays distinct from "never - reviewed"; -- `manual_define { source_sha256s, assign_song_id }` -- `split { from_song_id, into: [ { assign_song_id, source_sha256s } … ], supersedes_song_ids: [from_song_id] }` -- `merge { from_song_ids, into_song_id, source_sha256s, supersedes_song_ids: from_song_ids }` -- `correct { source_sha256s, new_song_id, supersedes_song_ids }` — the only - variant permitted to change a non-`None` label. - -**No invisible rewrites:** events are immutable and append-only; a `correct` / -`merge` / `split` is a **new** event referencing the `song_id`(s) it supersedes. -Apply replays events in order; the latest event for a `sha256` wins; a change from -a non-`None` label requires a `correct` (`ExistingLabelReplacementNotAuthorized` -otherwise). - -### 8.3 Plan artifact (the only thing Apply consumes) +- `next_song_seq` persists the issuance counter (§7). +- A batch's `input_corpus_fingerprint` is the fingerprint **all** its events were + made against; `previous_application_report_digest` chains it to the preceding + accepted application report (`null` for the first batch). +- **`action` is a tagged union**; each variant carries exact `source_sha256s`, + `supersedes_song_ids` (possibly empty), and its assignment(s): + - `accept_suggestion { candidate_id, source_sha256s, assign_song_id, supersedes_song_ids }` + - `reject_suggestion { candidate_id, reviewed_source_sha256s, reason? }` — a + review that produced **no** assignment (distinct from "never reviewed"); + - `manual_define { source_sha256s, assign_song_id }` + - `split { from_song_id, into: [ { assign_song_id, source_sha256s } … ], supersedes_song_ids: [from_song_id] }` + - `merge { from_song_ids, into_song_id, source_sha256s, supersedes_song_ids: from_song_ids }` + - `correct { source_sha256s, new_song_id, supersedes_song_ids }` — the only + variant permitted to change a non-`None` label. +- **No invisible rewrites:** events (and applied batches) are immutable and + append-only. Within one batch, `ordinal` is semantic (**latest event for a + `sha256` wins**); a change from a non-`None` label requires a `correct`. + Historical batches are audit history and are **never** replayed. + +### 8.3 Plan artifact — embeds one unapplied batch (Apply's decision source) ```json { @@ -353,34 +376,40 @@ otherwise). "policy_id": "…", "policy_version": "1", "input_corpus_fingerprint": "", - "decisions_digest": "", + "decision_batch": { + "batch_id": "batch-000004", + "input_corpus_fingerprint": "", + "previous_application_report_digest": null, + "events": [ "…the complete ordered event objects…" ] + }, + "decisions_digest": "", "plan_digest": "", "assignments": [ - { - "source_sha256": "…", - "song_id": "song-000042", - "expected_existing_song_id": null, - "affected_chunk_ids": ["…","…"] - } + { "source_sha256": "…", "song_id": "song-000042", "expected_existing_song_id": null, "affected_chunk_ids": ["…","…"] } ], "generated_songs_map": { "song-000042": ["", "…"] } } ``` -`plan_digest` is computed over the canonical plan bytes **with the `plan_digest` -field omitted** (§9 record encoding). Apply verifies it (blocker: "apply only a -validated plan" is otherwise decorative). `expected_existing_song_id` is how a -label change is authorized: Apply refuses if the on-disk label differs from it. +The plan **embeds the complete ordered batch**, so Apply verifies +`decisions_digest`, **replays the events, derives the assignments itself, and +compares** them to `assignments` / `generated_songs_map` (§5.5) — a digest cannot +be recomputed from its own field, and `plan_digest` alone proves only internal +integrity, not derivation from the decisions. `expected_existing_song_id` +authorizes any label change. -### 8.4 Application report (evidence, not authority) +### 8.4 Application report (evidence + chain link, not authority) ```json { "schema": "song-curation.apply-report.v1", + "report_digest": "", + "batch_id": "batch-000004", + "applied_event_ids": ["ev-000017", "…in order…"], "input_corpus_fingerprint": "", + "output_corpus_fingerprint": "", "decisions_digest": "", "plan_digest": "", - "output_corpus_fingerprint": "", "curated_manifest_path": "song-curation/manifest.json", "curated_manifest_digest": "", "assignments_applied": 0, @@ -391,37 +420,47 @@ label change is authorized: Apply refuses if the on-disk label differs from it. } ``` -The report is **evidence**, never a second authority for `song_id`. +`report_digest` (over the report with that field omitted) is what the **next** +batch's `previous_application_report_digest` references, closing the chain. The +report is **evidence and a chain link**, never a second authority for `song_id`. -## 9. Determinism and the corpus fingerprint (injective) +## 9. Determinism — three distinct canonicalizations The core already has `corpus_fingerprint()` (`core/src/curation_store.rs:228`), -but it deliberately hashes each chunk's **material** identity and excludes mutable -curation fields — so it cannot detect a `song_id` or manifest-membership change, -which is exactly what a curation plan must be invalidated by. This workflow -therefore defines its **own** fingerprint over the label-bearing inputs. - -**Encoding (injective, canonical).** Build these records, one per item, as -**compact UTF-8 JSON arrays** (JSON escaping makes the encoding injective — -`ChunkId` and `SongId` are unrestricted `String` and may contain tabs or -newlines, which an ad-hoc separator could not survive): +but it hashes each chunk's **material** identity and excludes mutable curation +fields, so it cannot detect a `song_id` or manifest-membership change — exactly +what a curation plan must be invalidated by. This workflow defines **three +separate** canonicalizations; they are **not** "the same scheme", because event +order is semantic while the corpus snapshot is set-like. + +**`corpus_fingerprint` — order-insensitive (set-like).** Build one **compact +UTF-8 JSON array** record per item (JSON escaping is injective — `ChunkId` / +`SongId` are unrestricted `String` and may contain tabs/newlines an ad-hoc +separator could not survive), **sort the record bytes**, join with `\n`, and +`source_sha256` the result: ```json -["manifest_songs", "absent"] // when CorpusManifest.songs is None -["manifest_songs", "present"] // when Some(...) — even empty {} +["manifest_songs", "absent"] // CorpusManifest.songs == None +["manifest_songs", "present"] // Some(...) — even empty {} ["chunk", "", "", ""] -["song", "", ["", "…"]] // one per songs-map entry +["song", "", ["", "…"]] ``` -Sort the record **bytes** as UTF-8 strings, join with `\n`, and `source_sha256` -the result. This detects added/removed chunks, changed source hashes, changed -existing labels, and changed manifest membership; directory timestamps and -traversal order cannot affect it; and the `manifest_songs` presence record -distinguishes **absent** (`None`, skips cross-check) from **present-empty** -(`Some({})`, must account for every labelled source) — a distinction core makes -deliberately. Suggestion, plan, and fingerprint output are byte-deterministic for -the same inputs and policy version. The same compact-JSON-record + sort + hash -scheme defines `plan_digest` and `decisions_digest`. +The `manifest_songs` record distinguishes **absent** (`None`, skips the +cross-check) from **present-empty** (`Some({})`, must account for every labelled +source) — a distinction core makes deliberately. + +**`decisions_digest` — order-*sensitive*.** Canonicalize the decision **batch** +with its events in **exact append order** (each event's `ordinal` included), **no +sorting of the event sequence**; SHA-256 the compact UTF-8 JSON. Two ledgers with +the same events in different order produce different assignments ("latest wins"), +so their digests **must** differ. + +**`plan_digest` — order-aware where it matters.** Canonicalize the complete plan +with the `plan_digest` field omitted; the embedded `decision_batch` keeps its +events in append order (per `decisions_digest`); `assignments` are sorted by +`source_sha256`, each `affected_chunk_ids` sorted, and `generated_songs_map` keys +and value lists sorted; SHA-256 the result. ## 10. Refusal taxonomy (typed, defined before implementation) @@ -429,12 +468,18 @@ scheme defines `plan_digest` and `decisions_digest`. - `ConflictingExistingSongIds` — one `sha256` carries more than one `song_id`. - `UnknownDecisionSource` — a decision names a `sha256` absent from the corpus. - `SourceAssignedToMultipleSongs` — one `sha256` assigned to two `SongId`s. -- `DecisionCorpusFingerprintMismatch` — decisions were made against a different - snapshot (per-event fingerprint mismatch). +- `DecisionBatchFingerprintMismatch` — a batch's `input_corpus_fingerprint` does + not match the corpus it is planned/applied against (or its events disagree on + it). - `PlanCorpusFingerprintMismatch` — the corpus changed between plan and apply. - `PlanDigestMismatch` — the plan bytes do not match `plan_digest`. +- `DecisionProjectionMismatch` — replaying the embedded batch does not reproduce + the plan's `assignments` / `generated_songs_map`. +- `DecisionBatchAlreadyApplied` — the batch is already recorded as applied. +- `ApplicationChainMismatch` — the batch's `previous_application_report_digest` + or `input_corpus_fingerprint` does not chain to the preceding accepted report. - `ExistingLabelReplacementNotAuthorized` — a non-`None` label would change - without an explicit `correct` (checked via `expected_existing_song_id`). + without a `correct` (checked via `expected_existing_song_id`). - `ManifestDisagreement` — `CorpusManifest.songs` disagrees with the per-source labels (propagated from the core preflight where applicable). - `IncompleteCoverage` — not every participating source is labelled. @@ -446,21 +491,21 @@ becomes a **refusal** when `--require-holdout-ready` (or equivalent) is set. ## 11. Implementation sequence after acceptance (separate RED→GREEN slices) -1. **Decision & validation core** — parse/validate a human-authored decisions - ledger; inventory sources by `sha256`; construct the deterministic serialized - **dry-run plan** (§8.3) and verify its digest; **no** suggestions; **no** - corpus writes. +1. **Decision & validation core** — parse/validate the batched decisions ledger; + inventory by `sha256`; construct the serialized dry-run plan (§8.3), embed the + unapplied batch, and verify `plan_digest` / `decisions_digest` / + `DecisionProjectionMismatch` by re-deriving assignments; **no** suggestions; + **no** corpus writes. 2. **Transactional application** — apply a validated plan to a **fresh** output - tree; update every chunk of each source; generate the deterministic `songs` - map at the distinct curated path; produce the application report; prove - **idempotence** and **no partial writes**. + tree; verify the application chain; update every chunk of each source; + generate the deterministic `songs` map at the distinct curated path; produce + the chained application report; prove **idempotence** and **no partial writes**. 3. **Suggestion generator** — deterministic **metadata-only** suggestions; - evidence-rich output; **no** write path and **no** implicit acceptance. + evidence-rich; **no** write path and **no** implicit acceptance. 4. **Controlled corpus pilot** — only after **independent acceptance** of slices - 1–3; operate on a small **copied subset**; human-confirm every assignment; - verify snapshot and the curated manifest by digest; run - `song_holdout_preflight`; **no full-corpus labeling** until the pilot is - independently accepted. + 1–3; a small **copied subset**; human-confirm every assignment; verify the + snapshot and curated manifest by digest; run `song_holdout_preflight`; **no + full-corpus labeling** until the pilot is independently accepted. ## 12. Explicit non-goals @@ -486,26 +531,27 @@ This proposal PR is **discussion only**. After independent review: Every choice below is **selected for v1**, not left open: - **Owner:** standalone isolated `song-curation/` tool (§4). -- **Ledger format:** one versioned JSON document with immutable, append-only - **per-event** records following `CurationStoreV1` (§8.2) — **not** JSONL, **not** - an envelope-only fingerprint. -- **Action model:** a **tagged `action` union** covering accept / reject / manual - define / split / merge / correct, each with exact hashes and - `supersedes_song_ids` (§8.2). -- **Plan:** a serialized, digest-verified `song-curation.plan.v1` artifact is the - **only** thing Apply consumes (§8.3, §5.5). -- **`SongId` encoding:** opaque ledger-issued `song-`, single-writer ledger; concurrent issuance **out of scope for v1** - (§7). -- **Fingerprint / digests:** SHA-256 over sorted **compact-JSON records** with a - `manifest_songs` absent/present marker (§9) — injective, presence-aware; the - same scheme defines `plan_digest` and `decisions_digest`. -- **Manifest guard:** the tool writes to a **distinct curated path** and refuses - ordinary `/manifest.json`; ordinary `griff manifest` cannot overwrite - it; strategy 2 (extend `griff manifest`) is a later, separately-accepted change - (§5.6). +- **Ledger:** one versioned JSON document of append-only **batches** of immutable + events, following `CurationStoreV1` (§8.2) — not JSONL. Each batch binds its + events to one `input_corpus_fingerprint` and is the unit of one application. +- **Action model:** a tagged `action` union (accept / reject / manual_define / + split / merge / correct), each with exact hashes and `supersedes_song_ids`. +- **Plan:** a serialized plan that **embeds exactly one unapplied batch**; Apply + replays it, re-derives assignments, and compares (§8.3, §5.5) — the plan is the + sole source of the *decisions*; a non-initial batch also consumes the preceding + application report for **chain** verification only. +- **Application chain:** batch → report → next batch, linked by + `previous_application_report_digest` and matching input/output fingerprints + (§6, §8.4). +- **`SongId` encoding:** opaque ledger-issued `song-`, + single-writer; concurrent issuance out of scope for v1 (§7). +- **Digests:** **three** distinct canonicalizations — order-insensitive + `corpus_fingerprint`, order-**sensitive** `decisions_digest`, and order-aware + `plan_digest` (§9). +- **Manifest guard:** distinct curated path; refuse ordinary + `/manifest.json`; strategy 2 later (§5.6). - **Suggestion normalization:** reuse the census `strip_version_suffix` rule - **exactly** for v1; any later divergence requires a policy-version bump (§5.2). + **exactly** for v1; later divergence requires a policy-version bump (§5.2). Genuinely deferred (out of scope for v1, **not** unresolved alternatives): multi-writer/concurrent `SongId` issuance; the strategy-2 `griff manifest` From df2978a144bc2549f848149228635b8f922e7a3b Mon Sep 17 00:00:00 2001 From: PhysShell Date: Wed, 29 Jul 2026 12:38:25 +0000 Subject: [PATCH 4/4] docs(proposal): close batch action and report-chain edge contracts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Close four local contradictions + a taxonomy gap inside the accepted chain model. Still docs-only, Status: for discussion. 1. merge/split are now authorized replacement actions alongside correct: a non-None label may be replaced by exactly the action whose supersession set covers it (correct: supersedes_song_ids; merge: from_song_ids; split: from_song_id). Removed "correct is the only variant permitted to change a non-None label" — all six actions are first-class. 2. report_digest now has a canonicalization contract: §9 defines one shared canonical JSON encoding (sorted object keys, compact UTF-8) under FOUR ordering contracts — corpus (set-like), decisions (append order), plan (order-aware), and report (applied_event_ids in batch order, refusals sorted by (kind, source_sha256)). 3. Event order has one source of truth: the events array order is authoritative, ordinal MUST equal array position, ordinals contiguous+unique, event_id unique — else InvalidDecisionBatchOrder / DuplicateDecisionEventId before digest/replay. 4. DecisionBatchAlreadyApplied is now provable: a new append-only application index (§8.5, song-curation.applications.v1) is an explicit Apply input, written transactionally with the report; Apply checks batch_id absence and chains via the index's last record (catches rejection-only / fingerprint- neutral batches and re-apply to a fresh copy). 5. Added DecisionDigestMismatch, distinct from PlanDigestMismatch and DecisionProjectionMismatch. Changed §5.5, §6, §8.2, §8.4 (+ new §8.5), §9, §10, §14. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_012T7SRMiXmZe5v1imtrKMPi --- docs/proposals/song-id-curation-workflow.md | 189 ++++++++++++++------ 1 file changed, 131 insertions(+), 58 deletions(-) diff --git a/docs/proposals/song-id-curation-workflow.md b/docs/proposals/song-id-curation-workflow.md index 7b65d815..f121a918 100644 --- a/docs/proposals/song-id-curation-workflow.md +++ b/docs/proposals/song-id-curation-workflow.md @@ -165,24 +165,28 @@ batch do not all share that fingerprint. ### 5.5 Apply (transactional, chain-verified) Apply consumes the **plan** (which embeds its decision batch — the sole source of -the *decisions* it applies) and, for a **non-initial** batch, the **preceding -application report** (for **chain verification only**, never for decisions). +the *decisions* it applies) and the **application index** (§8.5) — the append-only +registry of already-applied batches that makes the chain and already-applied +checks *provable* from a declared input, not inferred from corpus side-effects. Before writing, Apply: 1. recomputes `plan_digest` over the plan (`PlanDigestMismatch` on mismatch); -2. recomputes the embedded batch's `decisions_digest` from its ordered events; +2. recomputes the embedded batch's `decisions_digest` and checks it equals the + plan's `decisions_digest` field (`DecisionDigestMismatch`); 3. verifies the batch `input_corpus_fingerprint` equals the **current** corpus fingerprint (`PlanCorpusFingerprintMismatch`); -4. verifies the application chain — the batch's - `previous_application_report_digest` equals the digest of the supplied - preceding report, and that report's `output_corpus_fingerprint` equals this - batch's `input_corpus_fingerprint` (`ApplicationChainMismatch`); refuses a - batch already recorded as applied (`DecisionBatchAlreadyApplied`); +4. verifies the application chain against the index: the batch's `batch_id` is + **absent** from the index (`DecisionBatchAlreadyApplied`); and — for a + non-initial batch — the batch's `previous_application_report_digest` equals the + index's last `report_digest`, whose `output_corpus_fingerprint` equals this + batch's `input_corpus_fingerprint` (`ApplicationChainMismatch`); 5. **replays the embedded events itself**, derives assignments + `generated_songs_map`, and compares them to the plan's — any divergence is `DecisionProjectionMismatch` (the plan cannot assert an assignment the decisions do not produce); -6. checks every `expected_existing_song_id` against the on-disk label. +6. checks every `expected_existing_song_id` against the on-disk label, and that + each label change is covered by the acting event's authorized supersession set + (§8.2). Only then does it write: @@ -194,11 +198,15 @@ Only then does it write: changes); - **deterministic** file order and JSON rendering; **idempotent** for already-correct labels; -- **never** clearing a non-`None` label unless a `correct` event authorizes it +- **never** clearing a non-`None` label unless the acting event's supersession + set authorizes it — `correct`, `merge`, or `split` (§8.2) (`ExistingLabelReplacementNotAuthorized`; the plan's `expected_existing_song_id` is the check). -A **correction is a new curator decision**, not heuristic reconciliation. +On success Apply publishes the application report **and** appends a matching +record to the application index **transactionally** — both land or neither does, +so the registry never lies about what was applied. A **correction / merge / split +is a new curator decision**, not heuristic reconciliation. ### 5.6 Generate manifest (deterministic) — hard distinct-path guard @@ -255,10 +263,15 @@ question the flat-replay model could not: - every event in one batch binds to the **same** `input_corpus_fingerprint`; - **one plan embeds exactly one unapplied batch**; - **historical batches are audit history and are never replayed**; -- the next batch's `input_corpus_fingerprint` must equal the preceding accepted - report's `output_corpus_fingerprint` (`ApplicationChainMismatch` otherwise); -- reuse of an already-applied batch typed-refuses - (`DecisionBatchAlreadyApplied`). +- the **application index** (§8.5) is the append-only registry of applied + batches; the next batch's `input_corpus_fingerprint` must equal the index's + last `output_corpus_fingerprint`, and its `previous_application_report_digest` + the index's last `report_digest` (`ApplicationChainMismatch` otherwise); +- reuse of a batch whose `batch_id` is already in the index typed-refuses + (`DecisionBatchAlreadyApplied`) — the registry makes this **provable** rather + than inferred from corpus side-effects, so it also catches a rejection-only or + otherwise fingerprint-neutral batch, and re-applying an initial batch to a + fresh copy of the same snapshot. Partial output is `holdout_ready: false`, is **not** a valid song-holdout corpus, leaves uncurated sources `song_id: None`, never invents an "unknown" shared @@ -361,12 +374,28 @@ binds all its events to a single `input_corpus_fingerprint`: - `manual_define { source_sha256s, assign_song_id }` - `split { from_song_id, into: [ { assign_song_id, source_sha256s } … ], supersedes_song_ids: [from_song_id] }` - `merge { from_song_ids, into_song_id, source_sha256s, supersedes_song_ids: from_song_ids }` - - `correct { source_sha256s, new_song_id, supersedes_song_ids }` — the only - variant permitted to change a non-`None` label. + - `correct { source_sha256s, new_song_id, supersedes_song_ids }`. +- **Authorized replacement (three actions, not one).** Changing a non-`None` + existing label is permitted by exactly the action whose supersession set covers + it: + - `correct` — may replace exactly the labels named in its `supersedes_song_ids`; + - `merge` — may replace labels from `from_song_ids` with `into_song_id`; + - `split` — may replace `from_song_id` with the explicitly enumerated target + assignments. + + `ExistingLabelReplacementNotAuthorized` fires when the actual on-disk label is + **not** in the authorized supersession set of the acting event + (`accept_suggestion` / `manual_define` carry an empty set, so they may only + label `None`; `reject_suggestion` assigns nothing). All six actions are + first-class. +- **Ordering — one source of truth.** The `events` **array order is the + authoritative append order**; `ordinal` MUST equal the zero-based array + position; ordinals MUST be contiguous and unique; `event_id` MUST be unique + within the ledger. Any violation typed-refuses (`InvalidDecisionBatchOrder` / + `DuplicateDecisionEventId`) **before** any digest or replay. Within one batch, + the **latest event (highest ordinal) for a `sha256` wins**. - **No invisible rewrites:** events (and applied batches) are immutable and - append-only. Within one batch, `ordinal` is semantic (**latest event for a - `sha256` wins**); a change from a non-`None` label requires a `correct`. - Historical batches are audit history and are **never** replayed. + append-only; historical batches are audit history and are **never** replayed. ### 8.3 Plan artifact — embeds one unapplied batch (Apply's decision source) @@ -420,24 +449,53 @@ authorizes any label change. } ``` -`report_digest` (over the report with that field omitted) is what the **next** -batch's `previous_application_report_digest` references, closing the chain. The -report is **evidence and a chain link**, never a second authority for `song_id`. +`report_digest` (the report canonicalized per §9 with that field omitted) is what +the **next** batch's `previous_application_report_digest` references, closing the +chain. The report is **evidence and a chain link**, never a second authority for +`song_id`. -## 9. Determinism — three distinct canonicalizations +### 8.5 Application index — the append-only applied-batch registry + +The registry that makes `DecisionBatchAlreadyApplied` and the chain checks +**provable from a declared Apply input**, rather than inferred from corpus +side-effects (which a rejection-only or fingerprint-neutral batch would evade): + +```json +{ + "schema": "song-curation.applications.v1", + "applications": [ + { + "batch_id": "batch-000004", + "report_digest": "", + "input_corpus_fingerprint": "", + "output_corpus_fingerprint": "" + } + ] +} +``` + +Plan creation and Apply check the batch's `batch_id` is **absent** here; Apply +appends the new record **transactionally with report publication** (both land or +neither). The last record is the chain head (§5.5 step 4). + +## 9. Determinism — one canonical encoding, four ordering contracts The core already has `corpus_fingerprint()` (`core/src/curation_store.rs:228`), but it hashes each chunk's **material** identity and excludes mutable curation fields, so it cannot detect a `song_id` or manifest-membership change — exactly -what a curation plan must be invalidated by. This workflow defines **three -separate** canonicalizations; they are **not** "the same scheme", because event -order is semantic while the corpus snapshot is set-like. +what a curation plan must be invalidated by. This workflow therefore defines its +own digests. + +**Shared canonical JSON encoding.** All four contracts below serialize to +**compact UTF-8 JSON with object keys sorted lexicographically** and no +insignificant whitespace, then SHA-256. Fixed key ordering plus JSON escaping +(injective — `ChunkId` / `SongId` are unrestricted `String` and may contain +tabs/newlines an ad-hoc separator could not survive) makes the byte encoding +deterministic across implementations. What differs per contract is **which arrays +are sorted and which preserve semantic order**: -**`corpus_fingerprint` — order-insensitive (set-like).** Build one **compact -UTF-8 JSON array** record per item (JSON escaping is injective — `ChunkId` / -`SongId` are unrestricted `String` and may contain tabs/newlines an ad-hoc -separator could not survive), **sort the record bytes**, join with `\n`, and -`source_sha256` the result: +**1. `corpus_fingerprint` — order-insensitive (set-like).** One record per item; +**sort the record bytes**; join with `\n`: ```json ["manifest_songs", "absent"] // CorpusManifest.songs == None @@ -450,17 +508,21 @@ The `manifest_songs` record distinguishes **absent** (`None`, skips the cross-check) from **present-empty** (`Some({})`, must account for every labelled source) — a distinction core makes deliberately. -**`decisions_digest` — order-*sensitive*.** Canonicalize the decision **batch** -with its events in **exact append order** (each event's `ordinal` included), **no -sorting of the event sequence**; SHA-256 the compact UTF-8 JSON. Two ledgers with -the same events in different order produce different assignments ("latest wins"), -so their digests **must** differ. +**2. `decisions_digest` — order-*sensitive*.** Canonicalize the decision **batch** +with its `events` array in **append order** (never sorted; `ordinal` equals the +array position, §8.2). Two ledgers with the same events in different order produce +different assignments ("latest wins"), so their digests **must** differ. + +**3. `plan_digest` — order-aware.** Canonicalize the complete plan with the +`plan_digest` field omitted; the embedded `decision_batch` keeps its `events` in +append order (per contract 2); `assignments` sorted by `source_sha256`, each +`affected_chunk_ids` sorted, `generated_songs_map` keys and value lists sorted. -**`plan_digest` — order-aware where it matters.** Canonicalize the complete plan -with the `plan_digest` field omitted; the embedded `decision_batch` keeps its -events in append order (per `decisions_digest`); `assignments` are sorted by -`source_sha256`, each `affected_chunk_ids` sorted, and `generated_songs_map` keys -and value lists sorted; SHA-256 the result. +**4. `report_digest` — order-aware.** Canonicalize the complete application report +with the `report_digest` field omitted; `applied_event_ids` preserved in **batch +order**; `refusals` sorted by the fixed tuple `(kind, source_sha256)`. This +contract is load-bearing because the next batch's +`previous_application_report_digest` references it (§8.4, §8.5). ## 10. Refusal taxonomy (typed, defined before implementation) @@ -468,18 +530,26 @@ and value lists sorted; SHA-256 the result. - `ConflictingExistingSongIds` — one `sha256` carries more than one `song_id`. - `UnknownDecisionSource` — a decision names a `sha256` absent from the corpus. - `SourceAssignedToMultipleSongs` — one `sha256` assigned to two `SongId`s. +- `InvalidDecisionBatchOrder` — a batch's `ordinal`s are not contiguous, unique, + and equal to the array position (§8.2). +- `DuplicateDecisionEventId` — an `event_id` repeats within the ledger. - `DecisionBatchFingerprintMismatch` — a batch's `input_corpus_fingerprint` does not match the corpus it is planned/applied against (or its events disagree on it). - `PlanCorpusFingerprintMismatch` — the corpus changed between plan and apply. -- `PlanDigestMismatch` — the plan bytes do not match `plan_digest`. -- `DecisionProjectionMismatch` — replaying the embedded batch does not reproduce - the plan's `assignments` / `generated_songs_map`. -- `DecisionBatchAlreadyApplied` — the batch is already recorded as applied. +- `PlanDigestMismatch` — the canonical plan bytes do not match `plan_digest`. +- `DecisionDigestMismatch` — the embedded batch does not match the plan's + `decisions_digest` field (distinct from the two above). +- `DecisionProjectionMismatch` — the batch is valid, but replaying it does not + reproduce the plan's `assignments` / `generated_songs_map`. +- `DecisionBatchAlreadyApplied` — the batch's `batch_id` is already in the + application index (§8.5). - `ApplicationChainMismatch` — the batch's `previous_application_report_digest` - or `input_corpus_fingerprint` does not chain to the preceding accepted report. -- `ExistingLabelReplacementNotAuthorized` — a non-`None` label would change - without a `correct` (checked via `expected_existing_song_id`). + or `input_corpus_fingerprint` does not chain to the index's last record. +- `ExistingLabelReplacementNotAuthorized` — a non-`None` label would change, but + the actual on-disk label is **not** in the acting event's authorized + supersession set (`correct` / `merge` / `split`, §8.2; checked via + `expected_existing_song_id`). - `ManifestDisagreement` — `CorpusManifest.songs` disagrees with the per-source labels (propagated from the core preflight where applicable). - `IncompleteCoverage` — not every participating source is labelled. @@ -535,19 +605,22 @@ Every choice below is **selected for v1**, not left open: events, following `CurationStoreV1` (§8.2) — not JSONL. Each batch binds its events to one `input_corpus_fingerprint` and is the unit of one application. - **Action model:** a tagged `action` union (accept / reject / manual_define / - split / merge / correct), each with exact hashes and `supersedes_song_ids`. + split / merge / correct), all six first-class; `correct`, `merge`, and `split` + are the **authorized replacement** actions, each replacing exactly the labels in + its supersession set (§8.2). - **Plan:** a serialized plan that **embeds exactly one unapplied batch**; Apply replays it, re-derives assignments, and compares (§8.3, §5.5) — the plan is the - sole source of the *decisions*; a non-initial batch also consumes the preceding - application report for **chain** verification only. -- **Application chain:** batch → report → next batch, linked by - `previous_application_report_digest` and matching input/output fingerprints - (§6, §8.4). + sole source of the *decisions*. +- **Application chain + index:** batch → report → next batch, linked by + `previous_application_report_digest` and matching input/output fingerprints, + with an append-only **application index** (§8.5) as the provable already-applied + registry (an Apply input, written transactionally with the report). - **`SongId` encoding:** opaque ledger-issued `song-`, single-writer; concurrent issuance out of scope for v1 (§7). -- **Digests:** **three** distinct canonicalizations — order-insensitive - `corpus_fingerprint`, order-**sensitive** `decisions_digest`, and order-aware - `plan_digest` (§9). +- **Digests:** one **shared canonical JSON encoding** (sorted object keys, compact + UTF-8) under **four** ordering contracts (§9): order-insensitive + `corpus_fingerprint`; order-**sensitive** `decisions_digest`; order-aware + `plan_digest`; order-aware `report_digest`. - **Manifest guard:** distinct curated path; refuse ordinary `/manifest.json`; strategy 2 later (§5.6). - **Suggestion normalization:** reuse the census `strip_version_suffix` rule