Skip to content

docs: add Human Similarity Benchmark design for the preference-learning discussion - #153

Merged
PhysShell merged 5 commits into
mainfrom
claude/human-similarity-benchmark
Jul 26, 2026
Merged

docs: add Human Similarity Benchmark design for the preference-learning discussion#153
PhysShell merged 5 commits into
mainfrom
claude/human-similarity-benchmark

Conversation

@PhysShell

@PhysShell PhysShell commented Jul 26, 2026

Copy link
Copy Markdown
Owner

What

New design artifact docs/proposals/human-similarity-benchmark.md (Status: for discussion (v2.1); docs-only, binds nothing), the companion spec for Step 2 (§4) of the preference-and-similarity-learning proposal — the human benchmark that proposal names as the gate.

Docs-only: no code, no production change. A normative acceptance-gate paragraph sits near the top — designs the benchmark, promises no implementation/API/UI/delivery; benchmark results carry no production authority until the parent proposal's per-task gates run and it is accepted; every later slice earns its own red→green cycle.

Sections (§2)

  • Four tasks, closed set (similarity / variation / complementarity / copy-detection), each with the same field set and an honest baseline-expressiveness note. A metric passing one task earns no licence on another.
  • Evidence schema — versioned JSONL, identity griff.similarity-benchmark-judgement v1. Recipe/content/lineage identities; immutable bindings (item_manifest_hash, sampling_policy_id, presentation_profile_id, derivation_policy_id, dataset_split_id) with typed refusal on unknown versions; Skip distinct from reject; contextual B-over-C / C-over-B derived labels (never A-over-B), policy contextual_bc_preference_v1; closed question_variant registry; audition_order recorded independently of display_order; anchor + candidate audition; privacy contract before collection.
  • Presentation contract — versioned PresentationProfile + procedure; complementarity judges the A+B vs A+C mixes; candidate audition order randomized and recorded. Timbre/loudness are controlled presentation confounds, not N/A ("no audio similarity" remains the non-goal).
  • Samplingtask-specific difficulty strata (a single near/far scale isn't constructible for copy-detection or complementarity); corpus-axis stratification; source-identity (song) holdout; user-disjoint cross-user split independent of the song split; same-song pairs labelled.
  • Session — fatigue bound, order randomization, repeated probes; evidence floors per user/task and per stratum.
  • Gate (statistics) — a paired per-item difference on a clustered bootstrap (resampled over users and songs, repeated probes collapsed), read per stratum and aggregate at a fixed 95%/seed. Non-inferiority (≥ −δ) vs a real baseline; superiority (≥ ε, ε>0) vs a floor. Symmetric deterministic tie rule for both predictors. Ranking metrics (top-k, Kendall/Spearman, NDCG) are diagnostics, not gates.
  • Baselines — named code: similarity.rs v3 (similarity), novelty.rs (copy-detection), a declared composite (variation), and — honestly — no handcrafted distance for complementarity (complement.rs::validate_pair is a legality validator, not a wired gate); it runs against a declared floor under the superiority gate, or is declared ungateable.
  • Collection surface — one paragraph: a new A/B/C mode of the S8 playground; UI out of scope.
  • Non-goals / Prior art — no implementation/embeddings; no change to rerank.rs or the governing S9 Phase 1 EMA plan; no audio similarity; no quality claim. Prior art: timbremetrics triplet methodology, paired non-inferiority + cluster bootstrap, pairwise aggregation, listening-test hygiene (idea-only; GPL never contributes code).

Also updated

  • docs/proposals/preference-and-similarity-learning.mdStage NStep N relabel (avoids reading as canonical SN); Step 3 learner + Step 2 copy-detection wording synced to the companion's contextual/single-question forms.
  • docs/proposals/README.md — index entry (baseline-or-declared-floor wording).
  • docs/decisions.log.md — prior-art survey entries (benchmark methodology; non-inferiority / cluster bootstrap), Y-statement format.

Review history

Two arbiter rounds (REQUEST CHANGES) addressed: v2 (paired-difference gate, B-over-C semantics, presentation profile, immutable bindings, single-question copy-detection, Step rename) and v2.1 (parent sync, randomized audition order, task-specific strata, symmetric tie handling, complementarity superiority floor, per-stratum evidence floors, concrete v1 ids, metadata cleanup). All CodeRabbit inline threads auto-resolved; Codex v1 findings mapped to fixes. Four files changed. Docs-only → CI (Rust-only) green.

🤖 Generated with Claude Code

https://claude.ai/code/session_012T7SRMiXmZe5v1imtrKMPi

…ng discussion

Companion spec for Stage 2 of the preference-and-similarity-learning
proposal: the four separate benchmark tasks (similarity / variation /
complementarity / copy-detection), each with its own dataset, gate, and
named handcrafted baseline; a versioned JSONL evidence schema
(griff.similarity-benchmark-judgement v1) with recipe/content/lineage
identities, Skip kept distinct from reject, and derived-only pairwise
labels; the confound-controlled, source-identity sampling protocol; the
session/fatigue procedure with an evidence floor; and an explicit
metric-to-gate mapping.

Honest baselines: similarity.rs v3 axes for similarity, novelty.rs for
copy-detection, a declared composite for variation, and — stated plainly
rather than stretched — no handcrafted distance for complementarity
(complement.rs validate_pair is a legality validator, not a wired gate).

Docs-only; binds nothing. The S9 Phase 1 EMA plan remains governing;
rerank.rs untouched; every later slice earns its own red->green cycle.
Index and decisions.log (prior-art survey, Y-statement) updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012T7SRMiXmZe5v1imtrKMPi
@coderabbitai

coderabbitai Bot commented Jul 26, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@PhysShell, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 19 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: a82e9254-0042-4759-9fbe-34efd7a1a6c9

📥 Commits

Reviewing files that changed from the base of the PR and between 4da2daf and fbaae43.

📒 Files selected for processing (4)
  • docs/decisions.log.md
  • docs/proposals/README.md
  • docs/proposals/human-similarity-benchmark.md
  • docs/proposals/preference-and-similarity-learning.md
📝 Walkthrough

Walkthrough

Added a documentation-only Human Similarity Benchmark design specification, indexed it in the proposals README, and recorded its prior-art methodology decision. The specification defines four tasks, evidence and sampling protocols, session procedures, metric gates, baselines, collection scope, and non-goals.

Changes

Human Similarity Benchmark

Layer / File(s) Summary
Scope and benchmark task contracts
docs/proposals/human-similarity-benchmark.md
Defines the artifact’s acceptance boundaries, four benchmark tasks, task questions, sourcing rules, confound controls, and decision procedures.
Evidence schema and collection protocol
docs/proposals/human-similarity-benchmark.md
Defines judgement records, stimulus identities, response semantics, sampling, session procedures, evidence thresholds, and the intended A/B/C collection surface.
Metrics, gates, and handcrafted baselines
docs/proposals/human-similarity-benchmark.md
Maps metrics to task gates, uses bootstrap confidence-interval lower bounds, and specifies handcrafted baselines for each task.
Scope boundaries and proposal governance
docs/proposals/human-similarity-benchmark.md, docs/proposals/README.md, docs/decisions.log.md
Documents non-goals and prior-art constraints, indexes the proposal for discussion, and records the methodology survey decision.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • PhysShell/griff#149: Extends the parent preference-and-similarity-learning proposal with the benchmark gate covered by this specification.

Suggested reviewers: claude

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: adding the Human Similarity Benchmark design docs for the preference-learning discussion.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/human-similarity-benchmark

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 7

🧹 Nitpick comments (1)
docs/proposals/human-similarity-benchmark.md (1)

388-397: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Resolve the Mantel-test scope.

Prior art says the Mantel test is adopted as methodology, but the binding metric table assigns it no task or status. Either define its role or explicitly mark it informative/out of scope so implementations do not disagree.

Also applies to: 489-494

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/proposals/human-similarity-benchmark.md` around lines 388 - 397, Resolve
the Mantel-test ambiguity in the binding metric table and its corresponding
section around the methodology discussion: explicitly assign the test to
applicable tasks with its status and role, or label it informative/out of scope.
Ensure both references use the same scope so implementations interpret the
metric consistently.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/proposals/human-similarity-benchmark.md`:
- Around line 243-262: Update the schema v1 definition near the question_variant
field to enumerate the allowed question_variant identifiers for each of the four
tasks, or explicitly reference a versioned registry containing them. Keep the
vocabulary closed and unambiguous so producers cannot emit arbitrary identifiers
under the same schema version.
- Around line 263-269: Update the benchmark proposal’s user_id definition to
specify a non-reversible pseudonym and document its retention and access rules
before collection. Replace the current enrollment/pseudonymization TBD note with
this privacy contract, ensuring schema v1 records are not emitted until it is
defined.
- Around line 313-319: Update the benchmark record or archive manifest schema to
store the pairwise-label derivation policy identifier alongside the recorded
observations. Define the accepted policy identifiers and make downstream
processing reject archives containing unknown policies, preserving deterministic
reprocessing of existing JSONL data.
- Around line 395-404: Complete the bootstrap protocol before applying CI-based
gates: specify the confidence level, resampling unit, handling of repeated
triplets and users, and whether candidate-versus-baseline comparisons use paired
or independent resampling. Add these requirements to the benchmark protocol and
ensure the gate’s lower-bound comparison follows them consistently.
- Around line 158-160: Update the complementarity task description around the
question and corresponding non-goals to explicitly define how participants hear
or view the parts, including whether they receive notation, MIDI playback,
benchmark rendering, or another modality. Ensure the selected presentation
method is consistent across the task and makes the “played together” assessment
reproducible.
- Around line 102-105: Update the baseline comparison and scoring descriptions
in the affected sections, including the gate comparison and the referenced
baseline definitions, to specify one deterministic tie-handling rule for equal
scores. State how ties affect prediction and agreement scoring, and require that
this rule be applied consistently before collecting benchmark results.
- Around line 271-297: Extend the audition evidence model described in the
“Exposure and audition” section to record whether the anchor was heard or
inspected, including its listen duration and completion status where applicable.
Ensure anchor playback is represented explicitly as a precondition or dedicated
anchor fields so downstream analysis can filter judgements without treating it
as an exposed candidate.

---

Nitpick comments:
In `@docs/proposals/human-similarity-benchmark.md`:
- Around line 388-397: Resolve the Mantel-test ambiguity in the binding metric
table and its corresponding section around the methodology discussion:
explicitly assign the test to applicable tasks with its status and role, or
label it informative/out of scope. Ensure both references use the same scope so
implementations interpret the metric consistently.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 1169e0a6-3d58-422c-851d-d451948821de

📥 Commits

Reviewing files that changed from the base of the PR and between 459f542 and 4da2daf.

📒 Files selected for processing (3)
  • docs/decisions.log.md
  • docs/proposals/README.md
  • docs/proposals/human-similarity-benchmark.md

Comment thread docs/proposals/human-similarity-benchmark.md Outdated
Comment thread docs/proposals/human-similarity-benchmark.md Outdated
Comment thread docs/proposals/human-similarity-benchmark.md Outdated
Comment thread docs/proposals/human-similarity-benchmark.md Outdated
Comment thread docs/proposals/human-similarity-benchmark.md Outdated
Comment thread docs/proposals/human-similarity-benchmark.md Outdated
Comment thread docs/proposals/human-similarity-benchmark.md Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4da2dafa79

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +313 to +316
- **Pairwise labels are derived, never stored raw and never conflated.** A
record stores the triplet `choice` and the `unary` signals as *observations*;
an "A-over-B" pairwise label is computed downstream from a `choice` under a
documented derivation, and a unary favorite/reject is **not** a pairwise

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Derive B-over-C rather than A-over-B labels

For every task, the recorded choice states whether candidate B or C is preferred relative to anchor A, so it supports a B-over-C or C-over-B label; it provides no comparison between A and B. Implementing the stated A-over-B derivation would fabricate training labels and corrupt the pairwise agreement and reranker datasets.

Useful? React with 👍 / 👎.

Comment on lines +401 to +404
- **The gate is the CI lower bound.** A task passes only when the candidate
metric's bootstrap-CI lower bound on its gate metric is ≥ the baseline's on
the same items (§1 pass criteria). A point-estimate win inside overlapping
CIs does not pass.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Bootstrap the paired agreement difference

Comparing the two CI lower bounds does not test whether the candidate beats or matches the baseline: a lower point estimate with a tighter CI can pass, and even a tiny point-estimate win with heavily overlapping CIs can have the higher lower bound, contrary to the next sentence. Because both systems are evaluated on the same items, the gate should bootstrap their paired agreement difference and test its lower bound against zero or a specified non-inferiority margin.

Useful? React with 👍 / 👎.

Comment on lines +339 to +342
- **Source-identity discipline (the reachability-lab holdout law).** Any
evaluation split — train/validation/test for a learned metric, or held-out
users/items for generalization — is by **source identity (song), never by
chunk**. A split that puts one bar of a song in train and the next bar in

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Split held-out users by user identity

When evaluating the later Cross-user generalization metric, partitioning every split only by source song can leave the same listeners in training and evaluation, so the reported result is item generalization rather than held-out-user agreement. User generalization needs a user-ID holdout, while item leakage should independently remain source-song-disjoint.

Useful? React with 👍 / 👎.

[preference-and-similarity-learning](preference-and-similarity-learning.md)
proposal: the evidence contract, sampling protocol, session procedure, and
metric-to-gate mapping for the four-task human benchmark that proposal names
as *the gate* (its Stage 2). This document designs the benchmark; it does not

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Rename proposal steps to avoid canonical stage labels

Calling this benchmark “Stage 2” conflicts with the project glossary, where the roadmap stage S2 is the MIDI transport refactor; the new document similarly uses Stage 0 and Stage 1 for unrelated proposal steps. Rename these internal steps to phases/slices or refer to the parent proposal's section numbers so roadmap-stage references remain unambiguous.

AGENTS.md reference: AGENTS.md:L110-L110

Useful? React with 👍 / 👎.

Comment on lines +293 to +297
- `auditioned` — for each exposed candidate, whether it was actually played,
the `listen_ms` (audition duration in milliseconds), and a `completed` flag
(did it play to the end). A choice among candidates one of which was never
heard is a different datum from an informed choice, and downstream analysis
must be able to drop it.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Record audition data for anchor A

When a listener does not play A fully, replays it, or stops it early, this schema cannot identify the compromised judgement because auditioned covers only exposed candidates and the anchor is not part of exposure. Since every task depends on comparison against A, the anchor also needs played/listen-duration/completion evidence so downstream analysis can reject uninformed trials.

Useful? React with 👍 / 👎.

PhysShell and others added 2 commits July 26, 2026 06:44
… arbiter)

The proposal's internal sequence used `Stage 0..3`, which reads as canonical
roadmap stages (glossary §0 owns S0..S16). Relabel to `Step 0..3` — a
proposal-local sequence, introducing no `SN` stage — and add a Scope note
saying so. Substance unchanged; the only roadmap stage touched is S9, named
as such. Keeps the companion benchmark spec from implying a new canonical
stage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012T7SRMiXmZe5v1imtrKMPi
Address the arbiter's contract-level findings; the concept was accepted, the
v1 contract was not executable. Changes:

- Statistical gate corrected: replace the two-independent-CI comparison with
  a paired per-item non-inferiority test on a clustered bootstrap —
  Δ_i = correct(candidate,i) − correct(baseline,i), gate lower_CI(mean Δ) ≥ −δ
  with a pre-registered margin, fixed confidence/resamples/seed, resampling
  over users and source songs (not rows), repeated probes collapsed, read
  per difficulty stratum and aggregate; deterministic baseline-tie rule.
- Pairwise semantics fixed: the choice is B-vs-C relative to A, so the only
  derivable label is B-over-C / C-over-B in the context of A and the task —
  never A-over-B. Add PairwisePreference with a versioned derivation_policy_id.
- Presentation contract added: a versioned PresentationProfile (backend,
  patch, tempo/velocity/gain-pan/mix policy, loop count, normalization) and a
  fixed procedure; timbre/loudness are controlled presentation confounds, not
  N/A. Anchor audition evidence added; complementarity judges A+B vs A+C mixes.
- Immutable binding: judgement records bind item_manifest_hash,
  sampling_policy_id, presentation_profile_id, derivation_policy_id,
  dataset_split_id; unknown versions are a typed refusal. Closed
  question-variant registry; user_id privacy contract before collection.
- Copy-detection made a single-question 2AFC ("which is too close to A"); the
  remote candidate is a labelled control, no "too far" label derived.
- Sampling difficulty strata (easy..hard_negative, baseline_tie), fixed
  before collection and independent of the metric under test; cross-user
  generalization gets its own user-disjoint split, independent of the song split.
- Ranking metrics (top-k, Kendall/Spearman, NDCG) demoted to exploratory
  diagnostics; promoting them needs a separate versioned ranking-item protocol.
- decisions.log: append the non-inferiority / cluster-bootstrap prior-art
  survey (prior-art-first rule).

Still docs-only; binds nothing. S9 Phase 1 EMA remains governing; rerank.rs
untouched; every later slice earns its own red->green cycle.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012T7SRMiXmZe5v1imtrKMPi
@PhysShell

Copy link
Copy Markdown
Owner Author

Thanks — REQUEST CHANGES accepted in full; none of the eight findings warranted a refutation. Pushed v2 as two focused commits on head a8957387e54911fad02ab4cbbbcf06c258448339:

  • 51b4b86 — relabel the parent proposal's Stage NStep N
  • a895738 — the v2 benchmark contract + decisions.log prior-art addendum

Point by point:

  1. Pairwise semantics. Fixed. The choice is B-vs-C relative to A, so the only derivable label is B-over-C / C-over-B in the context of A and the task — "A-over-B" is explicitly called out as never observed and never minted (§2.2, "Derived pairwise labels — corrected semantics"). Added PairwisePreference { context_anchor, preferred, dispreferred, task, question_variant, derivation_policy_id }, and derivation_policy_id is now a bound field of every record.

  2. Statistical gate. Rewritten (§2.6). Now a paired per-item non-inferiority test: Δ_i = correct(candidate,i) − correct(baseline,i), gate lower_CI(mean Δ) ≥ −δ with a pre-registered margin, fixed confidence (95%) / resample count / seed; resampling is a paired hierarchical/cluster bootstrap over users and source songs, not rows; repeated probes collapsed within cluster; read per difficulty stratum and aggregate. Cross-user generalization now requires a user-disjoint split independent of the song split (§2.4). Deterministic baseline-tie rule (half-credit default; abstain / fixed-fallback named) in §2.6.

  3. Presentation contract. Added (§2.3). Versioned PresentationProfile (backend, patch, tempo/velocity/gain-pan, loop_count, lead_candidate_mix_policy, normalization_version) + fixed procedure; complementarity judges the mixes A+B vs A+C, never B/C alone (§1.3, §2.3). Timbre/loudness/mix reclassified as controlled presentation confounds, not N/A (§2.9 keeps only "no audio similarity" as the non-goal). Anchor audition evidence added (anchor_played/listen_ms/completed/replay_count), plus per-mix playback identities (§2.2).

  4. Self-describing JSONL. Records now bind item_id, item_manifest_schema + item_manifest_hash, sampling_policy_id, presentation_profile_id, derivation_policy_id, dataset_split_id; the item manifest holds source identities, same-source flags, strata, controlled dimensions; unknown version of any policy/manifest is a typed refusal (§2.2). question_variant is now a closed versioned registry, not a free string; user_id carries a privacy contract (irreversible pseudonym, access rules, retention) as a precondition of collection.

  5. Copy-detection. Made a single-question 2AFC — "Which candidate is too close to A to count as original?" The remote candidate is a labelled control/negative; no "too far" label is derived (§1.4). Took your preferred option, so the closed set stays four.

  6. Sampling. Difficulty strata added — easy / medium / hard / hard_negative / baseline_tiefixed before collection and independent of the candidate metric under test; the gate is read per-stratum so an easy-item win can't mask a hard-negative loss (§2.4, §2.6).

  7. Stage nomenclature. Relabelled Stage NStep N in both the parent proposal and this companion, with a Scope note that Step 0…3 are proposal-local, not canonical SN (glossary §0). PR body's "Stage 2" now reads "Step 2 (§4)".

  8. Remaining clarifications. Deterministic tie rule ✓; fixed confidence + bootstrap seed/repetition ✓; closed question-variant registry ✓; versioned pairwise derivation policy ✓; anchor audition ✓; privacy contract ✓. Ranking metrics (top-k, Kendall/Spearman, NDCG) are demoted to exploratory diagnostics, not gates — promoting them needs a separate versioned ranking-item protocol (named as future work); the only v1 gate metric is triplet agreement (§2.6).

Deliberately left as TBD at spec (with a named decision procedure, not a silent gap): the non-inferiority margin δ; the complementarity floor baseline (random vs legality-only vs a new handcrafted distance); the variation composite's middle-band breakpoints; the evidence-floor numbers; and the concrete pseudonym/retention mechanism.

Docs-only; still binds nothing; S9 Phase 1 EMA remains governing. Not merging without an explicit human GO.

@PhysShell

Copy link
Copy Markdown
Owner Author

@codex — your review was on the v1 commit 4da2daf; all five findings are fixed in the v2 head a8957387 (commits 51b4b86, a895738). Mapping, so you can re-verify against code:

  • Derive B-over-C rather than A-over-B labels (P1) → fixed in a895738, §2.2 "Derived pairwise labels — corrected semantics": the choice is B-vs-C relative to A, so the only derivable labels are B-over-C / C-over-B in the context of A and the task; an "A-over-B" label is explicitly called out as never observed and never minted. Added PairwisePreference { context_anchor, preferred, dispreferred, task, question_variant, derivation_policy_id }.
  • Bootstrap the paired agreement difference (P1) → fixed in a895738, §2.6: gate is now lower_CI(mean Δ) ≥ −δ on the per-item paired difference Δ_i = correct(candidate,i) − correct(baseline,i), via a paired hierarchical/cluster bootstrap over users and source songs (not rows), repeated probes collapsed, read per difficulty stratum and aggregate; fixed confidence/resamples/seed; deterministic baseline-tie rule.
  • Split held-out users by user identity (P1) → fixed in a895738, §2.4: cross-user generalization requires a user-disjoint split, independent of the source-song split, both fixed by dataset_split_id.
  • Rename proposal steps to avoid canonical stage labels (P1) → fixed in 51b4b86: parent proposal and companion relabelled Stage NStep N with a glossary-§0 Scope note; zero Stage N strings remain in the companion.
  • Record audition data for anchor A (P2) → fixed in a895738, §2.2: anchor_audition { anchor_played, anchor_listen_ms, anchor_completed, anchor_replay_count }, plus per-mix playback identities for the A+B / A+C complementarity task.

CI is green on the v2 head. Please re-review a8957387. @codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a8957387e5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +365 to +367
only labels derivable from a `choice` are therefore **B-over-C** or
**C-over-B** *in the context of `A` and the task* — an "A-over-B" label is not
observed and must never be minted:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep Step 3 on B-over-C labels

Although this paragraph correctly prohibits synthesizing A-over-B labels, the parent proposal's Step 3 still specifies a pairwise logistic loss on derived A-over-B labels (preference-and-similarity-learning.md:120-122). Following that learner contract would recreate exactly the unobserved comparison this schema rejects, so the parent must instead consume the contextual B-over-C/C-over-B preferences defined here.

Useful? React with 👍 / 👎.

Comment on lines +432 to +434
- similarity / variation / copy-detection: `A`, then `B`, then `C`.
- complementarity: `A` solo, then `A+B` together, then `A+C` together — the
listener judges the two mixes, never the candidates alone (§1.3).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Randomize candidate playback order

This procedure always auditions B before C (and A+B before A+C), while §2.5 randomizes only the on-screen display_order. When the item manifest's B/C identities are fixed, the human choice therefore remains confounded by playback recency/order even though the document claims early/late position is controlled; randomize and record the realized audition order as well.

Useful? React with 👍 / 👎.

Comment on lines +226 to +227
- **Candidate source**: one near-quotation of `A` (verbatim or
transposed/resolution-shifted) and one musically remote fragment. The remote

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Make copy-detection strata constructible

This mandates one near quotation and one remote candidate for every copy-detection item, but §2.4 requires every task to populate medium, hard, and hard_negative strata such as near-vs-near and fails the gate when any stratum fails. No copy-detection item set can satisfy both contracts, so define task-specific strata or broaden the candidate construction before making the per-stratum gate normative.

Useful? React with 👍 / 👎.

Comment on lines +554 to +558
**Baseline ties are handled deterministically.** When the baseline's two
candidate scores are equal (the `baseline_tie` stratum, or any exact tie), the
baseline's `correct(baseline, i)` is defined by a single pre-registered rule —
**half-credit** (`0.5`) by default, with *abstain* (drop the item from the
baseline arm) and a fixed fallback ordering as the named alternatives — chosen

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Define paired handling for every tie outcome

The gate is still undefined when the candidate metric ties, which is plausible for discrete symbolic scores, because this policy covers only baseline ties. Moreover, the named abstain alternative drops an item only from the baseline arm, making the per-item paired difference impossible to compute. Specify one deterministic rule for ties from either predictor, and if abstention is allowed, drop the entire pair rather than one arm.

Useful? React with 👍 / 👎.

Comment thread docs/decisions.log.md Outdated

- 2026-07-26 — In the context of designing the Human Similarity Benchmark
(`docs/proposals/human-similarity-benchmark.md`, the companion spec for
Stage 2 of the preference-and-similarity-learning proposal), facing the

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Relabel the decision-log reference as Step 2

The parent and companion now explicitly reserve Step 2 for this proposal-local sequence, but this canonical decision record still calls it Stage 2, which denotes the unrelated MIDI transport roadmap stage S2. The root AGENTS.md requires stage numbering to follow the glossary rather than inventing labels, so this remaining reference should be changed to Step 2.

AGENTS.md reference: AGENTS.md:L110-L110

Useful? React with 👍 / 👎.

Comment on lines +222 to +224
- **Question**: "**A** is an existing riff. Which of **B** and **C** is *too
close* to A to count as original?" A single two-alternative forced choice,
asking **one** thing.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Update the parent copy-detection question

This correctly changes copy detection to the single question “which is too close,” but the parent proposal still summarizes the task as “which is too close / too far?” (preference-and-similarity-learning.md:100). A collection slice following the parent can therefore reintroduce the unobserved “too far” judgment that v2 explicitly removed; update the parent summary to match this contract.

Useful? React with 👍 / 👎.

Architecture accepted; close the remaining executable-contract contradictions.

Parent proposal synced to the companion's evidence contract:
- Step 3 learner: pairwise loss over contextual B-over-C / C-over-B labels
  (conditioned on anchor, task, question variant), not the forbidden A-over-B.
- Step 2 copy-detection: the single observable question ("too close to A"),
  remote candidate a labelled control — no "too close / too far".

Companion (v2.1):
- Audition order randomized (A->B->C | A->C->B; mixes A->A+B->A+C | A->A+C->A+B)
  and recorded in a new `audition_order` field, independent of `display_order`,
  to kill the recency confound the fixed order left.
- Difficulty strata made task-specific — one near/far scale was not
  constructible for copy-detection (defined copy-vs-remote) or complementarity
  (no natural near/far order): similarity distance-margin, variation
  middle-band, complementarity feature-margin, copy-detection quotation-boundary.
- Paired tie handling made symmetric across BOTH predictors (half-credit
  default, or paired abstention that drops the whole item — never one arm),
  bound to the gate policy.
- Complementarity floor put behind a SUPERIORITY gate (lower_CI(Δ) >= ε, ε>0),
  or declared ungateable — matching random/legality is not evidence.
- Per-stratum evidence floors added (min users/source-groups/effective-items
  per stratum), not just per user/task.
- Concrete v1 ids bound: question_variant {similarity_v1, variation_v1,
  complementarity_mix_v1, copy_detection_v1}; derivation contextual_bc_preference_v1.

Metadata: decisions.log Stage 2 -> Step 2; README index notes the floor case
and the paired-difference gate.

Still docs-only; binds nothing. S9 Phase 1 EMA governing; rerank.rs untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012T7SRMiXmZe5v1imtrKMPi
@PhysShell

Copy link
Copy Markdown
Owner Author

Thanks — v2.1 pushed as d4417f8 (single focused consistency commit). All nine points accepted; none refuted. Mapping:

  1. Parent recreates forbidden labels (blocker) — fixed. Parent Step 3 now: pairwise logistic loss on the contextual B-over-C / C-over-B labels the benchmark derives, conditioned on anchor, task, and question variant, never a fabricated A-over-B.
  2. Parent copy-detection stale — fixed. Parent Step 2 task list now the single observable question ("which candidate is too close to A to count as original?"), remote candidate a labelled control.
  3. Deterministic playback order — fixed. Audition order randomized (A→B→C | A→C→B; mixes A→A+B→A+C | A→A+C→A+B) and recorded in a new audition_order field, independent of display_order (§2.2, §2.3); the analysis conditions on it.
  4. Strata not constructible per task — fixed. Replaced the one near/far scale with task-specific strata (§2.4): similarity distance-margin + same_stratum_distinct_motif; variation middle-band (clearly_better/close_call/both_off_band); complementarity feature-margin (wide/narrow/tie_margin, explicitly not near/far); copy-detection quotation-boundary (obvious_copy_vs_remote/threshold_adjacent_copy_vs_noncopy/two_threshold_adjacent). Each task populates only its own strata; §1 candidate-source wording updated to match.
  5. Tie handling breaks the paired gate — fixed (§2.6). One pre-registered rule, symmetric for either predictor, bound to the gate policy: half-credit 0.5 default, or paired abstention that drops the whole item (never one arm). correct ∈ {0, 0.5, 1}, Δ_i ∈ [−1, 1].
  6. Complementarity floor — fixed (§1.3, §2.6, §2.7). Against a random/legality-only floor the gate is superiority lower_CI(Δ) ≥ ε, pre-registered ε > 0 — beat the floor, don't match it — with the honest alternative to declare complementarity ungateable until a meaningful baseline exists. "Match random and enter production" is off the table.
  7. Per-stratum evidence floors — fixed (§2.5): no per-stratum gate is read until that stratum independently meets pre-registered minimum user, source-group, and effective-item counts (effective = after collapsing repeated probes).
  8. Concrete ids — fixed (§2.2): question_variant v1 = { similarity_v1, variation_v1, complementarity_mix_v1, copy_detection_v1 }; derivation_policy_id v1 = contextual_bc_preference_v1; unknown ids are typed refusals.
  9. Metadata — fixed: decisions.log Stage 2Step 2 (§4); README index now "named handcrafted baseline or, where none exists (complementarity), an explicitly declared floor"; PR body rewritten to v2.1 / Step 2 / paired clustered difference / four files.

Still docs-only; S9 Phase 1 EMA governing; rerank.rs untouched. Not merging without an explicit human GO.

…er review)

Final surgical consistency pass; no new architecture.

1. Copy-detection candidate source made constructible under its own boundary
   strata: only the obvious_copy stratum is quotation-vs-remote; the boundary
   strata use candidates on/near opposite sides of the pre-registered copy
   threshold — a non-copy candidate need not be remote. Parent + companion
   "remote candidate" softened to "non-chosen candidate".
2. complementarity_mix_v1 question made order-neutral ("A paired with B and C
   in a randomized order") so the closed wording no longer contradicts the
   randomized playback of §2.3.
3. Shared gate-comparison / pass-criterion (and the parent gate rule) now carry
   the baseline-vs-floor distinction: non-inferiority against a real baseline,
   superiority against a floor, or ungateable — instead of assuming a
   handcrafted baseline for every task.
4. Statistical parameters given their own immutable identity: a separate
   griff.similarity-benchmark-gate-run v1 manifest binds the judgement-archive
   hash + tie rule / δ-or-ε / gate form / confidence / resamples / seed /
   procedure version / per-stratum floors / baseline-or-floor id / candidate id.
   Judgement records deliberately bind no gate policy (collection precedes and
   is independent of evaluation), so the same evidence can be re-gated.

Wording cleanup: complementarity strata margin is a pre-registered
controlled-feature / sampler-policy margin independent of every evaluated
candidate metric (was ambiguously "candidate-metric margin").

Still docs-only; binds nothing. S9 Phase 1 EMA governing; rerank.rs untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012T7SRMiXmZe5v1imtrKMPi
@PhysShell

Copy link
Copy Markdown
Owner Author

Thanks — v2.2 pushed as fbaae43 (one surgical commit, no new architecture). All four corrections + the wording cleanup applied:

  1. Copy-detection strata now constructible — §1.4 candidate source rewritten: two candidates sampled under the copy-detection strata; obvious_copy_vs_remote uses quotation-vs-remote-control, the boundary strata (threshold_adjacent_copy_vs_noncopy, two_threshold_adjacent) use candidates on/near opposite sides of the pre-registered copy threshold, and a non-copy candidate need not be remote. The listener is still asked only "which is the copy"; the non-chosen candidate is a control. Parent + companion "remote candidate" language softened to "non-chosen candidate".
  2. Order-neutral complementarity questioncomplementarity_mix_v1 is now "You will hear A paired with B and A paired with C, in a randomized order. Which pairing works better…", so the closed wording no longer contradicts the randomized playback (§2.3); the UI labels pairings after playback without promising which was first.
  3. Floor case in the shared contract + parent — the per-task field-set now reads "candidate metric versus the task's named handcrafted baseline or explicitly declared floor" and "the task-appropriate §2.6 gate: non-inferiority against a real baseline, superiority against a floor, or ungateable". The parent gate rule now carries the same distinction (beat/match a baseline where one exists, beat a declared floor by a pre-registered margin where none does). Companion and parent no longer disagree about complementarity's baseline.
  4. Gate policy has an identity — added a separate immutable griff.similarity-benchmark-gate-run v1 manifest binding judgement_archive_hash, tie_rule, delta_or_epsilon, gate_form, confidence_level, bootstrap_resample_count, bootstrap_seed, resampling_procedure_version, per_stratum_evidence_floors, baseline_or_floor_id, candidate_metric_id, keyed by gate_policy_id. The judgement archive deliberately binds no gate policy (collection precedes and is independent of evaluation), so the same evidence can be re-gated without rewriting a byte; unknown gate-run versions are typed refusals (§2.2, §2.6).

Wording cleanup — the complementarity strata margin is now "a pre-registered controlled-feature or sampler-policy margin independent of every evaluated candidate metric", so the benchmark can't stratify itself with the contestant it is evaluating.

Still docs-only; S9 Phase 1 EMA governing; rerank.rs untouched. CI re-running on fbaae43. Standing by for your GO — will not merge without it.

@PhysShell

Copy link
Copy Markdown
Owner Author

Closing the loop on Codex's 07:53 re-review — all findings are resolved at head fbaae43:

  • Keep Step 3 on B-over-C labels → parent preference-and-similarity-learning.md:126-128 now specifies contextual B-over-C / C-over-B labels; the only A-over-B string is the explicit prohibition (fixed in d4417f8).
  • Update the parent copy-detection question → parent :100-102 is now the single "too close to A" question; the only "too far" is the explicit prohibition (fixed in d4417f8).
  • Make copy-detection strata constructible → §1.4 candidate source is now stratum-driven, non-copy need not be remote (fixed in fbaae43).
  • Randomize candidate playback orderaudition_order randomized + recorded independently of display_order (fixed in d4417f8).
  • Define paired handling for every tie outcome → symmetric tie rule for either predictor; paired abstention drops the whole item (fixed in d4417f8).
  • Relabel decision-log reference as Step 2decisions.log:1949 now "Step 2 (§4)" (fixed in d4417f8).

CI is fully green on fbaae43. Per the arbiter's conditional GO (four v2.2 corrections + green run = GO MERGE, no further review round), merging now.

@PhysShell
PhysShell merged commit a37332e into main Jul 26, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant