feat(core): flag near-duplicate phrases via novelty (#76, part 3)

[PhysShell]

Part 3 (final) of #76. After the over-long cap (#83) and trivial-drop (#84), this flags near-duplicate phrases — the playtest's "chorus/verse/filler repeats (p8≈p4, p9≈p5…)" — using the existing transposition-aware measure_novelty. Per the design call: flag, don't drop (non-destructive; the curator decides, aided by the #78 compare overlay).

What

• Core core/src/novelty.rs: new flag_phrase_duplicates(phrases, track_index, min_quote_share) -> Vec<Option<PhraseDuplicate>>. For phrase i it compares against phrases 0..i via measure_novelty and flags it as ≈ phrase N when the longest verbatim (transposition-/resolution-aware) quote covers ≥ min_quote_share of its notes. First occurrence is canonical; only later repeats are flagged. Both sides are reduced to the detected track (single_track_line) so a reference resolves to the same musical line even if an earlier track also sounds. PHRASE_DUPLICATE_SHARE = 0.8 default (tunable).
• Web web/src/lib.rs: the split emits duplicate_of / duplicate_share per phrase.
• UI app.js: the pager shows · ≈ phrase N (X% quote) in splitInfo, pointing the curator at which phrase to compare against.

Approach

Red → green (core algorithm): de8eaf8 adds failing tests (compile-fail, the repo's TDD idiom for a new API); 1840d89 implements it and wires the web path.

Verification

• core novelty 19/19 (3 new: a later repeat flagged ≈ the first at ~100%; distinct phrases unflagged; a track-isolation case proving the detected track is compared on both sides — it fails without single_track_line). Full core suite green; web split 5/5; JS 23/23; clippy -D warnings clean.
• End-to-end (rebuilt wasm + DGD GPX): of the 11 phrases, 6 are flagged — p6≈p2, p7≈p3, p8≈p4, p9≈p5 (100%), p10≈p1 (93%), p11≈p2 (89%) — exactly the song's verse/chorus repeats. Screenshot in the thread.

This completes #76's three parts (over-long / trivial / dedup). Flagging in the CLI chunk metadata (for the corpus pipeline) is a clean optional follow-up.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TTUbGjzD8ysnVnCJnZJE95

---

Generated by Claude Code

Summary by CodeRabbit

• New Features
  • Added phrase near-duplicate detection to identify verbatim melodic repeats using quote-coverage thresholds.
  • Split/auto-split output now includes duplicate metadata per phrase (original index and approximate quote share).
• UI Updates
  • Split phrase rendering now shows when a phrase is flagged as a near-duplicate, along with the source index and rounded quote coverage.
• Tests
  • Added coverage for phrase de-duplication behavior, including track-specific comparisons and unflagged distinct phrases.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: e9ac4c50-d492-4be6-a8f4-ed34c212af29

📥 Commits

Reviewing files that changed from the base of the PR and between 1840d89 and 62bb92c.

📒 Files selected for processing (1)

• core/tests/novelty.rs

🚧 Files skipped from review as they are similar to previous changes (1)

• core/tests/novelty.rs

---

📝 Walkthrough

Walkthrough

Adds phrase-level near-duplicate detection to the core novelty module via flag_phrase_duplicates, PhraseDuplicate, and PHRASE_DUPLICATE_SHARE. The web split pipeline calls this function and emits duplicate_of/duplicate_share JSON fields per phrase. The front-end renderPhrase displays duplicate metadata when present.

Changes

Phrase Near-Duplicate Detection

Layer / File(s)	Summary
Core detection: PhraseDuplicate and flag_phrase_duplicates core/src/novelty.rs	Adds LossReport import, PHRASE_DUPLICATE_SHARE constant, PhraseDuplicate struct, flag_phrase_duplicates function, and internal single_track_line helper. Each phrase is compared against all earlier phrases using single-track Score views and measure_novelty; phrases whose quote share meets min_quote_share are flagged.
Tests for flag_phrase_duplicates core/tests/novelty.rs	Imports flag_phrase_duplicates and adds three tests: verbatim repeat flagged at ~100% share, distinct phrases unflagged, and decoy-track scenario verifying correct track_index isolation.
Web pipeline: duplicate metadata emission web/src/lib.rs	Imports flag_phrase_duplicates and PHRASE_DUPLICATE_SHARE. In split_segments_to_json, derives phrase_scores, calls flag_phrase_duplicates, and emits duplicate_of/duplicate_share (or null) fields for each phrase in the JSON output.
UI: duplicate info in renderPhrase web/static/app.js	splitInfo.textContent in renderPhrase conditionally appends duplicate_of index and rounded duplicate_share percentage when a phrase chunk carries duplicate metadata.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related issues

• engine/curation: phrase segmentation — dedup repeats, drop trivial, split over-long phrases #76 — This PR directly implements the phrase near-duplicate detection mechanism (flag_phrase_duplicates using measure_novelty) requested in issue #76 for identifying and flagging duplicate/repeat phrases during curation.

Poem

🐇 Hopping through the melody stack,
I sniff each phrase and then look back—
"This tune I've heard! A copy, a clone!"
quote_share near one, it's not unknown.
I stamp it duplicate_of with care,
Fresh tunes get null—the original hare! 🎵

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately reflects the main change: adding phrase-level duplicate detection via novelty analysis. It references the feature work (#76, part 3) and clearly conveys the primary implementation.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/modest-maxwell-0zecwn

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@core/tests/novelty.rs`:
- Around line 427-440: The test
flag_phrase_duplicates_compares_the_detected_track_on_both_sides uses the same
decoy track in both phrases at position 0, so it will pass even if the duplicate
detection code incorrectly compares track 0 instead of track 1. To properly
validate track isolation, replace the decoy track in the second phrase with a
different decoy track (having different pitches than the first phrase's decoy)
while keeping the same REF_PITCHES at track 1. This ensures the test will fail
if the code mistakenly compares track 0 instead of the intended track 1.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 31137ee7-40d1-47b1-b209-849ec3f3ee7c

📥 Commits

Reviewing files that changed from the base of the PR and between d60982d and 1840d89.

📒 Files selected for processing (4)

• core/src/novelty.rs
• core/tests/novelty.rs
• web/src/lib.rs
• web/static/app.js

[PhysShell]

,@codex

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: 62bb92c2c4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".