feat(core,cli,web): persist near-duplicate link in ChunkMeta (schema v8) (#76)

[PhysShell]

Closes the curation gap surfaced by a DGD playtest export: the split flags near-duplicate phrases (#76) and shows them in the UI/CLI, but the signal lived only in the split envelope — a downloaded chunk.json or a built manifest lost which phrases are repeats. On that export, phrases p2≡p6, p3≡p7, p4≡p8 are byte-identical repeats, yet nothing in the chunk recorded it.

What

ChunkMeta gains an optional duplicate: Option<PhraseDuplicate> — the earlier split phrase a later one verbatim-quotes (transposition-aware) and the quote share. Both split paths now mirror the already-computed flag onto the persisted record:

"duplicate": { "of": 2, "quote_share": 1.0 }

Schema

Bumps SCHEMA_VERSION 7 → 8 under the established additive pattern (serde default + skip_serializing_if), exactly like structure (v2) / gesture (v3) / complexity (v6) / rights (v7). Pre-v8 records have no duplicate key, parse as None, and re-serialize byte-identically. PhraseDuplicate gains serde derives. Unlike the #75 tag additions — data within an existing field, deliberately unversioned — this is a structural field, so it bumps the schema (recorded in decisions.log).

Wiring

• core/src/corpus.rs — field + version + doc; core/src/novelty.rs — serde derives on PhraseDuplicate.
• web/src/lib.rs split_segments_to_json and cli/src/main.rs chunks_for_segments set meta.duplicate from the per-phrase flag_phrase_duplicates result, right where each already sets bar_range. The single-capture path stays None (no phrase-sequence context).

TDD

red (3b09acc) → green (54f70c5) → docs (18fd624). The serde round-trip (skip-when-absent / carry-when-present) and CLI persistence parity were red first; the round-trip proptest fuzzes the new field too via a new arb_duplicate() strategy.

Verification

• cargo test --workspace green; cargo clippy --workspace --all-targets -- -D warnings clean; cargo fmt --all --check clean
• web (wasm32): cargo check + cargo clippy clean
• No change to the live UI/CLI — the split envelope still carries duplicate_of/duplicate_share for the pager; this only adds the same signal to the persisted record.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TTUbGjzD8ysnVnCJnZJE95

---

Generated by Claude Code

Summary by CodeRabbit

Release Notes

• New Features

  • Chunk metadata now persists near-duplicate phrase relationships in downloaded manifests and JSON exports.

• Bug Fixes

  • Near-duplicate links are cleared/reset appropriately during re-splitting and invalidation to prevent stale duplicate data.

• Documentation

  • Corpus annotation schema updated to v8, adding an optional duplicate field using a forward-compatible pattern (omitted when unset).

• Tests

  • Updated and expanded schema and curation/similarity test coverage for the new optional field behavior.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: d8161f9a-1044-4c34-a92b-ed98f438edd9

📥 Commits

Reviewing files that changed from the base of the PR and between 18fd624 and 62739f6.

📒 Files selected for processing (2)

• preview/src/curation.rs
• preview/tests/curation.rs

---

📝 Walkthrough

Walkthrough

Adds an optional duplicate: Option<PhraseDuplicate> field to ChunkMeta, bumps SCHEMA_VERSION from 7 to 8, and derives Serialize/Deserialize on PhraseDuplicate. The web and CLI chunk-building paths are updated to populate or default meta.duplicate, curation reset logic clears the field on invalidation and split, and tests validate JSON omission, round-trip fidelity, and persistence.

Changes

ChunkMeta duplicate field — schema v8

Layer / File(s)	Summary
Schema contract: PhraseDuplicate Serde + ChunkMeta.duplicate field (v8) core/src/novelty.rs, core/src/corpus.rs, docs/decisions.log.md	PhraseDuplicate gains Serialize/Deserialize derives; ChunkMeta gains pub duplicate: Option<PhraseDuplicate> with #[serde(default, skip_serializing_if = "Option::is_none")]; SCHEMA_VERSION bumps to 8; decisions log records the additive serde pattern and meaning of duplicate.of.
Web and CLI: populate ChunkMeta.duplicate web/src/lib.rs, cli/src/main.rs	build_chunk_meta_record (web) and build_chunk_meta (CLI) initialize duplicate: None; split_segments_to_json (web) and phrase_chunks (CLI) assign near-duplicate detection results to meta.duplicate so the embedded chunk JSON persists the link.
Curation: clear duplicate on invalidation and split preview/src/curation.rs, preview/tests/curation.rs	fresh_extent now clears meta.duplicate when invalidating a chunk record. New test verifies that when a record with populated duplicate is split, both resulting halves have duplicate reset to None.
Schema round-trip and persistence tests core/tests/corpus_schema.rs, core/tests/similarity.rs, preview/tests/curation.rs, cli/src/main.rs	Schema v8 tests assert SCHEMA_VERSION == 8, verify absent duplicate is omitted from JSON (not written as null), and confirm populated values round-trip losslessly. Property tests gain arb_duplicate() with dyadic quote_share constraints. Fixture ChunkMeta structs in similarity and curation tests gain duplicate: None; the CLI near-duplicate test asserts chunks[2].meta.duplicate == chunks[2].duplicate.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related PRs

• PhysShell/griff#85: Introduced PhraseDuplicate and near-duplicate detection machinery in core/src/novelty.rs that this PR now serializes into embedded chunk metadata.
• PhysShell/griff#93: Wired near-duplicate detection into the CLI phrase-splitting flow, producing the PhraseDuplicate values that this PR persists into ChunkMeta.
• PhysShell/griff#84: Modifies web/src/lib.rs's auto-splitting path split_segments_to_json, the same path now extended to persist ChunkMeta.duplicate links to emitted chunks.

Poem

🐇 A duplicate hid in the envelope's fold,
Lost when the JSON was written and sold.
Now meta.duplicate carries the link,
Persisted in v8 before you can blink!
The schema is bumped, the round-trips are tight —
This bunny's chunk records are perfectly right. ✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: persisting near-duplicate links in ChunkMeta with a schema version bump to v8, which is the core objective across multiple files.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/modest-maxwell-0zecwn

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 18fd624649

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Split this schema change into red/green commits

The /workspace/griff AGENTS.md TDD workflow says “Never commit new pub fn / pub struct implementation in the same commit as the tests that cover it” and that reviewers must judge the commit sequence. This single commit adds the public ChunkMeta.duplicate schema field and its covering tests together, so the required failing-test commit is absent; please split the history into a red test commit followed by the implementation commit.

Useful? React with 👍 / 👎.

[PhysShell]

This reads the flattened PR diff rather than the commit sequence — the history is already red→green, as AGENTS.md asks reviewers to judge:

• 3b09acc test(core,cli): red — adds the covering test chunk_meta_persists_and_skips_the_near_duplicate_link (and the CLI persistence assertion). It does not compile: error[E0609]: no field 'duplicate' on type 'ChunkMeta'.
• 54f70c5 feat: green — adds pub duplicate: Option<PhraseDuplicate> and makes that test pass.

Evidence:

• git log -G 'pub duplicate: Option' -- core/src/corpus.rs → only 54f70c5
• git log -G 'chunk_meta_persists_and_skips' -- core/tests/corpus_schema.rs → only 3b09acc

The failing-test-first commit is present and precedes the implementation; the "single commit" is the squashed-diff view, not the branch history.

---

Generated by Claude Code

[chatgpt-codex-connector]

Clear stale duplicate links on extent edits

When a split-generated chunk carrying duplicate is later re-curated with preview split/merge, preview/src/curation.rs uses fresh_extent to reset invalidated whole-extent state but will now preserve this new field. Because duplicate.of is only meaningful for the original split sibling index, both halves or the merged record can keep a stale duplicate pointer/share after their source.bar_range and ids change; clear it in the extent-changing paths alongside the other reset metadata.

Useful? React with 👍 / 👎.

[PhysShell]

Good catch — fixed in 62739f6. fresh_extent is the single reset shared by both split halves (split_record) and merge_records, so clearing duplicate there covers every extent-changing path: a re-cut extent no longer keeps a stale sibling pointer, alongside the reviewer/structure/gesture/complexity resets. Covered by split_record_clears_the_near_duplicate_link (red 1a310e3 → green 62739f6).

---

Generated by Claude Code