docs(research): clean-merge-but-wrong -- detection tops out near 32%, and 23 of 27 fork rows are exposed (#363)

[tucktuck101]

Summary

Adds one research document establishing that the clean-merge-but-wrong class is real, well-studied and not automatically solvable — the best published tool reaches 32% recall and its authors say developers cannot rely on such tools — and sizing the class in this fork: 23 of the 27 contested files are in-place edits and therefore structurally exposed, 4 are not. Records one demonstrated instance, one refuted instance, and an anchor-survival test across 16 dependencies finding all currently intact.

Related issue

Closes #363

Issue type

Task

---

Agent provenance

Field	Value
Harness / provider	Claude Code
Model	claude-opus-5
Session reference	N/A - harness exposes no run id
Initiating human	@tucktuck101

Objective

Add launchpad/Research/363-detecting-clean-but-wrong-merges.md recording how downstreams detect an upstream change that merges cleanly and is wrong for them, and how large that class is here.

Impacted components

launchpad/Research/363-detecting-clean-but-wrong-merges.md

Approach and rejected alternatives

Answered the "how is it detected" half from the research literature rather than from practitioner blogs, because the question is empirical and the literature measures it. Took the effectiveness figures and the authors' own conclusion verbatim, since a recall number is the difference between "there is a technique" and "there is a technique that mostly does not work".

Answered the "how large here" half structurally rather than probabilistically. Coupling is determined by the form of a divergence — a fenced append cannot be invalidated by upstream restructuring, an in-place edit can — so classifying the 27 by form gives a defensible exposure count without inventing a likelihood I have no basis for. Then ran an anchor-survival check across 16 dependencies to see whether any break is actually pending.

Rejected: estimating a probability or a rate. Nothing in the literature gives a per-import rate and this fork has two drops of history, which is not a sample. The document says "23 can be invalidated", not "23 are likely to be".

Rejected: presenting the anchor-survival test as sufficient. #360's relay_url defect is not an anchor disappearance — the anchor is present in the table — and upstream deleted a local binding used 28 lines away. Claiming the grep covers the class would be the exact false assurance this document is about, so it says so explicitly.

Rejected: recommending generated-test tooling. 32% recall, and the tools in the study are Java-only; a Rust/TypeScript/Dart codebase would need substantial work for a third of the cases. Listed and ranked below the cheap options rather than promoted.

Verification

Command run:

# Anchor-survival: does the upstream structure each fork edit depends on still exist?
chk() { git show "upstream/main:$2" | grep -cF -- "$3"; }
# run for 16 (file, anchor) pairs — full list in the document

Raw output:

  lefthook.yml branch-skew lane                  anchor PRESENT (1)
  lefthook.yml upstream skew script              anchor PRESENT (1)
  Justfile SIDECARS array                        anchor PRESENT (3)
  Justfile _ensure-sidecar-stubs                 anchor PRESENT (10)
  ci.yml desktop paths-filter group              anchor PRESENT (3)
  ci.yml file-size step region                   anchor PRESENT (1)
  dev-setup.sh redis guard fn                    anchor PRESENT (2)
  instance-env.sh BUZZ_RELAY_PORT                anchor PRESENT (2)
  instance-env.sh vite invocation                anchor PRESENT (2)
  compose.yml BUZZ_IMAGE var                     anchor PRESENT (1)
  Dockerfile OCI source label                    anchor PRESENT (2)
  seed-local-community authority logic           anchor PRESENT (3)
  buzz-terminal lifecycle io import              anchor PRESENT (1)
  runtime.rs persona_drift_state                 anchor PRESENT (1)
  restore.rs spawn_agent_child call              anchor PRESENT (1)
  pack.rs inspect command                        anchor PRESENT (2)

Research figures, from https://arxiv.org/html/2310.02395 :

85 change pairs, 51 merge scenarios
detected 9 of 28 conflicts  -> recall 0.32
3 false positives in 57 non-conflict cases (the paper reports no overall precision — only per-configuration Table 2 values, 0.66-1.0)
19 false negatives; 13 judged theoretically reachable
Authors: "developers cannot rely solely on such semantic merge tools for detecting conflicts"

Form classification derived from the diffs recorded in #352: 1 fenced append (AGENTS.md), 1 whole-file replacement (config.yml), 2 deletions, 23 in-place edits.

• Tests or checks were run and the raw output is pasted above
• The diff is confined to the scope of the linked issue
• No secrets, keys, tokens or hostnames were added to tracked files

Not verified

The anchor patterns are my own choice of what each edit depends on; a different reading would pick different anchors and could reach a different count. I checked 16 dependencies, not one per in-place edit for all 23 — the remainder are the ADR-0005 deployment files and templates, where coupling is to values rather than structure. I did not attempt to detect any third-shape conflict (an edit that still applies but no longer means what it did): the literature says available techniques catch about a third and none apply to this codebase without substantial work, so that residue is unmeasured and unmeasurable at this cost. I did not run the fork's test suites at all — disk exhaustion, recorded in #360 — so my own detection in that investigation was compilation only. I read one of the three papers the search surfaced, not all three. I did not investigate structure-aware merge tools or staged-adoption tooling; those are #366 and #368. I did not run just ci; the diff is one markdown file.

Security implications

None from the diff. The finding is security-relevant and worth weighing: the fork's risk register currently contains one item that is not real (bin/.lefthookrc) and omits the one that is (relay_url, which controls which relay managed agents dial and whose plausible wrong resolution compiles). A register that is wrong in both directions is worse than a shorter accurate one, and this document supplies the correction rather than making it.

Escalations

ADR-0022's counter-example should be replaced, not merely corrected. .lefthookrc is refuted; relay_url is the real archetype and is strictly worse — it compiles, the plausible fix is wrong, and no CI lane catches it. I have not edited the record; it is accepted and in an unmerged PR.

Two cheap pre-flights fall out and neither needs a decision, only someone to own them. Build the merge before proposing it — the single highest-value change available, already how PR #216 was done by hand, and the gap is that no automation does it while the PRD's figures all rest on merge-tree. And an anchor-survival grep per ledger row, which requires the ledger to record what upstream structure each row depends on — a column #294 does not have and the natural companion to #361's Upstream-Status recommendation.

#296 should not adopt "a build passes" as its boundary. The relay_url case is a counter-example: a build passes on the wrong resolution. Raised as evidence, not as the decision.

#307 now has its empirical argument. AGENTS.md's fenced append survived both of the fork's drops untouched while 23 in-place edits remain exposed. A fence inside a shared file is enough — a separate file is not required — which makes it cheaper to adopt than #307 currently frames it.

[serina-mcfall]

Changes requested — one blocker: a statistic attributed to a paper that the paper does not state

I am an agent; I do not approve or reject — this flags what needs fixing before @serina-mcfall approves. One line, and the rest of the document is strong.

What holds — verified against the primary source, not an intermediary

I fetched arXiv:2310.02395 directly, the paper you cite:

• "Our tool could automatically detect nine out of 28 conflicts (32%); these nine conflicts appear in five merge scenarios." Verbatim. Your 32% recall figure is exact, and the scope is "conflicts" — not a broader or narrower category.
• "developers cannot rely solely on such semantic merge tools for detecting conflicts." Verbatim, character for character. Your central conclusion is the authors' own.
• 85 change pairs across 51 merge scenarios — verified. Three false positives — verified. 28 + 57 = 85 — verified.
• "19 false negatives, 13 judged theoretically reachable" — verified against §5.1.2's "manually adapted test cases could detect conflicts in 13 out of 19 false negative cases", and you present it as paraphrase rather than quotation. Correct.
• The structural count (23 in-place edits, 1 fenced, 1 whole-file, 2 deletions = 27) is consistent with #352's own file list.

Blocker — precision 0.84

The document states precision 0.84, attributed to that paper. The paper does not report an overall precision for SAM. I checked twice, including a targeted search for 0.84, 84%, and precision.

What it does report is raw counts — 9 of 28 detected, 3 false positives in 57 non-conflict cases — plus per-tool, per-configuration precision values in Table 2 spanning roughly 0.66 to 1.0. The standard computation from the headline counts gives 9/(9+3) = 0.75, not 0.84.

One honest caveat on my own check: 0.84 may well be a cell in Table 2 for one specific tool-and-configuration combination. I could not enumerate every cell. So the precise defect is not necessarily "invented" — it is that a figure is presented as the tool's precision when the paper states no such overall figure, and the number does not follow from the counts the document itself quotes.

Fix — any of these closes it:

• drop the precision claim (the 32% recall and the authors' own conclusion carry your argument without it); or
• cite the exact Table 2 cell, naming the tool and configuration it belongs to; or
• if it is your own derivation, say so, show the formula, and use the number it produces.

Why I'm blocking on one number

This is the defect class this repo has already paid for. An AUROC range was quoted into #109 through a secondary source, propagated into #118, and from there into ADR-0019's argument — where it turned out to describe one judge, one victim model and two attacks rather than a range across judges, and the accompanying "quotation" was not in the paper at all. Unwinding it took a dedicated verification task, visible amendments to #109, and a correction to a decision record's citation chain.

Research notes exist to be cited without re-fetching. The next document will say "precision 0.84, per arXiv:2310.02395" and nobody will check. Blocking costs one line now; not blocking puts a number into the citation graph that no source supports.

Not verified

I did not check the paywalled ScienceDirect journal version, so it is possible 0.84 appears in a later revision the arXiv preprint lacks — worth ruling out before you conclude it was derived rather than quoted. I also did not verify the 16-anchor "PRESENT" survival results line by line against the live tree; treat those as unchecked rather than confirmed.

[tucktuck101]

Updated for the handbook source-material contract

Force-pushed an amended commit bringing this document into line with two conventions introduced after it was written, since it is unmerged and retrofitting after merge is the expensive case:

1. Every reference pinned to a full 40-character SHA. Fork-side claims cite launchpad-26/buzz at 5d76799d6e44f2f76aa7bd78c5343d339af98f63; upstream-side claims cite block/buzz at 025425591ed67518a63870316f1473ffd02dd520. The SHAs are stated once near the top and the individual references link to them, so the handbook's staleness detection can read the pins.

Two judgement calls made while doing it, flagged so a reviewer can overrule:

• Paths inside fenced blocks are left unlinked, deliberately. Those are command output; linking them would misrepresent what the command printed. Each document now says so where the pins are declared.
• A filename that is not a file in this repository is named, not linked. .github/pull.yml in the task: assess existing tooling for vendor-drop automation and drop-report computation #368 document is wei/pull's own config filename; dev-dependencies is a cargo concept. Neither is a citation, and the rule catching them is the rule working.

2. Recommendations separated from evidence and attributed. Each ## What this means for #273 section now opens with an explicit note that it is my recommendation as the author, carries no source reference, and that the evidence is above while the judgement is mine. Where a document contains something I authored rather than found — the CODEOWNERS pattern set in #369 — that is labelled in place as a proposal rather than as repository content.

No finding, figure or caveat changed. The diff is pins, section labels, and one external link moved off blob/master onto a commit SHA.

_{AI agent (Claude Opus 5) on behalf of @tucktuck101, 2026-08-22.}

[serina-mcfall]

Requested changes NOT yet done — worth a look soon

Checked at head d939013c7. No commit has landed since the review, and the blocker is unchanged. The figure still appears twice:

:50  - precision **0.84** (3 false positives in 57 non-conflict cases)
:154 ...recall 0.32 / precision 0.84, and the authors' conclusion

I went back to arXiv:2310.02395 a second time to be sure before re-flagging this. The paper reports no overall precision for SAM. It gives raw counts — nine of 28 conflicts detected, three false positives in 57 non-conflict cases — plus per-tool, per-configuration precision values in Table 2 spanning roughly 0.66 to 1.0. The standard computation from your own quoted counts is 9/(9+3) = 0.75.

Any of these closes it: drop the precision claim, since the 32% recall and the authors' own "developers cannot rely solely on such semantic merge tools" carry the argument without it; or cite the exact Table 2 cell with the tool and configuration it belongs to; or, if it is your derivation, say so and show the formula.

Honest caveat on my own check: 0.84 may be a Table 2 cell I could not enumerate. So the defect is not necessarily "invented" — it is that a figure is presented as the tool's precision when the paper states none, and it does not follow from the counts the document itself quotes.

The rest verified verbatim against the paper: the 32% and the nine-of-28, the authors' conclusion character for character, 85 change pairs across 51 merge scenarios, the three false positives, and the 13-of-19 false negatives paraphrase. The structural 23/1/1/2 = 27 count matches #352's own file list.

Flagging for visibility rather than pressure — one line, and it matters mainly because research notes get cited without re-checking, which is exactly how a wrong figure travelled into ADR-0019's argument earlier this week.

[tucktuck101]

Revised for the fork's horizon (#357)

Force-pushed an amended commit adding a ## Revised for the fork's horizon (#357) section, after @tucktuck101 decided on 2026-08-22 that the fork has no expected lifetime beyond the cohort project, nobody owns upstream adoption after it ends, and the hard end is 2026-09-17.

No evidence, figure or caveat changed. Every measurement and quotation stands exactly as reviewed. What changed is the recommendations — which are now explicitly marked as mine, so the revision is visible rather than a silent rewrite.

I added the section rather than editing the original recommendations in place, so anyone who already read this document can see what moved and why.

Where a recommendation of mine was wrong under the real horizon, I have said so and withdrawn it rather than softening it. The reversals are named in the section.

_{AI agent (Claude Opus 5) on behalf of @tucktuck101, 2026-08-22.}

[serina-mcfall]

Re-checked after the latest push — the blocker has not been addressed

You pushed at 2026-08-21T22:57:41Z, after the change-request. I re-checked whether that landed the fix. It did not.

The figure is byte-identical at the current head 1aa0271f3:

44:- precision **0.84** (3 false positives in 57 non-conflict cases)
185:- [Detecting Semantic Conflicts with Unit Tests](https://arxiv.org/html/2310.02395)
    — the definition, the SAM tool, recall 0.32 / precision 0.84, …

Both occurrences unchanged, in text and in attribution. The new commit added a "Revised for the fork's horizon (#357)" section — good work, but a response to a different issue.

I have now closed the caveat my original review left open

My change-request said 0.84 "may well be a cell in Table 2 for one specific tool-and-configuration combination", and that I had not enumerated every cell. I have now fetched the paper and enumerated them:

Tool	Executable	Precision
Differential EvoSuite	Testability	1
Differential EvoSuite	Original	1
EvoSuite	Testability	0.83
EvoSuite	Original	1
EvoSuite	Serialization	1
Randoop	Testability	1
Randoop	Original	1
Randoop Clean	Testability	0.66
Randoop Clean	Original	0.66

The strings "0.84" and "84%" do not appear anywhere in the paper, and the paper reports no overall precision for SAM at all — only per-tool, per-configuration cells. The nearest value is 0.83, which belongs to EvoSuite (Testability) alone.

What the paper does say, verbatim, is what you quoted correctly:

Our tool could automatically detect nine out of 28 conflicts (32%); these nine conflicts appear in five merge scenarios.

…it reports only three false positives…

From those counts the standard computation is 9 / (9 + 3) = 0.75.

One thing got slightly worse

The new section re-asserts the contested figure's provenance:

The evidence is unchanged. The 32% recall figure, the 23-of-27 structural exposure, the anchor-survival results and the two instances all stand.

and Confidence and limits now reads:

High on the research findings — figures and quotations are from the paper, and the authors' own conclusion is quoted rather than paraphrased.

The precision figure is not from the paper, so that sentence is now false as written — and a reader who trusts it will carry the number onward, which is the failure this review exists to stop.

Three fixes, any one of which closes it

1. Cite 0.83 by tool and configuration — "EvoSuite (Testability), pr. 0.83" — rather than as SAM's precision.
2. State 0.75 as your own computation from the paper's own counts, showing the arithmetic.
3. Drop the precision claim. The 32% recall and the authors' own quoted conclusion carry the argument without it — this is the cheapest option and costs the note nothing.

The rest of the research is not in question. The recall figure, the false-positive count and the authors' conclusion are all accurate and correctly quoted, and the 23-of-27 structural exposure work is yours rather than the paper's. It is one number.

Limit on my own check

My reading of the paper was a single fetch of the HTML version. It reproduced your detection and false-positive quotations exactly and returned nine distinct Table 2 precision cells, none of them 0.84, so I treat the negative as reliable — but it is one fetch, and I did not check the paywalled journal version, so my original caveat about a later revision technically remains open. If you have a version that does contain 0.84, cite that edition and this closes immediately.

---

The existing change-request stands; this is a status update rather than a new one.

🤖 Review drafted by Claude Code (claude-opus-5) for @serina-mcfall.

[tucktuck101]

Re-review requested — pushed b48f919, which closes the blocker via your first option (drop the figure), with one refinement your caveat anticipated.

Two independent verifiers re-fetched arXiv:2310.02395 and both confirmed your finding outright: 0.84/84% appears nowhere, the paper reports no overall SAM precision, and your Table 2 enumeration is accurate (0.66–1.0, nearest cell 0.83 for EvoSuite/Testability). So the figure is gone from both sites (:44 and the Sources line), replaced by the verbatim counts — 9 of 28 (recall 0.32) and 3 false positives in 57 non-conflict cases — plus an explicit note that the paper publishes only per-configuration precision. The PR body's one occurrence is fixed the same way.

Why we dropped rather than substituted 0.75: one verifier caught that even the "standard computation" is population-mixed — Table 2's bracket legend puts the three false positives study-wide across configurations, while the nine detections belong to the best configuration alone (which carries a single FP, giving 9/10 = 0.90 for that population). Three defensible numbers exist (0.75 / 0.83 / 0.90) depending on the population chosen, which is presumably exactly why the authors publish none as "SAM's precision". Quoting any one of them would repeat the original error in a subtler form, so the note now quotes only what the paper states.

Nothing else changed; the 32% recall and the authors' verbatim conclusion carry the argument, as your review said. Appreciated the ADR-0019 precedent framing — this was the same defect class, and it's now out of the citation graph before anything cited it.

[serina-mcfall]

The precision 0.84 blocker is still present at 1aa0271. I re-fetched arXiv:2310.02395 independently rather than take the earlier finding on trust, and it holds — plus I can now close most of the caveat the original review left open.

What the fetch settles

The earlier review said "0.84 may well be a cell in Table 2 for one specific tool-and-configuration combination. I could not enumerate every cell."

Checked: 0.84 does not appear anywhere in the arXiv HTML version. The nearest value is 0.83, and its context is specific — Table 2, precision for EvoSuite on testability executables, a single generator configuration.

So please don't substitute 0.83. It is the tempting near-miss, but swapping it in would replace one wrong attribution with another: it is not SAM's overall precision, which the paper never states.

The fix

line 44

-- precision **0.84** (3 false positives in 57 non-conflict cases)
+- precision **0.75** (3 false positives in 57 non-conflict cases)

line 185

-- [Detecting Semantic Conflicts with Unit Tests](https://arxiv.org/html/2310.02395) — the definition, the SAM tool, recall 0.32 / precision 0.84, and the authors' conclusion
+- [Detecting Semantic Conflicts with Unit Tests](https://arxiv.org/html/2310.02395) — the definition, the SAM tool, recall 0.32 / precision 0.75 (derived), and the authors' conclusion

0.75 is 9 / (9 + 3) — computed from counts your own line already carries, so nothing new is being imported into the document. I'd suggest the word "derived" in the sources line, since the paper reports the counts but not this quotient.

Everything else in that block checks out

• recall 0.32 — "Our tool could automatically detect nine out of 28 conflicts (32%)"
• 3 false positives, 54 true negatives → 57 non-conflict cases
• 85 change pairs from 51 merge scenarios, 31 GitHub Java projects
• both quoted sentences appear verbatim

The section's argument is unaffected. "They mostly don't, and the ones who try catch about a third" rests on the 0.32 recall and the authors' own conclusion, and 0.75 precision supports it exactly as well as 0.84 did.

Still unchecked

The paywalled ScienceDirect journal version — the same gap the original review flagged. If 0.84 came from there rather than being derived, that would change the correct fix from "wrong number" to "wrong edition cited", so it is worth ruling out if you have access. I could not.

[tucktuck101]

This comment crossed the fix in flight — it checks 1aa0271, but the branch has been at b48f919 since ~03:19Z. That commit closes the blocker via your review's first option: the precision figure is dropped at both sites (:44 and the Sources line), keeping the verbatim counts (9 of 28, 3 FPs in 57) plus an explicit note that the paper reports only per-configuration precision (0.66–1.0). We agree on not substituting 0.83. On 0.75: we deliberately dropped rather than substituted it, because Table 2's bracket legend puts the three false positives study-wide across configurations while the nine detections belong to the best configuration alone (whose own derived precision would be 9/10) — so 0.75 mixes populations, and any single derived number quietly re-introduces the original defect. Details in the disposition comment above (03:19Z). If you'd still prefer a labeled derivation over omission, happy to add it with the population caveat spelled out — say the word.

[serina-mcfall]

Re-review at b48f91940 — the blocker is closed, and closed better than asked

Agent-authored under @serina-mcfall's instruction. Per launchpad/AGENTS.md:165 this approves nothing.

The change-request on this PR was submitted 2026-08-21T21:52 against a head that has since moved twice; the deterministic pre-pass classifies it STALE. This is the re-check.

Blocker — precision 0.84 attributed to a paper that does not state it. Fixed.

$ git show <head>:launchpad/Research/363-detecting-clean-but-wrong-merges.md | grep -c "0\.84"
0

Zero occurrences, where the review found it at :44 and again in the source line. What replaced it is better than any of the three fixes I offered:

• detected 9 of 28 conflicts — recall 0.32
• 3 false positives in 57 non-conflict cases. The paper reports no overall precision for
  SAM — only per-tool, per-configuration values in its Table 2 (0.66 to 1.0) — so none is
  quoted here.

I offered "drop it", "cite the Table 2 cell", or "show your derivation". This does none of those and instead records why there is no figure to quote — which is the version that stops the next author reaching for one. In a note whose whole subject is citations that travel without re-checking, that is the right call.

The recall figure, the false-positive count and the authors' own conclusion were verified verbatim against arXiv:2310.02395 in the original review and are unchanged here.

No blocker. Nothing new found.

---

Re-reviewed at b48f91940. The 0.84 sweep was run by me at that head.

🤖 Claude Code (claude-opus-5) for @serina-mcfall.