Skip to content

fix(patch): reject wrong-region matches in the fuzzy edit fallback - #54575

Open
MaxFreedomPollard wants to merge 2 commits into
NousResearch:mainfrom
MaxFreedomPollard:fix/fuzzy-match-wrong-region
Open

fix(patch): reject wrong-region matches in the fuzzy edit fallback#54575
MaxFreedomPollard wants to merge 2 commits into
NousResearch:mainfrom
MaxFreedomPollard:fix/fuzzy-match-wrong-region

Conversation

@MaxFreedomPollard

Copy link
Copy Markdown
Contributor

Fixes #54572

Problem

The patch tool in replace mode locates old_string through the nine strategy chain in tools/fuzzy_match.py. The last two strategies, block_anchor and context_aware, accept a region based on a shared first and last line (with a 0.50 middle similarity) or on half of its lines matching. They run only after the seven formatting-tolerant strategies have failed, which means old_string is genuinely not in the file at that point. When one of them fires, the tool replaces a region whose content differs from old_string and returns a successful single match. patch_replace verifies that the bytes it wrote landed, but not that the matched region was the one described, so the wrong edit is not caught.

A concrete case: a file whose middle line is audit_log(request.user_id), edited with an old_string that names validate(request.token) (a line not in the file) but shares the surrounding two lines. block_anchor matches, the audit_log line is replaced, and the call reports success.

Fix

After a match is produced by one of the two content divergent strategies, drop any matched region that is not substantially the same text as old_string, comparing with unicode form and whitespace runs normalized away (_normalized_similarity, floor 0.90). Legitimate reflow, indentation and smart quote drift normalize to about 1.0 and still match. A region that merely shares an anchor line falls below the floor, so the match is discarded and the chain falls through to the existing no match path, which already attaches a "did you mean" hint that prompts a re-read.

The guard runs after the existing escape-drift check, so the more specific escape-drift error still wins where it applies.

This is intentionally narrow. The seven formatting-tolerant strategies are untouched, since they already guarantee the region equals old_string apart from formatting. A genuine near miss where a single line drifted stays well above the floor and still applies, so real edits are not regressed.

Tests

tests/tools/test_fuzzy_match.py:

  • the block_anchor wrong-region case leaves the file unchanged and reports no match,
  • the context_aware half-matching case preserves the two unrelated lines,
  • a single line drift in an otherwise identical block still applies.

The full file suite passes (53 tests).

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the focused guard. The premise is confirmed on current main: block_anchor accepts shared-boundary candidates at the 0.50 middle-similarity threshold (tools/fuzzy_match.py:665-691), while context_aware accepts a block when only half its lines are highly similar (tools/fuzzy_match.py:716-726). fuzzy_find_and_replace then applies the selected spans (tools/fuzzy_match.py:85-153).

Problems

  • The new coverage is limited to fuzzy_find_and_replace in tests/tools/test_fuzzy_match.py. The externally visible mutation path is ShellFileOperations.patch_replace (tools/file_operations.py:1590-1616), so the no-write guarantee is not exercised through real file I/O.

Suggested changes

  • Add a live patch_replace regression in tests/tools/test_file_tools_live.py using the wrong-region fixture and assert both an error/no replacement and unchanged on-disk contents. The neighboring live tests establish this convention at tests/tools/test_file_tools_live.py:281-301.

Automated hermes-sweeper review.

"def handler(request):\n"
" rate_limit(request)\n"
" return process(request)"
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also cover this fixture through ShellFileOperations.patch_replace and assert the file remains byte-identical. This is the user-visible mutation path, whereas this test only verifies the pure matcher.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 15, 2026
@MaxFreedomPollard
MaxFreedomPollard force-pushed the fix/fuzzy-match-wrong-region branch 4 times, most recently from febd4b3 to 3f0c8be Compare July 31, 2026 03:46
@Likivik

Likivik commented Aug 6, 2026

Copy link
Copy Markdown

Wrote a live test for the no-write guarantee:

  • test_block_anchor_wrong_region_is_rejected — calls patch_replace through real file I/O. old_string shares first/last lines with the file but the middle differs. Fails on main with the expected silent-replace diff; passes on your branch.
  • test_legitimate_rescue_still_applies — sanity check that real edits (indent drift + value change) still apply.

Commit: Likivik@7610ac4ca

Disclosure: I wrote this test (and this comment) with help from an LLM. Reasoning is mine; wording has been polished.

Cherry-pick 7610ac4, or merge Likivik:fix/fuzzy-match-wrong-region — whatever's easier. Thanks.

MaxFreedomPollard and others added 2 commits August 13, 2026 17:15
The patch tool's replace mode falls through to block_anchor and
context_aware when the seven formatting-tolerant strategies fail. Those
two accept a region on a shared first/last line or on half its lines, so
they can overwrite a region whose content differs from old_string and
still report a successful single match, with no signal that an unrelated
line was destroyed.

Add a content-similarity guard (unicode + whitespace normalized, floor
0.90) applied to those two strategies, after the existing escape-drift
check. A region that only shares an anchor line falls below the floor and
is discarded, so the chain falls through to the no-match path and the
caller re-reads instead of editing the wrong place. The seven formatting
strategies are untouched and a genuine single-line drift still applies.
Adds a real-file-I/O test to tests/tools/test_file_tools_live.py that
exercises the patch_replace code path with a wrong-region pattern
(blocks anchored on first/last lines only). Before PR NousResearch#54575, the
block_anchor / context_aware strategies could silently overwrite a
different region and report success; the patch_replace post-write
verification only confirmed the bytes landed, not that the right
region was matched.

This test:

* asserts patch_replace returns a PatchResult with error set when
  old_string shares only the first/last lines with the file,
* asserts the on-disk file is byte-identical to its pre-patch
  state (audit_log line preserved, validate / rate_limit lines absent),
* includes a positive sanity test that a real edit (indentation
  drift + value change) still applies so the guard does not regress.

Without the fix this test fails on main with the silent-replace
diff ('audit_log(request.user_id)' -> 'rate_limit(request)').

Refs NousResearch#54572.
@MaxFreedomPollard
MaxFreedomPollard force-pushed the fix/fuzzy-match-wrong-region branch from 3f0c8be to c6c85d8 Compare August 13, 2026 21:15
@MaxFreedomPollard

Copy link
Copy Markdown
Contributor Author

@Likivik thank you, this is exactly the coverage the review asked for and it is a nicer test than the one I would have written. Cherry-picked as c6c85d8 with your authorship on the commit, and rebased the fix onto current main at aae5298.

I checked your claim rather than taking it on trust, and it holds: with the fix reverted, test_block_anchor_wrong_region_is_rejected fails on clean main with patch_replace returning success=True and no error for a wrong-region match, and test_legitimate_rescue_still_applies passes throughout. So it fails for the right reason and the sanity case shows the guard is not just rejecting everything.

That closes @teknium1's point directly. The gap was that the coverage stopped at fuzzy_find_and_replace in test_fuzzy_match.py while the externally visible mutation path is ShellFileOperations.patch_replace, and your test goes through real file I/O and asserts the on-disk bytes are untouched, which is the part that actually matters for a no-write guarantee.

The rebase conflict was a clean one, main added is_already_applied and _format_match_locations in the same region where this branch adds _CONTENT_VERIFIED_STRATEGIES and _normalized_similarity, so both sides are kept.

81 tests pass across test_file_tools_live.py and test_fuzzy_match.py.

Thanks also for the note about how you wrote it. Appreciated, and it did not change how I reviewed it, the test either reproduces the bug or it does not, and this one does.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades tool/file File tools (read, write, patch, search) type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

patch tool (replace mode) can edit the wrong region when old_string is not an exact match

4 participants