fix(git): stop silently truncating large diffs

[georgebashi]

Summary

• Root cause: compact_diff capped output at 100 lines per hunk and 500 lines total, silently replacing later content with truncation markers. This affected git diff/show/stash show, gh pr diff, glab mr diff, and the git-diff pipe filter.
• Remove both caps and preserve every diff line. Only lossless compaction remains: metadata removal plus per-file +N -M summaries. All six call sites were updated; existing passthrough options are unchanged.
• This completes the compact_diff side of the never-truncate fix, which patched the sibling condense_unified_diff but missed these paths. It also restores failed downstream searches such as gh pr diff | grep, and fixes a leading-context bug.
  • prior fix that missed this path: bug(critical): silent truncation causes data loss — LLM takes decisions on incomplete diffs #827
  • leading-context bug also fixed here: fix(git): compact_diff drops context lines before first change in hunk #1852
• This is the content-side fix: it changes what rtk emits. It is complementary to two other lines of work that change whether rtk runs — neither of which covers this case:
  • pipe-routing (intentionally excludes git/gh/glab): fix(hooks): don't rewrite pipe-feeding segments whose filter changes output shape #2965, fix: add is_unsupported_shape() guard before rewrite_compound() #2903
  • non-TTY passthrough (proposed, not implemented): Silent output corruption when stdout is piped or redirected (suggest isatty passthrough) #1282, No isatty() passthrough: rtk-decorated output flows into pipes (and the hook rewrites pipeline stages), breaking grep/xargs consumers — caused a destructive rm #2861
• Removing truncation also avoids RTK-specific recovery markers in parseable diff output, per the Transparency principle. guard::never_worse still prevents output from exceeding raw input.
• Related truncation / structured-output work:
  • git log -p patch bodies dropped: Hook-rewritten git log -p | rg … silently drops patch bodies → false-clean secret scans #2944, fix(git): preserve patch output from log commands #2951
  • --porcelain / --name-only compaction: rtk compacts machine-bound git output (--porcelain / --name-only / piped), silently corrupting consumers #2487
  • rtk diff breaks git apply / exit codes: rtk diff and rtk git diff produce output that breaks programmatic consumers (patch, git apply, --name-only, exit codes) #1918

Test plan

• cargo fmt --all --check && cargo clippy --all-targets && cargo test — 2,495 passed, 8 ignored, no warnings
• Replaced truncation assertions with content-preservation tests; added leading-context coverage
• Added git_diff_large_change_preserves_all_content: a 1,200-line diff retains and can grep its final-line marker
• Release binary: raw diff 2,407 lines; compact diff 2,411 lines (previously ~110); final-line grep succeeds; zero truncation markers
• Recovery-hint/cap check: N/A because diff truncation was removed

Closes #1852.

AI-assisted contribution.

[CLAassistant]

All committers have signed the CLA.

[KuSh]

Requesting changes. To be straight about where this stands: the diagnosis is right, but the patch as written does not do what it says — and I only found that by measuring, so this isn't a style objection.

I built the patched branch and compared its binary against the pre-patch binary on real diffs from this repo. The full suite passes. The problem is inline: every retained line still gains a two-space prefix, so once the caps are gone the compacted output is larger than the raw input, never_worse discards it, and rtk git diff silently becomes a raw passthrough.

The good news is that the fix is small and isn't yours to write. #3269 removes exactly that two-space prefix from hunk bodies, for unrelated reasons. With it applied, compaction is net-negative again and the arithmetic that breaks this PR stops applying. So the useful next step is to re-measure on top of it:

git fetch upstream pull/3269/head && git cherry-pick <the compact_diff commit>

then compare rtk git show <sha> byte counts against the pre-patch binary on a few commits. If you'd rather not carry a cherry-pick on your branch, say so and I'll run the measurement and post the numbers here — happy to do that either way, since it's the same handful of commands I already ran.

On the never-truncate policy itself: .claude/rules/cli-testing.md declares ≥60% savings a release blocker, and a compact_diff that never truncates can't guarantee that by construction. That's a project-level call rather than something to settle in this thread, and it's being discussed separately — so please leave your reasoning stated in the PR body. The argument that silent truncation costs more tokens downstream, because the agent re-fetches what was dropped, is the strongest form of the case and it's worth having on record.

Two smaller things that don't need their own threads:

• Unbounded allocation. result: Vec<String> now takes one heap String per input line, plus the join("\n") copy; it was previously bounded at 500 entries. gh pr diff feeds it up to the 10 MiB RAW_CAP (src/core/stream.rs:286), roughly 200k lines — several hundred MB transient, against the <5 MB target in CLAUDE.md. Streaming into a single String with push_str and a reserved capacity removes the per-line allocation.
• &out[out.len().saturating_sub(300)..] (tests/guard_integration_test.rs:297, same shape at git.rs:2454) is not char-boundary-safe. It's only evaluated when the assertion fails, which is exactly when you want the message rather than a byte index is not a char boundary panic. String::from_utf8_lossy on the tail fixes it.

[KuSh]

Measured, not inferred. With the caps removed, every retained line still gets format!(" {}", line) — two bytes per line — while the only bytes removed are roughly 140 per file (index, ---, +++, the shortened diff --git). Past about fifteen changed lines the compacted string exceeds the raw input, and never_worse(&raw, &printed) (git.rs:203, git.rs:318, runner.rs:141 for gh pr diff / glab mr diff, pipe_cmd.rs:276) discards it.

Bytes emitted by rtk git show <sha> on this repo:

commit	raw	pre-patch	post-patch
87702bf	31524	19375	31140
4147e99	31548	20162	31549 (= raw)
ab48a68	26121	13112	26122 (= raw)
d3f31dd	15385	11661	15386 (= raw)

And on this PR's own 1,200-line scenario, rtk git diff emits 42,641 bytes against 42,504 raw — the output carries diff --git/index headers and zero +/ - indented lines, i.e. it is the raw fallback.

Net effect: roughly 0-1% reduction instead of 35-50% on the main path, and compact_diff becomes dead code for anything non-trivial. A second-order consequence worth knowing about: the emitted format now flips between "stat + Changes: + indented" for small diffs and plain git output for large ones, so anything parsing rtk's diff output has to handle both shapes.

This also means the new regression test at tests/guard_integration_test.rs:270 passes through the raw fallback and never exercises compact_diff at all — CHANGED_0, CHANGED_1199, _finalize and "no truncation marker" are all satisfied by unfiltered git diff output, so re-introducing a cap inside compact_diff would not fail it. Once the prefix is gone, assert on the compacted shape (the +N -M summary) so the test can see the function it's guarding.

[KuSh]

"This never drops diff content" isn't accurate, and a doc comment is the wrong place to overstate it, since it's what future readers will trust.

Still dropped after this change: \ No newline at end of file (line 378), and everything between diff --git and the first @@ — Binary files a/x and b/x differ, similarity index / rename from / rename to, new file mode / deleted file mode. Verified: a staged binary change plus a rename compacts to two bare filenames, with no indication of what happened to either.

Separately, gh pr diff output above RAW_CAP (10 MiB) is still silently line-dropped in src/core/stream.rs:418, upstream of compact_diff — so "completes the never-truncate fix" overstates the reach even for the callers this does cover.

Suggest "drops diff metadata" rather than "never drops content", plus a sentence naming which metadata.

One more, low severity, but it belongs next to this comment because the comment now explicitly promises these lines survive: the !line.starts_with("+++") / !line.starts_with("---") guards mean a content line ++ foo (which arrives as +++ foo) or -- foo (arriving as --- foo) falls through to the context branch — printed, but never counted in added/removed. Verified: a diff turning -- old dashes into ++ new pluses produces no +N -M line at all, while git diff --stat reports one insertion and one deletion. Pre-existing, and orthogonal to the caps, but it undercuts the same promise.

[KuSh]

Correction to my earlier review — I cited the wrong policy, and it was the part of my comment most likely to discourage you.

I wrote that ".claude/rules/cli-testing.md declares ≥60% savings a release blocker". That file is stale. CONTRIBUTING.md is authoritative and the figure is 20%, and it is an admission criterion for what belongs in RTK at all, not a per-release gate every filter must clear:

Commands that produce text output (typically 100+ tokens) whose bytes can be compressed 20%+ without losing essential information for the LLM.

Two further things in CONTRIBUTING.md I should have pointed you to instead, because they support your position rather than block it:

• "When in doubt, preserve correctness."
• Never Block already settles the truncation question: "capping output at N items is only acceptable if accompanied by a hint that lets the agent recover the hidden data."

That last line reframes this PR. Truncation is not forbidden by policy — truncation without a recovery path is. rtk git diff already satisfies it: it emits [full diff: rtk git diff --no-compact], and that flag exists.

Where the policy is actually violated is the sibling callers. gh pr diff and glab mr diff route through the same compact_diff and print the same hint, but neither has a --no-compact equivalent — so the recovery hint names a command the user cannot run. Same for rtk pipe -f git-diff.

So I would narrow the ask rather than widen it: instead of removing truncation globally, give those callers a recovery path (a --no-compact passthrough, or a hint naming the real escape hatch for each). That is a policy defect with a concrete fix, it is far easier to review than a behaviour change across every caller, and it needs nobody to grant an exemption.

Unchanged from my review: the measured finding stands on its own and is unaffected by any of this. With the caps removed, every retained line still carries a two-space prefix, so the compacted output exceeds the raw input, never_worse discards it, and rtk git diff becomes a raw passthrough — 0-1% instead of 35-50%. #3269 removes that prefix; re-measuring on top of it is still the next step, and my offer to run those numbers for you stands.

Apologies for the detour — the "project-level decision being discussed separately" I described does not exist in the form I described it.

[KuSh]

Thanks for the careful write-up on diff truncation. compact_diff was reworked in #3269 (merged 2026-09-01): hunk bodies now emit at column 0 so anchored greps work again, dropped lines are disclosed split by sign as "... (N deletions, M additions truncated)", and the leading-context bug you point at is bounded by max_lines/10 inside the same function. The "[full diff: rtk git diff --no-compact]" marker and the --no-compact escape hatch for git diff and gh pr diff were already on develop before this PR opened, so the truncation is disclosed rather than silent, and the caps were kept deliberately. The patch here also reinstates the two-space indent that #3269 removed, so it can no longer be rebased as written. Two gaps are still real and would make good narrow issues or PRs: glab mr diff compacts at 500 lines with no --no-compact of its own (src/cmds/git/glab_cmd.rs:490), and the git-diff pipe filter caps at 200 lines with no opt-out (src/cmds/system/pipe_cmd.rs:56). Please comment or reopen if part of this was missed.