Skip to content

fix(registry): only rewrite last pipe segment; prevent data corruption in pipes/redirects - #2274

Closed
youbamj wants to merge 1 commit into
rtk-ai:developfrom
youbamj:fix/rewrite-last-pipe-segment
Closed

youbamj wants to merge 1 commit into
rtk-ai:developfrom
youbamj:fix/rewrite-last-pipe-segment

Conversation

@youbamj

@youbamj youbamj commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes critical data corruption bugs where RTK incorrectly filters piped or redirected commands, breaking downstream tools and producing malformed output files.

Real Bugs Fixed

Bug 1: Large JSON truncated, breaks jq

$ rtk curl https://jsonplaceholder.typicode.com/posts | jq
jq: parse error: Unfinished string at EOF at line 14, column 0

Root cause: RTK filters curl output, truncating large JSON responses. The truncated data is piped to jq, which fails to parse incomplete JSON.

Bug 2: Git diff corrupted when redirected to file

Before (RTK-filtered patch - BROKEN):

README.md | 2 ++
 1 file changed, 2 insertions(+)

--- Changes ---

README.md
  @@ -2,6 +2,8 @@
  +Change
  +
   <p align="center">
     <strong>High-performance CLI proxy that reduces LLM token consumption by 60-90%</strong>
   </p>
  +2 -0

After this fix (real patch - CORRECT):

diff --git a/README.md b/README.md
index d91e21b..dabe2cd 100644
--- a/README.md
+++ b/README.md
@@ -2,6 +2,8 @@
   <img src="https://avatars.githubusercontent.com/u/258253854?v=4" alt="RTK - Rust Token Killer" width="500">
 </p>
 
+Change
+
 <p align="center">
   <strong>High-performance CLI proxy that reduces LLM token consumption by 60-90%</strong>
 </p>

Result: git apply /tmp/rtk-test.patch fails with "No valid patches in input" on the RTK version, but succeeds on the real version.

Algorithm

  1. Find last | in entire command
  2. Everything before last | → stays raw (intermediate pipe segments)
  3. Everything after last | → normal rewrite rules:
    • Check for file redirect (>, >>, &>) → skip entirely
    • Check if supported → prepend rtk if yes
  4. Input redirect < is fine (reading from file, output still to terminal)

Examples

Input Output Reason
git log | cat && git stash git log | rtk cat && rtk git stash Only last segments get rtk
cargo test 2>&1 | head > file None Redirected to file, skip
cat < file.txt rtk read - < file.txt Input redirect is OK
find . | xargs grep pattern find . | xargs rtk grep pattern xargs is transparent
curl ... | jq None Pipe chain stays raw

Changes

  • src/discover/registry.rs (+198, -62 lines)
    • Refactored rewrite_compound() with last-pipe detection
    • Extracted rewrite_segments() helper
    • Added has_stdout_redirect() (checks >, >>, &> only)
    • Fixed < input redirect to not block rewrite
    • Added xargs to SHELL_PREFIX_BUILTINS
    • Added 14 new/updated tests

Testing

cargo test        # 2016 passed, 0 failed
cargo fmt --all   # clean
cargo clippy --all-targets  # clean

Breaking Change

This changes the behavior of rtk rewrite for piped commands. Previously all commands in a pipe got rtk prefix; now only the last segment does. This is the correct Unix behavior but may surprise users relying on the old (broken) behavior.

Implement Unix philosophy: when commands are piped, only the last
segment outputs to terminal. All commands before the last pipe stay
raw; only the portion after gets RTK processing.

Changes:
- rewrite_compound(): find last | globally, leave before raw,
  process after with normal per-segment logic
- has_stdout_redirect(): only block on > >> &> (not < input redirect)
- cat < file.txt: special case → rtk read - < file.txt
- xargs: add as transparent prefix (xargs grep → xargs rtk grep)
- Add tests for pipe chains with &&/||/; operators
- Add tests for input redirect (<) behavior

Fixes stdout passthrough for piped/redirected commands.
@youbamj

youbamj commented Aug 22, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #3128 (merged → 0.44.0): upstream shipped a native tokenizer-based pipeline rewrite (rewrite_pipeline_final_stage) that implements the same last-stage-only behavior. Closing as obsolete — the local patch is being dropped from the Nix derivation in favor of the upstream implementation.

@youbamj youbamj closed this Aug 22, 2026
@youbamj
youbamj deleted the fix/rewrite-last-pipe-segment branch August 22, 2026 12:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant