Repository navigation
Conversation
`grep -l` / `-L` / `--files` route through passthrough() and printed raw paths verbatim — one per line, each repeating the same long (often absolute) directory prefix. That redundancy was forwarded uncompressed (0% savings) even though it is the most strippable part of the output. Add `fold_path_prefix`: factor the longest common directory prefix (computed on whole path components, never a partial segment) into a one-line header and emit only the differing tails. The transform is lossless (every path = prefix + tail) so no truncation/recovery hint is needed, and it is wrapped in `never_worse` so non-path passthrough output that finds no common prefix is emitted verbatim. Switch passthrough's tracking from `track_passthrough` (0/0) to `track` so the new savings are measured. Example (`grep -ril websocket /home/u/proj/findings /home/u/proj/methodology`): before: 5 lines, each prefixed /home/u/proj/... after: /home/u/proj/ (5 files) + 5 short tails (~40-53% fewer tokens) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collaborator
|
Thank you for this. The file this changes, src/cmds/system/grep_cmd.rs, was removed on develop by #2641 (merged 2026-06-26), which rebuilt grep and rg around a shared Engine model in src/cmds/system/search.rs, and its passthrough path also gained a streaming mode that never buffers the whole output. That means the fold_path_prefix helper and its tests no longer have a place to attach, so I am closing this. The underlying gap is real: file-list passthrough still prints each path in full and records no savings, so an issue or a small PR against search.rs passthrough would be very welcome. If I have misread part of this, please comment or reopen. |
Contributor
Author
KuSh
pushed a commit
that referenced
this pull request
Sep 25, 2026
`rtk grep -l`, `grep -L`, `rg -l`, and `rg --files` print every path in full and record 0/0 savings. Agents search from absolute roots, so each line repeats the same long directory prefix. Fold that prefix into a `<prefix> (N files)` header and print each path's tail. The transform is lossless, keeps engine order, cuts only on whole components, and runs under `never_worse`. It applies only to a bare file list: any other shape flag (`-c`, `-Z`, `--null`, `--json`, `-q`) changes the line and leaves the list verbatim. Streaming stdin passthrough is untouched. Track the fold with real sizes so it shows in `rtk gain`; unchanged passthrough keeps recording 0/0. Follow-up to #2630, which targeted the pre-#2641 grep_cmd.rs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
rtk grep -l/-L/--filesroute throughpassthrough()and print paths verbatim — one per line, each repeating the same long directory prefix. When an agent passes an absolute search root (e.g.grep -ril foo /home/user/project/...), every line restates/home/user/project/.... That shared prefix is the single most strippable part of the output, yet it was forwarded uncompressed andtrack_passthroughrecorded 0 tokens saved for the command.Change
Add
fold_path_prefix(ingrep_cmd.rs): factor the longest common directory prefix into a one-line header, then group the remaining tails by sub-directory — the samedir/ file1 file2shapefind_cmdalready emits, so no new output format is invented (Transparency principle), and the path-grouping logic mirrors existing code (Extensibility principle).Wired into
passthrough(), which is where the raw-path leak happened (the-l/-L/--filesformat-flag branch and the unparseable-shape fallback both funnel through it).Correctness guarantees
header_prefix + dir + "/" + file. No truncation, so noCAP_*/ recovery hint needed.guard::never_worse; non-path passthrough output finds no common prefix (None) and is emitted verbatim./a/foobarand/a/foobazshare/a/, never/a/fooba.None(caller keeps raw) for <2 paths, no shared directory, bare filenames, or a degenerate/-only prefix.track_passthrough(0/0) totrackso the savings are actually measured. When nothing folds, in == out — no dilution.The default grouped-match grep path (
file:line:content) is unchanged.Savings
Honest numbers: pure file-lists are bounded by the filenames themselves (irreducible signal), so this lands ~30–55% depending on prefix depth and file count rather than the 60%+ typical of verbose-output filters — but it converts a measured 0% → positive on a common agent operation. Examples measured locally:
-ril, 5 files): ~50%Testing
fold_path_prefix(absolute, relative, sub-dir grouping, component-boundary, single-path/no-common/bare-filename/root-only →None, size-shrink invariant).cargo fmt --all --check && cargo clippy --all-targets && cargo testall pass (2268 passed, 8 ignored).grep -rilcase.Docs
Updated
src/cmds/system/README.md— the line that said-l/-L"bypass RTK filtering and run raw" now notes the path-prefix folding.