Skip to content

feat(grep): fold shared path prefix in file-list output - #2630

Closed
israellot wants to merge 1 commit into
rtk-ai:developfrom
israellot:feat/grep-fold-path-prefix
Closed

israellot wants to merge 1 commit into
rtk-ai:developfrom
israellot:feat/grep-fold-path-prefix

Conversation

@israellot

Copy link
Copy Markdown
Contributor

Problem

rtk grep -l / -L / --files route through passthrough() and print paths verbatim — one per line, each repeating the same long directory prefix. When an agent passes an absolute search root (e.g. grep -ril foo /home/user/project/...), every line restates /home/user/project/.... That shared prefix is the single most strippable part of the output, yet it was forwarded uncompressed and track_passthrough recorded 0 tokens saved for the command.

$ rtk grep -ril websocket /home/u/proj/findings /home/u/proj/methodology
/home/u/proj/findings/vpn_login_websocket_hang.md
/home/u/proj/findings/meet_realtime_listen_detect.md
/home/u/proj/methodology/client_server_time_sync.md
...                       # ← prefix repeated on every line, 0% savings

Change

Add fold_path_prefix (in grep_cmd.rs): factor the longest common directory prefix into a one-line header, then group the remaining tails by sub-directory — the same dir/ file1 file2 shape find_cmd already emits, so no new output format is invented (Transparency principle), and the path-grouping logic mirrors existing code (Extensibility principle).

/home/u/proj/ (3 files)
findings/ vpn_login_websocket_hang.md meet_realtime_listen_detect.md
methodology/ client_server_time_sync.md

Wired into passthrough(), which is where the raw-path leak happened (the -l/-L/--files format-flag branch and the unparseable-shape fallback both funnel through it).

Correctness guarantees

  • Lossless — every input path is recoverable as header_prefix + dir + "/" + file. No truncation, so no CAP_* / recovery hint needed.
  • Never-worse guarded — wrapped in guard::never_worse; non-path passthrough output finds no common prefix (None) and is emitted verbatim.
  • Component-boundary safe — the common prefix is computed on whole path components, so /a/foobar and /a/foobaz share /a/, never /a/fooba.
  • Returns None (caller keeps raw) for <2 paths, no shared directory, bare filenames, or a degenerate /-only prefix.
  • Tracking switched from track_passthrough (0/0) to track so the savings are actually measured. When nothing folds, in == out — no dilution.

The default grouped-match grep path (file:line:content) is unchanged.

Savings

Honest numbers: pure file-lists are bounded by the filenames themselves (irreducible signal), so this lands ~30–55% depending on prefix depth and file count rather than the 60%+ typical of verbose-output filters — but it converts a measured 0% → positive on a common agent operation. Examples measured locally:

  • Shallow 2-dir case (-ril, 5 files): ~50%
  • Deep Unity-style abs paths (3 files): 38% (252 → 156 bytes)

Testing

  • 9 new unit tests for fold_path_prefix (absolute, relative, sub-dir grouping, component-boundary, single-path/no-common/bare-filename/root-only → None, size-shrink invariant).
  • cargo fmt --all --check && cargo clippy --all-targets && cargo test all pass (2268 passed, 8 ignored).
  • grep smoke tests pass; the unrelated smoke failures (curl/wget/pnpm/rewrite/init) are pre-existing and environment-bound (verified identical with this change stashed).
  • Manual before/after verified on the motivating grep -ril case.

Docs

Updated src/cmds/system/README.md — the line that said -l/-L "bypass RTK filtering and run raw" now notes the path-prefix folding.

`grep -l` / `-L` / `--files` route through passthrough() and printed raw
paths verbatim — one per line, each repeating the same long (often
absolute) directory prefix. That redundancy was forwarded uncompressed
(0% savings) even though it is the most strippable part of the output.

Add `fold_path_prefix`: factor the longest common directory prefix
(computed on whole path components, never a partial segment) into a
one-line header and emit only the differing tails. The transform is
lossless (every path = prefix + tail) so no truncation/recovery hint is
needed, and it is wrapped in `never_worse` so non-path passthrough output
that finds no common prefix is emitted verbatim. Switch passthrough's
tracking from `track_passthrough` (0/0) to `track` so the new savings are
measured.

Example (`grep -ril websocket /home/u/proj/findings /home/u/proj/methodology`):

  before: 5 lines, each prefixed /home/u/proj/...
  after:  /home/u/proj/ (5 files)  +  5 short tails   (~40-53% fewer tokens)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@CLAassistant

CLAassistant commented Jun 25, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@KuSh

KuSh commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator

Thank you for this. The file this changes, src/cmds/system/grep_cmd.rs, was removed on develop by #2641 (merged 2026-06-26), which rebuilt grep and rg around a shared Engine model in src/cmds/system/search.rs, and its passthrough path also gained a streaming mode that never buffers the whole output. That means the fold_path_prefix helper and its tests no longer have a place to attach, so I am closing this. The underlying gap is real: file-list passthrough still prints each path in full and records no savings, so an issue or a small PR against search.rs passthrough would be very welcome. If I have misread part of this, please comment or reopen.

@israellot

Copy link
Copy Markdown
Contributor Author

Thanks @KuSh, that reading is correct. Re-targeted at the search.rs passthrough in #4170: same fold, now applied to grep -l/-L, rg -l, and rg --files under never_worse, with the streaming stdin path left untouched.

KuSh pushed a commit that referenced this pull request Sep 25, 2026
`rtk grep -l`, `grep -L`, `rg -l`, and `rg --files` print every path in
full and record 0/0 savings. Agents search from absolute roots, so each
line repeats the same long directory prefix.

Fold that prefix into a `<prefix> (N files)` header and print each
path's tail. The transform is lossless, keeps engine order, cuts only on
whole components, and runs under `never_worse`. It applies only to a
bare file list: any other shape flag (`-c`, `-Z`, `--null`, `--json`,
`-q`) changes the line and leaves the list verbatim. Streaming stdin
passthrough is untouched.

Track the fold with real sizes so it shows in `rtk gain`; unchanged
passthrough keeps recording 0/0.

Follow-up to #2630, which targeted the pre-#2641 grep_cmd.rs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants