Skip to content

feat(rewrite): rewrite multiline blocks - #3319

Merged
aeppling merged 6 commits into
developfrom
feat/rewrite-multiline-blocks
Aug 3, 2026
Merged

aeppling merged 6 commits into
developfrom
feat/rewrite-multiline-blocks

Conversation

@aeppling

@aeppling aeppling commented Jul 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Multi-line Bash blocks (a shape coding agents emit constantly) were treated as one command: the rtk prefix landed on the first line and every following line ran raw and unfiltered. This PR splits blocks at the newline tokens the quote-aware lexer emits and rewrites each line through the existing single-line path. Implements the line-by-line rewrite proposed in #1243.

$ rtk rewrite 'git status
# check history
  grep -rn foo src'
rtk git status
# check history
  rtk grep -rn foo src
  • Lines continued by a trailing && / || / | / |& are joined and rewritten as one logical command (git status && ⏎ git log -3 → rtk git status && rtk git log -3), exactly what develop produced for the single-line form.
  • Quoted multi-line strings (multi-line commit messages) keep rewriting as one prefixed command, as develop always did.
  • Blank lines, comment lines, indentation, and CRLF separators are preserved verbatim.
  • Conservative by design — the whole block passes through untouched (native permission handling sees the original) whenever a line can't be attested: control flow keywords, heredocs (including << split across a \<NL> continuation), grouping/arithmetic spanning lines, swallowed newlines (quote state opened by an apostrophe in a comment), a continuation operator hidden behind a trailing comment, $'…' containing \' (the one ANSI-C case where the lexer's view diverges from bash), and |& stderr pipes.

How this was built

This PR supersedes #3188 (@revopsrocks), taken over after the author became unresponsive; their three commits are cherry-picked with authorship preserved.

  1. feat(rewrite): rewrite each line of multi-line Bash blocks #3188 as submitted — line-by-line rewrite + two adversarial review rounds by the original author (swallowed newlines, cross-line arithmetic, array literals/function bodies, comment-stripped independence checks, ANSI-C bail).
  2. Maintainer review fixes (review on feat(rewrite): rewrite each line of multi-line Bash blocks #3188, applied here):
    • narrowed the $' bail to the actual hazard (\' inside a $'…' span) so an unrelated echo $'a\tb' line no longer disables the block;
    • cross-line &&/||/| lists are now joined and rewritten instead of declined (develop already handled the joined form);
    • comment_start recognizes # after operator bytes (|&;()), so git log |# keep ⏎ grep -f patterns.txt passes through instead of rebuilding a guarded pipeline;
    • the heredoc/$(( gate runs on the bash-faithful continuation join (\<NL> → nothing) before the space-join normalization erases the operator.
  3. Differential fuzz round (below) — found one genuine regression vs develop (quoted multi-line strings declined) and fixed it: zero emitted newline tokens + balanced quote state at EOF proves the block is one logical command, which is rewritten whole; unbalanced quote state (the apostrophe-comment hazard) still declines.

Differential fuzz validation

971-case corpus (real per-ecosystem commands, compounds, multiline blocks, cross-line continuations, control flow, deep quoting, heredoc/redirect matrix, permission-bypass attempts, plus the #3188 adversarial cases), run 3-way through the real rtk rewrite hook path — develop vs this PR vs #2781 — with every rewrite validated by bash -n:

binary rewrote invalid rewrites crashes
develop 598/971 (61.6%) 0 0
this PR 684/971 (70.4%) 0 0
#2781 (2c89559) 717/971 (73.8%) 1 0
  • 105 blocks newly rewritten; 208 upgraded from partial to full (of 579 cases both rewrote, develop had prefixed only the first line in 208 — silent incomplete coverage the percentage never showed).
  • 19 deliberately declined that develop "rewrote": apostrophe-in-comment blocks (develop's output left lines 2+ unfiltered anyway), degraded comment-in-continuation joins, no-op trims.
  • Pipe-consumer protection verified: zero cases where any stage feeding a pipe gained an rtk prefix; cross-line pipes join into the same rewrite_compound machinery (final-stage-only, pipeline_final_command_is_safe, grep -f rejected, |& raw).
  • feat(rewrite): rewrite commands bundles #2781's coverage lead is partly mis-rewrites its own bash -n check can't see; this PR declines all of them: cat <\ ⏎ <EOF → cat < <EOF (the corpus's only syntax error), $(\ ⏎ (1+2)) arithmetic corrupted into command substitution, rtk spliced inside a $'…' literal's value, grep -f rebuilt as the final stage of a pipeline behind a |# comment, and function bodies prefixed (changes x=$(foo) captured output).

Permission verdicts

Verified live against Claude Code allow/deny rules (.claude/settings.json): a block auto-allows (exit 0) only when every line independently matches an allow rule (#1213 aggregation); one unmatched line downgrades to ask; a deny match on any line wins outright; and a block whose newline was swallowed by quote state can never auto-allow — the same condition that blinds split_for_permissions forces the rewrite to decline, and exit 0 is impossible without a rewrite. split_for_permissions and the multiline splitter share the same newline-emitting tokenizer, so a verdict is always computed for exactly the lines being rewritten.

Test plan

  • cargo fmt --all && cargo clippy --all-targets && cargo test --all (2609 passed, clippy clean)
  • 971-case 3-way differential fuzz (above), every rewrite bash -n-validated
  • ~45 unit tests on the multiline path covering the rewrite and every passthrough gate

Follow-ups (out of scope here)

  • Replace the hand-rolled attestation checks with a real Bash grammar (tree-sitter-bash or a pure-Rust parser) — the apostrophe-comment cluster (18 declined cases, a very common agent shape) needs comment-aware parsing to rewrite safely. Being discussed with maintainers; startup-latency and binary-size budgets to be measured first.
  • Statement-level rewriting when a different line holds a redirect/substitution (feat(rewrite): rewrite commands bundles #2781's remaining legitimate coverage lead).
  • Check the fuzz corpus + runner into the repo as the regression suite for the above.

Ref #1243. Supersedes #3188. Related: #2781.

revopsrocks and others added 4 commits July 30, 2026 13:47
Coding agents routinely emit multi-line Bash blocks of sequential
commands. The rewriter tokenized newlines as plain whitespace, so a
block was treated as one command: the rtk prefix landed on the first
line and every following line ran raw and unfiltered.

Split multi-line input at the newline tokens the quote-aware lexer
emits (newlines inside quoted strings are never split points) and
rewrite each line through the existing single-line path. Blank lines,
comment lines, indentation, and CRLF separators are preserved verbatim.

Per-line rewriting only applies when every line is an independent
command. The whole block passes through untouched when:
- a shell keyword opens control flow (for/if/while/case/...)
- a list or pipeline continues across the line break (&&, ||, |, |&
  at a line edge)
- a subshell or group spans lines
- the block contains a heredoc (existing gate)

Ref #1243
Hardening from adversarial review of the multi-line rewrite:

- If any newline byte was swallowed by quote state (raw \n/\r count vs
  emitted newline tokens), pass the whole block through. The lexer has
  no comment awareness, so an apostrophe in a # comment opens quote
  state and hides subsequent lines; rewriting such a block would act on
  lines no permission verdict accounted for. Passthrough hands the
  original command to native permission handling. Quoted multi-line
  strings (commit messages) forgo their rewrite as the safe trade.
- Bail when (( or )) sits at a line edge: arithmetic spanning lines
  must not get a command prefix spliced into arithmetic context.
- Strip trailing unquoted comments before the independence checks:
  'git log | # keep pipeline' continues the pipeline across the
  newline even though the line ends in comment text, and rewriting
  the next line would rebuild a pipeline whose final stage the
  pipeline-safety guard may specifically reject (e.g. grep -f).
- Bail when a line's unquoted ()/{} don't balance: array literals
  (arr=(one), function bodies (foo() {), and groups span lines, so
  surrounding lines are not independent commands. Subsumes the
  previous single-char edge checks.
- Bail on any block containing $'...': inside ANSI-C quoting bash
  treats \' as a literal quote that does not close the string, but
  the lexer thinks it does — emitting a split point bash would never
  honor, the inverse of the swallowed-newline case the count check
  catches.
@aeppling
aeppling marked this pull request as draft July 30, 2026 12:23
@aeppling
aeppling marked this pull request as ready for review July 30, 2026 16:31
Comment thread src/discover/registry.rs
Comment thread src/discover/registry.rs
Comment thread src/discover/registry.rs
Comment thread src/discover/registry.rs

@KuSh KuSh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks

@aeppling
aeppling merged commit 3044911 into develop Aug 3, 2026
17 checks passed
@rtk-release-bot rtk-release-bot Bot mentioned this pull request Aug 3, 2026
@aeppling aeppling mentioned this pull request Aug 4, 2026
2 tasks
cubicj pushed a commit to cubicj/rtk that referenced this pull request Aug 11, 2026
seathatflowsinourveins pushed a commit to native-agent-stack/native-agent-stack that referenced this pull request Sep 27, 2026
Adds a stdlib, fail-open SubagentStart hook that returns
adoption/hooks/claude/token-lanes-block.md as additionalContext to every
subagent except agent types starting "blind-", registers it in the Claude
settings template beside ai-memory, and installs both files through the
existing sha256-checked guard step.

docs/token-session-handbook.md "Token lanes carried into subagents" is the
source of truth the block copies line for line. Its RTK line is the measured
RTK 0.50.0 rule (rtk hook check and the rtk hook claude PreToolUse hook;
rewrite_multiline_block, rtk-ai/rtk#3319): && chains and multi-line blocks
are rewritten segment by segment; $(...), backticks, <(...), file redirects
and heredocs are never rewritten; gh pipelines are not rewritten.

Native probe (Claude Code 2.1.283, Agent-tool Haiku children): with the hook
the child found and quoted the block's first 60 characters exactly; without
it, and for a blind-judge child with it, the child found nothing.

Decision record: docs/decisions/2026-09-27-token-lanes-subagent-start.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
seathatflowsinourveins pushed a commit to native-agent-stack/native-agent-stack that referenced this pull request Sep 27, 2026
Adds a stdlib, fail-open SubagentStart hook that returns
adoption/hooks/claude/token-lanes-block.md as additionalContext to every
subagent except agent types starting "blind-", registers it in the Claude
settings template beside ai-memory, and installs both files through the
existing sha256-checked guard step.

docs/token-session-handbook.md "Token lanes carried into subagents" is the
source of truth the block copies line for line. Its RTK line is the measured
RTK 0.50.0 rule (rtk hook check and the rtk hook claude PreToolUse hook;
rewrite_multiline_block, rtk-ai/rtk#3319): && chains and multi-line blocks
are rewritten segment by segment; $(...), backticks, <(...), file redirects
and heredocs are never rewritten; gh pipelines are not rewritten.

Native probe (Claude Code 2.1.283, Agent-tool Haiku children): with the hook
the child found and quoted the block's first 60 characters exactly; without
it, and for a blind-judge child with it, the child found nothing.

Decision record: docs/decisions/2026-09-27-token-lanes-subagent-start.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
seathatflowsinourveins pushed a commit to native-agent-stack/native-agent-stack that referenced this pull request Sep 27, 2026
Adds a stdlib, fail-open SubagentStart hook that returns
adoption/hooks/claude/token-lanes-block.md as additionalContext to every
subagent except agent types starting "blind-", registers it in the Claude
settings template beside ai-memory, and installs both files through the
existing sha256-checked guard step.

docs/token-session-handbook.md "Token lanes carried into subagents" is the
source of truth the block copies line for line. Its RTK line is the measured
RTK 0.50.0 rule (rtk hook check and the rtk hook claude PreToolUse hook;
rewrite_multiline_block, rtk-ai/rtk#3319): && chains and multi-line blocks
are rewritten segment by segment; $(...), backticks, <(...), file redirects
and heredocs are never rewritten; gh pipelines are not rewritten.

Native probe (Claude Code 2.1.283, Agent-tool Haiku children): with the hook
the child found and quoted the block's first 60 characters exactly; without
it, and for a blind-judge child with it, the child found nothing.

Decision record: docs/decisions/2026-09-27-token-lanes-subagent-start.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
seathatflowsinourveins added a commit to native-agent-stack/native-agent-stack that referenced this pull request Sep 27, 2026
…be, measured RTK rule) (#378)

* SubagentStart token-lanes carrier for non-blind subagents

Adds a stdlib, fail-open SubagentStart hook that returns
adoption/hooks/claude/token-lanes-block.md as additionalContext to every
subagent except agent types starting "blind-", registers it in the Claude
settings template beside ai-memory, and installs both files through the
existing sha256-checked guard step.

docs/token-session-handbook.md "Token lanes carried into subagents" is the
source of truth the block copies line for line. Its RTK line is the measured
RTK 0.50.0 rule (rtk hook check and the rtk hook claude PreToolUse hook;
rewrite_multiline_block, rtk-ai/rtk#3319): && chains and multi-line blocks
are rewritten segment by segment; $(...), backticks, <(...), file redirects
and heredocs are never rewritten; gh pipelines are not rewritten.

Native probe (Claude Code 2.1.283, Agent-tool Haiku children): with the hook
the child found and quoted the block's first 60 characters exactly; without
it, and for a blind-judge child with it, the child found nothing.

Decision record: docs/decisions/2026-09-27-token-lanes-subagent-start.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Native probe and RTK hook-check receipt for the token-lanes carrier

evidence/artifacts/token-lanes-subagent-start-20260927/ retains, sanitized,
the five native probe runs (Claude Code 2.1.283, Agent-tool Haiku children:
without the hook, with it, and a blind-judge child with it, plus the failed
v1 question kept as a failed design) and 17 RTK 0.50.0 inputs through both
`rtk hook check` and the `rtk hook claude` PreToolUse processor. Raw CLI
output and child transcripts stay private and are identified by SHA-256 and
counts. Evidence class local_integration; not native_proven for Workflow
children, which have not run with the hook.

The handbook's RTK sources paragraph and the decision record now cite the
receipt instead of unretained observations.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Count the carrier block's budget in UTF-8 bytes (cross-family review)

The GPT-6 review of #378 at 85dd27c found that the text test checked
len(block), a character count, although the decision record states a
3,500-byte budget: a 3,280-character, 3,520-byte block would pass. The test
now measures the file's bytes through fits_budget() and adds that case as a
regression test.

Controls: a character-counting fits_budget fails the regression test, and a
block padded to 3,560 bytes (3,360 characters) fails the byte check but
passed the old character check. The committed block is ASCII, 3,159 bytes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Register the carrier's files on main 1aa9776 (after #366 and #380)

Replayed onto main: the bootstrap.md note merges after #366's OTEL tool-details
sentence, manifests/evidence.json is re-registered, and the matrix and grand
list are regenerated. validate FAILS=0; 438 related tests OK.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Scout <scout@local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants