Repository navigation
feat(rewrite): rewrite multiline blocks - #3319
Merged
Merged
Conversation
Coding agents routinely emit multi-line Bash blocks of sequential commands. The rewriter tokenized newlines as plain whitespace, so a block was treated as one command: the rtk prefix landed on the first line and every following line ran raw and unfiltered. Split multi-line input at the newline tokens the quote-aware lexer emits (newlines inside quoted strings are never split points) and rewrite each line through the existing single-line path. Blank lines, comment lines, indentation, and CRLF separators are preserved verbatim. Per-line rewriting only applies when every line is an independent command. The whole block passes through untouched when: - a shell keyword opens control flow (for/if/while/case/...) - a list or pipeline continues across the line break (&&, ||, |, |& at a line edge) - a subshell or group spans lines - the block contains a heredoc (existing gate) Ref #1243
Hardening from adversarial review of the multi-line rewrite: - If any newline byte was swallowed by quote state (raw \n/\r count vs emitted newline tokens), pass the whole block through. The lexer has no comment awareness, so an apostrophe in a # comment opens quote state and hides subsequent lines; rewriting such a block would act on lines no permission verdict accounted for. Passthrough hands the original command to native permission handling. Quoted multi-line strings (commit messages) forgo their rewrite as the safe trade. - Bail when (( or )) sits at a line edge: arithmetic spanning lines must not get a command prefix spliced into arithmetic context.
- Strip trailing unquoted comments before the independence checks:
'git log | # keep pipeline' continues the pipeline across the
newline even though the line ends in comment text, and rewriting
the next line would rebuild a pipeline whose final stage the
pipeline-safety guard may specifically reject (e.g. grep -f).
- Bail when a line's unquoted ()/{} don't balance: array literals
(arr=(one), function bodies (foo() {), and groups span lines, so
surrounding lines are not independent commands. Subsumes the
previous single-char edge checks.
- Bail on any block containing $'...': inside ANSI-C quoting bash
treats \' as a literal quote that does not close the string, but
the lexer thinks it does — emitting a split point bash would never
honor, the inverse of the swallowed-newline case the count check
catches.
aeppling
marked this pull request as draft
July 30, 2026 12:23
aeppling
marked this pull request as ready for review
July 30, 2026 16:31
pszymkowiak
approved these changes
Jul 31, 2026
KuSh
reviewed
Aug 1, 2026
KuSh
requested changes
Aug 1, 2026
Merged
2 tasks
cubicj
pushed a commit
to cubicj/rtk
that referenced
this pull request
Aug 11, 2026
feat(rewrite): rewrite multiline blocks
4 tasks done
This was referenced Sep 5, 2026
Open
This was referenced Sep 13, 2026
seathatflowsinourveins
pushed a commit
to native-agent-stack/native-agent-stack
that referenced
this pull request
Sep 27, 2026
Adds a stdlib, fail-open SubagentStart hook that returns adoption/hooks/claude/token-lanes-block.md as additionalContext to every subagent except agent types starting "blind-", registers it in the Claude settings template beside ai-memory, and installs both files through the existing sha256-checked guard step. docs/token-session-handbook.md "Token lanes carried into subagents" is the source of truth the block copies line for line. Its RTK line is the measured RTK 0.50.0 rule (rtk hook check and the rtk hook claude PreToolUse hook; rewrite_multiline_block, rtk-ai/rtk#3319): && chains and multi-line blocks are rewritten segment by segment; $(...), backticks, <(...), file redirects and heredocs are never rewritten; gh pipelines are not rewritten. Native probe (Claude Code 2.1.283, Agent-tool Haiku children): with the hook the child found and quoted the block's first 60 characters exactly; without it, and for a blind-judge child with it, the child found nothing. Decision record: docs/decisions/2026-09-27-token-lanes-subagent-start.md. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
seathatflowsinourveins
pushed a commit
to native-agent-stack/native-agent-stack
that referenced
this pull request
Sep 27, 2026
Adds a stdlib, fail-open SubagentStart hook that returns adoption/hooks/claude/token-lanes-block.md as additionalContext to every subagent except agent types starting "blind-", registers it in the Claude settings template beside ai-memory, and installs both files through the existing sha256-checked guard step. docs/token-session-handbook.md "Token lanes carried into subagents" is the source of truth the block copies line for line. Its RTK line is the measured RTK 0.50.0 rule (rtk hook check and the rtk hook claude PreToolUse hook; rewrite_multiline_block, rtk-ai/rtk#3319): && chains and multi-line blocks are rewritten segment by segment; $(...), backticks, <(...), file redirects and heredocs are never rewritten; gh pipelines are not rewritten. Native probe (Claude Code 2.1.283, Agent-tool Haiku children): with the hook the child found and quoted the block's first 60 characters exactly; without it, and for a blind-judge child with it, the child found nothing. Decision record: docs/decisions/2026-09-27-token-lanes-subagent-start.md. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
seathatflowsinourveins
pushed a commit
to native-agent-stack/native-agent-stack
that referenced
this pull request
Sep 27, 2026
Adds a stdlib, fail-open SubagentStart hook that returns adoption/hooks/claude/token-lanes-block.md as additionalContext to every subagent except agent types starting "blind-", registers it in the Claude settings template beside ai-memory, and installs both files through the existing sha256-checked guard step. docs/token-session-handbook.md "Token lanes carried into subagents" is the source of truth the block copies line for line. Its RTK line is the measured RTK 0.50.0 rule (rtk hook check and the rtk hook claude PreToolUse hook; rewrite_multiline_block, rtk-ai/rtk#3319): && chains and multi-line blocks are rewritten segment by segment; $(...), backticks, <(...), file redirects and heredocs are never rewritten; gh pipelines are not rewritten. Native probe (Claude Code 2.1.283, Agent-tool Haiku children): with the hook the child found and quoted the block's first 60 characters exactly; without it, and for a blind-judge child with it, the child found nothing. Decision record: docs/decisions/2026-09-27-token-lanes-subagent-start.md. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
seathatflowsinourveins
added a commit
to native-agent-stack/native-agent-stack
that referenced
this pull request
Sep 27, 2026
…be, measured RTK rule) (#378) * SubagentStart token-lanes carrier for non-blind subagents Adds a stdlib, fail-open SubagentStart hook that returns adoption/hooks/claude/token-lanes-block.md as additionalContext to every subagent except agent types starting "blind-", registers it in the Claude settings template beside ai-memory, and installs both files through the existing sha256-checked guard step. docs/token-session-handbook.md "Token lanes carried into subagents" is the source of truth the block copies line for line. Its RTK line is the measured RTK 0.50.0 rule (rtk hook check and the rtk hook claude PreToolUse hook; rewrite_multiline_block, rtk-ai/rtk#3319): && chains and multi-line blocks are rewritten segment by segment; $(...), backticks, <(...), file redirects and heredocs are never rewritten; gh pipelines are not rewritten. Native probe (Claude Code 2.1.283, Agent-tool Haiku children): with the hook the child found and quoted the block's first 60 characters exactly; without it, and for a blind-judge child with it, the child found nothing. Decision record: docs/decisions/2026-09-27-token-lanes-subagent-start.md. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Native probe and RTK hook-check receipt for the token-lanes carrier evidence/artifacts/token-lanes-subagent-start-20260927/ retains, sanitized, the five native probe runs (Claude Code 2.1.283, Agent-tool Haiku children: without the hook, with it, and a blind-judge child with it, plus the failed v1 question kept as a failed design) and 17 RTK 0.50.0 inputs through both `rtk hook check` and the `rtk hook claude` PreToolUse processor. Raw CLI output and child transcripts stay private and are identified by SHA-256 and counts. Evidence class local_integration; not native_proven for Workflow children, which have not run with the hook. The handbook's RTK sources paragraph and the decision record now cite the receipt instead of unretained observations. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Count the carrier block's budget in UTF-8 bytes (cross-family review) The GPT-6 review of #378 at 85dd27c found that the text test checked len(block), a character count, although the decision record states a 3,500-byte budget: a 3,280-character, 3,520-byte block would pass. The test now measures the file's bytes through fits_budget() and adds that case as a regression test. Controls: a character-counting fits_budget fails the regression test, and a block padded to 3,560 bytes (3,360 characters) fails the byte check but passed the old character check. The committed block is ASCII, 3,159 bytes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Register the carrier's files on main 1aa9776 (after #366 and #380) Replayed onto main: the bootstrap.md note merges after #366's OTEL tool-details sentence, manifests/evidence.json is re-registered, and the matrix and grand list are regenerated. validate FAILS=0; 438 related tests OK. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Scout <scout@local> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Multi-line Bash blocks (a shape coding agents emit constantly) were treated as one command: the
rtkprefix landed on the first line and every following line ran raw and unfiltered. This PR splits blocks at the newline tokens the quote-aware lexer emits and rewrites each line through the existing single-line path. Implements the line-by-line rewrite proposed in #1243.&&/||/|/|&are joined and rewritten as one logical command (git status &&⏎git log -3→rtk git status && rtk git log -3), exactly what develop produced for the single-line form.<<split across a\<NL>continuation), grouping/arithmetic spanning lines, swallowed newlines (quote state opened by an apostrophe in a comment), a continuation operator hidden behind a trailing comment,$'…'containing\'(the one ANSI-C case where the lexer's view diverges from bash), and|&stderr pipes.How this was built
This PR supersedes #3188 (@revopsrocks), taken over after the author became unresponsive; their three commits are cherry-picked with authorship preserved.
$'bail to the actual hazard (\'inside a$'…'span) so an unrelatedecho $'a\tb'line no longer disables the block;&&/||/|lists are now joined and rewritten instead of declined (develop already handled the joined form);comment_startrecognizes#after operator bytes (|&;()), sogit log |# keep⏎grep -f patterns.txtpasses through instead of rebuilding a guarded pipeline;$((gate runs on the bash-faithful continuation join (\<NL>→ nothing) before the space-join normalization erases the operator.Differential fuzz validation
971-case corpus (real per-ecosystem commands, compounds, multiline blocks, cross-line continuations, control flow, deep quoting, heredoc/redirect matrix, permission-bypass attempts, plus the #3188 adversarial cases), run 3-way through the real
rtk rewritehook path — develop vs this PR vs #2781 — with every rewrite validated bybash -n:rtkprefix; cross-line pipes join into the samerewrite_compoundmachinery (final-stage-only,pipeline_final_command_is_safe,grep -frejected,|&raw).bash -ncheck can't see; this PR declines all of them:cat <\⏎<EOF→cat < <EOF(the corpus's only syntax error),$(\⏎(1+2))arithmetic corrupted into command substitution,rtkspliced inside a$'…'literal's value,grep -frebuilt as the final stage of a pipeline behind a|# comment, and function bodies prefixed (changesx=$(foo)captured output).Permission verdicts
Verified live against Claude Code allow/deny rules (
.claude/settings.json): a block auto-allows (exit 0) only when every line independently matches an allow rule (#1213 aggregation); one unmatched line downgrades to ask; a deny match on any line wins outright; and a block whose newline was swallowed by quote state can never auto-allow — the same condition that blindssplit_for_permissionsforces the rewrite to decline, and exit 0 is impossible without a rewrite.split_for_permissionsand the multiline splitter share the same newline-emitting tokenizer, so a verdict is always computed for exactly the lines being rewritten.Test plan
cargo fmt --all && cargo clippy --all-targets && cargo test --all(2609 passed, clippy clean)bash -n-validatedFollow-ups (out of scope here)
Ref #1243. Supersedes #3188. Related: #2781.