Skip to content

shell: remove only trailing line ends from a quoted command substitution - #45339

Open
robobun wants to merge 2 commits into
robobun/53f4200c/shell-cmdsubst-drop-nulfrom
robobun/6d873e57/shell-cmdsubst-keep-trailing-whitespace
Open

robobun wants to merge 2 commits into
robobun/53f4200c/shell-cmdsubst-drop-nulfrom
robobun/6d873e57/shell-cmdsubst-keep-trailing-whitespace

Conversation

@robobun

@robobun robobun commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #44752.

Problem

  • A quoted substitution in a command word drops every trailing space, tab and CR of the output. echo "a$(printf ' ')b" prints ab, bash prints a b.
  • echo hi > "$(echo 'out ')" writes out, and cut -d"$(printf '\t')" fails.
  • Cause: the trim loop in Expansion::child_done (src/runtime/shell/states/Expansion.rs:650). POSIX 2.6.3 removes only trailing newlines.

Fix

  • child_done removes only the trailing LFs, and each CR directly before them, from a quoted substitution in a command word, a redirect target or a [[ ]] operand. bash keeps that CR.
  • An assignment value keeps the old trim, because the shell does not split an unquoted $X.
  • exit accepts spaces and tabs around its number, as bash does.
  • Verified: test/js/bun/shell/bunshell.test.ts, exec.test.ts, commands/exit.test.ts (16 of 24 new tests fail without the change). Also all of test/js/bun/shell/. Self-reviewed: 22 concerns raised, 21 addressed.

Background

  • child_done reads the stdout of a substitution's child script. A quoted substitution or an assignment appends it to the word, an unquoted one splits it.
  • No user reported this. shell: take the command-substitution capture and drop its NUL bytes #44752 lists this trim as separate work.
  • Considered the exact value for assignments, as in bash: X=$(cmd); prog $X then passes a trailing blank that main and bash do not pass.

Downsides

  • ping "$(hostname -I)" now gets the trailing space, as in bash.
  • A variable still drops the blank, so X="$(cmd)"; [ "$X" = "$(cmd)" ] is now false for such output (main, bash: true).
  • Cost per quoted substitution: +6 instructions. Other forms: +1 to +2. .text +256 bytes.
Notes

Rule for a quoted substitution in a word. If the output ends in LF, the shell removes the trailing run of CR and LF bytes. It keeps spaces, tabs, and a CR with no LF after it. An interior CRLF stays.

output of the command main PR bash 5.2, dash
a<SP> a a<SP> a<SP>
a<TAB><LF> a a<TAB> a<TAB>
a<LF><SP><LF> a a<LF><SP> a<LF><SP>
a<CR> a a<CR> a<CR>
a<LF><LF><LF> a a a
a<CR><LF> a a a<CR>
a<CR><CR><LF> a a a<CR><CR>
a<SP><CR><LF> a a<SP> a<SP><CR>

What does not change. An assignment value (X=$(cmd), X="$(cmd)", export X=..., X=... prog) and an unquoted substitution give the same words as before. Measured over seven forms (below): only the quoted word changes.

Why assignment values keep the trim. bash stores the exact value and drops the blanks later, when it splits an unquoted $X into fields. Bun Shell does not split a variable (Expansion.rs, SimpleAtom::Var), and it has no ${X% } to cut the blank. The first version of this PR stored the exact value. Then IP=$(hostname -I); ping $IP, D=$(cmd); cd $D, touch $X and seq $N got a trailing blank that neither main nor bash pass. So the exact value waits for field splitting of an unquoted variable. Until then X=$(printf 'a '); echo "[$X]" prints [a] where bash prints [a ]. To keep the blank today, use the substitution in the word: echo "[$(printf 'a ')]".

Why the CR goes on every platform. The shell has removed a CR before a trailing LF on Linux, macOS and Windows since #7748. Output that ends in CRLF also reaches Linux and macOS with no Windows program involved: HTTP headers, pty output, CRLF files. echo "[$(printf 'Location: /x\r\n\r\n' | cut -d' ' -f2)]" prints [/x] on main and here, and bash keeps <CR><LF><CR> after /x. An LF-only strip gives that word back its CR and makes "$(hostname)" end in CR on Windows. For output without NUL bytes, this CR rule is the only place where the end of a quoted substitution in a word differs from bash, apart from the empty word below. Related to #29669 (CR in a script file), which this PR does not change. Windows was not run locally. The tests feed the substitution from the echo builtin, so they run the same code on Windows CI.

A lone CR stays. "$(printf '\r')" is the way to get a CR in a script, as "$(printf '\t')" is for a tab. bash keeps it too.

What changes for a Windows program. A word changes only when a blank or a lone CR stands before the last line end. a<SP><CR><LF> was a and is now a<SP>. Programs that pad their lines with spaces (table output, wmic) give such output. I did not run these programs.

exit. parse_exit_code trims spaces and tabs. exit "3 ", exit " 3" and exit "3<TAB>" give 3, as in bash and dash (main: exit: numeric argument required). A trailing CR, LF or FF is still an error, as in bash. The strip is what exposes this: on main the trim hid the blank in exit "$(echo '3 ')".

Costs, with outputs (bash / main / PR).

  • ./argv "$(hostname -I)" passes 172.30.0.2<SP> / 172.30.0.2 / 172.30.0.2<SP>. A quoted word is now the text of bash, so a command that failed on the blank in bash fails here too.
  • ./argv a "$(true)" b "$(printf ' ')" c passes <a><><b>< ><c> / <a><b><c> / <a><b>< ><c>. "$(printf ' ')" echo x ran echo on main. It now fails with command not found, and bash fails too.
  • exit "$(printf '3\r')" gives 2 / 3 / 1: the lone CR stays in the word, and exit rejects it as bash does.
  • A variable and a quoted substitution of the same output now differ when the output ends in a blank, because only the variable is trimmed. X="$(echo 'a ')"; [ "$X" = "$(echo 'a ')" ] && echo T || echo F prints T / T / F. F="$(echo 'out ')"; echo hi > "$(echo 'out ')"; cat "$F" prints hi / hi / cat: out: No such file or directory. [[ "$(echo 'a ')" == $(echo 'a ') ]] is true / true / false, because an unquoted [[ ]] operand is split.

Not changed here, on purpose.

  • Assignment values, see above. A todo test records X="$(echo 'a ')"; echo "[$X]" with the result of bash.
  • The unquoted arm still drops SP, TAB, CR and LF at both ends and splits on spaces only. shell: break words on a tab in the lexer #43861 covers the tab.
  • A quoted substitution with empty output still gives no argv word, so test -n "$(true)" is true (bash: false). shell: keep an empty word that has a double-quoted expansion #44904 covers it.
  • Field splitting of an unquoted variable. No open PR covers it.
  • An export operand whose name is not literal text (export $N="$(cmd)", export "X"="$(cmd)") is a command word, not an assignment (Cmd.rs, is_assignment_word). A quoted substitution in it keeps its blanks, so the exported value has them.
  • exit 1_0 exits 10 on main, because the number parser accepts _. bash rejects it. With the blank trim, exit '1_0 ' does the same.
  • Windows only, read from the code and not run: a .cmd or .bat command refuses an argument that contains CR or LF (Cmd.rs, batch_arg_has_cmd_metachars). A kept lone CR, or an LF that is no longer at the end because a blank follows it, now reaches that check in a quoted word. The tree has no test for that refusal today.

Scope of the gain. The fix matters for straight-line scripts: a tab or a blank from printf used in a word, a file name with a trailing blank, [[ "$(cmd)" == "x " ]]. The comparison with bash is in the table and the measurements here and not in the tree.

Measurements. Release builds, linux x64, #44752 vs this PR. The base binary is d66d352. The later commit on #44752 changes a comment and a test.

  • allocations per command substitution: quoted word 26 -> 26 mallocs, 32 -> 32 frees; quoted assignment 24 -> 24, 29 -> 29; unquoted assignment 24 -> 24, 29 -> 29; unquoted word 27 -> 27, 33 -> 33 (gdb hit counts on mi_malloc and mi_free, slope of 200 vs 400 builtin substitutions)
  • instructions in Interpreter::child_done per substitution: quoted word 1,434 -> 1,440, quoted assignment 1,531 -> 1,533, unquoted assignment 1,534 -> 1,536, unquoted word 1,681 -> 1,682 (gdb single-step, callees included)
  • release binary: text 88,976,827 -> 88,977,083 bytes (size); Interpreter::child_done 8,651 -> 8,735 bytes, Exit::start 466 -> 615 bytes, post_subshell_expansion 2,001 -> 2,001 bytes (nm -S)
  • syscall stops per builtin substitution: 4 -> 4 in each of the four forms (gdb catch syscall, slope of 200 vs 400)
  • words changed over all 3,906 captures of {a, SP, TAB, CR, LF} up to length 5, in seven forms (27,342 runs): quoted word 2,843; quoted assignment with a quoted and with an unquoted use 0 and 0; unquoted assignment with a quoted and with an unquoted use 0 and 0; unquoted word, alone and after a literal, 0 and 0. Each changed word is the old word plus a tail that ends in SP, TAB or CR. 2,731 of the 2,843 now equal the word of bash, 0 moved away from it, and 112 keep the CR difference.
  • quoted words equal to bash over the same captures: 975 -> 3,706 of 3,906. Of the 200 that differ, 194 have a CR before a trailing LF and 6 are the empty word.

Reviews. The first version kept the blanks in assignment values too. The review of that version found the X=$(cmd); prog $X regression, and this version keeps the trim there. A second review of this version raised 11 points. 10 are done: the unsorted ls in a test, the test names and comments, the docs wording, and the two limits above (the variable against a quoted substitution, export with a name that is not literal). Not done: exit 1_0, which is base behaviour. A later review comment asked for strings::trim_right in both trims. Not done: with the helper an assignment substitution costs +36 instructions in child_done (1,531 -> 1,567), against +2 with the loop of the base, and this PR does not change assignment values.

Tests.

  • 24 new tests and 1 todo. 16 fail without the change. The 8 controls pin what must stay: every LF, every CRLF and every CR before the LF are removed, an assignment value (X=$(cmd), X="$(cmd)", export X="$(cmd)") still drops the trailing space, exit rejects a trailing CR, LF and FF, and exit takes a captured status.
  • Without the CR part of the loop, 4 tests fail. With one CR per LF only, 2 fail.
  • test/js/bun/shell/ (43 files) and test/cli/run/run-quote.test.ts on release builds with and without the change: the same results in every file except the 16 tests above. commands/ls.test.ts has 2 failures on both (permission tests, the container runs as root).
  • Debug build: bunshell.test.ts (464 pass), exec.test.ts (21 pass) and commands/exit.test.ts (14 pass). The stdin redirect from a zero-length buffer cells of bunshell.test.ts sometimes time out at 5 s on a debug build, with and without the change.

no test proof · iteration 0 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/js/bun/shell/bunshell.test.ts

@robobun

robobun commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 12:05 AM PT - Oct 17th, 2026

❌ @robobun, your commit b7f6855 has 1 failures in Build #127118 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 45339

That installs a local version of the PR into your bun-45339 executable, so you can run:

bun-45339 --bun

@robobun

robobun commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: ready for review. Stacked on #44752.

Reproduced with (release build of main, bash 5.2 beside it):

bun exec "echo \"a\$(printf '  ')b\""
# main: ab. bash and this PR: a  b
bun exec "printf 'a\tb\n' | cut -d\"\$(printf '\t')\" -f2"
# main: cut: the delimiter must be a single character. bash and this PR: b

PR: #45339

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beyond the inline findings, I also checked whether the unquoted $(cmd) path changed: post_subshell_expansion (Expansion.rs:569-572) still trims spaces, tabs, CR and LF at both ends before field splitting, so only the quoted/assignment arm gains trailing blanks. The intentional CR-before-LF divergence from bash is documented in the helper comment, docs/runtime/shell.mdx, and pinned by the new "substitution removes trailing line ends" tests.

Extended reasoning...

The change replaces the whitespace trim in the quoted/assignment command-substitution arm of src/runtime/shell/states/Expansion.rs with a shared strip of trailing LF/CR runs, and makes the shell exit builtin accept whitespace around its number; no security-sensitive surface is touched. Three confirmed findings are posted inline, so a human should weigh the behavioral tradeoffs for unquoted variables and Windows .cmd arguments before merging.

Findings marked 🟡 are optional suggestions and need no follow-up push.

Comment thread src/runtime/shell/states/Expansion.rs
Comment thread src/runtime/shell/states/Expansion.rs
Comment thread src/runtime/shell/builtin/exit.rs Outdated
A quoted command substitution and an assignment value lost every
trailing space, tab and CR of the output. The shell now removes only
the trailing newlines and the CRs directly before them. Output that
ends in CRLF gives the same word as before on every platform.

`exit` accepts whitespace around its number, as bash and dash do, so
a captured status with a trailing blank still works.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Beyond the inline nit, I checked the follow-up commit: the assign_ctx branch in src/runtime/shell/states/Expansion.rs now applies the same trailing SP/TAB/CR/LF trim as the base for A=$(...) and B="$(...)", post_subshell_expansion is byte-identical to the base, and parse_exit_code in src/runtime/shell/builtin/exit.rs trims only space and tab, so 3\r, 3\n and 3\f are still rejected (pinned in exit.test.ts). The remaining behavior change is confined to "$(...)" in a command word; since that is a user-visible Bun Shell semantics change, a human look at that choice is still worthwhile.

Extended reasoning...

The PR now changes only the quoted command-substitution path in Expansion::child_done (keeps trailing blanks and a lone CR, strips trailing LF/CRLF runs) and lets the exit builtin accept space/tab around its number; assignment and unquoted paths match the base. No security-sensitive surface. The follow-up commit addressed the two substantive prior findings (assignment values handing a trailing blank to unquoted $X, and exit accepting \f/\v/\r); the open inline finding this run is a style nit, but the change is a user-facing shell semantics decision authored by a bot, so a maintainer should still weigh it.

Still open from earlier reviews (1):

  • Unresolved: 1 minor or pre-existing.

Comment thread src/runtime/shell/states/Expansion.rs
An unquoted variable is not split into fields. So an assignment value
still drops the trailing whitespace of a command substitution, and
`prog $X` gets what bash passes after it splits `$X`. A quoted
substitution in a command word, a redirect target or a `[[ ]]` operand
keeps its trailing spaces and tabs.

`exit` accepts a space or a tab around its number, as bash does, and
no other byte.
@robobun
robobun force-pushed the robobun/6d873e57/shell-cmdsubst-keep-trailing-whitespace branch from bbb2bc2 to b7f6855 Compare October 17, 2026 01:44
@robobun robobun changed the title shell: remove only trailing line ends from command substitution output shell: remove only trailing line ends from a quoted command substitution Oct 17, 2026
@robobun

robobun commented Oct 17, 2026

Copy link
Copy Markdown
Collaborator Author

Review follow-up, at b7f6855 (rebased on the new head of #44752):

  • Unquoted $X after an assignment: fixed. Assignment values keep the trim of the base, so IP=$(hostname -I); ping $IP passes the same word as before and as bash. Only a quoted substitution in a command word, a redirect target or a [[ ]] operand keeps its trailing spaces and tabs. Over 3,906 outputs in seven forms, 2,843 words change, all in the quoted word form, and none moves away from bash.
  • exit: trims only space and tab. 3\r, 3\n and 3\f stay errors, pinned in exit.test.ts. One correction: trim_ascii did not accept \x0B. It did accept CR, LF and FF.
  • Windows .cmd arguments and the trim_right suggestion: not changed. The reasons and the numbers are in the two threads.

The title and the body now describe the narrower scope. The choice that is left for a maintainer: a quoted substitution in a command word now gives the text of bash, so ping "$(hostname -I)" gets the trailing space as it does in bash. The body also lists one new limit: a variable is still trimmed, so X="$(cmd)"; [ "$X" = "$(cmd)" ] is false when the output ends in a blank.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant