perf: Read child output in bounded chunks instead of per-line

[anthonyshew]

Summary

Closes TURBO-5978.

wait_with_piped_async_outputs called read_until(b'\n') independently for stdout and stderr and wrote each completed line to the shared output sink — one async read, wakeup, delimiter scan, and sink write per line. For verbose tasks (hundreds of thousands of lines), that per-line machinery is the hot path, and every sink write becomes a channel event downstream (TUI) or a log-file write (cache).

Benchmarking first (per the issue): the current line-oriented reader vs chunk-oriented reads. Key finding that makes chunking safe: nothing downstream of the sink is line-aware — the TUI terminal parses the byte stream itself, the cache log writer forwards bytes, and the task-handle writer forwards raw buffers. So forwarding bounded chunks is byte-identical to per-line forwarding.

Changes

wait_with_piped_async_outputs now reads bounded 8 KiB chunks (read_buf) and writes each chunk to the sink, replacing per-line reads and writes. Preserved behavior:

• Trailing-newline append for unterminated final output (tracked per stream as last-byte state).
• The post-exit drain window and its deadline flush.
• Concurrent stdout/stderr draining, failure fast-path (skip reads on non-zero exit without shutdown), and zero-exit keep-reading semantics.
• Memory bounded at 8 KiB per stream even for output without newlines.
• The PTY (sync) path is untouched — it already read in chunks.

Benchmarks

Real child processes piped through the production wait_with_piped_outputs entry point, release build, macOS, 7 in-process runs per scenario averaged, then whole-binary A/B iterations. Harness was temporary and is not committed.

With a production-shaped sink (one unbounded channel send per write, drained by a background task):

Scenario	Before: wall / sink writes	After: wall / sink writes	Change
300k tiny lines	82.7 ms / 300,000	42.6 ms / 243	−48% wall, −99.9% writes
100k ordinary ~70 B lines	38.1 ms / 100,000	35.0 ms / 780	−8% wall, −99.2% writes

Byte-for-byte output was identical on every scenario in every run (asserted via sink byte counts). Control scenarios with a no-op counting sink:

Scenario	Before	After	Verdict
2k very long lines (64 KiB each, 128 MiB)	273 ms	269 ms	wash (per-write count was never high)
2 MiB invalid UTF-8	13.4 ms	14.5 ms	wash
No trailing newline	4.9 ms	5.3 ms	wash
200k interleaved stdout+stderr lines	170 ms	159 ms	wash to slight win

Chunked reads win where per-line overhead dominated (verbose line-heavy tasks) and are neutral elsewhere; the 100–1000× reduction in sink writes removes the corresponding downstream TUI-event/log-write costs.

Verification

• cargo test -p turborepo-process: 76 passed, 0 failed (includes pipe/PTY output, trailing-newline, and post-exit drain tests).
• cargo clippy -p turborepo-process --all-targets: clean for library code.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
examples-basic-web	Ready	Preview, v0	Sep 9, 2026 9:07pm UTC
examples-designsystem-docs	Ready	Preview, v0	Sep 9, 2026 9:07pm UTC
examples-gatsby-web	Ready	Preview, v0	Sep 9, 2026 9:07pm UTC
examples-kitchensink-blog	Ready	Preview, v0	Sep 9, 2026 9:07pm UTC
examples-nonmonorepo	Ready	Preview, v0	Sep 9, 2026 9:07pm UTC
examples-svelte-web	Ready	Preview, v0	Sep 9, 2026 9:07pm UTC
examples-tailwind-web	Ready	Preview, v0	Sep 9, 2026 9:07pm UTC
examples-vite-web	Ready	Preview, v0	Sep 9, 2026 9:07pm UTC
turbo-site	Ready	Preview, v0	Sep 9, 2026 9:07pm UTC
turborepo-factory	Ready	Preview, v0	Sep 9, 2026 9:07pm UTC