Skip to content

Read console stdin with ReadConsoleW on Windows so non-ASCII input survives (REPL, prompt(), bun -) - #38140

Open
robobun wants to merge 6 commits into
mainfrom
farm/6eb31172/windows-console-stdin-readconsolew
Open

robobun wants to merge 6 commits into
mainfrom
farm/6eb31172/windows-console-stdin-readconsolew

Conversation

@robobun

@robobun robobun commented Aug 13, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • On Windows, everything bun reads from an interactive console through its own stdin readers comes back garbled for non-ASCII characters: bun repl silently drops them, prompt() returns one junk byte per character (typing a中b😀 into prompt() returned "a b" on one run and a + U+FFFD + b on another; the junk byte varies per run and the emoji's two UTF-16 units come back as two NULs), and a script typed into bun - gets the same junk in its source. 한국어 came back as three junk bytes. This is the Windows half of bun repl: support more language characters #27556; repl: keep multi-byte UTF-8 input in the interactive line editor #31876 fixed the POSIX line editor.
  • Cause: bun_sys::read() for HANDLE-backed fds (src/sys/lib.rs, windows_impl::read) calls ReadFile. On a console handle that is the ANSI console API: conhost converts the typed UTF-16 through the console input code page, and its CP_UTF8 conversion (bun sets code page 65001 at startup in src/bun_core/output.rs) does not produce multi-byte UTF-8 on Windows 10 / Server 2019. Every native stdin reader ends up in this one function: the REPL's read_byte (File::stdin().read, 256-byte buffer), prompt() through the 4 KiB BufferedStdin, alert()/confirm()/bun init/bun update -i through one-byte StdinReader reads, bun publish's OTP prompt, and bun - through read_to_end_into.
  • process.stdin, Bun.stdin and fs.readSync(0) go through libuv (fd 0 is a Uv-kind fd) and were never affected; they are not touched.

Fix

  • read() hands the process stdin to a new bun_sys::windows::console_stdin module when GetConsoleMode succeeds on it. The module reads 8192 UTF-16 units at a time with ReadConsoleW into a process-wide buffer, transcodes the chunk with bun_core::strings::copy_utf16_into_utf8, and serves read() calls of any size from the transcoded bytes. Pipes and files still take the ReadFile path, so redirected stdin stays byte-exact. The dispatch is keyed on fd == Fd::stdin(), so ordinary file reads do not pay for a GetConsoleMode call; stdin is the only console handle bun reads this way.
  • Reading the console with the W API and converting in the program is what every Unicode-aware console program does on Windows (libuv's tty reader, Python since PEP 528, Rust's std); the ANSI conversion is the broken piece, not the code page.
  • The chunk size is fixed instead of derived from the caller's buffer because on the Windows 10 / Server 2019 conhost the count passed to a line-input read is also the longest line the console lets the user type. Measured here: prompt() accepted 4094 characters with ReadFile (its 4 KiB buffer), a first version of this change that requested buf.len() / 3 units cut that to 1022, and with 8192 units prompt() takes 8190 and bun - keeps the 8190 it had (its ReadFile requests were 8 KiB). A 5000-character line is part of the test.
  • A high surrogate that ends a chunk is held back and prepended to the next chunk, so a pair split across two reads (in raw mode the console returns whatever has been typed so far) becomes one character instead of two U+FFFD; a chunk consisting only of such a surrogate makes read() go straight back to the console rather than report end of input.
  • End-of-input behaviour is unchanged from ReadFile, measured on Server 2019 before the change: in line-input mode a line that starts with Ctrl+Z is end of input (prompt() returns null, bun - runs the script), Ctrl+Z elsewhere in a line is an ordinary character, and end of input is not sticky (the next line is readable). ReadConsoleW itself has no Ctrl+Z rule, so it is applied here, to chunks that start a line (a newer console delivers a line longer than one chunk over several reads, and the continuation chunks are not checked); raw mode (the REPL) gets byte 0x1A as before. Ctrl+C / Ctrl+Break cancel a console read with ERROR_OPERATION_ABORTED, which is retried exactly like the ReadFile path does; a Ctrl+C probe against prompt() and the REPL behaves identically before and after (details below).
  • Verified with test/js/bun/windows/console-stdin.test.ts. console-stdin-driver-fixture.ts gives bun a real headless console (AllocConsole, the child gets the console as its stdio) and types into it with WriteConsoleInputW; the program under test writes what it read to a file. Three cases: prompt()/alert()/confirm() with the Ctrl+Z lines above and the 5000-character line; the REPL in raw mode, where the emoji's high surrogate is typed only after the REPL has consumed everything before it (the driver waits for the console input queue to drain) so it arrives in a read of its own; and bun - ended with a Ctrl+Z line. On windows-x64 all three fail with bun 1.4.0-canary.1 (USE_SYSTEM_BUN=1, output below) and pass with bun bd test; in CI the file runs and passes on both the Windows Server 2019 x64 and the Windows 11 aarch64 test lanes (the old and the new console), and is skipped on other platforms, so the Linux gate cannot exercise it. A temporarily instrumented build confirmed the REPL case reaches the surrogate hold-back with a one-unit chunk.
  • Also ran cargo check -p bun_sys for x86_64-pc-windows-msvc and aarch64-pc-windows-msvc, clippy on the Windows target (nothing reported in the new code), test/internal/source-lints/, and the test typecheck for the new files. cargo test -p bun_sys cannot link (bun_core needs the C++ objects), so the module has no Rust unit tests; the console cases cover the large-buffer, one-byte, split-surrogate, raw-mode, long-line and end-of-input paths.

Background

  • Windows has two console input APIs. ReadConsoleW returns the typed text as UTF-16. ReadFile on a console handle is routed to the ANSI variant, which converts that text to bytes using the console input code page; that conversion is what misbehaves for code page 65001 on the conhost versions in question, which is why Unicode-aware programs read consoles with ReadConsoleW and convert themselves.
  • Console input mode: with ENABLE_LINE_INPUT set (the default, used by prompt() and the CLI prompts) a read returns one line at a time after Enter, terminated by \r\n, and on the older conhost the requested count is the line-length limit; newer consoles return a longer line over several reads. With the flag cleared (the REPL's raw mode) a read returns whatever keys are available. The mode is applied when the program reads, not when the user types, which is what lets the test queue keystrokes before launching the child.
  • On Windows, bun's Fd is either Uv-kind (a CRT fd owned by libuv, such as fd 0 used by process.stdin and fs) or System-kind (a raw HANDLE). The cached process stdin behind File::stdin() and Output's stdin readers is System-kind, which is why these readers reach the ReadFile branch of read().
Measurements behind the Ctrl+Z and Ctrl+C rules and the fail-before output (Windows Server 2019, typed through the same console harness)

prompt() with the released build (ReadFile) and with this change (ReadConsoleW) agree on every row:

typed prompt() returns following prompt()
^Z Enter null reads the next line
^Zabc Enter null reads the next line
abc^Z Enter "abc\x1a"
ab^Zcd Enter "ab\x1acd"
^Z without Enter blocks, as before

Line length in line-input mode: released build, prompt() 4094 characters, bun - first line about 8190; this change, prompt() 8190, bun - 8190 (a 9000-character line is cut at 8190 by the console in both cases).

Ctrl+C sent with GenerateConsoleCtrlEvent while blocked, released build and this change identical: prompt() with no SIGINT listener exits with STATUS_CONTROL_C_EXIT; prompt() with a process.on("SIGINT") listener keeps blocking in the read (the listener cannot run while the main thread is blocked, same as before); the REPL exits with STATUS_CONTROL_C_EXIT (pre-existing Windows REPL behaviour, reported separately).

New test against bun 1.4.0-canary.1+9a543cc18 (USE_SYSTEM_BUN=1) on windows-x64:

-     "prompt1": "a中b😀",
-     "prompt2": "한국어",
+     "prompt1": "a�b",
+     "prompt2": "���",
-     "prompt6": "xxxx... (5000 characters)
+     "prompt6": "xxxx... (cut at 4094 by the console)
...
-   "result": "a中b😀",      (REPL)
+   "result": "ab",
...
-   "result": "a中b😀",      (bun -)
+   "result": "ab",

@coderabbitai

coderabbitai Bot commented Aug 13, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@robobun, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 2 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 3433ae4e-41af-4f4e-a5db-7846508a42af

📥 Commits

Reviewing files that changed from the base of the PR and between b7a0431 and f914392.

📒 Files selected for processing (6)
  • src/sys/lib.rs
  • src/sys/windows/console_stdin.rs
  • src/sys/windows/mod.rs
  • test/js/bun/windows/console-stdin-driver-fixture.ts
  • test/js/bun/windows/console-stdin-prompt-fixture.js
  • test/js/bun/windows/console-stdin.test.ts

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Aug 13, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 10:57 AM PT - Aug 13th, 2026

✅ @robobun, your commit f91439296e73bc3c59b76c9d9e7be28c389e3605 passed in Build #94654! 🎉


🧪   To try this PR locally:

bunx bun-pr 38140

That installs a local version of the PR into your bun-38140 executable, so you can run:

bun-38140 --bun

@robobun

robobun commented Aug 13, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: ready for a maintainer.

Reproduced on Windows Server 2019 with bun 1.4.0-canary.1+9a543cc18 by giving bun a real console and typing into it (the harness is test/js/bun/windows/console-stdin-driver-fixture.ts): prompt() on a typed a中b😀 returned a + junk + b, the REPL dropped the non-ASCII characters, and bun - got the same junk in the typed source. With this branch all three return the typed text; test/js/bun/windows/console-stdin.test.ts fails on the released build and passes on a bun bd build on windows-x64 (it is skipped on other platforms, so only the Windows CI lanes exercise it).

Review: the two line comments on refill() were addressed (0be7d61 tracks whether a chunk starts a line before applying the Ctrl+Z rule; f914392 clears the carried surrogate only after the read succeeds), the comment-length threads in 7b675ab; all 18 threads are resolved and the bot review has no further findings. Each follow-up was re-run against the console test on windows-x64.

Comment thread src/sys/windows/console_stdin.rs Resolved
Comment thread src/sys/lib.rs Resolved
Comment thread src/sys/windows/console_stdin.rs Resolved
Comment thread src/sys/windows/console_stdin.rs Resolved
Comment thread src/sys/windows/console_stdin.rs Resolved
Comment thread src/sys/windows/console_stdin.rs Resolved
Comment thread src/sys/windows/console_stdin.rs Resolved
Comment thread src/sys/windows/console_stdin.rs Resolved
Comment thread src/sys/windows/console_stdin.rs Resolved
Comment thread src/sys/windows/console_stdin.rs Resolved
Comment thread src/sys/windows/console_stdin.rs Resolved
Comment thread src/sys/windows/console_stdin.rs Resolved
Comment thread src/sys/windows/console_stdin.rs Resolved
Comment thread src/sys/windows/console_stdin.rs Resolved
Comment thread src/sys/windows/console_stdin.rs Resolved
Comment thread src/sys/windows/console_stdin.rs Resolved
Comment thread src/sys/windows/console_stdin.rs Resolved
Comment thread src/sys/windows/console_stdin.rs Resolved

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — f914392 addresses the pending_lead ordering nit (now cleared only after read_units() succeeds), and 0be7d61 already handled the mid_line Ctrl+Z tracking. I have no further findings.

Given this adds a new Windows console-input subsystem in the core bun_sys::read() path (process-global Mutex-guarded buffer held across a blocking ReadConsoleW, conhost-version-dependent Ctrl+Z / line-length semantics, plus a hand-rolled FFI console driver in the test fixture), a human look is still worthwhile.

What was reviewed: the fd == Fd::stdin() dispatch keeps ordinary file reads off the GetConsoleMode probe; BYTES_CAPACITY = 3 * UNITS_CAPACITY covers the UTF-8 worst case so copy_utf16_into_utf8 cannot short-write; the OPERATION_ABORTED retry mirrors the existing ReadFile path; and the surrogate-hold-back / mid_line state now survives an error return.

Extended reasoning...

Overview

This PR adds src/sys/windows/console_stdin.rs, a new module that intercepts bun_sys::read() on Windows when the fd is the process stdin and GetConsoleMode succeeds on it, replacing the ReadFile path with ReadConsoleW + in-process UTF-16→UTF-8 transcoding. The dispatch is a six-line insertion in src/sys/lib.rs (windows_impl::read); src/sys/windows/mod.rs gains only the pub(crate) mod console_stdin; line. Three new test files exercise prompt()/alert()/confirm(), the REPL in raw mode, and bun - against a real headless console driven via WriteConsoleInputW over bun:ffi.

Security risks

None identified. The new code is read-only console input; the unsafe block around ReadConsoleW has a SAFETY comment and its buffer is a fixed [u16; 8193] owned by the State struct. out.len() as DWORD cannot overflow (8192). The transcode destination is sized for the 3-bytes-per-unit worst case, so copy_utf16_into_utf8 cannot write past bytes. No user-controlled sizes reach an allocation.

Level of scrutiny

Moderate-to-high. The change is small in line count but sits directly in the bun_sys::read() hot path for HANDLE-backed fds on Windows, introduces process-global mutable state guarded by a bun_core::Mutex that is held across a blocking syscall (acceptable here — concurrent readers of the one stdin must serialize anyway, and the comment says so), and encodes conhost behaviour that differs between Windows 10/Server 2019 and Windows 11 (line-length capping vs multi-chunk delivery, Ctrl+Z semantics). The test fixture is itself ~190 lines of raw Win32 FFI (CreateProcessW, job objects, INPUT_RECORD construction). This is not a mechanical change; a maintainer familiar with the Windows lanes should confirm the design (global buffer, 8192-unit chunk, fd == Fd::stdin() gating) and that CI on windows-x64 is green.

Other factors

Two prior nits from earlier runs of this reviewer were both addressed in follow-up commits: mid_line tracking so the Ctrl+Z→EOF rule only fires at start-of-line (0be7d61), and reordering pending_lead clearing to after the fallible read_units() call (f914392). All comment-cop bot threads about long comments were resolved (7b675ab, 7172e07). The PR description is unusually thorough — every design constant is backed by a measurement — and the tests are structured to fail on the released build (USE_SYSTEM_BUN=1) and pass on the debug build. Nonetheless the guidelines exclude auto-approval for changes of this scope in a core syscall wrapper.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant