Skip to content

parser: an a ? (b) : c => d : e attempt inside another attempt keeps its arrow function - #46311

Open
robobun wants to merge 12 commits into
mainfrom
robobun/7ff7b2bf/keep-conditional-arrow-attempt
Open

robobun wants to merge 12 commits into
mainfrom
robobun/7ff7b2bf/keep-conditional-arrow-attempt

Conversation

@robobun

@robobun robobun commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Fix

  • An attempt inside another attempt keeps the arrow function that it finds: it skips its restore, and parse_paren_expr uses the result. Each body is parsed 2 times: 69 ms and 45 MB at depth 2,560.
  • Output and errors do not change. The outermost attempt still restores the parser.
  • Verified: test/bundler/transpiler/transpiler.test.js (new cost test: 224 MB on main, bound 32 MB). 180,000 generated inputs print the same as main.
  • Self-reviewed: 43 concerns raised, 41 addressed. Not taken: a => check before the body, a table of cost tests.

Background

  • In a ? (b) : c => d, (b): c => d is an arrow function only if a second : follows the body. The parser takes a ParserSnapshot, parses with the lexer's log off, and restores.
  • Only an attempt parses an expression with the log off, so an attempt that starts with the log off is inside another one.
  • Considered: the attempt as the real parse (1 parse per body, branch robobun/7ff7b2bf/keep-successful-attempt): +216/-77 source, and 15% more instructions where an attempt fails after the body.

Downsides

  • Chains of attempts nest 14% less deep (334 levels at a 1 MB stack, was 387): the attempt is its own function now.
  • One attempt costs 46 to 59 more instructions. .text +1,280 B.
  • A chain where every attempt fails stays quadratic: 0.5 s and 818 MB at depth 1,280.
Notes

Repro

// chain.js, run: bun chain.js 2560
const depth = Number(process.argv[2] ?? 2560);
const tail = process.argv[3] ?? " : e";
let s = "d";
for (let i = 0; i < depth; i++) s = `a ? (b) : c => ${s}${tail}`;
const code = `x = ${s};`;
const t = new Bun.Transpiler({ loader: "ts" });
t.transformSync("x = 1;");
const rss0 = process.memoryUsage().rss;
const t0 = performance.now();
t.transformSync(code);
console.log(code.length, "bytes", (performance.now() - t0).toFixed(0), "ms", "RSS", (rss0 / 2 ** 20).toFixed(0), "->", (process.memoryUsage().rss / 2 ** 20).toFixed(0), "MB");

Release builds of main (11c3671) and this PR, linux x64. "Kept" is the chain above. "Failed" is the chain without : e (bun chain.js 1280 ""), where no attempt finds an arrow function. "No return types" is a ? (b) => ... : e, which runs no attempt.

chain depth main this PR
kept 640 183 ms, RSS 30 to 259 MB 6 ms, 32 to 36 MB
kept 1,280 539 ms, to 936 MB 17 ms, to 39 MB
kept 2,560 3,989 ms, to 3,639 MB 69 ms, to 45 MB
failed 640 141 ms, to 230 MB 144 ms, to 231 MB
failed 1,280 601 ms, to 817 MB 520 ms, to 818 MB
no return types 2,560 72 ms, to 41 MB 69 ms, to 42 MB

Bun 1.3.13 takes 11 ms and 38 MB for the kept chain of depth 640. Bun 1.4.2 takes 8.9 s and 3,617 MB at depth 2,560. The time on main at depth 2,560 moved between 2.8 s and 4.0 s over my runs. The memory did not move.

What changed

  • The attempt turns the lexer's log off before it parses. The other places that turn it off (the lexer backtrackers for types and the one-token lookaheads) parse no expression. So an attempt that finds the log off on entry runs inside another attempt, and the outermost attempt restores the parser when it ends.
  • Such an inner attempt, when it parses an arrow function, sees : and can read the token after it (can_read_next_token, the read that expect makes), returns the E::Arrow with no restore. parse_paren_expr pops the scope and returns it. A debug assertion there checks that the log is off. The memo entry is written as before.
  • Every other path is the one from main: the outermost attempt restores, a failed attempt restores, and the parse after a restore declares the arguments and reports the errors.

Body parses and restores (gdb breakpoint counts on parse_arrow_body and restore_parser_snapshot):

input main this PR
kept chain, depth 24 324 and 24 48 and 1
kept chain, depth 640 205,760 and 640 1,280 and 1
kept chain, depth 640, emitDecoratorMetadata 205,760 and 640 1,280 and 1
10 lines of x = a ? (b) : T => d : e; 20 and 10 20 and 10
10 lines of x = a ? (b) : c => d; 20 and 10 20 and 10
failed chain, depth 24 324 and 24 324 and 24

Stack

The attempt was inlined into parse_paren_expr, so every TypeScript call of that function had the 416 B ParserSnapshot in its frame. To return the arrow function from the inlined attempt made that frame larger. So the attempt is #[cold] #[inline(never)] now. Two more changes keep its own frame small: restore_parser_snapshot takes &mut ParserSnapshot (a by-value call in one branch made a second copy of the snapshot), and the read of the next token is in the non-inlined can_read_next_token (its lexer snapshot is 240 B).

frame main this PR
parse_paren_expr, TypeScript 1,088 B 752 and 768 B
parse_paren_expr, JavaScript 448 and 464 B 448 and 464 B
the attempt inlined 688 B

Deepest input that transformSync accepts at ulimit -s 1024. Past the limit both builds report Maximum call stack size exceeded.

input main this PR
a ? (b) : c => ... : e 387 334
a ? (b) : c => ... 389 333
a ? (b) => ... : e 387 453
a ? (x) : e, nested in x 426 501
((((1)))) 501 617
(a) => (a) => ... 451 543
((a) => ((a) => ...)) 237 287

Cost for other input

Instructions per line (perf_event_open, instructions:u, JIT off, scanImports of a 50,000 line file, the same count on every run):

line main this PR
x = a ? (b) : T => b + 1 : e; 9,157 9,208
x = a ? (b) : c => c + 1; 9,118 9,177
x = a ? (b) : c + 1; 5,454 5,500
x = a ? (b) : T => a ? (b) : T => b + 1 : e : e; 21,031 18,611
x = a ? (b) => b + 1 : e; 6,025 6,023
x = (b) => b + 1; 4,851 4,849
x = (b: number): number => b + 1; 5,704 5,698
x = (b + 1); 2,842 2,842
x = b + 1; 2,126 2,126
x = (b) => b + 1; as .js 4,411 4,411
x = (b + 1); as .js 2,542 2,542
  • size -A: .text 108,646,665 to 108,647,945 B. The JavaScript copies of parse_paren_expr have the same code size (3,064 and 3,050 B).
  • The change adds no field: P, Lexer and ParserSnapshot keep their layout.

perf and valgrind are not in the container, so the instruction counts come from a perf_event_open wrapper.

Same output

  • 180,000 generated inputs with attempts nested in attempts: parameter lists, return types, block bodies, legal comments, unreadable tokens after the second :, async, JSX after the :, types that log while they are skipped, classes with decorators. Loaders ts and tsx, trimUnusedImports, minifyWhitespace, scan, and 60,000 of them with emitDecoratorMetadata. 70,041 are valid. The output, the error list and the error positions are the same on both builds.

The larger shape

The branch robobun/7ff7b2bf/keep-successful-attempt (commit 28ed7ee) makes the attempt the real parse, as tsc does: a kept chain takes n body parses, and a kept site outside a chain takes 1 parse (7,896 instructions for the first line of the table above). It was the head of this PR before. It is not the head now for these reasons:

  • It is +216/-77 source: a lexer flag that makes expect fail with the log on, the end of parse_paren_expr as a function, and a second arm for emitDecoratorMetadata, because Hardening follow-up: fix regressions from #46021 and more crashes #46270 needs the attempt to be thrown away.
  • A site that fails after the body (a ? (b) : c => d) costs 15% more instructions (9,118 to 10,498) and keeps about 640 B more, because the attempt declares the arguments.
  • Chains of attempts nest 17% less deep (320 at a 1 MB stack).
  • As written it rejects valid TSX (x = a ? (b) : <out.Foo />), because its check before the snapshot logs outside of it.

The shape in this PR keeps the rule that the attempt's parse is thrown away at the top level, which #40818, #40985 and #46270 rely on.

Not in this PR

  • The outermost successful attempt still parses its arrow function 2 times.
  • A chain where every attempt fails. Each failed attempt parses the body, and the parse after it parses the same text again. tsc 6.0.2 is quadratic there too (50, 144 and 552 ms at depth 200, 400 and 800).
  • restore_parser_snapshot scans ref_to_ts_namespace_member for each restore that made a symbol. A file with many enums and many attempts with block bodies pays for that.
  • A failed attempt at cond ? (x) : y before a newline formats an error message that the restore drops. A check for => before the body would skip it.

Related open PRs

These change the same function or the lines around its call: #45743 (async(...) before :), #45939 (the memo key), #46209 ((b as B) before :), #46553 (decorator metadata), #47133 (draft, more places that run the attempt), #45964 (Lexer::restore and the log). This PR is output-identical and changes about 30 lines of the attempt, so it can merge before or after them. #46177 is closed.

Tests

  • Debug build with ASAN: test/bundler/transpiler/transpiler.test.js, 259 pass. With src/ from main the cost test fails: the chain with return types takes 41.6 MB more than the chain without (bound 10 MB).
  • Release build of main: the same test fails with 224.6 MB (bound 32 MB).

robobun and others added 5 commits October 21, 2026 00:43
…w function

Between the "?" and ":" of a conditional, TypeScript `(b) : c => d` is an
arrow function only when another ":" follows its body. The attempt that
decides this parsed the arrow function, always restored the parser, and
the caller parsed a kept body again. A chain of depth n took n * n / 2
parses of a level, and the parser frees nothing before the file is done.

The attempt is now the parse. It declares the real arguments and is kept
when a ":" follows the body. Only a failed attempt restores the snapshot.

- `Lexer::backtrack_on_mismatch` keeps the attempt strict while the log
  stays on, so a kept parse reports its diagnostics.
- The legal comments that wait for the next statement stay in the lexer
  buffer with a count of the taken ones, so a restore gives them back
  without a copy.
- The memo records only failed attempts.
- The restore scans `ref_to_ts_namespace_member` only when the attempt
  added a key.

The shape of the helper follows `parse_possible_arrow_fn` from #44685.

Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>
…on attempt

The type names of decorator metadata are bound as they are parsed. A name
that nothing declares becomes a member of the module scope. A failed
attempt left that member behind with the index of a symbol that the
restore removed, so the parse after it bound `Date` to another symbol or
read past the end of the symbol table.

The restore now removes the module scope members and the scope uses that
the attempt added.
…accesses

A decorator metadata type that a sibling namespace block exports is bound
to a new symbol in the `property_accesses` of that namespace. The entry
stayed after a failed attempt, with the index of a removed symbol.
The closure for the tail was built for every parenthesized arrow
function and was a call in the TypeScript parsers. The tail is now an
`#[inline(always)]` method, and only the attempt builds a closure. The
JavaScript copies of `parse_paren_expr` compile to the size they had.

The lookahead after the second ":" is out of line, so its lexer snapshot
is not in the frame of the attempt while the body is parsed.
@robobun

robobun commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: ready for review.

Reproduced on bun 1.4.2 and on main (11c3671) with the script in the Notes of the description: bun chain.js 2560 takes 4.0 s and 3,639 MB for a 48 KB input. With this PR it takes 69 ms and 45 MB.

CI: no failure comes from this diff. Build 134548 is red only on test/aot/* shards (the same test families are red on the build of main, 134092) and on test/js/bun/jsonl/jsonl-parse-chunk-utf8.test.ts, which compares two wall-clock times. test/bundler/transpiler/transpiler.test.js passes on every lane that ran it.

PR: #46311

@robobun

robobun commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 3:12 PM PT - Oct 28th, 2026

❌ @robobun, your commit 2c81583 has 5 failures in Build #134548 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 46311

That installs a local version of the PR into your bun-46311 executable, so you can run:

bun-46311 --bun

@robobun

robobun commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator Author

Update: #46270 fixed this abort on main. This PR no longer changes anything for emitDecoratorMetadata, so the comparison below is about an earlier version of it.

The abort with emitDecoratorMetadata that this PR fixes also has a fix on the branch robobun/71f79215/decorator-metadata-bind-at-lowering. No PR is open for that branch, so this PR stays the only one for the abort.

Diff summary of the branch: the parse pass stores each decorator metadata type name, and serialize_metadata binds it when the class is lowered. The parse pass then binds no name, so a failed attempt leaves nothing to remove.

I ran 51 decorated shapes on release builds of main (9634520), of this PR (c4484b7) and of the branch (c9ee543), as bun index.ts and as bun build --target=node:

  • This PR removes the abort and the false "K" has already been declared in all 13 shapes that have a conditional. 12 of them load. The other one names a type that a sibling namespace block exports, and throws the ReferenceError that the same code without the conditional throws on main. No other shape changes, and no shape goes from loading to failing.
  • The branch changes the same 13 shapes, and 33 shapes without a conditional. Examples: a type declared or imported after the class (Object on main), a type-only import named only in T[] (Export named 'IFoo' not found on main), and ns.T of import * as ns in a bundle (Object on main).
  • The branch has two known costs. A bundle can throw Cannot access 'Later' before initialization for a class that only metadata names (emitDecoratorMetadata can cause "Cannot access uninitialized variable." at runtime #7591). A type-only import written after the class fails to load, as it does before the class (SyntaxError: Import named 'ClassA' not found in module with emitDecoratorMetadata: true #6172).

If this PR merges first, the branch can rebase onto it and delete the code that removes the module scope members.

The attempt that decides whether `(b) : c => d` between the `?` and
`:` of a conditional is an arrow function parses the body, restores
the parser, and the caller parses the body again. An attempt in the
body did the same, so a chain of these arrow functions parsed the body
at depth n about n times, and the arena kept every parse.

An attempt that runs inside another attempt is thrown away with it.
When it finds an arrow function it now skips its own restore and hands
the arrow function to the caller. The outermost attempt still restores
the parser, and the parse after it still parses each body for real, so
each body is parsed two times.

The attempt is no longer inlined into `parse_paren_expr`, which keeps
the parser snapshot out of that frame.
…-conditional-arrow-attempt

# Conflicts:
#	src/js_parser/parse/parse_skip_typescript.rs
The attempt that decides whether `(b) : c => d` between the `?` and
`:` of a conditional is an arrow function parsed the arrow function,
restored the parser, and the caller parsed it again.

The attempt now parses the arrow function with its arguments, with the
log on and with a lexer flag that makes a token mismatch end the
attempt. When another `:` follows, that parse is the result and
nothing is restored. A check that only moves the lexer runs first, so
an attempt with no `=>` after the type takes no parser snapshot.

With emitDecoratorMetadata the attempt is thrown away as before and the
caller parses again: the snapshot does not undo what metadata adds.

The snapshot copies the pending legal comments, and they stay in the
lexer for the parse that is kept.
@robobun robobun changed the title parser: keep a successful a ? (b) : c => d : e attempt, restore only on failure parser: keep the arrow function that the a ? (b) : c => d : e attempt finds Oct 22, 2026
…empt

Go back to the smaller shape. The attempt for `(b) : c => d` between
the `?` and `:` of a conditional is the one from main again: it parses
with the log off and without arguments, and the outermost attempt
always restores the parser.

An attempt that starts with the log off runs inside another attempt,
which throws everything away. When such an attempt finds an arrow
function, it skips its restore and returns the arrow function. The
outer attempt then parses each nested body one time, not one more time
for every attempt around it.

This takes out the parse that was kept for real: the lexer flag, the
arrow tail function, the copy of the legal comments in the snapshot,
and the check before the snapshot, which could log outside of it.
@robobun robobun changed the title parser: keep the arrow function that the a ? (b) : c => d : e attempt finds parser: an a ? (b) : c => d : e attempt inside another attempt keeps its arrow function Oct 28, 2026
@robobun
robobun marked this pull request as ready for review October 28, 2026 17:46
@coderabbitai

coderabbitai Bot commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 11c3671 and 2c81583.

Files selected for processing: 4
  • src/js_parser/p.rs
  • src/js_parser/parse/mod.rs
  • src/js_parser/parse/parse_skip_typescript.rs
  • test/bundler/transpiler/transpiler.test.js

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 4 remain after this review.

Walkthrough

The TypeScript conditional-expression check can return an already-parsed arrow from a nested attempt. Snapshot restoration and token lookahead were updated. Tests cover nested parsing cases, expected output, and peak memory.

Changes

Conditional Arrow Parsing

Layer / File(s) Summary
Retain parsed arrows in nested attempts
src/js_parser/parse/parse_skip_typescript.rs, src/js_parser/p.rs, src/js_parser/parse/mod.rs, test/bundler/transpiler/transpiler.test.js
The TypeScript return-type check returns a parsed arrow when a nested attempt finds one, while other paths restore the parser snapshot. parse_paren_expr returns the parsed arrow when available. Tests cover nested attempts, errors, comments, follow-on parsing, expected output, and peak RSS.

Possibly related PRs

  • oven-sh/bun#45939: Both changes modify speculative parsing and parser-state restoration for TypeScript conditional arrows.
  • oven-sh/bun#46209: Both changes affect TypeScript arrow-versus-expression disambiguation in parse_paren_expr.
  • oven-sh/bun#47139: Both changes affect parenthesized-expression versus arrow-function decisions in parse_paren_expr.

Suggested reviewers: jarred-sumner, alii

Priority: Normal

Merge Risk: Minimal · up to 2c815

Nested conditional-arrow parsing avoids repeated speculative work. No material merge-blocking issue was identified.

Pre-merge checks: Passed 4
Check name Status Explanation
Title check Passed The title clearly identifies the parser change and the nested-attempt behavior that the pull request fixes.
Description check Passed The description explains the problem, fix, tradeoffs, and verification results. It does not use the template headings exactly, but it provides the required information in equivalent sections.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Finishing Touches: 💡 1
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Beyond the inline finding, I also traced the invariant the kept path depends on (log off on entry means an enclosing attempt will restore): every other site that sets is_log_disabled — the two lexer backtrackers in src/js_parser/parse/parse_skip_typescript.rs, next_token_matches in src/js_parser/parse/mod.rs, and look_ahead_next_token_is_open_paren_or_less_than_or_dot in src/js_parser/typescript.rs — only skips types or lexes one token and never reaches parse_paren_expr, so a kept arrow with no outer restore is not reachable. restore_parser_snapshot has a single caller, so the &mut signature change and the dropped non-restored snapshot affect no other path.

Extended reasoning...

The change touches the TypeScript parser's speculative conditional-arrow attempt in src/js_parser/parse/parse_skip_typescript.rs, its caller in src/js_parser/parse/mod.rs, the snapshot restore in src/js_parser/p.rs, and adds transpiler tests including an RSS bound; it touches no security-sensitive surface. The one posted finding is a pre-existing quadratic case the author documents; the kept-arrow path's correctness rests on the nested-detection premise, which I verified against every is_log_disabled writer in the crate.

Comment thread src/js_parser/parse/parse_skip_typescript.rs

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant