Abstract the pretty printer's ringbuffer to be infinitely sized #92923
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rustbot
added
the
T-compiler
|
(rust-highfive has picked a reviewer for you, use r? to override) |
rust-highfive
added
the
S-waiting-on-review
|
@bors r+ |
|
📌 Commit 7b69ad504592961a5b9e78426d8760f2d3eb475d has been approved by |
bors
added
S-waiting-on-bors
|
☔ The latest upstream changes (presumably #92996) made this pull request unmergeable. Please resolve the merge conflicts. |
bors
added
S-waiting-on-author
|
|
@bors r=petrochenkov |
|
📌 Commit 7b5b3cf has been approved by |
bors
added
S-waiting-on-bors
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Jan 18, 2022
…askrgr Rollup of 14 pull requests Successful merges: - rust-lang#92629 (Pick themes on settings page, not every page) - rust-lang#92640 (Fix ICEs related to `Deref<Target=[T; N]>` on newtypes) - rust-lang#92701 (Add some more attribute validation) - rust-lang#92803 (Hide mobile sidebar on some clicks) - rust-lang#92830 (Rustdoc style cleanups) - rust-lang#92866 ("Does exists" typos fix) - rust-lang#92870 (add `rustc_diagnostic_item` attribute to `AtomicBool` type) - rust-lang#92914 (htmldocck: Add support for `/text()` in ``@snapshot`)` - rust-lang#92923 (Abstract the pretty printer's ringbuffer to be infinitely sized) - rust-lang#92946 (Exclude llvm-libunwind from the self-contained set on s390x-musl targets) - rust-lang#92947 (rustdoc: Use `intersperse` in a `visit_path` function) - rust-lang#92997 (Add `~const` bound test for negative impls) - rust-lang#93004 (update codegen test for LLVM 14) - rust-lang#93016 (Stabilize vec_spare_capacity) Failed merges: - rust-lang#92924 (Delete pretty printer tracing) r? `@ghost` `@rustbot` modify labels: rollup
This was referenced Jan 18, 2022
matthiaskrgr
added a commit
to matthiaskrgr/rust
that referenced
this pull request
Jan 19, 2022
Pretty printer algorithm revamp step 2 This PR follows rust-lang#92923 as a second chunk of modernizations backported from https://github.com/dtolnay/prettyplease into rustc_ast_pretty. I've broken this up into atomic commits that hopefully are sensible in isolation. At every commit, the pretty printer is compilable and has runtime behavior that is identical to before and after the PR. None of the refactoring so far changes behavior. The general theme of this chunk of commits is: the logic in the old pretty printer is doing some very basic things (pushing and popping tokens on a ring buffer) but expressed in a too-low-level way that I found makes it quite complicated/subtle to reason about. There are a number of obvious invariants that are "almost true" -- things like `self.left == self.buf.offset` and `self.right == self.buf.offset + self.buf.data.len()` and `self.right_total == self.left_total + self.buf.data.sum()`. The reason these things are "almost true" is the implementation tends to put updating one side of the invariant unreasonably far apart from updating the other side, leaving the invariant broken while unrelated stuff happens in between. The following code from master is an example of this: https://github.com/rust-lang/rust/blob/e5e2b0be26ea177527b60d355bd8f56cd473bd00/compiler/rustc_ast_pretty/src/pp.rs#L314-L317 In this code the `advance_right` is reserving an entry into which to write a next token on the right side of the ring buffer, the `check_stack` is doing something totally unrelated to the right boundary of the ring buffer, and the `scan_push` is actually writing the token we previously reserved space for. Much of what this PR is doing is rearranging code to shrink the amount of stuff in between when an invariant is broken to when it is restored, until the whole thing can be factored out into one indivisible method call on the RingBuffer type. The end state of the PR is that we can entirely eliminate `self.left` (because it's now just equal to `self.buf.offset` always) and `self.right` (because it's equal to `self.buf.offset + self.buf.data.len()` always) and the whole `Token::Eof` state which used to be the value of tokens that have been reserved space for but not yet written. I found without these changes the pretty printer implementation to be hard to reason about and I wasn't able to confidently introduce improvements like trailing commas in `prettyplease` until after this refactor. The logic here is 43 years old at this point (Graydon translated it as directly as possible from the 1979 pretty printing paper) and while there are advantages to following the paper as closely as possible, in `prettyplease` I decided if we're going to adapt the algorithm to work better for Rust syntax, it was worthwhile making it easier to follow than the original.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR backports dtolnay/prettyplease@8e5e83c from the
prettypleasecrate intorustc_ast_pretty.Using a dedicated RingBuffer type with non-wrapping indices, instead of manually
%-ing indices into a capped sized buffer, unlocks a number of simplifications to the pretty printing algorithm implementation in followup commits such as dtolnay/prettyplease@fcb5968 and dtolnay/prettyplease@4427ced.This change also greatly reduces memory overhead of the pretty printer. The old implementation always grows its buffer to 205920 bytes even for files without deeply nested code, because it only wraps its indices when they hit the maximum tolerable size of the ring buffer (the size after which the pretty printer will crash if there are that many tokens buffered). In contrast, the new implementation uses memory proportional to the peak number of simultaneously buffered tokens only, not the number of tokens that have ever been in the buffer.
Speaking of crashing the pretty printer and "maximum tolerable size", the constant used for that in the old implementation is a lie:
rust/compiler/rustc_ast_pretty/src/pp.rs
Lines 227 to 228 in de9b573
It was raised from 3 to 55 in #33934 because that was empirically the size that avoided crashing on one particular test crate, but according to #33934 (comment) other syntax trees still crash at that size. There is no reason to believe that any particular size is good enough for arbitrary code, and using a large number like 55 adds overhead to inputs that never need close to that much of a buffer. The new implementation eliminates this tradeoff.