Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Some parser cleanups #127233

Merged
merged 12 commits into from
Jul 3, 2024
Merged

Some parser cleanups #127233

merged 12 commits into from
Jul 3, 2024

Commits on Jul 2, 2024

  1. Simplify CfgEval. 

    It can contain an owned value instead of a reference.
    @nnethercote
    nnethercote committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    1680b79 View commit details
    Browse the repository at this point in the history

  2. Fix a typo in a comment.

    @nnethercote
    nnethercote committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    d6c0b81 View commit details
    Browse the repository at this point in the history

  3. Change AttrTokenStream::to_tokenstream to to_token_trees. 

    I.e. change the return type from `TokenStream` to `Vec<TokenTree>`.

Most of the callsites require a `TokenStream`, but the recursive call
used to create `target_tokens` requires a `Vec<TokenTree>`. It's easy
to convert a `Vec<TokenTree>` to a `TokenStream` (just call
`TokenStream::new`) but it's harder to convert a `TokenStream` to a
`Vec<TokenTree>` (either iterate/clone/collect, or use `Lrc::into_inner`
if appropriate).

So this commit changes the return value to simplify that `target_tokens`
call site.
    @nnethercote
    nnethercote committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    f852568 View commit details
    Browse the repository at this point in the history

  4. Rename TokenStream::new argument. 

    `tts` is a better name than `streams` for a `Vec<TokenTree>`.
    @nnethercote
    nnethercote committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    0cfd247 View commit details
    Browse the repository at this point in the history

  5. Just push in AttrTokenStream::to_token_trees. 

    Currently it uses a mixture of functional style (`flat_map`) and
imperative style (`push`), which is a bit hard to read. This commit
converts it to fully imperative, which is more concise and avoids the
need for `smallvec`.
    @nnethercote
    nnethercote committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    7416c20 View commit details
    Browse the repository at this point in the history

  6. Fix comment. 

    Both the indenting, and the missing `)`.
    @nnethercote
    nnethercote committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    36c30a9 View commit details
    Browse the repository at this point in the history

  7. Flip an if/else in AttrTokenStream::to_attr_token_stream. 

    To put the simple case first.
    @nnethercote
    nnethercote committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    2342770 View commit details
    Browse the repository at this point in the history

  8. Move things around in collect_tokens_trailing_token. 

    So that the `capturing` state is adjusted immediately before and after
the call to `f`.
    @nnethercote
    nnethercote committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    8b5a7eb View commit details
    Browse the repository at this point in the history

  9. Move more things around in collect_tokens_trailing_token. 

    To make things a little clearer, and to avoid some `mut` variables.
    @nnethercote
    nnethercote committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    f5b2896 View commit details
    Browse the repository at this point in the history

  10. Shrink parser positions from usize to u32. 

    The number of source code bytes can't exceed a `u32`'s range, so a token
position also can't. This reduces the size of `Parser` and
`LazyAttrTokenStreamImpl` by eight bytes each.
    @nnethercote
    nnethercote committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    3d750e2 View commit details
    Browse the repository at this point in the history

  11. Rename make_token_stream. 

    And update the comment. Clearly the return type of this function was
changed at some point in the past, but its name and comment weren't
updated to match.
    @nnethercote
    nnethercote committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    6f60156 View commit details
    Browse the repository at this point in the history

  12. Import std::{iter,mem}.

    @nnethercote
    nnethercote committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    edeebe6 View commit details
    Browse the repository at this point in the history