Skip to content

Rust edition upgrade, DX fix for generated token macros #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Apr 22, 2024
Merged

Conversation

dannymcgee
Copy link
Owner

Breaking changes

Generated token macros are no longer #[macro_export]

The Token derive macro generates a macro_rules! macro for each of the enum variants. E.g., for Lox's Token enum in examples/lox:

// examples/lox/src/tokens.rs

#[derive(DebugLispToken, PartialEq, Token, Lexer)]
pub enum Token {
    #[subset_of(Ident)]
    #[pattern = "and|class|else|false|for|fun|if|nil|or|print|return|super|this|true|var|while"]
    Keyword(Substr, Span),

    #[pattern = "[a-zA-Z_][a-zA-Z0-9_]*"]
    Ident(Substr, Span),

    #[pattern = r"[(){}]"]
    Brace(Substr, Span),

    #[pattern = "[,.;]"]
    Punct(Substr, Span),

    #[pattern = "[=!<>]=?"]
    #[pattern = "[-+*/]"]
    Operator(Substr, Span),

    #[pattern = "[0-9]+"]
    NumLit(Substr, Span),

    #[pattern = r#""[^"]*""#]
    StrLit(Substr, Span),
}

We get macro_rules! macros named keyword, ident, brace, punct, operator, num_lit and str_lit. These are mostly useful for the Parse implementations, e.g.:

// examples/lox/src/decl.rs

impl Parse for ClassDecl {
    type Stream = ParseStream;

    fn parse(input: &mut Self::Stream) -> Result<Self> {
        // ...
        let superclass = if input.check(operator![<]) {
            input.consume(operator![<])?;
            Some(input.consume_kind(TokenKind::Ident)?)
        } else {
            None
        };
        // ...
    }
}

Previously, those generated macros were declared like this:

#(
    #[macro_export]
    macro_rules! #snake_case_variant_ident {
        // ...
    }
)*

That has always caused some headaches which I was never really sure how to deal with. In the examples here, and in my consuming projects, I had resorted to using * imports everywhere to work around the problem (in hindsight though, I think that likely just hides the lint by obscuring what we're doing instead of actually addressing the issue).

This PR attempts an alternative solution. #[macro_export] is intended for libraries to expose macros to external consumers, but I don't foresee the macros generated by the Token derive being actually useful in that context. So instead, the generated macros are now declared like this:

#(
    macro_rules! #snake_case_variant_ident {
        // ...
    }
    pub(crate) use #snake_case_variant_ident;
)*

This is a breaking change for two reasons:

  1. If you were depending on those #[macro_export] attributes to make the generated macros available to external consumers of your library, that is no longer going to work. Again, I don't imagine this was actually a real-world use case for anyone, but I've been wrong before! Let me know if this is a problem for you and I'll see what we can do about it.

  2. If you had been importing those macros from the root of your crate, but your Token enum is not declared in the crate root, you'll need to update your import paths to instead import them from the module where the enum is declared. E.g.:

mod token {
    use gramatika::{DebugLisp, Span, Substr, Token as _};

    #[derive(DebugLispToken, PartialEq, Token, Lexer)]
    pub enum Token {
        #[pattern = ".+"]
        Word(Substr, Span),
    }
}

mod foo {
    // use crate::word;     // 👎
    use crate::token::word; // 👍
}

On the bright side, tools like rust-analyzer should now find and automatically suggest the correct import paths for those macros, so the fastest way to migrate will probably be to just delete your existing use statement and invoke your editor's suggestion feature to re-import any unresolved symbols from their correct paths.

Other changes

  • Updated the Rust edition to 2021 and fixed any resulting errors

  • Fixed any new clippy lints as a result of upgrading my environment to v1.77.2

  • Performed some low-hanging-fruit dependency upgrades.

    regex-automata and syn are still out of date for now — I attempted to update the former, but couldn't figure out how to migrate after ~10 minutes of poking around, and unfortunately I have other priorities that need to take precedence. Didn't even attempt syn because it's a major version bump, and that crate is basically the backbone of this whole project, so it'll have to wait for now.

@dannymcgee dannymcgee merged commit 6f3737e into main Apr 22, 2024
@dannymcgee dannymcgee deleted the update branch April 22, 2024 00:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant