refactor(oxc_parser): introduce ParserConfig

[lilnasy]

• Introduces ParserConfig, a generic interface that concretely does nothing for most compilation targets and will store tokens in an arena when compiled for oxlint.
• Uses the code snippets from Linter plugins: Rustify token methods oxc-project/oxc#16207 as starting point.
• At this point, the goal is to ensure there is no change in behavior or performance. The oxlint-exclusive storage of tokens is not implemented yet.

[lilnasy]

I was running in circles trying to figure this out. This is what claude suggested but the function call probably has a cost (maybe #[inline] helps). I'll have to come back to this another time.

[overlookmotel]

This is tricky. The reason we want a static is that statics are stored in the binary only once, whereas a const is inlined everywhere it's used. Lexer::handle_byte has 2 call sites, so the danger is that we end up with 2 copies of the table in the binary, which is wasteful. The table is 2 KiB (256 x 8 bytes).

There doesn't seem to be any way to do that, though. Rust doesn't allow statics to be generic.

Maybe this is slightly better:

/// Byte handler.
#[expect(type_alias_bounds)]
type ByteHandler<Config: ParserConfig> = unsafe fn(&mut Lexer<'_, Config>) -> Kind;

impl<Config: ParserConfig> Lexer<'_, Config> {
    /// Byte handlers for this `ParserConfig`.
    const BYTE_HANDLERS: [ByteHandler<Config>; 256] = byte_handlers();

    /// Handle next byte of source.
    ///
    /// # SAFETY
    ///
    /// * Lexer must not be at end of file.
    /// * `byte` must be next byte of source code, corresponding to current position of `lexer.source`.
    /// * Only `BYTE_HANDLERS` for ASCII characters may use the `ascii_byte_handler!()` macro.
    // `#[inline(always)]` to ensure is inlined into `read_next_token`
    #[expect(clippy::inline_always)]
    #[inline(always)]
    pub(super) unsafe fn handle_byte(&mut self, byte: u8) -> Kind {
        // SAFETY: Caller guarantees to uphold safety invariants
        unsafe { Self::BYTE_HANDLERS[byte as usize](self) }
    }
}

/// Lookup table mapping any incoming byte to a handler function defined below.
/// <https://github.com/ratel-rust/ratel-core/blob/v0.7.0/ratel/src/lexer/mod.rs>
#[rustfmt::skip]
const fn byte_handlers<Config: ParserConfig>() -> [ByteHandler<Config>; 256] {
    [
        // ....
    ]
}

At least then byte handlers table is a const, rather than stored in a local var.

(side benefit: this version also doesn't have the weird for<'a> syntax)

Maybe compiler is smart enough to de-duplicate the tables and store it only once. To find out, we'll need to build it, and then check the binary to see if it increases in size or not.

[lilnasy]

We basically need a wrapper for std::mem::replace(&mut self.tokens, Vec::new_in(allocator)).

[overlookmotel]

Maybe:

pub trait TokenStore<'a> {
    // ...
    fn into_tokens(self, allocator: &'a Allocator) -> ArenaVec<'a, Token>;
}

impl<'a> TokenStore<'a> for NoopTokenStore {
    // ...
    fn into_tokens(self, allocator: &'a Allocator) -> ArenaVec<'a, Token> {
        oxc_allocator::Vec::new_in(allocator)
    }
}

impl<'a> TokenStore<'a> for RealTokenStore<'a> {
    // ...
    fn into_tokens(self, _allocator: &'a Allocator) -> ArenaVec<'a, Token> {
        self.tokens
    }
}

Then in ParserImpl::parse:

let tokens = self.lexer.tokens.into_tokens(self.ast.allocator);

into_tokens consumes self, so no need for mem::replace.

[lilnasy]

Clone can simply be derived but that causes trait-satisfaction errors in ParserCheckpoint that I can't make sense of yet.

[overlookmotel]

The auto-derived trait impl is this:

impl<Config: Clone> Clone for LexerCheckpoint<Config>
where
    Config: ParserConfig,
    Config::Tokens<'a>: Clone,
{
    fn clone(&self) -> Self {
        match self {
            LexerCheckpoint {
                source_position: source_position,
                token: token,
                errors_snapshot: errors_snapshot,
                token_checkpoint: token_checkpoint,
            } => LexerCheckpoint {
                source_position: source_position.clone(),
                token: token.clone(),
                errors_snapshot: errors_snapshot.clone(),
                token_checkpoint: token_checkpoint.clone(),
            },
        }
    }
}

This has bounds that both Config: Clone and Config::Tokens<'a>: Clone, and it doesn't implement Clone on LexerCheckpoint unless both those are satisfied.

Not sure why it includes those bounds, as they're not required. Probably it can't statically determine that they're not required - macros operate as text manipulators only - they don't have type information.

Neither Config nor Config::Tokens are Clone, and neither can be, because oxc_allocator::Vec isn't Clone.

So no way I can see to avoid manual implementation of Clone here. I'd suggest just adding a short comment saying why.

(If you use VS Code, you can see the expansion of any macro by putting cursor on the macro, and Command Palette -> "rust-analyser: Expand macro recursively at caret")

[lilnasy]

There is a separate impl block for StandardParserConfig to avoid having to add type annotations in the many places using Lexer::new. That usage remains unambiguous and continues to work as-is. It reduces the PR diff size and I imagine makes life slightly easier for non-oxc projects using our crates.

Parser and ParserImpl have similar refactors.

[lilnasy]

This is oxc_allocator::Vec.

[overlookmotel]

Wow this gets a bit complicated. I hadn't clocked that BYTE_HANDLERS would be impacted by the change.

A few broad comments on top of the smaller notes below:

No-op impl first

I suggest splitting this change into 2 parts:

1. Introduce ParserConfig as an empty trait (no tokens code). Only add StandardParserConfig.
2. Add all the tokens-related code.

Step (1) is pure refactor. After that PR alone, we can check that the changes to BYTE_HANDLERS aren't bloating the binary.

Once that's done, we'll be able to concentrate on the tokens-related code in isolation.

Shorten code

We have to add the generic in a lot of places. To make it less intrusive, I suggest:

// Before
use crate::ParserConfig;
impl<'a, Config: ParserConfig> ParserImpl<'a, Config> {
  // ...
}

// After
use crate::ParserConfig as Config;
impl<'a, C: Config> ParserImpl<'a, C> {
  // ...
}

ParserConfig future

FYI: I have other plans for ParserConfig aside from tokens.

I'd like to add more methods to ParserConfig, and make ParseOptions implement ParserConfig. Then you'll be able to set parser options at compile time instead of dynamically if you want. For instance:

trait ParserConfig {
  fn preserve_parens(&self) -> bool;
}

struct ParseOptions {
  preserve_parens: bool;
}

impl ParserConfig for ParseOptions {
  fn preserve_parens(&self) -> bool {
    self.preserve_parens
  }
}

This behaves exactly the same as at present, but for example in linter where preserve_parens is always true, we can implement a specialized ParserConfig which returns true from preserve_parens unconditionally. This will remove branches from the version of parser used in linter.

(I'm not suggesting we do that now - just to put you in the picture of what I have in mind)

[overlookmotel]

Please also update the comment above this showing what the macro expands to.

[overlookmotel]

Please add use oxc_allocator::Vec as ArenaVec; at top of file and use ArenaVec here. Ditto other places where we have oxc_allocator::Vec.

[overlookmotel]

Ha yes! How about:

• RealParserConfig -> ParserConfigWithTokens
• StandardParserConfig -> ParserConfigWithoutTokens
• RealTokenStore -> ActiveTokenStore

?

[overlookmotel]

Can leave this as it was (as we'll use ArenaVec for the other Vec).

[overlookmotel]

I think this is only used in 1 test. Could:

1. Convert that test to use new_with_config instead.
2. Remove new.
3. Rename new_with_config to new.

[overlookmotel]

Is this where clause required? You already require this bound in impl<'a, Config: ParserConfig> Lexer<'a, Config> above.

[overlookmotel]

Nit: Move this import down into the existing use crate::{ ... } block.