Build and publish the lexer to npm #20778
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
arusahni
force-pushed
the
main
branch
3 times, most recently
from
2896925
to
c6c1464
Compare
arusahni
changed the title
maddyblue
reviewed
Jul 25, 2023
benesch
reviewed
Jul 26, 2023
|
I've updated the PR to capture the above changes. |
|
@benesch between approval and merge we'll need to set the |
Done: https://github.com/MaterializeInc/i2/pull/1016 |
benesch
approved these changes
Jul 28, 2023
|
LGTM, modulo the debugging |
In an effort to ensure we're not maintaing two concurrent Lexer implementations, make the existing lexer available to more platforms. By building a WASM module, we can re-use it in JavaScript contexts. `wasm-pack` does most of the heavy lifting, but we need to ensure we're emitting types that can be serialized across the WASM interface, hence the flattening of the token array.
Certain crates, such as 'sql-lexer', can be compiled to WASM using the
'wasm-pack' tool. This commit:
1. Adds the relevant WASM libraries and tooling to the ci-builder.
2. Exposes additional CLI surface area to allow xcompile to run in the
ci-builder image.
Add CI hooks to build WASM packages alongside the crates in CI, and tooling to publish built artifacts to npm with the appropriate tags and version information. This will publish packages for _all_ builds on the deploy pipeline. Stable/official builds will not have a `-dev` suffix, and will receive the `latest` dist tag. Unstable builds will have a `dev` suffix that includes a build number, and will receive the `dev` dist tag.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Following #20345, let's publish the lexer to npm! This PR contains the three phases of the change, each in its own commit with details in the message. To summarize, they:
mz-sql-lexer. Perform the necessary updates to be able to compile the lexer to WASM targets as well as make it ergonomic for JS/TS callers. This includes a special allocator that reduces the bundle size.wasm-packtool. It also includes a wrapper script to codify invocation parameters.@materializincnamespace` be added to the Buildkite environment. A sample npm listing is here (please ignore the incorrect package name held over from testing).CI updates
Two additional CI jobs are added to the PR and deploy pipelines:
main. For dev builds, this uses pre-release versioning with a build identifier derived from the Buildkite build ID.cargo-deny exemption
The custom allocator (
lol_alloc) depends on spin 0.9, whilering(a dependency ofaws-config) depends on 0.5. This is fixed onring's main branch, but it has not yet made it to a stable release. The last stable release forringis 2+ years old, and the last unstable release is 8 months old. The ring dep may disappear altogether in a future version of aws-config. Given the alternate version of this crate is only used when targeting WASM, I believe this exemption is fine to carve out.Tips for reviewer
These are grouped by commit, and each commit message includes additional information.
Unfortunately, I had to disable TypeScript definition autogeneration for the
lexfunction asjs-sysdoes not provide adequate support for container types (i.e., it would returnArrayinstead ofToken[]).Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.