refactor(ast): Introduce flexible serialization encoding for AST #11100

quininer · 2025-09-18T08:25:25Z

Description:

Following our previous discussion, I am trying to introduce a self-describing serialization encoding for ast to replace #11069 .

Since serde is currently used for json serialization, it is hard to modify serde to support both json and binary encodings (due to the untagged and flatten attr). So I made our own Encode and Decode proc macro, which also facilitate unknown variant support.

I've done some simple benchmarks with a modified swc ast, and I believe we don't have an unacceptable performance loss relative to rkyv.
I'm using cbor4ii here, which I developed, so I'd prefer it if there's no performance difference. but it's easy to migrate to bincode or other scheme.

This also introduces an unknown feature for ecma_ast crate, which should only be enabled by the wasm plugin to ensure that old plugins can decode new AST.

BREAKING CHANGE:

This PR itself only introduces new encoding and will not cause any breakage. However, it may cause breakage when switch the wasm plugin communication scheme in future.

Related issue (if exists):

changeset-bot · 2025-09-18T08:25:32Z

🦋 Changeset detected

Latest commit: f23fdbf

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

codspeed-hq · 2025-09-18T12:46:13Z

CodSpeed Performance Report

Merging #11100 will not alter performance

_{Comparing quininer:r/self-desc-encode (f23fdbf) with main (dba23f5)}

Summary

✅ 138 untouched

socket-security · 2025-09-19T03:46:18Z

No dependency changes detected. Learn more about Socket for GitHub.

👍 No dependency changes detected in pull request

**Description:** This is the split of #11100. Since we need to add unknown variants to the ast enum, this requires a lot of code changes to handle non-exhaustive matches. I'm doing this in a separate PR for review.

quininer · 2025-10-14T07:42:11Z

This also requires a serialization test, include unknown variants.

Since I'm busy with other work, it will be a little slower.

github-actions · 2025-10-30T04:27:05Z

Binary Sizes

File	Size
`swc.linux-x64-gnu.node`	31M (32023392 bytes)

Commit: c4ed6d2

crates/jsdoc/src/ast.rs

crates/ast_node/src/encoding/encode.rs

crates/ast_node/src/encoding/mod.rs

crates/swc_common/src/lib.rs

Copilot

Pull Request Overview

Copilot reviewed 107 out of 112 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-10-31T07:32:51Z

crates/swc_common/tests/attr_interop.rs

    pub s: String,
 }

 #[ast_node("Tuple")]


The change from usize to u64 alters the semantics of the Tuple struct. On 32-bit platforms, usize is 32 bits while u64 is 64 bits, which could lead to different behavior. This appears to be unrelated to the encoding implementation and should be documented or justified.

Suggested change

#[ast_node("Tuple")]

#[ast_node("Tuple")]

// NOTE: We use `u64` instead of `usize` for cross-platform consistency and to avoid issues with serialization/deserialization on 32-bit vs 64-bit platforms. If you need platform-dependent sizing, consider using `usize` and document the rationale.

Copilot · 2025-10-31T07:32:52Z

crates/swc_common/src/serializer.rs

@@ -1,5 +1,5 @@
 #![allow(deprecated)]
-#![deprecated = "Not used by swc, and this will be removed with next breaking change"]
+


The deprecation attribute and warning comment were removed without explanation. If this module is no longer deprecated and is being actively used for CBOR encoding support, this should be documented in the PR description or in a code comment explaining why the deprecation was lifted.

Copilot · 2025-10-31T07:32:52Z

crates/swc_atoms/src/wtf8_atom.rs

+        // This is not sound, maybe Wtf8Buf should make bytes operations safe
+        Ok(Self(hstr::Wtf8Atom::from(unsafe {
+            Wtf8Buf::from_bytes_unchecked(s.0.into())
+        })))


The comment admits this is 'not sound' yet uses unsafe code without validation. CBOR decoding should validate that the bytes form valid WTF-8. Consider adding validation before the unsafe conversion or document why it's safe in this context.

Suggested change

// This is not sound, maybe Wtf8Buf should make bytes operations safe

Ok(Self(hstr::Wtf8Atom::from(unsafe {

Wtf8Buf::from_bytes_unchecked(s.0.into())

})))

// Validate that the bytes are valid WTF-8 before constructing Wtf8Buf

match Wtf8::from_bytes(s.0) {

Some(_) => Ok(Self(hstr::Wtf8Atom::from(Wtf8Buf::from_bytes_unchecked(s.0.into())))),

None => Err(cbor4ii::core::dec::Error::message("Invalid WTF-8 bytes")),

}

crates/swc_css_parser/tests/fixture/function/unknown/output.json

kdy1

Thank you so much for the hard work!

move to ast_node make encoding-impl optional fix fmt make css_ast fix tuple fix unknown fix fmt make html_ast make xml_ast make jsdoc fix fmt fix unit tag update visit fix parser make regexp_ast no unknown fix snap fix arbitrary encoding: ignore unknown field fix cfg fix enum struct derive

Refactor AST serialization to support flexible encoding.

Copilot

Pull Request Overview

Copilot reviewed 108 out of 113 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

**Description:** This is a follow up to #11100, which will result in plugin abi break. It currently passes the plugin tests, but I still need to do some code and configuration cleanup, and add a regression test. <img width="2053" height="336" alt="plugin tests" src="https://github.com/user-attachments/assets/868b2ae8-105c-44ad-8d18-641a2dd33c9f" /> **BREAKING CHANGE:** Since we switched serialization schemes, this will result in breaking changes on the plugin ABI and api. --------- Co-authored-by: Donny/강동윤 <[email protected]> Co-authored-by: Donny/강동윤 <[email protected]> Co-authored-by: Copilot <[email protected]>

## Summary by CodeRabbit * **Chores** * Updated `swc_core` dependency to version 47.0.3. * Adjusted `serde_json` dependency configuration. * **Refactor** * Enhanced string handling across compiler plugins for robust text processing. * Updated AST processing logic for improved compatibility. * Improved import alias handling across multiple plugin modules. * Refined attribute and literal value transformations.  Related SWC breaking changes: 1. `Atom` -> `Wtf8Atom`: swc-project/swc#11144 > `Wtf8Atom` does not expose a `to_string` method like the old `Atom` does. Internally, it stores the code points of the characters. You can call `code_points()` to get an iterator of the code points or call `to_string_lossy()` to get an lossy string in which all unpaired surrogates are replaced with `U+FFFD`(Replacement Character). 2. flexible AST: swc-project/swc#11100 ## Checklist  - [ ] Tests updated (or not required). - [ ] Documentation updated (or not required). - [x] Changeset added, and when a BREAKING CHANGE occurs, it needs to be clearly marked (or not required).

quininer force-pushed the r/self-desc-encode branch 2 times, most recently from c9d0a01 to de8261b Compare September 18, 2025 12:27

quininer force-pushed the r/self-desc-encode branch from de8261b to bc125dc Compare September 19, 2025 03:44

quininer force-pushed the r/self-desc-encode branch 12 times, most recently from 1b60f1d to c5fab18 Compare September 22, 2025 12:10

quininer mentioned this pull request Sep 22, 2025

refactor(ast_node): Make AST enums non_exhaustive #11115

Merged

quininer force-pushed the r/self-desc-encode branch 5 times, most recently from 4da0e40 to 815343f Compare September 23, 2025 09:47

quininer force-pushed the r/self-desc-encode branch from 815343f to 1babf03 Compare September 30, 2025 09:21

quininer force-pushed the r/self-desc-encode branch from 1babf03 to 9d04340 Compare October 14, 2025 04:31

quininer force-pushed the r/self-desc-encode branch 2 times, most recently from c736814 to 46b5ab6 Compare October 15, 2025 10:35

kdy1 mentioned this pull request Oct 30, 2025

perf(plugin): hstr::Atom has very inefficient serialization/deserialization logic #11207

Open

quininer force-pushed the r/self-desc-encode branch from 8d05be8 to 172a594 Compare October 30, 2025 08:20

kdy1 self-assigned this Oct 30, 2025

kdy1 reviewed Oct 31, 2025

View reviewed changes

crates/jsdoc/src/ast.rs Show resolved Hide resolved

kdy1 reviewed Oct 31, 2025

View reviewed changes

crates/ast_node/src/encoding/encode.rs Show resolved Hide resolved

kdy1 reviewed Oct 31, 2025

View reviewed changes

crates/ast_node/src/encoding/mod.rs Show resolved Hide resolved

crates/ast_node/src/encoding/mod.rs Show resolved Hide resolved

crates/swc_common/src/lib.rs Outdated Show resolved Hide resolved

Copilot AI review requested due to automatic review settings October 31, 2025 07:30

quininer force-pushed the r/self-desc-encode branch 2 times, most recently from 6d0f5ff to 3737124 Compare October 31, 2025 07:32

Copilot AI reviewed Oct 31, 2025

View reviewed changes

kdy1 reviewed Oct 31, 2025

View reviewed changes

crates/swc_css_parser/tests/fixture/function/unknown/output.json Show resolved Hide resolved

kdy1 approved these changes Oct 31, 2025

View reviewed changes

kdy1 changed the title ~~refactor(ast): Introduce flexible serialization encoding for ast~~ refactor(ast): Introduce flexible serialization encoding for AST Oct 31, 2025

kdy1 enabled auto-merge (squash) October 31, 2025 07:44

quininer and others added 4 commits October 31, 2025 15:58

Add encoding-impl test

20d92c6

fix fmt

eed0dca

Update serialization encoding for AST

f23fdbf

Refactor AST serialization to support flexible encoding.

Copilot AI review requested due to automatic review settings October 31, 2025 07:58

auto-merge was automatically disabled October 31, 2025 07:58
Head branch was pushed to by a user without write access

quininer force-pushed the r/self-desc-encode branch from f497f9e to f23fdbf Compare October 31, 2025 07:58

Copilot AI reviewed Oct 31, 2025

View reviewed changes

kdy1 approved these changes Oct 31, 2025

View reviewed changes

kdy1 merged commit 8ad3647 into swc-project:main Oct 31, 2025
180 checks passed

quininer deleted the r/self-desc-encode branch October 31, 2025 10:02

colinaaa mentioned this pull request Nov 5, 2025

fix(deps): bump swc_core v47 lynx-family/lynx-stack#1916

Merged

3 tasks

	#[ast_node("Tuple")]
	#[ast_node("Tuple")]
	// NOTE: We use `u64` instead of `usize` for cross-platform consistency and to avoid issues with serialization/deserialization on 32-bit vs 64-bit platforms. If you need platform-dependent sizing, consider using `usize` and document the rationale.

		@@ -1,5 +1,5 @@
		#![allow(deprecated)]
		#![deprecated = "Not used by swc, and this will be removed with next breaking change"]

-        // This is not sound, maybe Wtf8Buf should make bytes operations safe
-        Ok(Self(hstr::Wtf8Atom::from(unsafe {
-            Wtf8Buf::from_bytes_unchecked(s.0.into())
-        })))
+        // Validate that the bytes are valid WTF-8 before constructing Wtf8Buf
+        match Wtf8::from_bytes(s.0) {
+            Some(_) => Ok(Self(hstr::Wtf8Atom::from(Wtf8Buf::from_bytes_unchecked(s.0.into())))),
+            None => Err(cbor4ii::core::dec::Error::message("Invalid WTF-8 bytes")),
+        }

Uh oh!

refactor(ast): Introduce flexible serialization encoding for AST #11100

refactor(ast): Introduce flexible serialization encoding for AST #11100

Uh oh!

Conversation

quininer commented Sep 18, 2025

Uh oh!

changeset-bot bot commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

codspeed-hq bot commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed Performance Report

Merging #11100 will not alter performance

Summary

Uh oh!

socket-security bot commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

quininer commented Oct 14, 2025

Uh oh!

github-actions bot commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Binary Sizes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kdy1 left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

changeset-bot bot commented Sep 18, 2025 •

edited

Loading

codspeed-hq bot commented Sep 18, 2025 •

edited

Loading

socket-security bot commented Sep 19, 2025 •

edited

Loading

github-actions bot commented Oct 30, 2025 •

edited

Loading