Move JuliaSyntax + JuliaLowering into the main tree #59870
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* A vector of `Slot`s is now created and passed into the `CodeInfo` creation pass so that code doesn't need access to the `Bindings` anymore. This is a better separation of data structures between passes. * Use K"Placeholder" for unused slots. * Fix small bug which made argument slurping broken.
Also fix a bug in linearization of `K"isdefined"`
Co-authored-by: spaette <[email protected]>
…#511) Julia's ecosystem (including Base.Docs and flisp lowering) assumes that strings within `struct` definitions are per-field docstrings, but the flisp parser doesn't handle these - they are only recognized when the struct itself has a docstring and are processed by the `@doc` macro recursing into the struct's internals. For example, the following doesn't result in any docs attached to `A`. ```julia struct A "x_docs" x "y_docs" y end ``` This change adds `K"doc"` node parsing to the insides of a struct, making the semantics clearer in the parser tree and making it possible to address this problems in the future within JuliaLowering. Also ensure that the `Expr` form is unaffected by this change.
…ng/JuliaSyntax.jl#506) * Don't assume that `SubString` has `pointer` and copy instead * Still assume `Substring{String}` has `pointer` * Test with `Test.GenericString`
…liaLang/JuliaSyntax.jl#500) * Remove the method `convert(::Type{String}, ::Kind)` This patch removes the method `convert(::Type{String}, ::Kind)` used for converting kinds to strings and replaces it with the already existing method of `Base.string`. There are two reason for this: i) the method causes invalidations when loading the package and ii) `convert` is called implicitly in e.g. constructors and should therefore typically only be defined between similar enough types. * Remove the method `Base.convert(::Type{Kind}, ::String)` This patch removes the method `Base.convert(::Type{Kind}, ::AbstractString)` and replaces it with a `Kind(::AbstractString)` constructor. The reason for this is that `convert` is called implicitly in e.g. constructors and should therefore typically only be defined between similar enough types.
Also introduce `K"code_info"` to distinguish the `CodeInfo`-like form with indexed statements from the more symbolic cross references that are used internally by lowering within `K"lambda"` prior to statement+SSA renumbering.
Still todo: * inner constructors * outer constructors * doc binding Also included here is `K"alias_binding"` - a more general replacement for the `outerref` used in flisp lowering. `alias_binding` allows one to allocate a binding early during desugaring and make this binding an alias for a given name. Bindings don't participate in scope resolution, so this allows us to bypass the usual scoping rules. For example, to refer to a global struct_name from an outer scope, but within an inner scope where the identifier struct_name is bound to a local variable. (We could also replace outerref by generating a new scope_layer and perhaps that would be simpler?)
This form where `K"lambda"` has four children [args, static_parameters, body, ret_var] feels more natural as it keeps AST pieces within the AST rather than as auxiliary attributes. These pieces do still need special treatment in scope resolution, but lambdas are already special there.
Avoid creating `::` expressions - just add these directly to the function argument name and type lists instead.
Perhaps this was used historically but it's now only used for method tables in method overlays.
As much as alias_binding is a neat idea, it seems like using a scope layer to distinguish the global vs local bindings might be good enough and allow us to remove the alias_binding concept. As a side effect, this may allow us to avoid needing support arbitrary bindings in some early lowering code.
Detangling this ball of string ... felt quite epic 😬😅 Here we take a different approach from the flisp code - we don't try to reproduce the function signature matching logic of `expand_function_def` to rewrite constructor signatures within the struct expansion code. Instead, we harness that existing logic by calling expand_function_def with custom rewrite functions for the inner part of the signature expression and the function body where `new()` occurs.
* Remove outterref - this has been removed upstream * Make expand_unionall_def its own function - this will be required shortly to match some changes upstream. * JuliaSyntax has removed the `convert` overload for `Kind` in the latest dev version
…args an error Here we introduce a `meta` attribute rather than - or perhaps in addition to - the `K"meta"` kind and use it to tag local variables which derived from function argument destructuring. We use this to make it an error to have duplicate destructured argument names. This is technically breaking, but probably only a good thing - without this users will silently have the intial duplicate argument names overwritten with the result of the last destructuring assignment. Also add tests for the various variable scope conflict errors: argument/local, static-parameter/local, local/global etc.
…wering These macros are a part of the language itself because they emit special syntax trees which is known to lowering. This is regardless of the fact that they don't have a surface syntax form. Where a `Base` form of these exists we add a method to that macro so it can be used as usual without needing to import from JuliaLowering.
topolarity
force-pushed
the
caf/merge-julia-frontend
branch
from
ffb0f6f to
afc8cef
Compare
Member
|
Fully recreated the merge + commit replay. $ git filter-repo --path-rename :JuliaSyntax/ --tag-rename :JuliaSyntax- --replace-message ../replace.txt
$ cat replace.txt
regex:(\W)(#[0-9]+)==>\1JuliaLang/JuliaSyntax.jl\2The rest is a local When CI is green this should be ready to merge (if no one spots any new issues). |
These are the simplest possible adaptions to create the vendored Base.JuliaSyntax from an in-tree version of JuliaSyntax (JuliaLowering to be hooked up later). Also remove JuliaLowering / JuliaSyntax GHA actions and update JuliaLowering `[sources]` to use local `JuliaSyntax`.
The environment when running under Distributed is slightly different than running via `Pkg.test()` so this required some tweaks to error printing, etc.
topolarity
force-pushed
the
caf/merge-julia-frontend
branch
from
afc8cef to
d0f24f2
Compare
Member
Author
|
Oh nice, thanks Cody! So just to check I understand - you used the version of the filtered git history I originally pushed to this PR, but recreated the merges (plus the additional commits which have happened in the meantime?) If that's correct, I should probably push the branches I had from the JuliaSyntax PRs which haven't merged yet into some location so that people can pick them up and recreate PRs against JuliaLang/julia with ease. |
Member
I recreated the filtered history as well, since the latest merge seemed to be missing some of the issue / PR text replacements Hopefully I got the details right - you can see them in my previous comment
Sounds good to me - I think the other thing we still need to do is move JuliaLowering into the JuliaLang org |
aviatesk
added a commit
to aviatesk/JETLS.jl
that referenced
this pull request
Nov 19, 2025
With JuliaLang/julia#59870 merged now, we should also use `JuliaLang/julia` as the remote source of those libraries.
aviatesk
added a commit
to aviatesk/JETLS.jl
that referenced
this pull request
Nov 19, 2025
With JuliaLang/julia#59870 merged now, we should also use `JuliaLang/julia` as the remote source of those libraries.
This was referenced Nov 20, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There's been some interest in having the new Julia compiler frontend (JuliaSyntax + JuliaLowering) in the main Julia tree so that these are easier to work on together and so the new lowering code can co-evolve with changes to Core more easily.
Here's a simple sketch for moving both these libraries into the main tree as separate top level modules in the JuliaSyntax and JuliaLowering subdirectories. For git history, I've used
git-filter-repoto rewrite the history of both repositories into their respective subdirectories. At the same time some light rewriting was performed to avoid confusion for commit messages referring to issue numbers. For example, if a commit in the JuliaSyntax history refers to #256, that will be rewritten to the string JuliaLang/JuliaSyntax.jl#256. (Note for completeness that the history of these projects also includes the git history of Tokenize.jl which is the origin of the lexer.)There's a few questions / TODOs I'd like to consider before merging this:
How do we do CI of JuliaSyntax against old Julia versions?
JuliaSyntax currently supports Julia versions back to 1.0 (!!) Admittedly this may be excessive, but we should keep the JuliaSyntax registered in General working for at least some older Julia versions.
The problem is I know very little about how to set this up and I'd like advice or help :) @IanButterworth I can see you're active with both build kite and github actions infrastructure - I hoped you might have some thoughts or be able to point me in the right direction? Presumably we download pre-built versions from
julialang-s3.julialang.organd test the JuliaSyntax module against those in addition to the current dev version of Julia.Easing the archiving of JuliaLang/JuliaSyntax
There's enough open PRs on JuliaSyntax that it'd be nice to make migrating those to the main Julia repository easy. My rough plan is to filter all branches while running
git-filter-repoand push those filtered branches to JuliaLang/JuliaSyntax. Then PR authors should be able to grab the filtered version of their branch and apply it to the main Julia repo without issues. I haven't figured out the details of this yet but it should be done in onegit-filter-reporun to ensure consistency of version hashes.When this is done I'll also move c42f/JuliaLowering.jl into JuliaLang/JuliaLowering.jl and archive it so there's a more permanent home for the associated github issue and PR discussions.
What should these modules be called?
I hesitate to bring this up because it might become a distraction. But if we want to rename either of these modules it makes sense to do it now while we're moving git histories around.
Originally,
JuliaSyntaxwas named that way because there was a very old and obsoleteJuliaParseralready taking the name, and the prefix "Julia" was used for clarity given that it was going into the General registry. (Also, the parser work was started as an experimental side project and taking a canonical name seemed rather too bold 😅) If we want to claim a more canonical name at this point we might consider renaming it toParser. (Of course we could takeJuliaParseras a name, but that seems marginal enough that we may as well stick with the existing name.)JuliaLoweringwas named with the same convention but if we change the JuliaSyntax name to just Parser we might also consider renaming JuliaLowering to something likeLoweringorCodeLowering.CompilerFrontendis also a tempting name but not including the parser in the "compiler frontend" would be a bit weird.