Towards getting JuliaSyntax to work with --trim and static binaries.

[KristofferC]

Tools that are convenient to use from the command line (like code formatters) are likely to use JuliaSyntax and being able to statically compile these would be nice for latency and distribution. This issue is about the work required to get JuliaSyntax to that state.

Right now, using the branch at JuliaLang/julia#55047 together with the following script:

tmpdir = mktempdir()
img_path = joinpath(tmpdir, "img.a")
trim_path = joinpath(Sys.BINDIR, "../../test/trimming")

cmd = Base.julia_cmd()
cmd = `$cmd --startup-file=no --history-file=no`
cmd = addenv(`$cmd --output-o $img_path --output-incremental=no --strip-ir --strip-metadata --trim $(joinpath(trim_path,"buildscript.jl")) juliasyntax.jl --output-exe true`, "OPENBLAS_NUM_THREADS" => 1, "JULIA_NUM_THREADS" => 1)

run(cmd)

and the file

module MyApp

using JuliaSyntax

Base.@ccallable function main()::Cint
    JuliaSyntax.parseall(JuliaSyntax.SyntaxNode, "NaN")
    return 0
end

end

the following error is quickly encountered:

Dynamic call to Base.:(+)(Any, Int64)
In operators.jl:642

Stacktrace:
  [1] +;
    @ operators.jl:642 [inlined]
  [2] seek(io::GenericIOBuffer{GenericMemory{:not_atomic, UInt8, ?}}, n::Int64)
    @ Base iobuffer.jl:288
  [3] parse_julia_literal(txtbuf::Array{UInt8, 1}, head::SyntaxHead, srcrange::UnitRange{Int64})
    @ JuliaSyntax ~/.julia/packages/JuliaSyntax/q0tWf/src/literal_parsing.jl:394
  [4] _to_SyntaxNode(source::SourceFile, txtbuf::Array{UInt8, 1}, offset::Int64, raw::GreenNode{SyntaxHead}, position::Int64, keep_parens::Bool)
    @ JuliaSyntax ~/.julia/packages/JuliaSyntax/q0tWf/src/syntax_tree.jl:72
  [5] #SyntaxNode#106;
    @ ~/.julia/packages/JuliaSyntax/q0tWf/src/syntax_tree.jl:61 [inlined]
  [6] TreeNode;
    @ ~/.julia/packages/JuliaSyntax/q0tWf/src/syntax_tree.jl:56 [inlined]

For now, I think using JET and fixing as much as possible is the best way forward. I will play with it a bit and update the post here with some results.

[KristofferC]

The error above is fixed by JuliaLang/julia#55424. Hitting some new stuff now but won't spam this issue with updates unless something interesting happens.

[Keno]

Another interesting application is to generate wasm versions of the parser to ship for VSCode extensions and other editors, so that parser services work without an active language server.

[KristofferC]

This actually works now so I think it can be closed.