ElixirLS 0.19+ fails to compile projects with YamlElixir (and possible other) applications working at compile-time

[ymtszw]

Environment

• Elixir & Erlang versions (elixir --version): Erlang/OTP 26 [erts-14.2.2] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit]

Elixir 1.16.2 (compiled with Erlang/OTP 26)

• VSCode ElixirLS version: 0.20.0
• Operating System Version: darwin 22.6.0

Current behavior

As the title says, ElixirLS 0.19 onward fails to compile projects with YamlElixir application working at compile-time.

By "working at compile-time" I mean for instance, reading YAML-formatted config/data/etc. files and generate functions and modules from them.

Minimal repro repository here: https://github.com/ymtszw/repro

Example logs:

[Info  - 20:42:45] Starting build with MIX_ENV: test MIX_TARGET: host
Compiling 207 files (.ex)
== Compilation error in file lib/siiibo/model/static_template.ex ==
** (YamlElixir.ParsingError) malformed yaml
    (yaml_elixir 2.9.0) lib/yaml_elixir.ex:41: YamlElixir.read_from_string!/2
    lib/siiibo/model/static_template.ex:288: (module)
    (elixir 1.16.2) lib/kernel/parallel_compiler.ex:428: anonymous fn/5 in Kernel.ParallelCompiler.spawn_workers/8

Repro steps:

• (Prepare reasonably new Elixir and Erlang runtimes)
• Clone the repository
• mix deps.get
• Make sure it successfully compiles with mix compile
• Open the lib/repro.ex file in VSCode with the ElixirLS extension installed
• Observe the error message in the editor like so:
• OR, on initial opening of the file, the compilation may succeed. If so, give some arbitrary change to lib/repro.ex and save the file. The error should appear then.

Expected behavior

Compilations should succeed. At least it was compiled in version 0.18.x IIRC.

[ymtszw]

Projects described in OP can be compiled with mix compile but cannot via ElixirLS (or via VSCode-ElixirLS).

My observation as of now, is that YamlElixir is not a pure library but an Erlang application, which requires its own application tree to work (storing configs, intermediate data, and whatnot). That may be related.

[lukaszsamson]

It builds correctly on the first try

==> repro
Compiling 1 file (.ex)
Generated repro app
[Info  - 13:53:25] Compile took 1167 milliseconds

Then something gets unloaded and it crashes with a badmatch inside the lib

** (MatchError) no match of right hand side value: :undefined
    (yamerl 0.10.0) /Users/bug_repro/repro/deps/yamerl/src/yamerl_app.erl:69: :yamerl_app.get_param/1
    (yamerl 0.10.0) /Users/bug_repro/repro/deps/yamerl/src/yamerl_constr.erl:769: :yamerl_constr.setup_node_mods/2
    (yamerl 0.10.0) /Users/bug_repro/repro/deps/yamerl/src/yamerl_constr.erl:597: :yamerl_constr.construct/2
    (yamerl 0.10.0) /Users/bug_repro/repro/deps/yamerl/src/yamerl_parser.erl:4141: :yamerl_parser.do_emit_token/2
    (yamerl 0.10.0) /Users/bug_repro/repro/deps/yamerl/src/yamerl_parser.erl:3976: :yamerl_parser.emit_tokens2/4
    (yamerl 0.10.0) /Users/bug_repro/repro/deps/yamerl/src/yamerl_parser.erl:3659: :yamerl_parser.queue_token_check_doc/3
    (yamerl 0.10.0) /Users/bug_repro/repro/deps/yamerl/src/yamerl_parser.erl:1529: :yamerl_parser.queue_block_sequence_entry_token/5
    (yamerl 0.10.0) /Users/bug_repro/repro/deps/yamerl/src/yamerl_constr.erl:477: :yamerl_constr.string/2
    (yaml_elixir 2.9.0) lib/yaml_elixir.ex:72: YamlElixir.read/3
    (yaml_elixir 2.9.0) lib/yaml_elixir.ex:49: YamlElixir.do_read/3
    (yaml_elixir 2.9.0) lib/yaml_elixir.ex:39: YamlElixir.read_from_string!/2
    lib/repro.ex:4: (module)

the error is mishandled in the library and returned as {:error, %YamlElixir.ParsingError{message: "malformed yaml"}}

I don't know this lib nor what it is missing. ElixirLS tries to unload all apps, config and mix state and restart compilation with a clean slate. Maybe it's too aggressive (or not aggressive enough). It would be best if someone familiar with the library did some experiments. The code is in https://github.com/elixir-lsp/elixir-ls/blob/master/apps/language_server/lib/language_server/build.ex

[ymtszw]

** (MatchError) no match of right hand side value: :undefined

The error around this is what I have recreated in my debugging too.

YamlElixir itself is a thin wrapper around Erlang library yamerl, and :yamerl_app.get_param/1 depends on application env (here). I believe these are materials we must look around but haven't delved deeper.

(badmatch and "malformed yaml" errors are mere fallback errors within YamlElixir's error handling routes)

Does it gives enough context? I think YamlElixir/yamerl (and other possible affected apps) require access to application env, and that is what ElixirLS intercepts somehow?

[lukaszsamson]

ElixirLS loads app env in a similar way to what mix does. Then after the build it attempts to cleanup everything. I guess it cleans some yamerl entries and the lib is not recreating them correctly. The problem does not affect mix as it gets a fresh OS process each time

[ymtszw]

I'm Elixir person and open to provide fix, so I can tackle on the issue on my fork of YamlElixir.

What could be the most natural/least resistant way forward? Should ideal yaml lib, if it is called at compile-time in ElixirLS environment and detects missing yamerl application state, "resurrect" yamerl app on-demand?

Or, does ElixirLS have better tuning points to looking into?

(I believe the most ideal yaml and other file codec library should be pure and brings no problems like this from the very first but anyway. yamerl is already a popular choice)

[lukaszsamson]

@ymtszw It turned out that mix project reset was cleaning up app config yamerl but not unloading the app itself. On subsequent compiles it crashed due to no :node_mods key in app env. I addressed that by unloading all apps.

[ymtszw]

Thank you! 2024年5月4日(土) 14:26 Łukasz Samson ***@***.***>:

…

@ymtszw <https://github.com/ymtszw> It turned out that mix project reset was cleaning up app config yamerl but not unloading the app itself. On subsequent compiles it crashed due to no :node_mods key in app env. I addressed that by unloading all apps. — Reply to this email directly, view it on GitHub <#1083 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCML5VBI6AHQJ7GQV5OO23ZARWRPAVCNFSM6AAAAABFBJDCUGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJUGAZDMMBVGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>