Skip to content

elixir-ls: fix elixir stdlib code detection, elixir: fix deterministic builds and 1.17 max OTP#423588

Merged
adamcstephens merged 3 commits intoNixOS:masterfrom
adamcstephens:push-ukqlsyplorpn
Jul 17, 2025
Merged

elixir-ls: fix elixir stdlib code detection, elixir: fix deterministic builds and 1.17 max OTP#423588
adamcstephens merged 3 commits intoNixOS:masterfrom
adamcstephens:push-ukqlsyplorpn

Conversation

@adamcstephens
Copy link
Contributor

@adamcstephens adamcstephens commented Jul 8, 2025

  • Fixes enabling deterministic builds for Elixir
  • Cleans up and patches elixir-ls to properly find Elixir stdlib

After digging into this, I realized this elixir-ls package is wrong in multiple ways. It's built as a mix release, but packaged as a set of scripts which will Mix.install elixir-ls from git. I plan to clean this up in a future PR, but this should fix the immediate problems.

Closes #422529

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • Nixpkgs 25.11 Release Notes (or backporting 25.05 Nixpkgs Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
  • NixOS 25.11 Release Notes (or backporting 25.05 NixOS Release notes)
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other contributing documentation in corresponding paths.

Add a 👍 reaction to pull requests you find important.

@nixpkgs-ci nixpkgs-ci bot added 6.topic: erlang General-purpose, concurrent, functional high-level programming language 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 11.by: package-maintainer This PR was created by a maintainer of all the package it changes. labels Jul 8, 2025
@adamcstephens adamcstephens force-pushed the push-ukqlsyplorpn branch 2 times, most recently from 7079e3d to 6cd2198 Compare July 14, 2025 15:26
@adamcstephens adamcstephens changed the title elixir-ls: Fix Elixir stdlib code detection elixir-ls: fix elixir stdlib code detection Jul 14, 2025
@adamcstephens adamcstephens marked this pull request as ready for review July 14, 2025 15:31
@adamcstephens adamcstephens requested review from a team and Munksgaard July 14, 2025 15:31
Copy link
Contributor

@happysalada happysalada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing to add

@nixpkgs-ci nixpkgs-ci bot added 12.approvals: 1 This PR was reviewed and approved by one person. 12.approved-by: package-maintainer This PR was reviewed and approved by a maintainer listed in any of the changed packages. labels Jul 14, 2025
@Munksgaard
Copy link
Contributor

I'm on vacation this week, so I likely won't have a chance to review. If it works I'm happy 😊

@adamcstephens adamcstephens changed the title elixir-ls: fix elixir stdlib code detection elixir-ls: fix elixir stdlib code detection, elixir: fix deterministic builds and 1.17 max OTP Jul 15, 2025
@adamcstephens
Copy link
Contributor Author

adamcstephens commented Jul 15, 2025

It appears something changed in the most recent staging merge to master, and now some mixRelease packages are failing. e.g. next-ls, livebook and probably the others.

nixpkgs-review result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 423588
Commit: ffd70542a12b1d01bd103adeec3946dd0e6c3a79


aarch64-linux

❌ 10 packages failed to build:
  • firezone-server (firezone-server-domain)
  • firezone-server-api
  • firezone-server-web
  • livebook
  • next-ls
  • plausible
  • rabbitmq-server
  • rabbitmq-server.doc
  • rabbitmq-server.man
  • sonic-pi
✅ 47 packages built:
  • akkoma
  • beam26Packages.elixir (beam26Packages.elixir_1_18)
  • beam26Packages.elixir-ls
  • beam26Packages.elixir_1_14
  • beam26Packages.elixir_1_15
  • beam26Packages.elixir_1_16
  • beam26Packages.elixir_1_17
  • beam26Packages.elixir_1_19
  • beam26Packages.ex_doc
  • beam26Packages.hex
  • elixir (beam27Packages.elixir, elixir_1_18)
  • elixir-ls (beam27Packages.elixir-ls)
  • elixir_1_17 (beam27Packages.elixir_1_17)
  • elixir_1_19 (beam27Packages.elixir_1_19)
  • ex_doc (beam27Packages.ex_doc)
  • beam27Packages.hex
  • beam28Packages.elixir (beam28Packages.elixir_1_18)
  • beam28Packages.elixir-ls
  • beam28Packages.elixir_1_19
  • beam28Packages.ex_doc
  • beam28Packages.hex
  • beamMinimal26Packages.elixir (beamMinimal26Packages.elixir_1_18)
  • beamMinimal26Packages.elixir-ls
  • beamMinimal26Packages.elixir_1_14
  • beamMinimal26Packages.elixir_1_15
  • beamMinimal26Packages.elixir_1_16
  • beamMinimal26Packages.elixir_1_17
  • beamMinimal26Packages.elixir_1_19
  • beamMinimal26Packages.ex_doc
  • beamMinimal26Packages.hex
  • beamMinimal27Packages.elixir (beamMinimal27Packages.elixir_1_18)
  • beamMinimal27Packages.elixir-ls
  • beamMinimal27Packages.elixir_1_17
  • beamMinimal27Packages.elixir_1_19
  • beamMinimal27Packages.ex_doc
  • beamMinimal27Packages.hex
  • beamMinimal28Packages.elixir (beamMinimal28Packages.elixir_1_18)
  • beamMinimal28Packages.elixir-ls
  • beamMinimal28Packages.elixir_1_19
  • beamMinimal28Packages.ex_doc
  • beamMinimal28Packages.hex
  • lexical
  • mix2nix
  • mobilizon
  • pinchflat
  • pleroma
  • protoc-gen-elixir

Error logs: `aarch64-linux`
firezone-server
Running phase: unpackPhase
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking source archive /nix/store/0dmj2gdpv4fdr3a09kjr8k0lilbhplaw-source/elixir
source root is elixir
Running phase: patchPhase
@nix { "action": "setPhase", "phase": "patchPhase" }
Running phase: updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
Running phase: configurePhase
@nix { "action": "setPhase", "phase": "configurePhase" }
==> opentelemetry_semantic_conventions
Compiling 2 files (.erl)
Compiling 79 files (.ex)
Generated opentelemetry_semantic_conventions app
escript: Not an archive file
** (Mix) Could not compile dependency :otel_http, "/nix/store/972dfzqpyq29j418g3rs2qcsxrg7ajlj-rebar3-3.25.0/bin/rebar3 bare compile --paths /build/elixir/_build/prod/lib/*/ebin" command failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile otel_http --force", update it with "mix deps.update otel_http" or clean it with "mix deps.clean otel_http"
livebook
Running phase: configurePhase
@nix { "action": "setPhase", "phase": "configurePhase" }
==> earmark_parser
Compiling 2 files (.xrl)
Compiling 1 file (.yrl)
Compiling 3 files (.erl)
Compiling 47 files (.ex)
Generated earmark_parser app
==> pluggable
Compiling 4 files (.ex)
Generated pluggable app
==> decimal
Compiling 4 files (.ex)
Generated decimal app
==> mime
Compiling 1 file (.ex)
Generated mime app
==> livebook
escript: Not an archive file
** (Mix) Could not compile dependency :iso8601, "/nix/store/972dfzqpyq29j418g3rs2qcsxrg7ajlj-rebar3-3.25.0/bin/rebar3 bare compile --paths /build/source/_build/prod/lib/*/ebin" command failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile iso8601 --force", update it with "mix deps.update iso8601" or clean it with "mix deps.clean iso8601"
next-ls
==> spitfire
Compiling 2 files (.erl)
Compiling 3 files (.ex)
      warning: this clause of defp get_newlines/2 is never used
      │
 2581 │   defp get_newlines(parser, :current) do
      │        ~
      │
      └─ lib/spitfire.ex:2581:8: Spitfire.get_newlines/2

Generated spitfire app
==> mime
Compiling 1 file (.ex)
Generated mime app
==> nimble_options
Compiling 3 files (.ex)
Generated nimble_options app
==> next_ls
escript: Not an archive file
** (Mix) Could not compile dependency :gproc, "/nix/store/972dfzqpyq29j418g3rs2qcsxrg7ajlj-rebar3-3.25.0/bin/rebar3 bare compile --paths /build/source/_build/prod/lib/*/ebin" command failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile gproc --force", update it with "mix deps.update gproc" or clean it with "mix deps.clean gproc"

plausible
Generated mime app
==> x509
Compiling 25 files (.ex)
    warning: Jason.decode!/1 is undefined (module Jason is not available or is yet to be defined)
    │
  7 │     |> Jason.decode!()
    │              ~
    │
    └─ lib/x509/path_validation_test.ex:7:14: X509.PathValidationTest.all_scenarios/0

Generated x509 app
==> nimble_options
Compiling 3 files (.ex)
Generated nimble_options app
==> nimble_parsec
Compiling 4 files (.ex)
Generated nimble_parsec app
escript: Not an archive file
==> plausible
** (Mix) Could not compile dependency :gproc, "/nix/store/972dfzqpyq29j418g3rs2qcsxrg7ajlj-rebar3-3.25.0/bin/rebar3 bare compile --paths /build/source/_build/ce/lib/*/ebin" command failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile gproc --force", update it with "mix deps.update gproc" or clean it with "mix deps.clean gproc"

rabbitmq-server
/nix/store/flq84az61mazbgpxrlyhfwidwg07y3xp-rabbitmq-server-4.0.9/bin/rabbitmq-diagnostics: interpreter directive changed from "#!/bin/sh" to "/nix/store/fkxvlai6grfi6nngzgp0vahi235nhkq3-bash-5.2p37/bin/sh"
/nix/store/flq84az61mazbgpxrlyhfwidwg07y3xp-rabbitmq-server-4.0.9/bin/rabbitmq-env: interpreter directive changed from "#!/bin/sh -e" to "/nix/store/fkxvlai6grfi6nngzgp0vahi235nhkq3-bash-5.2p37/bin/sh -e"
/nix/store/flq84az61mazbgpxrlyhfwidwg07y3xp-rabbitmq-server-4.0.9/bin/rabbitmq-plugins: interpreter directive changed from "#!/bin/sh" to "/nix/store/fkxvlai6grfi6nngzgp0vahi235nhkq3-bash-5.2p37/bin/sh"
/nix/store/flq84az61mazbgpxrlyhfwidwg07y3xp-rabbitmq-server-4.0.9/bin/rabbitmq-queues: interpreter directive changed from "#!/bin/sh" to "/nix/store/fkxvlai6grfi6nngzgp0vahi235nhkq3-bash-5.2p37/bin/sh"
/nix/store/flq84az61mazbgpxrlyhfwidwg07y3xp-rabbitmq-server-4.0.9/bin/rabbitmq-server: interpreter directive changed from "#!/bin/sh" to "/nix/store/fkxvlai6grfi6nngzgp0vahi235nhkq3-bash-5.2p37/bin/sh"
/nix/store/flq84az61mazbgpxrlyhfwidwg07y3xp-rabbitmq-server-4.0.9/bin/rabbitmq-streams: interpreter directive changed from "#!/bin/sh" to "/nix/store/fkxvlai6grfi6nngzgp0vahi235nhkq3-bash-5.2p37/bin/sh"
/nix/store/flq84az61mazbgpxrlyhfwidwg07y3xp-rabbitmq-server-4.0.9/bin/rabbitmq-upgrade: interpreter directive changed from "#!/bin/sh" to "/nix/store/fkxvlai6grfi6nngzgp0vahi235nhkq3-bash-5.2p37/bin/sh"
/nix/store/flq84az61mazbgpxrlyhfwidwg07y3xp-rabbitmq-server-4.0.9/bin/rabbitmqctl: interpreter directive changed from "#!/bin/sh" to "/nix/store/fkxvlai6grfi6nngzgp0vahi235nhkq3-bash-5.2p37/bin/sh"
/nix/store/flq84az61mazbgpxrlyhfwidwg07y3xp-rabbitmq-server-4.0.9/bin/vmware-rabbitmq: interpreter directive changed from "#!/bin/sh" to "/nix/store/fkxvlai6grfi6nngzgp0vahi235nhkq3-bash-5.2p37/bin/sh"
stripping (with command strip and flags -S -p) in  /nix/store/flq84az61mazbgpxrlyhfwidwg07y3xp-rabbitmq-server-4.0.9/bin /nix/store/flq84az61mazbgpxrlyhfwidwg07y3xp-rabbitmq-server-4.0.9/sbin
shrinking RPATHs of ELF executables and libraries in /nix/store/s9qspvl73jc3gfxm5gy9hr19rihc6lv8-rabbitmq-server-4.0.9-man
checking for references to /build/ in /nix/store/s9qspvl73jc3gfxm5gy9hr19rihc6lv8-rabbitmq-server-4.0.9-man...
gzipping man pages under /nix/store/s9qspvl73jc3gfxm5gy9hr19rihc6lv8-rabbitmq-server-4.0.9-man/share/man/
patching script interpreter paths in /nix/store/s9qspvl73jc3gfxm5gy9hr19rihc6lv8-rabbitmq-server-4.0.9-man
shrinking RPATHs of ELF executables and libraries in /nix/store/daggak5as1qwa1vpikr7g62pwhqfb92h-rabbitmq-server-4.0.9-doc
checking for references to /build/ in /nix/store/daggak5as1qwa1vpikr7g62pwhqfb92h-rabbitmq-server-4.0.9-doc...
patching script interpreter paths in /nix/store/daggak5as1qwa1vpikr7g62pwhqfb92h-rabbitmq-server-4.0.9-doc
Running phase: installCheckPhase
@nix { "action": "setPhase", "phase": "installCheckPhase" }
escript: Not an archive file
sonic-pi
-- Installing: /build/source/app/server/beam/tau/priv/libsp_link.so
[ 99%] Completed 'sp_link'
[ 99%] Built target sp_link
[ 99%] Linking CXX executable sonic-pi
[100%] Built target sonic-pi
Compiling Erlang/Elixir files...
warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8. Please ensure your locale is set to UTF-8 (which can be verified by running "locale" in your shell) or set the ELIXIR_ERL_OPTIONS="+fnu" environment variable
warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8. Please ensure your locale is set to UTF-8 (which can be verified by running "locale" in your shell) or set the ELIXIR_ERL_OPTIONS="+fnu" environment variable
==> logger_file_backend
Compiling 1 file (.ex)
Generated logger_file_backend app
==> decimal
Compiling 4 files (.ex)
Generated decimal app
==> mime
Compiling 1 file (.ex)
Generated mime app
==> tau
escript: Not an archive file
** (Mix) Could not compile dependency :telemetry, "/nix/store/972dfzqpyq29j418g3rs2qcsxrg7ajlj-rebar3-3.25.0/bin/rebar3 bare compile --paths /build/source/app/server/beam/tau/_build/prod/lib/*/ebin" command failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile telemetry --force", update it with "mix deps.update telemetry" or clean it with "mix deps.clean telemetry"

@adamcstephens
Copy link
Contributor Author

adamcstephens commented Jul 15, 2025

Ok, tracked down the failures to a regression in shebang patching. The fix for this is in staging already, so many erlang/elixir packages are broken until that gets to master.

@amadejkastelic amadejkastelic mentioned this pull request Jul 16, 2025
3 tasks
@adamcstephens adamcstephens merged commit ef3cbae into NixOS:master Jul 17, 2025
27 of 28 checks passed
@adamcstephens adamcstephens deleted the push-ukqlsyplorpn branch July 17, 2025 17:35
@nixpkgs-ci
Copy link
Contributor

nixpkgs-ci bot commented Jul 18, 2025

@PedroHLC
Copy link
Member

PedroHLC commented Jul 22, 2025

I'm pretty sure the patch is being applied (I copied the ./apps folder to $out through postInstall in an overrideAttrs and inspected it), yet:

{"jsonrpc":"2.0","method":"window/logMessage","params":{"message":"** (FunctionClauseError) no function clause matching in IO.chardata_to_string/1    \n    \n    The following arguments were given to IO.chardata_to_string/1:\n    \n        # 1\n        nil\n    \n    Attempted function clauses (showing 2 out of 2):\n    \n        def chardata_to_string(+string+) when -is_binary(string)-\n        def chardata_to_string(+list+) when -is_list(list)-\n    \n    (elixir 1.14.5) lib/io.ex:670: IO.chardata_to_string/1\n    (elixir 1.14.5) lib/file.ex:219: File.exists?/2\n    (language_server 0.28.1) lib/language_server.ex:146: ElixirLS.LanguageServer.check_elixir_sources/0\n    (language_server 0.28.1) lib/language_server.ex:63: ElixirLS.LanguageServer.main/0\n    (elixir 1.14.5) lib/code.ex:1260: Code.require_file/2","type":2}}

Is someone else observing the same? (Nixpkgs 2baf8e1658cba84a032c3a8befb1e7b06629242a)

# Mix.install prints to stdout and reads from stdin
# we need to make sure it doesn't interfere with LSP/DAP
-echo "" | elixir "$SCRIPTPATH/quiet_install.exs" >/dev/null || exit 1
+echo "" | @elixir@/bin/elixir "$SCRIPTPATH/quiet_install.exs" >/dev/null || exit 1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, this line is weird.

By default, it will use hex to fetch the source again, and run with that instead of anything from Nixpkgs' package.

With ELS_LOCAL=1 it will run the interpreter in the project in $store/libexec/../, which will fail.

What exactly are we attempting to do here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This package needs more changes on top of what is in this PR. This was a first pass at cleanup, but this package is indeed weird in that it's not really a package but a set of wrapper scripts. Ideally we properly package it and set ELS_LOCAL so it uses code from the store instead of downloading. Until then I wouldn't expect setting that env var to work.

I've used these changes without issues though. Do you have a reproducer?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've used these changes without issues though. Do you have a reproducer?

I'm more curious what impurity made it work for you, since there is no codepath that uses the .beam built here, it always uses the one re-downloaded into the mix home path, without the 1216.patch we need.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand the problem now, but it's still not clear to me why it worked on my machine.

Anyway, can you try this PR? #427805

kivikakk added a commit to kivikakk/nixpkgs that referenced this pull request Aug 18, 2025
The motivation is similar to (and implementation taken partly from)
NixOS#271288 --- some rebar3-compiled `.beam` files contain debug information
exposing references to `.hrl` files from the Erlang distribution it
was compiled with.  This results in unnecessary store references.

See also NixOS#423588's change to Elixir's `generic-builder.nix` where the
same option is used when building to avoid the same issue.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: erlang General-purpose, concurrent, functional high-level programming language 8.has: port to stable This PR already has a backport to the stable release. 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. 11.by: package-maintainer This PR was created by a maintainer of all the package it changes. 12.approvals: 1 This PR was reviewed and approved by one person. 12.approved-by: package-maintainer This PR was reviewed and approved by a maintainer listed in any of the changed packages.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

elixir: build doesn't include source code for standard library

4 participants