beamPackages.mixRelease: deterministic erlang builds

[mattpolzin]

Description of changes

This PR sets an Erlang compiler option that makes Erlang builds "deterministic." The real upshot of this is that debug information inside compiled Erlang modules no longer contains full system paths. These system paths would sometimes refer to the Erlang Nix package making them one of the trickier reasons why Erlang was being kept around as a runtime dependency for built mix releases.

The new option can be disabled with erlangDeterministicBuilds = false.

As a side effect of adding the deterministic Erlang option, I exposed an optional attr so that it is easy to specify any additional options without worrying about conflict with the deterministic option.

In addition to removing those references previously located inside beam files, this PR reintroduces logic previously in place to clean Erlang references from other non-beam files. As far as I understand, removing the beam references was the biggest remaining unknown (ref. https://github.com/plastic-gun/nix-mix-release-unwanted-references). Speaking of which, I would be very grateful for eyes on this from @c4710n who worked on the issue of mistaken Erlang references most recently.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

Priorities

Add a 👍 reaction to pull requests you find important.

[c4710n]

Hey @mattpolzin, I just tested your code in the branch test-beamPackages-mixRelease-deterministic of plastic-gun/nix-mix-release-unwanted-references repo.

It works for all the .beam files built by mixRelease. This is great!

But it does not work for Elixir itself, we should also build Elixir in the deterministic way.

[mattpolzin]

Ah, interesting. If this PR merges, I'd be happy to look into a similar change for the Elixir build since I don't imagine that will be any more complicated.

[c4710n]

Hey, @mattpolzin . I have created a PR for deterministic Elixir builds.

After that PR is merged, we should do something on this PR:

1. add the logic for result/erts-13.2.2.4/bin/start back.
2. remove outdated comments at the footer of source code.

Although it's a bit late, I really appreciate you mentioning the 'deterministic' option. It's like a light in the darkness.

[mattpolzin]

After that PR is merged, we should do something on this PR…

I was hoping to keep this PR very targeted and follow it with another PR to do any remaining work to clean up Erlang references. I’m ok either way in the end.

[mattpolzin]

@NobbZ you've taken a crack at this before as well I believe. Pinging in case you've got any experience with the deterministic option that makes this change a bad idea.

[c4710n]

Hi, @mattpolzin. The PR of elixir deterministic build has been merged. Can you rebase your branch on master branch of nixpkgs?

Then, I can:

• contribute todos mentioned by beamPackages.mixRelease: deterministic erlang builds #271288 (comment) on your branch.
• create a test repo which targets on your branch. This will help the maintainers to review this PR.

Then, you can merge my contributes into your branch. And the code will appear on this PR.

[mattpolzin]

@c4710n sure thing. rebased.

[NobbZ]

Given that we had problems in the past with software that relied on certain stack information available, I am not a friend of making this an unalterable default.

We need to check whether all currently packaged BEAM applications can actually be built and run using this option.

Based on that we should choose a default that then can be toggled.

[mattpolzin]

@NobbZ i can throw in a Boolean toggle for this option. I'm guessing based on your comment that you would lean toward defaulting that off so that we don't break any existing packages rather than on so that any broken packages need to be updated to override the toggle, but want to confirm that with you.

[happysalada]

Ive read his comment as

• we should have a boolean flag that controls it
• we can default to true, but we need to make sure all the erlang applications actually work.

I agree with both of these statements.
What if we default to true on unstable for a bit to get some feedback from people on unstable ? Its been really hard to gather feedback on erlang elixir packages. Im not sure who uses what at this point. So in the end trying to test everything ourselves might be difficult.
My rationale is that people on unstable know that it is unstable and if we do this now, we might have enough time and information to make a decision by the next stable release .

[mattpolzin]

I've merged in changes from @c4710n, who wanted to go ahead and wrap up the remaining work to remove erlang references given that remaining work was actually just reintroducing some code that was recently removed and adopting the new Erlang Compiler option in the buildMix helper as well.

They asked me to post the following after merging their changes:

---

This PR is trying to add funtionality for:

1. removing erlang references from multiple places:
   • mixRelease output
   • buildMix output
2. changing some comment lines as echo calls, which are useful when checking the build logs via nix log

---

We can take plausible as an example:

Before this PR:

$ nix build .#legacyPackages.x86_64-darwin.plausible

$ rg "/nix/store/.*/erlang" ./result --files-with-matches --binary
./result/lib/gettext-0.19.1/ebin/gettext_po_parser.beam
./result/lib/ex_cldr-2.36.0/ebin/plural_rules_lexer.beam
./result/lib/ex_cldr-2.36.0/ebin/plural_rules_parser.beam
./result/lib/ex_cldr_numbers-2.30.1/ebin/decimal_formats_lexer.beam
./result/lib/ex_cldr_numbers-2.30.1/ebin/rbnf_parser.beam
./result/lib/ex_cldr_numbers-2.30.1/ebin/rbnf_lexer.beam
./result/lib/ex_cldr_numbers-2.30.1/ebin/decimal_formats_parser.beam
./result/erts-13.2.2.4/bin/start
./result/lib/gen_smtp-1.2.0/ebin/smtp_rfc822_parse.beam

$ nix-store --query --requisites ./result
/nix/store/24shds1bq2gdm6m9i9n3qb2pi0kxln4j-pcre2-10.42
/nix/store/ig962y9d643hwwx9xqhdadc1lll8gvig-zlib-1.3
/nix/store/2sccymh6w6i6cvbg6bpk2yy71df7cx56-libpng-apng-1.6.40
/nix/store/4j38m9c97qzf7zhamm7i7gyrgbikcfhn-libiconv-50
/nix/store/pka340xbqzk4r46si0hjkadk00c69a18-libcxxabi-16.0.6
/nix/store/x4s0wi6d2z38wa0jb4yf769sl1c0c50z-libcxx-16.0.6
/nix/store/3gk0qbrp8ykyais7vh5l9ybdc21grpz5-gettext-0.21.1
/nix/store/5k6m1scwqdagb3ga1j4c9bwypcdwig1z-apple-framework-CoreBluetooth
/nix/store/5pc1dz9049nnrkxflr6n7aapszb2cmg6-apple-framework-IOBluetooth
/nix/store/pq9jk1hijyxl0vxngxq2qqwyx371yzym-apple-framework-SecurityFoundation
/nix/store/jw4g9a1g7radz49aqfrsjzr1bgkkjh7q-apple-framework-CoreWLAN
/nix/store/gfbznf4avqs1b8ibiham8kjfkqcj89rp-apple-framework-Accelerate
/nix/store/mkk3r6fsk6rci1q2vi8jlr1cih90zpyn-apple-lib-xpc
/nix/store/rc3jiihyhqild41fqch5sp2vywrk4q15-apple-framework-IOSurface
/nix/store/yxxsb6w1bcz81pd5p9j8f1nv8p5qgj12-apple-framework-SystemConfiguration
/nix/store/daj699si9b9cl878m00qyb6cpnbhlfbl-apple-framework-CoreGraphics
/nix/store/48520h73b51q05693g7wr0dp2h4qc0nc-apple-framework-ImageIO
/nix/store/6vjmi9dclz7msbvs25fn6907f35kj9nv-apple-framework-CoreFoundation
/nix/store/91fx0vm45w9bnq23jx3yhvvyhifi5y9h-apple-framework-DiskArbitration
/nix/store/ak03f5m80wflrc5dq8l2a8fv4nwa5j8n-apple-framework-CFNetwork
/nix/store/bh6q3ns50ja9h85fip1s7k9myhw10ljy-apple-framework-CoreAudio
/nix/store/c2naq83q66mr1n78hfaq5y8s79ggd8qx-apple-framework-OpenDirectory
/nix/store/lisf7jnwhps2lkys314mm23j8spf1iyf-apple-framework-CoreData
/nix/store/q12ypkrl9zdy844qj0fpi5p15xgzinv9-apple-framework-NetFS
/nix/store/lirsb8mk6547hp5p393njnyjaqwq555h-apple-framework-CoreServices
/nix/store/pa4fdvm1h1jxgzv4mf4msibbl98s7b21-apple-framework-CoreText
/nix/store/920kn7zscj2n74jv34z834fnmcrvrsf5-apple-framework-ApplicationServices
/nix/store/h0hh79siz4r7d2xiz8jiffrhhc5w55x2-apple-framework-OpenGL
/nix/store/4rihd0aclri5a857d2vkiy6rq0nq0hyv-apple-framework-CoreVideo
/nix/store/5abl3bvwhd404y8v2x8dxkq794szjsgg-apple-framework-Metal
/nix/store/7irx6kipzwz3a2052dbmdbsghi3qvirw-gmp-with-cxx-6.3.0
/nix/store/sn50gf3jdq1v2cxcznqpcdgvy7py42jm-objc4-709.1
/nix/store/9jvz3ah7s4rx59agsh67hbzlp0sr9mw2-apple-framework-Foundation
/nix/store/g74b4h5ggcba1kxvdcyhclssjqxy1hgr-apple-framework-OpenCL
/nix/store/mq9djrx461w8dnz5wbwmzrbjfn0vw97r-apple-framework-CoreImage
/nix/store/xyw10whvf35m2fb7wwz7yk3hrxjr387s-apple-framework-QuartzCore
/nix/store/kkbsd4wk9aqkhms3j8q6k2pa4m6i8bwj-apple-framework-Carbon
/nix/store/icdl8wvmn5x8aak5b675qnk93qcjxhwd-apple-framework-AGL
/nix/store/kf9r7xvbkir07d444sqm0wi65333awhi-expat-2.5.0
/nix/store/md5h0zmmrf80bdhmmfm6wj0abprqghj6-pcre2-10.42
/nix/store/iyv0safi1jwp685y4yvrcfjhg0745a0z-libdeflate-1.19
/nix/store/r195pxi80cy1nn56vfyr4zcbblbyk9n1-xz-5.4.4
/nix/store/xdcb3yziwgs3laiv2n086ba98w20all9-libtiff-4.6.0
/nix/store/85mmabv77ix7ig8nqlvwcv5whlm2gm15-wxwidgets-3.2.4
/nix/store/913xh1h9swbd4b7lhnl0501mx3f8ikn4-ncurses-6.4
/nix/store/f27mhm2kpk37mpj88f77j0y2ncgb95a7-gawk-5.2.2
/nix/store/i7ksfq6yclpykcisl39ysyqv540l59hd-libcxx-16.0.6-dev
/nix/store/lpzfs8kvdf7fs2j2fbc8a0r2zjjlygna-gnused-4.9
/nix/store/n5k43in732h89ay4wzfdz0nqss49ji7x-openssl-3.0.12
/nix/store/8f67d698c75p5sk53s5mp2z4skkdqyjq-erlang-25.3.2.7            <- erlang found
/nix/store/ccnn45f8fjs7dcj53jgw8jmaa2dq9nrm-gnugrep-3.11
/nix/store/vd0b59wk8x56i8kmwda8jl5h37wnixn2-coreutils-9.3
/nix/store/avhh0vj2wgksyp9zwj695ayjkchsngqp-plausible-2.0.0

After this PR:

$ nix build .#legacyPackages.x86_64-darwin.plausible

$ rg "/nix/store/.*/erlang" ./result --files-with-matches --binary
# nothing

$ nix-store --query --requisites ./result
/nix/store/0ly5lpwanagc5qpmnqrw8aqxb4ivw7kf-bash-5.2-p15
/nix/store/24shds1bq2gdm6m9i9n3qb2pi0kxln4j-pcre2-10.42
/nix/store/4j38m9c97qzf7zhamm7i7gyrgbikcfhn-libiconv-50
/nix/store/pka340xbqzk4r46si0hjkadk00c69a18-libcxxabi-16.0.6
/nix/store/x4s0wi6d2z38wa0jb4yf769sl1c0c50z-libcxx-16.0.6
/nix/store/3gk0qbrp8ykyais7vh5l9ybdc21grpz5-gettext-0.21.1
/nix/store/7irx6kipzwz3a2052dbmdbsghi3qvirw-gmp-with-cxx-6.3.0
/nix/store/913xh1h9swbd4b7lhnl0501mx3f8ikn4-ncurses-6.4
/nix/store/ccnn45f8fjs7dcj53jgw8jmaa2dq9nrm-gnugrep-3.11
/nix/store/f27mhm2kpk37mpj88f77j0y2ncgb95a7-gawk-5.2.2
/nix/store/ig962y9d643hwwx9xqhdadc1lll8gvig-zlib-1.3
/nix/store/lpzfs8kvdf7fs2j2fbc8a0r2zjjlygna-gnused-4.9
/nix/store/n5k43in732h89ay4wzfdz0nqss49ji7x-openssl-3.0.12
/nix/store/vd0b59wk8x56i8kmwda8jl5h37wnixn2-coreutils-9.3
/nix/store/abr143mafih2l81q4zqbpdx7kcyl6f4j-plausible-2.0.0
# no erlang found

[NobbZ]

Yeah, basically what happysalada wrote.

We should perhaps also ping maintainers of affected packages to let them briefly check and opt for an override early if they know that there will be a breakage, such that they have time to fix.

When I completely disabled debugsymbols by default it took about 3 months until the broken packages have been recognized and the root cause identified. Then stripping got disabled by default and needs to be enabled by anyone who actually needs it, eg. to create a release in a minified OCI container.

This time we should work closer with affected maintainers before simply changing things.

[mattpolzin]

I'll have a change pushed that adds that default-on toggle to both mixRelase and now (as of the work merged by Zeke) buildMix as soon as I've tested the change.

Then I'll look into who I should ping as package maintainers.

[mattpolzin]

Here's a heads up to at least one maintainer from each of the following packages that use either mixRelease or buildMix. The full context of the description of this PR is useful, but in short, I want you to know that both mixRelease and buildMix will have a new erlangDeterministicBuilds option which will default to true. When true, the ERL_COMPILER_OPTIONS environment variable will contain "deterministic" which will cause the Erlang compiler to produce partial paths in debug statements instead of full paths. It also has other less explicit effects to ensure two builds of the same package at the same changeset will always be the same (i.e. "deterministic").

Hopefully this does not have a negative impact on any existing Nixpkgs projects, but please let me know if you foresee any problems with building or running your packages as a result. Any testing of this new feature against projects you maintain is greatly appreciated, but I realize that's a lot to ask depending on what else you have going on.

plausible / @softinio [mixRelease]
livebook / @Munksgaard [mixRelease]
akkoma / @illdefined [mixRelease, buildMix]
pleroma / @picnoir [mixRelease, buildMix]
mobilizon / @minijackson [mixRelease, buildMix]
elixir-ls / @ankhers [mixRelease]

[Munksgaard]

Exciting work @mattpolzin! Everything seems to be in order with the livebook build. @happysalada perhaps you'd like to chime in whether you think this will affect the livebook service?

Edit: Oops, you're already here @happysalada! I meant to tag @scvalex instead. Sorry!

[happysalada]

before I forget,
This is a really high quality PR, thank you for taking the time to contribute it!

Thank you also to everyone who took time out of their day to comment/review !

if everyone is fine, I'd like to wait for one week (it's an unofficially considered "enough" time for people to review) before merging this.

[scvalex]

I tested the livebook binary from this branch and I ran the NixOS test for the service. I think everything's fine as far as Livebook goes.

That said, while mentions of erlang have disappeared from the ./result, it does still appear in the --query --requisites output. I think this is because the livebook derivation explicitly uses wrapProgram to add elixir to the PATH, so it's expected, but I'm mentioning it just in case somebody disagrees.

$ nix build .#legacyPackages.x86_64-linux.livebook
$ rg "/nix/store/.*/erlang" ./result --files-with-matches --binary
NOTHING
$ nix-store --query --requisites ./result | grep -E 'erlang|elixir'
/nix/store/hadmlq92mvvlwbd3wbh4pxq8n8x637zd-erlang-25.3.2.7
/nix/store/4bqk8f9ai5ycbrxkig187a0h7pixdams-elixir-1.15.7
$ git log -n1
commit 1a510904963d83ebe2615c87b9c3ac43f2469669 (HEAD -> beamPackages-mixRelease-deterministic)
Author: Mathew Polzin <matt.polzin@gmail.com>
Date:   Sun Dec 3 09:53:06 2023 -0600

    beamPackages.mixRelease: Make determinism toggleable

[mattpolzin]

I think this is because the livebook derivation explicitly uses wrapProgram to add elixir to the PATH, so it's expected [...]

Agreed, that sounds intentional and the correct result.

[DianaOlympos]

The only thing i can think of in term of potential problems is if perf/ebpf/crash have the correct data if used with +JPperf true. I do not think it should fail, but it is the one I am deeply interested in 😄

Thanks so much for this PR, really appreciating it!

[mattpolzin]

It might be incompatible with +JPperf map, though I am not sure of it. It drops absolute paths in debug tuples but leaves them there as relative paths, so it is possible this means you can no longer map all the way back to the original source file. That said, folks weren't relying on absolute debug paths without rebuilding on their own machine anyway -- I observed some of these debug paths (coming from Nixpkgs) as having absolute paths for the maintainer's computer, not my own, so downstream couldn't use that full path for anything useful regardless.

[DianaOlympos]

Make sense, thanks for the contribution!