Skip to content

Beam packages mix release deterministic#1

Merged
mattpolzin merged 2 commits intomattpolzin:beamPackages-mixRelease-deterministicfrom
plastic-forks:beamPackages-mixRelease-deterministic
Dec 3, 2023
Merged

Beam packages mix release deterministic#1
mattpolzin merged 2 commits intomattpolzin:beamPackages-mixRelease-deterministicfrom
plastic-forks:beamPackages-mixRelease-deterministic

Conversation

@c4710n
Copy link

@c4710n c4710n commented Dec 3, 2023

@mattpolzin This is the code I mentioned.

You can also squash beamPackages.mixRelease: remove erlang references from output to your previous commit.


After you updating your branch, you can @ me again. I will post more instructions in NixOS#271288.

@mattpolzin
Copy link
Owner

Did you mean to delete the comments at the end of the file with this change? Also, if you're quite sure this takes care of the last of the references, should disallowedReferences = [ erlang ]; be uncommented?

@c4710n
Copy link
Author

c4710n commented Dec 3, 2023

Did you mean to delete the comments at the end of the file with this change?

Oh, sorry, I forgot the comment. I have changed that.

Also, if you're quite sure this takes care of the last of the references, should disallowedReferences = [ erlang ]; be uncommented?

Currently, we choose to remove erlang references by detecting ERTS is included or not. So we can't add disallowedReferences = [ erlang ] directly.

I'm not sure if there's some way to set disallowedReferences = [ erlang ] dynamically. So I remove it directly.

@mattpolzin, ping you again.

@c4710n c4710n force-pushed the beamPackages-mixRelease-deterministic branch from 9682092 to 9bb76ac Compare December 3, 2023 07:13
@c4710n
Copy link
Author

c4710n commented Dec 3, 2023

@mattpolzin I must go out now. So I can't reply you for a long time.


The following content is what I plan to post in the original PR. When you are ready, you can post it.

BEGIN OF POST

This PR is trying to add funtionality for:

  1. removing erlang references from multiple places:
    • mixRelease output
    • buildMix output
  2. changing some comment lines as echo calls, which are useful when checking the build logs via nix log

We can take plausible as an example:

Before this PR:

$ nix build .#legacyPackages.x86_64-darwin.plausible

$ rg "/nix/store/.*/erlang" ./result --files-with-matches --binary
./result/lib/gettext-0.19.1/ebin/gettext_po_parser.beam
./result/lib/ex_cldr-2.36.0/ebin/plural_rules_lexer.beam
./result/lib/ex_cldr-2.36.0/ebin/plural_rules_parser.beam
./result/lib/ex_cldr_numbers-2.30.1/ebin/decimal_formats_lexer.beam
./result/lib/ex_cldr_numbers-2.30.1/ebin/rbnf_parser.beam
./result/lib/ex_cldr_numbers-2.30.1/ebin/rbnf_lexer.beam
./result/lib/ex_cldr_numbers-2.30.1/ebin/decimal_formats_parser.beam
./result/erts-13.2.2.4/bin/start
./result/lib/gen_smtp-1.2.0/ebin/smtp_rfc822_parse.beam

$ nix-store --query --requisites ./result
/nix/store/24shds1bq2gdm6m9i9n3qb2pi0kxln4j-pcre2-10.42
/nix/store/ig962y9d643hwwx9xqhdadc1lll8gvig-zlib-1.3
/nix/store/2sccymh6w6i6cvbg6bpk2yy71df7cx56-libpng-apng-1.6.40
/nix/store/4j38m9c97qzf7zhamm7i7gyrgbikcfhn-libiconv-50
/nix/store/pka340xbqzk4r46si0hjkadk00c69a18-libcxxabi-16.0.6
/nix/store/x4s0wi6d2z38wa0jb4yf769sl1c0c50z-libcxx-16.0.6
/nix/store/3gk0qbrp8ykyais7vh5l9ybdc21grpz5-gettext-0.21.1
/nix/store/5k6m1scwqdagb3ga1j4c9bwypcdwig1z-apple-framework-CoreBluetooth
/nix/store/5pc1dz9049nnrkxflr6n7aapszb2cmg6-apple-framework-IOBluetooth
/nix/store/pq9jk1hijyxl0vxngxq2qqwyx371yzym-apple-framework-SecurityFoundation
/nix/store/jw4g9a1g7radz49aqfrsjzr1bgkkjh7q-apple-framework-CoreWLAN
/nix/store/gfbznf4avqs1b8ibiham8kjfkqcj89rp-apple-framework-Accelerate
/nix/store/mkk3r6fsk6rci1q2vi8jlr1cih90zpyn-apple-lib-xpc
/nix/store/rc3jiihyhqild41fqch5sp2vywrk4q15-apple-framework-IOSurface
/nix/store/yxxsb6w1bcz81pd5p9j8f1nv8p5qgj12-apple-framework-SystemConfiguration
/nix/store/daj699si9b9cl878m00qyb6cpnbhlfbl-apple-framework-CoreGraphics
/nix/store/48520h73b51q05693g7wr0dp2h4qc0nc-apple-framework-ImageIO
/nix/store/6vjmi9dclz7msbvs25fn6907f35kj9nv-apple-framework-CoreFoundation
/nix/store/91fx0vm45w9bnq23jx3yhvvyhifi5y9h-apple-framework-DiskArbitration
/nix/store/ak03f5m80wflrc5dq8l2a8fv4nwa5j8n-apple-framework-CFNetwork
/nix/store/bh6q3ns50ja9h85fip1s7k9myhw10ljy-apple-framework-CoreAudio
/nix/store/c2naq83q66mr1n78hfaq5y8s79ggd8qx-apple-framework-OpenDirectory
/nix/store/lisf7jnwhps2lkys314mm23j8spf1iyf-apple-framework-CoreData
/nix/store/q12ypkrl9zdy844qj0fpi5p15xgzinv9-apple-framework-NetFS
/nix/store/lirsb8mk6547hp5p393njnyjaqwq555h-apple-framework-CoreServices
/nix/store/pa4fdvm1h1jxgzv4mf4msibbl98s7b21-apple-framework-CoreText
/nix/store/920kn7zscj2n74jv34z834fnmcrvrsf5-apple-framework-ApplicationServices
/nix/store/h0hh79siz4r7d2xiz8jiffrhhc5w55x2-apple-framework-OpenGL
/nix/store/4rihd0aclri5a857d2vkiy6rq0nq0hyv-apple-framework-CoreVideo
/nix/store/5abl3bvwhd404y8v2x8dxkq794szjsgg-apple-framework-Metal
/nix/store/7irx6kipzwz3a2052dbmdbsghi3qvirw-gmp-with-cxx-6.3.0
/nix/store/sn50gf3jdq1v2cxcznqpcdgvy7py42jm-objc4-709.1
/nix/store/9jvz3ah7s4rx59agsh67hbzlp0sr9mw2-apple-framework-Foundation
/nix/store/g74b4h5ggcba1kxvdcyhclssjqxy1hgr-apple-framework-OpenCL
/nix/store/mq9djrx461w8dnz5wbwmzrbjfn0vw97r-apple-framework-CoreImage
/nix/store/xyw10whvf35m2fb7wwz7yk3hrxjr387s-apple-framework-QuartzCore
/nix/store/kkbsd4wk9aqkhms3j8q6k2pa4m6i8bwj-apple-framework-Carbon
/nix/store/icdl8wvmn5x8aak5b675qnk93qcjxhwd-apple-framework-AGL
/nix/store/kf9r7xvbkir07d444sqm0wi65333awhi-expat-2.5.0
/nix/store/md5h0zmmrf80bdhmmfm6wj0abprqghj6-pcre2-10.42
/nix/store/iyv0safi1jwp685y4yvrcfjhg0745a0z-libdeflate-1.19
/nix/store/r195pxi80cy1nn56vfyr4zcbblbyk9n1-xz-5.4.4
/nix/store/xdcb3yziwgs3laiv2n086ba98w20all9-libtiff-4.6.0
/nix/store/85mmabv77ix7ig8nqlvwcv5whlm2gm15-wxwidgets-3.2.4
/nix/store/913xh1h9swbd4b7lhnl0501mx3f8ikn4-ncurses-6.4
/nix/store/f27mhm2kpk37mpj88f77j0y2ncgb95a7-gawk-5.2.2
/nix/store/i7ksfq6yclpykcisl39ysyqv540l59hd-libcxx-16.0.6-dev
/nix/store/lpzfs8kvdf7fs2j2fbc8a0r2zjjlygna-gnused-4.9
/nix/store/n5k43in732h89ay4wzfdz0nqss49ji7x-openssl-3.0.12
/nix/store/8f67d698c75p5sk53s5mp2z4skkdqyjq-erlang-25.3.2.7            <- erlang found
/nix/store/ccnn45f8fjs7dcj53jgw8jmaa2dq9nrm-gnugrep-3.11
/nix/store/vd0b59wk8x56i8kmwda8jl5h37wnixn2-coreutils-9.3
/nix/store/avhh0vj2wgksyp9zwj695ayjkchsngqp-plausible-2.0.0

After this PR:

$ nix build .#legacyPackages.x86_64-darwin.plausible

$ rg "/nix/store/.*/erlang" ./result --files-with-matches --binary
# nothing

$ nix-store --query --requisites ./result
/nix/store/0ly5lpwanagc5qpmnqrw8aqxb4ivw7kf-bash-5.2-p15
/nix/store/24shds1bq2gdm6m9i9n3qb2pi0kxln4j-pcre2-10.42
/nix/store/4j38m9c97qzf7zhamm7i7gyrgbikcfhn-libiconv-50
/nix/store/pka340xbqzk4r46si0hjkadk00c69a18-libcxxabi-16.0.6
/nix/store/x4s0wi6d2z38wa0jb4yf769sl1c0c50z-libcxx-16.0.6
/nix/store/3gk0qbrp8ykyais7vh5l9ybdc21grpz5-gettext-0.21.1
/nix/store/7irx6kipzwz3a2052dbmdbsghi3qvirw-gmp-with-cxx-6.3.0
/nix/store/913xh1h9swbd4b7lhnl0501mx3f8ikn4-ncurses-6.4
/nix/store/ccnn45f8fjs7dcj53jgw8jmaa2dq9nrm-gnugrep-3.11
/nix/store/f27mhm2kpk37mpj88f77j0y2ncgb95a7-gawk-5.2.2
/nix/store/ig962y9d643hwwx9xqhdadc1lll8gvig-zlib-1.3
/nix/store/lpzfs8kvdf7fs2j2fbc8a0r2zjjlygna-gnused-4.9
/nix/store/n5k43in732h89ay4wzfdz0nqss49ji7x-openssl-3.0.12
/nix/store/vd0b59wk8x56i8kmwda8jl5h37wnixn2-coreutils-9.3
/nix/store/abr143mafih2l81q4zqbpdx7kcyl6f4j-plausible-2.0.0
# no erlang found

END OF POST


After you posting it, you can @ happysalada to review your code.


And, feel free to modify my code, I won't feel offended. ;)

@mattpolzin
Copy link
Owner

Bed time for me anyway. I'll get your code merged and prep the upstream PR tomorrow.

Thanks for the help.

@mattpolzin mattpolzin merged commit 3b8f7f7 into mattpolzin:beamPackages-mixRelease-deterministic Dec 3, 2023
mattpolzin pushed a commit that referenced this pull request Mar 2, 2024
Without the change `unnethack` startup crashes as:

    (gdb) bt
    #0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
    #1  0x00007f734250c0e3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
    NixOS#2  0x00007f73424bce06 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
    NixOS#3  0x00007f73424a58f5 in __GI_abort () at abort.c:79
    NixOS#4  0x00007f73424a67a1 in __libc_message (fmt=fmt@entry=0x7f734261e2f8 "*** %s ***: terminated\n") at ../sysdeps/posix/libc_fatal.c:150
    NixOS#5  0x00007f734259b1d9 in __GI___fortify_fail (msg=msg@entry=0x7f734261e2df "buffer overflow detected") at fortify_fail.c:24
    NixOS#6  0x00007f734259ab94 in __GI___chk_fail () at chk_fail.c:28
    NixOS#7  0x00000000005b2ac5 in strcpy (__src=0x7ffe68838b00 "Shall I pick a character's race, role, gender and alignment for you? [YNTQ] (y)",
        __dest=0x7ffe68838990 "\001") at /nix/store/B0S2LKF593R3585038WS4JD3LYLF2WDX-glibc-2.38-44-dev/include/bits/string_fortified.h:79
    NixOS#8  curses_break_str (str=str@entry=0x7ffe68838b00 "Shall I pick a character's race, role, gender and alignment for you? [YNTQ] (y)", width=width@entry=163,
        line_num=line_num@entry=1) at ../win/curses/cursmisc.c:275
    NixOS#9  0x00000000005b3f51 in curses_character_input_dialog (prompt=prompt@entry=0x7ffe68838cf0 "Shall I pick a character's race, role, gender and alignment for you?",
        choices=choices@entry=0x7ffe68838d70 "YNTQ", def=def@entry=121) at ../win/curses/cursdial.c:211
    NixOS#10 0x00000000005b9ca0 in curses_choose_character () at ../win/curses/cursinit.c:556
    NixOS#11 0x0000000000404eb1 in main (argc=<optimized out>, argv=<optimized out>) at ./../sys/unix/unixmain.c:309

which corresponds to `gcc` warning:

    ../win/curses/cursmisc.c: In function 'curses_break_str':
    ../win/curses/cursmisc.c:275:5: warning: '__builtin___strcpy_chk' writing one too many bytes into a region of a size that depends on 'strlen' [-Wstringop-overflow=]
      275 |     strcpy(substr, str);
          |     ^

I did not find a single small upstream change that fixes it. Let's
disable `fortify3` until next release.

Closes: NixOS#292113
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants