Add git add-to-store support for Nix by matthewbauer · Pull Request #3 · obsidiansystems/nix

matthewbauer · 2020-05-28T22:25:21Z

sets up addToStore to work with both remote and local daemons.

The base branch was changed.

src/libstore/daemon.cc

src/libstore/local-store.cc

src/libstore/remote-store.cc

src/libstore/daemon.cc

src/nix-store/nix-store.cc

src/nix/add-to-store.cc

This matches what we want for blobs. Trees are still in progress - we need a way to symlink to other objects, using that to determine ca.

This is really bad and dangerous! But Git migration to sha256 is still a ways away: https://lwn.net/Articles/811068/ So we need to allow it for the time being.

We need access to other things in the store. This is kind of dangerous though if things are added in the wrong order�.

This is needed to create files based on git permissions

This updates the remote protocol to try to handle sha1 hashes.

Need to use the htSHA1 we were given, don’t recompute.

this makes maintanence easier

This reverts commit 7b6186d.

narHash is the hash of the nar, not the git objects.

We need to dump path and dump git here so that the hashes match what is expected elsewhere.

This should always use sha1 hash type, but we want to make sure the caller knows that. So just throw an error instead of ignoring hashAlgo on Git ingestion.

symlinks should be relative so that they look like: ../s5c0hnz9qfnpnn1bszfxicgz21d1fam3-dummy3 instead of /build/nix-test/store/s5c0hnz9qfnpnn1bszfxicgz21d1fam3-dummy3 This way our hashes will work with any real store dir. Note that /nix/store is still embedded in the store entry function.

this shouldn’t be needed - FdSink handles this for us.

Symlinks are resolved in the nar format so we end up with references to the real store dir. Copying avoids this and gives us a stable hash. Also update the ParseSink api with two methods: - copyFile - copyDirectory both are needed to properly implement git parsing.

libc++fs isn’t alway available

This deadlocks ProgressBar, e.g. # nix run --impure --no-substitute --store '/tmp/nix2?store=/foo' --expr 'derivation { builder = /nix/store/zi90rxslsm4mlr46l2xws1rm94g7pk8p-busybox-1.31.1-x86_64-unknown-linux-musl/bin/busybox; }' leads to Thread 1 (Thread 0x7ffff6126e80 (LWP 12250)): #0 0x00007ffff7215d62 in __lll_lock_wait () from /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/libpthread.so.0 #1 0x00007ffff720e721 in pthread_mutex_lock () from /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/libpthread.so.0 #2 0x00007ffff7ad17fa in __gthread_mutex_lock (__mutex=0x6c5448) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/x86_64-unknown-linux-gnu/bits/gthr-default.h:749 #3 std::mutex::lock (this=0x6c5448) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/bits/std_mutex.h:100 #4 std::unique_lock<std::mutex>::lock (this=0x7fffffff09a8, this=0x7fffffff09a8) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/bits/unique_lock.h:141 #5 std::unique_lock<std::mutex>::unique_lock (__m=..., this=0x7fffffff09a8) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/bits/unique_lock.h:71 #6 nix::Sync<nix::ProgressBar::State, std::mutex>::Lock::Lock (s=0x6c5448, this=0x7fffffff09a0) at src/libutil/sync.hh:45 #7 nix::Sync<nix::ProgressBar::State, std::mutex>::lock (this=0x6c5448) at src/libutil/sync.hh:85 #8 nix::ProgressBar::logEI (this=0x6c5440, ei=...) at src/libmain/progress-bar.cc:131 #9 0x00007ffff7608cfd in nix::Logger::logEI (ei=..., lvl=nix::lvlError, this=0x6c5440) at src/libutil/logging.hh:88 #10 nix::getCodeLines (errPos=...) at src/libutil/error.cc:66 #11 0x00007ffff76073f2 in nix::showErrorInfo (out=..., einfo=..., showTrace=<optimized out>) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/optional:897 #12 0x00007ffff7ad19e7 in nix::ProgressBar::logEI (this=0x6c5440, ei=...) at src/libmain/progress-bar.cc:134 #13 0x00007ffff7ab9d10 in nix::Logger::logEI (ei=..., lvl=nix::lvlError, this=0x6c5440) at src/libutil/logging.hh:88 #14 nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) (programName="/home/eelco/Dev/nix/outputs/out/bin/nix", fun=...) at src/libmain/shared.cc:328 #15 0x000000000046226b in main (argc=<optimized out>, argv=<optimized out>) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/ext/new_allocator.h:80

We now see exception beeing thrown when remote building in master because of writing to a non-blocking file descriptor from our json logger. > #0 0x00007f2ea97aea9c in __pthread_kill_implementation () from /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libc.so.6 > #1 0x00007f2ea975c576 in raise () from /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libc.so.6 > #2 0x00007f2ea9744935 in abort () from /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libc.so.6 > #3 0x00007f2ea99e8c2b in __gnu_cxx::__verbose_terminate_handler() [clone .cold] () from /nix/store/ybjcla5bhj8g1y84998pn4a2drfxybkv-gcc-13.3.0-lib/lib/libstdc++.so.6 > #4 0x00007f2ea99f820a in __cxxabiv1::__terminate(void (*)()) () from /nix/store/ybjcla5bhj8g1y84998pn4a2drfxybkv-gcc-13.3.0-lib/lib/libstdc++.so.6 > #5 0x00007f2ea99f8275 in std::terminate() () from /nix/store/ybjcla5bhj8g1y84998pn4a2drfxybkv-gcc-13.3.0-lib/lib/libstdc++.so.6 > #6 0x00007f2ea99f84c7 in __cxa_throw () from /nix/store/ybjcla5bhj8g1y84998pn4a2drfxybkv-gcc-13.3.0-lib/lib/libstdc++.so.6 > #7 0x00007f2eaa5035c2 in nix::writeFull (fd=2, s=..., allowInterrupts=true) at ../unix/file-descriptor.cc:43 > #8 0x00007f2eaa5633c4 in nix::JSONLogger::write (this=this@entry=0x249a7d40, json=...) at /nix/store/4krab2h0hd4wvxxmscxrw21pl77j4i7j-gcc-13.3.0/include/c++/13.3.0/bits/char_traits.h:358 > #9 0x00007f2eaa5658d7 in nix::JSONLogger::logEI (this=<optimized out>, ei=...) at ../logging.cc:242 > #10 0x00007f2ea9c5d048 in nix::Logger::logEI (ei=..., lvl=nix::lvlError, this=0x249a7d40) at /nix/store/a7cq5bqh0ryvnkv4m19ffchnvi8l9qx6-nix-util-2.27.0-dev/include/nix/logging.hh:108 > #11 nix::handleExceptions (programName="nix", fun=...) at ../shared.cc:343 > #12 0x0000000000465b1f in main (argc=<optimized out>, argv=<optimized out>) at /nix/store/4krab2h0hd4wvxxmscxrw21pl77j4i7j-gcc-13.3.0/include/c++/13.3.0/bits/allocator.h:163 > (gdb) frame 10 > #10 0x00007f2ea9c5d048 in nix::Logger::logEI (ei=..., lvl=nix::lvlError, this=0x249a7d40) at /nix/store/a7cq5bqh0ryvnkv4m19ffchnvi8l9qx6-nix-util-2.27.0-dev/include/nix/logging.hh:108 > 108 logEI(ei); So far only drainFD sets the non-blocking flag on a "readable" file descriptor, while this is a "writeable" file descriptor. It's not clear to me yet, why we see logs after that point, but it's also not that bad to handle EAGAIN in read/write functions after all.

This makes it behave the same as nix-daemon. Opening the store in the parent can cause a SIGBUS in libsqlite in the child: #0 0x00007f141cf6f789 in __memset_avx2_unaligned_erms () from /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libc.so.6 #1 0x00007f141c322fe8 in walIndexAppend () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0 #2 0x00007f141c3711a2 in pagerWalFrames () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0 #3 0x00007f141c38317e in sqlite3PagerCommitPhaseOne.part.0 () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0 #4 0x00007f141c383555 in sqlite3BtreeCommitPhaseOne.part.0 () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0 #5 0x00007f141c384797 in sqlite3VdbeHalt () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0 #6 0x00007f141c3b8f60 in sqlite3VdbeExec () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0 #7 0x00007f141c3bbfef in sqlite3_step () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0 #8 0x00007f141c3bd0e5 in sqlite3_exec () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0 #9 0x00007f141da140e0 in nix::SQLiteTxn::commit() () from /nix/store/1m4r8s7s1v54zq9isncvjgia02bffxlz-determinate-nix-store-3.1.0/lib/libnixstore.so #10 0x00007f141d9ce69c in nix::LocalStore::registerValidPaths(std::map<nix::StorePath, nix::ValidPathInfo, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::ValidPathInfo> > > const&)::{lambda()#1}::operator()() const () from /nix/store/1m4r8s7s1v54zq9isncvjgia02bffxlz-determinate-nix-store-3.1.0/lib/libnixstore.so

This was removed in NixOS#11152. However, we need it for the multi-threaded evaluator, because otherwise Boehm GC will crash while scanning the thread stack: #0 GC_push_all_eager (bottom=<optimized out>, top=<optimized out>) at extra/../mark.c:1488 #1 0x00007ffff74691d5 in GC_push_all_stack_sections (lo=<optimized out>, hi=<optimized out>, traced_stack_sect=0x0) at extra/../mark_rts.c:704 #2 GC_push_all_stacks () at extra/../pthread_stop_world.c:876 #3 GC_default_push_other_roots () at extra/../os_dep.c:2893 #4 0x00007ffff746235c in GC_mark_some (cold_gc_frame=0x7ffee8ecaa50 "`\304G\367\377\177") at extra/../mark.c:374 #5 0x00007ffff7465a8d in GC_stopped_mark (stop_func=stop_func@entry=0x7ffff7453c80 <GC_never_stop_func>) at extra/../alloc.c:875 #6 0x00007ffff7466724 in GC_try_to_collect_inner (stop_func=0x7ffff7453c80 <GC_never_stop_func>) at extra/../alloc.c:624 #7 0x00007ffff7466a22 in GC_collect_or_expand (needed_blocks=needed_blocks@entry=1, ignore_off_page=ignore_off_page@entry=0, retry=retry@entry=0) at extra/../alloc.c:1688 #8 0x00007ffff746878f in GC_allocobj (gran=<optimized out>, kind=<optimized out>) at extra/../alloc.c:1798 #9 GC_generic_malloc_inner (lb=<optimized out>, k=k@entry=1) at extra/../malloc.c:193 #10 0x00007ffff746cd40 in GC_generic_malloc_many (lb=<optimized out>, k=<optimized out>, result=<optimized out>) at extra/../mallocx.c:477 #11 0x00007ffff746cf35 in GC_malloc_kind (bytes=120, kind=1) at extra/../thread_local_alloc.c:187 #12 0x00007ffff796ede5 in nix::allocBytes (n=<optimized out>, n=<optimized out>) at ../src/libexpr/include/nix/expr/eval-inline.hh:19 This is because it will use the stack pointer of the coroutine, so it will scan a region of memory that doesn't exist, e.g. Stack for thread 0x7ffea4ff96c0 is [0x7ffe80197af0w,0x7ffea4ffa000) (where 0x7ffe80197af0w is the sp of the coroutine and 0x7ffea4ffa000 is the base of the thread stack). We don't scan coroutine stacks, because currently they don't have GC roots (there is no evaluation happening in coroutines). So there is currently no need to restore the other parts of the original patch, such as BoehmGCStackAllocator.

… with newer nixpkgs Firstly, this is now available on darwin where the default in llvm 19. Secondly, this leads to very weird segfaults when building with newer nixpkgs for some reason. (It's UB after all). This appears when building with the following: mesonComponentOverrides = finalAttrs: prevAttrs: { mesonBuildType = "debugoptimized"; dontStrip = true; doCheck = false; separateDebugInfo = false; preConfigure = (prevAttrs.preConfigure or "") + '' case "$mesonBuildType" in release|minsize|debugoptimized) appendToVar mesonFlags "-Db_lto=true" ;; *) appendToVar mesonFlags "-Db_lto=false" ;; esac ''; }; And with the following nixpkgs input: nix build ".#nix-cli" -L --override-input nixpkgs "https://releases.nixos.org/nixos/unstable/nixos-25.11pre870157.7df7ff7d8e00/nixexprs.tar.xz" Stacktrace: #0 0x00000000006afdc0 in ?? () #1 0x00007ffff71cebb6 in _Unwind_ForcedUnwind_Phase2 () from /nix/store/41ym1jm1b7j3rhglk82gwg9jml26z1km-gcc-14.3.0-lib/lib/libgcc_s.so.1 #2 0x00007ffff71cf5b5 in _Unwind_Resume () from /nix/store/41ym1jm1b7j3rhglk82gwg9jml26z1km-gcc-14.3.0-lib/lib/libgcc_s.so.1 #3 0x00007ffff7eac7d8 in std::basic_ios<char, std::char_traits<char> >::~basic_ios (this=<optimized out>, this=<optimized out>) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/basic_ios.h:286 #4 std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream (this=<optimized out>, this=<optimized out>) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/sstream:806 #5 nix::SimpleLogger::logEI (this=<optimized out>, ei=...) at ../logging.cc:121 #6 0x00007ffff7515794 in nix::Logger::logEI (this=0x675450, lvl=nix::lvlError, ei=...) at /nix/store/bkshji3nnxmrmgwa4n2kaxadajkwvn65-nix-util-2.32.0pre-dev/include/nix/util/logging.hh:144 #7 nix::handleExceptions (programName=..., fun=...) at ../shared.cc:336 #8 0x000000000047b76b in main (argc=<optimized out>, argv=<optimized out>) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/new_allocator.h:88

… with newer nixpkgs Firstly, this is now available on darwin where the default in llvm 19. Secondly, this leads to very weird segfaults when building with newer nixpkgs for some reason. (It's UB after all). This appears when building with the following: mesonComponentOverrides = finalAttrs: prevAttrs: { mesonBuildType = "debugoptimized"; dontStrip = true; doCheck = false; separateDebugInfo = false; preConfigure = (prevAttrs.preConfigure or "") + '' case "$mesonBuildType" in release|minsize|debugoptimized) appendToVar mesonFlags "-Db_lto=true" ;; *) appendToVar mesonFlags "-Db_lto=false" ;; esac ''; }; And with the following nixpkgs input: nix build ".#nix-cli" -L --override-input nixpkgs "https://releases.nixos.org/nixos/unstable/nixos-25.11pre870157.7df7ff7d8e00/nixexprs.tar.xz" Stacktrace: #0 0x00000000006afdc0 in ?? () #1 0x00007ffff71cebb6 in _Unwind_ForcedUnwind_Phase2 () from /nix/store/41ym1jm1b7j3rhglk82gwg9jml26z1km-gcc-14.3.0-lib/lib/libgcc_s.so.1 #2 0x00007ffff71cf5b5 in _Unwind_Resume () from /nix/store/41ym1jm1b7j3rhglk82gwg9jml26z1km-gcc-14.3.0-lib/lib/libgcc_s.so.1 #3 0x00007ffff7eac7d8 in std::basic_ios<char, std::char_traits<char> >::~basic_ios (this=<optimized out>, this=<optimized out>) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/basic_ios.h:286 #4 std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream (this=<optimized out>, this=<optimized out>) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/sstream:806 #5 nix::SimpleLogger::logEI (this=<optimized out>, ei=...) at ../logging.cc:121 #6 0x00007ffff7515794 in nix::Logger::logEI (this=0x675450, lvl=nix::lvlError, ei=...) at /nix/store/bkshji3nnxmrmgwa4n2kaxadajkwvn65-nix-util-2.32.0pre-dev/include/nix/util/logging.hh:144 #7 nix::handleExceptions (programName=..., fun=...) at ../shared.cc:336 #8 0x000000000047b76b in main (argc=<optimized out>, argv=<optimized out>) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/new_allocator.h:88 (cherry picked from commit dce1a89)

Ericson2314 previously approved these changes May 29, 2020

View reviewed changes

Ericson2314 changed the base branch from ipfs-develop to git-objects-develop May 29, 2020 03:04

Ericson2314 changed the base branch from git-objects-develop to ipfs-develop May 29, 2020 03:05