Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2022-06 spack update #949

Closed
3 tasks done
cyrush opened this issue Jun 8, 2022 · 21 comments
Closed
3 tasks done

2022-06 spack update #949

cyrush opened this issue Jun 8, 2022 · 21 comments
Labels

Comments

@cyrush
Copy link
Member

cyrush commented Jun 8, 2022

Using this issue to track all steps for to update to latest Spack for our CI builds and deployment on ECP platforms.

Changes needed to existing dependent packages on spack develop:

Missing Public Packages:

  • genten (branch we need is not yet in main repo)

Ascent Public (spack develop) Package Changes:

  • add occa, raja variants
  • add umpire dep (for occa and raja)
  • wire as HIP Package

Ascent hot-copy (uberenv) Package Changes:

  • add genten variant
  • add genten package
  • remove all hot-copy pkgs except for uberenv-ascent and genten

CI Container Config and Build Updates

ECP Public Install Config and Build Updates:

  • Summit
  • Crusher (pending VTK 1.8?)
  • Perlmutter (pending VTK 1.8)
@cyrush
Copy link
Member Author

cyrush commented Jun 10, 2022

  • Needed to update uberenv (to deal with changed to spack config hardwire)
  • All externals in packages.yaml need to be given specific versions or else, we get:

Error: Spec version is not concrete: netlib-lapack

@cyrush
Copy link
Member Author

cyrush commented Jun 11, 2022

our genten needs updates due to kokkos API changes - wait for new higher order moments branch to be merged into genten master.

@cyrush
Copy link
Member Author

cyrush commented Jun 11, 2022

  • Resolved - Transient issue with spack mirror?

I am seeing repeated fetch errors for sphinx source from spack mirrors.

#10 1341.6 ==> No binary for py-sphinx-4.4.0-v5mdrelwgt7tk2y5u5mbssukgodwx4tk found: installing from source
#10 1343.2 ==> Fetching https://mirror.spack.io/_source-cache/archive/6c/6caad9786055cb1fa22b4a365c1775816b876f91966481765d7d50e9f0dd35cc.tar.gz
#10 1343.3 ==> Error: UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 45: ordinal not in range(128)

@cyrush
Copy link
Member Author

cyrush commented Jun 15, 2022

  • Resolved - Transient issue with python servers?

I also hit consistent errors with spack fetching python (both 3.8 and 3.9)

I disabled python to get a minimal container - but we need python to generate the unit test reports we rely on in CI. So CI tests fail with this container.

@cyrush
Copy link
Member Author

cyrush commented Jun 16, 2022

Python 3.9 built with newer spack, however hit an activation error:

==> Error: Package merge blocked by file: /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/python-3.9.13-m3tvtc2qdvgffcitl3isjr2u2jnww56i/lib/python3.9/site-packages/_distutils_hack/__pycache__/__init__.cpython-39.pyc

Same error in Python 3.8.

More details:

==> Activating extension [email protected]%[email protected]+blas+lapack patches=873745d arch=linux-ubuntu18.04-zen/sbojlbm for [email protected]%[email protected]+bz2+ctypes+dbm~debug+ensurepip+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93,4c24573,f2fd060 arch=linux-ubuntu18.04-zen/trauecu
==> Activating extension [email protected]%[email protected] arch=linux-ubuntu18.04-zen/sjai4bd for [email protected]%[email protected]+bz2+ctypes+dbm~debug+ensurepip+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93,4c24573,f2fd060 arch=linux-ubuntu18.04-zen/trauecu
==> Error: Package merge blocked by file: /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/python-3.8.13-trauecunid25vkppz25nstapcbssl7am/lib/python3.8/site-packages/_distutils_hack/__pycache__/__init__.cpython-38.pyc

Both the main python install:

ls  /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/python-3.8.13-trauecunid25vkppz25nstapcbssl7am/lib/python3.8/site-
packages/_distutils_hack/

And the setup tools install:

ls /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/py-setuptools-62.3.2-sjai4bdyrc2vuthp2vuhyietb65imgv4/lib/python3.8/site-packages/
_distutils_hack  distutils-precedence.pth  pkg_resources  setuptools  setuptools-62.3.2.dist-info

contain this _distutils_hack/ files

spack activate fails b/c when spack activates setuptools, it tries to link it into the main python dir, it will notice a conflict and bail.

I tried two older versions of pysetup tools 59.4.0 49.6.0 and the folder was still there, but it could still be entangled with the python some how (I didn't do a clean python install for each try)

I am not sure if this is an issue with spack environment views.

@white238 saw this spack issue in another project.

I created a spack issue with a simple recipe to reproduce:
spack/spack#31166

@cyrush
Copy link
Member Author

cyrush commented Jun 17, 2022

  • Resolved?

Hitting an issue bootstrapping clingo on Ubunutu 22.04 with gcc.

10 244.9 ==> Error: cannot bootstrap the "clingo" Python module from spec "clingo-bootstrap@spack+python %gcc target=x86_64" due to the follow
ing failures:
#10 244.9 github-actions-v0.2 raised AttributeError: 'NoneType' object has no attribute 'add_default_arg'
#10 244.9 github-actions-v0.1 raised ValueError: source is not trusted
#10 244.9 Run `spack --debug ...` for more detailed errors
#10 244.9 ==> Bootstrapping clingo from pre-built binaries

Here we are trying
spack install zlib

root@6a1628339d8f:/# spack/bin/spack install zlib
==> Bootstrapping clingo from pre-built binaries
==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.2/build_cache/linux-centos7-x86_64-gcc-10.2.1-clingo-bootstrap-spack-2l45t4kw3cqqwj6vbxhfwhzlo6b3q2p4.spec.json
==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.2/build_cache/linux-centos7-x86_64/gcc-10.2.1/clingo-bootstrap-spack/linux-centos7-x86_64-gcc-10.2.1-clingo-bootstrap-spack-2l45t4kw3cqqwj6vbxhfwhzlo6b3q2p4.spack
==> Installing "clingo-bootstrap@spack%[email protected]~docs~ipo+python+static_libstdcpp build_type=Release arch=linux-centos7-x86_64" from a buildcache
==> Bootstrapping patchelf from pre-built binaries
==> Bootstrapping [email protected]:0.13.99 %gcc target=x86_64 from sources
==> Installing patchelf-0.13.1-dsjzmt4wboogb6lrfycmit6tdxpni7pd
==> No binary for patchelf-0.13.1-dsjzmt4wboogb6lrfycmit6tdxpni7pd found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/08/08c0237e89be74d61ddf8f6ff218439cdd62af572d568fb38913b53e222831de.tar.gz
==> No patches needed for patchelf
==> patchelf: Executing phase: 'autoreconf'
==> patchelf: Executing phase: 'configure'
==> patchelf: Executing phase: 'build'
==> patchelf: Executing phase: 'install'
==> patchelf: Successfully installed patchelf-0.13.1-dsjzmt4wboogb6lrfycmit6tdxpni7pd
  Fetch: 0.40s.  Build: 4.63s.  Total: 5.03s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/patchelf-0.13.1-dsjzmt4wboogb6lrfycmit6tdxpni7pd
==> Bootstrapping clingo from sources
==> Installing libiconv-1.16-qgiqsj3me7n7a33idnkjhfrje6gtkzfc
==> No binary for libiconv-1.16-qgiqsj3me7n7a33idnkjhfrje6gtkzfc found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/e6/e6a1b1b589654277ee790cce3734f07876ac4ccfaecbee8afa0b649cf529cc04.tar.gz
==> No patches needed for libiconv
==> libiconv: Executing phase: 'autoreconf'
==> libiconv: Executing phase: 'configure'
==> libiconv: Executing phase: 'build'
==> libiconv: Executing phase: 'install'
==> libiconv: Successfully installed libiconv-1.16-qgiqsj3me7n7a33idnkjhfrje6gtkzfc
  Fetch: 0.93s.  Build: 11.63s.  Total: 12.56s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/libiconv-1.16-qgiqsj3me7n7a33idnkjhfrje6gtkzfc
==> Installing libsigsegv-2.13-dvx6jjg4rhvuwr6mzpbecfjt6hwiju2p
==> No binary for libsigsegv-2.13-dvx6jjg4rhvuwr6mzpbecfjt6hwiju2p found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/be/be78ee4176b05f7c75ff03298d84874db90f4b6c9d5503f0da1226b3a3c48119.tar.gz
==> No patches needed for libsigsegv
==> libsigsegv: Executing phase: 'autoreconf'
==> libsigsegv: Executing phase: 'configure'
==> libsigsegv: Executing phase: 'build'
==> libsigsegv: Executing phase: 'install'
==> libsigsegv: Successfully installed libsigsegv-2.13-dvx6jjg4rhvuwr6mzpbecfjt6hwiju2p
  Fetch: 0.84s.  Build: 2.66s.  Total: 3.50s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/libsigsegv-2.13-dvx6jjg4rhvuwr6mzpbecfjt6hwiju2p
==> Installing berkeley-db-18.1.40-emdzfk4edy5a5kkudnchca5xvz6n6hs6
==> No binary for berkeley-db-18.1.40-emdzfk4edy5a5kkudnchca5xvz6n6hs6 found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/0c/0cecb2ef0c67b166de93732769abdeba0555086d51de1090df325e18ee8da9c8.tar.gz
==> Applied patch /spack/var/spack/repos/builtin/packages/berkeley-db/drop-docs.patch
==> Ran patch() for berkeley-db
==> berkeley-db: Executing phase: 'autoreconf'
==> berkeley-db: Executing phase: 'configure'
==> berkeley-db: Executing phase: 'build'
==> berkeley-db: Executing phase: 'install'
==> berkeley-db: Successfully installed berkeley-db-18.1.40-emdzfk4edy5a5kkudnchca5xvz6n6hs6
  Fetch: 3.41s.  Build: 11.90s.  Total: 15.31s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/berkeley-db-18.1.40-emdzfk4edy5a5kkudnchca5xvz6n6hs6
==> Installing pkgconf-1.8.0-wsqf3pdlaknj3vt7bq7skjmztdd5flps
==> No binary for pkgconf-1.8.0-wsqf3pdlaknj3vt7bq7skjmztdd5flps found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/ef/ef9c7e61822b7cb8356e6e9e1dca58d9556f3200d78acab35e4347e9d4c2bbaf.tar.xz
==> No patches needed for pkgconf
==> pkgconf: Executing phase: 'autoreconf'
==> pkgconf: Executing phase: 'configure'
==> pkgconf: Executing phase: 'build'
==> pkgconf: Executing phase: 'install'
==> pkgconf: Successfully installed pkgconf-1.8.0-wsqf3pdlaknj3vt7bq7skjmztdd5flps
  Fetch: 0.92s.  Build: 2.55s.  Total: 3.47s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/pkgconf-1.8.0-wsqf3pdlaknj3vt7bq7skjmztdd5flps
==> Installing zlib-1.2.12-edugrkv3cwwfvj7um4p4pcb77joh6bun
==> No binary for zlib-1.2.12-edugrkv3cwwfvj7um4p4pcb77joh6bun found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/91/91844808532e5ce316b3c010929493c0244f3d37593afd6de04f71821d5136d9.tar.gz
==> Applied patch /spack/var/spack/repos/builtin/packages/zlib/configure-cc.patch
==> zlib: Executing phase: 'install'
==> zlib: Successfully installed zlib-1.2.12-edugrkv3cwwfvj7um4p4pcb77joh6bun
  Fetch: 0.31s.  Build: 0.64s.  Total: 0.95s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/zlib-1.2.12-edugrkv3cwwfvj7um4p4pcb77joh6bun
==> Installing ca-certificates-mozilla-2022-03-29-n67mrapikyuvp7lnatgsizro6a6qdbzj
==> No binary for ca-certificates-mozilla-2022-03-29-n67mrapikyuvp7lnatgsizro6a6qdbzj found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/19/1979e7fe618c51ed1c9df43bba92f977a0d3fe7497ffa2a5e80dfc559a1e5a29
==> No patches needed for ca-certificates-mozilla
==> ca-certificates-mozilla: Executing phase: 'install'
==> ca-certificates-mozilla: Successfully installed ca-certificates-mozilla-2022-03-29-n67mrapikyuvp7lnatgsizro6a6qdbzj
  Fetch: 0.12s.  Build: 0.02s.  Total: 0.14s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/ca-certificates-mozilla-2022-03-29-n67mrapikyuvp7lnatgsizro6a6qdbzj
[+] /usr (external python-3.10-e5bbvi6anav5ecs3qlpuba5fdbhiux4z)
==> Installing re2c-2.2-oxzzls2ckny5zaui2bacjo4qqvoiee45
==> No binary for re2c-2.2-oxzzls2ckny5zaui2bacjo4qqvoiee45 found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/0f/0fc45e4130a8a555d68e230d1795de0216dfe99096b61b28e67c86dfd7d86bda.tar.xz
==> No patches needed for re2c
==> re2c: Executing phase: 'autoreconf'
==> re2c: Executing phase: 'configure'
==> re2c: Executing phase: 'build'
==> re2c: Executing phase: 'install'
==> re2c: Successfully installed re2c-2.2-oxzzls2ckny5zaui2bacjo4qqvoiee45
  Fetch: 0.32s.  Build: 8.41s.  Total: 8.73s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/re2c-2.2-oxzzls2ckny5zaui2bacjo4qqvoiee45
==> Installing diffutils-3.8-4zbnujfiapq5yeuc7xdv4bejn57w4d5z
==> No binary for diffutils-3.8-4zbnujfiapq5yeuc7xdv4bejn57w4d5z found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/a6/a6bdd7d1b31266d11c4f4de6c1b748d4607ab0231af5188fc2533d0ae2438fec.tar.xz
==> No patches needed for diffutils
==> diffutils: Executing phase: 'autoreconf'
==> diffutils: Executing phase: 'configure'
==> diffutils: Executing phase: 'build'
==> diffutils: Executing phase: 'install'
==> diffutils: Successfully installed diffutils-3.8-4zbnujfiapq5yeuc7xdv4bejn57w4d5z
  Fetch: 0.35s.  Build: 14.91s.  Total: 15.26s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/diffutils-3.8-4zbnujfiapq5yeuc7xdv4bejn57w4d5z
==> Installing m4-1.4.19-slscbtmpt5tj75r2xezhcr2cbcw3kzvx
==> No binary for m4-1.4.19-slscbtmpt5tj75r2xezhcr2cbcw3kzvx found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/3b/3be4a26d825ffdfda52a56fc43246456989a3630093cced3fbddf4771ee58a70.tar.gz
==> Applied patch /spack/var/spack/repos/builtin/packages/m4/checks-198.sysval.1.patch
==> Applied patch /spack/var/spack/repos/builtin/packages/m4/checks-198.sysval.2.patch
==> Ran patch() for m4
==> m4: Executing phase: 'autoreconf'
==> m4: Executing phase: 'configure'
==> m4: Executing phase: 'build'
==> m4: Executing phase: 'install'
==> m4: Successfully installed m4-1.4.19-slscbtmpt5tj75r2xezhcr2cbcw3kzvx
  Fetch: 0.48s.  Build: 19.60s.  Total: 20.08s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/m4-1.4.19-slscbtmpt5tj75r2xezhcr2cbcw3kzvx
==> Installing ncurses-6.2-ylxqh4yqmlbrjiexg3ej3v7vibzvedbk
==> No binary for ncurses-6.2-ylxqh4yqmlbrjiexg3ej3v7vibzvedbk found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/30/30306e0c76e0f9f1f0de987cf1c82a5c21e1ce6568b9227f7da5b71cbea86c9d.tar.gz
==> No patches needed for ncurses
==> ncurses: Executing phase: 'autoreconf'
==> ncurses: Executing phase: 'configure'
==> ncurses: Executing phase: 'build'
==> ncurses: Executing phase: 'install'
==> ncurses: Successfully installed ncurses-6.2-ylxqh4yqmlbrjiexg3ej3v7vibzvedbk
  Fetch: 0.43s.  Build: 28.46s.  Total: 28.89s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/ncurses-6.2-ylxqh4yqmlbrjiexg3ej3v7vibzvedbk
==> Installing bzip2-1.0.8-ufk3ruarnyb7d5vkutpwgozseljwzmlw
==> No binary for bzip2-1.0.8-ufk3ruarnyb7d5vkutpwgozseljwzmlw found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/ab/ab5a03176ee106d3f0fa90e381da478ddae405918153cca248e682cd0c4a2269.tar.gz
==> Ran patch() for bzip2
==> bzip2: Executing phase: 'install'
==> bzip2: Successfully installed bzip2-1.0.8-ufk3ruarnyb7d5vkutpwgozseljwzmlw
  Fetch: 0.24s.  Build: 1.37s.  Total: 1.61s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/bzip2-1.0.8-ufk3ruarnyb7d5vkutpwgozseljwzmlw
==> Installing readline-8.1-6op7wd26tpq2olfhj2fm5n33nq6yk7pu
==> No binary for readline-8.1-6op7wd26tpq2olfhj2fm5n33nq6yk7pu found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/f8/f8ceb4ee131e3232226a17f51b164afc46cd0b9e6cef344be87c65962cb82b02.tar.gz
==> Ran patch() for readline
==> readline: Executing phase: 'autoreconf'
==> readline: Executing phase: 'configure'
==> readline: Executing phase: 'build'
==> readline: Executing phase: 'install'
==> readline: Successfully installed readline-8.1-6op7wd26tpq2olfhj2fm5n33nq6yk7pu
  Fetch: 0.46s.  Build: 3.62s.  Total: 4.08s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/readline-8.1-6op7wd26tpq2olfhj2fm5n33nq6yk7pu
==> Installing gdbm-1.19-ujhnyjcyshsddopgbi4o7jjyfxnwbbwn
==> No binary for gdbm-1.19-ujhnyjcyshsddopgbi4o7jjyfxnwbbwn found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/37/37ed12214122b972e18a0d94995039e57748191939ef74115b1d41d8811364bc.tar.gz
==> No patches needed for gdbm
==> gdbm: Executing phase: 'autoreconf'
==> gdbm: Executing phase: 'configure'
==> gdbm: Executing phase: 'build'
==> gdbm: Executing phase: 'install'
==> gdbm: Successfully installed gdbm-1.19-ujhnyjcyshsddopgbi4o7jjyfxnwbbwn
  Fetch: 0.20s.  Build: 2.95s.  Total: 3.16s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/gdbm-1.19-ujhnyjcyshsddopgbi4o7jjyfxnwbbwn
==> Installing perl-5.34.1-wbl2db56q2knu45sdbx4v7zn4ju6cuue
==> No binary for perl-5.34.1-wbl2db56q2knu45sdbx4v7zn4ju6cuue found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/35/357951a491b0ba1ce3611263922feec78ccd581dddc24a446b033e25acf242a1.tar.gz
==> Fetching https://mirror.spack.io/_source-cache/archive/9d/9da50e155df72bce55cb69f51f1dbb4b62d23740fb99f6178bb27f22ebdf8a46.tar.gz
==> Moving resource stage
        source: /tmp/root/spack-stage/resource-cpanm-wbl2db56q2knu45sdbx4v7zn4ju6cuue/spack-src/
        destination: /tmp/root/spack-stage/spack-stage-perl-5.34.1-wbl2db56q2knu45sdbx4v7zn4ju6cuue/spack-src/cpanm/cpanm
==> Ran patch() for perl
==> perl: Executing phase: 'configure'
==> perl: Executing phase: 'build'
==> perl: Executing phase: 'install'
==> perl: Successfully installed perl-5.34.1-wbl2db56q2knu45sdbx4v7zn4ju6cuue
  Fetch: 2.38s.  Build: 51.42s.  Total: 53.80s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/perl-5.34.1-wbl2db56q2knu45sdbx4v7zn4ju6cuue
==> Installing bison-3.8.2-c62tseejkkakoffdrh3b7uitrlqcx5ri
==> No binary for bison-3.8.2-c62tseejkkakoffdrh3b7uitrlqcx5ri found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/06/06c9e13bdf7eb24d4ceb6b59205a4f67c2c7e7213119644430fe82fbd14a0abb.tar.gz
==> No patches needed for bison
==> bison: Executing phase: 'autoreconf'
==> bison: Executing phase: 'configure'
==> bison: Executing phase: 'build'
==> bison: Executing phase: 'install'
==> bison: Successfully installed bison-3.8.2-c62tseejkkakoffdrh3b7uitrlqcx5ri
  Fetch: 0.63s.  Build: 15.03s.  Total: 15.66s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/bison-3.8.2-c62tseejkkakoffdrh3b7uitrlqcx5ri
==> Installing openssl-1.1.1o-fbbkvo5kbkmxfkpgrjwwbuh6k3fil3th
==> No binary for openssl-1.1.1o-fbbkvo5kbkmxfkpgrjwwbuh6k3fil3th found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/93/9384a2b0570dd80358841464677115df785edb941c71211f75076d72fe6b438f.tar.gz
==> Ran patch() for openssl
==> openssl: Executing phase: 'install'
==> openssl: Successfully installed openssl-1.1.1o-fbbkvo5kbkmxfkpgrjwwbuh6k3fil3th
  Fetch: 1.01s.  Build: 11.01s.  Total: 12.02s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/openssl-1.1.1o-fbbkvo5kbkmxfkpgrjwwbuh6k3fil3th
==> Installing cmake-3.23.2-42fawrjzxxeqqtuayz4ghguomyv6bxzc
==> No binary for cmake-3.23.2-42fawrjzxxeqqtuayz4ghguomyv6bxzc found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/f3/f316b40053466f9a416adf981efda41b160ca859e97f6a484b447ea299ff26aa.tar.gz
==> No patches needed for cmake
==> cmake: Executing phase: 'bootstrap'
==> cmake: Executing phase: 'build'
==> cmake: Executing phase: 'install'
==> cmake: Successfully installed cmake-3.23.2-42fawrjzxxeqqtuayz4ghguomyv6bxzc
  Fetch: 1.10s.  Build: 1m 35.84s.  Total: 1m 36.94s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/cmake-3.23.2-42fawrjzxxeqqtuayz4ghguomyv6bxzc
==> Installing clingo-bootstrap-spack-vz27bxzu76bwbefdb33nqsbw5osbirrc
==> No binary for clingo-bootstrap-spack-vz27bxzu76bwbefdb33nqsbw5osbirrc found: installing from source
==> Ran patch() for clingo-bootstrap
==> clingo-bootstrap: Executing phase: 'cmake'
==> clingo-bootstrap: Executing phase: 'build'
==> clingo-bootstrap: Executing phase: 'install'
==> clingo-bootstrap: Successfully installed clingo-bootstrap-spack-vz27bxzu76bwbefdb33nqsbw5osbirrc
  Fetch: 7.09s.  Build: 26.89s.  Total: 33.98s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/clingo-bootstrap-spack-vz27bxzu76bwbefdb33nqsbw5osbirrc
==> Warning: Spec clingo-bootstrap@spack%[email protected]~docs~ipo+python~static_libstdcpp build_type=Release arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected]+cxx~docs+stl patches=b231fcc arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected]~debug~pic+shared arch=linux-ubuntu22.04-x86_64 ^ca-certificates-mozilla@2022-03-29%[email protected] arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected]~doc+ncurses+ownlibs~qt build_type=Release arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected] libs=shared,static arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected]+sigsegv patches=9dc5fbd,bfdffa7 arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected]~symlinks+termlib abi=none arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected]~docs~shared certs=mozilla arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected]+cpanm+shared+threads arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected]+bz2+ctypes+dbm~debug+ensurepip+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93,7d40923,f2fd060 arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected]+optimize+pic+shared patches=0d38234 arch=linux-ubuntu22.04-x86_64 did not provide module clingo
==> Warning: Spec clingo-bootstrap@spack%[email protected]~docs~ipo+python~static_libstdcpp build_type=Release arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected]+cxx~docs+stl patches=b231fcc arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected]~debug~pic+shared arch=linux-ubuntu22.04-x86_64 ^ca-certificates-mozilla@2022-03-29%[email protected] arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected]~doc+ncurses+ownlibs~qt build_type=Release arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected] libs=shared,static arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected]+sigsegv patches=9dc5fbd,bfdffa7 arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected]~symlinks+termlib abi=none arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected]~docs~shared certs=mozilla arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected]+cpanm+shared+threads arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected]+bz2+ctypes+dbm~debug+ensurepip+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93,7d40923,f2fd060 arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu22.04-x86_64 ^[email protected]%[email protected]+optimize+pic+shared patches=0d38234 arch=linux-ubuntu22.04-x86_64 did not provide module clingo
==> Error: cannot bootstrap the "clingo" Python module from spec "clingo-bootstrap@spack+python %gcc target=x86_64" due to the following failures:
github-actions-v0.2 raised AttributeError: 'NoneType' object has no attribute 'add_default_arg'
github-actions-v0.1 raised ValueError: source is not trusted
Run `spack --debug ...` for more detailed errors

@cyrush
Copy link
Member Author

cyrush commented Jun 18, 2022

  • Resolved by not using kokkos

VTK-m Cuda build is failing in ubuntu 18.04 cuda 11.4 container builds:

+] /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/conduit-0.8.3-txnuvayyylh3ujihxfwzdxc4qzvr3hwa
==> Installing vtk-m-1.7.1-i6fdr5cmeooavmahuj75bbr637ktkuwm
==> No binary for vtk-m-1.7.1-i6fdr5cmeooavmahuj75bbr637ktkuwm found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/7e/7ea3e945110b837a8c2ba49b41e45e1a1d8d0029bb472b291f7674871dbbbb63.tar.gz
==> No patches needed for vtk-m
==> vtk-m: Executing phase: 'cmake'
==> vtk-m: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j8'

The spack build process hangs before it cleans up, the hang is due to some lock in use by spack.

KeyboardInterrupt
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
    with self._rlock:
  File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()

I hunted around and found the logs while the spack process was still hung.

Kokkos build errors. I need to explicitly disable kokkos in VTK-m (it was on for another package, not VTK-m)

/tmp/tmpxft_00005b45_00000000-6_CellLocatorBoundingIntervalHierarchy.cudafe1.stub.c:69:210: error: template argument 1 is invalid
 typedef Kokkos::Impl::ParallelFor< ::,  ::Kokkos::MDRangePolicy< ::Kokkos::Cuda,  ::Kokkos::Rank<(unsigned int)3u, ( ::Kokkos::Iterate)0, ( ::Kokkos::Iterate)0> ,  ::Kokkos::IndexType<int>  > ,  ::Kokkos::Cuda>  _ZN6Kokkos4Impl11ParallelForIZN4vtkm4cont22DeviceAdapterAlgorithmINS3_22DeviceAdapterTagKokkosEE12ScheduleTaskIKNS2_7worklet16spatialstructure19CellRangesExtracterEKNS2_8internal10InvocationINSC_17FunctionInterfaceIFvNS2_4exec22ConnectivityStructuredINS2_22TopologyElementTagCellENS2_23TopologyElementTagPointELi2EEENSF_24ExecutionWholeArrayConstINS2_3VecIdLi3EEENS3_21StorageTagMultiplexerIJNS3_15StorageTagBasicENS3_13StorageTagSOAENS3_23StorageTagUniformPointsENS3_26StorageTagCartesianProductISO_SO_SO_EENS3_14StorageTagCastINSL_IfLi3EEESO_EENST_ISU_SP_EENST_ISU_SS_EEEEEJEEENSC_21ArrayPortalBasicWriteINS2_5RangeEEES12_S12_NS10_IdEES13_S13_EEENSE_IFvNS8_18WorkletMapTopologyISH_SI_E9CellSetInENS8_8internal11WorkletBase12WholeArrayInENS17_8FieldOutES1C_S1C_S1C_S1C_S1C_EEENSE_IFvNS2_12placeholders3ArgILi1EEENS17_22IncidentElementIndicesENS1G_ILi2EEENS1G_ILi3EEENS1G_ILi4EEENS1G_ILi5EEENS1G_ILi6EEENS1G_ILi7EEENS1G_ILi8EEEEEELi1ENSC_19ArrayPortalImplicitINSC_12IndexFunctorEEENS1S_INS3_8internal15ConstantFunctorIiEEEES1U_S5_EEEEvRNSF_6kokkos8internal11TaskBasic3DIT_T0_EENSL_IiLi3EEEEUliiiE_NS_13MDRangePolicyIJNS_4CudaENS_4RankILj3ELNS_7IterateE0ELS2D_0EEENS_9IndexTypeIiEEEEES2B_EE;
                                                                                                                                                                                                                  ^
In file included from tmpxft_00005b45_00000000-6_CellLocatorBoundingIntervalHierarchy.cudafe1.stub.c:1:0:
/tmp/tmpxft_00005b45_00000000-6_CellLocatorBoundingIntervalHierarchy.cudafe1.stub.c:70:210: error: template argument 1 is invalid
 typedef Kokkos::Impl::ParallelFor< ::,  ::Kokkos::MDRangePolicy< ::Kokkos::Cuda,  ::Kokkos::Rank<(unsigned int)3u, ( ::Kokkos::Iterate)0, ( ::Kokkos::Iterate)0> ,  ::Kokkos::IndexType<int>  > ,  ::Kokkos::Cuda>  _ZN6Kokkos4Impl11ParallelForIZN4vtkm4cont22DeviceAdapterAlgorithmINS3_22DeviceAdapterTagKokkosEE12ScheduleTaskIKNS2_7worklet16spatialstructure19CellRangesExtracterEKNS2_8internal10InvocationINSC_17FunctionInterfaceIFvNS2_4exec22ConnectivityStructuredINS2_22TopologyElementTagCellENS2_23TopologyElementTagPointELi3EEENSF_24ExecutionWholeArrayConstINS2_3VecIdLi3EEENS3_21StorageTagMultiplexerIJNS3_15StorageTagBasicENS3_13StorageTagSOAENS3_23StorageTagUniformPointsENS3_26StorageTagCartesianProductISO_SO_SO_EENS3_14StorageTagCastINSL_IfLi3EEESO_EENST_ISU_SP_EENST_ISU_SS_EEEEEJEEENSC_21ArrayPortalBasicWriteINS2_5RangeEEES12_S12_NS10_IdEES13_S13_EEENSE_IFvNS8_18WorkletMapTopologyISH_SI_E9CellSetInENS8_8internal11WorkletBase12WholeArrayInENS17_8FieldOutES1C_S1C_S1C_S1C_S1C_EEENSE_IFvNS2_12placeholders3ArgILi1EEENS17_22IncidentElementIndicesENS1G_ILi2EEENS1G_ILi3EEENS1G_ILi4EEENS1G_ILi5EEENS1G_ILi6EEENS1G_ILi7EEENS1G_ILi8EEEEEELi1ENSC_19ArrayPortalImplicitINSC_12IndexFunctorEEENS1S_INS3_8internal15ConstantFunctorIiEEEES1U_S5_EEEEvRNSF_6kokkos8internal11TaskBasic3DIT_T0_EENSL_IiLi3EEEEUliiiE_NS_13MDRangePolicyIJNS_4CudaENS_4RankILj3ELNS_7IterateE0ELS2D_0EEENS_9IndexTypeIiEEEEES2B_EE;
                                                                                                       

@cyrush
Copy link
Member Author

cyrush commented Jun 18, 2022

  • Resolved - Fixed in makefile example by adding umpire to custom device link step and by using ~test~utils in spack-built dray

Device Link Errors with Dray + Umpire.

  >> 559     /tmp/tmpxft_00003d1e_00000000-6_ResourceManager.cudafe1.stub.c:14: undefined reference to `__cudaRegisterLinkedBinary_50_tmpxft_00003d1e_000000
             00_7_ResourceManager_cpp1_ii_42a9a1b2'
     560     /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/umpire-6.0.0-bf6cwaevo72rqisypouvy4j2i34cszcd/lib/libumpire.a(Replay.cpp.o): In f
             unction `__sti____cudaRegisterAll()':
  >> 561     /tmp/tmpxft_00003d1d_00000000-6_Replay.cudafe1.stub.c:14: undefined reference to `__cudaRegisterLinkedBinary_41_tmpxft_00003d1d_00000000_7_Repl
             ay_cpp1_ii_5eca6429'
     562     /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/umpire-6.0.0-bf6cwaevo72rqisypouvy4j2i34cszcd/lib/libumpire.a(Umpire.cpp.o): In f
             unction `__sti____cudaRegisterAll()':
  >> 563     /tmp/tmpxft_00003d1a_00000000-6_Umpire.cudafe1.stub.c:14: undefined reference to `__cudaRegisterLinkedBinary_41_tmpxft_00003d1a_00000000_7_Umpi
             re_cpp1_ii_4507af2f'
     564     /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/umpire-6.0.0-bf6cwaevo72rqisypouvy4j2i34cszcd/lib/libumpire.a(Allocator.cpp.o): I
             n function `__sti____cudaRegisterAll()':
  >> 565     /tmp/tmpxft_00003d1c_00000000-6_Allocator.cudafe1.stub.c:14: undefined reference to `__cudaRegisterLinkedBinary_44_tmpxft_00003d1c_00000000_7_A
             llocator_cpp1_ii_a17095a1'
  >> 566     collect2: error: ld returned 1 exit status
  >> 567     tests/thirdparty/CMakeFiles/t_umpire_smoke.dir/build.make:104: recipe for target 'tests/thirdparty/t_umpire_smoke' failed
  >> 568     make[2]: *** [tests/thirdparty/t_umpire_smoke] Error 1

@cyrush
Copy link
Member Author

cyrush commented Jun 18, 2022

  • Resolved - used new mfem spec ^mfem~cuda~petsc~sundials~slepc

mfem failed to build in cuda container.

problem with petsc, which comes in directly from mfem and also sundials, new mfem spec for cuda container:

^mfem~cuda~petsc~sundials~slepc

@cyrush
Copy link
Member Author

cyrush commented Jun 19, 2022

  • Resolved - added logic to look for both old and new style umpire install layouts

Umpire config issue in cuda container:

-- Looking for Umpire in: /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/umpire-2022.03.1-6fjbta4536ffgtebjiznj5vczjv5ks7p
-- Configuring incomplete, errors occurred!
See also "/__w/1/s/build/CMakeFiles/CMakeOutput.log".
See also "/__w/1/s/build/CMakeFiles/CMakeError.log".
CMake Error at cmake/thirdparty/SetupUmpire.cmake:12 (find_package):
  Could not find a package configuration file provided by "umpire" with any
  of the following names:

    umpireConfig.cmake
    umpire-config.cmake

@white238
Copy link

Umpire config issue in cuda container:

-- Looking for Umpire in: /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/umpire-2022.03.1-6fjbta4536ffgtebjiznj5vczjv5ks7p
-- Configuring incomplete, errors occurred!
See also "/__w/1/s/build/CMakeFiles/CMakeOutput.log".
See also "/__w/1/s/build/CMakeFiles/CMakeError.log".
CMake Error at cmake/thirdparty/SetupUmpire.cmake:12 (find_package):
  Could not find a package configuration file provided by "umpire" with any
  of the following names:

    umpireConfig.cmake
    umpire-config.cmake

They got moved from share/umpire/cmake to lib/cmake/umpire. Not sure why it is only broken in your CUDA build, the following line should be broken on all builds:

find_package(umpire REQUIRED
NO_DEFAULT_PATH
PATHS ${UMPIRE_DIR}/share/umpire/cmake)

@cyrush
Copy link
Member Author

cyrush commented Jun 24, 2022

@white238 thanks this also explains an issue I had building dray release with spack.

@cyrush
Copy link
Member Author

cyrush commented Jun 24, 2022

  • Resolved - added explicit support for camp_dir

now umpire can't find camp, I might have to explicitly set camp dir? Answer: Yes

CMake Error at /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/cmake-3.23.2-ljzqnmnctewf3yne2azj7f5u26ypdtoz/share/cmake-3.23/Modules/CMakeFindDependencyMacro.cmake:47 (find_package):
  Could not find a package configuration file provided by "camp" with any of
  the following names:

    campConfig.cmake
    camp-config.cmake

  Add the installation prefix of "camp" to CMAKE_PREFIX_PATH or set
  "camp_DIR" to a directory containing one of the above files.  If "camp"
  provides a separate development package or SDK, be sure it has been
  installed.
Call Stack (most recent call first):
  /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/umpire-2022.03.1-6fjbta4536ffgtebjiznj5vczjv5ks7p/lib/cmake/umpire/umpire-config.cmake:35 (find_dependency)
  cmake/thirdparty/SetupUmpire.cmake:21 (find_package)

@cyrush
Copy link
Member Author

cyrush commented Jun 24, 2022

  • Resolved - this was a typo causing the spack config not to be passed.

Current CUDA Spack TPL install size comparisons:

Old: 3.5GB
New: 8.1GB

Issue: System cuda is not being used, spack is installing cuda 11.7 , config was not being passed to uberenv due to typo in script.

@cyrush
Copy link
Member Author

cyrush commented Jun 25, 2022

  • Resolved - By using older RAJA

Now we have RAJA BLT issue:

 ==> Installing raja-2022.03.0-nek4v4jgkkavtfvpprao4pfv5peky7hc
#10 2237.5 ==> No binary for raja-2022.03.0-nek4v4jgkkavtfvpprao4pfv5peky7hc found: installing from source
#10 2239.7 ==> No patches needed for raja
...
#10 2241.2   >> 123    /usr/bin/ld: cannot find -lcuda_runtime
#10 2241.2   >> 124    collect2: error: ld returned 1 exit status
#10 2241.2   >> 125    CMakeFiles/RAJA.dir/build.make:196: recipe for target 'lib/libRAJA.
#10 2241.2             so' failed

@cyrush
Copy link
Member Author

cyrush commented Jul 15, 2022

problem using ubernev + spack on macos:

[exe: spack/bin/spack install uberenv-ascent@develop%apple-clang]
==> Error: No valid value for variant 'ownlibs' of package 'cmake'

concertize can't figure out what to use for ownlibs for cmake.

If I try to disable ownlibs, I get another problem:

236, in parse_specs
    raise spack.error.SpackError(msg)
spack.error.SpackError: Cannot find valid provider for virtual pkgconfig

uberenv-ascent inheriting from ascent seems to be completely confusing the clingo concretization process:

==> Error: No valid value for variant 'ownlibs' of package 'cmake'
(so i add to the spec)
==> Error: No valid value for variant 'shared' of package 'uberenv-ascent'
(so i add to spec)
==> Error: No valid value for variant 'cuda' of package 'uberenv-ascent'
(so I add to sepc)
==> Error: No versions available for package 'umpire'
umpire exists and has several versions
there is lots of logic in these packages to make the right selections about this stuff, but it seem I have to set them all by hand ...

@cyrush
Copy link
Member Author

cyrush commented Jul 16, 2022

the issue is related to disabling local configs:

env  SPACK_DISABLE_LOCAL_CONFIG=1 bin/spack spec uberenv-ascent 
==> Error: No valid value for variant 'ownlibs' of package 'cmake'

when we don't disable, it works.

bin/spack spec uberenv-ascent 
Input spec
--------------------------------
uberenv-ascent

Concretized
--------------------------------
[email protected]%[email protected]~adios~adios2~babelflow+blt_find_mpi~cuda~doc~dray~fides+fortran~genten~ipo~mfem+mpi~occa~openmp+python+raja+serial+shared+test+umpire+vtkh build_type=RelWithDebInfo arch=darwin-catalina-skylake

Still working on root cause, we are not using packages.yaml in this case only a compilers.yaml in the spack default place. So SPACK_DISABLE_LOCAL_CONFIG should only block ~/.spack/

@cyrush
Copy link
Member Author

cyrush commented Aug 3, 2022

To update this thread: We hit issues with rocm package layouts vs what the spack packages expected.
These are still unresolved for us, other folks may have been working on smoothing them out.

For hip builds, we are using a script.

@cyrush
Copy link
Member Author

cyrush commented Sep 2, 2022

Since we are tracking spack develop (which was an accident on my part), I hit an issue with macOS not being able to build openmpi. Consistently fails to fetch hwloc source. spack/spack#32503

Also raja tries to build with openmp support, but we don't have in default macOS compilers -- so we have to use
^raja~openmp there is solution to this already when we use the newest raja package -- spack/spack#32284.

@cyrush
Copy link
Member Author

cyrush commented Dec 1, 2022

Related PR #1012

@cyrush
Copy link
Member Author

cyrush commented Dec 1, 2022

Spack was updated for CI part of the VTK-m 1.9 update - #1015.

We still do not have a method for using spack for hip builds, instead we use build_ascent_hip.sh` and friends.

@cyrush cyrush closed this as completed Dec 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants