{tools}[dummy/,dummy/dummy] git-annex v6.20170519, git v2.12.3, asciidoc v8.6.9, ...

[ddkinnamon]

(created using eb --new-pr)
Easyconfigs for git-annex and dependencies

[boegel]

Test report by @boegel
FAILED
Build succeeded for 3 out of 4 (4 easyconfigs in this PR)
node2420.golett.os - Linux centos linux 7.3.1611, Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz, Python 2.7.5
See https://gist.github.com/26c429bd69689daac6d06a344162d219 for a full test report.

[boegel]

@ddkinnamon As shown in the submitted test report, the installation of git-annex is failing for me (haven't looked into the details).

Also, your easyconfigs specify quite a bit of OS dependencies, which is not how we usually handle dependencies in easyconfigs that are a part of the central repository, we try to resolve as much dependencies as possible through EasyBuild instead.

[ddkinnamon]

@boegel: I'll look into the git-annex failure.

As for the OS dependencies, I should probably explain my use case. My team manages a software stack for genomic analysis and software development on RHEL HPC systems where we cannot sudo root (i.e., no package manager access). The HPC staff cannot respond to all of our software installation requests as they serve users from across the entire state, so we need to manage our "local" stack in a separate filesystem ourselves.

In this context, I would rather use OS-level compilers and libraries for tools like git and git-annex so that I can load and use them whether I am working with any one of multiple compiler toolchains. Also, the efficiency of configuring a build is vastly increased by using OS dependencies. For example, I tried building Stack on the GCC toolchain. As I got to retooling easyconfig #8 for a standard OS library included with RHEL 6 and 7, I thought that this process seemed incredibly inefficient, particularly given the fact that I would have to repeat it for every compiler toolchain that I use. Then I happened to be working on the cURL dependency and noticed the use of OS dependencies for OpenSSL (https://github.com/hpcugent/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/c/cURL/cURL-7.49.1-foss-2016b.eb), which seemed to solve my problem. As an alternative, I suppose I could use dummy/dummy easyconfigs for dependencies like zlib in lieu of OS dependencies, but that does entail an increase in overhead, especially since I have to make new easyconfigs for dependencies like libgmp. Doing so seems like reinventing the wheel for something that is already there on many recent OSes, and it is also wasteful of space on our filesystem.

Whether these easyconfigs should be in the central repo is up to you, but the use case I described is a common one in my world, so they will probably be of use to others. Having easyconfigs for this model might also facilitate adoption of EasyBuild as a stack manager by other groups like mine, and anyone who is missing a required OS dependency can always modify them fairly easily. As an example, I know that I would have to use an EasyBuild dependency for a newer version of zlib if I were building R with our RHEL 6 compilers and libraries (but not RHEL 7). All that being said, if you could give me more specifics on how you think I should split dependencies between the OS and EasyBuild, I'd be happy to consider making those changes.

[ddkinnamon]

Test report by @ddkinnamon
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in this PR)
owens-login03.hpc.osc.edu - Linux RHEL 7.3, Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz, Python 2.7.13
See https://gist.github.com/4488e233edc168f2c776fff3266fb1df for a full test report.

[ddkinnamon]

@boegel: I am unable to reproduce the git-annex build error that you saw on our RHEL 7 system (see my gist above). Based on your gist, perhaps there was some issue with the build path in /tmp getting cleaned prematurely?

== 2017-06-12 14:29:23,161 run.py:137 DEBUG run_cmd: running cmd export STACK_ROOT=/tmp/vsc40023/easybuild_build/gitannex/6.20170519/dummy-/.stack && export GIT_MERGE_AUTOEDIT=no && git clone git://git-annex.branchable.com/ /tmp/vsc40023/easybuild_build/gitannex/6.20170519/dummy- && git checkout 6.20170519 && stack config set resolver lts-8.17 &&  make -j 1 build test install BUILDER=stack GHC='stack ghc --' PREFIX=/user/home/gent/vsc400/vsc40023/eb_phanpyscratch/CO7/haswell-ib/software/git-annex/6.20170519 (in /tmp/vsc40023/easybuild_build/gitannex/6.20170519/dummy-)
== 2017-06-12 14:29:23,161 run.py:162 INFO running cmd: export STACK_ROOT=/tmp/vsc40023/easybuild_build/gitannex/6.20170519/dummy-/.stack && export GIT_MERGE_AUTOEDIT=no && git clone git://git-annex.branchable.com/ /tmp/vsc40023/easybuild_build/gitannex/6.20170519/dummy- && git checkout 6.20170519 && stack config set resolver lts-8.17 &&  make -j 1 build test install BUILDER=stack GHC='stack ghc --' PREFIX=/user/home/gent/vsc400/vsc40023/eb_phanpyscratch/CO7/haswell-ib/software/git-annex/6.20170519 
== 2017-06-12 14:29:23,261 build_log.py:156 ERROR EasyBuild crashed with an error (at ?:124 in __init__): Failed to return to /tmp/vsc40023/easybuild_build/gitannex/6.20170519/dummy- after executing command: [Errno 2] No such file or directory: '/tmp/vsc40023/easybuild_build/gitannex/6.20170519/dummy-' (at easybuild/tools/run.py:199 in run_cmd)

[migueldiascosta]

@ddkinnamon, this PR is being closed for the following reason(s): no activity for > 1 year.
Please don't hesitate to reopen this PR or add a comment if you feel this contribution is still relevant.
For more information on our policy w.r.t. closing PRs, see https://easybuild.readthedocs.io/en/latest/Contributing.html#why-a-pull-request-may-be-closed-by-a-maintainer

[boegel]

@ddkinnamon This PR was closed as a part of a cleanup operation, since we had too many open PRs piled up.

I want to apologize for not following up on this, it got snowed under, most likely due to summer holidays approaching at the time.

The approach you took here (heavily relying on OS dependencies) does not match the policy we've maintained for the vast majority of the easyconfig files we have in the central repository. OpenSSL is one exception there, we do rely on the OS package in that case because of the security aspect (and we're actually considering to revisit that policy, due to problems like #5614).

Our main motivation for "taking control" of the software stack rather than relying on OS-provided packages is to try and ensure reproducibility of installations. We've learned the hard way that minor differences in software versions that are available can result in a breaking installation, so relying on OS packages doesn't work out in practice when sharing a large set of easyconfig files with a diverse community.

That being said, you can of course maintain a set of those easyconfig files locally, and leverage them for your use case, as you explained extensively.