make bootstrap script compatible with Python 3 (WIP)

[boegel]

TODO:

• only check for setuptools in stage 0 when an older EasyBuild version is being installed (via $EASYBUILD_BOOTSTRAP_FORCE_VERSION), don't try installing it
• stop using easy_install (provided by setuptools in stage 1
• figure out EasyBuild version when installing from provided source tarballs

[boegelbot]

Travis test report: 21/21 runs failed - see https://travis-ci.org/easybuilders/easybuild-framework/builds/588327985

Only showing partial log for 1st failed test suite run 3466.1;
full log at https://travis-ci.org/easybuilders/easybuild-framework/jobs/588327986

...
>> python -m easybuild.main --version
This is EasyBuild 4.0.1.dev0 (framework: 4.0.1.dev0, easyblocks: 0.0.UNKNOWN.EASYBLOCKS) on host travis-job-f4a0b0fa-3a6f-4b3d-b484-727fe80dda7a.
travis_time:end:01685183:start=1569224537231702272,finish=1569224538650636743,duration=1418934471,event=script
�[0K�[32;1mThe command "EB_VERBOSE=1 eb --version" exited with 0.�[0m
travis_time:start:20938d7a
�[0K$ eb --check-github --github-user=easybuild_test
== temporary log file in case of crash /tmp/eb-olaeV6/easybuild-t8T4_k.log

WARNING: Robot search path is empty!


Checking status of GitHub integration...

Making sure we're online...OK

* GitHub user...easybuild_test => OK
* GitHub token...(no token found) => FAIL
* git command...OK ("git version 2.23.0; ")
* GitPython module...OK (GitPython version 1.0.2)
* push access to easybuild_test/easybuild-easyconfigs repo @ GitHub...FAIL (unexpected exception: 'Checking out branch \'master\' from git@github.com:easybuild_test/easybuild-easyconfigs.git failed: "Failed to fetch branch \'master\' from git@github.com:easybuild_test/easybuild-easyconfigs.git: \'Error when fetching: fatal: Could not read from remote repository.\\n\' returned with exit code 2"')
* creating gists...FAIL (res: None)
* location to Git working dirs... not found (suboptimal)

One or more checks FAILed, GitHub configuration not fully complete!
See http://easybuild.readthedocs.org/en/latest/Integration_with_GitHub.html#configuration for help.

Status of GitHub integration:
* --from-pr: OK
* --new-pr: not supported
* --review-pr: OK
* --update-pr: not supported
* --upload-test-report: not supported

travis_time:end:20938d7a:start=1569224538654991891,finish=1569224540158801509,duration=1503809618,event=script
�[0K�[32;1mThe command "eb --check-github --github-user=easybuild_test" exited with 0.�[0m
travis_time:start:0fc77420
�[0K$ python -O -m test.framework.suite 2>&1 | tee test_framework_suite.log
................................Skipping test_empty_pr, no GitHub token available?
..........Skipping test_from_pr, no GitHub token available?
.Skipping test_from_pr, no GitHub token available?
.Skipping test_from_pr_x, no GitHub token available?
..................Skipping test_merge_pr, no GitHub token available?
.....Skipping test_new_pr_delete, no GitHub token available?
.Skipping test_new_pr_dependencies, no GitHub token available?
.Skipping test_new_pr_python, no GitHub token available?
.Skipping test_new_update_pr, no GitHub token available?
......Skipping test_preview_pr, no GitHub token available?
....Skipping test_review_pr, no GitHub token available?
................................(skipping SvnRepository test)
.............................................................
WARNING: 'eb' not found in $PATH, failed to determine installation prefix!

................................................................................................................................/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/urllib3/util/ssl_.py:369: SNIMissingWarning: An HTTPS request has been made, but the SNI (Server Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/urllib3/util/ssl_.py:160: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
......................................................Skipping test_from_pr, no GitHub token available?
...................................................Skipping test_close_pr, no GitHub token available?
..Skipping test_download_repo, no GitHub token available?
.Skipping test_fetch_easyconfigs_from_pr, no GitHub token available?
.Skipping test_fetch_latest_commit_sha, no GitHub token available?
.Skipping test_fetch_pr_data, no GitHub token available?
.Skipping test_find_easybuild_easyconfig, no GitHub token available?
..Skipping test_install_github_token, no GitHub token available?
.Skipping test_list_prs, no GitHub token available?
.Skipping test_read, no GitHub token available?
.Skipping test_read_api, no GitHub token available?
.Skipping test_reasons_for_closing, no GitHub token available?
.Skipping test_restclient, no GitHub token available?
.Skipping test_validate_github_token, no GitHub token available?
.Skipping test_walk, no GitHub token available?
...................................................................................................................................................................GC3Pie not available, skipping test
.....................................................
WARNING: Robot search path is empty!

.
WARNING: Robot search path is empty!

.
WARNING: Robot search path is empty!

.
----------------------------------------------------------------------
Ran 640 tests in 840.511s

OK
travis_time:end:0fc77420:start=1569224540167464930,finish=1569225381576231308,duration=841408766378,event=script
�[0K�[32;1mThe command "python -O -m test.framework.suite 2>&1 | tee test_framework_suite.log" exited with 0.�[0m
travis_time:start:0da50a10
�[0K$ IGNORE_PATTERNS="vsc\.install\.shared_setup|no GitHub token available|skipping SvnRepository test|lib/python2.6/site-packages|required Lmod as modules tool|GC3Pie not available, skipping test"
travis_time:end:0da50a10:start=1569225381586158694,finish=1569225381592127294,duration=5968600,event=script
�[0K�[32;1mThe command "IGNORE_PATTERNS="vsc\.install\.shared_setup|no GitHub token available|skipping SvnRepository test|lib/python2.6/site-packages|required Lmod as modules tool|GC3Pie not available, skipping test"" exited with 0.�[0m
travis_time:start:27343a6a
�[0K$ PRINTED_MSG=$(egrep -v "${IGNORE_PATTERNS}" test_framework_suite.log | grep '\.\n*[A-Za-z]' || true)
travis_time:end:27343a6a:start=1569225381599593024,finish=1569225381611470651,duration=11877627,event=script
�[0K�[32;1mThe command "PRINTED_MSG=$(egrep -v "${IGNORE_PATTERNS}" test_framework_suite.log | grep '\.\n*[A-Za-z]' || true)" exited with 0.�[0m
travis_time:start:1fcbb868
�[0K$ test "x$PRINTED_MSG" = "x" || (echo "Found printed messages in output of test suite\n${PRINTED_MSG}" && exit 1)
travis_time:end:1fcbb868:start=1569225381620534390,finish=1569225381625851931,duration=5317541,event=script
�[0K�[32;1mThe command "test "x$PRINTED_MSG" = "x" || (echo "Found printed messages in output of test suite\n${PRINTED_MSG}" && exit 1)" exited with 0.�[0m
travis_time:start:1e852558
�[0K$ EB_BOOTSTRAP_VERSION=$(grep '^EB_BOOTSTRAP_VERSION' $TRAVIS_BUILD_DIR/easybuild/scripts/bootstrap_eb.py | sed 's/[^0-9.]//g')
travis_time:end:1e852558:start=1569225381633908761,finish=1569225381647148098,duration=13239337,event=script
�[0K�[32;1mThe command "EB_BOOTSTRAP_VERSION=$(grep '^EB_BOOTSTRAP_VERSION' $TRAVIS_BUILD_DIR/easybuild/scripts/bootstrap_eb.py | sed 's/[^0-9.]//g')" exited with 0.�[0m
travis_time:start:1683329c
�[0K$ EB_BOOTSTRAP_SHA256SUM=$(sha256sum $TRAVIS_BUILD_DIR/easybuild/scripts/bootstrap_eb.py | cut -f1 -d' ')
travis_time:end:1683329c:start=1569225381655726922,finish=1569225381673857736,duration=18130814,event=script
�[0K�[32;1mThe command "EB_BOOTSTRAP_SHA256SUM=$(sha256sum $TRAVIS_BUILD_DIR/easybuild/scripts/bootstrap_eb.py | cut -f1 -d' ')" exited with 0.�[0m
travis_time:start:03d555a7
�[0K$ EB_BOOTSTRAP_FOUND="$EB_BOOTSTRAP_VERSION $EB_BOOTSTRAP_SHA256SUM"
travis_time:end:03d555a7:start=1569225381681379401,finish=1569225381685732081,duration=4352680,event=script
�[0K�[32;1mThe command "EB_BOOTSTRAP_FOUND="$EB_BOOTSTRAP_VERSION $EB_BOOTSTRAP_SHA256SUM"" exited with 0.�[0m
travis_time:start:048b3004
�[0K$ EB_BOOTSTRAP_EXPECTED="20190922.01 778762479d77df641c247aba33c13203c8c5b8275e47f59464bab93917aaaf22"
travis_time:end:048b3004:start=1569225381693484645,finish=1569225381697810824,duration=4326179,event=script
�[0K�[32;1mThe command "EB_BOOTSTRAP_EXPECTED="20190922.01 778762479d77df641c247aba33c13203c8c5b8275e47f59464bab93917aaaf22"" exited with 0.�[0m
travis_time:start:03791fc0
�[0K$ test "$EB_BOOTSTRAP_FOUND" = "$EB_BOOTSTRAP_EXPECTED" || (echo "Version check on bootstrap script failed $EB_BOOTSTRAP_FOUND" && exit 1)
Version check on bootstrap script failed 20190922.01 ef29df6211a2836d7c6434d015162c30301c47064bf192fb50b312d2d9d5cdea

• 3466.2 - failed => https://travis-ci.org/easybuilders/easybuild-framework/jobs/588327987
• 3466.3 - failed => https://travis-ci.org/easybuilders/easybuild-framework/jobs/588327988
• 3466.4 - failed => https://travis-ci.org/easybuilders/easybuild-framework/jobs/588327989
• 3466.5 - failed => https://travis-ci.org/easybuilders/easybuild-framework/jobs/588327990
• 3466.6 - failed => https://travis-ci.org/easybuilders/easybuild-framework/jobs/588327991
• 3466.7 - failed => https://travis-ci.org/easybuilders/easybuild-framework/jobs/588327992
• 3466.8 - failed => https://travis-ci.org/easybuilders/easybuild-framework/jobs/588327993
• 3466.9 - failed => https://travis-ci.org/easybuilders/easybuild-framework/jobs/588327994
• 3466.10 - failed => https://travis-ci.org/easybuilders/easybuild-framework/jobs/588327995
• 3466.11 - failed => https://travis-ci.org/easybuilders/easybuild-framework/jobs/588327996
• 3466.12 - failed => https://travis-ci.org/easybuilders/easybuild-framework/jobs/588327997
• 3466.13 - failed => https://travis-ci.org/easybuilders/easybuild-framework/jobs/588327998
• 3466.14 - failed => https://travis-ci.org/easybuilders/easybuild-framework/jobs/588327999
• 3466.15 - failed => https://travis-ci.org/easybuilders/easybuild-framework/jobs/588328000
• 3466.16 - failed => https://travis-ci.org/easybuilders/easybuild-framework/jobs/588328001
• 3466.17 - failed => https://travis-ci.org/easybuilders/easybuild-framework/jobs/588328002
• 3466.18 - failed => https://travis-ci.org/easybuilders/easybuild-framework/jobs/588328003
• 3466.19 - failed => https://travis-ci.org/easybuilders/easybuild-framework/jobs/588328004
• 3466.20 - failed => https://travis-ci.org/easybuilders/easybuild-framework/jobs/588328005
• 3466.21 - failed => https://travis-ci.org/easybuilders/easybuild-framework/jobs/588328006

*bleep, bloop, I'm just a bot (boegelbot v20180813.01)*Please talk to my owner @boegel if you notice you me acting stupid),or submit a pull request to https://github.com/boegel/boegelbot fix the problem.

[arielzn]

@boegel as i was dealing briefly with this today, just thinking out loud here.

Instead of having easybuild bootstrapping itself.
To reproduce the same result as the bootstrap.py install, wouldn't it make sense to have instead a bash script leveraging the normal python install that's supported for 4.x.y versions, something as:

EASYBUILD_PREFIX=$1
EB_PYTHON=python2 or python3
EB_VERSION=4.3.4  or another chosen as arg

wget [url]/easybuild-framework-${EB_VERSION}.tar.gz
wget [url]/easybuild-easyblocks-${EB_VERSION}.tar.gz
wget [url]/easybuild-easyconfigs-${EB_VERSION}.tar.gz

mkdir -p ${EASYBUILD_PREFIX}/software/EasyBuild/${EB_VERSION}

extract tarballs

do "cd each dir"
    ${EB_PYTHON} setup.py install --prefix=${EASYBUILD_PREFIX}/software/EasyBuild/${EB_VERSION}
done

create corresponding ${EASYBUILD_PREFIX}/modules/all/EasyBuild/${EB_VERSION}.lua

is just a very schematic thing to transmit the idea.

there's more stuff to add for sure, as checking if an install is there already and so on, also something to check if there's lmod or tcl that we have on the system.

dunno if adding all checks will make things more cumbersome than just finishing the bootstrap.py todo.

or maybe a script as above can be used to replace the stage0? so pointed to a temp folder, and do the real install with the eb from there?

[boegel]

Bootstrap script was deprecated in #3742, not planning to spend more time on it...