Use nodeenv to manage node version in Jenkins builds

[benpatterson]

This pull request

• gives us the flexibility of managing node from pull-request to pull-request. It'll be a huge lift to folks that want to test out node upgrades like @bjacobel or @andy-armstrong
• Sets us up for a port to Xenial, which will be accomplished in part by using nodeenv for managing the node environment. See a companion PR on the configuration repo for edxapp: https://github.com/edx/configuration/pull/3444

This pull request does not:

• Attempt to solve any stability issues in builds related to node.
• Make node installs any faster
• Create any caching infrastructure to speed up node installs. They will be approximately as fast as they were before. Downloading the node bits build-to-build adds about 3 seconds, which is worth it for the added flexibility.

[benpatterson]

@e0d @edx/testeng This is a WIP but I'd appreciate your feedback, esp in case there's anything about this overall approach that sticks out

[benpatterson]

@maxrothman as well

I'm pulling this out of 'wip' and into 'for real'. Please consider; open to feedback.

[maxrothman]

Looks good to me. 👍

[bjacobel]

This will indeed be very helpful when we upgrade node in platform.

Strictly a feature request/Q: does anybody know what happens if we pass a non-specific version to nodeenv? E.g., 6.9, or even just 6? Can we get upgrades without needing to bump the version in
jenkins-common.sh?

👍

[benpatterson]

@bjacobel The short answer is: a 404.

For posterity, two outputs:

\1. When specifying a non-existent node version:

$ nodeenv --node=100 --prebuilt node_100 --force
 * Install prebuilt node (100) .Traceback (most recent call last):
  File "/workspaces/foo/venv/bin/nodeenv", line 11, in <module>
    sys.exit(main())
  File "/workspaces/foo/venv/lib/python2.7/site-packages/nodeenv.py", line 927, in main
    create_environment(env_dir, opt)
  File "/workspaces/foo/venv/lib/python2.7/site-packages/nodeenv.py", line 768, in create_environment
    install_node(env_dir, src_dir, opt)
  File "/workspaces/foo/venv/lib/python2.7/site-packages/nodeenv.py", line 640, in install_node
    download_node_src(node_url, src_dir, opt, prefix)
  File "/workspaces/foo/venv/lib/python2.7/site-packages/nodeenv.py", line 527, in download_node_src
    tar_contents = io.BytesIO(urlopen(node_url).read())
  File "/workspaces/foo/venv/lib/python2.7/site-packages/nodeenv.py", line 545, in urlopen
    return urllib2.urlopen(req)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 475, in error
    return self._call_chain(*args)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found

\2. A generic version number:

$ nodeenv --node=6 --prebuilt node_6 --force
 * Install prebuilt node (6) .Traceback (most recent call last):
  File "/workspaces/foo/venv/bin/nodeenv", line 11, in <module>
    sys.exit(main())
  File "/workspaces/foo/venv/lib/python2.7/site-packages/nodeenv.py", line 927, in main
    create_environment(env_dir, opt)
  File "/workspaces/foo/venv/lib/python2.7/site-packages/nodeenv.py", line 768, in create_environment
    install_node(env_dir, src_dir, opt)
  File "/workspaces/foo/venv/lib/python2.7/site-packages/nodeenv.py", line 640, in install_node
    download_node_src(node_url, src_dir, opt, prefix)
  File "/workspaces/foo/venv/lib/python2.7/site-packages/nodeenv.py", line 527, in download_node_src
    tar_contents = io.BytesIO(urlopen(node_url).read())
  File "/workspaces/foo/venv/lib/python2.7/site-packages/nodeenv.py", line 545, in urlopen
    return urllib2.urlopen(req)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 475, in error
    return self._call_chain(*args)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found

[benpatterson]

got the cb never called error.

jenkins run python

[pwnage101]

if doCreateNodeEnv itself exits with nonzero status, this loop would assume a timeout. May need to restructure the loop to accomodate for that detail. timeout(1) will exit with status 127 124 on timeout, which is what you're really looking for.

[pwnage101]

refresh page just in case, i made an important typo fix in the above comment :)

[benpatterson]

cb() never called error; flaky JS test

jenkins run js
jenkins run quality

[benpatterson]

jenkins run bokchoy

[benpatterson]

@pwnage101 could you suggest an implementation that would trap the exit code in the case of a timeout vs not-a-timeout?

Depending on how easy/hard that is to accomplish, we could consider shipping this as-is or refactoring.

[pwnage101]

Quoting Ben Patterson (2016-11-09 21:24:46)

@pwnage101 could you suggest an implementation that would trap the
exit code in the case of a timeout vs not-a-timeout?

Something like this should work:

EXIT_STATUS = 0
TIMEOUT_STATUS = 124 # timed-out status, see man 1 timeout
WAIT_COUNT = 0
WAIT_LIMIT = 3
unless [ $EXIT_STATUS -eq $TIMEOUT_STATUS -o $WAIT_COUNT -eq $WAIT_LIMIT ]; do
timeout 30s $NODE_INSTALL_COMMAND
exit_status = $?
retries=$(( retries++ ))
sleep 5
done

Troy

[pwnage101]

oops, bugs and typos on so many levels. Let me try again...

TIMEOUT_STATUS = 124  # timed-out status, see `man 1 timeout`
EXIT_STATUS = $TIMEOUT_STATUS
WAIT_COUNT = 0
WAIT_LIMIT = 3
# two conditions for breaking out of the retry loop: (1) we didn't timeout, or
# (2) we reached the retry limit.
until [ $EXIT_STATUS -neq $TIMEOUT_STATUS -o $WAIT_COUNT -eq $WAIT_LIMIT ]; do
  timeout 30s $NODE_INSTALL_COMMAND
  EXIT_STATUS = $?
  WAIT_COUNT=$(( WAIT_COUNT++ ))
  sleep 5
done

[pwnage101]

This condition shouldn't be removed, I don't think. Without it, nothing gets done on max retries.

[benpatterson]

@pwnage101 Thanks for the help, but it never quite behaved as intended. I think because the exit status was captured in the pipe, it wasn't registering correctly (i.e., because it was technically still happening). I did a fair amount of testing on a Jenkins box but in the end wasn't able to make it that smart.

So I'm leaving the older implementation...wherein it'll exit 1 whether it's a timeout or an actual command failure.

[benpatterson]

Oh also I'm intentionally leaving a unique echo statement in there so we can hit splunk and see if this is catching an intermittent hang.

[pwnage101]

The less fun implementation will likely work for our use case, but keep in mind that a command failure will result in retries which hopefully are not bad to perform.

[benpatterson]

@pwnage101 you are right.

I figure retrying a failed command is better than risking occasional hangs. I was about to merge this yesterday when I ran into a hang. It was the first I'd seen out of a couple or few dozen builds, but that could amount to another flaky condition and I wanted to (attempt to) nip it in the bud.

I gave a smarter implementation a go, but after a couple of hours working on it, I'm going to let it go for now. Not sure what the emoticon is for that...I'll say 🐂 as a hard working, albeit slow-moving animal. (er, that's meant to reflect the implementation, not me!)

[pwnage101]

makes sense.

👍 provided that you add a comment like "command will also be retried on failure, even if it didn't timeout"

[benpatterson]

Done.

I'll let this run, then squash and run again, and merge at that point unless we discover something else.

[benpatterson]

github failures...

jenkins run lettuce

[benpatterson]

jenkins run bokchoy

ref: https://openedx.atlassian.net/browse/FEDX-99