test-requireio-osx1010-x64-1 is borked

[refack]

I took it offline for now https://ci.nodejs.org/computer/test-requireio-osx1010-x64-1/

It started with perma:

FATAL: command execution failed
java.nio.channels.ClosedChannelException
	at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
	at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:179)
	at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:721)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused: hudson.remoting.ChannelClosedException: channel is already closed
	at hudson.remoting.Channel.send(Channel.java:605)
	at hudson.remoting.Request.call(Request.java:130)
	at hudson.remoting.Channel.call(Channel.java:829)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257)
	at com.sun.proxy.$Proxy59.isAlive(Unknown Source)
	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1043)
	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1035)
	at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
	at hudson.model.Build$BuildExecution.build(Build.java:206)
	at hudson.model.Build$BuildExecution.doRun(Build.java:163)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
	at hudson.model.Run.execute(Run.java:1728)
	at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:405)
Build step 'Execute shell' marked build as failure

I logged in and made the mistake of rimrafing the workspace. The job setup is broken as it refers to /home/iojs/io.js.reference when the local mirror is actually at /Users/iojs/git/io.js.reference.
So I cloned the local mirror manually and got one step further to.

 > git config remote.origin.url https://github.com/nodejs/node.git # timeout=10
Fetching upstream changes from https://github.com/nodejs/node.git
 > git --version # timeout=10
using GIT_SSH to set credentials 
 > git fetch --tags --progress https://github.com/nodejs/node.git +refs/heads/*:refs/remotes/origin/* +refs/pull/15083/head:refs/remotes/origin/_jenkins_local_branch # timeout=20
Checking out Revision aaec74c55f00a233206059dc3cd69242e2e3a643 (refs/remotes/origin/_jenkins_local_branch)
java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins$MasterComputer
	at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:29)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:71)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:895)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:870)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:829)
	at hudson.remoting.UserRequest.perform(UserRequest.java:153)
	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
	at hudson.remoting.Request$2.run(Request.java:336)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at hudson.remoting.Engine$1$1.run(Engine.java:94)
	at java.lang.Thread.run(Thread.java:745)
	at ......remote call to JNLP4-connect connection from static-n49-176-213-208.sbr2.nsw.optusnet.com.au/49.176.213.208:52267(Native Method)
	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)
	at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
	at hudson.remoting.Channel.call(Channel.java:830)
Caused: java.io.IOException: Remote call on JNLP4-connect connection from static-n49-176-213-208.sbr2.nsw.optusnet.com.au/49.176.213.208:52267 failed
	at hudson.remoting.Channel.call(Channel.java:838)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257)
	at com.sun.proxy.$Proxy60.withRepository(Unknown Source)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl.withRepository(RemoteGitImpl.java:235)
	at hudson.plugins.git.GitSCM.printCommitMessageToLog(GitSCM.java:1195)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1159)
	at hudson.scm.SCM.checkout(SCM.java:496)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1281)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
	at hudson.model.Run.execute(Run.java:1728)
	at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:405)
Run condition [Always] enabling perform for step [[]]

But there I'm stuck...

/cc @rvagg @joaocgreis

[rvagg]

fwiw the lack of a git mirror isn't essential and should just be a quiet fail, while it'll slow it down if it's not in the right place it still should be fine.

I'm still messing with this machine, some weird network problems, can't even update the git mirror or clone a new one using --mirror or --bare although I have no idea if that's related! I've tried updating the machine, rebooting, rebooting jenkins master, deleting and recreating the node config on jenkins master, rebooting network hardware the machine is plugged in to. All to no avail.

I would try updating Java but a recent VMWare Fusion upgrade on that machine messed up access to the consoles, I only have black screens and have to use SSH to do everything like everyone else.

I'm out for 12+ hours but will look back at this soon. I'm not going to leave this machine offline because failures are better than a big queue in jenkins. If anyone else wants to tinker on the machine please do so.

[rvagg]

@nodejs/build did anyone install a new jenkins plugin recently?

[rvagg]

ok, really heading off now, my remaining theory is that a plugin was installed recently that's inteferring. If someone installed one in the last week or so, I'd appreciate you either drop a note in here or experiment with removing it and running node-test-commit-osx.

[gibfahn]

The fact that there are only two updates available suggests that the plugins were updated recently, so it might have been an update that changed things.

I haven't installed anything since the rebuild plugin (which was apparently in July #672).

[Trott]

The node-test-commit-osx jobs seem to be trying to build on test-packetnet-ubuntu1604-x64-1 which seems to minimally be a labeling error, but maybe a full-on larger problem? See https://ci.nodejs.org/job/node-test-commit-osx/12132/ for an example.

[gibfahn]

@Trott is that a problem? The parent job is running there because it has the jenkins-workspace, but that just runs the initial git clone. The actual job is running on test-requireio-osx1010-x64-1.

[Trott]

@Trott is that a problem? The parent job is running there because it has the jenkins-workspace, but that just runs the initial git clone. The actual job is running on test-requireio-osx1010-x64-1.

Nope, not a problem if it works. :-D

[rvagg]

@gibfahn I updated the plugins while trying to sort this out, which is why they look like that. Didn't help.

Interestingly there have been a few greens today and the last run only disconnected as it was already into the test running.

I'll see if I can figure out how to determine which plugins were most recently installed and start removing them to see if that makes a difference. Another alternative might be to try and get a different JVM installed but as I've already mentioned that's going to be tricky without a GUI!

[rvagg]

some more weird https errors on my local infra, this time on pi1's building binaries: https://ci-release.nodejs.org/job/iojs+release/1990/nodes=pi1-raspbian-wheezy/console (only accessible to people with release access)

hudson.plugins.git.GitException: Failed to fetch from https://github.com/nodejs/node.git

I haven't experienced anything similar on my own use of this network (I'm on it now) but this is very similar to what I was experiencing on the macOS box on the same network! I've done a cleanup on those two release machines and will keep an eye on those builds, I could conceive of a way this might be local to those machines but it's pretty suspicious on the face of it.

Re macOS, I've downloaded the latest Oracle JRE macOS tarball and unpacked it into /usr/local/lib with a /usr/local/bin/java symlink and changed the slave.jar startup to use that binary and it's running now. It's going @ https://ci.nodejs.org/job/node-test-commit-osx/12159/nodes=osx1010/console but taking a very long time to even get started which makes me still suspect a problematic plugin.

Will keep an eye on all of this and continue to work out what on earth is up. Again, any help or suggestions would be appreciated!

[gibfahn]

@rvagg is there anything in the master jenkins logs? If it's to do with the git clone it might be an issue moving clones around. Not sure what else could be going on here, haven't been seeing this on our internal Jenkins instance at all.

[rvagg]

Hm, yeah, a bunch of these just now for that host only java.util.concurrent.TimeoutException when trying to do the regular monitoring.

Maybe I just need to and shut everything down, including the internet connection and start it all up again slowly.

[refack]

Another alternative might be to try and get a different JVM installed but as I've already mentioned that's going to be tricky without a GUI!

RE GUI, did you try to VNC to the VM? I have some success with that...

[rvagg]

@refack got some command-line-fu to enable screen sharing? it's not enabled on the VMs unfortunately.

[rvagg]

@joaocgreis can you look at the shared binary repo and see if you can make it prune any more aggressively? It's at 3.2G at the moment and may be contributing somewhat to network problems (not likely the cause of this particular problem though).

[refack]

I think I used https://content.pivotal.io/blog/enabling-os-x-screen-sharing-from-the-command-line
Which Is based off https://apple.stackexchange.com/questions/30238/how-to-enable-os-x-screen-sharing-vnc-through-ssh

[rvagg]

✋ beaut! that worked nicely, I'm on the console now.

[rvagg]

https://ci.nodejs.org/job/node-test-commit-osx/12160/nodes=osx1010/console

More of the same, those errors are in jenkins.log in the master so I'm assuming they come from there rather than being reported from the slave. So, on the slave we have new java (same on master and slave), restarted (a few times), completely cycled all network hardware (none of the Pi's are even running as I'm writing this).

Other than rogue plugins, the only other thing I can think of that might coincide with this is the updated SSL cert on master. The timing might explain this and the errors are SSL related. But why just this host? Not even the OSX host on ci-release is behaving this way.

[rvagg]

I've tried removing, reconfiguring, resetting network hardware in vmware for that VM. I've tried using NAT on the host instead of bridged direct onto the network. I've tried making new nodes on Jenkins and pointing it to them (test-requireio-osx1010-x64-2, test-requireio-osx1010-x64-3) just in case there's latent config garbage on the master.

Same.

[rvagg]

Seeing similar errors now on almost all of the machines hosted on the same network, most of the Pi's are offline at the moment due to this and are labelled as "Ping response time is too long or timed out.". I did tweak the ping and timeout to lengthen it but apparently this hasn't helped matters, maybe even made it worse!?

[rvagg]

okie dokie, making progress finally

Installed an entirely new VM, from scratch, OSX 10.10 and we now have a string of greens on https://ci.nodejs.org/job/node-test-commit-osx/nodes=osx1010/ with no disconnects (yet). Crossing fingers that the problems were isolated to that VM and will go away now 🤞. I'm seeing similar disconnection messages from other hosts that are similar now to the ones I was worried about in my last message so I think it's just a new form of Jenkins message I'm not used to seeing, so for now I'll avoid assuming there's something bigger at play on the local network.

While installing the VM I came up with a new script that could be ansiblised. I avoided the UI as much as possible for this. These instructions will need to be used (and maybe improved) by whoever gets far enough with the MacStadium hosts that replace this single machine. (Note for people not in the know: the MacStadium setup is super complicated, vSphere stuff that we're having to level-up on, it's stalled because none of us have had the time to spend the extra time needed to get further than we have. I believe IBM may be assigning some staff to help with this.)

Steps listed below, I need to put these into the new doc/process/non-ansible-configuration-notes.md doc (or someone else is welcome to).

• (UI) Set up machine with an Apple ID (rvagg / [email protected], can share creds with nodejs_build_infra folks if helpful). Although I'm not sure this is strictly required, maybe just jump straight to the iojs user.
• Gave this user NOPASSWD: ALL in /etc/sudoers for setup convenience
• (UI) Create an iojs user, unprivileged
• Install Homebrew /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)", which has a nice side effect of installing the xcode command-line tools so no separate step needed for that
• brew install git (updated version)
• brew install ccache, then set up with (cd /usr/local/bin/ && for i in c++ cc clang g++ gcc; do ln -sf ccache $i; done) (there may be a nicer way to do this, maybe with brew, or maybe just use PATH in the start script)
• brew install caskroom/cask/java (👏 👏)
• As user iojs mkdir /Users/iojs/build && mkdir /Users/iojs/git
• As user iojs git clone --mirror https://github.com/nodejs/node.git /Users/iojs/git/io.js.reference
• As user iojs git clone https://chromium.googlesource.com/external/gyp /Users/iojs/gyp
• Create executable script /Users/iojs/start.sh for user iojs with contents below
• Create /Library/LaunchDaemons/org.nodejs.osx.jenkins.plist as root:wheel with contents below
• sudo launchctl load /Library/LaunchDaemons/org.nodejs.osx.jenkins.plist
• sudo launchctl start org.nodejs.osx.jenkins

(edit: these two were run later as a result of @refack's comment below)

• sudo launchctl unload -w /System/Library/LaunchAgents/com.apple.ReportCrash.plist
• sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.ReportCrash.Root.plist

start.sh

#!/bin/sh

curl -O https://ci.nodejs.org/jnlpJars/slave.jar

SECRET=<SECRET HERE>
SERVER_ID=<NODE NAME HERE>
CI_SERVER=ci.nodejs.org

export PATH="/usr/local/bin:$PATH"
export JOBS=2
export NODE_COMMON_PIPE=/Users/iojs/test.pipe
export OSTYPE=osx
export ARCH=x64
export DESTCPU=x64

java -jar slave.jar -jnlpUrl https://${CI_SERVER}/computer/${SERVER_ID}/slave-agent.jnlp -secret $SECRET

/Library/LaunchDaemons/org.nodejs.osx.jenkins.plist

<plist version="1.0">
        <dict>
                <key>Label</key>
                <string>org.nodejs.osx.jenkins</string>

                <key>UserName</key>
                <string>iojs</string>

                <key>WorkingDirectory</key>
                <string>/Users/iojs</string>

                <key>Program</key>
                <string>/Users/iojs/start.sh</string>

                <key>RunAtLoad</key>
                <true/>

                <key>KeepAlive</key>
                <true/>

                <key>StandardErrorPath</key>
                <string>/Users/iojs/jenkins_err.log</string>

                <key>StandardOutPath</key>
                <string>/Users/iojs/jenkins_out.log</string>
        </dict>
</plist>

[refack]

An old favorite is raising it's head:

not ok 75 async-hooks/test-callback-error
  ---
  duration_ms: 15.545
  severity: fail
  stack: |-
    start case 1
    end case 1: 109.402ms
    start case 2
    end case 2: 99.464ms
    start case 3
    end case 3: 15.766ms
    Error: test_callback_abort
        at ActivityCollector.initHooks.oninit.common.mustCall (/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1010/test/async-hooks/test-callback-error.js:36:45)
        at ActivityCollector.oninit (/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1010/test/common/index.js:509:15)
        at ActivityCollector._init (/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1010/test/async-hooks/init-hooks.js:182:10)
        at emitInitNative (async_hooks.js:446:43)
        at Object.emitInitScript [as emitInit] (async_hooks.js:349:3)
        at Object.<anonymous> (/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1010/test/async-hooks/test-callback-error.js:38:17)
        at Module._compile (module.js:549:30)
        at Object.Module._extensions..js (module.js:560:10)
        at Module.load (module.js:483:32)
        at tryModuleLoad (module.js:446:12)
     1: node::Abort() [/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1010/out/Release/node]
     2: node::Chdir(v8::FunctionCallbackInfo<v8::Value> const&) [/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1010/out/Release/node]
     3: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) [/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1010/out/Release/node]
     4: v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1010/out/Release/node]
     5: v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1010/out/Release/node]
     6: 0x16a6e9e846fd

Need to disable the crash reporter nodejs/node#13527 (comment)

This has been fixed since @rvagg ran these two commands on the host:

launchctl unload -w /System/Library/LaunchAgents/com.apple.ReportCrash.plist
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.ReportCrash.Root.plist

Note that they need to be run from a GUI login or else they error out.

There's an issue open to get this into the provisioning for OS X machines. But for now, the problem is resolved, so I'm going to close this.

[refack]

P.S. it's rejecting the test key:

debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Trying private key: ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,keyboard-interactive
debug2: we did not send a packet, disable method
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug2: we sent a keyboard-interactive packet, wait for reply
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 1
Password:
Killed by signal 2.

[joaocgreis]

@rvagg I cleaned the binary repo (can only do manually, because there's no way to tell if a branch is in use or not). Many branches were left behind when jobs were aborted, so I took a look and was able to change the way branches are deleted in arm-fanned and windows-fanned to run even if the job is aborted.

[rvagg]

@refack thanks, just ran those commands via ssh (no security complaints, didn't need to do it via UI!), this re-run of a previously failed job (due to that error) has an unrelated failure: https://ci.nodejs.org/job/node-test-commit-osx/12210/nodes=osx1010/console, so I think we might be good now. Will watch today's daily jobs to see if we get more green. I've updated my scripted list above to include these. 🤞

[refack]

1 green job https://ci.nodejs.org/job/node-test-commit-osx/12211/
Which IMO is a very possitive sign 🏆

[refack]

2nd green job in a row https://ci.nodejs.org/job/node-test-commit-osx/12212/
By Jove I think she got it

[Trott]

P.S. Let's get those commands added to ansible or whatever, yeah?

[rvagg]

👍

[rvagg]

Something's still up. The 3 failures in that list of most recent builds from today are disconnects during compile. Not super-early like before but still the same kind of disconnect.

Still mostly green but regular disconnects it seems. I haven't looked any further back at the failures other than today's, I only just caught the machine offline so went for a quick dig. If anyone else wants to, go to https://ci.nodejs.org/computer/test-requireio-osx1010-x64-1/builds, wait for it to load and click through to each of them and look at the console output to see if it's a test failure or disconnection.

I don't have an answer here except that we need to get on with our mac IaaS work and have more of these for redundancy and hopefully stability.

[refack]

@rvagg @joaocgreis
test-requireio-osx1010-x64-1 is still rejecting the test key for iojs

[joaocgreis]

I can't connect either. Missing nodejs_build_test?

[rvagg]

yes, sorry, new rebuild missed out on key, fixed now, reopen if it doesn't work for y'all