Failed stability upload causes remainder of build to not execute

[brson]

See http://buildbot.rust-lang.org/builders/auto-win-64-opt/builds/817.

Stability upload is set to haltOnFailure=False which should allow it to fail, while continuing to execute the remaining build steps, but what actually happens is that the overall build succeeds but none of the subsequent steps execute.

[brson]

This is actually failing the build completely because the step ends with an exception not a failure. @sa2ajj suggested overriding TEMP (or whatever windows uses to find the temp directory) in order to force each build's temp files to go to a random directory to try to avoid the conflicts causing the original bug.

[brson]

Just setting max_builds=1 for the windows slaves might also fix this.

[brson]

First I might try putting a lock around this step, assuming there is a bug in the DirectoryUpload buildstep.

[brson]

Putting a lock around DirectoryUpload did nothing.

[brson]

Here's a dump of the log

(view as text)

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/Twisted-14.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 382, in callback
    self._startRunCallbacks(result)
  File "/usr/local/lib/python2.7/dist-packages/Twisted-14.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 490, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/local/lib/python2.7/dist-packages/Twisted-14.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 577, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/Twisted-14.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1155, in gotResult
    _inlineCallbacks(r, g, deferred)
--- <exception caught here> ---
  File "/usr/local/lib/python2.7/dist-packages/Twisted-14.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1097, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python2.7/dist-packages/Twisted-14.0.0-py2.7-linux-x86_64.egg/twisted/spread/pb.py", line 470, in throwExceptionIntoGenerator
    return g.throw(RemoteError(self.type, self.value, self.traceback))
  File "/usr/local/lib/python2.7/dist-packages/buildbot-0.8.10_pre_86_g94ddde8-py2.7.egg/buildbot/process/buildstep.py", line 552, in runCommand
    res = yield command.run(self, self.remote)
twisted.spread.pb.RemoteError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\rustbu~1\\appdata\\local\\temp\\tmptmsble'

[sa2ajj]

@brson did you try to run only one instance?

(I need a Windows instance to check the things myself, but it will only happen on Monday.)

[brson]

@sa2ajj No I haven't. I'll try it today and see what happens, but I also intend to just remove the steps causing the bug since they are quite inessential (#19145).

[brson]

@sa2ajj We made it through yesterday with one build per slave and none of these failures. Seems like a good sign.

[sa2ajj]

Yes, it is a good sign :)

But that makes me wonder why the same temporary file can somehow be used by different processes...

[brson]

Buildbots no longer try to upload these metrics so this is moot.