Jenkins flake: no space left on device

[dcbw]

During Jenkins run of #6532 we get:

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin/8298/console

Stderr from the command:

2016-01-07 17:40:36.334592 I | profile: cpu profiling enabled, cpu.pprof
2016-01-07 17:40:36.400521 I | profile: cpu profiling enabled, cpu.pprof
2016-01-07 17:40:50.069137 I | profile: cpu profiling enabled, cpu.pprof
2016-01-07 17:40:51.655557 I | profile: cpu profiling enabled, cpu.pprof
2016-01-07 17:40:51.715588 I | profile: cpu profiling enabled, cpu.pprof
Error from server: User "system:anonymous" cannot list pods in project "default"
Dry run enabled - no modifications will be made. Add --confirm to remove images
hack/../test/cmd/../../hack/cmd_util.sh: line 276: cannot create temp file for here-document: No space left on device
!!! Error in hack/../test/cmd/../../hack/cmd_util.sh:195
'return 1' exited with status 1
Call stack:
1: hack/../test/cmd/../../hack/cmd_util.sh:195 os::cmd::expect_success(...)
2: hack/../test/cmd/newapp.sh:166 main(...)
Exiting with status 1
!!! Error in hack/test-cmd.sh:288
'${test}' exited with status 1
Call stack:
1: hack/test-cmd.sh:288 main(...)
Exiting with status 1
make: *** [test-int-plus] Error 1

[liggitt]

I think this is caused by hotloops in https://github.com/liggitt/origin/blob/master/pkg/image/controller/controller.go#L143, e.g.:

I0108 11:38:05.813092   53562 controller.go:149] requeuing default/ruby to the worklist
I0108 11:38:05.813117   53562 controller.go:133] popped default/ruby from the worklist
I0108 11:38:05.844772   53562 controller.go:149] requeuing default/ruby to the worklist
I0108 11:38:05.844801   53562 controller.go:133] popped default/ruby from the worklist
I0108 11:38:05.913240   53562 controller.go:149] requeuing default/ruby to the worklist
I0108 11:38:05.913269   53562 controller.go:133] popped default/ruby from the worklist
I0108 11:38:05.948484   53562 controller.go:149] requeuing default/ruby to the worklist
I0108 11:38:05.948510   53562 controller.go:133] popped default/ruby from the worklist
I0108 11:38:06.015106   53562 controller.go:149] requeuing default/ruby to the worklist
I0108 11:38:06.015227   53562 controller.go:133] popped default/ruby from the worklist
I0108 11:38:06.053133   53562 controller.go:149] requeuing default/ruby to the worklist
I0108 11:38:06.053599   53562 controller.go:133] popped default/ruby from the worklist
I0108 11:38:06.119370   53562 controller.go:149] requeuing default/ruby to the worklist
I0108 11:38:06.119401   53562 controller.go:133] popped default/ruby from the worklist
I0108 11:38:06.154055   53562 controller.go:149] requeuing default/ruby to the worklist
I0108 11:38:06.154090   53562 controller.go:133] popped default/ruby from the worklist
I0108 11:38:06.220404   53562 controller.go:149] requeuing default/ruby to the worklist
I0108 11:38:06.220435   53562 controller.go:133] popped default/ruby from the worklist
I0108 11:38:06.255132   53562 controller.go:149] requeuing default/ruby to the worklist
I0108 11:38:06.255170   53562 controller.go:133] popped default/ruby from the worklist
I0108 11:38:06.324035   53562 controller.go:149] requeuing default/ruby to the worklist
I0108 11:38:06.324070   53562 controller.go:133] popped default/ruby from the worklist
I0108 11:38:06.355627   53562 controller.go:149] requeuing default/ruby to the worklist
I0108 11:38:06.355716   53562 controller.go:133] popped default/ruby from the worklist
I0108 11:38:06.429212   53562 controller.go:149] requeuing default/ruby to the worklist
I0108 11:38:06.429308   53562 controller.go:133] popped default/ruby from the worklist
I0108 11:38:06.456273   53562 controller.go:149] requeuing default/ruby to the worklist
I0108 11:38:06.456315   53562 controller.go:133] popped default/ruby from the worklist
I0108 11:38:06.534155   53562 controller.go:149] requeuing default/ruby to the worklist
I0108 11:38:06.534194   53562 controller.go:133] popped default/ruby from the worklist
I0108 11:38:06.557005   53562 controller.go:149] requeuing default/ruby to the worklist
I0108 11:38:06.557048   53562 controller.go:133] popped default/ruby from the worklist
I0108 11:38:06.635484   53562 controller.go:149] requeuing default/ruby to the worklist
I0108 11:38:06.635516   53562 controller.go:133] popped default/ruby from the worklist
I0108 11:38:06.657820   53562 controller.go:149] requeuing default/ruby to the worklist
I0108 11:38:06.657868   53562 controller.go:133] popped default/ruby from the worklist
I0108 11:38:06.738781   53562 controller.go:149] requeuing default/ruby to the worklist
I0108 11:38:06.738837   53562 controller.go:133] popped default/ruby from the worklist

@deads2k

[deads2k]

Hmmm... unlucky. This could happen if the same imagestream was listed multiple times as the only work in the channel. The channel add should be de-duped.

[bparees]

(also we're going to work on fixing the test-cmd script to not gather large log files to jenkins)