s2i secrets injection test failure

[bparees]

• Failure [16.136 seconds]
[builds][Slow] can use build secrets
/data/src/github.com/openshift/origin/test/extended/builds/secrets.go:114
  build with secrets
  /data/src/github.com/openshift/origin/test/extended/builds/secrets.go:113
    should contain secrets during the source strategy build [It]
    /data/src/github.com/openshift/origin/test/extended/builds/secrets.go:71

    Expected error:
        <*errors.errorString | 0xc820d08ed0>: {
            s: "The build \"test-1\" status is \"Failed\"",
        }
        The build "test-1" status is "Failed"
    not to have occurred

    /data/src/github.com/openshift/origin/test/extended/builds/secrets.go:55

key failure cause is:

E0725 02:26:22.653688       1 postexecutorstep.go:315] /bin/sh: /tmp/rm-injections: Permission denied

as seen:
https://ci.openshift.redhat.com/jenkins/view/Origin%20Test%20Jobs/job/origin_extended_build_tests/357/consoleFull

not sure if this is a flake or we only see it occasionally because other test issues cause the test to timeout before we get here.

[bparees]

@jupierce can you make this your next item after whatever you're working on? I'm hoping you're the right person since you just crawled through the other secrets injection logic.

[jupierce]

Will do!

[jorgemoralespou]

@bparees @jupierce
Not sure if related, but I just build a VM with origin master, and I'm not able to do an s2i build to to problems with secret injection.

First builds failed when pushing to registry. I deleted the serviceaccounts (so the secret injection into the SA worked), and since then, build not even starts. Build seems to be scheduled (per the events) and after a while I've seen this:

This is output form journal:

Jul 26 07:19:59 origin openshift[20321]: E0726 07:19:59.869678   20321 kubelet.go:2611] Failed cleaning pods: [remove /var/lib/origin/openshift.local.volumes/pods/2bd6c362-5322-11e6-8247-525400b263eb/volumes/kubernetes.io~secret/builder-dockercfg-rab7g-pull: device or resource busy, remove /var/lib/origin/openshift.local.volumes/pods/9339faa4-5321-11e6-8247-525400b263eb/volumes/kubernetes.io~secret/builder-dockercfg-rab7g-pull: device or resource busy]
Jul 26 07:20:01 origin openshift[20321]: I0726 07:20:01.873407   20321 kubelet.go:2044] Failed to remove orphaned pod "2bd6c362-5322-11e6-8247-525400b263eb" dir; err: remove /var/lib/origin/openshift.local.volumes/pods/2bd6c362-5322-11e6-8247-525400b263eb/volumes/kubernetes.io~secret/builder-dockercfg-rab7g-pull: device or resource busy
Jul 26 07:20:01 origin openshift[20321]: I0726 07:20:01.873561   20321 kubelet.go:2044] Failed to remove orphaned pod "9339faa4-5321-11e6-8247-525400b263eb" dir; err: remove /var/lib/origin/openshift.local.volumes/pods/9339faa4-5321-11e6-8247-525400b263eb/volumes/kubernetes.io~secret/builder-dockercfg-rab7g-pull: device or resource busy
Jul 26 07:20:01 origin openshift[20321]: E0726 07:20:01.873588   20321 kubelet.go:2290] Failed cleaning up orphaned pod directories: [remove /var/lib/origin/openshift.local.volumes/pods/2bd6c362-5322-11e6-8247-525400b263eb/volumes/kubernetes.io~secret/builder-dockercfg-rab7g-pull: device or resource busy, remove /var/lib/origin/openshift.local.volumes/pods/9339faa4-5321-11e6-8247-525400b263eb/volumes/kubernetes.io~secret/builder-dockercfg-rab7g-pull: device or resource busy]
Jul 26 07:20:01 origin openshift[20321]: E0726 07:20:01.873601   20321 kubelet.go:2611] Failed cleaning pods: [remove /var/lib/origin/openshift.local.volumes/pods/2bd6c362-5322-11e6-8247-525400b263eb/volumes/kubernetes.io~secret/builder-dockercfg-rab7g-pull: device or resource busy, remove /var/lib/origin/openshift.local.volumes/pods/9339faa4-5321-11e6-8247-525400b263eb/volumes/kubernetes.io~secret/builder-dockercfg-rab7g-pull: device or resource busy]
Jul 26 07:20:03 origin openshift[20321]: I0726 07:20:03.869319   20321 kubelet.go:2044] Failed to remove orphaned pod "2bd6c362-5322-11e6-8247-525400b263eb" dir; err: remove /var/lib/origin/openshift.local.volumes/pods/2bd6c362-5322-11e6-8247-525400b263eb/volumes/kubernetes.io~secret/builder-dockercfg-rab7g-pull: device or resource busy
Jul 26 07:20:03 origin openshift[20321]: I0726 07:20:03.869446   20321 kubelet.go:2044] Failed to remove orphaned pod "9339faa4-5321-11e6-8247-525400b263eb" dir; err: remove /var/lib/origin/openshift.local.volumes/pods/9339faa4-5321-11e6-8247-525400b263eb/volumes/kubernetes.io~secret/builder-dockercfg-rab7g-pull: device or resource busy
Jul 26 07:20:03 origin openshift[20321]: E0726 07:20:03.869472   20321 kubelet.go:2290] Failed cleaning up orphaned pod directories: [remove /var/lib/origin/openshift.local.volumes/pods/2bd6c362-5322-11e6-8247-525400b263eb/volumes/kubernetes.io~secret/builder-dockercfg-rab7g-pull: device or resource busy, remove /var/lib/origin/openshift.local.volumes/pods/9339faa4-5321-11e6-8247-525400b263eb/volumes/kubernetes.io~secret/builder-dockercfg-rab7g-pull: device or resource busy]
Jul 26 07:20:03 origin openshift[20321]: E0726 07:20:03.869485   20321 kubelet.go:2611] Failed cleaning pods: [remove /var/lib/origin/openshift.local.volumes/pods/2bd6c362-5322-11e6-8247-525400b263eb/volumes/kubernetes.io~secret/builder-dockercfg-rab7g-pull: device or resource busy, remove /var/lib/origin/openshift.local.volumes/pods/9339faa4-5321-11e6-8247-525400b263eb/volumes/kubernetes.io~secret/builder-dockercfg-rab7g-pull: device or resource busy]
Jul 26 07:20:03 origin openshift[20321]: I0726 07:20:03.932165   20321 reconciler.go:253] MountVolume operation started for volume "kubernetes.io/secret/router-token-xi0yb" (spec.Name: "router-token-xi0yb") to pod "376996d1-5318-11e6-8247-525400b263eb" (UID: "376996d1-5318-11e6-8247-525400b263eb"). Volume is already mounted to pod, but remount was requested.
Jul 26 07:20:03 origin openshift[20321]: I0726 07:20:03.935908   20321 operation_executor.go:720] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/router-token-xi0yb" (spec.Name: "router-token-xi0yb") pod "376996d1-5318-11e6-8247-525400b263eb" (UID: "376996d1-5318-11e6-8247-525400b263eb").

[bparees]

@jorgemoralespou that looks like the secrets mounting issue that @pmorie is working on, it's an upstream bug. Sorry I don't know the issue offhand, maybe @pmorie can link it.