Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extended.[Conformance][registry][migration] manifest migration from etcd to registry storage registry can get access to manifest [local] #13183

Closed
stevekuznetsov opened this issue Mar 2, 2017 · 6 comments
Assignees
Labels
dependency/devicemapper kind/test-flake Categorizes issue or PR as related to test flakes. priority/P1

Comments

@stevekuznetsov
Copy link
Contributor

Flaking out as seen here:

/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/registry/registry.go:122
Expected error:
    <*docker.Error | 0xc4210b0300>: {
        Status: 500,
        Message: "{\"message\":\"failed to remove device 8019a36673d26e2f94ca11f122a2ad36964521755c60fb7cd0c6b6a5720b8985:Device is Busy\"}\n",
    }
    API error (500): {"message":"failed to remove device 8019a36673d26e2f94ca11f122a2ad36964521755c60fb7cd0c6b6a5720b8985:Device is Busy"}
    
not to have occurred
/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/registry/registry.go:121

Looks like it may be fallout from the change to docker-storage-setup or in general a failure with devicemapper. /cc @jwhonce @runcom @bparees @csrwng

Click for full logs
Stacktrace

/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/registry/registry.go:122
Expected error:
    <*docker.Error | 0xc4210b0300>: {
        Status: 500,
        Message: "{\"message\":\"failed to remove device 8019a36673d26e2f94ca11f122a2ad36964521755c60fb7cd0c6b6a5720b8985:Device is Busy\"}\n",
    }
    API error (500): {"message":"failed to remove device 8019a36673d26e2f94ca11f122a2ad36964521755c60fb7cd0c6b6a5720b8985:Device is Busy"}
    
not to have occurred
/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/registry/registry.go:121
Standard Output

[BeforeEach] [Top Level]
  /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/test.go:47
[BeforeEach] [Conformance][registry][migration] manifest migration from etcd to registry storage
  /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:141
STEP: Creating a kubernetes client
Mar  1 17:23:34.963: INFO: >>> kubeConfig: /tmp/openshift/test-extended/core/openshift.local.config/master/admin.kubeconfig

STEP: Building a namespace api object
Mar  1 17:23:34.984: INFO: configPath is now "/tmp/extended-test-registry-migration-xl9xv-9gkl6-user.kubeconfig"
Mar  1 17:23:34.984: INFO: The user is now "extended-test-registry-migration-xl9xv-9gkl6-user"
Mar  1 17:23:34.984: INFO: Creating project "extended-test-registry-migration-xl9xv-9gkl6"
STEP: Waiting for a default service account to be provisioned in namespace
[It] registry can get access to manifest [local]
  /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/registry/registry.go:122
STEP: set up policy for registry to have anonymous access to images
Mar  1 17:23:35.050: INFO: Running 'oc policy --config=/tmp/extended-test-registry-migration-xl9xv-9gkl6-user.kubeconfig --namespace=extended-test-registry-migration-xl9xv-9gkl6 add-role-to-user registry-viewer system:anonymous'
role "registry-viewer" added: "system:anonymous"
STEP: pushing image...
Step 1 : FROM scratch
 ---> 
Step 2 : COPY data1 /data1
 ---> cbf641f53e1c
Removing intermediate container 9b472246d259
Successfully built cbf641f53e1c
Mar  1 17:23:37.584: INFO: Running 'oc whoami --config=/tmp/extended-test-registry-migration-xl9xv-9gkl6-user.kubeconfig --namespace=extended-test-registry-migration-xl9xv-9gkl6 -t'
The push refers to a repository [172.30.103.69:5000/extended-test-registry-migration-xl9xv-9gkl6/app]
Preparing
Pushing [====================>                              ]    512 B/1.28 kB
Pushing
Pushing [==================================================>] 1.792 kB
Pushing
Pushing [==================================================>] 3.072 kB
Pushing
Pushing [==================================================>] 3.072 kB
Pushing
Pushed
latest: digest: sha256:d7cf5b995be8469310e05d5f636d7c8b18d946cdc0e62503bd9fca9a48f44848 size: 1536
matching digest string
STEP: checking that the image converted...
STEP: getting image manifest from docker-registry...
STEP: restoring manifest...
STEP: checking that the manifest is present in the image...
STEP: getting image manifest from docker-registry one more time...
STEP: waiting until image is updated...
STEP: checking that the manifest was removed from the image...
STEP: getting image manifest from docker-registry to check if he's available...
STEP: pulling image...
STEP: get secret list err <nil> 
STEP: secret name builder-dockercfg-l284m 
STEP: docker cfg token json {"172.30.103.69:5000":{"username":"serviceaccount","password":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJleHRlbmRlZC10ZXN0LXJlZ2lzdHJ5LW1pZ3JhdGlvbi14bDl4di05Z2tsNiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJidWlsZGVyLXRva2VuLW5sNWd6Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImJ1aWxkZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJiNTliNjcwNC1mZWNkLTExZTYtYjA1Yi0wZWI5OTU2MWJkZTgiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZXh0ZW5kZWQtdGVzdC1yZWdpc3RyeS1taWdyYXRpb24teGw5eHYtOWdrbDY6YnVpbGRlciJ9.aP_xGAOor3Y63NALamnpUTLpX3QnsJcsWOJKOCnWUWFSm2Y7r0uNBHH1_u0OAwhM8aG74neV8K0KCUSPOPzKxF1AZjSongF49qOKujfHv64m6n8Wl1C5ufP_aICA1erDk5wk61d7BndeUkbV15tdQBGEm4AdtPBl2imqduQlBkowXA7DkiGmfFG4NSOjh_nLfyQdJld5QZVogiqPVRJli5lrOqfgxv4QobDHkjrfSLVheWZ-j_jX10miJMBpmlIwTO6HcB0yNlT809-zwKf2Dkkatkkyw23XOxd_0O5-HElyxn8QlHJbFbC2KHtUBha78KO-7-WVEIOF-4UTF3YuJA","email":"[email protected]","auth":"c2VydmljZWFjY291bnQ6ZXlKaGJHY2lPaUpTVXpJMU5pSXNJblI1Y0NJNklrcFhWQ0o5LmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUpsZUhSbGJtUmxaQzEwWlhOMExYSmxaMmx6ZEhKNUxXMXBaM0poZEdsdmJpMTRiRGw0ZGkwNVoydHNOaUlzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVmpjbVYwTG01aGJXVWlPaUppZFdsc1pHVnlMWFJ2YTJWdUxXNXNOV2Q2SWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXpaWEoyYVdObExXRmpZMjkxYm5RdWJtRnRaU0k2SW1KMWFXeGtaWElpTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxjblpwWTJVdFlXTmpiM1Z1ZEM1MWFXUWlPaUppTlRsaU5qY3dOQzFtWldOa0xURXhaVFl0WWpBMVlpMHdaV0k1T1RVMk1XSmtaVGdpTENKemRXSWlPaUp6ZVhOMFpXMDZjMlZ5ZG1salpXRmpZMjkxYm5RNlpYaDBaVzVrWldRdGRHVnpkQzF5WldkcGMzUnllUzF0YVdkeVlYUnBiMjR0ZUd3NWVIWXRPV2RyYkRZNlluVnBiR1JsY2lKOS5hUF94R0FPb3IzWTYzTkFMYW1ucFVUTHBYM1Fuc0pjc1dPSktPQ25XVVdGU20yWTdyMHVOQkhIMV91ME9Bd2hNOGFHNzRuZVY4SzBLQ1VTUE9Qekt4RjFBWmpTb25nRjQ5cU9LdWpmSHY2NG02bjhXbDFDNXVmUF9hSUNBMWVyRGs1d2s2MWQ3Qm5kZVVrYlYxNXRkUUJHRW00QWR0UEJsMmltcWR1UWxCa293WEE3RGtpR21mRkc0TlNPamhfbkxmeVFkSmxkNVFaVm9naXFQVlJKbGk1bHJPcWZneHY0UW9iREhranJmU0xWaGVXWi1qX2pYMTBtaUpNQnBtbEl3VE82SGNCMHlObFQ4MDktendLZjJEa2thdGtreXcyM1hPeGRfME81LUhFbHl4bjhRbEhKYkZiQzJLSHRVQmhhNzhLTy03LVdWRUlPRi00VVRGM1l1SkE="},"docker-registry.default.svc:5000":{"username":"serviceaccount","password":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJleHRlbmRlZC10ZXN0LXJlZ2lzdHJ5LW1pZ3JhdGlvbi14bDl4di05Z2tsNiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJidWlsZGVyLXRva2VuLW5sNWd6Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImJ1aWxkZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJiNTliNjcwNC1mZWNkLTExZTYtYjA1Yi0wZWI5OTU2MWJkZTgiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZXh0ZW5kZWQtdGVzdC1yZWdpc3RyeS1taWdyYXRpb24teGw5eHYtOWdrbDY6YnVpbGRlciJ9.aP_xGAOor3Y63NALamnpUTLpX3QnsJcsWOJKOCnWUWFSm2Y7r0uNBHH1_u0OAwhM8aG74neV8K0KCUSPOPzKxF1AZjSongF49qOKujfHv64m6n8Wl1C5ufP_aICA1erDk5wk61d7BndeUkbV15tdQBGEm4AdtPBl2imqduQlBkowXA7DkiGmfFG4NSOjh_nLfyQdJld5QZVogiqPVRJli5lrOqfgxv4QobDHkjrfSLVheWZ-j_jX10miJMBpmlIwTO6HcB0yNlT809-zwKf2Dkkatkkyw23XOxd_0O5-HElyxn8QlHJbFbC2KHtUBha78KO-7-WVEIOF-4UTF3YuJA","email":"[email protected]","auth":"c2VydmljZWFjY291bnQ6ZXlKaGJHY2lPaUpTVXpJMU5pSXNJblI1Y0NJNklrcFhWQ0o5LmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUpsZUhSbGJtUmxaQzEwWlhOMExYSmxaMmx6ZEhKNUxXMXBaM0poZEdsdmJpMTRiRGw0ZGkwNVoydHNOaUlzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVmpjbVYwTG01aGJXVWlPaUppZFdsc1pHVnlMWFJ2YTJWdUxXNXNOV2Q2SWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXpaWEoyYVdObExXRmpZMjkxYm5RdWJtRnRaU0k2SW1KMWFXeGtaWElpTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxjblpwWTJVdFlXTmpiM1Z1ZEM1MWFXUWlPaUppTlRsaU5qY3dOQzFtWldOa0xURXhaVFl0WWpBMVlpMHdaV0k1T1RVMk1XSmtaVGdpTENKemRXSWlPaUp6ZVhOMFpXMDZjMlZ5ZG1salpXRmpZMjkxYm5RNlpYaDBaVzVrWldRdGRHVnpkQzF5WldkcGMzUnllUzF0YVdkeVlYUnBiMjR0ZUd3NWVIWXRPV2RyYkRZNlluVnBiR1JsY2lKOS5hUF94R0FPb3IzWTYzTkFMYW1ucFVUTHBYM1Fuc0pjc1dPSktPQ25XVVdGU20yWTdyMHVOQkhIMV91ME9Bd2hNOGFHNzRuZVY4SzBLQ1VTUE9Qekt4RjFBWmpTb25nRjQ5cU9LdWpmSHY2NG02bjhXbDFDNXVmUF9hSUNBMWVyRGs1d2s2MWQ3Qm5kZVVrYlYxNXRkUUJHRW00QWR0UEJsMmltcWR1UWxCa293WEE3RGtpR21mRkc0TlNPamhfbkxmeVFkSmxkNVFaVm9naXFQVlJKbGk1bHJPcWZneHY0UW9iREhranJmU0xWaGVXWi1qX2pYMTBtaUpNQnBtbEl3VE82SGNCMHlObFQ4MDktendLZjJEa2thdGtreXcyM1hPeGRfME81LUhFbHl4bjhRbEhKYkZiQzJLSHRVQmhhNzhLTy03LVdWRUlPRi00VVRGM1l1SkE="}} 
STEP: json unmarshal err <nil> 
STEP: found auth true with auth cfg len 1 
STEP: dockercfg with svrAddr 172.30.103.69:5000 user serviceaccount pass eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJleHRlbmRlZC10ZXN0LXJlZ2lzdHJ5LW1pZ3JhdGlvbi14bDl4di05Z2tsNiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJidWlsZGVyLXRva2VuLW5sNWd6Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImJ1aWxkZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJiNTliNjcwNC1mZWNkLTExZTYtYjA1Yi0wZWI5OTU2MWJkZTgiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZXh0ZW5kZWQtdGVzdC1yZWdpc3RyeS1taWdyYXRpb24teGw5eHYtOWdrbDY6YnVpbGRlciJ9.aP_xGAOor3Y63NALamnpUTLpX3QnsJcsWOJKOCnWUWFSm2Y7r0uNBHH1_u0OAwhM8aG74neV8K0KCUSPOPzKxF1AZjSongF49qOKujfHv64m6n8Wl1C5ufP_aICA1erDk5wk61d7BndeUkbV15tdQBGEm4AdtPBl2imqduQlBkowXA7DkiGmfFG4NSOjh_nLfyQdJld5QZVogiqPVRJli5lrOqfgxv4QobDHkjrfSLVheWZ-j_jX10miJMBpmlIwTO6HcB0yNlT809-zwKf2Dkkatkkyw23XOxd_0O5-HElyxn8QlHJbFbC2KHtUBha78KO-7-WVEIOF-4UTF3YuJA email [email protected] 
STEP: removing image...
STEP: Deleting images and image streams in project "extended-test-registry-migration-xl9xv-9gkl6"
[AfterEach] [Conformance][registry][migration] manifest migration from etcd to registry storage
  /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:142
STEP: Collecting events from namespace "extended-test-registry-migration-xl9xv-9gkl6".
STEP: Found 0 events.
Mar  1 17:23:39.585: INFO: POD                      NODE          PHASE    GRACE  CONDITIONS
Mar  1 17:23:39.585: INFO: docker-registry-1-vx13z  172.18.4.157  Running         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-03-01 16:59:23 -0500 EST  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-03-01 16:59:32 -0500 EST  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-03-01 16:59:23 -0500 EST  }]
Mar  1 17:23:39.585: INFO: router-2-bz6p6           172.18.4.157  Running         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-03-01 16:59:54 -0500 EST  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-03-01 17:00:14 -0500 EST  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-03-01 16:59:54 -0500 EST  }]
Mar  1 17:23:39.585: INFO: test-docker-1-build      172.18.4.157  Running         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-03-01 17:22:43 -0500 EST  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-03-01 17:22:47 -0500 EST  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-03-01 17:22:43 -0500 EST  }]
Mar  1 17:23:39.585: INFO: 
Mar  1 17:23:39.586: INFO: 
Logging node info for node 172.18.4.157
Mar  1 17:23:39.588: INFO: Node Info: &TypeMeta{Kind:,APIVersion:,}
Mar  1 17:23:39.588: INFO: 
Logging kubelet events for node 172.18.4.157
Mar  1 17:23:39.590: INFO: 
Logging pods the kubelet thinks is on node 172.18.4.157
Mar  1 17:23:39.596: INFO: docker-registry-1-vx13z started at 2017-03-01 16:59:23 -0500 EST (0+1 container statuses recorded)
Mar  1 17:23:39.596: INFO: 	Container registry ready: true, restart count 0
Mar  1 17:23:39.596: INFO: router-2-bz6p6 started at 2017-03-01 16:59:54 -0500 EST (0+1 container statuses recorded)
Mar  1 17:23:39.596: INFO: 	Container router ready: true, restart count 0
Mar  1 17:23:39.596: INFO: test-docker-1-build started at 2017-03-01 17:22:43 -0500 EST (0+1 container statuses recorded)
Mar  1 17:23:39.596: INFO: 	Container docker-build ready: true, restart count 0
Mar  1 17:23:39.906: INFO: 
Latency metrics for node 172.18.4.157
Mar  1 17:23:39.906: INFO: {Operation:sync Method:pod_worker_latency_microseconds Quantile:0.9 Latency:2m1.041255s}
Mar  1 17:23:39.906: INFO: {Operation:sync Method:pod_worker_latency_microseconds Quantile:0.99 Latency:2m1.041255s}
Mar  1 17:23:39.906: INFO: {Operation:sync Method:pod_worker_latency_microseconds Quantile:0.5 Latency:2m1.015743s}
Mar  1 17:23:39.906: INFO: {Operation: Method:pod_start_latency_microseconds Quantile:0.99 Latency:49.341606s}
Mar  1 17:23:39.906: INFO: {Operation:stop_container Method:docker_operations_latency_microseconds Quantile:0.99 Latency:30.201589s}
Mar  1 17:23:39.906: INFO: {Operation: Method:pod_start_latency_microseconds Quantile:0.9 Latency:27.722085s}
STEP: Dumping a list of prepulled images on each node
Mar  1 17:23:39.909: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "extended-test-registry-migration-xl9xv-9gkl6" for this suite.
Mar  1 17:23:44.961: INFO: namespace: extended-test-registry-migration-xl9xv-9gkl6, resource: bindings, ignored listing per whitelist
@stevekuznetsov stevekuznetsov added kind/test-flake Categorizes issue or PR as related to test flakes. priority/P1 labels Mar 2, 2017
@stevekuznetsov
Copy link
Contributor Author

@smarterclayton do we want a upstream/devicemapper label? Might help that team track outstanding issues

@smarterclayton
Copy link
Contributor

smarterclayton commented Mar 3, 2017 via email

@smarterclayton
Copy link
Contributor

@rhvgoyal do we have a docker bug for this already? if not, @stevekuznetsov can you open one and link this?

@stevekuznetsov
Copy link
Contributor Author

I can't really tell the difference between many of the devicemapper issues -- this may be #12923 in which case there is a BZ and Jhon is taking care of it ... not sure

@jwhonce
Copy link
Contributor

jwhonce commented Jun 14, 2017

@stevekuznetsov With the changes made for #12923 , have you continued to see this issue?

@stevekuznetsov
Copy link
Contributor Author

Doesn't seem like we've hit this since March 20. Reasonable to close it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependency/devicemapper kind/test-flake Categorizes issue or PR as related to test flakes. priority/P1
Projects
None yet
Development

No branches or pull requests

3 participants