BUG 1685704: assets: use internal apiserver name for all internal clients #1633

sjenning · 2019-04-16T20:38:41Z

BZ1685704 requires that all internal client ie openshift cluster-infra clients inside the cluster that talk to the apiserver on LB needs to move to using api-int.$cluster_domain so that customers can modify the external LB URL for apiserver without affecting the internal clients.

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template

All the internal clients which includes the etcd cert agent moved to contacting the apiserver on api-int.$cluster_domain therefore the etcd signer on the bootstrap node needs to use the serving cert for the api-int

pkg/asset/kubeconfig/

The admin kubeconfig that installer provides its users needs to continue the apiserver on api.$cluster_domain

The kubelet kubeconfig is moved to use api-int.$cluster_domain as kubelets are internal clients to apiserver as kubelets are internal clients to apiserver.

pkg/asset/manifests

This change changes the .status.apiServerURL for cluster infrastructures.config.openshift.io 1 to point all internal client to use api-int.$cluster_domain for contacting apiserver.

@deads2k @abhinavdahiya @wking

see if this works now that the groundwork is laid
xref https://bugzilla.redhat.com/show_bug.cgi?id=1685704

wking · 2019-04-16T21:17:28Z

/hold

We don't want to land this until we have everyone on board (which may already be the case, but I want to be sure ;)

sjenning · 2019-04-16T21:18:40Z

also @deads2k @eparis

abhinavdahiya · 2019-04-16T21:44:08Z

you need to change the kubeconfig for kubelet to use api-int

$ rg 'https://api' ./pkg/asset
pkg/asset/kubeconfig/kubeconfig.go
34:                                     Server: fmt.Sprintf("https://api.%s:6443", installConfig.ClusterDomain()),

pkg/asset/kubeconfig/kubeconfig_test.go
65:    server: https://api.test-cluster-name.test.example.com:6443
89:    server: https://api.test-cluster-name.test.example.com:6443

pkg/asset/manifests/utils.go
37:     return fmt.Sprintf("https://api.%s:6443", ic.ClusterDomain())

abhinavdahiya · 2019-04-16T23:18:44Z

/test e2e-metal

sjenning · 2019-04-17T14:47:07Z

Interesting. All the worker kubelets are Ready but the master kubelets are Unknown

https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_installer/1633/pull-ci-openshift-installer-master-e2e-aws/5392/artifacts/e2e-aws/nodes.json

sjenning · 2019-04-17T15:21:46Z

Unfortunately the kubelet logs were not captured from the masters so it is hard to tell what happened. I'll try to recreate manually and see what is going on.

sjenning · 2019-04-17T22:56:49Z

I spent most of the day trying to figure this out to no avail. There must be some side effect I'm not intending. The only effect I intend is that the /etc/kubernetes/kubeconfig created for the nodes and an API URL that point to api-int.

Any help would be appreciated! 🙏

… API The `.status.apiServerURL` [1] states the this value can be used by components like kubelet on machines, to contact the `apisever` using the infrastructure provider rather than the kubernetes networking. Therefore, this will be set to `api-int.$cluster_domain` as that is URL that should be used by the LB consumers of apiserver. currently the external serving cert controller is using this value to generate the certificate, when it should be the one doing the replace magic from [2] and not the internal serving cert controller and which causes wrong certificate generation when we finally switch to `api-int` [3]: ```console $ oc get secrets -n openshift-kube-apiserver external-loadbalancer-serving-certkey -oyaml apiVersion: v1 data: tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURYekNDQWtlZ0F3SUJBZ0lJYzJwVFVHSWw3Mmd3RFFZSktvWklodmNOQVFFTEJRQXdOekVTTUJBR0ExVUUKQ3hNSmIzQmxibk5vYVdaME1TRXdId1lEVlFRREV4aHJkV0psTFdGd2FYTmxjblpsY2kxc1lpMXphV2R1WlhJdwpIaGNOTVRrd05ERTNNak0xT0RNeVdoY05NVGt3TlRFM01qTTFPRE16V2pBbk1TVXdJd1lEVlFRREV4eGhjR2t0CmFXNTBMbUZrWVdocGVXRXRNQzUwZEM1MFpYTjBhVzVuTUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEEKTUlJQkNnS0NBUUVBb293WHNVMW51TWRMSkhneUVWNTZXZmpTSDVCdmpqQ3c4eGJWYTVKNlRPUDhOTWVPaEtuOAowRVF6VDYzTTdQdTFtdlhqVmJ5T1JlcWZhL3Y4ZXJjdzIwS1liUDY3QXoyY2JrUXplUmpuVUVsVC8yU2hYa1E2CkVhS3Y4VGM4dlM3SFRhYkZTZVRmTW5RWHhNT0FOUDRyTW51NmpLUmw3aC90WU5Jck1xZFB3YUJaK0UyOWpBSW0KR0VaSUdCWDJWLy9Vb0hyK05vM2hHL0c1Ykdua3JhaUdUYkhTSjdQcExoNGJFYmFPTWlJWmR3K215WEtvS2ZScQo5SkVKd3llTDNueUNlQVQ1Y0tDa0NGZTR0eDQvcGMweTdQYmtHTmJzMG8yaHpMTkhHdTBNYVgwV2NYZkdObnhvCno4Y1p0S0ZGNjdaT25ZSGRtdGErVStjNWllcHZLTkJzY1FJREFRQUJvMzh3ZlRBT0JnTlZIUThCQWY4RUJBTUMKQmFBd0V3WURWUjBsQkF3d0NnWUlLd1lCQlFVSEF3RXdEQVlEVlIwVEFRSC9CQUl3QURBZkJnTlZIU01FR0RBVwpnQlNFQ09ZaFdUNDZObTdCOGQ0cHdYbEpQUXpDa3pBbkJnTlZIUkVFSURBZWdoeGhjR2t0YVc1MExtRmtZV2hwCmVXRXRNQzUwZEM1MFpYTjBhVzVuTUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFCbUVIOXQ5dGZDOXBMSVVXQUUKakE0dHBsZXpLVi9LR0ppZVh1dW5Td0NKU0FiNUcyclBMallVRGRRa0pJeks4cHdtNW1oMys2Vzd3bEo4ZVhWaAptVVJWT1RHNUlKQkkzdk1RT0hwWkt6YjY3a0ZZajVleSs3U2FmNjdKZzJ2L0lxS09QdmpVeU9VcUd6ckFWanI5Cm5FM2g3cVNwQXRYZk50Mk5IOGxuMTVKNFlsN1hhZkd6cGppUVZ2UVdBUHpoSnYxc2ZRVElpc0lSb2FvK2ZCUEcKRmtLcDUxRXRvc0xvbmtYRFZ5UGdzQVR4MW5jU2RvRE94WWhueXJteWEyT0Q2MVlMSDloalpjY2lENzBMdTVnOApFcUhrenhCL1F3S0JzbFdDUkNxYktpWkRVd0UvcWtEMmVST0NiS0hLckVKME8wWEVHVCtNRk1wU1RYcWIyaDA5CjhUV3IKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQotLS0tLUJFR0lOIENFUlRJRklDQVRFLS0tLS0KTUlJRE1qQ0NBaHFnQXdJQkFnSUliRHFHUFpCdlk0a3dEUVlKS29aSWh2Y05BUUVMQlFBd056RVNNQkFHQTFVRQpDeE1KYjNCbGJuTm9hV1owTVNFd0h3WURWUVFERXhocmRXSmxMV0Z3YVhObGNuWmxjaTFzWWkxemFXZHVaWEl3CkhoY05NVGt3TkRFM01qTTBPREExV2hjTk1qa3dOREUwTWpNME9EQTFXakEzTVJJd0VBWURWUVFMRXdsdmNHVnUKYzJocFpuUXhJVEFmQmdOVkJBTVRHR3QxWW1VdFlYQnBjMlZ5ZG1WeUxXeGlMWE5wWjI1bGNqQ0NBU0l3RFFZSgpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFPL1E0b3hYTjVvYkJ4WEptR3ZuOWkrRWpJRzJWN0RxCnFrTFFqMHU5STdnUDFMV3dEUE4rcUtFRWwzZnJjTGl2d0EvdTd5MlJ2Z0gxcDByRmFEUmhPYnBDWHU0VVN2aUYKLzVJZjl3dzFvMGlDRlNCczFmQm9GMERTNE1kMWc3cnhCVzVlTDlNVllsMGU1QzB0YkNVc3BWamkyNnR5K0dTWgpKNlRYVS9idEErV044STNnOUhQUHRZcnRLVVpycnBCUFpTWDNjZWxIWDlDczRFaDdFdXdrRFR6T2N4VGRsMUdoCjB4U1V5bE9lOU92eVVNVWM3SHpOS29QMGlRZE9scXNwL0ZvNjFwdHBOM0xUS1FCWUU5VUR2SUNFZ1NscXlzWFMKR3I1UzU0cFVMYzdva21LaXVLc0lkK0ZYWDd5STFGQTY1VVFUZDJOc2sweTFJNTF4VGp3LzN3OENBd0VBQWFOQwpNRUF3RGdZRFZSMFBBUUgvQkFRREFnS2tNQThHQTFVZEV3RUIvd1FGTUFNQkFmOHdIUVlEVlIwT0JCWUVGSVFJCjVpRlpQam8yYnNIeDNpbkJlVWs5RE1LVE1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQ1dSdVpTaURaYjQ4NzcKNmZhVWpvQTg5eld5Rm15L0VodTBURFpSSTFTSElaQjhmM1FINGlMdEhzMU53VGZDcU5XSTNOeFNxQ0Ntdmx6ZAp0ZGppVjdPUkxRZjc3WVVWdU9ZclF2M0RuK2ozcmZ1dCtoSzMyM3hjTDdYbXBpRnRIUXhac2NQZ2JqaUd1dS9XCmdhYlZBOXF3MWUxQit3b25UWlY0YWM1bjJ5N1QxbzdoeWE3WnY5RWJ5RFp3cnJPN2NBSmJjWS9pUDJnM0pnMG4KS0hJT2FVSnRZZjF0RDNCSWhzWVIzL0tQVStLU0lyQ2R3Y0ptbXYyZ1V5eGJLTzlFb2QyU2R0RDRJZEVKVTNobQpFUjhkcmFFK3Y5MzJHVHZYWWYwRUFOdHExam5ycFRxajNIVUFMajVZTm1UdE5DQTVEdHhWa2xaZ0I4RkJpdS9NCitjNmJ3UjJVCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K tls.key: .... kind: Secret metadata: annotations: auth.openshift.io/certificate-hostnames: api-int.adahiya-0.tt.testing auth.openshift.io/certificate-issuer: kube-apiserver-lb-signer auth.openshift.io/certificate-not-after: 2019-05-17T23:58:33Z auth.openshift.io/certificate-not-before: 2019-04-17T23:58:32Z creationTimestamp: 2019-04-17T23:58:37Z labels: auth.openshift.io/managed-certificate-type: target name: external-loadbalancer-serving-certkey namespace: openshift-kube-apiserver resourceVersion: "2560" selfLink: /api/v1/namespaces/openshift-kube-apiserver/secrets/external-loadbalancer-serving-certkey uid: b783479c-616c-11e9-8bba-52fdfc072182 type: kubernetes.io/tls $ xclip -sel c -o | base64 -d | openssl x509 -noout -text Certificate: Data: Version: 3 (0x2) Serial Number: 8316551266602184552 (0x736a53506225ef68) Signature Algorithm: sha256WithRSAEncryption Issuer: OU = openshift, CN = kube-apiserver-lb-signer Validity Not Before: Apr 17 23:58:32 2019 GMT Not After : May 17 23:58:33 2019 GMT Subject: CN = api-int.adahiya-0.tt.testing X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication X509v3 Basic Constraints: critical CA:FALSE X509v3 Authority Key Identifier: keyid:84:08:E6:21:59:3E:3A:36:6E:C1:F1:DE:29:C1:79:49:3D:0C:C2:93 X509v3 Subject Alternative Name: DNS:api-int.adahiya-0.tt.testing ``` This commit moves the replace magic [2] from internal to external serving cert controller. [1]: https://github.com/openshift/api/blob/13b403bfb6ce84ddc053bd3b401b5d67bf175efa/config/v1/types_infrastructure.go#L55-L58 [2]: openshift#405 [3]: openshift/installer#1633

sjenning · 2019-04-18T01:22:19Z

making the changes @abhinavdahiya suggested and now my etcd-members won't come up, stuck in the certs init container with

agent.go:116] error sending CSR to signer: Post https://api-int.lab.variantweb.net:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: x509: certificate is valid for api.lab.variantweb.net, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, localhost, not api-int.lab.variantweb.net

sounds like the bootstrap apiserver is not configured with the cert for api-int. fyi @deads2k

sjenning · 2019-04-18T01:31:17Z

from /opt/openshift/kube-apiserver-bootstrap/config on the bootstrap node

  namedCertificates:
  - certFile: /etc/kubernetes/secrets/kube-apiserver-service-network-server.crt
    keyFile: /etc/kubernetes/secrets/kube-apiserver-service-network-server.key
    names:
    - kubernetes
    - kubernetes.default
    - kubernetes.default.svc
    - kubernetes.default.svc.cluster.local
  - certFile: /etc/kubernetes/secrets/kube-apiserver-localhost-server.crt
    keyFile: /etc/kubernetes/secrets/kube-apiserver-localhost-server.key
    names:
    - localhost
    - 127.0.0.1
    - ::1
  - certFile: /etc/kubernetes/secrets/kube-apiserver-lb-server.crt
    keyFile: /etc/kubernetes/secrets/kube-apiserver-lb-server.key
  - certFile: /etc/kubernetes/secrets/kube-apiserver-internal-lb-server.crt
    keyFile: /etc/kubernetes/secrets/kube-apiserver-internal-lb-server.key

sjenning · 2019-04-18T01:40:20Z

gah, this isn't the real apiserver (since there is no etcd yet). it is the kube-etcd-signer-server.

# openssl x509 -in /opt/openshift/tls/apiserver.crt -noout -text
Certificate:
    Data:
...
        Issuer: OU = bootkube, CN = kube-ca
...
        Subject: O = kube-master, CN = system:kube-apiserver
...
            X509v3 Subject Alternative Name: 
                DNS:api.lab.variantweb.net, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:localhost, IP Address:172.30.0.1, IP Address:127.0.0.1
...

sjenning · 2019-04-18T13:49:55Z

rebased due to collision with #1640

sjenning · 2019-04-18T15:50:14Z

/retest

sjenning · 2019-04-18T16:50:29Z

i'm seeing this locally now

Apr 18 16:30:56 bootstrap bootkube.sh[1410]: All self-hosted control plane components successfully started
Apr 18 16:30:56 bootstrap bootkube.sh[1410]: Sending bootstrap-success event.Tearing down temporary bootstrap control plane...
Apr 18 16:30:56 bootstrap bootkube.sh[1410]: Waiting for remaining assets to be created.
Apr 18 16:31:56 bootstrap bootkube.sh[1410]: Failed to create /v1, Resource=namespaces: Post https://api.lab.variantweb.net:6443/api/v1/namespaces: x509: certificate is valid for kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, 172.30.0.1, not api.lab.variantweb.net
Apr 18 16:31:56 bootstrap bootkube.sh[1410]: Failed to create apiextensions.k8s.io/v1beta1, Resource=customresourcedefinitions: Post https://api.lab.variantweb.net:6443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions: x509: certificate is valid for kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, 172.30.0.1, not api.lab.variantweb.net
Apr 18 16:31:56 bootstrap bootkube.sh[1410]: Failed to create apiextensions.k8s.io/v1beta1, Resource=customresourcedefinitions: Post https://api.lab.variantweb.net:6443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions: x509: certificate is valid for kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, 172.30.0.1, not api.lab.variantweb.net

self-hosted apiserver doesn't have a cert for api. external LB name
@abhinavdahiya @deads2k ?

openshift-ci-robot · 2019-04-18T21:48:03Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya, sjenning

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [abhinavdahiya]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

abhinavdahiya · 2019-04-18T21:56:13Z

/hold cancel

abhinavdahiya · 2019-04-18T22:03:31Z

/hold

Looks like we want to merge this after beta4 ships ie after monday

abhinavdahiya · 2019-04-22T16:50:56Z

/hold

Looks like we want to merge this after beta4 ships ie after monday

/hold cancel

we are ready to merge this.

openshift-ci-robot · 2019-04-22T17:57:35Z

@sjenning: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/prow/e2e-metal	371283200c584b13fd101f6ed3d0bcc721f86a78	link	`/test e2e-metal`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-bot · 2019-04-22T18:17:42Z

/retest

Please review the full test history for this PR and help us cut down flakes.

Currently the etcd-signer-server does not support serving TLS traffic based on SNI. Therefore, since openshift's kube-apiserver serves traffic on api.$cluster_domain and api-int.$cluster_domain [1] and because during bootstrapping when etcd-signer-server is mimic-ing the kube-apiserver to sign the etcd clients certificates it can only serve traffic on single domain, external clients trying to connect to `:6443` from api.$cluster_domain see errors like, ```console time="2019-04-24T13:25:11-07:00" level=debug msg="Still waiting for the Kubernetes API: Get https://api.adahiya-0.tt.testing:6443/version?timeout=32s: x509: certificate is valid for api-int.adahiya-0.tt.testing, not api.adahiya-0.tt.testing" ``` as the etcd-signer-server is using certs for `api-int.$cluster_domain` as internal clients ie etcd agent is contacting it on that domain, the external clients ie the installer hits the etcd-signer on `api.$cluster_domain` Allowing etcd-signer-server to accept multiple certs and serve TLS based on SNI allows it to correctly mimic the kube-apiserver's capability. [1]: openshift/installer#1633

…-int Catching up with 13e4b70 (data/aws: create an api-int dns name, 2019-04-11, openshift#1601) and 052fcee (asset/manifests: use internal apiserver name, 2019-04-17, openshift#1633).

Catching up with 13e4b70 (data/aws: create an api-int dns name, 2019-04-11, openshift#1601), now that 052fcee (asset/manifests: use internal apiserver name, 2019-04-17, openshift#1633) has moved some internal assets over to that name.

Add the hosts plugin to coredns so that we can create a static entry for api-int.$CLUSTER_DOMAIN. hosts is used because it doesn't have to be authoritative for the zone and can allow fallthrough of records that are not found. For more details, see https://bugzilla.redhat.com/show_bug.cgi?id=1685704 and openshift/installer#1633

openshift-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Apr 16, 2019

openshift-ci-robot requested review from flaper87 and staebler April 16, 2019 20:38

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 16, 2019

abhinavdahiya mentioned this pull request Apr 18, 2019

certoperator: fix incorrect use of infrastructure.config.openshift.io API openshift/cluster-kube-apiserver-operator#419

Merged

sjenning force-pushed the use-internal-apiserver-name branch 2 times, most recently from aff9253 to 668f56b Compare April 18, 2019 00:56

sjenning force-pushed the use-internal-apiserver-name branch from 668f56b to 20464f8 Compare April 18, 2019 01:49

openshift-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Apr 18, 2019

sjenning force-pushed the use-internal-apiserver-name branch from 20464f8 to 5e6c5e9 Compare April 18, 2019 02:27

openshift-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Apr 18, 2019

sjenning force-pushed the use-internal-apiserver-name branch from 5e6c5e9 to c8c463c Compare April 18, 2019 04:15

asset/manifests: use internal apiserver name

052fcee

sjenning force-pushed the use-internal-apiserver-name branch from c8c463c to 052fcee Compare April 18, 2019 13:49

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Apr 18, 2019

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 18, 2019

abhinavdahiya changed the title ~~asset/manifests: use internal apiserver name~~ BUG 1685704: assets: use internal apiserver name for all internal clients Apr 18, 2019

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 18, 2019

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 18, 2019

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 22, 2019

openshift-merge-robot merged commit f1f2bbf into openshift:master Apr 22, 2019

deads2k mentioned this pull request Apr 23, 2019

switch to internal hostname openshift/cluster-kube-apiserver-operator#432

Merged

abhinavdahiya mentioned this pull request Apr 24, 2019

etcd-signer-server: allow server to serve TLS based on SNI coreos/kubecsr#28

Merged

wking mentioned this pull request Apr 30, 2019

pkg/asset/ignition/machine/node: Move pointer Ignition configs to api-int #1693

Merged

This was referenced May 15, 2019

Add api-int DNS record to coredns openshift-metal3/dev-scripts#450

Merged

Add api-int DNS entry to dnsmasq openshift-metal3/dev-scripts#462

Closed

BUG 1685704: assets: use internal apiserver name for all internal clients #1633

BUG 1685704: assets: use internal apiserver name for all internal clients #1633

Uh oh!

Conversation

sjenning commented Apr 16, 2019 • edited by abhinavdahiya Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wking commented Apr 16, 2019

Uh oh!

sjenning commented Apr 16, 2019

Uh oh!

abhinavdahiya commented Apr 16, 2019

Uh oh!

abhinavdahiya commented Apr 16, 2019

Uh oh!

sjenning commented Apr 17, 2019

Uh oh!

sjenning commented Apr 17, 2019

Uh oh!

sjenning commented Apr 17, 2019

Uh oh!

sjenning commented Apr 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sjenning commented Apr 18, 2019

Uh oh!

sjenning commented Apr 18, 2019

Uh oh!

sjenning commented Apr 18, 2019

Uh oh!

sjenning commented Apr 18, 2019

Uh oh!

sjenning commented Apr 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Apr 18, 2019

Uh oh!

abhinavdahiya commented Apr 18, 2019

Uh oh!

abhinavdahiya commented Apr 18, 2019

Uh oh!

abhinavdahiya commented Apr 22, 2019

Uh oh!

openshift-ci-robot commented Apr 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-bot commented Apr 22, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

sjenning commented Apr 16, 2019 •

edited by abhinavdahiya

Loading

sjenning commented Apr 18, 2019 •

edited

Loading

sjenning commented Apr 18, 2019 •

edited

Loading

openshift-ci-robot commented Apr 22, 2019 •

edited

Loading