WIP: Add ostreedev/ostree and projectatomic/rpm-ostree

[cgwalters]

Currently, these projects are using PAPR
which today uses RHT-internal OpenStack which has reliability problems.

There's no reason not for us to add some usage of Prow as an additional
CI context; all we're doing is a build, no tests.

I am strongly considering switching to Prow as a "merge bot" too,
i.e. using /lgtm etc. But this is just a first step.

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: cgwalters
To complete the pull request process, please assign bparees
You can assign the PR to them by writing /assign @bparees in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• ci-operator/config/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[cgwalters]

I think we need to add the Prow webhooks to the repository as well, will look at that.

[cgwalters]

Current status is

walters@toolbox ~/s/g/o/r/c/c/p/rpm-ostree> ci-operator --config projectatomic-rpm-ostree-master.yaml --git-ref cgwalters/rpm-ostree@prow-ci --target=src
2019/08/29 02:10:22 Resolved cgwalters/rpm-ostree@prow-ci to commit 010cfe7df45318e76567abbaa280d37c001b9185
2019/08/29 02:10:22 Resolved source https://github.com/cgwalters/rpm-ostree to prow-ci@010cfe7d
2019/08/29 02:10:23 Resolved rhcos/coreos-assembler:master to sha256:02a1e8642beafbbd4523f8c1757c8c8b83dbad19dab30ed0027c78d157d47585
2019/08/29 02:10:23 Resolved rhcos/coreos-assembler:master to sha256:02a1e8642beafbbd4523f8c1757c8c8b83dbad19dab30ed0027c78d157d47585
2019/08/29 02:10:23 Using namespace ci-op-i4blw6f2
2019/08/29 02:10:23 Running [input:root], src
2019/08/29 02:10:23 Creating namespace ci-op-i4blw6f2
2019/08/29 02:10:23 Setting a soft TTL of 1h0m0s for the namespace
2019/08/29 02:10:23 Setting a hard TTL of 12h0m0s for the namespace
2019/08/29 02:10:23 warning: Could not add annotations because you do not have permission to update the namespace (details: namespaces "ci-op-i4blw6f2" is forbidden: User "cgwalters" cannot update namespaces in the namespace "ci-op-i4blw6f2": no RBAC policy matched)
2019/08/29 02:10:23 Setting up pipeline imagestream for the test
2019/08/29 02:10:23 Tagging rhcos/coreos-assembler:master into pipeline:root
E0829 02:10:23.473940   16302 event.go:191] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:".15bf42c84f6cd048", GenerateName:"", Namespace:"ci-op-i4blw6f2", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"", Namespace:"ci-op-i4blw6f2", Name:"", UID:"", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"CiJobStarted", Message:"Running job dev for PRs () in namespace ci-op-i4blw6f2 from authors ()", Source:v1.EventSource{Component:"ci-op-i4blw6f2", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf51eae3da52fa48, ext:694546155, loc:(*time.Location)(0x2670540)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf51eae3da52fa48, ext:694546155, loc:(*time.Location)(0x2670540)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events is forbidden: User "cgwalters" cannot create events in the namespace "ci-op-i4blw6f2": no RBAC policy matched' (will not retry!)
2019/08/29 02:10:23 Building src
2019/08/29 02:11:49 Build src failed, printing logs:
Pulling image "registry.svc.ci.openshift.org/ci/clonerefs@sha256:d68e1c6c2de5c1167a79b24d5ba4f909349ca7a44fb634e214bdadc2c8b010cd" ...
Replaced Dockerfile FROM image pipeline:root

Pulling image docker-registry.default.svc:5000/ci-op-i4blw6f2/pipeline@sha256:02a1e8642beafbbd4523f8c1757c8c8b83dbad19dab30ed0027c78d157d47585 ...
Pulled 0/16 layers, 0% complete
Pulled 1/16 layers, 6% complete
Pulled 2/16 layers, 12% complete
Pulled 3/16 layers, 23% complete
Pulled 4/16 layers, 30% complete
Pulled 5/16 layers, 31% complete
Pulled 6/16 layers, 38% complete
Pulled 7/16 layers, 44% complete
Pulled 8/16 layers, 51% complete
Pulled 9/16 layers, 59% complete
Pulled 10/16 layers, 66% complete
Pulled 11/16 layers, 73% complete
Pulled 12/16 layers, 81% complete
Pulled 13/16 layers, 88% complete
Pulled 14/16 layers, 95% complete
Pulled 15/16 layers, 97% complete
Pulled 16/16 layers, 100% complete
Extracting
--> FROM docker-registry.default.svc:5000/ci-op-i4blw6f2/pipeline@sha256:02a1e8642beafbbd4523f8c1757c8c8b83dbad19dab30ed0027c78d157d47585 as 0
--> ENV "CLONEREFS_OPTIONS"="{\"src_root\":\"/go\",\"log\":\"/dev/null\",\"git_user_name\":\"ci-robot\",\"git_user_email\":\"ci-robot@openshift.io\",\"refs\":[{\"org\":\"cgwalters\",\"repo\":\"rpm-ostree\",\"base_ref\":\"prow-ci\",\"base_sha\":\"010cfe7df45318e76567abbaa280d37c001b9185\"}]}"
--> ADD ./app.binary /clonerefs
--> RUN umask 0002 && /clonerefs && find /go/src -type d -not -perm -0775 | xargs -r chmod g+xw
{"component":"clonerefs","file":"prow/pod-utils/clone/clone.go:34","func":"k8s.io/test-infra/prow/pod-utils/clone.Run","level":"info","msg":"Cloning refs","refs":{"org":"cgwalters","repo":"rpm-ostree","base_ref":"prow-ci","base_sha":"010cfe7df45318e76567abbaa280d37c001b9185"},"time":"2019-08-29T02:11:48Z"}
{"command":"mkdir -p /go/src/github.com/cgwalters/rpm-ostree","component":"clonerefs","error":"exit status 1","file":"prow/pod-utils/clone/clone.go:42","func":"k8s.io/test-infra/prow/pod-utils/clone.Run.func1","level":"info","msg":"Ran command","output":"mkdir: cannot create directory '/go': Permission denied\n","time":"2019-08-29T02:11:48Z"}
{"component":"clonerefs","file":"prow/cmd/clonerefs/main.go:43","func":"main.main","level":"info","msg":"Finished cloning refs","time":"2019-08-29T02:11:48Z"}
find: '/go/src': No such file or directory
--> WORKDIR /go/src/github.com/cgwalters/rpm-ostree/
--> ENV GOPATH=/go
--> RUN git submodule update --init
/bin/sh: line 0: cd: /go/src/github.com/cgwalters/rpm-ostree/: No such file or directory
error: build error: running 'git submodule update --init' failed with exit code 1
E0829 02:11:50.264379   16302 event.go:191] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:".15bf42dc8497273f", GenerateName:"", Namespace:"ci-op-i4blw6f2", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"", Namespace:"ci-op-i4blw6f2", Name:"", UID:"", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"CiJobFailed", Message:"Running job dev for PRs () in namespace ci-op-i4blw6f2 from authors ()", Source:v1.EventSource{Component:"ci-op-i4blw6f2", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf51eaf98de2ab3f, ext:87485859373, loc:(*time.Location)(0x2670540)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf51eaf98de2ab3f, ext:87485859373, loc:(*time.Location)(0x2670540)}}, Count:1, Type:"Warning", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events is forbidden: User "cgwalters" cannot create events in the namespace "ci-op-i4blw6f2": no RBAC policy matched' (will not retry!)
2019/08/29 02:11:51 Ran for 1m28s
error: could not run steps: step src failed: could not wait for build: the build src failed after 1m26s with reason DockerBuildFailed: Docker build strategy has failed.

--> WORKDIR /go/src/github.com/cgwalters/rpm-ostree/
--> ENV GOPATH=/go
--> RUN git submodule update --init
/bin/sh: line 0: cd: /go/src/github.com/cgwalters/rpm-ostree/: No such file or directory
error: build error: running 'git submodule update --init' failed with exit code 1

Something in the ci-operator high levle seems to be expecting golang?

[openshift-ci-robot]

@cgwalters: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/prow/generated-config	c6c0a2d	link	/test generated-config

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[cgwalters]

Oh no wait I see, it's trying to run in the context of cosa but it doesn't have the USER set and it's trying to write to /go. Hmm. I guess this is the time to make that cosa-buildroot container which would also reset USER root.

[cgwalters]

OK, I created coreos/coreos-assembler#730 which fixes that. And I built it in a new coreos namespace in api.ci but I can't actually run it in a useful way because the default SCC doesn't allow RunAsUser: 0.

So the next blocker there is https://github.com/openshift/release/issues/4892

[vrutkovs]

I can't actually run it in a useful way because the default SCC doesn't allow RunAsUser: 0.

Could this script connect to CentOS CI and start the pipeline there instead of running in on api.ci? This way we'll get prow integration and RunAsUser on CentOS CI cluster?
The hard part would be purging SA token from build logs, but I think its doable via mounting a secret with a remote cluster token

[cgwalters]

Could this script connect to CentOS CI

https://github.com/openshift/release/issues/4892

[cgwalters]

Closing this in favor of #4900