Skip to content

WIP: Symlink authfile with private pause #537

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
cgwalters wants to merge 2 commits into from
Closed

WIP: Symlink authfile with private pause #537

cgwalters wants to merge 2 commits into from

Conversation

@cgwalters
Copy link
Member

@cgwalters cgwalters commented Mar 8, 2019 •
edited
Loading

Additional hack to test #535

@openshift-ci-robot openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 8, 2019
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Mar 8, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cgwalters

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 8, 2019
@openshift-ci-robot openshift-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Mar 8, 2019
@cgwalters
Copy link
Member Author

cgwalters commented Mar 8, 2019

/hold

@openshift-ci-robot openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 8, 2019
@cgwalters cgwalters mentioned this pull request Mar 8, 2019
Merged
@cgwalters
controller: Symlink /root/.docker/config.json to kubelet auth
03af0e8 
This is a workaround for us switching to an authenticated pause
image and crio not knowing how to use the kubelet's auth for this.

https://bugzilla.redhat.com/show_bug.cgi?id=1686556
@cgwalters cgwalters force-pushed the symlink-authfile-with-private-pause branch from b4ca9c8 to 99eaeb0 Compare March 8, 2019 19:30
@cgwalters
Copy link
Member Author

cgwalters commented Mar 8, 2019

/retest e2e-aws-op

@cgwalters
Copy link
Member Author

cgwalters commented Mar 8, 2019

/test e2e-aws-op

@cgwalters
Copy link
Member Author

cgwalters commented Mar 8, 2019

Hum. This seems to be failing for a reason I don't understand:

Mar 08 20:51:33 osiris-s7ks9-master-0 hyperkube[3680]: E0308 20:51:33.465332    3680 pod_workers.go:186] Error syncing pod 041392a926382976036e2d579c30316e ("etcd-member-osiris-s7ks9-master-0_kube-system(041392a926382976036e2d579c30316e)"), skipping: failed to "CreatePodSandbox" for "etcd-member-osiris-s7ks9-master-0_kube-system(041392a926382976036e2d579c30316e)" with CreatePodSandboxError: "CreatePodSandbox for pod \"etcd-member-osiris-s7ks9-master-0_kube-system(041392a926382976036e2d579c30316e)\" failed: rpc error: code = Unknown desc = error creating pod sandbox with name \"k8s_etcd-member-osiris-s7ks9-master-0_kube-system_041392a926382976036e2d579c30316e_0\": empty trailing digest or ID in \"@\": invalid reference"
Mar 08 20:51:33 osiris-s7ks9-master-0 hyperkube[3680]: E0308 20:51:33.465242    3680 kuberuntime_sandbox.go:65] CreatePodSandbox for pod "etcd-member-osiris-s7ks9-master-0_kube-system(041392a926382976036e2d579c30316e)" failed: rpc error: code = Unknown desc = error creating pod sandbox with name "k8s_etcd-member-osiris-s7ks9-master-0_kube-system_041392a926382976036e2d579c30316e_0": empty trailing digest or ID in "@": invalid reference

[root@osiris-s7ks9-master-0 ~]# grep -i pause /etc/crio/crio.conf 
# pause_image is the image which we use to instantiate infra containers.
pause_image = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:810ded5c25b9ec252dba6a2497d1eff9ad13a19cc3ac290ef8943b7d658803f"
# pause_command is the command to run in a pause_image to have a container just
pause_command = "/usr/bin/pod"

@cgwalters
Copy link
Member Author

cgwalters commented Mar 8, 2019

More logs with crio debug:

Mar 08 21:11:20 osiris-s7ks9-master-0 hyperkube[3680]: E0308 21:11:20.422284    3680 pod_workers.go:186] Error syncing pod 041392a926382976036e2d579c30316e ("etcd-member-osiris-s7ks9-master-0_kube-system(041392a926382976036e2d579c30316e)"), skipping: failed to "CreatePodSandbox" for "etcd-member-osiris-s7ks9-master-0_kube-system(041392a926382976036e2d579c30316e)" with CreatePodSandboxError: "CreatePodSandbox for pod \"etcd-member-osiris-s7ks9-master-0_kube-system(041392a926382976036e2d579c30316e)\" failed: rpc error: code = Unknown desc = error creating pod sandbox with name \"k8s_etcd-member-osiris-s7ks9-master-0_kube-system_041392a926382976036e2d579c30316e_0\": empty trailing digest or ID in \"@\": invalid reference"
Mar 08 21:11:20 osiris-s7ks9-master-0 hyperkube[3680]: E0308 21:11:20.422190    3680 kuberuntime_manager.go:660] createPodSandbox for pod "etcd-member-osiris-s7ks9-master-0_kube-system(041392a926382976036e2d579c30316e)" failed: rpc error: code = Unknown desc = error creating pod sandbox with name "k8s_etcd-member-osiris-s7ks9-master-0_kube-system_041392a926382976036e2d579c30316e_0": empty trailing digest or ID in "@": invalid reference
Mar 08 21:11:20 osiris-s7ks9-master-0 hyperkube[3680]: E0308 21:11:20.422154    3680 kuberuntime_sandbox.go:65] CreatePodSandbox for pod "etcd-member-osiris-s7ks9-master-0_kube-system(041392a926382976036e2d579c30316e)" failed: rpc error: code = Unknown desc = error creating pod sandbox with name "k8s_etcd-member-osiris-s7ks9-master-0_kube-system_041392a926382976036e2d579c30316e_0": empty trailing digest or ID in "@": invalid reference
Mar 08 21:11:20 osiris-s7ks9-master-0 hyperkube[3680]: E0308 21:11:20.422080    3680 remote_runtime.go:96] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = error creating pod sandbox with name "k8s_etcd-member-osiris-s7ks9-master-0_kube-system_041392a926382976036e2d579c30316e_0": empty trailing digest or ID in "@": invalid reference
Mar 08 21:11:20 osiris-s7ks9-master-0 crio[4372]: time="2019-03-08 21:11:20.421480183Z" level=info msg="Attempting to run pod sandbox with infra container: kube-system/etcd-member-osiris-s7ks9-master-0/POD"
Mar 08 21:11:20 osiris-s7ks9-master-0 crio[4372]: time="2019-03-08 21:11:20.421395323Z" level=debug msg="RunPodSandboxRequest &RunPodSandboxRequest{Config:&PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:etcd-member-osiris-s7ks9-master-0,Uid:041392a926382976036e2d579c30316e,Namespace:kube-system,Attempt:0,},Hostname:,LogDirectory:/var/log/pods/041392a926382976036e2d579c30316e,DnsConfig:&DNSConfig{Servers:[192.168.126.1],Searches:[osiris.verbum.local],Options:[],},PortMappings:[&PortMapping{Protocol:TCP,ContainerPort:2380,HostPort:2380,HostIp:,} &PortMapping{Protocol:TCP,ContainerPort:2379,HostPort:2379,HostIp:,}],Labels:map[string]string{io.kubernetes.pod.name: etcd-member-osiris-s7ks9-master-0,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 041392a926382976036e2d579c30316e,k8s-app: etcd,},Annotations:map[string]string{kubernetes.io/config.hash: 041392a926382976036e2d579c30316e,kubernetes.io/config.seen: 2019-03-08T20:51:32.397228524Z,kubernetes.io/config.source: file,},Linux:&LinuxPodSandboxConfig{CgroupParent:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041392a926382976036e2d579c30316e.slice,SecurityContext:&LinuxSandboxSecurityContext{NamespaceOptions:&NamespaceOption{Network:NODE,Pid:CONTAINER,Ipc:POD,},SelinuxOptions:nil,RunAsUser:nil,ReadonlyRootfs:false,SupplementalGroups:[],Privileged:false,SeccompProfilePath:,RunAsGroup:nil,},Sysctls:map[string]string{},},},RuntimeHandler:,}"

@cgwalters cgwalters force-pushed the symlink-authfile-with-private-pause branch from 99eaeb0 to 0bdee52 Compare March 8, 2019 21:16
@cgwalters
Copy link
Member Author

cgwalters commented Mar 8, 2019 •
edited
Loading

Miloslav spotted that the checksum was truncated here; copy-paste error when I extracted it from the release image. Let's take another swing at this!

@cgwalters
Copy link
Member Author

cgwalters commented Mar 8, 2019

Cluster formed:

$ oc -c setup logs e2e-aws-op 
level=warning msg="Found override for ReleaseImage. Please be warned, this is not advised"
level=info msg="Consuming \"Install Config\" from target directory"
level=info msg="Creating cluster..."
level=info msg="Waiting up to 30m0s for the Kubernetes API..."
level=info msg="API v1.12.4+8284053 up"
level=info msg="Waiting up to 30m0s for the bootstrap-complete event..."
level=info msg="Destroying the bootstrap resources..."
level=info msg="Waiting up to 30m0s for the cluster to initialize..."

@cgwalters
Copy link
Member Author

cgwalters commented Mar 8, 2019 

$ oc debug node/ip-10-0-134-235.ec2.internal                                                                                                                                                                      
Starting pod/ip-10-0-134-235ec2internal-debug ...                                                                                                                                                                  
To directly access the host PATH, try `chroot /host /bin/bash`                                                                                                                                                    
If you don't see a command prompt, try pressing enter.                                                                                                                                                             
sh-4.2# chroot /host                                                                                                                                                                                                                                                                                                                                                                                            
sh-4.2# ls -al /root/.docker/                                                                                                                                                                                     
total 0                                                                                                                                                                                                           
drwxr-xr-x. 2 root root  25 Mar  8 21:33 .                                                                                                                                                                        
drwx------. 3 root root 123 Mar  8 21:33 ..                                                                                                                                                                       
lrwxrwxrwx. 1 root root  28 Mar  8 21:33 config.json -> /var/lib/kubelet/config.json      
sh-4.2# podman images| grep quay
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256        810ded5c25b9ec252dba6a2497d1eff9ad13a19cc3ac290ef8943b7d658803f2   68487363bd9a   4 days ago          236 MB                                         
quay.io/coreos/kube-client-agent                             36c62ccd7b16b522450c61e96fc556b217ee24f5                           b02de22ff740   4 months ago        33.8 MB

This looks good to me!

@cgwalters cgwalters force-pushed the symlink-authfile-with-private-pause branch from 0bdee52 to bbf8139 Compare March 8, 2019 22:00
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Mar 9, 2019

@cgwalters: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
ci/prow/e2e-aws bbf8139 link /test e2e-aws

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@runcom
Copy link
Member

runcom commented Mar 9, 2019

Awesome! Still WIP?

@cgwalters
Copy link
Member Author

cgwalters commented Mar 9, 2019

Awesome! Still WIP?

This is just a testing PR for #535

Closing this one, let's get the other one merged!

@cgwalters cgwalters closed this Mar 9, 2019
@cgwalters cgwalters mentioned this pull request Mar 12, 2019
Merged
@mtrmac mtrmac mentioned this pull request Mar 12, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jlebon jlebon Awaiting requested review from jlebon

@kikisdeliveryservice kikisdeliveryservice Awaiting requested review from kikisdeliveryservice

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@cgwalters @openshift-ci-robot @runcom