Codewind unable to deploy projects on IKS with Kube 1.15

[johnmcollier]

Codewind version: latest

IDE extension version: latest
Kubernetes cluster: IKS with Kube 1.15

Description:
I deployed hybrid Codewind onto IKS running Kube 1.15 and found that Codewind was unable to deploy Codewind-style projects it had built. The projects would fail to deploy with the following error:

Failed to pull image "<image>": rpc error: code = Unknown desc = failed to pull and unpack image "<image>": failed to unpack image on snapshotter overlayfs: failed to extract layer sha256:<hash>: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount799987480: archive/tar: invalid tar header: unknown

This has been reported against buildah on clusters running containerd 1.2 previously (see containers/buildah#1589 (comment)). Work arounds like disabling image compression on buildah push are not feasible for us since it will drastically slow the time it takes to push and deploy a project.

However, IKS clusters running Kube 1.16 (and newer) are running a newer version of containerd and are not affected by this issue. OpenShift on IBM Cloud is also not affected as it runs cri-o as its container runtime.

Since this seems to only affect older IKS versions, we should just update our docs (maybe troubleshooting?) to tell the users to upgrade to a newer version of IKS.

[sishida]

Hi, @johnmcollier, I have created a PR for this issue: eclipse-archived/codewind-docs#415 Could you please look at it and review/approve/merge if it looks good to go? Thank you!

[sishida]

Documentation complete with merging of eclipse-archived/codewind-docs#415.

[bsteinfeld]

I ran into a similar error trying to deploy the default go template.

In short when I tried to build my go project it tried to start the k8s pod but it failed to start due to: ErrImagePull.

I am running this in IKS version 1.15.10_1531.

Here are all the logs (describe go pod, pfe logs, list of serviceaccounts): https://gist.github.ibm.com/Bradley-Steinfeld/c61b76988a77ca0c8e8b5422ed37a9e3

[johnmcollier]

@bsteinfeld Yup, that's the issue you're hitting. The ErrImagePull is due to containerd being unable to extract the image that was built by buildah.

Unfortunately, there's not much I think we can do for 1.15 installs. Is moving up to Kube 1.16 (or higher) a possibility for you?

[bsteinfeld]

@johnmcollier Luckily I think we will be able to upgrade to 1.16! We are testing this right now - but so far so good.

[sishida]

Hi, team, Any updates on this issue? :)

[johnmcollier]

Closing as the doc changes were merged a while back

[micgibso]

Reopening and closing to fix issue appearing in Open list.

[micgibso]

Success, issue no longer appears in list of Open issues despite being closed.