Building with podman-remote hangs

[afbjorklund]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Trying to do a podman build over podman-remote, and the connection gets stuck.

This happens with both 1.9.0 and 1.9.1, but works OK with 1.8.2. On Ubuntu 19.10

The workaround is running "sudo podman build" remotely (over ssh), which works.

But that means that the user needs to transport the entire build context themselves.

From kubernetes/minikube#8063

Tried to debug, but not sure how ?

Steps to reproduce the issue:

1. minikube start --driver podman

2. eval $(minikube podman-env)

3. podman-remote build .

Describe the results you received:

 podman-remote build .
^C

Describe the results you expected:

 podman-remote build .
STEP 1: FROM busybox
STEP 2: RUN true
--> Using cache 98bc298a4242e40d67a4f37d2a2d37f2cf7ab3b71e1499a67a278bd131e20398
98bc298a4242e40d67a4f37d2a2d37f2cf7ab3b71e1499a67a278bd131e20398

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:            1.9.1
RemoteAPI Version:  1
Go Version:         go1.12.10
OS/Arch:            linux/amd64

Output of podman info --debug:

Package info (e.g. output of rpm -q podman or apt list podman):

podman/unknown,now 1.9.1~1 amd64 [installed]

Additional environment details (AWS, VirtualBox, physical, etc.):

[mheon]

@baude PTAL

[baude]

can you provide the dockerfile in question?

[afbjorklund]

Here it is:

FROM busybox
RUN true

[afbjorklund]

Thanks @baude:

I tried upgrading (to 904ee39) and it seems to have fixed the first issue.

But it now seems to hang on the second step, and eventually time-out:

STEP 1: FROM busybox
Getting image source signatures
Copying blob sha256:d9cbbca60e5f0fc028b13c01e70e96cc2eefe7858ccc5a33c9a8faa8b03286aa
Copying config sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42
Writing manifest to image destination
Storing signatures
STEP 2: RUN true
Error: unexpected EOF

Is there something similar needed, for other parts of the protocol too ?

Client:
Version:            1.9.1
RemoteAPI Version:  1
Go Version:         go1.13.8
Git Commit:         37a3aa6c83603ec774939a8d07c950dd3b26110a
Built:              Sun May 10 19:03:51 2020
OS/Arch:            linux/amd64

Server:
Version:            1.9.1
RemoteAPI Version:  1
Go Version:         go1.13.8
Git Commit:         904ee392f1cdf4368d8554b149415906dbb25ba0
Built:              Mon May 18 22:18:08 2020
OS/Arch:            linux/amd64

[afbjorklund]

Here is how it looks with 1.8.2: (i.e. when not cached, same VM as above)

STEP 1: FROM busybox
Getting image source signatures
Copying blob sha256:d9cbbca60e5f0fc028b13c01e70e96cc2eefe7858ccc5a33c9a8faa8b03286aa
Copying config sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42
Writing manifest to image destination
Storing signatures
STEP 2: RUN true
STEP 3: COMMIT
--> 3797f36bb81
3797f36bb8166a81a8335c925512148d0994f17437bf815a24bddc949297ead5

[baude]

@afbjorklund it finished for me ... ill try multiple times.

[baude]

STEP 1: FROM busybox
STEP 2: RUN true
--> Using cache 8050b40a3a397abc62ccaac3b2f89b8fed96f71ce6d8aa8bb8fe83ac443ff64b
STEP 3: COMMIT foo
--> 8050b40a3a3
8050b40a3a397abc62ccaac3b2f89b8fed96f71ce6d8aa8bb8fe83ac443ff64b

[baude]

Version:            1.9.1
RemoteAPI Version:  1
Go Version:         go1.14.2
Git Commit:         904ee392f1cdf4368d8554b149415906dbb25ba0
Built:              Mon May 18 15:38:09 2020
OS/Arch:            linux/amd64

Server:
Version:            1.9.1
RemoteAPI Version:  1
Go Version:         go1.14.2
Git Commit:         904ee392f1cdf4368d8554b149415906dbb25ba0
Built:              Mon May 18 15:38:01 2020
OS/Arch:            linux/amd64

[afbjorklund]

The environment is still a bit shaky in general, it's not unheard of to get timeouts and such.

ssh: exit status 255

It is running a privileged container (in podman), and in this one it runs the varlink bridge...

Both started with sudo.

---

I don't think the VM-based environment has similar issues, then again it still runs 1.8.2.

So I will check if we have the same problems in VirtualBox too, after upgrading to 1.9.x

[afbjorklund]

Upgraded to Ubuntu 20.04 and podman 1.9.2 (from the opensuse kubic repositories), same thing.

So most likely we will downgrade this to 1.8.2 for the release. Will try to rebuild the source package.

UPDATE: Was fixed in 1.9.3

See kubernetes/minikube#8063 (comment)

[rhatdan]

@afbjorklund Have you had a chance to try this with podman v2?

[afbjorklund]

@rhatdan : Haven't tried 2.0 RC yet, but it works OK with podman 1.9.3