"Permission denied !" on a binary that has permission !

[medyagh]

while doing this PR to add podman as a driver to minikube (which will run a systemd inside a container using podman)
kubernetes/minikube#6515

I noticed the binary files that minikube copies into the podman container, are not executable.

root@minikube:/var/lib/minikube/binaries/v1.17.2# ./kubectl
bash: ./kubectl: Permission denied

# ls -lah kubectl
-rwxrwxrwx 1 root root 42M Jan 21 18:07 kubectl

because I had other issues with podman cp command
I decided to go into the container myself and use wget to download them and then chmod +x manually but I still get permission denied !!!

this seems to be the only blocker to add run minikube in podman and I would appreciate any help on this

sudo podman exec -it minikube /bin/bash

root@minikube:/var/lib/minikube/binaries/v1.17.2# wget https://storage.googleapis.com/kubernetes-release/release/v1.17.2/bin/linux/amd64/kubectl
 41.48M   223MB/s    in 0.2s
2020-02-06 07:53:03 (223 MB/s) - 'kubectl' saved [43491328/43491328]

root@minikube:/var/lib/minikube/binaries/v1.17.2# chmod +x kubectl
root@minikube:/var/lib/minikube/binaries/v1.17.2# ./kubectl
bash: ./kubectl: Permission denied
root@minikube:/var/lib/minikube/binaries/v1.17.2# chmod a+x kubectl
root@minikube:/var/lib/minikube/binaries/v1.17.2# ./kubectl
bash: ./kubectl: Permission denied

it worth noting that minikube creates the container with --cgroup-manager cgroupfs
and --privileged and with sudo
(unlike docker it doesn't let me create my container without sudo)

versions :
$ podman version
Version: 1.7.0
RemoteAPI Version: 1
Go Version: go1.12.10
OS/Arch: linux/amd64

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 19.10
Release: 19.10

[rhatdan]

Could you check if this is an SELinux issue? setenforce 0?
Could this be seccomp blocking it?

[mheon]

Does minikube have file capabilities? That could conflict with no new privileges

[medyagh]

interestingly it only has this issue in /var/lib folder but if I move it to /usr/local/bin I can execute it !

[rhatdan]

Is /var/lib directory mounted with the noexec option?

[mheon]

Can you run mount in the container and provide the output?

[mheon]

I think this is probably a volume mount of /var or /var/lib - you may need to add the exec option onto the volume mount

[medyagh]

@mheon you are right ! for anyone having same issue
this fixed my problem

 mount -o remount,exec /var

I am just curious why same image and same container doesnt act same way on docker and podman

[mheon]

Our volume mounts are, by default, mounted noexec,nodev,nosuid for security; Docker does not do this. You can append the exec option to mounts if you need it; podman run -v /var:/var:exec for example.

[medyagh]

@mheon thank you very much for clarifying it ! it makes sense now ! I consider this issue closed for me, but I also recommend adding this to your website documentation for others