Unable to start with podman driver on rpi 4 - panic: runtime error: index out of range [1] with length 1

[htjain]

Steps to reproduce the issue:

1. Install podman
2. run minikube --driver=podman start
3. crash

Full output of minikube logs command:

Running on machine: raspberrypi Binary: Built with gc go1.16.4 for linux/arm64 Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg I0901 18:23:35.384852 44520 out.go:286] Setting OutFile to fd 1 ... I0901 18:23:35.385675 44520 out.go:338] isatty.IsTerminal(1) = true I0901 18:23:35.385696 44520 out.go:299] Setting ErrFile to fd 2... I0901 18:23:35.385713 44520 out.go:338] isatty.IsTerminal(2) = true I0901 18:23:35.386423 44520 root.go:312] Updating PATH: /home/pi/.minikube/bin W0901 18:23:35.386932 44520 root.go:291] Error reading config file at /home/pi/.minikube/config/config.json: open /home/pi/.minikube/config/config.json: no such file or directory I0901 18:23:35.387180 44520 out.go:286] Setting OutFile to fd 1 ... I0901 18:23:35.387225 44520 out.go:338] isatty.IsTerminal(1) = true I0901 18:23:35.391133 44520 out.go:165] I0901 18:23:35.393467 44520 out.go:165] ==> Audit <== I0901 18:23:35.397192 44520 out.go:165] |---------|------|---------|------|---------|------------|----------| | Command | Args | Profile | User | Version | Start Time | End Time | |---------|------|---------|------|---------|------------|----------| |---------|------|---------|------|---------|------------|----------|

I0901 18:23:35.399582 44520 out.go:165]
I0901 18:23:35.401831 44520 out.go:165] ==> Last Start <==
I0901 18:23:35.404326 44520 out.go:165] Last start log file not found at /home/pi/.minikube/logs/lastStart.txt
I0901 18:23:35.406694 44520 out.go:165]
I0901 18:23:35.406783 44520 out.go:286] Setting OutFile to fd 1 ...
I0901 18:23:35.406812 44520 out.go:338] isatty.IsTerminal(1) = true
I0901 18:23:35.407047 44520 mustload.go:65] Loading cluster: minikube
I0901 18:23:35.409465 44520 cli_runner.go:115] Run: sudo -n podman container inspect minikube --format={{.State.Status}}
W0901 18:23:35.563843 44520 cli_runner.go:162] sudo -n podman container inspect minikube --format={{.State.Status}} returned with exit code 125
I0901 18:23:35.568694 44520 out.go:165]
W0901 18:23:35.569027 44520 out.go:230] ❌ Exiting due to GUEST_STATUS: state: unknown state "minikube": sudo -n podman container inspect minikube --format=: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container minikube

W0901 18:23:35.569087 44520 out.go:230]

Full output of failed command:

$ minikube --driver=podman start 😄 minikube v1.22.0 on Raspbian 11.0 (arm64) ✨ Using the podman driver based on user configuration 👍 Starting control plane node minikube in cluster minikube 🚜 Pulling base image ... 💾 Downloading Kubernetes v1.21.2 preload ... > preloaded-images-k8s-v11-v1...: 522.45 MiB / 522.45 MiB 100.00% 7.59 MiB > gcr.io/k8s-minikube/kicbase...: 326.19 MiB / 326.19 MiB 100.00% 3.36 MiB E0901 18:21:21.546806 43811 cache.go:200] Error downloading kic artifacts: not yet implemented, see issue #8426 🔥 Creating podman container (CPUs=2, Memory=2200MB) ...- panic: runtime error: index out of range [1] with length 1

goroutine 300 [running]:
k8s.io/minikube/pkg/drivers/kic/oci.podmanNetworkInspect(0x1a77125, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1fdab68, ...)
/app/pkg/drivers/kic/oci/network_create.go:242 +0x5c4
k8s.io/minikube/pkg/drivers/kic/oci.containerNetworkInspect(0x4001ba9110, 0x6, 0x1a77125, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/app/pkg/drivers/kic/oci/network_create.go:161 +0x138
k8s.io/minikube/pkg/drivers/kic/oci.CreateNetwork(0x4001ba9110, 0x6, 0x4001ba90d8, 0x8, 0x2, 0x4022a0fce4, 0x6, 0x1be2c, 0x2e637e8)
/app/pkg/drivers/kic/oci/network_create.go:73 +0x108
k8s.io/minikube/pkg/drivers/kic.(*Driver).Create(0x4000f43e00, 0x18, 0x40001a2db0)
/app/pkg/drivers/kic/kic.go:95 +0x37c
k8s.io/minikube/pkg/minikube/machine.(*LocalClient).Create(0x4000572e80, 0x4000d213e0, 0x0, 0x0)
/app/pkg/minikube/machine/client.go:242 +0x418
k8s.io/minikube/pkg/minikube/machine.timedCreateHost.func2(0x2037598, 0x4000572e80, 0x4000d213e0, 0x4000f0e7c0, 0x4000c6cbd0)
/app/pkg/minikube/machine/start.go:193 +0x38
created by k8s.io/minikube/pkg/minikube/machine.timedCreateHost
/app/pkg/minikube/machine/start.go:192 +0xd4

System details:
OS: Debian GNU/Linux 11 (bullseye) ARM 64 bit
Podman version: 3.0.1
minikube version: v1.22.0

I checked existing issues but nothing helped.

[afbjorklund]

Looks like an issue with creating the private podman network, when running on arm64... Apparently the real error is not logged properly, and then we get an empty list of networks

[htjain]

Looks like somehow minikube container is not getting created. Any pointers how to debug it further?

Error: error inspecting object: no such container minikube

[ayeks]

I get the same error for:

OS: Ubuntu 21.04 5.11.0-31-generic x86_64
Podman version: 3.0.1
minikube version: v1.23.0

Full output:

$ minikube start
😄  minikube v1.23.0 on Ubuntu 21.04
✨  Automatically selected the podman driver
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
E0907 22:32:25.737877   61543 cache.go:200] Error downloading kic artifacts:  not yet implemented, see issue #8426
🔥  Creating podman container (CPUs=2, Memory=3900MB) ...| panic: runtime error: index out of range [1] with length 1

goroutine 232 [running]:
k8s.io/minikube/pkg/drivers/kic/oci.podmanNetworkInspect({0x1e8561f, 0x6})
	/app/pkg/drivers/kic/oci/network_create.go:242 +0x7f0
k8s.io/minikube/pkg/drivers/kic/oci.containerNetworkInspect({0xc000c4dc80, 0x418946}, {0x1e8561f, 0x0})
	/app/pkg/drivers/kic/oci/network_create.go:161 +0x19e
k8s.io/minikube/pkg/drivers/kic/oci.CreateNetwork({0xc000c4dc80, 0x6}, {0xc000c4dc50, 0x1e808d6})
	/app/pkg/drivers/kic/oci/network_create.go:73 +0x1ed
k8s.io/minikube/pkg/drivers/kic.(*Driver).Create(0xc00042c140)
	/app/pkg/drivers/kic/kic.go:95 +0x496
k8s.io/minikube/pkg/minikube/machine.(*LocalClient).Create(0xc000f4e180, 0xc000c83620)
	/app/pkg/minikube/machine/client.go:242 +0x5f2
k8s.io/minikube/pkg/minikube/machine.timedCreateHost.func2()
	/app/pkg/minikube/machine/start.go:193 +0x38
created by k8s.io/minikube/pkg/minikube/machine.timedCreateHost
	/app/pkg/minikube/machine/start.go:192 +0x17d

The referenced issue does not really help: #8426

[spowelljr]

Hi @htjain, this PR should fix your issue #12756

[htjain]

Thanks, its working now.