Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

limactl start template://docker no longer boots (virtio-serial seems intermittently breaking virtio-net-pci) #1689

Closed
AkihiroSuda opened this issue Jul 28, 2023 · 5 comments · Fixed by #1693
Closed

limactl start template://docker no longer boots (virtio-serial seems intermittently breaking virtio-net-pci) #1689

AkihiroSuda opened this issue Jul 28, 2023 · 5 comments · Fixed by #1693
Labels
component/dns component/qemu QEMU priority/high

Comments

@AkihiroSuda
Copy link
Member

AkihiroSuda commented Jul 28, 2023 •
edited
Loading

  • host: macOS 13.5 (Intel)
  • version: master (08bece2)
  • QEMU: 8.0.3 (Homebrew)
  • /var/log/cloud-init-output.log in the guest:
...
LIMA| Executing /mnt/lima-cidata/boot/30-install-packages.sh
+ INSTALL_IPTABLES=0
+ [  = 1 ]
+ [  = 1 ]
+ [ 58690 -ne 0 ]
+ INSTALL_IPTABLES=1
+ [ -d /mnt/lima-cidata/provision.dependency ]
+ [  = 1 ]
+ command -v apt-get
+ hexdump -C -n 4 /usr/bin/apt-get
+ grep -qF ELF
+ pkgs=
+ [ reverse-sshfs = reverse-sshfs ]
+ [ 2 -gt 0 ]
+ command -v sshfs
+ pkgs= sshfs
+ [ 1 = 1 ]
+ [ ! -e /usr/sbin/iptables ]
+ [  = 1 ]
+ [ -n  sshfs ]
+ DEBIAN_FRONTEND=noninteractive
+ export DEBIAN_FRONTEND
+ apt-get update
Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease
Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [108 kB]
Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [14.1 MB]
Get:6 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [634 kB]
Get:7 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [149 kB]
Get:8 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.0 kB]
Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [656 kB]
Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [104 kB]
Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [532 B]
Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [764 kB]
Get:13 http://archive.ubuntu.com/ubuntu jammy/universe Translation-en [5652 kB]
Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [137 kB]
Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.3 kB]
Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB]
Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7060 B]
Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 c-n-f Metadata [286 kB]
Get:19 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [217 kB]
Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B]
Get:21 http://archive.ubuntu.com/ubuntu jammy/multiverse Translation-en [112 kB]
Get:22 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 c-n-f Metadata [8372 B]
Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [855 kB]
Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [209 kB]
Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [15.4 kB]
Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [668 kB]
Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [106 kB]
Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [528 B]
Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [962 kB]
Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [207 kB]
Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [21.4 kB]
Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB]
Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9768 B]
Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [476 B]
Get:35 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [40.9 kB]
Get:36 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.2 kB]
Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B]
Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/restricted amd64 c-n-f Metadata [116 B]
Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [22.2 kB]
Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [15.4 kB]
Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [580 B]
Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/multiverse amd64 c-n-f Metadata [116 B]
Fetched 26.4 MB in 8s (3454 kB/s)
Reading package lists...
+ apt-get install -y --no-upgrade --no-install-recommends -q sshfs
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  sshfs
0 upgraded, 1 newly installed, 0 to remove and 86 not upgraded.
Need to get 45.1 kB of archives.
After this operation, 126 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 sshfs amd64 3.7.1+repack-2 [45.1 kB]
Fetched 45.1 kB in 2s (22.2 kB/s)
Selecting previously unselected package sshfs.
(Reading database ... ^M(Reading database ... 5%^M(Reading database ... 10%^M(Reading database ... 15%^M(Reading database ... 20%^M(Reading database ... 25%^M(Reading database ... 30%^M(Reading database ... 35%^M(Reading database ... 40%^M(Reading database ... 45%^M(Reading database ... 50%^M(Reading database ... 55%^M(Reading database ... 60%^M(Reading database ... 65%^M(Reading database ... 70%^M(Reading database ... 75%^M(Reading database ... 80%^M(Reading database ... 85%^M(Reading database ... 90%^M(Reading database ... 95%^M(Reading database ... 100%^M(Reading database ... 64219 files and directories currently installed.)
Preparing to unpack .../sshfs_3.7.1+repack-2_amd64.deb ...
Unpacking sshfs (3.7.1+repack-2) ...
Setting up sshfs (3.7.1+repack-2) ...
Processing triggers for man-db (2.10.2-1) ...
NEEDRESTART-VER: 3.5
NEEDRESTART-KCUR: 5.15.0-72-generic
NEEDRESTART-KEXP: 5.15.0-72-generic
NEEDRESTART-KSTA: 1
LIMA| Executing /mnt/lima-cidata/boot/35-setup-packages.sh
+ SETUP_DNS=0
+ [ -n 58690 ]
+ [ 58690 -ne 0 ]
+ SETUP_DNS=1
+ [ -n 63183 ]
+ [ 63183 -ne 0 ]
+ SETUP_DNS=1
+ [ 1 = 1 ]
+ /mnt/lima-cidata/boot/09-host-dns-setup.sh
+ readonly chain=LIMADNS
+ command -v iptables
+ chain_exists
+ iptables --table nat -n --list LIMADNS
+ iptables --table nat --flush LIMADNS
+ iptables --table nat --append LIMADNS --destination 192.168.5.3 --protocol udp --dport 53 --jump DNAT --to-destination 192.168.5.2:58690
+ iptables --table nat --append LIMADNS --destination 192.168.5.3 --protocol tcp --dport 53 --jump DNAT --to-destination 192.168.5.2:63183
+ [ reverse-sshfs = reverse-sshfs ]
+ update_fuse_conf
+ [ 2 -gt 0 ]
+ fuse_conf=/etc/fuse.conf
+ [ -e /etc/fuse3.conf ]
+ grep -q ^user_allow_other /etc/fuse.conf
+ echo user_allow_other
LIMA| Executing /mnt/lima-cidata/boot/40-install-containerd.sh
+ '[' '' '!=' 1 ']'
+ '[' '' '!=' 1 ']'
+ exit 0
LIMA| Executing /mnt/lima-cidata/provision.system/00000000
LIMA| Executing /mnt/lima-cidata/provision.system/00000001
+ command -v docker
+ export DEBIAN_FRONTEND=noninteractive
+ DEBIAN_FRONTEND=noninteractive
+ curl -fsSL https://get.docker.com
+ sh
# Executing docker install script, commit: c2de0811708b6d9015ed1a2c80f02c9b70c8ce7b
+ sh -c apt-get update -qq >/dev/null
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
+ sh -c install -m 0755 -d /etc/apt/keyrings
+ sh -c curl -fsSL "https://download.docker.com/linux/ubuntu/gpg" | gpg --dearmor --yes -o /etc/apt/keyrings/docker.gpg
+ sh -c chmod a+r /etc/apt/keyrings/docker.gpg
+ sh -c echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu jammy stable" > /etc/apt/sources.list.d/docker.list
+ sh -c apt-get update -qq >/dev/null
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease  Could not connect to archive.ubuntu.com:80 (185.125.190.36). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (185.125.190.39). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (91.189.91.83). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (91.189.91.82). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (91.189.91.81). - connect (101: Network is unreachable)
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease  Unable to connect to archive.ubuntu.com:http:
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-backports/InRelease  Unable to connect to archive.ubuntu.com:http:
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/jammy-security/InRelease  Could not connect to security.ubuntu.com:80 (185.125.190.39). - connect (101: Network is unreachable) Could not connect to security.ubuntu.com:80 (185.125.190.36). - connect (101: Network is unreachable) Could not connect to security.ubuntu.com:80 (91.189.91.38). - connect (101: Network is unreachable) Could not connect to security.ubuntu.com:80 (91.189.91.39). - connect (101: Network is unreachable)
W: Failed to fetch https://download.docker.com/linux/ubuntu/dists/jammy/InRelease  Could not connect to download.docker.com:443 (18.172.31.124). - connect (101: Network is unreachable) Could not connect to download.docker.com:443 (18.172.31.28). - connect (101: Network is unreachable) Could not connect to download.docker.com:443 (18.172.31.47). - connect (101: Network is unreachable) Could not connect to download.docker.com:443 (18.172.31.22). - connect (101: Network is unreachable)
W: Some index files failed to download. They have been ignored, or old ones used instead.
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq docker-ce docker-ce-cli containerd.io docker-compose-plugin docker-ce-rootless-extras docker-buildx-plugin >/dev/null
E: Package 'docker-ce' has no installation candidate
E: Unable to locate package docker-ce-cli
E: Unable to locate package containerd.io
E: Couldn't find any package by glob 'containerd.io'
E: Couldn't find any package by regex 'containerd.io'
E: Unable to locate package docker-compose-plugin
E: Unable to locate package docker-ce-rootless-extras
E: Unable to locate package docker-buildx-plugin
LIMA| WARNING: Failed to execute /mnt/lima-cidata/provision.system/00000001
LIMA| Executing /mnt/lima-cidata/provision.user/00000002 (as user suda)
+ systemctl --user start dbus
+ dockerd-rootless-setuptool.sh install
/home/suda.linux/.lima-user-script: line 4: dockerd-rootless-setuptool.sh: command not found
LIMA| WARNING: Failed to execute /mnt/lima-cidata/provision.user/00000002 (as user suda)
LIMA| Exiting with code 1
2023-07-28 13:55:40,403 - cc_scripts_per_boot.py[WARNING]: Failed to run module scripts-per-boot (per-boot in /var/lib/cloud/scripts/per-boot)
2023-07-28 13:55:40,403 - util.py[WARNING]: Running module scripts-per-boot (<module 'cloudinit.config.cc_scripts_per_boot' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_per_boot.py'>) failed
Cloud-init v. 23.1.2-0ubuntu0~22.04.1 finished at Fri, 28 Jul 2023 13:55:40 +0000. Datasource DataSourceNoCloud [seed=/dev/sr0][dsmode=net].  Up 70.48 seconds
@AkihiroSuda AkihiroSuda mentioned this issue Jul 28, 2023
Closed
@AkihiroSuda
Copy link
Member Author

AkihiroSuda commented Jul 28, 2023 •
edited
Loading

The issue seems to happen with Lima v0.16, so this might not be a regression on Lima side.
(EDIT: my testing was wrong. Not reproducible with Lima v0.16)

Can anybody repro this?

@AkihiroSuda AkihiroSuda changed the title [master] limactl start template://docker no longer boots due to a DNS issue limactl start template://docker no longer boots due to a DNS issue Jul 28, 2023
@AkihiroSuda
Copy link
Member Author

The issue doesn't seem to happen with --vm-type vz

@AkihiroSuda AkihiroSuda changed the title limactl start template://docker no longer boots due to a DNS issue limactl start template://docker no longer boots due to a DNS (?) issue Jul 28, 2023
@AkihiroSuda
Copy link
Member Author

template://k3s boots but every pods fails with ImagePullBackOff: failed to do request: Head "https://registry-1.docker.io/v2/rancher/mirrored-metrics-server/manifests/v0.6.3": dial tcp: lookup registry-1.docker.io: Try again

@AkihiroSuda
Copy link
Member Author

Seems a regression in

@AkihiroSuda AkihiroSuda changed the title limactl start template://docker no longer boots due to a DNS (?) issue limactl start template://docker no longer boots (virtio-serial seems intermittently breaking virtio-net-pci) Jul 28, 2023
This was referenced Jul 28, 2023
Merged
Closed
@AkihiroSuda
Copy link
Member Author

AkihiroSuda commented Jul 28, 2023 •
edited
Loading

This seems to work, but not sure why

diff --git a/pkg/qemu/qemu.go b/pkg/qemu/qemu.go
index a120b6e..813a93d 100644
--- a/pkg/qemu/qemu.go
+++ b/pkg/qemu/qemu.go
@@ -830,7 +830,7 @@ func Cmdline(cfg Config) (string, []string, error) {
        }
        const serialvChardev = "char-serial-virtio"
        args = append(args, "-chardev", fmt.Sprintf("socket,id=%s,path=%s,server=on,wait=off,logfile=%s", serialvChardev, serialvSock, serialvLog))
-       args = append(args, "-device", "virtio-serial-pci,id=virtio-serial0")
+       args = append(args, "-device", "virtio-serial-pci,id=virtio-serial0,max_ports=2")
        args = append(args, "-device", fmt.Sprintf("virtconsole,chardev=%s,id=console0", serialvChardev))
 
        // We also want to enable vsock here, but QEMU does not support vsock for macOS hosts

Possibly relevant:

@AkihiroSuda AkihiroSuda mentioned this issue Jul 30, 2023
Merged
@AkihiroSuda AkihiroSuda linked a pull request Jul 30, 2023 that will close this issue
qemu: virtio-serial-pci: set max_ports=1 #1693
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component/dns component/qemu QEMU priority/high
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

qemu: virtio-serial-pci: set max_ports=1 AkihiroSuda/lima-vm-lima
1 participant
@AkihiroSuda