Enabling a `shared` or `bridged` network causes `lima kubectl cluster-info` to fail #375

spkane · 2022-07-25T21:55:03Z

Description

colima is unable to verify that the Kubernetes has started correctly when a shared or bridged network is used for the underlying VM.

Version

Colima Version: 0.4.4
Lima Version: 0.11.2
Qemu Version: 7.0.0

Operating System

macOS Intel
macOS M1
Linux

Reproduction Steps

This used to work, but started failing at some point.

I have set up networking for lima as discussed here:

https://github.com/lima-vm/lima/blob/master/docs/network.md

And then in this file: .lima/_config/override.yaml I have this:

networks:
  - lima: bridged

When I try to start colima with the network uncommented then I get:

INFO[0000] starting colima
INFO[0000] runtime: containerd+k3s
INFO[0000] preparing network ...                         context=vm
INFO[0000] starting ...                                  context=vm
INFO[0030] provisioning ...                              context=containerd
INFO[0030] starting ...                                  context=containerd
INFO[0036] provisioning ...                              context=kubernetes
INFO[0037] starting ...                                  context=kubernetes
FATA[0062] error starting kubernetes: error running [lima kubectl cluster-info], output: "The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?", err: "exit status 1"

Expected behaviour

I would expect the VM to come up with no errors and everything in a useable state.

Additional context

I am able to reproduce this even after recompiling and installing vde-2 and vde_vmnet.

The text was updated successfully, but these errors were encountered:

abiosoft · 2022-07-26T05:21:33Z

Why not simply use --network-address flag?
Is there anything extra you have in your Lima override config other than specifying the network?

spkane · 2022-07-26T15:52:49Z

I do not have anything else in the override file. As a matter of fact, I completely removed my lima and colima installs and started from scratch while I was trying to troubleshoot this.

This had been working fine in the past, so I was surprised that it stopped working and caused colima to generate an error during startup. I'll take a look at that flag and see if I can make it work.

https://github.com/abiosoft/colima/blob/main/docs/FAQ.md#the-virtual-machines-ip-is-not-reachable

abiosoft · 2022-07-26T16:44:12Z

If all you need is a reachable IP address, I would recommend you do that with Colima directly.
Lima overrides are for advanced use-cases and they are not always guaranteed to work, especially when Colima also generates similar config.

I'll take a look at that flag and see if I can make it work.

It is easier than you think. Simply start with colima start --network-address and view the IP address with colima list.

spkane · 2022-07-26T18:36:08Z

I am a bit confused with the solution that you are recommending, only because this ticket is about the fact the colima generates an error during startup in this situation.

FATA[0062] error starting kubernetes: error running [lima kubectl cluster-info], output: "The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?", err: "exit status 1"

I am not trying to run lima kubectl cluster-info myself.

abiosoft · 2022-07-26T18:43:05Z

Yeah, colima runs the command as part of the startup for kubernetes to ensure that the startup is successful.
Does it work if you comment out the network config in the lima override file?

What I suspect is that k3s bootstraps using the vmnet IP address and thereby unreachable without using the IP.

spkane · 2022-07-26T19:06:30Z

Yes. It works when the network override is commented out. That additional info is helpful. Thanks.

Adding --network-address seems to have resolved the error, although I see INFO[0000] dependencies missing for setting up reachable IP address context=vm in the output. It still isn't clear to me exactly what is going on, but I'll experiment a bit with it.

colima start --cpu 8 --memory 8 --disk 100 --runtime containerd --with-kubernetes --network-address

ipv1337 · 2023-03-04T07:52:08Z

This is the closest issue I can find related to what I'm trying to achieve setting up a bridged network mode.

I have yet to see how to configure colima to start up the VM using my own network dhcp so I can get a real IP off my network. When I do try to add the network override.yaml for lima, I can start colima fine, but my network override in lima does not setup a bridge the way I'd expect it to.

My override is simple:

networks:
   - lima: bridged

Is this even possible? I've googled this many times and have found little information on this topic. Any help is much appreciated.

jandubois · 2023-03-04T09:44:56Z

This should work, as long as your host network interface is en0. Maybe you are using a different one. You can find the most likely candidate with (assuming you have jq installed; otherwise you have to pick it out yourself):

$ system_profiler SPNetworkDataType -json | jq -r 'first(.SPNetworkDataType[] | select(.ip_address) | .interface)'
en0

Then check your ~/.lima/_config/networks.yaml file for the definition of the bridged network:

networks:
  bridged:
    mode: bridged
    interface: en0

Make sure your host interface name matches the one in the bridged definition.

ipv1337 · 2023-03-04T19:59:18Z

Hi @jandubois thanks for you quick response. I am using en0 actually as you can see here also:

└[~]> system_profiler SPNetworkDataType -json | jq -r 'first(.SPNetworkDataType[] | select(.ip_address) | .interface)'
en0

Here's what my ~/.lima/_config/networks.yaml file looks like which is the default:

└[~]> cat ~/.lima/_config/networks.yaml
# Path to socket_vmnet executable. Because socket_vmnet is invoked via sudo it should be
# installed where only root can modify/replace it. This means also none of the
# parent directories should be writable by the user.
#
# The varRun directory also must not be writable by the user because it will
# include the socket_vmnet pid file. Those will be terminated via sudo, so replacing
# the pid file would allow killing of arbitrary privileged processes. varRun
# however MUST be writable by the daemon user.
#
# None of the paths segments may be symlinks, why it has to be /private/var
# instead of /var etc.
paths:
# socketVMNet requires Lima >= 0.12 .
# socketVMNet has precedence over vdeVMNet.
  socketVMNet: "/opt/socket_vmnet/bin/socket_vmnet"
# vdeSwitch and vdeVMNet are DEPRECATED.
  vdeSwitch: /opt/vde/bin/vde_switch
  vdeVMNet: /opt/vde/bin/vde_vmnet
  varRun: /private/var/run/lima
  sudoers: /private/etc/sudoers.d/lima

group: everyone

networks:
  shared:
    mode: shared
    gateway: 192.168.105.1
    dhcpEnd: 192.168.105.254
    netmask: 255.255.255.0
  bridged:
    mode: bridged
    interface: en0
    # bridged mode doesn't have a gateway; dhcp is managed by outside network
  host:
    mode: host
    gateway: 192.168.106.1
    dhcpEnd: 192.168.106.254
    netmask: 255.255.255.0

This is how I'm trying to start colima:

$ colima start --cpu 8 --memory 8 --disk 100 --vm-type qemu --runtime containerd --kubernetes --activate --network-driver slirp --network-address

Here's something I observed from /etc/sudoers.d/colima that might help?

└[/etc/sudoers.d]> cat colima
# starting vmnet daemon
%staff ALL=(root:wheel) NOPASSWD:NOSETENV: /opt/colima/bin/socket_vmnet --vmnet-mode shared --socket-group staff --vmnet-gateway 192.168.106.1 --vmnet-dhcp-end 192.168.106.254 *
# terminating vmnet daemon
%staff ALL=(root:wheel) NOPASSWD:NOSETENV: /usr/bin/pkill -F /opt/colima/run/*.pid
# validating vmnet daemon
%staff ALL=(root:wheel) NOPASSWD:NOSETENV: /usr/bin/pkill -0 -F /opt/colima/run/*.pid

That is the network IP my colima instance keeps getting instead of my own home network IP that en0 gets.

└[~]> colima status
INFO[0000] colima is running using QEMU
INFO[0000] arch: x86_64
INFO[0000] runtime: containerd
INFO[0000] mountType: virtiofs
INFO[0000] address: 192.168.106.2
INFO[0000] kubernetes: enabled

If I attempt to change /etc/sudoers.d/colima, it just reverts itself when I stop/start colima. I don't see how or where I can persist a change there for colima.

jandubois · 2023-03-04T20:17:43Z

I'm sorry @ipv1337, I don't know how colima manages the networking setup; I was just responding why the bridged network might not get an IP address if you bridge through the wrong host adapter, but that doesn't seem to be a problem for you. Maybe @abiosoft can chime in with the correct commandline options needed to use the bridged networking.

One (unrelated) thing I find curious that you are using virtiofs with QEMU; I thought that would only work with VZ. Or maybe that is also some colima-specific extension.

ipv1337 · 2023-03-05T01:18:52Z

I'm sorry @ipv1337, I don't know how colima manages the networking setup; I was just responding why the bridged network might not get an IP address if you bridge through the wrong host adapter, but that doesn't seem to be a problem for you. Maybe @abiosoft can chime in with the correct commandline options needed to use the bridged networking.

One (unrelated) thing I find curious that you are using virtiofs with QEMU; I thought that would only work with VZ. Or maybe that is also some colima-specific extension.

We can ignore the virtiofs, that I know is incorrect since I originally wanted to use vz but I was giving QEMU a try to see if they made any difference with the network setup I wanted.

abiosoft · 2023-03-05T14:06:49Z

If I attempt to change /etc/sudoers.d/colima, it just reverts itself when I stop/start colima. I don't see how or where I can persist a change there for colima.

Yeah, true. It is not expected a user would modify the file directly.

Maybe an introduction of --network-type flag (and config) would help with the options being shared (default) and bridged.

One (unrelated) thing I find curious that you are using virtiofs with QEMU; I thought that would only work with VZ. Or maybe that is also some colima-specific extension.

It is a misinformation from Colima's end, it is already corrected in main and would be part of the imminent next release.

zack-is-cool · 2023-07-31T22:18:00Z

Easy bridged networking would be an awesome feature. I have a usecase where I need to run x86 containers in k8s on an m1 mac and have services be exposed to another host on my network.

I was able to get bridged networking working with this config
en7 is my interface that I have via ethernet:

system_profiler SPNetworkDataType -json | jq -r 'first(.SPNetworkDataType[] | select(.ip_address) | .interface)'
en7

~/.colima/default/colima.yaml:

...
network:
  address: true

~/.lima/_config/override.yaml:

networks:
- lima: bridged
  interface: en7
- vzNAT: false
  interface: col0

~/.lima/_config/networks.yaml:

paths:
# socketVMNet requires Lima >= 0.12 .
# socketVMNet has precedence over vdeVMNet.
  socketVMNet: "/opt/homebrew/Cellar/socket_vmnet/1.1.2/bin/socket_vmnet"
# vdeSwitch and vdeVMNet are DEPRECATED.
  vdeSwitch: /opt/vde/bin/vde_switch
  vdeVMNet: /opt/vde/bin/vde_vmnet
  varRun: /private/var/run/lima
  sudoers: /private/etc/sudoers.d/lima

group: everyone

networks:
  user-v2:
    mode: user-v2
    # user-v2 network is experimental network mode which supports all functionalities of default usernet network and also allows vm -> vm communication.
    # Doesn't support configuration of custom gateway; hardcoded to 192.168.5.0/24
  shared:
    mode: shared
    gateway: 192.168.105.1
    dhcpEnd: 192.168.105.254
    netmask: 255.255.255.0
  bridged:
    mode: bridged
    interface: en7
    # bridged mode doesn't have a gateway; dhcp is managed by outside network
  host:
    mode: host
    gateway: 192.168.106.1
    dhcpEnd: 192.168.106.254
    netmask: 255.255.255.0

torarnv · 2024-03-06T00:15:18Z

I can't get this to work. The Lima override in ~/.lima/_config/override.yaml is not picked up by Colima, as Colima seems to use a separate $LIMA_HOME in ~/.colima/_lima/.

Modifying ~/.colima/_lima/_config/override.yaml does "work", in that the change is picked up, but since ~/.colima/_lima/_config/networks.yaml only contains a user-v2 network (https://github.com/abiosoft/colima/blob/9b0809d0ed9ad3ff1e57c405f27324e6298ca04f/embedded/network/networks.yaml) it fails with:

errors inspecting instance: [field networks[2].lima references network "bridged" which is not defined in networks.yaml]

And the networks.yaml can't be edited, as it's overwritten each time Colima starts.

So how is this supposed to work?

FTPiano · 2024-03-15T15:59:03Z

I can't get this to work. The Lima override in ~/.lima/_config/override.yaml is not picked up by Colima, as Colima seems to use a separate $LIMA_HOME in ~/.colima/_lima/.

Modifying ~/.colima/_lima/_config/override.yaml does "work", in that the change is picked up, but since ~/.colima/_lima/_config/networks.yaml only contains a user-v2 network (https://github.com/abiosoft/colima/blob/9b0809d0ed9ad3ff1e57c405f27324e6298ca04f/embedded/network/networks.yaml) it fails with:

errors inspecting instance: [field networks[2].lima references network "bridged" which is not defined in networks.yaml]

And the networks.yaml can't be edited, as it's overwritten each time Colima starts.

So how is this supposed to work?

Yep I got exactly the same error when I was trying to start a colima VM with bridged network. Found this issue when I was googling for more information.

It looks like the code is always preparing the vmnet in "shared" mode rather than "bridged". Finally, after some investigation, I got what I wanted with a local build after patching the code like this:

diff --git a/daemon/process/vmnet/vmnet.go b/daemon/process/vmnet/vmnet.go
index 12d4030..e66703d 100644
--- a/daemon/process/vmnet/vmnet.go
+++ b/daemon/process/vmnet/vmnet.go
@@ -75,10 +75,11 @@ func (*vmnetProcess) Start(ctx context.Context) error {
 	go func() {
 		// rootfully start the vmnet daemon
 		command := cli.CommandInteractive("sudo", BinaryPath,
-			"--vmnet-mode", "shared",
-			"--socket-group", "staff",
-			"--vmnet-gateway", NetGateway,
-			"--vmnet-dhcp-end", NetDHCPEnd,
+			"--vmnet-mode", "bridged",
+			"--vmnet-interface", "en0",
+//			"--socket-group", "staff",
+//			"--vmnet-gateway", NetGateway,
+//			"--vmnet-dhcp-end", NetDHCPEnd,
 			"--pidfile", pid,
 			socket,
 		)
diff --git a/embedded/network/networks.yaml b/embedded/network/networks.yaml
index abd37f1..20f1241 100644
--- a/embedded/network/networks.yaml
+++ b/embedded/network/networks.yaml
@@ -2,7 +2,6 @@
 # This file would be replaced by Colima on startup.
 
 networks:
-  user-v2:
-    mode: user-v2
-    gateway: 192.168.5.2
-    netmask: 255.255.255.0
+  bridged:
+    mode: bridged
+    interface: en0
diff --git a/embedded/network/sudo.txt b/embedded/network/sudo.txt
index 3fbd4ab..f8091fc 100644
--- a/embedded/network/sudo.txt
+++ b/embedded/network/sudo.txt
@@ -1,5 +1,5 @@
 # starting vmnet daemon
-%staff ALL=(root:wheel) NOPASSWD:NOSETENV: /opt/colima/bin/socket_vmnet --vmnet-mode shared --socket-group staff --vmnet-gateway 192.168.106.1 --vmnet-dhcp-end 192.168.106.254 *
+%staff ALL=(root:wheel) NOPASSWD:NOSETENV: /opt/colima/bin/socket_vmnet --vmnet-mode bridged *
 # terminating vmnet daemon
 %staff ALL=(root:wheel) NOPASSWD:NOSETENV: /usr/bin/pkill -F /opt/colima/run/*.pid
 # validating vmnet daemon

The patch is definitely hacky thus far from a mature PR. However, I suppose it's a good start for proof-of-concept and further discussions regarding this feature that some people are looking for 😄 @abiosoft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enabling a `shared` or `bridged` network causes `lima kubectl cluster-info` to fail #375

Enabling a `shared` or `bridged` network causes `lima kubectl cluster-info` to fail #375

spkane commented Jul 25, 2022 •

edited

Loading

abiosoft commented Jul 26, 2022

spkane commented Jul 26, 2022

abiosoft commented Jul 26, 2022

spkane commented Jul 26, 2022 •

edited

Loading

abiosoft commented Jul 26, 2022

spkane commented Jul 26, 2022 •

edited

Loading

ipv1337 commented Mar 4, 2023

jandubois commented Mar 4, 2023

ipv1337 commented Mar 4, 2023 •

edited

Loading

jandubois commented Mar 4, 2023

ipv1337 commented Mar 5, 2023

abiosoft commented Mar 5, 2023

zack-is-cool commented Jul 31, 2023 •

edited

Loading

torarnv commented Mar 6, 2024

FTPiano commented Mar 15, 2024

Enabling a shared or bridged network causes lima kubectl cluster-info to fail #375

Enabling a shared or bridged network causes lima kubectl cluster-info to fail #375

Comments

spkane commented Jul 25, 2022 • edited Loading

Description

Version

Operating System

Reproduction Steps

Expected behaviour

Additional context

abiosoft commented Jul 26, 2022

spkane commented Jul 26, 2022

abiosoft commented Jul 26, 2022

spkane commented Jul 26, 2022 • edited Loading

abiosoft commented Jul 26, 2022

spkane commented Jul 26, 2022 • edited Loading

ipv1337 commented Mar 4, 2023

jandubois commented Mar 4, 2023

ipv1337 commented Mar 4, 2023 • edited Loading

jandubois commented Mar 4, 2023

ipv1337 commented Mar 5, 2023

abiosoft commented Mar 5, 2023

zack-is-cool commented Jul 31, 2023 • edited Loading

torarnv commented Mar 6, 2024

FTPiano commented Mar 15, 2024

Enabling a `shared` or `bridged` network causes `lima kubectl cluster-info` to fail #375

Enabling a `shared` or `bridged` network causes `lima kubectl cluster-info` to fail #375

spkane commented Jul 25, 2022 •

edited

Loading

spkane commented Jul 26, 2022 •

edited

Loading

spkane commented Jul 26, 2022 •

edited

Loading

ipv1337 commented Mar 4, 2023 •

edited

Loading

zack-is-cool commented Jul 31, 2023 •

edited

Loading