Support for lima usernet network #1383

balajiv113 · 2023-02-24T13:28:15Z

This PR adds support for the following,

Lima Usernet (Provides host -> vm, vm <-> vm communication)
Usernet as daemon and support for reconciler similar to socket_vmnet
New property added in yaml usernet.enabled, defaults to false. If its set to true, lima usernet will replace the slirp network

Todo items

pkg/limayaml/limayaml.go

AkihiroSuda · 2023-02-24T15:14:48Z

pkg/limayaml/limayaml.go

@@ -31,6 +31,7 @@ type LimaYAML struct {
 	Env          map[string]string `yaml:"env,omitempty" json:"env,omitempty"`
 	DNS          []net.IP          `yaml:"dns,omitempty" json:"dns,omitempty"`
 	HostResolver HostResolver      `yaml:"hostResolver,omitempty" json:"hostResolver,omitempty"`
+	UserNet      UserNet           `yaml:"userNet,omitempty" json:"userNet,omitempty"`


Seems confusing to have two network options both called "user"...

https://wiki.qemu.org/Documentation/Networking#User_Networking_(SLIRP)

Originally posted by @afbjorklund in #1222 (comment)

Bikeshedding:

gvisor (confusing because it does not use gvisor's process sandbox)

gvisor-tap-vsock (confusing because it is not gvisor, nor tap, nor vsock)

netstack (confusing because any networking can be called "net(work) stack"

user-v2 (confusing because there was no user-v1)

modern (confusing because it can be eventually legacy)

Edit

builtin

multivm

multi-vm

vm-to-vm

vm2vm

v2v

Actually it would be OK to name both of them "user", and then toggle the slirp vs gvisor later ?

Just was afraid that we would have one called "user" and a different one called "usernet"...

Am bit confused on this "two network options both called user"

As of now we don't have any configuration for slirp right?
And the one we provide now will be more a toggle, if enabled will use the latest or revert to the old modal ?

It is called user in qemu, is what I meant. If there is an option called usernet in lima too, it is confusing.

Or "multivm", "multi-vm", "vm-to-vm", "vm2vm", "v2v"

To me the following suggestions looks good,

user-v2 as this give support for giving further versions as well like user-v3 etc.

builtin since usernet command is within limactl it kind of makes sense as well

@afbjorklund @jandubois
WDYT?

I don't know what happened with "user-v1", is that the normal one ? But "user-v2" seems complicated to me, as a yaml default. But maybe it needs versioning (as per above, "user-v3", "user-v4") otherwise "builtin" is simpler.

What about calling it "builtin"

My vote is for builtin

balajiv113 · 2023-02-28T18:34:42Z

Marking PR as ready for review as the code looks lot better compared to the initial version.

There are some todo items but i think it can be done incrementally as well. Should not affect the current code much.
Will start with documentation as a last item once naming are finalised

balajiv113 · 2023-03-03T08:06:09Z

Testing for performance are done as well.

I tested in M1 with some long running instance (for more than a day with sleep). Haven't faced any issues (There was slight degrade when did iperf3, but on restarting iperf3 -s on host the performance was back to more meaningful value)

Also performance of QEMU is improved with this PR containers/gvisor-tap-vsock#188

Performance report
VZ <-> Host

$ iperf3 -c host.lima.internal
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.64 GBytes  2.27 Gbits/sec    0             sender
[  5]   0.00-10.02  sec  2.64 GBytes  2.26 Gbits/sec                  receiver

iperf Done.
$ iperf3 -c host.lima.internal -R
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.02 GBytes  1.73 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  2.02 GBytes  1.73 Gbits/sec                  receiver

QEMU <-> Host

$ iperf3 -c host.lima.internal
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.84 GBytes  1.58 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  1.84 GBytes  1.58 Gbits/sec                  receiver

iperf Done.
$ iperf3 -c host.lima.internal -R
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.21 GBytes  1.04 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  1.21 GBytes  1.04 Gbits/sec                  receiver

iperf Done.

VZ <-> QEMU

$ iperf3 -c 192.168.5.1
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.44 GBytes  1.24 Gbits/sec    0             sender
[  5]   0.00-10.07  sec  1.44 GBytes  1.23 Gbits/sec                  receiver

iperf Done.
$ iperf3 -c 192.168.5.1 -R
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec  1.83 GBytes  1.57 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  1.83 GBytes  1.57 Gbits/sec                  receiver

iperf Done.

AkihiroSuda · 2023-03-07T05:25:51Z

Support video display for vz

Seems mixed up with a different PR

examples/experimental/usernet.yaml

AkihiroSuda · 2023-03-07T05:28:19Z

examples/experimental/usernet.yaml

@@ -0,0 +1,15 @@
+# Example to run vz instance with lima usernet enabled


usernet.yaml

usernet-v2.yaml, or maybe net-user-v2.yaml
(If we are going to call the new network as user-v2)

Will rename this once the name is decided in above comment

afbjorklund · 2023-03-07T13:03:29Z

Here is what I got:

? Creating an instance "usernet" Proceed with the current configuration
FATA[0001] the YAML is invalid, saved the buffer as "lima.REJECTED.yaml": field `networks[0].lima` is only supported on macOS right now 
FATA[0000] vm driver 'vz' needs macOS 13 or later (Hint: try recompiling Lima if you are seeing this error on macOS 13) 
FATA[0000] field `mountType` must be "reverse-sshfs" or "9p" for QEMU driver , got "virtiofs"

And eventually:

"level":"fatal","msg":"template: :1:21: executing \"\" at \u003cfd_connect \"/home/anders/.lima/_networks/usernet_default_qemu.sock\"\u003e: error calling fd_connect: fd_connect: dial unix /home/anders/.lima/_networks/usernet_default_qemu.sock: connect: no such file or directory"

balajiv113 · 2023-03-07T13:21:24Z

Missed support for linux with networks.lima property.
Will fix it. Thanks 👍

balajiv113 · 2023-03-13T04:42:19Z

@afbjorklund
Done changes to support linux, can you check if it works now :)

jandubois · 2023-03-14T05:50:31Z

@afbjorklund @jandubois
WDYT?

To be honest, I kind of like the name "usernet". I'm also not confusing it with QEMU networking, as I always think of that one as SLIRP. "User mode networking" is just the general category, and SLIRP is a specific instance of it.

Now, you could argue that we should not use the category name ("user mode networking") and usurp it for the specific Lima implementation, but again personally I'm not bothered by it because it is disambiguated by context (it is the Lima usernet). I think descriptive names are good.

And if we ever need another name, and still can't come up with anything better, then we can still use usernet-v2. But maybe (hopefully) we will just have some different configuration parameter to tweak the implementation instead of having to replace it wholesale again.

AkihiroSuda · 2023-03-14T05:55:53Z

To be honest, I kind of like the name "usernet". I'm also not confusing it with QEMU networking, as I always think of that one as SLIRP. "User mode networking" is just the general category, and SLIRP is a specific instance of it.

"slirp" has become a general category too.
Docker calls VPNKit as "slirp", although its implementation is different from the original slirp.
(Not an important topic here though)

jandubois · 2023-03-14T06:13:48Z

I have no strong feelings about the name. I prefer usernet, but I can also live with user-v2 if that is what everyone else prefers.

Just to throw something else out: How about tunnel, because that's what it does, or at least what we use it for.

Not even sure if I like it myself though; might delete later. 🤣

AkihiroSuda · 2023-04-07T10:45:28Z

pkg/networks/usernet/usernet-client.go

+			for ipAddr, leaseAddr := range leases {
+				if vmMacAddr == leaseAddr {
+					err = c.delegate.Expose(&types.ExposeRequest{
+						Local:    fmt.Sprintf(":%d", sshPort),


Suggested change

Local: fmt.Sprintf(":%d", sshPort),

Local: fmt.Sprintf("127.0.0.1:%d", sshPort),

AkihiroSuda · 2023-04-07T10:46:07Z

pkg/networks/usernet/usernet-client.go

s/-/_/ in the filename

AkihiroSuda · 2023-04-07T13:37:14Z

examples/experimental/net-user-v2.yaml

+- location: "~"
+- location: "/tmp/lima"
+  writable: true
+mountType: "virtiofs"


Shouldn't be needed

AkihiroSuda · 2023-04-07T13:37:29Z

examples/experimental/net-user-v2.yaml

@@ -0,0 +1,15 @@
+# Example to run vz instance with experimental user-v2 network enabled
+vmType: "vz"


Shouldn't be needed

AkihiroSuda · 2023-04-20T12:22:07Z

Sorry needs another rebase

Signed-off-by: Balaji Vijayakumar <[email protected]>

balajiv113 · 2023-04-21T07:56:03Z

@AkihiroSuda - Done :)

AkihiroSuda

Thanks

AkihiroSuda · 2023-05-03T12:41:23Z

@jandubois @afbjorklund May I merge this, or do you want to take the final look before merging?

jandubois · 2023-05-04T05:59:46Z

May I merge this, or do you want to take the final look before merging?

I did want to review it, but I continue to be out of time. Feel free to merge if you are ok with it. At least it gets it into more people's hands for testing.

alejandroiglesias · 2023-05-18T18:09:39Z

Any chance of releasing this? It seems it would fix some issues I'm having.

AkihiroSuda · 2023-05-18T18:40:43Z

Any chance of releasing this? It seems it would fix some issues I'm having.

Yes, probably next week

balajiv113 force-pushed the full-network branch 6 times, most recently from 156fcce to 46ac22d Compare February 24, 2023 14:36

AkihiroSuda reviewed Feb 24, 2023

View reviewed changes

pkg/limayaml/limayaml.go Outdated Show resolved Hide resolved

AkihiroSuda reviewed Feb 24, 2023

View reviewed changes

balajiv113 force-pushed the full-network branch from 46ac22d to f85f804 Compare February 26, 2023 16:43

AkihiroSuda added this to the v0.16.0 milestone Feb 28, 2023

balajiv113 force-pushed the full-network branch 2 times, most recently from 16ad9b5 to ebda445 Compare February 28, 2023 18:32

balajiv113 marked this pull request as ready for review February 28, 2023 18:34

balajiv113 force-pushed the full-network branch from ebda445 to 2ea48c9 Compare March 7, 2023 05:19

AkihiroSuda reviewed Mar 7, 2023

View reviewed changes

examples/experimental/usernet.yaml Outdated Show resolved Hide resolved

AkihiroSuda reviewed Mar 7, 2023

View reviewed changes

balajiv113 force-pushed the full-network branch from 2ea48c9 to 400129a Compare March 7, 2023 07:09

AkihiroSuda mentioned this pull request Mar 7, 2023

Proof-of-concept implementation of socket network #1185

Closed

balajiv113 mentioned this pull request Mar 9, 2023

vz: Has been stuck in Waiting for the essential requirement 1 of 3: "ssh" on Start lima #1200

Open

balajiv113 force-pushed the full-network branch from 400129a to b0ae1bc Compare March 13, 2023 04:39

balajiv113 force-pushed the full-network branch from b0ae1bc to 53a6924 Compare March 26, 2023 11:03

balajiv113 force-pushed the full-network branch 2 times, most recently from 48c408d to 00a3f9d Compare April 7, 2023 07:34

AkihiroSuda reviewed Apr 7, 2023

View reviewed changes

balajiv113 force-pushed the full-network branch 2 times, most recently from 9a5535a to 0a7451f Compare April 7, 2023 11:20

AkihiroSuda reviewed Apr 7, 2023

View reviewed changes

balajiv113 force-pushed the full-network branch 2 times, most recently from 15953b8 to 20b34b5 Compare April 8, 2023 07:01

balajiv113 requested a review from AkihiroSuda April 8, 2023 07:39

Support for lima usernet network

800e450

Signed-off-by: Balaji Vijayakumar <[email protected]>

balajiv113 force-pushed the full-network branch from 20b34b5 to 800e450 Compare April 21, 2023 05:27

AkihiroSuda approved these changes Apr 25, 2023

View reviewed changes

AkihiroSuda requested a review from jandubois April 25, 2023 07:53

AkihiroSuda merged commit 79bc1e1 into lima-vm:master May 4, 2023

AkihiroSuda mentioned this pull request May 4, 2023

Experiencing slow network under VZ virtualization #1333

Closed

abiosoft mentioned this pull request May 5, 2023

Colima hangs a short while after starting abiosoft/colima#552

Open

5 tasks

balajiv113 mentioned this pull request May 17, 2023

VM network hang and panic on restart (gvisor-tap-vsock/pkg/tap.(*Switch).Accept ) #1358

Closed

balajiv113 mentioned this pull request May 18, 2023

Support for configurable network address for user-v2 #1551

Closed

balajiv113 deleted the full-network branch May 18, 2023 18:59

AkihiroSuda added component/network/user-v2 impact/changelog labels May 19, 2023

fskaeh mentioned this pull request May 31, 2023

SLIRP nw driver stops working quite often abiosoft/colima#678

Open

5 tasks

		@@ -0,0 +1,15 @@
		# Example to run vz instance with lima usernet enabled

	Local: fmt.Sprintf(":%d", sshPort),
	Local: fmt.Sprintf("127.0.0.1:%d", sshPort),

		@@ -0,0 +1,15 @@
		# Example to run vz instance with experimental user-v2 network enabled
		vmType: "vz"

Support for lima usernet network #1383

Support for lima usernet network #1383

Conversation

balajiv113 commented Feb 24, 2023 • edited Loading

Choose a reason for hiding this comment

AkihiroSuda Feb 24, 2023 • edited Loading

Choose a reason for hiding this comment

afbjorklund Feb 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

balajiv113 commented Feb 28, 2023

balajiv113 commented Mar 3, 2023

AkihiroSuda commented Mar 7, 2023

AkihiroSuda Mar 7, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

afbjorklund commented Mar 7, 2023 • edited Loading

balajiv113 commented Mar 7, 2023

balajiv113 commented Mar 13, 2023

jandubois commented Mar 14, 2023

AkihiroSuda commented Mar 14, 2023 • edited Loading

jandubois commented Mar 14, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AkihiroSuda commented Apr 20, 2023

balajiv113 commented Apr 21, 2023

AkihiroSuda left a comment

Choose a reason for hiding this comment

AkihiroSuda commented May 3, 2023

jandubois commented May 4, 2023

alejandroiglesias commented May 18, 2023

AkihiroSuda commented May 18, 2023

balajiv113 commented Feb 24, 2023 •

edited

Loading

AkihiroSuda Feb 24, 2023 •

edited

Loading

afbjorklund Feb 24, 2023 •

edited

Loading

AkihiroSuda Mar 7, 2023 •

edited

Loading

afbjorklund commented Mar 7, 2023 •

edited

Loading

AkihiroSuda commented Mar 14, 2023 •

edited

Loading