Support --subnet in network create

[siikamiika]

Caution

Just noticed you can actually create a network that conflicts with an existing network not managed by network create. Unless that is fine do not merge!

Currently containerization only supports IPv4.

Example:

Docker ref: https://docs.docker.com/reference/cli/docker/network/create/
Podman ref: https://docs.podman.io/en/v5.0.3/markdown/podman-network-create.1.html#subnet-subnet

Fixes #458

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update

Motivation and Context

[Why is this change needed?]

Scripted creation of deterministic network environments for containers

Testing

• Tested locally
• Added/updated tests
• Added/updated docs

[jglogan]

@siikamiika Thank you for the PR!

Just noticed you can actually create a network that conflicts with an existing network not managed by network create. Unless that is fine do not merge!

Could you give a concrete example of this? Are you referring to a network that exists on your Mac that was created by an app other than container?

[siikamiika]

Explaining the process to myself to make sure I understand it:

• user runs container network create ...
• ClientNetwork.create is called with a NetworkConfiguration struct
• XPC message with JSON serialized NetworkConfiguration is sent to com.apple.container.apiserver
• NetworksService.create receives the NetworkConfiguration after it's parsed
  • registerService registers the configuration with launchd and also checks if there are overlapping networks in other configurations
  • NetworkClient checks if the network was created
  • if successful, configuration is persisted in filesystem

I don't think it's checking if any existing networks for interfaces like LAN or VPN overlap the created network. Actually I don't even know if the default mode where a new /24 network is picked next to the previous (starting from the default 192.168.64.0/24 and going through 65, 66, ...) handles this, but it's not relevant for these changes.

Who's responsible for making sure container networks don't overlap with other networks?

• user
• container
  • which component?
• vmnet_network_create
• kernel
• something else?

If it's the user, then this change should be ok in my opinion, but there could be a bug in the default mode where the network address is chosen automatically (sorry didn't have time to verify this).

If it's container, getifaddrs could maybe be used to query active networks and compared to the new configuration. But how early should it be caught? Client or server?

The rest are out of my reach.

---

edit: verified automatically addressed container networks can also overlap existing ones not created by it:

> container network create auto1
auto1
> container network create auto2
auto2
> container run --rm -it --network auto2 debian

> container network ls
NETWORK  STATE    SUBNET
auto2    running  192.168.66.0/24
default  running  192.168.64.0/24
auto1    running  192.168.65.0/24

(en12 is a usb ethernet adapter connected to a router and bridge100 is the vmnet bridge)

en12: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=404<VLAN_MTU,CHANNEL_IO>
        ether xx:xx:xx:xx:xx:xx
        inet 192.168.66.238 netmask 0xffffff00 broadcast 192.168.66.255
        media: autoselect (1000baseT <full-duplex>)
        status: active
bridge100: flags=8a63<UP,BROADCAST,SMART,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
        options=63<RXCSUM,TXCSUM,TSO4,TSO6>
        ether xx:xx:xx:xx:xx:xx
        inet 192.168.66.1 netmask 0xffffff00 broadcast 192.168.66.255
        media: autoselect
        status: active

en12 network was unavailable while bridge100 was active but came back when I stopped the container and bridge100 disappeared.

[jglogan]

@siikamiika That's some really nice foresight to call out the network shadowing challenge, and some useful analysis as well.

don't think it's checking if any existing networks for interfaces like LAN or VPN overlap the created network.

Your understanding of the flow looks solid, and you are 100% correct:

• container defaults to not specifying a network interface as that provides the best UX, in that the new-for-Tahoe vmnet APIs should vend us a conflict free /24 subnet.
• If we do ask for a specific subnet, vmnet is supposed to ensure that it doesn't conflict with existing subnets.
• At least for now the user is the ultimate backstop; if for some reason vmnet allows a conflicting subnet to be created, container isn't going to go to heroic measures (maybe we could mine the system config dynamic store later) to validate things.

As I mentioned your analysis is useful because it seems to indicate that vmnet might not be doing the validation I thought it would.

Can you confirm that that Ethernet interface was up with the assigned subnet prior to your creating the NAT bridge with the overlapping subnet? I'll have an internal chat with some associates and see if we can get to the bottom of this.

As far as the PR, I think that what I described above means that what you've written should be sufficient. I'm talking tomorrow off but if someone else can't review this, I'll do so after the weekend. Thanks for the contribution!

[jglogan]

There will be one case btw that will be pretty much impossible to prevent:

• I'm at the office and do network create foo and vmnet vends me 192.168.65.0/24.
• I close up my laptop for the day, go home, open up my laptop and it connects to my wifi which uses the 65 subnet.

[siikamiika]

Can you confirm that that Ethernet interface was up with the assigned subnet prior to your creating the NAT bridge with the overlapping subnet? I'll have an internal chat with some associates and see if we can get to the bottom of this.

Yes. I just verified it again to be sure.

• starting point: container is not installed, computer is MacBook Pro M1, macOS 26.0.1 (25A362)
• connected to Wi-Fi where the network is below 192.168.64.0/24
• connect usb ethernet adapter (AX88179A) to the computer and a router where LAN DHCP addresses are from 192.168.66.0/24 (network service order: Thunderbolt Bridge, Wi-Fi, AX88179A, iPhone USB)
• computer gets an address from 192.168.66.0/24 and can open the router's web interface (the router is not connected to the internet)
• install container and run container system start
• create 2 networks with container network create, now container network ls shows 3 networks 192.168.64.0/24, 192.168.65.0/24, 192.168.66.0/24
• in ifconfig there are still just Wi-Fi and USB ethernet
• (see below) start a container with the network in 192.168.66.0/24
• now ifconfig shows both USB ethernet and bridge100 (member of vmenet0) in 192.168.66.0/24
• 192.168.66.1 is unreachable from host but the container reaches the host from that address
• container can also connect to the internet
• after stopping the container, host can reach the router from 192.168.66.1 again

This also happens if I start with container system stopped and then disconnect my Wi-Fi, configure my offline USB ethernet connected test router to use 192.168.64.0/24, start container system and run a container with the default network. I'll end up with two networks in 192.168.64.0/24. Same if I connect to the test router's Wi-Fi.

There will be one case btw that will be pretty much impossible to prevent:

• I'm at the office and do network create foo and vmnet vends me 192.168.65.0/24.
• I close up my laptop for the day, go home, open up my laptop and it connects to my wifi which uses the 65 subnet.

Good point, it's also a problem the other way around. This became even more interesting. If I have a container running in the 192.168.66.0/24 network and try to connect to the router with USB ethernet, I won't get an IP address from the router before I close the container (removing the bridge100 interface) and waiting for DHCP to do its thing again without the conflicting network active. After that I can restart the container and everything works like in the original steps starting from "(see below)".

192.168.0.0-192.168.255.255 is quite crowded but I don't know if defaulting to some other private range would be any better.

Signature was missing from my commit, I've rebased and signed my commit now.

Thanks for your help!

[jglogan]

Thanks, @siikamiika, this looks good to me.

Just wanted to make sure that AllocationOnlyVmnetNetwork fails when you try to run container-network-vmnet start --subnet some-cidr and it does.

[jglogan]

192.168.0.0-192.168.255.255 is quite crowded but I don't know if defaulting to some other private range would be any better.

More about all of this to explain where things are for anyone that wants to know a bit more:

• Copying this from the above post: container defaults to not specifying a network interface as that provides the best UX, in that the new-for-Tahoe vmnet APIs should vend us a conflict free /24 subnet. What I mean about this is simply that the easy thing for users is to not have to care about choosing subnets unless they want to.
• I've confirmed that the current vmnet implementation complicates this a little in that today, it indeed only checks against other vmnet networks as supposed. Hopefully this will be fixed in a future macOS release but we'll need to live with this for now. For us, adding logic to container auto-assign by probing the system is really low as we've already got a huge backlog of community issues we're trying to address and we'd like to focus at first on bugs, especially where there's no workaround.
• The 192.168 space is problematic as most folks' home LANs default to something in that space. vmnet starts vending at 192.168.64.0/0 and generally goes up from there. I'd say that if you want a better chance of randomly choosing a network so as not to conflict with home LANs, go for something from 10/8 or 172.16/12.