No internet inside container? #6

tmpm697 · 2020-09-30T09:14:41Z

Steps to produce:

git clone .../mkosi-files
cd mkosi-files
sudo ./update-nspawn-images buildspawn/
sudo systemd-nspawn -UM buildspawn
passwd
exit

# start to container
sudo machinectl start buildspawn
# will failed to start due to # Bind=/ramdisk/scratch/:/scratch/
# I have to comment out Bind=/ramdisk/scratch/:/scratch/ in /etc/systemd/nspawn/buildspawn.nspawn to make it work

# login to container
sudo machinectl login buildspawn
root
<password.

ip ad
ping google.com
# failed

on host I have ve-* interface, and inside container I have another interface without IP address, I would expect the non-lo interface should have and ip address assigned to it and I'll be able to ping google.com from container.

all setup are mostly default and I haven't changed anything.

I use archlinux latest lts-linux, systemd-networkd to manage my wifi connection via wpa_supplicant profile.

The text was updated successfully, but these errors were encountered:

nosada · 2020-10-01T15:06:39Z

@tmpm697 Argh, my approgies, I've not documented about veth...
Try below rules for iptables:

-A INPUT -i ve-buildspawn -j ACCEPT

From my experience (sorry not to show you any sources), you must add rule INPUT for veth with -j ACCEPT to communitcate outside container with veth.
(i.e. VirtualEthernet=yes in /etc/systemd/nspawn/*.nspawn)

I added above to README.md at 18bed45.

Thanks for your reporting ☺️

tmpm697 · 2020-10-01T18:41:44Z

This is weird. I don't have iptables or nftables services running.

How to setup that container will have static IP address?

My setup:

00-mynet.network in /etc/systemd/network that match against a wifi nic wlxxx. (On host).

I use archlinux latest with linux-lts, systemd-networkd to manage network for both host and container. I have [email protected] to start wifi profile and the profile in /etc/wpa_supplicant on host.
Everything else has default config.

I want build and start buildspawn with a specific ipadress but failed.
I followed this article: https://wiki.archlinux.org/index.php/Systemd-networkd#Usage_with_containers

Basically:
On host:
Add Bridge=br0 to 00-mynet.network
Create br0.netdev:

#/etc/systemd/network/MyBridge.netdev
[NetDev]
Name=br0
Kind=bridge

Create bridge network:
br0.network

#/etc/systemd/network/MyBridge.network
[Match]
Name=br0

[Network]
DNS=192.168.1.254
Address=192.168.1.87/24
Gateway=192.168.1.1
#192.169.1.1 is also my host's gateway, 192.168.1.0/24 is my host's LAN network.

I build and start buildspawn container as default, the add static ip adress as in article:
On container:

#/etc/systemd/network/80-container-host0.network
[Match]
Name=host0

[Network]
DNS=192.168.1.254
Address=192.168.1.94/24
Gateway=192.168.1.1

Ip route on host:

wlxxx (container 192.168.1.xx)
ve-buidspawn@f13 status DOWN, no ipadress

Ip route on container:

lo
host0 DOWN, no ipadress

I think there's something thing wrong with my setup, do you have same issue before?

P/s I'll put code block when in desktop.

Thanks.

nosada · 2020-10-02T00:02:21Z

I see.
I haven't ever used bridge interface with nspawn + networkd, so there's no tips I can show you...

I'll try to use bridge + nspawn + networkd later when I have time.

nosada · 2020-10-11T05:14:00Z

I haven't tried bridge + nspawn + networkd yet, but found my mistaken (or forgot) point for using veth.
If you use veth and then go outside from container, you must set up NAT with iptables (or nftables), as described in https://wiki.archlinux.org/index.php/systemd-nspawn#Use_a_virtual_Ethernet_link.

Below's an example for iptables I used before (cf. https://wiki.archlinux.org/index.php/Simple_stateful_firewall) :

Install this to /etc/iptables/iptables.rules

*filter
:INPUT DROP [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:TCP - [0:0]
:UDP - [0:0]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

-A INPUT -i lo -j ACCEPT
-A INPUT -i ve-+ -j ACCEPT

-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -p icmp -m icmp --icmp-type 8 -m conntrack --ctstate NEW -j ACCEPT
-A INPUT -p udp -m conntrack --ctstate NEW -j UDP
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j TCP
-A INPUT -j REJECT --reject-with icmp-proto-unreachable
-A INPUT -p tcp -m recent --set --name TCP-PORTSCAN --mask 255.255.255.255 --rsource -j REJECT --reject-with tcp-reset
-A INPUT -p udp -m recent --set --name UDP-PORTSCAN --mask 255.255.255.255 --rsource -j REJECT --reject-with icmp-port-unreachable
-A TCP -p tcp -m recent --update --seconds 60 --name TCP-PORTSCAN --mask 255.255.255.255 --rsource -j REJECT --reject-with tcp-reset
-A UDP -p udp -m recent --update --seconds 60 --name UDP-PORTSCAN --mask 255.255.255.255 --rsource -j REJECT --reject-with icmp-port-unreachable
COMMIT

and below's for nftables I'm using now (cf. https://wiki.archlinux.org/index.php/Nftables#Simple_stateful_firewall):

Install this to /etc/nftables.conf

table inet filter {
        chain input {
                type filter hook input priority filter; policy drop;
                ct state established,related accept
                iifname "lo" accept
                iifname "ve-*" accept
                ct state invalid drop
                ip6 nexthdr ipv6-icmp icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, mld-listener-query, mld-listener-report, mld-listener-done, nd-router-solicit, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, ind-neighbor-solicit, ind-neighbor-advert, mld2-listener-report } accept
                ip protocol icmp icmp type { destination-unreachable, router-advertisement, router-solicitation, time-exceeded, parameter-problem } accept
                ip protocol igmp accept
                ip protocol udp ct state new jump UDP
                ip protocol tcp tcp flags & (fin | syn | rst | ack) == syn ct state new jump TCP
                ip protocol udp reject
                ip protocol tcp reject with tcp reset
                meta nfproto ipv4 counter packets 0 bytes 0 reject with icmp type prot-unreachable
        }

        chain forward {
                type filter hook forward priority filter; policy accept;
        }

        chain output {
                type filter hook output priority filter; policy accept;
        }

        chain TCP {
        }

        chain UDP {
        }
}

tmpm697 · 2020-10-11T18:24:52Z

I don't use iptables or nftables in my system.

I have ve-xxxx on host and host0 in container but as the archwiki:

"When you start the container, an IP address has to be assigned to both interfaces (on the host and in the container). If you use systemd-networkd on the host as well as in the container, this is done out-of-the-box"

This is not true in my case, both use systemd-networkd but there's no ip under container.

ve-xxx and host0 both in DOWN state.

I use wifi profile with wpa-supplicant but I think it doesn't relate to this case.

nosada · 2020-10-12T15:43:24Z

Ah, sorry, I misread your comment.

I could maybe reproduce your issue (no IP address on ve-* in host, host0 in container) by doing below (seems to be identical inside systemd) in host:

remove /usr/lib/systemd/network/80-container-ve.network
Create symlink /etc/systemd/network/80-container-ve.network to /dev/null

"networkctl" result

# In Host

$ networkctl
IDX LINK          TYPE     OPERATIONAL SETUP
  1 lo            loopback carrier     unmanaged
  2 enp0s25       ether    no-carrier  configuring
  4 wlan0         wlan     routable    configured
  8 ve-buildspawn ether    off         unmanaged     <- veth for nspawn container

4 links listed.

$ networkctl status ve-buildspawn
● 8: ve-buildspawn
                     Link File: /usr/lib/systemd/network/99-default.link
                  Network File: n/a
                          Type: ether
                         State: off (unmanaged)
                        Driver: veth
                    HW Address: 8a:41:5e:XX:XX:XX
                           MTU: 1500 (min: 68, max: 65535)
                         QDisc: noop
  IPv6 Address Generation Mode: eui64
          Queue Length (Tx/Rx): 1/1
              Auto negotiation: no
                         Speed: 10Gbps
                        Duplex: full
                          Port: tp

Oct 13 00:07:30 HOST systemd-networkd[4744]: ve-buildspawn: Link DOWN
Oct 13 00:07:30 HOST systemd-networkd[4744]: ve-buildspawn: Lost carrier
Oct 13 00:07:30 HOST systemd-networkd[4744]: ve-buildspawn: DHCPv6 lease lost
Oct 13 00:12:10 HOST systemd-networkd[23786]: ve-buildspawn: IPv6 successfully enabled
Oct 13 00:12:10 HOST systemd-networkd[23786]: ve-buildspawn: Link UP
Oct 13 00:12:12 HOST systemd-networkd[23786]: ve-buildspawn: Gained carrier
Oct 13 00:12:14 HOST systemd-networkd[23786]: ve-buildspawn: Gained IPv6LL
Oct 13 00:12:24 HOST systemd-networkd[23786]: ve-buildspawn: Link DOWN
Oct 13 00:12:24 HOST systemd-networkd[23786]: ve-buildspawn: Lost carrier
Oct 13 00:12:24 HOST systemd-networkd[23786]: ve-buildspawn: DHCPv6 lease lost

# In container

$ networkctl
IDX LINK  TYPE     OPERATIONAL SETUP
  1 lo    loopback carrier     unmanaged
  2 host0 ether    no-carrier  configuring

2 links listed.

$ networkctl status host0
● 2: host0
                     Link File: n/a
                  Network File: /usr/lib/systemd/network/80-container-host0.network
                          Type: ether
                         State: no-carrier (configuring)
                    HW Address: 7a:4d:b2:YY:YY:YY
                           MTU: 1500 (min: 68, max: 65535)
                         QDisc: noqueue
  IPv6 Address Generation Mode: eui64
          Queue Length (Tx/Rx): 1/1
              Auto negotiation: no
                         Speed: 10Gbps
                        Duplex: full
                          Port: tp
             DHCP6 Client DUID: DUID-EN/Vendor:0000ab11fe986dfda77474780000

Oct 13 00:12:51 buildspawn systemd-networkd[19]: host0: IPv6 successfully enabled
Oct 13 00:12:51 buildspawn systemd-networkd[19]: host0: Link UP

"ip address show" result

# In host

$ ip address show dev ve-buildspawn
8: ve-buildspawn@if2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 8a:41:5e:XX:XX:XX brd ff:ff:ff:ff:ff:ff link-netnsid 0

# In container

$ ip address show dev host0
2: host0@if8: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default qlen 1000
    link/ether 7a:4d:b2:YY:YY:YY brd ff:ff:ff:ff:ff:ff link-netnsid 0

Both could do them.
After reproduced, I restored 80-container-ve.network then restart systemd-networkd.service and things recovered.

Here's /etc/systemd/nspawn/buildspawn.nspawn I used:

[Exec]
PrivateUsers=true
NotifyReady=true

[Files]
# Commented out below also works fine
Bind=/ramdisk/scratch/:/scratch/

[Network]
Private=yes
VirtualEthernet=yes

Maybe you couldn't treat veth by systemd-networkd properly.
Could you try checking your *.network in /etc/systemd/network/ and /usr/lib/systemd/network/?

P.S. I use iwd to manage wifi profile instead of wpa_supplicant, I've used before. But when wpa_supplicant I haven't experience issue like you reported.
So I think so too that wpa_supplicant doesn't relate to this issue.

tmpm697 · 2020-10-13T14:33:37Z

ALL of my configs:

ON HOST:

cat /etc/systemd/network/00-wireless.network
# NOTE: I use static ip address.
# /etc/systemd/network/00-wireless.network
[Match]
Name=wlp1sX
SSID=<my wifi ssid>

[Network]
Address=192.168.Y.XXX/24
Gateway=192.168.E.F
DNS=8.8.8.8 8.8.4.4

cat /etc/wpa_supplicant/wpa_supplicant-wlp1sX.conf
network={
        ssid="<my wifi ssid>"
        scan_ssid=1
        psk=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
}

networkctl
IDX LINK          TYPE     OPERATIONAL SETUP
  1 lo            loopback carrier     unmanaged
  # ...
  8 ve-buildspawn ether    no-carrier  configuring

ip address show dev ve-buildspawn
8: ve-buildspawn@if2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default qlen 1000
    link/ether ee:bc:06:a5:c1:fc brd ff:ff:ff:ff:ff:ff link-netnsid 0

cat /etc/systemd/nspawn/buildspawn.nspawn
[Exec]
PrivateUsers=true
NotifyReady=true

[Files]
# Commented out below also works fine
# Bind=/ramdisk/scratch/:/scratch/

[Network]
Private=yes
VirtualEthernet=yes

networkctl status ve-buildspawn
● 8: ve-buildspawn
                     Link File: /usr/lib/systemd/network/99-default.link
                  Network File: /usr/lib/systemd/network/80-container-ve.network
                          Type: ether
                         State: no-carrier (configuring)
                        Driver: veth
                    HW Address: ee:bc:06:a5:c1:fc
                           MTU: 1500 (min: 68, max: 65535)
                         QDisc: noqueue
  IPv6 Address Generation Mode: eui64
          Queue Length (Tx/Rx): 1/1
              Auto negotiation: no
                         Speed: 10Gbps
                        Duplex: full
                          Port: tp
             DHCP6 Client DUID: DUID-EN/Vendor:0000ab11127bcb58b9aa75fa0000
           Offered DHCP leases: none

Oct 13 14:11:43 localhost systemd-networkd[389]: ve-buildspawn: IPv6 successfully enabled
Oct 13 14:11:43 localhost systemd-networkd[389]: ve-buildspawn: Link UP

cat /usr/lib/systemd/network/99-default.link
#  SPDX-License-Identifier: LGPL-2.1+
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Match]
OriginalName=*

[Link]
NamePolicy=keep kernel database onboard slot path
AlternativeNamesPolicy=database onboard slot path
MACAddressPolicy=persistent

ON CONTAINER:

networkctl
WARNING: systemd-networkd is not running, output will be incomplete.

IDX LINK  TYPE     OPERATIONAL SETUP
  1 lo    loopback n/a         unmanaged
  2 host0 ether    n/a         unmanaged

cat /usr/lib/systemd/network/80-container-ve.network
#  SPDX-License-Identifier: LGPL-2.1+
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

# This network file matches the host-side of the virtual Ethernet link
# created by systemd-nspawn's --network-veth switch. See systemd-nspawn(1) for
# details.

[Match]
Name=ve-*
Driver=veth

[Network]
# Default to using a /28 prefix, giving up to 13 addresses per container.
Address=0.0.0.0/28
LinkLocalAddressing=yes
DHCPServer=yes
IPMasquerade=yes
LLDP=yes
EmitLLDP=customer-bridge

cat /usr/lib/systemd/network/80-container-host0.network
#  SPDX-License-Identifier: LGPL-2.1+
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

# This network file matches the container-side of the virtual Ethernet link
# created by systemd-nspawn's --network-veth switch. See systemd-nspawn(1) for
# details.

[Match]
Virtualization=container
Name=host0

[Network]
DHCP=yes
LinkLocalAddressing=yes
LLDP=yes
EmitLLDP=customer-bridge

[DHCP]
UseTimezone=yes

ip address show host0
2: host0@if9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a2:32:d8:4f:17:cf brd ff:ff:ff:ff:ff:ff link-netnsid 0

networkctl status host0
WARNING: systemd-networkd is not running, output will be incomplete.

Failed to query link bit rates: The name org.freedesktop.network1 was not provided by any .service files
Failed to query link DHCP leases: The name org.freedesktop.network1 was not provided by any .service files
● 2: host0
                     Link File: n/a
                  Network File: n/a
                          Type: ether
                         State: n/a (unmanaged)
                    HW Address: a2:32:d8:4f:17:cf
                           MTU: 1500 (min: 68, max: 65535)
                         QDisc: noop
  IPv6 Address Generation Mode: eui64
          Queue Length (Tx/Rx): 1/1
              Auto negotiation: no
                         Speed: 10Gbps
                        Duplex: full
                          Port: tp

systemctl status systemd-networkd
● systemd-networkd.service - Network Service
     Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2020-10-13 14:25:08 UTC; 14s ago
TriggeredBy: ● systemd-networkd.socket
       Docs: man:systemd-networkd.service(8)
    Process: 488 ExecStart=/usr/lib/systemd/systemd-networkd (code=exited, status=200/CHDIR)
   Main PID: 488 (code=exited, status=200/CHDIR)

Oct 13 14:25:08 buildspawn systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 5.
Oct 13 14:25:08 buildspawn systemd[1]: Stopped Network Service.
Oct 13 14:25:08 buildspawn systemd[1]: systemd-networkd.service: Start request repeated too quickly.
Oct 13 14:25:08 buildspawn systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
Oct 13 14:25:08 buildspawn systemd[1]: Failed to start Network Service.

cat /etc/systemd/network/all-ethernet.network
[Match]
Type=ether

[Network]
DHCP=yes

It's oposite of yours as my host0 is completely in down state and ve-* is in configuring state. I have no idea why it just don't work.
Note that I'm using static ip for nic on host but it shouldn't be problem.

nosada · 2020-10-14T14:44:34Z

How about /usr/lib/systemd/network/80-container-ve.network in host? You showed it in container, not host.

And your log shows systemd-networkd not running in container. Would you show me your journalctl -u systemd-nspawn@buildspawn after starting container?

Also, any changes when you use kernel providedlinux instead of linux-lts?

Updated: last line may not related for this issue because of https://wiki.archlinux.org/index.php/Linux_Containers#Privileged_containers_or_unprivileged_containers...:

The Arch linux, linux-lts and linux-zen kernel packages currently provide out-of-the-box support for unprivileged containers.

tmpm697 · 2020-10-14T17:38:47Z

on HOST:

cat /usr/lib/systemd/network/80-container-ve.network
#  SPDX-License-Identifier: LGPL-2.1+
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

# This network file matches the host-side of the virtual Ethernet link
# created by systemd-nspawn's --network-veth switch. See systemd-nspawn(1) for
# details.

[Match]
Name=ve-*
Driver=veth

[Network]
# Default to using a /28 prefix, giving up to 13 addresses per container.
Address=0.0.0.0/28
LinkLocalAddressing=yes
DHCPServer=yes
IPMasquerade=yes
LLDP=yes
EmitLLDP=customer-bridge

vi /etc/systemd/nspawn/buildspawn.nspawn

[Exec]
PrivateUsers=true
NotifyReady=true

[Files]
# Commented out below also works fine
# Bind=/ramdisk/scratch/:/scratch/

[Network]
Private=yes
VirtualEthernet=yes

journalctl -u systemd-nspawn@buildspawn log here

I use linux-lts, not tried with linux yet.

iptables.service and nftables.service are disabled.

nosada · 2020-10-18T09:08:40Z

Thank you.

Your 80-container-ve.network seems to be fine, so I think there's no fault for your configuration.
It seems there are some problems in networkd & resolved in nspawn container with error: Failed to connect stdout to the journal socket, ignoring: Permission denied. I googled the error message, but there's no result for nspawn.

I don't have any answer / solution for you now...
This may not be caused by mkosi or this repository, I think.

You might want to consider asking for your issue in ArchWiki or https://github.com/systemd/systemd/issues.

nosada mentioned this issue Oct 18, 2020

Can't get builder shell? #8

Closed

nosada closed this as completed Apr 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

No internet inside container? #6

No internet inside container? #6

tmpm697 commented Sep 30, 2020

nosada commented Oct 1, 2020

tmpm697 commented Oct 1, 2020 •

edited

Loading

nosada commented Oct 2, 2020

nosada commented Oct 11, 2020 •

edited

Loading

tmpm697 commented Oct 11, 2020

nosada commented Oct 12, 2020

tmpm697 commented Oct 13, 2020 •

edited

Loading

nosada commented Oct 14, 2020 •

edited

Loading

tmpm697 commented Oct 14, 2020 •

edited

Loading

nosada commented Oct 18, 2020 •

edited

Loading

No internet inside container? #6

No internet inside container? #6

Comments

tmpm697 commented Sep 30, 2020

nosada commented Oct 1, 2020

tmpm697 commented Oct 1, 2020 • edited Loading

nosada commented Oct 2, 2020

nosada commented Oct 11, 2020 • edited Loading

tmpm697 commented Oct 11, 2020

nosada commented Oct 12, 2020

tmpm697 commented Oct 13, 2020 • edited Loading

nosada commented Oct 14, 2020 • edited Loading

tmpm697 commented Oct 14, 2020 • edited Loading

nosada commented Oct 18, 2020 • edited Loading

tmpm697 commented Oct 1, 2020 •

edited

Loading

nosada commented Oct 11, 2020 •

edited

Loading

tmpm697 commented Oct 13, 2020 •

edited

Loading

nosada commented Oct 14, 2020 •

edited

Loading

tmpm697 commented Oct 14, 2020 •

edited

Loading

nosada commented Oct 18, 2020 •

edited

Loading