Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(netlink): detect ipv6 support level #2523

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

feat(netlink): detect ipv6 support level #2523

wants to merge 2 commits into from

Conversation

qdm12
Copy link
Owner

@qdm12 qdm12 commented Oct 14, 2024

  • 'supported' if one ipv6 route is found that is not loopback and not a default route
  • 'internet' if one default ipv6 route is found

@diamkil
Copy link

diamkil commented Nov 11, 2024

I just tried this and the IPv6 detection does not seem to correctly work for me. I have some link-locals IPv6 but not a routable IPv6, yet it appears to detect IPv6 support as Internet.

========================================
========================================
=============== gluetun ================
========================================
=========== Made with ❤️ by ============
======= https://github.com/qdm12 =======
========================================
========================================

Running version pr-2523 built on 2024-10-15T13:27:08.150Z (commit 8c28d9e)

📣 All control server routes will become private by default after the v3.41.0 release

🔧 Need help? ☕ Discussion? https://github.com/qdm12/gluetun/discussions/new/choose
🐛 Bug? ✨ New feature? https://github.com/qdm12/gluetun/issues/new/choose
💻 Email? [email protected]
💰 Help me? https://www.paypal.me/qmcgaw https://github.com/sponsors/qdm12
2024-11-11T20:11:55Z INFO [routing] default route found: interface eth0, gateway 172.25.0.1, assigned IP 172.25.0.2 and family v4
2024-11-11T20:11:55Z INFO [routing] default route found: interface eth0, gateway fd34:5678:9abc::1, assigned IP fd34:5678:9abc::2 and family v6
2024-11-11T20:11:55Z INFO [routing] local ethernet link found: eth0
2024-11-11T20:11:55Z INFO [routing] local ipnet found: 172.25.0.0/16
2024-11-11T20:11:55Z INFO [routing] local ipnet found: fd34:5678:9abc::/64
2024-11-11T20:11:55Z INFO [routing] local ipnet found: fe80::/64
2024-11-11T20:11:55Z INFO [firewall] enabling...
2024-11-11T20:11:55Z DEBUG [firewall] /sbin/iptables --policy INPUT DROP
2024-11-11T20:11:55Z DEBUG [firewall] /sbin/iptables --policy OUTPUT DROP
2024-11-11T20:11:55Z DEBUG [firewall] /sbin/iptables --policy FORWARD DROP
2024-11-11T20:11:55Z DEBUG [firewall] /sbin/ip6tables --policy INPUT DROP
2024-11-11T20:11:55Z DEBUG [firewall] /sbin/ip6tables --policy OUTPUT DROP
2024-11-11T20:11:55Z DEBUG [firewall] /sbin/ip6tables --policy FORWARD DROP
2024-11-11T20:11:55Z DEBUG [firewall] /sbin/iptables --append INPUT -i lo -j ACCEPT
2024-11-11T20:11:55Z DEBUG [firewall] /sbin/ip6tables --append INPUT -i lo -j ACCEPT
2024-11-11T20:11:55Z DEBUG [firewall] /sbin/iptables --append OUTPUT -o lo -j ACCEPT
2024-11-11T20:11:55Z DEBUG [firewall] /sbin/ip6tables --append OUTPUT -o lo -j ACCEPT
2024-11-11T20:11:55Z DEBUG [firewall] /sbin/iptables --append OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
2024-11-11T20:11:55Z DEBUG [firewall] /sbin/ip6tables --append OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
2024-11-11T20:11:55Z DEBUG [firewall] /sbin/iptables --append INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
2024-11-11T20:11:55Z DEBUG [firewall] /sbin/ip6tables --append INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
2024-11-11T20:11:55Z DEBUG [firewall] /sbin/iptables --append OUTPUT -o eth0 -s 172.25.0.2 -d 172.25.0.0/16 -j ACCEPT
2024-11-11T20:11:55Z DEBUG [firewall] /sbin/ip6tables --append OUTPUT -o eth0 -d ff02::1:ff00:0/104 -j ACCEPT
2024-11-11T20:11:55Z DEBUG [firewall] /sbin/ip6tables --append OUTPUT -o eth0 -s fd34:5678:9abc::2 -d fd34:5678:9abc::/64 -j ACCEPT
2024-11-11T20:11:55Z DEBUG [firewall] /sbin/ip6tables --append OUTPUT -o eth0 -s fd34:5678:9abc::2 -d fe80::/64 -j ACCEPT
2024-11-11T20:11:55Z DEBUG [firewall] /sbin/iptables --append INPUT -i eth0 -d 172.25.0.0/16 -j ACCEPT
2024-11-11T20:11:55Z DEBUG [firewall] /sbin/ip6tables --append INPUT -i eth0 -d fd34:5678:9abc::/64 -j ACCEPT
2024-11-11T20:11:55Z DEBUG [firewall] /sbin/ip6tables --append INPUT -i eth0 -d fe80::/64 -j ACCEPT
2024-11-11T20:11:55Z INFO [firewall] enabled successfully
2024-11-11T20:11:57Z INFO [storage] merging by most recent 20553 hardcoded servers and 20553 servers read from /gluetun/servers.json
2024-11-11T20:11:58Z DEBUG [netlink] IPv6 is supported by link eth0
2024-11-11T20:11:58Z DEBUG [netlink] IPv6 is supported by link eth0
2024-11-11T20:11:58Z DEBUG [netlink] IPv6 internet access is enabled on link eth0

ip route on the host has a default route
ip -6 route on the host has no default route

@qdm12
Copy link
Owner Author

qdm12 commented Nov 15, 2024

@diamkil thanks for trying this out! 💯

Can you try docker pull qmcgaw/gluetun:pr-2523 and restart the gluetun container to see what debug logs it produces? I've added:

  • in d7f047e a debug log of each route being checked for IPv6
  • a possible fix to skip IPv4 routes properly in 22be52e

Thank you!!

@diamkil
Copy link

diamkil commented Nov 15, 2024

Here's the logs of the new version!

========================================
=============== gluetun ================
========================================
=========== Made with ❤️ by ============
======= https://github.com/qdm12 =======
========================================
========================================

Running version pr-2523 built on 2024-11-15T15:43:42.025Z (commit 5cec3c0)

📣 All control server routes will become private by default after the v3.41.0 release

🔧 Need help? ☕ Discussion? https://github.com/qdm12/gluetun/discussions/new/choose
🐛 Bug? ✨ New feature? https://github.com/qdm12/gluetun/issues/new/choose
💻 Email? [email protected]
💰 Help me? https://www.paypal.me/qmcgaw https://github.com/sponsors/qdm12
2024-11-15T21:57:47Z INFO [routing] default route found: interface eth0, gateway 172.25.0.1, assigned IP 172.25.0.2 and family v4
2024-11-15T21:57:47Z INFO [routing] default route found: interface eth0, gateway fd34:5678:9abc::1, assigned IP fd34:5678:9abc::2 and family v6
2024-11-15T21:57:47Z INFO [routing] local ethernet link found: eth0
2024-11-15T21:57:47Z INFO [routing] local ipnet found: 172.25.0.0/16
2024-11-15T21:57:47Z INFO [routing] local ipnet found: fd34:5678:9abc::/64
2024-11-15T21:57:47Z INFO [routing] local ipnet found: fe80::/64
2024-11-15T21:57:47Z INFO [firewall] enabling...
2024-11-15T21:57:47Z DEBUG [firewall] /sbin/iptables --policy INPUT DROP
2024-11-15T21:57:47Z DEBUG [firewall] /sbin/iptables --policy OUTPUT DROP
2024-11-15T21:57:47Z DEBUG [firewall] /sbin/iptables --policy FORWARD DROP
2024-11-15T21:57:47Z DEBUG [firewall] /sbin/ip6tables --policy INPUT DROP
2024-11-15T21:57:47Z DEBUG [firewall] /sbin/ip6tables --policy OUTPUT DROP
2024-11-15T21:57:47Z DEBUG [firewall] /sbin/ip6tables --policy FORWARD DROP
2024-11-15T21:57:47Z DEBUG [firewall] /sbin/iptables --append INPUT -i lo -j ACCEPT
2024-11-15T21:57:47Z DEBUG [firewall] /sbin/ip6tables --append INPUT -i lo -j ACCEPT
2024-11-15T21:57:47Z DEBUG [firewall] /sbin/iptables --append OUTPUT -o lo -j ACCEPT
2024-11-15T21:57:47Z DEBUG [firewall] /sbin/ip6tables --append OUTPUT -o lo -j ACCEPT
2024-11-15T21:57:47Z DEBUG [firewall] /sbin/iptables --append OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
2024-11-15T21:57:47Z DEBUG [firewall] /sbin/ip6tables --append OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
2024-11-15T21:57:47Z DEBUG [firewall] /sbin/iptables --append INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
2024-11-15T21:57:47Z DEBUG [firewall] /sbin/ip6tables --append INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
2024-11-15T21:57:47Z DEBUG [firewall] /sbin/iptables --append OUTPUT -o eth0 -s 172.25.0.2 -d 172.25.0.0/16 -j ACCEPT
2024-11-15T21:57:47Z DEBUG [firewall] /sbin/ip6tables --append OUTPUT -o eth0 -d ff02::1:ff00:0/104 -j ACCEPT
2024-11-15T21:57:47Z DEBUG [firewall] /sbin/ip6tables --append OUTPUT -o eth0 -s fd34:5678:9abc::2 -d fd34:5678:9abc::/64 -j ACCEPT
2024-11-15T21:57:47Z DEBUG [firewall] /sbin/ip6tables --append OUTPUT -o eth0 -s fd34:5678:9abc::2 -d fe80::/64 -j ACCEPT
2024-11-15T21:57:47Z DEBUG [firewall] /sbin/iptables --append INPUT -i eth0 -d 172.25.0.0/16 -j ACCEPT
2024-11-15T21:57:47Z DEBUG [firewall] /sbin/ip6tables --append INPUT -i eth0 -d fd34:5678:9abc::/64 -j ACCEPT
2024-11-15T21:57:47Z DEBUG [firewall] /sbin/ip6tables --append INPUT -i eth0 -d fe80::/64 -j ACCEPT
2024-11-15T21:57:47Z INFO [firewall] enabled successfully
2024-11-15T21:57:49Z INFO [storage] merging by most recent 20776 hardcoded servers and 20553 servers read from /gluetun/servers.json
2024-11-15T21:57:49Z DEBUG [netlink] Checking route (link eth0): netlink.Route{LinkIndex:122, Dst:netip.Prefix{ip:netip.Addr{addr:netip.uint128{hi:0xfd3456789abc0000, lo:0x0}, z:unique.Handle[net/netip.addrDetail]{value:(*netip.addrDetail)(0xc0000ea228)}}, bitsPlusOne:0x41}, Src:netip.Addr{addr:netip.uint128{hi:0x0, lo:0x0}, z:unique.Handle[net/netip.addrDetail]{value:(*netip.addrDetail)(nil)}}, Gw:netip.Addr{addr:netip.uint128{hi:0x0, lo:0x0}, z:unique.Handle[net/netip.addrDetail]{value:(*netip.addrDetail)(nil)}}, Priority:256, Family:10, Table:254, Type:1}
2024-11-15T21:57:49Z DEBUG [netlink] IPv6 is supported by link eth0
2024-11-15T21:57:49Z DEBUG [netlink] Checking route (link eth0): netlink.Route{LinkIndex:122, Dst:netip.Prefix{ip:netip.Addr{addr:netip.uint128{hi:0xfe80000000000000, lo:0x0}, z:unique.Handle[net/netip.addrDetail]{value:(*netip.addrDetail)(0xc0000ea228)}}, bitsPlusOne:0x41}, Src:netip.Addr{addr:netip.uint128{hi:0x0, lo:0x0}, z:unique.Handle[net/netip.addrDetail]{value:(*netip.addrDetail)(nil)}}, Gw:netip.Addr{addr:netip.uint128{hi:0x0, lo:0x0}, z:unique.Handle[net/netip.addrDetail]{value:(*netip.addrDetail)(nil)}}, Priority:256, Family:10, Table:254, Type:1}
2024-11-15T21:57:49Z DEBUG [netlink] IPv6 is supported by link eth0
2024-11-15T21:57:49Z DEBUG [netlink] Checking route (link eth0): netlink.Route{LinkIndex:122, Dst:netip.Prefix{ip:netip.Addr{addr:netip.uint128{hi:0x0, lo:0x0}, z:unique.Handle[net/netip.addrDetail]{value:(*netip.addrDetail)(0xc0000ea228)}}, bitsPlusOne:0x1}, Src:netip.Addr{addr:netip.uint128{hi:0x0, lo:0x0}, z:unique.Handle[net/netip.addrDetail]{value:(*netip.addrDetail)(nil)}}, Gw:netip.Addr{addr:netip.uint128{hi:0xfd3456789abc0000, lo:0x1}, z:unique.Handle[net/netip.addrDetail]{value:(*netip.addrDetail)(0xc0000ea228)}}, Priority:1024, Family:10, Table:254, Type:1}
2024-11-15T21:57:49Z DEBUG [netlink] IPv6 internet access is enabled on link eth0

@qdm12
Copy link
Owner Author

qdm12 commented Nov 16, 2024

Thanks! It looks like the last route checked as a destination ::0 "unspecified", which is usually for default routes. But given my debug logs are rather hard to read (whoops, I should had wrote a little helping formatting code), what do you get from:

docker run --rm alpine:3.20 ip -6 route show table all

To find out why the heck that ::0 destination route is present, but not really the "internet" route.

@diamkil
Copy link

diamkil commented Nov 16, 2024

Here's the results of
docker run --network=gluetun_network --rm alpine:3.20 ip -6 route show table all (Using the same network as the gluetun container):

fd34:5678:9abc::/64 dev eth0  metric 256
fe80::/64 dev eth0  metric 256
default via fd34:5678:9abc::1 dev eth0  metric 1024
local ::1 dev lo table local  metric 0
local fd34:5678:9abc::b dev eth0 table local  metric 0
local fe80::42:acff:fe19:b dev eth0 table local  metric 0
multicast ff00::/8 dev eth0 table local  metric 256

Start of my docker network inspect gluetun_network

"Name": "gluetun_network",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": true,
"IPAM": {
    "Driver": "default",
    "Options": {},
    "Config": [
        {
            "Subnet": "172.25.0.0/16",
            "Gateway": "172.25.0.1"
        },
        {
            "Subnet": "fd34:5678:9abc::/64",
            "Gateway": "fd34:5678:9abc::1"
        }

Gateway is present but that's used to contact the host over IPv6, not sure if it's the network config that's problematic?

Just to be safe, here's the result of docker run --rm alpine:3.20 ip -6 route show table all (without the custom network specified):

fd12:3456:789a::/64 dev eth0  metric 256
fe80::/64 dev eth0  metric 256
default via fd12:3456:789a::1 dev eth0  metric 1024
local ::1 dev lo table local  metric 0
local fd12:3456:789a::242:ac11:3 dev eth0 table local  metric 0
local fe80::42:acff:fe11:3 dev eth0 table local  metric 0
multicast ff00::/8 dev eth0 table local  metric 256

@qdm12
Copy link
Owner Author

qdm12 commented Nov 18, 2024

Thank you for the detailed investigation 💯
The responsible route triggering the "internet detected" is default via fd34:5678:9abc::1 dev eth0 metric 1024 (destination is default aka unspecified aka ::0). Unfortunately, cross comparing with my setup, I can't really tell just from ip routes if your have ipv6 internet access or not. One last thing to try:

docker run --rm alpine:3.20 ip a
docker run --rm alpine:3.20 ip -6 a

(referencing my previous comment):

So it seems you're right, the two factors indicating ipv6 support are inet6 fc00::242:ac11:2/7 scope global nodad (ipv6 address with scope global on default interface eth0) or/and default via fc00::1 dev eth0 metric 1024 pref medium (ipv6 route with default destination on default interface eth0). I can check for both in the code, but for now I limited it to find a default route for simplicity's sake.

@diamkil
Copy link

diamkil commented Nov 25, 2024

Sorry for the delay, here's the result of both commands:

docker run --rm alpine:3.20 ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
63: eth0@if64: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:ac:11:00:03 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.3/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fd12:3456:789a::242:ac11:3/64 scope global flags 02
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:3/64 scope link tentative
       valid_lft forever preferred_lft forever

docker run --rm alpine:3.20 ip -6 a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
65: eth0@if66: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 state UP
    inet6 fd12:3456:789a::242:ac11:3/64 scope global flags 02
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:3/64 scope link tentative
       valid_lft forever preferred_lft forever

IPv6 seems to show scope global too, unfortunately.

@qdm12
Copy link
Owner Author

qdm12 commented Dec 10, 2024

Sorry for the delay getting back; so clearly after comparing everything, there is no way to discern your routes/links from someone with full IPv6 support. As a consequence, if a default IPv6 route is found, then a final check dialing an IPv6 address (configurable, for now cloudflare.com IPv6 address) would be done, with the firewall only allowing output traffic to that address. I'm working on it, should be done around tomorrow!

@qdm12
Copy link
Owner Author

qdm12 commented Dec 12, 2024

@diamkil when you have the time, can you try repulling qmcgaw/gluetun:pr-2523 and see how it does?
Now if a default non-loopback IPv6 route is found, gluetun allows through the firewall tcp output traffic only to the ipv6:port defined in IPV6_CHECK_ADDRESS (defaults to cloudflare.com [2606:4700::6810:84e5]:443, something that can be changed, happy to hear suggestions), and does a TCP dial to that ip:port. If it succeeds, then it's "ipv6 internet supported", otherwise it's just "ipv6 supported".

@qdm12 qdm12 force-pushed the ivp6-level branch 4 times, most recently from 70a9bb7 to e3e6431 Compare December 14, 2024 20:53
- 'supported' if one ipv6 route is found that is not loopback and not a default route
- 'internet' if one default ipv6 route is found
- If a default IPv6 route is found, query the ip:port defined by `IPV6_CHECK_ADDRESS` to check for internet access
@Tuxie
Copy link

Tuxie commented Dec 25, 2024

I get this with the latest :pr-2523

2024-12-25T22:47:32+01:00 INFO [routing] default route found: interface eth0, gateway 172.20.0.1, assigned IP 172.20.0.2 and family v4
2024-12-25T22:47:32+01:00 DEBUG [netlink] ip -4 rule list
2024-12-25T22:47:32+01:00 DEBUG [netlink] ip -6 rule list
2024-12-25T22:47:32+01:00 DEBUG [netlink] ip -f 0 rule add from 172.20.0.2/32 lookup 200 pref 100
2024-12-25T22:47:32+01:00 INFO [routing] adding route for 0.0.0.0/0
2024-12-25T22:47:32+01:00 DEBUG [routing] ip route replace 0.0.0.0/0 via 172.20.0.1 dev eth0 table 200
2024-12-25T22:47:32+01:00 INFO [firewall] setting allowed subnets...
2024-12-25T22:47:32+01:00 INFO [routing] default route found: interface eth0, gateway 172.20.0.1, assigned IP 172.20.0.2 and family v4
2024-12-25T22:47:32+01:00 DEBUG [netlink] ip -4 rule list
2024-12-25T22:47:32+01:00 DEBUG [netlink] ip -6 rule list
2024-12-25T22:47:32+01:00 DEBUG [netlink] ip -f 0 rule add to 172.20.0.0/16 lookup 254 pref 98
2024-12-25T22:47:32+01:00 DEBUG [netlink] ip -4 rule list
2024-12-25T22:47:32+01:00 DEBUG [netlink] ip -6 rule list
2024-12-25T22:47:32+01:00 DEBUG [netlink] ip -f 0 rule add to fe80::/64 lookup 254 pref 98
2024-12-25T22:47:32+01:00 INFO [dns] using plaintext DNS at address 1.1.1.1
2024-12-25T22:47:32+01:00 INFO [http server] http server listening on [::]:8000
2024-12-25T22:47:32+01:00 DEBUG [wireguard] Wireguard server public key: i6ejwlJ3Qz9woJeypdOXgPVlnl7Kr3TNKZa9NyGlgFw=
2024-12-25T22:47:32+01:00 DEBUG [wireguard] Wireguard client private key: oFe...30=
2024-12-25T22:47:32+01:00 DEBUG [wireguard] Wireguard pre-shared key: [not set]
2024-12-25T22:47:32+01:00 INFO [firewall] allowing VPN connection...
2024-12-25T22:47:32+01:00 INFO [healthcheck] listening on 127.0.0.1:9999
2024-12-25T22:47:32+01:00 DEBUG [firewall] /sbin/iptables --append OUTPUT -d 217.64.148.71 -o eth0 -p udp -m udp --dport 9929 -j ACCEPT
2024-12-25T22:47:32+01:00 DEBUG [firewall] /sbin/iptables --append OUTPUT -o tun0 -j ACCEPT
2024-12-25T22:47:32+01:00 DEBUG [firewall] /sbin/ip6tables --append OUTPUT -o tun0 -j ACCEPT
2024-12-25T22:47:32+01:00 INFO [wireguard] Using available kernelspace implementation
2024-12-25T22:47:32+01:00 INFO [wireguard] Connecting to 217.64.148.71:9929
2024-12-25T22:47:32+01:00 DEBUG [wireguard] closing controller client...
2024-12-25T22:47:32+01:00 DEBUG [wireguard] shutting down link...
2024-12-25T22:47:32+01:00 ERROR [vpn] cannot add route for interface: adding route for destination ::/0: adding route for link tun0, destination ::/0 and table 51820: no such device
2024-12-25T22:47:32+01:00 INFO [vpn] retrying in 15s
2024-12-25T22:47:32+01:00 DEBUG [wireguard] deleting link...
2024-12-25T22:47:38+01:00 INFO [healthcheck] program has been unhealthy for 6s: restarting VPN (healthcheck error: dialing: dial tcp4: lookup cloudflare.com on 1.1.1.1:53: write udp 172.20.0.2:54367->1.1.1.1:53: write: operation not permitted)
2024-12-25T22:47:38+01:00 INFO [healthcheck] 👉 See https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md
2024-12-25T22:47:38+01:00 INFO [healthcheck] DO NOT OPEN AN ISSUE UNLESS YOU READ AND TRIED EACH POSSIBLE SOLUTION

Then it healthcheck-loops from there until it dies.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants