Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GET http://localhost:9993/status X-ZT1-Auth:... fails over IPv4 but works over IPv6 #2151

Closed
dch opened this issue Oct 18, 2023 · 18 comments
Closed

Comments

@dch
Copy link
Contributor

dch commented Oct 18, 2023

Please let us know

There's a regression between 1.12.0 and 1.12.1 that breaks ipv4 localhost query
via zerotier-cli info. I've only just noticed while adding a new node that
this is broken.

Under v1.12.1:

# sudo zerotier-cli status
401 status {}

# zerotier-cli dump
Error connecting to the ZeroTier service: {}

Please check that the service is running and that
TCP port 9993 can be contacted via 127.0.0.1.

# sockstat -46lp 9993
USER     COMMAND    PID   FD  PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
root     zerotier-o 85664 7   tcp4   *:9993                *:*
root     zerotier-o 85664 8   tcp46  *:9993                *:*
root     zerotier-o 85664 14  udp4   10.0.0.112:9993       *:*

But clearly the daemon is up, and listening.

We'd like to see, as under previous versions, the info.

Under v1.12.0:

> sudo zerotier-cli info
200 info 7c5005f22b 1.12.0 ONLINE
  • environment is ansible-managed
  • runs ZT since over a decade probably
  • OS: FreeBSD (various versions)
  • ZT: v1.12.2 (everywhere)
  • authtoken.secret hasn't changed in years on nodes

Running the same thing using tcpdump/ngrep, we see that this is basically the same as running an HTTP GET to http://localhost:9993/status with header X-ZT1-Auth:my_secret so let's do that with curl:

$ curl -v http://localhost:9993/status -4HX-ZT1-Auth:...
< HTTP/1.1 401 Unauthorized
...
{}

$ curl -v http://localhost:9993/status -6HX-ZT1-Auth:...
< HTTP/1.1 200 OK
...
{"address":"9bbbdbfdd2","clock":1697648307359,"config":{"settings": ....}

The -v4 version should be authorized correctly, but isn't, whether via zerotier-cli or curl.

@dch dch changed the title GET http://localhost:9993/status X-ZT1-Auth:... fails over IPv4 but works over IPv6 GET http://localhost:9993/status X-ZT1-Auth:... fails over IPv4 but works over IPv6 Oct 18, 2023
@laduke
Copy link
Contributor

laduke commented Oct 18, 2023

Hello! Thanks for reporting. This isn't happening on my mac; both curl -4 and curl -6 work. So that's strange.

Are there any error messages when zerotier is starting up?

"authCheck" is around here:

if (remoteAddr.ipScope() == InetAddress::IP_SCOPE_LOOPBACK) {

@dch
Copy link
Contributor Author

dch commented Oct 19, 2023

No, there's no error messages that I see running a gmake -j debug flavoured build. This only occurs on FreeBSD ofc. I'll throw some printfs in and see what happens. the only change beween 1.12.0 and 1.12.1 is this splitting of address functionality for linux.

@dch
Copy link
Contributor Author

dch commented Oct 19, 2023

ok I think we can see why the auth fails :D but do the numbers here mean anything to you?

if (remoteAddr.ipScope() == InetAddress::IP_SCOPE_LOOPBACK) {

zt1flo98dm17np8
authCheck: /status
remoteAddr.ipScope() = 4
IP_SCOPE_LOOPBACK() = 2

@dch
Copy link
Contributor Author

dch commented Oct 19, 2023

ifconfig output if that's relevant

lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet 127.0.0.1 netmask 0xff000000
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
wlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=0
	ether 00:28:f8:d0:91:52
	inet 172.16.2.21 netmask 0xffffff00 broadcast 172.16.2.255
	inet6 fe80::228:f8ff:fed0:9152%wlan0 prefixlen 64 scopeid 0x2
	groups: wlan
	ssid skunkwerks channel 48 (5240 MHz 11a) bssid 80:2a:a8:85:e2:a3
	regdomain ETSI2 country AT authmode WPA2/802.11i privacy ON
	deftxkey UNDEF AES-CCM 2:128-bit AES-CCM 3:128-bit txpower 17 bmiss 10
	mcastrate 6 mgmtrate 6 scanvalid 60 wme roaming MANUAL
	parent interface: iwm0
	media: IEEE 802.11 Wireless Ethernet OFDM/54Mbps mode 11a
	status: associated
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo1: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet 100.64.0.0 netmask 0xffff8000
	inet 100.64.0.1 netmask 0xffffffff
	inet 100.64.0.2 netmask 0xffffffff
	inet 100.64.0.3 netmask 0xffffffff
	inet 100.64.0.4 netmask 0xffffffff
	inet 100.64.0.5 netmask 0xffffffff
	inet 100.64.0.6 netmask 0xffffffff
	inet 100.64.0.7 netmask 0xffffffff
	inet 100.64.0.8 netmask 0xffffffff
	inet 100.64.0.9 netmask 0xffffffff
	inet 100.64.0.10 netmask 0xffffffff
	inet 100.64.0.11 netmask 0xffffffff
	inet 100.64.0.12 netmask 0xffffffff
	inet 100.64.0.13 netmask 0xffffffff
	inet 100.64.0.14 netmask 0xffffffff
	inet 100.64.0.15 netmask 0xffffffff
	inet6 fe80::1%lo1 prefixlen 64 scopeid 0x3
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ztagim5o45dhe4c: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 5000 mtu 2800
	options=80000<LINKSTATE>
	ether 8e:23:63:d1:3c:17
	hwaddr 58:9c:fc:10:ff:d3
	inet6 fca2:927d:4d9b:bbdb:fdd2::1 prefixlen 40
	inet6 fe80::8c23:63ff:fed1:3c17%ztagim5o45dhe4c prefixlen 64 scopeid 0x4
	groups: tap
	media: Ethernet 1000baseT <full-duplex>
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	Opened by PID 5486
zt1flo98dm17np8: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 5000 mtu 2800
	options=80000<LINKSTATE>
	ether 2a:44:a8:b7:be:db
	hwaddr 58:9c:fc:00:0f:27
	inet6 fc7b:c4d6:6b9b:bbdb:fdd2::1 prefixlen 40
	inet6 fe80::2844:a8ff:feb7:bedb%zt1flo98dm17np8 prefixlen 64 scopeid 0x5
	groups: tap
	media: Ethernet 1000baseT <full-duplex>
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	Opened by PID 5486

@laduke
Copy link
Contributor

laduke commented Oct 19, 2023

but do the numbers here mean anything to you?

Not really but they're in here

InetAddress::IpScope InetAddress::ipScope() const

4 is IP_SCOPE_GLOBAL. When the ip doesn't match anything else in that switch

I'm curious what remoteAddr actually is? I think you can print it like this:

char buf[64];
fprintf(stderr, "remoteAddr %s\n", remoteAddr.toIpString(buf));

@dch
Copy link
Contributor Author

dch commented Oct 20, 2023

interesting:

$ doas ngrep -qid lo0 -W byline port 9993
interface: lo0 (127.0.0.0/255.0.0.0)
filter: (ip or ip6) and ( port 9993 )

T 127.0.0.1:10178 -> 127.0.0.1:9993 [AP]
GET /status HTTP/1.1.
X-ZT1-Auth: qrasbyedsabltduqypok8lqo.
.


T 127.0.0.1:9993 -> 127.0.0.1:10178 [AP]
HTTP/1.1 401 Unauthorized.
Content-Length: 2.
Content-Type: application/json.
Keep-Alive: timeout=5, max=5.
.


T 127.0.0.1:9993 -> 127.0.0.1:10178 [AP]
{}
$  doas ./zerotier-one /var/db/zerotier-one/
Starting Control Plane...
Starting V6 Control Plane...
ztagim5o45dhe4c
zt1flo98dm17np8
authCheck: /status
remoteAddr.ipScope() = 4
IP_SCOPE_LOOPBACK() = 2
remoteAddr ::ffff:127.0.0.1

something in between the IP stack (as seen by ngrep output) and zerotier turns this into a
v6v4 sort of thing.

@laduke
Copy link
Contributor

laduke commented Oct 20, 2023

That's really strange. I don't think we'll be able to do a fix and release in a timely fashion. You might be able to put ::ffff:127.0.0.1 in your local.conf allowManagementFrom as a work around.

@dch
Copy link
Contributor Author

dch commented Oct 23, 2023

That specific pattern didn't work, in the end I used :: and rely on the per-node authsecret and local firewall rules for security. Is that sufficient? It's certainly not ideal :-(

@laduke
Copy link
Contributor

laduke commented Oct 24, 2023

Seems like it to me, but I don't know the threat model, your requirements, etc...

remove 127.0.0.1 localhost from /etc/hosts :)

@dch
Copy link
Contributor Author

dch commented Mar 14, 2024

This basically broke new zerotier installs on all FreeBSD and TrueNAS systems, you can't add nodes unless find this issue.

Any chance this could get addressed for 1.14? The fix is likely to be quite trivial, to add the missing match to InetAddress.cpp.

In OneService.cpp#L1601-L1610 if (remoteAddr.ipScope() == InetAddress::IP_SCOPE_LOOPBACK) {

is called with these values:

zt1flo98dm17np8
authCheck: /status
remoteAddr.ipScope() = 4
IP_SCOPE_LOOPBACK() = 2
remoteAddr ::ffff:127.0.0.1

According to InetAddress.hpp#L67C1-L67C75 ::ffff:127.0.0.1 is identified as IP_SCOPE_GLOBAL which seems incorrect.

AFAICT this is an error in InetAddress.cpp which decides that 0xff_ff_7f_00_00_01 or something similar to that in whatever internal representation is used, is not local.

I will test on FreeBSD 13.x and 14.x again to see what I can track down, in particular what the memory representation of ::ffff:127.0.0.1 is, which is what is needed to check against.

@laduke
Copy link
Contributor

laduke commented Mar 14, 2024

yeah lets take a look

@dch
Copy link
Contributor Author

dch commented Mar 14, 2024

OK it aint pretty, but it does fix the issue... over to you c++ gurus to make it pretty/correct:

diff --git a/node/InetAddress.cpp b/node/InetAddress.cpp
index da1c7294..7a786e49 100644
--- a/node/InetAddress.cpp
+++ b/node/InetAddress.cpp
@@ -112,6 +112,20 @@ InetAddress::IpScope InetAddress::ipScope() const
                }       break;

                case AF_INET6: {
+         // Check for IPv4-mapped IPv6 addresses (::ffff::/96 prefix)
+            const struct sockaddr_in6* sa6 = reinterpret_cast<const struct sockaddr_in6*>(this);
+            if (sa6->sin6_addr.s6_addr[0] == 0 && sa6->sin6_addr.s6_addr[1] == 0 &&
+                sa6->sin6_addr.s6_addr[2] == 0 && sa6->sin6_addr.s6_addr[3] == 0 &&
+                sa6->sin6_addr.s6_addr[4] == 0 && sa6->sin6_addr.s6_addr[5] == 0 &&
+                sa6->sin6_addr.s6_addr[6] == 0 && sa6->sin6_addr.s6_addr[7] == 0 &&
+                sa6->sin6_addr.s6_addr[8] == 0 && sa6->sin6_addr.s6_addr[9] == 0 &&
+                sa6->sin6_addr.s6_addr[10] == 0xff && sa6->sin6_addr.s6_addr[11] == 0xff) {
+                // next 4 bytes should be IPv4 address 0x7f000001 / 127.0.0.1
+                uint32_t ipv4Part = *(reinterpret_cast<const uint32_t*>(&sa6->sin6_addr.s6_addr[12]));
+                if (ipv4Part == htonl(0x7f000001)) {
+                    return IP_SCOPE_LOOPBACK;
+                }
+            }
                        const unsigned char *ip = reinterpret_cast<const unsigned char *>(reinterpret_cast<const struct sockaddr_in6 *>(this)->sin6_addr.s6_addr);
                        if ((ip[0] & 0xf0) == 0xf0) {
                                if (ip[0] == 0xff) {
diff --git a/service/OneService.cpp b/service/OneService.cpp
index b06bcb9b..8f52ce58 100644
--- a/service/OneService.cpp
+++ b/service/OneService.cpp
@@ -1697,8 +1697,14 @@ public:
                                bool ipAllowed = false;
                                bool isAuth = false;
                                // If localhost, allow
+                               char buf[64];
+                               fprintf(stderr, "remoteAddr %s\n", remoteAddr.toIpString(buf));
                                if (remoteAddr.ipScope() == InetAddress::IP_SCOPE_LOOPBACK) {
+                                       fprintf(stderr, "ALLOWED\n");
                                        ipAllowed = true;
+                               } else
+                               {
+                                       fprintf(stderr, "DENIED\n");
                                }

                                if (!ipAllowed) {

@dch
Copy link
Contributor Author

dch commented Mar 14, 2024

was this perhaps a change in the underlying httplib that passes a different form of remote address through?

@laduke
Copy link
Contributor

laduke commented Mar 14, 2024

Not sure yet. Thanks for working on it.
Even curl -v -4 "http://127.0.0.1:9993/status pops out as ::ffff:127.0.0.1 inside of zerotier-one, but not on other operating systems.

@dch
Copy link
Contributor Author

dch commented Mar 14, 2024

once there's a more appropriate patch I'll backport it to the current FreeBSD port.

@laduke
Copy link
Contributor

laduke commented Mar 14, 2024

The httplib unsets the IPV6_V6ONLY socket option.
yhirose/cpp-httplib@b2203bb

This seems to make all IPv4 requests come in with the ::ffff:, as a v6 to 4 mapping thing.

Making it parse ::ffff:127.0.0.1 as loopback works.

But it also does the 6 to 4 thing to other ip addresses:

curl -4 "http://45.32.69.185:9993/ -> ::ffff:45.32.69.185
which might make configuring things like "allowManagementFrom" a little weird.

this patch also makes it work. Maybe we can make that a FreeBSD specific patch or something.

laduke added a commit that referenced this issue Mar 15, 2024
cpp-httplib  sets IPV6_V6ONLY to false on it's sockets.
On FreeBSD, this makes all ipv4 addresses get get prefixed with ::ffff:
it makes them IPv6 addresses mapped to v4.

This is a partial fix for #2151. The cli will work again.
Something should probably also be adjusted with the httplib.

If you want to, for example, use the `allowManagementFrom` option in
local.conf
you will need to prefix it with "::ffff:", "::ffff:1.2.3.4"
which is a little surprising and inconsistent between BSD and other OSs.
dch pushed a commit to skunkwerks/zt that referenced this issue Mar 16, 2024
cpp-httplib  sets IPV6_V6ONLY to false on it's sockets.
On FreeBSD, this makes all ipv4 addresses get get prefixed with ::ffff:
it makes them IPv6 addresses mapped to v4.

This is a partial fix for zerotier#2151. The cli will work again.
Something should probably also be adjusted with the httplib.

If you want to, for example, use the `allowManagementFrom` option in
local.conf
you will need to prefix it with "::ffff:", "::ffff:1.2.3.4"
which is a little surprising and inconsistent between BSD and other OSs.
@dch
Copy link
Contributor Author

dch commented Mar 16, 2024

Your improved version of my ugly hack looks great, let's roll - thanks!

BTW I tested this patch and wasn't able to run info or join ....

Perhaps, for a dual-stack address like localhost the AF_INET6 branch
never actually gets reached?

@dch
Copy link
Contributor Author

dch commented Mar 22, 2024

thanks, works a treat. I pulled this into FreeBSD 1.12.2 as an upstream patch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants