tor instances not running on the primary/first IP will be announced on the wrong IP and fail reachability test ('Address' needed) #153

nusenu · 2018-03-24T15:16:08Z

Tim reports that on a multi public IP exit only the first 2 instances on the first public IP confirm ORPort reachability:
https://lists.torproject.org/pipermail/tor-relays/2018-March/014891.html

Mär 22 02:19:47 tor Tor-185.220.100.25*3*_9000[586]: Your server (185.220.100.25*2*:9000) has not managed to confirm that its ORPort is reachable. Relays do not publish descriptors until their ORPort and DirPort are reachable. Please check your firewalls, ports, address, /etc/hosts file, etc.

Note the mismatch between the IP addresses.

We expect this to be covered by our use of OutboundBindAddress

Tim worked around the issue by adjusting the torrc template to add the Address config parameter which we do not add by default.

The text was updated successfully, but these errors were encountered:

RedDog99 · 2018-03-24T15:25:25Z

Here the requested data:

ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 52:54:00:21:08:b6 brd ff:ff:ff:ff:ff:ff
3: ens9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:df:68:24 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fedf:6824/64 scope link 
       valid_lft forever preferred_lft forever
4: ens10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:53:e1:e3 brd ff:ff:ff:ff:ff:ff
    inet 185.220.100.252/32 scope global ens10
       valid_lft forever preferred_lft forever
    inet 185.220.100.253/32 scope global ens10
       valid_lft forever preferred_lft forever
    inet 185.220.100.254/32 scope global ens10
       valid_lft forever preferred_lft forever
    inet 185.220.100.255/32 scope global ens10
       valid_lft forever preferred_lft forever
    inet6 2a0b:f4c0:16c:4::1/48 scope global 
       valid_lft forever preferred_lft forever
    inet6 2a0b:f4c0:16c:3::1/48 scope global 
       valid_lft forever preferred_lft forever
    inet6 2a0b:f4c0:16c:2::1/48 scope global 
       valid_lft forever preferred_lft forever
    inet6 2a0b:f4c0:16c:1::1/48 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe53:e1e3/64 scope link 
       valid_lft forever preferred_lft forever

ip ro

default via 185.220.100.128 dev ens10 
185.220.100.128 dev ens10 scope link

ip -6 ro

2a0b:f4c0:16c::/48 dev ens10 proto kernel metric 256  pref medium
fe80::/64 dev ens9 proto kernel metric 256  pref medium
fe80::/64 dev ens10 proto kernel metric 256  pref medium
default via 2a0b:f4c0:16c::1 dev ens10 metric 1024  pref medium

$ cat tor.yml

---
- hosts: relays
  vars:
    tor_ContactInfo: [email protected]
    tor_signingkeylifetime_days: 90
    tor_nickname: "F3 Netze"
    tor_ExitRelay: True
    tor_ExitPolicy: accept *:80,accept *:443,reject *:*
    tor_ExitNoticePage: True
    tor_AbuseEmailAddress: [email protected]
    tor_maxPublicIPs: 4
    tor_IPv6: False
    tor_IPv6Exit: False
  roles:
   - nusenu.relayor

nusenu · 2018-03-24T15:41:57Z

Thanks for the fast reply, could you also add

the log output (more lines up to the line already in the first post here)
ansible version
target OS (i.e. Debian 9)
the content of /etc/tor/instances/185.220.100.253_9000/torrc.... (you will find the old torrc file in that folder before you added the Address because old ansible makes backups of them before overriding)

thanks!

RedDog99 · 2018-03-24T15:49:17Z

the log output (more lines up to the line already in the first post here)

Mär 22 08:39:47 tor Tor-185.220.100.253_9000[586]: Your server (185.220.100.252:9000) has not managed to confirm that its ORPort is reachable. Relays do not publish descriptors until their ORPort and DirPort are reachable. Please 
check your firewalls, ports, address, /etc/hosts file, etc.
Mär 22 08:59:46 tor Tor-185.220.100.253_9000[586]: Heartbeat: It seems like we are not in the cached consensus.
Mär 22 08:59:46 tor Tor-185.220.100.253_9000[586]: Heartbeat: Tor's uptime is 11:59 hours, with 0 circuits open. I've sent 5.07 MB and received 23.43 MB.
Mär 22 08:59:46 tor Tor-185.220.100.253_9000[586]: Average packaged cell fullness: 13.655%. TLS write overhead: 10%
Mär 22 08:59:46 tor Tor-185.220.100.253_9000[586]: Circuit handshake stats since last time: 0/0 TAP, 0/0 NTor.
Mär 22 08:59:46 tor Tor-185.220.100.253_9000[586]: Since startup, we have initiated 0 v1 connections, 0 v2 connections, 0 v3 connections, and 183 v4 connections; and received 0 v1 connections, 0 v2 connections, 0 v3 connections, a
nd 0 v4 connections.
Mär 22 08:59:46 tor Tor-185.220.100.253_9000[586]: DoS mitigation since startup: 0 circuits rejected, 0 marked addresses. 0 connections closed. 0 single hop clients refused.
Mär 22 08:59:47 tor Tor-185.220.100.253_9000[586]: Your server (185.220.100.252:9000) has not managed to confirm that its ORPort is reachable. Relays do not publish descriptors until their ORPort and DirPort are reachable. Please 
check your firewalls, ports, address, /etc/hosts file, etc.
Mär 22 09:06:47 tor Tor-185.220.100.253_9000[586]: The IPv4 ORPort address 185.220.100.253 does not match the descriptor address 185.220.100.252. If you have a static public IPv4 address, use 'Address <IPv4>' and 'OutboundBindAddr
ess <IPv4>'. If you are behind a NAT, use two ORPort lines: 'ORPort <PublicPort> NoListen' and 'ORPort <InternalPort> NoAdvertise'.
Mär 22 09:06:47 tor Tor-185.220.100.253_9000[586]: The IPv4 DirPort address 185.220.100.253 does not match the descriptor address 185.220.100.252. If you have a static public IPv4 address, use 'Address <IPv4>' and 'OutboundBindAdd
ress <IPv4>'. If you are behind a NAT, use two DirPort lines: 'DirPort <PublicPort> NoListen' and 'DirPort <InternalPort> NoAdvertise'.
Mär 22 09:19:47 tor Tor-185.220.100.253_9000[586]: Your server (185.220.100.252:9000) has not managed to confirm that its ORPort is reachable. Relays do not publish descriptors until their ORPort and DirPort are reachable. Please 
check your firewalls, ports, address, /etc/hosts file, etc.
Mär 22 09:35:48 tor Tor-185.220.100.253_9000[586]: No circuits are opened. Relaxed timeout for circuit 1366 (a Testing circuit 3-hop circuit in state doing handshakes with channel state open) to 60000ms. However, it appears the ci
rcuit has timed out anyway. [1 similar message(s) suppressed in last 3600 seconds]
Mär 22 09:39:47 tor Tor-185.220.100.253_9000[586]: Your server (185.220.100.252:9000) has not managed to confirm that its ORPort is reachable. Relays do not publish descriptors until their ORPort and DirPort are reachable. Please 
check your firewalls, ports, address, /etc/hosts file, etc.
Mär 22 09:59:47 tor Tor-185.220.100.253_9000[586]: Your server (185.220.100.252:9000) has not managed to confirm that its ORPort is reachable. Relays do not publish descriptors until their ORPort and DirPort are reachable. Please 
check your firewalls, ports, address, /etc/hosts file, etc.
Mär 22 10:19:47 tor Tor-185.220.100.253_9000[586]: Your server (185.220.100.252:9000) has not managed to confirm that its ORPort is reachable. Relays do not publish descriptors until their ORPort and DirPort are reachable. Please 
check your firewalls, ports, address, /etc/hosts file, etc.
Mär 22 10:37:47 tor Tor-185.220.100.253_9000[586]: The IPv4 ORPort address 185.220.100.253 does not match the descriptor address 185.220.100.252. If you have a static public IPv4 address, use 'Address <IPv4>' and 'OutboundBindAddr
ess <IPv4>'. If you are behind a NAT, use two ORPort lines: 'ORPort <PublicPort> NoListen' and 'ORPort <InternalPort> NoAdvertise'.
Mär 22 10:37:47 tor Tor-185.220.100.253_9000[586]: The IPv4 DirPort address 185.220.100.253 does not match the descriptor address 185.220.100.252. If you have a static public IPv4 address, use 'Address <IPv4>' and 'OutboundBindAdd
ress <IPv4>'. If you are behind a NAT, use two DirPort lines: 'DirPort <PublicPort> NoListen' and 'DirPort <InternalPort> NoAdvertise'.
Mär 22 10:39:47 tor Tor-185.220.100.253_9000[586]: Your server (185.220.100.252:9000) has not managed to confirm that its ORPort is reachable. Relays do not publish descriptors until their ORPort and DirPort are reachable. Please 
check your firewalls, ports, address, /etc/hosts file, etc.
Mär 22 10:59:47 tor Tor-185.220.100.253_9000[586]: Your server (185.220.100.252:9000) has not managed to confirm that its ORPort is reachable. Relays do not publish descriptors until their ORPort and DirPort are reachable. Please 
check your firewalls, ports, address, /etc/hosts file, etc.
Mär 22 11:19:47 tor Tor-185.220.100.253_9000[586]: Your server (185.220.100.252:9000) has not managed to confirm that its ORPort is reachable. Relays do not publish descriptors until their ORPort and DirPort are reachable. Please 
check your firewalls, ports, address, /etc/hosts file, etc.
Mär 22 11:39:47 tor Tor-185.220.100.253_9000[586]: Your server (185.220.100.252:9000) has not managed to confirm that its ORPort is reachable. Relays do not publish descriptors until their ORPort and DirPort are reachable. Please 
check your firewalls, ports, address, /etc/hosts file, etc.
Mär 22 11:59:47 tor Tor-185.220.100.253_9000[586]: Your server (185.220.100.252:9000) has not managed to confirm that its ORPort is reachable. Relays do not publish descriptors until their ORPort and DirPort are reachable. Please 
check your firewalls, ports, address, /etc/hosts file, etc.
Mär 22 12:08:47 tor Tor-185.220.100.253_9000[586]: The IPv4 ORPort address 185.220.100.253 does not match the descriptor address 185.220.100.252. If you have a static public IPv4 address, use 'Address <IPv4>' and 'OutboundBindAddr
ess <IPv4>'. If you are behind a NAT, use two ORPort lines: 'ORPort <PublicPort> NoListen' and 'ORPort <InternalPort> NoAdvertise'.
Mär 22 12:08:47 tor Tor-185.220.100.253_9000[586]: The IPv4 DirPort address 185.220.100.253 does not match the descriptor address 185.220.100.252. If you have a static public IPv4 address, use 'Address <IPv4>' and 'OutboundBindAdd
ress <IPv4>'. If you are behind a NAT, use two DirPort lines: 'DirPort <PublicPort> NoListen' and 'DirPort <InternalPort> NoAdvertise'.
Mär 22 12:11:47 tor Tor-185.220.100.253_9000[586]: No circuits are opened. Relaxed timeout for circuit 1391 (a Testing circuit 3-hop circuit in state doing handshakes with channel state open) to 60000ms. However, it appears the ci
rcuit has timed out anyway. [1 similar message(s) suppressed in last 3600 seconds]
Mär 22 12:19:47 tor Tor-185.220.100.253_9000[586]: Your server (185.220.100.252:9000) has not managed to confirm that its ORPort is reachable. Relays do not publish descriptors until their ORPort and DirPort are reachable. Please 
check your firewalls, ports, address, /etc/hosts file, etc.
Mär 22 12:39:47 tor Tor-185.220.100.253_9000[586]: Your server (185.220.100.252:9000) has not managed to confirm that its ORPort is reachable. Relays do not publish descriptors until their ORPort and DirPort are reachable. Please 
check your firewalls, ports, address, /etc/hosts file, etc.
Mär 22 12:59:47 tor Tor-185.220.100.253_9000[586]: Your server (185.220.100.252:9000) has not managed to confirm that its ORPort is reachable. Relays do not publish descriptors until their ORPort and DirPort are reachable. Please 
check your firewalls, ports, address, /etc/hosts file, etc.
Mär 22 13:19:47 tor Tor-185.220.100.253_9000[586]: Your server (185.220.100.252:9000) has not managed to confirm that its ORPort is reachable. Relays do not publish descriptors until their ORPort and DirPort are reachable. Please 
check your firewalls, ports, address, /etc/hosts file, etc.
Mär 22 13:39:47 tor Tor-185.220.100.253_9000[586]: The IPv4 ORPort address 185.220.100.253 does not match the descriptor address 185.220.100.252. If you have a static public IPv4 address, use 'Address <IPv4>' and 'OutboundBindAddr
ess <IPv4>'. If you are behind a NAT, use two ORPort lines: 'ORPort <PublicPort> NoListen' and 'ORPort <InternalPort> NoAdvertise'.
Mär 22 13:39:47 tor Tor-185.220.100.253_9000[586]: The IPv4 DirPort address 185.220.100.253 does not match the descriptor address 185.220.100.252. If you have a static public IPv4 address, use 'Address <IPv4>' and 'OutboundBindAdd
ress <IPv4>'. If you are behind a NAT, use two DirPort lines: 'DirPort <PublicPort> NoListen' and 'DirPort <InternalPort> NoAdvertise'.
Mär 22 13:39:47 tor Tor-185.220.100.253_9000[586]: Your server (185.220.100.252:9000) has not managed to confirm that its ORPort is reachable. Relays do not publish descriptors until their ORPort and DirPort are reachable. Please 
check your firewalls, ports, address, /etc/hosts file, etc.
Mär 22 13:59:47 tor Tor-185.220.100.253_9000[586]: Your server (185.220.100.252:9000) has not managed to confirm that its ORPort is reachable. Relays do not publish descriptors until their ORPort and DirPort are reachable. Please 
check your firewalls, ports, address, /etc/hosts file, etc.
Mär 22 14:19:47 tor Tor-185.220.100.253_9000[586]: Your server (185.220.100.252:9000) has not managed to confirm that its ORPort is reachable. Relays do not publish descriptors until their ORPort and DirPort are reachable. Please 
check your firewalls, ports, address, /etc/hosts file, etc.
Mär 22 14:21:48 tor Tor-185.220.100.253_9000[586]: No circuits are opened. Relaxed timeout for circuit 1433 (a Testing circuit 3-hop circuit in state doing handshakes with channel state open) to 60000ms. However, it appears the ci
rcuit has timed out anyway. [1 similar message(s) suppressed in last 3600 seconds]
Mär 22 14:39:47 tor Tor-185.220.100.253_9000[586]: Your server (185.220.100.252:9000) has not managed to confirm that its ORPort is reachable. Relays do not publish descriptors until their ORPort and DirPort are reachable. Please 
check your firewalls, ports, address, /etc/hosts file, etc.
Mär 22 14:59:46 tor Tor-185.220.100.253_9000[586]: Heartbeat: It seems like we are not in the cached consensus.
Mär 22 14:59:46 tor Tor-185.220.100.253_9000[586]: Heartbeat: Tor's uptime is 17:59 hours, with 0 circuits open. I've sent 5.69 MB and received 30.27 MB.
Mär 22 14:59:46 tor Tor-185.220.100.253_9000[586]: Average packaged cell fullness: 13.655%. TLS write overhead: 10%
Mär 22 14:59:46 tor Tor-185.220.100.253_9000[586]: Circuit handshake stats since last time: 0/0 TAP, 0/0 NTor.
Mär 22 14:59:46 tor Tor-185.220.100.253_9000[586]: Since startup, we have initiated 0 v1 connections, 0 v2 connections, 0 v3 connections, and 204 v4 connections; and received 0 v1 connections, 0 v2 connections, 0 v3 connections, a
nd 0 v4 connections.

ansible version
ansible 2.4.3.0
target OS (i.e. Debian 9)
Debian 9.4
the content of /etc/tor/instances/185.220.100.253_9000/torrc.... (you will find the old torrc file in that folder before you added the Address because old ansible makes backups of them before overriding)

# ansible-relayor generated torrc configuration file
# Note: manual changes will be OVERWRITTEN on the next ansible-playbook run

OfflineMasterKey 1
RunAsDaemon 0
Log notice syslog
OutboundBindAddress 185.220.100.253
SocksPort 0
User _tor-185.220.100.253_9000
DataDirectory /var/lib/tor-instances/185.220.100.253_9000
ORPort 185.220.100.253:9000


DirPort 185.220.100.253:9001

SyslogIdentityTag 185.220.100.253_9000

ControlSocket 0
CookieAuthentication 0

Nickname F3Netze

Sandbox 1
# we are an exit relay!
ExitRelay 1
ExitPolicy accept *:80,accept *:443,reject *:*
DirPortFrontPage /etc/tor/instances/tor-exit-notice.html

ContactInfo [email protected]

MyFamily 0436eb493d58a21f22621c916b4e2285a10b513e,2017b0df257a9d6501f084b6d7aeaaf6852e9dc1,2bd4aaef7b149ee7c290dedd1cfa3259aa1db66b,6bcb964ab74e23f8986bda905697d3a6be08af28,719fd0fa327f3ccbcda0d4ea74c15ea110338942,939126ea4d25cb212a79746c133194f8a24c435b,9973e1e9730a58fdba9e112d2b3342d2c0d921b5,9bfc62520f4db61f5df99063ef95b142037d7aa4
# end of torrc

nusenu · 2018-03-24T16:19:03Z

thanks for providing that info!

tor version for the record: 0.3.2.10

An easy fix would be to simply add the following to the template:

Address {{ item.0.ipv4}}

if we run on public IPv4 addresses (no NAT) and if the user did not set tor_Address (#141) but before doing so I'd like to get to the bottom of this.

I sent an email to tor-dev:
https://lists.torproject.org/pipermail/tor-dev/2018-March/013031.html

nusenu · 2018-03-24T16:25:48Z

An easy fix would be to simply add the following to the template:

It is less obvious than I thought, because currently it would work even if we are behind NAT on public IPs, if we start doing what I suggested above things would no longer work in such (rare) cases.

…#153) Due to a bug in tor, instances that are not binding on the first public IPv4 address are not announcing their correct IP address (they use the first IP instead of the one configured via `OutboundBindAddress`) and therefor fail their reachability test and never upload a valid descriptor. This commit works around this by explicitly setting `Address` to the same IP address as the one used for `OutboundBindAddress`. Once the upstream bugs are fixed we should revert this change so we might also work in setups where public IPs are NATed to other public IPs. This bug affected all users which set `tor_maxPublicIPs` to a value greater than 1. Upstream bugs: https://trac.torproject.org/projects/tor/ticket/13953 https://trac.torproject.org/projects/tor/ticket/19919 Thanks to Tim Niemeyer (f3netze.de) for bringing this to our attention.

nusenu · 2018-03-25T10:27:19Z

This is apparently an old upstream bug in tor reported in 2014 here:
https://trac.torproject.org/projects/tor/ticket/13953
In this bug they decided to add a warning only to avoid breaking existing configs.
The actual fix is supposed to be in (not fixed yet):
https://trac.torproject.org/projects/tor/ticket/19919

Until upstream fixed this we will workaround this issue by sitting Address under certain conditions.

We set the Address parameter to {{ item.0.ipv4}} if

the operator intents to use more than one public IP by setting tor_maxPublicIPs to something >1
the server actually has more than one public IP address (tor_available_public_ipv4s | length > 1)
the operator did not specifically set tor_Address (Add support for manually specifying "Address" torrc option (for rare environments) #141)

Note: We do not set Address if the user has the default value of tor_maxPublicIPs (1) or has only one public IP

The fix will not require the operator to make any change to their configs (it should work out of the box).

@RedDog99 Could you test b90109a and confirm that it works for you? (you could use the -C and -D switches to simulate the change)

RedDog99 · 2018-03-25T11:18:19Z

I think -C prevents the "Check if /etc/resolv.conf contains any Google DNS servers (Exits only)" to run. Without it, the "Abort if Google DNS server is in use (Exits only)" fails. I removed the include for the google check temporary. There are some more things why -C -D may not work: "msg": "AnsibleUndefinedVariable: 'dict object' has no attribute 'stdout'" :(

After removing all stdouts (there is even one in the template), i got a new torrc and just the removed stdout changed. So yes, the change solves the problem for me. Thank you for your fast and kind help.

nusenu · 2018-03-25T12:54:25Z

thanks for testing, I'll be putting out a new release with this fix in the next few hours.

nusenu · 2018-03-25T15:11:11Z

workaround is in v0.3.3.

This ticket will remain open until the upstream issue
https://trac.torproject.org/projects/tor/ticket/19919
is solved.

nusenu · 2021-08-22T12:34:43Z

the upstream issue https://gitlab.torproject.org/tpo/core/tor/-/issues/21044 has been closed as wontfix.

we should be fine with our workaround b90109a

nusenu changed the title ~~potential bug: relayor fails to~~ potential bug: relayor fails to confirm ORPort reachability on 2th public IP Mar 24, 2018

nusenu changed the title ~~potential bug: relayor fails to confirm ORPort reachability on 2th public IP~~ potential bug: tor fails to confirm ORPort reachability on 2th public IP Mar 24, 2018

nusenu self-assigned this Mar 24, 2018

nusenu added bug upstream labels Mar 24, 2018

nusenu changed the title ~~potential bug: tor fails to confirm ORPort reachability on 2th public IP~~ tor instances not running on the primary/first IP will be announced on the wrong IP and fail reachability test ('Address' needed) Mar 25, 2018

nusenu added workaround_applied and removed bug labels Mar 25, 2018

nusenu added upstream_bug and removed upstream labels Apr 10, 2018

nusenu closed this as completed Aug 22, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tor instances not running on the primary/first IP will be announced on the wrong IP and fail reachability test ('Address' needed) #153

tor instances not running on the primary/first IP will be announced on the wrong IP and fail reachability test ('Address' needed) #153

nusenu commented Mar 24, 2018 •

edited

Loading

RedDog99 commented Mar 24, 2018 •

edited by nusenu

Loading

nusenu commented Mar 24, 2018 •

edited

Loading

RedDog99 commented Mar 24, 2018

nusenu commented Mar 24, 2018 •

edited

Loading

nusenu commented Mar 24, 2018 •

edited

Loading

nusenu commented Mar 25, 2018

RedDog99 commented Mar 25, 2018

nusenu commented Mar 25, 2018

nusenu commented Mar 25, 2018

nusenu commented Aug 22, 2021

tor instances not running on the primary/first IP will be announced on the wrong IP and fail reachability test ('Address' needed) #153

tor instances not running on the primary/first IP will be announced on the wrong IP and fail reachability test ('Address' needed) #153

Comments

nusenu commented Mar 24, 2018 • edited Loading

RedDog99 commented Mar 24, 2018 • edited by nusenu Loading

ip addr

ip ro

ip -6 ro

nusenu commented Mar 24, 2018 • edited Loading

RedDog99 commented Mar 24, 2018

nusenu commented Mar 24, 2018 • edited Loading

nusenu commented Mar 24, 2018 • edited Loading

nusenu commented Mar 25, 2018

RedDog99 commented Mar 25, 2018

nusenu commented Mar 25, 2018

nusenu commented Mar 25, 2018

nusenu commented Aug 22, 2021

nusenu commented Mar 24, 2018 •

edited

Loading

RedDog99 commented Mar 24, 2018 •

edited by nusenu

Loading

nusenu commented Mar 24, 2018 •

edited

Loading

nusenu commented Mar 24, 2018 •

edited

Loading

nusenu commented Mar 24, 2018 •

edited

Loading