Got various errors running in docker

[ofirule]

ver: dns-cache (0.2.0)

uname -a
Linux tegra-ubuntu 4.4.38-tegra #8 SMP PREEMPT Fri Nov 9 13:01:46 CST 2018 aarch64 aarch64 aarch64 GNU/Linux

Docker host and instance are both ubuntu 16.04, on aarch64
The instance is running with flag --dns=8.8.8.8

So with dnspython (2.0.0) this is the given err:

Traceback (most recent call last):
  File "watchdog.py", line 13, in <module>
    import dns_cache
  File "/usr/local/lib/python3.5/dist-packages/dns_cache/__init__.py", line 6, in <module>
    from dns.resolver import override_system_resolver as upstream_override_system_resolver
  File "/usr/local/lib/python3.5/dist-packages/dns/resolver.py", line 32, in <module>
    import dns.flags
  File "/usr/local/lib/python3.5/dist-packages/dns/flags.py", line 24, in <module>
    class Flag(enum.IntFlag):
AttributeError: module 'enum' has no attribute 'IntFlag'

and with dnspython (1.16.0) this is the given err:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/dns/resolver.py", line 1187, in _getaddrinfo
    addr = dns.ipv6.inet_aton(ahost)
  File "/usr/local/lib/python3.5/dist-packages/dns/ipv6.py", line 155, in inet_aton
    raise dns.exception.SyntaxError
dns.exception.SyntaxError: Text input is malformed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/dns/resolver.py", line 1193, in _getaddrinfo
    addr = dns.ipv4.inet_aton(host)
  File "/usr/local/lib/python3.5/dist-packages/dns/ipv4.py", line 52, in inet_aton
    raise dns.exception.SyntaxError
dns.exception.SyntaxError: Text input is malformed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/dns/resolver.py", line 1211, in _getaddrinfo
    raise_on_no_answer=False)
  File "/usr/local/lib/python3.5/dist-packages/dns_cache/resolver.py", line 45, in query
    qname, rdtype, rdclass, **kwargs
  File "/usr/local/lib/python3.5/dist-packages/dns_cache/resolver.py", line 86, in query
    qname, rdtype, rdclass, **kwargs
  File "/usr/local/lib/python3.5/dist-packages/dns/resolver.py", line 1002, in query
    raise NXDOMAIN(qnames=qnames_to_try, responses=nxdomain_responses)
dns.resolver.NXDOMAIN: The DNS query name does not exist: localhost.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "watchdog.py", line 530, in <module>
    watchdog = Watchdog()
  File "watchdog.py", line 499, in __init__
    self.netdata_metric_handler = NetdataMetricHandler(enabled=True, logger=self.logger)
  File "/opt/infrastructure/common/netdata/netdata_metric_handler.py", line 11, in __init__
    self.statsd_client = statsd.StatsClient(prefix=METRICS_PREFIX)
  File "/usr/local/lib/python3.5/dist-packages/statsd/client/udp.py", line 35, in __init__
    host, port, fam, socket.SOCK_DGRAM)[0]
  File "/usr/local/lib/python3.5/dist-packages/dns/resolver.py", line 1218, in _getaddrinfo
    raise socket.gaierror(socket.EAI_NONAME)
socket.gaierror: -2

[jayvdb]

The error from dnspython 1.16.0 is

dns.resolver.NXDOMAIN: The DNS query name does not exist: localhost.

That looks like a "config" problem, which could be on your side, or also could be in this package. Either way, we can very likely find a workaround for this.

I did do some testing on dnspython 2 before it was released, and submitted some issues I found. I see it has come out only supporting py36+
https://github.com/rthalley/dnspython/blob/master/setup.py#L68

If you really need py35, I can probably investigate it, and maybe get it working for you, but if you are using Docker then maybe you can use py36.

[ofirule]

It worked on my pc. host ubuntu16.04 instance ubuntu18.04 with both dnspython versions with python36, I suspect that the issue is coming from either the python version or the docker version which is quiet old on my aarch64 devices.

I will try to go back to this issue in the following weeks

[ofirule]

Update:
OS ubuntu 16.04, on aarch64
docker OS : ubuntu 18.04,
python ver 3.6

I got the following error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/dns/resolver.py", line 1347, in _getaddrinfo
    raise_on_no_answer=False)
  File "/usr/local/lib/python3.6/dist-packages/dns/resolver.py", line 1030, in resolve
    (request, answer) = resolution.next_request()
  File "/usr/local/lib/python3.6/dist-packages/dns/resolver.py", line 585, in next_request
    responses=self.nxdomain_responses)
dns.resolver.NXDOMAIN: The DNS query name does not exist: localhost.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "watchdog.py", line 529, in <module>
    watchdog = Watchdog()
  File "watchdog.py", line 498, in __init__
    self.netdata_metric_handler = NetdataMetricHandler(enabled=True, logger=self.logger)
  File "/opt/infrastructure/common/netdata/netdata_metric_handler.py", line 11, in __init__
    self.statsd_client = statsd.StatsClient(prefix=METRICS_PREFIX)
  File "/usr/local/lib/python3.6/dist-packages/statsd/client/udp.py", line 35, in __init__
    host, port, fam, socket.SOCK_DGRAM)[0]
  File "/usr/local/lib/python3.6/dist-packages/dns/resolver.py", line 1354, in _getaddrinfo
    raise socket.gaierror(socket.EAI_NONAME, 'Name or service not known')
socket.gaierror: [Errno -2] Name or service not known

Using both dnspython (1.16.0) dnspython (2.0.0) even after removing all docker run Flags.
So I need a fix and/or a workaround for aarch64

Maybe it's related to the fact that dnspython doesn't use "/etc/hosts" (but it does not reproduce on x86_64 so I'm not sure)

[jayvdb]

I've pushed dnspython 2.0 support onto master, if you want to test with that. The CI failures are primarily to do with dnspython 2 doing a slightly better job of caching than dnspython 1, so my tests are expecting the wrong counts at various stages of the tests on dnspython 2. I have fixes for these test failures, but I am working on better tests for the aggressive caching class, and that is all intermingled so it will be about another day before I push that and get the tree green again, and then do a release of dns-cache.

It would not surprise me that there was some small but significant bugs in dnspthon+dns-cache on aarch64 - I have seen socket.gaierror behave differently on different arch for same kernel of Linux . I have access to aarch64, so I will run my own test suite against that arch as soon as the tree is green, and if that doesnt flush out the bug we can try to build extra tests which better isolate your case (something to do with https://github.com/netdata/netdata I assume)

Note master only supports Resolver.query in dnspython 2 . The new Resolver.resolve isnt supported yet for the custom dns-cache resolvers, but it is coming.

[jayvdb]

@ofirule , master has another update, especially adding .resolve() and lots of tests , but few other functional changes. Change log started at https://github.com/jayvdb/dns-cache/blob/master/CHANGELOG.md

[ofirule]

Error is almost the same after using dns-cache from master

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/dns/resolver.py", line 1347, in _getaddrinfo
    raise_on_no_answer=False)
  File "/usr/local/lib/python3.6/dist-packages/dns/resolver.py", line 1030, in resolve
    (request, answer) = resolution.next_request()
  File "/usr/local/lib/python3.6/dist-packages/dns/resolver.py", line 585, in next_request
    responses=self.nxdomain_responses)
dns.resolver.NXDOMAIN: The DNS query name does not exist: localhost.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "watchdog.py", line 529, in <module>
    watchdog = Watchdog()
  File "watchdog.py", line 498, in __init__
    self.netdata_metric_handler = NetdataMetricHandler(enabled=True, logger=self.logger)
  File "/opt/infrastructure/common/netdata/netdata_metric_handler.py", line 11, in __init__
    self.statsd_client = statsd.StatsClient(prefix=METRICS_PREFIX)
  File "/usr/local/lib/python3.6/dist-packages/statsd/client/udp.py", line 35, in __init__
    host, port, fam, socket.SOCK_DGRAM)[0]
  File "/usr/local/lib/python3.6/dist-packages/dns/resolver.py", line 1354, in _getaddrinfo
    raise socket.gaierror(socket.EAI_NONAME, 'Name or service not known')
socket.gaierror: [Errno -2] Name or service not known

Note: that your getting started example is working in my docker container, the problem is only with localhost

[ofirule]

Added easy tests:

test1.py:

import socket
print(socket.gethostbyname('google.com'))
print(socket.gethostbyname('localhost'))

test2.py:

import dns_cache
import socket
dns_cache.override_system_resolver()

print(socket.gethostbyname('google.com'))
print(socket.gethostbyname('localhost'))

results when running dns-cache from master with dnsython==2.0.0 :

# python3 test1.py 
172.217.18.46
127.0.0.1
# python3 test2.py 
172.217.18.46
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/dns/resolver.py", line 1347, in _getaddrinfo
    raise_on_no_answer=False)
  File "/usr/local/lib/python3.6/dist-packages/dns_cache/resolver.py", line 52, in resolve
    raise_on_no_answer, source_port, lifetime,
  File "/usr/local/lib/python3.6/dist-packages/dns_cache/resolver.py", line 145, in resolve
    raise_on_no_answer, source_port, lifetime,
  File "/usr/local/lib/python3.6/dist-packages/dns/resolver.py", line 1030, in resolve
    (request, answer) = resolution.next_request()
  File "/usr/local/lib/python3.6/dist-packages/dns/resolver.py", line 585, in next_request
    responses=self.nxdomain_responses)
dns.resolver.NXDOMAIN: The DNS query name does not exist: localhost.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "b.py", line 6, in <module>
    print(socket.gethostbyname('localhost'))
  File "/usr/local/lib/python3.6/dist-packages/dns/resolver.py", line 1457, in _gethostbyname
    return _gethostbyname_ex(name)[2][0]
  File "/usr/local/lib/python3.6/dist-packages/dns/resolver.py", line 1464, in _gethostbyname_ex
    socket.SOL_TCP, socket.AI_CANONNAME)
  File "/usr/local/lib/python3.6/dist-packages/dns/resolver.py", line 1354, in _getaddrinfo
    raise socket.gaierror(socket.EAI_NONAME, 'Name or service not known')
socket.gaierror: [Errno -2] Name or service not known

[jayvdb]

Ah, interesting. This is also a 'bug' in dnspython, but I will fix it here.
(gonna sleep on where this feature fits in)

[jayvdb]

@ofirule , branch localhost has a WIP fix, but you also need to install Python packages python-hosts and lazyutils to activate it (I'm still looking at possible alternatives). It needs some cleanup, and may have some bugs as there are only a few tests. I'll get it tidied up in the next day and pushed to master.

[jayvdb]

A bit more cleaned up, localhost2 which installs different deps, and does it automatically.

[ofirule]

I will try it in the following weeks

[ofirule]

Both branches localhost and localhost2 fixed the errors on my device (with localhost requiring to install python-hosts and lazyutils).

This is great!

Please add the fixes to dnspython for ver 2.0.0 and hosts issue to a new release, and I will definitely use it

[jayvdb]

Good to hear. I have tidied up localhost2 some more, and will get it onto master in the next day, and released soon after. I have one other feature I'd like to squeeze in, but if you need to be using PyPI, I can defer that and get it onto PyPI asap.

[ofirule]

Is it possible to create a new release with the localhost fix soon?

[jayvdb]

Yup; I am working on it right now ;-)

[jayvdb]

v0.3.0 tagged and released.

[jayvdb]

Assuming all is well, so closing. Let me know if you encounter any further trouble.

[jayvdb]

Noting the dnspython upstream project decided not to support /etc/hosts/. rthalley/dnspython#149